I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Looking closely, we see that there are a number of very visible "spikes" on the plot. These are particular emission lines associated with atomic transitions. These specific spectral features are a direct result of the energetic photons emitted from all the new/young stars in the galaxy. Try and zoom in on the particular region on the plot where most of these are located. Can you identify any lines in particular based on this list? Some of the most common ones are also defined below.
Extra challenge: Overplot the line wavelengths on the galaxy spectrum using the plt.vlines function.
Can you write Python code for it?

# defining some common emission lines
ha = 6564.6  # H-alpha [A]
n2 = 6549.86  # NII [A]
o3_1 = 5008.240  # OIII doublet (1) [A]
o3_2 = 4960.295  # OIII doublet (2) [A]
hb = 4862.7  # H-beta [A]
o2 = 3728.4  # approximate center of (blended) OII doublet [A]

# consolidating results in an array (in order of decreasing wavelength)
emlines = np.array([ha, n2, o3_1, o3_2, hb, o2])
emline_names = np.array(['Ha', 'NII', 'OIII (1)', 'OIII (2)', 'Hb', 'OII (1,2)'])
Nlines = len(emlines)


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Now that we have all these bits and pieces, see if you can plot the average photon energy in each filter as a function of wavelength. A short example is shown below using plt.errorbar along with a short printout summary that utilizes the round function, but you're welcome to be as creative as you like!
Can you write Python code for it?

# figure
plt.figure(figsize=(16, 6))
xlow, xhigh = filt_bounds.T
xe_low, xe_high = filt_cent - xlow, xhigh - filt_cent
plt.errorbar(filt_cent, ephot_cent, xerr=[xe_low, xe_high], linestyle='none', marker='o')

# printout
print('Filter', 'Center[A]', 'Width({0})[A]'.format(fbound), 'Low[A]', 'High[A]', 'E_phot[1e-12 erg]')
for i in xrange(Nfilt):
    print(filt_names[i], round(filt_cent[i], 1), filt_width[i], filt_bounds[i][0],
          filt_bounds[i][1], round(ephot_cent[i] * 1e12, 2))


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Now that we've changed the defaults, we notice a number of issues with our plot to do with our font size. This is because all outputs from pyplot are intrinsically drawn on a Figure object. If one of these are not initialized explicitly at the beginning, a default one is created. With our new larger fonts, the default figure feels a little squished.
Play around with changing the size of our figure using the commented line. Feel free to also mess around with the arguments passed to the axes labels and titles.
Galaxies and Emission Lines
Now that we have some familiarity with plotting, let's move on to a bit more of the science. In the seds folder under data, there are a bunch of files containing galaxy spectral energy distributions (SEDs). Let's load in UGCA 166, a nearby blue compact dwarf galaxy that's actively forming stars.
Can you write Python code for it?

# load in text data
data = np.loadtxt('seds/brown_UGCA_166_spec.dat')
data


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Our previous galaxy file was just a bunch of numbers, which we were able to load in using np.loadtxt. Our list of filters, however, is a bunch of names. This means we need to read in the file a bit differently. Let's do this line by line just to be very explicit about it.
Can you write Python code for it?

# initialize lists
filt_names = []  # empty list
filt_files = []  # empty list

# read in our filter names and file paths line-by-line
filt_list = open(filt_path+'Euclid.list') # open list of filter files for Euclid+LSST
for line in filt_list:
    l = line.strip()  # strip out the end-carriage '\n' if present
    ls = l.split()  # split the line into component strings
    filt_names.append(ls[0]) # filter name
    filt_files.append(ls[1]) # filter file
filt_list.close()  # close file (**ALWAYS REMEMBER TO DO THIS**)
Nfilt = len(filt_files)  # number of filters


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Photometry
We now want to compute some basic properties of our set of filters. We'll start with the effective wavelength. This can be seen as the approximate "mean" wavelength of the filter, accounting for the differing transmission as a function of wavelength. Defining our wavelengths as $\lambda$, our frequencies as $\nu$, and our transmission at a particular frequency as $T_\nu$, the "standard" defition for this effective wavelength $\lambda_{\textrm{eff}}$:
$$ \lambda_{\textrm{eff}} = \exp \left[ \frac{\int T_\nu \, \ln \lambda \, d(\ln \nu)}{\int T_\nu \, d(\ln \nu)} \right] \quad . $$
Although this definition might seem a bit weird, the basic idea is we want to compromise between averaging as a function of wavelength compared to as a function of frequency, which don't give the same result since $ \lambda = c / \nu \propto \nu^{-1}$.
We can break down this computation into four steps:
1. compute the integral in the denominator,
2. compute the integral in the numerator,
3. exponentiate their ratio, and
4. iterate over all of our filters.
Python has a bunch of numerical integration packages available as part of scipy for more general applications. There also is a basic numerical integration tool trapz as part of numpy, which should suffice for our purposes here. An example is shown below.
Can you write Python code for it?

# integrate our function from the beginning of the notebook
plt.plot(x, y)  # plot our original function
plt.fill_between(x, y, color='blue', alpha=0.3)  # fill between y and 0 over our x's
y_area = np.trapz(y, x)  # numerically integrate our function

print('Integral = {0}'.format(y_area))


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Play around with the components above to get a sense of what we just did. What happens if you remove parts or don't close the file? What operations can you do with lists? How about with arrays? Some quick examples are below. We'll be coming back to some of these subtleties later, but please don't hesitate to ask if you have any questions.
Can you write Python code for it?

#print(filt_names, filt_names * 2)
#print(emlines, emlines * 2)
#print(filt_list)


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Play around with the parameters above to get some more familiarity with plotting. If you have time, see if you can:
- Change the original relationship to a quadratic one.
- Change the type of random noise we are adding to the data.
- Change the colors used for plotting.
- Change the "linestyle" used for plotting from connected lines to unconnected dots.
- Change the x and y limits in the plot.
and any other changes you'd like to experiment with.
Feel free to use any resources you want to figure this out. For immediate results, try the help function (shown below) or Shift-Tab within a function for in-line documentation. There's also some official documentation online.
Can you write Python code for it?


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Plotting
Before we get started, we need to initialize the environment. First, let's import pyplot from matplotlib, which we will use to plot things. Then, we can use one of the "magic commands" to enable in-line plotting, which ensures our plots will show up in our actual notebook (rather than externally).
Can you write Python code for it?

# import plotting utility and define our naming alias
from matplotlib import pyplot as plt

# plot figures within the notebook rather than externally
%matplotlib inline


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Spend some time breaking down the above snippet of code so that you can (ideally) explicitly summarize what exactly each line is doing and why. One way I like to make sense of unfamiliar code is by pulling it apart and plotting some of the intermediate results. An example of this is shown below.
Can you write Python code for it?

i = 3
cdf = np.cumsum(ft[i])
cdf /= cdf[-1]
fremain = (1 - fbound) / 2.

plt.figure(figsize=(16, 4))
plt.plot(fw[i], ft[i] / max(ft[i]), color='black')  # normalize to 1.
plt.plot(fw[i], cdf, color='blue')  # normalize to 1.

temp = np.abs(cdf - fremain)
idx_left = np.argmin(temp)
plt.plot(fw[i], temp, color='red', linestyle='--')  # function we will minimize
plt.vlines(fw[i][idx_left], 0., 1., color='red')

temp = np.abs(cdf - (1. - fremain))
idx_right = np.argmin(temp)
plt.plot(fw[i], temp, color='red', linestyle='--')  # function we will minimize
plt.vlines(fw[i][idx_right], 0., 1., color='red')

plt.xlabel('Wavelength')
plt.ylabel('Fraction')


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
The data we loaded in is a $N \times 2$-dimensional numpy array, where the first column is the wavelength (measured in Angstroms, $10^{-10}$ meters) while the second column is the relative flux density (energy per time per area per wavelength, i.e. the relative intensity of the light at that particular wavelength).
Arrays are fixed-size data structures in Python that allow you to quickly manipulate lots of numbers. We'll be exploiting them here and you'll definitely be using them if you code in Python more regularly.
The way data is currently structured is a bit awkward: the relevant quantities are the columns here, which makes plotting a bit awkward. One possible way to get around this is to make some new variables by (1) iterating through the array (using an implicit for loop) or (2) slicing through the array. Both of these are deomnstrated below.
Can you write Python code for it?

# iteration through the array using an implicit for loop (list comprehension)
print('Wavelength:', np.array([d[0] for d in data]))

# slice the array along the 0th entry in the 1st dimension (i.e. by column, not row)
print('Wavelength:', data[:, 0])


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
The set of commands above probably didn't work on the first try. What gives? Looking at the error, it's telling us that the name 'rcParams' is not defined in any capacity. This makes some sense: we never defined this variable anywhere. See if you can find out where to import it.
Let's re-plot our results to see what updating our defaults has changed.
Can you write Python code for it?

# plot our results
plt.figure(figsize=(10, 4))
plt.plot(x, yobs)
plt.plot(x, y)
plt.xlabel('x', fontsize=20, color='darkviolet')
plt.ylabel('y', fontsize=40, color='red')
plt.title('A (Noisy) Line with Larger Font', y=1.05, color='navy')


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
To facilitate plotting and later manipulation, let's redefine our data to instead be 2 $N$-dimensional arrays called wave and fgal_wave. There are a bunch of ways to do this besides the two shown above, but the most direct way is to use numpy's built-in array manipulation functions.
Extra challenge: can you code up your own custom method to do this? How many lines of code does your method take compared with the one line implementations shown above?
Can you write Python code for it?

# take the transpose of the array (N x M) -> (M x N)
print(data.T)
wave, fgal_wave = data.T


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Try plotting some of these below building off of the basic layout. The basic Figure setup has been initialized along with two possible color schemes and a bunch of keyword arguments to the methods. Try to play around with the styles to see how they change things.
Can you write Python code for it?

# initialize figure
plt.figure(figsize=(16, 5))

# define a sequence of colors
colors = ['blue', 'magenta', 'red', 'orange', 'brown',  # this works because python implicitly
          'green', 'teal', 'goldenrod', 'coral', 'black']  # continues bracketed statements

# define our colormap (see: https://matplotlib.org/examples/color/colormaps_reference.html)
color_scale = np.linspace(0, 1, Nfilt)
colors = plt.get_cmap('viridis')(color_scale)

# plot results
for i in range(Nfilt):
    plt.plot(fw[i], ft[i], ls='-', lw=2, label=filt_names[i], color=colors[i])
plt.ylim([0, 1.5])
plt.legend(loc=1, ncol=5, fontsize=16)
plt.tight_layout()


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Using the example above, see if you can compute (1) the denominator, (2) numerator, and (3) effective wavelength for a particular filter in the style shown below. Then see if you can turn this into an array of effective wavelengths over all filters.
Can you write Python code for it?

# denominator
denominator = ...

# numerator
numerator = ...

# effective wavelength (Schneider et al. 1983; Fukugita et al. 1996)
filt_cent = ...


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Using this, let's redo our integral from above. I've provided a one-statement solution below, but I would highly encourage everyone to try their hand at implementing something themselves.
Can you write Python code for it?

# compute relative photometry
phot = np.array([np.trapz(np.interp(wave, fw[i], ft[i], left=0., right=0.) * fgal_nu, np.log(freq)) /
                 np.trapz(np.interp(wave, fw[i], ft[i], left=0., right=0.), np.log(freq))
                 for i in range(Nfilt)])

print(phot)


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Why this conversion between $S_\nu$ and $S_\lambda$? A mix between historical reasons and plotting usefulness (sometimes it's easier to visualize trends as per unit wavelength instead of per unit frequency, and vice versa). Plotting things in $S_\nu$ tends to highlight behavior in the optical range (which becomes "flatter" the more star formation a galaxy tends to have).
Using everything we've covered up to this point, we're now ready to simulate our galaxy observation. First, let's integrate our galaxy over the filters to get the relative photometric flux density $F_\nu$.
Can you write Python code for it?

freq = c / wave
phot = np.array([np.trapz(ft[i] * fgal_nu, np.log(freq)) /
                 np.trapz(ft[i], np.log(freq))
                 for i in range(Nfilt)])


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
A solution is given below.
Can you write Python code for it?

# one possible solution using boolean algebra (0=False, 1=True)

# slit dimensions [arcsec]
slit_width = 0.8
slit_height = 7.

# slit model
model_slit = (abs(x) <= slit_width / 2.)  # set all x's within slit_width / 2. of 0. to 1; otherwise set to 0
model_slit *= (abs(y) <= slit_height / 2.)  # *also* set all y's within slit_height / 2. to 1; otherwise set to 0

# post-slit observed galaxy model
model_spec = model_slit * model_obs

# plot results (basic)
plt.figure(figsize=(14, 6))  # create figure object
plt.subplot(1, 2, 1)  # split the figure into a grid with 1 row and 2 columns; pick subplot 1
plt.imshow(model_slit.T)  # plot slitmask model
plt.colorbar(label='Transmission')
plt.subplot(1, 2, 2)  #  pick subplot 2
plt.imshow(model_spec.T)  # plot combined galaxy+slit model
plt.colorbar(label='Intensity')
plt.tight_layout()


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
As with many problems that come up often but are tedious to implement by hand, Python has a function to do this! The solution is shown below.
Can you write Python code for it?

# mesh (x_grid, y_grid) into a new set of 2-D (x, y) arrays
x, y = np.meshgrid(x_grid, y_grid) # x,y for our 2-D grid
print(x, x.shape)
print(y, y.shape)


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Check that these give the same result using a grid of radii. Feel free to use np.linspace, np.arange, or any other tool to generate the set of radii for testing.
Can you write Python code for it?

re = 1.  # effective radius
radius = ...  # radii
gal1 = ...  # function 1 (def)
gal2 = ...  # function 2 (lambda)

# numerical checks

# plot results


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Compare our "convolved" image with the original galaxy model and the original PSF. This "smeared out" image now represents the galaxy we'd actually observe through the atmosphere from our ground-based telescope!
Slitmask
Let's imagine we're interested in taking a spectrum of this galaxy. Typically, this works by putting a slitmask on the telescope. These are typically made of metal, and are designed to blocks out everything except for the small amount of light passing through slits that have been drilled in the mask. The reason this has to be done is that the light from each slit is going to be spread out on the physical detector as a function of wavelength, so that blue light (i.e. shorter wavelengths) on one end and red light (i.e. longer wavelengths) are on the other. A general picture of this is shown below.

Using the slit dimensions specified below, see if you can compute (and plot) the galaxy image that would be seen by our telescope through the slitmask. I've included a solution below, but see if you can come up with your own version before moving on.
Extra Challenge: Try and compute the "slit loss" (the fraction of light that is not blocked by the slit). This tells us the overall "efficiency" of our observation so far.
Can you write Python code for it?

# slit dimensions [arcsec]
slit_width = 0.8
slit_height = 7.

# post-slit galaxy model
model_spec =

# plotting the results
...


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Galaxy Model
The model for the spatial intensity of the galaxy we observe (i.e. the distribution of brightness on the sky) has two basic components:
1. An intrinsic galaxy model, which governs the distribution of brightness away from the center of the galaxy. This needs to depend on the expected angular size of our galaxy, which determines how large an object is on the sky.
2. A model of the observed point-spread function (PSF), which governs how much a "point" on the sky is smeared out into a blob by the atmosphere/instrument. This needs to depend on the expected seeing (i.e. the typical size of a PSF blob).
Let's initialize each of these in turn using Python's built-in ability to store/define functions.
Our galaxy intensity will be a radially-symmetric exponential profile defined as
$$ I_{\textrm{gal}} = I_0 \exp \left(- a \frac{r}{r_e} \right) $$
where $I_0$ is a normalization factor (which we will take to be 1 here), $a = 1.68$ is a constant that governs how quickly light "falls off" towards the outskirts, and $r_e$ is the effective radius of the galaxy.
Can you write Python code for it?

# Galaxy intensity model: Exponential
def prof_expo(r, re):
    a = 1.68
    return np.exp(-a * r / re)


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Let's bin this using np.histogram to see what a histogram output looks like.
Can you write Python code for it?

counts, bin_edges = np.histogram(radius, weights=gal1, normed=True)
print(counts)
print(bin_edges)


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Let's now visually check the results.
Can you write Python code for it?


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Observed Galaxy Model
As mentioned above, our final observed image is a convolution of our galaxy model and our PSF model. Convolutions can be tricky to deal with. Naively, we could imagine doing a convolution as follows:
1. Compute the standard PSF.
2. Turn every point in our 2-D r(x, y) grid into a PSF, making sure we keep the same overall amplitude $I(r(x,y))$ for each PSF.
3. Add up all the PSFs together.
This type of thinking represents the type of thinking that is useful when coding: breaking down each part of a larger problem into a bite-size chunk that is easier to code up before combining things together at the end. (Figuring out where loops are needed versus where array operations can be done instead is also incredibly useful.)
In addition to this discrete, "computing"-oriented way of thinking about the problem, convolutions also lend themselves naturally to a more theoretical/math-y approach (i.e. we're smearing out an infinite collection of points from one continuous function using another continuous function). From that perspective, it turns out there are some really neat tricks we can do to actually compute more "exact" convolutions even with a small set of data points.
As with most applications that are relatively common, Python has a bunch of functions/packages useful for convolutions! We'll use a specific version fftconvolve (which uses fast Fourier transforms) from scipy.signal (a set of functions in scipy useful for signal processing) to do our convolution.
Import fftconvolve from scipy.signal, use it to compute the observed galaxy model, and plot the results.
Note: make sure to compute the convolution using mode='same', otherwise you won't be able to plot your results! Check the documentation for additional details.
Can you write Python code for it?

# compute convolution of galaxy and PSF
model_obs = fftconvolve()
model_obs /= np.max(model_obs)  # normalize result to 1.

# plot our result
...


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Pixelation
We're almost done simulating our galaxy. The last step is to account for the pixel scale of our observation. Telescopes don't have infinite resolving power, but have a minimum resolution determined by the size of the pixels on the charge-coupled device (CCD). This usually is expressed in units of arcseconds per pixel. Since our simulated observation above likely has a much higher resolution than the typical resolving scale, we need to bin it to lower resolution.
numpy has a few functions to accomplish this. The most relevant for our purposes are histogram as histogram2d, which compute the histogram of input data. There are also some nifty shortcuts for plotting these using the plt.hist and plt.hist2d functions.
As a warm up, let's first bin a set of normally distributed random numbers just to get familiar with the function.
Can you write Python code for it?

rand = np.random.normal(loc=0., scale=1., size=100000)  # normally distributed random numbers
plt.hist(rand);  # plot a histogram


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Rather than trying to struggle with getting images to look good in plt.plot (or, alternately, plt.scatter), we're instead going to use plt.imshow (which is designed for this). Some examples are shown here.
Can you write Python code for it?

# plotting our galaxy profile
plt.figure()

# default plot
plt.imshow(model_gal)

# more detailed plot
#plt.imshow(model_gal.T,  # take the transpose to flip x and y in plot
#           origin='lower',  # specify the origin to be at the bottom not the top
#           extent=[x_grid[0], x_grid[-1], y_grid[0], y_grid[-1]], # specify [left, right, bottom, top] positions
#           cmap='magma', interpolation='none')  # additional options
#plt.xlabel('x [arcsec]')
#plt.ylabel('y [arcsec]')
#plt.title('Intrinsic Galaxy Profile')
#plt.colorbar(label='Intensity')  # add a colorbar


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
So with that in hand, let's now see what our grid looks like (thinned by 20). To help out with plotting, we're also going to "flatten" our arrays from 2-D to 1-D before plotting them.
Can you write Python code for it?

# thin grid by a factor of 20
x_temp, y_temp = x[::20, ::20].flatten(), y[::20, ::20].flatten()  # slicing/thinning/flattening
plt.figure()
plt.plot(x_temp, y_temp, '.', markersize=2)

# print array shape (2-D vs flattened)
print(x[::20, ::20].shape, x_temp.shape)


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Take a look at some of the additional arguments that can be passed to plt.hist. See if you can:
- adjust the number of bins,
- set the locations of the bins by hand,
- change the color  of the histogram,
- change the "style" (type) of histogram, and
- plot the histogram horizontally.
Extra Challenge: Ensure that each histogram is properly normalized (i.e. integrates to 1.), cumulative (each bin is the sum of all previous bins), has 100 evenly spaced bins from -5 to 5, and ignores any values outside of those boundaries.
Extra Extra Challenge: Plot 10 realizations of random numbers such that they "stack" on top of each other, each with a different color based on a pre-defined colormap (see the previous notebook for an example of how to define a colormap).
Now let's look at weighted histograms. Our original exponential profile has a series of radii (stored in radius) corrsponding to a set of intensities (gal1 and gal2). We want to bin this to lower resolution using plt.hist. To do this, we just bin up the radius to lower resolution, where we weight each point by the intensity.
Can you write Python code for it?

plt.plot(radius, gal1)  # plot original relation
plt.hist(radius, weights=gal1, normed=True);  # plot a histogram, normalized based on our weights


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
Using the bins/bin centers computed above:
1. Bin our model_spec observation to the MMIRS pixel scale using np.histogram2d.
2. Plot the result using plt.imshow.
Note that the code below should not work as is: not only are the variables placeholders (i.e. they're undefined), histogram2d also requires that all the input data is 1-D, which means we have to flatten the input arrays (if needed).
Can you write Python code for it?

# convert from arcsec to pixels
model_pix, x_edges, y_edges = np.histogram2d(Xs, Ys, weights=Model, bins=[x_bin, y_bin]) # bin over pixel scale

# plot results
plt.imshow(model_pix.T)


I am working on the file "Part 2/galaxy_spec.ipynb".
The first task is:
With all that done, let's now define the bins set by the pixel scale resolution of the Multiple Mirror Telescope (MMT) and Magellan Infrared Spectrograph (MMIRS) instrument. This is a spectrograph operated by a joint venture between the Smithsonian and the University of Arizona that you could one day use if you end up going to either institution!
Note that the code below uses a number of methods that you might not be familiar with. Take some time to see if you understand everything that's going on well enough that you could explain what the code below is doing to a friend.
Can you write Python code for it?

# the pixel scale of the MMIRS instrument
pix_scale = 0.2012  # [arcsec/pix]

# define our bins
x_bin = np.arange(x_grid[0], x_grid[-1] + pix_scale, pix_scale)  # bin edges in x with width=pix_scale
y_bin = np.arange(y_grid[0], y_grid[-1] + pix_scale, pix_scale)  # bin edges in y with width=pix_scale

# define the centers of each bin
x_cent, y_cent = 0.5 * (x_bin[1:] + x_bin[:-1]), 0.5 * (y_bin[1:] + y_bin[:-1])

# the total number of bins
Nx_pix, Ny_pix = len(x_cent), len(y_cent)

print(Nx_pix, Ny_pix)


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Finally, re-compute the number of goals needed to get to the finals for these new $\boldsymbol{\Theta}$ samples and plot the distribution of the results as a histogram using plt.hist with Nbins=50 bins.
Can you write Python code for it?

# compute goals needed
final_preds_resamp = ...

# plot histogram of results
# Remember to label your axes!
Nbins = 50
...


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Finding the Best-Fit Line
The best-fit line ($a$ and $b$) is the one that minimizes our loss function. To compute this, we need to locate the critical points where
$$ \frac{\partial f(\boldsymbol{\Theta})}{\partial \Theta_i} = 0 $$
for all parameters $\Theta_i$ of interest within $\boldsymbol{\Theta}$. In our case, we get:
$$ \frac{\partial L}{\partial \alpha} = \sum_{i=1}^{n} 2(\alpha + \beta x_i - y_i) = 0 $$
$$ \frac{\partial L}{\partial \beta} = \sum_{i=1}^{n} 2\beta(\alpha + \beta x_i - y_i) = 0 $$
This gives us two linear equations with two unknowns, which we can solve exactly using linear algebra to get an analytic best-fit solution $\hat{\alpha}$ and $\hat{\beta}$. You're welcome to try and solve this explicitly now; we'll come back to this later.
Can you write Python code for it?

# space if you want to try your hand at solving the linear system explicitly


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Now that our data is in an accessible format, let's try and get it into something we can do math with.  Copy over the data from our dictionary into numpy arrays called x and y. Then plot the results.
Can you write Python code for it?

# copy over our data into x and y
x = ...
y = ...

# plot our results
# remember to label your axes!
...


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Once you're done, plot the data as a 2-D histogram as before.
Can you write Python code for it?

# Plot draws as 2-D histogram.
# Remember to label your axes!
...


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Plot the weighted 2-D histogram of the fits and the weighted 1-D histogram of the goals needed to get to the finals. Note that meshgrid might be helpful here.
Can you write Python code for it?

Nbins = 20

# Plot draws as 2-D histogram.
# Remember to label your axes!
...

# compute goals needed
final_preds_grid = ...
final_preds_weights = ...

# plot histogram of results
# Remember to label your axes!
Nbins = 50
...


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Problem
Our starting goal here is going to be pretty simple: try to determine a simple linear relationship between the number of goals scored and total number of games played. In other words, we want a model like
$$ y = ax + b $$
where $y$ is the number of games played, $x$ is the number of goals scored, and $a$ and $b$ are coefficients of the fit. We are using the number of games played as a proxy for ranking.
First, we need to figure out how many goals were scored during the group stages for each team. We are going to compute the "effective" number of goals by taking the difference between the number of goals scored "for" the team $x_{\rm for}$ and the number scored "against" $x_{\rm against}$:
$$ x = x_{\rm for} - x_{\rm against} $$
Let's create a dictionary for this.
Can you write Python code for it?

data = dict()

# read out world cup 2014 data from the dictionary
for matchup in cup_data['rounds']:
    for match in matchup['matches']:
        team1, team2 = match['team1']['name'], match['team2']['name']  # team names
        score1, score2 = match['score1'], match['score2']  # scores
        score = score1 - score2  # effective score
        try:
            data[team1]['goals'] += score
            data[team2]['goals'] -= score
            data[team1]['games'] += 1
            data[team2]['games'] += 1
        except:
            data[team1] = {'goals': score, 'games': 1}
            data[team2] = {'goals': -score, 'games': 1}
        if matchup['name'] == 'Match for third place':
            data[team1]['games'] -= 1
            data[team2]['games'] -= 1

# Check data.
data


I am working on the file "Part 3/Introduction to Linear Regression.ipynb".
The first task is:
Test out a few lines and see what the loss function is. Combined with the plot above, see if you can get a reasonable fit to the data.
Can you write Python code for it?

# testing out the linear fit and loss function


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Brute Force
Alternately, we can just brute force the problem by computing $\chi^2$ for tons of possible parameter combinations $(\Omega_{m,0}, \Omega_{\Lambda,0})$, and then plot how our $\chi^2$ changes as $\Omega_{m,0}$ and $\Omega_{\Lambda,0}$ change.
Based on the code below, fill in each element of the chi2_arr with the corresponding $\chi^2$ value.
Can you write Python code for it?

# define our grid
Omega_m = arange(0, 1.01, 0.04)  # grid in Omega_m
Omega_L = arange(0, 1.01, 0.04)  # grid in Omega_L

# do our brute-force grid search
chi2_arr = np.zeros((len(Omega_m), len(Omega_L)))
for ...:  # loop over Omega_m
    for ...:  # loop over Omega_L
        # compute chi2
        chisquare = ...

        # fill  in array
        chi2_arr[i, j] = chisquare

        # Print progress.
        sys.stderr.write('\r Omega_m = {0}, Omega_L = {1}     '
                         .format(Om, OL))


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Just like with our galaxy example, we can convert each of our $\chi^2$ values to a corresponding weight that we assign each combination of parameters. This is computed via
$$ w(\chi^2) = e^{-\chi^2/2} $$
Using the formula above, convert the $\chi^2$ values we just computed to a set of weights $w$. Then, plot the resulting 2-D weighted array using plt.imshow based on the code below. Remember to set origin to the right location and extent to ensure you get the right bounds.
Can you write Python code for it?

# convert chi2 to weights
weights = ...

# plot results
# remember to label your axes!
plt.imshow(weights.T, origin=..., extent=...)


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Now, compute, plot, and label the luminosity distance $d_L(z; \Omega_{m,0}, \Omega_\Lambda)$ for different combinations of $(\Omega_{m,0}, \Omega_\Lambda)$:
1. $(0.3, 0.0)$,
2. $(0.3, 0.4)$,
3. $(0.3, 0.7)$,
4. $(0.5, 0.7)$,
5. $(0.8, 0.8)$.
Can you write Python code for it?

# compute dists for each combination (1->5)
...

# plot results
# remember to label your axes!
...


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Part 2: Inspecting Data
We now want to fit the luminosity distance and magnitude predictions to some Type Ia Supernovae data. This is stored under sne1a_data.txt and contains the redshifts, observed magnitudes, and magnitude errors from 50 (mock) supernovae observations.
Using whatever methods you prefer, read in the supernovae data. Then, plot:
1. a histogram of the redshift distribution of our sample with redshift bins of width $\Delta z = 0.2$ and
2. the observed magnitudes as a function of redshift.
Can you write Python code for it?

# read in data
z_spec, mag_spec, magerr_spec = ...

# plot histogram of redshifts
...

# plot magnitudes vs redshifts
...


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Part 4: Fitting the Data
We now want to fit the data. We will do this in two parts:
Minimizing $\chi^2$
First, we want to run an analysis similar to that from our previous linear regression exercise where we attempt to find the $(\Omega_{m,0}, \Omega_{\Lambda,0}$ values where our $\chi^2$ is smallest. I've defined a modified function below that matches the format needed to pass into scipy.optimize.minimize.
Can you write Python code for it?

# Define our modified function to match the format for `minimize`.
def calc_chi2(theta):
    Om, OL = theta
    dpred = d_L(z_spec, Omega_m=Om, Omega_L=OL)
    mpred = mag(dpred)
    chisquare = chi2(mpred, mag_spec, magerr_spec)

    return chisquare


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Plot a 2-D histogram of 10000 draws from the resulting Multivariate Normal distribution using numpy.random.multivariate_normal. Limit the x and y axes to be from 0 to 1.
Can you write Python code for it?

# plot 2-D histogram
# remember to label your axes!


I am working on the file "Part 4/Supernova Cosmology.ipynb".
The first task is:
Using scipy.optimize.minimize, find the best-fit combination of $(\Omega_{m,0}, \Omega_{\Lambda,0})$ using the loss function defined above starting from $(0.5, 0.5)$ and subject to the constraints $0 < \Omega_{m,0} < 1$ and $0 < \Omega_{\Lambda,0} < 1$.
Can you write Python code for it?

# Find the best fit.
x0 = ...  # initial guess
bounds = ...  # bounds for guesses (see documentation)
results = minimize(calc_chi2, x0, bounds=bounds)  # minimize!
theta_best = results['x']  # get best fit
theta_cov = results['hess_inv'].todense()  # get covariance matrix

# print results
print(theta_best)
print(theta_cov)


I am working on the file "Part-1-Data-Preprocessing/P1_Data_Preprocessing.ipynb".
The first task is:
But there is problem here, the mathmatical model may
see 2>1 instead of taking them as labels. So we are going to use
one hot encoder.
Can you write Python code for it?

from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder(categorical_features = [0])
# this means which column you are taking as having
# categories. Here 1st column i.e. Column 0

# Now our object is ready, lets replace

X = onehotencoder.fit_transform(X).toarray()

X


I am working on the file "Part-1-Data-Preprocessing/P1_Data_Preprocessing.ipynb".
The first task is:
Feature Scaling
Columns Age and Salary are not in same scale which may create problems in
ML models. Types- Standardisation and Normalisation
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
# for training set fit and transform, for test
# set only transform
X_test = sc_X.transform(X_test)

X_train

X_test


I am working on the file "Part-1-Data-Preprocessing/P1_Data_Preprocessing.ipynb".
The first task is:
Handle Categorical Data
As seen earlier Country and Purchased are categorical variables
Encode the categorical variables into numbers
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])

X


I am working on the file "Part-1-Data-Preprocessing/P1_Data_Preprocessing.ipynb".
The first task is:
Now we will encode the dependent variable. We dont need to hot encode this
Can you write Python code for it?

labelencoder_Y = LabelEncoder()
Y = labelencoder_Y.fit_transform(Y)

Y


I am working on the file "Part-1-Data-Preprocessing/P1_Data_Preprocessing.ipynb".
The first task is:
Split the dataset into train and test
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state = 0)
# You can use random_state=0 if you want the exactly same result as mine

X_train

X_test

Y_train

Y_test


I am working on the file "Part-2-Regression/P2-Multiple_Linear_Regression.ipynb".
The first task is:
Building the optimal model using Backward elimination
Previously to build the multiple regression model we used all the independent variable.
Out of these some independent variable are higly statistically significant and some are not
Can you write Python code for it?

import statsmodels.formula.api as sm

# The multiple linear regression equation is
# y = b0 + b1.X^1 + b2.x^2 + ... + bn.x^n
# The stats model requires that b0 is actually b0.x^0
# So, X^0 here is column vector of 1's
# we will add this vector of 1's to X
X = np.append(arr = np.ones((50, 1)).astype(int), values = X, axis = 1)
# append methode adds a new row or column
# np.ones adds a column of 1's if axis=1
# We want to keep column of 1's as first column so, np.ones is first argument and then append X to it
# X has 50 rows
# astype[int] required to convert the 1's to int otherwise
# you will get type error

X[1]

# Backward elimination steps
# Step 1: Select a significance level to stay in the model(e.g. SL = 0.05)
X_opt = X[:, [0,1,2,3,4,5]]

# Step 2: Fit the full model with all possible predictors
regressor_OLS = sm.OLS(endog = Y, exog = X_opt).fit()

# Step 3: Consider the predictor with the highest P-value.
# If P>SL, go to STEP4, otherwise go to Finish(model is final)
regressor_OLS.summary()

# P-value of X2 is highest, so we remove X2
# Step4: Remove the predictor whose p-value is highest and more than SL = 0.05
# here we have to remove index 2 column
# Step 5: Fit the model without this variable
X_opt = X[:, [0,1,3,4,5]]
regressor_OLS = sm.OLS(endog = Y, exog = X_opt).fit()
regressor_OLS.summary()

# here X1 has highest p-value. So we will remove column index
# 1
X_opt = X[:, [0,3,4,5]]
regressor_OLS = sm.OLS(endog = Y, exog = X_opt).fit()
regressor_OLS.summary()

# here X2 has highest p-value. So we will remove column index
# 4
X_opt = X[:, [0,3,5]]
regressor_OLS = sm.OLS(endog = Y, exog = X_opt).fit()
regressor_OLS.summary()

# here X2 has highest p-value. So we will remove column index
# 5
X_opt = X[:, [0,3]]
regressor_OLS = sm.OLS(endog = Y, exog = X_opt).fit()
regressor_OLS.summary()

X[1]

# So column R&D spent has the max impact on profit


I am working on the file "Part-2-Regression/P2-Multiple_Linear_Regression.ipynb".
The first task is:
Fitting multiple linear regression to the training set
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train,Y_train)

# Predicting the test set results
Y_pred = regressor.predict(X_test)


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Comparing our implementation with Surprise
Can you write Python code for it?

sim_options = {
    'name': 'MSD',
    'user_based' : False
}
algo = KNNWithMeans(sim_options = sim_options, k = 5, min_k =2)

perf = evaluate(algo, data2)


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Data Preparation/ Cleaning <br>
Removing ratings equal to zero, since Book Crossing Dataset has rating scale from 1-10. Taking Inner Join with books dataframe to maintain books whose details exist.
Can you write Python code for it?

newdf = ratings[ratings.book_rating>0]
joined = books.merge(newdf, on ='isbn')
print(newdf.shape)


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Function for Evaluation Metrics: MAE and RMSE
Can you write Python code for it?

def rmse(ypred, ytrue):
    ypred = ypred[ytrue.nonzero()].flatten()
    ytrue = ytrue[ytrue.nonzero()].flatten()
    return np.sqrt(mean_squared_error(ypred, ytrue))

def mae(ypred, ytrue):
    ypred = ypred[ytrue.nonzero()].flatten()
    ytrue = ytrue[ytrue.nonzero()].flatten()
    return mean_absolute_error(ypred, ytrue)


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Grid Search for best K for item-item CF using all the similarity metric implemented.
Can you write Python code for it?

sims = []
sims_rmse = []
for arg in ['euc','cos','','pr','adjcos']:
    each_sims = []
    each_sims_rmse = []
    for k in [2, 3, 4, 5, 10, 15, 20, 25]:
        print(arg, k)
        ans1, ans2  = get_results(nmat, arg, len(rows), len(cols), 5 ,k)
        each_sims.append(ans1)
        each_sims_rmse.append(ans2)

    print()
    print("Best K Value for ", arg)
    print()
    print("Min MAE")
    print(np.min(each_sims), np.argmin(each_sims))
    print("Min RMSE")
    print(np.min(each_sims_rmse), np.argmin(each_sims_rmse))
    print()
    sims.append(each_sims)
    sims_rmse.append(each_sims_rmse)

cos_res = sims[1]
euc_res = sims[0]
pr_res = sims[3]
adjcos_res = sims[4]
k = [2, 3, 4, 5, 10, 15, 20, 25]


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Loading the Book Ratings Dataset
Can you write Python code for it?

ratings = pd.read_csv('../raw-data/BX-Book-Ratings.csv', encoding='iso-8859-1', sep = ';')
ratings.columns = ['user_id', 'isbn', 'book_rating']

print(ratings.dtypes)
print()
print(ratings.head())
print()
print("Data Points :", ratings.shape[0])


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Loading the Books Dataset
Can you write Python code for it?

books = pd.read_csv('../raw-data/BX-Books.csv', sep=';', encoding = 'iso-8859-1', dtype =str)
del books['Image-URL-L']
del books['Image-URL-M']
del books['Image-URL-S']
del books['Book-Author']
del books['Publisher']


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Our Naive Baseline for any user i, item j prediction is to assign it with (sum of mean rating given by user i (umean[i]), mean rating received by item j (imean[j]) substracting average rating over entire dataset. (amean))

-------------- Naive Baseline ---------------
Can you write Python code for it?

def predict_naive(user, item):
    prediction = imean1[item] + umean1[user] - amean1
    return prediction

x1, x2 = train_test_split(nmat, test_size = 0.2, random_state =42)
naive = np.zeros((len(rows),len(cols)))
for row in x1:
    naive[row[0], row[1]] = row[2]

predictions = []
targets = []

amean1 = np.mean(naive[naive!=0])
umean1 = sum(naive.T) / sum((naive!=0).T)
imean1 = sum(naive) / sum((naive!=0))

umean1 = np.where(np.isnan(umean1), amean1, umean1)
imean1 = np.where(np.isnan(imean1), amean1, imean1)

print('Naive---')
for row in x2:
    user, item, actual = row[0], row[1], row[2]
    predictions.append(predict_naive(user, item))
    targets.append(actual)

print('rmse %.4f' % rmse(np.array(predictions), np.array(targets)))
print('mae %.4f' % mae(np.array(predictions), np.array(targets)))
print()


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Plot of MAE
Can you write Python code for it?

results_df1 = pd.DataFrame({'K': k, 'COS': cos_res, 'EUC': euc_res, 'Pearson': pr_res, 'Adjusted Cosine': adjcos_res})
plot1 = results_df1.plot(x='K', y=['COS', 'EUC', 'Pearson', 'Adjusted Cosine'], ylim=(0.95, 1.1), title = 'Item-Item CF: MAE for different similarity metrics at different Ks')
fig = plot1.get_figure()
fig.savefig('../figures/Kmae_item.png')


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Plot of RMSE
Can you write Python code for it?

cos_res = sims_rmse[1]
euc_res = sims_rmse[0]
pr_res = sims_rmse[3]
adjcos_res = sims_rmse[4]
k = [2, 3, 4, 5, 10, 15, 20, 25]
results_df1 = pd.DataFrame({'K': k, 'COS': cos_res, 'EUC': euc_res, 'Pearson': pr_res, 'Adjusted Cosine': adjcos_res})
plot1 = results_df1.plot(x='K', y=['COS', 'EUC', 'Pearson', 'Adjusted Cosine'], ylim=(1.5, 1.6), title = 'Item-Item CF: RMSE for different similarity metrics at different Ks')
fig = plot1.get_figure()
fig.savefig('../figures/Krmse_item.png')


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Plot of varying evaluation metrics vs number of latent factors for NMF
Can you write Python code for it?

results_df = pd.DataFrame({'Factors': factors, 'MAE': maelist, 'RMSE': rmselist})
plot1 = results_df.plot(x='Factors', y=['MAE', 'RMSE'], ylim=(0.9, 1.7), title = 'NMF: evaluation metrics vs number of latent factors')
fig = plot1.get_figure()
fig.savefig('../figures/NMFfactor.png')

from collections import defaultdict
def get_top_n(predictions, n=10):

    top_n = defaultdict(list)
    for uid, iid, true_r, est, _ in predictions:
        top_n[uid].append((iid, est))

    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]

    return top_n

trainset = data2.build_full_trainset()
algo = NMF(n_epochs = 60, n_factors = 50, reg_pu = 0.001, reg_qi = 3)
algo.train(trainset)

# Than predict ratings for all pairs (u, i) that are NOT in the training set.
testset = trainset.build_anti_testset()
predictions = algo.test(testset)

top_n = get_top_n(predictions, n=10)

# Print the recommended items for each user


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
RecBooks is used to recommend books to a user.
Can you write Python code for it?

def recbooks(mat, user_id, idict, cov = False):
    full_mat = np.zeros((len(rows),len(cols)))
    for row in mat:
        full_mat[row[0], row[1]] = row[2]

    nzero = full_mat[user_id].nonzero()[0]

    flipped_dict = dict(zip(idict.values(), idict.keys()))
    ans = [flipped_dict[i[0]] for i in top_n[user_id]]
    if not cov:
        print("Books Read -----")
        for i in nzero:
            print(flipped_dict[i])
        print()
        print("Recs -----")
        for i in ans:
            print(i)
    return ans

recbooks(nmat, 1,idict)


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
Some Books don't have unique ISBN, creating a 1:1 maping between books-title and ISBN
Can you write Python code for it?

print('Number of Books == Number of ISBN ? ', books["Book-Title"].nunique() == books["ISBN"].nunique())
book_dict = books[["Book-Title","ISBN"]].set_index("Book-Title").to_dict()["ISBN"]
books['new_isbn'] = books["Book-Title"].apply(lambda x: book_dict[x])
print('Number of Books == Number of ISBN ? ', books["Book-Title"].nunique() == books["new_isbn"].nunique())

books['isbn'] = books['new_isbn']

del books['ISBN']
del books['new_isbn']


I am working on the file "Part1/analysis/CF-Data.ipynb".
The first task is:
getmrec function is used to get top m recommendation for a user_id based on the similarity matrix (option), k neighbours.
Can you write Python code for it?

full_mat = np.zeros((len(rows),len(cols)))
for row in nmat:
    full_mat[row[0], row[1]] = row[2]
item_similarity, amean, umean, imean = itemsimilar(full_mat, 'euc')

def getmrec(full_mat, user_id, item_similarity, k, m, idict,  cov = False):

    n = item_similarity.shape[0]
    nzero = full_mat[user_id].nonzero()[0]

    preds = {}
    for row in range(n):
        preds[row] = predict(user_id, row, full_mat, item_similarity, amean, umean, imean, k)

    flipped_dict = dict(zip(idict.values(), idict.keys()))

    if not cov:
        print("Books Read -----")
        for i in nzero:
            print(flipped_dict[i])
            del preds[i]

    res = sorted(preds.items(), key=lambda x: x[1], reverse = True)

    ans = [flipped_dict[i[0]] for i in res[:m]]
    return ans

for m in [5, 8, 10, 15]:
    cov = []
    for i in range(len(rows)):
        cov.extend(getmrec(full_mat, i, item_similarity, 5, m, idict, True))
    print("Coverage with", m, "recs:", len(set(cov)), "%")

getmrec(full_mat, 313, item_similarity, 5, 10, idict)


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
... from delimited files
Can you write Python code for it?

import csv
csv.
with open('data/periods.csv', 'rt') as f:
    reader = csv.DictReader(f, delimiter=',')
    for row in reader:
        print (row)


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
... from the web
Use View Source in your browser to figure out where the age range is on the page, and what it looks like.
Try to find the same string here.
Can you write Python code for it?

url = "http://en.wikipedia.org/wiki/Cretaceous"

import requests
r = requests.get(url)
r.text[:2000]


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
*iteration, *iterable
List comprehension
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Checking the type of a variable
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Exercise:
a) write a function called make_curves that takes the data as input and return a dictionary of the curves. Keys are the curve name, values are the columns
b) write a function that converts the sonic log (us/ft) to velocity (m/s)
Can you write Python code for it?

# Acoustic impedance
vp = data['DT']
rhob = well['RHOB']
acimp = vp * rhob


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Exercise: Make a function to get the start and end ages of any geologic period, taking the name of the period as an argument.
Can you write Python code for it?

def get_age(period):
    url = "http://en.wikipedia.org/wiki/" + period
    r = requests.get(url)
    start, end = re.search(r'<i>([\.0-9]+)–([\.0-9]+) million years ago</i>', r.text).groups()
    return float(start), float(end)

period = "Cretaceous"
get_age(period)

def duration(period):
    t0, t1 = get_age(period)
    duration = t0 - t1
    response = "According to Wikipedia, the {0} period was {1:.2f} Ma long. ".format(period, duration)
    return response

duration('Cretaceous')


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Exercise: what is the expected output of:

a) age_intervals[:2]

b) age_intervals[6]

c) what command would you type to return the age of the end of the Permian, 248?

d) the start of the Cretaceous is wrong (it should be 144). Change it to the correct value

e) We've lost the dates for the Quaternary Period [1.8 mya to present (0)]. Index into that entry, and append it.
Can you write Python code for it?

# your code here

ages = {'Cambrian': {"Abbreviation": "C", "Start": "544", "End": "495"},
        'Ordivician': {"Abbreviation": "O", "Start": "495", "End": "492"},
        'Silurian': {"Abbreviation": "S", "Start": "442", "End": "416"},
        'Devonian': {"Abbreviation": "D", "Start": "416", "End": "354"},
        'Mississipian': {"Abbreviation": "M", "Start": "354", "End": "324"},
        'Pennsylvanian': {"Abbreviation": "IP", "Start": "324", "End": "295"},
        'Permian': {"Abbreviation": "P", "Start": "304", "End": "248"},
        'Triassic': {"Abbreviation": "Tr", "Start": "248", "End": "205"},
        'Jurassic': {"Abbreviation": "J", "Start": "205", "End": "144"},
        'Cretaceous': {"Abbreviation": "C", "Start": "144", "End": "65"},
        'Tertiary': {"Abbreviation": "T", "Start": "65", "End": "1.8"},
        'Quaternary': {"Abbreviation": "Q", "Start": "1.8", "End": "0"}
}


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Exercise: write a function called process_tops that takes a
filename as input and return a dictionary of the tops
Can you write Python code for it?

topsfile = 'data/L30_tops.txt'

def process_tops(file):
    """
    Takes a file as input and returns a dictionary of tops
    f : a filename path
    """
    # Enter code here
    return my_tops

# process_tops(topsfile)

import numpy as np
import matplotlib.pyplot as plt

% matplotlib inline

topo = np.load('data/topography.npy')
plt.imshow(topo)
plt.show()


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
External python languges
The Python Package Index, PyPI

SciPy -  a collection of often-used libraries

Using external libraries
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Getting started with bruges
Writing and running programs
Objects and Classes
Can you write Python code for it?

class Layers(object):

    def __init__(self, layers, label=None):
        # Just make sure we end up with an array
        self.layers = np.array(layers)
        self.label = label or "My log"
        self.length = self.layers.size  # But storing len in an attribute is unexpected...

    def __len__(self):  # ...better to do this.
        return len(self.layers)

    def rcs(self):
        uppers = self.layers[:-1]
        lowers = self.layers[1:]
        return (lowers-uppers) / (uppers+lowers)

    def plot(self, lw=0.5, color='#6699ff'):
        fig = plt.figure(figsize=(2,6))
        ax = fig.add_subplot(111)
        ax.barh(range(len(self.layers)), self.layers, color=color, lw=lw, align='edge', height=1.0, alpha=1.0, zorder=10)
        ax.grid(zorder=2)
        ax.set_ylabel('Layers')
        ax.set_title(self.label)
        ax.set_xlim([-0.5,1.0])
        ax.set_xlabel('Measurement (units)')
        ax.invert_yaxis()
        #ax.set_xticks(ax.get_xticks()[::2])    # take out every second tick
        ax.spines['right'].set_visible(False)  # hide the spine on the right
        ax.yaxis.set_ticks_position('left')    # Only show ticks on the left and bottom spines

        plt.show()

velocities = [0.23, 0.34, 0.45, 0.25, 0.23, 0.35]

l = Layers(velocities, label='Well # 1')

l.label

l.plot()


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Here's a good time to take a break

Variables and Assignment
Native data types
Operators and Expressions
Data collections and data structures
Procedures and control: Loops and Making choices
Getting data, manipulating data
Defining functions and calling functions
Writing and running programs
Objects and classes

Exercise manipulating and viewing lists:

a) create a new list that has the integer 1 for sand and 2 for shale.
b) using plt.plot() create a plot of the porosity values
c) make a depth vs porosity plot so depth is vertically downwards
d) Use plt.scatter() to make a cross plot of gamma vs porosity
f) Explore other keyword arguments for plt.plot() and plt.scatter to pretty things up
bonus e) Use the keyword c in the call to scatter to distinguish between sand and shale
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Indexing practice

Exercise:

return the string:
> Triassic (Tr)

return just the word:
> Triassic

return the abbreviation:
> (Tr) enclosed in parenthesis

return just the abbreviation:
> Tr

(bonus points if you can do (d) all in one line)
Nested list
lists can contain anything*
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Indexing, slicing, striding
In addition to accessing a single element in a list or string, we can also slice or stride into data structures to access multiple elements at once.
Can you write Python code for it?

name = 'Cambrian (C)'
name

name = 'Cambrian (C)'


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Making choices
The if statement
Can you write Python code for it?

layers = ['shale','shale','shale','sand','sand','sand','sand','shale','shale','shale']
# the if statement:


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Native data types
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Procedures and control: Loops and Making choices
Loops
Doing stuff many times
the while loop
the for loop
Can you write Python code for it?

layers = ['shale','shale','shale','sand','sand','sand','sand','shale','shale','shale']

# for loop syntax
for val in gamma:
    print(val)


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Sets
Instances of the set type are equivalent to mathematical sets. Like their math counterparts, literal sets in Python are defined by comma seperated values between curly braces ({}). Sets are unordered containers of unique values. Duplicated elements are ignored. Beacuse they unordered, sets are not sequences and cannot be duplicated.
Can you write Python code for it?

# a literal set formed with elements of various types
{1.0, 10, "one hundred", (1, 0, 0, 0)}

# a literal set OF special values
{True, False, None, "", 0.0, 0}

# conversion from a list to a set
set([2.0, 4, "eight", (16,), 4, 4, 2.0])


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
String methods and string formatting

Exercise: Use a combination of string methods on s text formatting to produce the following output:
> The Nordegg limestone has minor shale fragments
(ensure sentence case and remove '#', ':', `'\n')

Operators and Expressions

mathematical operations

comparison operations

bitwise operations

augmented assignment, copies, and pointers

boolean expressions

conversion functions

mathematical operations
comparison operations
augmented assignment, copies and pointers
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
The Python standard library
Built-in functions
Built-in Types
docs.python.org
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Using a regular expression:
Can you write Python code for it?

import re

s = re.search(r'<i>(.+?million years ago)</i>', r.text)
text = s.group(1)


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
Writing and running programs
Put the previous function in a text file and give it the name, load_tops.py
Your first module
Can you write Python code for it?

import my_module

topsfile = 'data/L30_tops.txt'
tops = load_tops.my_tops(topsfile)


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
You can concatenate tuples together in the same way as lists, but be careful about the order of opeartions. This is where parentheses come in handy,
(1, 2) + (3, 4)
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
boolean expressions
Can you write Python code for it?


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
conversion functions
Data collections and data structures
list, dict, tuples, sets
list
Lists in Python are one-dimensional, ordered containers whose elements may be any Python objects. Lists are mutable and have methods for adding and removing elements to and from themselves. The literal syntax for lists is surround commas seperated values with square brackets ([]). The square brackets are a syntactic hint that lists are indexable.
Can you write Python code for it?

# [1,1] + [3,3] + [4, 4]
list(str(10))

fib = [1, 1, 2, 3, 5, 8] + [13]
fib.append(13)

del(fib)

fib.extend([21.0, 34.0, 55.0])
fib

fib += [89.0, 144.0]
fib

fibm = np.array(fib[:-1])
fibp = np.array(fib[1:])
plt.plot(fibp/fibm)
plt.title('Golden Ratio')
fibp/fibm


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
dicts
Dictionaries are hands down the most important data structure in Python. Everything in Python is a dictionary. A dictionary, or dict, is a mutable, unordered collection of unique key / value pairs.
timescale = dict([(k1,v1),(k1,v1),(k1,v1)])
tuples
Tuples are the immutable form of lists. They behave almost exactly the same as lists in every way except that you cannot change any of their values. There are no append() or extend() methods, and there are no in-place operators.
They also differ from lists in their syntax. They are so central to how Python works, that tuples are defined by commas. Oftentimes, tuples will be seen surrounded by parentheses. These parentheses only serve to group actions or make the code more readable, not to actually define tuples.
Can you write Python code for it?

a = 1,2,3,4  # a length-4 tuple
b = (42,)    # length-1 tuple defined by the comma
c = (42)     # not a tuple, just the number 42
d = ()       # length-0 tuple- no commas means no elements


I am working on the file "Part1a_Fundamentals_of_Programming.ipynb".
The first task is:
str indexing (how to count, part 1)

Exercise: return the e character in s

Try help(s), s1?, s1??, s1.<tab>, s1.upper() , s1.strip(), s1.startswith(), s1.pop()
Can we do add two 'strings' together?
Can you write Python code for it?

s2, s3 = 'Limestone \n', 'Shale'

s2 + s3

print(s2 + s3)

print(s2 * 5)

lithology = s + s2 + 'has minor ' + s3 + ' fragments'
lithology

'{0} {1} has minor {2} fragments'.format(s,s2,s3)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
60 times faster on my machine!
Aside: more performance with numba
Can you write Python code for it?

from numba import jit

@jit
def compute_rc_numba(layers):
    uppers = layers[:-1]
    lowers = layers[1:]
    return (lowers - uppers) / (uppers + lowers)

%timeit compute_rc_numba(biglog)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
But some people might find it more inuitive to do it this way:
Can you write Python code for it?

from functools import partial

cuber = partial(power, b=3)

cuber(2)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Exercise: Make a function to get the start and end ages of any geologic period, taking the name of the period as an argument.
Can you write Python code for it?

def get_age(period):
    url = "http://en.wikipedia.org/wiki/" + period
    r = requests.get(url)
    start, end = re.search(r'<i>([\.0-9]+)–([\.0-9]+) million years ago</i>', r.text).groups()
    return float(start), float(end)

period = "Jurassic"
get_age(period)

def duration(period):
    t0, t1 = get_age(period)
    duration = t0 - t1
    response = "According to Wikipedia, the {0} period was {1:.2f} Ma long.".format(period, duration)
    return response

duration('Cretaceous')


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Functions
Definition, inputs, side-effects, returning, scope, docstrings
Can you write Python code for it?

# Exercise
def compute_rc(layers):
    """
    Computes reflection coefficients given
    a list of layer impedances.
    """
    uppers = layers[:-1]
    lowers = layers[1:]
    rcs = []
    for pair in zip(lowers, uppers):
        rc = (pair[1] - pair[0]) / (pair[1] + pair[0])
        rcs.append(rc)
    return rcs

compute_rc(layers)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
However, you can't speed up our original list-based function this way.
Can you write Python code for it?

@jit
def compute_rc_hopeful(layers):
    """
    Computes reflection coefficients given
    a list of layer impedances.
    """
    uppers = layers[:-1]
    lowers = layers[1:]
    rcs = []
    for pair in zip(lowers, uppers):
        rc = (pair[1] - pair[0]) / (pair[1] + pair[0])
        rcs.append(rc)
    return rcs

%timeit compute_rc_hopeful(biglog)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
In practice, we'd often write this as a list comprehension. Then we can skip the creation of the funciton or lambda entirely:
Can you write Python code for it?


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Note that the log has to be fairly big for the benchmarking to work properly, because otherwise the CPU caches the computation and this skews the results.
Now we can re-write our function using arrays instead of lists.
Can you write Python code for it?

# Exercise
def compute_rc_vector(layers):
    uppers = layers[:-1]
    lowers = layers[1:]
    return (lowers - uppers) / (uppers + lowers)

%timeit compute_rc_vector(biglog)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
One of the advantages of map is that it is 'lazy', so if you map a function to a giant list, you don't get a giant list back, you get an iterator. A list-comp would give you a giant list, possibly jamming the memory on your box.
reduce
reduce takes a sequence and applies some function to it recursively. You could think of it like a running sum, say, but for any function, not just summing.
Can you write Python code for it?

def runsum(a, b):
    return a + b

# For some reason reduce is not in the main namespace like map
from functools import reduce

reduce(runsum, l)

def runmult(a, b):
    return a * b

reduce(runmult, l)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Put in a file and import into a new notebook
Numpy
Before continuing, do some basic NumPy array stuff in the interpreter.
Let's make a really big 'log' from random numbers:
Can you write Python code for it?

import numpy as np  # Just like importing file

biglog = np.random.random(10000000)
%timeit compute_rc(biglog)


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Use View Source in your browser to figure out where the age range is on the page, and what it looks like.
Try to find the same string here.
Can you write Python code for it?

import requests
r = requests.get(url)
r.text[:500]


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
Using a regular expression:
Can you write Python code for it?


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
We can get around defining that tiny function sq() with a lambda, which you can think of as a temporary, 'throwaway' function:
Can you write Python code for it?


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
We could rewrite that tiny function as a lambda, which is basically a little unnamed function:
Can you write Python code for it?


I am working on the file "Part1b_Intro_to_scientific_computing.ipynb".
The first task is:
What if we want to sort based on the number of characters? Don't ask why, we just do. Then we write a function that returns a key which, when sorted, will give the ordering we want.
Can you write Python code for it?

def strlen(n):
    return len(str(n))

sorted(l, key=strlen)


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Benchmark Traditional Item-Item Collaborative Filtering
Can you write Python code for it?

import time
start = time.time()
naive = np.zeros((len(rows),len(cols)))
for row in x1:
    naive[row[0], row[1]] = row[2]
items, amean, umean, imean = itemsimilar(naive,'cos')
end = time.time()
print(end-start)

print(end - start)

items.shape


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Coverage for Content Based CF
Can you write Python code for it?

covcon1 = []
covcon2 = []
covcon3 = []
covcon4 = []
covcon5 = []
mlist = [5,10,15,20,25]
for i in range(len(rows)):
    if(i%100 == 0):
        print(i)
    ans = coverage(full_mat, i, items_features, 10, mlist, flipped_dict, True)
    covcon1.extend(ans[0])
    covcon2.extend(ans[1])
    covcon3.extend(ans[2])
    covcon4.extend(ans[3])
    covcon5.extend(ans[4])

print("Coverage with recommending 5 books", len(set(covcon1))/4959 *100 ,"%")

print("Coverage with recommending 10 books", len(set(covcon2))/4959 *100 ,"%")

print("Coverage with recommending 15 books", len(set(covcon3))/4959 *100 ,"%")

print("Coverage with recommending 20 books", len(set(covcon4))/4959 *100 ,"%")

print("Coverage with recommending 25 books", len(set(covcon5))/4959 *100 ,"%")


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Data Preparation/ Cleaning <br>
Removing ratings equal to zero, since Book Crossing Dataset has rating scale from 1-10. Taking Inner Join with books dataframe to maintain books whose details exist.
Can you write Python code for it?

newdf = ratings[ratings.book_rating>0]
joined = books.merge(newdf, on ='isbn')
print(joined.shape)


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Function for Evaluation Metrics: MAE and RMSE
Can you write Python code for it?

def rmse(ypred, ytrue):
    ypred = ypred[ytrue.nonzero()].flatten()
    ytrue = ytrue[ytrue.nonzero()].flatten()
    return np.sqrt(mean_squared_error(ypred, ytrue))

def mae(ypred, ytrue):
    ypred = ypred[ytrue.nonzero()].flatten()
    ytrue = ytrue[ytrue.nonzero()].flatten()
    return mean_absolute_error(ypred, ytrue)


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Function item similar returns matrix of pairwise similarity between all items based on the option provided. Also return amean (global mean rating), umean (average rating of each user), imean (Average rating of each item)
Can you write Python code for it?

def itemsimilar(mat, option):
    amean = np.mean(mat[mat!=0])
    umean = sum(mat.T) / sum((mat!=0).T)
    imean = sum(mat) / sum((mat!=0))

    umean = np.where(np.isnan(umean), amean, umean)
    imean = np.where(np.isnan(imean), amean, imean)

    n = mat.shape[1]
    sim_mat = np.zeros((n, n))

    if option == 'pr':
        #print("PR")
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = pr(mat, i, j, imean)
        sim_mat = (sim_mat + 1)/2
    elif option == 'cos':
        #print("COS")
        print(n)
        for i in range(n):
            if(i%100 == 0):
                print(i)
            for j in range(n):
                sim_mat[i][j] = cos(mat, i, j)
    elif option == 'adjcos':
        #print("ADJCOS")
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = adjcos(mat, i, j, umean)
        sim_mat = (sim_mat + 1)/2
    elif option == 'euc':
        #print("EUCLIDEAN")
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = euc(mat, i, j)
    else:
        #print("Hello")
        sim_mat = cosine_similarity(mat.T)

    return sim_mat, amean, umean, imean


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Loading the Amazon Data Set
Can you write Python code for it?

data1 = pd.read_csv('../clean-data/ratings_Books.csv', )

data1.columns = ['user_id', 'isbn', 'book_rating', 'timestamp']


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Predict function is used to get recommended rating by user i for item j.
Can you write Python code for it?

def predict(user, item, mat, item_similarity, amean, umean, imean,  k=20):
    nzero = mat[user].nonzero()[0]
    if len(nzero) == 0:
        return amean
    baseline = imean + umean[user] - amean
    choice = nzero[item_similarity[item, nzero].argsort()[::-1][1:k+1]]

    prediction = ((mat[user, choice] - baseline[choice]).dot(item_similarity[item, choice])/ sum(item_similarity[item, choice])) + baseline[item]

    if np.isnan(prediction):
        prediction = imean[item] + umean[user] - amean
    if prediction > 5:
        prediction = 5
    if prediction < 1:
        prediction = 1
    return prediction

predict(0,1, naive, items, amean, umean, imean,5)

def get_results1(X, rows, cols, folds, k, item_similarity, amean, umean, imean):
    kf =  KFold(n_splits=folds, shuffle = True, random_state=95)
    count = 1
    rmse_list = []
    mae_list = []
    trmse_list = []
    tmae_list = []
    for train_index, test_index in kf.split(X):
        print("----------   Fold ", count, "---------------")
        train_data, test_data = X[train_index], X[test_index]

        full_mat = np.zeros((rows, cols))

        for row in train_data:
            full_mat[row[0], row[1]] = row[2]

        preds = []
        real = []

        for row in train_data:
            user_id, isbn, rating = row[0], row[1], row[2]
            preds.append(predict(user_id, isbn, full_mat, item_similarity, amean, umean, imean, k))
            real.append(rating)

        err1 = rmse(np.array(preds), np.array(real))
        err2 = mae(np.array(preds), np.array(real))
        trmse_list.append(err1)
        tmae_list.append(err2)

        print('Train Errors')
        print('RMSE : %.4f' % err1)
        print('MAE : %.4f' % err2)

        preds = []
        real = []

        for row in test_data:
            user_id, isbn, rating = row[0], row[1], row[2]
            preds.append(predict(user_id, isbn, full_mat, item_similarity, amean, umean,...


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Sampling wiht each item being rated atleast 53 times
Can you write Python code for it?

temp = data3[data3['isbn'].isin(data3['isbn'].value_counts()[data3['isbn'].value_counts()>50].index)]
# print(len(temp.user_id.unique()))
# print(len(temp.isbn.unique()))
temp1 = temp[temp['user_id'].isin(temp['user_id'].value_counts()[temp['user_id'].value_counts()>49].index)]
# print(len(temp1.user_id.unique()))
# print(len(temp1.isbn.unique()))
temp2 = temp1[temp1['isbn'].isin(temp1['isbn'].value_counts()[temp1['isbn'].value_counts()>53].index)]
print(len(temp2.user_id.unique()))
print(len(temp2.isbn.unique()))

print(temp2.groupby(['user_id']).count()['book_rating'].mean())
print(temp2.groupby(['isbn']).count()['book_rating'].mean())


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Similarity Matrix Using Only Book Features
Can you write Python code for it?

naive = np.zeros((len(rows),len(cols)))
for row in x1:
    naive[row[0], row[1]] = row[2]
items_features, temple1, temple2, temple3 = itemsimilar(inputscores,'')

s1 = time.time()
get_results1(nmat, len(rows), len(cols), 5 ,20,items_features, amean,umean, imean)
e1 = time.time()

print("Time to test the recommendation over 5 folds cross validation of the data", (e1-s1)/5, "seconds")

each_sims_con = []
each_sims_rmse_con = []
for k in [5, 10, 15, 20, 25]:
    print("Nearest Neighbors: ",k)
    ans1, ans2  = get_results1(nmat, len(rows), len(cols), 5 ,k,items_features, amean,umean, imean)
    each_sims_con.append(ans1)
    each_sims_rmse_con.append(ans2)

print()
print("Best K Value for")
print()
print("Min MAE")
print(np.min(each_sims_con), np.argmin(each_sims_con))
print("Min RMSE")
print(np.min(each_sims_rmse_con), np.argmin(each_sims_rmse_con))
print()

results_df2 = pd.DataFrame({'Nearest Neighbors': [5, 10, 15, 20, 25], 'MAE': each_sims_con, 'RMSE': each_sims_rmse_con })
plot2 = results_df2.plot(x='Nearest Neighbors', y=['MAE', 'RMSE'], ylim=(0.5,0.9), title = 'Content Based Item-Item CF: Metrics over different K')
fig = plot2.get_figure()
fig.savefig('MetricsContentCFK.png')


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Some Books don't have unique ISBN, creating a 1:1 maping between books-title and ISBN
Can you write Python code for it?

print('Number of Books == Number of ISBN ? ', books["Book-Title"].nunique() == books["ISBN"].nunique())
book_dict = books[["Book-Title","ISBN"]].set_index("Book-Title").to_dict()["ISBN"]
books['new_isbn'] = books["Book-Title"].apply(lambda x: book_dict[x])
print('Number of Books == Number of ISBN ? ', books["Book-Title"].nunique() == books["new_isbn"].nunique())

books['isbn'] = books['new_isbn']

del books['ISBN']
del books['new_isbn']

books.shape


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Sparsity, Number of Users and Items in Amazon Dataset
Can you write Python code for it?

rows = data1.user_id.unique()
cols = data1.isbn.unique()
print(data1.user_id.nunique(), data1.isbn.nunique())
print("Sparsity :", 100 - (data1.shape[0]/(data1.user_id.nunique()* data1.isbn.nunique())))

data1 = data1[['user_id', 'isbn', 'book_rating']]

data1.shape

data2 = joined[['user_id', 'isbn', 'book_rating']]

data2.book_rating = data2.book_rating / 2.0

data2.shape

data2 = data2.drop_duplicates()

data2.shape


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Sparsity, Number of Users and Items in Book Crossing Dataset
Can you write Python code for it?

rows = joined.user_id.unique()
cols = joined['Book-Title'].unique()
print(joined.user_id.nunique(), joined.isbn.nunique())

print("Sparsity :", 100 - (joined.shape[0]/(joined.user_id.nunique()* joined.isbn.nunique())))


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
Time to Test the recommendations
Can you write Python code for it?

s = time.time()
get_results1(nmat, len(rows), len(cols), 5 ,20,items, amean,umean, imean)
e=time.time()

print("Time to test the recommendation over 5 fold cross validation of the data", (e-s)/5, "seconds")


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
get_results function is our function to cross_val setup and changing the parameter of this function will help to tune hyperparameter k (nearest neighbours)
Grid Search for best K for item-item CF using all the similarity metric implemented.
Can you write Python code for it?

each_sims = []
each_sims_rmse = []
for k in [5, 10, 15, 20, 25]:
    print("Nearest Neighbors: ",k)
    ans1, ans2  = get_results1(nmat, len(rows), len(cols), 5 ,k,items, amean,umean, imean)
    each_sims.append(ans1)
    each_sims_rmse.append(ans2)

print()
print("Best K Value for")
print()
print("Min MAE")
print(np.min(each_sims), np.argmin(each_sims))
print("Min RMSE")
print(np.min(each_sims_rmse), np.argmin(each_sims_rmse))
print()

print(each_sims[2], each_sims_rmse[2])

results_df1 = pd.DataFrame({'Nearest Neighbors': [5, 10, 15, 20, 25], 'MAE': each_sims, 'RMSE': each_sims_rmse })
plot1 = results_df1.plot(x='Nearest Neighbors', y=['MAE', 'RMSE'], ylim=(0.5,0.85), title = 'Item-Item CF: Metrics over different K')
fig = plot1.get_figure()
fig.savefig('MetricsCFK.png')


I am working on the file "Part2/analysis/DatasetCreation_Benchmark_ContentBased.ipynb".
The first task is:
getmrec function is used to get top m recommendation for a user_id based on the similarity matrix (option), k neighbours.
Can you write Python code for it?

full_mat = np.zeros((len(rows),len(cols)))
for row in nmat:
    full_mat[row[0], row[1]] = row[2]
#item_similarity, amean, umean, imean = itemsimilar(full_mat, 'euc')

def getmrec(full_mat, user_id, item_similarity, k, m, idict,  cov = False):

    n = item_similarity.shape[0]
    nzero = full_mat[user_id].nonzero()[0]

    preds = {}
    for row in range(n):
        preds[row] = predict(user_id, row, full_mat, item_similarity, amean, umean, imean, k)

    flipped_dict = dict(zip(idict.values(), idict.keys()))

    if not cov:
        print("Books Read -----")
        for i in nzero:
            print(flipped_dict[i])
            del preds[i]

    res = sorted(preds.items(), key=lambda x: x[1], reverse = True)

    ans = [flipped_dict[i[0]] for i in res[:m]]
    return ans

flipped_dict = dict(zip(idict.values(), idict.keys()))


I am working on the file "Part2/analysis/feature_extraction_from_api.ipynb".
The first task is:
Importing the data file
This data file contains the original data that needs to be enriched. It contains the ISBN data for books using which we will be extracting additional information from the API.
Can you write Python code for it?

df = pd.read_csv('Combine.csv',index_col=0)

all_isbn = df.isbn.unique()

isbn_df = pd.DataFrame(all_isbn,columns=['isbn'])

c = 0

def get_details(isbn):

    global c

    c+=1

    if(c%100 == 0):
        print(c)

    try:
        b = gc.book(isbn=isbn)
        return pd.Series({'title':b.title,'description':b.description,'num_pages':b.num_pages})
    except:
        return pd.Series({'title':'none','description':'none','num_pages':'none'})

isbn_df[['description','num_pages','title']] = isbn_df.isbn.apply(get_details)

isbn_df.to_pickle('ibsn_features_full.pickle')

isbn_df = pd.read_pickle('ibsn_features_full.pickle')


I am working on the file "Part2/analysis/feature_extraction_from_api.ipynb".
The first task is:
Retrying to get info for null records
Can you write Python code for it?

dfx = isbn_df[isbn_df['title'] == 'none']

dfx.head()

dfx[['description','num_pages','title']] = dfx.isbn.apply(get_details)

dfx[dfx['title'] == 'none'].shape


I am working on the file "Part2/analysis/feature_extraction_from_api.ipynb".
The first task is:
Saving the data file
Can you write Python code for it?


I am working on the file "Part2/analysis/feature_extraction_from_api.ipynb".
The first task is:
There are 34 records that still remain with no information from the API. We shall remove these records from our dataset as they form a negligible portion of our sample of books.
Merging dfx with ibsn_df
Can you write Python code for it?

for i, row in dfx.iterrows():
    isbn_df.loc[i] = dfx.loc[i]

# Checking if the newly created dataset contains the same number of empty records as in dfx

isbn_df[isbn_df['title'] == 'none'].shape == dfx[dfx['title'] == 'none'].shape

# dfx = isbn_df[isbn_df['title'] == 'none']


I am working on the file "Part2/analysis/tree_based_ann.ipynb".
The first task is:
Function for Evaluation Metrics: MAE and RMSE
Can you write Python code for it?


I am working on the file "Part2/analysis/tree_based_ann.ipynb".
The first task is:
Grid Search for best K for item-item CF using all the similarity metric implemented.
Can you write Python code for it?

start = time.time()
full_mat = np.zeros((len(rows), len(cols)))

for row in nmat:
    full_mat[row[0].astype(int), row[1].astype(int)] = row[2]

#item_similarity, amean, umean, imean = itemsimilar(full_mat, 'cos')

# this .pkl file can be accessed on google drive. Takes 5-10 hours to create this file: http://bit.ly/2CkBOjk
item_similarity = pickle.load(open("items_similarity_mat.pkl", "rb" ))

amean = np.mean(full_mat[full_mat!=0])
umean = sum(full_mat.T) / sum((full_mat!=0).T)
imean = sum(full_mat) / sum((full_mat!=0))

umean = np.where(np.isnan(umean), amean, umean)
imean = np.where(np.isnan(imean), amean, imean)

from sklearn.random_projection import GaussianRandomProjection
grp = GaussianRandomProjection(n_components=10, eps=0.1, random_state=None)
W = grp.fit_transform(np.transpose(full_mat))
print(W.shape)

rte = RandomTreesEmbedding(n_estimators=10, max_depth=1000, min_samples_split=30, min_samples_leaf=30,sparse_output=False, random_state=12)
tree = rte.fit_transform(W)#np.transpose(full_mat)
print(tree.shape)
tree = tree[:,~(sum(tree)==tree.shape[0])]
sum(tree)

ks = [2, 3, 5, 8, 10, 15, 20, 25]
trmse_list = []
tmae_list = []
for k in ks:
    preds = []
    real = []
    item_similarity_ann = itemsimilar_ann(tree, full_mat, k=k, items = item_similarity)
    print('Item similarity for items in same leaf computed')
    for row in nmat:
        user_id, isbn, rating = row[0], row[1], row[2]
        preds.append(predict(user_id.astype(int), isbn.astype(int), full_mat, item_similarity_ann, amean, umean, imean, k=20))
        real.append(rating)

    err1 = rmse(np.array(preds), np.array(real))
    err2 = mae(np.array(preds), np.array(real))
    trmse_list.append(err1)
    tmae_list.append(err2)

    print(k)
    print('Errors')
    print('RMSE : %.4f' % err1)
    print('MAE : %.4f' % err2)

df_results = pd.DataFrame({'k':ks, 'MAE':tmae_list, 'RMSE':trmse_list})
df_results

plot1 = results_df1.plot(x='Nearest Neighbors', y=['MAE', 'RMSE'], ylim=(0,1), title = 'Item-Item CF: Metrics over different...


I am working on the file "Part2/analysis/tree_based_ann.ipynb".
The first task is:
itemsimilar function is used to get similarities between items
Can you write Python code for it?

def itemsimilar(mat, option, indices='nope'): #item-item
    amean = np.mean(mat[mat!=0])
    umean = sum(mat.T) / sum((mat!=0).T)
    imean = sum(mat) / sum((mat!=0))

    umean = np.where(np.isnan(umean), amean, umean)
    imean = np.where(np.isnan(imean), amean, imean)

    n = mat.shape[1]
    sim_mat = np.ones((n, n))*0

    if option == 'pr':
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = pr(mat, i, j, imean)
        sim_mat = (sim_mat + 1)/2
    elif option == 'cos':
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = cos(mat, i, j)
    elif option == 'adjcos':
        for i in range(n):
            for j in range(n):
                sim_mat[i][j] = adjcos(mat, i, j, umean)
        sim_mat = (sim_mat + 1)/2
    elif option == 'euc':
        if indices=='nope':
            for i in range(n):
                for j in range(n):
                    sim_mat[i][j] = euc(mat, i, j)
        else:
            for i in indices:
                for j in indices:
                    sim_mat[i][j] = euc(mat, i, j)
    else:
        sim_mat = cosine_similarity(mat.T)

    return sim_mat, amean, umean, imean


I am working on the file "Part2/analysis/tree_based_ann.ipynb".
The first task is:
itemsimilar_ann() uses itemsimilar() to find closest items using tree based ANN
Can you write Python code for it?

def itemsimilar_ann(tree, mat, k, items, recalculate=False):
    n=tree.shape[1]
    for i in range(n):
        indices = np.argwhere(tree[:, i]==1).flatten()
        indices_none = np.argwhere(tree[:, i]!=1).flatten()
        if recalculate==True:
            item_similarity1, amean1, umean1, imean1 = itemsimilar(mat, 'cos', indices)
        else:
            item_similarity1 = items.copy()
            for m in indices_none:
                item_similarity1[m] = 0
        if i==0:
            item_similarity_all = item_similarity1.copy()
            item_similarity_all[:,:] = 0
            item_similarity_agg = item_similarity_all.copy()
            closest_items = item_similarity_all.copy()
            item_similarity_final = item_similarity_all.copy()

        no_data = item_similarity_all==0
        item_similarity_all[no_data] = item_similarity_all[no_data] + item_similarity1[no_data]

        for j in indices:
            a = item_similarity1[j].copy()
            # item_similarity_agg counts number of times the item came as closest to the jth item.
            item_similarity_agg[j] = item_similarity_agg[j] + (a>=np.amax([0.00001, (-1*np.partition(-a, k-1, axis=-1)[k-1])])).astype(int)

    for j in range(mat.shape[1]):
        b = item_similarity_agg[j].copy()
        closest_items[j] = (b>=np.amax([0.00001, (-1*np.partition(-b, k-1, axis=-1)[k-1])])).astype(int)

    item_similarity_final[closest_items>0] = item_similarity_all[closest_items>0]
    return(item_similarity_final)


I am working on the file "Part2a__Well_log_calculations.ipynb".
The first task is:
Q. Create an acoustic impedance log
Can you write Python code for it?


I am working on the file "Part2a__Well_log_calculations.ipynb".
The first task is:
Q. Use the porosity log, NPHISS to determine the average value of porosity for a) sands, b) shales, c) carbonates.
Can you write Python code for it?


I am working on the file "Part2a__Well_log_calculations.ipynb".
The first task is:
Q. Using a Sand-Shale cutoff for Gamma-Ray of 60 API, what is the Gross sand thickness from 2200 - 3000 m?
Can you write Python code for it?


I am working on the file "Part2a__Well_log_calculations.ipynb".
The first task is:
Q. Write a function that converts slowness (us/m) into velocity (m/s) your code here
Can you write Python code for it?


I am working on the file "Part2a__Well_log_calculations.ipynb".
The first task is:
Q: from welly's Well module, use the from_las function to create a well object from the las file
Can you write Python code for it?

# your code here

# it should looks like this
well

pslow = well.data['DT']
rhob = well.data['RHOB']
gr = well.data['GRD']


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
And then use this to convert the logs to a time basis:
Can you write Python code for it?

dt = 0.004  # Sample interval.
maxt = 3  # Max time that we need; just needs to be longer than the log.

# Make a regular time basis: the seismic time domain.
seis_time = np.arange(0, maxt, dt)

# Interpolate the AI log onto this basis.
ai_t = np.interp(seis_time, dt_time, ai)

plt.figure(figsize=(16, 2))
plt.plot(seis_time, ai_t)
plt.show()


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
If we are recording with dynamite or even an airgun, this might be an acceptable model of the seismic. But if we're using Vibroseis, things get more complicated.
Compare with the seismic
Can you write Python code for it?

seismic = np.loadtxt('data/Penobscot_xl1155.txt')

syn.shape

# Sample index space
idx = np.arange(0, syn.size)

# The synthetic is at trace number 77
tr = 77
gain = 50

# Make a shifted version of the synthetic to overplot.
s = tr + gain*syn

plt.figure(figsize=(10,20))
plt.imshow(seismic.T, cmap='Greys')
plt.plot(s, idx)
plt.fill_betweenx(idx, tr, s, where=syn>0, lw=0)
plt.xlim(0, 400)
plt.ylim(800, 0)
plt.show()


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Impulsive wavelet
Convolve with a wavelet.
Can you write Python code for it?


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Load petrophysical data
We'll use welly to faciliate loading curves from an LAS file.
Can you write Python code for it?

from welly import Well
L30 = Well.from_las('data/L30.las')


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Now use this to scale the DT values to 'seconds per step' (instead of µs/m).
Can you write Python code for it?

scaled_dt = step * np.nan_to_num(dt) / 1e6  # Convert to seconds per step


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Plotting these is a bit more fiddly, because we would like to show them as a sequence of spikes, rather than as a continuous curve, and matplotlib's axvline method wants everything in terms of fractions of the plot's dimensions, not as values in the data space.
Can you write Python code for it?

fig = plt.figure(figsize=(16, 2))
ax = fig.add_subplot(111)
ax.axhline()
for i, c in enumerate(rc):
    mi, ma = (0.5, 0.5+c) if c > 0 else (0.5+c, 0.5)
    ax.axvline(i, mi, ma)
ax.set_ylim(-0.3, 0.3)
plt.show()


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Q. Compute velocity and thus acoustic impedance.
Can you write Python code for it?

# CALCULATE VELOCITY
ai = # PUT THE EXPRESSION FOR IMPEDANCE HERE

plt.figure(figsize=(16, 2))
plt.plot(z, ai)
plt.show()

# YOU SHOULD GET SOMETHING LIKE THIS:


I am working on the file "Part2b__Synthetic_seismogram.ipynb".
The first task is:
Q. What other curve do we need? Inspect the contents of L30 and assign any new curves to new variables.
Can you write Python code for it?


I am working on the file "Part2c__Seismic_arrays.ipynb".
The first task is:
Exercise: Get rid of some bad data for the last few samples on each trace; take the first 450 samples (of the last dimension)
Can you write Python code for it?


I am working on the file "Part2c__Seismic_arrays.ipynb".
The first task is:
Exercise: Plot this seismic trace (vertically!)
- make a "time axis": `np.arange(first_sample, last_sample, sample_rate)

- use matplotlib's `plt.plot` function (documentation!)

- other `plt` functions to check out: `plt.xlim`, `plt.title`, `plt.ylabel`, `plt.invert_yaxis()`, 'plt.grid`

Trace statistics

Exercise: use the plt.hist(s, bins) function to create a histogram with 100 bins
As suspected, most of the data values are close to zero, and fan outward (more or less symmetrically) to larger positive and negative values.
To be sure, we can include the values from the entire line to build up better statitics. However, in order to pass a 2D array to the plt.hist function, we have to unravel it first,
Trace Bandwidth
Let's look at the frequency content of this trace. To do this we will need to use the the fast fourier transform function from the Scipy FFT module. It helps to know that the sample rate of the trace is 0.04 s
Can you write Python code for it?

from scipy import fft
S = abs(fft(s))
power = 20*np.log10(a)
faxis = np.fft.fftfreq( len(power), d = dt)


I am working on the file "Part2c__Seismic_arrays.ipynb".
The first task is:
Exercise: Use numpy's reshape function to turn this this array into a 3D array. Print the shape attribute of the reshaped object to verify that this object has the correct number of inlines and crosslines
Can you write Python code for it?


I am working on the file "Part2c__Seismic_arrays.ipynb".
The first task is:
Some information about the seismic cube:

Number of inlines: 194

Number of crosslines: 299

Number of samples per trace: 463

Sample rate in seconds: 0.004

I'm giving you this information, but in practice, this would probably come from the file's header or meta-data section
Can you write Python code for it?

nIL = 194  # number of inlines
nXL = 299  # number of crosslines
nt = 463   # number of samples per trace
dt = 0.004 # sample rate in seconds


I am working on the file "Part_1.ipynb".
The first task is:
2.) Take a look at the files we have to work with
Can you write Python code for it?


I am working on the file "Part_1.ipynb".
The first task is:
3.) Time to do a little house cleaning and transformation
a.) First the clean up
Can you write Python code for it?

all_data['discount'].fillna(0, inplace=True)          #  Let's fill mising values with zeros
all_data['ext price'] = abs(all_data['ext price'])    #  Let's make the negative numbers go bye bye
all_data['unit price'] = abs(all_data['unit price'])
all_data['quantity'] = abs(all_data['quantity'])
all_data['date'] = pd.to_datetime(all_data['date'])   #  Let's make this a date object so we can do fancy stuff
all_data.head()


I am working on the file "Part_1.ipynb".
The first task is:
The road so far

Merged 3 data files of different format and scale
Cleaned and normalized data
Added calculated columns
Only 11 lines of "real" code used so far

4.) Let's join our data to another data set
Can you write Python code for it?

customer_data = pd.read_excel('./input/customer-status.xlsx')
customer_data.head()

all_data = pd.merge(all_data, customer_data, how='left')    #  Just like in SQL, we can left join data sets
all_data.head()


I am working on the file "Part_1.ipynb".
The first task is:
a.) Did you know that you can cross section pivots?
Can you write Python code for it?

pt.xs('Will LLC', level=0)    #  Let's cut out just the data for one account


I am working on the file "Part_1.ipynb".
The first task is:
a.) Now let's grab all the files and concatenate them together
Can you write Python code for it?

all_data = pd.DataFrame()
for f in glob.glob('./input/sales-*.xlsx'):
    df = pd.read_excel(f)
    all_data = pd.concat([all_data, df])
all_data.info()


I am working on the file "Part_1.ipynb".
The first task is:
b.) Now let's add a couple calculated columns
Can you write Python code for it?

all_data['total price'] = all_data['unit price'] * all_data['quantity'] * (1 - all_data['discount'])
all_data['month'] = all_data['date'].dt.month
all_data['month'] = all_data['month'].apply(lambda x: calendar.month_name[x])
all_data.tail()


I am working on the file "Part_1.ipynb".
The first task is:
b.) Open a JSON file that has abbreviations for all the states
Can you write Python code for it?

import json
states = json.loads(open('./input/state_abbreviations.json').read())


I am working on the file "Part_1.ipynb".
The first task is:
b.) We can get quick stats on numeric columns just by running a "describe"
Can you write Python code for it?


I am working on the file "Part_1.ipynb".
The first task is:
c.) An Excel file with a different cross-section on each tab
Can you write Python code for it?

writer = pd.ExcelWriter('./output/pivoted_x-sections.xlsx')
for name in pt.index.get_level_values(0).unique():
    temp_df = pt.xs(name, level=0)
    temp_df.to_excel(writer,name)
writer.save()


I am working on the file "Part_1.ipynb".
The first task is:
c.) Let's look at the raw data
Can you write Python code for it?


I am working on the file "Part_1.ipynb".
The first task is:
d.) Let's apply our function to populate the correct abbreviations
Can you write Python code for it?

all_data['abbrev'] = all_data.apply(convert_state, axis=1)
all_data.head()


I am working on the file "Part_2.ipynb".
The first task is:
2.) Preview the raw data
Can you write Python code for it?

df = pd.read_excel('./input/complete_data.xls')
df.head()