I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
Dedispersion
When pulses reach Earth they reach the observer at different times due to dispersion. This dispersion is the result of the interstellar medium causing time delays. This creates a "swooping curve" on the radio spectrogram instead of plane waves. If we are going to fold the pulses to increase the SNR then we're making the assumption that the pulses arrive at the same time. Thus we need to correct the dispersion by shifting each channel down a certain time delay relative to its frequency channel. We index a frequency column in the spectrogram. Then we split it between a time delay and original data and swap the positions.
However, the problem is, we don't know the dispersion measure DM of the signal. The DM is the path integral of the signal through the interstellar medium with an electron density measure of.
$$DM =\int_0^d n_e dl$$
What we do is we brute force the DM by executing multiple trials DMs and we take the highest SNR created by the dedispersion with the given trial DM.
Can you write Python code for it?

def delay_from_DM(DM, freq_emitted):
    if (type(freq_emitted) == type(0.0)):
        if (freq_emitted > 0.0):
            return DM / (0.000241 * freq_emitted * freq_emitted)
        else:
            return 0.0
    else:
        return Num.where(freq_emitted > 0.0,
                         DM / (0.000241 * freq_emitted * freq_emitted), 0.0)

def de_disperse(data,DM,fchan,width,tsamp):
  clean = deepcopy(data)
  for i in range(clean.shape[1]):
    end = clean.shape[0]
    freq_emitted = i*width+ fchan
    time = int((delay_from_DM(DM, freq_emitted))/tsamp)
    if time!=0 and time<clean.shape[0]:
      # zero_block = np.zeros((time))
      zero_block = clean[:time,i]
      shift_block = clean[:end-time,i]
      clean[time:end,i]= shift_block
      clean[:time,i]= zero_block

    elif time!=0:
      clean[:,i]= np.zeros(clean[:,i].shape)
  return clean

def DM_can(data, data_base, sens, DM_base, candidates, fchan,width,tsamp ):
  snrs = np.zeros((candidates,2))
  for i in range(candidates):
    DM = DM_base+sens*i
    data = de_disperse(data, DM, fchan,width,tsamp)
    time_series = data.sum(axis=1)
    snrs[i,1] = SNR(time_series)
    snrs[i,0] =DM
    if int((delay_from_DM(DM, fchan))/tsamp)+1 > data.shape[0]:
      break
    if i %1==0:
      print("Candidate "+str(i)+"\t SNR: "+str(round(snrs[i,1],4)) + "\t Largest Time Delay: "+str(round(delay_from_DM(DM, fchan), 6))+' seconds'+"\t DM val:"+ str(DM)+"pc/cm^3")
    data = data_base
  return snrs

# Functions to determine SNR and TOP candidates
def SNR(arr):
  index = np.argmax(arr)
  average_noise = abs(arr.mean(axis=0))
  return math.log(arr[index]/average_noise)

def top(arr, top = 10):
  candidate = []
  # Delete the first and second element fourier transform
  arr[0]=0
  arr[1]=0
  for i in range(top):
    # We add 1 as the 0th index = period of 1 not 0
    index = np.argmax(arr)
    candidate.append(index+1)
    arr[index]=0
  return candidate


I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
Dedispersion Trials
The computer now checks multiple DM values and adjust each frequency channel where it records its SNR. We increment the trial DM by a tunable parameter sens. After the trials, we take the largest SNR created by adjusting the time delays. We use that data to perform the FFT's and record the folded profiles.
Can you write Python code for it?

small_data = data_adjust[:,0,:]
data_base = data_adjust[:,0,:]
sens =0.05
DM_base =  6.4
candidates = 50
fchan = obs.header['fch1']
width = obs.header['foff']
tsamp = obs.header['tsamp']
fchan = fchan+ width*small_data.shape[1]
snrs = DM_can(small_data, data_base, sens, DM_base, candidates, fchan, abs(width),tsamp)
plt.plot(snrs[:,0], snrs[:,1])
plt.title('DM values vs SNR')
plt.xlabel("DM values")
plt.ylabel("SNR of Dedispersion")

DM = snrs[np.argmax(snrs[:,1]),0]
print(DM)
fchan = fchan+ width*small_data.shape[1]
data_adjust[:,0,:] = de_disperse(data_adjust[:,0,:], DM, fchan,abs(width),tsamp)
fig = plt.figure(figsize=(10, 8))
plt.imshow(data_adjust[:,0,:], aspect='auto')
plt.title('De-dispersed Data')
plt.xlabel("Fchans")
plt.ylabel("Time")
plt.colorbar()
plt.show()


I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
Detecting Pulses - Fourier Transforms and Folding
Next, we apply the discrete Fourier transform on the data to detect periodic pulses. To do so, we look for the greatest magnitude of the Fourier transform. This indicates potential periods within the data. Then we check for consistency by folding the data by the period which the Fourier transform indicates.
The folding algorithm is simple. You take each period and you fold the signals on top of itself. If the period you guessed matches the true period then by the law of superposition it will increase the SNR. This spike in signal to noise ratio appears in the following graph. This algorithm is the following equation.
Can you write Python code for it?

#  Preforming the fourier transform.
%matplotlib inline
import scipy.fftpack
from scipy.fft import fft
N = 1000
T = 1.0 / 800.0
x = np.linspace(0.0, N*T, N)
y = abs(data_adjust[:,0,:].mean(axis=1))
yf = fft(y)

xf = np.linspace(0.0, 1.0/(2.0*T), N//2)

# Magintude of the fourier transform
# Between 0.00035 and 3.5 seconds
mag = np.abs(yf[:60000])
candidates = top(mag, top=15)
plt.plot(2.0/N * mag[1:])
plt.grid()
plt.title('Fourier Transform of Signal')
plt.xlabel("Periods")
plt.ylabel("Magnitude of Fourier Transform")
plt.show()

print("Signal To Noise Ratio for the Fourier Transform is: "+str(SNR(mag)))
print("Most likely Candidates are: "+str(candidates))


I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
Folding Algorithm
The idea of the folding algorithm is to see if the signal forms a consistent profile as you fold/integrate the values together. If the profile appears consistent/stable then you're looking at an accurate reading of the pulsar's period. This confirms the implications drawn from the Fourier transform. This is profiling the pulsar. When folding the pulses it forms a "fingerprint" of the pulsar. These folds are unique to the pulsar detected.
$$s_j = \sum^{N/P-1}{K=0} D{j+kP} $$
We are suming over the regular intervals of period P. This is implemented below.
Can you write Python code for it?

#  Lets take an example of such a period!
#  The  0th candidate is the top ranked candidate by the FFT
period = 895
fold = np.zeros((period, data.shape[2]))
multiples = int(data.data.shape[0]/period)
results = np.zeros((period))

for i in range(multiples-1):
  fold[:,:]=data_adjust[i*period:(i+1)*period,0,:]+ fold

results = fold.mean(axis=1)
results = results - results.min()
results = results / results.max()

print(SNR(results))

plt.plot(results)
plt.title('Folded Signal Profile With Period: '+str(round(period*0.000349,5)))
plt.xlabel("Time (Multiples of 0.00035s)")
plt.ylabel("Normalized Integrated Signal")

#  Lets take an example of such a period!
#  The  0th candidate is the top ranked candidate by the FFT
can_snr =[]
for k in range(len(candidates)):
  period = candidates[k]
  fold = np.zeros((period, data.shape[2]))
  multiples = int(data.data.shape[0]/period)
  results = np.zeros((period))

  for i in range(multiples-1):
    fold[:,:]=data[i*period:(i+1)*period,0,:]+ fold

  results = fold.mean(axis=1)
  results = results - results.min()
  results = results / results.max()
  can_snr.append(SNR(results))
  # print(SNR(results))

print("Max SNR of Fold Candidates: "+ str(max(can_snr)))

# Generates multiple images saved to create a GIF
from scipy import stats
data = data
period = candidates[0]
fold = np.zeros((period, data.shape[2]))
multiples = int(data.data.shape[0]/period)
results = np.zeros((period))

for i in range(multiples-1):
  fold[:,:]=data[i*period:(i+1)*period,0,:]+ fold
  results = fold.mean(axis=1)
  results = results - results.min()
  results = results / results.max()
  #  Generates multiple frames of the graph as it folds!
  plt.plot(results)
  plt.title('Folded Signal Period '+str(period*0.000349)+" seconds| Fold Iteration: "+str(i))
  plt.xlabel("Time (Multiples of 0.00035s)")
  plt.ylabel("Normalized Integrated Signal")
  plt.savefig('/content/drive/My Drive/Deeplearning/Pulsars/output/candidates/CAN_3/multi_chan_'+str(period)+'_'+str(i)+'.png')
  plt.close()

results =...


I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
NOTICE
Notice how the signal to noise ratio is a lot smaller, It's smaller by 2 orders of magnitude (100x) than the original pulsar fold. Typically with a SNR of 1, it isn't considered a signal of interest as it's most likely just noise.
Can you write Python code for it?

#  Lets take an example of such a period!
#  The  0th candidate is the top ranked candidate by the FFT
can_snr =[]
for k in range(len(candidates)):
  period = candidates[k]
  fold = np.zeros((period, data.shape[2]))
  multiples = int(data.data.shape[0]/period)
  results = np.zeros((period))

  for i in range(multiples-1):
    fold[:,:]=data[i*period:(i+1)*period,0,:]+ fold

  results = fold.mean(axis=1)
  results = results - results.min()
  results = results / results.max()
  can_snr.append(SNR(results))

print("Max SNR of Fold Candidates: "+ str(max(can_snr)))


I am working on the file "GBT/pulsar_searches/Pulsar_Search/Pulsar_DedisperseV3.ipynb".
The first task is:
What Happens If The Data Doesn't Contain Pulses?
Below we will show you that this algorithm detects pulses and excludes targets that do not include this feature. We will do so by loading a target that isn't known to be a pulsar. HIP65960 is a target that doesn't contain repeating signals.
Below we will repeat and apply the same algorithm but on a target that isn't a pulsar. We won't reiterate the explanations again.
Can you write Python code for it?

!wget http://blpd13.ssl.berkeley.edu/dl/GBT_58402_66282_HIP65960_time.h5

from blimpy import Waterfall
import pylab as plt
import numpy as np
import math
from scipy import stats, interpolate

%matplotlib inline

obs = Waterfall('/content/GBT_58402_66282_HIP65960_time.h5',
               f_start=0,f_stop= 361408,max_load=5)
obs.info()
# Loads data into numpy array
data = obs.data
coarse_channel_width = np.int(np.round(187.5/64/abs(obs.header['foff'])))
obs.plot_spectrum()

average_power = np.zeros((data.shape[2]))
shifted_power = np.zeros((int(data.shape[2]/8)))
x=[]
spl_order = 2
print("Fitting Spline")
data_adjust = np.zeros(data.shape)
average_power = data.mean(axis=0)
#  Note the value 8 is the COARSE CHANNEL WIDTH
#  We adjust each coarse channel to correct the bandpass artifacts
for i in range(0, data.shape[2], coarse_channel_width):
    average_channel = average_power[0,i:i+coarse_channel_width]
    x = np.arange(0,coarse_channel_width,1)
    knots = np.arange(0, coarse_channel_width, coarse_channel_width//spl_order+1)

    tck = interpolate.splrep(x, average_channel, s=knots[1:])
    xnew = np.arange(0, coarse_channel_width,1)
    ynew = interpolate.splev(xnew, tck, der=0)
    data_adjust[:,0,i:i+coarse_channel_width]  = data[:,0,i:i+coarse_channel_width] - ynew

from copy import deepcopy
small_data = data[:,0,:]
data_base = data[:,0,:]
sens =0.05
DM_base = 6.4
candidates = 50
fchan = obs.header['fch1']
width = obs.header['foff']
tsamp = obs.header['tsamp']
# fchan = fchan+ width*small_data.shape[1]
fchan = 7501.28173828125
snrs = DM_can(small_data, data_base, sens, DM_base, candidates, fchan, abs(width),tsamp)
plt.plot(snrs[:,0], snrs[:,1])
plt.title('DM values vs SNR')
plt.xlabel("DM values")
plt.ylabel("SNR of Dedispersion")

DM = snrs[np.argmax(snrs[:,1]),0]
print(DM)
fchan = fchan+ width*small_data.shape[1]
data_adjust[:,0,:] = de_disperse(data_adjust[:,0,:], DM, fchan,abs(width),tsamp)

#  Preforming the fourier transform.
%matplotlib inline
import scipy.fftpack
from scipy.fft import fft
N =...


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
Now, let's read the observation data using Waterfall():
Can you write Python code for it?


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
The data has the shape (2, 66060288), meaning that is has 2 time integrations, and  66,060,288 channels, over a bandwidth of 187 MHz. For the record, that's a lot of channels! The corresponding channel bandwidth is far more than that required for most astrophysical observations.
Plotting the data
The Waterfall() class provides a plot_spectrum() function to plot the data:
Can you write Python code for it?


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
The specifics are discussed in detail in the SIGPROC user guide. Briefly, astronomers use a Celestial coordinate system to specify the location of objects in outer space. The src_raj and src_dej specify the J2000 coordinates, in terms of Right Ascension and Declination (RA & DEC), toward which the telescope is pointing. tstart specifies the Julian Date of the observation, and fch1 and foff specify the frequency of the first and frequency increment of each data channel respectively, in MHz.
We can also look directly at the header and data, which are stored as objects in the Waterfall() class:
Can you write Python code for it?


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
This is it: the telemetry signal from Voyager 1! What you see is the carrier (center) and two sidebands that carry the data. We can zoom in on these to see more features:
Can you write Python code for it?

plt.figure(figsize=(8, 6))
plt.subplot(3,1,1)
obs.plot_spectrum(f_start=8420.193, f_stop=8420.195) # left sideband
plt.subplot(3,1,2)
obs.plot_spectrum(f_start=8420.2163, f_stop=8420.2166) # carrier
plt.subplot(3,1,3)
obs.plot_spectrum(f_start=8420.238, f_stop=8420.24) # right sideband
plt.tight_layout()


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
We can get basic info about the file by reading the filterbank header. The info() method will do this:
Can you write Python code for it?


I am working on the file "GBT/voyager/voyager.ipynb".
The first task is:
What we see here is the power spectral density (or simply "spectrum") of the data contained in the filterbank file. Apparent in the plot are discontinuities due to the BL digital filters, but not a lot else. Voyager is a very narrowband signal, so we can't easily see it in this plot. However, if we zoom in on where Voyager's telemetry is known to be, we see some very narrowband signals:
Can you write Python code for it?


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Define Functions Used in R
Can you write Python code for it?

%%R

########################
## Convert ppm to mols C

calc_mol_C = function(ppm,ion,volume.L){
  # moles (n) = PV / RT
  # x = ppm
  temp.K=294.261
  pressure.atm=1
  R=0.08206

  ppm = as.numeric(ppm)
  ion = as.numeric(ion)
  mol.volume = (pressure.atm * volume.L) / (R * temp.K)  # mol gas in container
  mol.CO2 = mol.volume * (ppm / 1000000)  # fraction of mol as CO2

  if (ion == 44){
    mol.C = mol.CO2 * 12/44   # fraction of CO2 that is C
  } else {
    mol.C = mol.CO2 * 13/45   # fraction of CO2 that is C
  }
  return(mol.C)
}

#########################
#Calculate Standard Error
# Taken from http://www.cookbook-r.com/Graphs/Plotting_means_and_error_bars_(ggplot2)/

summarySE <- function(data=NULL, measurevar, groupvars=NULL, na.rm=FALSE, conf.interval=.95, .drop=TRUE) {
  library(plyr)

  # New version of length which can handle NA's: if na.rm==T, don't count them
  length2 <- function (x, na.rm=FALSE) {
    if (na.rm) sum(!is.na(x))
    else       length(x)
  }

  # This does the summary. For each group's data frame, return a vector with
  # N, mean, and sd
  datac <- ddply(data, groupvars, .drop=.drop,
                 .fun = function(xx, col) {
                   c(N    = length2(xx[[col]], na.rm=na.rm),
                     mean = mean   (xx[[col]], na.rm=na.rm),
                     sd   = sd     (xx[[col]], na.rm=na.rm)
                   )
                 },
                 measurevar
  )

  # Rename the "mean" column
  datac <- rename(datac, c("mean" = measurevar))

  datac$se <- datac$sd / sqrt(datac$N)  # Calculate standard error of the mean

  # Confidence interval multiplier for standard error
  # Calculate t-statistic for confidence interval:
  # e.g., if conf.interval is .95, use .975 (above/below), and use df=N-1
  ciMult <- qt(conf.interval/2 + .5, datac$N-1)
  datac$ci <- datac$se * ciMult

  return(datac)
}

##############################################
### Join Data and Time into Single POSIX stamp

time_converter <- function(date, time){
    x <-...


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 3: Import and Work-up in R
Can you write Python code for it?

## Setup R-Magic for Jupyter Notebooks
import rpy2
import pandas as pd
%load_ext rpy2.ipython

## Description: use Pandas to create a dataframe and then pipe that to R

# Import CO2 Data
co2_data = pd.read_csv(directory+"/"+output_name+".raw.co2.table.tsv", sep="\t")
%R -i co2_data

# Segregate Standards
%R raw_standards <- co2_data[grep("standard",co2_data$sampleID),]
%R co2_data <- co2_data[-grep("standard",co2_data$sampleID),]

# Import Standards
%R -i standards

# Import Microcosm Size
%R -i microcosm_size

# Import Analysis Directory
%R -i directory
%R -i output_name

# Import Events
try:
    events = pd.read_csv(directory+"/"+events, sep="\t")
    %R -i events

except:
    pass


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 4: Calculate Durations
Can you write Python code for it?

%%R

# Combine all dates and times into single 'POSIX' time-stamp
posix <- time_converter(events$date, events$time)
events <- cbind(events, posix)

# Calculate Duration for Each Timepoint
start <- subset(events, timepoint == "T0")
events <- subset(events, timepoint != "T0")
events$duration <- duration_calculator(start$posix, events$posix) # duration_calculator(start_time, all_time_points)

# Merge CO2 Data with duration
co2_data <- merge(co2_data, events, by = "timepoint")

# print current work-up
print(head(co2_data))


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 5: Calculate Standard Curve
Can you write Python code for it?

%%R
## Note: This script assumes very low inter-run variability
## From my experience, preparing the standards by hand introduces greater inter-run variability than the instrument.
## Therefore, this script will average all standard data and calculate ppm based off of this average

## a bit of clean-up
raw_standards$ion <- gsub("m/z ","",raw_standards$ion)
raw_standards <- subset(raw_standards, ion != "TIC")

## Concentrations are calculated based on total CO2  (converting 13C to 12C-equivalent is negligible)
# sum ion 44 and ion 45
raw_standards <- ddply(raw_standards, ~ timepoint + sampleID, summarise, total.peak.area = sum(peak.area))
raw_standards$combo <- paste(raw_standards$timepoint, raw_standards$sampleID,sep="_")

## regress standards
standards$sampleID <- rownames(standards)
raw_standards <- merge(raw_standards, standards, by = "sampleID")

## Plot Curve
plot <- ggplot(raw_standards, aes(total.peak.area, ppm, label = combo)) + geom_point() + geom_smooth(method=lm, se=F) + ggtitle("Raw Standards")
print(plot + geom_label(size = 4, hjust = -0.1))

## Remove Outliers from Standard Curve
remove_me <- c("T2_standard7","T2_standard6")

if (length(remove_me) > 0){
    refined_standards <- raw_standards[-which(raw_standards$combo %in% remove_me),]
    plot <- ggplot(refined_standards, aes(total.peak.area, ppm, label = combo)) + geom_point() + geom_smooth(method=lm, se=F) + ggtitle("Refined Standards")
    print(plot + geom_label(size = 4, hjust = -0.1))
} else {
    refined_standards <- raw_standards
}

## Calculate regression coefficients (force through zero)
m <- as.numeric(coef(lm(ppm ~ total.peak.area -1, refined_standards))[1])
#b <- as.numeric(coef(lm(ppm ~ total.peak.area, refined_standards))[1])


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 6: Convert peak area to ppm
Can you write Python code for it?

%%R

## a bit of clean-up
co2_data$ion <- gsub("m/z ","", co2_data$ion)
co2_data <- subset(co2_data, ion != "TIC")

## Calculate ppm based on curve
co2_data$adj.conc <- m*co2_data$peak.area

print(head(co2_data))


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 7: Convert ppm to mols C
Can you write Python code for it?

%%R

# Calculate mol
co2_data$mol.C <- apply(co2_data[,c("adj.conc","ion")], 1, function(x) calc_mol_C(x[1],x[2],microcosm_size))

print(head(co2_data))


I am working on the file "GC-MS_Data_Processing/GC-MS CO2 Data Processing Pipeline.ipynb".
The first task is:
Step 8: Calculate Cumulative Respiration
Can you write Python code for it?

%%R

## Calculate Cumulative Respiration
count = 1

for (sample in unique(co2_data$sampleID)){
    for (i in c(44, 45)){
        foo<-subset(co2_data, sampleID == sample & ion == i)
        foo<-foo[order(foo$duration),]
        foo$cum.mol.C <- cumsum(foo$mol.C)

        if (count == 1){
            cumulative <- foo
            count = count + 1
        } else {
            cumulative <- rbind(cumulative, foo)
        }
    }
}

print(head(cumulative))


I am working on the file "GC/flux_fit_and_integral.ipynb".
The first task is:
$$\frac{dN}{dE}=N_0\times\left(\frac{E}{E_0}\right)^{-\alpha}$$
$$\frac{dN}{dE}=N_0\times\left(\frac{E}{E_0}\right)^{-\alpha}\times e^{-\frac{E}{E_{cut}}}$$
$$\frac{dN}{dE}=N_0\times\left(\frac{E}{E_0}\right)^{-\alpha}\times \frac{1}{1+\left(\frac{E}{E_{break}}\right)^{\gamma-\alpha}}$$
To Do: compare Crab run with the same elevation range
<h1>Spectra fitting
Can you write Python code for it?

home = expanduser("~")
gc_dir = home + "/Dropbox/GalacticCenter/"

data = np.genfromtxt(gc_dir+"/spectralPoints/SgrA_4tels_noPoor_dNdE_TeV.txt")

E_array = np.array(data[:,0], dtype=np.float64)
flux_obs_array = np.array(data[:,1], dtype=np.float64)
#flux_err_array = np.array(data[:2], dtype=np.float64)

logE = list(map(lambda l: math.log10(l), E_array))
logF = list(map(lambda l: math.log10(l), flux_obs_array))

flux_powlaw = dNdE_powerlaw(flux_obs_array, 1.0, 6.113e-8, 2.57)

pyplot.plot(E_array, flux_obs_array, 'bo')
pyplot.plot(E_array, flux_powlaw)
pyplot.xlabel(r"$E\left(TeV\right)$")
pyplot.ylabel(r"flux $\frac{gamma}{s}$") # gamma / m^2*s*TeV
pyplot.xscale('log')
pyplot.yscale('log')
#pyplot.xlim(1.5e0, 5.e1)
#pyplot.ylim(1.e-13, 1.e-7)

powlaw_guess = (1., 1e-9, 2.5)
powlaw_bounds = ((0.5,20.), (1.e-7,1.e-11), (1.5,4.5))
p_opt_powlaw, p_cov_powlaw = optimize.curve_fit(dNdE_powerlaw, E_array, flux_obs_array, p0=powlaw_guess)

print(tuple(p_opt_powlaw))

#res = optimize.minimize(err, powlaw_guess )#bounds=powlaw_bounds)

cutoff_guess = (15, 1e-9, 2.5)
p_opt_cutoff, p_cov_cutoff = optimize.curve_fit(dNdE_cutoff, E_array, flux_obs_array, p0=cutoff_guess, maxfev=2500)
print(p_opt_cutoff)

broken_guess = (15, 1e-9, 2.5, 2.)
p_opt_broken, p_cov_broken = optimize.curve_fit(dNdE_broken, E_array, flux_obs_array, p0=broken_guess, maxfev=2500)
print(p_opt_broken)

flux_array_fit = dNdE_powerlaw(E_array, *p_opt_powlaw)

pyplot.plot(E_array, flux_obs_array, 'bo') # star because it is a map
pyplot.plot(E_array, flux_array_fit)
#pyplot.plot(E_array, )
pyplot.xlabel(r"$E\left(TeV\right)$")
pyplot.ylabel(r"flux $\frac{gamma}{s}$") # gamma / m^2*s*TeV
pyplot.xscale('log')
pyplot.yscale('log')
pyplot.xlim(1.5e0, 5.e1)
pyplot.ylim(1.e-13, 1.e-7)


I am working on the file "GC/flux_fit_and_integral.ipynb".
The first task is:

Source
Flux from 2TeV to Infiniti

Sgr A*
$$1.393\times10^{-10}$$

Crab
$$8.01\times10^{-10}$$

Ratio
$$0.1739$$

$$Flux_{SgrA}={{a}}\frac{gamma}{{m^2s}}$$

 Misc Code
Can you write Python code for it?

err = lambda p: np.mean((powerlaw(E_array, *p)-F_array)**2)

def powerlaw(E, *params): # energy (TeV), E0, norm, index
    E0, N0, A = params
    dNdE = N0*np.power(E/E0,-A)
    return dNdE

xarray = []
for i in x:
    xarray.append(i)

logE = []
for i, v in enumerate(E_array):
    logE.append(math.log10(v))

logF = []
for i, v in enumerate(F_array):
    logF.append(math.log10(v))

par = (5, 5)
print(type(par))
brack = [5, 5]
print(type(brack))

optimize.minimize?


I am working on the file "GC/flux_fit_and_integral.ipynb".
The first task is:
The above fit is not weighted by statistics of each point
<h2>Integral Flux
Can you write Python code for it?

# set the fit parameters for each source
norm_SgrA = 6.113e-8
index_SgrA = 2.57
norm_Crab = 3.157e-7
index_Crab = 2.525

Edec = 7 # estimate of decorrelation energy in TeV

Emin_Crab = 0.5 # TeV
Emin_SgrA = 2.0 # TeV
Emax_both = 40. # TeV

gamma_rate_SgrA = 0.155 # gamma / min
gamma_rate_Crab = 2.63  # gamma / min

# integrate the differential energy distribution for Sgr A*
int_tuple_SgrA = integrate.quad(dNdE_powerlaw, Emin_SgrA, math.inf, args=(Edec, norm_SgrA, index_SgrA))
int_flux_SgrA = int_tuple_SgrA[0]
int_upperBoundErr_SgrA = int_tuple_SgrA[1]

# now for the Crab with the same bounds as for Sgr A*
int_tuple_Crab_1 = integrate.quad(dNdE_powerlaw, Emin_SgrA, math.inf, args=(Edec, norm_Crab, index_Crab))
int_flux_Crab_1 = int_tuple_Crab_1[0]
int_upperBoundErr_Crab_1 = int_tuple_Crab_1[1]

# do the Crab with its full, larger energy range
int_tuple_Crab_2 = integrate.quad(dNdE_powerlaw, Emin_Crab, math.inf, args=(Edec, norm_Crab, index_Crab))
int_flux_Crab_2 = int_tuple_Crab_2[0]
int_upperBoundErr_Crab_2 = int_tuple_Crab_2[1]

print(int_flux_SgrA)
print(int_flux_Crab_1)
print(int_flux_Crab_2)

print("The ratio of Sgr A* flux to the Crab flux, integrated from 2TeV to 40TeV for both:")
print(int_flux_SgrA/int_flux_Crab_1)

print("The ratio of Sgr A* flux to the Crab flux, integrated from 0.5TeV to 40TeV for Crab")
print(int_flux_SgrA/int_flux_Crab_2)

print("The ratio of Sgr A* gamma rate to Crab gamma rate, as reported by VEGAS")
print(gamma_rate_SgrA/gamma_rate_Crab)


I am working on the file "GC/flux_lvl_detection_rough.ipynb".
The first task is:
$$ \frac{fN-N}{\sqrt{fN}} = t $$
$$ \left(f-1\right)^2N^2 = t^2fN $$
Can you write Python code for it?


I am working on the file "GC/flux_lvl_detection_rough.ipynb".
The first task is:
Finding rough minimum flux level required to detect
Can you write Python code for it?

sig = 34. # roughly
time = 6406. / 60. # hours
sens = sig / np.sqrt(time) # sensitivity
t = (5./sens)**2 # time required to find 5 sigma

print(sens)
print(t)


I am working on the file "GCC_FaciesClassification/01 - Facies Classification - GCC-VALIDATION.ipynb".
The first task is:
Feature construction and data clean-up.
    1. Z-score normalisation of data.
    2. Group each of the measurement parameters into quartiles. Most of the classification methods find data like this easier to work with.
    3. Create a series of 'adjacent' parameters by looking for the above and below depth sample for each well. Create a series of features associated with the above and below parameters.
Can you write Python code for it?

train_columns = train.columns[1:]
std_scaler = preprocessing.StandardScaler().fit(train[train_columns])

train_std = std_scaler.transform(train[train_columns])

train_std_frame = train
for i, column in enumerate(train_columns):
    train_std_frame.loc[:, column] = train_std[:, i]

train = train_std_frame
master_columns = train.columns[4:]

def in_range(row, vmin, vmax, variable):
    if vmin <= row[variable] < vmax:
        return 1
    else:
        return 0

for i, column in train[master_columns].iteritems():
    ds = np.linspace(0, 1.0, 5)
    quantiles = [column.quantile(n) for n in ds]

    for j in range(len(quantiles) - 1):
        train[i + '_{0}'.format(j)] = train.apply(lambda row: in_range(row, ds[j], ds[j + 1], i), axis = 1)

master_columns = train.columns[4:]

above = []
below = []
for i, group in train.groupby('Well Name'):

    df = group.sort_values('Depth')
    u = df.shift(-1).fillna(method = 'ffill')
    b = df.shift(1).fillna(method = 'bfill')

    above.append(u[master_columns])
    below.append(b[master_columns])

above_frame = pd.concat(above)
above_frame.columns = ['above_'+ column for column in above_frame.columns]
below_frame = pd.concat(below)
below_frame.columns = ['below_'+ column for column in below_frame.columns]

frame = pd.concat((train, above_frame, below_frame), axis = 1)


I am working on the file "GCI_kadai00.ipynb".
The first task is:
課題2への回答として1900年から2001年までのうるう年を探してみる。
Can you write Python code for it?

for i in range(1900,2001):
    leap_year(i)


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
(optional) Save/load data from local pkl-file
Sometime it is convients to have a saved copy of query result.
Here is two helpers to save/load GDELT dataframe to local pkl-file
Can you write Python code for it?

# Save dataframe to the file to avoid often queries
gdelt_df.to_pickle('./tf_gdelt_quad_countries_2000_now.pkl')

# Load dataframe from the file
gdelt_df = pd.read_pickle('./tf_gdelt_quad_countries_2000_now.pkl')
print gdelt_df.info()
gdelt_df.describe()


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
Below is a plot of our new dataset.
It is clear that data is centered around zero and have a comparable values.
Machine learning algorithms will work much better on this dataset.
Can you write Python code for it?

# Plot Growth Rate to have a look at the LOG-normalized data
_ = event_by_codes_log.resample(rule='1M').mean().plot(figsize=(15, 5), fontsize=20)


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
Classification Model
Let's try to make a more general prediction. instead of trying to predict actual Growth Rate value, let's predict just Growth direction:
Will Growth Rate increase or decrease?
To do that, we convert Growth Rate to 2 classes: 1 - INCREASE, 0 - DECREASE or not changing.
All other steps will be the same as for Regression Model
Can you write Python code for it?

# RNN Classification
# Helper function to convert Growth Rate value into class
def value_to_class(x):
  if x<=0:
    return 0.0
  else:
    return 1.0

# Prepare X (all quads classes' Growth Rate) and y (Quad Class '4' - Material Conflict Increase/Decrease)
X = event_by_codes_log.values
y = event_by_codes_log['4'].shift(periods=-1)
y = y.apply(value_to_class).values

# There no info from the past for the very first day.
# Let's drop out first day.
X = X[1:-1]
y = y[1:-1]

# Verify that shape of both sets are compatible
print("X shape: {}, y shape: {}".format(X.shape, y.shape))

# Split dataset to Train and Test partion in a ratio 80/20.
# All model training will be done on Train part, and only evalution will be done on Test part.
test_split = int(0.2 * len(X))
X_train = X[:test_split]
y_train = y[:test_split]
X_test = X[test_split:]
y_test = y[test_split:]

# Helper function to prepare input data for RNN
def rnn_input_fn(x):
  return tf.split(1, 4, x)  # 4 - number of features in X vector

# Validation monitor to do early stopping (https://en.wikipedia.org/wiki/Early_stopping)
val_monitor = skflow.monitors.ValidationMonitor(X_test, y_test,
                                                early_stopping_rounds=200,
                                                n_classes=2,
                                                print_steps=100)
# RNN Tensorflow classifier model
classifier = tf.contrib.learn.TensorFlowRNNClassifier(
                rnn_size=20,
                cell_type="lstm",
                n_classes=2,
                input_op_fn=rnn_input_fn,
                num_layers=7,
                steps=1000)

# Run  model training
classifier.fit(X_train, y_train, monitor=val_monitor, logdir='./rnn_clf/')

# Predict Growth Rate class on Test part
y_pred = classifier.predict(X_test)
accuracy_score = metrics.accuracy_score(y_test, y_pred)

# Measure RNN Classifier performance on Test part (see more: https://en.wikipedia.org/wiki/F1_score#Diagnostic_Testing)
print("* Classification report *\n{}".format(
   ...


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
It is quite clear now that we have some signal in our data.
But let's try one more useful and fast visual test to be sure that we are on the right way.
Scatter Matrix allows us to see if there is a correlation between Quad Classes. See more details here: http://www.originlab.com/doc/Tutorials/ScatterMatrix
Scatter plot of perfectly correlated variables will be close to the line, and for the opposite - shapeless cloud.
Our scatter plot matrix shows that all quad classes are correlated and it's a good sign.
Can you write Python code for it?

_ = scatter_matrix(event_by_codes_log, figsize=(20, 20), diagonal='kde', alpha=0.2)


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
Prepare dataset for analysis
Here we are converting SQLDATE string with datetime stamp to datetime index.
Then we are extracting only records for Egypt (country code 'EG'). This will be out dataset for further analysis and machine learning applications
Can you write Python code for it?

# We will create new column for datetime series with helper function
def date_from_int(date_int):
  """
  Helper function to convert date from int to datetime object.
  E.g. 20160125 -> 2016-01-25 datetime obj
  """
  date_str = str(date_int)
  date = pd.to_datetime(date_str, format='%Y%m%d')
  return date

# Make a new index with date
gdelt_df['Date'] = gdelt_df['SQLDATE'].apply(date_from_int)
gdelt_df.index = gdelt_df['Date']
gdelt_df.drop(['SQLDATE'], axis=1, inplace=True)

# Prepare dataset  - take events for country EG
def split_events(gdelt_df, measure_name = 'NumArticles', country = 'EG'):
  event_root_codes = gdelt_df.QuadClass.unique()
  event_series = [gdelt_df[(gdelt_df.QuadClass==event_code) & (gdelt_df.Actor1Geo_CountryCode==country)][[measure_name]]
                 for event_code in event_root_codes]
  event_by_codes = pd.concat(event_series, axis=1).sort_index()
  event_by_codes.columns = map(str, event_root_codes)
  event_by_codes = event_by_codes.fillna(method='ffill')
  event_by_codes = event_by_codes.fillna(method='bfill')
  return event_by_codes

event_by_codes = split_events(gdelt_df)


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
Recurrent Neural Network in TensorFlow
From our everyday life's experience we know that if there is a buzz in news, it has a tail at leat for several days.
So let's see if information about the past Growth Rate can help predict the future Growth Rate.
Or, more precisely, let's try to
predict Material Conflict Growth Rate using information about all 4 quads' Growth Rate from the past.

Reccurent Neural Network shows good perfomance in the tasks where prediction of new value depends in the previous values (more http://colah.github.io/posts/2015-08-Understanding-LSTMs/).
So let's use TensorFlow build-in Datalab to train RNN regression model to predict Material Conflict Growth Rate using information from the last 7 days.
Regression Model
Can you write Python code for it?

# RNN regression
# Prepare X (all quads classes' Growth Rate) and y (Quad Class '4' - Material Conflict real Growth Rate)
X = event_by_codes_log.values
y = event_by_codes_log['4'].shift(periods=-1).values

# There no info from the past for the very first day.
# Let's drop out first day.
X = X[1:-1]
y = y[1:-1]

# Verify that shape of both sets are compatible
print("X shape: {}, y shape: {}".format(X.shape, y.shape))

# Split dataset to Train and Test partion in a ratio 80/20.
# All model training will be done on Train part, and only evalution will be done on Test part.
test_split = int(0.2 * len(X))
X_train = X[:test_split]
y_train = y[:test_split]
X_test = X[test_split:]
y_test = y[test_split:]

# Helper function to prepare input data for RNN
def rnn_input_fn(x):
  return tf.split(1, 4, x)  # 4 - number of features in X vector

# Validation monitor to do early stopping (https://en.wikipedia.org/wiki/Early_stopping)
val_monitor = skflow.monitors.ValidationMonitor(X_test, y_test,
                                                early_stopping_rounds=200,
                                                n_classes=0,
                                                print_steps=100)

# RNN Tensorflow regression model
regressor = tf.contrib.learn.TensorFlowRNNRegressor(rnn_size=20,              # How many nodes in a layer (aka cells)
                                                    cell_type="lstm",         # Cell type (lstm has better 'memory', https://en.wikipedia.org/wiki/Long_short-term_memory)
                                                    input_op_fn=rnn_input_fn, # Input data transform function
                                                    num_layers=7,             # How many layers RNN will have (think '# of days in the past RNN will look into')
                                                    steps=1000)               # Max learning steps
# Run model training
regressor.fit(X_train, y_train, monitor=val_monitor)

# Predict Growth Rate values on Test part
y_pred =...


I am working on the file "GDC Meetup Paris GDELT-skflow-RNN.ipynb".
The first task is:
There are several very visible charactersrics of our dataset:
1. There are much more data after 2010, then before. In order of magnitude.
2. Number of articels for each quad behave in a similar way: in 2011 there is a peak for "Quad Class 1" and there are peaks fol all other classes. Seems like we can try to see if there is a pattern in our dataset.
Really big difference in the numbers over the time can make further analysis difficult.
We need "normalized" data.
One way to do data normalization is to apply a logarithme function.
Log function has 2 advantage:
1. It smooths peak values, and highlights small values
2. It "centralizes" data around zero
Then it is importnat to remeber, that we have events with vector in time. We should exploit date-time information.
Let's use time vector to transform our data from raw Number of Articles to the relative Growth Rate. And then normalize Growth Rate with Log function.
For example, in pseudocode our new measure Growth Rate will be:
Growth_Rate = log( Num_Articles(Day-1) / Number_Arcticles(Day) )
Can you write Python code for it?

# Let's use time component and transform data with log
def log_scale_dataframe(df):
  new_df = pd.DataFrame()
  for col in df.columns:
    new_df[col] = np.log(df[col] / df[col].shift() )
  return new_df

event_by_codes_log = log_scale_dataframe(event_by_codes)


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
But I don't want scare people with spooky, scary math

Iris Dataset
The Iris dataset is a small dataset of 150 Iris flowers. For each data point we have the following features:

1.Sepal Length (cm)
2.Sepal Width (cm)
3.Petal Length (cm)
4.Petal Width (cm)

We want to predict whether the Iris is of type:

1.Setosa
2.Vesicolor
3.Virginica

Photo Credits @ Kaggle
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
But let's say we have an arrangement of balls that no line can separate. What can we do?
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
But now if we add another red ball, the boundary does not hold
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Could this problem have been avoided? Is there a way to choose the line in the initial problem to prevent this?
The answer is yes! If we choose a line that has a large margin, we can confidently label new data points.
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
For this example, we will only use the Sepal Length and Sepal Width feature, so we can plot the data in a 2D grid.
Can you write Python code for it?

x_ind = 0
y_ind = 1

X = iris.data[:,(x_ind, y_ind)]
labels = iris.target

print X.shape
print labels.shape

# this formatter will label the colorbar with the correct target names
formatter = plt.FuncFormatter(lambda i, *args: iris.target_names[int(i)])

plt.scatter(X[:, x_ind], X[:, y_ind],
            c=labels, cmap=plt.cm.get_cmap('RdYlBu', 3))
plt.colorbar(ticks=[0, 1, 2], format=formatter)
plt.clim(-0.5, 2.5)
plt.xlabel(iris.feature_names[x_ind])
plt.ylabel(iris.feature_names[y_ind]);


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
If we want to separate the two cluster of points with lines, there are several lines we can choose from. How would we determine which line is the best line.
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
K-NN is easy to understand and implement, but because it requires us to test each new data point against each point in the training set, it is slow and scales poorly to larger datasets.
Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
K-Nearest Neighbors
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Let's create a new dataset
Can you write Python code for it?

X, y = make_blobs(n_samples=50, centers=2,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring');


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Let's look at a different dataset. In this dataset, it is obvious that the data points are not linearly separable.
Can you write Python code for it?

from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')
plot_svc_decision_function(clf);


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Like the previous algorithms, the decision tree can partition up the space of points.
Can you write Python code for it?

def visualize_tree(estimator, X, y, boundaries=True,
                   xlim=None, ylim=None):
    estimator.fit(X, y)

    if xlim is None:
        xlim = (X[:, 0].min() - 0.1, X[:, 0].max() + 0.1)
    if ylim is None:
        ylim = (X[:, 1].min() - 0.1, X[:, 1].max() + 0.1)

    x_min, x_max = xlim
    y_min, y_max = ylim
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
                         np.linspace(y_min, y_max, 100))
    Z = estimator.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure()
    plt.pcolormesh(xx, yy, Z, alpha=0.2, cmap='rainbow')
    plt.clim(y.min(), y.max())

    # Plot also the training points
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='rainbow')
    plt.axis('off')

    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)
    plt.clim(y.min(), y.max())

    # Plot the decision boundaries
    def plot_boundaries(i, xlim, ylim):
        if i < 0:
            return

        tree = estimator.tree_

        if tree.feature[i] == 0:
            plt.plot([tree.threshold[i], tree.threshold[i]], ylim, '-k')
            plot_boundaries(tree.children_left[i],
                            [xlim[0], tree.threshold[i]], ylim)
            plot_boundaries(tree.children_right[i],
                            [tree.threshold[i], xlim[1]], ylim)

        elif tree.feature[i] == 1:
            plt.plot(xlim, [tree.threshold[i], tree.threshold[i]], '-k')
            plot_boundaries(tree.children_left[i], xlim,
                            [ylim[0], tree.threshold[i]])
            plot_boundaries(tree.children_right[i], xlim,
                            [tree.threshold[i], ylim[1]])

    if boundaries:
        plot_boundaries(0, plt.xlim(), plt.ylim())


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Logistic Regression is the classification cousin of linear regression. Our boundaries are made up of lines.
In logistic regression, we predict the likelihood of a given label y based on our features.
Can you write Python code for it?

clf = LogisticRegression(C=1e5)
clf.fit(X, labels)


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Logistic regression is quick and simple, but sometimes does not perform well due to the strict partitioning of the decision boundary.
Support Vector Machines: Motivation
Let's say we have red balls and blue balls on a table, and we want to separate them.
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
One of the big breakthroughs in machine learning is the kernel function. Using the kernel function, we can map the input data into a new feature space.
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
SVMs are a mathematically beautiful algorithm that are guaranteed to find a wide margin decision boundary, but they do not perform well on large datasets due to scaling issues.
Decision Trees
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Support Vector Machine: Example
Can you write Python code for it?

from sklearn.datasets.samples_generator import make_blobs


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Ta-Da!
Can you write Python code for it?


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
The idea of combining multiple models together is a good idea, but as we can see from above, it has some overfitting issues.
Ensemble Method
In video games, you pick party members to cover your weaknesses.

In machine learning, we want to pick classifiers that cover each other's "weaknesses".
That's the idea behind a random forest. Using a meta-heuristic known as the ensemble method, we can average the results of a group of over-fitted decision tree to create a random forest that has more generalizable results.
Can you write Python code for it?

clf = RandomForestClassifier(n_estimators=100, random_state=0)
visualize_tree(clf, X, y, boundaries=False);


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
These points are called the Support vectors. The support vectors are used by the algorithm to determine the margin boundary. Now, let's look at a more interactive demo.
Can you write Python code for it?

from IPython.html.widgets import interact

def plot_svm(N=10):
    X, y = make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    clf = SVC(kernel='linear')
    clf.fit(X, y)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')
    plt.xlim(-1, 4)
    plt.ylim(-1, 6)
    plot_svc_decision_function(clf, plt.gca())
    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=200, facecolors='none')

interact(plot_svm, N=[10, 200], kernel='linear');


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
This tutorial is based on example in Jake Vanderplas' PyCon 2015 tutorial.
What is Machine Learning
Machine Learning is a subfield of computer science that utilizes statistics and mathemathical optimization to learn generalizable patterns from data. In machine learning, we develop models with adjustable parameters, so we can learn the parameters that best fit the data. In this tutorial, I will cover five commonly used algorithms in supervised learning.
It's a simple idea that has led to some impressive results.

The goal of this talk is for you to understand what these algorithms are learning from the data.
Many machine learning tutorials just show classifier's performance with no in-depth explanation of how the algorithm works.

Photo Credits @ Randy Olsen
Imagine you were taught how to build a birdhouse, but never taught how to use a hammer.
Can you write Python code for it?

YouTubeVideo("IFACrIx5SZ0", start = 85, end = 95)


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
We can plot the decision boundary of the SVM along with its margin.
Can you write Python code for it?

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')
plot_svc_decision_function(clf);


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
We created a K-NN classifier that uses 13 neighbors to decide each test point. Let's see how it would label the space.
Can you write Python code for it?

x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1

xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                        np.arange(y_min, y_max, h))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.get_cmap('RdYlBu', 3), alpha = 0.2)

    # Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=labels, cmap=plt.cm.get_cmap('RdYlBu', 3), s = 40)

plt.colorbar(ticks=[0, 1, 2], format = formatter)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xlabel(iris.feature_names[x_ind])
plt.ylabel(iris.feature_names[y_ind]);


I am working on the file "GDG_MachineLearning.ipynb".
The first task is:
Well, we can put a line down the middle that separates them.
Can you write Python code for it?


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
$\lambda_t$: Following appendix equation A.9 (List, 1949), I can estimate $\lambda_t\ [J m^{-1} s^{-1} K^{-1}]$ as function of $T_a [K]$:
$$\lambda_t = 0.000063 * T_a + 0.00673$$
Can you write Python code for it?

lambda_t = 0.000063 * Tair_K + 0.00673
print(D_m2ps, lambda_t, D_m2ps/lambda_t)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
$\rho_{T_a}$, $\rho_{sat(T_s)}$ : To estimate water vapour density $[kg\ m^{-3}]$ in the free atmosphere, I calculate the water vapour pressure, e $[Pa]$, using Dingman (2015), eq 3.9a, p.113:
$$e = \frac{RH}{100} * 611\ exp\left(\frac{17.27\ T}{T + 237.3}\right)$$
where:
RH = relative humidity = 70%
T = air temperature, $°C$
Can you write Python code for it?

RH = 60.
saturated_vapour_pressure_Pa = 611 * np.exp((17.27 * Tair_C)/(Tair_C + 237.3))
free_air_vapour_pressure_Pa = (RH / 100.) * saturated_vapour_pressure_Pa
print("saturated vapour pressure [Pa] = %.2f Pa" % (saturated_vapour_pressure_Pa))
print("free air vapour pressure [Pa] for %d%% RH = %.2f Pa" % (
        RH, free_air_vapour_pressure_Pa))


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
And finally, using b and c from middle plot (hourly) of Fig. 6:
Can you write Python code for it?

b = 2.50286
c = 0.125006
rainfall_fraction = 1. / (1. + (b * c**T_i))
precip_mm = 5
print(rainfall_fraction, rainfall_fraction * precip_mm, \
      ((1 - rainfall_fraction) * precip_mm))


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
Finally, the subcanopy net radiation $Q^\ast = K^\ast + L^\ast$ = 14.50 $W m^{-1}$.
Question 2
Use equations presented in the paper by Harder and Pomeroy (2013) to estimate precipitation phase:
During a precipitation measurement of 5 mm over an hour into a remote unattended Alter-shielded weighing precipitation gauge in a forested clearing you measure an air temperature of +1.0 C, RH of 60%, with very low wind speed. What is the water equivalent depth of snowfall? What is the depth of rainfall?
Harder and Pomeroy (2013) derive a psychrometric equation to derive phase change, expressed as rainfall fraction, $f_r$, as a sigmoidal function:
$$f_r(T_i) = \frac{1}{1 + b * c^{T_{i}}}$$
where:
$b$, $c$ = best fit coefficients, both are functions of time scale of measurement (15-min, hourly, daily)
$T_i$ = hydrometeor temperature
An iterative solution for $T_i$, Appendix eq. A.5, is:
$$T_i = T_a + \frac{D}{\lambda_t}  L (\rho_{T_a} - \rho_{sat(T_i)})$$
where:
$T_a$ = air temperature $[K]$
$D$ = diffusivity of water vapour in air $[m^2 s^{-1}]$
$\lambda_t$ = thermal conductivity of air $[J\ m^{-1} s^{-1} K^{-1}]$
$L$ = latent heat of vaporisation or sublimation $[J\ kg^{-1}]$
$\rho_{T_a}$ = water vapour density in the free atmosphere $[kg\ m^{-3}]$
$\rho_{sat(T_i)}$ = saturated water vapour density at the hydrometeor surface $[kg\ m^{-3}]$
Since the wind speed is low, I will assume thermodynamic equilibrium.
I am given $T_a$ = 1.0 °$C$ = 274.15 $K$.  For each of the next terms:
$D$: Following the appendix equation A.6 (Thorpe and Mason, 1966), I can estimate $D [m^2 s^{-1}]$ as function of $T_a [K]$:
$$D = 2.06 * 10^{-5} * \left(\frac{T_a}{273.15}\right)^{1.75}$$
Can you write Python code for it?

C_to_K_offset = 273.15
Tair_C = 1.0
Tair_K = Tair_C + C_to_K_offset
D_m2ps = 2.06e-5 * (Tair_K/273.15)**1.75
C_to_K_offset, Tair_K, D_m2ps


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
Linearly interpolate $K_A$ at 56 N to July 2 between Jun22 and Jul15:
Can you write Python code for it?

import datetime
jun22_doy = datetime.datetime(2017, 6, 22).timetuple().tm_yday
jul2_doy = datetime.datetime(2017, 7, 2).timetuple().tm_yday
jul15_doy = datetime.datetime(2017, 7, 15).timetuple().tm_yday
print(jun22_doy, jul2_doy, jul15_doy)

KA_56N_jul2 = linear_model_value_at(jul2_doy,
                                    jun22_doy, KA_56N_jun22,
                                    jul15_doy, KA_56N_jul15)
KA_56N_jul2


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So I estimate $\tau$ = 0.12. This is consistent with my notes from class, which indicate that a good rule of thumb for $\tau$ in forests is 0.1.
For daily average incoming solar radiation and incoming longwave radiation fluxes to the sub-canopy floor under an adjacent pine canopy at this location on this day, incoming shortwave will be $K^\ast$ times the transmittance, and net shortwave will be incoming shortwave times $(1 - \alpha)$:
Can you write Python code for it?

subcanopy_shortwave_down = K_net * forest_tau
net_subcanopy_shortwave = subcanopy_shortwave_down * (1. - albedo)
print("shortwave down to subcanopy %.2f W/m^2" % subcanopy_shortwave_down)
print("net subcanopy shortwave %.2f W/m^2" % net_subcanopy_shortwave)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So I estimate average net radiation in this location over the day to be 159.67 $W m^{-2}$.
Q1. Part v
<i>v) what would the daily average incoming solar radiation and incoming longwave radiation fluxes be to the sub-canopy floor under an adjacent pine canopy at this location on this day, if the pine canopy had a leaf area index of 2.1 and a sky view factor of 0.2? If the sub-canopy surface temperature and albedo are the same as in the field, what is the sub-canopy net radiation?</i>
Per Pomeroy and Dion, 1996, the transmittance through a forest canopy can be modeled with an extinction coefficient:
$$\tau = exp^{\frac{-Q_{ext}\ {LAI}'}{\sin\theta}}$$
where:
$\tau$ = transmittance through forest canopy
$Q_{ext}$ = extinction efficiency (dimensionless)
$LAI'$ = effective winter leaf area index
$\theta$ = solar angle above the horizon
From Pomeroy lecture on Day 1, slide 42, apparently $Q_{ext}$ cancels $\sin\theta$ , so $Q_{ext} \approx \sin\theta$ so their ratio will $\rightarrow$1. and $\tau$ is:
Can you write Python code for it?

eff_lai = 2.1
forest_tau = exp(-1 * eff_lai)
forest_tau


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So I estimate daily average clear sky incoming longwave radiation, $L\downarrow$, to be 307.49 $W m^{-2}$, which is greater than the incoming shortwave from part i).
For the actual daily average incoming longwave radiation to this field, since it there are only 7 hours of bright sunshine, I think the actual atmospheric emissivity will need to be adjusted upward, to account for increased emission from the clouds.  I will use the adjustment used in Sicart et al 2006, equation (9).  The adjustment for emissivity from clouds is:
$$\varepsilon_{cloudy} = \varepsilon_{clear}(1 + 0.44 RH - 0.18 \tau_{atm})$$
where:
$RH$ = relative humidity (as a percent)
$\tau_{atm}$ = shortwave transmissivity of the atmosphere
Shortwave atmospheric transmissivity is the ratio of my answers from part i) and ii):
Can you write Python code for it?

tau_atm = K_down / KA_56N_jul2
print("KA = %.2f, K_down = %.2f, tau_atm = %.2f" %(
        KA_56N_jul2, K_down, tau_atm))


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So I estimate the friction velocity $u^*$ = 0.0098 $m\ s^{-1}$
Using Dingman eq. 3.35, the momentum flux is:
$$F_M = - \rho_a\ {u^*}^2$$
Letting
$\rho_a$ = 1.2 $kg\ m^{-3}$
$u^*$ = friction velocity $m\ s^{-1}$
Can you write Python code for it?

# Units kg/m3 * m2/s2 = kg/m/s2  ?? flux units make sense,
#                                   kg m/s per unit area per unit time
rho_a = 1.2 # kg/m-3
momentum_flux = -1 * rho_a * friction_u
print("Momentum flux is %f kg/m/s^2" % momentum_flux)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So given a gauge precip measurement of 5 $mm$, I estimate it fell as 3.8 $mm$ of rainfall, and 1.2 $mm$ water equivalent from snowfall.
Question 3
You are measuring air temperature and water vapour content over a natural grassland (roughness length = 0.03 m) to get an idea of how available energy is being used at this site. You have installed the minimum of two levels of measurements of wind speed, temperature and humidity. The daily average measurements are (heights above the ground):</i

- 1-m height, wind speed is 0.6 $m\ s^{-1}$, air temperature 20 $^oC$, and vapour pressure 2.0 $kPa$;
- 2-m height, wind speed is 0.62 $m\ s^{-1}$, air temperature 19 $^oC$, and vapour pressure 1.5 $kPa$.
Assume Pa = 101.3 kPa, $\rho_a$ = 1.2 $kg\ m^{-3}$, and $c_p$ = 1005 $J\ kg^{-1} K^{-1}$ and $L_v$ = 2.454 $MJ\ kg^{-1}$.
(a) Estimate a $z_0$ for momentum, heat and water vapour from the vegetation height. Then using bulk transfer flux-gradient calculations (example in Helgason and Pomeroy, 1995) and ignoring stability corrections find $Q_E$ and $Q_H$.
(b) Calculate the footprint representative of these measurements which accounts for 80% of the flux. Recall that you will first need to calculate u*.
Per Helgason and Pomeroy, 2005, a typical method to estimate momentum roughness length $z_{0m}$ is to plot $ln(z)$ vs. $\bar{u}$ for neutral conditions and then determine the value of $z_{0m}$ as the y-intercept where $\bar{u}$ =0.
Can you write Python code for it?

import numpy as np

u1 = 0.6  # m/s
u2 = 0.62 # m/s
z1 = 1.0  # m
z2 = 2.0  # m

# quick linear interpolation function
# returns slope and intercept
def linear_model_parms(x1, y1, x2, y2):
    slope = (y2 - y1) / (x2 - x1)
    # y = mx + b ==> b = y - mx
    intercept =  y1 - (slope * x1)
    return {'m':slope, 'b':intercept}

parms = linear_model_parms(u1, np.log(z1), u2, np.log(z2))
print("y-intercept: ln(z) = %f" % parms['b'])

# Make the plot:
plt.rc('text', usetex=True)
fig, ax = plt.subplots(1)
ax.plot([u1, u2], [np.log(z1), np.log(z2)], "co-")
#ax.set_title("$ln(z) vs. \bar{u}")
ax.set_xlabel('Averge wind speed $(m\ s^{-1})$')
ax.set_ylabel("$ln(z)$")
ax.set_xlim([0,0.63])
ax.set_ylim([-20, 1.0])

plt.show()
fig.savefig('HW1.3_fig1.png')


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So my calculated momentum roughness length is about 10 Angstroms, which seems impossibly small.  But I am unsure as to what I've done wrong, here.
Taking a different approach, the problem states that the roughness length is 0.03 m.  Using Dingman eq (3.28 and 3.29), p. 122, if this roughness length is $z_0$, then the average vegetation height $z_{veg}$ = 0.3 m = 30 cm , which seems reasonable for a "natural grassland".  And, the zero-plane displacement height, $z_d$ is therefore 0.7 * 0.3 = 0.21 m.
Using equation 3.30b to estimate friction velocity, $u^*$:
$$u^* = \frac{\kappa [u(z_2) - u(z_1)]}{ln(\frac{z_2 - z_d}{z_1 - z_d})}$$
where:
$\kappa$ = 0.4
Can you write Python code for it?

z_veg = 10. * 0.03
z_d = 0.7 * z_veg
kappa = 0.4
friction_u = (kappa * (u2 - u1)) / np.log((z2 - z_d) / (z1 - z_d))
print("friction velocity u* = %f m/s" % friction_u)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So the estimated daily average solar radiation to the top of the Earth's atmosphere at 56$^{\circ}$ N on July 2nd is 466.90 $W m^{-2}$.
Q1. Part ii
ii) if there are 7 hours of bright sunshine, what is the average incoming solar radiation to the surface of a flat unvegetated field at this location on this day?
Eq 7. from "Calculating Evaporation Notes" gives the equation for mean incoming shortwave radiation $K\downarrow [W m^{-2}]$ as:
$K\downarrow = K_A[a + b(\frac{n}{N})]$
Looking at latitudes of locations in Table B, I think the closest location would be that of Central SK, Canada from Mudiare(1985). I will use the rainfree values for a and b.
Let:

a = 0.27 (From Table B, Mudiare, rain-free, May-Aug, central SK)
b = 0.47 (From Table B, Mudiare, rain-free, May-Aug, central SK)
n = 7. (actual sunshine hours)
N = maximum possible number of sunshine hours

Estimate N from Table C, values given for latitude=56N, interpolate values from Jul1 - Jul5 to a value for Jul2, and remember to convert (hours,minutes) to decimal hours)

Date:  N at 56N
Jul1: 17h 31m
Jul5: 17h 25m

Can you write Python code for it?


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So this matches my expectation from class notes that $\varepsilon_{clear} < \varepsilon_{cloudy}$, and I estimate actual daily average incoming longwave radiation to this field as 376.84 $W m^{-2}$.
Q1. Part iv
<i>iv) if the surface temperature is 20°C and albedo is 0.15 what is the average net radiation over the day?</i>
The average net radiation over the day is the sum of net shortwave and net longwave:
$$Q^\ast = K^\ast - L^\ast = K\downarrow(1 - \alpha) + L\downarrow - L\uparrow$$
where:
$K^\ast$ = net shortwave
$L^\ast$ = net longwave
$K\downarrow$ = incoming shortwave
$\alpha$ = shortwave albedo
$L\downarrow$ = incoming longwave
$L\uparrow$ = outgoing longwave from the terrain surface
Air temperature is not a factor for the incoming shortwave, so I can use incoming shortwave from part ii:
$K\downarrow = 213.88 W m^{-2}$
$\alpha = 0.15$
Can you write Python code for it?

albedo = 0.15
K_net = K_down * (1. - albedo)
K_net


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So using emissivity of the cloudy atmosphere in Stephan-Boltzmann:
Can you write Python code for it?

e_cloudy = e_clear * (1. + 0.44 * (RH/100.) - 0.18* (tau_atm))
longwave_cloudysky = e_cloudy * sigma * T_K**4
print("Clear-sky emissivity %.2f" % e_clear)
print("Cloudy-sky emissivity %.2f" % e_cloudy)
print("Incoming longwave radiation %.2f" % longwave_cloudysky)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So using the Stephan-Boltzmann Law, let:
$\varepsilon_{clear}(T=15°C,e=11.91mb)$ = 0.79
$\sigma$ = 5.67 x $10^{-8} W m^{-2} K^{-4}$
$T$ = 288.15 $K$
Can you write Python code for it?

sigma = 5.67e-08
longwave_clearsky = e_clear * sigma * T_K**4
print("Incoming longwave clearsky %.2f [W/m^2]" % longwave_clearsky)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
So with 7 hours of direct sunshine at this location, the mean incoming solar radiation to the surface of a flat unvegetated field at about 56$^{\circ}$ N on July 2nd is 213.88 $W m^{-2}$.  So atmospheric transmittance reduces the shortwave by more than half of that reaching TOA.
Q1. Part iii
<i>iii) if the mean daily air temperature is 15°C and the mean daily relative
humidity is 70%, what is the daily average clear sky incoming longwave
radiation and what is the actual daily average incoming longwave radiation to this field?
</i>
From our class notes, incoming longwave clear sky radiation from the atmosphere, $L_{0,clear} [W m^{-2}]$, is expressed by the Stephan-Boltzmann Law:
$L_{0,clear} = \varepsilon_{clear}(T,e) \sigma T^4$
Where apparent clear-sky emissivity, $\varepsilon_{clear}$, is a function of air temperature near the ground, $T [K]$, and water vapour pressure, $e [mb]$, near the ground. Brutsaert(1975) derived an empirical relationship for $\varepsilon_{clear}$ as a function of $T$ and $e$:
$$\varepsilon_{clear} = C(\frac{e}{T})^{1/m}$$
where:
$T$ = air temperature, $[K]$
$e$ = vapour pressure near the ground, $[mb]$
$C$ = 1.24
$m$ = 7
From lectures, I can use the Magnus Formula to approximate saturated water vapour pressure $e_s [hPa=mb]$, at 15-deg C:
$e_s [mb] = 6.1094 \exp{\frac{17.625 T}{T + 243.04}}$
where:
$T$ = air temperature $[°C] = 15.$
Can you write Python code for it?

T_C = 15. # air temperature, degrees C
es_15C = 6.1094 * np.exp((17.625 * T_C) / (T_C + 243.04))
print("Saturated water vapour pressure at 15°C is %.2f mb" % es_15C)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
Solving for hydrometeor temperature:
Can you write Python code for it?

T_i = Tair_C + (D_m2ps/lambda_t) * Lv_Jpkg * (
    free_air_vapour_density_kgpm3 - saturated_vapour_density_kgpm3)
print("hydrometeor temperature = %.2E deg-C" % T_i)


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
Solving for the corresponding height:
Can you write Python code for it?

print("Momentum roughness length z_0m = %e meters" % np.exp(parms['b']))


I am working on the file "GEOG827/Brodzik_assignment1.ipynb".
The first task is:
The incoming longwave to the subcanopy will come from a combination of the sky and the trees.  Sicart et al. 2006 (equation 6) found that terrain and vegetation emissions of longwave radiation could be represented as separate components weighted by sky view factor:
$$L = V_f L_0 + (1 - V_f) \varepsilon_s \sigma T_{s}^4$$
let:
$V_f$ = 0.2 (given)
$L_0$ = longwave from the cloudysky at 20°C from part iv, $W m^{-2}$
$\varepsilon_s$ = 0.98 emissivity of the forest, assume it is close to 1, from Sicart "terrain emissivity is close to 1 for snow and most natural surfaces"
$T_{s}$ = 20 + 273.15 $K$ (same temperature as surrounding field)
The outgoing longwave from the subcanopy floor will be the same as that from the surrounding field.
Can you write Python code for it?

sky_view_factor = 0.2
subcanopy_longwave_down = sky_view_factor * longwave_cloudysky_20 + \
       (1. - sky_view_factor) * surface_emissivity * sigma * T_K**4
net_subcanopy_longwave = subcanopy_longwave_down - longwave_up
print("longwave down to subcanopy %.2f W/m^2" % subcanopy_longwave_down)
print("longwave up from subcanopy %.2f W/m^2" % longwave_up)
print("net subcanopy longwave %.2f W/m^2" % net_subcanopy_longwave)


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Define functions:
(1) Get the log scale axis of pressure
(2) Get the temperature tendency variables from the model
Can you write Python code for it?

def zstar(lev):
    return -np.log(lev / climlab.constants.ps)
def get_tendencies(model):
    '''Pack all the subprocess tendencies into xarray.Datasets
    and convert to units of K / day'''
    tendencies_atm = xr.Dataset()
    tendencies_sfc = xr.Dataset()
    for name, proc, top_proc in climlab.utils.walk.walk_processes(model, topdown=False):
        if proc.time_type is not 'diagnostic':
            tendencies_atm[name] = proc.tendencies['Tatm'].to_xarray()
            tendencies_sfc[name] = proc.tendencies['Ts'].to_xarray()
    for tend in [tendencies_atm, tendencies_sfc]:
        #  convert to K / day
        tend *= climlab.constants.seconds_per_day
    return tendencies_atm, tendencies_sfc

yticks = np.array([1000., 750., 500., 250., 100., 50., 20., 10.])

plt.style.use('ggplot')
plt.figure( figsize=(8,8) )
plt.plot( Tglobal + 273.15, zstar(level),label='observations',color='C2',linestyle=':',linewidth=4)
plt.yticks(zstar(yticks),yticks)
plt.xlabel('Temperature (K)')
plt.ylabel('Pressure (hPa)')
plt.title('Global, annual mean sounding from NCEP Reanalysis')


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Let's look at what radiatively active gases are in this model.
Can you write Python code for it?

#  Volumetric mixing ratios
rad.absorber_vmr


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Let's see how the surface temperature changes as the number of layers increases.
Can you write Python code for it?

# number of layers
# it is equivalent to the amount of greenhouse gas in the atmosphere
layers = np.arange(100)
T_layer = nlayer(layers,S,albedo,4)

# plot number of layers vs. surface temperature
plt.plot(layers,T_layer)
plt.xlabel('number of layers')
plt.ylabel('Surface temperature (K)')


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Now let's get the temperature tendencies. A tendency is the rate of change of a quantity due to different processes; in this case radiation (shortwave and longwave) and (later on) convection. This allows us to quantify the contribution of different physical processes to the evolution of temperature: when a process contriutes to warming the atmosphere, its tendency is positive. When it contributes to cooling the atmosphere, its tendency is negative.
Can you write Python code for it?

tendencies_atm, tendencies_sfc = get_tendencies(rad_model)

# plot the temperature profile and temperature tendencies

# prepare the legend names of different temperature tendencies
plot_names = {'top': 'Total', 'LW': 'LW radiation',
              'SW': 'SW radiation',
              'Radiation': 'Radiation (net)',
              'Convection': 'Convection'}

plt.style.use('ggplot')
plt.figure( figsize=(12,6) )
# plot the first subplot
plt.subplot(1, 2, 1)
plt.plot( Tglobal + 273.15, zstar(level),label='observations',color='C2',linestyle=':',linewidth=4 )
plt.plot(rad_model.Tatm,zstar(rad_model.lev),label='pure radiation',color='C0')
plt.legend()
plt.yticks(zstar(yticks),yticks)
plt.xlabel('Temperature (K)')
plt.ylabel('Pressure (hPa)')
plt.title('Temperature profile')

# plot the second subplot
plt.subplot(1, 2, 2)
for i, name in enumerate(tendencies_atm.data_vars):
    plt.plot(tendencies_atm[name], zstar(rad_model.lev),label=plot_names[name])
plt.legend()
plt.yticks(zstar(yticks),yticks)
plt.xlabel('Temperature tendency (K/day)')
plt.ylabel('Pressure (hPa)')
plt.title('Temperature tendency of the pure radiation case')


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Question 3: Please describe the vertical temperature distribution.
Answer 3:
2. Radiative equilibrium
Now we only use the radiation model (RRTMG) to simulate the temperature profile and compare it with the observation. This model can consider the radiation effect of different gases, at different wavelengths (like CO$_2$ mainly absorbs long-wave radiation, but O$_3$ mainly absorbs short-wave radiation).
Can you write Python code for it?

#  Choose the surface albedo
alb = 0.2
#  State variables (Air and surface temperature)
state = climlab.column_state(num_lev=50)
#  Parent model process
rad_model = climlab.TimeDependentProcess(state=state)
#  Set the radiation model
rad = climlab.radiation.RRTMG(state=state, specific_humidity=q_obs, albedo=alb)
#  Couple the radiation model
rad_model.add_subprocess('Radiation', rad)


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Question 4: What gases are considered in the radiation model?
Answer 4:
Let's run this model until it reaches equilibrium (i.e. energy fluxes balance each other out, so the temperature is constant). We need to specify an initial state, which will be an isothermal one (the atmosphere starts out with the same temperature everywhere - obviously not realistic, but instructive to see how radiation changes that situation).
Can you write Python code for it?

#  Start from isothermal state
rad_model.state.Tatm[:] = rad_model.state.Ts

# Integrate the model for 4 years
rad_model.integrate_years(4)

# Check whether the model reaches the equilibrium
# usually it does when the number smaller than 0.1
rad_model.ASR - rad_model.OLR


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Question 6: Does adding the dry convection process improve the simulation of vertical temperature profile? Look at the temperature tendencies, what's the role of convection process? At what level does this role stop, and why do you think that is?
Answer 6:
4. Adding moist convection
Dry convection is fine and good, but in the real world there is often moisture, which condenses once the temperature gets cold enough, so that changes the game. How so? Let us see. We start with some model gobbledygook to set things up proper, then we run the model for 4 years, so it has a chance to reach equilibrium.
Can you write Python code for it?

state3 = climlab.column_state(num_lev=50)
#  Parent model process
rcm = climlab.TimeDependentProcess(state=state3)
#  Fixed relative humidity
h2o = climlab.radiation.ManabeWaterVapor(state=state3)
#  Couple water vapor to radiation
rad = climlab.radiation.RRTMG(state=state3, specific_humidity=h2o.q, albedo=alb)
#  Convective adjustment
conv = climlab.convection.ConvectiveAdjustment(state=state3, adj_lapse_rate=6)
#  Couple everything together
rcm.add_subprocess('Radiation', rad)
rcm.add_subprocess('WaterVapor', h2o)
rcm.add_subprocess('Convection', conv)

rcm.state.Tatm[:] = rcm.state.Ts
rcm.integrate_years(4)
rcm.ASR - rcm.OLR

tendencies_atm, tendencies_sfc = get_tendencies(rcm)

plt.style.use('ggplot')
plt.figure( figsize=(12,6) )
plt.subplot(1, 2, 1)
plt.plot( Tglobal + 273.15, zstar(level),label='observations',color='C2',linestyle=':',linewidth=4 )
plt.plot(rad_model.Tatm,zstar(rad_model.lev),label='pure radiation',color='C0')
plt.plot(rcm_dry.Tatm,zstar(rad_model.lev),label='dry adjust',color='C3',linestyle='--')
plt.plot(rcm.Tatm,zstar(rcm.lev),label='moist adjust',color='C1')
plt.legend()
plt.yticks(zstar(yticks),yticks)
plt.xlabel('Temperature (K)')
plt.ylabel('Pressure (hPa)')
plt.title('Temperature profile')

plt.subplot(1, 2, 2)
for i, name in enumerate(tendencies_atm.data_vars):
    plt.plot(tendencies_atm[name], zstar(rad_model.lev),label=plot_names[name])
plt.legend()
plt.yticks(zstar(yticks),yticks)
plt.xlabel('Temperature tendency (K/day)')
plt.ylabel('Pressure (hPa)')
plt.title('Temperature tendency of the moist adjustment case')


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
Take global averages and time averages.
Can you write Python code for it?

Tzon = np.mean(ncep_air.variables['air'],axis=(0,3))
Tglobal = np.sum( Tzon * np.cos(np.deg2rad(lat)), axis=1 ) / np.sum( np.cos(np.deg2rad(lat) ) )

q_obs = np.loadtxt('q_profile.txt')


I am working on the file "GEOL157L/GEOL157L_lab4_Energy_balance_model.ipynb".
The first task is:
We can plot the how surface temperature responses as the power changes
Can you write Python code for it?

# what if the power number is not 4?
n_power = np.arange(1,7,0.2)
T_power = nlayer(2,S,albedo,n_power) # choose a two-layer model

plt.semilogy(n_power,T_power)
plt.axhline(273,color='gray',linestyle='--')
plt.axhline(373,color='gray',linestyle='--')
plt.xlabel('Stefan-Boltzmann Exponent')
plt.ylabel('Surface temperature (K)')


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Here we list the name of different reconstructions. Each name represent one temperature recontruction of the northern hemisphere in 1881-1980.
Can you write Python code for it?


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Let us load the reconstructed temperature and $CO_2$ over the last 800,000 years, then plot them.
Can you write Python code for it?

f1 = pd.read_csv('edc3deuttemp2007.txt',delim_whitespace=True,skiprows=91)

f1


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Let's plot the reconstruction and the fitted trend lines.
Can you write Python code for it?

plt.figure(figsize=(10,6))
plt.plot(years,lmr_nht,'k-',linewidth=2,label='LMR '+lmc,alpha=.5)
for i in range(len(segment_idxs)):
    slope_all_idxs = np.arange(segment_idxs[i,0],segment_idxs[i,1]+1)
    slope_segment_years = years[slope_all_idxs]
    slope_segment_values = (slope_all_idxs*segment_slopes[i])+segment_intercepts[i]
    line3, = plt.plot(slope_segment_years,slope_segment_values,color='Purple',linewidth=1)
plt.xlabel('year CE')
plt.ylabel('Temperature anomaly (K)')


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Load the reconstructed carbon dioxide concentration in the last 800,000 years.
Can you write Python code for it?

f2 = pd.read_csv('edc-co2-2008.txt',delim_whitespace=True,skiprows=773)

f2

plt.style.use('ggplot')
fig, ax = plt.subplots(2,sharex=True,figsize=(10,6))
ax[0].plot(f1['Age']/1000,f1['Temperature']-54.5)
# Make the y-axis label, ticks and tick labels match the line color.
ax[0].set_ylabel(r'Temperature ($^{\circ}$C)')
#ax1.tick_params('y')
plt.gca().invert_xaxis()

ax[1].plot(f2['Age(yrBP)']/1000,f2['CO2(ppmv)'],color='C1')
ax[1].set_ylabel(r'CO$_2$ concentration (ppmv)')
ax[0].set_xlabel('1000 years before present')
fig.subplots_adjust(hspace=0)

from bokeh.plotting import figure, show, output_file
from bokeh.layouts import gridplot
from bokeh.io import output_notebook

output_notebook()

p1 = figure(title='Temperature anomaly',width=800, height=300,x_range=(800, 0))
p1.xaxis.axis_label = '1000 years before present'
p1.yaxis.axis_label = 'Temperature anomaly'
p1.line(f1['Age']/1000,f1['Temperature'],color='red')

p2 = figure(title='CO2',width=800, height=300,x_range=(800, 0))
p2.xaxis.axis_label = '1000 years before present'
p2.yaxis.axis_label = 'CO2 concentration (ppmv)'
p2.line(f2['Age(yrBP)']/1000,f2['CO2(ppmv)'])
show(gridplot([[p1], [p2]]))


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Now calculate the correlation.
Can you write Python code for it?


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Now we wish to see if the latest trend is unprecented.
Can you write Python code for it?

plt.figure(figsize=(8,8))
ngrid = 100
gmt_blockTrend_older = segment_length*segment_slopes[0:-1].flatten()
gmt_blockTrend_newest = segment_length*segment_slopes[-1]
cutoff_date = str(years[-1]-segment_length) # extract date string for cutoff
old_kws = {"color": 'DarkGray', "label": "pre-"+cutoff_date,"shade": True,"gridsize": ngrid,"legend":False}
sns.distplot(gmt_blockTrend_older, hist=False, kde_kws= old_kws, color = 'DarkGray')
plt.axvline(gmt_blockTrend_newest, color = 'DarkRed',label='Most recent')
plt.title(str(segment_length)+"-year trends")
plt.xlim(-.5,1.5)
plt.legend(loc='upper right')

plt.xlabel(u'GMST (\xb0 C/century)');  plt.ylabel("Probability Density")
#fig.savefig('HowUnusualIsCurrentTrend.pdf',dpi=300, bbox_inches='tight',pad_inches=.3)


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Question 6b: Does this agree with what you obtained in 6a?
Answer 6b:
2. Climate of the Common Era
The Common Era refers to the past 2,000 years of Earth's history. As we will see in class, one can use data from all manner of sources (tree rings corals, icea cores, lake and marine sediments, etc) to piece together some estimate of global or hemispheric temperature over this interval. This allows us to answer an important question: Do we live in extraordinary times?.
Specifically, are current temperature levels unprecedented in the past 2,000 years? Is the rate of warming unpredecented?
To do this, we use the work of Hakim et al. [2016], who reconstructed Northern Hemisphere 2m air temperature at yearly resolution.
Details here: https://www.researchgate.net/project/Last-Millennium-Reanalysis
Can you write Python code for it?

ddict = pd.read_pickle('hakim_lmr_jgra_2016_figure_2.pckl')

years = ddict['recon_times']
lmr_nht = ddict['sanhmt']
lmc = ddict['lmc']
nhmt_max = ddict['nhmt_max']
nhmt_min = ddict['nhmt_min']
xvar = ddict['xvar']
lmr_trend = ddict['lmr_trend']
offset = ddict['offset']
allyears = ddict['allyears']
ipcc_mean = ddict['ipcc_mean']
ipcc_max = ddict['ipcc_max']
ipcc_min = ddict['ipcc_min']

plt.figure(figsize=(10,6))
plt.plot(years,lmr_nht,'r-',linewidth=1,label='LMR mean',alpha=0.8)
plt.fill_between(years, nhmt_max, nhmt_min, facecolor='red', alpha=0.3, linewidth=0)
plt.title('Reconstructed temperature in the Last Millennium Reanalysis, version 1')
plt.ylim([-1.25,1])
plt.xlabel('year CE')
plt.ylabel('NH Temperature anomaly (K)')
plt.savefig('hakim_lmr_jgra_2016_figure_2.pdf', dpi=300, format='pdf', bbox_inches='tight')


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Question 7: Does this suggest that the most recent temperature trend is unprecented? Why?
Answer 7:
It turns out that there are many estimates of NHT over the past 1,000 to 2,000 years -- these estimates vary in the input data and statistical methods used.
Now let's redo this analysis with other reconstructions as in northern hemisphere 1881-1980
Can you write Python code for it?

nh_dict = pd.read_pickle('NH1881-1980.pickle')

nh_dict


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
Question 8: Let's compare our previous result with one using one of these curves. Pick your favorite and apply the same analysis as before.
Can you write Python code for it?

nht = nh_dict['Ju07cvm']['temp']
years = nh_dict['Ju07cvm']['year']
segment_means, segment_slopes, segment_intercepts, segment_idxs = means_and_slopes(nht,segment_length,step,years)

plt.figure(figsize=(10,6))
plt.plot(years,nht,'k-',linewidth=2,label='LMR '+lmc,alpha=.5)
for i in range(len(segment_idxs)):
    slope_all_idxs = np.arange(segment_idxs[i,0],segment_idxs[i,1]+1)
    slope_segment_years = years[slope_all_idxs]
    slope_segment_values = (slope_all_idxs*segment_slopes[i])+segment_intercepts[i]
    line, = plt.plot(slope_segment_years,slope_segment_values,color='Purple',linewidth=1)
plt.xlabel('year CE')
plt.ylabel('Temperature anomaly (K)')

plt.figure(figsize=(8,8))
ngrid = 100
gmt_blockTrend_older = segment_length*segment_slopes[0:-1].flatten()
gmt_blockTrend_newest = segment_length*segment_slopes[-1]
cutoff_date = str(years[-1]-segment_length) # extract date string for cutoff
old_kws = {"color": 'DarkGray', "label": "pre-"+cutoff_date,"shade": True,"gridsize": ngrid,"legend":False}
sns.distplot(gmt_blockTrend_older, hist=False, kde_kws= old_kws, color = 'DarkGray')
plt.axvline(gmt_blockTrend_newest, color = 'DarkRed',label='Most recent')
plt.title(str(segment_length)+"-year trends")
plt.xlim(-.5,1.5)
plt.legend(loc='upper right')

plt.xlabel(u'GMST (\xb0 C/century)');  plt.ylabel("Probability Density")


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
The code below chops the reconstruction in blocks of length "segment_length" and steps them every "step" steps, to find the mean and trend over each interval.
Can you write Python code for it?

def means_and_slopes(variable,segment_length,step,years):
    # This function calculates the means and slopes for the specified segments.  Outputs:
    #    segment_means:      Means of every segment.
    #    segment_slopes:     Slopes of every segment.
    #    segment_intercepts: Y-intercepts of every segment, for plotting purposes.
    #    segment_idxs:       The first and last index of every segment, for record-keeping.
    #
    import statsmodels.api as sm
    import numpy as np
    n_years = variable.shape[0]
    n_segments = int(((n_years-segment_length)/step)+1)
    skip_idx = np.remainder(n_years-segment_length,step)  # If the segments don't cover the entire time-span, skip years at the beginning.
    #
    # Initialize variables
    segment_means      = np.zeros((n_segments))
    segment_slopes     = np.zeros((n_segments))
    segment_intercepts = np.zeros((n_segments))
    segment_idxs       = np.zeros((n_segments,2),dtype=int)
    segment_means[:]      = np.nan
    segment_slopes[:]     = np.nan
    segment_intercepts[:] = np.nan
    segment_idxs[:]       = np.nan
    #
    # Compute the means and slopes for every location
    for k in range(n_segments):
        start_idx = skip_idx+(k*step)
        end_idx   = start_idx+segment_length
        #print("Calculating: "+str(k+1)+"/"+str(n_segments)+", years: "+str(years[start_idx])+"-"+str(years[end_idx-1]))
        segment_means[k] = np.mean(variable[start_idx:end_idx],axis=0)
        segment_slopes[k], segment_intercepts[k] = np.polyfit(np.arange(start_idx,end_idx),variable[start_idx:end_idx],1)
        segment_idxs[k,:] = start_idx,end_idx-1
    #
    return segment_means, segment_slopes, segment_intercepts, segment_idxs

step = 10; segment_length = 100
segment_means, segment_slopes, segment_intercepts, segment_idxs = means_and_slopes(lmr_nht,segment_length,step,years)


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
We can first plot the scatterplot to see the relationship between temperature and CO$_2$ concentration.
Can you write Python code for it?

# combine temperature and co2 to a dataframe first
d = {'temperature': temp_interp, 'CO2': co2_interp, 'age': age}
df = pd.DataFrame(data=d)
# plot scatterplot
sns.jointplot(x="temperature", y="CO2", data=df, kind = "hex")


I am working on the file "GEOL157L/GEOL157_Lab5_paleclimate_records.ipynb".
The first task is:
You all know that correlation is not causation. However, in cases where one variable (A) causes changes in another (B), one often sees A lead the changes in B. Let us hunt for leads in the relationship between temeperature and $CO_2$. We do this via lag-correlations, which are simply correlations between A and lagged copies of B.
Can you write Python code for it?

# lag correlations (co2 leads 10 kyrs to lags 10 kyrs)

lags = np.arange(50,-51,-1)
lag_corr = np.zeros(len(lags))

for i in np.arange(len(lags)):
    lag_corr[i] = df['temperature'].corr(df['co2'].shift(lags[i]))

plt.plot(lags*100,lag_corr)
plt.xlabel(r'CO$_2$ lags temperature (years)')
plt.ylabel('correlation')


I am working on the file "GEOL351/ENSO_recharge.ipynb".
The first task is:
Now, let's define the model itself. For reasons that wil be clear later, we'll include a couple of extra bells and whistles, which we'll ignore for now.
Can you write Python code for it?

def recharge_deriv((T,h),t, mu, en, c = 1.0,r= 0.25,alpha = 0.125, b0 = 2.5, gamma = 0.75):
    """Compute the time-derivative of the recharge system system."""
    b = b0*mu # coupling coefficient
    R = gamma*b-c;  #Bjerknes coefficient

    return [R*T + gamma*h - en*(h + b*T)**3, -r*h - alpha*b*T]


I am working on the file "GEOL351/ENSO_recharge.ipynb".
The first task is:
Question 1 ###
Describe the behavior of the system at criticality.
Answer 1:
Write your answer here
What happens for supercritical values $ \mu > 2/3$?
Can you write Python code for it?


I am working on the file "GEOL351/ENSO_recharge.ipynb".
The first task is:
Question 2 ###
Describe the supercritical behavior of the system.
Answer 2:
Write your answer here
Nonlinear effects
Linear theories are useful when perturbations are small, but they often break down once perturbations grow. Often, nonlinear terms will kick in at that point and will help stabilize the system. One of the strongest nonlinearities in the tropical Pacific ocean-atmosphere system is the fact that subsurface temperatures depend on thermocline depth in a sigmoidal fashion (flat for large values of $|h|$, and linear in between). The essence of this behavior may be incorporated in our toy model by including an extra, cubic term in the SST equation:
$$ \frac{d T_E}{dt} = R T_E + \gamma h_W -e_n (h_W + b T_E)^3 $$
Now, to save myself some coding, it turns out that I have already included this term in the solver. All you need to do to activate it is enter a positive value of $e_n$, say  $e_n = 3$.
Can you write Python code for it?

mu, en = 0.7, 3.0  # set variable parameters
x0 = [[0.1,-0.1]]   # initial conditions
t, x_t = recharge_solver(mu,en)  # solve the system
nt = t.shape[0];  last_time = int(nt/5)
recharge_plotter(t, x_t, mu, en, x0, last_time) # plot it


I am working on the file "GEOL351/ENSO_recharge.ipynb".
The first task is:
Question 3.1 ###
Describe the system's behavior with the inclusion of a nonlinear term, comparing to Question 2.
Question 3.2 ###
Change the values of $e_n$ and $\mu$ and describe qualitatively and quantitatively  what happens, particulary regarding the period of oscillation.
Answer 3.1, 3.2:
Write your answers here
Frequency entraiment of the recharge oscillator
So far you should have found many combinations of parameters that lead to so-called "self-sustained" oscillations. That is, in a nonlinear regime, the system has the ability to excite itself, and even in the absence of external forcing this oscillation will keep going. There aren't enough variables in this system to create chaos (it turns out that you need at least three), but at least this model contains the basic ingredients for an ENSO-like oscillatory instability.
As your explorations should have revealed, the period of oscillation is not a particulary robust feature of this model: depending on arbitrary paramter choices, one can get oscillation periods that vary quite a bit. Besides, none of the oscillations actually line up with the calendar year, unlike in reality.
This had lead some investigators to explore the possibility that phase-locking the calendar year (and it some cases chaos) arise because of interaction with the seasonal cycle. The underlying physical mechanism is called "mode locking", and was first described in the 17th century. Dixit Wikipedia:

Dutch physicist Christiaan Huygens, the inventor of the pendulum clock, introduced the concept after he noticed, in 1666, that the pendulums of two clocks mounted on a common board had synchronized, and subsequent experiments duplicated this phenomenon. He described this effect as "odd sympathy". The two pendulum clocks synchronized with their pendulums swinging in opposite directions, 180° out of phase, but in phase states can also result. Entrainment occurs because small amounts of energy are transferred between the two systems when they are out of phase in such a way as to produce...
Can you write Python code for it?

def recharge_deriv((T,h),t, mu, en, Af, Pf, c = 1.0,r= 0.25,alpha = 0.125, b0 = 2.5, gamma = 0.75):
    """Compute the time-derivative of the recharge system system."""
    b = b0*mu # coupling coefficient
    R = gamma*b-c;  #Bjerknes coefficient
    forcing = Af*np.sin(2*np.pi*t/Pf)
    return [R*T + gamma*h - en*(h + b*T)**3 + forcing, -r*h - alpha*b*T]

def recharge_solver(mu,en,x0, Af=1.0, Pf=6.0,max_time = 3000):
    """ Solve recharge oscillator ODE system
        mu = effective coupling
        en = strength of nonlinear feedback
        x0 = initial conditions
        Af = forcing amplitude
        Pf = forcing period
    """
    # define time axis
    t = np.linspace(0, max_time, int(4*max_time),endpoint=False)
    # Solve for the trajectories
    x_t = np.asarray([integrate.odeint(recharge_deriv, x0i, t, (mu, en, Af, Pf))
                      for x0i in x0])
    return t, x_t

def recharge_plotter(T, h, t, mu, en, Fs, Af=0.0, Pf=6.0):
    """ Plot the output of recharge_solver
        Fs = sampling frequency
    """
    # extract output
    ts = t*tscale; nt = t.shape[0]
    last_time = np.round(nt/5.0)
    # obtain spectrum
    f_T, S_T, nu = tsa.multi_taper_psd(T,Fs)
    # compute average period
    DominantPeriod =  "{:4.2f}".format(1.0/f_T[np.argmax(S_T)])
    # PLOT IT OUT
    plt.close('all')
    fig = plt.figure(figsize=(8,8))
    # plot traces
    ax1 = plt.subplot2grid((2, 2), (0, 0),colspan = 2)
    hdl = ax1.plot(ts[-last_time:],T[-last_time:],ts[-last_time:],h[-last_time:])
    ax1.legend(hdl,[r'$T_E$',r'$h_W$'],loc = 'upper left')
    ax1.set_title(r'Recharge oscillator solution, $\mu$ = %s, $e_n$ = %s, $A_f=$ %s'%(mu, en, Af),fontsize=14)
    # phase portrait
    ax2 = plt.subplot2grid((2, 2), (1, 0))
    ax2.plot(x_t[0,:,0],x_t[0,:,1])
    ax2.plot(x0[0][0],x0[0][1],'ro')
    ax2.set_xlabel(r'$T_E$'); ax2.set_ylabel(r'$h_W$')
    ax2.set_title("Phase portrait",fontsize=14)
    #
    ax3 = plt.subplot2grid((2, 2), (1, 1))
    ax3.loglog(f_T,S_T,...


I am working on the file "GEOL351/ENSO_recharge.ipynb".
The first task is:
We are now ready to start tinkering around. The analysis by Jin [1997a] identifies $\mu_c = 2/3$ as the critical coupling coefficient.  Below this value, the system should be damped, and above, it should be unstable. Right at criticality, one should observe another remarkable phenomenon.
What happens just below this value?
set $\mu = 0.65$ and watch what happens by executing the next cell.
Can you write Python code for it?

mu, en = 0.65, 0  # set variable parameters
x0 = [[0.1,-0.1]]   # initial conditions
t, x_t = recharge_solver(mu,en)  # solve the system
nt = t.shape[0];  last_time = nt  # plot only the last "last_time" steps in case the plot is too dense
recharge_plotter(t, x_t, mu, en, x0, last_time) # plot it


I am working on the file "GEOL351/MOC_stability.ipynb".
The first task is:
Because of a plotting glitch, we had to plot E as a function of $\delta$, but you could easily flip the axes in your head. also, we plotted $q = 1-\delta$, because what we care about is whether the ocean is circulating, and in which direction.
Question 4 What can you say of the region left of the green point? Between the green and blue points? To the right of the blue point? (in each case, indicate whether the solution is stable or unstable.
 Answer 4 : here
Approaches to equilibrium
Now that we have established where the fixed points are, and their stability, let us solve equation (5) numerically (via odeint) and graph the solutions. Let's do this first for $E<1/4$, to make things a little interesting.  Let's define 2 functions: one to obtain the fixed points as a function of $E$, and one containing $f(\delta)$, which is what odeint wants.
Can you write Python code for it?

def fixed_pts(E):
     # compute fixed points given E
    C1 = np.lib.scimath.sqrt(0.25-E) # defined even for E > .25
    C2 = np.sqrt(0.25+E)

    if (E < .25):
        nsol = 3; x = np.zeros((nsol))
        x[0] = .5 + C2 # always stable
        x[1] = .5 + C1
        x[2] = .5 - C1

    elif (E == .25):
        nsol = 2; x = np.zeros((nsol))
        x[0] = .5 + C2 # always stable
        x[1] = .5

    else:
        nsol = 1; x = np.zeros((nsol))
        x[0] = .5 + C2 # always stable
    return  x

def delta_eqn(y,t,E=0.2):
    return E-abs(1-y)*y


I am working on the file "GEOL351/MOC_stability.ipynb".
The first task is:
Now investigate the behavior of the solution as you change A and d0.
Question 7 For which values of these parameters do you obtain periodic solutions? Are they symmetric about 0? If so, why not? Do the switches happen when the forcing changes sign? If not, why?
Based on the stability diagram above, explain physically what happens as the system is moved across the $E_c$ threshold (dashed line).
Answer 7 ...
Question 8
Does the same behavior happen for a linear system? Repeat the above exercise with the following function:
Can you write Python code for it?

def delta_lin_forced(y,t,A=0.5,period=5):
    forc = 0.25+A*np.sin(2*np.pi*t/period)
    dydt = forc-y
    return dydt


I am working on the file "GEOL351/MOC_stability.ipynb".
The first task is:
Now let's integrate equation (5). For a given freshwater forcing $E$, it's instructive to do this for an array of initial conditions, and see where we end up.
Can you write Python code for it?

E = 0.2

tmax = 20
t = np.linspace(0,tmax); nt = len(t)
d0vec = np.linspace(0,1.5); nd = len(d0vec)
d = np.zeros((nt,nd)) # define array

for k in range(nd):  # loop over initial conditions
    y = odeint(delta_eqn, d0vec[k], t, args=(E,)) # invoke ODE solver
    d[:,k] = y.flatten() # store result in an array

# Now plot it
f,ax = plt.subplots(1,1,figsize=(8,6))
ax.set_color_cycle([plt.cm.inferno(s) for s in np.linspace(0, 1, nd)])
ax.plot(t,d)
ax.hlines(fixed_pts(E),0,tmax,linestyle='dashed',zorder=50)
ax.set_title('Behavior of Stommel solution for E='+str(E))
ax.set_xlabel(r'$t$ (non-dimensional)')
ax.set_ylabel(r'$\delta(t)$')


I am working on the file "GEOL351/MOC_stability.ipynb".
The first task is:
Pretty, eh? The color of each line indicates its origin, and dashed black lines indicate the fixed points. You can see that depending on this origin, the salinity converges to two equilibrium values: one around 1.17, the other around 0.28.  In between, curves that start around the third fixed point over around it for a brief time, then go join their friends around the other fixed points.
 Question 5 Why do you think that is the case? (Hint: Look at the previous stability diagram of $E=0.2$)
 Answer 5 here
 Question 6 Repeat the analysis above for $E = 0.25$ and $E=0.26$.  How many fixed points do you obtain in each case? Does it make sense given the stability diagram? How quickly do trajectories converge to the fixed points, and what does this depend on?
 Answer 6 here
Hysteresis
As mentioned in class, the system can exhibit hysteresis if forced appropriately across thresholds. In this section we force the system with a periodically varying freshwater flux to investigate this behavior.
$$E = E_c + A*cos(\frac{2\pi}{T}t)$$
where $E_c=1/4$ is the critical value (see above), $A$ is some amplitude, and $T$ is a forcing
Next we define a slightly modified version of the funciton delta_eqn() that accounts for this forcing
Can you write Python code for it?

def delta_eqn_forced(y,t,A=0.5,period=5):
    forc = 0.25+A*np.sin(2*np.pi*t/period)
    return forc-abs(1-y)*y