I am working on the file "GEOL351/MOC_stability.ipynb".
The first task is:
Voilà. All you need is to specify values of A, period, and the starting point $d0 = \delta(0)$. Here we do so for some choice of parameters:
Can you write Python code for it?

tmax = 50
t = np.linspace(0,tmax,100)
A = .3
period = 20
d0 = 0.2

forc = 0.25+A*np.sin(2*np.pi*t/period)
df = odeint(delta_eqn_forced, d0, t, args=(A,period)).flatten()

zer = 0*t  # define zero line
q = 1-df   # define transport

f,ax1 = plt.subplots(1,1,figsize=(8,6))
ax1.plot(t,q,'k:')
ax1.set_title('Behavior of Stommel solution with periodic forcing')
ax1.set_xlabel(r'$t$ (non-dimensional)')
ax1.set_ylabel(r'$q=1-\delta(t)$')
ax1.fill_between(t, zer, q , where= (q >= zer), facecolor='Crimson', interpolate=True,alpha=0.3)
ax1.fill_between(t, zer, q , where= (q < zer), facecolor='DodgerBlue', interpolate=True,alpha=0.3)
ax2 = ax1.twinx()
ax2.plot(t,forc,'k',label='forcing')
ax2.set_ylabel(r'$E = 1/4+cos(2\pi\omega t)$')
ax2.hlines(0.25,0,tmax,linestyle='dashed',zorder=50)

ax1.set_yticks(np.linspace(-.6, 1.2, 10))
ax2.set_yticks(np.linspace(-.6, 1.2, 10))


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Answer 4
write your answer here
Finally, we could ask what kind of climate would be obtained by varying the atmosphere's opacity to outgoing longwave radiation (i.e. its concentration of greenhouse gases). We seek a stability diagram similar to the one above, but cast in terms of radiative pressure $p_{rad}$ instead of $L$.
Question 5
 What is the effect of lowering $p_{rad}$ on the surface temperature $T_s$?
You may answer using purely mathematical, physical, or heuristic arguments (one way to do this is to vary $p_{rad}$ and see how it affects the OLR).
Answer 5
write your answer here
Greenhouse Bifurcation
To draw the bifurcation diagram in terms of $p_{rad}$ we need a bit more machinery. Let's define a function that solves for radiative pressure given the solar flux and the temperature. (if you really want to know, it uses Newton-Raphson's method, but you needn't worry about that)
Can you write Python code for it?

def radPress(flux,T):
    def g(p,const):
        return OLR(const.T,p) - const.flux
    root = newtSolve(g)
    const= Dummy()
    const.T = T
    const.flux = flux
    root.setParams(const)
    return root(500.)

L = 1370.
TList = [220.+i for i in range(131)]
gList = [radPress((1.-albedo(T))*L/4.,T) for T in TList]
cG = Curve()
cG.addCurve(gList,'pRad')
cG.addCurve(TList,'T')
#Just for variety, here I've shown that instead of
# switching axes, you can just put the data into
# the Curve in a different order
cG.PlotTitle = 'Bifurcation diagram, L = %f W/m**2'%L
cG.Ylabel = 'Surface Temperature (K)'
cG.Xlabel = 'Radiating pressure (mb)'
cG.reverseX = True #Reverse pressure axis so warmer is to the right
plot(cG)


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Energy Balance
Next, let's look at energy balance. What's coming in, what's going out?
Can you write Python code for it?

c1 = Curve()
c1.addCurve(Tlist)
c1.addCurve(SabsList,'Sabs','Absorbed Solar')
c1.addCurve(OLRlist,'OLR','Outgoing Longwave')
c1.Xlabel = 'Temperature (K)'
c1.Ylabel = 'Flux ($W/m^2$)'
c1.PlotTitle = 'Energy Balance Diagram'
w1 = plot(c1)


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Notice how the function uses "global" arguments  alpha_ice, alpha_ice, T1, T2. We need to define those for the function to work. Let's do this, along with defining pRad.
Can you write Python code for it?

L = 1.*1370.
alpha_ice = .6
alpha_0 = .2
T1 = 260. # in Kelvins
T2 = 290. # in Kelvins
pRad = 1000. # in mb


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Once again, wherever this balance function (blue curve) inersects the incident solar radiation (dashed green curve), we get one or more equilibrium temperature(s). From this it is easy to find the values of the solar constant for which we can have 1, 2 or 3 equilibria. Note that some of the branches in this diagram are unstable.  For instance, as you vary L from 1370 up to 3000, say, at some point you will leave the lower branch and land abruptly and without warning, on the upper branch.  This is one reason why bifurcations are relevant to climate risk assessment.
Question 4
The current greenhouse effect yields a radiative pressure pRad in the neighborhood of 670mb. Repeat the calculations above for this value of $p_{Rad}$, (it will be cleaner if you copy and paste the relevant cells below).
 For which value of L, approximately, would we enter a stable snowball state?
(to answer this, redraw the bifurcation diagram above with $p_{Rad}=670$mb. For $L = 1370 W.m^{-2}$ you should find 3 possible states. Assume that you are on the upper, toasty branch. For which value of $L$ would you fall onto the lower, icy branch?)
Can you write Python code for it?


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Outgoing Long-wave radiation
Let us define a function that will compute outgoing longwave emission from the planet for a given emission temperature $T$ (dropping the subscript for convenience).
Can you write Python code for it?

def OLR(T,param=None):
    pRad = param
    return phys.sigma * (T**4.)*(pRad/1000.)**(4.*2./7.)


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
The next cell generates lists of values for OLR, the next flux, and the incoming radiation given a range of temperature values from 200 to 340 K.
Can you write Python code for it?

Tlist = [200.+ 2.*i for i in range(70)]
SabsList = []
OLRlist = []
NetFluxlist = []
Glist = []
aList = []

for T in Tlist:
    aList.append(albedo(T))  # albedo function
    SabsList.append((L/4.)*(1.-albedo(T))) # incident solar values
    OLRlist.append(OLR(T,pRad)) # outgoing longwave emissions
    NetFluxlist.append((L/4.)*(1.-albedo(T)) - OLR(T,pRad))  # net flux
    Glist.append(4.*OLR(T,pRad)/(1.-albedo(T)))  # balance function


I am working on the file "GEOL351/ZeroD.ipynb".
The first task is:
Time to plot a few things! First, the albedo:
Can you write Python code for it?

c1 = Curve()
c1.addCurve(Tlist)
c1.addCurve(aList,'Albedo','Alb')
c1.Xlabel = 'Temperature (K)'
c1.Ylabel = 'Alpha'
c1.PlotTitle = 'Planetary Albedo'
w1 = plot(c1)
plt.ylim((0, 0.8))


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
(Once again, Y is on top of X. )
Let us now plot the probability of occurence of states in the $(X,Y)$ plane. If all motions were equally likely, this probability would be uniform. Is that what we observe?
Can you write Python code for it?

cmap = sns.cubehelix_palette(light=1, as_cmap=True)
skip = 10
sns.jointplot(xl[0::skip,0],xl[0::skip,1], kind="kde", color="#4CB391")


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
Let's plot this solution
Can you write Python code for it?


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
Let's start by plotting the solutions for X, Y and Z when $f_0 = 0$ (no pertubation):
Can you write Python code for it?

sns.set_style("darkgrid")
sns.set_palette("Dark2")
t, x_t = forced_lorenz(fnot = 0, theta = 50,max_time = 100.00)

xv = x_t[0,:,:]
# time filter
lab = 'X', 'Y', 'Z'

col = sns.color_palette("Paired")

fig = plt.figure(figsize = (8,8))
xl = np.empty(xv.shape)
for k in range(3):
     xl[:,k] = blf.filter(xv[:,k],0.5,fs=25)
     plt.plot(t,xv[:,k],color=col[k*2])
     #plt.plot(t,xl[:,k],color=col[k*2+1],lw=3.0)

plt.legend(lab)
plt.show()


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
Question 1 ###
How would you describe the probability of visiting states? What do "dark" regions correspond to?
Answer 1:
Write your answer here
2. Visualizing climate change in the Lorenz system
We now wish to see if this changes once we apply a non-zero forcing. Specifically:
1. Does the attractor change with the applied forcing?
2. If not, can we say something about how frequently some states are visited?
In all the following we set $f_0 = 2.5$ and we tweak $\theta$ to see how the system responds
To ease experimentation, let us first define a function to compute and plot the results:
Can you write Python code for it?

def plot_lorenzPDF(fnot, theta, max_time = 1000.00, skip = 10):
    t, x_t = forced_lorenz(fnot = fnot, theta = theta,max_time = max_time)
    xv = x_t[0,:,:]; xl = np.empty(xv.shape)

    for k in range(3):
        xl[:,k] = blf.filter(xv[:,k],0.5,fs=25)

    g = sns.jointplot(xl[0::skip,0],xl[0::skip,1], kind="kde", color="#4CB391")
    return g


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
Very pretty. If you are curious, you can even change the plot angle within the function, and examine this strange attractor under all angles. There are several key properties to this attractor:
A forced Lorenz system
As a metaphor for anthropogenic climate change, we now consider the case of a forced lorenz system. We wish to see what happens if a force  is applied in the directions $X$ and $Y$.   The magnitude of this force is $f_0$ and we may apply it at some angle $\theta$. Hence:
$f = f_0 \left ( \cos(\theta),sin(\theta) \right) $
The new system is thus:
$$
\begin{aligned}
\dot{x} & = \sigma(y-x) + f_0  \cos(\theta)\
\dot{y} & = \rho x - y - xz +  f_0  \sin(\theta)\
\dot{z} & = -\beta z + xy
\end{aligned}
$$
Does the attractor change? Do the solutions change? If so, how?
Solving the system
Let us define a function to solve this new system:
Can you write Python code for it?

def forced_lorenz(N=3, fnot=2.5, theta=0, max_time=100.0, sigma=10.0, beta=8./3, rho=28.0):

    def lorenz_deriv((x, y, z), t0, sigma=sigma, beta=beta, rho=rho):
        """Compute the time-derivative of a forced Lorentz system."""
        c = 2*np.pi/360
        return [sigma * (y - x) + fnot*np.cos(theta*c), x * (rho - z) - y + fnot*np.sin(theta*c), x * y - beta * z]

    # Choose random starting points, uniformly distributed from -15 to 15
    np.random.seed(1)
    x0 = -15 + 30 * np.random.random((N, 3))

    # Solve for the trajectories
    t = np.linspace(0, max_time, int(25*max_time))
    x_t = np.asarray([integrate.odeint(lorenz_deriv, x0i, t)
                      for x0i in x0])
    return t, x_t


I am working on the file "GEOL351/lorenz_climatechange.ipynb".
The first task is:
What happened to X? Well in this case X and Y are so close to each other that they plot on top of each other.
Baiscally, the system orbits around some fixed points near $X, Y = \pm 10$. The transitions between these "regimes" are quite random. Sometimes the system hangs out there a while, sometimes not.
Isolating climate fluctuations
Furthermore, you may be overwhelmed by the short term variability in the system. In the climate system, we call this shprt-term variability "weather", and often what is of interest is the long-term behavior of the system (the "climate"). To isolate that, we need to filter the solutions. More precisely, we will apply a butterworth lowpass filter to $X, Y ,Z$ to highlight their slow evolution. (if you ever wonder what it is, take GEOL425L: Data Analysis in the Earth and Environmental Sciences)
Can you write Python code for it?

# Timeseries PLOT
sns.set_palette("Dark2")
t, x_t = forced_lorenz(fnot = 0, theta = 50,max_time = 1000.00)

xv = x_t[0,:,:]
# time filter
lab = 'X', 'lowpass-filtered X', 'Y', 'lowpass-filtered Y', 'Z','lowpass-filtered Z'

col = sns.color_palette("Paired")

fig = plt.figure(figsize = (8,8))
xl = np.empty(xv.shape)
for k in range(3):
     xl[:,k] = blf.filter(xv[:,k],0.5,fs=25)
     plt.plot(t,xv[:,k],color=col[k*2])
     plt.plot(t,xl[:,k],color=col[k*2+1],lw=3.0)

plt.legend(lab)
plt.xlim(0,100)

# Be patient... this could take a few seconds to complete.


I am working on the file "GHCND.ipynb".
The first task is:
Analysing the plot above, we can see that the 3 cities experienced a big variation of temperature in the time of observation. The variation was more expressive in Lee Vining.
Can you write Python code for it?

june = stations[(stations.DATE>=20150601) & (stations.DATE<=20150630)]
rain = june.groupby(['STATION'])
rain.plot('DATE','PRCP')


I am working on the file "GHCND.ipynb".
The first task is:
In order to select the stations, we can select the following data from the initial amount:
Can you write Python code for it?


I am working on the file "GHCND.ipynb".
The first task is:
So we can print data3 and, then, select the stations in the table that will be printed.
Can you write Python code for it?

stations = data2[(data2.STATION=='GHCND:USC00047326') | (data2.STATION=='GHCND:USC00047902') | (data2.STATION=='GHCND:USC00044881')]

st = stations.groupby(['STATION'])
temp = st.agg({'TMIN' : [np.min], 'TMAX' : [np.max]})
temp.plot(kind='bar')


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
3. Density plots
Can you write Python code for it?

mask = df_prcp.Accumulated > 0.0

# N.b. the KDE (kernel density estimate) is Gaussian - which is not true
# for precip data (log or power law data)...
daily_dp = plt.figure()
daily_dp.set_size_inches(12,5)
ax = daily_dp.add_subplot(111)

sns.distplot(df_prcp.Accumulated[mask],bins=100,norm_hist = True,kde=False,color = 'r')
sns.kdeplot(df_prcp.Accumulated[mask],shade=True,kernel='cos',cumulative=False,color='b')
leg1=ax.legend(['KDE','Accumulated mean'],prop={'size':11},
                numpoints=1,markerscale=5.,frameon=True,fancybox=True)

ax.set_xlim(0,30)
ax.set_title("East Africa Mean(Zone1) Precipitation")
ax.set_xlabel(r'Precip. (mm day$^{-1}$)')
ax.set_ylabel('Density (0-1)')

plt.show(daily_dp)
#daily_dp.savefig('Zone1_Densityplot.pdf',dpi=300)

sns.kdeplot

#Histogram
#source code from https://github.com/benlaken/Tanzania/blob/master/Precipitation_Tanzania.ipynb
hist_dp = plt.figure()
hist_dp.set_size_inches(5,5)          # Specify the output size
ax1 = hist_dp.add_subplot(211)        # Add an axis frame object to the plot (i.e. a pannel)
ax2 = hist_dp.add_subplot(212)        # Add an axis frame object to the plot (i.e. a pannel)

# the histogram of the data
ax1.set_title(r'PDFs of Daily Precipitation for Zone 2', fontsize=13)
n, bins, patches = ax1.hist(df_prcp.Accumulated[mask], 100, normed=True, facecolor='blue', alpha=0.75,
                            histtype='stepfilled')
n, bins, patches = ax1.hist(eraint, 100, normed=True, facecolor='red', alpha=0.75,
                            histtype='stepfilled')
ax1.grid(True)
ax1.set_ylabel('Density', fontsize =15)
n, bins, patches = ax2.hist(df_prcp.Accumulated[mask], 100, normed=True, facecolor='blue', alpha=0.75,
                            histtype='stepfilled',cumulative=True)
n, bins, patches = ax2.hist(eraint, 100, normed=True, facecolor='red', alpha=0.75,
                            histtype='stepfilled',cumulative=True)
leg1=ax1.legend(['Station...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
4. Seasonality
*Calculate the Day of Year (DOY) mean over the data-period (climatology).
Can you write Python code for it?

doy_mean=[]
doy_sem =[]

for doy in range(366):
    index = df_prcp['1961-01-01':'1990-12-31'].index.dayofyear == doy+1
    #print(index)
    doy_mean.append(np.nanmean(df_prcp['Accumulated']['1961-01-01':'1990-12-31'][index]))
    doy_sem.append(calc_SEM(df_prcp['Accumulated']['1961-01-01':'1990-12-31'][index]))

doy_mean = np.array(doy_mean)
doy_sem = np.array(doy_sem)

doy_eramean=[]
doy_erasem =[]

for doy in range(366):
    index = times_index.dayofyear == doy+1
    #print(index)
    doy_eramean.append(np.nanmean(eraint[index]))
    doy_erasem.append(calc_SEM(eraint[index]))

doy_eramean = np.array(doy_eramean)
doy_erasem = np.array(doy_erasem)

ssn_rmean = pd.rolling_mean(doy_mean, window=30, min_periods=0, center = True)
ssn_eramean = pd.rolling_mean(doy_eramean, window=30, min_periods=0, center = True)
#ssn_rmean[-30:] = np.nan

#Plot the seasonal climatology East Africa precip data
mnths= ['Jan','Feb','Mar','Apr','May','June','Jul','Aug','Sep','Oct','Nov','Dec']
#mrange = arange(12)

my_sclim = plt.figure(dpi=300)
my_sclim.set_size_inches(10,5)        # Specify the output size
ax1 = my_sclim.add_subplot(111)        # Add an axis frame object to the plot (i.e. a pannel)

ax1.errorbar(range(366),doy_mean,xerr=None, yerr=doy_sem, c='b', alpha=1., lw=1.5)
ax1.errorbar(range(366),doy_eramean,xerr=None, yerr=doy_erasem, c='r', alpha=1.,lw=1.5)
#ax1.plot(range(366), ssn_rmean,'b-', alpha=1.0)
#ax1.plot(range(366), ssn_eramean,'r-', alpha=1.0)

leg1=ax1.legend(['Station data','RCA4_ERA'],prop={'size':12},numpoints=1,markerscale=5.,frameon=True,fancybox=True)
plt.xlim(0,max(range(366)))
plt.title("Climatology Precipitation for Zone 2 by Day (& 30day smooth)", fontsize=15)
plt.ylabel("Precipitation (mm day$^{-1}$)", fontsize=15)
plt.xlabel("Day of Year (DOY)", fontsize=15)
plt.grid(True)
#my_sclim.savefig('Zone2_SeasonalClimatology_+ERA_plot.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
5. Extreme Precip Events
Extreme events have been defined  by absolute threshhold set by SWFDP-RSMC-Nairobi (Rnn mm = Count of days where RR ≥ user-defined threshold in mm)
Can you write Python code for it?

#A mask for the df_prcp to identify categories of Extreme rainfall events
"""
The thresholds used in here are based on the definitions as used
by SWFDP-EA. It should be noted that this hold under natural conditions
No risk  - <5mm
Low risk - 5mm-20mm
Medium risk 20 - 50mm
High risk >=50mm
"""
cond1 = df_prcp.Accumulated < 5
cond2 = df_prcp.Accumulated > 5
cond3 = df_prcp.Accumulated < 20
highmed_risk = df_prcp.Accumulated[df_prcp.Accumulated > 20] #Medium to high risk
high_risk = df_prcp.Accumulated[df_prcp.Accumulated > 50]
medium_risk = df_prcp.Accumulated[(df_prcp.Accumulated > 20) & (df_prcp.Accumulated < 50)]
low_risk = df_prcp.Accumulated[cond2 & cond3]
no_risk = df_prcp.Accumulated[cond1]

#A mask for the df_era to identify if there are similar Extreme rainfall events in RCA4 data
nrisk= df_era.ERAINT[df_era.ERAINT < 5]
yrisk = df_era.ERAINT[(df_era.ERAINT>5) & (df_era.ERAINT<20) ]
print(len(yrisk))

daily_floodrisk = plt.figure(dpi=72)
daily_floodrisk.set_size_inches(12,7)      # Specify the figure size
ax1 = daily_floodrisk.add_subplot(111)     #

ax1.plot(high_risk.index, high_risk,'ro',alpha=1.,linewidth = 3, ms=3)
ax1.plot(medium_risk.index, medium_risk,'bo',alpha=0.9,ms=3)
ax1.plot(low_risk.index, low_risk,'co',alpha=0.9,ms=3)
ax1.plot(no_risk.index, no_risk,'go',alpha=0.9,ms=3)
leg1=ax1.legend(['high risk','medium risk','low risk','no risk'],
                prop={'size':12},numpoints=1,markerscale=5.,frameon=True,fancybox=True)
#plt.xlim('1953-01-01','1990-12-31')
plt.title(r" Zone 2 Station data Extreme Precipitation Events based on Absolute Threshhold", fontsize=15)
plt.ylabel(r"Amounts (mm day$^{-1}$)", fontsize=15)
plt.xlabel("Years", fontsize=15)
plt.grid(True)
plt.show(daily_floodrisk)
#daily_floodrisk.savefig('Zone1_ExtremeEvent.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
CDD & CWD
Consecutive dry days  & Consecutive wet days
Can you write Python code for it?

cdd_ERA = add_days_since(df=df_era, truth_array=df_era['ERAINT']<1 , name_to_add='RCA4_CDD')
cdd_Obs = add_days_since(df=df_prcp, truth_array=df_prcp['Accumulated']<1 , name_to_add='CDD')
cwd_ERA = add_days_since(df=df_era, truth_array=df_era['ERAINT']>=1 , name_to_add='RCA4_CWD')
cwd_Obs = add_days_since(df=df_prcp, truth_array=df_prcp['Accumulated']>=1 , name_to_add='CWD')

annera_cdd = df_era.RCA4_CDD.resample('A', how='mean')
annobs_cdd = df_prcp.CDD.resample('A', how='mean')
annera_cwd = df_era.RCA4_CWD.resample('A', how='mean')
annobs_cwd = df_prcp.CWD.resample('A', how='mean')

cddobs_rav = pd.rolling_mean(annobs_cdd, window=10, min_periods=0, center = True)
cddera_rav = pd.rolling_mean(annera_cdd, window=10, min_periods=0, center = True)
cwdobs_rav = pd.rolling_mean(annobs_cwd, window=10, min_periods=0, center = True)
cwdera_rav = pd.rolling_mean(annera_cwd, window=10, min_periods=0, center = True)

cdd = plt.figure()
cdd.set_size_inches(15,5)
ax1 = cdd.add_subplot(121)
ax2 = cdd.add_subplot(122)

ax1.plot(df_prcp.CDD.index, df_prcp.CDD,'b',ms=3.0,alpha=1.0,lw=2)
ax1.plot(df_prcp.CWD.index, df_prcp.CWD,'r',ms=3.0,alpha=1.0,lw=2)

leg=ax1.legend(['CDD','CWD'],prop={'size':10},numpoints=1,markerscale=5.,
                frameon=True,fancybox=True)

ax1.set_ylabel(r'Duration since last event (Days) ')
ax1.set_xlabel('Years')
ax1.set_title(r'CDD & CWD over Zone 1 (Station data)')
ax1.set_ylim(0,200)
#ax1.set_xlim('1953-01-01','1990-12-31')
ax1.grid(True)

ax2.plot(df_era.RCA4_CDD.index, df_era.RCA4_CDD,'b',ms=3.0,alpha=1.0, lw=2)
ax2.plot(df_era.RCA4_CWD.index, df_era.RCA4_CWD,'r',ms=3.0,alpha=1.0, lw=2)
ax2.set_ylabel(r'Duration since last event (Days) ')
ax2.set_xlabel('Years')
ax2.set_title(r'CDD & CWD over Zone 1 (RCA4_ERA)')
ax2.set_ylim(0,200)
ax2.grid(True)

leg=ax2.legend(['CDD','CWD'],prop={'size':10},numpoints=1,markerscale=5.,
                frameon=True,fancybox=True)
plt.show(cdd)
#cdd.savefig('Zone1_cdd.pdf',dpi=300)

#Changes of annual maximum CWD and CDD
cdd_mean1 =...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Call the Get_combined() function to create dataframes out of all data in a folder.
Can you write Python code for it?

%%time
df_tmax = get_combined_df(fpth="Data/zone2/*.csv",var="TMAX")
df_tmin = get_combined_df(fpth="Data/zone2/*.csv",var="TMIN")
df_prcp = get_combined_df(fpth="Data/zone2/*.csv",var="PRCP")

df_prcp.tail(3)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Correlation with ENSO
Can you write Python code for it?


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Correlation with IOD
Can you write Python code for it?

stats.linregress(dmi_mon,dmi_mon)

#Linear Regression of Monthly anomalized data with IOD
gd_data = acc_monthly['1981-11-30':'1990-12-31'] == acc_monthly['1981-11-30':'1990-12-31']
iod_stats = stats.linregress(dmi_mon['1981-11-30':'1990-12-31'][gd_data],
                             acc_monthly['1981-11-30':'1990-12-31'][gd_data])
iodera_stats = stats.linregress(dmi_mon['1981-11-30':'2010-12-31'][good_era],
                             era_monthly['1981-11-30':'2010-12-31'][good_era])
print(iod_stats)
print(iodera_stats)

corr_iod = plt.figure(dpi=72)
corr_iod.set_size_inches(5,5)
ax1 = corr_iod.add_subplot(111)

ax1.plot(dmi_mon['1981-11-30':'1990-08-31'],acc_monthly['1981-11-30':'1990-08-31'],'r.')
fit = np.arange(-3,3,0.1)*iod_stats[0] + iod_stats[0]
ax1.plot(np.arange(-3,3,0.1), fit, 'r-')
ax1.set_ylabel(r'Monthly Precip (Anomalies)')
ax1.set_title('IOD Scatter plot for Zone 1')
ax1.set_xlabel(r'IOD Index ')

corr_iod.text(0.4, 0.85, r'r = -0.0097' ,fontsize=12,color='b')
#corr_iod.savefig('Zone1_Index_IOD.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Correlation with QBO
Can you write Python code for it?


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Correlation with SOI
Can you write Python code for it?


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Duration of extreme events

Here I calculate the time between heavy precipitation (risk) from the observed time directly.
Can you write Python code for it?

low_risk.index[2], low_risk.index[1]

test = low_risk.index[2] - low_risk.index[1]
print("diffrence in days between first and second lowrisk:",test.days)

lowrisk_times = 1
for n, date in enumerate(low_risk.index[lowrisk_times - 1:3]):
    print(date.date(), (date - low_risk.index[n -1]).days)

#Duration
lowrisk_dur = []
mediumrisk_dur = []
risk_time = 1
for n, date in enumerate(low_risk.index[risk_time - 1:]):
    lowrisk_d = (date - low_risk.index[n -1]).days

    lowrisk_dur.append(lowrisk_d)

lowrisk_dur=np.array(lowrisk_dur)
for n, date in enumerate(highmed_risk.index[risk_time - 1:]):
    mediumrisk_d = (date - highmed_risk.index[n -1]).days

    mediumrisk_dur.append(mediumrisk_d)

mediumrisk_dur=np.array(mediumrisk_dur)

#Duration plot
freq = plt.figure(dpi=72)
freq.set_size_inches(20,5)        # Specify the output size
ax1 = freq.add_subplot(121)
ax2 = freq.add_subplot(122)

ax1.plot(low_risk.index, lowrisk_dur ,'b-',ms=2.0,alpha=1.)
ax1.set_title('Zone 2 Duration of consecutive Low risk precip \n events based on absolute threshhold', fontsize=15)
ax1.set_ylabel(r'Days between low risk events',fontsize = 14)
ax1.set_xlabel(r'Years',fontsize = 14)
#ax1.set_xlim('1953-01-03','1990-12-31')
ax1.set_ylim(0,200)
ax1.grid(True)

ax2.plot(highmed_risk.index, mediumrisk_dur,'b-',ms=3.0,alpha=1.)
ax2.set_title(' Zone 2 Duration of consecutive Medium-High risk precip \n based on absolute threshhold',fontsize=15)
ax2.set_ylabel('Days between Medium-High risk events', fontsize = 14)
ax2.set_ylim(0,1500)
ax2.set_xlabel(r"Years", fontsize = 14)
#ax2.set_xlim('1953-01-03','1990-12-31')
ax2.grid(True)
#freq.savefig('Zone2_Duration_Swfdp1.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Duration of extreme rainfall  events based on defined statistical extreme threshold
Duration
  * Operations on time differences using timedelta in the Pandas / datetime packages.
Can you write Python code for it?

test = tmp_yr_data[drought].index[1] - tmp_yr_data[drought].index[0]
print("diffrence in days between first and second flood:",test.days)

def add_days_since(df, truth_array, name_to_add):
    """
    This function takes a dataframe (df) as input, and a truth array
    related to that dataframe (e.g. df=df_prcp, truth_array=drought)
    and will then go through each date in the dataframe, look if the
    truth value is True, and if it is, it will see how long since the
    last truth value occured, and give an integer value (for day count)
    which is then placed in an array, and appended to the original
    dataframe at the end, with the name_to_add as the column name.
    """
    days_since_list = []
    last_day = df.index[0]  # Initilise the state of the 'last' day
    for day in df.index:
        if truth_array[day] == True:
            days_since = day - last_day
            days_since = int(days_since.days)
            last_day = day  # update the state of last day to current true day
            if days_since > 1000: # Just check the values aren't silly
                days_since = np.NAN
        else:
            days_since = np.NAN
        days_since_list.append(days_since)
    days_since_list = np.array(days_since_list)
    df[name_to_add]=days_since_list #adding column to the df_prcp data frame
    return

add_days_since(df=df_prcp, truth_array=drought, name_to_add = 'DS_Last_D')#Days  since last drought
add_days_since(df=df_prcp, truth_array=flood, name_to_add = 'DS_Last_F') #Days since last flood
#add_days_since(df=df_prcp, truth_array=low_risk, name_to_add = 'DS_Last_R')

my_duration = plt.figure()
my_duration.set_size_inches(10,5)
ax1 = my_duration.add_subplot(111)
#ax2 = my_duration.add_subplot(122)

ax1.plot(df_prcp.DS_Last_F.index, df_prcp.DS_Last_F,'-b',ms=3.0,alpha=1.0, lw=2)
ax1.plot(df_prcp.DS_Last_D.index, df_prcp.DS_Last_D,'-r',ms=3.0,alpha=1.0,lw=2)

#ax1.bar(df_prcp.DS_Last_F.index, df_prcp.DS_Last_F, width=0.8, color = 'b')
#ax1.bar(df_prcp.DS_Last_D.index,...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Duration of flood risk events based on cumulative statistical values (boxcar approach)
Can you write Python code for it?

####From the Frequency of occurance its evident that duration between flood events takes upto 22 years (1967-1989)

running_mean = pd.rolling_mean(df_prcp["Accumulated"], window=10, min_periods=3, center = True)
print(np.max(running_mean), np.max(df_prcp["Accumulated"]))


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Extreme events based on statistical values of daily anomalies and percentiles
Can you write Python code for it?

flood_threshold = np.percentile(df_prcp['Acc_anomaly'][mask],90)
drought_threshold = np.percentile(df_prcp['Acc_anomaly'][mask],10)

print('90th percentile = ',flood_threshold)
print('10th percentile = ',drought_threshold)
print('50th percentile = ',np.percentile(df_prcp['Acc_anomaly'][mask],50))
print('99th percentile = ',np.percentile(df_prcp['Acc_anomaly'][mask],99))
#sns.distplot(df_prcp['Acc_anomaly'][mask])

flood_era = np.percentile(df_era['ERA_anomaly'],90)
drought_era = np.percentile(df_era['ERA_anomaly'],10)

print('90th percentile = ',flood_era)
print('10th percentile = ',drought_era)
print('50th percentile = ',np.percentile(df_era['ERA_anomaly'],50))
print('99th percentile = ',np.percentile(df_era['ERA_anomaly'],99))
#sns.distplot(df_prcp['Acc_anomaly'][mask])

my_dist = plt.figure()
my_dist.set_size_inches(10,5)               # Specify the output size
ax1 = my_dist.add_subplot(111)              # Add an axis frame object to the plot (i.e. a pannel)

#Univeriate distribution of Observed daily precipitation.
sns.distplot(df_prcp['Acc_anomaly'][mask],bins=100, norm_hist=True, kde=False) # Filled bars
sns.kdeplot(df_prcp['Acc_anomaly'][mask],shade=False,kernel='gau',cumulative=False,color='k',lw=3)
sns.distplot(era_anom,bins=100, norm_hist=True, kde=False) # Filled bars
sns.kdeplot(era_anom,shade=False,kernel='gau',cumulative=False,color='r',lw=2)
ax1.vlines(np.percentile(df_prcp['Acc_anomaly'][mask],90), 0.00, 0.40, colors='b',linestyle='--',lw=2.0)
ax1.vlines(np.percentile(df_prcp['Acc_anomaly'][mask],50), 0.00, 0.40, colors='b',lw=2.0) #Marker line of Median
ax1.vlines(np.percentile(df_prcp['Acc_anomaly'][mask],10), 0.00, 0.40, colors='b',linestyle='-.',lw=3.0)
leg1=ax1.legend(['Station data KDE','90th percentile','50th percentile','10th percentile','observed anomalies'],
                prop={'size':11},numpoints=1,markerscale=5.,frameon=True,fancybox=True)
ax1.vlines(np.percentile(df_era['ERA_anomaly'],90), 0.00, 0.40,...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Fetching data from web source using Pandas
IOD data,
ENSO data,
SOI data,
QBO30mb data,
QBO50mb data.
Can you write Python code for it?

"""
This is aimed at finding the relationship between extreme weather events
in EA and climate indices(ENSO, QBO, IOD), Also what is the correlation
between the data (and their statistical significance)?
"""
#iod=pd.read_fwf("Data/dmi.nc")
enso=pd.read_fwf("Data/noaa_mei.txt")
soi = pd.read_fwf("Data/noaa_soi.txt", index_col='YEAR')
qbo30 =pd.read_fwf("Data/noaa_qbo30.txt", index_col='YEAR')
qbo50 = pd.read_fwf("Data/noaa_qbo50.txt", index_col='YEAR')
# Make the year integers the index
enso.index = pd.date_range(start='1950-01-01',end='2015-07-01',freq='M')

month_list = soi.keys()
tmp_data = []
for year in soi.index:
    for month in month_list:
        tmp_data.append(soi[month][year])
        #print(month,year,soi[month][year])
tmp_data = np.array(tmp_data)

mon_list = qbo30.keys()
qbo3_data = []
qbo5_data = []
for year in qbo30.index:
    for month in mon_list:
        qbo3_data.append(qbo30[month][year])
        qbo5_data.append(qbo50[month][year])
        #print(month,year,qbo30[month][year])
qbo3_data = np.array(qbo3_data)
qbo5_data = np.array(qbo5_data)

#Creating their data frames
df_soi = pd.DataFrame(data=tmp_data,index=pd.date_range(start='1951-01-01',end='2015-12-31',freq='M'),columns=['SOI'])
df_qbo3=pd.DataFrame(data=qbo3_data,index=pd.date_range(start='1979-01-01',end='2015-12-31',freq='M'),columns=['QBO3'])
df_qbo5=pd.DataFrame(data=qbo5_data,index=pd.date_range(start='1979-01-01',end='2015-12-31',freq='M'),columns=['QBO5'])

#to convert a DataFrame to a TimeSeries
soii = df_soi.unstack()
qbo3 = df_qbo3.unstack()
qbo5 = df_qbo5.unstack()
print(np.mean(soii), np.mean(qbo3), np.mean(qbo5))

#Indian Ocean Dipole Index
f = netCDF4.Dataset("Data/dmi.nc") # Assign the netcdf file
#print(f.variables)                # Show what is in the netcdf file
dmi = f["DMI"][:]                  # Call the DMI and dates and assign them to a variable
dmi_date = f["WEDCEN2"][:]
dmi_dates = netCDF4.num2date(f['WEDCEN2'][:],units = f['WEDCEN2'].units)
dmi_index = pd.to_datetime(dmi_dates)
f.close()              ...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Frequency of extreme rainfall  events based on defined statistical extreme threshold
Can you write Python code for it?

#Frequency
my_ = plt.figure(dpi=72)
my_.set_size_inches(15,5)        # Specify the output size
ax = my_.add_subplot(121)        # Add an axis frame object to the plot (i.e. a pannel)
ax1 = my_.add_subplot(122)

ax.plot(years,flood_freq, 'b-', alpha=0.8)
ax.plot(years[yr_day_count > 350],drought_freq[yr_day_count > 350], 'r-',alpha=0.8)
leg=ax.legend(['Floods','Drought',],prop={'size':10},numpoints=1,markerscale=1.,
                frameon=True,fancybox=True)

ax.set_ylim(0,100)
ax.set_title('Integer count of Extreme precip events(Frequency) \n based on threshold value detection (Zone2)')
ax.set_ylabel(r'Number of Extreme events (counts)')
ax.set_xlabel('Years')
ax.grid(True)

ax1.plot(years[yr_day_count > 350], flood_freq[yr_day_count > 350]/ #Normalized frequency
        yr_day_count[yr_day_count > 350], 'b-', alpha=1.)
ax1.plot(years[yr_day_count > 350],drought_freq[yr_day_count > 350]/
        yr_day_count[yr_day_count > 350],'r-', alpha=1.)
leg=ax1.legend(['Floods','Drought',],prop={'size':10},numpoints=1,markerscale=1.,
                frameon=True,fancybox=True)

ax1.set_title('Extreme precip events Frequency \n based on threshold value detection(Zone2)')
ax1.set_ylabel(r'Normalized count of Extreme events')
ax1.set_xlabel('Years')
ax1.grid(True)
plt.show(my_)
#my_.savefig('Zone2_Percentile_Frequency_plot.pdf',dpi=300)

#Frequency
my_ = plt.figure(dpi=300)
my_.set_size_inches(10,5)        # Specify the output size
ax1 = my_.add_subplot(111)

ax1.plot(years[yr_day_count > 350], fre_rmean[yr_day_count > 350]/yr_day_count[yr_day_count > 350],'b-', lw=2)
ax1.plot(years[yr_day_count > 350], fre_drmean[yr_day_count > 350]/yr_day_count[yr_day_count > 350],'r-', lw=2)
ax1.plot(years_era, erafreq_rmean/365,'g-',  lw=2)
ax1.plot(years_era, erafreq_drmean/365,'k-',  lw=2)

ax1.plot(years[yr_day_count > 350], flood_freq[yr_day_count > 350]/yr_day_count[yr_day_count > 350], 'b.', alpha=1.)
ax1.plot(years[yr_day_count > 350],drought_freq[yr_day_count > 350]/yr_day_count[yr_day_count > 350],'r.',...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Intensity of extreme rainfall  events based on defined statistical extreme threshold
Can you write Python code for it?

#Intensity
my_int = plt.figure(dpi=72)
my_int.set_size_inches(15,5)        # Specify the output size
ax1 = my_int.add_subplot(121)
ax2 = my_int.add_subplot(122)

ax1.errorbar(years,flood_mean,  #Masking missing values
             xerr=None, yerr=flood_sem,color='b', fmt='.', alpha=1.)
ax1.errorbar(years_era, flood_eramean,xerr=None, yerr=flood_erasem,color='k', fmt='.', alpha=1.)

ax1.plot(years, int_rmean,'g--', lw=2)
ax1.plot(years_era, era_rmean,'k--', lw=2)
ax1.set_title('Zone 2 Intensity of R90p events based on \n statistical values of daily anomalies and percentiles', fontsize=14)
ax1.set_ylabel(r'Intensity (mm day$^{-1}$)')
ax1.set_xlabel(r'Year')
leg1=ax1.legend(['Station data','RCA4_ERA'],prop={'size':11},numpoints=1,markerscale=5.,frameon=True,fancybox=True)
#ax1.set_ylim(0,12)
ax1.grid(True)

ax2.errorbar(years,drought_mean,
             xerr=None, yerr=drought_sem, color='r', fmt='.', alpha=1.)
ax2.errorbar(years_era, drought_eramean,xerr=None, yerr=drought_erasem,color='k', fmt='.', alpha=1.)
ax2.plot(years, int_drmean,'g--',alpha=1., lw=2)
ax2.plot(years_era, era_drmean,'k--')
ax2.set_title(' Zone 2 Intensity of R10p events based on \n statistical values of daily anomalies and percentiles ', fontsize =14)
ax2.set_xlabel(r"Years")
ax2.set_ylabel('Intensity(mm day$^{-1}$)')
leg1=ax2.legend(['Station data','RCA4_ERA'],prop={'size':11},numpoints=1,markerscale=5.,frameon=True,fancybox=True)
ax2.grid(True)
#my_int.savefig('Zone2_Percentile_intensity+ERA1_plot.pdf',dpi=300)

##calc the trendline (linear fitting)
trend = np.polyfit(flood_freq[yr_day_count > 350], years[yr_day_count > 350], len(flood_freq[yr_day_count > 350]))

#running_test = pd.rolling_mean(synthetic["vals"], window=10, min_periods=3, center = True)
fre_rmean = pd.rolling_mean(flood_freq, window=10, min_periods=0, center = True)
fre_drmean = pd.rolling_mean(drought_freq, window=10, min_periods=0, center = True)
#ERA
erafreq_rmean = pd.rolling_mean(flood_erafreq, window=10, min_periods=0, center = True)
erafreq_drmean =...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Monthly Correlations
Can you write Python code for it?

selector = acc_monthly.index.month
selectors = enso['ANOM.3'].index.month
selectorss  = dmi_mon.index.month
selector_era = era_monthly.index.month

era_monthly[selector_era == 1].head()

monthly_corr = plt.figure(dpi=300)
monthly_corr.set_size_inches(15,5)
ax = monthly_corr.add_subplot(111)

ax.plot(acc_monthly[selector == 11].index, acc_monthly[selector ==11], 'b', ms=3.0,alpha=0.75)
ax.plot(enso['ANOM.3'][selectors==11].index, enso['ANOM.3'][selectors==11], 'r', ms=3.0,alpha=0.75)
ax.plot(dmi_mon[selectorss==11].index,dmi_mon[selectorss==11], 'g', ms=3.0,alpha=0.75)
leg=ax.legend(['Precip','ENSO3.4 Index','IOD Index'],prop={'size':12},numpoints=1,markerscale=1.,
                frameon=True,fancybox=True)

ax.set_ylabel(r'Anomalies')
ax.set_title(r'Relationship of EA precips Vs ENSO Index & IOD Index for the Month of November (Zone 2) ')
ax.set_xlabel(r'Time (months_Nov)')
ax.grid(True)
#monthly_corr.savefig('Zone1_Correlation_plot.pdf',dpi=300)

#gd = acc_monthly[selector == 5]['1955-01-01':'1990-12-31'] == acc_monthly[selector == 5]['1955-01-01':'1990-12-31']
#stats = stats.linregress(enso['ANOM.3'][selectors==5]['1955-01-01':'1990-12-31'][gd],
#                             acc_monthly[selector==5]['1955-01-01':'1990-12-31'][gd])
#stats
#High r values Mar=-30% April=25% May =44% Oct=35%, Nov=34% ENSO
#High r values June=29% May=42% Oct=42%, Nov=70% Dec=20% IOD

corr_mon = plt.figure(dpi=72)
corr_mon.set_size_inches(5,5)
ax1 = corr_mon.add_subplot(111)

ax1.plot(enso['ANOM.3'][selectors==11]['1955-01-01':'1990-12-31'],
         acc_monthly[selector==11]['1955-01-01':'1990-12-31'],'r.')
#fit = np.arange(-3,3,0.1)*stats[0] + stats[0]
ax1.plot(np.arange(-3,3,0.1), fit, 'r-')
ax1.set_ylabel(r'May precip (mm)')
ax1.set_title('MAY (ENSO) Scatter plot Zone 2')
ax1.set_xlabel(r' ENSO Index ($^{o}$C)')

corr_mon.text(0.2, 0.85, r'r = -0.20968898413341716' ,fontsize=12,color='b')
#corr_mon.savefig('Zone1_Index_MAY.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Plot time series of precipitation for all stations, and also accumulate the data and plot the average rainfall.
2. Time series plots
Daily Mean and SEM values: Mean uncertainty is given by SEM, where:
$SEM = \frac{\sigma}{\sqrt{n-1}}$
Can you write Python code for it?

def calc_SEM(data):
    """
    Calculate Standard error of the mean. No nan's
    should be in the input (numpy) array.
    """
    return np.std(data)/np.sqrt(len(data) - 1)

def gather_daily_stats(date, df):
    """
    For a specified day, given by date, create a short array of
    observed values (obs) excluding the NANs. Return the mean,
    and SEM value.
    Restrictions: more than one observation on a day, not a missing
    value, less than 200 mm per day (which is erroneous).
    """
    obs = np.array([df_prcp[key][day] for key in df_prcp.keys()])
    obs = obs[(obs > -1) & (obs < 200)]

    if len(obs) < 2:
        return np.NAN, np.NAN
    return np.mean(obs), calc_SEM(obs)

# Create an accumulated time series (with SEM uncertainty values)
means = []
sems = []
for day in df_prcp.index:
    tmp_mean, tmp_sem = gather_daily_stats(date=day, df=df_prcp)#['1961-01-01':'1990-12-31'])
    means.append(tmp_mean)
    sems.append(tmp_sem)
means = np.array(means)
sems = np.array(sems)
df_prcp['Accumulated']=pd.Series(means,index=df_prcp.index)  #adding columns to the dataframe!
df_prcp['Acc_SEM']=pd.Series(sems,index=df_prcp.index)

print(np.max(df_prcp['Accumulated']))

# SEM uncertainty values for RCA4_ERA
sems_era = []
for day in df_era.index:
    tmp_sem = np.std(eraint)/np.sqrt(len(eraint) - 1)
    sems_era.append(tmp_sem)
sems_era = np.array(sems_era)
print(sems_era)

ts= plt.figure(dpi=300)
ts.set_size_inches(15,5)      # Specify the figure size
ax1 = ts.add_subplot(111)     # Add an axis frame object to the plot (i.e. a pannel)

#ax1.plot(df_prcp.index, df_prcp.Accumulated,'.g',ms=2.0)

ax1.errorbar(df_era.index, eraint,
             yerr=sems_era, c='b', alpha=1.,  fmt='-')
ax1.set_ylim(0,20)
plt.title("RCA4 ERAINT (Zone 1) Mean Daily Precipitation", fontsize=15)
plt.ylabel("Precipitation (mm day$^{-1}$)", fontsize=15)
plt.xlabel("Years", fontsize=15)
plt.grid(True)
#ts.savefig('Zone1_ERAts.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Possible dependent Variable ###
Can you write Python code for it?

acc_monthly = df_prcp.Acc_anomaly.resample('M', how='mean') #monthly Deseasonalized precip
era_monthly = df_era.ERA_anomaly.resample('M', how = 'mean')
extreme_monthly = df_prcp["Acc_anomaly"][extremes].resample('M', how='mean')


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
RX1day: maximum 1-d Precipitation : Highest precipitation amount in 1-d period
Can you write Python code for it?

for station in df_prcp:
    print(station, np.max(df_prcp[station]))


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
RX5day: maximum 5-d Precipitation : Highest precipitation amount in 5-d period
Can you write Python code for it?

running_total5d = pd.rolling_sum(df_prcp["Accumulated"], window=5, min_periods=5, center = True) #inverse=[::-1]
print(np.max(running_total5d), np.max(df_prcp["Accumulated"]))

my_total = plt.figure()
my_total.set_size_inches(15,5)        # Specify the output size
ax1 = my_total.add_subplot(121)
ax2 = my_total.add_subplot(122)

ax1.plot(running_total.index, running_total,'.g',ms=3.0,alpha=0.75)
ax1.plot(running_total[running_total > 50].index, running_total[running_total > 50],'.r',ms=3.0,alpha=1.0)
leg=ax1.legend(['No risk','flood risk',],prop={'size':10},numpoints=1,markerscale=5.,
                frameon=True,fancybox=True)
ax1.set_ylabel(r'Precipitation (mm day$^{-1}$)',fontsize =14)
ax1.set_xlabel('Years',fontsize =14)
ax1.set_title(r'RX3day:Station data maximum 3-d Precipitation in East Africa',fontsize =14)
ax1.set_ylim(0, 120)
ax1.set_xlim('1952-01-01','2015-12-31')
ax1.grid(True)

ax2.plot(running_total5d.index, running_total5d,'.g',ms=3.0,alpha=0.75)
ax2.plot(running_total5d[running_total5d > 50].index, running_total5d[running_total5d > 50],'.r',ms=3.0,alpha=1.0)
leg=ax2.legend(['No risk','flood risk',],prop={'size':10},numpoints=1,markerscale=5.,
                frameon=True,fancybox=True)
ax2.set_ylabel(r'Precipitation (mm day$^{-1}$)',fontsize =14)
ax2.set_xlabel('Years', fontsize =14)
ax2.set_title(r'RX5day:Station data Maximum 5-d Precipitation in East Africa',fontsize =14)
ax2.set_ylim(0, 120)
ax2.set_xlim('1952-01-01','2015-12-31')
ax2.grid(True)
plt.show(my_total)
#my_total.savefig('Zone1_total_ts.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Reading  the RCA4 - CORDEX data downloaded ICPAC-Kenya data repository
Data obtained from http://197.254.113.174:8081/repository
Can you write Python code for it?

# RCA4_ERAINT
era = netCDF4.Dataset("CORDEX/ECMWF-ERAINT/evaluation/pr30_43_4_13_mean.nc") #zone 1 model data
era2 = netCDF4.Dataset("CORDEX/ECMWF-ERAINT/evaluation/pr28_42_-7_4_mean.nc") #Zone 2 model data
eraint = era2['pr'][:,0,0]

time = era2["time"]
times = netCDF4.num2date(era2['time'][:],units = era2['time'].units)
times_index = pd.to_datetime(times)
#print(times_index)

#creating a dataframes for RCA4 data
df_era = pd.DataFrame(data=eraint,index=pd.date_range(start='1980-01-01',end='2010-12-31'),columns=['ERAINT'])


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Seasonal Correlarion statistics
Can you write Python code for it?

#Expoloring here the seasonal correlations
'''
To do seasonal anomaly correlation
'''
acc_seasonal = acc_monthly.resample('Q', how='mean') #Seasons "JFM, AMJ, JAS, OND"
xtrm_seasonal = extreme_monthly.resample('Q', how='mean') #Seasons "JFM, AMJ, JAS, OND"
era_seasonal = era_monthly.resample('Q', how='mean')
enso_seasonal = enso['ANOM.3'].resample('Q', how='mean')
dmi_seasonal = df_iod['IOD'].resample('Q', how='mean')
soi_seasonal = df_soi['SOI'].resample('Q', how='mean')
qbo_seasonal = df_qbo3['QBO3'].resample('Q', how='mean')

#xtrm_seasonal = extreme_monthly['1954-01-01':'2015-12-31'].resample('4M', how='mean') #Seasons "FMAM, JJAS, ONDJ"

#acc_ssn = acc_monthly.resample('BQ-FEB', how = 'mean') #Seasons "DJF, MAM, JJA, SON"
#enso_ssn = enso['ANOM.3'].resample('BQ-FEB', how='mean') #Seasons "DJF, MAM, JJA, SON"
#dmi_ssn = df_iod['IOD'].resample('BQ-FEB', how='mean')

x = acc_seasonal.index.month
x1 = enso_seasonal.index.month
x2 = era_seasonal.index.month
x3 = xtrm_seasonal.index.month
x4 = dmi_seasonal.index.month
x5 = soi_seasonal.index.month
x6 = qbo_seasonal.index.month

OND_Obs = acc_seasonal[x == 12]
OND_ens = enso_seasonal[x1 == 12]
OND_era = era_seasonal[x2 == 12]
OND_xtr = xtrm_seasonal[x3 == 12]
OND_iod = dmi_seasonal[x4 == 12]
OND_soi = soi_seasonal[x5 == 12]
OND_qbo = qbo_seasonal[x6 == 12]
#JJAS season was used for zone 1 while OND used in Zone 2

clima = plt.figure()
clima.set_size_inches(15,5)
ax = clima.add_subplot(111)
ax.plot(OND_xtr.index, OND_xtr, 'r', ms=3.0,alpha=1.0,lw=2)
ax.plot(OND_ens.index, OND_ens, 'b', ms=3.0,alpha=1.0,lw=2)
ax.plot(OND_iod.index, OND_iod, 'g', ms=3.0,alpha=1.0,lw=2)
ax.set_title('OND Average ENSO/SOI Indices Correlation with Zone 2 Extreme Precipitation ', fontsize = 15)
ax.set_xlabel(r'Years', fontsize = 15)
ax.set_ylabel(r'Anomaly ', fontsize = 15)
#ax.set_xlim('1950' '1990')
#ax.set_ylim(-5,10)
leg=ax.legend(['Station data','ENSO 3.4','IOD'],prop={'size':10},
             ...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Summary statistics
Frequency of extreme events based on absolute threshhold set by SWFDP-RSMC-Nairobi
Can you write Python code for it?

#low_risk.groupby( [low_risk.index.year, low_risk.index.month, low_risk.index.day] ).count()
lwrisk_freq = low_risk.groupby(low_risk.index.year).count()/365 #To get normalized freq per annum
mdrisk_freq = medium_risk.groupby(medium_risk.index.year).count()/365
mdhgrisk_freq = highmed_risk.groupby(highmed_risk.index.year).count()/365
hgrisk_freq = high_risk.groupby( high_risk.index.year ).count()
#print(hgrisk_freq)
#ERA
yrisk_freq = yrisk.groupby(yrisk.index.year).count()/365

frq_rmean_low = pd.rolling_mean(lwrisk_freq, window=10, min_periods=0, center = True)
frq_rmean_med = pd.rolling_mean(mdrisk_freq, window=10, min_periods=0, center = True)
frq_rmean_medhig = pd.rolling_mean(mdhgrisk_freq, window=10, min_periods=0, center = True)
frq_rmean_hig = pd.rolling_mean(hgrisk_freq, window=10, min_periods=0, center = True)

#ERA
frq_rmean_yrisk = pd.rolling_mean(yrisk_freq, window=10, min_periods=0, center = True)

#Frequency plot
freq = plt.figure(dpi=300)
freq.set_size_inches(15,5)        # Specify the output size
ax1 = freq.add_subplot(121)
ax2 = freq.add_subplot(122)

ax1.plot(lwrisk_freq.index, lwrisk_freq ,'b-',ms=10.0,alpha=1., lw=1.5)
ax1.plot(yrisk_freq.index, yrisk_freq ,'g-',ms=3.0,alpha=1., lw=1.5)
ax1.plot(lwrisk_freq.index, frq_rmean_low,'r--', linewidth=2, alpha=1.)
ax1.plot(yrisk_freq.index, frq_rmean_yrisk,'r--', linewidth=2, alpha=1.)
ax1.set_title('Zone 2 Frequency of Low risk extreme precip \n based on Absolute Threshhold')
ax1.set_ylabel(r' Normalized counts of low risk events', fontsize = 12)
ax1.set_xlabel(r'Years', fontsize = 12)
#ax1.set_xlim(1953, 1990)
leg1=ax1.legend(['Station data','RCA4_ERA'],prop={'size':11},numpoints=1,markerscale=5.,frameon=True,fancybox=True)

ax1.grid(True)

ax2.plot(mdhgrisk_freq.index, mdhgrisk_freq ,'b-',ms=5.0,alpha=1.,lw=1.5)
ax2.plot(mdhgrisk_freq.index, frq_rmean_medhig,'r--', linewidth=2, alpha=1.,lw=1.5)
ax2.set_title('Zone 2 Frequency of Medium-High risk extreme precip \n based on Absolute Threshhold')
ax2.set_xlabel(r"Years", fontsize =...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Summary statistics
Frequency of flood risk events based on cumulative statistical values (boxcar approach)
Can you write Python code for it?

flood_risk = running_total[running_total > 50]   #Gives days when precipitation 3-d precip totals exceeded the flood
flood_risk5d = running_total5d[running_total5d > 50] #threshold

floodrisk_freq = flood_risk.groupby( flood_risk.index.year).count()/365   #Gives Noramlized frequency
floodrisk_freq5d = flood_risk5d.groupby( flood_risk5d.index.year).count()/365

r3mean = pd.rolling_mean(flood_risk, window=10, min_periods=0, center = True)
r5mean = pd.rolling_mean(flood_risk5d, window=10, min_periods=0, center = True)

#Frequency plot
flood_freq = plt.figure()
flood_freq.set_size_inches(15,5)
ax1 = flood_freq.add_subplot(121)
ax2 = flood_freq.add_subplot(122)

ax1.plot(floodrisk_freq.index, floodrisk_freq ,'r.',ms=7.0,alpha=1.)
#ax1.plot(r3mean.index, r3mean ,'r-',ms=3.0,alpha=1.)
ax1.set_title('Flood risk events Frequency RX3day (Zone1) based on\n cumulative statistical values (boxcar approach)')
ax1.set_ylabel(r'Normalized count of events')
ax1.set_xlabel(r'Year')
ax1.grid(True)

ax2.plot(floodrisk_freq5d.index, floodrisk_freq5d ,'r.',ms=7.0,alpha=1.)
#ax2.plot(floodrisk_freq5d.index, r3mean ,'r-',ms=3.0,alpha=1.)
ax2.set_title('Flood risk events Frequency RX5day (Zone1) based on\n cumulative statistical values (boxcar approach)')
ax2.set_ylabel(r'Normalized count of events')
ax2.set_xlabel(r'Year')
ax2.grid(True)
#flood_freq.savefig('Zone1_RX3d.pdf',dpi=300)


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
Summary statistics
Intensity, Duration and Frequency (IDF) of extreme events based on defined statistical extreme threshold
Can you write Python code for it?

#groupby can be used to querey the dataset
#OR can write hacks like this, to pull out data based on the index
#---Splitting the data into groups based on extreme threshhold
for year in range(min(df_prcp.index.year),max(df_prcp.index.year)):
    wet_extreme = df_prcp["Acc_anomaly"][flood][df_prcp["Acc_anomaly"][flood].index.year == year]
    dry_extreme = df_prcp["Acc_anomaly"][drought][df_prcp["Acc_anomaly"][drought].index.year == year]

    print(year,len(wet_extreme), year,len(dry_extreme))
    break

#For station dataset
flood_freq = []
drought_freq = []
yr_day_count = []
years = []
flood_mean=[]
flood_sem =[]
drought_mean=[]
drought_sem =[]

for year in range(min(df_prcp.index.year),max(df_prcp.index.year)):
    tmp_yr_data = df_prcp["Acc_anomaly"][df_prcp.index.year == year]  # pool data for each year
    #print(tmp_yr_data)
    yr_day_count.append(tmp_yr_data.count())
    years.append(year)
    if tmp_yr_data.count() > 1:
        flood_freq.append(len(tmp_yr_data[flood]))
        #print(tmp_yr_data[flood])
        drought_freq.append(len(tmp_yr_data[drought]))
        flood_mean.append(np.nanmean(tmp_yr_data[flood]))
        flood_sem.append(calc_SEM(tmp_yr_data[flood]))
        drought_mean.append(np.nanmean(tmp_yr_data[drought]))
        drought_sem.append(calc_SEM(tmp_yr_data[drought]))
    else:
        flood_freq.append(np.NAN)
        drought_freq.append(np.NAN)

        flood_mean.append(np.NAN)
        flood_sem.append(np.NAN)
        drought_mean.append(np.NAN)
        drought_sem.append(np.NAN)

flood_freq = np.array(flood_freq)
drought_freq = np.array(drought_freq)
yr_day_count = np.array(yr_day_count)
years = np.array(years)
flood_mean = np.array(flood_mean)
flood_sem = np.array(flood_sem)
drought_mean = np.array(drought_mean)
drought_sem = np.array(drought_sem)

flood_erafreq = []
drought_erafreq = []
yr_day_eracount = []
years_era = []
flood_eramean=[]
flood_erasem =[]
drought_eramean=[]
drought_erasem =[]

for year in...


I am working on the file "GHCN_Data_Analysis.ipynb".
The first task is:
extreme event based on cumulative statistical values (boxcar approach)
Can you write Python code for it?

# Type of boxcar function
"""
Interested in the sum of mean precip for current day plus the next 2 days
If the sum is greater than 100mm, then this could be a signal
for potential medium-high risk for flood, low risk if greater than 50mm but
less than 100.
"""
#RX3day: maximum 3-d Precipitation : Highest precipitation amount in 3-d period
running_total = pd.rolling_sum(df_prcp["Accumulated"], window=3, min_periods=3, center = True) #inverse=[::-1]
print(np.max(running_total), np.max(df_prcp["Accumulated"]))

print(running_total[7000:7005])


I am working on the file "GINI.ipynb".
The first task is:
Comparing with GDP, population, gender inequality, even subjective "satisfaction indexes" and the like would be much more interesting. To tell a real story, we need to show some correlation, and provide some narrative and/or visualization to explain Gini. At the end of the day, perhaps the real story is that Gini is not a great universal indicator.
Colors
Where the graph at http://www.informationisbeautiful.net/visualizations/what-are-wallst-protestors-angry-about/ was using a very gradual change in hue based on the value (redundant, the width and the number already shows this), it is so subtle that it doesn't show any significant difference between two consecutive rows.
A better use of color is to highlight our focus, or for reference lines. With that in mind, let's enhance our bar plot with judicious use of color for making it quicker to spot the US data.
Can you write Python code for it?

current_ax = gini_df.plot(kind='barh', color=sns.color_palette()[0])
current_ax.patches[list(gini_df.index).index("United States")].set_facecolor('#cc5555')
current_ax.set_title('Gini index (%) in 2009')
current_ax.vlines(39.99, 0, len(gini_df), color=sns.color_palette()[2])
current_ax.annotate('Average for\n1989-2010',
                    (40, 2),
                    xytext=(20, 10),
                    textcoords='offset points',
                    arrowprops=dict(arrowstyle='-|>'))


I am working on the file "GINI.ipynb".
The first task is:
Getting the data
Before going into the purely visual aspects and how effective they are at conveying a story, I want to understand what data we are dealing with. At the bottom of the graph, there is a bit.ly URL that points to a google drive document. Adding export?format=xlsx will allow us to download this document as an excel spreadsheet, which can then be sliced and diced easily with the pandas analytics module.
Can you write Python code for it?

!wget 'https://docs.google.com/spreadsheets/d/1N_Hc-xKr7DQc8bZAvLROGWr5Cr-A6MfGnH91fFW3ZwA/export?format=xlsx&id=1N_Hc-xKr7DQc8bZAvLROGWr5Cr-A6MfGnH91fFW3ZwA' -O wallstreet.xlsx

df = pd.read_excel('wallstreet.xlsx', skiprows=1, index_col = 'Country')

df.describe()


I am working on the file "GINI.ipynb".
The first task is:
We will get just the data for 2009. Not only it is recent, but it is plenty of data points to represent at once. This will also address the other issue with the data: in the raw form, it is too numerous and will overload the reader if presented as is. We will also load the US data, since it is supposed to tell the story of <b>'occupy wallstreet'</b>. If we are missing further critical data, we can always add a specific data point later, as we are keeping the original data frame untouched.
Can you write Python code for it?

gini_df = df[(df.Year==2009)|(df.index=='United States')]['Gini']  # Only 2009, and choose only the gini columns (and the index, country)

gini_df

current_ax = gini_df.plot(kind='barh', color=sns.color_palette()[0])
current_ax.set_title('Gini index (%) in 2009')
current_ax.vlines(39.99, 0, len(gini_df), color=sns.color_palette()[2])


I am working on the file "GLM-AL-2015/Flash sorting - AL.ipynb".
The first task is:
Case-specific time series data
 Locations of all flashes contributing to the analyses
Lets you identify that only one cell was tracked
Can you write Python code for it?


I am working on the file "GLM-AL-2015/Flash sorting - AL.ipynb".
The first task is:
Each of the grid type folders contains a CSV file with statistics of the pixels making up the image.
Can you write Python code for it?


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
... or the flash IDs for the first hundred sources.
Can you write Python code for it?


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
Inside the HDF5 files
To look inside the HDF5 file, we'll use the Python library known as PyTables.
The file object itself can be printed to describe the file's contents.
Can you write Python code for it?

import os
import tables, pandas
import numpy as np

results_dir = "/data/GLM-wkshp/flashsort/results/"
h5name = os.path.join(results_dir, "h5_files/2009/Apr/10/LYLOUT_090410_180000_3600.dat.flash.h5")
h5 = tables.open_file(h5name)
print h5


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
Interactive animation
Even better, we can use the notebook's widget system to make an interactive version of the above.
Can you write Python code for it?

from IPython.html.widgets import interactive
from IPython.display import display

from matplotlib.colors import LogNorm

def plot_for_frame(frame=0):
    # get the data
    t = NCs.times[frame]
    xedge, yedge, data = NCs.data_for_time(t)

    # plot a frame of data
    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(111)
    mesh = ax.pcolormesh(xedge, yedge, data,
                         vmin=1, vmax=100, norm=LogNorm(), cmap='gray_r')
    ax.axis(limits)
    plt.colorbar(mesh, ax=ax)
    title = "{1} at {0}".format(t.isoformat(), nc_field)
    ax.set_title(title)
    return fig

N_frames = len(NCs.times)
w = interactive(plot_for_frame, frame=(0, N_frames-1))
display(w)


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
Let's look at flash #27. The indexing operation (colon) is what actually lets us access the data, instead of just the description of the data column.
Can you write Python code for it?

print event_table.cols.flash_id
query = (event_table.cols.flash_id[:] == 27)
fl_27_events = event_table[query][:]
lats, lons, alts, times = fl_27_events['lat'], fl_27_events['lon'], fl_27_events['alt'], fl_27_events['time']
for lat, lon in zip(lats, lons):
    print lat, lon

import matplotlib
matplotlib.rc('font', size=12)
%matplotlib inline

import matplotlib.pyplot as plt
plt.scatter(lons, lats, c=times)
plt_range =  (lons.min(),lons.max(),lats.min(),lats.max())
print plt_range
plt.axis(plt_range)


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
Now, let's get the associated flash properties. Use pandas to make an appealing print of the the data table.
Can you write Python code for it?

flash_table = h5.root.flashes.LMA_090410_180000_3600
fl_27 = flash_table[flash_table.cols.flash_id[:] == 27] # This is a numpy array with a named dtype
fl_27 = pandas.DataFrame(fl_27)

fl_27

fl_27_events = pandas.DataFrame(fl_27_events)
fl_27_events


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
Or the first ten sources ...
Can you write Python code for it?


I am working on the file "GLM-AL-2015/Reading the flash-sorted files.ipynb".
The first task is:
We can get all the altitudes ...
Can you write Python code for it?


I am working on the file "GNB.ipynb".
The first task is:
Regresión Logística
Can you write Python code for it?

# Transform training outputs
fun = lambda x: 1 if x == 1 else 0
Y = np.array(list(map(fun, iris.target)))
X = iris.data[:, :1]

# Logistic classifier and training
log_classifier = None
@interact(loss=['log', 'hinge', 'perceptron'], penalty=['none', 'l1', 'l2'])
def log_classify(loss='log', penalty=None):
    global log_classifier
    log_classifier = SGDClassifier(loss=loss, penalty=penalty, max_iter=1000)
    log_classifier.fit(X, Y)
    print(log_classifier.coef_)
    print(log_classifier.intercept_)

sample_x = np.expand_dims(np.linspace(0, 8), axis=1)
sample_y = log_classifier.intercept_ + (sample_x * log_classifier.coef_)
plt.scatter(X, Y)
plt.plot(sample_x, sample_y, color='r')

log_classifier.coef_[:,]


I am working on the file "GNB.ipynb".
The first task is:
The Iris Dataset
This data sets consists of 3 different types of irises'
(Setosa, Versicolour, and Virginica) petal and sepal
length, stored in a 150x4 numpy.ndarray
The rows being the samples and the columns being:
Sepal Length, Sepal Width, Petal Length and Petal Width.
The below plot uses the first two features.
See here <https://en.wikipedia.org/wiki/Iris_flower_data_set>_ for more
information on this dataset.
Can you write Python code for it?

print(__doc__)

# Code source: Gaël Varoquaux
# Modified for documentation by Jaques Grobler
# License: BSD 3 clause

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn import datasets
from sklearn.decomposition import PCA

# import some data to play with
iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
y = iris.target

x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5

plt.figure(2, figsize=(8, 6))
plt.clf()

# Plot the training points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Set1,
            edgecolor='k')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())

# To getter a better understanding of interaction of the dimensions
# plot the first three PCA dimensions
fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)
X_reduced = PCA(n_components=3).fit_transform(iris.data)
ax.scatter(X_reduced[:, 0], X_reduced[:, 1], X_reduced[:, 2], c=y,
           cmap=plt.cm.Set1, edgecolor='k', s=40)
ax.set_title("First three PCA directions")
ax.set_xlabel("1st eigenvector")
ax.w_xaxis.set_ticklabels([])
ax.set_ylabel("2nd eigenvector")
ax.w_yaxis.set_ticklabels([])
ax.set_zlabel("3rd eigenvector")
ax.w_zaxis.set_ticklabels([])

plt.show()

# Plot nth feature
@interact(n=(1,4))
def plot_nth(n=1):
    X = iris.data[:, n-1:n]
    Y = np.expand_dims(iris.target, axis=1)
    plt.scatter(X, Y)

# Gaussian classifier and training
classifier = GaussianNB()
classifier.fit(iris.data, iris.target)

classifier.predict(np.array([[1,9,0,1]]))


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Agora vamos salvar a base para um arquivo CSV usando a função "to_csv" do Pandas:
Can you write Python code for it?


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Agora vamos usar a função "read_csv" do Pandas para abrir o arquivo "receitas_candidatos_2016_brasil.txt". Esse arquivo possui o consolidado da prestação de contas de todos os candidatos nas eleições de 2016. É um arquivo de quase 1Gb. Temos alguns parâmetros:

delimiter: ";" (os arquivos brasileiros possuem o delimitador ";" em vez da ",")
decimal: "," (preciamos informar que no formato numérico brasileiro a casa decimal é separada por vírgula)
encoding: "latin1" (o padrão brasileiro de encoding de caracteres usado pelo governo equivale ao "latin1" em Python)
low_memory=False (o Pandas tenta adivinhar qual é o tipo de dado em cada coluna para economizar memória - número, texto etc - como uma das colunas possui mais de um tipo de dado, precisamos dizer ao Pandas que ele vai precisar usar mais memória)
Can you write Python code for it?

base = pd.read_csv(
    'data/receitas_candidatos_2016_brasil.txt',
    delimiter=";",
    decimal=",",
    encoding="latin1",
    low_memory=False)


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Buscando o sexo
Agora que temos a base que precisamos para fazer as análises, precisamos buscar o sexo de todos os candidatos.
Vamos usar a coluna "Sequencial Candidato" na base que acabamos de criar e vamos compará-la com uma coluna de mesmo nome na base "consulta_cand_2016_brasil.txt".
Essa base foi criada unindo todos os arquivos TXT do TSE com um simples comando no terminal:
cat *.txt > consulta_cand_2016_brasil.csv
Você pode renomear depois para .txt (como abaixo) ou deixar .csv mesmo. Agora basta criarmos um novo DataFrame com a base que acabamos de definir:
Can you write Python code for it?

candidatos = pd.read_csv(
    'data/consulta_cand_2016_brasil.txt',
    delimiter=";",
    decimal=",",
    encoding="latin1",
    header=None,
    low_memory=False)

candidatos.columns


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Como percebemos acima, o arquivo de candidaturas não possui cabeçalho. Foi por isso que um dos parâmetros na hora de importar o arquivo foi "header=None".
Com isso, o Pandas nomeou em sequência as colunas do 0 até o 45, totalizando 46 colunas.
Para a consulta de sexo dos candidados vamos renomear as colunas 10, 11 e 30. Assim, criaremos uma nova base apenas com os nomes, o número sequencial e o sexo dos candidatos. Vamos usar essa base para acrescentar a informação de sexo na nossa tabela de doações:
Can you write Python code for it?

candidatos.rename(columns={10: "Nome Candidato", 11:"Sequencial Candidato", 30:"Sexo"}, inplace=True)

candidatos_sexo = pd.DataFrame(candidatos, columns=["Sequencial Candidato", "Nome Candidato", "Sexo"])

candidatos_sexo


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Dividindo entre prefeitos e vereadores
Os valores de doação para prefeitos e vereadores são muito diferentes.
Vamos criar uma base apenas com os cargos de prefeito e outra apenas com vereadores:
Can you write Python code for it?

merge_filtrado = merge[merge['Tipo receita'] == u"Recursos de partido político"]
prefeitos = merge_filtrado[merge_filtrado['Cargo'] == u"Prefeito"]
vereadores = merge_filtrado[merge_filtrado['Cargo'] == u"Vereador"]


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Essa tabela tem muitas colunas que não precisamos usar na nossa análise. Vamos recuperar os nomes das colunas e depois vamos fazer um recorte para gerar uma base apenas com as colunas que queremos. Para recuperar as colunas de uma base, digite:
Can you write Python code for it?


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Fatiando a base
Agora, vamos criar uma nova variável chamada "base_cortada" e vamos criar um novo DataFrame - o nome que o Pandas dá para as bases de dados que você manipula por meio do módulo - com as seguintes colunas:

Sequencial Candidato (essa é a chave que vai nos ajudar a buscar os dados de gênero depois)
Nome Candidato (nome para referência mais rápida)
UF (estado do candidato)
Nome da UE (cidade)
Sigla Partido
Cargo (Prefeito ou Vereador)
Nome do doador (nome do diretório ou comissão que doou)
Número partido doador (número da sigla registrado junto ao TSE)
Número candidato doador (número do candidato doador, se aplicável)
Valor receita (valor da doação)
Tipo de receita (categoria que nos ajuda a identificar se a doação foi de pessoa física ou de um partido, por exemplo)
Fonte receita (se aplicável)
Especie recurso (depósito em espécie? transferência eletrônica? etc)
Can you write Python code for it?

base_cortada = pd.DataFrame(base, columns=[
                    u"Sequencial Candidato",
                    u"Nome Candidato",
                    u"UF",
                    u"Nome da UE",
                    u"Sigla  Partido",
                    u"Cargo",
                    u"Nome do doador",
                    u"Número partido doador",
                    u"Número candidato doador",
                    u"Valor receita",
                    u"Tipo receita",
                    u"Fonte receita",
                    u"Especie recurso"
                ])


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Qual foi o valor máximo doado de uma vez só?
Também queremos saber quais foram os valores máximos doados por cada partido em cada um dos cargos.
Geramos outra tabela dinâmica, assim:
Can you write Python code for it?

maximos_prefeitos = prefeitos.pivot_table(index=[u'Número partido doador'],
                            values=['Valor receita'],
                            columns=["Sexo"],
                            aggfunc=max)

maximos_vereadores = vereadores.pivot_table(index=[u'Número partido doador'],
                            values=['Valor receita'],
                            columns=["Sexo"],
                            aggfunc=max)

maximos_prefeitos.to_csv("maximos_prefeitos.csv", encoding = "utf-8")
maximos_vereadores.to_csv("maximos_vereadores.csv", encoding = "utf-8")

maximos_vereadores


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Unindo tabelas pelo número sequencial
Vamos unir as duas tabelas usando a coluna que elas possuem em comum: Sequencial Candidato. À esquerda, vamos usar a nossa base cortada, à direita a base de candidatos com o sexo. O ponto de união das duas bases será a coluna Sequencial Candidato.
O Pandas já faz o trabalho pra gente: onde o número sequencial na tabela com os sexos for X, por exemplo, ele vai acrescentar o valor do sexo automaticamente. Funciona como um VLOOKUP/PROCV ou index/match:
Can you write Python code for it?

left = base_cortada
right = candidatos_sexo

merge = pd.merge(left,right, on="Sequencial Candidato")


I am working on the file "GN_receitas_candidatos_2016.ipynb".
The first task is:
Vamos ver como ficou a nossa nova base cortada:
Can you write Python code for it?


I am working on the file "GOT_character_prediction_dim_reduction.ipynb".
The first task is:
Group by Popularity
Can you write Python code for it?

# Reorder columns so popularity is last
df = pd.concat([df.ix[:,:-2], df.ix[:,-1:], df.ix[:,-2]], axis=1)

# Split Data into X and Y vectors, where Y is popularity
y = df.ix[:,-1].values
x = df.ix[:,:-1].values

# Standardize features by removing the mean and scaling to unit variance
standard_scaler = StandardScaler()
x_std = standard_scaler.fit_transform(x)

# t-distributed Stochastic Neighbor Embedding (t-SNE)
# Compress X vector to two dimensions
tsne = TSNE(n_components=2, random_state=0)
x_2d = tsne.fit_transform(x_std)

# Scatter Plot
plt.figure()

# Create continous color map for the 0-1 popularity
cmap = cm.seismic

# Fill Scatter Plot
for idx, cl in enumerate(np.unique(y)):
    plt.scatter(x=x_2d[y==cl,0], y=x_2d[y==cl,1], c=cmap(cl), marker='o', label=cl, lw=0)

# Labels and Legend
plt.xlabel('X in t-SNE')
plt.ylabel('Y in t-SNE')
plt.title('t-SNE visualization of data')

normalize = clr.Normalize(vmin=y.min(), vmax=y.max())
scalarmappaple = cm.ScalarMappable(norm=normalize, cmap=cmap)
scalarmappaple.set_array(y)
plt.colorbar(scalarmappaple, label='Popularity')

plt.show()


I am working on the file "GOT_deaths_dim_reduction.ipynb".
The first task is:
Group by House
Can you write Python code for it?

# Reorder columns so Allegiance is last
df = pd.concat([df.ix[:,1:], df.ix[:,0]], axis=1)

# Split Data into X and Y vectors, where Y is allegiance
y = df.ix[:,-1].values
x = df.ix[:,:-1].values

# Standardize features by removing the mean and scaling to unit variance
standard_scaler = StandardScaler()
x_std = standard_scaler.fit_transform(x)

# t-distributed Stochastic Neighbor Embedding (t-SNE)
# Compress X vector to two dimensions
tsne = TSNE(n_components=2, random_state=0)
x_2d = tsne.fit_transform(x_std)

# Scatter Plot
plt.figure()

# Create color map for the 20 Allegiances and two different alternating markers
colors = cm.rainbow(np.linspace(0, 1, int((max(y)+1))))
markers = np.tile(['o','s'], 11)

# Fill Scatter Plot
for idx, cl in enumerate(np.unique(y)):
    plt.scatter(x=x_2d[y==cl,0], y=x_2d[y==cl,1], c=colors[cl], marker=markers[idx], label=le.inverse_transform(cl))

# Labels and Legend
plt.xlabel('X in t-SNE')
plt.ylabel('Y in t-SNE')
plt.legend(loc='right', bbox_to_anchor=(1.7, 0.6), ncol=2)
plt.title('t-SNE visualization of test data')

plt.show()


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
A sanity check for the extrapolated code phase at the start of the recording. In units of chips it should be close to the code offset that we have indicated to track-gps-l1.py.
Can you write Python code for it?

(np.polyval(poly_code, 0) % 1e-3) * chip_rate - code_offset


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
Another cross-check. Now that we know that the UHD timestamp is reasonably accurate, we use it to compute the time of flight of each satellite at the beginning of the recording. We check the difference between the code offset reported by acquire-gps-l1.py and the code offset computed by subtracting from the UHD timestamp the time of flight. We show the result in units of chips. All the satellites should show a very similar result (with a fraction of a chip). Moreover, the results should be small (a few chips), since the UHD timestamp error is a few microseconds.
Note that G11, which was marked as unhealthy, is off by ~2 chips.
Can you write Python code for it?


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
Now we use the compute binary, which uses RTKLIB to compute the time of flight of the signal of the satellite at the start of the recording, using the known position of the antenna and the satellite ephemerides. The calculated time of flight is inserted here and used to derive the GPS time at the start of the recording.
Can you write Python code for it?

time_of_flight = 0.074602692582
gps_time_start = tx_time_start + time_of_flight
gps_time_start


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
Read some of the columns of the output of track-gps-l1.py into numpy arrays.
Can you write Python code for it?

with open('track.txt') as f:
    track = f.readlines()

iq = np.array([float(a.split()[1]) + 1j*float(a.split()[2]) for a in track])
samp = np.array([int(a.split()[-1]) for a in track]) + samples_skipped
carrier_f = np.array([float(a.split()[3]) for a in track])
code_cyc = np.array([int(a.split()[-5]) for a in track])
code_p = np.array([float(a.split()[-4]) for a in track])

samp_seconds = samp / samp_rate


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
Some parameters:
Can you write Python code for it?

chip_rate = 1.023e6  # GPS L1 C/A chip rate
samp_rate = 4e6  # This should match the IQ recording sample rate


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
Symbols are 20 dumps long. We find where symbols start by integrating the symbols for all possible offsets and choosing the one that maximizes the amplitude.
Can you write Python code for it?

t0 = 8000  # Use data mid-way the recording
length = 4000  # Use only 4 seconds of data
dumps_per_symbol = 20
symbol_sums = np.empty(dumps_per_symbol)
for j in range(dumps_per_symbol):
    symbols = np.average(iq.real[t0+j:][:length].reshape(-1, dumps_per_symbol),
                         axis=1)
    symbol_sums[j] = np.average(np.abs(symbols))

plt.plot(symbol_sums, 'o-')
plt.xticks(np.arange(dumps_per_symbol))
plt.grid()
plt.xlabel('Offset (dumps)')
plt.ylabel('Amplitude')
plt.title('Symbol amplitude');


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
The carrier frequency can be used to check if the tracking loops are locked. After a second or so, the carrier frequency should follow the Doppler of the satellite (plus receiver clock drift).
Can you write Python code for it?

plt.plot(samp_seconds, carrier_f)
plt.title('Carrier frequency')
plt.xlabel('Time (s)')
plt.ylabel('Frequency (Hz)');


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
The track-gps-l1.py script skips some samples at the beginning of the IQ file in order to start at the beginning of a PRN repetition. Here we calculate how many samples were skipped.
Can you write Python code for it?

code_offset = 151.6  # This should match the code offset used in track-gps-l1.py

samples_skipped = int(samp_rate*0.001*((1023-code_offset)/1023))
samples_skipped


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
To get the TOW, we need to read a 17-bit number somewhere in the subframe (in the 2nd word of the subframe), and XOR it with the last bit of the previous word. The TOW is given in units of 6 seconds, so we multiply it by 6 to obtain seconds.
Can you write Python code for it?

def get_tow(bits, preamble_start):
    # handle inversion according to the last bit of the previous word
    tow = bits[preamble_start + 30:][:17] * (-1) * bits[preamble_start + 29]
    tow = ((tow + 1) / 2).astype('uint8')
    tow = np.packbits(tow)
    tow = ((np.int32(tow[0]) << 16) | (np.int32(tow[1]) << 8) | np.int32(tow[2])) >> 7
    tow = tow * 6
    return tow


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
We fit a polynomial of degree 2 ignoring the first 5 seconds in order to extrapolate back a more accurate value for the code phase at the start of the recording.
Can you write Python code for it?

sel_fit = samp_seconds >= 5
poly_code = np.polyfit(samp_seconds[sel_fit], code_secs[sel_fit], 2)
fig, axs = plt.subplots(2, 1, sharex=True)
axs[0].plot(samp_seconds, code_secs - np.polyval(poly_code, samp_seconds))
axs[1].plot(samp_seconds, code_secs - np.polyval(poly_code, samp_seconds))
axs[1].set_ylim((-1e-9, 1e-9))
axs[0].set_ylabel('Difference (s)')
axs[1].set_ylabel('Difference (s)')
plt.xlabel('Time (s)')
plt.suptitle('Difference between code phase and degree 2 polynomial fit');


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
We now compare the GPS time with the timestamp that UHD has included in the recording metadata using the PPS synchronization (we have converted it to TOW taking into account the 18 second difference between GPS and UTC).
Can you write Python code for it?

usrp_timestamp = 41542.2147593125

gps_time_start - usrp_timestamp


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
We now integrate and plot the symbols.
Can you write Python code for it?

symbols = iq[symbol_offset:]
symbols = symbols[:symbols.size//dumps_per_symbol*dumps_per_symbol]
symbols = np.average(symbols.reshape(-1, dumps_per_symbol), axis=1)

symbols_seconds = samp_seconds[symbol_offset:][::dumps_per_symbol][:symbols.size]

plt.plot(symbols_seconds, symbols.real, '.', label='I')
plt.plot(symbols_seconds, symbols.imag, '.', label='Q')
plt.title('Symbols')
plt.legend()
plt.xlabel('Time (s)')
plt.ylabel('Amplitude');


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
We try to search for correlations that achieve the maximum amplitude and that differ by 300 symbols (6 seconds) to guess which correlations really correspond to the preamble.
Can you write Python code for it?

np.where(corr_preamble == 1)[0]

np.where(corr_preamble == -1)[0]

preamble_start = 92


I am working on the file "GPS_timing/GPS timing.ipynb".
The first task is:
We use this extrapolation and the TOW at the first dump to compute the satellite transmission time at the beginning of the recording.
Can you write Python code for it?

# Here -1e-3 accounts for the fact that code_secs starts counting
# at the PRN repetition that starts just before the recording start,
# while tow_first_dump refers to the first PRN repetition that starts
# after the recording start.
tx_time_start = np.polyval(poly_code, 0) + tow_first_dump - 1e-3
tx_time_start


I am working on the file "GPU_Introduction.ipynb".
The first task is:
At the end of the calcuation, use the copy_to_host method to copy back to the CPU array.
Can you write Python code for it?


I am working on the file "GPU_Introduction.ipynb".
The first task is:
GPU Acceleration using CUDA
We can run this faster using CPU hardware.
The first step is to set up the arrays in GPU device memory.
The input array needs to be copied over, and we need to allocate space for
the output array on the GPU device.
After the GPU operation, we will need to copy the output back to host memory
Can you write Python code for it?

from numba import cuda

dev_a = cuda.to_device(a) # Copy the input data to GPU (device) memory
dev_b = cuda.device_array_like(a) # Allocate but do not initialize output in GPU memory

dev_a.alloc_size, dev_b.alloc_size # Ask how many bytes are allocated on the GPU device


I am working on the file "GPU_Introduction.ipynb".
The first task is:
Next we write a CUDA kernel. This looks a lot like the vectorized function.
CUDA will set this up with blocks of threads.
In C++/CUDA, we'd need to set up the blocks and threads and do a bit of bookkeeping.
In numba cuda we can simply use the cuda.grid(1) to automatically turn the block
and thread ids into an index i.
Can you write Python code for it?

@cuda.jit
def cu_silly_operation(a, b):
    """
    Some mathematical busywork to illustrate GPU vs CPU
    """
    i = cuda.grid(1)
    b[i] = math.sqrt(a[i] % 13)
    for n in range(4):
        b[i] += math.sin(a[i] % (n + 50))


I am working on the file "GP_GAN.ipynb".
The first task is:
Huzzah!
Here is another way of doing it (unsupervised):
Can you write Python code for it?

!python run_gp_gan.py --src_image images/test_images/src.jpg --dst_image images/test_images/dst.jpg --mask_image images/test_images/mask.png --blended_image images/test_images/result_image_unsupervised.png --supervised False

from IPython.display import Image
Image('images/test_images/result_image_unsupervised.png')


I am working on the file "GPy-phil/GPy Intro.ipynb".
The first task is:
Changing parameters is as easy as assigning new values to the respective parameter:
Can you write Python code for it?


I am working on the file "GPy-phil/GPy Intro.ipynb".
The first task is:
Scalability

GPys parameterized framework can handle as many parameters as you like and is memory and speed efficient in setting parameters by having only one copy of the parameters in memory.

There are many scalability based Gaussian process methods implemented in GPy, have a look at
Can you write Python code for it?


I am working on the file "GPy-phil/GPy Intro.ipynb".
The first task is:
The parameters can be optimized using gradient based optimization. The optimization routines are taken over from scipy. Running the optimization in a GPy model is a call to the models own optimize method.
Can you write Python code for it?

m.optimize(messages=1)

_ = m.plot()

# You can use different kernels to use on the data.
# Try out three different kernels and plot the result after optimizing the GP:
# See kernels using GPy.kern.<tab>

# Type your code here


I am working on the file "GPy-phil/GPy Intro.ipynb".
The first task is:
The whole model gets updated automatically, when updating a parameter, without you having to interfere at all.
Change some parameters and plot the results, using the models plot() function
What do the different parameters change in the result?
Can you write Python code for it?


I am working on the file "GPy-phil/GPy Intro.ipynb".
The first task is:
We have put a lot of effort in stability of execution, so try to randomize a model using its randomize() function, which randomized the models parameters. After optimization the result whould be very close to previous model optimizations.
Can you write Python code for it?


I am working on the file "GPy-phil/ease-of-use.ipynb".
The first task is:
As we can see, the GP can fit the data ok, although the noise on the data is clearly poisson distributed. You can try to run the Poisson regression on this data now (see e.g. Posson regression). We will go on using more inputs and showing the parameter handling.
Can you write Python code for it?

def log_likelihood(m, X_test, Y_test):
    import scipy as sp
    mu, var = m.predict(X_test, full_cov=True)
    return sp.stats.multivariate_normal.logpdf(Y_test.flatten(), mu.flatten(), var).sum()

def RMSE(m, X_test, Y_test):
    mu, var = m.predict(X_test, full_cov=False)
    return np.sqrt(((mu - Y_test)**2).mean())

result_df = pd.DataFrame(index=['Log-likelihood', 'Root-mean-squared-error'])
result_df['simple'] = [log_likelihood(simple_model, X_test[[selected_input]].values, Y_test.values), RMSE(simple_model, X_test[[selected_input]].values, Y_test.values)]

result_df


I am working on the file "GPy-phil/ease-of-use.ipynb".
The first task is:
For the purpose of this notebook we will use the regression task of the abalone dataset - the prediction of age of Abalone (edible snails) from physical measurements.
We can download the dataset from http://www.cs.toronto.edu/%7Edelve/data/abalone/desc.html. For convenience we include alittle script which downloads the data and saves it locally as a pickle file. The data gets saved in this directory. When running the script it loads the data into a pandas data frame called data:
Can you write Python code for it?

%run load_abalone
data.head()

# Make training and test possible, by subsetting the data randomly.
# We will use 1/8 of the data as training set:
train_idx = np.random.choice(data.shape[0], data.shape[0]/8)
test_idx = np.setdiff1d(np.arange(data.shape[0]), train_idx)[:400]
# make sex a categorical factor, for usage in regression task
_tmp = data.copy()
sex_labels, sex_uniques = pd.factorize(data['sex'], size_hint=3)
_tmp['sex'] = sex_labels
data_train, data_test = _tmp.loc[train_idx].copy(), _tmp.loc[test_idx].copy()


I am working on the file "GPy-phil/ease-of-use.ipynb".
The first task is:
In order to add coregionalization to a model we merely have to create the right kernel, which understands the coregionalization structure in the data:
Can you write Python code for it?

kern_coreg = GPy.kern.Coregionalize(1, # Dimensionality of coregionalize input
                                    len(sex_uniques), # How many different tasks do we have?
                                    rank=3, # How deep is the coregionaliztion,
                                    # the higher the rank, the more flexible the
                                    # coregionalization, rank >= 1
                                    active_dims=[X_train.columns.get_loc('sex')] ,
                                    # Which dimension of the input carries the task information?
                                    name = 'sex',
                                   ) * kern_numeric.copy()

full_model = GPy.models.SparseGPRegression(X_train.values, Y_train.values, kernel=kern_coreg, num_inducing=40)

full_model.optimize(messages=1, max_iters=3e3)

W, k = full_model.kern.sex.parameters
coregionalization_matrix = GPy.util.linalg.tdot(W)
GPy.util.diag.add(coregionalization_matrix, k)

fig, ax = plt.subplots()
c = ax.matshow(coregionalization_matrix, cmap=plt.cm.hot)
ax.set_xticks([0, 1, 2])
ax.set_xticklabels(sex_uniques[[0, 1, 2]], fontdict=dict(size=23))
ax.set_yticks([0, 1, 2])
ax.set_yticklabels(sex_uniques[[0, 1, 2]], fontdict=dict(thickness='bold'))
ax.xaxis.tick_top()
plt.colorbar(c, ax=ax)
fig.tight_layout()

full_model.kern.kernel.plot_ARD()
_ = plt.legend()

# Evaluate the log likelihood for the whole numerical model with coregionalization:
result_df['coreg'] = (
    log_likelihood(full_model, X_test.values, Y_test.values),
    RMSE(full_model, X_test.values, Y_test.values)
)

result_df


I am working on the file "GPy-phil/ease-of-use.ipynb".
The first task is:
The first thing which we can see, is that the diameter kernel was added properly into the model and we can see its name in the output.
Can you write Python code for it?

# Optimize the simple model and plot the result
simple_model.optimize(messages=1)

_ = simple_model.plot(plot_training_data=True)


I am working on the file "GPy-phil/ease-of-use.ipynb".
The first task is:
When comparing to the model with only the one selected input we have a slight improvement in RMSE. Now we have an additional input, which seperates the data into three different tasks. We can resolve different tasks in GPs using the so-called coregionalization technique.
Can you write Python code for it?


I am working on the file "GPy/Integral_kernel.ipynb".
The first task is:
We now have a set of
Can you write Python code for it?

outputmean = np.mean(outputs)
outputs -= outputmean

#we have twice as many dimensions as the actual data as each
#pair of dimensions is used to specify one dimension in the domain of each bin.
#the input matrix above is used, so dimension 0 and 1 specify the start
#and end of each bin in the 'age' direction. etc...
kernel = GPy.kern.Multidimensional_Integral_Limits(input_dim=4, variances=1.0, lengthscale=[8.0,4.0])
m = GPy.models.GPRegression(1.0*inputs,1.0*area*outputs[:,None],kernel)
m.optimize()

a = 4.0
h = 100.0
Xpred = []
test_heights = range(70,140)
for h in test_heights:
    Xpred.append([a,0,h,0])
Xpred = np.array(Xpred)
weight, err = m.predict_noiseless(Xpred)
plt.plot(test_heights,weight+outputmean)
plt.plot(test_heights,weight+outputmean+np.sqrt(err),':')
plt.plot(test_heights,weight+outputmean-np.sqrt(err),':')
plt.xlabel('Height / cm')
plt.ylabel('Weight / kg')
plt.title('Results from Children Age, Height, Weight simulation')


I am working on the file "GPy/MagnificationFactor.ipynb".
The first task is:
As you can see, the magnification factor tells us about the 'bending' of the GP embedding. Where there is data, the embedding is dense and bends towards the data. This means the magnification shows us, how the plane got warped (with the data pinned on it) to recreate the latent space. You can see the magnification as stretching and bending. The higher the magnification the more stretched the space is at those places. Still, you need to keep in mind, that the magnification is summarized for all latent dimensions and therefore not always full represents what is going on, as a stretching in one dimension could be counteracted by a contraction of another.
Bayesian GPLVM:
We can plot the magnification also for the Bayesian GPLVM. Here it is defined through the inducing inputs, shown as little white triangles in the plot. You can see, that the magnification evolves around the inducing inputs.
Can you write Python code for it?

GPy.plotting.change_plotting_library('matplotlib')
m = GPy.models.bayesian_gplvm_minibatch.BayesianGPLVMMiniBatch(Y, input_dim, num_inducing=30, missing_data=True)

m.optimize(messages=1, max_iters=5e3)

fig, axes = plt.subplots(1, 2, figsize=(15,5))
axes[0].set_title('Latent Space')
v1 = m.plot_latent(labels=labels, ax=axes[0], updates=False)
axes[1].set_title('Magnification')
v2 = m.plot_magnification(labels=labels, ax=axes[1], updates=False, resolution=120)


I am working on the file "GPy/MagnificationFactor.ipynb".
The first task is:
Following from the properties of Gaussian processes (and given that we are using a differentiable kernel) for a test point $\textbf{x}_*$ we can compute the predictive Jacobian $\textbf{J}$ as follows:
$$ \mathbf{J} \sim \prod_p \mathcal{N}( \partial\mathbf{K}{,f}\mathbf{K}{f,f}^{-1}\mathbf{Y}{:,p}, \partial^2\mathbf{K}_{,} - \partial\mathbf{K}_{,f}\mathbf{K}^{-1}{f,f}\partial\mathbf{K}_{f,*} ) $$
Given that the distribution of the Jacobian is Gaussian and considering the fact that its $p$ rows are independent,  we compute the $\textit{expected metric tensor}$:
$$ \mathbf{G} = E(\mathbf{J}^T \mathbf{J}) = E(\mathbf{J}^T)E(\mathbf{J}) + d*Cov(\mathbf{J}) $$
For the visualization of the metric tensor we use the square root of its determinant, called the $\textit{magnification factor}$.
Predictive metric (including covariance)
The computation of the expected metric tensor $\mathbf{G}$ contains two terms, one related to the mean of the Jacobian $\mathbf{J}$ and one related to its covariance. The plotting tool below allows to separately visualize  the effect of each term. If we use covariance= False, we visualize only the mean and, vice versa, if we use mean = False we plot the effect of the covariance term into the metric.
Can you write Python code for it?

Q = input_dim
m_gplvm = GPy.models.GPLVM(Y, Q, kernel=GPy.kern.RBF(Q))
m_gplvm.kern.lengthscale = .2
m_gplvm.kern.variance = 1
m_gplvm.likelihood.variance = 1.
#m2.likelihood.variance.fix(.1)
m_gplvm

m_gplvm.optimize(messages=1, max_iters=5e4)

figure = GPy.plotting.plotting_library().figure(1, 2,
                        shared_yaxes=True,
                        shared_xaxes=True,
                        subplot_titles=('Latent Space',
                                        'Magnification',
                                        )
                            )

canvas = m_gplvm.plot_latent(labels=labels, figure=figure, col=(1), legend=False)
canvas = m_gplvm.plot_magnification(labels=labels, figure=figure, col=(2), legend=False)

GPy.plotting.show(canvas, filename='wishart_metric_notebook')

figure = GPy.plotting.plotting_library().figure(1, 3,
                        shared_yaxes=True,
                        shared_xaxes=True,
                        subplot_titles=('Full Magnification',
                                        'Magnification Mean',
                                        'Magnification Variance'
                                        )
                            )

canvas = m_gplvm.plot_magnification(figure=figure,
                      covariance=True,
                      updates=False,
                      labels = labels,
                      col=1,
                     )

canvas = m_gplvm.plot_magnification(figure=figure,
                      covariance=False,
                      updates=False,
                      labels = labels,
                      col=2,
                     )

canvas = m_gplvm.plot_magnification(figure=figure,
                      mean=False,
                      covariance=True,
                      updates=False,
                      labels = labels,
                      col=3,
                     )

GPy.plotting.show(canvas, filename='wishart_metric_notebook_mean_variance')


I am working on the file "GPy/MagnificationFactor.ipynb".
The first task is:
Oil Flow Data:
Running the Oil Flow dataset
Can you write Python code for it?

m_oil = GPy.examples.dimensionality_reduction.bgplvm_oil(N=300, plot=False)

fig, axes = plt.subplots(1, 2, figsize=(15,5))
axes[0].set_title('Latent Space')
v1 = m_oil.plot_latent(labels=m_oil.data_labels, ax=axes[0], updates=False)
axes[1].set_title('Magnification')
v2 = m_oil.plot_magnification(labels=m_oil.data_labels, ax=axes[1], updates=False, resolution=120)


I am working on the file "GPy/MagnificationFactor.ipynb".
The first task is:
The magnification factor is less fine grided, as it goes through the inducing inputs, instead of through every data point. Still you can see, the structure of the embedding, being less bent, where there is data and bending towards data. Around the data clusters the magnification (and therefore the embedding) is more flat. Importantly, in the areas far from the data clusters the mean of the embedding is flat, but still the values of the magnification factors show some distortion; this means that out of the data range there is some uncertainty over the structure of the embedding, thanks to the role played by the covariance term.
Stick demo:
Running the stick demo code and plotting the magnification factor for it:
Can you write Python code for it?

m_stick = GPy.examples.dimensionality_reduction.stick_bgplvm(plot=False)

m_stick.kern.plot_ARD()

fig, axes = plt.subplots(1, 2, figsize=(15,5))
wi = [0,3]
axes[0].set_title('Latent Space')
v1 = m_stick.plot_latent(labels=None, ax=axes[0], updates=False, which_indices=wi)
axes[1].set_title('Magnification')
v2 = m_stick.plot_magnification(labels=None, ax=axes[1], updates=False, resolution=120, which_indices=wi)


I am working on the file "GPy/ParametricNonParametricInference.ipynb".
The first task is:
We will assume the prior over $w_i\sim\mathcal{N}(0, 3)$ and a Matern32 structure in the non-parametric part. Additionally, we add a half parametric part, which is a periodic effect only active between $x\in[3,8]$:
Can you write Python code for it?

k = GPy.kern.Matern32(1, .3)
Kf = k.K(x)

k_per = GPy.kern.PeriodicMatern32(1, variance=100, period=1)
k_per.period.fix()
k_dom = DomainKernel(1, 1., 5.)
k_perdom = k_per * k_dom
Kpd = k_perdom.K(x)

np.random.seed(1234)
alpha = np.random.gamma(3, 1, Phi.shape[1])
w = np.random.normal(0, alpha)[:,None]

f_SE = np.random.multivariate_normal(np.zeros(x.shape[0]), Kf)[:, None]
f_perdom = np.random.multivariate_normal(np.zeros(x.shape[0]), Kpd)[:, None]
f_w = Phi.dot(w)

f = f_SE + f_w + f_perdom

y = f + np.random.normal(0, .1, f.shape)

plt.plot(x, f_w)
_ = plt.plot(x, y)
# Make sure the function is driven by the linear trend, as there can be a difficulty in identifiability.


I am working on the file "GPy/Poisson regression tutorial.ipynb".
The first task is:
As we can see, the model as it stands is not all that great, the lengthscale parameter is soo small. We noted before that the kernels hyperparameters we chose were not the correct ones, only an initial guess. Next we try maximizing the likelihood of the model with respect to the parameters. Optimization within GPy is straightforward.
Can you write Python code for it?

m.optimize()
print m

#Predictions of counts of buses, including noise arising from Poisson noise
m.plot()
pb.xlabel('Time')
pb.ylabel('Buses counts per 30 minutes')