I am working on the file "Documentation/Yahtzee Analysis.ipynb".
The first task is:
Now we will plot the axis labels.
Can you write Python code for it?

# Label the graph
bins = len(count)
plt.ylabel('Count', fontsize=12)
plt.xlabel('Score', fontsize=12)

# Plot the axis ticks
plt.yticks(range(0,8501,500), fontsize=8)
plt.xticks(range(bins),fontsize = 8, ha='center')

# set the x axis so it fits the data better
ax.set(xlim=[-1,bins]);


I am working on the file "Documentation/Yahtzee Analysis.ipynb".
The first task is:
Now we will separate the filtered scores into their own bins so that we can graph a histogram.
Can you write Python code for it?


I am working on the file "Documentation/Yahtzee Analysis.ipynb".
The first task is:
Therefore percentage Yahztee was scored in this data set is:
Can you write Python code for it?

percentage_Yahtzee = yahtzee_count/total_turns * 100

percentage_Yahtzee


I am working on the file "Documentation/Yahtzee Analysis.ipynb".
The first task is:
We will use this function throught our analysis.
It can be imported with the following code (Note: this code must be in the same location as the code where you invoke it)
from _nameoffile_ import fileIntegerParser
Visualize the Data
Next we will graph the data. I am curious to see how many times a given score has occured. We can easily plot this in Python.
First, we Import the required packages
Can you write Python code for it?

import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import FormatStrFormatter
import numpy as np
import pandas as pd
import re


I am working on the file "Documents/IPython notebooks/uncertainty.ipynb".
The first task is:
Computing uncertainty for a derived quantity
From Coleman and Steele (2009) equation 3.15, the uncertainty of the power coefficient is
$$ u_{C_P}^2 = s_{C_P}^2 + b_{C_P}^2, $$
where
$$ b_{C_P}^2 = \sum_{i=1}^J \left( \frac{\partial C_P}{\partial X_i} \right)^2 b_{X_i}^2. $$
For our case, we consider $X = [T, \omega, U_\infty ] $. The partial derivatives are then
$$ \frac{\partial C_P}{\partial T} = \frac{\omega}{\frac{1}{2} \rho A U_\infty^3}, $$
$$ \frac{\partial C_P}{\partial \omega} = \frac{T}{\frac{1}{2} \rho A U_\infty^3}, $$
$$ \frac{\partial C_P}{\partial U_\infty} = \frac{-3 T \omega}{\frac{1}{2} \rho A U_\infty^4}. $$
Power coefficient
Can you write Python code for it?

rho = 1000.0
area = 1.0
omega = run.omega.mean()
torque = run.torque.mean()
u_infty = run.tow_speed.mean()
const = 0.5*rho*area

b_cp = np.sqrt((omega/(const*u_infty**3))**2*b_torque**2 + \
               (torque/(const*u_infty**3))**2*b_angle**2 + \
               (-3*torque*omega/(const*u_infty**4))**2*b_car_pos**2)

unc_cp = calc_uncertainty(run.cp_per_rev, b_cp)

print("Uncertainty of C_P = {:.3f}".format(unc_cp))


I am working on the file "Documents/IPython notebooks/uncertainty.ipynb".
The first task is:
Drag coefficient
Drag coefficient is calculated using
$$ C_D = \frac{F_L + F_R}{\frac{1}{2} \rho A U_\infty^2} . $$
The measured variables used to calculate $C_D$ are $[F_L, F_R, U_\infty]$, therefore the necessary partial derivatives are
$$ \frac{\partial C_D}{\partial F_L} = \frac{1}{\frac{1}{2} \rho A U_\infty^2} ,$$
$$ \frac{\partial C_D}{\partial F_R} = \frac{1}{\frac{1}{2} \rho A U_\infty^2} ,$$
$$ \frac{\partial C_D}{\partial U_\infty} = \frac{-2(F_L + F_R)}{\frac{1}{2} \rho A U_\infty^3} .$$
Can you write Python code for it?

drag = run.drag.mean()
u_infty = run.tow_speed.mean()
const = 0.5*rho*area

b_cd = np.sqrt((1/(const*u_infty**2))**2*b_force**2 + \
               (1/(const*u_infty**2))**2*b_force**2 +
               (-2*drag/(const*u_infty**3))**2*b_car_pos**2)

unc_cd = calc_uncertainty(run.cd_per_rev, b_cd)

print("Uncertainty of C_D = {:.3f}".format(unc_cd))


I am working on the file "Documents/IPython notebooks/uncertainty.ipynb".
The first task is:
Tip speed ratio
The necessary partial derivatives are
$$ \frac{\partial \lambda}{\partial \omega} = \frac{R}{U_\infty} , $$
$$ \frac{\partial \lambda}{\partial U_\infty} = \frac{-\omega R}{U_\infty^2} . $$
Can you write Python code for it?

omega = run.omega.mean()
u_infty = run.tow_speed.mean()
R = 0.5

b_tsr = np.sqrt((R/(u_infty))**2*b_angle**2 + \
                (-omega*R/(u_infty**2))**2*b_car_pos**2)

unc_tsr = calc_uncertainty(run.tsr_per_rev, b_tsr)

print("Uncertainty of TSR = {:.3f}".format(unc_tsr))


I am working on the file "Documents/IPython notebooks/uncertainty.ipynb".
The first task is:
Torque
Can you write Python code for it?

run = Run("Wake-1.0", 20)
torque = run.torque
unc_torque = calc_uncertainty(torque, b_torque)
print("Torque uncertainty (using all samples) = {:0.1e} Nm".format(unc_torque))
torque = run.torque_per_rev
unc_torque = calc_uncertainty(torque, b_torque)
print("Torque uncertainty (averaging per rev) = {:0.1e} Nm".format(unc_torque))


I am working on the file "Dorina_preprocessing.ipynb".
The first task is:
Add eClip data on 16 April 2018
Add eClip data from Van Nostrand, Eric L., et al. "Robust transcriptome-wide discovery of RNA-binding protein binding sites with enhanced CLIP (eCLIP)." Nature methods 13.6 (2016): 508.
Can you write Python code for it?

for cell in ('K562', 'HepG2'):
    path = Path(f'/Volumes/biodb/encode/encode_hg38_clip_peaks/{cell}/replicates/')
    bed_files = path.glob('*.bed')
    regulators = [x.stem[: x.stem.find('_')] for x in bed_files]
    for regulator in regulators:
        bt1 = pybedtools.BedTool(str(path / f'{regulator}_{cell}_rep01.bed'))
        bt2 = pybedtools.BedTool(str(path / f'{regulator}_{cell}_rep02.bed'))
        bt_inter = bt1.intersect(bt2, s=True).sort()
        bt_merged = bt1.cat(bt2, postmerge=False).sort()
        bt_inter = bt_inter.map(b=bt_merged, c='5', s=True, o='median').bed6()
        new_name = f'{cell}-{regulator}|eClip'
        bt_inter.each(change_name, new_name=new_name).saveas().moveto(
            f'/Volumes/tbrittoborges/dorina_eclip/eClip_{cell}_{regulator}_vanNostrand2016_hg38.bed')

!cat /Volumes/tbrittoborges/dorina_eclip/* > /Volumes/prj/dorina2/regulators/h_sapiens/hg38/eClip_RBP_hg38.bed

pd.read_json('/Volumes/prj/dorina2/regulators/h_sapiens/hg38/TargetScanCons_mirna_hg19.json').head()

eclip_json = []
for cell in ('K562', 'HepG2'):
    path = Path(f'/Volumes/biodb/encode/encode_hg38_clip_peaks/{cell}/replicates/')
    bed_files = path.glob('*.bed')
    regulators = [x.stem[: x.stem.find('_')] for x in bed_files]
    for regulator in regulators:
        n_sites = len(pybedtools.BedTool(
            f'/Volumes/tbrittoborges/dorina_eclip/eClip_{cell}_{regulator}_vanNostrand2016_hg38.bed'))
        eclip_json.append(
            {
            "description": f'sites regulated by {n_sites} obtained from {cell} cells. The '
            'duplicated were merged with intersect and the median score is presented.'
            'The original data was mapped to hg38 assembly',
            "experiment": 'eClip',
            "id": f'{cell}-{regulator}|eClip',
            "references": 'Van Nostrand, Eric L., et al. Robust transcriptome-wide discovery '
            'of RNA-binding protein binding sites with enhanced CLIP (eCLIP). Nature methods '
            '13.6 (2016): 508.',
           ...


I am working on the file "Dorina_preprocessing.ipynb".
The first task is:
Add miRNA expression defined in Fehlmann2017 et al
Downloaded supplementar file btx814_supp and used Supplemental_Data_2.gff3
Can you write Python code for it?

!head /Volumes/prj/dorina2/raw/Fehlmann2017etal.gff3

bt = pybedtools.BedTool('/Volumes/prj/dorina2/raw/Fehlmann2017etal.gff3')

bt_new = bt.each(change_name_fehlmannetal2017).saveas()

'/Volumes/prj/dorina2/regulators/h_sapiens/hg38/targetScan_miRNA_hg38.bed'
for entry in bt:
    entry[2] = entry.name

bt.head()


I am working on the file "Dorina_preprocessing.ipynb".
The first task is:
Expression from Gtex
Can you write Python code for it?

%%script false
# this one was processed in the computer cluster
!cd /biodb/gtex/
import pandas as pd

rnaseq = 'GTEx_Analysis_2016-01-15_v7_RNASeQCv1.1.8_gene_tpm.gct.gz'
sample = 'GTEx_v7_Annotations_SampleAttributesDS.txt'
samples = pd.read_table(sample)
gtex = pd.read_table(rnaseq, skiprows=2, compression='gzip', header=0, )
samples_mapping = dict(zip(samples['SAMPID'], samples['SMTS']))
gtex.set_index('Name', inplace=True)
gtex.drop(columns=['Description'], inplace=True)
gtex = gtex.rename(columns=samples_mapping)
gtex = gtex.T
names = [x.split('.')[0] for x in gtex.columns]
#  alternative solution for id mapping

import numpy as np
from bioservices import BioDBNet
bioservice = BioDBNet()
input_db = "Ensembl Gene ID"
output_db = ["Gene Symbol"]
input_values = ['ENSG00000214688']
xref = [bioservice.db2db(input_db, output_db, list(x), 9606) for x in np.array_split(names, 562)]
xref = pd.concat(xref)

gtex = pd.read_csv('/Volumes/tbrittoborges-1/gtex_posprocessed2.csv',
                   index_col=0, header=0)

gtex = gtex.groupby(level=0).median()

gtex.head()

gtex.loc['Heart'][gtex.loc['Heart'] > 1].shape

id_map = pd.read_csv('/Volumes/tbrittoborges-1/gtexv7_id_mapping.csv', header=0, index_col=0,sep=';')

mapping = id_map.set_index(['ensembl_id'])['hgnc_symbol']

mapped_names = mapping[names]

mapped_names.head()

gtex.columns = [x[:x.index('.')] for x in gtex.columns]
gtex.head()

# mapped_names.fillna(value={k:k for k in mapped_names.index}, inplace=True)

gtex.rename(columns=mapped_names, inplace=True)

gtex[gtex.columns.dropna()].head()

gtex_clean =  gtex[gtex.columns.dropna()]

# gtex.loc['Heart'][gtex.loc['Heart']  > 1].index.tolist()

gtex_clean[gtex_clean > 1]


I am working on the file "Dorina_preprocessing.ipynb".
The first task is:
Fix for mm10 regulators on 13 April 2018
dorina and webdorina were not working with mm10 regulatos due to mismatch of the regulators record name and .bed file name.
This section fixes this issue
Can you write Python code for it?

bed_files = Path('/Volumes/prj/dorina2/regulators/m_musculus/mm10/').glob('*.bed')

for bed_file in bed_files:
    bt = pybedtools.BedTool(str(bed_file))
    new_name = bed_file.stem + '*'
    bt.each(change_name, new_name=new_name).saveas().moveto(str(bed_file))


I am working on the file "Dotstar-LED.ipynb".
The first task is:
Create Dotstar object
You can pass several arguments to the Dotstar class constructor to change the behavior of the LED class.

ds = dotstar.Dotstar(led_count=72, bus=0, init_data=0, init_brightness=0)
Parameters:
led_count = some_number_of_leds
Change the number of LEDs in your strip. Note that this counts the raw number of individual LEDs, not how many strips/devices you have. Make sure this is set so all the LEDs are used.
bus = 0
Change the SPI bus. If you do not specify one, it will be initialized on bus 0, which is the default for the Minnowboard.
init_data = some_brightness_value + some_hue
Change the initial value of the LED strip. By default all the LEDS are initialized to the first color pushed. If you plan on having all the LEDs start off dark, don't set anything here.
init_brightness = some_brightness
Change the initial brightness of the LEDs. Valid brightness settings range from 0 to 10, representing the intensity of the LEDs from 0% to 100%. If you want the LEDs to start off dark, set this to 0 at the start.

Here is a typical initialization, starting all 72 LEDS (or 2 Adafruit Dotstar LED strips connected together) turned off:
Can you write Python code for it?


I am working on the file "Double Slit Correlation Model.ipynb".
The first task is:
Define a single function to explore the FFT:
Can you write Python code for it?

def plotFFT(d,a,b):
    """Single function version of generating the FFT output"""
    TotalField = E_dg(y,a,b)+E_lo(y,d)
    TotalIntensity=TotalField*TotalField.conj()
    plt.plot(abs(fft(TotalIntensity)),".-")
    plt.ylim([0,1e-2])
    plt.xlim([0,650])
    plt.title("FFT output")

plotFFT(d=0.046,a=0.5e-3,b=0.08e-3)


I am working on the file "Double Slit Correlation Model.ipynb".
The first task is:
Replace with Gaussian LO: import gaussian beam function, and repeat:
Can you write Python code for it?

# bopt.gaussian_beam(x, y, z, E0, wavelambda, w0, k)
# set to evaluate gaussian at L (full distance to CCD) with waist width of 2 cm
# using d=0.046 for agreement with experiment
d=0.046
E_lo_gauss = bopt.gaussian_beam(0,y,L,E0=1,wavelambda=795e-9,w0=0.02,k=[0,k*d/L,k])

frames = 59
rounds = 20

drift_type= 3
# SG I made a few drift modes to model the phase drift that would be present in the lab
# drift mode two appears to be the most similar to the phase shifts we observe in the lab

time=linspace(0,2*pi,rounds*frames)

phase=[]

if drift_type == 0:
    phase= [sin(t) for t in time]
#mode 0 is just a sine wave in time

elif drift_type == 1:
    phase= [sin(t+random.randn()/2) for t in time]
    #phase= [sin(t)+random.randn()/2 for t in time]
#mode 1 is a sine wave with some randomness added to each data point

elif drift_type == 2:
    phase=[0]
    for i in range(len(time)-1):
        phase.append(phase[-1]+random.randn()/4*sin(time[i]))
#mode 2 is a sine wave with some randomness added to each data point, and also considering
#the location of the previous data point

elif drift_type == 3:
    phase=[0]
    for i in range(len(time)-1):
        phase.append(phase[-1]+0.1*(random.randn()))
#mode 2 is a sine wave with some randomness added to each data point, and also considering
#the location of the previous data point

raw_intensity_data = zeros([1300,frames,rounds],dtype=complex128)
scaled = zeros([1300,frames,rounds],dtype=complex128)
i=0
for r in range(rounds):
    for f in range(frames):
        TotalField = E_dg(y,a,b)*exp(-1j*phase[i]) + E_lo_gauss #adds the appropriate phase
        #TotalField = E_dg(y,a,b) + E_lo_gauss
        TotalIntensity = TotalField * TotalField.conj()
        raw_intensity_data[:,f,r] = TotalIntensity
        scaled[:,f,r]=fft(TotalIntensity)
        i=i+1 #increases index

#checking how phase moves around
plt.polar(phase,time,'-')
plt.title("phase shift with (simulated)...


I am working on the file "Double Slit Gaussian LO.ipynb".
The first task is:
Import gaussian beam function:
Can you write Python code for it?

import BeamOptics as bopt

d=.045

pixwidth = 1e-5  # CCD pixel width

y = linspace(-pixwidth*ccdsize/2,pixwidth*ccdsize/2,ccdsize)

#E_lo_gauss = bopt.gaussian_beam(0,y,10*L,E0=0.005,wavelambda=795e-9,w0=0.0030,k=[0,k*d/L,k])
E_lo_gauss = bopt.gaussian_beam(0,y,L,E0=0.005,wavelambda=795e-9,w0=0.0060,k=[0,k*d/L,k])

#TODO: change to normal LO and angled signal

plt.plot(y,unwrap(angle(E_lo_gauss)))
#plt.ylim([0,0.005])


I am working on the file "Double Slit Gaussian LO.ipynb".
The first task is:
Interesting, the phase fronts are pretty flat under these conditions. Need to revisit the Zernike model we started?
Can you write Python code for it?

plt.plot(y,abs(E_lo_gauss))
plt.ylim([0,0.005])

TotalIntensity=(E(y)+E_lo_gauss) * (E(y)+E_lo_gauss).conj()

plt.figure(figsize=(14,4))
plt.plot(y,TotalIntensity,".-")

#plt.xlim([-.002,0])

plt.plot(abs(fft(TotalIntensity)),".-")

plt.ylim([0,.0005]) # Had to lower the LO power quite a bit, and then zoom way in.

plt.xlim([300,500])


I am working on the file "Double Slit Model.ipynb".
The first task is:
Define a single function to explore the FFT:
Can you write Python code for it?


I am working on the file "Double Slit Model.ipynb".
The first task is:
This agrees well with Matt's code using symbolic calculations. The main difference I see is in the size of the low-frequency peak. It's much smaller here than in his version.
d=0.035
Can you write Python code for it?


I am working on the file "Double Slit Model.ipynb".
The first task is:
This does not agree with experimental results.
Replace with Gaussian LO: import gaussian beam function, and repeat:
Can you write Python code for it?

bopt.gaussian_beam?

# bopt.gaussian_beam(x, y, z, E0, wavelambda, w0, k)
# set to evaluate gaussian at L (full distance to CCD) with waist width of 2 cm
# using d=0.046 for agreement with experiment
d=0.046
E_lo_gauss = bopt.gaussian_beam(0,y,L,E0=1,wavelambda=795e-9,w0=0.02,k=[0,k*d/L,k])

plt.plot(y,abs(E_lo_gauss))

TotalIntensity=(E_dg(y,a,b)+E_lo_gauss) * (E_dg(y,a,b)+E_lo_gauss).conj()

plt.figure(figsize=(14,4))
plt.plot(y,TotalIntensity,".-")

#plt.xlim([-.002,0])

plt.plot(abs(fft(TotalIntensity)),".-")

plt.ylim([0,0.01]) # Had to lower the LO power quite a bit, and then zoom way in.

plt.xlim([0,650])


I am working on the file "Double-DQN/tensorflow-deepq/notebooks/.ipynb_checkpoints/karpathy_game-checkpoint.ipynb".
The first task is:
Average Reward over time
Can you write Python code for it?


I am working on the file "Double-DQN/tensorflow-deepq/notebooks/karpathy_game.ipynb".
The first task is:
Average Reward over time
Can you write Python code for it?


I am working on the file "Double-DQN/tensorflow-deepq/notebooks/karpathy_game.ipynb".
The first task is:
Visualizing what the agent is seeing
Starting with the ray pointing all the way right, we have one row per ray in clockwise order.
The numbers for each ray are the following:
- first three numbers are normalized distances to the closest visible (intersecting with the ray) object. If no object is visible then all of them are $1$. If there's many objects in sight, then only the closest one is visible. The numbers represent distance to friend, enemy and wall in order.
- the last two numbers represent the speed of moving object (x and y components). Speed of wall is ... zero.
Finally the last two numbers in the representation correspond to speed of the hero.
Can you write Python code for it?

g.__class__ = KarpathyGame
np.set_printoptions(formatter={'float': (lambda x: '%.2f' % (x,))})
x = g.observe()
new_shape = (x[:-2].shape[0]//g.eye_observation_size, g.eye_observation_size)
print(x[:-2].reshape(new_shape))
print(x[-2:])
g.to_html()


I am working on the file "Double-Dueling-DQN.ipynb".
The first task is:
Above is an example of a starting environment in our simple game. The agent controls the blue square, and can move up, down, left, or right. The goal is to move to the green square (for +1 reward) and avoid the red square (for -1 reward). The position of the three blocks is randomized every episode.
Implementing the network itself
Can you write Python code for it?

class Qnetwork():
    def __init__(self,h_size):
        #The network recieves a frame from the game, flattened into an array.
        #It then resizes it and processes it through four convolutional layers.
        self.scalarInput =  tf.placeholder(shape=[None,21168],dtype=tf.float32)
        self.imageIn = tf.reshape(self.scalarInput,shape=[-1,84,84,3])
        self.conv1 = slim.conv2d( \
            inputs=self.imageIn,num_outputs=32,kernel_size=[8,8],stride=[4,4],padding='VALID', biases_initializer=None)
        self.conv2 = slim.conv2d( \
            inputs=self.conv1,num_outputs=64,kernel_size=[4,4],stride=[2,2],padding='VALID', biases_initializer=None)
        self.conv3 = slim.conv2d( \
            inputs=self.conv2,num_outputs=64,kernel_size=[3,3],stride=[1,1],padding='VALID', biases_initializer=None)
        self.conv4 = slim.conv2d( \
            inputs=self.conv3,num_outputs=h_size,kernel_size=[7,7],stride=[1,1],padding='VALID', biases_initializer=None)

        #We take the output from the final convolutional layer and split it into separate advantage and value streams.
        self.streamAC,self.streamVC = tf.split(self.conv4,2,3)
        self.streamA = slim.flatten(self.streamAC)
        self.streamV = slim.flatten(self.streamVC)
        xavier_init = tf.contrib.layers.xavier_initializer()
        self.AW = tf.Variable(xavier_init([h_size//2,env.actions]))
        self.VW = tf.Variable(xavier_init([h_size//2,1]))
        self.Advantage = tf.matmul(self.streamA,self.AW)
        self.Value = tf.matmul(self.streamV,self.VW)

        #Then combine them together to get our final Q-values.
        self.Qout = self.Value + tf.subtract(self.Advantage,tf.reduce_mean(self.Advantage,axis=1,keep_dims=True))
        self.predict = tf.argmax(self.Qout,1)

        #Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.
        self.targetQ = tf.placeholder(shape=[None],dtype=tf.float32)
        self.actions =...


I am working on the file "Double-Dueling-DQN.ipynb".
The first task is:
Load the game environment
Feel free to adjust the size of the gridworld. Making it smaller provides an easier task for our DQN agent, while making the world larger increases the challenge.
Can you write Python code for it?

from gridworld import gameEnv

env = gameEnv(partial=False,size=5)


I am working on the file "Double-Dueling-DQN.ipynb".
The first task is:
These functions allow us to update the parameters of our target network with those of the primary network.
Can you write Python code for it?

def updateTargetGraph(tfVars,tau):
    total_vars = len(tfVars)
    op_holder = []
    for idx,var in enumerate(tfVars[0:total_vars//2]):
        op_holder.append(tfVars[idx+total_vars//2].assign((var.value()*tau) + ((1-tau)*tfVars[idx+total_vars//2].value())))
    return op_holder

def updateTarget(op_holder,sess):
    for op in op_holder:
        sess.run(op)


I am working on the file "Double-Dueling-DQN.ipynb".
The first task is:
This is a simple function to resize our game frames.
Can you write Python code for it?

def processState(states):
    return np.reshape(states,[21168])


I am working on the file "Double_Maxwell_Lognormal_prior.ipynb".
The first task is:
First, we need to set up our test data. We'll use two relaxation modes that are themselves log-normally distributed.
Can you write Python code for it?

def H(tau):
    h1 = 1; tau1 = 0.03; sd1 = 0.5;
    h2 = 7; tau2 = 10; sd2 = 0.5;
    term1 = h1/np.sqrt(2*sd1**2*np.pi) * np.exp(-(np.log10(tau/tau1)**2)/(2*sd1**2))
    term2 = h2/np.sqrt(2*sd2**2*np.pi) * np.exp(-(np.log10(tau/tau2)**2)/(2*sd2**2))
    return term1 + term2

Nfreq = 50
Nmodes = 30
w = np.logspace(-4,4,Nfreq).reshape((1,Nfreq))
tau = np.logspace(-np.log10(w.max()),-np.log10(w.min()),Nmodes).reshape((Nmodes,1))

# get equivalent discrete spectrum
delta_log_tau = np.log10(tau[1]/tau[0])
g_true = (H(tau) * delta_log_tau).reshape((1,Nmodes))

plt.loglog(tau,H(tau), label='Continuous spectrum')
plt.plot(tau.ravel(),g_true.ravel(), 'or', label='Equivalent discrete spectrum')
plt.legend(loc=4)
plt.xlabel(r'$\tau$')
plt.ylabel(r'$H(\tau)$ or $g$')
plt.savefig('Original_relax_spec.png',dpi=500)


I am working on the file "Double_Maxwell_Lognormal_prior.ipynb".
The first task is:
Now we can sample the models to get our parameter distributions:
Can you write Python code for it?

Nsamples = 2000
trueMapEstimate = pm.find_MAP(model=trueModel)
with trueModel:
    trueTrace = pm.sample(Nsamples, start=trueMapEstimate)
pm.backends.text.dump('./Double_Maxwell_v3_true', trueTrace)

noisyMapEstimate = pm.find_MAP(model=noisyModel)
with noisyModel:
    noisyTrace = pm.sample(Nsamples, start=noisyMapEstimate)
pm.backends.text.dump('./Double_Maxwell_v3_noisy', noisyTrace)

burn = 500
trueQ = pm.quantiles(trueTrace[burn:])
noisyQ = pm.quantiles(noisyTrace[burn:])


I am working on the file "Double_Maxwell_Lognormal_prior.ipynb".
The first task is:
Now, let's calculate the moduli. We'll have both a true version and a noisy version with some random noise added to simulate experimental variance.
Can you write Python code for it?

wt = tau*w
Kp = wt**2/(1+wt**2)
Kpp = wt/(1+wt**2)
noise_level = 0.02
Gp_true = np.dot(g_true,Kp)
Gp_noise = Gp_true + Gp_true*noise_level*np.random.randn(Nfreq)
Gpp_true = np.dot(g_true,Kpp)
Gpp_noise = Gpp_true + Gpp_true*noise_level*np.random.randn(Nfreq)
plt.loglog(w.ravel(),Gp_true.ravel(),label="True G'")
plt.plot(w.ravel(),Gpp_true.ravel(), label='True G"')
plt.plot(w.ravel(),Gp_noise.ravel(),'xr',label="Noisy G'")
plt.plot(w.ravel(),Gpp_noise.ravel(),'+r',label='Noisy G"')
plt.xlabel(r'$\omega$')
plt.ylabel("Moduli")
plt.legend(loc=4)
plt.savefig('Original_Moduli_spec.png',dpi=500)


I am working on the file "Double_Maxwell_Lognormal_prior.ipynb".
The first task is:
Plotting the quantiles gives us a sense of the uncertainty in our estimation of $g_i$:
Can you write Python code for it?

def plot_quantiles(Q,ax):
    ax.fill_between(tau.ravel(), y1=Q['g'][2.5], y2=Q['g'][97.5], color='c',
               alpha=0.25)
    ax.fill_between(tau.ravel(), y1=Q['g'][25], y2=Q['g'][75], color='c',
               alpha=0.5)
    ax.plot(tau.ravel(), Q['g'][50], 'b-')
    # sampling localization lines:
    ax.axvline(x=np.exp(np.pi/2)/w.max(), color='k', linestyle='--')
    ax.axvline(x=(np.exp(np.pi/2)*w.min())**-1, color='k', linestyle='--')

fig,ax = plt.subplots(nrows=2, sharex=True,
                      subplot_kw={'xscale':'log','yscale':'log',
                                  'ylabel':'$g_i$'})
plot_quantiles(trueQ,ax[0])
plot_quantiles(noisyQ,ax[1])
# true spectrum
trueSpectrumline0 = ax[0].plot(tau.ravel(), g_true.ravel(),'xr',
                           label='True Spectrum')
trueSpectrumline1 = ax[1].plot(tau.ravel(), g_true.ravel(),'xr',
                           label='True Spectrum')

ax[0].legend(loc=4)
ax[0].set_title('Using True Moduli')

ax[1].set_xlabel(r'$\tau$')
ax[1].legend(loc=4)
ax[1].set_title('Using Noisy Moduli')

fig.set_size_inches(5,8)
fig.savefig('True,Noisy_moduli.png',dpi=500)

noisySample = pm.sample_ppc(noisyTrace[burn:],model=noisyModel,samples=250)

fig,ax = plt.subplots()
for logg1,logg2 in zip(noisySample['logGp'].reshape(250,50),
                 noisySample['logGpp'].reshape(250,50)):
    ax.plot(w.ravel(),np.exp(logg1),'b-',alpha=0.05)
    ax.plot(w.ravel(),np.exp(logg2),'r-',alpha=0.01)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel(r'$\omega$')
ax.set_ylabel('G\',G"')
ax.plot(w.ravel(), Gp_true.ravel(),'xk', label='True G\'' )
ax.plot(w.ravel(), Gpp_true.ravel(), '+k', label='True G"')
ax.set_title('Moduli estimated from noisy sample')
plt.legend(loc=4)
plt.savefig('Re-estimated_moduli.png',dpi=500)


I am working on the file "Double_Maxwell_Uniform_prior.ipynb".
The first task is:
Load trace:
Can you write Python code for it?

noisyTrace = pm.backends.text.load('./Double_Maxwell_noisy',model=noisyModel)
trueTrace = pm.backends.text.load('./Double_Maxwell_true', model=trueModel)

burn = 500
trueQ = pm.quantiles(trueTrace[burn:])
noisyQ = pm.quantiles(noisyTrace[burn:])

def plot_quantiles(Q,ax):
    ax.fill_between(tau.ravel(), y1=Q['g'][2.5], y2=Q['g'][97.5], color='c',
               alpha=0.25)
    ax.fill_between(tau.ravel(), y1=Q['g'][25], y2=Q['g'][75], color='c',
               alpha=0.5)
    ax.plot(tau.ravel(), Q['g'][50], 'b-')
    # sampling localization lines:
    ax.axvline(x=np.exp(np.pi/2)/w.max(), color='k', linestyle='--')
    ax.axvline(x=(np.exp(np.pi/2)*w.min())**-1, color='k', linestyle='--')

fig,ax = plt.subplots(nrows=2, sharex=True,
                      subplot_kw={'xscale':'log','yscale':'log',
                                  'ylabel':'$g_i$'})
plot_quantiles(trueQ,ax[0])
plot_quantiles(noisyQ,ax[1])
# true spectrum
trueSpectrumline0 = ax[0].plot(tau.ravel(), g_true.ravel(),'xr',
                           label='True Spectrum')
trueSpectrumline1 = ax[1].plot(tau.ravel(), g_true.ravel(),'xr',
                           label='True Spectrum')

ax[0].legend(loc=4)
ax[0].set_title('Using True Moduli')

ax[1].set_xlabel(r'$\tau$')
ax[1].legend(loc=4)
ax[1].set_title('Using Noisy Moduli')

fig.set_size_inches(5,8)
fig.savefig('True,Noisy_moduli_uniform_prior.png',dpi=500)


I am working on the file "Double_Maxwell_Uniform_prior.ipynb".
The first task is:
Now, let's construct the moduli. We'll have both a true version and a noisy version with some random noise added to simulate experimental variance.
Can you write Python code for it?

wt = tau*w
Kp = wt**2/(1+wt**2)
Kpp = wt/(1+wt**2)
noise_level = 0.02
Gp_true = np.dot(g_true,Kp)
Gp_noise = Gp_true + Gp_true*noise_level*np.random.randn(Nfreq)
Gpp_true = np.dot(g_true,Kpp)
Gpp_noise = Gpp_true + Gpp_true*noise_level*np.random.randn(Nfreq)
plt.loglog(w.ravel(),Gp_true.ravel(),label="True G'")
plt.plot(w.ravel(),Gpp_true.ravel(), label='True G"')
plt.plot(w.ravel(),Gp_noise.ravel(),'xr',label="Noisy G'")
plt.plot(w.ravel(),Gpp_noise.ravel(),'+r',label='Noisy G"')
plt.xlabel(r'$\omega$')
plt.ylabel("Moduli")
plt.legend(loc=4)


I am working on the file "Download recent crimes from the data portal.ipynb".
The first task is:
Now let's request the data from the API, using our query
Can you write Python code for it?

import json
import requests

def get_rows_url(start, count):
    url_tpl = "https://data.cityofchicago.org/api/views/INLINE/rows.json?method=getRows&start={start}&length={length}"
    return url_tpl.format(
      start=start,
      length=count
    )

def get_rows(query, start=0, count=1000):
    url = get_rows_url(start, count)
    headers = { 'content-type' : 'application/json' }
    response = requests.post(url, data=json.dumps(query), headers=headers, verify=False)
    return response.json()

def transform_row(row, transforms):
    transformed_row = row
    for transform in transforms:
        transformed_row = transform(transformed_row)

    return transformed_row

def get_all_rows(query, transforms=[]):
    continue_fetching = True
    page_size = 1000
    start = 0

    while continue_fetching:
        rows = get_rows(query, start, page_size)
        if len(rows) < page_size:
            continue_fetching = False

        start += page_size

        for row in rows:
            yield(transform_row(row, transforms))

crimes = list(get_all_rows(query, transforms=[lambda r: humanize_columns(r, human_columns)]))

import pprint

print("There are {} crimes since {}".format(len(crimes), date_30_days_ago.strftime("%Y-%m-%d")))

print("The first one looks like: ")
pprint.pprint(crimes[0])


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
1. Download molecule from PDBbind
Can you write Python code for it?

%%bash
wget -q -N www.pdbbind.org.cn/v2007/10gs/10gs_ligand.mol2


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
2. Variables controling images
There are two main variables for controling the drawing, namely image_size and image_backend located in a oddt.toolkit namespace. There are two backends available PNG (default) and SVG.
Can you write Python code for it?

print('Default size: ', oddt.toolkit.image_size)
print('Default backend: ', oddt.toolkit.image_backend)


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
3. OpenBabel Drawing
OpenBabel is the default toolkit, although just for assurance will asign is as the toolkit of choice.
Can you write Python code for it?

oddt.toolkit = oddt.toolkits.ob
mol = next(oddt.toolkit.readfile('mol2', '10gs_ligand.mol2'))
mol.removeh()


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
3.1 OB PNG (default)
Can you write Python code for it?


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
3.2 OB SVG
Can you write Python code for it?


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
4. RDKit molecules are drawn in the same way
Can you write Python code for it?

oddt.toolkit = oddt.toolkits.rdk
mol = next(oddt.toolkit.readfile('mol2', '10gs_ligand.mol2'))
mol.removeh()


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
4.1 RDKit PNG
Can you write Python code for it?


I am working on the file "Drawing_Molecules.ipynb".
The first task is:
5. Changing a size of a molecule
A size can be arbitrally set to an int or two element tuple/list. It is a global setting appplied in all subsequent images.
Can you write Python code for it?

oddt.toolkit.image_backend = 'png'
oddt.toolkit.image_size = 100

mol

oddt.toolkit.image_size = (600, 400)

mol


I am working on the file "Drift_Fourier_Analysis_Bootstrap.ipynb".
The first task is:
Implementation
Can you write Python code for it?

def fourier_basis(x, degree, half_period):
    """Returns a 2-d array of fourier basis."""
    A = np.ones((x.size, 2 * degree + 1))

    for d in range(1, degree + 1):
        A[:, 2*d-1] = np.cos(d * np.pi * x / half_period)
        A[:, 2*d] = np.sin(d * np.pi * x / half_period)

    return A

def fit_driftfourier(x, data, weights, degree, half_period=0.5):
    """
    Fit y = f(x - x.min()) to data where f is given by
    fourier series + drift.

    Parameters
    ----------
    x : 1-d array
        x-coordinates
    data : 1-d array
        data values
    weights : 1-d array
        weights (>=0)
    degree : int
        degree of fourier series
    half_period : float
        half period

    Returns
    -------
    intercept : float
        intercept at x.min()
    slope : float
        slope (drift) for the normalized data
        (x - x.min())
    pfourier : 1-d array
        Fourier series parameters for the
        normalized data
    f_drift : callable
        Can be used to calculate the drift
        given any (non-normalized) x
    f_fourier : callable
        Can be used to calculate fourier series
    f_driftfourier : callable
        Can be used to calculate drift + fourier
    residual_std : float
        estimated standard deviation of residuals
    A : 2-d array
        matrix of "coefficients"

    """
    xmin = x.min()
    xnorm = x - xmin

    # coefficient matrix
    A = np.ones((x.size, 2 * degree + 2))
    A[:, 1] = xnorm
    A[:, 2:] = fourier_basis(xnorm, degree, half_period)[:, 1:]

    # linear weighted least squares
    results = np.linalg.lstsq(A * weights[:, np.newaxis],
                              data * weights)

    params = results[0]
    intercept = params[0]
    slope = params[1]
    pfourier = params[2:]

    f_drift = lambda t: slope * (t - xmin) + intercept
    f_fourier = lambda t: np.sum(fourier_basis(t - xmin, degree,
                                               half_period)[:, 1:]
                                 *...


I am working on the file "Drift_Fourier_Analysis_Bootstrap.ipynb".
The first task is:
Test data
A test with a dataset generated from given trend and intra-annual variability, with random noise.
Can you write Python code for it?

sns.set_context("notebook")
np.random.seed(seed=0)

intercept = 2.
trend = 0.1
variability = 0.4
noise = 0.05

x = np.arange(2009, 2012, 0.01)
x += np.random.random(size=x.size) * 0.1
y = variability * np.cos(x * np.pi / 0.5) + trend * (x - x.min()) + intercept
y += np.random.normal(scale=noise, size=y.size)
w = np.ones_like(x)

res = fit_driftfourier(x, y, w, 2)

print("fitted trend (slope): {}".format(res[1]))
print("fitted intercept at xmin: {}".format(res[0]))
print("fitted fourier parameters: {}".format(res[2]))
print("std of residuals: {}".format(res[6]))

f_drift, f_fourier, f_driftfourier = res[3:6]

plt.scatter(x, y, label='data')
plt.plot(x, f_drift(x), label='fitted annual trend')
plt.plot(x, f_driftfourier(x),
         label='fitted annual trend + intra-annual variability')
plt.legend()


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA.ipynb".
The first task is:
Plot data + fitted line
Can you write Python code for it?

ccl4_nya.plot(x='time', y='col_density', kind='scatter', label='data')
plt.plot(x, f_drift(x), label='fitted annual trend')
plt.plot(x, f_driftfourier(x),
         label='fitted annual trend + intra-annual variability')
plt.legend()


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA.ipynb".
The first task is:
Print the results
Can you write Python code for it?

print("95% CI for trend: ({:.3E}, {:.3E}) molec./cm2/yr".format(*cf['slope']))
print("95% CI for trend: ({:.3f}, {:.3f}) %/yr".format(*cf['slope_percent']))
print("95% CI for intercept: ({:.3E}, {:.3E}) molec./cm2".format(*cf['intercept']))
print("percentiles for fourier parameters: {}".format(cf['pfourier']))


print("fitted trend: {:.3E} molec./cm2/yr | {:.3f} %/yr".format(slope, slope_percent))
print("fitted intercept at xmin: {:.3E} molec./cm2".format(intercept))
print("fitted fourier parameters: {}".format(pfourier))
print("std of residuals: {:.3E} molec./cm2".format(residuals_std))


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Fit trend only (no fourier parameters added to the model), without bootstrapping. Also express trend as percentage.
Can you write Python code for it?

x = ccl4_nya_daily_means['time'].values
y = ccl4_nya_daily_means['col_density'].values
w = np.ones_like(x)
fourier_degree = 0   # fit trend only

res = fit_driftfourier(x, y, w, fourier_degree)
intercept, slope, pfourier, residuals_std = res[0:3] + (res[6],)
f_drift, f_fourier, f_driftfourier = res[3:6]
slope_percent = get_slope_percent(slope, intercept)


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Fit trend only, with bootstrap resampling. Compute 2.5% and 97.5% percentiles (i.e., the 95% interval) for each fitted parameter. Compute also the percentiles for trend in %.
Can you write Python code for it?

ci95_percentiles = (2.5, 97.5)

cf, boot_intercept, boot_slope, boot_pfourier = cf_driftfourier(
    x, y, w, fourier_degree, nboot=20000,
    percentiles=ci95_percentiles
)

boot_slope_percent = get_slope_percent(boot_slope, boot_intercept)
cf['slope_percent'] = np.percentile(boot_slope_percent, ci95_percentiles)


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Histograms of the distributions of the annual trend and intercept estimates obtained from bootstrap resampling.
Can you write Python code for it?

fig, axes = plt.subplots(1, 3, figsize=(13, 3))

def plot_bootdist(v, ax, label):
    sns.distributions.distplot(v, ax=ax)
    plt.setp(ax, xlabel=label, yticks=[])

plot_bootdist(boot_slope, axes[0], 'trend (molec./cm2/yr)')
plot_bootdist(boot_slope_percent, axes[1], 'trend (%/yr)')
plot_bootdist(boot_intercept, axes[2], 'intercept (molec./cm2)')


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Plot data + fitted line
Can you write Python code for it?

ccl4_nya_daily_means.plot(x='time', y='col_density', kind='scatter', label='data')
plt.plot(x, f_drift(x), label='fitted annual trend')
plt.plot(x, f_driftfourier(x),
         label='fitted annual trend + intra-annual variability')
plt.legend()


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Plot of daily mean column densities with errorbars (stdev).
Can you write Python code for it?

ccl4_nya_daily_stds = ccl4_nya.groupby(by_day).apply(np.std)

ccl4_nya_daily_means.plot(x='time', y='col_density',
                          kind='scatter', color='r',
                          yerr=ccl4_nya_daily_stds)


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Plot of daily means vs. raw data
Can you write Python code for it?

ccl4_nya_daily_stds = ccl4_nya.groupby(by_day).apply(np.std)
ax = plt.subplot(111)

ccl4_nya.plot(x='time', y='col_density', kind='scatter', ax=ax,
              label='raw measurements')
ccl4_nya_daily_means.plot(x='time', y='col_density',
                          kind='scatter', ax=ax, color='r',
                          label='daily means')


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
Print the results
Can you write Python code for it?


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
The fit below is performed from the daily mean column densities:
Can you write Python code for it?

by_day = [ccl4_nya.index.year, ccl4_nya.index.month, ccl4_nya.index.day]

ccl4_nya_daily_means = ccl4_nya.groupby(by_day).apply(np.mean)

ccl4_nya_daily_means.head()


I am working on the file "Drift_Fourier_Analysis_Bootstrap_CCL4_NYA_Daily_Means.ipynb".
The first task is:
The values in the time axis are given in decimal years. We first need to express them as real date & time values.
Can you write Python code for it?

def decimal_year_to_datetime(val):
    """
    Convert `val` given in decimal years into a
    `datetime.datetime` object, via the computation of the
    total duration of the year in seconds.
    """
    year = int(val)
    frac = val - year

    year_dt = datetime(year, 1, 1)
    year_delta = datetime(year + 1, 1, 1) - year_dt
    year_delta_seconds = year_delta.total_seconds()

    return year_dt + timedelta(seconds=year_delta_seconds * frac)

decimal_years_to_datetimes = np.vectorize(decimal_year_to_datetime)

# assign the datetimes as index of the dataset
ccl4_nya.index = pd.DatetimeIndex(decimal_years_to_datetimes(ccl4_nya['time'].values))

ccl4_nya.head()


I am working on the file "Drill Central Limit Theorem.ipynb".
The first task is:
What changes, and why?
The t-value decreases in the second case (when p1=0.4, p2=0.5) and the p-value is much higher. The t value decreases because the difference between means is lower and the increase if the p-value shows the that the noise due to variability is growing in weach case
Can you write Python code for it?

#Change the distribution of your populations from binomial to a distribution of your choice

pop3 = np.random.standard_t(25, 10000)
pop4 = logistic = np.random.logistic(9,2, 10000)
plt.hist(pop1, alpha=0.5, label='Population 3')
plt.hist(pop2, alpha=0.5, label='Population 4')
plt.legend(loc='upper right')
plt.show()

print(pop1.mean())
print(pop2.mean())
print(pop1.std())
print(pop2.std())

#Samples of the new pop1 and pop2.

sample7 = np.random.choice(pop3, 100, replace=True)
sample8 = np.random.choice(pop4, 100, replace=True)

print(sample7.mean())
print(sample8.mean())
print(sample7.std())
print(sample8.std())

plt.hist(sample7, alpha=0.5, label='sample 7')
plt.hist(sample8, alpha=0.5, label='sample 8')
plt.legend(loc='upper right')
plt.show()

#Compare samples Calculate t-value & p-value.

from scipy.stats import ttest_ind
from scipy.stats import ttest_1samp

mean1 = pop3.mean()
mean2 = pop4.mean()

print(ttest_1samp(sample7, mean1))
print(ttest_1samp(sample8, mean2))

print(ttest_ind(sample8, sample7, equal_var=False))


I am working on the file "Drill Central Limit Theorem.ipynb".
The first task is:
What values change, and what remain the same?
When running the one sample t test, p is always higher than 0.05, meaning that at 5% significane level the sample provides sufficient evidence to conclude that the mean of the sample is the calculated mean in all cases (for both populations)
Regarding the samples: means change, the lower the number of datapoints the less accurate the representation of the population.
T-values increase with the size while p-values tend to zero showing that the difference in means is due to the difference in the populations and not just variability.
Can you write Python code for it?

#Change the population value for pop1 to 0.3

pop1 = np.random.binomial(10, 0.3, 10000)
pop2 = np.random.binomial(10,0.5, 10000)
plt.hist(pop1, alpha=0.5, label='Population 1')
plt.hist(pop2, alpha=0.5, label='Population 2')
plt.legend(loc='upper right')
plt.show()

print(pop1.mean())
print(pop2.mean())
print(pop1.std())
print(pop2.std())

#Samples of the new pop1 and pop2

sample5 = np.random.choice(pop1, 100, replace=True)
sample6 = np.random.choice(pop2, 100, replace=True)

print(sample5.mean())
print(sample6.mean())
print(sample5.std())
print(sample6.std())

plt.hist(sample5, alpha=0.5, label='sample 5')
plt.hist(sample6, alpha=0.5, label='sample 6')
plt.legend(loc='upper right')
plt.show()

#Compare samples Calculate t-value & p-value.

from scipy.stats import ttest_ind
from scipy.stats import ttest_1samp

mean1 = pop1.mean()
mean2 = pop2.mean()

print(ttest_1samp(sample5, mean1))
print(ttest_1samp(sample6, mean2))

print(ttest_ind(sample6, sample5, equal_var=False))

#Then change the population value p for group 1 to 0.4, and do it again

pop1 = np.random.binomial(10, 0.4, 10000)
pop2 = np.random.binomial(10,0.5, 10000)
plt.hist(pop1, alpha=0.5, label='Population 1')
plt.hist(pop2, alpha=0.5, label='Population 2')
plt.legend(loc='upper right')
plt.show()

print(pop1.mean())
print(pop2.mean())
print(pop1.std())
print(pop2.std())

#Samples of the new pop1 and pop2

sample5 = np.random.choice(pop1, 100, replace=True)
sample6 = np.random.choice(pop2, 100, replace=True)

print(sample5.mean())
print(sample6.mean())
print(sample5.std())
print(sample6.std())

plt.hist(sample5, alpha=0.5, label='sample 5')
plt.hist(sample6, alpha=0.5, label='sample 6')
plt.legend(loc='upper right')
plt.show()

#Compare samples Calculate t-value & p-value.

from scipy.stats import ttest_ind
from scipy.stats import ttest_1samp

mean1 = pop1.mean()
mean2 = pop2.mean()

print(ttest_1samp(sample5, mean1))
print(ttest_1samp(sample6, mean2))

print(ttest_ind(sample6, sample5, equal_var=False))


I am working on the file "Drill+and+Challenge+Present+the+same+data+multiple+ways.ipynb".
The first task is:
Challenge
Can you write Python code for it?

#Clean the Airbnb dataset by dropping all the varibles that are unrelated to the challenge
df3 = df.drop(["id", "name", "host_id", 'host_name', 'neighbourhood','neighbourhood_group', 'latitude', 'longitude', 'last_review', 'calculated_host_listings_count','reviews_per_month' ], axis=1)
df3.head()

g = sns.pairplot(df3, hue="room_type", diag_kind="hist")
for ax in g.axes.flat:
    plt.setp(ax.get_xticklabels(), rotation=45)


I am working on the file "Drill+and+Challenge+Present+the+same+data+multiple+ways.ipynb".
The first task is:
Choose one continuous variable and one categorical variable, and plot them six different ways
Can you write Python code for it?

#Bar plot with error of the categorical variable: room type and price
sns.set(style="whitegrid")

# Draw a nested barplot to show prices per room type in Airbnb
g = sns.factorplot(x="room_type", y="price", data=airbnb, size=4, kind="bar", palette="muted")
g.despine(left=True)
g.set_ylabels("prices")
plt.ylim([0, 250])
plt.title('Price (GBP) per room type')

#Boxplot
#Show price per room type in horizontal boxplot with log scale
sns.set(style="ticks")

# Initialize the figure with a logarithmic x axis
f, ax = plt.subplots(figsize=(7, 6))
ax.set_xscale("log")

# Plot the price with horizontal boxes
sns.boxplot(x="price", y="room_type", data=airbnb, whis=np.inf, palette="vlag")

# Add in points to show each observation
#sns.swarmplot(x="price", y="room_type", data=airbnb, size=2, color=".3", linewidth=0)

# Tweak the visual presentation
ax.xaxis.grid(True)
ax.set(ylabel="")
sns.despine(trim=True, left=True)
plt.title('Price (GBP) per room type')

#Histogram to see the frequency of pricese per room type
g = sns.FacetGrid(df, col="room_type")
g.map(plt.hist, "price")
plt.xlim([0, 1000])
plt.ylim([0, 30000])
plt.title('Prices (GBP) per room type')

#Count the number of instances for each room type
sns.countplot(x="room_type", data=df, palette="Greens_d")

#Show the prices with a stripplot
sns.stripplot(x="room_type", y="price", data=df, jitter=True)
plt.ylim([0, 500])
plt.title('Prices (GBP) per room type')

#Count the number of instances for each type of room with a horizontal bar plot
f, ax = plt.subplots(figsize=(7, 3))
sns.countplot(y="room_type", data=df, color="c")


I am working on the file "Drill+and+Challenge+Present+the+same+data+multiple+ways.ipynb".
The first task is:
Choose one variable and plot that variable four different ways
Can you write Python code for it?

#Dataset from Airbnb showing the listings for London (date: March 3017)
airbnb = pd.read_csv('https://raw.githubusercontent.com/borja876/Thinkful-DataScience-Borja/master/listings.csv')
df = pd.DataFrame(airbnb)

#Distribution of the variable price
sns.distplot(df['price'])
plt.xlim([0, 600])
plt.title('More frequent prices (GBP)')

#A random distribution has been created and plotted
x = np.random.normal(size=1000)
sns.distplot(x)

#Desnity function showing the bins that have been used to plot it
sns.distplot(x, hist=False, rug=True)

#Plot of a gamma distribution
x = np.random.gamma(2, size=200)
sns.distplot(x, kde=False, fit=stats.gamma)


I am working on the file "Drill+and+Challenge+Present+the+same+data+multiple+ways.ipynb".
The first task is:
Choose two continuous variables, and plot them three different ways
Can you write Python code for it?

#The dataset of Airbnb has been cleaned leaving only the 4 variables that are required for plotting
df = pd.DataFrame(airbnb)
df1 = df.drop(["id", "name", "host_id", 'host_name', 'neighbourhood','neighbourhood_group', 'latitude', 'longitude', 'room_type', 'last_review', 'calculated_host_listings_count','reviews_per_month' ], axis=1)
df1.head()

#Plot the relationship between availability and price
g = sns.lmplot(x="price", y="availability_365", col="room_type", data=df)
plt.xlim([0, 1000])
plt.ylim([-50, 500])

#Dropped the minimum nights and number of reviews to avoid problems. Plot the distributions and bivariate relation between both variables
df2 = df1.drop(['minimum_nights','number_of_reviews'], axis=1)
# Set style of scatterplot
sns.set_context("notebook", font_scale=1.1)
sns.set_style("ticks")

# Create scatterplot of dataframe
sns.pairplot(df2)

# title
plt.title('Relationship between Price (GBP) and availability (Days)')

#Density regions for the relationship between availability and price
sns.jointplot("price", "availability_365", data=df2, xlim=250, ylim=None, kind="kde")


I am working on the file "Ducts.ipynb".
The first task is:
Find the best $\alpha$ for $p = 15, N_y = 128, N_z = 1024$
Can you write Python code for it?

alphs = list(np.linspace(0,pi/2, 16, endpoint=False))
Re=2000;
N = 16
Nl = 129
Nz = 1025
yms = []; y1s = []; y10s = []; zms = []; cms = []
for alph in alphs:
    yl=mesh(alph, Nl)
    ym, y1, y10, zm, cm = wall_units(yl,Nz, N,Re)
    yms.append(ym)
    y1s.append(y1)
    y10s.append(y10)
    zms.append(zm)
    cms.append(cm)

alpha = 0.22
plot_units(yms, y1s, y10s, zms, alpha)
yl=mesh(alpha, Nl)
ym, y1, y10, zm, cm = wall_units(yl,Nz, N,Re)
print("Final values:", ym, y1, y10, zm, cm)

yl=mesh(0.22, 129)
plt.figure(figsize=(128,1))
plot_mesh(yl)
plt.savefig('mesh_16.png')
print(yl)


I am working on the file "Ducts.ipynb".
The first task is:
Find the best $\alpha$ for $p = 31, N_y = 64, N_z = 512$
Can you write Python code for it?

alphs = list(np.linspace(0,pi/2, 16, endpoint=False))
Re=2000;
N = 31
Nl = 65
Nz = 513
yms = []; y1s = []; y10s = []; zms = []
for alph in alphs:
    yl=mesh(alph, Nl)
    ym, y1, y10, zm, cm = wall_units(yl,Nz, N,Re)
    yms.append(ym)
    y1s.append(y1)
    y10s.append(y10)
    zms.append(zm)

alpha = 0.36
plot_units(yms, y1s, y10s, zms, alpha)
yl=mesh(alpha, Nl)
ym, y1, y10, zm, cm = wall_units(yl,Nz, N,Re)
print("Final values:", ym, y1, y10, zm, cm)

yl=mesh(.36, 65)
plt.figure(figsize=(64,1))
plot_mesh(yl)
plt.savefig('mesh_32.png')
print(yl)


I am working on the file "Dummy.ipynb".
The first task is:
例題9-2 「係数ダミー」
以下のようにモデルを設定して回帰分析を行う。
$Y_{i} = \alpha + \beta X_{i} + \gamma D_{i} + \delta D_{i} X_{i} + u_{i}$
Can you write Python code for it?

# データ読み込み
data = pd.read_csv('example/k0902.csv')
data

# 説明変数設定
X = data[['X', 'D', 'DX']]
X = sm.add_constant(X)
X

# 被説明変数設定
Y = data['Y']
Y

# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())

# ダミー別データ
data_d0 = data[data["D"] == 0]
data_d1 = data[data["D"] == 1]

# グラフ生成
plt.plot(data["X"], data["Y"], 'o', label="data")
plt.plot(data_d0.X, results.fittedvalues[data_d0.index], label="D=0")
plt.plot(data_d1.X, results.fittedvalues[data_d1.index], label="D=1")
plt.xlim(min(data["X"])-1, max(data["X"])+1)
plt.ylim(min(data["Y"])-1, max(data["Y"])+1)
plt.title('9-2: Dummy Variable')
plt.legend(loc=2)
plt.show()


I am working on the file "Dummy.ipynb".
The first task is:
例題9-3 「t検定による構造変化のテスト」
例題9-2において$\gamma = 0$に関するP値は0.017であり、$\delta = 0$に関するP値は0.003であることから、標準的な有意水準を設定すれば、いずれのダミー変数も有意であるといえる。
例題9-4 「F検定による構造変化のテスト」
Can you write Python code for it?

# ダミー変数を加えない時のOLSモデル作成
X = data[['X']]
X = sm.add_constant(X)
model2 = sm.OLS(Y,X)
results2 = model2.fit()

# anova(Analysis of Variance)
print(sm.stats.anova_lm(results2, results))


I am working on the file "Duplicate Detection LSH Cosine.ipynb".
The first task is:
Globals
Reading the data

data has to be a .h5 data file.
data_path should contain the path to this .h5 data file.
Can you write Python code for it?

# 1 million summary data. Takes long!
data_path = 'msd_summary_file.h5'
# subset(10000 songs) summary data
data_path = 'MillionSongSubset/AdditionalFiles/subset_msd_summary_file.h5'

features = ['duration', 'end_of_fade_in','key', 'loudness', 'mode', 'start_of_fade_out','tempo','time_signature']

debug_print = True

# Comment out, when debugging.
np.random.seed(42)


I am working on the file "Duplicate Detection LSH Cosine.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

'''
Reads the data and the feature names and returns the track ids and the feature matrix.
The track_id field from the data is mandatory, therefore it will always be included
Args:
    feature_names(list of strings): list containing the feature names that will be included in the feature matrix.
    data(pandas.io.pytables.HDFStore table): table containing the data.
Returns:
    (numpy.ndarray, numpy.ndarray): matrix(#samples x 1) of track_ids, feature matrix(#samples x #features).
'''
def get_feature_matrix(feature_names,data):
    if 'track_id' in feature_names:
        songs = np.asarray(data['/analysis/songs'][feature_names])
    else:
        songs = np.asarray(data['/analysis/songs'][['track_id'] + feature_names])
    #indices = np.random.choice(range(np.size(songs,0)), sample_size)
    return np.array(songs[:,0]),np.array(songs[:,1:],dtype=np.float64)

'''
Returns a vector with normal distributed {0,1} values.
Args:
    n(int) : size of the vector.
Returns:
    list(int): list of length n of normal distributed {0,1} values.
'''
def get_random_vector(n):
    #rv = np.random.normal(0, 1, n)
    #rv[rv < 0] = -1
    #rv[rv > 0] = 1
    #rv[rv == 0] = np.random.randint(0,2)*2-1
    rv = 2*np.random.randint(0,2,n)-1
    return rv

'''
Returns the cosine of the angle of two given vectors
Args:
    a(numpy.ndarray): vector of real values.
    b(numpy.ndarray): vector of real values.
Returns:
    double: Returns the cosine of the angle of a and b.
'''
def cosine_angle(a,b):
    return np.dot(a,b) / (np.linalg.norm(a) * np.linalg.norm(b))

'''
Returns the cosine distance between two vectors.
Args:
    a(numpy.ndarray): vector of real values.
    b(numpy.ndarray): vector of real values.
Returns:
    double: Returns the cosine distance of a and b.
'''
def cosine_distance(a,b):
    return 1.0-cosine_angle(a,b)

'''
Returns a matrix(#samples x #bands) with hash-values for the different song based on the respective bands.
Args:
    Sketch(numpy.ndarray): Sketch matrix(#samples x #rows*#bands) with values in...


I am working on the file "Duplicate Detection with LSH Cosine Similarity.ipynb".
The first task is:
Algorithm Description

for each band b we store a hash_table in the dictionary buckets[b].
for each band b:
for each song s:
add song s to the bucket to which it hashes in the dictionary buckets[b].

for each key in buckets[b]:
if it contains more than one element, it contains candidates for duplication.
for all elements in this list, generate all unordered pairs.
for each such pair, if it is not contained in the candidates dictionary, add the pair with the corressponding cosine distance.

for each pair in candidates:
check if its cosine distance < cosine distance of epsilon, meaning we consider it a duplicate.
Can you write Python code for it?

'''
candidates(dict): Dictionary with key=(song_id,song_id), value=cosine_distance(song_id,song_id)
duplicates(list): List of tuples (songid, songid)
buckets(dict)   : Dictionary with key=band_id, value=dict with key=hash_key, value = list of song_id
'''

candidates = {}
duplicates = []
buckets = { i : {} for i in range(b) }

start = time.time()
for i in range(b):
    for j in range(N):
        hash_key = hb[j,i]
        if hash_key not in buckets[i]:
            buckets[i][hash_key] = []
        buckets[i][hash_key].append(j)
    for candidates_list in buckets[i].values():
        if len(candidates_list) > 1:
            for _i in range(len(candidates_list)):
                for _j in range(_i+1,len(candidates_list)):
                    songA = candidates_list[_i]
                    songB = candidates_list[_j]
                    if (songA,songB) not in candidates:
                        candidates[(songA,songB)] = cosine_distance(feature_matrix[songA,:],feature_matrix[songB,:])

cos_eps_dist = 1-math.cos(math.radians(eps))
for key in candidates.keys():
    if candidates[key] < cos_eps_dist:
        songA = key[0]
        songB = key[1]
        duplicates.append((songA,songB))

print("LSH Duration:", time.time() - start,"sec")
print("Nr. candidates:", len(candidates.keys()))
print("Nr. duplicates:",len(duplicates))


I am working on the file "Duplicate Detection with LSH Cosine Similarity.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

def get_feature_matrix(feature, data):
    '''
        Reads the data and the feature names and returns the track ids and the feature matrix.
        The track_id field from the data is mandatory, therefore it will always be included
        Args:
            feature_names(list of strings): list containing the feature names that will be included
            in the feature matrix.
            data(pandas.io.pytables.HDFStore table): table containing the data.
        Returns:
            (numpy.ndarray, numpy.ndarray): (N, 1) of track_ids, feature matrix (N, D).
    '''
    if 'track_id' in feature:
        songs = np.asarray(data[osp.join('analysis','songs')][feature])
    else:
        songs = np.asarray(data[osp.join('analysis','songs')][['track_id'] + feature])

    return np.array(songs[:, 0]), np.array(songs[:, 1:], dtype=np.float64)

def get_random_vector(n):
    '''
        Returns a vector with normal distributed values {-1,1}.
        Args:
            n (int) : size of the vector.
        Returns:
            ndarray : list of length n of normal distributed values {-1,1}.
    '''
    return 2*np.random.randint(0, 2, n) - 1

def cosine_angle(a, b):
    '''
        Returns the cosine of the angle of two given vectors
        Args:
            a(numpy.ndarray): vector of real values.
            b(numpy.ndarray): vector of real values.
        Returns:
            double: the cosine of the angle between a and b.
    '''
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def cosine_distance(a, b):
    '''
        Returns the cosine distance between two vectors.
        Args:
            a(numpy.ndarray): vector of real values.
            b(numpy.ndarray): vector of real values.
        Returns:
            double: the cosine distance of a and b.
    '''
    return 1.0 - cosine_angle(a, b)

def get_hash_bands(sketch, r, b):
    '''
        Computes the signature matrix (#samples, #bands) with hash-values for the different song based on the respective bands.
        Args:
       ...


I am working on the file "Duplicate Detection with LSH Cosine Similarity.ipynb".
The first task is:
LSH Cosine Similarity
Can you write Python code for it?

# data and algorithm parameters
'''
D = number of features
N = number of samples
b = number of bands
r = number of rows
eps = angle threshold(degrees)
'''
D = np.size(feature_matrix,1)
N = np.size(feature_matrix, 0)
b = 3
r = 64
eps = 2

'''
Operations:
- Generate matrix of random vectors with values in {-1,1}.

RV: matrix(#bands*#rows x n_features)
'''
RV = np.array([get_random_vector(D) for i in range(b*r)])

if verbose:
    print("Shape RV:",np.shape(RV))
    print("Random vectors matrix RV:\n",RV)

'''
Operations:
    - Generate sketch matrix, by performing
Clip sketch matrix to 0-1 range for hashing.
Dimensionality: n_samples x n_bands*n_rows
'''
Sketch = np.dot(feature_matrix, RV.T)
Sketch[Sketch < 0] = -1
Sketch[Sketch > 0] = 1
Sketch[Sketch == 0] = np.random.randint(0,2)*2-1

if verbose:
    print("Shape Sketch:",Sketch.shape)
    print("Sketch:\n",Sketch)

# clip values of Sketch matrix in domain {0,1} to easily hash them.
Sketch[Sketch < 0] = 0

if verbose:
    print("Shape Binary Sketch:",Sketch.shape)
    print("Binary Sketch:\n",Sketch)

hb = get_hash_bands(Sketch,r, b)

if verbose:
    print("Shape hb:",hb.shape)
    print("hb:\n",hb)


I am working on the file "Dynamic Equilibrium Example-Bitcoin.ipynb".
The first task is:
Dynamic equilibrium model
This applies the entropy minimization method of finding the dynamic equilibrium.
Can you write Python code for it?

forecastDuration = 0.25
forecastResolution = 100
forecastTime = np.array(list(range(int(output['data'][-1,0]*forecastResolution),int((output['data'][-1,0]+forecastDuration)*forecastResolution),1)))/forecastResolution

temp = dynamic_equilibrium_optimize(output['data'],alphaRange=(-2.9,-2.4),binWidth=0.5,alphaDelta=0.001,method='brute')
transform = log_linear_transform(output['data'], temp)

fitFunction = two_shock

transform[:,1] = np.exp(transform[:,1])
guess = [1.0,-0.01,2017.35,1.0,-0.01,2017.6,0.0]
result = dynamic_eq_fit(fitFunction, transform, guess)

fitData = result['fit']
popt = result['params']
pcov = result['cov']

forecastFit = np.array(list(map(lambda x:np.exp(fitFunction(x,*popt)),forecastTime)))

plotData[:,1] = np.exp(np.log(transform[:,1])+temp*(plotData[:,0]-plotData[0,0]))
plotScale = 1
plotLabel = output['name']
plotData2 = np.exp(np.log(fitData)+temp*(plotData[:,0]-plotData[0,0]))
plotData3 = np.exp(np.log(forecastFit)+temp*(forecastTime-plotData[0,0]))

pl.figure(figsize=(12,6))
for index in range(len(result['transitions'])):
    pl.axvline(x=result['transitions'][index],color='0.5',linewidth=0.5)
pl.plot(plotData[:,0],plotData[:,1],plotData[:,0],plotData2,forecastTime,plotData3)
pl.ylabel(plotLabel)
pl.show()

print('Dynamic equilibrium growth rate (entropy min) = ',np.round(100*temp,decimals=2))
print('Shock centers = ',np.round(result['transitions'],decimals=1))
print('Shock widths = ',np.round(result['shock_widths'],decimals=2))
print('Shock magnitudes = ',np.round(result['shock_mags'],decimals=2))


I am working on the file "Dynamic Equilibrium Example-Bitcoin.ipynb".
The first task is:
Example: bitcoin
Here we read in the bitcoin exchange rate data and plot it. We take only the most recent data because the version of python IEtools used here could only look at two shocks.
Can you write Python code for it?

filename='C:/econdata/btcUSDtimeseries_recent.xls'

book = xlrd.open_workbook(filename)
sheet = book.sheet_by_index(0)
outputList = []
outputName = 'USD/btc'
for rowIndex in range(sheet.nrows):
    outputList.append([sheet.cell(rowIndex,0).value, sheet.cell(rowIndex,1).value])
dataOutput =  np.array(outputList)
output = {'name':outputName,'data':dataOutput}

plotData = output['data']
plotType = pl.plot
plotScale = 1
plotLabel = output['name']
pl.figure(figsize=(12,6))
plotType(plotData[:,0],plotData[:,1]*plotScale)
pl.ylabel(plotLabel)
pl.show()


I am working on the file "Dynamic Equilibrium Examples.ipynb".
The first task is:
Dynamic equilibrium models
This applies the entropy minimization method of finding the dynamic equilibrium.
Can you write Python code for it?

temp = dynamic_equilibrium_optimize(output['data'],alphaDelta=0.001,method='brute')
transform = log_linear_transform(output['data'], temp)

transform[:,1] = np.exp(transform[:,1])
guess = [1.0,-0.1,2008.0,0.1,0.1,2014.0, 0.0]
result = dynamic_eq_fit(two_shock, transform, guess)

fitData = result['fit']
popt = result['params']
pcov = result['cov']

plotData = transform
plotScale = 1
plotLabel = output['name']+' [transformed]'
plotData2 = fitData

pl.figure(figsize=(12,6))
for index in range(len(result['transitions'])):
    pl.axvline(x=result['transitions'][index],color='0.5',linewidth=0.5)
pl.plot(plotData[:,0],np.log(plotData[:,1]),plotData[:,0],np.log(plotData2))
pl.ylabel(plotLabel)
pl.show()

print('Dynamic equilibrium growth rate (entropy min) = ',np.round(100*temp,decimals=2))
print('Shock centers = ',np.round(result['transitions'],decimals=1))
print('Shock widths = ',np.round(result['shock_widths'],decimals=2))
print('Shock magnitudes = ',np.round(result['shock_mags'],decimals=2))


I am working on the file "Dynamic Equilibrium Examples.ipynb".
The first task is:
Example: UNRATE
Here we read in the unemployment rate data and plot it. We take only the most recent data because the version of python IEtools used here could only look at two shocks.
Can you write Python code for it?

filename='C:/econdata/UNRATE.csv'

output0 = FREDcsvRead(filename)
output = {
    'data': output0['data'][-170:-1],
    'name': output0['name']
}

plotData = output['data']
plotType = pl.plot
plotScale = 1
plotLabel = output['name']+' [$ p.c.]'
pl.figure(figsize=(12,6))
plotType(plotData[:,0],plotData[:,1]*plotScale)
pl.ylabel(plotLabel)
pl.show()


I am working on the file "Dynamic Equilibrium Examples.ipynb".
The first task is:
This uses the global fit approach.
Can you write Python code for it?

guess = [1.0,-1.0,1980.0,0.1,0.1,2010.0,0.04,0.0]
result = dynamic_eq_fit(two_shock_eq, output['data'], guess)

fitData = result['fit']
popt = result['params']
pcov = result['cov']

growthRate=np.round(popt[-2]*100,decimals=2)
growthError=np.round(2*100*np.sqrt(np.diag(pcov))[-2],decimals=2)

plotData = output['data']
plotType = pl.plot
plotType = pl.semilogy
plotScale = 10**6
plotLabel = output['name']+' [$ p.c.]'
plotData2 = fitData

pl.figure(figsize=(12,6))
for index in range(len(result['transitions'])):
    pl.axvline(x=result['transitions'][index],color='0.5',linewidth=0.5)
plotType(plotData[:,0],plotData[:,1]*plotScale,plotData[:,0],plotData2*plotScale)
pl.ylabel(plotLabel)
pl.show()

print('Dynamic equilibrium growth rate = ',growthRate,'+/-',growthError,'%')
print('Shock centers = ',np.round(result['transitions'],decimals=1))
print('Shock widths = ',np.round(result['shock_widths'],decimals=2))
print('Shock magnitudes = ',np.round(result['shock_mags'],decimals=2))


I am working on the file "Dynamic Programming.ipynb".
The first task is:
Apples on a table
A table composed of N x M cells, each having a certain quantity of apples, is given. You start from the upper-left corner. At each step you can go down or right one cell. Find the maximum number of apples you can collect.
Can you write Python code for it?

def wrapper(mat, N, M):
    states = [[0 for c in range(M)] for r in range(N)]
    return apples(mat,0,0,N,M, states)

def apples(mat,i,j,N,M, states):
    if i >= N or j >= M:
        return 0
    if states[i][j] > 0:
        return states[i][j]
    states[i][j] = mat[i][j] + max([apples(mat,i+1,j,N,M, states), apples(mat,i,j+1,N,M, states)])
    return states[i][j]

arr = [int(k) for k in raw_input("Array(comma seperated): ").split(',')]
N, M = [int(k) for k in raw_input("N, M (comma seperated): ").split(',')]
mat = [arr[i*N:i*N+M] for i in range(N)]
print mat
wrapper(mat, N, M)


I am working on the file "Dynamic Programming.ipynb".
The first task is:
Longest sequence problem
Given a sequence of $N$ numbers – $A[1] , A[2] , …, A[N]$ . Find the length of the longest non-decreasing sequence.
Approach
$len_{LSS}(A, 0, N) = min({len_{LSS}(A, i, N-i) | i \in [1,N]})$
Can you write Python code for it?

def wrapper(arr):
    states = [1]*len(arr)
    return longest_sub(arr, len(arr)-1, states)

def longest_sub(arr, i, states):
    if i <= 0:
        return 1
    if states[i] > 1:
        return states[i]
    for j in range(i):
        lj = longest_sub(arr, j, states)
        if arr[j] <= arr[i]:
            print j, i, states
            states[i] = lj + 1
        else:
            states[i] = lj
    return states[i]

arr = [int(k) for k in raw_input("Array(comma seperated): ").split(',')]
print arr
print wrapper(arr)


I am working on the file "Dynamic programming class.ipynb".
The first task is:
$$
s_{new} = \begin{cases} (t+1,h-1,i) & \text{if } x = 0  \ (t+1,h+1,i) & \text{if } x = 1 \ (t+1,h+1.5,i) & \text{if } x = 2\end{cases}
$$
Can you write Python code for it?

assert(hjb((settings['T'],settings['H_init']-1)) == 10000)
assert(hjb((settings['T'],settings['H_init'])) == 0)
assert(hjb((settings['T']-1,settings['H_min']-1)) == 10000)
assert(hjb((settings['T']-1,settings['H_max']+1)) == 10000)


I am working on the file "Dynamic programming class.ipynb".
The first task is:
$$
v(s) = \min_x(cost(x,s) + v(new state(x,s)))
$$
Can you write Python code for it?

class DynamicProgram(object):
    """
    Generate a dynamic program to find a set of optimal descissions using the HJB.

    define the program by:

    Setting intial states via:            set_inital_state(list or int)
    Setting the number of steps via:      set_step_number(int)

    Add a set of descissions:             add_decisions_set(set)
    Add a cost function:                  add_cost_function(function in terms of state )

    Add a state change equation:          add_state_eq(function(state))
    Add an expression for the last value: add_final_value_expression(function(state,settings))
    Add limits on the states:             add_state_limits(lower=list or int,upper = list or int)

    See below for examples:

    """
    def __init__(self):
        self.settings = {
                        'Lower state limits' : [],
                        'Upper state limits' : [],
                        'x_set' : set(),
                        'cache' : {},}

    def add_state_eq(self,function):
        """Returns a tuple describing the states.
        Remember to increment the first state, b convention the number of steps.
        Load additional parameters (usually needed for cost and state value with global variables)

        Example of a function that changes the state by the decission:
        def new_state(x,s):
            return (s[0]+1,s[1]+x) #Return a tuple, use (s[:-1]+(5,)) to slice tuples.

        """
        self.settings['State eq.'] = function

    def add_cost_function(self,function):
        """
        Returns a float or integer describing the cost value.
        Load additional parameters (usually needed for cost and state value with global variables)

        Example is a function that simply returns the decision as cost:
        def cost(x,s):
            return x

        """
        self.settings['Cost eq.'] = function

    def add_final_value_expression(self, function,):
        """
     ...


I am working on the file "Dynamic programming class.ipynb".
The first task is:
The cost of operating a pump with a given wind turbine power input given by a certain state is given by:
$$
cost(x,t,h,j) := \begin{cases} T(t) \times (x \times P_p - W(t,j)) & \text{if} +ve \
E_{xp} \times (x \times P_p - W(t,j)) & \text{if} -ve\end{cases}
$$
where $W(t,j)$ is the wind power output at time $t$ with an error state $j$.
Can you write Python code for it?

# Convention s = t,h,j
def stoch_simple_state(x,s):
    #print s
    if x == 0:
        return (s[0]+1,s[1]-1,s[2])
    elif x == 1:
        return (s[0]+1,s[1]+1,s[2])
    elif x == 2:
        return (s[0]+1,s[1]+1.5,s[2])

def err_corr_wind_power_cost(x,s):
    Tariff = [5,5,5,5,5,8,8,8,8,8,12,12,12,12,12,50,50,50,50,20,20,6,5,5]
    Wind = [46,  1,  3, 36, 30, 19,  9, 26, 35,  5, 49,  3,  6, 36, 43, 36, 14,
       34,  2,  0,  0, 30, 13, 36]

    diff = np.array([-1,0,1])*3

    Export_price = 5.5

    wind_out = Wind[s[0]]+diff[s[2]]
    if wind_out <= 0:
        wind_out = 0

    power_con = x*60-wind_out
    if power_con >= 0:
        return power_con*Tariff[s[0]]
    else:
        return power_con*Export_price

def val_T(s,settings):
    if s[1] < settings['Initial state'][1]:
        return float('inf')
    else:
        return 0

transition = np.array([[0.4,0.5,0.1],[0.2,0.6,0.2],[0.1,0.5,0.4]])

pumping_stoch = StochasticProgram()
pumping_stoch.add_decisions_set({0,1,2})
pumping_stoch.add_cost_function(err_corr_wind_power_cost)
pumping_stoch.add_state_eq(stoch_simple_state)
pumping_stoch.add_final_value_expression(val_T)
pumping_stoch.add_state_limits(lower=[0,0],upper = [200,3])
pumping_stoch.set_inital_state([100,1])
pumping_stoch.set_step_number(24)
pumping_stoch.add_transition_matrix(transition)
#print pumping_stoch.settings
pumping_stoch.solve()
#pumping_stoch.retrieve_decisions()

pumping_stoch.retrieve_decisions()


I am working on the file "Dynamic programming class.ipynb".
The first task is:
The cost of operating a pump with a given wind turbine power input is given by:
$$
cost(x,t,h) := \begin{cases} T(t) \times (x \times P_p - W(t)) & \text{if} +ve \
E_{xp} \times (x \times P_p - W(t)) & \text{if} -ve\end{cases}
$$
where $x$ is the descision variable, $W(t)$ is the wind turbine output in time step $t$. $P_p$ is the pump power, $E_{xp}$ is the export price.
Can you write Python code for it?


I am working on the file "Dynamic programming class.ipynb".
The first task is:
We can also solve classic dynamic programming problems such as the knapsack problem, hannoi towers or the fibonacci number calculation. Blank functions are outlined below.
The functions must fulfill a range of conditions:
$$
f:\mathcal{S}^n\times x \rightarrow  \mathbb{R}
$$
where $\mathcal{S}$ is the set of permissable states, $x$ the decision variable and $n$ the number of dimensions of the state. These are defined by $x \in \mathcal{X}$ the \texttt{set()} of permissable decisions. The state $s \in \mathcal{S}^n$ where $n \geq 2$ and $\mathcal{S} \subset \mathbb{R} $ and is finite.
The new state is defined by a function such that:
$$
f:\mathcal{S}^n\times x \rightarrow  \mathcal{S}^n
$$
The value of the final stage is defined as:
$$
f:\mathcal{S}^n \rightarrow  \mathbb{R}
$$
which as an impimentation detail has the required argument settings, to which features can be added through add_setting(key,value) where key must be a new and unique dictionary key and value may be any permissable dictionary entry.
Can you write Python code for it?

#help(DynamicProgram)

def cost(x,s):
    """Return a float or integer"""
    pass

def new_state(x,s):
    """Return a tuple"""
    pass

def val_T(s,settings):
    """Return a float or int"""
    pass


I am working on the file "Dynamic programming class.ipynb".
The first task is:
We can have more than one state variable. For example we can add a second tank and now pump to either of them:
Here they have similar equations, but they can be completly independant, of each other. Any cost and state function that meets the requirments above is allowed.
$$
s_{new} = \begin{cases} (t+1,h_1-1,h_2-1) & \text{if } x = 0  \ (t+1,h_1+1,h_2) & \text{if } x = 1 \ (t+1,h_1,h_2+2) & \text{if } x = 2\end{cases}
$$
Can you write Python code for it?

def simple_state2(x,s):
    if x == 0:
        return (s[0]+1,s[1]-1,s[2]-1)
    elif x == 1:
        return (s[0]+1,s[1]+1,s[2])
    elif x == 2:
        return (s[0]+1,s[1]  ,s[2]+2)

# We also need to update the final value function.
def val_T2(s,settings):
    if s[1] < settings['Initial state'][1] or s[2] < settings['Initial state'][2]:
        return float('inf')
    else:
        return 0

# For now we leave the cost function the same, but that could also be changed.


I am working on the file "Dynamic programming class.ipynb".
The first task is:
We can solve a very simple pump optimsiation where the state of water in a tank is given by h and described by:
$$
s_{new} = \begin{cases} (t+1,h-1) & \text{if } x = 0  \ (t+1,h+1) & \text{if } x = 1 \ (t+1,h+1.5) & \text{if } x = 2\end{cases}
$$
The operating cost are described by:
$$
cost = tarrif(t)\times x
$$
where $x$ is the descission variable.
The final value is given by:
$$
V_T = \begin{cases} 0 & \text{if: } h_T \geq h_0 \ Inf &\text{otherwise} \end{cases}
$$
Can you write Python code for it?


I am working on the file "Dynamic programming class.ipynb".
The first task is:
state is given by $s = (t,h,i)$
$$
v(s) = \min_x(cost(x,s) + \sum_j p_{ij} v(new state(x,s)))
$$
Can you write Python code for it?

def make_schedule(settings):
    sched = np.ones(settings['T'])*np.nan
    cost_calc= 0
    elev = np.ones(settings['T']+1)*np.nan

    s = settings['Initial state']
    t = 0

    while t < settings['T']:
        sched[t] = settings['cache'][s][1]
        print sched[t]

        cost_calc += settings['Cost eq.'](sched[t],s)
        elev[t] = s[1]

        s = settings['State eq.'](sched[t],s)
        t += 1

    elev[settings['T']] = s[1]

    return cost_calc, sched, elev

def make_schedule2(settings):

    sched_stack =  []
    cost_summary = []
    string_stack = []
    elev = np.ones(settings['T']+1)*np.nan
    for ij in [0,1,2]:
        cost_calc = 0
        string_stack.insert(i,[])
        s = settings['Initial state']
        t = 0
        #string_stack[ij].insert(0,[])
        #string_stack[ij].insert(0,'{0:2} {1} {2}'.format(t,settings['cache'][s][1], s[1:]))
        string_stack[ij].insert(0,'{0}'.format(s))
        while t < settings['T']:

            state = tuple(sk if sk in s[:-1] else ij for sk in s  )
            print s, s[:-1], state
            x = settings['cache'][state][1]
            print x
            cost_calc += settings['Cost eq.'](x,s)
            elev[t] = s[1]

            s = settings['State eq.'](x,s)
            t += 1
            string_stack[ij].insert(t,'{0} {1} {2}'.format(x,s[1:],ij) )

    elev[settings['T']] = s[1]

    return string_stack, cost_calc

make_schedule2(settings)

settings['cache']

def stoch_hjb(s):
    global settings
    if settings['cache'].has_key(s):
        return settings['cache'][s][0]

    if s[0] == settings['T'] and s[1] < settings['H_init']:
        return 10000

    elif s[0] == settings['T'] and s[1] >= settings['H_init']:
        return 0

    elif s[1] < settings['H_min'] or s[1] > settings['H_max']:
        return 10000

    else:
        p=[]
        for x in settings['x_set']:
            future = sum(stoch_hjb(settings['State eq.'](x,(s[0],s[1],i)))
                         ...


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
2b. Find the minimum levenshtein distance
Use your levenshtein_distance function to find the closest_match between a candidate word and an iterable of words. Note that if multiple words from words share the minimal edit distance to the candidate, you should return the word which would come first in a dictionary.
As a concrete example, zark has an edit distance of 1 with both ark and bark, but you would return ark as it comes lexicographically before bark.
Your function should return a tuple of two values - first the closest word match, and secondly the edit distance between this word and the candidate.
python
closest, distance = closest_match('zark', ['ark', 'bark', ...])
assert closest == 'ark'
assert distance == 1
Can you write Python code for it?

def closest_match(candidate, words):
    # complete function without changing signature
    n = len(words)
    max_edit_dist = len(candidate)+1
    closest_word = ""
    for word in words:
        temp = levenshtein_distance(candidate, word)
        if temp < max_edit_dist :
            max_edit_dist = temp
            closest_word = word
    return closest_word, max_edit_dist


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
DP Top Down Implementation ($\mathcal{O}(nc)$)
Can you write Python code for it?

def coin_change_top_down(n, coins):
    change = [0 for x in range(n + 1)]
    sol = [0 for x in range(n + 1)]

    def coin_change_helper(n, coins):
        # n is 0, basecase. We need no coins
        if n == 0:
            return 0, change

        # We have already computed the solution, just return it
        if sol[n] != 0:
            return sol[n], change

        # Try every possible coin and recursively compute the best solution
        for c in reversed(coins):
            if n - c >= 0:
                sol_t, _ = coin_change_helper(n - c, coins)
                sol_t = sol_t + 1
                if sol_t < sol[n] or sol[n] == 0:
                    sol[n] = sol_t
                    change[n] = c

        return sol[n], change

    return coin_change_helper(n, coins)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
From the chart above we can see how both the top down and the bottom up dp approaches are linear (given a fixed $c$, as it is in our case). Moreover, we can see that the greedy approach is constant in time, and therefore much more efficient.
However, even though the two dp approaches have the same complexity, from the chart it is easy to notice how the bottom up approach is faster in practice. This is not a contradiction, since it simply means that they have different coefficients in their linearity.
Analysis between two Top Down implementations
Can you write Python code for it?

data = range(1,4000,1)

import sys

sys.setrecursionlimit(23000)

dp_top_down_res = []
dp_top_down_incr_res = []

for i in data:
    dp_top_down_res.append(time_f(lambda: coin_change_top_down(i, UK_coin_set)[0]))
    dp_top_down_incr_res.append(time_f(lambda: coin_change_top_down_increasing_ord(i, UK_coin_set)[0]))

%matplotlib inline
from matplotlib import pyplot as plt

top_down = plt.scatter(data, dp_top_down_res, c='red')
top_down_incr = plt.scatter(data, dp_top_down_incr_res, c='blue')

plt.xlabel('n')
plt.ylabel('time (/s)')
plt.xlim(0)
plt.ylim(0)

plt.legend((top_down, top_down_incr),
           ('Top Down', 'Top Down Increasing Order'))


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
From the chart above we can see the exponential nature of the naive solution with respect to the top down dp approach.
Analysis with big input numbers (naive excluded)
Can you write Python code for it?

data = range(1,20000,100)

dp_top_down_res = []
dp_bottom_up_res = []
greedy_res = []

for i in data:
    dp_top_down_res.append(time_f(lambda: coin_change_top_down(i, UK_coin_set)))
    dp_bottom_up_res.append(time_f(lambda: coin_change_bottom_up(i, UK_coin_set)))
    greedy_res.append(time_f(lambda: coin_change_greedy(i, UK_coin_set)))

%matplotlib inline
from matplotlib import pyplot as plt

top_down = plt.scatter(data, dp_top_down_res, c='red')
bottom_up = plt.scatter(data, dp_bottom_up_res, c='blue')
greedy = plt.scatter(data, greedy_res, c='green')

plt.xlabel('n')
plt.ylabel('time (/s)')
plt.xlim(0)
plt.ylim(0)

plt.legend((top_down, bottom_up, greedy),
           ('Top Down', 'Bottom Up', 'Greedy'))


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
General implementation (can choose which specific one to call)
Can you write Python code for it?

def coin_change(n, coins):
    return coin_change_top_down(n, coins)

def print_change(n,coins):
    counts, change = coin_change_naive(n,coins)
    while n > 0:
        print(change[n])
        n = n - change[n]

UK_coin_set = [1,2,5,10,20,50,100,200,500]
print_change(25,UK_coin_set)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
Run the below cell to test your implementation
Can you write Python code for it?

# A one liner that queries closest_match and then prints the result
print_closest = lambda w, wl: print('{}: {} ({})'.format(w, *closest_match(w, wl)))

print_closest('zilophone', wordlist)
print_closest('inconsidrable', wordlist)
print_closest('bisamfiguatd', wordlist)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
Test all efficient solutions (DP and greedy) against themselves
Can you write Python code for it?

test_coin_set = [1,2,5,10,20,50,100,200,500]
test_data = [i for i in range(1200)]

for t in test_data:
    assert coin_change_bottom_up(t, test_coin_set)[0] == coin_change(t, test_coin_set)[0]
    assert coin_change_bottom_up(t, test_coin_set)[0] == coin_change_greedy(t, test_coin_set)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
Test all solutions against the naive solution, therefore with small input numbers
Can you write Python code for it?

test_coin_set = [1,2,5,10,20,50,100,200,500]
test_data = [i for i in range(22)]

for t in test_data:
    assert coin_change_naive(t, test_coin_set)[0] == coin_change(t, test_coin_set)[0]
    assert coin_change_naive(t, test_coin_set)[0] == coin_change_bottom_up(t, test_coin_set)[0]
    assert coin_change_bottom_up(t, test_coin_set)[0] == coin_change_greedy(t, test_coin_set)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
Top Down Implementation that tries coins in increasing order ($\mathcal{O}(nc)$)
Can you write Python code for it?

def coin_change_top_down_increasing_ord(n, coins):
    change = [0 for x in range(n + 1)]
    sol = [0 for x in range(n + 1)]

    def coin_change_helper(n, coins):
        # n is 0, basecase. We need no coins
        if n == 0:
            return 0, change

        # We have already computed the solution, just return it
        if sol[n] != 0:
            return sol[n], change

        # Try every possible coin and recursively compute the best solution
        for c in coins:
            if n - c >= 0:
                sol_t, _ = coin_change_helper(n - c, coins)
                sol_t = sol_t + 1
                if sol_t < sol[n] or sol[n] == 0:
                    sol[n] = sol_t
                    change[n] = c

        return sol[n], change

    return coin_change_helper(n, coins)


I am working on the file "DynamicProgramming/Dynamic Programming.ipynb".
The first task is:
Use this test to confirm your implementation is correct.
Can you write Python code for it?

print(levenshtein_distance('sunny', 'snowy') == 3)
print(levenshtein_distance('algorithm', 'altruistic') == 6)
print(levenshtein_distance('imperial', 'empirical') == 3)
print(levenshtein_distance('weird', 'wired') == 2)


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
From the information criteria outlined above for lag order selection, it seems that the most prevalent lag order is 5. Hence, we proceed with the VAR Model with this specification.
Initial Results
Can you write Python code for it?

# Fit VAR model with 5 lags (simpler model of those identified)
result = var.fit(5)

print 'Results Summary of the VAR model: '
result.summary()

# Forecasting
lag = result.k_ar
fc = result.forecast(df.values[-lag:], 36)

# Impulse Response Function
irf = result.irf(36)
irf.plot()

# Cummulative Impulse Response Function
irf.plot_cum_effects(orth=False)


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
In contrast to the simple ARIMA and ARIMA + SWN, this model does worse in predicting the last 12 months of CCI data.
2) Dynamic VAR - CCI + SWN + LDA Topics:
We now include the LDA topics into the VAR setting. As was discussed in previous Notebooks, LDA topics are not only non-stationary but they behave with a strong seasonal component. For this reason, we include the 12 month differences of the LDA topics into our VAR model.
Can you write Python code for it?

# Load datasets
top = np.load('topicsByMonthBigrams8.npy')
top = pd.DataFrame(data = top, index = sen.index)
top.columns = ['T1','T2','T3','T4','T5','T6','T7','T8']

top1 = np.load('topicsByMonthBigrams8.npy')
top1 = pd.DataFrame(data = top1, index = sen.index)
top1.columns = ['T1','T2','T3','T4','T5','T6','T7','T8']
top1 = top1 - top1.shift(12)
top1.dropna(inplace=True)

top2 = np.load('topicsByMonthBigrams8.npy')
top2 = pd.DataFrame(data = top2, index = sen.index)
top2.columns = ['T1','T2','T3','T4','T5','T6','T7','T8']

for i in ['T1','T2','T3','T4','T5','T6','T7','T8']:
    temp1 = tsa.seasonal_decompose(top2[i].values, freq = 12)
    top2[i] = temp1.trend + temp1.resid
top2.dropna(inplace=True)

dfs = pd.concat([cci, sen], axis = 1)
dfs = dfs - dfs.shift(1)
dfs = pd.concat([dfs, top1], axis = 1)
dfs.dropna(inplace=True)
dfs.columns = ['CCI', 'SEN', 'T1','T2','T3','T4','T5','T6','T7','T8']


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
Lag Order Specification
Can you write Python code for it?

# Define application of VAR to relevant dataset
var = tsa.VAR(dfs)

# Select lag order by information criteria
var.select_order(12)


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
Lag order specification
Can you write Python code for it?

# Define application of VAR to relevant dataset
var = tsa.VAR(dfs)

# Select lag order by information criteria
var.select_order(10)


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
Surprisingly, we cannot reject the null hypothesis that the SWN series for the full month follows a random walk, while we were able to confidently do so with the data for the first 15 days. For this reason, we will work with both series in one-month differences for our Dynamic VAR models.
Can you write Python code for it?


I am working on the file "DynamicVAR_Final.ipynb".
The first task is:
Test stationarity of CCI and SWN
Can you write Python code for it?

def test_stationarity(timeseries):

    #Determing rolling statistics
    rolmean = pd.rolling_mean(timeseries, window=12)
    rolstd = pd.rolling_std(timeseries, window=12)

    #Plot rolling statistics:
    fig, ax = plt.subplots(1,1, figsize=(20, 5))
    ax = plt.plot(timeseries, color='blue',label='Original')
    ax = plt.plot(rolmean, color='red', label='Rolling Mean')
    #ax = plt.plot(rolstd, color='black', label = 'Rolling Std')
    #ax = plt.plot(timeseries-rolmean, color='green', label = 'Noise')
    ax = plt.legend(loc='best')
    ax = plt.title('CCI and its Rolling Mean')
    plt.show(block=False)

    #Perform Dickey-Fuller test:
    for j in ['nc', 'c', 'ct']:
        print ''
        print 'Results of Dickey-Fuller Test (Reg. {}):'.format(j)
        dftest = tsa.adfuller(timeseries, autolag='AIC', regression = j)
        dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])
        for key,value in dftest[4].items():
            dfoutput['Critical Value (%s)'%key] = value
        print dfoutput

def is_unit_root(data, threshold="5%"):

    nc = tsa.adfuller(data, regression="nc")
    c = tsa.adfuller(data, regression="c")
    ct = tsa.adfuller(data, regression="ct")

    votes = 0
    for test in [nc, c, ct]:
        if(abs(test[0]) < abs(test[4][threshold])):
            votes += 1

    return votes >= 2, {"nc": nc, "c":c, "ct":ct}

print 'We cannot reject null hypothesis that CCI is Unit Root: '
test_stationarity(df['CCI'])

print ''
print ''
print 'We cannot reject null hypothesis that SWN is Unit Root: '
test_stationarity(df['SEN'])


I am working on the file "Dynamical chaos.ipynb".
The first task is:
Logistic map
Here we have very simple equation:
$$x_{n+1} = k x_n (1 - x_n)$$
where $k$ is some fixed constant.
Can you write Python code for it?

logistic = lambda k, x: k*x*(1-x)

limits = [0,1]
for k in [2.8, 3.2, 3.5, 3.9]:
    l = lambda x: logistic(k, x)
    cobweb_plot(take(iterator(l, x0=0.1), n=30, skip=500), limits, 'plot', boundary(l, limits))

# let's plot the bifurcation diagram

dots = []
for k in linspace(2.5, 4, 0.001):
    for dot in set(take(iterator(lambda x: logistic(k, x), x0=0.5), n=50, skip=500)):
        dots.append((k, dot))

df = pd.DataFrame(dots, columns=('k', 'xs'))
df.plot(x='k', y='xs', kind='scatter', style='.', figsize=(16,12), s=1, xlim=[2.5,4], ylim=[0,1])


I am working on the file "Dynamical chaos.ipynb".
The first task is:
Sawtooth map
Let's examine simple system where each next element is derived by previous element by the following rule:
$$x_{n+1}={2 x_n}$$
where operator ${}$ means taking decimal part of a number.
Can you write Python code for it?

#from math import trunc
assert trunc(1.5) == 1.0
assert trunc(12.59) == 12.0
assert trunc(1) == 1.0

sawtooth = lambda x: round(2*x-trunc(2*x),8)

sawtooth_borders = [pd.DataFrame([(0,0),(0.5,1)]), pd.DataFrame([(0.5,0),(1,1)])]
for x0 in [0.4, 0.41, 0.42, 0.43]:
    cobweb_plot(take(iterator(sawtooth, x0=x0), n=30, skip=500), [0,1], *sawtooth_borders)


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
# Rebinning in Frequency
Can you write Python code for it?

print("The current frequency resolution is {}".format(dynspec.df))


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
All starts with a lightcurve..
Open the event file with astropy.io.fits
Can you write Python code for it?


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Let's create a Lightcurve from the Events time of arrival witha a given time resolution
Can you write Python code for it?

lc = stingray.Lightcurve.make_lightcurve(toa=toa, dt=dt)

lc.plot()


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Let's plot it on top of the dynamic spectrum
Can you write Python code for it?

extent = min(dynspec.time), max(dynspec.time), min(dynspec.freq), max(dynspec.freq)
plt.imshow(dynspec.dyn_ps, origin="lower", aspect="auto", vmin=2.0, vmax=3.0,
           interpolation="none", extent=extent, alpha=0.7)
plt.colorbar()
plt.ylim(740,800)
plt.plot(dynspec.time, dynspec.freq[tracing], color='red', lw=3, alpha=1)
plt.show()


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Let's rebin to a frequency resolution of 2 Hz and using the average of the power
Can you write Python code for it?

dynspec.rebin_frequency(df_new=2.0, method="average")

print("The new frequency resolution is {}".format(dynspec.df))


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Let's rebin to a time resolution of 64 s
Can you write Python code for it?

dynspec.rebin_time(dt_new=64.0, method="average")

print("The new time resolution is {}".format(dynspec.dt))

extent = min(dynspec.time), max(dynspec.time), min(dynspec.freq), max(dynspec.freq)
plt.imshow(dynspec.dyn_ps, origin="lower", aspect="auto", vmin=2.0, vmax=3.0,
           interpolation="none", extent=extent)
plt.colorbar()
plt.ylim(700,850)


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Rebin time
Let's try to improve the visualization by rebinnin our matrix in the time axis
Can you write Python code for it?

print("The current time resolution is {}".format(dynspec.dt))


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
The collumn TIME of the first extension stores the time of each event
Can you write Python code for it?


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
The dyn_ps attribute stores the power matrix, each column corresponds to the powerspectrum of each segment of the light curve
Can you write Python code for it?


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
This is how the trace function looks like
Can you write Python code for it?

plt.plot(dynspec.time, dynspec.freq[tracing], color='red', alpha=1)
plt.show()


I am working on the file "DynamicalPowerspectrum/DynamicalPowerspectrum_tutorial_[real_data].ipynb".
The first task is:
Trace maximun
Let's use the method trace_maximum() to find the index of the maximum on each powerspectrum in a certain frequency range. For example, between 755 and 782Hz)
Can you write Python code for it?

tracing = dynspec.trace_maximum(min_freq=755, max_freq=782)


I am working on the file "DynamicalSystems.ipynb".
The first task is:
A random switching system
\begin{eqnarray}
A(0) & = & \left(\begin{array}{cc} 0.444 & -0.3733 \ 0.06 & 0.6000 \end{array}\right) \
B(0) & = & \left(\begin{array}{c} 0.3533 \ 0 \end{array}\right) \
A(1) & = & \left(\begin{array}{cc} -0.8 & -0.1867 \ 0.1371 & 0.8 \end{array}\right) \
B(1) & = & \left(\begin{array}{c} 1.1 \ 0.1 \end{array}\right) \
w & = & 0.2993
\end{eqnarray}
\begin{eqnarray}
c_t & \sim & \mathcal{BE}(c; w) \
x_t & = & A(c_t) x_{t-1} + B(c_t)
\end{eqnarray}
Can you write Python code for it?


I am working on the file "DynamicalSystems.ipynb".
The first task is:
A simple ecological network
Food (Clover): 🍀
Prey (Rabbit): 🐰
Predator (Wolf): 🐺

🐰🍀 $\rightarrow$ 🐰🐰

🐰 $\rightarrow$ 🍀

🐰🐺 $\rightarrow$ 🐺🐺

🐺 $\rightarrow$  🍀

The number of objects in this system are constant
Can you write Python code for it?

#%matplotlib nbagg
%matplotlib inline

import numpy as np
import matplotlib.pylab as plt

A = np.array([[1,0,1],[1,0,0],[1,1,0],[0,1,0]])
B = np.array([[2,0,0],[0,0,1],[0,2,0],[0,0,1]])

#k = np.array([0.02,0.09, 0.001, 0.3])
#x0 = np.array([1000,1000,10000])
k = np.array([0.02,0.19, 0.001, 2.8])
x0 = np.array([1000,1,10000])

X,T = simulate_skm(A,B,k,x0,STEPS=50000)

plt.figure(figsize=(10,5))
plt.plot(T,X[0,:], '.y',ms=2)
plt.plot(T,X[1,:], '.r',ms=2)
plt.plot(T,X[2,:], '.g',ms=2)
plt.legend([u'Rabbit',u'Wolf',u'Clover'])
plt.show()

sm = int(sum(X[:,0]))+1
Hist = np.zeros((sm,sm))
STEPS = X.shape[1]

for i in range(STEPS):
    Hist[int(X[1,i]),int(X[0,i])] = Hist[int(X[1,i]),int(X[0,i])] + 1

plt.figure(figsize=(10,5))
#plt.plot(X[0,:],X[1,:], '.',ms=1)
plt.imshow(Hist,interpolation='nearest')
plt.xlabel('# of Rabbits')
plt.ylabel('# of Wolfs')
plt.gca().invert_yaxis()
#plt.axis('square')
plt.show()

%matplotlib inline
import networkx as nx
import numpy as np
import matplotlib.pylab as plt

# Maximum number of rabbits or wolves
N = 30

k = np.array([0.005,0.06, 0.001, 0.1])

G = nx.DiGraph()
pos = [u for u in product(range(N),range(N))]
idx = [u[0]*N+u[1] for u in pos]

G.add_nodes_from(idx)
edge_colors = []
edges = []
for y,x in product(range(N),range(N)):
    clover = N - (x+y)
    source = (x,y)
    rho = k*np.array([source[0]*clover, source[0], source[0]*source[1], source[1]])
    srho = np.sum(rho)
    if srho==0:
        srho = 1.

    if x<N-1: # Rabbit Birth
        target = (x+1,y)
        edges.append((source[0]*N+source[1], target[0]*N+target[1]))
        edge_colors.append(rho[0]/srho)

    if y<N-1 and x>0: # Consumption
        target = (x-1,y+1)
        edges.append((source[0]*N+source[1], target[0]*N+target[1]))
        edge_colors.append(rho[2]/srho)

#    if y>0: # Wolf Death
#        target = (x,y-1)
#        edges.append((source[0]*N+source[1], target[0]*N+target[1]))
#        edge_colors.append(rho[3]/srho)

#    if x>0: # Rabbit Death
#        target = (x-1,y)
#       ...