I am working on the file "SCGExperiment.ipynb".
The first task is:
We set up the loss on both $p(\xi)$ (here x) and $q(\xi)$ (here z). We then train with Adam with a learning rate of $10^{-3}$.
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=(None, x_dim))
z = tf.random_normal(tf.shape(x))

Lx, _, px, output = propose(x, dynamics, do_mh_step=True)
Lz, _, pz, _ = propose(z, dynamics, do_mh_step=False)

loss = 0.

v1 = (tf.reduce_sum(tf.square(x - Lx), axis=1) * px) + 1e-4
v2 = (tf.reduce_sum(tf.square(z - Lz), axis=1) * pz) + 1e-4
scale = 0.1

loss += scale * (tf.reduce_mean(1.0 / v1) + tf.reduce_mean(1.0 / v2))
loss += (- tf.reduce_mean(v1) - tf.reduce_mean(v2)) / scale

global_step = tf.Variable(0., name='global_step', trainable=False)
learning_rate = tf.train.exponential_decay(1e-3, global_step, 1000, 0.96, staircase=True)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
train_op = optimizer.minimize(loss, global_step=global_step)


I am working on the file "SCRAPE/review_gather.ipynb".
The first task is:
Start with rope and tug
Can you write Python code for it?

def get_cat_link(cat_id):
    try:
        df = DF_CAT
    except NameError:
        df = pd.read_csv("s3://dogfaces/reviews/category.csv")
    link = df[df['cat_id']==cat_id]['link'].values[0]
    page_range = df[df['cat_id']==cat_id]['page range'].values[0]
    return link, page_range

link, page_range = get_cat_link(1)

def get_toys(cat_id):
    link, page_range = get_cat_link(cat_id)
    res = []
    for i in xrange(page_range):
        toys_url = link+'&page={}'.format(i+1)
        r = requests.get(toys_url)
        if r.status_code == 200:
            soup = BeautifulSoup(r.content)
            for item in soup.select("article.product-holder.cw-card.cw-card-hover"):
                num_reviews = int(item.select('p.rating.item-rating')[0].find('span').get_text())
                if num_reviews > 10:
                    toy = {}
                    toy['num_reviews'] = num_reviews
                    raw_id = item.select('a')[0]['href']
                    toy['toy_link'] = "https://www.chewy.com"+item.select('a')[0]['href']
                    toy['toy_id'] = raw_id.split('/')[-1]
                    toy['toy_name'] = raw_id.split('/')[1]
                    toy['picture_link'] = "https:" + item.select('img')[0]['src']
                    toy['price'] = item.select('p.price')[0].get_text().split()[0]
                    res.append(toy)
    return res

temp = get_toys(3)
df_test = pd.DataFrame.from_dict(temp)

print df_test['toy_link'][10]

from list_toys import *

get_toys(4)

print url


I am working on the file "SCRAPE/review_gather_1.ipynb".
The first task is:
Data base structures at chewy.com
categories:
|cat id|category | link| page range|
|:-|:--------|:---:|:----------|
|1|Balls and Fetch Toys| https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A317 | 19|
|2|Chew Toys|https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A316 | 15|
|3|Plush Toys|https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A320 | 17|
|4|Interactive Toys|https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A319 |8|
|5|Rope and Tug | https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A321 |9|
for chewy.com the toy page is: catepage + '&page=?'
for each toy, build a toy table:
toys:
|toyid |name | link| catid | picture_link|price |reviews|review_link|review_page_num|
|:-----|:----|:----|:------|:------------|:-----|:------|:----------|:--------------|
|114660|usa-bones-chews-cotton-rope-bones|https://www.chewy.com/usa-bones-chews-cotton-rope-bones/dp/114660 |5| https://img.chewy.com/is/catalog/86885_MAIN.AC_SL400_V1477926797.jpg |5.69|568|/usa-bones-chews-cotton-rope-bones/product-reviews/114660?reviewSort=NEWEST&reviewFilter=ALL_STARS&pageNumber=1|58|
also build a review table:
|reviewid|toyid|toy_name | user_name| stars| review_text| time| image|image_id| useful|
|:-------|:----|:--------|:---------|:------|:-----------|:----|:---------|:------|:---|
|9809823|114660|usa-bones-chews-cotton-rope-bones|Steffy|5|We have...|Sep2, 2017|http://chewy.ugc.bazaarvoice.com/0090-en_us/2540529/photo.jpg |0090-en_us_2540529| 0 |
also build a image table:
|image_id|image_link|image_name|
|:-------|:---------|:---------|
|0090-en_us_254052|http://chewy.ugc.bazaarvoice.com/0090-en_us/2540529/photo.jpg | 0090-en_us_254052.jpg|
Can you write Python code for it?

# create the category data frame
cat_id = [1,2,3,4,5]
category = ['Balls and Fetch Toys','Chew Toys','Plush Toys','Interactive Toys','Rope and Tug']
link = ['https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A317','https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A316',
       'https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A320','https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A319',
       'https://www.chewy.com/s?rh=c%3A288%2Cc%3A315%2Cc%3A321']
pagerange = [19,15,17,8,9]
df_cat = pd.DataFrame({'cat_id':cat_id,'category':category,'link':link,'page range':pagerange})

df_data = df_cat.to_csv(index=False)
s3_res = boto3.resource('s3')
s3_res.Bucket('dogfaces').put_object(Key='reviews/category.csv', Body=df_data)

df_cat = pd.read_csv("s3://dogfaces/reviews/category.csv")
df_cat.head()


I am working on the file "SCRAPE/review_gather_1.ipynb".
The first task is:
Scrape reviews and images
Can you write Python code for it?

df_toys = pd.read_csv("s3://dogfaces/reviews/toys.csv")

df_toys.head()

test_url = df_toys['toy_link'][0]
review_url = test_url+'?'+'reviewSort=NEWEST&reviewFilter=ALL_STARS&pageNumber=1'
r = requests.get(test_url)

#|reviewid|toyid|toy_name | user_name| starts| review_text| time| image|image_id| useful|
num_reviews =

review_url

new_url = 'https://www.chewy.com/kong-airdog-squeakair-ball-dog-toy/product-reviews/47728?reviewSort=NEWEST&reviewFilter=ALL_STARS&pageNumber=3'
new_url

test_url.split('/')


I am working on the file "SCRAPE/review_gather_1.ipynb".
The first task is:
Start with rope and tug
Can you write Python code for it?

def get_cat_link(cat_id):
    try:
        df = DF_CAT
    except NameError:
        df = pd.read_csv("s3://dogfaces/reviews/category.csv")
    link = df[df['cat_id']==cat_id]['link'].values[0]
    page_range = df[df['cat_id']==cat_id]['page range'].values[0]
    return link, page_range

link, page_range = get_cat_link(1)

def get_toys(cat_id):
    link, page_range = get_cat_link(cat_id)
    res = []
    for i in xrange(page_range):
        toys_url = link+'&page={}'.format(i+1)
        r = requests.get(toys_url)
        if r.status_code == 200:
            soup = BeautifulSoup(r.content)
            for item in soup.select("article.product-holder.cw-card.cw-card-hover"):
                num_reviews = int(item.select('p.rating.item-rating')[0].find('span').get_text())
                if num_reviews > 10:
                    toy = {}
                    toy['num_reviews'] = num_reviews
                    raw_id = item.select('a')[0]['href']
                    toy['toy_link'] = "https://www.chewy.com"+item.select('a')[0]['href']
                    toy['toy_id'] = raw_id.split('/')[-1]
                    toy['toy_name'] = raw_id.split('/')[1]
                    toy['picture_link'] = "https:" + item.select('img')[0]['src']
                    toy['price'] = item.select('p.price')[0].get_text().split()[0]
                    res.append(toy)
    return res

temp = get_toys(3)
df_test = pd.DataFrame.from_dict(temp)

print df_test['toy_link'][10]


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Hand follow.ipynb".
The first task is:
Following the left hand
To follow the left hand, the script will do the following steps :
* Find the 3D position of the left hand, with Forward Kinematics
* Assign this position ( + a gap to avoid collision) as the target of the right hand
* Tell the right hand to reach this target
That's exactly what we do in the hand_follow function :
Can you write Python code for it?

def follow_hand(poppy, delta):
    """Tell the right hand to follow the left hand"""
    right_arm_position = poppy.l_arm_chain.end_effector + delta
    poppy.r_arm_chain.goto(right_arm_position, 0.5, wait=True)


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Hand follow.ipynb".
The first task is:
Initialize your robot positions to 0 :
Can you write Python code for it?

for m in poppy.motors:
    m.goto_position(0, 2)


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Hand follow.ipynb".
The first task is:
Then, create your Pypot robot :
Can you write Python code for it?


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Inverse Kinematics with ErgoJr.ipynb".
The first task is:
First, instantiate your Creature
Can you write Python code for it?

from poppy.creatures import PoppyErgoJr

poppy = PoppyErgoJr(simulator='vrep')


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Inverse Kinematics with ErgoJr.ipynb".
The first task is:
Forward Kinematics
With Forward Kinematics, you can access the 3D position of your robot, from the angles of the motors :

For the Ergo Jr, you will get the 3D position of the tip (the blue dot on the picture above):

To access the Forwards Kinematics of the Ergo Jr :
Can you write Python code for it?


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Inverse Kinematics with ErgoJr.ipynb".
The first task is:
Inverse Kinematics
More than only accessing the position of the tip of the ErgoJr, you can use Inverse Kinematics to set this position.
Simply, you give a desired position in space, and you will get a motor configuration that will move the robot to the target!

The goto function just does this :
Can you write Python code for it?

poppy.chain.goto([-0.00670386, -0.17439168,  0.15083097],2.)

from ipywidgets import interact, FloatSlider

poppy.reset_simulation()

c = poppy.chain

x, y, z = c.end_effector
size = 0.3

def goto(x, y, z):
    c.goto((x, y, z), .1)

interact(goto,
         x=FloatSlider(min=x-size, max=x+size, value=x, step=0.01),
         y=FloatSlider(min=y-size, max=y+size, value=y, step=0.01),
         z=FloatSlider(min=z-size, max=z+size, value=z, step=0.01))


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
And their respective motors:
Can you write Python code for it?

[m.name for m in poppy.l_arm_chain.motors]


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
Drawing a circle
This is particularly useful when you want to directly control your robot in the cartesian space. For instance, it become rather easy to make the hand of the robot follow a circle: (two times here!)
Can you write Python code for it?

import numpy as np
r = .13
x0, y0, z0 = (0.2, -0.2, 0.2)

poppy.l_arm_chain.goto((x0, y0, z0), 1., wait=True)

for alpha in np.arange(0, 4*np.pi, .08):
    x = r * np.cos(alpha) + x0
    z = r * np.sin(alpha) + z0

    poppy.l_arm_chain.goto((x, y0, z), 0.03, wait=True)


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
Forward kinematics
You can directly retrieve the current cartesian position of the end effector of a chain. For instance, assuming your robot is still in the rest positions:
Can you write Python code for it?


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
Note the use of the convert_to_ik_angles method which converts from pypot representation (using degrees, motor offset and orientation) to IKPy internal representation.
Inverse kinematics
You can also directly ask the robot, more precisely one of its kynematic chain, to reach a specific cartesian position. The IKPy library will use an optimization technique to try to find the configuration of the joints that best match your target position.
For instance, if you want to move the robot left hand in front of its head, you can ask it to go the position [0, -0.15, 0.35]:
Can you write Python code for it?


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
Plotting
Thanks to the IKPy library, you can also directly plot a kinematic chain configuration using matplotlib.
For instance, you can plot the current position of your robot :
Can you write Python code for it?

import matplotlib
from mpl_toolkits.mplot3d import Axes3D
%matplotlib notebook
zero = [0] * 7

ax = matplotlib.pyplot.figure().add_subplot(111, projection='3d')
ax.scatter([0], [0],[0])

poppy.l_arm_chain.plot(poppy.l_arm_chain.convert_to_ik_angles(poppy.l_arm_chain.joints_position), ax, target = (0.2, -0.2, 0.2))
poppy.r_arm_chain.plot(poppy.r_arm_chain.convert_to_ik_angles(poppy.r_arm_chain.joints_position), ax)


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Moving the Poppy Torso using Inverse Kinematics.ipynb".
The first task is:
The optimizer used by IKPy can be tweaked depending on what you want to do. This is beyond the scope of this tutorial, please refer directly to IKPy documentation for details.
Finally, thanks to widgets you can also easily design a user interface for directly controlling your robot's hand!
Can you write Python code for it?

from ipywidgets import interact, FloatSlider

c = poppy.l_arm_chain

x, y, z = c.end_effector
size = 0.3

def goto(x, y, z):
    c.goto((x, y, z), .1)

interact(goto,
         x=FloatSlider(min=x-size, max=x+size, value=x, step=0.01),
         y=FloatSlider(min=y-size, max=y+size, value=y, step=0.01),
         z=FloatSlider(min=z-size, max=z+size, value=z, step=0.01))


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Quickstart.ipynb".
The first task is:
Note : as mentioned before, here we use a file in the resource folder.
Inverse kinematics
In Inverse Kinematics, you want your kinematic chain to reach a 3D position in space.
To have a more general representation of position, IKPy works with homogeneous coordinates. It is a 4x4 matrix storing both position and orientation.
Prepare your desired position as a 4x4 matrix. Here we only consider position, not orientation of the chain.
Can you write Python code for it?

target_vector = [ 0.1, -0.2, 0.1]
target_frame = np.eye(4)
target_frame[:3, 3] = target_vector

print("The angles of each joints are : ", my_chain.inverse_kinematics(target_frame))


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Quickstart.ipynb".
The first task is:
Plotting
And finally plot the result :
(If the code below doesn't work, comment the %maplotlib notebook line, and uncomment the %matplotlib inline line)
Can you write Python code for it?

# If there is a matplotlib error, uncomment the next line, and comment the line below it.
# %matplotlib inline
%matplotlib notebook
import matplotlib.pyplot as plt
ax = plot_utils.init_3d_figure()
my_chain.plot(my_chain.inverse_kinematics(target_frame), ax, target=target_vector)
plt.xlim(-0.1, 0.1)
plt.ylim(-0.1, 0.1)


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Quickstart.ipynb".
The first task is:
The basic element of IKPy is the kinematic Chain.
To create a chain from an URDF file :
Can you write Python code for it?


I am working on the file "SCS_Documents/ikpy-master/tutorials/ikpy/Quickstart.ipynb".
The first task is:
You can check that the Inverse Kinematics is correct by comparing with the original position vector :
Can you write Python code for it?

real_frame = my_chain.forward_kinematics(my_chain.inverse_kinematics(target_frame))
print("Computed position vector : %s, original position vector : %s" % (real_frame[:3, 3], target_frame[:3, 3]))


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
Alternatively we can use a derivative-free version of Milsteins method as a two-stage kind-of Runge-Kutta method, documented in wikipedia (https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_method_%28SDE%29) or the original in arxiv.org https://arxiv.org/pdf/1210.0933.pdf.
Can you write Python code for it?

q0 = 0
v0 = 0
X = np.zeros([len(tArray), 2])
X[0, 0] = q0
X[0, 1] = v0

def a(t, X):
    q, v = X
    return np.array([v, -(Gamma0 - Omega0*eta*q**2)*v - Omega0**2*q])

def b(t, X):
    q, v = X
    return np.array([0, np.sqrt(2*Gamma0*k_b*T_0/m)])

%%timeit

S = np.array([-1,1])
for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    K1 = a(t, X[n])*dt + b(t, X[n])*(dw - S*np.sqrt(dt))
    Xh = X[n] + K1
    K2 = a(t, Xh)*dt + b(t, Xh)*(dw + S*np.sqrt(dt))
    X[n+1] = X[n] + 0.5 * (K1+K2)

q = X[:, 0]
v = X[:, 1]

plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")

plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
Apply Milstein's method (Euler Maruyama if $b'(Y_{n}) = 0$ as is the case here):
recursively define $Y_{n}$ for $ 1\leq n\leq N $ by
$ Y_{{n+1}}=Y_{n}+a(Y_{n})\Delta t+b(Y_{n})\Delta W_{n}+{\frac  {1}{2}}b(Y_{n})b'(Y_{n})\left((\Delta W_{n})^{2}-\Delta t\right)$
Perform this for the 2 first order differential equations:
Can you write Python code for it?

#%%timeit

for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    v[n+1] = v[n] + a_v(t, v[n], q[n])*dt + b_v(t, v[n], q[n])*dw + 0
    q[n+1] = q[n] + a_q(t, v[n], q[n])*dt + 0


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
Generate independent and identically distributed normal random variables with expected value 0 and variance dt
Can you write Python code for it?

np.random.seed(88)

dwArray = np.random.normal(0, np.sqrt(dt), len(tArray)) # independent and identically distributed normal random variables with expected value 0 and variance dt


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
The form of $F_{feedback}(t)$ is still questionable
On page 49 of 'Dynamics of optically levitated nanoparticles in high vacuum' - Thesis by Jan Gieseler he uses the form: $F_{feedback}(t) = \Omega_0 \eta q^2 \dot{q}$
On page 2 of 'Parametric feeedback cooling of levitated optomechancs in a parabolic mirror trap' Paper by Jamie and Muddassar they use the form: $F_{feedback}(t) = \dfrac{\Omega_0 \eta q^2 \dot{q}}{q_0^2}$ where $q_0$ is the amplitude of the motion: $q(t) = q_0(sin(\omega_0t)$
However it always shows up as a term $\delta \Gamma$ like so:
$\dfrac{d^2q(t)}{dt^2} + (\Gamma_0 + \delta \Gamma)\dfrac{dq(t)}{dt} + \Omega_0^2 q(t) - \sqrt{\dfrac{2\Gamma_0 k_B T_0}{m}} \dfrac{dW(t)}{dt} = 0$
By fitting to data we extract the following 3 parameters:
1) $A = \gamma^2 \dfrac{k_B T_0}{\pi m}\Gamma_0 $
Where:

$\gamma$ is the conversion factor between Volts and nanometres. This parameterises the amount of light/ number of photons collected from the nanoparticle. With unchanged allignment and the same particle this should remain constant with changes in pressure.
$m$ is the mass of the particle, a constant
$T_0$ is the temperature of the environment
$\Gamma_0$ the damping due to the environment only

2) $\Omega_0$ - the natural frequency at this trapping power
3) $\Gamma$ - the total damping on the system including environment and feedback etc...
By taking a reference save with no cooling we have $\Gamma = \Gamma_0$ and therefore we can extract $A' = \gamma^2 \dfrac{k_B T_0}{\pi m}$. Since $A'$ should be constant with pressure we can therefore extract $\Gamma_0$ at any pressure (if we have a reference save and therefore a value of $A'$) and therefore can extract $\delta \Gamma$, the damping due to cooling, we can then plug this into our SDE instead in order to include cooling in the SDE model.
For any dataset at any pressure we can do:
$\Gamma_0 = \dfrac{A}{A'}$
And then $\delta \Gamma = \Gamma - \Gamma_0$
Using this form and the same derivation as above we arrive at the following form of the 2 1st order...
Can you write Python code for it?

def a_q(t, v, q):
    return v

def a_v(t, v, q):
    return -(Gamma0 + deltaGamma)*v - Omega0**2*q

def b_v(t, v, q):
    return np.sqrt(2*Gamma0*k_b*T_0/m)


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
Using values obtained from fitting to data from a real particle we set the following constant values describing the system. Cooling has been assumed to be off by setting $\eta = 0$.
Can you write Python code for it?

Gamma0 = 4000 # radians/second
Omega0 = 75e3*2*np.pi # radians/second
eta = 0.5e7
T_0 = 300 # K
k_b = scipy.constants.Boltzmann # J/K
m = 3.1e-19 # KG


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
We now have an array of positions, $v$, and velocities $p$ with time $t$.
Can you write Python code for it?

plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")

plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
partition the interval [0, T] into N equal subintervals of width $\Delta t>0$:
$ 0=\tau {0}<\tau {1}<\dots <\tau {N}=T{\text{ with }}\tau {n}:=n\Delta t{\text{ and }}\Delta t={\frac {T}{N}}$
Can you write Python code for it?

dt = 1e-10
tArray = np.arange(0, 100e-6, dt)

print("{} Hz".format(1/dt))


I am working on the file "SDE_Solution_Derivation.ipynb".
The first task is:
values below are taken from a ~1e-2 mbar cooled save
Can you write Python code for it?

Gamma0 = 15 # radians/second
deltaGamma = 2200
Omega0 = 75e3*2*np.pi # radians/second
eta = 0.5e7
T_0 = 300 # K
k_b = scipy.constants.Boltzmann # J/K
m = 3.1e-19 # KG

dt = 1e-10
tArray = np.arange(0, 100e-6, dt)

q0 = 0
v0 = 0
q = np.zeros_like(tArray)
v = np.zeros_like(tArray)
q[0] = q0
v[0] = v0

np.random.seed(88)

dwArray = np.random.normal(0, np.sqrt(dt), len(tArray)) # independent and identically distributed normal random variables with expected value 0 and variance dt

for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    v[n+1] = v[n] + a_v(t, v[n], q[n])*dt + b_v(t, v[n], q[n])*dw + 0
    q[n+1] = q[n] + a_q(t, v[n], q[n])*dt + 0

plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")

plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
By swapping the axes of the array, I can put all the index-sharing values in the same inner array. For example, the first value of each of the scans are now in the first inner array and the second values of all the scans are in the second inner array. Averaging the values in each of those arrays would now be a simple task.
We now have a single array of values ripe for plotting.
Spectrum Improvement and Plotting
Let's see a visual of our plot of the average power values against our frequency band.
Can you write Python code for it?

plt.figure(figsize=(15,5))
plt.title('Time-Averaged Radio Spectrum over 90 Scans')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.plot(TotalBand, avg_data)
plt.show()


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
FMstations is a separate .py file storing a dictionary to be used in a later function. While dictionaries can be included in the executing script without functional issues, their potential large size warrants storage in a separate file.
stationsdata.csv is the file containing the raw data, an array of lines. Each line is an array itself, consisting of a number of power values corresponding to the sampling rate. Specifically for my experiment, the sampling rate is 2.5 MHz and each eight lines constitute a scan of the bandwidth of interest, the 20 MHz band between 87.9 and 107.9 MHz. U.S. radio stations are allocated broadcast frequencies in that range, and these values should be changed according to the experiment at hand. Data is collected every 10 seconds for every 610.35 (raw_data[0][4]) frequency difference in that range over a total of 900 seconds.
The raw data file consists of 720 lines because the scan is performed 90 times, the total number of seconds divided by the scan interval. Each line unfortunately comes with an initial six outputs, such the date of when the data was taken, that we have no interest of for purposes of data processing and plotting. Those outputs would have to be truncated.
Processing Raw Data
First, we have to idenfity the frequency range we are interested in nd have it be consistent with the type and number of data in the file for plotting. There are DATAPOINTS number of data (32776 for my experiment) for each scan, with each line having SAMP_RATE / FREQ_BIN number of data (4107).
Can you write Python code for it?

def total_freq(samp_rate, line):
    """total frequency band of the data

    samp_rate: sample rate used to collect the data
    freq_bin: difference in frequency between each data point
    line: number of iterations to create the band;
    each sampling may sample over only a part of the desired bandwidth
    """
    if samp_rate < (MAXFREQ - MINFREQ):
        allfreq = []
        i = 0
        while i < line:
            freqband = np.arange(MINFREQ + (samp_rate * i), MINFREQ + (samp_rate * (i+1)), FREQ_BIN)
            allfreq = np.append(allfreq, freqband)
            i += 1
        return allfreq
    else:
        return np.arange(MINFREQ, MAXFREQ, FREQ_BIN)

TotalBand = total_freq(raw_data[0][3] - raw_data[0][2], 8)


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
THe function iterates over all the values in the spectrum and stores any frequency (and its associated index) of the peaks (values above a certain threshold that's a local maxima) in arrays. The indices will be useful for corresponding the frequencies with their peak location. The majority of the function is ordering the frequencies in descending order of the peak intensity by iterating over the array of peaks and determining each's corresponding peak value. The ordered list is then rounded to one decimal and is in units of MHz or in a desired unit as determined by xreducer.
With a list of peaks, we can use them to physically mark their sources on our plot, provided a dictionary of the sources.
Can you write Python code for it?

def mark_peaks(src_dict, spec, threshold, line, title, xreducer, error=.01, bound1='left', bound2='bottom', rot=90):
    """returns both a plot and a dictionary

    plot: shows the stations next to the marked peaks
    dictionary: matches the relevant peak frequencies with the corresponding station(s)

    src_dict: input dictionary of frequencies and stations from which the results are selected from
    spec: input spectrum data
    threshold: only data above which are taken in account to ignore the noise level
    title: title for the plot
    xreducer: the values of the x-axis divided by which to simpler units
    error: within which the obtained frequencies are acceptable as equivalent to that of a station
    remaining parameters: used the adjust the markings and labels of the plots
    """
    stations = []
    peakfreqs = []
    stations_i = []
    peaker = get_peaks(spec, threshold, xreducer)
    p0 = peaker[0]
    p1 = peaker[1]
    for i in np.arange(len(p1)):
        if p1[i] in src_dict.keys():
            stations.append(src_dict[p1[i]])
            peakfreqs.append(p1[i])
            stations_i.append(p0[i])
        else:
            for x in np.arange(p1[i]-error, p1[i]+error, error):
                if x in src_dict.keys():
                    stations.append(src_dict[x])
                    peakfreqs.append(p1[i])
                    stations_i.append(p0[i])
                else:
                    continue
    peaks = [spec[y] for y in stations_i]
    plt.figure(figsize=(15,5))
    plt.title(title)
    plt.xlabel('Frequency (MHz)')
    plt.ylabel('Reduced Power (dBm)')
    yoffset = (np.amax(spec) - np.amin(spec)) / 4
    plt.ylim(np.amin(spec) - yoffset, np.amax(spec) + yoffset)
    plt.plot(np.array(total_freq(line)) / 1000000, spec)
    plt.scatter(peakfreqs, peaks, marker = 'o', color = 'r', s = 40)
    text_bounds = {'ha':bound1, 'va':bound2}
    for i in np.arange(len(peakfreqs)):
        plt.text(peakfreqs[i], peaks[i] + (yoffset / 10), stations[i], text_bounds,...


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
The code evaluates based on what type of smoothing we're using. Numpy has a number of built-in smoothing functions. For my plots, I used the kaiser kernel, which is derived from Bessel functions. If flat smoothing is chosen, then our plot is smoothed by a line. The result is a much smoother graph, with the side effect of reduced power at every point.
Can you write Python code for it?

SmooSpec = smooth(ReduSpec, 150, 'kaiser')

plt.figure(figsize=(15,5))
plt.title('Smoothed, Time-Averaged Radio Spectrum')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.plot(TotalBand, SmooSpec)
plt.show()

plt.figure(figsize=(15,5))
plt.title('Second Highest Peak Zoomed-In')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.ylim(-43.5,-42.5)
plt.plot(total_freq(8)[14000:14200]/1000000, SmooSpec[14000:14200],8)
plt.show()


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
The code, though looks nicer, is still pretty rough due to all the noise. This can be simply solved by applying a median averaging across the data; the noisy peaks that correspond to a single frequency would thus be filtered out.
Can you write Python code for it?

def Reduce(spec, n):
    """takes the median of a set of points, removing noise-produced peaks and much noise

    n: half the number of a set of points that the median is taken over
    """
    R = []
    i = 0
    for x in spec:
        if i < n:
            R.append(np.median(spec[0 : i + n + 1]))
            i += 1
        elif i >= 32769:
            R.append(np.median(spec[i - n :]))
            i += 1
        else:
            R.append(np.median(spec[i - n : i + n + 1]))
            i += 1
    return R

ReduSpec = Reduce(FlatSpec, 10)

plt.figure(figsize=(15,5))
plt.title('Noise-Reduced, Time-Averaged Radio Spectrum')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.plot(TotalBand, ReduSpec)
plt.show()


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
There are many ways to go about processing the raw data values to correspond with our frequency range. Since my data came in the form of an array of arrays, that's how I initially kept my data as, though with the fir several unneeded elements truncated from each line. The follow code does just that.
Can you write Python code for it?

def power_total(data, line, trunc):
    """returns an array of arrays; all power values from the scans

    data: input raw data; usually an array of arrays
    line: number of lines of data corresponding to a full scan over the bandwidth
    trunc: truncate number of, if any, of unneeded data, such as time, at the start of each line
    """
    P_tot = []
    j = 0
    while j < SCAN:
        P_band = []
        i = 0
        while i < line:
            P_band = np.append(P_band, data[j*line+i].tolist()[trunc:])
            i += 1
        P_tot.append(P_band)
        j += 1
    return P_tot

total_data = power_total(raw_data, 8, 6)


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
This is seemingly our desired plot, but upon closer inspection...
Can you write Python code for it?

plt.figure(figsize=(15,5))
plt.title('Second Highest Peak Zoomed-In')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.ylim(-43,-42)
plt.plot(TotalBand[14000:14200], ReduSpec[14000:14200])
plt.show()


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
Unlike the raw data, however, each inner array now consists of a full scan of the relevant band (as opposed to each line in the raw data corresponding to only an eighth of the band). For my data, there's now only 90 inner arrays, each a scan at a specific time, down from 720 lines in the raw data.
Having this result is useful for, as an example, plotting multiple graphs to see the time-evolution of the signal. For our immediate purposes, we look at the time-averaged values to produce a single plot.
Can you write Python code for it?

def power_avg(data, xax=0, yax=1):
    """returns the time-average power values of each index-sharing set of numbers

    data: input data; usually an array of arrays
    """
    indices_arr = np.swapaxes(data, xax, yax)
    y = []
    for index in indices_arr:
        y.append(np.average(index))
    return y

avg_data = power_avg(total_data)


I am working on the file "SDR/stations/.ipynb_checkpoints/sdr_stations-checkpoint.ipynb".
The first task is:
We have a functional plot, but it's riddled with issues associated with signal process. We need to flatten the noise floor, remove the noisy peaks, and smooth out the graph so that we can systematically identify the peaks, which correspond to frequencies which signals are received at. Power is in units of dBm, decibels referencing milliwatts. Online resources have more in-depth explanation of the bel unit and power representation in signal processing.
To flatten the noise floor, we have the libery of choosing the agent for our task. We have to divide our plot by a fitted curve or any set of values that can accomplish the task.
Can you write Python code for it?

def Flatten(spec, flatter, n):
    """flattens the noise level of our original plot

    dataSpec: input spectrum to be flatted
    flatter: an array derived from the input spectrum used to divided by the spectrum
    n: half the number of a set of points that the median is taken over
    """
    MED = []
    i = 0
    for x in flatter:
        if i < n:
            MED.append(np.median(flatter[0 : i + n + 1]))
            i += 1
        elif i >= (len(spec) - (n + 1)):
            MED.append(np.median(flatter[i - n :]))
            i += 1
        else:
            MED.append(np.median(flatter[i - n : i + n + 1]))
            i += 1
    Spectrum = np.array(spec) / np.array(MED) * np.average(spec)
    return Spectrum


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
FMstations is a separate .py file storing a dictionary to be used in a later function. While dictionaries can be included in the executing script without functional issues, their potential large size warrants storage in a separate file.
logfile.csv is the file containing the raw data, an array of lines. Each line is an array itself, consisting of a number of power values corresponding to the sampling rate. Specifically for my experiment, the sampling rate is 2.5 MHz and each eight lines constitute a scan of the bandwidth of interest, the 21 MHz band between 87 and 108 MHz. U.S. radio stations are allocated broadcast frequencies in that range, and these values should be changed according to the experiment at hand. Data is collected every 10 seconds for every 610.35 (raw_data[0][4]) frequency difference in that range over a total of 300 seconds.
The raw data file consists of 240 lines because the scan is performed 30 times, the total number of seconds divided by the scan interval. Each line unfortunately comes with an initial six outputs, such the date of when the data was taken, that we have no interest of for purposes of data processing and plotting. Those outputs would have to be truncated.
Processing Raw Data
First, we have to idenfity the frequency range we are interested in nd have it be consistent with the type and number of data in the file for plotting by generating a list of frequencies between our min and max frequencies that correspond to the amount of samples we have for each time step
Can you write Python code for it?

def total_freq(num_samples_per_time):
    """total frequency band of the data

    num_samples_per_time: number of data samples over entire frequency spectrum per timestep
    """
    return np.linspace(MINFREQ, MAXFREQ, num_samples_per_time)

TotalBand = total_freq(len(raw_data[0]) * LINES_PER_TIME)


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
Next block visualizes the data
Can you write Python code for it?


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
THe function iterates over all the values in the spectrum and stores any frequency (and its associated index) of the peaks (values above a certain threshold that's a local maxima) in arrays. The indices will be useful for corresponding the frequencies with their peak location. The majority of the function is ordering the frequencies in descending order of the peak intensity by iterating over the array of peaks and determining each's corresponding peak value. The ordered list is then rounded to one decimal and is in units of MHz or in a desired unit as determined by xreducer.
With a list of peaks, we can use them to physically mark their sources on our plot, provided a dictionary of the sources.
Can you write Python code for it?

def mark_peaks(src_dict, spec, threshold, line, title, xreducer, error=.01, bound1='left', bound2='bottom', rot=90):
    """returns both a plot and a dictionary

    plot: shows the stations next to the marked peaks
    dictionary: matches the relevant peak frequencies with the corresponding station(s)

    src_dict: input dictionary of frequencies and stations from which the results are selected from
    spec: input spectrum data
    threshold: only data above which are taken in account to ignore the noise level
    title: title for the plot
    xreducer: the values of the x-axis divided by which to simpler units
    error: within which the obtained frequencies are acceptable as equivalent to that of a station
    remaining parameters: used the adjust the markings and labels of the plots
    """
    stations = []
    peakfreqs = []
    stations_i = []
    peaker = get_peaks(spec, threshold, xreducer)
    p0 = peaker[0]
    p1 = peaker[1]
    for i in np.arange(len(p1)):
        if p1[i] in src_dict.keys():
            stations.append(src_dict[p1[i]])
            peakfreqs.append(p1[i])
            stations_i.append(p0[i])
        else:
            for x in np.arange(p1[i]-error, p1[i]+error, error):
                if x in src_dict.keys():
                    stations.append(src_dict[x])
                    peakfreqs.append(p1[i])
                    stations_i.append(p0[i])
                else:
                    continue
    peaks = [spec[y] for y in stations_i]
    plt.figure(figsize=(15,5))
    plt.title(title)
    plt.xlabel('Frequency (MHz)')
    plt.ylabel('Reduced Power (dBm)')
    yoffset = (np.amax(spec) - np.amin(spec)) / 4
    plt.ylim(np.amin(spec) - yoffset, np.amax(spec) + yoffset)
    plt.plot(TotalBand / 1000000, spec)
    plt.scatter(peakfreqs, peaks, marker = 'o', color = 'r', s = 40)
    text_bounds = {'ha':bound1, 'va':bound2}
    for i in np.arange(len(peakfreqs)):
        plt.text(peakfreqs[i], peaks[i] + (yoffset / 10), stations[i], text_bounds, rotation=rot)
   ...


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
Unlike the raw data, however, each inner array now consists of a full scan of the relevant band (as opposed to each line in the raw data corresponding to only an eighth of the band). For my data, there's now only 30 inner arrays, each a scan at a specific time, down from 240 lines in the raw data.
Having this result is useful for, as an example, plotting multiple graphs to see the time-evolution of the signal. For our immediate purposes, we look at the time-averaged values to produce a single plot.
We now take the average across time of all these arrays
Can you write Python code for it?


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
We finally have our smoothed plot and can systematically identify characteristics with numpy without dealing with much of the interference due to rough data patterns.
Identifying and Marking Peaks and their Frequencies
We can isolate the peak frequencies with simple methods.
Can you write Python code for it?

def get_peaks(spec, threshold, xreducer, rounder=2):
    """identifies the peaks of a plot. Returns an array of 2 lists:

    1. the indices of the frequencies corresponding to the peaks;
    2. said frequencies, divided by xreducer for simpler units and rounded to rounder decimals

    spec: input data spectrum
    threshold: only data above which are taken into account to ignore the noise level
    """
    Peaks = []
    spec = spec.tolist()
    for i in np.arange(len(spec)):
        if spec[i] > threshold and spec[i] > spec[i-1] and spec[i] > spec[i+1]:
            Peaks.append(spec[i])
        else:
            continue
    Ordered_Indices = []
    while True:
        if np.array(Peaks).tolist() == []:
            Ordered_Freq = [(x * FREQ_BIN + MINFREQ) for x in Ordered_Indices]
            Reduced_Freq = np.around((np.array(Ordered_Freq) / xreducer), rounder)
            return [Ordered_Indices, Reduced_Freq.tolist()]
        elif len(Peaks) == 1:
            Ordered_Indices.append(spec.index(Peaks[0]))
            Peaks = np.delete(Peaks, 0)
        else:
            Ordered_Indices.append(spec.index(np.amax(Peaks)))
            Peaks = np.delete(Peaks, np.array(Peaks).tolist().index(np.amax(Peaks)))


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
We got our final result. This particular approach is useful for identifying nearby signal sources for any purpose. Another approach uses not the average power values, but all the power values from all the scans. Visualizing the time-evolution can be useful for other purposes, such as noting the fluctuations of the peaks' strengths or finding sudden peaks arising at certain moments. The "waterfall" plot displays, in this case, intensity (power) with color codes and has time instead on the y-axis.
Can you write Python code for it?

def waterfall(data, title, axesrange, gridshape='auto'):
    """returns a waterfall grid consisting off all the spectra from the input

    data: an array of arrays
    title: title of grid
    axesrange: boundaries of the values of the grid
    """
    fig = plt.figure(figsize=(15, 5))
    ax = fig.add_subplot(111)
    ax.set_title(title)
    ax.set_xlabel('Frequency (MHz)')
    ax.set_ylabel('Time (s)')
    plt.imshow(data, extent=axesrange)
    ax.set_aspect(gridshape)
    plt.colorbar(orientation='vertical')
    plt.show()

waterfall(total_data, 'Radio Spectra Waterfall', [MINFREQ*1e-6, MAXFREQ*1e-6, 0, TOTAL_TIME])


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
We have a functional plot, but it's riddled with issues associated with signal process. We need to  remove the noisy peaks and smooth out the graph so that we can systematically identify the peaks, which correspond to frequencies which signals are received at. Power is in units of dBm, decibels referencing milliwatts. Online resources have more in-depth explanation of the bel unit and power representation in signal processing.
Can you write Python code for it?

def Reduce(spec, n):
    """takes the median of a set of points, removing noise-produced peaks and much noise

    n: half the number of a set of points that the median is taken over
    """
    R = []
    i = 0
    for x in spec:
        if i < n:
            R.append(np.median(spec[0 : i + n + 1]))
            i += 1
        elif i >= 32769:
            R.append(np.median(spec[i - n :]))
            i += 1
        else:
            R.append(np.median(spec[i - n : i + n + 1]))
            i += 1
    return R

ReduSpec = Reduce(avg_data, 10)

plt.figure(figsize=(15,5))
plt.title('Noise-Reduced, Time-Averaged Radio Spectrum')
plt.xlabel('MHz')
plt.ylabel('Power (dBm)')
plt.plot(TotalBand, ReduSpec)
plt.show()


I am working on the file "SDR/stations/sdr_stations.ipynb".
The first task is:
We now have a single array of values ripe for plotting.
Spectrum Improvement and Plotting
Let's see a visual of our plot of the average power values against our frequency band.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
<span style="color:red">Though I didn't know what these data are, I found the mass of these chemicals are nearly normal distribution!
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Get a plot voc v.s. grouped data
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Get another plot.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Group the data by the standard diviation of the mass.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Make  id  column  the  index.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Now I calculate how many data are wihtin 1 standard diviation, 2 standard diviatioa, and 3 standard diviation respectively.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Read the first 10 rows.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Show the column names.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-1-danielfather7/SEDS-HW1 Tai-Yu Pan.ipynb".
The first task is:
Show the number of elements.
Can you write Python code for it?


I am working on the file "SEDS_Hw/seds-hw-2-procedural-python-part-1-danielfather7/SEDS-HW2.ipynb".
The first task is:
Set 2: analyza data
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import math
alldata['(xi-x)^2'] = (alldata['mass'] - alldata['mass'].mean())**2
SD = math.sqrt(sum(alldata['(xi-x)^2'])/alldata.shape[0])
M = alldata['mass'].mean()
print('standard diviation of mass = ',SD,', mean of mass = ',M,"\n")
alldata['mass_group'] = pd.cut(alldata['mass'],bins=[min(alldata['mass']),M-3*SD,M-2*SD,M-SD,M+SD,M+2*SD,M+3*SD,max(alldata['mass'])],labels=["<(-3SD)","-3SD~-2SD","-2SD~-SD","-SD~+SD","+SD~+2SD","+2SD~+3SD",">(+3SD)"])
count = pd.value_counts(alldata['mass_group'],normalize=True)
print("Count numbers in each group(%)\n",count,"\n")
print("within 1 standard diviation:",count[3],"\nwithin 2 standard diviation:",count[2]+count[3]+count[4],"\nwithin 3 standard diviation:",count[2]+count[3]+count[4]+count[1]+count[5],"\n")
print("Conclusions: mass is nearly normal distribution!")


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
2. corr_rowi_rowj( df , i , j ): It can calculate the Pearson correlation coefficients(P) for selected 2  rows.

df : data frame input
i : the first row you select
j : the second row you select

Note: i and j are rows in a data frame, not index in python.
Can you write Python code for it?

#Try now
"""
def corr_rowi_rowj(df,i,j):

    #Print data first to make it easy to check.

    print("data:")
    print(df,"\n\nP value:")

    #'np.corrcoef(X,Y)' is used to caculate the Pearson correlation coefficient.
     It will yield the matrix:
            P(X,X) P(X,Y)
            P(Y,X) P(Y,Y)
     So I use 'np.corrcoef(X,Y)[0,1]' just to get value P(X,Y) at positon (0,1) in matrix.
     It can also be used if there are three or more arrays, like 'np.corrcoef(X,Y,Z,......)'.
    #'iloc[i-1]'is used to locate the row.
     'i-1' is used because the difference between index (from 0) and actual row is 1.
    #Because the output of rows from 'iloc[i-1]' is a series, I use 'tolist()' to make it bacome an array,
     and then can be used in 'np.corrcoef(X,Y)'.

    P = np.corrcoef(df.iloc[i-1].tolist(), df.iloc[j-1].tolist())[0,1]
    print("P value of row%d and row%d is %.2f." %(i,j,P))
    return P
"""
du.corr_rowi_rowj(df,2,3)
#Note: In this case, row2(row of index 1) and row3 (row of index 2) are compared.


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
3. corr_rowi_vs_all1( df ):
Can you write Python code for it?

def test_corr_rowi_vs_all1():
    df = pd.DataFrame([[-1, 0, 1], [1, 0, -1], [.5, 0, .5]])
    assert type(du.corr_rowi_vs_all1(df).iloc[0,0]) == str, "Diagonal elements not handled properly"
    assert int(du.corr_rowi_vs_all1(df).iloc[0,1]) == -1, "Anticorrelated elements not handled properly"
    assert int(du.corr_rowi_vs_all1(df).iloc[0,2]) == 0, "Uncorrelated elements not handled properly"
    assert int(du.corr_rowi_vs_all1(df).iloc[1,2]) == int(du.corr_rowi_vs_all1(df).iloc[2,1]), "Data not appended properly"
    return

test_corr_rowi_vs_all1()


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
3. corr_rowi_vs_all1( df ): It can calculate the Pearson correlation coefficients(P) for each 2 rows (not including itself) in data frame.

df : data frame input
Can you write Python code for it?

#Try now
"""
def corr_rowi_vs_all1(df):

    #Print data first to make it easy to check.

    print("data:")
    print(df,"\n\nP value:")
    metrix = pd.DataFrame()
    labels = []

    #Use 'iterrows()' to get rows repeatedly.
    #There are two for loops.
     The outer for loop is used to get each row,
     and the inner for loop is to make each row from the outer loop compare to each row(including itself).
    #There is a if loop in the inner for loop to avoid calculating P between 2 same rows
     by checking the index values between 2 rows. I use "P=1" instead of "1" to clearly show it.
    #'np.corrcoef(X,Y)' is used to caculate the Pearson correlation coefficient.
     It will yield the matrix:
            P(X,X) P(X,Y)
            P(Y,X) P(Y,Y)
     So I use 'np.corrcoef(X,Y)[0,1]' just to get value P(X,Y) at positon (0,1) in matrix.
     It can also be used if there are three or more arrays, like 'np.corrcoef(X,Y,Z,......)'.
    #Because the output of rows from 'interrows()' is a series, I use 'tolist()' to make it bacome an array,
     and then can be used in 'np.corrcoef(X,Y)'.
    #I remain the original way I did printing out all the P valuse line by line.
     After Monday class, I appended all Ps into a data frame,
     and then change the labels of index and columns by using a list 'labels',
     and use 'labels.append('row'+str(index+1))' in outer for loop to create labels.

    for index, row in df.iterrows():
        labels.append('row'+str(index+1))
        for index2, row2 in df.iterrows():
            if index==index2:
                metrix.loc[index,index] = "P=1"
            else:
                P = np.corrcoef(row.tolist(), row2.tolist())[0,1]
                print("P value of row%d and row%d is %.2f." %(index+1,index2+1,P))
                metrix.loc[index, index2] = P
    metrix.columns = labels
    metrix.index = labels
    print("\nTables for P values:")
    print(metrix)
    return metrix

"""

du.corr_rowi_vs_all1(df)


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
4. corr_rowi_vs_all2( df , i ):
Can you write Python code for it?

def test_corr_rowi_vs_all2():
    df = pd.DataFrame([[-1, 0, 1], [1, 0, -1], [.5, 0, .5]])
    assert type(du.corr_rowi_vs_all2(df,2).iloc[0,1]) == str, "Diagonal elements not handled properly"
    assert int(du.corr_rowi_vs_all2(df,2).iloc[0,0]) == -1, "Anticorrelated elements not handled properly"
    assert int(du.corr_rowi_vs_all2(df,2).iloc[0,2]) == 0, "Uncorrelated elements not handled properly"
    return

test_corr_rowi_vs_all2()


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
Functions
1. pairwise_correlation( df ): It can calculate the Pearson correlation coefficients(P) for each 2 rows in data frame.

df : data frame input
Can you write Python code for it?

#Try now

"""
def pairwise_correlation(df):

    #Print data first to make it easy to check.

    print("data:")
    print(df,"\n\nP value:")
    metrix = pd.DataFrame()
    labels=[]

    #Use 'iterrows()' to get rows repeatedly.
    #There are two for loops.
     The outer for loop is used to get each row,
     and the inner for loop is to make each row from the outer loop compare to each row(including itself).
    #'np.corrcoef(X,Y)' is used to caculate the Pearson correlation coefficient.
     It will yield the matrix:
            P(X,X) P(X,Y)
            P(Y,X) P(Y,Y)
     So I use 'np.corrcoef(X,Y)[0,1]' just to get value P(X,Y) at positon (0,1) in matrix.
     It can also be used if there are three or more arrays, like 'np.corrcoef(X,Y,Z,......)'.
    #Because the output of rows from 'interrows()' is a series, I use 'tolist()' to make it bacome an array,
     and then can be used in 'np.corrcoef(X,Y)'.
    #I remain the original way I did printing out all the P valuse line by line.
     After Monday class, I appended all Ps into a data frame,
     and then change the labels of index and columns by using a list 'labels',
     and use 'labels.append('row'+str(index+1))' in outer for loop to create labels.

    for index, row in df.iterrows():
        labels.append('row'+str(index+1))
        for index2, row2 in df.iterrows():
            P = np.corrcoef(row.tolist(), row2.tolist())[0,1]
            print("P value of row%d and row%d is %.2f." %(index+1,index2+1,P))
            metrix.loc[index,index2] = P
    metrix.columns = labels
    metrix.index = labels
    print("\nTable of P values")
    print(metrix)
    return metrix
"""
du.pairwise_correlation(df)


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
Test
1. pairwise_correlation( df ):
Can you write Python code for it?

def test_pairwise_correlation():
    df = pd.DataFrame([[-1, 0, 1], [1, 0, -1], [.5, 0, .5]])
    assert int(du.pairwise_correlation(df).iloc[0,0]) == 1, "Diagonal elements not handled properly"
    assert int(du.pairwise_correlation(df).iloc[0,1]) == -1, "Anticorrelated elements not handled properly"
    assert int(du.pairwise_correlation(df).iloc[0,2]) == 0, "Uncorrelated elements not handled properly"
    assert int(du.pairwise_correlation(df).iloc[1,2]) == int(du.pairwise_correlation(df).iloc[2,1]), "Data not appended properly"
    return

test_pairwise_correlation()


I am working on the file "SEDS_Hw/seds-hw-3-danielfather7/SEDS-HW3.ipynb".
The first task is:
Try test_df_utils.py:
Can you write Python code for it?

import test_df_utils as ts

ts.test_pairwise_correlation()

ts.test_corr_rowi_rowj()

ts.test_corr_rowi_vs_all1()

ts.test_corr_rowi_vs_all2()


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
1.
Can you write Python code for it?

tk.test_dist()

"""
def test_dist():
    import knn
    vec1 = [0, 0, 0, 1]
    vec2 = [0, 0, 0, 0]
    assert int(knn.dist(vec1, vec2)) == 1, "Distance calculate not correct."
"""
print()


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
2.
Can you write Python code for it?

tk.test_sortdist()

"""
def test_sortdist():
    import knn
    import pandas as pd
    df = pd.DataFrame([[0,3], [0,1], [0,2]])
    vec = [0,0]
    assert int(knn.sortdist(df,vec)[0][0]) == 1, "Sort not handled properly"
    assert int(knn.sortdist(df,vec)[0][1]) == 1, "Index finding not handled properly."
"""
print()


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
3.
Can you write Python code for it?

tk.test_classify()

"""
def test_classify():
    import knn
    import pandas as pd
    df = pd.DataFrame([[0,3,'C'], [0,1,'B'], [0,2,'A']],columns = [1,2,'Type'])
    vec = [0,0]
#Test if there are multiple group type by a given k.
    assert knn.classify(1,df,vec) == 'B', "Group finding not handled properly."
    assert knn.classify(2,df,vec) == 'B', "Group finding not handled properly."
    assert knn.classify(3,df,vec) == 'B', "Group finding not handled properly."
"""
print()


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
3. classify(k,df,vec): give 1-D array a group by choosing a k value and training dataframe.

input: (k: k values; df: training dataframe, there should be a column named 'Type'; vec: 1-D array with 2 elements)
output: group which vec belongs to.

Note: If there are 2 or more max values, it will choose the closet point as its group among those max values.(Please example below)
Can you write Python code for it?

"""
def classify(k,df,vec):
    import itertools as it
    countlist = []
    grouplist = []
    gpsorted = []
    makechoice = []
#Choose k numbers of sorted data.
    kdistlist = sortdist(df,vec)[0:k]
#Find out the group it belongs to.
    for i in range(k):
        grouplist.append(df.loc[kdistlist[i][1]]['Type'])
#Use groupby to get numbers of the same group and unduplicated group name.
    gp = sorted(grouplist)
#    print(grouplist)
    for key, group in it.groupby(gp):
        countlist.append(len(list(group)))
        gpsorted.append(key)
#    print(countlist)
#    print(gpsorted)
#Use a if loop to deal with 2 or more max values in countlist.
    if countlist.count(max(countlist))>1:
        for j in range(len(countlist)):
            if countlist[j] == max(countlist):
                makechoice.append(grouplist.index(gpsorted[j]))
#        print(makechoice)
#It will choose the group which appeared ahead in grouplist.
#(In most case, it is the shortest distance among equal numbers of same
#groups.)
        group = grouplist[min(makechoice)]
    else:
        group = gpsorted[countlist.index(max(countlist))]
    return group
"""
print()

# Try
knn.classify(8,data,[0.5,0.5])


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
4.
Can you write Python code for it?

tk.test_knn()

"""
def test_knn():
    import knn
    import pandas as pd
    df1 = pd.DataFrame([[0,3,'C'], [0,1,'B'], [0,2,'A']],columns = [1,2,'Type'])
    df2 = pd.DataFrame([[0,3.1], [0,1.1], [0,2.1]])
    assert list(knn.knn(df1,df2,1)['Knn Type']) == ['C', 'B', 'A'], "Knn not handled properly."
    assert knn.knn(df1,df2,1).shape[1] == 3, "Column appending not handled properly."
"""
print()


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
5. knnac(traindf,inputdf,k): calculate accuracy by different k value.

input: (traindf: training dataframe with a column named 'Type'; inputdf: dataframe which needs to be classified; k: a list of k value)
output: a dictionary with k as the keys and the training accuracy of the test set.
Can you write Python code for it?

"""
def knnac(traindf,inputdf,k):
#Creat a dictionary, and calculate accuracy.
    k_ac = {}
    for i in k:
        n = 0
        for j in range(inputdf.shape[0]):
            if knn(traindf,inputdf,i)['Type'][j] == knn(traindf,inputdf,i)['Knn $
                n += 1
        k_ac[i] = '{0:.1%}'.format(n/inputdf.shape[0])
    return k_ac
"""
print()

#Try
knn.knnac(data,test,range(1,data.shape[0]))


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
Functions:
1. dist(vec1,vec2): caculate Euclidean distance.

input: 2 same size 1-D arrays.
output: Euclidean distance between them.
Can you write Python code for it?

"""
def dist(vec1,vec2):
#Use scipy package to calculate Euclidean distance.
    from scipy.spatial import distance
    return(distance.euclidean(vec1,vec2))
"""
print()

#Try
knn.dist(data.loc[0][0:2],test.loc[0][0:2])


I am working on the file "SEDS_Hw/seds-hw4-coding-and-testing-big-league-danielfather7/SEDS-HW4.ipynb".
The first task is:
Unit tests:
Can you write Python code for it?


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
<b>apts</b> is a list of 120 renting objects. We can take one out for our sample. We work on this sample to have the sense how to apply all these things to the whole list.
Can you write Python code for it?


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
Apply everything will did above to the whole list of rental objects.

Object: is every single apartment posted on Craigslist that we are working on.
Each page will show 120 objects by default.
We can iterate over 1000 objects to get the information
Can you write Python code for it?

page_index = np.arange(0, 1000, 120)

def set_price(x): # Avoid getting None value
    price = x.find('span', {'class':'result-price'})
    if price is not None:
        price = float(price.text.strip('$'))
    else:
        price = np.nan
    return price

def set_time(x): # Avoid getting None value
    time = x.find('time')['datetime']
    if time is not None:
        time = pd.to_datetime(time)
    else:
        time = np.nan
    return time


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
Put everything into Pandas DataFrame
Can you write Python code for it?

col_names = ['time', 'price', 'size', 'brs', 'title', 'link']
result = pd.DataFrame(data=result, columns=col_names)
result.set_index('time')
result.head()

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

plt.figure(figsize=(12,10))
sns.distplot(result.price.dropna(),bins=np.arange(0, 10000, 100),color='red',axlabel='SF Bay Area - Apartments\' Rental Price ($)').set(xlim=(0, 12000))


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
STEP 02: Get necessary information from the web
We can see that every single post will be stored in p attribute with class result-info

From that observation, we can retrieve all information by getting them from this attribute.
Can you write Python code for it?

apts = html.find_all('p', attrs={'class':'result-info'})
print(len(apts))


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
Some notices in this step:

We need to have a delay when we query Craigslist because if we keep querying too often, our IP address will be blocked (to avoid DDoS attack). In this step, I set the delay time is 1 second for each query.
In params, s parameter indicates the page we are currently working on. By increasing this variable, we will iterate over 1000 objects via multiple pages.
After everything's done, we store the result into result array. This is the input data for us to put in pandas DataFrame for later analytics purpose.
Can you write Python code for it?

import time as tp
result = []
for j in page_index:
    url_base = 'https://sfbay.craigslist.org/search/apa'
    params = dict(bedrooms=1, is_furnished=1,s=j)
    resq = requests.get(url_base, params=params)
    html = bs4(resq.text, 'html.parser')
    apts = html.find_all('p', attrs={'class':'result-info'})
    for this_aprt in apts:
        size = this_aprt.find_all(attrs={'class':'housing'})[0].text
        apt_size, n_brs = size_brs(size)
        time = set_time(this_aprt)
        price = set_price(this_aprt)
        title = this_aprt.find('a', {'class':'result-title hdrlnk'}).text
        link = this_aprt.find('a', {'class':'result-title hdrlnk'})['href']
        data = np.array([time, price, apt_size, n_brs,title, link])
        result.append(data)
    tp.sleep(1)


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
We get the information about number of bedroom and area
Can you write Python code for it?

size = this_aprt.find_all(attrs={'class':'housing'})[0].text
print(size)


I am working on the file "SF_BayArea_Rental/Craiglist - Apartments Rental Prices SF BayArea.ipynb".
The first task is:
We will create a function to get those information and store into apt_size and n_brs variables:
Can you write Python code for it?

def size_brs(size):
    split_var = size.strip('\n ').strip('\n - ').split('-')
    if len(split_var) == 2:
        n_brs = split_var[0].replace('br', '')
        apt_size = split_var[1].replace('ft2', '')
    elif 'br' in split_var[0]:
        n_brs = split_var[0].replace('br', '')
        apt_size = np.nan
    elif 'ft2' in split_var[1]:
        apt_size = split_var[1].replace('ft2', '')
        n_brs = np.nan
    return float(apt_size), float(n_brs)

apt_size, n_brs = size_brs(size)

time = this_aprt.find('time')['datetime']
time = pd.to_datetime(time)

price = float(this_aprt.find('span', {'class':'result-price'}).text.strip('$'))

title = this_aprt.find('a', {'class':'result-title hdrlnk'}).text

link = this_aprt.find('a', {'class':'result-title hdrlnk'})['href']


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:

Can you write Python code for it?

pokeapi = 'http://pokeapi.co/api/v2/generation/1/'

pokedata = requests.get(pokeapi).json()

# Take that data, print out a nicely formatted version of the first 5 results
print(json.dumps(pokedata['pokemon_species'][:5], indent=4))

# Let's get more info about the first pokemon on the list
# By following the chain of linked data

# Narrow down the url we'd like to get
bulbasaur_url = pokedata['pokemon_species'][0]['url']

# request data from that URL
bulbasaur_data = requests.get(bulbasaur_url).json()

# Let's remove the 'flavor text' because that's really long
del bulbasaur_data['flavor_text_entries']

bulbasaur_data


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
Other SHARE APIs
SHARE has a host of other APIs that provide direct access to the data stored in SHARE.
You can read more about the SHARE Data Models here: http://share-research.readthedocs.io/en/latest/share_models.html
Can you write Python code for it?

SHARE_API = 'https://staging-share.osf.io/api/'

share_endpoints = requests.get(SHARE_API).json()

share_endpoints


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
SHARE Search API
Also a fantastic resource!
One Way to Access Data
Instead of writing custom code to  parse both data coming from JSON and XML APIs
The SHARE Search Schema
The SHARE search API is built on a tool called elasticsearch. It lets you search a subset of SHARE's normalized metadata in a simple format.
Here are the fields available in SHARE's elasticsearch endpoint:
- 'title'
- 'language'
- 'subject'
- 'description'
- 'date'
- 'date_created'
- 'date_modified
- 'date_updated'
- 'date_published'
- 'tags'
- 'links'
- 'awards'
- 'venues'
- 'sources'
- 'contributors'

You can see a formatted version of the base results from the API by visiting the SHARE Search API URL.
Can you write Python code for it?

SHARE_SEARCH_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'

from furl import furl

search_url = furl(SHARE_SEARCH_API)
search_url.args['size'] = 3
recent_results = requests.get(search_url.url).json()

recent_results = recent_results['hits']['hits']

recent_results

print('The request URL is {}'.format(search_url.url))
print('----------')
for result in recent_results:
    print(
        '{} -- from {}'.format(
            result['_source']['title'],
            result['_source']['sources']
        )
    )


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
Sending a Query to the SHARE Search API
First, we'll define a function to do the hard work for us.
It will take 2 parameters, a URL, and a query to send to the search API.
Can you write Python code for it?

import json

def query_share(url, query):
    # A helper function that will use the requests library,
    # pass along the correct headers, and make the query we want

    headers = {'Content-Type': 'application/json'}
    data = json.dumps(query)
    return requests.post(url, headers=headers, data=data).json()

search_url.args = None  # reset the args so that we remove our old query arguments.
search_url.url # Show the URL that we'll be requesting to make sure the args were cleared

tags_query = {
    "query": {
        "exists": {
            "field": "tags"
        }
    }
}

missing_tags_query = {
    "query": {
        "bool": {
            "must_not": {
                "exists": {
                    "field": "tags"
                }
            }
        }
    }
}

with_tags = query_share(search_url.url, tags_query)
missing_tags = query_share(search_url.url, missing_tags_query)

total_results = requests.get(search_url.url).json()['hits']['total']

with_tags_percent = (float(with_tags['hits']['total'])/total_results)*100
missing_tags_percent = (float(missing_tags['hits']['total'])/total_results)*100

print(
    '{} results out of {}, or {}%, have tags.'.format(
        with_tags['hits']['total'],
        total_results,
        format(with_tags_percent, '.2f')
    )
)

print(
    '{} results out of {}, or {}%, do NOT have tags.'.format(
        missing_tags['hits']['total'],
        total_results,
        format(missing_tags_percent, '.2f')
    )
)

print('------------')
print('As a little sanity check....')
print('{} + {} = {}%'.format(with_tags_percent, missing_tags_percent, format(with_tags_percent + missing_tags_percent, '.2f')))


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
Visit the API In Your Browser
You can visit https://staging-share.osf.io/api/ and see the data formatted in "pretty printed" JSON
SHARE Providers API
Access the information about the providers that SHARE harvests from
Can you write Python code for it?

SHARE_PROVIDERS = 'https://staging-share.osf.io/api/providers/'

data = requests.get(SHARE_PROVIDERS).json()

data


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
We can print that out a little nicer
Using a loop and using the lookups that'd we'd like!
Can you write Python code for it?

print('Here are the first 10 Providers:')
for source in data['results']:
    print(
        '{}\n{}\n{}\n'.format(
            source['long_title'],
            source['home_page'],
            source['provider_name']
        )
    )


I am working on the file "SHARE_Curation_Associates_Overview.ipynb".
The first task is:
open your terminal
pip install arrow
Can you write Python code for it?

from arrow.arrow import Arrow

for item in data['response']:
    datetime = Arrow.fromtimestamp(item['risetime'])
    print(
        'The ISS will be visable over Charlottesville on {} at {} for {} seconds.'.format(
            datetime.date(),
            datetime.time(),
            item['duration']
        )
    )


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
"Traditional" Conceptual Model
Can you write Python code for it?

# Manual Conceptual Model
My_Manual = IPd.Image(path+'/Manual_Diagram.png')
My_Manual


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
1.     Optimization & GIS
What is Spatial Optimization?
       • Geographic Information Systems & Science
       • Operations Research / Decision Science
       • Optimization & Mathematical Programming

Maximize

        9x + 7y

Subject To

       10x + 5y <= 50
       6x + 6y <= 36
       4.5x + 18y <= 81

Bounds

       x >= 0
       y >= 0

End

The equations above represent a basic linear program designed to maximize profit.
The Shop Problem

x is Product 1;  y is Product 2

\$9 for each unit of x and \$7 for each unit of y

Constraint 1 is Machine 1 time (hrs.)

Constraint 2 is Machine 2 time (hrs.)
Constraint 3 is Machine 3 time (hrs.)

Solution:

objective value = \$50
x = 4 units
y = 2 units

Originally Published:

Greenwald, D. U.
1957
Linear Programming: An Explanation of the Simplex Algorithm
New York, NY
The Ronald Press Company

       • Location Allocation / Facility Location
|Linear Programming (LP)|Integer Programming (IP)|  Mixed-Integer Programming(MIP)  |
|:-------------------------:|:-------------------------:|:----------------------:
|   The Traveling Salesperson Problem   |p-Median Problem|p-Center Problem|
|The Transportation Problem|  Set Cover Location Problem  |Capacitated p-Center Problem|

How do we solve these models and is the workflow reproducible?
Later in this section there is an example of the workflow for another paper I am currently working on using an accepted, traditional method.  This involves manual data input and processing at every step in the schema which tends to decrease efficiency and reproducibility. This is very similar to an Operations Research approach to model...
Can you write Python code for it?


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1      Data preparation and creation
2.1.1       Instantiate a network
First, I start out by instantiating a network by reading a shapefile in as a network object with PySAL.  I use Waverly Hills, a smallish neighborhood in Tallahassee, FL where quite a few professors at Florida State University own homes.  This shapefile was clipped from a US Census TIGER/Line file [see references].  Additionally, there are some good topological challenges in Waverly Hills that make for good test cases, as seen below.
Note:

Steps 3.1.3 through 3.1.7 are needed to create a toy dataset of aggregated population demand points (i.e. households) and service facilities (i.e. school bus stops). These steps are eliminated when reading in actual datasets.
Can you write Python code for it?

# Instanitate network of Waverly Hills
ntw = ps.Network(path+'Waverly/Waverly.shp')


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.10      Instantiate client and service shapefiles
Using the python package shapefile, I instantiate .shp files of the client and service location data that was created above.
Can you write Python code for it?

# Client Shapefile
client = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_client:
    client.point(i,j)
# Add Fields
client.field('client_ID')
client.field('Weight')
client.field('LAT')
client.field('LON')
counter = 0
for i in range(len(rand_coords_client)):
    counter = counter + 1
    client.record('client_' + str(counter), Ai[i], lat_client[i], lon_client[i])
# Save Shapefile
client.save(path+'Simulated/RandomPoints_CLIENT')

#Service Shapefile
service = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_service:
    service.point(i,j)
# Add Fields
service.field('y_ID')
service.field('x_ID')
service.field('LAT')
service.field('LON')
counter = 0
for i in range(len(rand_coords_service)):
    counter = counter + 1
    service.record('y' + str(counter), 'x' + str(counter), lat_service[i], lon_service[i])
# Save Shapefile
service.save(path+'Simulated/RandomPoints_SERVICE')


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.11a      Snap observations to ntw
Next, I snap the client and service observations to the network with the ntw.snapobservations method within the instantiated network object for visualization.
Can you write Python code for it?

t1 = time.time()
Snap_C = ntw.snapobservations(path+'Simulated/RandomPoints_CLIENT.shp',
                     'Rand_Points_CLIENT', attribute=True)
Snap_S = ntw.snapobservations(path+'Simulated/RandomPoints_SERVICE.shp',
                     'Rand_Points_SERVICE', attribute=True)
print round(time.time()-t1, 4), 'seconds'


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.11b      Snap observations to SimNet
Also, I snap the client and service observations to the simplified network with the SimNet.snapobservations method. This allows for the creation of a distance matrix.
Can you write Python code for it?

t1 = time.time()
Snap_C = SimNet.snapobservations(path+'Simulated/RandomPoints_CLIENT.shp',
                     'Rand_Points_CLIENT', attribute=True)
Snap_S = SimNet.snapobservations(path+'Simulated/RandomPoints_SERVICE.shp',
                     'Rand_Points_SERVICE', attribute=True)
print round(time.time()-t1, 4), 'seconds'


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.12      Draw ntw, snapped coords, & random coords
When the graph is redrawn we can the client and service location in their original locations represented by the smaller points, and the network-snapped location represented with larger points.
Can you write Python code for it?

# Instantiate Figure
figsize(10,11)

# Draw Graph of Roads
for e in ntw.edges:
    g.add_edge(*e)
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)

# Draw Graph of Snapped Client Nodes
g_client = nx.Graph()
for p,coords in ntw.pointpatterns['Rand_Points_CLIENT'].snapped_coordinates.iteritems():
    g_client.add_node(p)
    g_client.node[p] = coords
nx.draw(g_client, ntw.pointpatterns['Rand_Points_CLIENT'].snapped_coordinates,
        node_size=75, alpha=1, node_color='b')

# Draw Graph of Snapped Service Nodes
g_service = nx.Graph()
for p,coords in ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates.iteritems():
    g_service.add_node(p)
    g_service.node[p] = coords
nx.draw(g_service, ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates,
        node_size=75, alpha=1, node_color='c')

# Draw Graph of Random Client Points
nx.draw(GRAPH_client, points_client,
    node_size=20, alpha=1, node_color='y')

# Draw Graph of Random Service Points
nx.draw(GRAPH_service, points_service,
    node_size=20, alpha=1, node_color='w')

# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
LEGEND['Snapped Client']=g_client
LEGEND['Snapped Service']=g_service
LEGEND['Client Nodes']=GRAPH_client
LEGEND['Service Nodes']=GRAPH_service
legend(LEGEND, loc='lower right', fancybox=True, framealpha=0.5, scatterpoints=1)

# Title
title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')

# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(-84.281, 30.49, 0.0, 0.005, width=.0003, head_width=0.001,
          head_length=0.0015, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(-84.2815, 30.498), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.13      Instantiate shapefile of service nodes snapped to ntw
This code chunk is key for representing the results of the optimization problems accurately.  I create a shapefile of the snapped service variable names and their coordinates to be used for comparison later.
Can you write Python code for it?

# Create Lat & Lon dictionaries of the snapped service locations
lat_snapped = []
lon_snapped = []
for i,j in ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates.iteritems():
    lat_snapped.append(j[0])
    lon_snapped.append(j[1])

# Snapped Service Shapefile
service_SNAP = shp.Writer(shp.POINT)
# Add Points
for i,j in ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates.iteritems():
    service_SNAP.point(j[0],j[1])
# Add Fields
service_SNAP.field('y_ID')
service_SNAP.field('x_ID')
service_SNAP.field('LAT')
service_SNAP.field('LON')
counter = 0
for i in range(len(ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates)):
    counter = counter + 1
    service_SNAP.record('y' + str(counter), 'x' + str(counter), lat_snapped[i], lon_snapped[i])
# Save Shapefile
service_SNAP.save(path+'Snapped/SERVICE_Snapped')


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.14      Create distance matrices
Finally, I create a client to service distance matrix with SimNet.allneighbordistances() from the instantiated network object of the simplified network of Waverly Hills [Waverly_Simplified.shp].  The cost matrix in created on the to reduce the computational burden, although it is not correct in it's current form.  The matrix is 100x15 and the values are converted from decimal degrees to miles.
A current limitation of this is the run-time to calculate accurate road segment length which is hindered by the number of nodes and edges of the network.  When tested with much larger client and service point sets the shortest-path  Dijkstra algorithm ran in similar time.  This demonstrates the limitation of calculating a cost matrix with the actual road network.  As a potential improvement, the Dijkstra may be refactored with a different data structure to see decreases in run-time. The run-time may decrease with another method of data storage.
Can you write Python code for it?

t1 = time.time()

# Define Client to Service Matrix Function
def c_s_matrix():
    global All_Dist_MILES
    All_Neigh_Dist = SimNet.allneighbordistances(sourcepattern=SimNet.pointpatterns['Rand_Points_CLIENT'],
                                             destpattern=SimNet.pointpatterns['Rand_Points_SERVICE'])
    All_Dist_MILES = All_Neigh_Dist * float(10000/90) * 0.6214

# Call Client to Service Matrix Function
c_s_matrix()

seconds = round(time.time()-t1, 4)
print seconds, 'seconds'
print 'Client to Service Matrix Shape --> ', All_Dist_MILES.shape


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.6      Create empty random points dictionaries
Dictionaries are instantiated to house the point location data for the simulated client and service facilities.
Can you write Python code for it?

# Empty Clients dictionary
points_client = {}

# Empty Service dictionary
points_service = {}


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.7      Fill dictionaries of random roints
The dictionaries that have just been created are now filled with the ids and lat/lons of the points.
Can you write Python code for it?

# CLIENT with {id: [lon, lat], } dictionary format
for idx, coords in enumerate(rand_coords_client):
    GRAPH_client.add_node(idx)
    points_client[idx] = coords
    GRAPH_client.node[idx] = coords

# SERVICE with {id: [lon, lat], } dictionary format
for idx, coords in enumerate(rand_coords_service):
    GRAPH_service.add_node(idx)
    points_service[idx] = coords
    GRAPH_service.node[idx] = coords


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.1.8      Draw roads, simplified network, and random client & service nodes
The following is a visualization of the data I have just created and the roads in Waverly Hills using matplotlib.  The roads and nodes are derived for network object created in [2.1.1].
Nodes in pink represent the actual intersections and the roads are represented in light red. The simplified network (connecting graph nodes) is gray and the remaining graph nodes (where a choice in forward destination is possible) are red.  These are necessary for determination of a shortest path and may greatly would diminish the computational time. The client nodes are represented in blue and the service nodes are represented in cyan.
Can you write Python code for it?

print dt.datetime.now()

#Instantiate Figure
figsize(10,11)

#Draw Graph of Actual Nodes and Roads
for e in ntw.edges:
    g.add_edge(*e)
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)

#Draw only unique edges in graph
for e in ntw.graphedges:
    g1.add_edge(*e)
    # highlights cases where start and end node are the same
    if e[0]==e[1]:
        g1.add_node(e[0])

for node_id in g1.node:
    g1.node[node_id] = ntw.node_coords[node_id]
nx.draw(g1, ntw.node_coords, node_size=20, alpha=0.5)

# Draw Graph of Random Client Points
nx.draw(GRAPH_client, points_client,
        node_size=75, alpha=1, node_color='b')

# Draw Graph of Random Service Points
nx.draw(GRAPH_service, points_service,
    node_size=75, alpha=1, node_color='c')

# Legend (Ordered Dictionary) from collections
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
LEGEND['Graph Vertices']=g1
LEGEND['Graph Edges']=g1
LEGEND['Client Nodes']=GRAPH_client
LEGEND['Service Nodes']=GRAPH_service
legend(LEGEND,
       loc='lower right',
       fancybox=True,
       framealpha=0.5,
       scatterpoints=1)

# Title
title('Waverly Hills\nTallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')

# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(-84.281, 30.49, 0.0, 0.005, width=.0003, head_width=0.001,
          head_length=0.0015, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(-84.2815, 30.498), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2      Mathematical Optimization
Set Parameters within Gurobi
Can you write Python code for it?

# Set Parameters
gbp.setParam('MIPFocus', 2)                 # Set MIP focus to 'Optimal' --> 2
gbp.setParam('MIPGapAbs', 0)                # Set Absolute MIP Gap  --> 0
gbp.setParam('GomoryPasses', 0)             # Set Number of Gomory Cuts  --> 0
gbp.setParam('ZeroHalfCuts', 0)             # Set Number of Zero Half Cuts  --> 0
gbp.setParam('ImpliedCuts', 0)              # Set Number of Implied Cuts  --> 0
gbp.setParam('BarConvTol', .000000001)      # Set Barrier Convergence Tolerence
gbp.setParam('FeasibilityTol', .000000001)  # Set Feasibility Tolerence
gbp.setParam('IntFeasTol', .000000001)      # Set Integer Feasibility Tolerence
gbp.setParam('OptimalityTol', .000000001)   # Set Optimality Tolerence
gbp.setParam('Method', 4)                   # Set Algorithm to 'concurrent': Dual Simplex and Barrier
gbp.setParam('DisplayInterval', 1)          # Set Display Interval to 1


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.1b     Instantiate selected Set Cover location shapefile
Following optimization, I created a new shapefile of the selected candidate service locations.
Can you write Python code for it?

# Define SCLP shapefile function
def Create_SCLP():
    try:
        # Instantiate SCLP shapefile
        SHP_SetCover = shp.Writer(shp.POINT)
        # Add Points
        for idy,idx,x,y in NEW_Records_SCLP:
            SHP_SetCover.point(float(x), float(y))
        # Add Fields
        SHP_SetCover.field('y_ID')
        SHP_SetCover.field('x_ID')
        SHP_SetCover.field('LAT')
        SHP_SetCover.field('LON')
        # Add Records
        for idy,idx,x,y in NEW_Records_SCLP:
            SHP_SetCover.record(idy,idx,x,y)
        # Save Shapefile
        SHP_SetCover.save(path+'Results/Selected_Locations_SetCover')
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
        print '   ################################################################'
        print ' < ISSUE : ', e, ' >'
        print '   ################################################################'

# Call SCLP shapefile function
try:
    Create_SCLP()
except Exception as e:
    exc_type, exc_obj, exc_tb = sys.exc_info()
    fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
    print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
    print '   ################################################################'
    print ' < ISSUE : ', e, ' >'
    print '   ################################################################'


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.3a       p-Center test [p = 2]
The p-center problem (PCP), also referred to as the minimax problem, sites facilities with a focus on equity and has been widely utilized in research as a counter-measure to the efficiency-based p-median problem.  The objective of the PCP is the minimization of the maximum (worst-case) travel cost from client nodes to service facilities [see references].
Minimize
         $W$
Subject to
         $\displaystyle\sum_{j\in m} x_{ij} = 1,$                $\forall i \in n$
         $\displaystyle\sum_{i \in n} y_j = p$
         $x_{ij} - y_j \geq 0,$              $\forall i\in n, j \in m$
         $\displaystyle W \geq \sum_{j \in m} c_{ij} x_{ij}$          $\forall i \in n$
         $x_{ij}, y_j \in (0,1)$           $\forall i \in n, j \in m$
where
         −  $W$ = the maximum travel cost between client and service nodes
         −  $i$ = a specific origin
         −  $j$ = a specific destination
         −  $n$ = the set of origins
         −  $m$ = the set of destinations
         −  $a_i$ = weight at each node
         −  $c_{ij}$ = travel costs between nodes
         −  $x_{ij}$ = the decision variable at each node in the...
Can you write Python code for it?

# define PCP shapefile function
def gbpPCP():

    t1 = time.time()

    # Define Global Variables
    global Cij
    global mPCP
    global dbf1
    global selected_PCP
    global NEW_Records_PCP

    #     1. Data
    Cij = All_Dist_MILES

    # Total Client and Service nodes
    client_nodes = range(len(Cij))
    service_nodes = range(len(Cij[0]))

    #    2. Create Model, Add Variables, & Update Model
    # Instantiate Model
    mPCP = gbp.Model(' -- P-Center -- ')

    # Add Client Decision Variables (iXj)
    client_var = []
    for orig in client_nodes:
        client_var.append([])
        for dest in service_nodes:
            client_var[orig].append(mPCP.addVar(vtype=gbp.GRB.BINARY,
                                                lb=0,
                                                ub=1,
                                                obj=Cij[orig][dest],
                                                name='x'+str(orig+1)+'_'+str(dest+1)))
    # Add Service Decision Variables (j)
    serv_var = []
    for dest in service_nodes:
        serv_var.append([])
        serv_var[dest].append(mPCP.addVar(vtype=gbp.GRB.BINARY,
                                          lb=0,
                                          ub=1,
                                          name='y'+str(dest+1)))

    # Add the Maximum travel cost variable
    W = mPCP.addVar(vtype=gbp.GRB.CONTINUOUS,
                    lb=0.,
                    name='W')

    # Update the model
    mPCP.update()

    #     3. Set the Objective function
    mPCP.setObjective(W, gbp.GRB.MINIMIZE)

    #     4. Add Constraints
    # Add Assignment Constraints
    for orig in client_nodes:
        mPCP.addConstr(gbp.quicksum(client_var[orig][dest]
                            for dest in service_nodes) == 1)

    # Add Opening constraints
    for orig in service_nodes:
        for dest in client_nodes:
            mPCP.addConstr((serv_var[orig][0] - client_var[dest][orig] >= 0))

    # Add...


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.3b      Instantiate selected p-Center shapefile
Following optimization, I created a new shapefile of the selected candidate service locations.
Can you write Python code for it?

# define PCP shapefile function
def Create_PCP():
    try:
        # Instantiate Shapefile
        SHP_Center = shp.Writer(shp.POINT)
        # Add Points
        for idy,idx,x,y in NEW_Records_PCP:
            SHP_Center.point(float(x), float(y))
        # Add Fields
        SHP_Center.field('y_ID')
        SHP_Center.field('x_ID')
        SHP_Center.field('LAT')
        SHP_Center.field('LON')
        # Add Records
        for idy,idx,x,y in NEW_Records_PCP:
            SHP_Center.record(idy,idx,x,y)
        # Save Shapefile
        SHP_Center.save(path+'Results/Selected_Locations_Pcenter')
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
        print '   ################################################################'
        print ' < ISSUE : ', e, ' >'
        print '   ################################################################'

# Call PCP shapefile function
try:
    Create_PCP()
except Exception as e:
    exc_type, exc_obj, exc_tb = sys.exc_info()
    fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
    print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
    print '   ################################################################'
    print ' < ISSUE : ', e, ' >'
    print '   ################################################################'


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.4b      Manual p-Center .lp File Solved
The .lp file created above is then read into Gurobi through gurobipy and solved.
Can you write Python code for it?

# Define manual LP read PCP Function
def Manual_LP_PCP():

    global Cij
    global manualPCP
    global dbf1
    global selected_PCP_manual
    global NEW_Records_PCP_Man

    t1 = time.time()

    # Instantiate Optimization model from .lp file
    manualPCP = gbp.read(path+'LP_Files/pCenter_Manual.lp')

    # Solve
    try:
        manualPCP.optimize()
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
        print '   ################################################################'
        print ' < ISSUE : ', e, ' >'
        print '   ################################################################'

    t2 = time.time()-t1

    # Record and Display Results
    print '\n*************************************************************************'
    selected_PCP_manual = []
    dbf1 = ps.open(path+'Snapped/SERVICE_Snapped.dbf')
    NEW_Records_PCP_Man = []
    for v in manualPCP.getVars():
        if 'x' in v.VarName:
            pass
        elif 'W' in v.VarName:
            pass
        elif v.x > 0:
            var = '%s' % v.VarName
            selected_PCP_manual.append(var)
            for i in range(dbf1.n_records):
                if var in dbf1.read_record(i):
                    x = dbf1.read_record(i)
                    NEW_Records_PCP_Man.append(x)
                else:
                    pass
            print '    |                                            ', var,  '         '
    print '    | Selected Facility Locations -------------- ^^^^ ',  '         '
    print '    | Candidate Facilities [p] ----------------- ', len(selected_PCP_manual), '           '
    print '    | Objective Value (miles) ------------------ ', manualPCP.objVal, '     '
    print '    | Real Time to Optimize (sec.) ------------- ', t2
    print '    | Date/Time -------------------------------- ', dt.datetime.now()
    print...


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.4c      Instantiate selected manual p-Center shapefile
Following optimization, I create another new shapefile for the selected manual candidate service locations.
Can you write Python code for it?

# define Manual PCP shapefile function
def Create_PCP_Man():
    try:
        # Instantiate Shapefile
        SHP_Center_Man = shp.Writer(shp.POINT)
        # Add Points
        for idy,idx,x,y in NEW_Records_PCP_Man:
            SHP_Center_Man.point(float(x), float(y))
        # Add Fields
        SHP_Center_Man.field('y_ID')
        SHP_Center_Man.field('x_ID')
        SHP_Center_Man.field('LAT')
        SHP_Center_Man.field('LON')
        # Add Records
        for idy,idx,x,y in NEW_Records_PCP_Man:
            SHP_Center_Man.record(idy,idx,x,y)
        # Save Shapefile
        SHP_Center_Man.save(path+'Results/Selected_Locations_Pcenter_Man')
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
        print '   ################################################################'
        print ' < ISSUE : ', e, ' >'
        print '   ################################################################'

# Call Manual PCP shapefile function
try:
    Create_PCP_Man()
except Exception as e:
    exc_type, exc_obj, exc_tb = sys.exc_info()
    fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
    print exc_type, fname, 'Line Number -- ',exc_tb.tb_lineno
    print '   ################################################################'
    print ' < ISSUE : ', e, ' >'
    print '   ################################################################'


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
2.2.4d      p-Center Results compared
Comaprison of the automated and manual p-Center solution results:
Can you write Python code for it?

print ' \n Automated p-Center'
print '    | Candidate Facilities [p] ----------------- ', len(selected_PCP), '           '
print '    | Selected Facility Locations -------------- ', selected_PCP
print '    | Objective Value (miles) ------------------ ', mPCP.objVal, '     '

print '\n Manual p-Center'
print '    | Candidate Facilities [p] ----------------- ', len(selected_PCP_manual), '           '
print '    | Selected Facility Locations -------------- ', selected_PCP_manual
print '    | Objective Value (miles) ------------------ ', manualPCP.objVal, '     '

print '\n\n    | Same Objective Value (str) --------------- ', str(mPCP.ObjVal) == str(manualPCP.ObjVal)
print '    | Same Objective Value (float)  ------------ ', mPCP.ObjVal.real == manualPCP.ObjVal.real
print '    | Same Facilities  ------------------------- ', selected_PCP == selected_PCP_manual

print '\nIdentical Parameters? ', str(mPCP.Params) == str(manualPCP.Params)

print '\n1: ', ("%.50f" % mPCP.ObjVal)
print '2: ', ("%.50f" % manualPCP.ObjVal)
print '3: ', ("%.50f" % A_to_MPCP.ObjVal)


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
3. Next Steps

Potential for refactoring the Dijkstra algorithm in PySAL.Network to increase efficiency

Improve cases for determining the simplified network to tackle to issue of network size

Integrate PySAL with an open source solver

GitHub
https://github.com/jGaboardi/SHRUG__2015
Can you write Python code for it?


I am working on the file "SHRUG_pres_Gaboardi.ipynb".
The first task is:
<b>So why do things differently?
When solving spatial problems in a GIS, there are typically lots of files to keep track of and a lot of button clicking. This can lead to compounded errors at every stage in the process.
The following flowchart is a conceptual model of my current workflow.  This workflow mimics the one described above, but the processes within are largely automated to maximize efficiency and reproducibility by minimizing the potential for human error in data handling, etc.  This workflow will be demonstrated next.
Automated Model
Can you write Python code for it?

# Automated Conceptual Model
My_Auto = IPd.Image(path+'/Auto_Diagram.png')
My_Auto