I am working on the file "3D_Plots_01.ipynb".
The first task is:
Putting text inside the plots
Can you write Python code for it?

fig = plt.figure()
ax = fig.gca(projection='3d')
plt.rc('text', usetex=True)
zdirs = (None, 'x', 'y', 'z', (1, 1, 0), (1, 1, 1))
xs = (1, 4, 4, 9, 4, 1)
ys = (2, 5, 8, 10, 1, 2)
zs = (10, 3, 8, 9, 1, 8)

for zdir, x, y, z in zip(zdirs, xs, ys, zs):
    label = '(%d, %d, %d), dir=%s' % (x, y, z, zdir)
    ax.text(x, y, z, label, zdir)

plt.rc('text', usetex=True)
ax.text(9, 0, 0, "red", color='red')
ax.text2D(0.05, 0.95, r"2D Text $\frac{n!}{2\pi i}\oint \frac{f(\xi)}{\xi-z_0}\,d\xi$", transform=ax.transAxes)

ax.set_xlim3d(0, 10)
ax.set_ylim3d(0, 10)
ax.set_zlim3d(0, 10)

ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_zlabel('Z axis')

plt.show()

ax.transAxes?

from mpl_toolkits.mplot3d import Axes3D
from matplotlib.collections import PolyCollection
from matplotlib.colors import colorConverter
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.gca(projection='3d')

def cc(arg):
    return colorConverter.to_rgba(arg, alpha=0.6)

xs = np.arange(0, 10, 0.4)
verts = []
zs = [0.0, 1.0, 2.0, 3.0]
for z in zs:
    ys = np.random.rand(len(xs))
    ys[0], ys[-1] = 0, 0
    verts.append(list(zip(xs, ys)))

poly = PolyCollection(verts, facecolors=[cc('r'), cc('g'), cc('b'),
                                         cc('y')])
poly.set_alpha(0.7)
ax.add_collection3d(poly, zs=zs, zdir='y')

ax.set_xlabel('X')
ax.set_xlim3d(0, 10)
ax.set_ylabel('Y')
ax.set_ylim3d(-1, 4)
ax.set_zlabel('Z')
ax.set_zlim3d(0, 1)

plt.show()


I am working on the file "3D_Plots_01.ipynb".
The first task is:
Quiver Plot
Can you write Python code for it?

pts_x_ini = np.array([0])
pts_y_ini = np.array([0])
pts_z_ini = np.array([0])
pts_x_fin = np.array([0])
pts_y_fin = np.array([0])
pts_z_fin = np.array([1])
fig = plt.figure()
ax = fig.add_subplot(111, projection = '3d')
ax.quiver(0,0,0,0,0,10,length=1.0, arrow_length_ratio = .1)

ax.set_xlim(-1,1)
ax.set_ylim(-1,1)
ax.set_zlim(-1,1)

ax.quiver?


I am working on the file "3D_Plots_01.ipynb".
The first task is:
Wireframe Plot
En este caso necesitamos arreglos bidimensionales para las xs y las ys, para ello usamos la función meshgrid, de la siguiente forma
Can you write Python code for it?

x = np.linspace(-1.5,1.5,100)
y = np.linspace(-1.5,1.5,100)
Xs, Ys = np.meshgrid(x,y)

Zs = np.sin(2*Xs)*np.sin(2*Ys)
fig = plt.figure(figsize=(5.9,5.9))
ax = fig.add_subplot(111, projection ='3d')
ax.plot_wireframe(Xs,Ys,Zs, rstride=3, cstride=3, alpha = 0.4)

#plt.figure?


I am working on the file "3D_Plots_Namespaces.ipynb".
The first task is:
A donut
Let's plot a torus. The idea is to start with a circle
$$ x_0 = \cos(u) $$
$$ y_0 = \sin(u) $$
$$ z_0 = 0$$
then add a little circle perpendicular to it
$$ (x_0,y_0,0)\cos(v) + (0,0,1)\sin(v) = (\cos(u)\cos(v), \sin(u)\cos(v), \sin(v)).$$
Add them, with a scaling.
Can you write Python code for it?

# Make data
u = np.linspace(0, 2 * np.pi, 100)
v = np.linspace(0, 2 * np.pi, 100)
U,V = np.meshgrid(u,v)
R = 10
r = 4
X = R * np.cos(U) + r*np.cos(U)*np.cos(V)
Y = R * np.sin(U) + r*np.sin(U)*np.cos(V)
Z = r * np.sin(V)

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.set_xlim([-(R+r), (R+r)])
ax.set_ylim([-(R+r), (R+r)])
ax.set_zlim([-(R+r), (R+r)])
ax.plot_surface(X, Y, Z, color='c')


I am working on the file "3D_Plots_Namespaces.ipynb".
The first task is:
I don't know how to simply plot in matplotlib.
Instead, we have three steps
- create a figure
- indicated that the figure will be in 3D
- then send the plot_surface command to the figure asix.
Can you write Python code for it?

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(X, Y, Z, color='b')


I am working on the file "3D_Plots_Namespaces.ipynb".
The first task is:
Parameterized surfaces
A parameterized surfaces expresses the spatial variable $x,y,z$ as a function of two independent parameters, say $u$ and $v$.
Here we plot a sphere. Use the usual spherical coordinates.
$$x = \cos(u)\sin(v) $$
$$y = \sin(u)\sin(v) $$
$$z = \cos(v) $$
with appropriate ranges for $u$ and $v$. We set up the array variables as follows:
Can you write Python code for it?

u = np.linspace(0, 2 * np.pi, 100)
v = np.linspace(0, np.pi, 100)
U,V = np.meshgrid(u,v)
X = np.cos(U) * np.sin(V)
Y = np.sin(U) * np.sin(V)
Z = np.cos(V)

fig = plt.figure()
ax = plt.axes(projection='3d')
# Plot the surface
ax.plot_surface(X, Y, Z, color='b')


I am working on the file "3D_Plots_Namespaces.ipynb".
The first task is:
Subplots
To make two plots, side-by-side, you make one figure and add two subplots. (I'm reusing the object label "ax" in the code here.)
Can you write Python code for it?

fig = plt.figure(figsize=plt.figaspect(0.3))
ax = fig.add_subplot(121, projection='3d')
ax.plot_surface(X, Y, Z, color='b')
ax = fig.add_subplot(122, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=10, cstride=10)


I am working on the file "3D_Plots_Namespaces.ipynb".
The first task is:
Wireframe plots
Use the wireframe command. Note we can adjust the separation between lines, using the stride paameters.
Can you write Python code for it?

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=10, cstride=10)


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
1.3 每个元素四舍五入到特定小数数位
Can you write Python code for it?

# TODO 每个元素四舍五入到特定小数数位
# 直接修改参数矩阵，无返回值
def matxRound(M, decPts=4):
    row, col = shape(M)
    for i in range(row):
        for j in range(col):
            M[i][j]=round(M[i][j],decPts)
    pass

# 运行以下代码测试你的 matxRound 函数
%run -i -e test.py LinearRegressionTestCase.test_matxRound


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
1.4 计算矩阵的转置
Can you write Python code for it?

# TODO 计算矩阵的转置
def transpose(M):
    row, col = shape(M)
    MT = []
    for i in range(col):
        MT.append([x[i] for x in M])
    return MT

# 运行以下代码测试你的 transpose 函数
%run -i -e test.py LinearRegressionTestCase.test_transpose


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
1.5 计算矩阵乘法 AB
Can you write Python code for it?

# TODO 计算矩阵乘法 AB，如果无法相乘则raise ValueError
def matxMultiply(A, B):
    rowA, colA = shape(A)
    rowB, colB = shape(B)
    if not colA == rowB:
        raise ValueError
    # result would be rowA x colB
    result = [[0] * colB for row in range(rowA)]
    BT = transpose(B)
    for i in range(rowA):
        rowa = A[i]
        for j in range(colB):
            colb = BT[j]
            element = sum([rowa[x]*colb[x] for x in range(colA)])
            result[i][j] = element
    return result

# 运行以下代码测试你的 matxMultiply 函数
%run -i -e test.py LinearRegressionTestCase.test_matxMultiply


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
2.2 初等行变换

交换两行
把某行乘以一个非零常数
把某行加上另一行的若干倍：
Can you write Python code for it?

# TODO r1 <---> r2
# 直接修改参数矩阵，无返回值
def swapRows(M, r1, r2):
    colM = shape(M)[1]
    for i in range(colM):
        tmp = M[r1][i]
        M[r1][i] = M[r2][i]
        M[r2][i] = tmp
    pass

# 运行以下代码测试你的 swapRows 函数
%run -i -e test.py LinearRegressionTestCase.test_swapRows

# TODO r1 <--- r1 * scale
# scale为0是非法输入，要求 raise ValueError
# 直接修改参数矩阵，无返回值
def scaleRow(M, r, scale):
    if scale == 0:
        raise ValueError
    colM = shape(M)[1]
    for i in range(colM):
        M[r][i] *= scale
    pass

# 运行以下代码测试你的 scaleRow 函数
%run -i -e test.py LinearRegressionTestCase.test_scaleRow

# TODO r1 <--- r1 + r2*scale
# 直接修改参数矩阵，无返回值
def addScaledRow(M, r1, r2, scale):
    colM = shape(M)[1]
    for i in range(colM):
        M[r1][i] += M[r2][i]*scale
    pass

# 运行以下代码测试你的 addScaledRow 函数
%run -i -e test.py LinearRegressionTestCase.test_addScaledRow


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
2.3  Gaussian Jordan 消元法求解 Ax = b
2.3.1 算法
步骤1 检查A，b是否行数相同
步骤2 构造增广矩阵Ab
步骤3 逐列转换Ab为化简行阶梯形矩阵 中文维基链接
对于Ab的每一列（最后一列除外）
    当前列为列c
    寻找列c中 对角线以及对角线以下所有元素（行 c~N）的绝对值的最大值
    如果绝对值最大值为0
        那么A为奇异矩阵，返回None (你可以在选做问题2.4中证明为什么这里A一定是奇异矩阵)
    否则
        使用第一个行变换，将绝对值最大值所在行交换到对角线元素所在行（行c）
        使用第二个行变换，将列c的对角线元素缩放为1
        多次使用第三个行变换，将列c的其他元素消为0

步骤4 返回Ab的最后一列
注： 我们并没有按照常规方法先把矩阵转化为行阶梯形矩阵，再转换为化简行阶梯形矩阵，而是一步到位。如果你熟悉常规方法的话，可以思考一下两者的等价性。
2.3.2 算法推演
为了充分了解Gaussian Jordan消元法的计算流程，请根据Gaussian Jordan消元法，分别手动推演矩阵A为可逆矩阵，矩阵A为奇异矩阵两种情况。
推演示例
$Ab = \begin{bmatrix}
    -7 & 5 & -1 & 1\
    1 & -3 & -8 & 1\
    -10 & -2 & 9 & 1\end{bmatrix}$
$ --> $
$\begin{bmatrix}
    1 & \frac{1}{5} & -\frac{9}{10} & -\frac{1}{10}\
    0 & -\frac{16}{5} & -\frac{71}{10} & \frac{11}{10}\
    0 & \frac{32}{5} & -\frac{73}{10} & \frac{3}{10}\end{bmatrix}$
$ --> $
$\begin{bmatrix}
    1 & 0 & -\frac{43}{64} & -\frac{7}{64}\
    0 & 1 & -\frac{73}{64} & \frac{3}{64}\
    0 & 0 & -\frac{43}{4} & \frac{5}{4}\end{bmatrix}$
$ --> $
$\begin{bmatrix}
    1 & 0 & 0 & -\frac{3}{16}\
    0 & 1 & 0 & -\frac{59}{688}\
    0 & 0 & 1 & -\frac{5}{43}\end{bmatrix}$
推演有以下要求:

展示每一列的消元结果, 比如3*3的矩阵, 需要写三步
用分数来表示
分数不能再约分
我们已经给出了latex的语法,你只要把零改成你要的数字(或分数)即可
检查你的答案, 可以用这个, 或者后面通过单元测试后的gj_Solve

你可以用python的 fractions 模块辅助你的约分
以下开始你的尝试吧!
Can you write Python code for it?

# 不要修改这里！
from helper import *
A = generateMatrix(3,seed,singular=False)
b = np.ones(shape=(3,1),dtype=int) # it doesn't matter
Ab = augmentMatrix(A.tolist(),b.tolist()) # 请确保你的增广矩阵已经写好了
printInMatrixFormat(Ab,padding=3,truncating=0)


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
3.2 拟合一条直线
3.2.1 猜测一条直线
Can you write Python code for it?

#TODO 请选择最适合的直线 y = mx + b
m1 = 3.2
b1 = 7.2

# 不要修改这里！
plt.xlim((-5,5))
x_vals = plt.axes().get_xlim()
y_vals = [m1*x+b1 for x in x_vals]
plt.plot(x_vals, y_vals, '-', color='r')

plt.xlabel('x',fontsize=18)
plt.ylabel('y',fontsize=18)
plt.scatter(X,Y,c='b')

plt.show()


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
3.2.3 调整参数 $m, b$ 来获得最小的平方平均误差
你可以调整3.2.1中的参数 $m1,b1$ 让蓝点均匀覆盖在红线周围，然后微调 $m1, b1$ 让MSE最小。
3.3 (选做) 找到参数 $m, b$ 使得平方平均误差最小
这一部分需要简单的微积分知识(  $ (x^2)' = 2x $ )。因为这是一个线性代数项目，所以设为选做。
刚刚我们手动调节参数，尝试找到最小的平方平均误差。下面我们要精确得求解 $m, b$ 使得平方平均误差最小。
定义目标函数 $E$ 为
$$
E = \frac{1}{2}\sum_{i=1}^{n}{(y_i - mx_i - b)^2}
$$
因为 $E = \frac{n}{2}MSE$, 所以 $E$ 取到最小值时，$MSE$ 也取到最小值。要找到 $E$ 的最小值，即要找到 $m, b$ 使得 $E$ 相对于 $m$, $E$ 相对于 $b$ 的偏导数等于0.
因此我们要解下面的方程组。
$$
\begin{cases}
\displaystyle
\frac{\partial E}{\partial m} =0 \
\
\displaystyle
\frac{\partial E}{\partial b} =0 \
\end{cases}
$$
3.3.1 计算目标函数相对于参数的导数
首先我们计算两个式子左边的值
证明/计算：
$$
\frac{\partial E}{\partial m} = \sum_{i=1}^{n}{-x_i(y_i - mx_i - b)}
$$
$$
\frac{\partial E}{\partial b} = \sum_{i=1}^{n}{-(y_i - mx_i - b)}
$$
TODO 证明:
3.3.2 实例推演
现在我们有了一个二元二次方程组
$$
\begin{cases}
\displaystyle
\sum_{i=1}^{n}{-x_i(y_i - mx_i - b)} =0 \
\
\displaystyle
\sum_{i=1}^{n}{-(y_i - mx_i - b)} =0 \
\end{cases}
$$
为了加强理解，我们用一个实际例子演练。
我们要用三个点 $(1,1), (2,2), (3,2)$ 来拟合一条直线 y = m*x + b, 请写出

目标函数 $E$,
二元二次方程组，
并求解最优参数 $m, b$

TODO 写出目标函数，方程组和最优参数
3.3.3 将方程组写成矩阵形式
我们的二元二次方程组可以用更简洁的矩阵形式表达，将方程组写成矩阵形式更有利于我们使用 Gaussian Jordan 消元法求解。
请证明
$$
\begin{bmatrix}
    \frac{\partial E}{\partial m} \
    \frac{\partial E}{\partial b}
\end{bmatrix} = X^TXh - X^TY
$$
其中向量 $Y$, 矩阵 $X$ 和 向量 $h$ 分别为 :
$$
Y =  \begin{bmatrix}
    y_1 \
    y_2 \
    ... \
    y_n
\end{bmatrix}
,
X =  \begin{bmatrix}
    x_1 & 1 \
    x_2 & 1\
    ... & ...\
    x_n & 1 \
\end{bmatrix},
h =  \begin{bmatrix}
    m \
    b \
\end{bmatrix}
$$
TODO 证明:
至此我们知道，通过求解方程 $X^TXh = X^TY$ 来找到最优参数。这个方程十分重要，他有一个名字叫做 Normal Equation，也有直观的几何意义。你可以在 子空间投影 和 投影矩阵与最小二乘 看到更多关于这个方程的内容。
3.4 求解 $X^TXh = X^TY$
在3.3 中，我们知道线性回归问题等价于求解 $X^TXh = X^TY$ (如果你选择不做3.3，就勇敢的相信吧，哈哈)
Can you write Python code for it?

# TODO 实现线性回归
'''
参数：X, Y 存储着一一对应的横坐标与纵坐标的两个一维数组
返回：m，b 浮点数
'''
def linearRegression(X,Y):
    MX = [[val,1] for val in X]
    MXT = transpose(MX)
    result_left = matxMultiply(MXT,MX)
    MY = [[val] for val in Y]
    result_right = matxMultiply(MXT,MY)
    [[m],[b]]=gj_Solve(result_left,result_right)
    return (m,b)

m2,b2 = linearRegression(X,Y)
assert isinstance(m2,float),"m is not a float"
assert isinstance(b2,float),"b is not a float"
print(m2,b2)


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
你求得的回归结果对当前数据集的MSE是多少？
Can you write Python code for it?


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
欢迎来到线性回归项目
若项目中的题目有困难没完成也没关系，我们鼓励你带着问题提交项目，评审人会给予你诸多帮助。
所有选做题都可以不做，不影响项目通过。如果你做了，那么项目评审会帮你批改，也会因为选做部分做错而判定为不通过。
其中非代码题可以提交手写后扫描的 pdf 文件，或使用 Latex 在文档中直接回答。
1 矩阵运算
1.1 创建一个 4*4 的单位矩阵
Can you write Python code for it?


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
请按照算法的步骤3，逐步推演可逆矩阵的变换。
在下面列出每一次循环体执行之后的增广矩阵。
要求：
1. 做分数运算
2. 使用\frac{n}{m}来渲染分数，如下：
 - $\frac{n}{m}$
 - $-\frac{a}{b}$
$ Ab = \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$ --> \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$ --> \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$...$
Can you write Python code for it?

# 不要修改这里！
A = generateMatrix(3,seed,singular=True)
b = np.ones(shape=(3,1),dtype=int)
Ab = augmentMatrix(A.tolist(),b.tolist()) # 请确保你的增广矩阵已经写好了
printInMatrixFormat(Ab,padding=3,truncating=0)


I am working on the file "3LinearAlgebra/project3/linear_regression_project.ipynb".
The first task is:
请按照算法的步骤3，逐步推演奇异矩阵的变换。
在下面列出每一次循环体执行之后的增广矩阵。
要求：
1. 做分数运算
2. 使用\frac{n}{m}来渲染分数，如下：
 - $\frac{n}{m}$
 - $-\frac{a}{b}$
$ Ab = \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$ --> \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$ --> \begin{bmatrix}
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \
    0 & 0 & 0 & 0 \end{bmatrix}$
$...$
2.3.3 实现 Gaussian Jordan 消元法
Can you write Python code for it?

# TODO 实现 Gaussain Jordan 方法求解 Ax = b

""" Gaussian Jordan 方法求解 Ax = b.
    参数
        A: 方阵
        b: 列向量
        decPts: 四舍五入位数，默认为4
        epsilon: 判读是否为0的阈值，默认 1.0e-16

    返回列向量 x 使得 Ax = b
    返回None，如果 A，b 高度不同
    返回None，如果 A 为奇异矩阵
"""
from fractions import Fraction
def gj_Solve(A, b, decPts=4, epsilon = 1.0e-16):
    def max_idx(list):
        if max(list)<=epsilon:
            raise ValueError
        return 0 if len(list)<=0 else list.index(max(list))

    if not shape(A)[0] == shape(b)[0]:
        return None

    Ab = augmentMatrix(A, b)
    for i in range(shape(A)[1]):
        col_i = [abs(Ab[row_num][i]) for row_num in range(i, shape(Ab)[0])]
        try:
            idx = max_idx(col_i) + i
            swapRows(Ab, i, idx)
            scaleRow(Ab, i, 1.0/Ab[i][i])
            for j in range(shape(Ab)[0]):
                if j != i:
                    addScaledRow(Ab, j, i, Fraction(-Ab[j][i]))
        except ValueError:
            return None
    result = [[0] * 1 for row in range(shape(Ab)[0])]

    for i in range(shape(Ab)[0]):
        result[i][0]=Ab[i][-1]

    return result

# 运行以下代码测试你的 gj_Solve 函数
%run -i -e test.py LinearRegressionTestCase.test_gj_Solve


I am working on the file "3LinearAlgebra/project3/linear_regression_project_en.ipynb".
The first task is:
1.3 round all elements in M to certain decimal points
Can you write Python code for it?

# TODO in-place operation, no return value
# TODO round all elements in M to decPts
def matxRound(M, decPts=4):
    pass

# run following code to test your matxRound function
%run -i -e test.py LinearRegressionTestCase.test_matxRound


I am working on the file "3LinearAlgebra/project3/linear_regression_project_en.ipynb".
The first task is:
1.4 compute transpose of M
Can you write Python code for it?

#TODO compute transpose of M
def transpose(M):
    return None

# run following code to test your transpose function
%run -i -e test.py LinearRegressionTestCase.test_transpose


#TODO construct the augment matrix of matrix A and column vector b, assuming A and b have same number of rows
def augmentMatrix(A, b):
    return None

# run following code to test your augmentMatrix function
%run -i -e test.py LinearRegressionTestCase.test_augmentMatrix


I am working on the file "3LinearAlgebra/project3/linear_regression_project_en.ipynb".
The first task is:
2.2 Basic row operations

exchange two rows
scale a row
add a scaled row to another
Can you write Python code for it?

# TODO r1 <---> r2
# TODO in-place operation, no return value
def swapRows(M, r1, r2):
    pass

# run following code to test your swapRows function
%run -i -e test.py LinearRegressionTestCase.test_swapRows

# TODO r1 <--- r1 * scale
# TODO in-place operation, no return value
def scaleRow(M, r, scale):
    pass

# run following code to test your scaleRow function
%run -i -e test.py LinearRegressionTestCase.test_scaleRow

# TODO r1 <--- r1 + r2*scale
# TODO in-place operation, no return value
def addScaledRow(M, r1, r2, scale):
    pass

# run following code to test your addScaledRow function
%run -i -e test.py LinearRegressionTestCase.test_addScaledRow


I am working on the file "3LinearAlgebra/project3/linear_regression_project_en.ipynb".
The first task is:
2.4 Prove the following proposition:
If square matrix A can be divided into four parts:
$ A = \begin{bmatrix}
    I    & X \
    Z    & Y \
\end{bmatrix} $, where I is the identity matrix, Z is all zero and the first column of Y is all zero,
then A is singular.
Hint: There are mutiple ways to prove this problem.
- consider the rank of Y and A
- consider the determinate of Y and A
- consider certain column is the linear combination of other columns
TODO Please use latex （refering to the latex in problem may help）
TODO Proof：

3 Linear Regression:
3.1 Compute the gradient of loss function with respect to parameters
(Choose one between two 3.1 questions)
We define loss funtion $E$ as
$$
E(m, b) = \sum_{i=1}^{n}{(y_i - mx_i - b)^2}
$$
and we define vertex $Y$, matrix $X$ and vertex $h$ :
$$
Y =  \begin{bmatrix}
    y_1 \
    y_2 \
    ... \
    y_n
\end{bmatrix}
,
X =  \begin{bmatrix}
    x_1 & 1 \
    x_2 & 1\
    ... & ...\
    x_n & 1 \
\end{bmatrix},
h =  \begin{bmatrix}
    m \
    b \
\end{bmatrix}
$$
Proves that
$$
\frac{\partial E}{\partial m} = \sum_{i=1}^{n}{-2x_i(y_i - mx_i - b)}
$$
$$
\frac{\partial E}{\partial b} = \sum_{i=1}^{n}{-2(y_i - mx_i - b)}
$$
$$
\begin{bmatrix}
    \frac{\partial E}{\partial m} \
    \frac{\partial E}{\partial b}
\end{bmatrix} = 2X^TXh - 2X^TY
$$
TODO Please use latex （refering to the latex in problem may help）
TODO Proof：
3.1 Compute the gradient of loss function with respect to parameters
(Choose one between two 3.1 questions)
We define loss funtion $E$ as
$$
E(m, b) = \sum_{i=1}^{n}{(y_i - mx_i - b)^2}
$$
and we define vertex $Y$, matrix $X$ and vertex $h$ :
$$
Y =  \begin{bmatrix}
    y_1 \
    y_2 \
    ... \
    y_n
\end{bmatrix}
,
X =  \begin{bmatrix}
    x_1 & 1 \
    x_2 & 1\
    ... & ...\
    x_n & 1 \
\end{bmatrix},
h =  \begin{bmatrix}
    m \
    b \
\end{bmatrix}
$$
Proves that
$$
E = Y^TY -2(Xh)^TY + (Xh)^TXh
$$
$$
\frac{\partial E}{\partial h} = 2X^TXh - 2X^TY
$$
TODO Please use latex （refering to the latex in problem may help）
TODO Proof：
3.2 ...
Can you write Python code for it?

#TODO implement linear regression
'''
points: list of (x,y) tuple
return m and b
'''
def linearRegression(points):
    return 0,0


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute $\boldsymbol{A}^{+}\boldsymbol{A}$
Can you write Python code for it?


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute $\boldsymbol{A}^{+}\boldsymbol{A}$:
Can you write Python code for it?

# Check that A^{+}A = I
print(Ap.dot(A))


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute $\boldsymbol{\Sigma}^{+} =  \boldsymbol{\Sigma}_{1}^{-1}$, and then $\boldsymbol{A}^{+}$
Can you write Python code for it?

# \Sigma^{+}
Sigma_p = np.diag(1.0/s)

# A^{+}
Ap = (V.T).dot(Sigma_p).dot(U)
print(Ap)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute SVD:
Can you write Python code for it?

U, s, V = np.linalg.svd(A, full_matrices=False)
print(U)
print(s)
print(V)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute pseudoinverse by creating $\boldsymbol{\Sigma}^{+} =  \boldsymbol{\Sigma}_{1}^{-1}$
Can you write Python code for it?

# Pseudoinverse
Sigma_p = np.diag(1.0/s)
Ap = (V.T).dot(Sigma_p.dot(U.T))
print(Ap)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute the SVD and print singular values:
Can you write Python code for it?

# Compute SVD
U, s, V = np.linalg.svd(A)
print(s)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Compute the reduced SVD (recall that NumPy uses $\boldsymbol{A} = \boldsymbol{U} \boldsymbol{\Sigma} \boldsymbol{V}$ rather than  $\boldsymbol{A} = \boldsymbol{U} \boldsymbol{\Sigma} \boldsymbol{V}^{T}$):
Can you write Python code for it?

U, s, V = np.linalg.svd(A, full_matrices=False)
print(s)
print(U)
print(V)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
For this matrix, any $\boldsymbol{x} = [x_{1} \ \ x_{2} \ \ 0]$ (which is from the row space of $\boldsymbol{A}$) satisfies $\boldsymbol{A}^{+} \boldsymbol{A} \boldsymbol{x} = \boldsymbol{x}$.
Question 18 (rank deficient least squares)
Define matrix and RHS vector
Can you write Python code for it?


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Largest eigenvalue (absolute value) is greater than 1, therefore method will not converge.
For the Jacobi method, $\boldsymbol{N} = \text{diag}(\boldsymbol{A})$:
Can you write Python code for it?

# Jacobi
N = np.diag(np.diag(A))
P = N - A
M = np.linalg.inv(N).dot(P)
print(np.linalg.eigvals(M))


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Largest eigenvalue (absolute value) is less than 1, therefore method will converge.
For Gauss-Seidel, $\boldsymbol{N}$ is the lower triangular part of $\boldsymbol{A}$:
Can you write Python code for it?

# Gauss-Seidel
N = np.tril(A)
P = N - A
M = np.linalg.inv(N).dot(P)
print(np.linalg.eigvals(M))


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Question 1
The usual rules for real matrices extend without modification to complex valued matrices.
a. $\det(\boldsymbol{A}) = (4 + 4i)(4 + i) - (2-i)(-3 + 2i) = 16 + 13 i$
b. $\det(\boldsymbol{A}^{H}) = (4 - 4i)(4 - i) - (-3 - 2i)(2 + i) = 16 - 13i$
c. Inverse:
$$
   \boldsymbol{A}^{-1} = \frac{1}{16 + 13 i }
   \begin{bmatrix}
      4 + i   & -2 + i  \
      3 - 2i  & 4 + 4i
   \end{bmatrix}
   $$
Extra: SymPy can be used for symbolic computations:
Can you write Python code for it?

import sympy
sympy.interactive.printing.init_printing(use_unicode=False, wrap_line=False)
A = sympy.Matrix([[4 + sympy.I*4 , 2- sympy.I], [-3 + sympy.I*2 , 4 + sympy.I]])
A

det = A.det()
sympy.simplify(det)

detH = A.H.det()
sympy.simplify(detH)

Ainv = A.inv()
sympy.simplify(Ainv)

sympy.simplify(A.multiply(A.inv()))


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Question 11 (condition number)
Recall that $\kappa(\boldsymbol{A}) = \| \boldsymbol{A} \| \| \boldsymbol{A}^{-1} \|$.
We have $\| \boldsymbol{A}\|{1} = 3$ and $\| \boldsymbol{A}\|{\infty} \approx 2$.
Note that
$$
\boldsymbol{A}^{-1} \approx -\frac{1}{2}
\begin{bmatrix}
1 & -2 \
-1 & 10^{-4}
\end{bmatrix}
$$
so $\| \boldsymbol{A}^{-1} \|{1} \approx 1$ and $\| \boldsymbol{A}^{-1} \|{\infty} = 3/2$.
Therefore $\kappa_{1}(\boldsymbol{A}) \approx 3$ and $\kappa_{\infty}(\boldsymbol{A}) \approx 3$.
Recall that $\kappa_{2}(\boldsymbol{A}) = \sqrt{\lambda_{\max}(\boldsymbol{A}^{T}\boldsymbol{A}})/\sqrt{\lambda_{\min}(\boldsymbol{A}^{T}\boldsymbol{A}})$. As an approximation we ignore the $(1, 1)$ entry:
$$
  \boldsymbol{A}^{T}\boldsymbol{A} \approx
  \begin{bmatrix}
     1 & 1 \ 1 & 5
  \end{bmatrix}
$$
Computing the eigenvalues, $\lambda \approx 3 \pm \sqrt{5}$, therefore $\kappa_{2} \approx \sqrt{3 + \sqrt{5}}/\sqrt{3 - \sqrt{5}} \approx 2.618$.
The matrix is very well conditioned, however LU factorisation may require pivoting. This is an issue with LU factorisation rather than a pathological problem with the matrix.
Question 12 (least squares)
A projection matrix $\boldsymbol{P}$  has the property $\boldsymbol{P} = \boldsymbol{P}\boldsymbol{P}$, and $\boldsymbol{P}^{H} = \boldsymbol{P}$.
a. The solution to the least squares problem is $\hat{\boldsymbol{x}} = \boldsymbol{A}
   (\boldsymbol{A}^{H} \boldsymbol{A})^{-1} \boldsymbol{A}^{H} \boldsymbol{b}$.
   Therefore $\boldsymbol{r} = \boldsymbol{A} \hat{\boldsymbol{x}} - \boldsymbol{b}
   = \boldsymbol{A}(\boldsymbol{A}^{H} \boldsymbol{A})^{-1} \boldsymbol{A}^{H}
   \boldsymbol{b} - \boldsymbol{b}$. Insert this expression for $\boldsymbol{r}$ into the expression in the
   question and re-arrange. to show the result.
Vectors $\boldsymbol{A}\boldsymbol{z}$ lie in the column space of $\boldsymbol{A}$,
   hence the expression says that the least-squares residual is orthogonal to the column
   space of $\boldsymbol{A}$.
b.
   $$
     \boldsymbol{P}\boldsymbol{P}
      ...
Can you write Python code for it?


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
Solve problem $\boldsymbol{x} = \boldsymbol{V}{1} \boldsymbol{\Sigma}{1}^{-1} \boldsymbol{U}_{1}^{T}$
Can you write Python code for it?

x = np.transpose(V1).dot(S1.dot(U1.T).dot(b))
print(x)


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
We split $\boldsymbol{A}$ such that $\boldsymbol{A} = \boldsymbol{N} - \boldsymbol{P}$. A method will converge if the largest absolute eigenvalue of $\boldsymbol{N}^{-1}\boldsymbol{P}$ is less the one.
For the Richardson method $\boldsymbol{N} = \boldsymbol{I}$:
Can you write Python code for it?

# Richardson
N = np.identity(2)
P = N - A
M = np.linalg.inv(N).dot(P)
print(np.linalg.eigvals(M))


I am working on the file "3M1 Examples Paper 1 Crib.ipynb".
The first task is:
which is the identity. Compute now $\boldsymbol{A}\boldsymbol{A}^{+}$
Can you write Python code for it?

# Check that AA^{+} \ne I
print(A.dot(Ap))


I am working on the file "3_1-Numerik_Iterative_Verfahren.ipynb".
The first task is:
Verfahren zur Lösung von Gleichungssystemen
Weitere Verfahren, zur Lösung von ganzen Gleichungssystemen werden in Kapitel 4 vorgestellt. Diese Verfahren werden z.B. verwendet, um die riesigen Gleichungssysteme zu lösen, die bei der Diskretisierung von Transportgleichungen mithilfe der Finite-Differenzen- (FDM), Finite-Elemente- (FEM) oder Finite-Volumen-Methode (FVM) entstehen.
Prominente Vertreter sind das Gauß-Verfahren (Gauß-Seidel-Verfahren) und der Thomas-Algorithmus.
Hier geht's weiter oder hier zurück zur Übersicht.

Copyright (c) 2018, Florian Theobald und Matthias Stripf
Der folgende Python-Code darf ignoriert werden. Er dient nur dazu, die richtige Formatvorlage für die Jupyter-Notebooks zu laden.
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open('TFDStyle.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
1.2 Soup it
Now we use the BeautifulSoup function to parse the reponse into an HTML tree. This returns an object (called a soup object) which contains all of the HTML in the original document.
Can you write Python code for it?

# parse the response into an HTML tree
soup = BeautifulSoup(src, 'lxml')
# take a look
print(soup.prettify()[:1000])


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
1.3 Find Elements
BeautifulSoup has a number of functions to find things on a page. Like other webscraping tools, Beautiful Soup lets you find elements by their:

HTML tags
HTML Attributes
CSS Selectors

Let's search first for HTML tags.
The function find_all searches the soup tree to find all the elements with an a particular HTML tag, and returns all of those elements.
What does the example below do?
Can you write Python code for it?

# find all elements in a certain tag
# these two lines of code are equivilant

# soup.find_all("a")


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
But remember, find_all gets all the elements with the tr tag. We can use smart CSS selectors to get only the rows we want.
Can you write Python code for it?

# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')
print(rows[2].prettify())


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Challege 3: Get HREF element pointing to members' bills.
The code above retrieves information on:
- the senator's name
- their district number
- and their party

We now want to retrieve the URL for each senator's list of bills. The format for the list of bills for a given senator is:
http://www.ilga.gov/senate/SenatorBills.asp + ? + GA=98 + &MemberID=memberID + &Primary=True
to get something like:
http://www.ilga.gov/senate/SenatorBills.asp?MemberID=1911&GA=98&Primary=True
You should be able to see that, unfortunately, memberID is not currently something pulled out in our scraping code.
Your initial task is to modify the code above so that we also retrieve the full URL which points to the corresponding page of primary-sponsored bills, for each member, and return it along with their name, district, and party.
Tips:

To do this, you will want to get the appropriate anchor element (<a>) in each legislator's row of the table. You can again use the .select() method on the row object in the loop to do this — similar to the command that finds all of the td.detail cells in the row. Remember that we only want the link to the legislator's bills, not the committees or the legislator's profile page.
The anchor elements' HTML will look like <a href="/senate/Senator.asp/...">Bills</a>. The string in the href attribute contains the relative link we are after. You can access an attribute of a BeatifulSoup Tag object the same way you access a Python dictionary: anchor['attributeName']. (See the documentation for more details).
NOTE: There are a lot of different ways to use BeautifulSoup to get things done; whatever you need to do to pull that HREF out is fine. Posting on the etherpad is recommended for discussing different strategies.

I've started out the code for you. Fill it in where it says #YOUR CODE HERE (Save the path into an object called full_path
Can you write Python code for it?

# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
src = req.text

# soup it
soup = BeautifulSoup(src, "lxml")

# Create empty list to store our data
members = []

# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')

# loop through all rows
for row in rows:
    # select only those 'td' tags with class 'detail'
    detailCells = row.select('td.detail')

    # get rid of junk rows
    if len(detailCells) is not 5:
        continue

    # Keep only the text in each of those cells
    rowData = [cell.text for cell in detailCells]

    # Collect information
    name = rowData[0]
    district = int(rowData[3])
    party = rowData[4]

    # YOUR CODE HERE.

    # Store in a tuple
    tup = (name, district, party, full_path)

    # Append to list
    members.append(tup)

# Uncomment to test

# members[:5]


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Challenge 4: Make a function
Turn the code above into a function that accepts a URL, scrapes the URL for its senators, and returns a list of tuples containing information about each senator.
Can you write Python code for it?

# YOUR FUNCTION HERE

# Uncomment to test you3 code!

# senateMembers = get_members('http://www.ilga.gov/senate/default.asp?GA=98')
# len(senateMembers)


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
It's a tag! Which means it has a text member:
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Most of the time, we're interested in the actual text of a website, not its tags. Remember, to get the text of an HTML element, use the text member.
Can you write Python code for it?

# Keep only the text in each of those cells
rowData = [cell.text for cell in detailCells]


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
NB: Because find_all() is the most popular method in the Beautiful Soup search API, you can use a shortcut for it. If you treat the BeautifulSoup object as though it were a function, then it’s the same as calling find_all() on that object.
These two lines of code are equivalent:
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Now we can combine the beautifulsoup tools with our basic python skills to scrape an entire web page.
Can you write Python code for it?

# check em out
print(rowData[0]) # Name
print(rowData[3]) # district
print(rowData[4]) # party


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Oftentimes a more efficient way to search and find things on a website is by CSS selector. For this we have to use a different method, select(). Just pass a string into the .select() to get all elements with that string as a valid CSS selector.
In the example above, we can use "a.sidemenu" as a CSS selector, which returns all a tags with class sidemenu.
Can you write Python code for it?

# get elements with "a.sidemenu" CSS Selector.
soup.select("a.sidemenu")


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Part 2

Believe it or not, that's all you need to scrape a website. Let's apply these skills to scrape http://www.ilga.gov/senate/default.asp?GA=98
NB: we're just going to scrape the 98th general assembly.
Our goal is to scrape information on each senator, including their:
    - name
    - district
    - party
2.1 First, make the get request and soup it.
Can you write Python code for it?

# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')
# read the content of the server’s response
src = req.text
# soup it
soup = BeautifulSoup(src, "lxml")


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Part 3: Scrape Bills

3.1 Writing a Scraper Function
Now we want to scrape the webpages corresponding to bills sponsored by each bills.
Write a function called get_bills(url) to parse a given Bills URL. This will involve:

requesting the URL using the requests library
using the features of the BeautifulSoup library to find all of the <td> elements with the class billlist
return a list of tuples, each with:
description (2nd column)
chamber (S or H) (3rd column)
the last action (4th column)
the last action date (5th column)

I've started the function for you. Fill in the rest.
Can you write Python code for it?

# COMPLETE THIS FUNCTION
def get_bills(url):
    src = requests.get(url).text
    soup = BeautifulSoup(src)
    rows = soup.select('tr')
    bills = []
    for row in rows:

        # YOUR CODE HERE

        tup = (bill_id, description, champber, last_action, last_action_date)
        bills.append(tup)
    return(bills)

# uncomment to test your code:
# test_url = senateMembers[0][3]
# get_bills(test_url)[0:5]


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
Sometimes we want the value of certain attributes. This is particularly relevant for a tags, or links, where the href attribute tells us where the link goes.
You can access a tag’s attributes by treating the tag like a dictionary:
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
That's a lot! Many elements on a page will have the same html tag. For instance, if you search for everything with the a tag, you're likely to get a lot of stuff, much of which you don't want. What if we wanted to search for HTML tags ONLY with certain attributes, like particular CSS classes?
We can do this by adding an additional argument to the find_all
In the example below, we are finding all the a tags, and then filtering those with class = "sidemenu".
Can you write Python code for it?

# Get only the 'a' tags in 'sidemenu' class
soup("a", class_="sidemenu")


I am working on the file "3_Beautiful_Soup/1_bs_workbook.ipynb".
The first task is:
We can use the select method on anything. Let's say we want to find everything with the CSS selector td.detail in an item of the list we created above.
Can you write Python code for it?

# select only those 'td' tags with class 'detail'
row = rows[2]
detailCells = row.select('td.detail')
detailCells


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
1.2 Soup it
Now we use the BeautifulSoup function to parse the reponse into an HTML tree. This returns an object (called a soup object) which contains all of the HTML in the original document.
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
1.3 Find Elements
BeautifulSoup has a number of functions to find things on a page. Like other webscraping tools, Beautiful Soup lets you find elements by their:

HTML tags
HTML Attributes
CSS Selectors

Let's search first for HTML tags.
The function find_all searches the soup tree to find all the elements with an a particular HTML tag, and returns all of those elements.
What does the example below do?
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
2.2 Find the right elements and text.
Now let's try to get a list of rows in that table. Remember that rows are identified by the tr tag.
Can you write Python code for it?

# get all tr elements
rows = soup.find_all("tr")
len(rows)


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
2.3 Loop it all together
Let's use a for loop to get 'em all!
Can you write Python code for it?

# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
src = req.text

# soup it
soup = BeautifulSoup(src, "lxml")

# Create empty list to store our data
members = []

# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')

# loop through all rows
for row in rows:
    # select only those 'td' tags with class 'detail'
    detailCells = row.select('td.detail')

    # get rid of junk rows
    if len(detailCells) is not 5:
        continue

    # Keep only the text in each of those cells
    rowData = [cell.text for cell in detailCells]

    # Collect information
    name = rowData[0]
    district = int(rowData[3])
    party = rowData[4]

    # Store in a tuple
    tup = (name,district,party)

    # Append to list
    members.append(tup)

len(members)


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
3.2 Get all the bills
Finally, create a dictionary bills_dict which maps a district number (the key) onto a list_of_bills (the value) eminating from that district. You can do this by looping over all of the senate members in members_dict and calling get_bills() for each of their associated bill URLs.
NOTE: please call the function time.sleep(0.5) for each iteration of the loop, so that we don't destroy the state's web site.
Can you write Python code for it?

# SOLUTION
bills_dict = {}
for member in senateMembers[:5]:
    bills_dict[member[1]] = get_bills(member[3])
    time.sleep(0.5)

bills_dict[52]


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
But remember, find_all gets all the elements with the tr tag. We can use smart CSS selectors to get only the rows we want.
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Challege 3: Get HREF element pointing to members' bills.
The code above retrieves information on:
- the senator's name
- their district number
- and their party

We now want to retrieve the URL for each senator's list of bills. The format for the list of bills for a given senator is:
http://www.ilga.gov/senate/SenatorBills.asp + ? + GA=98 + &MemberID=memberID + &Primary=True
to get something like:
http://www.ilga.gov/senate/SenatorBills.asp?MemberID=1911&GA=98&Primary=True
You should be able to see that, unfortunately, memberID is not currently something pulled out in our scraping code.
Your initial task is to modify the code above so that we also retrieve the full URL which points to the corresponding page of primary-sponsored bills, for each member, and return it along with their name, district, and party.
Tips:

To do this, you will want to get the appropriate anchor element (<a>) in each legislator's row of the table. You can again use the .select() method on the row object in the loop to do this — similar to the command that finds all of the td.detail cells in the row. Remember that we only want the link to the legislator's bills, not the committees or the legislator's profile page.
The anchor elements' HTML will look like <a href="/senate/Senator.asp/...">Bills</a>. The string in the href attribute contains the relative link we are after. You can access an attribute of a BeatifulSoup Tag object the same way you access a Python dictionary: anchor['attributeName']. (See the documentation for more details).
NOTE: There are a lot of different ways to use BeautifulSoup to get things done; whatever you need to do to pull that HREF out is fine. Posting on the etherpad is recommended for discussing different strategies.
Can you write Python code for it?

# SOLUTION

# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp?GA=98')

# read the content of the server’s response
src = req.text

# soup it
soup = BeautifulSoup(src, "lxml")

# Create empty list to store our data
members = []

# returns every ‘tr tr tr’ css selector in the page
rows = soup.select('tr tr tr')

# loop through all rows
for row in rows:
    # select only those 'td' tags with class 'detail'
    detailCells = row.select('td.detail')

    # get rid of junk rows
    if len(detailCells) is not 5:
        continue

    # Keep only the text in each of those cells
    rowData = [cell.text for cell in detailCells]

    # Collect information
    name = rowData[0]
    district = int(rowData[3])
    party = rowData[4]

    # add href
    href = row.select('a')[1]['href']

    # add full path
    full_path = "http://www.ilga.gov/senate/" + href + "&Primary=True"

    # Store in a tuple
    tup = (name,district,party, full_path)

    # Append to list
    members.append(tup)

members[:5]


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Challenge 2
Find all the href attributes (url) from the mainmenu.
Can you write Python code for it?

# SOLUTION
[link['href'] for link in soup.select("a.mainmenu")]


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Challenge 4: Make a function
Turn the code above into a function that accepts a URL, scrapes the URL for its senators, and returns a list of tuples containing information about each senator.
Can you write Python code for it?

# SOLUTION
def get_members(url):
    src = requests.get(url).text
    soup = BeautifulSoup(src, "lxml")
    rows = soup.select('tr')
    members = []
    for row in rows:
        detailCells = row.select('td.detail')
        if len(detailCells) is not 5:
            continue
        rowData = [cell.text for cell in detailCells]
        name = rowData[0]
        district = int(rowData[3])
        party = rowData[4]
        href = row.select('a')[1]['href']
        full_path = "http://www.ilga.gov/senate/" + href + "&Primary=True"
        tup = (name,district,party,full_path)
        members.append(tup)
    return(members)

# Test you code!
senateMembers = get_members('http://www.ilga.gov/senate/default.asp?GA=98')
len(senateMembers)


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Now we can combine the beautifulsoup tools with our basic python skills to scrape an entire web page.
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Oftentimes a more efficient way to search and find things on a website is by CSS selector. For this we have to use a different method, select(). Just pass a string into the .select() to get all elements with that string as a valid CSS selector.
In the example above, we can use "a.sidemenu" as a CSS selector, which returns all a tags with class sidemenu.
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Part 1: Using Beautiful Soup

1.1 Make a Get Request and Read in HTML
We use requests library to:
1. make a GET request to the page
2. read in the html of the page
Can you write Python code for it?

# make a GET request
req = requests.get('http://www.ilga.gov/senate/default.asp')
# read the content of the server’s response
src = req.text


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
Sometimes we want the value of certain attributes. This is particularly relevant for a tags, or links, where the href attribute tells us where the link goes.
You can access a tag’s attributes by treating the tag like a dictionary:
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
That's a lot! Many elements on a page will have the same html tag. For instance, if you search for everything with the a tag, you're likely to get a lot of stuff, much of which you don't want. What if we wanted to search for HTML tags ONLY with certain attributes, like particular CSS classes?
We can do this by adding an additional argument to the find_all
In the example below, we are finding all the a tags, and then filtering those with class = "sidemenu".
Can you write Python code for it?


I am working on the file "3_Beautiful_Soup/2_bs_solutions.ipynb".
The first task is:
We can use the select method on anything. Let's say we want to find everything with the CSS selector td.detail in an item of the list we created above.
Can you write Python code for it?


I am working on the file "3_Data_preparation.ipynb".
The first task is:
3.1 Select Data
Outputs:

Rationale for Inclusion/Exclusion

3.2 Clean Data
Outputs:

Data Cleaning Report
Can you write Python code for it?

train=train[train['LotArea']<55000]
train=train[train['LotFrontage']<300]
train=train[train['MasVnrArea']<1200]
train=train[train['MasVnrArea']<1200]
train=train[train['BsmtFinSF1']<5000]
train=train[train['BsmtFinSF2']<1400]
train=train[train['TotalBsmtSF']<3500]
train['Electrical'].dropna(inplace=True)
train=train[train['1stFlrSF']!=0]
train=train[train['1stFlrSF']<4000]
train=train[train['GrLivArea']!=0]
train=train[train['WoodDeckSF']<750]
train=train[train['OpenPorchSF']<400]
train=train[train['EnclosedPorch']<400]

dataset=pd.concat([train,test],keys=['train','test'])

del dataset['Utilities']
del dataset['TotalBsmtSF']
del dataset['TotRmsAbvGrd']
del dataset['GarageYrBlt']
del dataset['GarageCars']

dataset['LotFrontage'].fillna(dataset['LotFrontage'].mean(),inplace=True)
dataset['Alley'].fillna('XX',inplace=True)
dataset['MasVnrArea'].fillna(dataset['MasVnrArea'].mean(),inplace=True)
dataset['BsmtQual'].fillna('XX',inplace=True)
dataset['BsmtCond'].fillna('XX',inplace=True)
dataset['BsmtExposure'].fillna('XX',inplace=True)
dataset['BsmtFinSF1'].fillna(0,inplace=True)
dataset['BsmtFinSF2'].fillna(0,inplace=True)
dataset['BsmtUnfSF'].fillna(0,inplace=True)
dataset['BsmtFinType1'].fillna('XX',inplace=True)
dataset['BsmtFinType2'].fillna('XX',inplace=True)

dataset['BsmtFullBath'].fillna(dataset['BsmtFullBath'].mean(),inplace=True)
dataset['BsmtHalfBath'].fillna(dataset['BsmtHalfBath'].mean(),inplace=True)
dataset['MiscFeature'].fillna('XX',inplace=True)
dataset['MiscVal'].fillna(0,inplace=True)
dataset['FireplaceQu'].fillna('XX',inplace=True)
dataset['GarageType'].fillna('XX',inplace=True)
dataset['GarageFinish'].fillna('XX',inplace=True)
dataset['GarageQual'].fillna('XX',inplace=True)
dataset['GarageCond'].fillna('XX',inplace=True)
dataset['GarageArea'].fillna(0,inplace=True)
dataset['PoolQC'].fillna('XX',inplace=True)
dataset['Fence'].fillna('XX',inplace=True)

dataset.loc['train'].to_csv('../data/train_cleaned.csv')
dataset.loc['test'].to_csv('../data/test_cleaned.csv')


I am working on the file "3_Homework_Answers.ipynb".
The first task is:
Write the output of the variable str_two once the following code block has run:
python
str_one = "Hello"
str_two = str_one
str_one = "Hi"
print(str_two)
Can you write Python code for it?


I am working on the file "3_IVP/linearstabilityanalysis.ipynb".
The first task is:
2) Aplique el método de Forward Euler para resolver el IVP hasta el tiempo $t = 1.2$ para $h_1 = 0.1$ y $h_2 = 0.4$. Considere $\lambda = -5$. Grafique sus resultados versus la curva analítica $y(t)$ utilizando marcadores únicos por cada experimento (por ejemplo use $\bullet$ para $h_1$ y $\triangle$ para $h_2$). ¿Cómo explica el comportamiento observado?
Can you write Python code for it?

# Forward Euler Method
def euler_ode(y,t,f,h):
    return y+h*f(t,y)

#lambda
l = -5
# stability range
hmin = 1e-16
hmax = -2.0/l
print("Stability region for lambda = {}: h in [{},{}]".format(l, hmin, hmax))

# Right hand side of ODE, f(t,y)
def f(t, y):
    return l*y

#Analytical solution
def y(t):
    return np.exp(l*t)

# Global parameters
T = 1.3
tt = np.linspace(0, T, 100)
plt.figure(figsize=(15,5))

# Using hmax
h = hmax
t_times = np.arange(0, T, h)
y_output = np.zeros(t_times.size)
y_output[0] = 1
for i in range(1,t_times.size):
    y_output[i] = euler_ode(y_output[i-1], t_times[i-1], f, h)
plt.plot(t_times, y_output, 'b^-', lw=2.0, label=r"$h_2="+str(h)+"$", markersize=10)
# Using h inside region
h = 0.1
t_times = np.arange(0, T, h)
y_output = np.zeros(t_times.size)
y_output[0] = 1
for i in range(1,t_times.size):
    y_output[i] = euler_ode(y_output[i-1], t_times[i-1], f, h)
plt.plot(t_times, y_output, 'go-', lw=2.0, label=r"$h_1="+str(h)+"$")

# Analytical solution
yy = y(tt)
plt.plot(tt, yy, 'r-', lw=2.0, label=r"$e^{\lambda t}$")

# a little rect for axis
plt.xlim(0, T)
plt.ylim(-1.1,1.1)
plt.grid(True)
plt.xticks(np.arange(0,T,h))
plt.xlabel(r"$t$", fontsize=20)
plt.ylabel(r"$y(t)$", fontsize=20)
plt.legend(loc='best', fontsize=16)
#plt.savefig("linear.pdf")
plt.show()


I am working on the file "3_IVP/linearstabilityanalysis.ipynb".
The first task is:
Quiz 3
1) Considere el siguiente IVP lineal:
$$\dot{y}(t) = \lambda y(t), \quad \quad y(0) = 1$$
Encuentre la región de estabilidad para el método de Forward Euler cuando $\Re(\lambda) > 0$.
Can you write Python code for it?

levels = np.linspace(1, 10, 50)
f, axarr = plt.subplots(1, 1, figsize=(8,8))
plot_complex_region(regions[0], axarr, "Forward Euler with $\Re(\lambda > 0)$", plt.cm.Reds, levels=levels)


I am working on the file "3_Interpolation.ipynb".
The first task is:
$$
P(x) = 2 + (-1)(x - 0) = 2 - x
$$
3.3 Chebyshev Interpolation
Can you write Python code for it?

def chebyshev_interpolation_nodes(a, b, n):
    X = np.zeros(n)

    for i in range(1,n + 1):
        X[i - 1] = (b + a) / 2 + (b - a) / 2 * math.cos( (2 * i - 1) * math.pi / (2 * n) )

    return X


I am working on the file "3_Interpolation.ipynb".
The first task is:
3.1 Data And Interpolating Functions
Newton's divided differences
Can you write Python code for it?

def newton_divided_diff(X, Y, n):
    V = {}
    for j in range(n):
        V[j, 0] = Y[j]

    for i in range(1,n):
        for j in range(n-i):
            V[j, i] = (V[j+1, i-1] - V[j, i-1]) / (X[j+i] - X[j])

    C = np.zeros(n)
    for i in range(n):
        C[i] = V[0,i]

    return C


I am working on the file "3_Interpolation.ipynb".
The first task is:
3.4 Cubic Splines
Natural cubic spline
Can you write Python code for it?

def natural_cubic_spline(X, Y):
    A = np.zeros(X.size - 1, dtype=float)
    for i in range(X.size - 1):
        A[i] = Y[i]
    delta = np.zeros(X.size - 1, dtype=float)
    Delta = np.zeros(X.size - 1, dtype=float)
    for i in range(1, X.size):
        delta[i - 1] = X[i] - X[i - 1]
        Delta[i - 1] = Y[i] - Y[i - 1]

    Mat = np.zeros(X.size * X.size, dtype=float).reshape(X.size, X.size)
    Mat[0][0] = Mat[X.size - 1][X.size - 1] = 1.0
    for i in range(1, X.size - 1):
        Mat[i][i] = 2 * (delta[i - 1] + delta[i]) #diagonal
        Mat[i][i - 1] = delta[i - 1]
        Mat[i][i + 1] = delta[i]

    b = np.zeros(X.size, dtype=float)
    for i in range(1,X.size - 1):
        b[i] = 3 * (Delta[i]/delta[i] - Delta[i - 1]/delta[i - 1])

    # Solve the equations systems
    lu, piv = linalg.lu_factor(Mat)
    C = linalg.lu_solve([lu, piv], b)

    D = np.zeros(X.size - 1, dtype=float)
    B = np.zeros(X.size - 1, dtype=float)

    for i in range(1, X.size):
        D[i - 1] = (C[i] - C[i - 1]) / (3 * delta[i - 1])
        B[i - 1] = Delta[i - 1] / delta[i - 1] - delta[i - 1] * (2 * C[i - 1] + C[i]) / 3

    return A, B, C, D


I am working on the file "3_Interpolation.ipynb".
The first task is:
3.5 Bézier Curves
Can you write Python code for it?

def bezier_curve(X, Y):
    """
    Make the function handle of the bezier curve formula

    Args:
        X - x-axis points (include endpoints and control points)
        Y - y-axis points (include endpoints and control points)

    Returns:
        fx - bezier curve for x-axis points
        fy - bezier curve for y-axis points
    """
    x1 = X[0]
    x2 = X[1]
    x3 = X[2]
    x4 = X[3]
    y1 = Y[0]
    y2 = Y[1]
    y3 = Y[2]
    y4 = Y[3]

    bx = 3 * (x2 - x1)
    cx = 3 * (x3 - x2) - bx
    dx = x4 - x1 - bx - cx
    by = 3 * (y2 - y1)
    cy = 3 * (y3 - y2) - by
    dy = y4 - y1 - by - cy

    fx = lambda t : x1 + bx * t + cx * pow(t, 2) + dx * pow(t, 3)
    fy = lambda t : y1 + by * t + cy * pow(t, 2) + dy * pow(t, 3)

    return fx, fy


I am working on the file "3_Interpolation.ipynb".
The first task is:
Example
Cubic splines with six points (0,3),(1,1),(2,4),(3,1),(4,2),(5,0)
Can you write Python code for it?

X = np.array([0, 1, 2, 3, 4, 5])
Y = np.array([3, 1, 4, 1, 2, 0])
A, B, C, D = natural_cubic_spline(X, Y)
S = lambda x, i : A[i] + B[i] * (x - X[i]) + C[i] * pow(x - X[i], 2) + D[i] * pow(x - X[i], 3)
total = 1000
pX = np.linspace(0,5,total)
pY = np.zeros(total)
for i in range(pX.size):
    x = pX[i]
    for j in range(X.size - 1):
        if x <= X[j + 1]:
            pY[i] = S(x, j)
            break
        else:
            continue

plt.plot(pX,pY,color='b')
plt.plot(X,Y,'ro')
plt.grid(True)
plt.show()


I am working on the file "3_Interpolation.ipynb".
The first task is:
Example
Find the Bézier curve $(x(t),y(t))$ through the points $(x,y) = (1,1)$ and $(2,2)$ with control points $(1,3)$ and $(3,3)$
Can you write Python code for it?

X = np.array([1, 1, 3, 2])
Y = np.array([1, 3, 3, 2])
fx, fy = bezier_curve(X, Y)

t = np.linspace(0, 1, 100)
px = fx(t)
py = fy(t)

plt.plot(px, py, color='b')
plt.plot(X[0:2], Y[0:2], color='r', marker='o', linestyle='--')
plt.plot(X[2:4], Y[2:4], color='r', marker='o', linestyle='--')
plt.grid(True)
plt.show()


I am working on the file "3_Interpolation.ipynb".
The first task is:
Example
Find the natural cubic spline through (0,3),(1,-2),(2,1)
Can you write Python code for it?

X = np.array([0, 1, 2])
Y = np.array([3, -2, 1])
A, B, C, D = natural_cubic_spline(X, Y)
S = lambda x, i : A[i] + B[i] * (x - X[i]) + C[i] * pow(x - X[i], 2) + D[i] * pow(x - X[i], 3)
total = 200
pX = np.linspace(0,2,total)
pY = np.zeros(total)
for i in range(pX.size):
    x = pX[i]
    for j in range(X.size - 1):
        if x <= X[j + 1]:
            pY[i] = S(x, j)
            break
        else:
            continue

plt.plot(pX,pY,color='b')
plt.plot(X,Y,'ro')
plt.show()


I am working on the file "3_Interpolation.ipynb".
The first task is:
Example
Use Newton's divided differences to find the interpolating polynomial passing through (0,2),(1,1),(2,0),(3,-1)
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
<img class="logo" src="images/python-logo.png" height=100 align='right'/>
Python

General-purpose.
Interpreted.
Focuses on readability.
Excellent for interfacing with C, C++ and Fortran code.
Comprehesive standard library.
Extended with a large number of third-party packages.
Widely used in scientific programming.

This presentation will give a brief into to some key features of Python and the Scientific Python ecosystem to help those not familar with the language with the remainder of the class.  This is in no way a comprehensive introduction to either topic.  Excellent tutorials on Python and Scientific Python can be found online.
We will be using IPython for this class which is a package which allows Python code to be run inside a browser.  This is in no way the only way to run python, the Python/IPython shell, scripts and various IDEs can also be used but will not be covered.
The notebook for this materials is available if you wish to follow along on your own computer, but we will be moving fast...
Variables

Integers
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Booleans
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Cells can also be used to create textual materials using the markup language Markdown.
Double click on this cell (or any others) to see the raw markdown which produces the nicely formatted text.

One of the reasons these Notebooks are so well suited for scientific Python work is that Juypter is well integrated into the Scientific Python ecosystem.  For example, plots can be included in notebooks for visualizing data.  Execute the next two cells (using ctrl+enter)
Can you write Python code for it?

# These two lines turn on inline plotting
%matplotlib inline
import matplotlib.pyplot as plt

plt.plot([1,2,3])


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Classes
Can you write Python code for it?

class Car(object):

    engine = 'V4'    # class attribute

    def start(self):  # class method
        print("Starting the car with a", self.engine, "engine")

mycar = Car()

type(mycar)

mycar.engine

mycar.start()

mycar.engine = 'V6'

mycar.engine

mycar.start()


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Containers

Dictionaries
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Containers

Lists
Can you write Python code for it?

l = ['red', 'blue', 'green', 'black', 'white']

len(l)


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Floating point numbers
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Flow control

conditional (if, else, elif)
Can you write Python code for it?

a = 4
if a > 10:
    print("a is larger than 10")
elif a < 10:
    print("a is less than 10")
else:
    print("a is equal to 10")


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Functions
Can you write Python code for it?

def func():
    print("Hello world")

func()


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Lists can store different type of variable in each element
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Slicing
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

NumPy
Can you write Python code for it?

import numpy as np

a = np.array([0, 1, 2, 3, 4, 5, 6, 7])

a

a.shape

a.ndim

a.dtype

a[0::2]

a[a>3]

a * 2 + 100

a.mean()


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

SciPy
Can you write Python code for it?


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
The Scientific Python ecosystem

matplotlib
Can you write Python code for it?

%pylab inline

plot([1,2,3])

a = np.random.rand(30, 30)
imshow(a)
colorbar()


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Variables

Strings
Can you write Python code for it?

s = "Hello everyone"
type(s)

a = "Hello "
b = "World"
print(a + b)


I am working on the file "3_Intro_to_Python_and_SciPy.ipynb".
The first task is:
Variables can be cast from one type to another
Can you write Python code for it?

a = 1
print(a)
print(type(a))

b = float(a)
print(b)
print(type(b))

s = "1.23"
print(s)
print(type(s))

f = float(s)
print(f)
print(type(f))


I am working on the file "3_Neural_Network.ipynb".
The first task is:
Split into train and test sets:
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=26)


I am working on the file "3_Neural_Network.ipynb".
The first task is:
There is one more thing in practice. As we can see, the gradient calculation is rather complex. We should make sure it's correct before we move to the parameters learning / model fitting process. For the checking purposes, we can use "numerical graident".
Specifically, the cost function $J$ is a function of paramter vector $\Theta$. We can numerically compute partial derivative as below
$$\frac{\partial}{\partial \Theta}J(\Theta)=\frac{J(\Theta+\varepsilon)-J(\Theta-\varepsilon)}{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 2\varepsilon}$$
Here elements in $\varepsilon$ need to be really small to be able to approximate the derivative with slope.
In practice, we can first calculate the partial derivatives using back propagation, and compare it with the numerial gradients. If they are approximately the same, we can continue to use back propagation to perform gradient-based learning.
Also, we need to be aware that the computation cost for numeric gradient is high. Thus we should turn off this graident checking after we confirm the correctness of the back propagation.
Let's do the gradient checking by initializing a simple $353$ network:
Can you write Python code for it?

debug_nn = ThreeLayerNeuralNetwork(layer1_size=3, layer2_size=5, layer3_size=3, lamda=1)
debug_nn.grad_check()


I am working on the file "3_Neural_Network.ipynb".
The first task is:
Train the neural network with 10 nodes in the hidden layer:
Can you write Python code for it?

nn = ThreeLayerNeuralNetwork(layer1_size=X.shape[1], layer2_size=10, layer3_size=len(np.unique(y)), lamda=1)
nn = nn.fit(X_train, y_train)
y_predict = nn.predict(X_test)
print "True Values:      {}".format(y_test)
print "Predicted Values: {}".format(y_predict)
print "Prediction Accuracy: {:.2%}".format(np.mean((y_predict == y_test).astype(float)))


I am working on the file "3_Neural_Network.ipynb".
The first task is:
We can see that the difference between numerial and back-propagation-calculated gradient are very tiny. Thus we can clonclude that our back propagation is working correctly.
Let's now load the real data for demo:
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

X = iris['data']
y = iris['target']

print X.shape
print y.shape
print "Number of Classes: {}".format(len(np.unique(y)))


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
Boxplots
We had created simple boxplots such as the ones shown below. Now, let's plot multiple boxplots and see what they can tell us the distribution of variables across categories.
Can you write Python code for it?

# boxplot of a variable
sns.boxplot(y=market_df['Sales'])
plt.yscale('log')
plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
Note that, by default, seaborn plots the mean value across categories, though you can plot the count, median, sum etc. Also, barplot computes and shows the confidence interval of the mean as well.
Can you write Python code for it?

# Create 2 subplots for mean and median respectively

# increase figure size
plt.figure(figsize=(12, 6))

# subplot 1: statistic=mean
plt.subplot(1, 2, 1)
sns.barplot(x='Product_Category', y='Sales', data=df)
plt.title("Average Sales")

# subplot 2: statistic=median
plt.subplot(1, 2, 2)
sns.barplot(x='Product_Category', y='Sales', data=df, estimator=np.median)
plt.title("Median Sales")

plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
Now that we've compared Sales and Profits across product categories, let's drill down further and  do the same across another categorical variable - Customer_Segment.
We'll need to add the customer-related attributes (dimensions) to this dataframe.
Can you write Python code for it?

# merging with customers df
df = pd.merge(df, customer_df, how='inner', on='Cust_id')
df.head()

# boxplot of a variable across various product categories
sns.boxplot(x='Customer_Segment', y='Profit', data=df)
plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
Now, let's say you want to compare the (distribution of) sales of various product categories. Let's first merge the product data into the main dataframe.
Can you write Python code for it?

# merge the dataframe to add a categorical variable
df = pd.merge(market_df, product_df, how='inner', on='Prod_id')
df.head()

# boxplot of a variable across various product categories
sns.boxplot(x='Product_Category', y='Sales', data=df)
plt.yscale('log')
plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
So this tells you that the sales of office supplies are, on an average, lower than the other two categories. The sales of technology and furniture categories seem much better. Note that each order can have multiple units of products sold, so Sales being higher/lower may be due to price per unit or the number of units.
Let's now plot the other important variable - Profit.
Can you write Python code for it?

# boxplot of a variable across various product categories
sns.boxplot(x='Product_Category', y='Profit', data=df)
plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
You can see that though the category 'Technology' has better sales numbers than others, it is also the one where the most loss making transactions happen. You can drill further down into this.
Can you write Python code for it?

# adjust figure size
plt.figure(figsize=(10, 8))

# subplot 1: Sales
plt.subplot(1, 2, 1)
sns.boxplot(x='Product_Category', y='Sales', data=df)
plt.title("Sales")
plt.yscale('log')

# subplot 2: Profit
plt.subplot(1, 2, 2)
sns.boxplot(x='Product_Category', y='Profit', data=df)
plt.title("Profit")

plt.show()


I am working on the file "3_Plotting_Categorical_Data.ipynb".
The first task is:
You can visualise the distribution across two categorical variables using the hue= argument.
Can you write Python code for it?

# set figure size for larger figure
plt.figure(num=None, figsize=(12, 8), dpi=80, facecolor='w', edgecolor='k')

# specify hue="categorical_variable"
sns.boxplot(x='Customer_Segment', y='Profit', hue="Product_Category", data=df)
plt.show()


I am working on the file "3_Position_and_Label_Based_Indexing.ipynb".
The first task is:
As mentioned in the documentation, the inputs x, y to df.iloc[x, y] can be:
* An integer, e.g. 3
* A list or array of integers, e.g. [3, 7, 8]
* An integer range, i.e. 3:8
* A boolean array
Let's see some examples.
Can you write Python code for it?

# Selecting a single element
# Note that 2, 4 corresponds to the third row and fifth column (Sales)
market_df.iloc[2, 4]


I am working on the file "3_Position_and_Label_Based_Indexing.ipynb".
The first task is:
As mentioned in the documentation, the inputs x, y to df.loc[x, y] can be:
* A single label, e.g. '3' or 'row_index'
* A list or array of labels, e.g. ['3', '7', '8']
* A range of labels, where row_x and row_y both are included, i.e. 'row_x':'row_y'
* A boolean array <br>
Let's see some examples.
Can you write Python code for it?

# Selecting a single element
# Select row label = 2 and column label = 'Sales
market_df.loc[2, 'Sales']

# Selecting a single row using a single label
# df.loc reads 5 as a label, not index
market_df.loc[5]

# or equivalently
market_df.loc[5, :]

# or market_df.loc[5, ]

# Select multiple rows using a list of row labels
market_df.loc[[3, 7, 8]]

# Or equivalently
market_df.loc[[3, 7, 8], :]

# Selecting rows using a range of labels
# Notice that with df.loc, both 4 and 8 are included, unlike with df.iloc
# This is an important difference between iloc and loc
market_df.loc[4:8]

# Or equivalently
market_df.loc[4:8, ]

# Or equivalently
market_df.loc[4:8, :]

# The use of label based indexing will be more clear when we have custom row indices
# Let's change the indices to Ord_id
market_df.set_index('Ord_id', inplace = True)
market_df.head()

# Select Ord_id = Ord_5406 and some columns
market_df.loc['Ord_5406', ['Sales', 'Profit', 'Cust_id']]

# Select multiple orders using labels, and some columns
market_df.loc[['Ord_5406', 'Ord_5446', 'Ord_5485'], 'Sales':'Profit']

# Using booleans
# This selects the rows corresponding to True
market_df.loc[[True, True, False, True, True, False, True]]


I am working on the file "3_Position_and_Label_Based_Indexing.ipynb".
The first task is:
Note that simply writing df[2, 4] will throw an error, since pandas gets confused whether the 2 is an integer index (the third row), or is it a row with label = 2?
On the other hand, df.iloc[2, 4] tells pandas explicitly that it should assume integer indices.
Can you write Python code for it?

# Selecting a single row, and all columns
# Select the 6th row, with label (and index) = 5
market_df.iloc[5]

# The above is equivalent to this
# The ":" indicates "all rows/columns"
market_df.iloc[5, :]

# equivalent to market_df.iloc[5, ]

# Select multiple rows using a list of indices
market_df.iloc[[3, 7, 8]]

# Equivalently, you can use:
market_df.iloc[[3, 7, 8], :]

# same as market_df.iloc[[3, 7, 8], ]

# Selecting rows using a range of integer indices
# Notice that 4 is included, 8 is not
market_df.iloc[4:8]

# or equivalently
market_df.iloc[4:8, :]

# or market_df.iloc[4:8, ]

# Selecting a single column
# Notice that the column index starts at 0, and 2 represents the third column (Cust_id)
market_df.iloc[:, 2]

# Selecting multiple columns
market_df.iloc[:, 3:8]

# Selecting multiple rows and columns
market_df.iloc[3:6, 2:5]

# Using booleans
# This selects the rows corresponding to True
market_df.iloc[[True, True, False, True, True, False, True]]


I am working on the file "3_Position_and_Label_Based_Indexing.ipynb".
The first task is:
Position (Integer) Based Indexing
Pandas provides the df.iloc functionality to index dataframes using integer indices.
Can you write Python code for it?


I am working on the file "3_Position_and_Label_Based_Indexing.ipynb".
The first task is:
To summarise, df.iloc[x, y] uses integer indices starting at 0.
The other common way of indexing is the label based indexing, which uses df.loc[].
Label Based Indexing
Pandas provides the df.loc[] functionality to index dataframes using labels.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_Equatorial_Coordinates.ipynb".
The first task is:
3.1 Equatorial Coordinates (RA,DEC)
3.1.1 The Celestial Sphere
We can use a geographical coordinate system to uniquely identify a position on earth. We normally use the coordinates latitude $L_a$ (to measure north and south) and longitude $L_o$ (to measure east and west) to accomplish this. The equatorial coordinate system is  depicted in Fig. 3.1.1 ⤵.

Figure 3.1.1: The geographical coordinates latitude $L_a$ and longitude $L_o$.
We also require a coordinate system to map the celestial objects. For all intents and purposes we may think of our universe as being projected onto a sphere of arbitrary radius. This sphere surrounds the Earth and is known as the celestial sphere. This is not a true representation of our universe, but it is a very useful approximate astronomical construct. The celestial equator is obtained by projecting the equator of the earth onto the celestial sphere. The stars themselves do not move on the celestial sphere and therefore have a unique location on it. The Sun is an exception, it changes position in a periodic fashion during the year (as the Earth orbits around the Sun). The path it traverses on the celestial sphere is known as the ecliptic.
3.1.2 The NCP and SCP
The north celestial pole (NCP) is an important location on the celestial sphere and is obtained by projecting the north pole of the earth onto the celestial sphere. The star Polaris is very close to the NCP and serves as a reference when positioning a telescope.
The south celestial pole (SCP) is obtained in a similar way. The imaginary circle known as the celestial equator is in the same plane as the equator of the earth and is obtained by projecting the equator of the earth onto the celestial sphere. The southern hemisphere
counterpart of Polaris is Sigma Octanis.
We use a specific point on the celestial equator from which we measure the location of all other celestial objects. This point is known as the first point of Aries...
Can you write Python code for it?

arcturus = ephem.star('Arcturus')
arcturus.compute('2016/2/8',epoch=ephem.J2000)
print('J2000: RA:%s DEC:%s' % (arcturus.ra, arcturus.dec))
arcturus.compute('2016/2/8', epoch=ephem.B1950)
print('B1950: RA:%s DEC:%s' % (arcturus.a_ra, arcturus.a_dec))


I am working on the file "3_Positional_Astronomy/3_1_Equatorial_Coordinates.ipynb".
The first task is:
3.1.4 Example: The 408 MHz Haslam map
To finish things off, let's make sure that given the concepts we have learned in this section we are able to interpret a radio skymap correctly. We will be plotting and inspecting the Haslam 408 MHz map ⤴. We load the Haslam map with read_map and view it with cartview. These two functions form part of the healpy package ⤴.
Can you write Python code for it?

haslam = hp.read_map('../data/fits/haslam/lambda_haslam408_nofilt.fits')

matplotlib.rcParams.update({'font.size': 10})
proj_map = hp.cartview(haslam,coord=['G','C'], max=2e5, xsize=2000,return_projected_map=True,title="Haslam 408 MHz with no filtering",cbar=False)
hp.graticule()


I am working on the file "3_Positional_Astronomy/3_1_Equatorial_Coordinates.ipynb".
The first task is:
The cartview function also produces a projected map as a byproduct (it takes the form of a 2D numpy array). We can now replot this projected map using matplotlib (see Fig. 3.1.3 ⤵ <!--\ref{pos:fig:haslam_map}-->). We do so in the code snippet that follows.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_equatorial_coordinates.ipynb".
The first task is:
3.1 Equatorial Coordinates (RA,DEC)
3.1.1 The Celestial Sphere
We can use a geographical coordinate system to uniquely identify a position on earth. We normally use the coordinates latitude $L_a$ (to measure north and south) and longitude $L_o$ (to measure east and west) to accomplish this. The equatorial coordinate system is  depicted in Fig. 3.1.1 ⤵.

Figure 3.1.1: The geographical coordinates latitude $L_a$ and longitude $L_o$.
We also require a coordinate system to map the celestial objects. For all intents and purposes we may think of our universe as being projected onto a sphere of arbitrary radius. This sphere surrounds the Earth and is known as the celestial sphere. This is not a true representation of our universe, but it is a very useful approximate astronomical construct. The celestial equator is obtained by projecting the equator of the earth onto the celestial sphere. The stars themselves do not move on the celestial sphere and therefore have a unique location on it. The Sun is an exception, it changes position in a periodic fashion during the year (as the Earth orbits around the Sun). The path it traverses on the celestial sphere is known as the ecliptic.
3.1.2 The NCP and SCP
The north celestial pole (NCP) is an important location on the celestial sphere and is obtained by projecting the north pole of the earth onto the celestial sphere. The star Polaris is very close to the NCP and serves as a reference when positioning a telescope.
The south celestial pole (SCP) is obtained in a similar way. The imaginary circle known as the celestial equator is in the same plane as the equator of the earth and is obtained by projecting the equator of the earth onto the celestial sphere. The southern hemisphere
counterpart of Polaris is KT:GM: Do you want to add Sigma Octanis to the Glossary? Sigma Octanis.
We use a specific point on the celestial equator from which we...
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_equatorial_coordinates.ipynb".
The first task is:
Equatorial Coordinates (RA/DEC)
The Celestial Sphere
We can use a geographical coordinate system to uniquely identify a position on earth. We normally use the coordinates latitude $L_a$ (to measure north and south) and longitude $L_o$ (to measure east and west) to accomplish this. The equatorial coordinate system is  depicted in Figure 3.1.1.
</a> <img src='figures/geo.svg' width=60%>
Figure 3.1.1: The geographical coordinates latitude $L_a$ and longitude $L_o$.
We also require a coordinate system to map the celestial objects. For all intents and purposes we may think of our universe as being projected onto a sphere of arbitrary radius. This sphere surrounds the Earth and is known as the celestial sphere. This is not a true representation of our universe, but it is a very useful approximate astronomical construct. The celestial equator is obtained by projecting the equator of the earth onto the celestial sphere. The stars themselves do not move on the celestial sphere and therefore have a unique location on it. The Sun is an exception, it changes position in a periodic fashion during the year (as the Earth orbits around the Sun). The path it traverses on the celestial sphere is known as the ecliptic.
The NCP and SCP
The north celestial pole (NCP) is an important location on the celestial sphere and is obtained by projecting the north pole of the earth onto the celestial sphere. The star Polaris is very close to the NCP and serves as a reference when positioning a telescope.
The south celestial pole (SCP) is obtained in a similar way. The imaginary circle known as the celestial equator is in the same plane as the equator of the earth and is obtained by projecting the equator of the earth onto the celestial sphere. The southern hemisphere counterpart of Polaris is Sigma Octanis.
We use a specific point on the celestial equator from which we measure the location of all other celestial objects. This point is known as the first point of Aries ($\gamma$) <!--\vernal--> or the vernal equinox. The vernal equinox is...
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_equatorial_coordinates.ipynb".
The first task is:
Example: The 408 MHz Haslam Map
To finish things off, let's make sure that given the concepts we have learned in this section we are able to interpret a radio skymap correctly. We will be plotting and inspecting the Haslam 408 MHz map ⤴. We load the Haslam map with read_map and view it with cartview. These two functions form part of the healpy package ⤴.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_equatorial_coordinates.ipynb".
The first task is:
The cartview function also produces a projected map as a byproduct (it takes the form of a 2D numpy array). We can now replot this projected map using matplotlib (see  ). We do so in the code snippet that follows.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_1_equatorial_coordinates.ipynb".
The first task is:
The cartview function also produces a projected map as a byproduct (it takes the form of a 2D numpy array). We can now replot this projected map using matplotlib (see Fig. 3.1.3 ⤵ <!--\ref{pos:fig:haslam_map}-->). We do so in the code snippet that follows.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_2_Hour_Angle.ipynb".
The first task is:
3.2 Hour Angle (HA) and Local Sidereal Time (LST)
As we already mentioned in the chapter introduction, the static equatorial coordinates right ascension and declination are not enough if one wants to pin point the location of an object in the night sky from earth. We introduce two concepts here (namely hour angle and local sidereal time) which together with the static equatorial coordinates allow an observer to track celestial objects from earth.
3.3.1 Hour Angle (HA)
Zenith is the position on the celestial sphere which lies directly above an observer on earth. Nadir is the position on the celestial sphere which lies directly below an observer on earth. The local meridian is the hour circle on the celestial sphere which we form when we connect the NCP with zenith. Celestial objects rise in the east reach their highest point when they cross the local meridian (we say they are at transit) and set in the west.

Note: The *hour angle* (denoted with $H$) of a celestial body is the angular distance (measured in hours) between the hour circle of a celesteal object and the local meridian measured along the celestial equator in a westerly direction.

In other words, the hour angle is the time that still needs to pass before a celestial body is at transit if it is negative or the time since it was at transit if it is positive. We can therefore use it instead of right ascension as a celestial longitude to keep track of the stars as they move accros the sky of an earth bound observer.
3.3.2 Local Sidereal Time (LST)
Our wrist watches keep track of when the sun will be at transit. The solar time-keeping system we use ensures that our working day aligns with the rising and setting of the sun. Astronomers are however more intereseted in keeping track of the stars, since they want to observe them. They therefore use a sidereal time-keeping system to keep track of the vernal equinox...
Can you write Python code for it?

#Setting up the observer
JB = ephem.Observer()
JB.lat = '53:14:10'
JB.lon = '-02:18:26'
JB.elevation = 0.0

months = ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]
h_number = np.zeros((14,),dtype=float)

#Calculating the lst at differnt times of the year
for k in xrange(1,14):
    if k < 10:
       JB.date = '2016/'+str(k)+'/22  23:59:59'
       str_sid = str(JB.sidereal_time()).split(":")
       h_number[k-1] = float(str_sid[0]) + float(str_sid[1])/60. + float(str_sid[2])/3600.
    elif k > 10:
       JB.date = '2016/'+str(k-1)+'/22  23:59:59'
       str_sid = str(JB.sidereal_time()).split(":")
       h_number[k-1] = float(str_sid[0]) + float(str_sid[1])/60. + float(str_sid[2])/3600

#Plot
matplotlib.rcParams.update({'font.size': 13.75})

fig, ax = plt.subplots()
h_number[-1] = h_number[0]
x = np.arange(14)
x[9:] = x[9:]-1

ax.plot(x,h_number)

ax.set_ylim([0,24])

ticks = np.array([0,2,4,6,8,10,12,14,16,18,20,22,24])
plt.yticks(ticks)

labels = [item.get_text() for item in ax.get_xticklabels()]
labels = np.array(["Jan","Mar","May","Jul","Sep","Nov","Jan"])
ax.set_xticklabels(labels)

labels = [item.get_text() for item in ax.get_yticklabels()]
labels = np.array(['$0^h$','$2^h$','$4^h$','$6^h$','$8^h$','$10^h$','$12^h$','$14^h$','$16^h$','$18^h$','$20^h$','$22^h$','$24^h$'])
ax.set_yticklabels(labels)

plt.grid('on')


I am working on the file "3_Positional_Astronomy/3_2_hour_angle.ipynb".
The first task is:
Hour Angle (HA) and Local Sidereal Time (LST)
As we already mentioned in the chapter introduction, the static equatorial coordinates right ascension and declination are not enough if one wants to pin point the location of an object in the night sky from earth. We introduce two concepts here (namely hour angle and local sidereal time) which together with the static equatorial coordinates allow an observer to track celestial objects from earth.
Hour Angle (HA)
Zenith is the position on the celestial sphere which lies directly above an observer on earth. Nadir is the position on the celestial sphere which lies directly below an observer on earth. The local meridian is the hour circle on the celestial sphere which we form when we connect the NCP with zenith. Celestial objects rise in the east reach their highest point when they cross the local meridian (we say they are at transit) and set in the west.

Note: The *hour angle* (denoted with $H$) of a celestial body is the angular distance (measured in hours) between the hour circle of a celestial object and the local meridian measured along the celestial equator in a westerly direction.

In other words, the hour angle is the time that still needs to pass before a celestial body is at transit if it is negative or the time since it was at transit if it is positive. We can therefore use it instead of right ascension as a celestial longitude to keep track of the stars as they move across the sky of an earth bound observer.
Local Sidereal Time (LST)
Our wrist watches keep track of when the sun will be at transit. The solar time-keeping system we use ensures that our working day aligns with the rising and setting of the sun. Astronomers are however more intereseted in keeping track of the stars, since they want to observe them. They therefore use a sidereal time-keeping system to keep track of the vernal equinox (instead of the sun). KT:XX: The switching between the Vernal Equinox and the First...
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_3_Horizontal_Coordinates.ipynb".
The first task is:
Figure 3.3.3: The altitude angle of ten well know stars during 2016/5/30 as observed by the KAT-7 array. The altitude angle was computed by employing pyephem. The peaks of the curves indicate the times at which the stars were at transit. The black rectangle represents the fundamental horizon. Any star that stays below the horizon would not be observable at all (see the curve associated with Thuban for an example). Any star that stays above the horizon for the entire day is a circumpolar star. Mimosa can almost be classified as a circumpolar star.
We have not yet plotted the azimuth coordinates for the randomly selected stars. We do so by using the code snippet below.
Can you write Python code for it?

#Plot az
matplotlib.rcParams.update({'font.size': 13.75})
fig, ax = plt.subplots()

c = ["r","b","g","y","m","c","k"]
l = ["-","--"]
l_ind = 0
c_ind = 0

for i in xrange(10):
    if c_ind == 7:
       c_ind = 0
       l_ind = 1
    plt.plot(time_v,alt_az_mat[i,:,1]*(180/np.pi),c[c_ind]+l[l_ind],lw=2,label=star_names[i])
    c_ind = c_ind +1

ax.legend()
ax.set_xlim([0,24])
ax.set_ylim([0,360])
ticks = np.array([0,60,120,180,240,300,360])
plt.yticks(ticks)
ticks = np.array([0,2,4,6,8,10,12,14,16,18,20,22,24])
plt.xticks(ticks)
plt.xlabel("UTC [$h$]")
plt.ylabel("Azimuth [$^{\circ}$]")
plt.title("KAT-7: 2016/5/30")

labels = [item.get_text() for item in ax.get_yticklabels()]
labels = np.array(["0$^{\circ}$","60$^{\circ}$","120$^{\circ}$","180$^{\circ}$","240$^{\circ}$","300$^{\circ}$","360$^{\circ}$"])
ax.set_yticklabels(labels)

ax.grid('on')


I am working on the file "3_Positional_Astronomy/3_3_Horizontal_Coordinates.ipynb".
The first task is:
Figure 3.3.4: The azimuth angle of ten well know stars during 2016/5/30 as observed by the KAT-7 array. The azimuth angle was computed by employing pyephem.
In the second use case we determine the horizontal coordinates of Betelgeuse
for different arrays around the world at a specific moment in time (2016/5/30 00:00:00). We again use pyephem to accomplish this. See the code snippet below for the exact details of how this can be achieved. We plot the main result of the code snippet in Fig. 3.3.5 ⤵ .
Can you write Python code for it?

#Preliminaries
matplotlib.rcParams.update({'font.size': 13.75})
observatories = ["LOFAR","KAT7","MWA","VLA","ALMA","GMRT"]
lat_v = ["52:54:32","-30:43:17","-26:42:12","34:04:43","-23:01:09","19:05:47"]
lon_v = ["06:52:08","21:25:40.08","116:40:16","-107:37:05","-67:45:12","74:02:59"]

alt_az = np.zeros((len(observatories),2),dtype=float)

#Loading different observatories and calculating alt/az of Betelgeuse for each of them
for k in xrange(len(observatories)):
    obs = ephem.Observer()
    obs.lat = lat_v[k]
    obs.lon = lon_v[k]
    obs.elevation = 0.0
    obs.date = '2016/5/30 00:00:00' #UTC
    betelgeuse = ephem.star("Betelgeuse",obs)
    alt_az[k,0] = float(betelgeuse.alt)
    alt_az[k,1] = float(betelgeuse.az)

#Plotting
cluster = ['o','^','>','s','*','v']
col = ['b','r','g','k','c','m']

fig, ax = plt.subplots()

for xp, yp, m, n, col_v in zip(alt_az[:,0]*(180/np.pi), alt_az[:,1]*(180/np.pi), cluster, observatories,col):
    ax.plot([xp],[yp], marker=m, c = col_v, label = n, markersize = 20, linestyle='None')

ax.legend(numpoints=1)
ax.set_xlim([-90,90])
ax.set_ylim([0,360])
ticks = np.array([0,60,120,180,240,300,360])
plt.yticks(ticks)
ticks = np.array([-90,-80,-70,-60,-50,-40,-30,-20,-10,0,10,20,30,40,50,60,70,80,90])
plt.xticks(ticks)

labels = [item.get_text() for item in ax.get_yticklabels()]
labels = np.array(["0$^{\circ}$","60$^{\circ}$","120$^{\circ}$","180$^{\circ}$","240$^{\circ}$","300$^{\circ}$","360$^{\circ}$"])
ax.set_yticklabels(labels)

labels = [item.get_text() for item in ax.get_xticklabels()]
labels = np.array(["-90$^{\circ}$","-80$^{\circ}$","-70$^{\circ}$","-60$^{\circ}$","-50$^{\circ}$","-40$^{\circ}$","-30$^{\circ}$","-20$^{\circ}$","-10$^{\circ}$","0$^{\circ}$","10$^{\circ}$","20$^{\circ}$","30$^{\circ}$","40$^{\circ}$","50$^{\circ}$","60$^{\circ}$","70$^{\circ}$","80$^{\circ}$","90$^{\circ}$"])
ax.set_xticklabels(labels)

plt.xlabel("Altitude [$^{\circ}$]")
plt.ylabel("Azimuth [$^{\circ}$]")
plt.title("Betelgeuse: 2016/5/30 - 00:00:00 UTC")

ax.grid('on')


I am working on the file "3_Positional_Astronomy/3_3_horizontal_coordinates.ipynb".
The first task is:
3.3 Horizontal Coordinates (ALT,AZ)
3.3.1 Coordinate Definitions
In $\S$ 3.2.1 ➞ we introduced the concept of an hour angle, which allows us to determine the time that still needs to elapse before a source crosses the local meridian. This however does not tell us where we should point a telescope on earth in order to observe a source with a specific hour angle. The horizontal coordinates azimuth $\mathcal{A}$ and altitude $\mathcal{E}$ (elevation) is used to enable an observer on earth to locate celestial objects in the observer's local sky. The observer's horizontal plane is the fundamental plane of this coordinate system and is known as the celestial horizon. The azimuth angle is measured in the celestial horizon from due north towards the east, while the altitude of a celestial object is the angle between it and the celestial horizon. Both azimuth and elevation are measured in degrees. The azimuth and elevation angle are depicted in  Fig. 3.3.1 ⤵

Figure 3.3.1: The horizontal coordinates. KT:XX: Azimuth and elevation symbols seem to have been exchanged in the figure.
The equations below allow us to convert between equatorial and horizontal coordinates

   Converting between equatorial and horizontal

\begin{eqnarray}
\cos\delta\cos H &=& \cos L_a\sin \mathcal{E} - \sin L_a\cos \mathcal{E}\cos \mathcal{A}\\
-\cos\delta\sin H&=& \cos \mathcal{E}\sin \mathcal{A}\\
\sin\delta &=& \sin L_a\sin \mathcal{E}+\cos L_a \cos \mathcal{E} \cos \mathcal{A}
\end{eqnarray}

Note: In the conversion equations above $L_a$ denotes latitude (see $\S$ 3.1 ➞).

The above equations were derived by applying the spherical trigonometric identities in <a...
Can you write Python code for it?

#Creating the observer: KAT-7
KAT7 = ephem.Observer()
KAT7.lat = '-30:43:17'
KAT7.lon = '21:25:40.08'
KAT7.elevation = 0.0
KAT7.date = '2016/5/30 00:00:00' #UTC

#Creating the celestial bodies
star_names = np.array(["Rigel","Thuban","Mimosa","Procyon","Sirius","Achernar","Menkar","Zaurak","Aldebaran","Betelgeuse"])
star_objects = np.empty((len(star_names),),dtype=object)
for k in xrange(len(star_names)):
    star_objects[k] = ephem.star(star_names[k],KAT7)

#Creating the time-strings at which we observe
hours = np.empty((96,),dtype=object)
minutes = np.empty((96,),dtype=object)
alt_az_mat = np.zeros((len(star_names),len(hours)+1,2),dtype=float) #(sources,hours,horz_coord)

hours_c = 0
for k in xrange(len(hours)):

    if k % 4 == 0:
       if hours_c < 10:
          hours[k] = '0'+str(hours_c)
       else:
          hours[k] = str(hours_c)
       minutes[k] = "00"
    elif k % 4 == 1:
       if hours_c < 10:
          hours[k] = '0'+str(hours_c)
       else:
          hours[k] = str(hours_c)
       minutes[k] = "15"
    elif k % 4 == 2:
       if hours_c < 10:
          hours[k] = '0'+str(hours_c)
       else:
          hours[k] = str(hours_c)
       minutes[k] = "30"
    elif k % 4 == 3:
       if hours_c < 10:
          hours[k] = '0'+str(hours_c)
       else:
          hours[k] = str(hours_c)
       hours_c = hours_c + 1
       minutes[k] = "45"

#Compute the alt/az for different stars observed by KAT-7 at different times on 2016/5/30
for k in xrange(len(hours)):

    #Set new time
    n_date = '2016/5/30 ' + hours[k] + ':' + minutes[k] + ':00'

    KAT7.date = n_date

    #Calculate new alt/az
    for j in xrange(len(star_names)):
        star_objects[j].compute(KAT7)
        alt_az_mat[j,k,0] = float(star_objects[j].alt)
        alt_az_mat[j,k,1] = float(star_objects[j].az)

#Copy first value to last value
alt_az_mat[:,-1,:] = alt_az_mat[:,0,:]
time_v = np.linspace(0,24,len(hours)+1,endpoint=True)

#Plot alt
matplotlib.rcParams.update({'font.size':...


I am working on the file "3_Positional_Astronomy/3_3_horizontal_coordinates.ipynb".
The first task is:
Figure 3.3.3: The altitude angle of ten well known stars during 2016/5/30 as observed by the KAT-7 array. The altitude angle was computed by employing pyephem. The peaks of the curves indicate the times at which the stars were at transit. The black rectangle represents the fundamental horizonKT:XX: What is the fundamental horizon? Do you mean the celestial horizon? . Any star that stays below the horizon would not be observable at all (see the curve associated with Thuban for an example). Any star that stays above the horizon for the entire day is a circumpolar star. Mimosa can almost be classified as a circumpolar star.
We have not yet plotted the azimuth coordinates for the randomly selected stars. We do so by using the code snippet below.
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_3_horizontal_coordinates.ipynb".
The first task is:
Horizontal Coordinates (ALT/AZ)
In   we introduced the concept of an hour angle, which allows us to determine the time that still needs to elapse before a source crosses the local meridian. This however does not tell us where we should point a telescope on earth in order to observe a source with a specific hour angle. The horizontal coordinates azimuth $\mathcal{A}$ and altitude $\mathcal{E}$ (elevation) is used to enable an observer on earth to locate celestial objects in the observer's local sky. The observer's horizontal plane is the fundamental plane of this coordinate system and is known as the celestial horizon. The azimuth angle is measured in the celestial horizon from due north towards the east, while the altitude of a celestial object is the angle between it and the celestial horizon. Both azimuth and elevation are measured in degrees. The azimuth and elevation angle are depicted in

Figure 3.3.1: The horizontal coordinates. KT:XX: Azimuth and elevation symbols seem to have been exchanged in the figure.
The equations below allow us to convert between equatorial and horizontal coordinates

   Converting between equatorial and horizontal

\begin{eqnarray}
\cos\delta\cos H &=& \cos L_a\sin \mathcal{E} - \sin L_a\cos \mathcal{E}\cos \mathcal{A}\\
-\cos\delta\sin H&=& \cos \mathcal{E}\sin \mathcal{A}\\
\sin\delta &=& \sin L_a\sin \mathcal{E}+\cos L_a \cos \mathcal{E} \cos \mathcal{A}
\end{eqnarray}

Note: In the conversion equations above $L_a$ denotes latitude (see  ).

The above equations were derived by applying the spherical trigonometric identities in <a...
Can you write Python code for it?


I am working on the file "3_Positional_Astronomy/3_3_horizontal_coordinates.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import HTML
HTML('../style/code_toggle.html')
import ephem
import matplotlib
%pylab inline
pylab.rcParams['figure.figsize'] = (15, 10)


I am working on the file "3_Positional_Astronomy/3_4_Direction_Cosine_Coordinates.ipynb".
The first task is:
3.4 Direction Cosine Coordinates ($l$,$m$,$n$)
There is another useful astronomical coordinate system that we ought to introduce at this juncture, namely the direction cosine coordinate system. The direction cosine coordinate system is quite powerful and allows us to redefine the fundamental reference point on the celestial sphere, from which we measure all other celestial objects, to an arbitrary location (i.e. we can make local sky-maps around our own chosen reference point; the vernal equinox need not be our fundamental reference point). Usually this arbitrary location is chosen to be the celestial source that we are interested in observing. We generally refer to this arbitrary location as the field center or phase centre.

Note: The direction cosine coordinate system is useful for another reason, when we use
it to image interferometric data, then it becomes evident that there exists a Fourier relationship between the sky brightness function and the measurements that an interferometer makes (see Chapter 4 ➞).

We use three coordinates in the direction cosine coordinate system, namely $l$, $m$ and $n$. The coordinates $l$, $m$ and $n$ are dimensionless direction cosines, i.e.
\begin{eqnarray}
l &=& \cos(\alpha) = \frac{a_1}{|\mathbf{a}|}\
m &=& \cos(\beta) = \frac{a_2}{|\mathbf{a}|}\
n &=& \cos(\gamma) = \frac{a_3}{|\mathbf{a}|}
\end{eqnarray}

Figure 3.4.1: Definition of direction cosines.
The quantities $\alpha$, $\beta$, $\gamma$, $a_1$, $a_2$, $a_3$ and $\mathbf{a}$ are all defined in Fig. 3.4.1 ⤵. Moreover, $|\cdot|$ denotes the magnitude of its operand. The definitions above also imply that $l^2+m^2+n^2 = 1$. When $|\mathbf{a}|=1$ then we may simply interpret $l$, $m$ and $n$ as   Cartesian coordinates, i.e. we may simply relabel the axes $x$, $y$ and $z$ (in Fig....
Can you write Python code for it?