I am working on the file "4_BVP/shooting_method.ipynb".
The first task is:
Ejemplo 4
Aplicar el shooting method al sistema no lineal de BVPs
$$
\begin{align}
y'_1 &= \frac{1}{y_2}\
y'_2 &= t+\tan(y_1)\
y_1(0) &= 0\
y_2(1) &= 2
\end{align}
$$
Notemos que en este caso no tenemos una ecuación de segundo orden, simplemente es un BVP (tenemos valores de fronteras para las funciones que buscamos). Nos interesaría encontrar alguna condición inicial faltante, en este caso $y_2(0)$, con lo cual obtendríamos el IVP que debemos encontrar.
Al ser un sistema de ecuaciones, no es necesario el cambio de variable. Simplemente el solver deberá contener los valores explícitos de las derivadas $y'_1$ e $y'_2$.
Ojo: El solver devuelve las estimaciones de $y_1$ e $y_2$. Anteriormente ignoramos ese hecho pues en ejercicios anteriores el solver devolvía estimaciones $y, v$, y nosotros descartábamos $v$ pues nos interesaba $y$.
Como la condición de borde en $t=b$ está dada para $y_2$, debemos comparar el resultado del solver para dicha función, es decir, necesitamos el retorno del solver completo (Ver el slicing realizado)
Can you write Python code for it?

# Condiciones de borde e intervalo de tiempo
a, b = (0, 1)
y2_1 = 2.0
t = np.linspace(a, b, 101)

def yprime(_y, t):
    # En este caso "y" y "v" representan a y1 e y2
    y1, y2 = _y
    return [1.0 / y2,  t + np.tan(y1)]

def F(s):
    # y1_0 = 0, y2_0 = s
    y0 = [0, s]
    sol = odeint(yprime, y0, t, mxstep=100)
    return sol[:,1][-1] - y2_1

print("F(0.9) = ", F(0.9))
print("F(1.1) = ", F(1.1))

# Solución
sbest = bisect(F, 0.9, 1.1)
print("sbest =",sbest)
print("F(sbest) = ", F(sbest))

yy = odeint(yprime, [0, sbest], t)

# Plot de resultados
plt.figure(figsize=(10, 7))
plt.plot(t, yy[:, 0], lw=2.0, label="$y_1$")
plt.plot(t, yy[:, 1], lw=2.0, label="$y_2$")

# Plot analitico
y1 = np.arctan(t)
y2 = t**2.0 + 1
plt.plot(t, y1,  "r--", lw=2.0, label="$y_1$ Analítico")
plt.plot(t, y2,  "m--", lw=2.0, label="$y_2$ Analítico")
plt.legend(loc='best', fontsize=20)
plt.xlabel("$t$", fontsize=20)
plt.ylabel("$y(t)$", fontsize=20)
plt.xlim([0, 1])
plt.show()


I am working on the file "4_BVP/shooting_method.ipynb".
The first task is:
Ejemplo: Encuentre la altura máxima para un proyectil lanzado desde 30 metros de altura y que alcanza el suelo en 4 segundos.
En este caso, el problema puede plantearse dadas las leyes de Newton de movimiento como:
$$
\begin{align}
y'' &= -g\
y(0) &= 30\
y(4) &= 0
\end{align}
$$
Resolvemos vía integración de $y''$ pues ésta depende únicamente de $t$.
$$
\begin{align}
y'(t) &= \int y''(t)\,dt = -gt + v_0\
y(t) &= \int -gt + v_0 \, dt = -\frac{1}{2}gt^2 + v_0t + y_0
\end{align}
$$
Las condiciones de borde nos permiten descubrir las constantes $v_0$ y $y_0$ pues $y(0) = y_0 = 30$ y $y(4) = -8g+v_0+30$, luego $v_0 \approx 12.12 $. Al obtener la solución analítica podemos descubrir la altura máxima del proyectil.
$$-gt + 12.12 = 0 \Rightarrow t \approx 1.236 \Rightarrow y_{max} \approx 37.5$$
Can you write Python code for it?

# Solución analítica para el BVP presentado
def y(t, v0, y0):
    return -0.5*9.8*t**2.0 + v0*t + y0

# Condiciones iniciales
a = 0
b = 4
y0 = 30
# Condiciones encontradas vía integración
v0 = 12.12
# Plot
tt = np.linspace(a, b, 100)
yy = y(tt, v0, y0)
plt.plot(tt, yy, lw=2.0)
plt.plot(1.236, 37.5, "ro", lw=2.0)
plt.xlabel("$t$", fontsize=20)
plt.ylabel("$y(t)$", fontsize=20)
plt.show()


I am working on the file "4_BVP/shooting_method.ipynb".
The first task is:
Shooting method
En general los problemas de valor inicial (IVP) tienen soluciones únicas. En cambio los problemas de valor de frontera (BVP) podrían no tener solución o tener infinitas soluciones. El shooting method es un método numérico para resolver BVP que trabaja bajo la premisa de encontrar el IVP que tiene la misma solución.
Dada una pendiente o slope $s_a$ inicial y un valor inicial $y_a$ generamos un IVP con solución $\hat{y}(t)$ y vemos si dicha solución tentativa se acerca a $y_b$. Por prueba y error iremos corrigiendo la pendiente $s_a$ hasta que la solución al IVP generado calce con $y_b$. El problema por lo tanto puede definirse como una función de $s$
$$
\begin{align}
F(s) = \text{Diferencia entre } y_b \text{ y } y(b) \text{, donde } y(t) \text{ es la solución del IVP con } y(a) = y_a \text{ y } y'(a) = s
\end{align}
$$

Como la función calcula una diferencia entre el valor de la condición de frontera y el valor de frontera del IVP, cuando ésta diferencia sea $0$ entonces habremos encontrado la solución al problema, es decir:
$$F(s) = 0$$
Por lo tanto nuestro problema de resolver BVP se reduce a una búsqueda de ceros para $F(s)$. Un método clásico para tal efecto es el método de bisección. Obviamente necesitamos dos pendientes $s_0$ y $s_1$ para establecer un intervalo con cambio de signo y buscar $s$ acá. Finalmente podemos buscar la solución final para el BVP como el IVP relacionado a la pendiente $s$:
$$\begin{align}
y'' = f(t, y, y')\
y(a) = y_a\
y'(a) = s^*
\end{align}
$$
Ejemplo 1
Aplique el shooting method al BVP siguiente $$\begin{align}y'' = 4y\y(0) = 1 \ y(1) = 3\end{align}$$
Solución: Como vimos antes, necesitamos tratar al BVP como un IVP e intentar encontrar la pendiente que calce con el IVP que resuelve el BVP. Se requiere por lo tanto un cambio de variable afín, para que aparezca como restricción explícitamente la pendiente. Sea $y' = v$, el problema cambia entonces a:
$$
\begin{align}v'=4y\y' = v
\end{align}
$$
Recordar que las condiciones iniciales deben ser $y(0) = 1$ y $y'(0)...
Can you write Python code for it?

from scipy.integrate import odeint
from scipy.optimize import bisect

# Límites de t, la condición de borde yb y un array con muchos valores de t entre a y b
a, b = (0, 1)
yb = 3.0
t = np.linspace(a, b, 101)

# Recordar cambio de variable!
# v' = 4y
# y' = v

# Función para el solver donde declaramos el IVP
def yprime(_y, t):
    y, v = _y
    return [v, 4*y]

# Función donde buscamos el cero, i.e., donde buscamos pillar la 2da condición de borde.
def F(s):
    # Condición inicial del IVP que iremos cambiando, y'(0) = s, y(a0) = 1
    y0 = [1, s]
    # y, v
    sol = odeint(yprime, y0, t)
    return sol[:,0][-1] - yb

# Chequear F(-1)F(0) < 0 para método de bisección

# -1, 0
print(F(-1))
print(F(0))
sbest = bisect(F, -1, 0)
print(F(sbest))

## Gráficos
# 1) Para comparar generamos soluciones a los IVP con las pendientes que sospechamos
y1 = odeint(yprime, [1, -1], t)
y2 = odeint(yprime, [1, 0], t)
# 2) La gráfica de nuestra solución con mejor pendiente
ybest = odeint(yprime, [1, sbest], t)
# 3) Cháchara plotística (o sea, graficar las funciones)
plt.figure(figsize=(10,7))
plt.plot(1, 3, 'ro', lw=3.0)
plt.plot(t, y1[:,0], 'g--', lw=2.0, label="$s=-1$")
plt.plot(t, y2[:,0], 'b--', lw=2.0, label="$s=0$")
plt.plot(t, ybest[:,0],'r-', lw=2.0, label="$s=s_{best}$")
plt.legend(loc='best', fontsize=20)
plt.xlabel("$t$", fontsize=20)
plt.ylabel("$y(t)$", fontsize=20)
plt.show()


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Answer: the prediction is very close to the true value, and the error doesn't tend to change in a non-random way as the true value changes. the high R^2 value confirms this intuition.
Implementation: Shuffle and Split Data
Your next implementation requires that you take the Boston housing dataset and split the data into training and testing subsets. Typically, the data is also shuffled into a random order when creating the training and testing subsets to remove any bias in the ordering of the dataset.
For the code cell below, you will need to implement the following:
- Use train_test_split from sklearn.cross_validation to shuffle and split the features and prices data into training and testing sets.
  - Split the data into 80% training and 20% testing.
  - Set the random_state for train_test_split to a value of your choice. This ensures results are consistent.
- Assign the train and testing splits to X_train, X_test, y_train, and y_test.
Can you write Python code for it?

# TODO: Import 'train_test_split'
from sklearn.cross_validation import train_test_split

# TODO: Shuffle and split the data into training and testing subsets
X_train, X_test, y_train, y_test = train_test_split(features, prices, test_size = 0.2,
                                                   random_state = 42)

# Success
print "Training and testing split was successful."


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Data Exploration
In this first section of this project, you will make a cursory investigation about the Boston housing data and provide your observations. Familiarizing yourself with the data through an explorative process is a fundamental practice to help you better understand and justify your results.
Since the main goal of this project is to construct a working model which has the capability of predicting the value of houses, we will need to separate the dataset into features and the target variable. The features, 'RM', 'LSTAT', and 'PTRATIO', give us quantitative information about each data point. The target variable, 'MEDV', will be the variable we seek to predict. These are stored in features and prices, respectively.
Implementation: Calculate Statistics
For your very first coding implementation, you will calculate descriptive statistics about the Boston housing prices. Since numpy has already been imported for you, use this library to perform the necessary calculations. These statistics will be extremely important later on to analyze various prediction results from the constructed model.
In the code cell below, you will need to implement the following:
- Calculate the minimum, maximum, mean, median, and standard deviation of 'MEDV', which is stored in prices.
  - Store each calculation in their respective variable.
Can you write Python code for it?

# TODO: Minimum price of the data
minimum_price = np.min(prices)

# TODO: Maximum price of the data
maximum_price = np.max(prices)

# TODO: Mean price of the data
mean_price = np.mean(prices)

# TODO: Median price of the data
median_price = np.median(prices)

# TODO: Standard deviation of prices of the data
std_price = np.std(prices)

# Show the calculated statistics
print "Statistics for Boston housing dataset:\n"
print "Minimum price: ${:,.2f}".format(minimum_price)
print "Maximum price: ${:,.2f}".format(maximum_price)
print "Mean price: ${:,.2f}".format(mean_price)
print "Median price ${:,.2f}".format(median_price)
print "Standard deviation of prices: ${:,.2f}".format(std_price)


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Developing a Model
In this second section of the project, you will develop the tools and techniques necessary for a model to make a prediction. Being able to make accurate evaluations of each model's performance through the use of these tools and techniques helps to greatly reinforce the confidence in your predictions.
Implementation: Define a Performance Metric
It is difficult to measure the quality of a given model without quantifying its performance over training and testing. This is typically done using some type of performance metric, whether it is through calculating some type of error, the goodness of fit, or some other useful measurement. For this project, you will be calculating the coefficient of determination, R2, to quantify your model's performance. The coefficient of determination for a model is a useful statistic in regression analysis, as it often describes how "good" that model is at making predictions.
The values for R2 range from 0 to 1, which captures the percentage of squared correlation between the predicted and actual values of the target variable. A model with an R2 of 0 always fails to predict the target variable, whereas a model with an R2 of 1 perfectly predicts the target variable. Any value between 0 and 1 indicates what percentage of the target variable, using this model, can be explained by the features. A model can be given a negative R2 as well, which indicates that the model is no better than one that naively predicts the mean of the target variable.
For the performance_metric function in the code cell below, you will need to implement the following:
- Use r2_score from sklearn.metrics to perform a performance calculation between y_true and y_predict.
- Assign the performance score to the score variable.
Can you write Python code for it?

from sklearn.metrics import r2_score

def performance_metric(y_true, y_predict):
    """ Calculates and returns the performance score between
        true and predicted values based on the metric chosen. """

    # TODO: Calculate the performance score between 'y_true' and 'y_predict'
    score = r2_score(y_true, y_predict)

    # Return the score
    return score


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Question 1 - Feature Observation
As a reminder, we are using three features from the Boston housing dataset: 'RM', 'LSTAT', and 'PTRATIO'. For each data point (neighborhood):
- 'RM' is the average number of rooms among homes in the neighborhood.
- 'LSTAT' is the percentage of homeowners in the neighborhood considered "lower class" (working poor).
- 'PTRATIO' is the ratio of students to teachers in primary and secondary schools in the neighborhood.
Using your intuition, for each of the three features above, do you think that an increase in the value of that feature would lead to an increase in the value of 'MEDV' or a decrease in the value of 'MEDV'? Justify your answer for each.
Hint: Would you expect a home that has an 'RM' value of 6 be worth more or less than a home that has an 'RM' value of 7?
Answer:
I would expect that RM would have a positive correlation with MEDV, while both LSTAT and PTRATIO would have a negative correlation.
Generally, larger houses have more rooms and more expensive.
Generally, lower class homes are less expensive, and school conditions in poor neighbourhoods tend to be worse.
Can you write Python code for it?

# Code for Review feedback
import matplotlib.pyplot as plt
import numpy as np

for col in features.columns:

    fig, ax = plt.subplots()
    fit = np.polyfit(features [col], prices, deg=1) # We use a linear fit to compute the trendline
    ax.scatter(features [col],  prices)
    plt.plot(features [col], prices, 'o', color='black')
    ax.plot(features[col], fit[0] * features[col] + fit[1], color='blue', linewidth=3) # This plots a trendline with the regression parameters computed earlier. We should plot this after the dots or it will be covered by the dots themselves
    plt.title('PRICES vs  '+ str(col)) # title here
    plt.xlabel(col) # label here
    plt.ylabel('PRICES') # label here


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Question 2 - Goodness of Fit
Assume that a dataset contains five data points and a model made the following predictions for the target variable:
| True Value | Prediction |
| :-------------: | :--------: |
| 3.0 | 2.5 |
| -0.5 | 0.0 |
| 2.0 | 2.1 |
| 7.0 | 7.8 |
| 4.2 | 5.3 |
Would you consider this model to have successfully captured the variation of the target variable? Why or why not?
Run the code cell below to use the performance_metric function and calculate this model's coefficient of determination.
Can you write Python code for it?

# Calculate the performance of this model
score = performance_metric([3, -0.5, 2, 7, 4.2], [2.5, 0.0, 2.1, 7.8, 5.3])
print "Model has a coefficient of determination, R^2, of {:.3f}.".format(score)


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Question 3 - Training and Testing
What is the benefit to splitting a dataset into some ratio of training and testing subsets for a learning algorithm?
Hint: What could go wrong with not having a way to test your model?
Answer:  splitting a dataset helps ensure less variance in the model by reducing the likelihood of overfitting. it does this by giving us an independent dataset on which the model hasn't learned from to test our model. without this, we would likely overfit the model and it would not generalize well. however, it's important to note than simply splitting data into test and train may not be enough, as we can succumb to issues such as p-hacking, where we essentially inadverdently overfit our model to the test data.

Analyzing Model Performance
In this third section of the project, you'll take a look at several models' learning and testing performances on various subsets of training data. Additionally, you'll investigate one particular algorithm with an increasing 'max_depth' parameter on the full training set to observe how model complexity affects performance. Graphing your model's performance based on varying criteria can be beneficial in the analysis process, such as visualizing behavior that may not have been apparent from the results alone.
Learning Curves
The following code cell produces four graphs for a decision tree model with different maximum depths. Each graph visualizes the learning curves of the model for both training and testing as the size of the training set is increased. Note that the shaded region of a learning curve denotes the uncertainty of that curve (measured as the standard deviation). The model is scored on both the training and testing sets using R2, the coefficient of determination.
Run the code cell below and use these graphs to answer the following question.
Can you write Python code for it?

len(features)

# Produce learning curves for varying training set sizes and maximum depths
vs.ModelLearning(features, prices)


I am working on the file "4_Boston_Housing/boston_housing.ipynb".
The first task is:
Question 5 - Bias-Variance Tradeoff
When the model is trained with a maximum depth of 1, does the model suffer from high bias or from high variance? How about when the model is trained with a maximum depth of 10? What visual cues in the graph justify your conclusions?
Hint: How do you know when a model is suffering from high bias or high variance?
Answer:  When the model is trained with a max depth of 1, the model suffers from high-bias, as the model is not complex enough to accurately model the training data, as the training score converges with the test data and shrinks as additional data is provided, and the score is not very high.
When the model is trained with a max-depth of 10, it suffers from high-variance, as the training score remains very high and does not change, while the test score does not converge closely.
Question 6 - Best-Guess Optimal Model
Which maximum depth do you think results in a model that best generalizes to unseen data? What intuition lead you to this answer?
Answer:  max_depth = 4 as the test score is as the maximum there.

Evaluating Model Performance
In this final section of the project, you will construct a model and make a prediction on the client's feature set using an optimized model from fit_model.
Question 7 - Grid Search
What is the grid search technique and how it can be applied to optimize a learning algorithm?
Answer:  The grid search technique allows for the tuning of different algorithm parameters. For example, in a regression tree we can tune the cp or max_depth values. Using the grid search technique, we can explore the model
evaluation metric, such as R-squared, against these changing parameters to see how best to optimize a learning algorithm against test data.
Question 8 - Cross-Validation
What is the k-fold cross-validation training technique? What benefit does this technique provide for grid search when optimizing a model?
Hint: Much like the reasoning behind having a testing set, what could go wrong with using grid search without a cross-validated...
Can you write Python code for it?

# TODO: Import 'make_scorer', 'DecisionTreeRegressor', and 'GridSearchCV'
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import make_scorer
from sklearn.grid_search import GridSearchCV

def fit_model(X, y):
    """ Performs grid search over the 'max_depth' parameter for a
        decision tree regressor trained on the input data [X, y]. """

    # Create cross-validation sets from the training data
    cv_sets = ShuffleSplit(X.shape[0], n_iter = 10, test_size = 0.20, random_state = 0)

    # TODO: Create a decision tree regressor object
    regressor = DecisionTreeRegressor()

    # TODO: Create a dictionary for the parameter 'max_depth' with a range from 1 to 10
    params = {'max_depth': range(1,11)}

    # TODO: Transform 'performance_metric' into a scoring function using 'make_scorer'
    scoring_fnc = make_scorer(performance_metric)

    # TODO: Create the grid search object
    grid = GridSearchCV(regressor, params, scoring_fnc, cv=cv_sets)

    # Fit the grid search object to the data to compute the optimal model
    grid = grid.fit(X, y)

    # Return the optimal model after fitting the data
    return grid.best_estimator_


I am working on the file "4_Cython.ipynb".
The first task is:
Pure Python
Can you write Python code for it?

def mandelbrot_python(m, size, iterations):
    for i in range(size):
        for j in range(size):
            c = -2 + 3./size*j + 1j*(1.5-3./size*i)
            z = 0
            for n in range(iterations):
                if np.abs(z) <= 10:
                    z = z*z + c
                    m[i, j] = n
                else:
                    break

%%timeit -n1 -r1 m = np.zeros((size, size))
mandelbrot_python(m, size, iterations)


I am working on the file "4_Cython.ipynb".
The first task is:
Take 1
First, we just add the %%cython magic.
Can you write Python code for it?

%%cython -a
import numpy as np

def mandelbrot_cython(m, size, iterations):
    for i in range(size):
        for j in range(size):
            c = -2 + 3./size*j + 1j*(1.5-3./size*i)
            z = 0
            for n in range(iterations):
                if np.abs(z) <= 10:
                    z = z*z + c
                    m[i, j] = n
                else:
                    break

%%timeit -n1 -r1 m = np.zeros((size, size), dtype=np.int32)
mandelbrot_cython(m, size, iterations)


I am working on the file "4_K_Nearest_Neighbors.ipynb".
The first task is:
Again, let's split the entire data into training and test sets:
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.025, random_state=26)


I am working on the file "4_K_Nearest_Neighbors.ipynb".
The first task is:
Let's first look at a demo for classification problem.
The Iris data set with three differe classes will be loaded here:
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

X = iris['data']
y = iris['target']

print X.shape
print y.shape
print "Number of Classes: {}".format(len(np.unique(y)))


I am working on the file "4_K_Nearest_Neighbors.ipynb".
The first task is:
Now we will utilize the KNearestNeighbors() class again to train the model and predict for test examples:
Can you write Python code for it?

knn = KNearestNeighbors(k=20, model_type='regression', weights='uniform')
knn.fit(X_train, y_train)
y_predict = knn.predict(X_test)
print "True Values:\n{}".format([round(elem, 1) for elem in y_test])
print "Predicted Values:\n{}".format([round(elem, 1) for elem in y_predict])
print "RMSE is {:.4}".format(np.sqrt(np.mean((y_test.reshape((len(y_test), 1)) - y_predict) ** 2)))


I am working on the file "4_K_Nearest_Neighbors.ipynb".
The first task is:
We will define an object of the KNearestNeighbors() class to train the model and predict for test examples:
Can you write Python code for it?

knn = KNearestNeighbors(k=3, model_type='classification', weights='uniform')
knn = knn.fit(X_train, y_train)
y_predict = knn.predict(X_test)
print "True Values:      {}".format(y_test)
print "Predicted Values: {}".format(y_predict)
print "Prediction Accuracy: {:.2%}".format(np.mean((y_predict == y_test).astype(float)))


I am working on the file "4_Least_Squares.ipynb".
The first task is:
4.1 Least Squares and the normal equations
$$
A^TA\bar{X} = A^Tb
$$
for the least squares solution $\bar{x}$ that minimizes the Euclidean length if the residual $r = b -Ax$
Example
Solve the least squares problem
$$
\begin{bmatrix}
1 & -4 \
2 & 3  \
2 & 2
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2
\end{bmatrix}
=
\begin{bmatrix}
-3 \
15 \
9
\end{bmatrix}
$$
Can you write Python code for it?


I am working on the file "4_Least_Squares.ipynb".
The first task is:
4.5 Nonlinear Least Squares
Gauss-Newton Method
Example
Consider the three circles in the plane with centers $(x_1,y_1) = (-1,0)$,$(x_2,y_2) = (1,1/2)$,$(x_3,y_3) = (1,-1/2)$

and radii $R_1 = 1,R_2 = 1/2,R3 = 1/2$,respectively.

Use the Gauss-Newton Method to find the point for which the sum of the squared distances to the three circles is miniminzed.
Can you write Python code for it?

def R_xy(x, y):
    A = np.zeros(3)
    f = lambda xf, yf, R : math.sqrt(pow(x - xf, 2) + pow(y - yf, 2)) - R
    A[0] = f(-1,   0, 1)
    A[1] = f( 1, 0.5, 0.5)
    A[2] = f( 1,-0.5, 0.5)
    return A

def DR_xy(x, y):
    A = np.zeros(6).reshape(3, 2)
    fx = lambda xf, yf : (x - xf) / math.sqrt(pow(x - xf, 2) + pow(y - yf, 2))
    fy = lambda xf, yf : (y - yf) / math.sqrt(pow(x - xf, 2) + pow(y - yf, 2))
    A[0][0] = fx(-1, 0)
    A[0][1] = fy(-1, 0)
    A[1][0] = fx(1, 0.5)
    A[1][1] = fy(1, 0.5)
    A[2][0] = fx(1, -0.5)
    A[2][1] = fy(1, -0.5)
    return A

def gauss_newton_method(x0, y0, k):

    xk = np.array([x0, y0])

    for _ in range(k):
        x = xk[0]
        y = xk[1]
        A = DR_xy(x, y)
        r = R_xy(x, y)
        v = np.matmul(linalg.inv(np.matmul(A.T, A)), -np.matmul(A.T, r))
        xk = xk + v

    return xk

x = gauss_newton_method(0, 0, 8)
print('x = %s' %x)


I am working on the file "4_Least_Squares.ipynb".
The first task is:
Example
Find the full QR factorization of
$
A =
\begin{bmatrix}
1 & -4 \
2 & 3  \
2 & 2
\end{bmatrix}
$
Can you write Python code for it?

A = np.array([1, -4, 2, 3, 2, 2]).reshape(3, 2)
Q, R = linalg.qr(A)
print('Q =')
print(Q)
print('R =')
print(R)


I am working on the file "4_Least_Squares.ipynb".
The first task is:
Example
Find the reduced QR factorization by applying Gram-Schmidt orthogonalization to the columns of
$
A =
\begin{bmatrix}
1 & -4 \
2 & 3  \
2 & 2
\end{bmatrix}
$
Can you write Python code for it?

A = np.array([1, -4, 2, 3, 2, 2]).reshape(3, 2)
Q, R = classical_gram_schmidt_orthogonalization(A)
print('Q =')
print(Q)
print('R =')
print(R)


I am working on the file "4_Least_Squares.ipynb".
The first task is:
Householder reflector
Example
Let $x = [3,4]$ and $w =[5,0]$. Find a householder reflector H that satisfies $Hx = w$.
Can you write Python code for it?

x = np.array([3, 4]).reshape(2, 1)
w = np.array([5, 0]).reshape(2, 1)
v = w - x

# Projection matrix
P = np.matmul(v, v.T) / np.matmul(v.T, v)

# Householder reflector
H = np.identity(P.shape[0]) - 2 * P

print('H=\n', H)

def householder_reflector(x):
    w = np.zeros(x.size)
    w[0] = linalg.norm(x, 2)
    v = (w - x).reshape(x.size, 1)

    # Projection matrix
    P = np.matmul(v, v.T) / np.matmul(v.T, v)

    # Householder reflector
    H = np.identity(P.shape[0]) - 2 * P

    return H


I am working on the file "4_Least_Squares.ipynb".
The first task is:
Modified Gram-Schmidt orthogonalization
Can you write Python code for it?

def modified_gram_schmidt_orthogonalization(A):
    Q = np.zeros(A.size).reshape(A.shape)
    R = np.zeros(A.shape[1] ** 2).reshape(A.shape[1], A.shape[1])
    for j in range(A.shape[1]):
        y = A[:,j]
        for i in range(j):
            R[i][j] = np.matmul(Q[:,i], y)
            y = y - R[i][j] * Q[:,i]
        R[j][j] = linalg.norm(y, 2)
        Q[:,j] = y / R[j][j]
    return Q, R


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
(We will go deeper into indexing in a while)

A NumPy array has a number of dimensions (or axes). To obtain the number of axes and the size of each of them you use the command shape. For 2-dimensional arrays (matrices), the order corresponds to (rows, columns)
There are two different ways of calling the shape command, either with np.shape(arr) or arr.shape. This is not the only command that works in both formats, and we will be finding some more in our way.
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
A very useful tool is conditional indexing, where we apply a function, assignment... only to those elements of an array that satisfy some condition
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Also (and obviously) these symbols can be used to operate between two arrays, which must be of the same shape. If this is the case, they also do element-wise operations
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Array Methods
To not have to go from NumPy arrays to lists back and forth, NumPy contains some functions to know properties of your arrays. Actually, there are more of these functions than in standard Python.
Can you write Python code for it?

a = np.array([-4, -2, 1, 3, 5])
print(a.max())
print(a.min())
print(a.sum())
print(a.mean())
print(a.std())


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
As you might have noticed, it is a bit ugly to deal with large dataframes. There are however some functions that allows to have an idea of the data in a frame.
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
But, as an addition, Pandas has special commands to load and save Excel spreadsheets (yay!). However, to use it you'll need the openpyxl and xlrd packages.
Can you write Python code for it?

u.to_excel('test.xlsx', sheet_name='My sheet')
pd.read_excel('test.xlsx', 'My sheet', index_col=0).head()


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Combining Arrays
The most general command for combining arrays is concatenate(arrs, d). It takes a list of arrays and concatenates them along axis $d$
Can you write Python code for it?

p = np.ones([2, 2, 2])
p

np.concatenate([p, 2 * p], 0)

np.concatenate([p, 2 * p], 1)

np.concatenate([p, 2 * p], 2)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Computing With DataFrames
You can calculate with DataFrames or their columns (which are Series) the same way you could with arrayss
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Copying Data
Be very careful with copying and modifying arrays in NumPy! You will see the reason right now. Let's begin defining r2 as a slice of r
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Exercise 2: Take a 10x2 matrix representing $(x1,x2)$ coordinates and transform them into polar coordinates $(r,\theta)$.
Hint 1: the inverse transformation is given by $x1 = r\cos\theta$, $x2 = r\sin\theta$
Hint 2: generate random numbers with the functions in numpy.random
Can you write Python code for it?

z = np.random.random((10, 2))
x1, x2 = z[:, 0], z[:, 1]
R = np.sqrt(x1 ** 2 + x2 ** 2)
T = np.arctan2(x2, x1)
print(R)
print(T)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Exercise 5: Download this dataset and load it, using the first column as the index. Take a look at it, and do the following things:
- Choose the columns 'Identifier', 'BaseStamina', 'BaseAttack', 'BaseDefense', 'Type1' and 'Type2'
- Create a function that lowercases strings and apply it to 'Type1' and 'Type2' (Extra: just capitalize the strings, i.e., leave the first letter uppercase and lowercase the rest)
- Create a function that returns a Boolean value (don't be afraif by this, it is a function that returns either True or False) that tells if a Pokémon has high stamina (BaseStamina>170) or not. Store this information in a new column and show the list of Pokémon with high stamina
- Show the instructor the last 15 rows of your dataset
Can you write Python code for it?

df = pd.read_csv('https://raw.githubusercontent.com/ChihChengLiang/pokemongor/master/data-raw/pokemons.csv',
                 index_col=0)

df = df[['Identifier', 'BaseStamina', 'BaseAttack', 'BaseDefense', 'Type1', 'Type2']]

capitalize = lambda st: st.capitalize()

for col in ['Type1', 'Type2']:
    df[col] = df[col].apply(capitalize)

def highstamina(x):
    return True if x > 170 else False

df['HighStamina'] = df.BaseStamina.apply(highstamina)

print(df[df['HighStamina'] == True].Identifier)

df.tail(15)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Grouping Data
Can you write Python code for it?

for h, group in u.groupby('G'):
    print('{}: {}'.format(h, np.mean(group['F'])))

u.groupby('G').describe()

u.pivot_table(index='G', aggfunc='mean')


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
However, there are a few different ways of accessing the data in a Pandas dataframe, that typically have a more "direct" connection with the actual content fo the dataframe. Individual or sets of columns can also be accessed by their column names. Choosing one single column will give a Series, while two or more will produce a Dataset
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
In the same way as with lists, you can use zip to iterate over multiple iterables.
Can you write Python code for it?

test2 = test**2
test2

for i, j in zip(test, test2):
    print('{} + {} = {}'.format(i, j, i + j))


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Indexing/Slicing
We have already seen briefly that to access individual elements you use the bracket notation: array[ax_0, ax_1, ...], where the ax_i denotes the coordinate in the i-th axis. You can even use this to assign new values to your elements.
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Loading and Saving Data
To load and save data NumPy has the loadtxt and savetxt commands. However, they only work for two-dimensional arrays
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Loading and saving dataframes
To load and save Pandas dataframes we will use the to_csv and read_csv commands
Can you write Python code for it?

u.to_csv('test.csv')
v = pd.read_csv('test.csv', index_col=0)
v.head()


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Not only that, you can access a single column without the need of brackets []
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Numeric Python (NumPy)
NumPy is an open-source add-on module to Python that provides common mathematical and numerical routines in pre-compiled, fast functions. These are growing into highly mature packages that provide functionality that meets, or perhaps exceeds, that
associated with common commercial software like MATLAB. The NumPy (Numeric Python) package provides basic routines for manipulating large arrays and matrices of numeric data. The main object NumPy works with is a homogeneous multidimensional array. Despite its intimidating name, these are nothing but tables of numbers, each labelled by a tuple of indices.
We will now explore some capabilities of NumPy that will prove very useful not only for data analyisis, but throughout all our life with Python.
Creating Arrays
As mentioned before, the main object in NumPy is the array. Creating one is as easy as calling the command array
Can you write Python code for it?

mylist = [1, 2, 3]
x = np.array(mylist)
x

type(x)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
On the other hand, DataFrames can be built from two-dimensional arrays, with the ability of labelling columns and indexing the rows
Can you write Python code for it?

u = pd.DataFrame(np.random.randn(1000,6), index=np.arange(0, 3000, 3),
                 columns=['A', 'B', 'C', 'D', 'E', 'F'])
u


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Operations
You can perform easily element-wise operations on arrays of any shape. Use the typical symbols, +, -, *, / and ** to perform element-wise addition, subtraction, multiplication, division and power.
Can you write Python code for it?

x = np.array([1, 2, 3])
print(x)
print(x + 10)
print(3 * x)
print(1 / x)
print(x ** (-2 / 3))
print(2 ** x)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Or by row and index:
Can you write Python code for it?

for i, row in enumerate(test):
    print('Row {} is {}'.format(i, row))


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Pandas
When dealing with numeric matrices and vectors in Python, NumPy makes life a lot easier. For more complex data, however, it leaves a bit to be desired. For those used to working with dedicated languages like R, doing data analysis directly with numpy feels like a step back. Fortunately, some nice folks have written the Python Data Analysis Library (a.k.a. pandas). Pandas provides an R-like DataFrame, produces high quality plots with matplotlib, and integrates nicely with other libraries that expect NumPy arrays.
Pandas works with Series of data, that then are arranged in DataFrames. A dataframe will be the object closest to an Excel spreadsheet that you will see throughout the course (but of course, given that it is integrated in Python and can be combined with so many different packages, dataframes are much more powerful than Excel spreadsheets). The data in the series can be either qualitative or quantitative data. Creating a series is as easy as creating a NumPy array from a one-dimensional list.
Can you write Python code for it?

animals = ['Tiger', 'Bear', 'Moose']
pd.Series(animals)

numbers = [1, 2, 3]
pd.Series(numbers)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Pandas can also generate random DataFrames for testing:
Can you write Python code for it?

import pandas.util.testing as tm

tm.makeDataFrame().head()


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
Special Arrays
Now we review some built-in functions that create matrices commonly used. ones and zeros return arrays of given shape and type (default is float64), filled with ones or zeros, respectively.
Can you write Python code for it?

array_zeros=np.zeros((3, 2))
array_ones=np.ones((3, 2, 4), dtype=np.int8)

array_zeros

array_ones


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
The proper way of handling selections without modifying the original arrays is through the copy command.
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
The same result can be obtained with concat.
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
The usual [] will select specific rows according to the row number
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
When looking at r, we see that it has also been changed!
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
With python 3.5 matrix multiplication got it's own operator
Can you write Python code for it?

x@y

X = np.array([[i + j for i in range(3, 6)] for j in range(3)])
Y = np.diag([1, 1], 1) + np.diag([1], -2)

print('{}\n'.format(X))
print(X * Y)
np.dot(X, Y)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
You can also choose specific rows according to their indices with the loc command
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
arange(begin, end, step) returns evenly spaced values within a given interval. Note that the beginning point is included, but not the ending.
Can you write Python code for it?

n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30
n

len(n)


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
diag, depending on the input, either extracts a diagonal from a matrix (if the input is a 2-D array), or constructs a diagonal array (if the input is a vector).
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
eye can also create arrays with ones in upper and lower diagonals. To achieve this, you must call eye(d, d, k) where $k$ denotes the diagonal (positive for above the center diagonal, negative for below), or eye(d, k=num)
Can you write Python code for it?


I am working on the file "4_Numpy_Pandas.ipynb".
The first task is:
reshape changes the shape of an array, but not its data. This is another of the commands that can be called before or after the array.
Can you write Python code for it?

print(n.reshape(3, 5))
np.reshape(n, (5, 3))


I am working on the file "4_Titanic.ipynb".
The first task is:
В именах есть приставки, с ними тоже можно кое-что сделать, т.к. социальный статус может быть важным признаком выживаемости.
Can you write Python code for it?

Title_Dictionary = {
                    "Capt":       "Officer",
                    "Col":        "Officer",
                    "Major":      "Officer",
                    "Jonkheer":   "Nobel",
                    "Don":        "Nobel",
                    "Sir" :       "Nobel",
                    "Dr":         "Officer",
                    "Rev":        "Officer",
                    "the Countess":"Nobel",
                    "Dona":       "Nobel",
                    "Mme":        "Mrs",
                    "Mlle":       "Miss",
                    "Ms":         "Mrs",
                    "Mr" :        "Mr",
                    "Mrs" :       "Mrs",
                    "Miss" :      "Miss",
                    "Master" :    "Master",
                    "Lady" :      "Nobel"
                    }

#train_df['Title'] =
#test_df['Title'] =

train_df.head()


I am working on the file "4_Titanic.ipynb".
The first task is:
Вместо двух полей указывающий на наличие партнера (Parch) или родственника (SibSp), сделаем одно поле FamilySize
Can you write Python code for it?

#train_df['FamilySize'] =
#test_df['FamilySize'] =

train_df.head()


I am working on the file "4_Titanic.ipynb".
The first task is:
Загружаем наши данные и смотрим на их состояние
Can you write Python code for it?

# your code here
# train_df
# test_df

train_df.head()

train_df.info()
print("----------------------------")
test_df.info()


I am working on the file "4_Titanic.ipynb".
The first task is:
Легко заметить, что в тренировочном датасете у нас не хватает данных о возрасте, каюте и месте погружения пассажира на корабль.
В тестовом датасете нам не хватает данных о возрасте, каюте и плате за пребывание на корабле.
Для начала разберемся с полем Embarked в тренировочном датасете, которое отвечает за место погружения.
Проверим, в каких строках у нас отсутствуют данные.
Can you write Python code for it?


I am working on the file "4_Titanic.ipynb".
The first task is:
Пол тоже очень важный признак, но если вы смотрели фильм титаник, то наверное помните "Сначала женщин и детей." Поэтому предлагаю сооздать новый признак, который будет учитывать как пол, так и возраст
Can you write Python code for it?

def get_person(passenger):
    # your code here
    pass

train_df['Person'] = train_df[['Age','Sex']].apply(get_person,axis=1)
test_df['Person']    = test_df[['Age','Sex']].apply(get_person,axis=1)

train_df.head()

# print status of datasets


I am working on the file "4_Titanic.ipynb".
The first task is:
Посмотрим на общею зависимость шанса выживания от пункта погружения.
Can you write Python code for it?

fig, (axis1,axis2,axis3) = plt.subplots(1,3,figsize=(15,5))

sns.countplot(x='Embarked', data=train_df, ax=axis1)
sns.countplot(x='Survived', hue="Embarked", data=train_df, order=[1,0], ax=axis2)

# group by embarked, and get the mean for survived passengers for each value in Embarked
embark_perc = train_df[["Embarked", "Survived"]].groupby(['Embarked'],as_index=False).mean()
sns.barplot(x='Embarked', y='Survived', data=embark_perc,order=['S','C','Q'],ax=axis3)


I am working on the file "4_Titanic.ipynb".
The first task is:
Смотрим на другие возможные зависимости, которые могли б нам указать на то, где пассажиры попали на корабль.
Can you write Python code for it?

# проверить пассажиров с билетом 113572

# вывести кол-во пассажиров с ценами от 75 до 85

# присвоить пустым значениям те, которые мы решили

# проверить что пустых больше нет


I am working on the file "4_Titanic.ipynb".
The first task is:
Создадим функцию, которая будет строить зависимость обучаемости от кол-ва тестовых семплов.
Can you write Python code for it?

# не вдаемся в подробности, используем готовый код

def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5), scoring='accuracy'):
    plt.figure(figsize=(10,6))
    plt.title(title)
    if ylim is not None:
        plt.ylim(*ylim)
    plt.xlabel("Training examples")
    plt.ylabel(scoring)
    train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, scoring=scoring,
                                                            n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation score")

    plt.legend(loc="best")
    return plt


I am working on the file "4_Titanic.ipynb".
The first task is:
Строим простую модель
Random forest
Модель случайного леса это ничто иное как ансамбль решающих деревьев:

Который в результате будет выглядить где-то так:

Настоятельно рекомендую просмотреть это видео (https://www.youtube.com/watch?v=loNcrMjYh64) если вас интерисует как работает этот алгоритм
Посмотрим модель случайного леса. Параметры укажем обычные, потом благодаря GridSearchCV подберем оптимальные. Ну и в конце взглянем на то, что у нас вышло
Can you write Python code for it?

# создадим обьект нашего классификатора
clf = RandomForestClassifier()

# задаем параметры случайным образом
parameters = {'n_estimators': [4, 6, 9],
              'max_features': ['log2', 'sqrt','auto'],
              'criterion': ['entropy', 'gini'],
              'max_depth': [2, 3, 5, 10],
              'min_samples_split': [2, 3, 5],
              'min_samples_leaf': [1,5,8]
             }

# параметр для расчета точности нашей модели
acc_scorer = make_scorer(accuracy_score)

# подбираем оптимальные параметры для нашей модели с помощью GridSearchCV
# grid_obj =

# устанавливаем полученные параметры для нашей модели
clf = grid_obj.best_estimator_

clf.fit(X_train, y_train)

# Посмотрим скорость обучения нашей модели

predictions = clf.predict(X_test)
print(accuracy_score(y_test, predictions))
plot_learning_curve(clf, 'Random Forest', X, y, cv=4);


I am working on the file "4_Titanic.ipynb".
The first task is:
Теперь исправим пустое поле с платой за путешествение в тестовом датасете.
Can you write Python code for it?

# показать пустые в тестовом датасете с полем Fare


I am working on the file "4_Titanic.ipynb".
The first task is:
Теперь разберемся с полем Возраста в тренировочном датасете. Ему нужно уделить больше внимания, т.к. это очень важный признак, который сильно влияет на выживаемость пассажиров.
Can you write Python code for it?

# вывести пустые с возрастом

#fig, (axis1,axis2) = plt.subplots(1,2,figsize=(15,4))
#axis1.set_title('Original Age values')
#axis2.set_title('New Age values')

# среднее, дисперсия и пустые значение в тестовом датасете ( .mean() .std() .isnull().sum() )
#average_age_train   =
#std_age_train       =
#count_nan_age_train =

# среднее, дисперсия и пустые значение в тестовом датасете
#average_age_test   =
#std_age_test       =
#count_nan_age_test =

# генерируем случайные значения (mean - std) & (mean + std)
rand_1 = np.random.randint(average_age_train - std_age_train, average_age_train + std_age_train, size = count_nan_age_train)
rand_2 = np.random.randint(average_age_test - std_age_test, average_age_test + std_age_test, size = count_nan_age_test)

# строим гистограму возраста до изменений (пустые конвертим в инты)
train_df['Age'].dropna().astype(int).hist(bins=70, ax=axis1)
test_df['Age'].dropna().astype(int).hist(bins=70, ax=axis1)

# заполняем случайными значениями пустые поля с возрастом
train_df["Age"][np.isnan(train_df["Age"])] = rand_1
test_df["Age"][np.isnan(test_df["Age"])] = rand_2

# конвертим флоаты в инты
train_df['Age'] = train_df['Age'].astype(int)
test_df['Age']    = test_df['Age'].astype(int)

# гистограма нового возраста
train_df['Age'].hist(bins=70, ax=axis2)
test_df['Age'].hist(bins=70, ax=axis2)

# Еще немного графиков

# пик выживаемости в зависимости от возраста
facet = sns.FacetGrid(train_df, hue="Survived",aspect=4)
facet.map(sns.kdeplot,'Age',shade= True)
facet.set(xlim=(0, train_df['Age'].max()))
facet.add_legend()

# средняя выживаемость по возрасту
fig, axis1 = plt.subplots(1,1,figsize=(18,4))
average_age = train_df[["Age", "Survived"]].groupby(['Age'],as_index=False).mean()
sns.barplot(x='Age', y='Survived', data=average_age)

# проверим статус наших датасетов после преобразование данных


I am working on the file "4_Titanic.ipynb".
The first task is:
У нас есть дискретные переменные и нам стоило б их закодировать. Для этого в пандас уже существует функция get_dummies
Can you write Python code for it?

# кодировать переменные с помощью get_dummies, http://pandas.pydata.org/pandas-docs/version/0.18.1/generated/pandas.get_dummies.html

#dummies_person_train =
#dummies_embarked_train =
#dummies_title_train =
#dummies_pclass_train =

# обьеденяем добавляем новые переменные в наш датафрейм с помощью concat(), http://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html
#train_df =
#train_df =

train_df.head()

# Аналогично для тестового датасета

#dummies_person_test =
#dummies_embarked_test =
#dummies_title_test =
#dummies_pclass_test =

#test_df =
#test_df =

test_df.head()


I am working on the file "4_Train_Model.ipynb".
The first task is:
Build the StarNet model architecture
The StarNet architecture is built with:
- input layer
- 2 convolutional layers
- 1 maxpooling layer followed by flattening for the fully connected layer
- 2 fully connected layers
- output layer
First, let's define some model variables.
Can you write Python code for it?

# activation function used following every layer except for the output layers
activation = 'relu'

# model weight initializer
initializer = 'he_normal'

# number of filters used in the convolutional layers
num_filters = [4,16]

# length of the filters in the convolutional layers
filter_length = 8

# length of the maxpooling window
pool_length = 4

# number of nodes in each of the hidden fully connected layers
num_hidden = [256,128]

# number of spectra fed into model at once during training
batch_size = 64

# maximum number of interations for model training
max_epochs = 30

# initial learning rate for optimization algorithm
lr = 0.0007

# exponential decay rate for the 1st moment estimates for optimization algorithm
beta_1 = 0.9

# exponential decay rate for the 2nd moment estimates for optimization algorithm
beta_2 = 0.999

# a small constant for numerical stability for optimization algorithm
optimizer_epsilon = 1e-08

# Input spectra
input_spec = Input(shape=(num_fluxes,), name='starnet_input_x')

# Reshape spectra for CNN layers
cur_in = Reshape((num_fluxes, 1))(input_spec)

# CNN layers
cur_in = Conv1D(kernel_initializer=initializer, activation=activation,
                padding="same", filters=num_filters[0], kernel_size=filter_length)(cur_in)
cur_in = Conv1D(kernel_initializer=initializer, activation=activation,
                padding="same", filters=num_filters[1], kernel_size=filter_length)(cur_in)

# Max pooling layer
cur_in = MaxPooling1D(pool_size=pool_length)(cur_in)

# Flatten the current input for the fully-connected layers
cur_in = Flatten()(cur_in)

# Fully-connected layers
cur_in = Dense(units=num_hidden[0], kernel_initializer=initializer,
               activation=activation)(cur_in)
cur_in = Dense(units=num_hidden[1], kernel_initializer=initializer,
               activation=activation)(cur_in)

# Output nodes
output_label = Dense(units=num_labels, activation="linear",
                     input_dim=num_hidden[1], name='starnet_output_y')(cur_in)

model = Model(input_spec,...


I am working on the file "4_Train_Model.ipynb".
The first task is:
More model techniques
* The Adam optimizer is the gradient descent algorithm used for minimizing the loss function
* EarlyStopping uses the cross-validation set to test the model following every iteration and stops the training if the cv loss does not decrease by min_delta after patience iterations
* ReduceLROnPlateau is a form of learning rate decay where the learning rate is decreased by a factor of factor if the training loss does not decrease by epsilon after patience iterations unless the learning rate has reached min_lr
Can you write Python code for it?

# Default loss function parameters
early_stopping_min_delta = 0.0001
early_stopping_patience = 4
reduce_lr_factor = 0.5
reuce_lr_epsilon = 0.0009
reduce_lr_patience = 2
reduce_lr_min = 0.00008

# loss function to minimize
loss_function = 'mean_squared_error'

# compute mean absolute deviation
metrics = ['mae']

optimizer = Adam(lr=lr, beta_1=beta_1, beta_2=beta_2, epsilon=optimizer_epsilon, decay=0.0)

early_stopping = EarlyStopping(monitor='val_loss', min_delta=early_stopping_min_delta,
                                       patience=early_stopping_patience, verbose=2, mode='min')

reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.5, epsilon=reuce_lr_epsilon,
                                  patience=reduce_lr_patience, min_lr=reduce_lr_min, mode='min', verbose=2)


I am working on the file "4_Train_Model.ipynb".
The first task is:
Normalization
Write a function to normalize the output labels. Each label will be normalized to have approximately have a mean of zero and unit variance.
NOTE: This is necessary to put output labels on a similar scale in order for the model to train properly, this process is reversed in the test stage to give the output labels their proper units
Can you write Python code for it?

mean_and_std = np.load(normalization_data)
mean_labels = mean_and_std[0]
std_labels = mean_and_std[1]

def normalize(labels):
    # Normalize labels
    return (labels-mean_labels) / std_labels


I am working on the file "4_Train_Model.ipynb".
The first task is:
Obtain training data
Here we will collect the output labels for the training and cross-validation sets, then normalize each.
Next we will create an HDF5Matrix for the training and cross-validation input spectra rather than loading them all into memory. This is useful to save RAM when training the model.
Can you write Python code for it?

# Define the number of output labels
num_labels = np.load(datadir+'mean_and_std.npy').shape[1]

# Define the number of training spectra
num_train = 41000

# Load labels
with  h5py.File(training_set, 'r') as F:
    y_train = np.hstack((F['TEFF'][0:num_train], F['LOGG'][0:num_train], F['FE_H'][0:num_train]))
    y_cv = np.hstack((F['TEFF'][num_train:], F['LOGG'][num_train:], F['FE_H'][num_train:]))

# Normalize labels
y_train = normalize(y_train)
y_cv = normalize(y_cv)

# Create the spectra training and cv datasets
x_train = HDF5Matrix(training_set, 'spectrum',
                           start=0, end=num_train)
x_cv = HDF5Matrix(training_set, 'spectrum',
                           start=num_train, end=None)

# Define the number of output labels
num_labels = y_train.shape[1]

num_fluxes = x_train.shape[1]

print('Each spectrum contains ' + str(num_fluxes) + ' wavelength bins')
print('Training set includes ' + str(x_train.shape[0]) +
      ' spectra and the cross-validation set includes ' + str(x_cv.shape[0])+' spectra')


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
4.4.1 UV coverage : UV tracks
The objectives of $\S$ 4.4.1 ⤵ and $\S$ 4.4.2 ➞ is to give you a glimpse into the process of accumulation of measurements. An interferometer measures components of the sky Fourier Transform through the sampling of the Visibility function $\mathcal{V}$. This collection of samples lives in the ($u$, $v$, $w$) space and are often projected into a plane, the $uv$-plane.
In $\S$ 4.4.1 ⤵, we will focus on the way the visibility function is sampled as a function of the interferometer configuration, the direction of the source and observation time.
In $\S$ 4.4.2 ➞, we will see how this sampling can be improved by specific observing techniques.
4.4.1.A The projected baseline with time: the UV track
The combination of a baseline and a direction in the sky enable us to define a projected baseline, which is the baseline as seen from the source. This projected baseline is associated with the measurement of a spatial frequency of the source. As the Earth rotates, this projected baseline and the corresponding spatial frequency at coordinate ($u$, $v$) slowly vary in time, generating a path in the $uv$-plane.
In the following we will generate test cases to see what are the shape of the path and how it can be predicted according to the baseline geometry.
Baseline projection as seen from the source
Let's generate one baseline from two antennas Ant$_1$ and Ant$_2$.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Below we sample our visibility plane on the $uv$-track derived in the first section, i.e. $V(u_t,v_t)$.
Can you write Python code for it?

plt.subplot(121)
plt.imshow(zz.real,extent=[-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10,-1*(np.amax(abs(v_60)))-10, \
                           np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Real part of visibilities")

plt.subplot(122)
plt.imshow(zz.imag,extent=[-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10,-1*(np.amax(abs(v_60)))-10, \
                           np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Imaginary part of visibilities")


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Computation of the projected baselines in ($u$,$v$,$w$) coordinates along time
As seen previously, we convert the baseline coordinates using the previous matrix transformation.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Figure 4.4.2: $uvw$ track derived from the simulation and projection in the $uv$-plane.
The track in the $uvw$ space are curves and the projection in the $uv$ plane are arcs. Let's focus more on the projection of the track in this plane. To get an observation-independent knowledge of the track, we can try to combine the three equations of $u$, $v$ and $w$ with the objective of eliminating $H$ from the equation. We end up an equation linking $u$, $v$, $X$ and $Y$ (the full derivation can be found in $\S$ A.3 ➞):
$$\boxed{u^2 + \left[ \frac{v -\frac{Z}{\lambda} \cos \delta}{\sin \delta} \right]^2 = \left[ \frac{X}{\lambda} \right]^2 + \left[ \frac{Y}{\lambda} \right]^2}$$
One can note that in this particular case, the $uv$ tracks is described by the equation of an ellipsis.
This ellipsis is centered at $(0,\frac{Z}{\lambda} \cos \delta)$ in the ($u$,$v$) plane.
The major axis of  $a=\frac{\sqrt{X^2 + Y^2}}{\lambda}$.
The minor axis along the axis $v$ will be a function of $Z$, $\delta$ and $a$.
To verify this fact, let's plot the theoretical ellipsis over the observed portion of the track. (You can fall back to the duration of the observation to see that the track is mapping exactly the ellipsis).
Can you write Python code for it?

%matplotlib inline
from matplotlib.patches import Ellipse

# parameters of the UVtrack as an ellipse
a=np.sqrt(X**2+Y**2)/lam/1e3 # major axis
b=a*np.sin(dec)              # minor axis
v0=Z/lam*np.cos(dec)/1e3     # center of ellipse

plotBL.UVellipse(u,v,w,a,b,v0)


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Figure 4.4.3: The blue (resp. the red) curve is the $uv$ track of the baseline $\mathbf{b}{12}$ (resp. $\mathbf{b}{21}$). As $I_\nu$ is real, the real part of the visibility $\mathcal{V}$ is even and the imaginary part if odd making $\mathcal{V}(-u,-v)=\mathcal{V}^*$. It implies that one baseline provide a measurement of its complex conjugate for free at ($-u$,$-v$).
4.4.1.B Special cases
The Polar interferometer
Let settle one baseline at the North pole. The local zenith corresponds to the North Celestial Pole at $\delta=90^\circ$. As seen from the NCP, the baseline will rotate and the projected baseline will correspond to the physical baseline. This is the only case where it is happening.
If $\mathbf{b}$ rotates, we can guess that the $uv$ tracks will be perfect circles. Let's verify that.
Can you write Python code for it?

L=np.radians(90.)
ant1 = np.array([25.095,-9.095,0.045])
ant2 = np.array([90.284,26.380,-0.226])
b_ENU = ant2-ant1
D = np.sqrt(np.sum((b_ENU)**2))

A=np.arctan2(b_ENU[0],b_ENU[1])
print "Azimuth=",A*(180/np.pi)
E=np.arcsin(b_ENU[2]/D)
print "Elevation=",E*(180/np.pi)

X = D*(np.cos(L)*np.sin(E)-np.sin(L)*np.cos(E)*np.cos(A))
Y = D*np.cos(E)*np.sin(A)
Z = D*(np.sin(L)*np.sin(E)+np.cos(L)*np.cos(E)*np.cos(A))


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Figure 4.4.7: Real and imaginary part of the visibility function. The black curve is the portion of the $uv$ track crossing the visibility.
We now plot the sampled visibilites as a function of time-slots, i.e $V(u_t(t_s),v_t(t_s))$.
Can you write Python code for it?

plt.subplot(121)
plt.plot(z.real)
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Real: sampled visibilities")

plt.subplot(122)
plt.plot(z.imag)
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Imag: sampled visibilities")


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Let's compute the $uv$ tracks when observing the NCP ($\delta=90^\circ$):
Can you write Python code for it?

dec=np.radians(90.)

uNCP = lam**(-1)*(np.sin(h)*X+np.cos(h)*Y)/1e3
vNCP = lam**(-1)*(-np.sin(dec)*np.cos(h)*X+np.sin(dec)*np.sin(h)*Y+np.cos(dec)*Z)/1e3
wNCP = lam**(-1)*(np.cos(dec)*np.cos(h)*X-np.cos(dec)*np.sin(h)*Y+np.sin(dec)*Z)/1e3

# parameters of the UVtrack as an ellipse
aNCP=np.sqrt(X**2+Y**2)/lam/1e3 # major axis
bNCP=aNCP*np.sin(dec)              # minor axi
v0NCP=Z/lam*np.cos(dec)/1e3     # center of ellipse


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Let's compute the uv tracks when observing a source at $\delta=30^\circ$:
Can you write Python code for it?

dec=np.radians(30.)

u30 = lam**(-1)*(np.sin(h)*X+np.cos(h)*Y)/1e3
v30 = lam**(-1)*(-np.sin(dec)*np.cos(h)*X+np.sin(dec)*np.sin(h)*Y+np.cos(dec)*Z)/1e3
w30 = lam**(-1)*(np.cos(dec)*np.cos(h)*X-np.cos(dec)*np.sin(h)*Y+np.sin(dec)*Z)/1e3

a30=np.sqrt(X**2+Y**2)/lam/1e3 # major axis
b30=a*np.sin(dec)              # minor axi
v030=Z/lam*np.cos(dec)/1e3     # center of ellipse

%matplotlib inline
plotBL.UVellipse(u30,v30,w30,a30,b30,v030)
plotBL.UVellipse(uNCP,vNCP,wNCP,aNCP,bNCP,v0NCP)


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
Let's place the interferometer at a latitude $L_a=+45^\circ00'00''$.
Can you write Python code for it?

L = (np.pi/180)*(45+0./60+0./3600)      # Latitude in radians

A = np.arctan2(b_ENU[0],b_ENU[1])
print "Baseline Azimuth="+str(np.degrees(A))+"°"

E = np.arcsin(b_ENU[2]/D)
print "Baseline Elevation="+str(np.degrees(E))+"°"

%matplotlib nbagg
plotBL.sphere(ant1,ant2,A,E,D,L)


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
The coordinates of the sources and the phase center are now in degrees.
Can you write Python code for it?

%matplotlib inline
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plt.xlim([-4,4])
plt.ylim([-4,4])
plt.xlabel("$l$ [degrees]")
plt.ylabel("$m$ [degrees]")
plt.plot(l[0],m[0],"bx")
plt.hold("on")
plt.plot(l[1:]*(180/np.pi),m[1:]*(180/np.pi),"ro")
counter = 1
for xy in zip(l[1:]*(180/np.pi)+0.25, m[1:]*(180/np.pi)+0.25):
    ax.annotate(Flux_sources_labels[counter], xy=xy, textcoords='offset points',horizontalalignment='right',
                verticalalignment='bottom')
    counter = counter + 1

plt.grid()


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
We create the dimensions of our visibility plane.
Can you write Python code for it?

s = point_sources.shape
for counter in xrange(1, s[0]+1):
    A_i = point_sources[counter-1,0]
    l_i = point_sources[counter-1,1]
    m_i = point_sources[counter-1,2]
    zz += A_i*np.exp(-2*np.pi*1j*(uu*l_i+vv*m_i))
zz = zz[:,::-1]


I am working on the file "4_Visibility_Space/4_5_1_UV_Coverage_UV_Tracks.ipynb".
The first task is:
We then convert the ($\alpha$,$\delta$) to $l,m$ with
* $l = \cos \delta \sin \Delta \alpha$
* $m = \sin \delta\cos\delta_0 -\cos \delta\sin\delta_0\cos\Delta \alpha$
* $\Delta \alpha = \alpha - \alpha_0$
Can you write Python code for it?

RA_rad = np.array(RA_sources)*(np.pi/12)
DEC_rad = np.array(DEC_sources)*(np.pi/180)
RA_delta_rad = RA_rad-RA_rad[0]

l = np.cos(DEC_rad)*np.sin(RA_delta_rad)
m = (np.sin(DEC_rad)*np.cos(DEC_rad[0])-np.cos(DEC_rad)*np.sin(DEC_rad[0])*np.cos(RA_delta_rad))
print "l=",l*(180/np.pi)
print "m=",m*(180/np.pi)

point_sources = np.zeros((len(RA_sources)-1,3))
point_sources[:,0] = Flux_sources
point_sources[:,1] = l[1:]
point_sources[:,2] = m[1:]


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
4.5.1 UV coverage : UV tracks
The objective of $\S$ 4.5.1 ⤵ and $\S$ 4.5.2 ➞ is to give you a glimpse into the process of aperture synthesis. <span style="background-color:cyan">TLG:GM: Check if the italic words are in the glossary. </span> An interferometer measures components of the Fourier Transform of the sky by sampling the visibility function, $\mathcal{V}$. This collection of samples lives in ($u$, $v$, $w$) space, and are often projected onto the so-called $uv$-plane.
In $\S$ 4.5.1 ⤵, we will focus on the way the visibility function is sampled. This sampling is a function of the interferometer's configuration, the direction of the source and the observation time.
In $\S$ 4.5.2 ➞, we will see how this sampling can be improved by using certain observing techniques.
4.5.1.1 The projected baseline with time: the $uv$ track
A projected baseline depends on a baseline's coordinates, and the direction being observed in the sky. It corresponds to the baseline as seen from the source. The projected baseline is what determines the spatial frequency of the sky that the baseline will measure. As the Earth rotates, the projected baseline and its corresponding spatial frequency (defined by the baseline's ($u$, $v$)-coordinates) vary slowly in time, generating a path in the $uv$-plane.
We will now generate test cases to see what locus the path takes, and how it can be predicted depending on the baseline's geometry.
4.5.1.1.1 Baseline projection as seen from the source
Let's generate one baseline from two antennas Ant$_1$ and Ant$_2$.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
4.5.1 UV coverage : UV tracks
The objective of $\S$ 4.5.1 ⤵ and $\S$ 4.5.2 ➞ is to give you a glimpse into the process of aperture synthesis. TLG:GM: Check if the italic words are in the glossary.  An interferometer measures components of the Fourier Transform of the sky by sampling the visibility function, $\mathcal{V}$. This collection of samples lives in ($u$, $v$, $w$) space, and are often projected onto the so-called $uv$-plane.
In $\S$ 4.5.1 ⤵, we will focus on the way the visibility function is sampled. This sampling is a function of the interferometer's configuration, the direction of the source and the observation time.
In $\S$ 4.5.2 ➞, we will see how this sampling can be improved by using certain observing techniques.
4.5.1.1 The projected baseline with time: the $uv$ track
A projected baseline is obtained via a baseline and a direction in the sky. It corresponds to the baseline as seen from the source. The projected baseline is associated with the measurement of a spatial frequency of the source. TLG:RC: Rewrite previous sentence. As the Earth rotates, the projected baseline and its corresponding spatial frequency (defined by the baseline's ($u$, $v$)-coordinates) vary slowly in time, generating a path in the $uv$-plane.
We will now generate test cases to see what locus the path takes, and how it can be predicted depending on the baseline's geometry.
4.5.1.1.1 Baseline projection as seen from the source
Let's generate one baseline from two antennas Ant$_1$ and Ant$_2$.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
4.5.1.1.3 Computing of the projected baselines in ($u$,$v$,$w$) coordinates as a function of time
As seen previously, we convert the baseline coordinates using the previous matrix transformation.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
TLG:AC: Add the following figures. This is specifically for an EW array. They will add some more insight.

4.5.1.3.2 Simulating the sky
Let us populate our sky with three sources, with positions given in RA ($\alpha$) and DEC ($\delta$):
* Source 1: (5h 32m 0.4s,60$^{\circ}$-17' 57'') - 1 Jy
* Source 2: (5h 36m 12.8s,-61$^{\circ}$ 12' 6.9'') - 0.5 Jy
* Source 3: (5h 40m 45.5s,-61$^{\circ}$ 56' 34'') - 0.2 Jy
We place the field center at $(\alpha_0,\delta_0) = $ (5h 30m,60$^{\circ}$).
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
As the $u$, $v$, $w$ coordinates explicitly depend on $H$, we must evaluate them for each observational time step. We will use the equations defined in $\S$ 4.2.2 ➞:

$\lambda u = X \sin H + Y \cos H$
$\lambda v= -X \sin \delta \cos H + Y \sin\delta\sin H + Z \cos\delta$
$\lambda w= X \cos \delta \cos H -Y \cos\delta\sin H + Z \sin\delta$
Can you write Python code for it?

u = lam**(-1)*(np.sin(h)*X+np.cos(h)*Y)/1e3
v = lam**(-1)*(-np.sin(dec)*np.cos(h)*X+np.sin(dec)*np.sin(h)*Y+np.cos(dec)*Z)/1e3
w = lam**(-1)*(np.cos(dec)*np.cos(h)*X-np.cos(dec)*np.sin(h)*Y+np.sin(dec)*Z)/1e3


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Below we sample our visibility plane on the $uv$-track derived in the first section, i.e. $V(u_t,v_t)$.
TLG:RC: The graphs below intersect. Axis labels inside other
        graphs.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Below we sample our visibility plane on the $uv$-track derived in the first section, i.e. $V(u_t,v_t)$.
Can you write Python code for it?

plt.figure(figsize=(12,6))

plt.subplot(121)
plt.imshow(zz.real,extent=[-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10,-1*(np.amax(abs(v_60)))-10, \
                           np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Real part of visibilities")

plt.subplot(122)
plt.imshow(zz.imag,extent=[-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10,-1*(np.amax(abs(v_60)))-10, \
                           np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Imaginary part of visibilities")


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.1: A baseline located at +45$^\circ$ as seen from the sky. This plot is interactive and can be rotated in 3D to see different baseline projections, depending on the position of the source w.r.t. the physical baseline.
On the interactive plot above, we represent a baseline located at +45$^\circ$, aligned with the local south-west/north-east as seen from the celestial sphere. TLG:RC: Rewrite previous sentence. By rotating the sphere westward, you can simulate the variation of the projected baseline as seen from a source in apparent motion on the celestial sphere.
4.5.1.1.2  Coordinates of the baseline in the ($u$,$v$,$w$) plane
We will now simulate an observation to study how a projected baseline will change with time. We will position this baseline at a South African latitude. We first need the expression of the physical baseline in a convenient reference frame, attached to the source in the sky.
In $\S$ 4.2 ➞, we linked the equatorial coordinates of the baseline to the ($u$,$v$,$w$) coordinates through the transformation matrix:
\begin{equation}
\begin{pmatrix}
u\
v\
w
\end{pmatrix}
=
\frac{1}{\lambda}
\begin{pmatrix}
\sin H_0 & \cos H_0 & 0\
-\sin \delta_0 \cos H_0 & \sin\delta_0\sin H_0 & \cos\delta_0\
\cos \delta_0 \cos H_0 & -\cos\delta_0\sin H_0 & \sin\delta_0\
\end{pmatrix}
\begin{pmatrix}
X\
Y\
Z
\end{pmatrix}
\end{equation}

\begin{equation}
\begin{bmatrix}
X\
Y\
Z
\end{bmatrix}
=|\mathbf{b}|
\begin{bmatrix}
\cos L_a \sin \mathcal{E} - \sin L_a \cos \mathcal{E} \cos \mathcal{A}\nonumber\
\cos \mathcal{E} \sin \mathcal{A} \nonumber\
\sin L_a \sin \mathcal{E} + \cos L_a \cos \mathcal{E} \cos \mathcal{A}\
\end{bmatrix}
\end{equation}
Equation 4.5.1
This expression of $\mathcal{b}$ is a function of ($\mathcal{A}$,$\mathcal{E}$) in the equatorial ($X$,$Y$,$Z$) systems. TLG:RC: Rewrite previous sentence as the meaning is unclear. Notice the unbold...
Can you write Python code for it?

# Observation parameters
c = 3e8                                        # Speed of light
f = 1420e9                                     # Frequency
lam = c/f                                      # Wavelength
dec = (np.pi/180)*(-30-43.0/60-17.34/3600)     # Declination

time_steps = 600                               # Time Steps
h = np.linspace(-4,4,num=time_steps)*np.pi/12  # Hour angle window


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.1: A baseline located at +45$^\circ$ as seen from the sky. This plot is interactive and can be rotated in 3D to see different baseline projections, depending on the position of the source w.r.t. the physical baseline.
On the interactive plot above, we represent a baseline located at +45$^\circ$. It is aligned with the local south-west/north-east axis, as seen from the sky frame of reference. By rotating the sphere westward, you can simulate the variation of the projected baseline as seen from a source in apparent motion on the celestial sphere.
4.5.1.1.2  Coordinates of the baseline in the ($u$,$v$,$w$) plane
We will now simulate an observation to study how a projected baseline will change with time. We will position this baseline at a South African latitude. We first need the expression of the physical baseline in a convenient reference frame, attached to the source in the sky.
In $\S$ 4.2 ➞, we linked the equatorial coordinates of the baseline to the ($u$,$v$,$w$) coordinates through the transformation matrix:
\begin{equation}
\begin{pmatrix}
u\
v\
w
\end{pmatrix}
=
\frac{1}{\lambda}
\begin{pmatrix}
\sin H_0 & \cos H_0 & 0\
-\sin \delta_0 \cos H_0 & \sin\delta_0\sin H_0 & \cos\delta_0\
\cos \delta_0 \cos H_0 & -\cos\delta_0\sin H_0 & \sin\delta_0\
\end{pmatrix}
\begin{pmatrix}
X\
Y\
Z
\end{pmatrix}
\end{equation}

\begin{equation}
\begin{bmatrix}
X\
Y\
Z
\end{bmatrix}
=|\mathbf{b}|
\begin{bmatrix}
\cos L_a \sin \mathcal{E} - \sin L_a \cos \mathcal{E} \cos \mathcal{A}\nonumber\
\cos \mathcal{E} \sin \mathcal{A} \nonumber\
\sin L_a \sin \mathcal{E} + \cos L_a \cos \mathcal{E} \cos \mathcal{A}\
\end{bmatrix}
\end{equation}
Equation 4.5.1
This expression of $\mathbf{b}$ is a function of ($\mathcal{A}$,$\mathcal{E}$), and therefore of ($X$,$Y$,$Z$) in the equatorial frame of reference.
4.5.1.1.2 Observation parameters
Let's define an arbitrary set of observation parameters to mimic a real observation.

Latitude of the baseline:...
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.2: $uvw$ track derived from the simulation and projection in the $uv$-plane.
The track in $uvw$ space are curves and the projection in the $uv$ plane are arcs. Let us focus on the track's projection in this plane. To get observation-independent knowledge of the track we can try to combine the three equations of $u$, $v$ and $w$, the aim being to eliminate $H$ from the equation. We end up with an equation linking $u$, $v$, $X$ and $Y$ (the full derivation can be found in $\S$ A.3 ➞):
$$\boxed{u^2 + \left[ \frac{v -\frac{Z}{\lambda} \cos \delta}{\sin \delta} \right]^2 = \left[ \frac{X}{\lambda} \right]^2 + \left[ \frac{Y}{\lambda} \right]^2}$$
One can note that in this particular case, the $uv$ track takes on the form of an ellipse.
<span style="background-color:cyan">TLG:GM: Check if the italic words are in the glossary. </span>
This ellipse is centered at $(0,\frac{Z}{\lambda} \cos \delta)$ in the ($u$,$v$) plane.
The major axis is  $a=\frac{\sqrt{X^2 + Y^2}}{\lambda}$.
The minor axis (along the axis $v$) will be a function of $Z$, $\delta$ and $a$.
We can check this by plotting the theoretical ellipse over the observed portion of the track. (You can fall back to the duration of the observation to see that the track is mapping this ellipse exactly).
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.4: $uv$ track for a baseline at the pole observing at $\delta=90^\circ$ (NCP) and at $\delta=30^\circ$ with the same color conventions as the previous figure.
When observing a source at declination $\delta$, we still have an elliptical shape but centered at (0,0). In the case of a polar interferometer, the full $uv$ track can be covered in 12 hours only due to the symmetry of the baseline.
4.5.1.2.2 The Equatorial interferometer
Let's consider the other extreme scenario: this time, we position the interferometer at the equator. The local zenith is crossed by the Celestial Equator at $\delta=0^\circ$. As seen from the celestial equator, the baseline will not rotate and the projected baseline will no longer correspond to the physical baseline. This configuration is the only case where this happens.
If $\mathbf{b}$ is not rotating, we can intuitively guess that the $uv$ tracks will be straight lines.
Can you write Python code for it?

L=np.radians(90.)
X = D*(np.cos(L)*np.sin(E)-np.sin(L)*np.cos(E)*np.cos(A))
Y = D*np.cos(E)*np.sin(A)
Z = D*(np.sin(L)*np.sin(E)+np.cos(L)*np.cos(E)*np.cos(A))

# At local zenith == Celestial Equator
dec=np.radians(0.)

uEQ = lam**(-1)*(np.sin(h)*X+np.cos(h)*Y)/1e3
vEQ = lam**(-1)*(-np.sin(dec)*np.cos(h)*X+np.sin(dec)*np.sin(h)*Y+np.cos(dec)*Z)/1e3
wEQ = lam**(-1)*(np.cos(dec)*np.cos(h)*X-np.cos(dec)*np.sin(h)*Y+np.sin(dec)*Z)/1e3

# parameters of the UVtrack as an ellipse
aEQ=np.sqrt(X**2+Y**2)/lam/1e3 # major axis
bEQ=aEQ*np.sin(dec)              # minor axi
v0EQ=Z/lam*np.cos(dec)/1e3     # center of ellipse

# Close to Zenith
dec=np.radians(10.)

u10 = lam**(-1)*(np.sin(h)*X+np.cos(h)*Y)/1e3
v10 = lam**(-1)*(-np.sin(dec)*np.cos(h)*X+np.sin(dec)*np.sin(h)*Y+np.cos(dec)*Z)/1e3
w10 = lam**(-1)*(np.cos(dec)*np.cos(h)*X-np.cos(dec)*np.sin(h)*Y+np.sin(dec)*Z)/1e3

a10=np.sqrt(X**2+Y**2)/lam/1e3 # major axis
b10=a*np.sin(dec)              # minor axi
v010=Z/lam*np.cos(dec)/1e3     # center of ellipse

%matplotlib inline
plotBL.UVellipse(u10,v10,w10,a10,b10,v010)
plotBL.UVellipse(uEQ,vEQ,wEQ,aEQ,bEQ,v0EQ)


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.5: $uv$ track for a baseline at the equator observing at $\delta=0^\circ$ and at $\delta=10^\circ$, with the same color conventions as the previous figure.
An equatorial interferometer observing its zenith will see radio sources crossing the sky on straight, linear paths. Therefore, they will produce straight $uv$ coordinates.
4.5.1.1.3 The East-West array
The East-West array is the special case of an interferometer with physical baselines aligned with the East-West direction in the ground-based frame of reference. They have the convenient property of giving a $uv$ coverage which lies entirely on a plane.
If the baseline is aligned with the East-West direction, then the Elevation $\mathcal{E}$ of the baseline is zero and the Azimuth $\mathcal{A}$ is $\frac{\pi}{2}$. Eq. 4.5.1 ⤵ then simplifies considerably:
The only non-zero component of the baseline will be its $Y$-component.
\begin{equation}
\frac{1}{\lambda}
\begin{bmatrix}
X\
Y\
Z
\end{bmatrix}
=
|\mathbf{b_\lambda}|
\begin{bmatrix}
\cos L_a \sin 0 - \sin L_a \cos 0 \cos \frac{\pi}{2}\nonumber\
\cos 0 \sin \frac{\pi}{2} \nonumber\
\sin L_a \sin 0 + \cos L_a \cos 0 \cos \frac{\pi}{2}\
\end{bmatrix}
=
\begin{bmatrix}
0\
|\mathbf{b_\lambda}|\
0 \
\end{bmatrix}
\end{equation}
If we observe a source at declination $\delta_0$ with varying Hour Angle, $H$, we obtain:
\begin{equation}
\begin{pmatrix}
u\
v\
w\
\end{pmatrix}
=
\begin{pmatrix}
\sin H & \cos H & 0\
-\sin \delta_0 \cos H & \sin\delta_0\sin H & \cos\delta_0\
\cos \delta_0 \cos H & -\cos\delta_0\sin H & \sin\delta_0\
\end{pmatrix}
\begin{pmatrix}
0\
|\mathbf{b_\lambda}| \
0
\end{pmatrix}
\end{equation}
\begin{equation}
\begin{pmatrix}
u\
v\
w\
\end{pmatrix}
=
\begin{pmatrix}
|\mathbf{b_\lambda}| \cos H \
|\mathbf{b_\lambda}| \sin\delta_0 \sin H\
-|\mathbf{b_\lambda}|\cos\delta_0\sin H\
\end{pmatrix}
\end{equation}
when $H = 6^\text{h}$ (West)
\begin{equation}
\begin{pmatrix}
u\
v\
w\
\end{pmatrix}
=
\begin{pmatrix}
0 \...
Can you write Python code for it?

H = np.linspace(-6,6,600)*(np.pi/12) #Hour angle in radians
d = 100 #We assume that we have already divided by wavelength

delta = 60*(np.pi/180) #Declination in degrees
u_60 = d*np.cos(H)
v_60 = d*np.sin(H)*np.sin(delta)


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.6: Distribution of the simulated sky in the $l$,$m$ plane.
4.5.1.3.3 Simulating an observation
We will now create a fully-filled $uv$-plane, and sample it using the EW-baseline track we created in the first section. We will be ignoring the $w$-term for the sake of simplicity.
Can you write Python code for it?

u = np.linspace(-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10, num=step_size, endpoint=True)
v = np.linspace(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10, num=step_size, endpoint=True)
uu, vv = np.meshgrid(u, v)
zz = np.zeros(uu.shape).astype(complex)


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.7: Real and imaginary parts of the visibility function. The black curve is the portion of the $uv$ track crossing the visibility.
We now plot the sampled visibilites as a function of time-slots, i.e $V(u_t(t_s),v_t(t_s))$.
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.subplot(121)
plt.plot(z.real)
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Real: sampled visibilities")

plt.subplot(122)
plt.plot(z.imag)
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Imag: sampled visibilities")


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.8: Real and imaginary parts of the visibility sampled by the black curve in Fig. 4.5.7, plotted as a function of time.
Can you write Python code for it?

plt.subplot(121)
plt.imshow(abs(zz),
    extent=[-1*(np.amax(np.abs(u_60)))-10,
            np.amax(np.abs(u_60))+10,
            -1*(np.amax(abs(v_60)))-10,
            np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Amplitude of visibilities")

plt.subplot(122)
plt.imshow(np.angle(zz),
    extent=[-1*(np.amax(np.abs(u_60)))-10,
            np.amax(np.abs(u_60))+10,
            -1*(np.amax(abs(v_60)))-10,
            np.amax(abs(v_60))+10])
plt.plot(u_60,v_60,"k")
plt.xlim([-1*(np.amax(np.abs(u_60)))-10, np.amax(np.abs(u_60))+10])
plt.ylim(-1*(np.amax(abs(v_60)))-10, np.amax(abs(v_60))+10)
plt.xlabel("u")
plt.ylabel("v")
plt.title("Phase of visibilities")


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Figure 4.5.9: Amplitude and Phase of the visibility function. The black curve is the portion of the $uv$ track crossing the visibility.
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.subplot(121)
plt.plot(abs(z))
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Abs: sampled visibilities")

plt.subplot(122)
plt.plot(np.angle(z))
plt.xlabel("Timeslots")
plt.ylabel("Jy")
plt.title("Phase: sampled visibilities")


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Let's compute the $uv$ tracks of an observation of the NCP ($\delta=90^\circ$):
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Let's compute the uv tracks when observing a source at $\delta=30^\circ$:
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Let's express the corresponding physical baseline in ENU coordinates.
Can you write Python code for it?

b_ENU = ant2-ant1                # baseline
D = np.sqrt(np.sum((b_ENU)**2))  # |b|
print str(D/1000)+" km"


b_ENU = ant2-ant1                # baseline
D = np.sqrt(np.sum((b_ENU)**2))  # |b|
print(str(D/1000)+" km")


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
Let's place the interferometer at a latitude $L_a=+45^\circ00'00''$.
Can you write Python code for it?


L = (np.pi/180)*(45+0./60+0./3600)      # Latitude in radians

A = np.arctan2(b_ENU[0],b_ENU[1])
print("Baseline Azimuth="+str(np.degrees(A))+"°")

E = np.arcsin(b_ENU[2]/D)
print("Baseline Elevation="+str(np.degrees(E))+"°")

%matplotlib nbagg
plotBL.sphere(ant1,ant2,A,E,D,L)


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
The source and phase centre coordinates are now given in degrees.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
We create our fully-filled visibility plane. With a "perfect" interferometer, we could sample the entire $uv$-plane. Since we only have a finite amount of antennas, this is never possible in practice. Recall that our sky brightness $I(l,m)$ is related to our visibilites $V(u,v)$ via the Fourier transform. For a bunch of point sources we can therefore write:
$$V(u,v)=\mathcal{F}{I(l,m)} = \mathcal{F}{\sum_k A_k \delta(l-l_k,m-m_k)} = \sum_k A_k e^{-2\pi i (ul_i+vm_i)}$$
Let's compute the total visibilities for our simulated sky.
Can you write Python code for it?

u_track = u_60
v_track = v_60
z = np.zeros(u_track.shape).astype(complex)

s = point_sources.shape
for counter in xrange(1, s[0]+1):
    A_i = point_sources[counter-1,0]
    l_i = point_sources[counter-1,1]
    m_i = point_sources[counter-1,2]
    z += A_i*np.exp(-1*2*np.pi*1j*(u_track*l_i+v_track*m_i))


u_track = u_60
v_track = v_60
z = np.zeros(u_track.shape).astype(complex)

s = point_sources.shape
for counter in range(1, s[0]+1):
    A_i = point_sources[counter-1,0]
    l_i = point_sources[counter-1,1]
    m_i = point_sources[counter-1,2]
    z += A_i*np.exp(-1*2*np.pi*1j*(u_track*l_i+v_track*m_i))


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
We create the dimensions of our visibility plane.
Can you write Python code for it?


s = point_sources.shape
for counter in range(1, s[0]+1):
    A_i = point_sources[counter-1,0]
    l_i = point_sources[counter-1,1]
    m_i = point_sources[counter-1,2]
    zz += A_i*np.exp(-2*np.pi*1j*(uu*l_i+vv*m_i))
zz = zz[:,::-1]


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
We now have everything that describes the $uvw$-track of the baseline (over an 8-hour observational period). It is hard to predict which locus the $uvw$ track traverses given only the three mathematical equations from above. Let's plot it in $uvw$ space and its projection in $uv$ space.
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_1_uv_coverage_uv_tracks.ipynb".
The first task is:
We then convert the ($\alpha$,$\delta$) to $l,m$: TLG:AC:Point to Chapter 3.
* $l = \cos \delta \sin \Delta \alpha$
* $m = \sin \delta\cos\delta_0 -\cos \delta\sin\delta_0\cos\Delta \alpha$
* $\Delta \alpha = \alpha - \alpha_0$
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:

Figure 4.4.14: Distribution of the antennas in a $ENU$ reference frame. Each dot represents an antenna which will be part of $N$-1 baselines.
As for the previous simulations, we will simulate an observation using the VLA as the default case.
 Observation configuration
Can you write Python code for it?

# Observation parameters
c=3e8                                         # Speed of light
f=1420e6                                      # Frequency
lam = c/f                                     # Wavelength

time_steps = 1200                              # time steps
h = np.linspace(-6,6,num=time_steps)*np.pi/12 # Hour angle window

# declination convert in radian

L = np.radians(34.0790)      # Latitude of the VLA
dec = np.radians(34.)


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:

Figure 4.4.15: Snapshot $uv$ coverage of the interferometer. Red and blue points corresponds to symmetric $uv$ points due to the fact that each baseline gives us the measurement of $V_\nu$ at ($u$,$v$) and its complex conjugate $V^*_\nu$ at ($-u$,$-v$).
Depending on the number of elements and their relative distribution on the ground, the shape of the snapshot $uv$ coverage can be very different from on interferometer to the other. One may prefer an antenna distribution which maximizes an instantaneous coverage which spans all over the $uv$ plane (e.g. VLA in Fig. 4.4.15 ⤵) rather than a compact array which samples a smaller portion of the $uv$ plane. With the raw increase of the total sensitivity, improving the instantaneous $uv$ coverage is the main motivation which leads to increase the number of elements and to optimize their relative positions. A good $uv$ coverage can be obtained if the $uv$ plane is sufficiently and smoothly sampled. In the next section, we will see how the Earth contributes to enrich the $uv$ coverage with a variety of samples.

4.4.2.B Integration in time: Earth Rotation Synthesis
The observation of one source during several hours requires the tracking of this source with fringe (or delay) tracking. During the motion of the source in the sky, the projected baselines, as seen from the source, will continuously vary along the elliptical $uv$ tracks.
If the time sampling is high enough, it is possible to sample each $uv$ track along the observation and accumulate a number of $\frac{N(N-1)}{2}\times N_\text{times}$ independent measurements.
The following experiment will give you the possibility to change the number of time steps collected during an observation, from a snapshot observation up to a 12 hour observation.
In the next block, a snapshot observation will be plotted. You can increase the duration of the observation with the slider which defines the total number of time steps for which the $uv$ coverage is computed. Upon a change...
Can you write Python code for it?

from ipywidgets import *
from IPython.display import display

def Interactplot(key,Ntimes):
    print "Ntimes="+str(Ntimes)
    plotBL.plotuv(antennaPosition,L,dec,h,Ntimes,lam)

slider=IntSlider(description="Ntimes",min=2,max=1200,step=100,continuous_update=False)

slider.on_trait_change(Interactplot,'value')
display(slider)
Interactplot("",2)


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:

Figure 4.4.16: Sampled $uv$ coverage of the interferometer for various values of time samples. Red and blue points corresponds to symmetric $uv$ points due to the fact that each baseline give us the measurement of $V_\nu$ at ($u$,$v$) and its complex conjugate $V^*_\nu$ at ($-u$,$-v$).
With an increasing number of time steps, you can see each individual baselines describing its own elliptic $uv$ track in the $uv$ plane. It is easy to understand that the longer the observation, the better the sampling of the visibility function.

4.4.2.C Integration in frequency: Frequency Synthesis
We have seen in $\S$ 4.2 ➞ that the $u$, $v$, $w$ coordinates are usually normalized by $\lambda$. As a consequence, observing with the same interferometer but at a different frequency, give a different set of spatial frequencies. Sampling at a different $u$, $v$ coordinates results in a scaling of the uv coverage of the original frequency. Indeed, for the same snapshot $uv$ coverage at wavelength $\lambda_1$, the $uv$ coverage at $\lambda_2 < \lambda_1$ will be a shrunk version of that a $\lambda_1$.
If the distribution of frequencies is (quasi) continuous, e.g. if the observing system operates in a bandwidth [$f_\text{min}$,$f_\text{max}$], with $N_\text{freqs}$ channels, we can sample contiguously portions of the $uv$ plane with $\frac{N(N-1)}{2}\times N_\text{freqs}$ independent measurements. For each baseline, a radial track will be generated due to the frequency scaling effect of $u$ and $v$.

Warning: It is possible to do this up to a certain point (see the *bandwidth smearing* in [$\S$ 9.3 ➞](../9_Practical/9_3_Observing_smearing.ipynb)).

In the next block, a snapshot observation will be plotted. You can increase the number of contiguous frequency channels  (at 10 MHz steps) with the slider. Upon a change on the slider, a new plot will be generated next to the previous one to enable visual comparisons (Note: allow some time to generate...
Can you write Python code for it?

df=10e6 # frequency step
f0=c/lam # starting frequency
lamb0=lam # starting wavelength

def Interactplot(key,Nfreqs):
    print "Nfreqs="+str(Nfreqs)
    plotBL.plotuv_freq(antennaPosition,L,dec,h,Nfreqs,lamb0,df)

slider=IntSlider(description="Nfreqs",min=1,max=200,step=1,continuous_update=False)

slider.on_trait_change(Interactplot,'value')
display(slider)
Interactplot("",1)


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:
From the list define above, you can select different configurations corresponding to real instrumental layout.
However, if you want to specify yourself the relative positions of the antenna in a $ENU$ reference frame, you can edit the following block and set the variable "custom" to "1".
Can you write Python code for it?

# you need to re-evaluate this box if you modify the array.
antennaPosition=np.genfromtxt(config.value)

# Custom antenna distribution
custom=0
if (custom):
    antennaPosition = np.zeros((10, 2), dtype=float)
    antennaPosition[0,:] = [0,0]
    antennaPosition[1,:] = [-4, 5]
    antennaPosition[2,:] = [4, 5]
    antennaPosition[3,:] = [-10,0]
    antennaPosition[4,:] = [-8,-3]
    antennaPosition[5,:] = [-4,-5]
    antennaPosition[6,:] = [0,-6]
    antennaPosition[7,:] = [4,-5]
    antennaPosition[8,:] = [8,-3]
    antennaPosition[9,:] = [10,0]


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:
Let's plot the distribution of the antennas from the selected (or customized) interferometer:
Can you write Python code for it?

%matplotlib inline
mxabs = np.max(abs(antennaPosition[:]))*1.1;
# make use of pylab librery to plot
fig=plt.figure(figsize=(6,6))
plt.plot((antennaPosition[:,0]-np.mean(antennaPosition[:,0]))/1e3, \
         (antennaPosition[:,1]-np.mean(antennaPosition[:,1]))/1e3, 'o')
plt.axes().set_aspect('equal')
plt.xlim(-mxabs/1e3, mxabs/1e3)
plt.ylim(-mxabs/1e3, (mxabs+5)/1e3)
plt.xlabel("E (km)")
plt.ylabel("N (km)")
plt.title("Antenna positions")


I am working on the file "4_Visibility_Space/4_5_2_UV_Coverage_Improving_Your_Coverage.ipynb".
The first task is:
The snapshot $\boldsymbol{uv}$ coverage
Can you write Python code for it?

%matplotlib inline
Ntimes=3
plotBL.plotuv(antennaPosition,L,dec,h,Ntimes,lam)