I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Choosing the best value of k using a validation set
There remains a question of choosing the value of k to use in making predictions. Here, we use a validation set to choose this value. Write a loop that does the following:

For k in [1, 2, ..., 15]:
Makes predictions for each house in the VALIDATION set using the k-nearest neighbors from the TRAINING set.
Computes the RSS for these predictions on the VALIDATION set
Stores the RSS computed above in rss_all

Report which k produced the lowest RSS on VALIDATION set.

(Depending on your computing environment, this computation may take 10-15 minutes.)
Can you write Python code for it?

def find_best_k(features_train, output_train, features_valid, output_valid, ks):
    rss_all = []
    for k in ks:
        predictions = predict_all_by_nearest_k(features_train, output_train, features_valid, k)
        errors = predictions - output_valid
        rss_all.append(np.dot(errors, errors))
        print rss_all
    return ks[np.argmin(rss_all)]

print find_best_k(features_train, output_train, features_valid, output_valid, range(1,16))


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Compare this predicted value using 4-nearest neighbors to the predicted value using 1-nearest neighbor computed earlier.
Make multiple predictions
Write a function to predict the value of each and every house in a query set. (The query set can be any subset of the dataset, be it the test set or validation set.) The idea is to have a loop where we take each house in the query set as the query house and make a prediction for that specific house. The new function should take the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature matrix for the query set.
The function should return a set of predicted values, one for each house in the query set.
Hint: To get the number of houses in the query set, use the .shape field of the query features matrix. See the documentation.
Can you write Python code for it?

def predict_all_by_nearest_k(features_train, output_train, query_houses, k):
    return [predict_by_nearest_k(features_train, output_train, query_house, k) for query_house in query_houses]

predict_all_by_nearest_k(features_train, output_train, features_test[2:3], 4)


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Compute a single distance
To start, let's just explore computing the "distance" between two given houses.  We will take our query house to be the first house of the test set and look at the distance between this house and the 10th house of the training set.
To see the features associated with the query house, print the first row (index 0) of the test feature matrix. You should get an 18-dimensional vector whose components are between 0 and 1.
Can you write Python code for it?

query_house = features_test[0]

query_house


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Compute multiple distances
Of course, to do nearest neighbor regression, we need to compute the distance between our query house and all houses in the training set.
To visualize this nearest-neighbor search, let's first compute the distance from our query house (features_test[0]) to the first 10 houses of the training set (features_train[0:10]) and then search for the nearest neighbor within this small set of houses.  Through restricting ourselves to a small set of houses to begin with, we can visually scan the list of 10 distances to verify that our code for finding the nearest neighbor is working.
Write a loop to compute the Euclidean distance from the query house to each of the first 10 houses in the training set.
Can you write Python code for it?

[get_euclidean_distance(query_house, house) for house in features_train[:10]]


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Extract features and normalize
Using all of the numerical inputs listed in feature_list, transform the training, test, and validation SFrames into Numpy arrays:
Can you write Python code for it?

feature_list = ['bedrooms',
                'bathrooms',
                'sqft_living',
                'sqft_lot',
                'floors',
                'waterfront',
                'view',
                'condition',
                'grade',
                'sqft_above',
                'sqft_basement',
                'yr_built',
                'yr_renovated',
                'lat',
                'long',
                'sqft_living15',
                'sqft_lot15']
features_train, output_train = get_numpy_data(train, feature_list, 'price')
features_test, output_test = get_numpy_data(test, feature_list, 'price')
features_valid, output_valid = get_numpy_data(validation, feature_list, 'price')


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Import useful functions from previous notebooks
To efficiently compute pairwise distances among data points, we will convert the SFrame into a 2D Numpy array. First import the numpy library and then copy and paste get_numpy_data() from the second notebook of Week 2.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]

    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe[output]

    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
In computing distances, it is crucial to normalize features. Otherwise, for example, the sqft_living feature (typically on the order of thousands) would exert a much larger influence on distance than the bedrooms feature (typically on the order of ones). We divide each column of the training feature matrix by its 2-norm, so that the transformed column has unit norm.
IMPORTANT: Make sure to store the norms of the features in the training set. The features in the test and validation sets must be divided by these same norms, so that the training, test, and validation sets are normalized consistently.
Can you write Python code for it?

features_train, norms = normalize_features(features_train) # normalize training set features (columns)
features_test = features_test / norms # normalize test set by training set norms
features_valid = features_valid / norms # normalize validation set by training set norms


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
It is computationally inefficient to loop over computing distances to all houses in our training dataset. Fortunately, many of the Numpy functions can be vectorized, applying the same operation over multiple values or vectors.  We now walk through this process.
Consider the following loop that computes the element-wise difference between the features of the query house (features_test[0]) and the first 3 training houses (features_train[0:3]):
Can you write Python code for it?

for i in xrange(3):
    print features_train[i]-features_test[0]
    # should print 3 vectors of length 18


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Load in house sales data
For this notebook, we use a subset of the King County housing dataset created by randomly selecting 40% of the houses in the full dataset.
Can you write Python code for it?


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Make a single prediction by averaging k nearest neighbor outputs
Now that we know how to find the k-nearest neighbors, write a function that predicts the value of a given query house. For simplicity, take the average of the prices of the k nearest neighbors in the training set. The function should have the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature vector of the query house, whose price we are predicting.
The function should return a predicted value of the query house.
Hint: You can extract multiple items from a Numpy array using a list of indices. For instance, output_train[[6, 10]] returns the prices of the 7th and 11th training houses.
Can you write Python code for it?

def predict_by_nearest_k(features_train, output_train, query_house, k):
    distances = compute_distances(features_train, query_house)
    argsorted = np.argsort(distances)
    nearest_k = argsorted[:k]
    return np.average(output_train[nearest_k])


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Note that the output of this vectorized operation is identical to that of the loop above, which can be verified below:
Can you write Python code for it?

# verify that vectorization works
results = features_train[0:3] - features_test[0]
print results[0] - (features_train[0]-features_test[0])
# should print all 0's if results[0] == (features_train[0]-features_test[0])
print results[1] - (features_train[1]-features_test[0])
# should print all 0's if results[1] == (features_train[1]-features_test[0])
print results[2] - (features_train[2]-features_test[0])
# should print all 0's if results[2] == (features_train[2]-features_test[0])


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Now you are ready to write a function that computes the distances from a query house to all training houses. The function should take two parameters: (i) the matrix of training features and (ii) the single feature vector associated with the query.
Can you write Python code for it?

def compute_distances(features_train, query_house):
    return np.sqrt(np.sum((features_train - query_house)**2, axis=1))


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Perform k-nearest neighbor regression
For k-nearest neighbors, we need to find a set of k houses in the training set closest to a given query house. We then make predictions based on these k nearest neighbors.
Fetch k-nearest neighbors
Using the functions above, implement a function that takes in
 * the value of k;
 * the feature matrix for the training houses; and
 * the feature vector of the query house
and returns the indices of the k closest training houses. For instance, with 2-nearest neighbor, a return value of [5, 10] would indicate that the 6th and 11th training houses are closest to the query house.
Hint: Look at the documentation for np.argsort.
Can you write Python code for it?

def find_nearest_k(features_train, query_house, k):
    distances = compute_distances(features_train, query_house)
    argsorted = np.argsort(distances)
    nearest_k = argsorted[:k]
    return nearest_k, distances[nearest_k]


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
QUIZ QUESTION
Among the first 10 training houses, which house is the closest to the query house?
Can you write Python code for it?

np.argmin([get_euclidean_distance(query_house, house) for house in features_train[:10]])


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
QUIZ QUESTION
Make predictions for the first 10 houses in the test set using k-nearest neighbors with k=10.

What is the index of the house in this query set that has the lowest predicted value?
What is the predicted value of this house?
Can you write Python code for it?

predictions = predict_all_by_nearest_k(features_train, output_train, features_test[:10], 10)

np.argmin(predictions)

predictions[np.argmin(predictions)]


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
QUIZ QUESTION
Take the query house to be third house of the test set (features_test[2]).  What are the indices of the 4 training houses closest to the query house?
Can you write Python code for it?

top_k, distances = find_nearest_k(features_train, features_test[2], 4)

top_k

distances


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
QUIZ QUESTION
What is the Euclidean distance between the query house and the 10th house of the training set?
Note: Do not use the np.linalg.norm function; use np.sqrt, np.sum, and the power operator (**) instead. The latter approach is more easily adapted to computing multiple distances at once.
Can you write Python code for it?

def get_euclidean_distance(house1, house2):
    delta = house1 - house2
    return np.sqrt(np.sum(delta ** 2))

get_euclidean_distance(query_house, house_10th)

get_euclidean_distance(np.array([0,0]), np.array([3,4]))

np.array([2,2]) ** 2


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
QUIZ QUESTION
What is the RSS on the TEST data using the value of k found above?  To be clear, sum over all houses in the TEST set.
Can you write Python code for it?

'{:.3E}'.format(67361678735491.5)

predictions = predict_all_by_nearest_k(features_train, output_train, features_test, 8)

errors = predictions - output_test

RSS = np.dot(errors, errors)

'{:.3E}'.format(RSS)


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Split data into training, test, and validation sets
Can you write Python code for it?

(train_and_validation, test) = sales.random_split(.8, seed=1) # initial train/test split
(train, validation) = train_and_validation.random_split(.8, seed=1) # split training set into training and validation sets


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
The next step in computing the Euclidean distances is to take these feature-by-feature differences in diff, square each, and take the sum over feature indices.  That is, compute the sum of square feature differences for each training house (row in diff).
By default, np.sum sums up everything in the matrix and returns a single number. To instead sum only over a row or column, we need to specifiy the axis parameter described in the np.sum documentation. In particular, axis=1 computes the sum across each row.
Below, we compute this sum of square feature differences for all training houses and verify that the output for the 16th house in the training set is equivalent to having examined only the 16th row of diff and computing the sum of squares on that row alone.
Can you write Python code for it?

print np.sum(diff**2, axis=1)[15] # take sum of squares across each row, and print the 16th sum
print np.sum(diff[15]**2) # print the sum of squares for the 16th row -- should be same as above


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
To test the code above, run the following cell, which should output a value -0.0934339605842:
Can you write Python code for it?

print diff[-1].sum() # sum of the feature differences between the query and last training house
# should print -0.0934339605842


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
To test the code above, run the following cell, which should output a value 0.0237082324496:
Can you write Python code for it?

print distances[100] # Euclidean distance between the query house and the 101th training house
# should print 0.0237082324496


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
To visualize the performance as a function of k, plot the RSS on the VALIDATION set for each considered k value:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

kvals = range(1, 16)
plt.plot(kvals, rss_all,'bo-')


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
We will also need the normalize_features() function from Week 5 that normalizes all feature columns to unit norm. Paste this function below.
Can you write Python code for it?

def normalize_features(feature_matrix):
    norms = np.linalg.norm(feature_matrix, axis=0)
    return feature_matrix/norms, norms


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
With this result in mind, write a single-line expression to compute the Euclidean distances between the query house and all houses in the training set. Assign the result to a variable distances.
Hint: Do not forget to take the square root of the sum of squares.
Can you write Python code for it?

distances = np.sqrt(np.sum((features_train - query_house)**2, axis=1))


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Above: Fit looks pretty wild, too.  Here's a clear example of how overfitting is associated with very large magnitude estimated coefficients.

#
#
Ridge Regression
Ridge regression aims to avoid overfitting by adding a cost to the RSS term of standard least squares that depends on the 2-norm of the coefficients $\|w\|$.  The result is penalizing fits with large coefficients.  The strength of this penalty, and thus the fit vs. model complexity balance, is controled by a parameter lambda (here called "L2_penalty").
Define our function to solve the ridge objective for a polynomial regression model of any degree:
Can you write Python code for it?

def polynomial_ridge_regression(data, deg, l2_penalty):
    model = graphlab.linear_regression.create(polynomial_features(data,deg),
                                              target='Y', l2_penalty=l2_penalty,
                                              validation_set=None,verbose=False)
    return model


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Add random Gaussian noise to y
Can you write Python code for it?

random.seed(1)
e = graphlab.SArray([random.gauss(0,1.0/3.0) for i in range(n)])
y = y + e


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Compute y
Can you write Python code for it?


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Create random values for x in interval [0,1)
Can you write Python code for it?

random.seed(98103)
n = 30
x = graphlab.SArray([random.random() for i in range(n)]).sort()


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Define a function to fit a polynomial linear regression model of degree "deg" to the data in "data":
Can you write Python code for it?

def polynomial_regression(data, deg):
    model = graphlab.linear_regression.create(polynomial_features(data,deg),
                                              target='Y', l2_penalty=0.,l1_penalty=0.,
                                              validation_set=None,verbose=False)
    return model


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Explore the lasso solution as a function of a few different penalty strengths
We refer to lambda in the lasso case below as "l1_penalty"
Can you write Python code for it?

for l1_penalty in [0.0001, 0.01, 0.1, 10]:
    model = polynomial_lasso_regression(data, deg=16, l1_penalty=l1_penalty)
    print 'l1_penalty = %e' % l1_penalty
    print 'number of nonzeros = %d' % (model.coefficients['value']).nnz()
    print_coefficients(model)
    print '\n'
    plt.figure()
    plot_poly_predictions(data,model)
    plt.title('LASSO, lambda = %.2e, # nonzeros = %d' % (l1_penalty, (model.coefficients['value']).nnz()))


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Fit a degree-2 polynomial
Fit our degree-2 polynomial to the data generated above:
Can you write Python code for it?

model = polynomial_regression(data, deg=2)


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Fit a degree-4 polynomial
Can you write Python code for it?

model = polynomial_regression(data, deg=4)
print_coefficients(model)
plot_poly_predictions(data,model)


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Form and plot our predictions along a grid of x values:
Can you write Python code for it?


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Inspect learned parameters
Can you write Python code for it?


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Let's look at fits for a sequence of increasing lambda values
Can you write Python code for it?

for l2_penalty in [1e-25, 1e-10, 1e-6, 1e-3, 1e2]:
    model = polynomial_ridge_regression(data, deg=16, l2_penalty=l2_penalty)
    print 'lambda = %.2e' % l2_penalty
    print_coefficients(model)
    print '\n'
    plt.figure()
    plot_poly_predictions(data,model)
    plt.title('Ridge, lambda = %.2e' % l2_penalty)

data


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Perform a ridge fit of a degree-16 polynomial using a "good" penalty strength
We will learn about cross validation later in this course as a way to select a good value of the tuning parameter (penalty strength) lambda.  Here, we consider "leave one out" (LOO) cross validation, which one can show approximates average mean square error (MSE).  As a result, choosing lambda to minimize the LOO error is equivalent to choosing lambda to minimize an approximation to average MSE.
Can you write Python code for it?

# LOO cross validation -- return the average MSE
def loo(data, deg, l2_penalty_values):
    # Create polynomial features
    data = polynomial_features(data, deg)

    # Create as many folds for cross validatation as number of data points
    num_folds = len(data)
    folds = graphlab.cross_validation.KFold(data,num_folds)

    # for each value of l2_penalty, fit a model for each fold and compute average MSE
    l2_penalty_mse = []
    min_mse = None
    best_l2_penalty = None
    for l2_penalty in l2_penalty_values:
        next_mse = 0.0
        for train_set, validation_set in folds:
            # train model
            model = graphlab.linear_regression.create(train_set,target='Y',
                                                      l2_penalty=l2_penalty,
                                                      validation_set=None,verbose=False)

            # predict on validation set
            y_test_predicted = model.predict(validation_set)
            # compute squared error
            next_mse += ((y_test_predicted-validation_set['Y'])**2).sum()

        # save squared error in list of MSE for each l2_penalty
        next_mse = next_mse/num_folds
        l2_penalty_mse.append(next_mse)
        if min_mse is None or next_mse < min_mse:
            min_mse = next_mse
            best_l2_penalty = l2_penalty

    return l2_penalty_mse,best_l2_penalty


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Plot results of estimating LOO for each value of lambda
Can you write Python code for it?

plt.plot(l2_penalty_values,l2_penalty_mse,'k-')
plt.xlabel('$\ell_2$ penalty')
plt.ylabel('LOO cross validation error')
plt.xscale('log')
plt.yscale('log')


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Put data into an SFrame to manipulate later
Can you write Python code for it?

data = graphlab.SFrame({'X1':x,'Y':y})
data


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Run LOO cross validation for "num" values of lambda, on a log scale
Can you write Python code for it?

l2_penalty_values = numpy.logspace(-4, 10, num=10)
l2_penalty_mse,best_l2_penalty = loo(data, 16, l2_penalty_values)


I am working on the file "Linear_Regression_Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Woah!!!!  Those coefficients are crazy!  On the order of 10^6.
Can you write Python code for it?


I am working on the file "Linear_SVM.ipynb".
The first task is:
2.1.4 Gradient Descent
Write a function my-gradient-descent that implements the gradient descent algorithm with a constant step-size η. The function is initialised at $w_0 = 0$. The function takes as input η and the maximum number of iterations maxiter. The function returns the output $w_T$ with T = maxiter.
Can you write Python code for it?

def my_gradient_descent(x, y, F, dF, eta=0.001, maxiter=1000):
    w = np.zeros(x.shape[1])
    for i in range(maxiter):
        w = w - eta*dF(x,y,w)
    return w

%time my_gradient_descent(x, y, compute_obj, compute_grad);


I am working on the file "Linear_SVM.ipynb".
The first task is:
Explain how the huberized hinge loss relates to the regular hinge loss and to the misclassification error loss.
Well, Huberized Hinge Loss is a smooth version of Hinge Loss that is differentiable everywhere. As can be seen in the plot, Hinge Loss and Huberized Hinge Loss give loss even when the function output is less than 1. Misclassification Error Loss only acts after the function output falls below 0.
Write a mathematical proof that the huberized hinge loss is differentiable (hint: the huberized hinge loss is defined piecewise).
Assuming yt = $x$ and function $f$ as the huberized hinge loss,
$$
f(x) = \left{\begin{aligned}
&0 &: x > 1+h\
&\frac{(1+h-x)^2}{4h} &: 1-h \leq x \leq 1+h\
&1-x &: x < 1-h
\end{aligned}
\right.$$
Piecewise differentiation of $f$,
$$
f'(x) = \left{\begin{aligned}
&0 &: x > 1+h\
&-\frac{2(1+h-x)}{4h} &: 1-h \leq x \leq 1+h\
&-1 &: x < 1-h
\end{aligned}
\right.$$
Clearly, $f$ is differentiable on each piece.
Let's check $f$'s differentiability on the kinks using $f'(x)$ $= \lim_{p\to x}\frac{f(p)}{p}$.
For $1+h$,
\begin{align}
f'(1+h) =& \lim_{p\to 1+h}\frac{f(p)}{p}
\end{align}
Right limit is,
\begin{align}
f'(1+h) =& \lim_{p\to (1+h)^+}\frac{0}{1+h}
= 0
\end{align}
Left limit is,
\begin{align}
f'(1+h) =& \lim_{p\to (1+h)^-}\frac{\frac{(1+h-(1+h))^2}{4h}}{1+h}
= 0
\end{align}
Clearly, Right limit = Left limit at $1+h$. Therefore, the function is differentiable at this point.
For $1-h$ right limit is,
\begin{align}
f'(1-h) =& \lim_{p\to (1-h)^+}\frac{\frac{(1+h-(1-h))^2}{4h}}{1-h} = \frac{h}{1-h}
\end{align}
Left limit is,
\begin{align}
f'(1-h) =& \lim_{p\to (1-h)^-}\frac{1-(1-h)}{1-h} = \frac{h}{1-h}
\end{align}
Clearly, Right limit = Left limit at $1-h$. Therefore, the function is differentiable at this point.
This proves that the function is differentiable at all points.
Write the analytic expression(s) of the gradient of the huberized hinge loss.
Analytic expression(s) of the gradient of the huberized hinge loss is,
$$
l_{huber-hinge}'(yt) = \left{\begin{aligned}
&0 &: yt >...
Can you write Python code for it?

def compute_obj(x, y, w, C=1.0, h=0.5):
    loss = np.vectorize(huberizedHingeLoss, excluded=['h'])
    return np.dot(w, w) + (C/float(x.shape[0]))*sum(loss(y*np.dot(x,w), h))

def compute_grad(x, y, w, C=1.0, h=0.5):
    p = y*np.dot(x, w)
    gradW = np.zeros(w.shape[0], dtype=float)

    def gradHuberHinge(i, j):
        if p[i] > 1+h:
            return 0
        elif abs(1-p[i]) <= h:
            return ((1+h-p[i])/(2*h))*(-y[i]*x[i][j])
        else:
            return (-y[i]*x[i][j])

    for j in range(w.shape[0]):
        sum_over_i = 0.0
        for i in range(x.shape[0]):
            sum_over_i += gradHuberHinge(i,j)
        gradW[j] = 2*w[j] + (C/float(x.shape[0]))*sum_over_i
    return gradW

def add_bias_column(x):
    return np.append(x, np.ones(x.shape[0]).reshape(x.shape[0],1), axis=1)

n_samples = 1000
n_features = 30
x, y = make_blobs(n_samples, n_features, centers=2)
y[y==0] = -1
x = add_bias_column(x)

w = np.zeros(n_features+1)

compute_obj(x, y, w)

compute_grad(x, y, w)


I am working on the file "Linear_SVM.ipynb".
The first task is:
Implement backtracking line search (google it). Profile your
code, optimise in terms of speed.
Can you write Python code for it?

def backtracked_gradient_descent(x, y, F, dF, maxiter=100):
    w = np.zeros(x.shape[1])
    beta = 0.8
    F_vals = np.zeros(maxiter)
    for i in range(maxiter):
        eta = 1
        val = F(x,y,w)
        grad = dF(x,y,w)
        while F(x, y, (w - eta * grad)) > val - ((eta/2.) * grad.dot(grad)):
            eta = beta * eta
        #print eta
        w = w - eta*grad
        F_vals[i] = F(x,y,w)
    return w, F_vals

x_copy = add_bias_column(x_train)
_, F_vals = backtracked_gradient_descent(x_copy, y_train, compute_obj, compute_grad)
iterations = np.arange(100)
plt.plot(iterations, F_vals)


I am working on the file "Linear_SVM.ipynb".
The first task is:
Run experiments for various values of the fixed step-size η.
Can you write Python code for it?

svm = my_svm()
for k in range(0,9):
    eta = 0.1**k
    svm.fit(x_train, y_train, eta)
    print eta, svm.score(x_test, y_test)


I am working on the file "Linear_SVM.ipynb".
The first task is:
You will use here Linear SVM with huberized hinge loss, trained using your gradient descent algorithm. Write a function my-svm for Linear SVM, that can used for training (by calling my-gradient-descent) and testing.
Can you write Python code for it?

class my_svm(object):
    def __init__(self):
        self.learnt_w = None

    def fit(self, x_train, y_train, eta=0.01, max_iter=1000):
        x_copy = add_bias_column(x_train)
        self.learnt_w = my_gradient_descent(x_copy, y_train, compute_obj, compute_grad, eta, max_iter)

    def predict(self, x_test):
        x_copy = add_bias_column(x_test)
        y = np.dot(x_copy, self.learnt_w)
        y[y<0] = -1
        y[y>0] = 1
        return y

    def score(self, x_test, y_test):
        y_predict = self.predict(x_test)
        bools = y_predict == y_test
        accuracy = bools[bools == True].shape[0]/float(bools.shape[0])
        return accuracy


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
1/ Math fundamentals
Linear algebra builds upon high school math concepts like:
  - Numbers (integers, rationals, reals, complex numbers)
  - Functions ($f(x)$ takes an input $x$ and produces an output $y$)
  - Basic rules of algebra
  - Geometry (lines, curves, areas, triangles)
  - The cartesian plane
2/ Intro to linear algebra
Linear algebra is the study of vectors and matrices.
Vectors
Can you write Python code for it?

# define two vectors
u = Vector([2,3])
v = Vector([3,0])
u

v

plot_vecs(u, v)
autoscale_arrows()


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
4/ Geometrical linear algebra
Points, lines, and planes are geometrical objects that are conveniently expressed using the language of vectors.
Points
A point $p=(p_x,p_y,p_z)$ refers to a single location in $\mathbb{R}^3$.
Can you write Python code for it?


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Adjugate-matrix formula
The adjugate matrix of the matrix $A$ is obtained by replacing each entry of the matrix with a partial determinant calculation (called minors). The minor $M_{ij}$ is the determinant of $A$ with its $i$th row and $j$th columns removed.
Can you write Python code for it?


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Augmented matrix approach
$$
 \left[ \, A \, | \,  \mathbb{1} \, \right]
 \qquad
 -\textrm{Gauss-Jordan elimination}\rightarrow
 \qquad
 \left[ \, \mathbb{1} \, | \,  A^{-1} \, \right]
$$
Can you write Python code for it?

AUG = A.row_join(eye(2))
AUG

# perform row operations until left side of AUG is in RREF
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG[1,:] = AUG[1,:]/3
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG

# the inverse of A is in the right side of RREF(AUG)
AUG[:,2:5]  # == A-inverse

# verify A times A-inverse gives the identity matrix...
A*AUG[:,2:5]


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Basis
When we describe the vector as the coordinate pair $(4,6)$, we're implicitly using the standard basis $B_s = { \hat{\imath}, \hat{\jmath} }$. The vector $\hat{\imath} \equiv (1,0)$ is a unit-length vector in the $x$-direciton,
and $\hat{\jmath} \equiv (0,1)$ is a unit-length vector in the $y$-direction.
To be more precise when referring to vectors, we can indicate the basis as a subscript of every cooridnate vector $\vec{v}=(4,6)_{B_s}$, which tells $\vec{v}= 4\hat{\imath}+6\hat{\jmath}=4(1,0) +6(0,1)$.
Can you write Python code for it?

# the standard basis
ihat = Vector([1,0])
jhat = Vector([0,1])

v = 4*ihat + 6*jhat
v

# geomtrically...
plot_vecs(ihat, jhat, 4*ihat, 6*jhat, v)
autoscale_arrows()


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Determinant
Can you write Python code for it?

a, b, c, d  = symbols('a b c d')
A = Matrix([[a,b],
            [c,d]])

A.det()

# Consider the parallelogram with sides:
u1 = Vector([3,0])
u2 = Vector([2,2])

plot_vecs(u1,u2)
plot_vec(u1, at=u2, color='k')
plot_vec(u2, at=u1, color='b')
autoscale_arrows()

# What is the area of this parallelogram?

# base = 3, height = 2, so area is 6

# Compute the area of the parallelogram with sides u1 and u2 using the deteminant
A = Matrix([[3,0],
            [2,2]])
A.det()


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Lines
A line is a one dimensional infinite subset of $\mathbb{R}^3$ that can be described as
$$
   \ell:  { p_o + \alpha \vec{v} \ | \ \forall \alpha \in \mathbb{R} }.
$$
Can you write Python code for it?

po = Point([1,1,1])
v = Vector([1,1,0])

plot_line(v, po)


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Matrix operations

Addition (denoted $A+B$)
Subtraction, the inverse of addition (denoted $A-B$)
Scaling by a constant $\alpha$  (denoted $\alpha A$)
Matrix-vector product (denoted $A\vec{x}$, related to linear transformations)
Matrix product (denoted $AB$)
Matrix inverse (denoted $A^{-1}$)
Trace (denoted $\textrm{Tr}(A)$)
Determinant (denoted $\textrm{det}(A)$ or $|A|$)

In linear algebra we'll extend the notion of funttion $f:\mathbb{R}\to \mathbb{R}$, to functions that act on vectors called linear transformations.  We can understand the properties of linear transformations $T$ in analogy with ordinary functions:
\begin{align}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}! \to \mathbb{R}^{m}
    \end{array} \
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
     \textrm{output }      T(\vec{x})\in \mathbb{R}^m \
    g\circ! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{inverse transformation } T^{-1} \
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T \
    \textrm{image of } f
         & \ \Leftrightarrow \
             \begin{array}{l}
             \textrm{image of } T
              \end{array}
\end{align}
Linear property
$$
  T(a\mathbf{x}_1 + b\mathbf{x}_2) = aT(\mathbf{x}_1) + bT(\mathbf{x}_2)
$$
Matrix-vector product representation of linear transformations
Equivalence between linear transformstions $T$ and matrices $M_T$:
$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$
$$
  \vec{y} = T(\vec{x})
  \qquad
  \Leftrightarrow
  \qquad
  \vec{y} = M_T\vec{x}
$$
3/ Computational...
Can you write Python code for it?

# represent as an augmented matrix
AUG = Matrix([
  [1, 2,   5],
  [3, 9,  21]])
AUG

# eliminate x_1 in second equation by subtracting 3x times the first equation
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG

# simplify second equation by dividing by 3
AUG[1,:] = AUG[1,:]/3
AUG

# eliminate x_2 from first equation by subtracting 2x times the second equation
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Projection operation
A projection of the vector $\vec{v}$ in the direction $\vec{d}$ is denoted $\Pi_{\vec{d}}(\vec{v})$. The formula for computing the projections uses the dot product operation:
$$
   \Pi_{\vec{d}}(\vec{v})
      \ \equiv \
      (\vec{v} \cdot \hat{d}) \hat{d}
      \ =  \
      \left(\vec{v} \cdot \frac{\vec{d}}{\|\vec{d}\|} \right) \frac{\vec{d}}{\|\vec{d}\|}.
$$
Can you write Python code for it?

def proj(v, d):
    """Computes the projection of vector `v` onto direction `d`."""
    return v.dot( d/d.norm() )*( d/d.norm() )

v = Vector([2,2])
d = Vector([3,0])
proj_v_on_d = proj(v,d)

plot_vecs(d, v, proj_v_on_d)
autoscale_arrows()


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
Recall definition $A^{-1}A=\mathbb{1}$, and we just observed that $E_3E_2E_1 A =\mathbb{1}$, so it must be that $A^{-1}=E_3E_2E_1$.
Can you write Python code for it?


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
The basic projection operation can be used to compute projection onto planes, and compute distances between geomteirc objects (page 192).
Bases and coordinate projections
See page 225 in v2.2 of the book:

Different types of bases
Orthonormal
Orthogonal
Generic

Change of basis operation

Vector spaces
See page 231 in v2.2 of the book.
Vector space techniques
See page 244 in the book.
5/ Linear transformations
See page 257 in v2.2 of the book.
Vector functions
Functions that take vectors as inputs and produce vectors as outputs:
$$
  T:\mathbb{R}^{n}! \to \mathbb{R}^{m}
$$
Matrix representation of linear transformations
$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$
Input and output spaces
We can understand the properties of linear transformations $T$, and their matrix representations $M_T$ in analogy with ordinary functions:
\begin{align}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}! \to \mathbb{R}^{m}        \
    \textrm{represented by the matrix } M_T \in \mathbb{R}^{m \times n}
    \end{array} \
    %
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \
    %\textrm{compute }
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
    %      \textrm{compute matrix-vector product }
     \textrm{output }      T(\vec{x}) \equiv M_T\vec{x} \in \mathbb{R}^m \
    %\textrm{function composition  }
    g\circ! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \equiv M_SM_T \vec{x} \
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{matrix inverse } M_T^{-1} \
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T  \equiv  \textrm{null space of } M_T \equiv \mathcal{N}(A)  \
    \textrm{image of } f
   ...
Can you write Python code for it?


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
The deteminant is non-zero so inverse exists.
Can you write Python code for it?


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
The same vector $\vec{v}$ will correspond to the a different pair of coefficients if a differebt basis is used.
For example, if we use the basis $B^\prime = { (1,1), (1,-1) }$, the same vector $\vec{v}$ must be expressed as $\vec{v} = 5\vec{b}1 +(-1)\vec{b}_2=(5,-1){B^\prime}$.
Can you write Python code for it?

# another basis B' = { (1,1), (1,-1) }
b1 = Vector([ 1, 1])
b2 = Vector([ 1, -1])

v = 5*b1 + (-1)*b2
v

# How did I know 5 and -1 are the coefficients w.r.t basis {b1,b2}?
# Matrix([[1,1],[1,-1]]).inv()*Vector([4,6])

# geomtrically...
plot_vecs(b1, b2, 5*b1, -1*b2, v)
autoscale_arrows()


I am working on the file "Linear_algebra_chapters_overview.ipynb".
The first task is:
This augmented matrix is in reduced row echelon form (RREF), and corresponds to the system of equations:
\begin{align}
1x_1  \ \  \qquad &= 1 \
       1x_2 &= 2,
\end{align}
so the the solution is $x_1=1$ and $x_2=2$.
Matrix equations
See page 177 in v2.2 of the book.
Matrix product
Can you write Python code for it?

a,b,c,d,e,f, g,h,i,j = symbols('a b c d e f  g h i j')
A = Matrix([[a,b],
            [c,d],
            [e,f]])
B = Matrix([[g,h],
            [i,j]])

A, B

A*B

def mat_prod(A, B):
    """Compute the matrix product of matrices A and B."""
    assert A.cols == B.rows, "Error: matrix dimensions not compatible."
    m, ell = A.shape  # A is a  m x ell  matrix
    ell, n = B.shape  # B is a  ell x n  matrix
    C = zeros(m,n)
    for i in range(0,m):
        for j in range(0,n):
            C[i,j] = A[i,:].dot(B[:,j])
    return C

mat_prod(A,B)

# mat_prod(B,A)


I am working on the file "Linear_regression/Linear_regression.ipynb".
The first task is:
梯度下降法求解

目标函数选取要合适一些, 前边乘以适当的系数.
注意检验梯度的计算是否正确...
Can you write Python code for it?

def f(w,X,y):
    return ((X.dot(w)-y)**2/(2*1000)).sum()

def grad_f(w,X,y):
    return  (X.dot(w) - y).dot(X)/1000

w0 = np.array([100.0,100.0,100.0,100.0])

epsilon = 1e-10

alpha = 0.1

check_condition = 1

while  check_condition > epsilon:
    w0 += -alpha*grad_f(w0,X,y)
    check_condition = abs(grad_f(w0,X,y)).sum()
print w0


I am working on the file "Linear_regression/Linear_regression.ipynb".
The first task is:
随机梯度下降法求解

Stochastic gradient descent
使用了固定步长
一开始用的0.1, 始终达不到给定的精度

于是添加了判定条件用来更新步长.
Can you write Python code for it?

def cost_function(w,X,y):
    return (X.dot(w)-y)**2/2

def grad_cost_f(w,X,y):
    return  (np.dot(X, w) - y)*X

w0 = np.array([1.0, 1.0, 1.0, 1.0])

epsilon = 1e-3

alpha = 0.01

# 生成随机index,用来随机索引数据.
random_index = np.arange(1000)
np.random.shuffle(random_index)

cost_value = np.inf #初始化目标函数值

while abs(grad_f(w0,X,y)).sum() > epsilon:
    for i in range(1000):
        w0 += -alpha*grad_cost_f(w0,X[random_index[i]],y[random_index[i]])

    #检查目标函数变化趋势, 如果趋势变化达到临界值, 更新更小的步长继续计算
    difference = cost_value - f(w0, X, y)
    if difference < 1e-10:
        alpha *= 0.9
    cost_value = f(w0, X, y)
print w0


I am working on the file "Linear_sampling.ipynb".
The first task is:

Geometric reasoning
If we think of a 2D line, we can reason that if a feature lies more than its radius from the line then it is not intersected. Here's the situation for a grid:

If there's a set of lines, then the problem is symmetric across the gaps between lines. The width of the 'invisible strip' (grey in the figure, which shows a grid rather than a swath) is the size of the gap minus the width of the feature. If we divide the width of the invisible strip by the width of the gap, we get a probability of randomly distributed features falling into the invisible strip.
Parallel 2D lines
Can you write Python code for it?

line_spacing = 3.0  # km, the width of the gap

# 'Invisible' means 'not intersected'
width_invisible = line_spacing - feature_width
prob_invisible = width_invisible / line_spacing
prob_visible = 1 - prob_invisible

print "Probability of intersecting a given feature:", prob_visible


I am working on the file "Linear_sampling.ipynb".
The first task is:
<hr />
Interpretation accuracy
We can apply Bayes' theorem to update the prior probability (above) with the reliability of our interpretation (due to lack of resolution, data quality, or skill).
Can you write Python code for it?

reliability = 0.75
trials = 120

intersect_interpret = prob_visible * reliability * trials
intersect_xinterpret = prob_visible * (1 - reliability) * trials
xintersect_interpret = (1 - prob_visible) * (1 - reliability) * trials
xintersect_xinterpret = (1 - prob_visible) * reliability * trials

t = [[intersect_interpret, intersect_xinterpret], [xintersect_interpret, xintersect_xinterpret]]


I am working on the file "Linear_sampling.ipynb".
The first task is:
Just for fun, we can — using the original formula in the reference — calculate the population size from an observation:
Can you write Python code for it?

observed = 37.5
population = (observed * area) / (2. * length * feature_width)

print "Population:", population


I am working on the file "Linear_sampling.ipynb".
The first task is:
We can compute the probability of a given feature being correctly interpreted:
Can you write Python code for it?

prob_correct = intersect_interpret / (intersect_interpret + xintersect_interpret)
print "Probability of a feature existing if interpreted:", prob_correct


I am working on the file "Linear_sampling.ipynb".
The first task is:
We can use a pandas DataFrame to show a quick table:
Can you write Python code for it?

from pandas import DataFrame
df = DataFrame(t, index=['Intersected', 'Not intersected'], columns=['Interpreted','Not interpreted'])
df


I am working on the file "Linear_sampling.ipynb".
The first task is:
We can use the distribution to estimate the probability of seeing no features. Then we can use the survival function (or, equivalently, 1 - the cumulative distribution function), sf(x, n, p), to tell us the probability of drawing more than x in n trials, given a success probability p:
Can you write Python code for it?

p = "Probability of intersecting"
print p, "no features:", scipy.stats.binom.pmf(0, population, prob_visible)
print p, "at least one:", scipy.stats.binom.sf(0, population, prob_visible)
print p, "at least two:", scipy.stats.binom.sf(1, population, prob_visible)
print p, "all features:", scipy.stats.binom.sf(population-1, population, prob_visible)


I am working on the file "Linear_sampling.ipynb".
The first task is:
We're going to need a binomial distribution, scipy.stats.binom.
Can you write Python code for it?


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Again the eight, although this was not planned. But now let's quickly save the data to file, so that we can go on and inspect the findings further (commandline covers this command via python3 autocogs.py alignments4):
Can you write Python code for it?

alm.output('tsv', filename='mikronesian-aligned', ignore='all', prettify=False)


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Checking Coverage
For cognate detection, it is not only important to have good phonetic transcriptions (ideally segmented in such a form that they were checked by an experienced linguist), but also to make sure that there are enough words in your data. If the data is too sparse, even human linguists would not be able to find any signal based on regular sound correspondences, provided they see the languages the first time and don't know their history (which is the situation for every algorithm). Following an earlier study by List (2014b), we know now that at least 100 word pairs for languages as disparate as English and French are needed to provide a solid basis for automatic cognate detection. But when dealing with a large dataset of different languages, which necessarily contains a number of gaps (not all concepts can be elicited in the sources, field work has not provided enough details, etc.), it can be deleterious if the mutual coverage between the languages is low.
By mutual coverage, I mean the number of comparable word pairs (with the same concept) for each language pair in a given dataset. We can compare different aspects of mutual coverage, such as the average mutual coverage, where we average the number of available word pairs, or the minimal mutual coverage which provides the smallest mutual coverage of any pair of languages. In addition, one can also ask for the subset fulfilling a minimal mutual coverage for all language pairs, and this task would return the subset of languages in a Wordlist which all have at least the mutual coverage specified by the user. LingPy offers now (in version 2.5.1) solutions for all these problems, but since the last problem is considerably hard and computationally intensive, we won't discuss it here, but will instead simply check the minimal mutual coverage which holds for all languages in our sample. So we try to find the lower bound of concept pairs which all languages have in common (type $ python3 autocogs.py coverage1 in commandline).
Can you write Python code for it?

from lingpy.compare.sanity import mutual_coverage_check, mutual_coverage_subset
for i in range(210, 0, -1):
    if mutual_coverage_check(wl, i):
        print("Minimal mutual coverage is at {0} concept pairs.".format(i))
        break


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Evaluation with LingPy
LingPy has a couple of evaluation methods implemented, and since we have original expert cognate judgments from ABVD, we can compare our findings against theirs. Comparing cognate set accuracy is not necessarily a trivial problem, as we deal with cluster comparisons, which is a topic that was debated a lot in circles outside of linguistics, and it would lead us too far away if we discussed it in detail here now. For a linguistic viewpoint with a brief working example of our preferred method, the B-Cubed scores (see Hauer and Kondrak 2011, Bagga and Baldwin 1998, and Amigo et al. 2009), see List, Greenhill, and Gray (2017). What you need to know, however, is that evaluation in NLP circles usually comes along with the concepts of precision, recall, and f-score. Initially, I found them rather difficult to grasp, as historical linguists usually think in terms of false positives and false negatives. In order to understand the idea, one should think that an algorithm for cognate detection can basically do two things either right or wrong: it could cluster words which are not cognate, or it could fail to cluster words as cognate. In the first case, we would measure this in terms of precision, by counting, how often the algorithm proposes correct or incorrect answers, and in the latter case, we measure the proportion of cognate sets which are missed. In the B-Cubed measure we use, this translates roughly to a measure of false/true positives vs. false/true negatives, but it is not entirely the same. The f-score computes the harmonic mean, which summarizes both values, and we usually want to improve the f-score and we use it to compare different algorithms with each other.
Let's start and do this comparison now, by loading the respective functions from the LingPy evaluation module, and computing precision, recall, and f-scores for all our different automatically inferred cognate sets with respect to the gold standard. The gold standard is located in the column COGID of the input file, so we need...
Can you write Python code for it?

from lingpy.evaluate.acd import bcubes, diff
wl = Wordlist('mikronesian-lexstat.tsv')

for res in ['turchinid', 'scaid', 'lexstatid', 'infomap']:
    print('{0:10}\t{1[0]:.2f}\t{1[1]:.2f}\t{1[2]:.2f}'.format(
        res,
        bcubes(wl, 'cogid', res, pprint=False)
    ))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Further Use
This is by far not all you can do with LingPy, but it should be almost enough for a first demonstration. Yet in the reminder of this tutorial, I want to show quickly, what you can do, once the data has been analysed. We will start by showing how to visualize and inspect the data word by word and alignment by alignment, using the EDICTOR tool, which I have been developing for the last couple of years, and which has now been released in a first beta version (List 2017). Then I will show how we can evaluate the accuracy of our findings with help of LingPy's evaluation module. Last not least, I will show how you can export distance data, Nexus files, and trees with help of LingPy's export facilities.
Inspection with the EDICTOR
The EDICTOR is a web-based tool written entirely in JavaScript which helps to load LingPy's tsv-files and inspect them in various ways. In order to use it, make sure to use FireFox, GoogleChrome, or Safari as your web browser, open the url http://edictor.digling.org, and you will see a welcome screen as follows:

In order to inspect your data, all you have to do is to take your file, and drag it onto the "browse" field. Alternatively, you can also click on the browse field and then select the file by navigating to its location. Once this has been done, you will need to apply some tweaks with the EDICTOR which would take to long to be discussed here in text. However, for those who only have the chance to read this tutorial online, consider having a look at the help instructions of the EDICTOR itself, or by checking out the little demo video (of which the music was cut, due to copy-right problems, play "Like a Rolling Stone" in Dylan's original version, when watching it to get the right feeling).
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('IyZuf6SmQM4')


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
If you have problems in your code, you will be asked if you want to exclude the sequences automatically. As a result, a logfile, called errors.log will be created and point you to all erroneous sequences which contain segments which LingPy does not recognize. Let us quickly introduce some bad sequences by just converting randomly all [e] sounds to the latter A (capitals are never accepted in the normal sound class models of LingPy) and see what we get then. For this, we even do not need to re-write the data, we just add another row where we change the content, give it a random name (we call it "tokens", as this also signals LingPy that the input should be treated as a sequence and not as a string), and specify this for the LexStat instance method as the column in the file where the segments are. We first load the data as Wordlist and then pass that data directly to LexStat ($ python3 autocogs.py trigger-errors in commandline):
Can you write Python code for it?

wl = Wordlist('mikronesian.tsv')

# add new column "segments" and replace data from column "tokens"
wl.add_entries('segments', 'tokens', lambda x: ['A' if y == 'e' else y for y in x])

lex = LexStat(wl, segments='segments', check=True)


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
If you now check the file errors.log, you will find a long file with the following first ten lines:
text
ID  Tokens  Error-Type
1572    <bad character in tokens at «A»>    g a t o a A l i m a
3320    <bad character in tokens at «A»>    p a A _ ʔ a ʔ u r u
5145    <bad character in tokens at «A»>    l i m a s A f u l u
5696    <bad character in tokens at «A»>    r i m a _ t A k a u
12  <bad character in tokens at «A»>    p a A
3327    <bad character in tokens at «A»>    p a A
5153    <bad character in tokens at «A»>    A _ f aː
Each row starts with the ID of the word which is contaminated (and this links to the row-ID of your input file), it is followed by a description of the error-type, and then by a segmented form of the word form. LingPy then also creates a file called lingpy-DATE-cleaned.tsv (DATE meaning the date of the day you run LingPy), in which all contaminated words have been excluded, and this file is read in again, if you pressed "y", and will be the one to run the analysis.
LingPy thus tries to make the enterprise of cognate detection quite convenient for you as a user, but you should be warned not to use files containing errors for publications, but only for personal test purposes, in order to improve your data. If LingPy does not recognize characters, you should not globally exclude them as a reaction, but instead you should try to improve your data until it is publication-ready. Otherwise, the results will much likely be disappointing anyway.
Overview on Algorithms
LingPy comes along with four pre-defined cognate detection algorithms. These algorithms are all contained in the LexStat class which often confuses users, as one of the algorithms provided by LexStat is also called lexstat. Internally, however, it makes sense, as all algorithms were created at the same time, when I published the LexStat algorithm (List 2012b), so I would write it into one class which I called "LexStat" and for the paper, I then also decided to call the algorithm "LexStat"...
Can you write Python code for it?

lex = LexStat('mikronesian.tsv', segments='tokens', check=True)

# run the dolgopolsky (turchin) analysis, which is threshold-free
lex.cluster(method='turchin')

# show the cognate sets, stored in "turchinid" for the words for "Eight"
eight = lex.get_dict(row='Eight') # get a dictionary with language as key for concept "eight"
for k, v in eight.items():
    idx = v[0] # index of the word, it gives us access to all data
    print("{0:20} \t {1} \t{2}".format(lex[idx, 'doculect'], lex[idx, 'value'], lex[idx, 'turchinid']))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Loading the Data into a Wordlist Object
Loading the data into LingPy is straightforward. LingPy has a couple of classes which are specifically designed to handle word list data, and these classes provide all similar basic functions, plus additional ones for specific purposes:

Wordlist: Basic class with core functionality, allows to read, modify, and write word list data, also allows to calculate distance matrices from cognate sets as well as rudimentary tree reconstruction functions (UPGMA, Sokal and Michener 1958, Neighbor-joining, Saitou and Nei 1987).
Alignments: Class allows to align all cognate sets in a word list. Requires one column which stores the cognate sets as well as a column for doculect, concept, and transcription (default: ipa) or user-defined segmented transcription (default: tokens). Alignments can be carried out in different ways, the algorithms follow the ones first described in List (2012a).
LexStat: Core class for automatic cognate detection, following the algorithm first described in List (2012b) and later expanded in List (2014), and List, Greenhill, and Gray (2017).
Partial: Recent algorithm proposed in List, Lopez, and Bapteste (2016), allows -- provided data is morpheme-segmented -- to search for partial cognates in the data.

We will start with the basic Wordlist object to illustrate some core facilities below (commandline: $ python3 autocogs.py wordlist1).
Can you write Python code for it?

from lingpy import *

# load the wordlist
wl = Wordlist('polynesian.tsv')

# count number of languages, number of rows, number of concepts
print("Wordlist has {0} languages and {1} concepts across {2} rows.".format(wl.width, wl.height, len(wl)))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Note as a last point that the conversion to sound classes is the major check whether LingPy has "understood" your input. If LingPy does not find a class symbol corresponding to a given segment, it will use the default character "0" to indicate this failure of converting a given sound sequence. This zero will be treated as an uninvited guest in most comparisons. It won't be aligned with other elements and will score negatively in the automatic cognate detection routines. You should thus try to avoid this by making sure that your sequences do not contain any errors. When carrying out cognate the detection analysis, we have a specific keyword check which you can set to True to make sure that all sequences with zeros in sound classes are excluded before the analysis is carried out. But you can easily write a Python function to check yourself in only a few lines (write $ python3 autocogs.py errors in commandline):
Can you write Python code for it?

from collections import defaultdict

def check_sequence(seq):
    """Takes a segmented string as input and returns erroneously converted segments."""
    cls = tokens2class(seq, 'dolgo') # doesn't matter which model to take, all cover the same character range
    errors = defaultdict(int)
    for t, c in zip(seq, cls):
        if c == '0':
            errors[t] += 1
    return errors

word = "θiɣatEra"
seq = ipa2tokens(word)
for error, count in check_sequence(seq).items():
    print("The symbol <{0}> occurs {1} times and is not recognized.".format(error, count))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
So by accessing the attributes width we retrieve the number of languages and with height, we retrieve the number of concepts. This follows the logic inherent in the classical format in which linguists prepare their spreadsheets, namely by placing concepts in the first column and languages in the rest of the columns. Classical linguists would thus represent the data from the table above as follows:
CONCEPT | Emae_1030 | RennellBellona_206 | Tuvalu_753 | Sikaiana_243 | Penrhyn_235 | Kapingamarangi_217
--- | --- | --- | --- | --- | --- | ---
eight | βaru | baŋu | valu | valu | varu | waru
... | ... | ... | ... | ... | ... | ...
The disadvantage of this annotation is, however, that we can only store one value in each cell, and we will create inconsistencies if we try to mix information per cell. For that reason, we maintain the strict tabular representation where each word is placed in one row, but internally, LingPy represents the data in multi-dimensional tables in which languages are thought to be placed in the columns and concepts in the rows.
There are multiple ways in LingPy to inspect and manipulate data using the Wordlist class, but it would go to far to mention them all here, so we will restrict it to one example, by which we retrieve the values from the six languages above for the entry "Eight", using the wordlist.get_dict() function, and refer the users to a longer tutorial which is online.
Can you write Python code for it?

# get all indices for concept "hand", `row` refers to the concepts here, while `col` refers to languages
eight = wl.get_dict(row='Eight', entry='value')
for taxon in ['Emae_1030', 'RennellBellona_206', 'Tuvalu_753', 'Sikaiana_243', 'Penrhyn_235',  'Kapingamarangi_217']:
    print('{0:20}'.format(taxon), '  \t', ', '.join(eight[taxon]))


wl.calculate('dst', ref='infomap', mode='swadesh')
wl.output('dst', filename='mikronesian')


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
There are more complicated algorithms available, for example, library-based alignment, following the T-Coffee algorithm (Notredame et al. 2000, based on a so-called "library" which is created before the tree is built. This algorithm, you can use like this ($ python3 autocogs.py alignments2 in commandline):
Can you write Python code for it?


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
This value is definitely good enough for our purpose, given the rule of thumb which says that below a minimal mutual coverage of 100 one should not do language-specific cognate detection analyses. If the coverage is lower, this does not mean you need to give up automatic cognate detection, but it means you should not use the language-specific LexStat method but rather a language-independent method, which does not require the information on potential sound correspondences (but will also tend to identify more false positives).
Although, as I just said, the value is good enough, we should further reduce the data a bit to make sure we can inspect them better later on (otherwise, the analyses may also take a lot of time if you run them on computers with insufficient power). So what we will do right now is testing the mutual_coverage_subset method which returns a subset of languages for which a given minimal mutual coverage holds. We will then export our Wordlist object to file by specifying these languages as our subset (type $ python3 autocogs.py coverage2 in commandline):
Can you write Python code for it?

count, results = mutual_coverage_subset(wl, 200)
coverage, languages = results[0]
print('Found {0} languages with an average mutual coverage of {1}.'.format(count, coverage))

# write word list to file
wl.output("tsv", filename="mikronesian", subset=True, rows=dict(doculect = "in "+str(languages)),
          ignore='all', prettify=False)

# load the smaller word list
wl = Wordlist('mikronesian.tsv')

# print basic characteristics
print("The new word list has {0} languages and {1} concepts across {2} words.".format(
    wl.width, wl.height, len(wl)))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
This was not very spectacular, as we have not yet seen what happened. We can visualize the alignments from the commandline by picking a particular cognate set and printing the alignments on screen. The alignments are added in a specific column called alignments as a default (but which can be modified by specifying another value with the keyword alignments passed to the initialization method for the Alignments class). Additionally, they are in a class attribute, which is called Alignments.msa and which can store multiple different alignment analyses. This code is a bit hacky, due to the nested dictionary, so we won't go into the details right now but just illustrate how we can print a couple of the aligned cognate sets (python3 autocogs.py alignments5 in commandline):
Can you write Python code for it?

for cog in ['1']:

    msa = alm.msa['infomap'][cog]
    for i, idx in enumerate(msa['ID']):
        print(
            '{0:20}'.format(msa['taxa'][i]),
            '\t',
            alm[idx, 'concept'],
            '\t',
            '\t'.join(msa['alignment'][i])
        )


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
This will create a file called "mikronesian-lexstat.tsv.diff" in your folder. If you open this file, you will see that it contrasts the "cogid" with the "infomap" numbers by putting them with words and languages in two columns:
text
Concept: above, False Positives: no, False Negatives: yes
Austral_128             nuʔa           1       1
Tahitian_173            niʔa           1       1
EastFutuna_210          luga           1       2
Hawaiian_52             i_luna         1       2
Maori_85                i_ruŋa         1       2
NorthMarquesan_38       ʔuna           1       2
Nukuria_1212            i_aruna        1       2
Pukapuka_152            luŋa           1       2
Rapanui_264             ruga           1       2
Rarotongan_58           ruŋa           1       2
RennellBellona_206      guŋa           1       2
Samoan_118              i_luŋa         1       2
TongaTongaIslands_136   ʔi_ʔoluŋa      1       2
VaeakauTaumako_375      luŋa           1       2
I assume that the format is more or less self-explaining. Note only one thing: the cognate-ids are always re-computed for each concept set, so you cannot compare them with the ones you find in the text. This is, however, justified, as it helps to compare, since LingPy's diff method re-numbers the cognate identifiers for each concept in order to maximally contrast findings by the algorithm and the ones proposed by the experts.
Exporting Distances and Nexus Files
It is clear that for many of those who consult automatic cognate detection, they use the methods in order to be able to do more with the data afterwards. LingPy so far supports quite a few different ways to write your data to file for further use in other software packages. A complete integration of Nexus files which transport all information which might be relevant for BEAST, however, does not exist yet (but will be added at some point sooner than later).
You can nevertheless export to nexus, and what the algorithm exports are your presence-absence patterns extracted from the cognate sets ($...
Can you write Python code for it?

wl = Wordlist('mikronesian-lexstat.tsv')
wl.output('paps.nex', filename='mikronesian', ref='infomap', missing='?')


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Unfortunately, we do not have enought time to go into all the details of alignment analyses, and especially more complex aspects of different alignment modes (global, local, semi-global, etc.), different basic algorithms (extended algorithm for secondary alignments), but also how sound classes, the internal representation format in LingPy are integrated into the algorithms, cannot be treated here in full. I can only refer the interested users to both the extensive online documentation at lingpy.org, as well as my aforementioned book on sequence comparison.
Cognate Detection
Checking the Data
I assume that you have thoroughly checked your data manually before running cognate detection analyses. I also assume that you do not have any of the following problems in your data:

extensive number of synonyms in one language
multiple variant forms for the same word form
data merged from different sources without adjusting the phonetic transcription
mutual coverage below 100 words per language pair

Before running the cognate detection analysis, you may, however, still want to check whether LingPy recognizes all your data correctly. Here, a very simple way to achieve this, is to load the LexStat class with the specific keyword check set to True:
Can you write Python code for it?

lex = LexStat('mikronesian.tsv', check=True, segments='tokens')


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
We now do the same for the "sca" method, but since this methods are not threshold free, we will need to define a threshold. We follow the default value we know from experience, which is 0.45. We then print out the same data, but this time including the cognate jugdments by all three methods ($ python3 autocogs.py cognates-sca in commandline):
Can you write Python code for it?

lex.cluster(method="sca", threshold=0.45)

for k, v in eight.items():
    idx = v[0]
    print("{0:20} \t {1} \t{2} \t {3} ".format(
        lex[idx, 'doculect'],
        lex[idx, 'value'],
        lex[idx, 'turchinid'],
        lex[idx, 'scaid']))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
Well, no improvement for "eight", but we will see later in detail, and for now, we just write the data to file, this time in plain text, without the additional information, but with the additional columns with our analyses.
Can you write Python code for it?

lex.output('tsv', filename='mikronesian-lexstat', ignore='all', prettify=False)


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
You can see from these examples, that LingPy's ipa2tokens function automatically identifies diacritics and the like, but that you can also tweak it to some extent. If the sequence contains white space, as in the first example, ipa2tokens will split by white space and assume that the data is already segmented. We won't go into the details of this and other functions here, but you should consider giving the documentation a proper read before you start spending time on segmenting your data manually. At the same time, when trusting LingPy's default algorithm for segmentation, you should always make sure after using it that the segmentations make sense. If they are largely wrong or problematic, you should refine them before running any automatic cognate detection method.
An alternative is to use the segments package by Steven Moran, whose main idea is more comprehensively described in Moran and Cysouw (2017). We have in fact been using it for our working example in order to segment the ABVD data on Polynesian languages properly, but it would require more time to introduce you to all the details at this point. Just keep in mind that segmentation is crucial for automatic cognate detection, and that it is your responsibility to make sure that the algorithms "understand" the data you give them to analyse.
Above, I wrote that LingPy takes care of the word form as one of the units of the linguistic sign in the classical "Saussurean" model. But how can we know whether LingPy recognizes a symbol or not? For this, we need to understand what LingPy does internally with word forms. Here, LingPy follows Dolgopolsky's (1964) idea of "sound classes", namely the idea that we can break down the complexity inherent in phonetic transcription to some major classes of sounds so that those sounds represent some kind of a coherent unit. Dolgopolsky was thinking of sounds which often occur in correspondence relation to each other, assuming that there is a certain sound-correspondence probability inherent in all sounds (see also Brown,...
Can you write Python code for it?

word = "θiɣatɛra"
segs = ipa2tokens(word)

# iterate over sound class models and write them in converted version
for m in ['dolgo', 'sca', 'asjp', 'art']:
    print(word, ' -> ', ''.join(tokens2class(segs, m)), '({0})'.format(m))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
You can see that there is not much difference in the results for this very item, but you should not underestimate the different power of the methods, as we will see later on when running an evaluatin analysis. For now, trust me that in general the results are quite different.
Let us now run (for those who managed to install the python-igraph package) an additional analysis which was shown to yield even better results. Here, we do still use the "lexstat" approach, but we use "infomap" (Rosvall and Bergstroem 2008) as our cluster method. This method is network-based rather than agglomerative (as is LingPy's default), and was shown to yield consistently better results in combination with "lexstat" (List, Greenhill, and Gray 2017). In order to avoid that we override the content of the column "lexstatid", we now pass a specific keyword, called ref (the "reference" of the output) and set its value to "infomap". We also choose a different threshold, the one we empirically determined from tests on different language families (see ibd. for details, ($ python3 autocogs.py cognates-infomap in commandline):
Can you write Python code for it?

lex.cluster(method="lexstat", threshold=0.55, ref="infomap", cluster_method='infomap')
for k, v in eight.items():
    idx = v[0]
    print("{0:20} \t {1} \t{2} \t {3} \t {4} \t {5}".format(
        lex[idx, 'doculect'],
        lex[idx, 'value'],
        lex[idx, 'turchinid'],
        lex[idx, 'scaid'],
        lex[idx, 'lexstatid'],
        lex[idx, 'infomap']
    ))


I am working on the file "LingPy/Sequence Comparison with LingPy (Johann-Mattis List).ipynb".
The first task is:
You can see, that the "infomap" method is in fact working one point better than the normal "lexstat" method, and you can also see how deep the difference between the correspondence-informed methods and the other methods is. As a last way to inspect the data, we will now use the diff function to create a file that contrasts the expert cognate sets with the ones inferred by Infomap ($ python3 autocogs.py diff in terminal).
Can you write Python code for it?

diff(wl, 'cogid', 'infomap', pprint=False)


I am working on the file "Linking_and_brushing.ipynb".
The first task is:
Now go select the "Western arm" of the star-forming region (in Glue) and make a subset of it
Can you write Python code for it?

catalog = dc[1]

j_minus_h = catalog['Jmag'] - catalog['Hmag']


I am working on the file "Linking_and_brushing.ipynb".
The first task is:
See many examples of configuring plot tools at http://bokeh.pydata.org/en/latest/docs/user_guide/tools.html
Interacting with Glue
Can you write Python code for it?

#import glue

# Quick way to launch Glue
#from glue import qglue
#qglue()


I am working on the file "Linking_and_brushing.ipynb".
The first task is:
We can add something to our catalog and it shows up in Glue.
Can you write Python code for it?

catalog['jmh'] = j_minus_h

hmag = catalog.id['Hmag']
jmag = catalog.id['Jmag']


I am working on the file "Linking_and_brushing.ipynb".
The first task is:
We can define a new subset group here or in Glue
Can you write Python code for it?

jmhred = (jmag - hmag) > 1.5
dc.new_subset_group('j - h > 1.5', jmhred)

dc.subset_groups

dc.subset_groups[2].label

catalog.subsets

catalog.subsets[0]['Jmag']

mask = catalog.subsets[0].to_mask()

new_catalog = w5catalog[mask]


I am working on the file "Linking_and_brushing.ipynb".
The first task is:
We will output to a static html file.
The output_notebook() function can output to the notebook,
but with 50,000 points it really slows down.
Can you write Python code for it?

output_file('sdss_color_color.html')

TOOLS = "pan,wheel_zoom,reset,box_select,poly_select,help"

# create a new plot and add a renderer
left = figure(tools=TOOLS, width=400, height=400, title='SDSS g-r vs u-g', webgl=True)
left.x('umg', 'gmr', source=source)

# create another new plot and add a renderer
right = figure(tools=TOOLS, width=400, height=400, title='SDSS i-z vs r-i')
right.x('rmi', 'imz', source=source)

p = gridplot([[left, right]])

show(p)


I am working on the file "Linrad resampler.ipynb".
The first task is:
Function to stack up the pulses and plot.
Can you write Python code for it?

def plot_audio(file):
    audio = np.memmap(file, offset=0x28, dtype=np.int16)
    #audio = np.abs(hilbert(audio)) # improves display, but takes computation time
    pulse_len = 4800
    lines = len(audio)//pulse_len
    plt.imshow(np.abs(audio)[:lines*pulse_len].reshape(lines, pulse_len), cmap='viridis', vmin = 0, vmax=1e4)
    del audio


I am working on the file "Linrad resampler.ipynb".
The first task is:
Generate a WAV IQ file at 48ksps containing pulses with 10Hz repetition rate. The duration of each pulse is 10ms.
Can you write Python code for it?

samp_rate = 48000
pulse_on = samp_rate//100
pulse_off = samp_rate//10 - pulse_on
duration = 600 # seconds
n_pulses = duration * 10
amplitude = 0.01

pulse = np.array([1]*pulse_on + [0]*pulse_off, dtype='float32')
i = amplitude*np.tile(pulse, n_pulses)

iq = np.zeros((len(i),2), dtype='float32')
iq[:,0] = i

wavfile.write('/home/daniel/pulses.wav', rate=samp_rate, data=iq)
del iq
del i


I am working on the file "Linrad resampler.ipynb".
The first task is:
This is with ntpd running. The resampling rate variations sometimes seem much greater.
Can you write Python code for it?


I am working on the file "Linrad resampler.ipynb".
The first task is:
We open the WAV IQ file in Linrad using adwav. The output soundcard is a snd-aloop. The pulses are tuned in SSB mode and the audio output is recorded at 48ksps in Linrad using 'W'.
The pulses are stacked and plotted. They are slanted, which means that the resampling rate is constant but not correct (not 1:1 from input to output). The results are the same regardless of whether ntpd is running.
Can you write Python code for it?


I am working on the file "Linux Tools & Tricks.ipynb".
The first task is:
Get the Active Window on Linux
Get active window title in X
- Original Code had the following error: TypeError: can't use a string pattern  on a bytes-like object
<br/>
Obtain Active window using Python

Corected code is now here

"import wnck" only works with python 2.x, using python3.x pypie and wx were the only options I found so far
Can you write Python code for it?

import sys
import os
from subprocess import PIPE, Popen
import re

def get_active_window_title():
    root = Popen(['xprop', '-root', '_NET_ACTIVE_WINDOW'], stdout=PIPE)

    for line in root.stdout:
        m = re.search(b'^_NET_ACTIVE_WINDOW.* ([\w]+)$', line)
        if m != None:
            id_ = m.group(1)
            id_w = Popen(['xprop', '-id', id_, 'WM_NAME'], stdout=PIPE)
            break

    if id_w != None:
        for line in id_w.stdout:
            match = re.match(b"WM_NAME\(\w+\) = (?P<name>.+)$", line)
            if match != None:
                return match.group("name")

    return "Active window not found"

get_active_window_title()

import time
time.sleep(2)
get_active_window_title()


I am working on the file "LissajousFigures.ipynb".
The first task is:
The parameters of the oscillators. You need not change anything starting from the time periods.
If the graph is not smooth enough, smoothen it.
Can you write Python code for it?

# amplitudes
A = np.array([1,1])

# frequencies
f = np.array([3,5])

# phased difference
delta = 0*np.pi/4

# time periods
T = 1/f

# the time grid
tmin, tmax = 0, lcm(f[0], f[1])
dt=(tmax-tmin)/2000
t=np.arange(tmin, tmax+dt, dt)

# the oscillations - mutually perpendicular
x = A[0]*np.cos(2*np.pi*f[0]*t )
y = A[1]*np.sin(2*np.pi*f[1]*t+delta)

fig = pl.figure(1, figsize=(7,7))
ax = fig.add_subplot(111)
pl.plot(x,y)
pl.grid('on')
d=1.1*max(A)
pl.xlim((-d, d))
pl.ylim((-d, d))
ax.set_aspect(1)


I am working on the file "List Comprehensions.ipynb".
The first task is:
Example 3
Lets see how to add in if statements:
Can you write Python code for it?

# Check for even numbers in a range
lst = [x for x in range(11) if x % 2 == 0]

lst


I am working on the file "List Comprehensions.ipynb".
The first task is:
Example 4
Can also do more complicated arithmetic:
Can you write Python code for it?

# Convert Celsius to Fahrenheit
celsius = [0,10,20.1,34.5]

fahrenheit = [ ((float(9)/5)*temp + 32) for temp in Celsius ]

fahrenheit


I am working on the file "List Comprehensions.ipynb".
The first task is:
This is the basic idea of a list comprehension. If you're familiar with mathematical notation this format should feel familiar for example: x^2 : x in { 0,1,2...10}
Lets see a few more example of list comprehensions in Python:
Example 2
Can you write Python code for it?

# Square numbers in range and turn into list
lst = [x**2 for x in range(0,11)]

lst


I am working on the file "List+Comprehensions.ipynb".
The first task is:
Example 2
Can you write Python code for it?

# Generate Squares for 10 numbers
list1 = [x**2 for x in range(10)]
print(list1)


I am working on the file "List+Comprehensions.ipynb".
The first task is:
Example 3
Can you write Python code for it?

# List comprehension with a filter condition
list2 = [x**2 for x in range(10) if x%2 == 0]
print(list2)


I am working on the file "List+Comprehensions.ipynb".
The first task is:
Example 4
Can you write Python code for it?

# Use list comprehension to filter out numbers
words = "Hello 12345 World".split()
numbers = [w for w in words if w.isdigit()]
print(numbers)


I am working on the file "List+Comprehensions.ipynb".
The first task is:
Example 6
Can you write Python code for it?


I am working on the file "ListaEX_02.ipynb".
The first task is:
Exercício 2: Crie uma função para determinar se um ano é bissexto.
O ano é bissexto se for múltiplo de 400 ou múltiplo de 4 e não múltiplo de 100. Utilize o operador de resto da divisão (%) para determinar se um número é múltiplo de outro.
Can you write Python code for it?

print (Bissexto(2000)) # True
print (Bissexto(2004)) # True
print (Bissexto(1900)) # False


I am working on the file "ListaEX_02.ipynb".
The first task is:
Exercício 3: Crie uma função que receba três valores x, y, z como parâmetros e retorne-os em ordem crescente.
O Python permite que você faça comparações relacionais entre as 3 variáveis em uma única instrução:
Python
x < y < z
Can you write Python code for it?

print (Crescente(1,2,3))
print (Crescente(1,3,2))
print (Crescente(2,1,3))
print (Crescente(2,3,1))
print (Crescente(3,1,2))
print (Crescente(3,2,1))
print (Crescente(1,2,2))


I am working on the file "ListaEX_02.ipynb".
The first task is:
Exercício 5: Crie uma função que receba as coordenadas cx, cy, o raio r correspondentes ao centro e raio de uma circunferência e receba também coordenadas x, y de um ponto.
A função deve retornar True se o ponto está dentro da circunferência e False, caso contrário.
Can you write Python code for it?

print (Circunferencia(0,0,10,5,5) ) # True
print (Circunferencia(0,0,10,15,5))  # False


I am working on the file "ListaEX_02.ipynb".
The first task is:
Exercício 6:
A Estrela da Morte é uma arma desenvolvida pelo império para dominar o universo.

Um telescópio digital foi desenvolvido pelas forças rebeldes para detectar o local dela.
Mas tal telescópio só consegue mostrar o contorno das circunferências encontradas indicando o centro e o raio delas.
Sabendo que uma Estrela da Morte é definida por:

O raio de uma circunferência for 10 vezes maior que o raio da outra
A circunferência menor se encontrar totalmente dentro da maior
O contorno da circunferência menor está a pelo menos 2 unidades de distância do contorno da maior

Faça uma função (utilizando os exercícios anteriores), para detectar se duas circunferências definidas por (cx1,cy1,r1) e (cx2,cy2,r2) podem formar uma Estrela da Morte.
Bônus: plote as circunferências utilizando a biblioteca gráfica.
Can you write Python code for it?

import math

print (EstrelaMorte(0,0,20,3,3,10))
print (EstrelaMorte(0,0,200,3,3,10))
print (EstrelaMorte(0,0,200,195,3,10))


I am working on the file "ListaEX_02.ipynb".
The first task is:
Exercício 7: Crie uma função para determinar as raízes reais da equação do segundo grau:
$$
a.x^{2} + b.x + c = 0
$$
Faça com que a função retorne:

Uma raíz quando $b^2 = 4ac$
Raízes complexas quando $b^2 < 4ac$
Raízes reais, caso contrário

Utilize a biblioteca cmath para calcular a raíz quadrada para números complexos.
Can you write Python code for it?

import math, cmath

print (RaizSegundoGrau(2,4,2) ) # -1.0
print (RaizSegundoGrau(2,2,2))  # -0.5 - 0.9j, -0.5+0.9j
print (RaizSegundoGrau(2,6,2))  # -2.6, -0.38


I am working on the file "ListaEX_03.ipynb".
The first task is:
Avançados
Exercício 07: Dado um número $x$, crie a função Segmenta() que divida esse número em segmentos de $d$ dígitos. Ex.:
$$
x = 123456 \
d = 2 \
[12,34,56]
$$
Note que é mais fácil extrair os $d$ dígitos da direita para a esquerda. Nesse caso, crie também uma função Inverte() que gera o inverso de um número.
Crie então uma outra função chamada Crescente() que verifica se a lista de valores estão em ordem crescente.
Finalmente, crie uma função que verifique qual o menor valor de $d$ que gera um segmento de números crescentes.
Can you write Python code for it?


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 02: Crie uma função Fatorial() que calcule n!.
$$n! = n \cdot (n-1) \cdot (n-2) \dots 1$$
Pense primeiro nos casos $0!$ e $1!$ antes de criar o algoritmo.
Can you write Python code for it?

print (Fatorial(0))
print (Fatorial(1))
print (Fatorial(5))
print (Fatorial(10))


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 03: Crie uma função para calcular o $n$-ésimo termo da sequência de Fibonacci.
A sequência de Fibonacci é definida por:
$$
F_0 = F_1 = 1 \
F_n = F_{n-1} + F_{n-2}
$$
Can you write Python code for it?

for i in range(10):
    print (Fibonacci(i),)
print()


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 04: Crie uma função SomaSerie() que efetue a soma $S$ das seguintes séries para um valor de $n$:
$$
S_1 = 1 + \frac{1}{3} + \frac{1}{5} + \dots + \frac{1}{2n+1} \
S_2 = \frac{1}{2} + \frac{1}{4} + \dots + \frac{1}{2n} \
S = S_1 - S_2
$$
Can you write Python code for it?

print (SomaSeries(3))
print (SomaSeries(10))
print (SomaSeries(100))


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 05: Crie 2 funções para calcular o Coeficiente Binomial:
$$\binom nk = \frac{n!}{k!\,(n-k)!} \quad \text{for }\ 0\leq k\leq n.$$
Na primeira versão, utilize a função fatorial do Exercício 02.
Na segunda versão, tente simplificar ao máximo a equação para reduzir o número de operações efetuadas.
Can you write Python code for it?

print (Binomial1(4,2), Binomial2(4,2))
print (Binomial1(10,4), Binomial2(10,4))


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 09: Um Fractal é uma imagem com padrões auto-similares. Uma forma de desenhar um fractal é através do Conjunto Julia, que trabalha com números complexos. Podemos construir uma matriz com valores do conjunto de Julia da seguinte forma:
m = []
Para i de -1.5 até 1.5 com passo p:
    l = []
    Para j de 1.0 até -1.0 com passo -p:
        l.append( Julia( complex(i,j), c ) )
     m.append(l)
Para percorrermos os valores em uma faixa de números reais, podemos utilizar o comando arange() da biblioteca numpy:
python
import numpy as np
m = []
for i in np.arange(-1.5,1.5,p):
    l = []
    for j in np.arange(1.0,-1.0,-p):
        l.append( Julia( complex(i,j), c ) )
     m.append(l)
Ou usando syntatic sugar:
python
m = [ [Julia(complex(i,j),c) for j in np.arange(1.0,-1.0,-step)] for i in np.arange(-1.5,1.5, step) ]
O comando complex() cria um número complexo através dos valores passados como parâmetro. A variável $c$ é um número complexo de base passada como parâmetro.
A função Julia(x,y) é definida como:

Para n em range(200)
se o valor absoluto de x for maior que 4, retorne n
senão, faça x = x*x + y
ao final das repetições, retorne n

Implemente a função Julia(x,y) e a função GeraFractal(c, passo ) que recebe o número complexo semente e o tamanho do passo.
Essa função deve gerar a tal matriz m, os comandos para plotagem já são dados!
Não utilize um passo menor que 0.005, pois poderá ocupar muita memória do computador.
Experimente com diversos valores de $c$.
Can you write Python code for it?

%pylab inline
import matplotlib.pylab as plt
import numpy as np

def Julia(x,y):
    ...
    return n

def GeraFractal( c, step ):
    m = [ [Julia(complex(i,j),c) for j in np.arange(1.0,-1.0,-step)] for i in np.arange(-1.5,1.5, step) ]
    fig = plt.figure(figsize=(15,15))
    plt.imshow(m, cmap="RdGy", extent=[-2,2,-2,2])

GeraFractal(-0.06+0.67j, 0.005)


I am working on the file "ListaEX_03.ipynb".
The first task is:
Exercício 10: A música We Will Rock You do grupo Queen inovou na criação de uma técnica de mixagem hoje conhecida como reverberação não-harmônica.
Para criar o efeito de uma multidão, o guitarrista e físico Brian May, gravou o efeito de batida e multiplicou essas batidas com pequenos atrasos.
Para evitar que as múltiplas cópias se sobrepusessem, ele fez com que os atrasos fossem múltiplos de números primos.
Vamos utilizar para essa tarefa as bibliotecas wave e audioop do Python e o som Palmas.wav fornecido no site do curso.
A primeira tarefa é criar uma função Primo(n) que verifica se o número $n$ é primo.
A segunda tarefa é completar a função GeraMusica( ecos ) que gera $n$ repetições do som contido no arquivo, com $n$ sendo a quantidade de números primos de 1 até ecos.
Can you write Python code for it?

import wave, audioop

def GeraMusica( ecos ):

    # abre o arquivo fornecido
    som1 = wave.open('Palmas.wav')
    # criar o arquivo destino, o segundo parâmetro 'w'
    # indica que o arquivo vai ser aberto para escrita
    som2 = wave.open('WeWillRockYou.wav', 'w')

    # faz com que o som criado tenha os mesmos parâmetros do som original
    som2.setnchannels( som1.getnchannels() )
    som2.setsampwidth( som1.getsampwidth() )
    som2.setframerate( som1.getframerate() )

    # lê o arquivo de audio e normaliza o som
    frames = som1.readframes( som1.getnframes() )
    taxa = 0x7fff/ audioop.max(frames, som1.getsampwidth())
    frames = audioop.mul(frames, som1.getsampwidth(), taxa)

    # gera uma lista de números primos de 1 até ecos
    primos = ???

    # o som está em 44100Hz com apenas 1 canal, isso significa
    # que cada decimo de segundo contem 44100/10 posições da lista
    # cada eco será colocado com atraso de mult*p, onde p é cada número primo
    mult = som1.getframerate()*som1.getnchannels()/10

    # cada eco tem um decaimento de amplitude
    decaimento = 0.3

    # para cada numero primo
    for p in primos:
        # a partir de onde começaremos a colocar o eco
        atraso = ???

        # frame2 será o trecho que copiaremos para o eco
        # ele sera o conteúdo da lista frames do elemento 0 até len()-atraso
        frame2 = ???

        # Vamos multiplicar frame2 pelo decaimento, para reduzir o som
        frame2 = audioop.mul( frame2, som1.getsampwidth(), decaimento)

        # frame2 será adicionado a frames a partir do atraso
        # utilize a função audioop.add( frame1, frame2, som1.getsampwidth() )
        frame1 = ???
        frame2 = audioop.add( frame1, frame2, som1.getsampwidth())

        # concatene frames de 0 até atraso com frame2
        frames = ???

        # reduza o decaimento pela metade
        decaimento = ???

    # grava o novo som
    som2.writeframes( frames )

    som1.close()
   ...


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 02: Crie uma função ConverteData() que recebe uma string no formato DIA-MES-ANO e retorne uma string no formato DIA-MES_NUMERO-ANO. Exemplo:
'01-MAI-2000' => '01-05-2000'.
Você pode separar a string em uma lista de strings da seguinte maneira:
data = '01-MAI-2000'
lista = data.split('-')
print lista # ['01','MAI','2000']
E pode juntar novamente usando join:
lista = ['01','05', '2000']
data = '-'.join(lista)
print data # '01-05-2000'
Can you write Python code for it?

# converter data no formato 01-MAI-2000 em 01-05-2000
def ConverteData(data):

print (ConverteData('01-MAI-2000'))


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 03: Crie um dicionário chamado Dados que tenha como chave um número de 2 até 12 e o valor seja uma lista contendo todas as combinações dos valores de dois dados que resulta nessa chave.
Can you write Python code for it?

# crie um dicionário em que a chave é um número de 2 a 12
# e o valor é uma lista de combinações de dois dados que resulta na chave
Dados =...

for chave, valor in Dados.iteritems():
    print (chave, valor)


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 04: Crie um dicionário onde as chaves são palavras em português e os valores sua tradução para o inglês. Use todas as palavras do texto do exercício 01.
Crie uma função Traduz() que recebe o nome do arquivo texto como parâmetro e retorna uma string com a tradução.
Can you write Python code for it?

# crie um pequeno dicionário de inglês para português e use para traduzir frases simples
import codecs

def Traduz(texto):

print (Traduz('exemplo.txt'))


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 05: A Cifra de César é uma forma simples de criptografar um texto. O procedimento é simples:

dado um número $n$
crie um mapa de substituição em que cada letra será substituida pela n-ésima letra após ela no alfabeto. Ex.:

n = 1
A -> B
B -> C
...
n = 2
A -> C
B -> D
...
A Codificação é feita substituindo cada letra da frase pelo correspondente do mapa.
Para Decodificar uma frase, basta criar um mapa utilizando $-n$ ao invés de $n$.
Crie uma função ConstroiDic() que recebe um valor n como entrada e cria um mapa de substituição. Utilize a constante string.ascii_letters para obter todas as letras do alfabeto.
Note que o mapa é cíclico, ou seja, para n=1, a letra Z tem que ser substituida pela letra A. Isso pode ser feito utilizando o operador '%'.
Crie uma função Codifica() que recebe como parâmetros uma string contendo uma frase e um valor para n, essa função deve construir o dicionário e retornar a frase codificada.
Para Decodificar o texto, basta chamar a função Codifica() pasando -n como parâmetro.
Can you write Python code for it?

# cifra de César
import string

def ConstroiDic(n):

def Codifica(frase, n):

l = Codifica('Vou tirar dez na proxima prova', 5)
print (l)
print (Codifica(l,-5))


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 06: Faça uma função que leia a tabela periódica de um arquivo (você construirá esse arquivo) e armazene em um dicionário.
Can you write Python code for it?


I am working on the file "ListaEX_04.ipynb".
The first task is:
Exercício 07: Assista o vídeo abaixo e crie uma lista com os personagens da letra da música.
Em seguida, utilizando dois laços for percorra essa lista e escreva a letra da música.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('BZzNBNoae-Y', 640,480)

# velha a fiar


I am working on the file "ListaEX_05.ipynb".
The first task is:
Ele é representado por uma matriz tridimensional de dimensões $133 \times 100 \times 3$, representando a altura, largura e as cores. A dimensão das cores representa a intensidade das cores Vermelho, Verde e Azul, respectivamente, com um valorde 0 a 255.
Exemplo:
mario[0,0,:] = [255, 255, 255]
Temos intensidade máxima nas três cores, que forma a cor branca.
Faça um algoritmo que percorra a imagem do Mario e, toda vez que encontrar um pixel de cor (112, 104, 0), troque ele para azul ([0,0,255]) ou preto ([0,0,0]) dependendo da altura desse pixel (o cabelo, bigode e sapato devem ser pretos, a blusa deve ser azul).
Can you write Python code for it?

plt.figure(figsize=(4, 4))
plt.imshow(mario)
plt.axis('off')
plt.show()


I am working on the file "ListaEX_05.ipynb".
The first task is:
Exercício 03: A matriz Z representa uma matriz com coordenadas cartesianas, converta cada linha para coordenada polar, dado que:
$$
X, Y -> \rho, \theta \
\rho = \sqrt(X^2 + Y^2) \
\theta = arctan2(Y,X)
$$
Faça utilizando laço for (em Z2) e sem usar laços (Z3).
Can you write Python code for it?

# converter matriz N x 2 representando coordenadas cartesianas em polares
Z = np.array([[10.,2.],[5.,3.5],[1.2,4.1],[3.2,3.14]])
Z2 = np.zeros_like(Z)
Z3 = np.zeros_like(Z)

print (Z2)
print ()
print (Z3)


I am working on the file "ListaEX_05.ipynb".
The first task is:
Exercício 05: Dado um vetor X e uma matriz Y, construa um vetor D em que cada elemento i é a distância Euclidiana entre X e o i-ésimo elemento de Y.
Can you write Python code for it?