I am working on the file "Assessment.ipynb".
The first task is:
Below I use resample and count to create a by-minute count of requests and then do the same for several subsets filtered by various clues to malicious behavior I have discovered while exploring the dataset.
Can you write Python code for it?

overall_by_minute = merged_test_df['IP'].resample('T').count()
top_ip_by_minute = merged_test_df.loc[merged_test_df['IP'] ==  '225.19.49.85']['IP'].resample('T').count()
auth_by_minute = merged_test_df.loc[merged_test_df['path'] == '/auth']['IP'].resample('T').count()
zh_by_minute = merged_test_df.loc[merged_test_df['Accept-Language'].str.contains('zh', na = False)]['IP'].resample('T').count()
mozillia_by_minute = merged_test_df.loc[merged_test_df['User-Agent'].str.contains('Mozillia', na = False)]['IP'].resample('T').count()
tc_by_minute = merged_test_df.loc[merged_test_df["Content-Length"].isin(top_content.index)]['IP'].resample('T').count()
thousand_by_minute = merged_test_df.loc[merged_test_df["X-Forwarded-For"].isin(thousand_arr)]['IP'].resample('T').count()

graph_df = pd.concat([overall_by_minute, top_ip_by_minute, auth_by_minute, zh_by_minute, mozillia_by_minute, tc_by_minute, thousand_by_minute], axis=1)

graph_df.columns = ['overall', 'top_ip', 'auth', 'zh', 'mozillia', 'top_content', 'thousand']

import matplotlib
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import numpy as np

hours = mdates.HourLocator()
Fmt = mdates.DateFormatter('%Y')

t = graph_df.index
a = graph_df['overall']
b = graph_df['top_ip']
c = graph_df['auth']
d = graph_df['zh']
e = graph_df['mozillia']
f = graph_df['top_content']
g = graph_df['thousand']

fig, ax = plt.subplots(figsize=(20, 10))
ax.plot(t, a, label='overall')
ax.plot(t, b, label='top_ip')
ax.plot(t, c, label='auth')
ax.plot(t, d, label='zh')
ax.plot(t, e, label='mozillia')
ax.plot(t, f, label='top_content')
ax.plot(t, g, label='thousand')

ax.set(xlabel='time (m)', ylabel='traffic')
ax.grid()
ax.xaxis.set_major_locator(hours)
ax.format_xdata = mdates.DateFormatter('%Y')
fig.autofmt_xdate()

plt.legend(loc=1)
plt.show()

fig, ax = plt.subplots(figsize=(20, 10))
ax.plot(t, a, label='overall')
ax.plot(t, b, label='top_ip')
ax.plot(t, c, label='auth')
ax.plot(t, d, label='zh')
ax.plot(t, e, label='mozillia')
ax.plot(t, f,...


I am working on the file "Assessment.ipynb".
The first task is:
Checking Subnets
Below, I will use regex to check if any of the IPs from the IP or X-Forwarded-For columns come from the same /24 networks.
Can you write Python code for it?

ip_nets = merged_test_df['IP'].str.extract(r'(\d{,3}.\d{,3}.\d{,3})')
ip_nets.columns = ['network']
ip_nets['network'].value_counts().head()

forward_nets = merged_test_df['X-Forwarded-For'].str.extract(r'(\d{,3}.\d{,3}.\d{,3})')
forward_nets.columns = ['network']
forward_nets['network'].value_counts().head()


I am working on the file "Assessment.ipynb".
The first task is:
In the chart above, we can see that there are several Content-Lengths which have unusually high rates of occurence, all grouped in a range between 150-160 bytes.
Can you write Python code for it?

five_ip_df = merged_test_df['X-Forwarded-For'].value_counts() == 5
five_ip_arr = []
for i in range(len(five_ip_df)):
    if five_ip_df.values[i]:
        five_ip_arr.append(five_ip_df.index[i])


I am working on the file "Assessment.ipynb".
The first task is:
Then import json and use json_normalize to convert the Headers data into columns of the csv. Merge this back with the original data.
Can you write Python code for it?

import json
headers_df = pd.io.json.json_normalize(test_df['Headers'].apply(json.loads))
merged_test_df = pd.merge(test_df, headers_df, left_index=True, right_index=True)


I am working on the file "Assessment.ipynb".
The first task is:
What are the columns about?
I'll begin by taking a look at summary statistics for the columns which stand out as most immediately relevant. As I discover insights from these, I will look back to these columns and others for more details and parse further to identify malicious traffic.
I will use merged_test_df['column'].value_counts().sample(5) to identify the top few values for each column and look for patterns and outliers.

IP - 225.19.49.85 stands out right away with 2231 instances. Interesting that all others besides this one have exactly 5 or fewer.
path - /auth is most common by far, followed by first image and product pages, in ascending order.
Accept-Language - Anything containing zh is potentially suspicious since the website is presumably in english, for an American company and zh indicates Chinese language.
Content-Length - not sure what to make of this at first, lots of values in a smallish range, 909 is a notable outlier
X-Forwarded-For - It is suspicious how many IPs have exactly 5
User-Agent - Mozillia is very suspicious, since it is not a real User-Agent and shows that someone is spoofing it. The top three User-Agents overall by a significant margin are:

Mozilla/5.0 (Linux i686) AppleWebKit/505.0 (KHTML, like Gecko) Chrome/1.0143.94 Safari/505 - 8902
Mozilla/5.0 (compatible; MSIE 5.0; Linux x86_64; Tablet PC;Trident/4.0) - 2262
Mozilla/5.0 Chrome/156.0 - 2201

Authorization - Basic + character string that looks like it is most likely a base64 string truncated to protect PII. Each of these instances indicates a login attempt.
Connection - about 1/3 close, 1/3 keep alive, and 1/3 NaN - this could be useful somehow later.

Below I start looking deeper into Content-Length. I picked the top 11 instances because there is a significant gap between number of occurences of the 11th and 12th.
Can you write Python code for it?

content_length_df = pd.DataFrame(merged_test_df['Content-Length'].value_counts())
top_content = content_length_df.head(11)


I am working on the file "Assignment 07.ipynb".
The first task is:
Approximation error is roughly at 800000,
Can you write Python code for it?

figure("Problem1.d")
i, err = problem01("d")
print("It took {} terms to reach 4 digit accuracy. The error value with 4 digits is {}.".format(i, err))


I am working on the file "Assignment 07.ipynb".
The first task is:
By analyzing the graph we can determine that: x1 = 9.6 and x2 = 6.2
Problem 03
Study graphically (do not use any algorithms) the following function
$$f(x) = 10.0e^{-x}sin(2.0\pi x)+ 0.5x$$
within the range $0 ≤ x ≤ 10$.
What is (a) the maximum, (b) the minimum, and (c) the largest root of the function within the given range. Give the answers with four significant figures.
Can you write Python code for it?


I am working on the file "Assignment 07.ipynb".
The first task is:
Maximum:
x = 0.2264
y = 7.9999
Minimum:
x = 0.7223
y = -4.4218
Largest root:
x = 1.8920
Problem 04
The following code implements the forward elimination part of theGaussian elimination algorithm. Complete the code with the backward substitution part, e.g. implement the following equations
$$x_i = \frac{(b_i - \sum_{j = i+1}^{n} a_{ij}x_j)}{a_{ii}}$$
and test and verify your algorithm by solving your choice of 3 x 3 linear
equation system.
Can you write Python code for it?

def GaussianElimination(a , b):
    a = a.copy().astype(float)
    b = b.copy().astype(float)
    n = len(a)
    for k in range(n):
        # Check
        if a[k, k] == 0 :
            print(' Unable to solve ')
            return None
        # Forward eliminate
        for i in range(k+1, n):
            factor = a[i, k]/a[k , k]
            for j in range(k+1, n):
                a[i , j] = a[i , j] - factor*a[k , j]
            b[i] = b[i] - factor*b[k]

    # Backward substitution
    # Do the calculations in place, e.g. x = b
    for i in range(n-1, -1, -1):
        for j in range(i+1, n):
            b[i] = b[i] - a[i, j]*b[j]
        b[i] = b[i]/a[i, i]
    return b

A = array(([5.0, -1.0, 82.0], [-41.8, 22.0, 9000.01], [1, 0.0, -1.0]))
b = array(([0.0, -800.98, 172.5]))
v1 = GaussianElimination(A, b)
v2 = dot(inv(A), b)

print("Results of function: {}\nResults of dot-product: {}".format(v1, v2))


I am working on the file "Assignment 07.ipynb".
The first task is:
Now that we have the values for $x_i$, we can use them to confirm the equations are correct.
Can you write Python code for it?


I am working on the file "Assignment 07.ipynb".
The first task is:
Problem 02
Use graphical method to solve
$$2x1 − 6x2 = −18$$
$$−x1 + 8x2 = 40$$
Can you write Python code for it?


I am working on the file "Assignment 07.ipynb".
The first task is:
Problem 07
The equilibrium equations of the blocks in the spring-block system are
$$ 3(x_2 − x_1) − 2x_1 = −80 $$
$$ 3(x_3 − x_2) − 3(x_2 − x_1) = 0 $$
$$ 3(x_4 − x_3) − 3(x_3 − x_2) = 0 $$
$$ 3(x_5 − x_4) − 3(x_4 − x_3) = 60 $$
$$ −2x_5 − 3(x_5 − x_4) = 0 $$
where $x_i$ are the horizontal displacements of the blocks measured in mm. Solve these equations using any functions of your choice found in scipy.linalg package. Show the steps and verify your solution.
Before we start to solve the problem we should convert the equations into matrix format.
$$\begin{bmatrix}
-5.0 & 3.0 & 0.0 & 0.0 & 0.0 \
3.0 & -6.0 & 3.0 & 0.0 & 0.0 \
0.0 & 3.0 & -6.0 & 3.0 & 0.0 \
0.0 & 0.0 & 3.0 & -6.0 & 3.0 \
0.0 & 0.0 & 0.0 & 3.0 & -5.0 \
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2 \
x_3 \
x_4 \
x_5 \
\end{bmatrix}
=
\begin{bmatrix}
-80 \
0 \
0 \
60 \
0 \
\end{bmatrix}$$
Can you write Python code for it?


I am working on the file "Assignment 07.ipynb".
The first task is:
Problem 1
The sine function can be approximated with following series:
$$sin(x) \approx x - \dfrac{x^3}{3!} + \dfrac{x^5}{5!} - \dfrac{x^7}{7!} + ...$$
(a) Draw in the same graph the sine function and its 7th order polynomial
approximation in range $x ∈ [−π, π]$.
(b) Where (x-value) in the given range does the the 7th order polynomial
approximation differ most? How much is the true error in
that point?
(c) What is the largest relative approximation error (in absolute values)
between the 5th order and 7th order polynomials within the
given range?
(d) How many terms we need in the approximation in order that we
have the value of $sin(x)$ correct at least 4 significant figures everywhere
in the given range?
Can you write Python code for it?

def plot_sin():
    x = linspace(-pi, pi, 1000)
    y = sin(x)
    plot(x, y)
    show()

def sinabout(x, N):

    y = x
    for n in range(1, N):
        plus = (-1)**n
        y = y + plus*x**(2*n+1)/fact(2*n+1)
    return y

def problem01(letter):

    x = linspace(-pi, pi, 1000)

    if letter == "a":

        # Plot sin graph
        plot_sin()

        # Plot graph with 4 terms
        y = sinabout(x, 4)
        plot(x, y)
        grid()

    if letter == "b":
        y1 = sin(x)
        y2 = sinabout(x, 4)

        # Plot the difference between y1 and y2
        plot(x, y1 - y2)
        show()
        grid()

        # Calculate true error
        error = max(abs(y1 - y2))
        print("True error is: {}".format(error))

    if letter == "c":
        y1 = sinabout(x, 3)
        y2 = sinabout(x, 4)

        plot(x, y1)
        plot(x, y2)

        grid()
        show()

        return y1, y2, x
    if letter == "d":

        # Define precision
        prec = 0.0001

        y1 = sin(x)
        for j in range(10):
            y2 = sinabout(x, j)
            err = max(abs(y1 - y2))
            if err < prec:
                      break

        # Plot graphs
        plot(x, y1)
        plot(x, y2)
        grid()
        show()

        # Return error and iteration count
        return j, err

    else:
        "Unable to parse problem!"

figure("Problem 1.a")
problem01("a")

figure("Problem 1.b")
problem01("b")


I am working on the file "Assignment 07.ipynb".
The first task is:
We can analyze the graph to to see that the approximation differs the most at x = pi or x = -pi. True error is either 0.07522 or -0.07522.
Can you write Python code for it?

figure("Problem 1.c")
y1, y2, x  = problem01("c")

figure("Problem 1.c - Approximation error")

# Calculate approximation error
err = (y2 - y1)/y2

# Plot graphs
plot(x, err*100)
ylabel('Relative appr error (%)')
xlabel('x-axis')

# Show graphs
grid()
show()


I am working on the file "Assignment 1. Pixel regression.ipynb".
The first task is:
Our objective is to train a deep MLP that is able to reconstruct the image:
Can you write Python code for it?

import keras
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout

# your model here
# hints: k*10^2 neurons per layer + good initialization + deep network (>2 layers)

# use this cell to find the best model architecture

model.fit(X_train, Y_train, nb_epoch=1, shuffle=True, verbose=1, batch_size=10)
Y = model.predict(X_train, batch_size=10000)
k = 0
im_out = im[:]
for i in range(im.shape[0]):
    for j in range(im.shape[1]):
        im_out[i,j]= Y[k]
        k += 1

plt.imshow(im_out)
plt.show()


I am working on the file "Assignment 1. Pixel regression.ipynb".
The first task is:
Ourt training dataset will be composed of pixels locatiions and input and pixel values as output:
Can you write Python code for it?

X_train = []
Y_train = []
for i in range(im.shape[0]):
    for j in range(im.shape[1]):
        X_train.append([float(i),float(j)])
        Y_train.append(im[i][j])

X_train = np.array(X_train)
Y_train = np.array(Y_train)
print 'Samples:', X_train.shape[0]
print '(x,y):', X_train[0],'\n', '(r,g,b):',Y_train[0]


I am working on the file "Assignment 1.ipynb".
The first task is:
Question 0 (Example)
How many features does the breast cancer dataset have?
This function should return an integer.
Can you write Python code for it?

# You should write your whole answer within the function provided. The autograder will call
# this function and compare the return value against the correct solution value
def answer_zero():
    # This function returns the number of features of the breast cancer dataset, which is an integer.
    # The assignment question description will tell you the general format the autograder is expecting
    return len(cancer['feature_names'])

# You can examine what your function returns by calling it in the cell. If you have questions
# about the assignment formats, check out the discussion forums for any FAQs
answer_zero()


I am working on the file "Assignment 1.ipynb".
The first task is:
Question 5
Using KNeighborsClassifier, fit a k-nearest neighbors (knn) classifier with X_train, y_train and using one nearest neighbor (n_neighbors = 1).
*This function should return a * sklearn.neighbors.classification.KNeighborsClassifier.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

def answer_five():
    X_train, X_test, y_train, y_test = answer_four()

    knn = KNeighborsClassifier(n_neighbors = 1)
    knn.fit(X_train, y_train)

    return knn


I am working on the file "Assignment 1.ipynb".
The first task is:
Question 6
Using your knn classifier, predict the class label using the mean value for each feature.
Hint: You can use cancerdf.mean()[:-1].values.reshape(1, -1) which gets the mean value for each feature, ignores the target column, and reshapes the data from 1 dimension to 2 (necessary for the precict method of KNeighborsClassifier).
This function should return a numpy array either array([ 0.]) or array([ 1.])
Can you write Python code for it?

def answer_six():
    cancerdf = answer_one()

    knn = answer_five()
    means = cancerdf.mean()[:-1].values.reshape(1, -1)

    return knn.predict(means)


I am working on the file "Assignment 1.ipynb".
The first task is:
Question 8
Find the score (mean accuracy) of your knn classifier using X_test and y_test.
This function should return a float between 0 and 1
Can you write Python code for it?

def answer_eight():
    X_train, X_test, y_train, y_test = answer_four()
    knn = answer_five()

    # Your code here

    return knn.score(X_test, y_test)


I am working on the file "Assignment 2-joost.ipynb".
The first task is:
A benchmark study (3 points (2+1))
A benchmark study is an experiment in which multiple algorithms are evaluated on multiple datasets. The end goal is to study whether one algorithm is generally better than the others. Meaningful benchmark studies can grow quite complex, here we do a simplified variant.

Download OpenML datasets 37, 470, 1120, 1464 and 1471. They are sufficiently large (e.g., at least 500 data points) so that the performance estimation is trustworthy. Select at least three classifiers that we discussed in class, e.g. kNN, Logistic Regression, Random Forests, Gradient Boosting, SVMs, Naive Bayes. Note that some of these algorithms take longer to train. Evaluate all classifiers (with default parameter settings) on all datasets, using a 10-fold CV and AUC. Show the results in a table and interpret them. Which is the best algorithm in this benchmark?
Note that these datasets have categorical features, different scales, missing values, and (likely) irrelevant features. You'll need to build pipelines to correctly build all models. Also remove any row identifiers (see, e.g., https://www.openml.org/d/1120)
Hint: You can either compare the performances directly, or (better) use a statistical significance test, e.g. a pairwise t-test or (better) Wilcoxon signed ranks test, to see whether the performance differences are significant. This is covered in statistics courses. You can then count wins, ties and losses.

Repeat the benchmark, but now additionally optimize the main hyperparameters of each algorithm in a grid or random search (explore at least 5 values per hyperparameter, where possible). Does this affect the ranking of the algorithms?

Gaussian Processes (2 points (1+1))
Consider the RAM prices dataset (included in the data folder). Separate the data in a training set of all data points up until the year 2000, and a test set with all points after that.

Train several of the algorithms we have covered in the course that can handle regression. Include at least linear regression, decision tree,...
Can you write Python code for it?

ram_prices = pd.read_csv('data/ram_price.csv')

plt.semilogy(ram_prices.date, ram_prices.price)
plt.xlabel("Year")
plt.ylabel("Price in $/Mbyte");


I am working on the file "Assignment 2-joost.ipynb".
The first task is:
Results
As we can see, the linear kernel and the polynomial kernel fits the data very well, whereas the RBF kernel performs the slightly worse. We can see in the figure that the ideal classifier here would be a linear classifier, as the data is somewhat linearly seperable. A polynomial kernel can act like a linear kernel with high level polynomials, so that is why it performs similar to the linear kernel.
But the RBF kernel wants to look for the distance with respect to other points, which does not work so well in this case. It cannot find the optimal linear classifier, but instead tries to look for points near other points.
Can you write Python code for it?

# First performing a 10CV grid search to obtain the ROC_AUC values.
from sklearn.model_selection import GridSearchCV

param_grid = {
    "C" : [2e-15, 2, 2e15],
    "gamma" : [2e-15, 2, 2e15]
}

svm_clf = SVC(kernel="rbf")

grid_search = GridSearchCV(svm_clf, param_grid, n_jobs=-1, cv=3, scoring="roc_auc")
_ = grid_search.fit(X, y)

results = pd.DataFrame(grid_search.cv_results_)
scores = np.array(results.mean_test_score)

# Using a slightly adapted version of the plot_svm_kernels function from mglearn.
def plot_svm_rbf_kernel(X, y, clf, C, gamma):
    # figure number
    fignum = 1

    # plot the line, the points, and the nearest vectors to the plane
    plt.figure(fignum, figsize=(4, 3))
    plt.suptitle('C = ' + str(C) + ', gamma = ' + str(gamma))

    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=85, edgecolors='k', c='w', zorder=10)
    plt.scatter(X[:, 0], X[:, 1], c=y, zorder=10, cmap=plt.cm.bwr)

#         for i, coef in enumerate(clf.dual_coef_[0]):
#             plt.annotate("%0.2f" % (coef), (clf.support_vectors_[i, 0]+0.15,clf.support_vectors_[i, 1]), fontsize=8, zorder=11)

    plt.axis('tight')
    x_min = np.min(X, axis=0)[0] - 1
    x_max = np.max(X, axis=0)[0] + 1
    y_min = np.min(X, axis=0)[1] - 1
    y_max = np.max(X, axis=0)[1] + 1

    XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]
    Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(XX.shape)
    plt.figure(fignum, figsize=(4, 3))
    #plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.bwr, alpha=0.1)
    plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'],
                levels=[-.5, 0, .5])

    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)

    plt.xticks(())
    plt.yticks(())

    plt.show()

# Question: moest je het handmatig scoren op 75-25 of moet je grid_search gebruiken?
# Zo ja, als je grid_search gebruikt, hoe krijg je dan alle estimators terug voor de support vectors?
idx = 0
for C...


I am working on the file "Assignment 2-joost.ipynb".
The first task is:
Robots and SVMs (4 points (2+1+1))
The Wall Robot Navigation dataset contains about 5500 readings of an ultrasound sensor array mounted on a robot, and your task is to finetune and train an SVM classifier to predict how the robot should move next.

Make a stratified 80-20 split of the data. On the training set alone, optimize the main hyperparameters of the SVM for Accuracy with a random search. Vary at least the main kernel types (linear, polynomial, and RBF), the C parameter, the $\gamma$ parameter for the RBF kernel and the exponent/degree for the polynomial kernel. Report the optimal hyperparameter settings and Accuracy performance.
The degree of the polynonial is typically in the range 2..10.
Hint: note that the hyperparameter ranges depend on each other. For instance, $\gamma$ only makes sense if you have selected the RBF kernel as well. We've seen in class how to define multiple hyperparameter spaces in a random/grid search.

Use a 5x3-fold (5 outer, 3 inner) nested cross-validation (CV) on the whole dataset to obtain a clean evaluation. What is the mean optimized performance? Is this in line with the optimized result of the random search of the previous question?
Train an SVM using the optimal hyperparameter configuration you found (in part 1 of this question) and test it on the held out (20%) test set. Compare this Accuracy result with the (mean) result of the nested CV. If you would build this robot in practice, how would you find the hyperparameters to use, and which performance would you expect? Is it truly necessary to tune the hyperparameters? Which hyperparameters were most important to tune?
Can you write Python code for it?

robot_data = oml.datasets.get_dataset(1497) # Download Robot data
# Get the predictors X and the labels y
X, y = robot_data.get_data(target=robot_data.default_target_attribute);

from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

# We'll use stratify=None for the built-in stratify of train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

param_grid = [
    {
        'kernel':['linear'],
        'C' : [2*10**(i) for i in range(-12, 13, 1)],
    },
    {
        'kernel':['poly'],
        'degree': [i for i in range(2, 11)],
        'gamma' : [2*10**(i) for i in range(-12, 13, 1)],
        'C' : [2*10**(i) for i in range(-12, 13, 1)],
    },
    {
        'kernel':['rbf', 'sigmoid'],
        'C' : [2*10**(i) for i in range(-12, 13, 1)],
        'gamma' : [2*10**(i) for i in range(-12, 13, 1)]
    }
]

random_search = RandomizedSearchCV(SVC(), param_distributions=param_grid, n_iter=30, n_jobs=-1, cv=3)
random_search.fit(X_train, y_train)
print("Best Score (3CV accuracy): " + str(best_score_))
print("Best parameters: ")
print(best_params_)

param_grid


I am working on the file "Assignment 2.ipynb".
The first task is:
A benchmark study (3 points (2+1))
A benchmark study is an experiment in which multiple algorithms are evaluated on multiple datasets. The end goal is to study whether one algorithm is generally better than the others. Meaningful benchmark studies can grow quite complex, here we do a simplified variant.

Download OpenML datasets 37, 470, 1120, 1464 and 1471. They are sufficiently large (e.g., at least 500 data points) so that the performance estimation is trustworthy. Select at least three classifiers that we discussed in class, e.g. kNN, Logistic Regression, Random Forests, Gradient Boosting, SVMs, Naive Bayes. Note that some of these algorithms take longer to train. Evaluate all classifiers (with default parameter settings) on all datasets, using a 10-fold CV and AUC. Show the results in a table and interpret them. Which is the best algorithm in this benchmark?
Note that these datasets have categorical features, different scales, missing values, and (likely) irrelevant features. You'll need to build pipelines to correctly build all models. Also remove any row identifiers (see, e.g., https://www.openml.org/d/1120)
Hint: You can either compare the performances directly, or (better) use a statistical significance test, e.g. a pairwise t-test or (better) Wilcoxon signed ranks test, to see whether the performance differences are significant. This is covered in statistics courses. You can then count wins, ties and losses.

Repeat the benchmark, but now additionally optimize the main hyperparameters of each algorithm in a grid or random search (explore at least 5 values per hyperparameter, where possible). Does this affect the ranking of the algorithms?

Gaussian Processes (2 points (1+1))
Consider the RAM prices dataset (included in the data folder). Separate the data in a training set of all data points up until the year 2000, and a test set with all points after that.

Train several of the algorithms we have covered in the course that can handle regression. Include at least linear regression, decision tree,...
Can you write Python code for it?


I am working on the file "Assignment 2.ipynb".
The first task is:
Kernel selection (4 points (1+2+1))
SVMs can be trained with different kernels. Generate a 2-dimensional dataset as shown below and study the effect of the choice of kernel by visualizing the results.

Train a SVM classifier on the dataset using respectively a linear, polynomial and radial basis function (RBF) kernel, evaluate the performance of each kernel using 10-fold cross-validation and AUC. Which one works best? Visualize the results. Can you intuitively explain why one kernel is more suited than another?
Hint: you can use the visualization code used in class. It is under mglearn/plot_svm.py > plot_svm_kernels().

Take the RBF kernel and vary both the C parameter and the kernel width ($\gamma$). Use 3 values for each (a very small, default, and very large value). For each of the 9 combinations, create the same RBF plot as before, report the number of support vectors, and the AUC performance. Explain the performance results. When are you over/underfitting?
Hint: values for C and $\gamma$ are typically in [$2^{-15}..2^{15}$] on a log scale.
Hint: don't count the support vectors manually, retrieve them from the trained SVM.

Vary C and $\gamma$ again, but this time use a grid of at least 20x20, vary both parameters uniformly on a log scale, and visualise the results using a $C \times \gamma \rightarrow AUC$ heatmap. Explain the performance results, and compare them to the 9 results obtained in the previous subquestion. Can you also tell in which regions of the heatmap you are over/underfitting?
Hint: We've constructed such a heatmap in class and in assignment 1.
Can you write Python code for it?

X, y = make_blobs(centers=2, n_samples=1000, random_state=0)


I am working on the file "Assignment 2.ipynb".
The first task is:
Part 2 - Classification
Here's an application of machine learning that could save your life! For this section of the assignment we will be working with the UCI Mushroom Data Set stored in mushrooms.csv. The data will be used to train a model to predict whether or not a mushroom is poisonous. The following attributes are provided:
Attribute Information:

cap-shape: bell=b, conical=c, convex=x, flat=f, knobbed=k, sunken=s
cap-surface: fibrous=f, grooves=g, scaly=y, smooth=s
cap-color: brown=n, buff=b, cinnamon=c, gray=g, green=r, pink=p, purple=u, red=e, white=w, yellow=y
bruises?: bruises=t, no=f
odor: almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, none=n, pungent=p, spicy=s
gill-attachment: attached=a, descending=d, free=f, notched=n
gill-spacing: close=c, crowded=w, distant=d
gill-size: broad=b, narrow=n
gill-color: black=k, brown=n, buff=b, chocolate=h, gray=g, green=r, orange=o, pink=p, purple=u, red=e, white=w, yellow=y
stalk-shape: enlarging=e, tapering=t
stalk-root: bulbous=b, club=c, cup=u, equal=e, rhizomorphs=z, rooted=r, missing=?
stalk-surface-above-ring: fibrous=f, scaly=y, silky=k, smooth=s
stalk-surface-below-ring: fibrous=f, scaly=y, silky=k, smooth=s
stalk-color-above-ring: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y
stalk-color-below-ring: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y
veil-type: partial=p, universal=u
veil-color: brown=n, orange=o, white=w, yellow=y
ring-number: none=n, one=o, two=t
ring-type: cobwebby=c, evanescent=e, flaring=f, large=l, none=n, pendant=p, sheathing=s, zone=z
spore-print-color: black=k, brown=n, buff=b, chocolate=h, green=r, orange=o, purple=u, white=w, yellow=y
population: abundant=a, clustered=c, numerous=n, scattered=s, several=v, solitary=y
habitat: grasses=g, leaves=l, meadows=m, paths=p, urban=u, waste=w, woods=d

<br>
The data in the mushrooms dataset is currently encoded with strings. These values will need to be encoded to numeric to work with...
Can you write Python code for it?

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

mush_df = pd.read_csv('mushrooms.csv')
mush_df2 = pd.get_dummies(mush_df)

X_mush = mush_df2.iloc[:,2:]
y_mush = mush_df2.iloc[:,1]

# use the variables X_train2, y_train2 for Question 5
X_train2, X_test2, y_train2, y_test2 = train_test_split(X_mush, y_mush, random_state=0)

# For performance reasons in Questions 6 and 7, we will create a smaller version of the
# entire mushroom dataset for use in those questions.  For simplicity we'll just re-use
# the 25% test split created above as the representative subset.
#
# Use the variables X_subset, y_subset for Questions 6 and 7.
X_subset = X_test2
y_subset = y_test2

# mush_df2.head()


I am working on the file "Assignment 2.ipynb".
The first task is:
Question 1
Write a function that fits a polynomial LinearRegression model on the training data X_train for degrees 1, 3, 6, and 9. (Use PolynomialFeatures in sklearn.preprocessing to create the polynomial features and then fit a linear regression model) For each model, find 100 predicted values over the interval x = 0 to 10 (e.g. np.linspace(0,10,100)) and store this in a numpy array. The first row of this array should correspond to the output from the model trained on degree 1, the second row degree 3, the third row degree 6, and the fourth row degree 9.

The figure above shows the fitted models plotted on top of the original data (using plot_one()).

This function should return a numpy array with shape (4, 100)
Can you write Python code for it?

def answer_one():
    from sklearn.linear_model import LinearRegression
    from sklearn.preprocessing import PolynomialFeatures

    X_train_reshaped = X_train.reshape(-1, 1)
    # Your code here
    poly_1 = PolynomialFeatures(degree=1)
    X_1_poly = poly_1.fit_transform(X_train_reshaped)

    poly_3 = PolynomialFeatures(degree=3)
    X_3_poly = poly_3.fit_transform(X_train_reshaped)

    poly_6 = PolynomialFeatures(degree=6)
    X_6_poly = poly_6.fit_transform(X_train_reshaped)

    poly_9 = PolynomialFeatures(degree=9)
    X_9_poly = poly_9.fit_transform(X_train_reshaped)

    # fitting
    linreg_1 = LinearRegression().fit(X_1_poly, y_train.reshape(-1, 1))
    linreg_3 = LinearRegression().fit(X_3_poly, y_train.reshape(-1, 1))
    linreg_6 = LinearRegression().fit(X_6_poly, y_train.reshape(-1, 1))
    linreg_9 = LinearRegression().fit(X_9_poly, y_train.reshape(-1, 1))

    #
    t = np.linspace(0,10,100)
    t = t.reshape(-1, 1)
    t_1 = poly_1.fit_transform(t)
    t_3 = poly_3.fit_transform(t)
    t_6 = poly_6.fit_transform(t)
    t_9 = poly_9.fit_transform(t)

    p_1 = linreg_1.predict(t_1)
    p_3 = linreg_3.predict(t_3)
    p_6 = linreg_6.predict(t_6)
    p_9 = linreg_9.predict(t_9)

    return np.array([p_1, p_3, p_6, p_9]).reshape(4, -1)

# x = answer_one()

# x.reshape(4, -1).shape

# feel free to use the function plot_one() to replicate the figure
# from the prompt once you have completed question one
def plot_one(degree_predictions):
    plt.figure(figsize=(10,5))
    plt.plot(X_train, y_train, 'o', label='training data', markersize=10)
    plt.plot(X_test, y_test, 'o', label='test data', markersize=10)
    for i,degree in enumerate([1,3,6,9]):
        plt.plot(np.linspace(0,10,100), degree_predictions[i], alpha=0.8, lw=2, label='degree={}'.format(degree))
    plt.ylim(-1,2.5)
    plt.legend(loc=4)

# plot_one(answer_one())


I am working on the file "Assignment 2.ipynb".
The first task is:
Question 2
Write a function that fits a polynomial LinearRegression model on the training data X_train for degrees 0 through 9. For each model compute the $R^2$ (coefficient of determination) regression score on the training data as well as the the test data, and return both of these arrays in a tuple.
This function should return one tuple of numpy arrays (r2_train, r2_test). Both arrays should have shape (10,)
Can you write Python code for it?

def answer_two():
    from sklearn.linear_model import LinearRegression
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.metrics.regression import r2_score

    # Your code here
    r2_train = []
    r2_test = []
    for i in range(0, 10):

        poly = PolynomialFeatures(degree=i)
        X_poly = poly.fit_transform(X_train.reshape(-1, 1))
        linreg = LinearRegression().fit(X_poly, y_train.reshape(-1, 1))

        s_train = linreg.score(X_poly, y_train.reshape(-1, 1))

        X_poly_test = poly.fit_transform(X_test.reshape(-1, 1))
        s_test = linreg.score(X_poly_test, y_test.reshape(-1, 1))

        r2_train.append(s_train)
        r2_test.append(s_test)

    return (np.array(r2_train), np.array(r2_test))

# answer_two()


I am working on the file "Assignment 2.ipynb".
The first task is:
Question 5
Using X_train2 and y_train2 from the preceeding cell, train a DecisionTreeClassifier with default parameters and random_state=0. What are the 5 most important features found by the decision tree?
As a reminder, the feature names are available in the X_train2.columns property, and the order of the features in X_train2.columns matches the order of the feature importance values in the classifier's feature_importances_ property.
This function should return a list of length 5 containing the feature names in descending order of importance.
Can you write Python code for it?

def answer_five():
    from sklearn.tree import DecisionTreeClassifier

    clf = DecisionTreeClassifier(random_state=0).fit(X_train2, y_train2)

    max_index_5 = clf.feature_importances_.argsort()[-5:][::-1]

    return list(X_train2.columns[max_index_5])

# ind = answer_five()

# list(ind)


I am working on the file "Assignment 2.ipynb".
The first task is:
Question 6
For this question, we're going to use the validation_curve function in sklearn.model_selection to determine training and test scores for a Support Vector Classifier (SVC) with varying parameter values.  Recall that the validation_curve function, in addition to taking an initialized unfitted classifier object, takes a dataset as input and does its own internal train-test splits to compute results.
Because creating a validation curve requires fitting multiple models, for performance reasons this question will use just a subset of the original mushroom dataset: please use the variables X_subset and y_subset as input to the validation curve function (instead of X_mush and y_mush) to reduce computation time.
The initialized unfitted classifier object we'll be using is a Support Vector Classifier with radial basis kernel.  So your first step is to create an SVC object with default parameters (i.e. kernel='rbf', C=1) and random_state=0. Recall that the kernel width of the RBF kernel is controlled using the gamma parameter.
With this classifier, and the dataset in X_subset, y_subset, explore the effect of gamma on classifier accuracy by using the validation_curve function to find the training and test scores for 6 values of gamma from 0.0001 to 10 (i.e. np.logspace(-4,1,6)). Recall that you can specify what scoring metric you want validation_curve to use by setting the "scoring" parameter.  In this case, we want to use "accuracy" as the scoring metric.
For each level of gamma, validation_curve will fit 3 models on different subsets of the data, returning two 6x3 (6 levels of gamma x 3 fits per level) arrays of the scores for the training and test sets.
Find the mean score across the three models for each level of gamma for both arrays, creating two arrays of length 6, and return a tuple with the two arrays.
e.g.
if one of your array of scores is
array([[ 0.5,  0.4,  0.6],
       [ 0.7,  0.8,  0.7],
       [ 0.9,  0.8,  0.8],
       [ 0.8,  0.7,  0.8],
       [ 0.7,  0.6,  0.6],
       [ 0.4,  0.6, ...
Can you write Python code for it?

def answer_six():
    from sklearn.svm import SVC
    from sklearn.model_selection import validation_curve

    clf = SVC(kernel='rbf', C=1, random_state=0)

    param_range = np.logspace(-4, 1, 6)

    train_scores, test_scores = validation_curve(clf, X_subset, y_subset,
                                            param_name='gamma',
                                            param_range=param_range, cv=3, scoring='accuracy')
    train_s = [x.mean() for x in train_scores]
    test_s = [x.mean() for x in test_scores]
    return (np.array(train_s), np.array(test_s))

# answer_six()


I am working on the file "Assignment 2.ipynb".
The first task is:
Question 7
Based on the scores from question 6, what gamma value corresponds to a model that is underfitting (and has the worst test set accuracy)? What gamma value corresponds to a model that is overfitting (and has the worst test set accuracy)? What choice of gamma would be the best choice for a model with good generalization performance on this dataset (high accuracy on both training and test set)? Note: there may be multiple correct solutions to this question.
(Hint: Try plotting the scores from question 6 to visualize the relationship between gamma and accuracy.)
This function should return one tuple with the degree values in this order: (Underfitting, Overfitting, Good_Generalization)
Can you write Python code for it?


I am working on the file "Assignment 3.ipynb".
The first task is:
Question 1
Import the data from fraud_data.csv. What percentage of the observations in the dataset are instances of fraud?
This function should return a float between 0 and 1.
Can you write Python code for it?

def answer_one():

    # Your code here
    df = pd.read_csv('fraud_data.csv')

    return df['Class'].sum()/len(df['Class'])

# answer_one()

# Use X_train, X_test, y_train, y_test for all of the following questions
from sklearn.model_selection import train_test_split

df = pd.read_csv('fraud_data.csv')

X = df.iloc[:,:-1]
y = df.iloc[:,-1]

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)


I am working on the file "Assignment 3.ipynb".
The first task is:
Question 3
Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?
This function should a return a tuple with three floats, i.e. (accuracy score, recall score, precision score).
Can you write Python code for it?

def answer_three():
    from sklearn.metrics import recall_score, precision_score, accuracy_score
    from sklearn.svm import SVC

    scv = SVC()

    scv.fit(X_train, y_train)

    y_p = scv.predict(X_test)

    return (scv.score(X_test, y_test), recall_score(y_test, y_p), precision_score(y_test, y_p))

# answer_three()


I am working on the file "Assignment 3.ipynb".
The first task is:
Question 4
Using the SVC classifier with parameters {'C': 1e9, 'gamma': 1e-07}, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.
This function should return a confusion matrix, a 2x2 numpy array with 4 integers.
Can you write Python code for it?

def answer_four():
    from sklearn.metrics import confusion_matrix
    from sklearn.svm import SVC

    # Your code here
    scv = SVC(C=1e9, gamma=1e-07)

    scv.fit(X_train, y_train)

    scores = scv.decision_function(X_test)

    y_pred_with_threshold = scores > -220
    confusion = confusion_matrix(y_test, y_pred_with_threshold)

    return confusion

# answer_four()


I am working on the file "Assignment 3.ipynb".
The first task is:
Question 5
Train a logisitic regression classifier with default parameters using X_train and y_train.
For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).
Looking at the precision recall curve, what is the recall when the precision is 0.75?
Looking at the roc curve, what is the true positive rate when the false positive rate is 0.16?
This function should return a tuple with two floats, i.e. (recall, true positive rate).
Can you write Python code for it?

def draw_pr_curve():
    from sklearn.linear_model import LogisticRegression
    from sklearn.metrics import precision_recall_curve
    from sklearn.metrics import roc_curve, auc

    lr = LogisticRegression()
    lr.fit(X_train, y_train)
    y_scores_lr = lr.decision_function(X_test)

    precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)
    closest_zero = np.argmin(np.abs(thresholds))
    closest_zero_p = precision[closest_zero]
    closest_zero_r = recall[closest_zero]

    import matplotlib.pyplot as plt
    plt.figure()
    plt.xlim([0.0, 1.01])
    plt.ylim([0.0, 1.01])
    plt.plot(precision, recall, label='Precision-Recall Curve')
    plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3)
    plt.xlabel('Precision', fontsize=16)
    plt.ylabel('Recall', fontsize=16)
    plt.axes().set_aspect('equal')
    plt.show()

# draw_pr_curve()

def draw_roc_curve():
    %matplotlib notebook
    import matplotlib.pyplot as plt
    from sklearn.metrics import roc_curve, auc

    fpr_lr, tpr_lr, _ = roc_curve(y_test, y_scores_lr)
    roc_auc_lr = auc(fpr_lr, tpr_lr)

    plt.figure()
    plt.xlim([-0.01, 1.00])
    plt.ylim([-0.01, 1.01])
    plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)
    plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
    plt.legend(loc='lower right', fontsize=13)
    plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
    plt.axes().set_aspect('equal')
    plt.show()

# draw_roc_curve()

def answer_five():

    # Your code here

    return (0.8, 0.9)


I am working on the file "Assignment 4.ipynb".
The first task is:
Train a NeuralNet and see the performance
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

clf = MLPClassifier(hidden_layer_sizes = [50], alpha = 5,
                   random_state = 0,
                   solver='lbfgs')
clf.fit(X_train_scaled, y_train)
print(clf.loss_)

clf.score(X_train_scaled, y_train)

clf.score(X_test_scaled, y_test)

from sklearn.metrics import recall_score, precision_score, f1_score

train_pred = clf.predict(X_train_scaled)
print(precision_score(y_train, train_pred),
      recall_score(y_train, train_pred),
      f1_score(y_train, train_pred))

from sklearn.metrics import recall_score, precision_score, f1_score

test_pred = clf.predict(X_test_scaled)
print(precision_score(y_test, test_pred),
      recall_score(y_test, test_pred),
      f1_score(y_test, test_pred))

test_pro = clf.predict_proba(X_test_scaled)

def draw_roc_curve():
    %matplotlib notebook
    import matplotlib.pyplot as plt
    from sklearn.metrics import roc_curve, auc

    fpr_lr, tpr_lr, _ = roc_curve(y_test, test_pro[:,1])
    roc_auc_lr = auc(fpr_lr, tpr_lr)

    plt.figure()
    plt.xlim([-0.01, 1.00])
    plt.ylim([-0.01, 1.01])
    plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)
    plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
    plt.legend(loc='lower right', fontsize=13)
    plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
    plt.axes().set_aspect('equal')
    plt.show()

draw_roc_curve()

test_pro[0:10]

clf.predict(X_test_scaled[0:10])

y_test[0:10]

1 - y_train.sum()/len(y_train)

from sklearn.metrics import recall_score, precision_score, f1_score

test_pred = clf.predict(X_test_scaled)
print(precision_score(y_test, test_pred),
      recall_score(y_test, test_pred),
      f1_score(y_test, test_pred))

def...


I am working on the file "Assignment 4.ipynb".
The first task is:
Train, keep, test split
Can you write Python code for it?

from sklearn.model_selection import train_test_split
train_features = df_train.columns.drop('compliance')
train_features

X_data, X_keep, y_data, y_keep = train_test_split(df_train[train_features],
                                                    df_train.compliance,
                                                    random_state=0,
                                                    test_size=0.05)

print(X_data.shape, X_keep.shape)

X_train, X_test, y_train, y_test = train_test_split(X_data[train_features],
                                                    y_data,
                                                    random_state=0,
                                                    test_size=0.2)

print(X_train.shape, X_test.shape)


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Create Train and Test Data  [from categories-medical and automobiles]
Can you write Python code for it?

cats = ['sci.med' , 'rec.motorcycles']
newsgroups_train = fetch_20newsgroups(subset='train', categories=cats, remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test', categories=cats, remove=('headers', 'footers', 'quotes'))

print("Categories to classify\n-----------------------")
print(list(newsgroups_train.target_names))

print("TRAIN DATA\n---------------")
print("Data Type:", type(newsgroups_train))
print("%d documents" % len(newsgroups_train.filenames))
print("%d categories" % len(newsgroups_train.target_names))
print("X shape :", newsgroups_train.filenames.shape)
print("Y shape :",newsgroups_train.target.shape)
print("Y head  :", newsgroups_train.target[:10])

print("TEST DATA\n---------------")
print("Data Type:", type(newsgroups_test))
print("%d documents" % len(newsgroups_test.filenames))
print("%d categories" % len(newsgroups_test.target_names))
print("X shape :", newsgroups_test.filenames.shape)
print("Y shape :",newsgroups_test.target.shape)
print("Y head  :", newsgroups_test.target[:10])


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Evaluate Classifier
Can you write Python code for it?

y_true = newsgroups_test.target
y_pred = clf.predict(vectors_test)

metrics.f1_score(y_true, y_pred, average='macro')


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Explore the data
Can you write Python code for it?

print(newsgroups_train.data[0])

print(newsgroups_test.data[0])

print(type(newsgroups_test.data))
print(type(newsgroups_test.data[0]))


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Interpretation:

From 398 Automobile related articles, we classifier 385 Correctly.
From 396 Medicine related articles, we classified, 363 Correctly.

Plot: Precision-Recall Curve
Can you write Python code for it?

def plot_precision_recall(y_true,y_score):
    """
    Plot a precision recall curve

    Parameters
    ----------
    y_true: ls
        ground truth labels
    y_score: ls
        score output from model
    """
    precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_true,y_score[:,1])
    plt.plot(recall_curve, precision_curve)
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    auc_val = auc(recall_curve,precision_curve)
    print('AUC-PR: {0:1f}'.format(auc_val))
    plt.show()
    plt.clf()

y_score = clf.predict_proba(vectors_test)
plot_precision_recall(y_true, y_score)


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Interpretation:  An F-1 score of 0.94 is high. Our model is performant.
Plot Confusion Matrix
Can you write Python code for it?


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Load Dataset
Can you write Python code for it?

all_newsgroups= fetch_20newsgroups()

pprint(list(all_newsgroups.target_names))


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Pre-process Data
Can you write Python code for it?

train_labels = newsgroups_train.target #0, 1 array
#print(train_labels)

test_labels = newsgroups_test.target
#print(test_labels)

RE_PREPROCESS = r'\W+|\d+' #the regular expressions that matches all non-characters
#train_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_train.jobDescription.values])
#test_corpus = np.array( [re.sub(RE_PREPROCESS, ' ', text).lower() for text in df_test.jobDescription.values])

labels = np.append(train_labels, test_labels)


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Pretty Print Confusion Matrix
Can you write Python code for it?

def print_cm(cm, labels, hide_zeroes=False, hide_diagonal=False, hide_threshold=None):
    """pretty print for confusion matrixes"""
    columnwidth = max([len(x) for x in labels] + [5])  # 5 is value length
    empty_cell = " " * columnwidth
    # Print header
    print("    " + empty_cell, end=" ")
    for label in labels:
        print("%{0}s".format(columnwidth) % label, end=" ")
    print()

    # Print rows
    for i, label1 in enumerate(labels):
        print("    %{0}s".format(columnwidth) % label1, end=" ")
        for j in range(len(labels)):
            cell = "%{0}.1f".format(columnwidth) % cm[i, j]
            if hide_zeroes:
                cell = cell if float(cm[i, j]) != 0 else empty_cell
            if hide_diagonal:
                cell = cell if i != j else empty_cell
            if hide_threshold:
                cell = cell if cm[i, j] > hide_threshold else empty_cell
            print(cell, end=" ")
        print()

print_cm(cm, labels = ['Automobiles', 'Medical'])

pd.crosstab(y_true, y_pred, rownames=['True'], colnames=['Predicted'], margins=True)


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
There are 18000+ features for each document. And on average, 87 out of 18000 features are non-zeros. This is a sparse matrix
Train Classifier
Can you write Python code for it?

clf = MultinomialNB(alpha=.01)
clf.fit(vectors_train, newsgroups_train.target)


I am working on the file "Assignment 7 - Part A.ipynb".
The first task is:
Transform Data (Vectorize)
Can you write Python code for it?

vectorizer = TfidfVectorizer()
vectors_train = vectorizer.fit_transform(newsgroups_train.data)
vectors_train.shape

vectors_train.nnz / float(vectors_train.shape[0])

vectors_test = vectorizer.transform(newsgroups_test.data)


I am working on the file "Assignment+3_Hui+Lyu.ipynb".
The first task is:
Below, construct a dataset class.  You will see stub functions.  You may reuse what we have worked on in class, or you may develop something new.
You must implement the routines convert, columns, filter_eq, filter_lt, filter_gt, filter_ne, size and split.
Can you write Python code for it?

class Dataset:
    def __init__(self, data):
        self.data = data.copy()

    def convert(self, column, dtype):
        # YOUR CODE HERE
        self.data[column] = np.array(self.data[column], dtype=dtype)
        #raise NotImplementedError()

    def columns(self):
        # YOUR CODE HERE
        return self.data.keys()
        #raise NotImplementedError()

    def filter_eq(self, column, value):
        # YOUR CODE HERE
        good = (self.data[column] == value)
        new_data = {}
        for column in self.data:
            new_data[column] = self.data[column][good]
        return Dataset(new_data)
        #raise NotImplementedError()

    def filter_lt(self, column, value):
        # YOUR CODE HERE
        good = (self.data[column] < value)
        new_data = {}
        for column in self.data:
            new_data[column] = self.data[column][good]
        return Dataset(new_data)
        #raise NotImplementedError()

    def filter_gt(self, column, value):
        # YOUR CODE HERE
        good = (self.data[column] > value)
        new_data = {}
        for column in self.data:
            new_data[column] = self.data[column][good]
        return Dataset(new_data)
        #raise NotImplementedError()

    def filter_ne(self, column, value):
        # YOUR CODE HERE
        good = (self.data[column] != value)
        new_data = {}
        for column in self.data:
            new_data[column] = self.data[column][good]
        return Dataset(new_data)
        #raise NotImplementedError()

    def size(self):
        # YOUR CODE HERE
        for key in self.data:
            return self.data[key].size
        #raise NotImplementedError()

    def split(self, column):
        # YOUR CODE HERE
        new_datasets = {}
        for split_value in np.unique(self.data[column]):
            new_datasets[split_value] = self.filter_eq(column, split_value)
        return new_datasets
        #raise NotImplementedError()

    def stats(self):
        statistics = {}
       ...


I am working on the file "Assignment+3_Hui+Lyu.ipynb".
The first task is:
The column Location is not in a useful form.  You may use the code below to convert it to Latitude and Longitude.
The original form is in a string such as:
'(40.15271592360, -88.25708727630)'
We will use "split" on these and strip off the leading and trailing parenthesis.  Because this is material that we have not covered in class before, you may use this code directly.
Can you write Python code for it?

data["Latitude"] = []
data["Longitude"] = []
for loc in data["Location"]:
    lat, lon = loc.split(",")
    data["Latitude"].append(float(lat[1:]))
    data["Longitude"].append(float(lon[:-1]))


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
(Note that the first ratio reported is meaningless.)
Convergence might be easier to see in a plot.  If a method is p'th order accurate then we expect the error to behave like $E\approx Ch^p$ for some constant $C$, for small $h$.  This is hard to visualize.  It is much easier to see what order accuracy we are achieving if we produce a log-log plot instead, since $E = Ch^p~$ means that $\log E = \log C + p\log h$
In other words $\log E~$ is a linear function of $\log h~$.
Can you write Python code for it?

def error_plot(f,a,b,nvals,int_true):
    errvals = np.zeros(nvals.shape)  # initialize to right shape
    for i in range(len(nvals)):
        n = nvals[i]
        int_trap = trapezoid(f,a,b,n)
        error = abs(int_trap - int_true)
        errvals[i] = error
    hvals = (b - a) / (nvals - 1)  # vector of h values for each n
    mpl.loglog(hvals,errvals, 'o-')
    mpl.xlabel('spacing h')
    mpl.ylabel('error')
    mpl.show()

error_plot(f1,a1,b1,nvals,int_true1)


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
An oscillatory function
If the function $f(x)$ is not as smooth (has larger second derivative at various places) then the accuracy with a small number of points will not be nearly as good.  For example, consider this function:
Can you write Python code for it?


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
For the previous choice of nvals the method does not seem to be doing well:
Can you write Python code for it?

nvals = np.array([5, 10, 20, 40, 80, 160, 320])
print "nvals = ",nvals
error_plot(f2,a2,b2,nvals,int_true2)


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
Making k larger will make it more oscillatory.  For this function with k=50, using n=10 points is not going to give a very good approximation:
Can you write Python code for it?


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
Now make the function much more oscillatory with a larger value of $k$...
Can you write Python code for it?


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
This doesn't look very good, but for larger values of $n$, we still see the expected convergence rate:
Can you write Python code for it?


I am working on the file "Assignment-03/question-1/Numerical-Quadrature-with-the-Trapezoid-Rule.ipynb".
The first task is:
We can test it out for the points used in the figure above:
Can you write Python code for it?

n = 5
int_trap = trapezoid(f1,a1,b1,n)
error = abs(int_trap - int_true1)
print "trapezoid rule approximation: %22.14e,  error: %10.3e" % (int_trap, error)


I am working on the file "Assignment-03/question-2/Quadrature-in-SciPy.ipynb".
The first task is:
As another alternative, the routine scipy.integrate.trapz accepts numerical data in arrays as inputs and computes a trapezoidal approximation of a different integral. An analytical integrand is chosen and defined below as a lambda function for comparison. The numerical data is obtained by randomly drawing numbers from a Gaussian distribution and using those values (scaled and translated) as quadrature points from which to build up a trapezoidal approximation of an integral.
Can you write Python code for it?

(a,b) = (0,5)  # Observe use of tuple for multiple assignment statements
N = 5
xdat = np.unique(np.sort(np.random.rand(N) * (b-a) + a)) # random ordinates uniformly sampled from a to b
fun = lambda x: np.sin(x) * np.cos(x**2) + 1
ydat = fun(xdat)
# Generate plot on slighlty wider axes
delta = 0.1*(b-a)
x = np.linspace(a-delta,b+delta,301)
y = fun(x)
mpl.plot(x,y,'r-',xdat,ydat,'bo',xdat,0*xdat,'bx')
mpl.plot([x.min(),x.max()],[0.,0.],'k-')
mpl.xlim((x.min(),x.max()))
mpl.ylim((y.min()-delta,y.max()+delta))
mpl.show()
print "length = %s" % len(xdat)

(fun_solution, err_fun) = quad(fun, 0, 5) # quad requires a callable function as an argument
from scipy.integrate import trapz
dat_solution = trapz(ydat, xdat)          # trapz requires numerical data
print 'Solution obtained by adaptive quadrature: %s' % str(fun_solution)
print 'Trapezoidal solution obtained using random samples: %s' % str(dat_solution)
print "Relative error: %s" % str(abs(fun_solution-dat_solution)/fun_solution)


I am working on the file "Assignment-03/question-2/Quadrature-in-SciPy.ipynb".
The first task is:
Quadrature using scipy
In the following example, an explicit integrand (notably $\cos^2(e^x)$) is defined as a lambda function that is passed to the quadrature routine scipy.integrate.quad for evaluation. The quadrature routine adaptively chooses quadrature points based on an error estimator; where the error estimate for the integral over a given subinterval is too large, the subinterval is subdivided recursively and the approximate integral recomputed on each subinterval until the total error estimate is below a prescribed tolerance. Notice that this quadrature routine expects a function $f$ as input that can be evaluated at any quadrature point in the domain. Given the oscillatory nature of the integrand toward the right end of the interval, more quadrature points are required there.
Can you write Python code for it?

fun = lambda x: np.cos(np.exp(x))**2 # define as a one-line "lambda" function
a = 0
b = 3
x = np.linspace(a,b,200)
y = fun(x)
mpl.plot(x,y)
mpl.show()
print fun(b)

from scipy.integrate import quad
(I_val, err) = quad(fun, 0, 3) # Notice return value is a Python *tuple* (here an ordered pair)
print "Approximate value of integral is %15.13g with error estimate %15.10g" % (I_val,err)


I am working on the file "Assignment/.ipynb_checkpoints/AssignmentSteps-checkpoint.ipynb".
The first task is:
Discharge at $x_w, yw = 0, 0$
Can you write Python code for it?

subttl = f'\nkD={kD:.0f} m2/d, S={S:4g}, xw = {xw:.0f} m'

ax0 = newfig(title='Extraction from well' + subttl, xlabel='time [y]',
             ylabel='Q [m3/d]   infiltration -->')
ax1 = newfig(title='Infiltration [m2/d] at y=0' + subttl, xlabel='time [y]',
             ylabel='Qin [m2/d] --> in')

x1, y1 = -300, 0
x2, y2 = -x1, 0
xp, yp =  0, 0
r1 = np.sqrt((x1 - xp) ** 2 + (y1 - yp) ** 2)
r2 = np.sqrt((x2 - xp) ** 2 + (y2 - yp) ** 2)

Qin = np.zeros_like(time)
Q = np.zeros_like(time)
for iw in range(len(welldata)):
    dQ, tch = welldata.iloc[iw][['dQ', 'tch']]
    n = np.sum(time > tch)
    u1 = r1 ** 2 * S / (4 * kD * (time[time > tch] - tch))
    u2 = r2 ** 2 * S / (4 * kD * (time[time > tch] - tch))
    Q[time > tch]   += dQ
    dQin = -dQ * (np.exp(-u1)/ (2 * np.pi * r1) + np.exp(-u2) / (2 * np.pi * r2))
    #ax1.plot(time[time > tch] / ayear, dQin)
    Qin[time > tch] += dQin
ax0.plot(time / ayear, Q, 'r', lw=5, alpha=0.3, label='total')
ax1.plot(time / ayear, Qin, 'r', lw=5, alpha=0.3, label='total')

ax2 = ax1.twiny()
ax2.plot(time / ayear, Q/np.abs(Q), 'g', lw=5, alpha=0.3, label='season')
ax3 = ax0.twiny()
ax3.plot(time / ayear, Q/np.abs(Q), 'g', lw=5, alpha=0.3, label='season')
ax0.legend()
ax1.legend()

secaxyaxis

ax1.twiny(


I am working on the file "Assignment/.ipynb_checkpoints/AssignmentSteps-checkpoint.ipynb".
The first task is:
Including the mirror well
Can you write Python code for it?

subttl = f'\nkD={kD:.0f} m2/d, S={S:4g}'

ax0 = newfig(title='Extraction from well' + subttl, xlabel='time [y]',
             ylabel='Q [m3/d] infiltration -->')
ax1 = newfig(title='head due to extraction of one well',
             xlabel='time [y]', ylabel='s [m]')

x1, y1 = -200, 0
x2, y2 =  200, 0
xp, yp =  100, 100
r1 = np.sqrt((x1 - xp) ** 2 + (y1 - yp) ** 2)
r2 = np.sqrt((x2 - xp) ** 2 + (y2 - yp) ** 2)

s = np.zeros_like(time)
Q = np.zeros_like(time)
for iw in range(len(welldata)):
    dQ, tch = welldata.iloc[iw][['dQ', 'tch']]
    n = np.sum(time > tch)
    u1 = r1 ** 2 * S / (4 * kD * (time[time > tch] - tch))
    u2 = r2 ** 2 * S / (4 * kD * (time[time > tch] - tch))
    Q[time > tch]  += dQ
    ds = -dQ/(4 * np.pi * kD) * (exp1(u1) - exp1(u2))
    #ax1.plot(time[time > tch] / ayear, ds)
    s[time > tch] += ds
ax0.plot(time / ayear, Q, 'r', lw=5, alpha=0.3, label='total')
ax1.plot(time / ayear, s, 'r', lw=5, alpha=0.3, label='total')
ax0.legend()
ax1.legend()


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Assigment
The assignment is solved step by step. First steady state, one point. Then more points. Then points a long the river bank. Then integrate along the river. Then make it transient. And finally compute for all river-bank points and all times, and integrate along the river. As very larst, show it for a few well distances and conclude what a proper distance is so that now water is flows into the ground (is extracted from the river) during the summer.
Can you write Python code for it?

kD = 900. # m2/d
S = 0.2 # [-]

xw = -500.
x1, y1 = xw, 0 # pumping station, the well
x2, y2 = -xw, y1 # mirror well

Q1 = -1500 # extraction by PS
Q2 = -Q1   # extraction by mirror well


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
At this point the Qinto array is filled
Each row has the specific inflow for all times and one point
Each column as the specific inflow at one time and all rows

Now we integrate over the points, along the y-axis just as we did before. The only difference is that we now have an array not just a vector. Anyway if we sum vertically, along the columns we get the total inflow. The only thing  we need to do make broadcasting work is to make $\Delta y$ ans array with rows corresponding to the points and one column. You can just ad an axis to the the vector to align the vector
Intermezzo, showing how to turn a row-vector vertically to align it with the columns in the array
Can you write Python code for it?

a = np.array([1, 3, 6, 0.2]) # standard vector (has no orientation)
b = a[np.newaxis, :]  # turn a in a 2d array with only one row (horizontal orientation)
c = a[:, np.newaxis] # turn a in a 2d array with one column (vertical orientation)
d = a[np.newaxis, :, np.newaxis] # turn it into a plane in a 3d array, with only one planec (advanced)
print('a = ', a, 'note, only single []')
print()
print('b = ', b, ' note, the double [[]]')
print()
print('c = ', c, ' note, the double [[]]')
print()
print('d = ', d, ' note the triple [[[]]]')


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Head change along the x-axis for several times
Can you write Python code for it?


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Inflow to the river
Now that we can compute the specific discharge at any point, we can also plot them for points at the river bank. These are the point where the flow is perpendicular to the river bank.
The only thing we need to do to show them, is to plot them not for random points but only for points at the river bank. So just change the coordinates of the points to those at the river bank ($x=0$).
Nothing else is changed!
Can you write Python code for it?

# New points, along the river bank
y = np.linspace(5 * xw, -5 * xw, 200)  # y values, print them to see what's happening here.
x = np.zeros_like(y) # A of all zeros, the vector is the same size as y.

ax = newfig('Specific discharge points','x', 'y')
ax.plot(x, y, '.')
ax.plot(x1, y1, 'o')
ax.plot(x2, y2, 'o')

r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

q1 = Q1  / (2 * np.pi * r1)
q2 = Q2  / (2 * np.pi * r2)

qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2
qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2

a = 100. # scale factor
for i, (xi, yi) in enumerate(zip(x, y)):
    #print(i, [xi, xi + a * qx[i]],
    #         [yi, yi + a * qy[i]] )
    ax.plot([xi, xi + a * qx[i]],
            [yi, yi + a * qy[i]])


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Integrate the filled-in Qintot array along the columns (i.e.vertically)
Can you write Python code for it?

dY = np.diff(Y)[:, np.newaxis]  # align vertically, to apply broadcasting

Qin = np.sum( (Qintot[:-1] + Qintot[1:]) * dY  / 2, axis=0) # That's all

# Now plot the end result
ax = newfig('Total inflow from river versus time','time [d]', 'Qintot m3/d')

ax.plot(tsim, Qin)


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Now let's get the changetimes in welldata in days after tstart as floating point numbers
Can you write Python code for it?

tchs = welldata.index - tstart
tchs # as timedeltas

# And now as floating point values
tchs = (welldata.index - tstart) / np.timedelta64(1, 'D')
tchs


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Ok, this should get you familiar, now all together, in practice
Can you write Python code for it?

welldata = pd.read_excel('PumpingStationQ.xlsx', index_col=0, parse_dates=True)

tstart = welldata.index[ 0] - np.timedelta64(7 , 'D')
tend   = welldata.index[-1] + np.timedelta64(28, 'D')
tsim = np.arange(   (tend - tstart) / np.timedelta64(1, 'D')   )
tchs = np.asarray(  (welldata.index - tstart) / np.timedelta64(1, 'D')   )
tchs


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Ok, we've now seen some freatures of  pandas. Make sure you get familiar with it, because it's extremely useful for a great many purposes.
What we'll do now is loop over the lines in the table. Each line is a so-called change times, on which the flow from the well changes.
But because we simulate using days, not dates, we need the time in dates. Clearly this requries fixing a
starting time of the simulation, which will get the value zero, while the following values are increasing by 1.
A logical starting date would be the first change time, i.e. the first value of the index of the table. But we
could taken any value.
The point to know is that the only calculation you can do with datetimes, is subtract two datetimes, to get the
time difference between them, this will be a np.timedelta64 object. Lets subtract the two dt's we define above
to show this
Can you write Python code for it?


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Specific discharge $[m^3/d]$ at arbitrary points
Th specific discharge is here the discharge over the entire heigh of the aqufier per m perpdendicular to the flow. Its dimension is, therefore, m3/d/m = m2/d. For a point at a distance $r$ from a well and due to only this well, it is, the flow divided by the length of the circle with radius $r$ around the well:
$$q = \frac {Q} {2 \pi r}$$
We may illustrate the specific discharge by showing it at some arbitrary points. We just take a set of random points and show the specific discharge as a vector (a small line-piece) from each point into the direction of the vector. It will be scaled by a factor $a$ to make it visible. Scaling is necessary, because the dimension of the specific discharge differs from that of the $x$ and $y$ coordinates. Because we only want to illustrate these vectors, the value of the scale factor does not matter, just try a few values until you like the result.
So, the specific discharge has an $x$ and an $y$ component, which equal $q_x = q \cos(\alpha)$ and $q_y = q \sin(\alpha)$ with $\alpha$ the angle between the specific discharge vector and the $x$-axis:
$$\cos \alpha = \frac {x - x_w}{r}$$
$$\sin \alpha = \frac {y - y_w}{r}$$
so that
$$q_x = q \frac {x - x_w}{r}$$
$$q_y = q \frac {y - y_w}{r}$$
The plan is as follows:
    1. First we'll compute 200 random point locations
    2. Then we'll plot the points as dots and the well and its mirror as thicker circles
    3. Compute the specific discharge
    4. Compute the x and y components of the specific discharge
    5. For each point plot the specific dischrage vector multiplied by a scale vector $#a$
Can you write Python code for it?

# random points
x = (np.random.rand(200) - 0.5) * 3 * xw
y = (np.random.rand(200) - 0.5) * 3 * xw

# Set up the figure, show the points, the well and its mirror
ax = newfig('Specific discharge points','x', 'y')
ax.plot(x, y, '.')
ax.plot(x1, y1, 'o') # well
ax.plot(x2, y2, 'o') # mirror well

# distances from well to each point and form mirror well to each point
r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

# Specific discharge (magnitute only) at all points
q1 = Q1  / (2 * np.pi * r1)
q2 = Q2  / (2 * np.pi * r2)

# Super positionn, add well and mirror well
qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2
qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2

a = 100. # scale factor for speciic discharge.

# For each point plot the scaled discharge vector
for i, (xi, yi) in enumerate(zip(x, y)):
    #print(i, [xi, xi + a * qx[i]],
    #         [yi, yi + a * qy[i]] )
    ax.plot([xi, xi + a * qx[i]],
            [yi, yi + a * qy[i]])


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
The discharge at the river
Instead of printing the q vectors along the river bank on the x, y map, we can print them in a figure where the y-values are the y coordinates and where the x-values are the specific discharge. The y-axis then has dimension [m] and the x-axis has dimension [m2/d]. We can then do away with the scale factor $a$.

Change the x-label to 'specific discharge [m2/d]'
Chante the x in the plot to just qx[i]

But we can now also do away with the loop over the points, and, instead, plot y versus qx directly. This gives a graph showing the specific discharge at the river bank.
The rest is unaltered.
Can you write Python code for it?

y = np.linspace(5 * xw, -5 * xw, 200)
x = np.zeros_like(y)

ax = newfig('Specific discharge points','specific discharge [m2/d]', 'y')

r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

q1 = Q1  / (2 * np.pi * r1)
q2 = Q2  / (2 * np.pi * r2)

qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2
qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2

# Plot the individual vectors
for i, (xi, yi) in enumerate(zip(x, y)):
    ax.plot([0,  qx[i]],
            [yi, yi])

# Plot as one (thick) line linewidth = 5.
ax.plot(qx, y, lw=5) # thick line


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
The specific discharge to the well at any point
Steady state first
Can you write Python code for it?

x, y = 0., 100 # any point

r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

q1 = Q1  / (2 * np.pi * r1)
q2 = Q2  / (2 * np.pi * r2)

qx1 = q1 * (x - x1) / r1 # x-component of the specific discharge due to the well
qx2 = q2 * (x - x2) / r2 # same due to the mirror well
qy1 = q1 * (y - y1) / r1 # y-component of the specific discharge due to the well
qy2 = q2 * (y - y2) / r2 # same due to the mirror well

# We can compute the x and y compents due to both well at once, don't need the 4 lines above
qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2  # added together, x-component
qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2  # same for mirror well

print(qx1, qx2, qy1, qy2, qx1 + qx2, qy1 + qy2)


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Total inflow as function of time
Now we have everything in place to compute the total extraction from the river as a function of time. We just have to compute the total inflow for a large number of times, store the values in an array Qintot and then show the results in a figure.
Can you write Python code for it?

y = np.linspace(5 * xw, -5 * xw, 200)
x = np.zeros_like(y)

ax = newfig(f'Total extraction from the river versus time over {y[0]:.0f}<y<{y[-1]:.0f}',
            'time [d]', 'Qintot m3/d')

r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

# We will now use a large number of times to compute the total inflow for.
times = np.linspace(0, 1000, 1001)
times[0] = 0.001 # prevent division by zero in computing u

# Initialize the Qintot array by zeros so that we can fill in the values in the loop below
Qintot = np.zeros_like(times)

for i, t in enumerate(times):
    u1 = r1 ** 2 * S / (4 * kD * t)
    u2 = r2 ** 2 * S / (4 * kD * t)

    q1 = Q1 * np.exp(-u1) / (2 * np.pi * r1)
    q2 = Q2 * np.exp(-u2) / (2 * np.pi * r2)

    qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2
    qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2

    dy = np.diff(y)
    # Here we store the values for t=times[i] in Qintot[i]
    Qintot[i] = np.sum(0.5 * dy * qx[:-1] + 0.5 * dy * qx[1:])

# Afterwards, we just have to make a graph of Qintot versus time
# However, the sign of extraction is negative, the same as that of the well extraction.
# This is fine. You could call it "infiltration into the river instead", then it makes sense that
# extraction is negative. It has the same sign as the extraction from the well, which too is negative
# as in infiltration would be positive. So the curve below will just be negative (=extraction from the river).
# You can just think of Qinto = the total inflow into the river. Then the sign makes sense.
ax.plot(times, Qintot) # change sign because it's inflow


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Total inflow from river
The total inflow from the river is obtained by integrating the inflow along the $y$-axis.
We have an inflow value at every $y$-value. Integrating is equal to multiplying the inflow value with the length it represents along the $y$-axis and then summing it all. Using a simple trapezium rule would be
$$ Q_{intot} = \intop_{-\infty}^{+\infty} q_x(y) dy \approx \sum_{all\, y} q_x(y) * \Delta y $$
In the code below, we compute $\Delta y = diff(y) = y[1:] - y[:-1]$. The number of $\Delta y$ values is one less than the number of $q_x$-values. So we compute the sum as follows
$$ Q_{totin} = \sum_{\Delta y}  \frac {q_x[:-1] + q_x[1:]}{0.5} \times \Delta y $$
The rest is the same as before.
Can you write Python code for it?

y = np.linspace(5 * xw, -5 * xw, 200)
x = np.zeros_like(y)

ax = newfig('Specific discharge points','specific discharge [m2/d]', 'y')

r1 = np.sqrt((x - x1) ** 2 + (y - y1) ** 2)
r2 = np.sqrt((x - x2) ** 2 + (y - y2) ** 2)

q1 = Q1  / (2 * np.pi * r1)
q2 = Q2  / (2 * np.pi * r2)

qx = q1 * (x - x1) / r1 + q2 * (x - x2) / r2
qy = q1 * (y - y1) / r1 + q2 * (y - y2) / r2

ax.plot(qx, y, lw=5)

dy = np.diff(y)

# Here is the integration:
Qintot = np.sum((qx[:-1] + qx[1:]) / 2 * dy)

# Here is the result
print(f'The total extraction from the river over {y[0]:.0f}<y<{y[-1]:.0f} is {Qintot:.0f} m3/d')
print(f'The total extraction from the river over the full y-axis would be Qw={Q1:.0f} m3/d')


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
Using time-dependent well flow
Data in the Excel workbook
Till now, the well rates were constant. However, in the assignment it varies over time as defined in the accompanying workbook "PumpingStationQ.xlsx". So we have to use superposition in time using the data in the Excel workbook.
En Excel workbook is clearly excellent to prepare the data so that you will have table that contains exactly the data that you'll need in Python. This prevents a lot of clutter in Python, and utilizes the power of Excel to prepare and order all kind of information to your needs. The only point is that you have to read in the data in Python.
Using pandas
The most convenient way to read data from a .csv file or an Excel workbook into Python is by using Pandas. The whole world uses Pandas to work with tabled informatin, even if you'd have to deal with a million lines, Pandas will do this efficiently.
Pandas has an enormous amount of functionality, which you can find int the book on Pandas on the Internet or look at the documentation for Pandas on the Internet or even under the help menu of Jupyter. Pressing the pandas menu item under the help dropdown menu above, brings you immedately to the page that says:
pandas: powerful Python data analysis toolkit
I'd say have a look, it's powerful for any data analyse you'd ever need to do.
Reading in the Excel workbook
Reading in the spreadsheet from the Exccel workbook in pandas can be as simple as running the line
welldata = pd.read_excel('PumpingStationQ.xlsx')
Just try it.
But there is a large number of optioinal arguments to this function to fine-tune reading in your data if things are more difficult. To see, them make a fresh cell, type pd.read_excel() and then with the cursor between the parentheses press shift-tab to see them.
What we'll do extra, is tell which column will be used as our index, which will be the data column, and make sure the values are not just strings but datenums, so we can use them in our computation.
Can you write Python code for it?

# Readin the first spreadsheet in the workbook, use the first column as its index and parse the
# values which are strings into real date (datenums). All in one line. It's so simple because we
# prepared our data so well in Excel.

welldata = pd.read_excel('PumpingStationQ.xlsx', index_col=0, parse_dates=True)

# Show the index
# The dtype, which is the type of the values in the index, shows 'datetime64[ns]', which is numpy's
# datenum objects (which is a 64 bit long integer number telling the date and time in nanoseconds accuracy)
# (In Excel datenums are floating point numbers, but in Python its an integer, but you should not look at it
# as an integer but as a np.datetime64 object), with its own functionality.
welldata.index

# The header of the coliums is also an index, that of the columns, here it is
welldata.columns

# To get the so many-th line of the table you use the iloc (i is always a indication that we are dealing with
# a whole number, something we can count)
# The fourth line of our table (remmeber the first line has inedex 0)
welldata['n'].iloc[3]

# The 7 th line of the table
welldata.iloc[6]

# A few lines
welldata.iloc[4:8]

# Some columns and lines
welldata[['n', 'month', 'dQ']].iloc[4:9]


I am working on the file "Assignment/.ipynb_checkpoints/Inclass020200129-checkpoint.ipynb".
The first task is:
You see it's still an index and not an array. This may be ok, if not you can convert it into an array by passing it the funtion np.asarray(), like so:
Can you write Python code for it?

tchs = np.asarray((welldata.index - tstart) / np.timedelta64(1, 'D'))
tchs


I am working on the file "Assignment/VScode/AssJan2017.ipynb".
The first task is:
Because the river is regarded as a straight fixed-head boundary along the y-axis at x=0, it can be simulated using a well and a mirror wel. The well is at (-xw, 0) and the mirror well ast `(xw, 0)
Can you write Python code for it?


I am working on the file "Assignment/VScode/AssJan2017.ipynb".
The first task is:
Compute the infiltration from the river if the extration were continuous after $t=0$
$$Q_{r,t} = Q_0 \, e^{-u}\text{, where }u=\frac{r^2 S}{4 kD t}$$
Hence,
$$q_{r,t} = \frac{Q_0}{2 \pi r} \, e^{-u}\text{, where }u=\frac{r^2 S}{4 kD t}$$
Separating out the components in x- and y-direction:
$$q_x(x, y,t) = \frac{Q_0}{2 \pi r} \, e^{-u} \cos(\alpha)$$
$$q_y(x, y,t) = \frac{Q_0}{2 \pi r} \, e^{-u} \sin(\alpha)$$
And, with $Q < 0$ meaning extration, the specific discharge due to a single well becomes
$$q_x(x, y,t) = -\frac{Q_0}{2 \pi} \frac{x - x_w}{r^2}\,e^{-u}$$
$$q_y(x, y,t) = -\frac{Q_0}{2 \pi} \frac{y - y_w}{r^2}\,e^{-u}$$
Let the river coincide with the y axis and x=0.
Then we generate points along the river where we'll comopute the exchange between river and aquifer.
Can you write Python code for it?

a = 15 # (-), a scale factor for y-coordinates, further no meaning.
Y = np.hstack(( -np.logspace(0, np.log10(a * L))[::-1],
                 np.logspace(0, np.log10(a * L))))
X = np.zeros_like(Y)

ax = newfig(f"Inflow from the river due to well at x={L:.0f} m, Qd={Qd:.0f} m3/d", "qx = m2/d", "y [m]")

for t in show_times:
    qx = np.zeros_like(Y)
    qy = np.zeros_like(Y)
    for k in wells:
        well = wells[k]
        R2 = (X - well['x']) ** 2 + (Y - well['y']) ** 2
        u = R2 * S / (4 * kD * t)
        qx += well['Q'] / (2 * np.pi) * (X - well['x']) / R2 *  np.exp(-u)
        qy += well['Q'] / (2 * np.pi) * (Y - well['y']) / R2 *  np.exp(-u)

    ax.plot(qx, Y, label=f'{t:.0f} d')
ax.legend()


I am working on the file "Assignment/VScode/AssJan2017.ipynb".
The first task is:
First show the heads and the specific discharge to gain overview
Can you write Python code for it?

N, a = 100, 2.
x = np.linspace(-a * L, a * L, N + 1)
y = np.linspace(-a * L, a * L, N + 1)
X, Y = np.meshgrid(x, y)

show_times = [10, 30, 90]

titles = [f'Head contours and stream vectors at t = {t:.0f} d' for t in show_times]
xlabels = ['x [m]' for _ in show_times]
ylabels = ['y [m[' for _ in show_times]

axs = newfigs((1, 3), titles, xlabels, ylabels, sharex=True, sharey=True, figsize=(12, 3), fontsize=8)

for ax, t in zip(axs.ravel(), show_times):
    s = np.zeros_like(X)
    for k in wells:
        well = wells[k]
        r2 = (X - well['x']) ** 2 + (Y - well['y']) ** 2
        u = r2 * S / (4 * kD * t)
        s += well['Q'] / (4 * np.pi * kD) * W(u)
    cs = ax.contour(X, Y, s, levels=20)
    ax.clabel(cs)

plt.show()

# Quiver plot to show the flow field
titles = [f'Stream vextors at t = {t:.0f} d' for t in show_times]
axs = newfigs((1, 3), titles, xlabels, ylabels, sharex=True, sharey=True, figsize=(12, 3), fontsize=8)

xq = np.linspace(-1000, 1000, 21)
yq = np.linspace(-1000, 1000, 20)
Xq, Yq = np.meshgrid(xq, yq)

for ax, t in zip(axs.ravel(), show_times):
    qx = np.zeros_like(Xq)
    qy = np.zeros_like(Yq)
    for k in wells:
        well = wells[k]
        r2 = (Xq - well['x']) ** 2 + (Yq - well['y']) ** 2
        u = r2 * S / (4 * kD * t)
        qx += well['Q'] * np.exp(-u) * (Xq - well['x']) / r2
        qy += well['Q'] * np.exp(-u) * (Yq - well['y']) / r2
    ax.quiver(Xq, Yq, qx, qy)

plt.show()


I am working on the file "Assignment/VScode/AssJan2017.ipynb".
The first task is:
Now simulate the pumping regime by superposition
Flow changes need to be suerimposed:
Can you write Python code for it?

# Carry out superposition in time

# Coordinates
L = 1500.
wells = {1: {'x':+L, 'y':0., 'Q':-Qd},
         2: {'x':-L, 'y':0., 'Q':+Qd}
}

a = 10
yy = np.logspace(2, np.log10(L * a), 50)

Yr = np.hstack((-yy[::-1], 0, yy))[:, np.newaxis]
Xr = np.zeros_like(Yr)
qx = np.zeros((len(Yr), len(tsim)))
qy = np.zeros((len(Yr), len(tsim)))

for dQw, change_date in zip(regime['dQw'], regime['dates']):
    t = (tsim[tsim > change_date] - change_date) / np.timedelta64(1, 'D')
    for k in wells:
        well = wells[k]
        dx_, dy_ = Xr - well['x'], Yr - well['y']
        r2 = dx_ ** 2 + dy_ ** 2
        u = r2 * S / (4 * kD * t) # (ny x nt)
        sign = well['Q'] / abs(well['Q']) # Make  sure the sign for the mirror well is reversed, its -dQw
        qx[:, tsim > change_date] += sign * dQw / (2 * np.pi) * np.exp(-u) * dx_ / r2
        qy[:, tsim > change_date] += sign * dQw / (2 * np.pi) * np.exp(-u) * dy_ / r2
Qx = np.sum(0.5 * (qx[:-1, :] + qx[1:, :]) * np.diff(Yr, axis=0), axis=0)
Qy = np.sum(0.5 * (qy[:-1, :] + qy[1:, :]) * np.diff(Yr, axis=0), axis=0)

ax = newfig("Infiltration/exfiltration rates at change_dates [m/d]", "infiltration/exfiltration rate [m/d", "y [m]")
for change_date in regime['dates']:
    it = np.sum(tsim <= change_date)
    ax.plot(qx[:, it], Yr, label=str(change_date))
ax.legend()
plt.show()

ax = newfig("Pumping regime and inflow into aquifer", "time", "Q[m3/d")
ax.plot(tsim, Qx, label='Qx [m3/d]')
ax.plot(tsim, Qy, label='Qy [m3/d]')
ax.step(regime['dates'], regime['Qw'], where='post', label='regime')
ax.set_ylim((-200, 200))
ax.legend()
plt.show()


I am working on the file "Assignment/VScode/AssJan2017.ipynb".
The first task is:
The next step is to integrate the flow along the river. We do that numerically. One can use the function quad() or just apply the Simpson rule by hand.
Using both quad and the hand method to compute:
$$ Q_{0,y} = \intop_{-\infty}^{+\infty} q_x dy $$
Can you write Python code for it?

ax = newfig(f"Total inflow from the river due to well at x={L:.0f} m, Qd={Qd:.0f} m3/d", "time [d]", "Q_total [m3/d]")

dy = np.diff(Y)

times = np.arange(0, 4001, 10.)
Qrin = np.zeros_like(times)
for it, t in enumerate(times):
    qx = np.zeros_like(Y)
    qy = np.zeros_like(Y)
    for k in wells:
        well = wells[k]
        R2 = (X - well['x']) ** 2 + (Y - well['y']) ** 2
        u = R2 * S / (4 * kD * t)
        qx += well['Q'] / (2 * np.pi) * (X - well['x']) / R2 *  np.exp(-u)
        qy += well['Q'] / (2 * np.pi) * (Y - well['y']) / R2 *  np.exp(-u)
    Qrin[it] = np.sum(0.5 * (qx[:-1] + qx[1:]) * dy)

ax.plot(times, Qrin, label=f'well extraction = {Qd:.0f} m3/d')
ax.hlines([Qd], xmin=0., xmax=times[-1], colors=['r'], label=f'Qd={Qd:.0f} m3/d')
ax.legend()


I am working on the file "Assignment/VScode/AssJan2019.ipynb".
The first task is:
Convenient plotting functions
Can you write Python code for it?

def newfig(title='?', xlabel='?', ylabel='?', xlim=None, ylim=None, xscale=None, yscale=None, figsize=(10, 8),
            fontsize=16):
    sizes = ['xx-small', 'x-small', 'small', 'medium', 'large', 'x-large', 'xx-large']

    assert isinstance(fontsize, int) or fontsize in sizes, \
        "fontsize not int and not in [{}]".format(', '.join(sizes))

    _, ax = plt.subplots(figsize=figsize)
    ax.set_title(title)
    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    if xlim: ax.set_xlim(xlim)
    if ylim: ax.set_ylim(ylim)
    if xscale: ax.set_xscale(xscale)
    if yscale: ax.set_yscale(yscale)
    ax.grid()

    for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
             ax.get_xticklabels() + ax.get_yticklabels()):
        item.set_fontsize(fontsize)
    return ax

def newfigs(layout, titles=['?'], xlabels=['?'], ylabels=['?'], xscales=None, yscales=None,
            xlim=None, ylim=None, sharex=None, sharey=None, fontsize=16, figsize=(10, 8)):
    sizes = ['xx-small', 'x-small', 'small', 'medium', 'large', 'x-large', 'xx-large']

    assert isinstance(fontsize, int) or fontsize in sizes, \
        "fontsize not int and not in [{}]".format(', '.join(sizes))

    fig, axs = plt.subplots(*layout, sharex=sharex, sharey=sharey)
    fig.set_size_inches(figsize)

    assert isinstance(layout, tuple) and len(layout) == 2, 'layout must be a 2-tuple (nrows, ncols) not {}'.format(str(layout))
    n_axes = np.prod(layout)

    if xscales is None: xscales = [None for _ in range(n_axes)]
    if yscales is None: yscales = [None for _ in range(n_axes)]

    for items_name, items in zip(['titles', 'xlabels', 'ylabels', 'xscales', 'yscales'], [titles, xlabels, ylabels, xscales, yscales]):
        assert len(items) == np.prod(layout), 'len({}) == {} != len(layout) == {}'.format(items_name, len(items), len(np.prod(layout)))

    for ax, title, xlabel, ylabel, xscale, yscale in zip(axs, titles, xlabels, ylabels, xscales, yscales):
       ...


I am working on the file "Assignment/VScode/AssJan2019.ipynb".
The first task is:
Impact of river tide
The impact is shown below. The penetration depth of the tide is so small compared to the distance between the river and the building pit, that no measures have to be taken.
Of course, in the case of a lower frequency tide, like two-monthly extra high tides, and storm tides, pentration depth will be larger and this compensation may become necessary.
The same is true for high river stages that persist over longer times than only one month and with much higher values. during wet periods.
Can you write Python code for it?

A, T = 1.0, 0.5 # Amplitude (m), cycle time (d)

omega = 2 * np.pi / T
a = np.sqrt(omega * S / (2 * kD))

show_times = np.arange(24, dtype=float) / 24

Y = np.linspace(0, 2 * L, 101) # keep sign postiive use negative Y only in the graph

ax =newfig("Effect of river tide on the adjacent groundwater", "distance from river[m]", "efect of tide")
env = A * np.exp(-a * Y)
for show_time in show_times:
    s = env * np.sin(omega * show_time - a * Y )
    ax.plot(Y, s, label=f'tide at t={show_time * 24:.0f} h')
ax.plot(Y, +env, 'k', lw=2, label='top envelope')
ax.plot(Y, -env, 'k', lw=2, label='bottom env.')
ax.legend(loc='upper right')
plt.show()


I am working on the file "Assignment/VScode/AssJan2019.ipynb".
The first task is:
Impact of the month-long wave in the river stage
Q6: Compensate for the effect of the change of the river stage.
We should superimpose this change on the drawdown by the wells and adapt the well regime to compensate for this stage change.
We should generate an extra drawdown of 0.4 m at the end of month 4. For this the regime has to be adapated
Can you write Python code for it?

regime['Q'] = np.array([-1325., -1200., -1150., -1100., -1100., -1100., 0., 0., 0., 0., 0., 0.,]) * 5 # Initially guessed pumping regime.

# Extraction that compensates for the river srage change
# Set both to 1 to see what happens without extra pumping.
regime['Q'][3] *= 1.10
regime['Q'][4] *= 1.05

regime['dQ'] = np.diff(np.hstack((0, regime['Q'])))

ax = newfig(f'Water level at critical points', 'time', 'water level [m]')

for kp in ['mid', 'ctr', 'crit']:
    xp, yp = pit[kp]['x'], pit[kp]['y']
    dy = 0 - yp # distance from the river (river is at x=0 m)

    s = np.zeros_like(tsim, dtype=float)

    for dQ, change_date in zip(regime['dQ'], regime['dates']):
        t = (tsim[tsim > change_date] - change_date) / np.timedelta64(1, 'D')

        for kw in wells:
            well = wells[kw]
            dx_, dy_ = xp - well['x'], yp - well['y']
            r2 = dx_ ** 2 + dy_ ** 2
            u = r2 * S / (4 * kD * t)
            s[tsim > change_date] += well['sign'] * dQ / (4 * np.pi * kD) * exp1(u)

    for dstage, change_date in zip(dstages, change_dates):
        t = (tsim[tsim > change_date] -change_date) / np.timedelta64(1, 'D')
        u = dy * np.sqrt(S / (4 * kD * t))
        s[tsim > change_date] += dstage * erfc(u)

    ax.plot(tsim, s, label=f'at point pit[{kp}]=({x0:.0f},{y0:.0f})')

ax1 = ax.twinx()
ax1.set_ylabel('Extraction m3/d')

nwells = np.sum([wells[k]['type']=='wel' for k in wells])

ax1.step(regime['dates'], nwells * regime['Q'], 'r--', lw=2, where='post', label='Extraction')
ax1.legend(loc='lower right')


I am working on the file "Assignment/VScode/AssJan2019.ipynb".
The first task is:
Setup the situation, define the wells and observation point
Can you write Python code for it?


I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
1.  Make a picture of the water table at a line through the pumping well perpendicular to the river at t the following times 0.001, 1day, 1 week, 1 month, 1 year, 10 years after the start of the extraction.
We will use the show_times defined above, to show how to work with dates.
Can you write Python code for it?

# The head at differnt times along the x-axis (from left to right thorugh pumping station, x=0 at river)
Np = 601
Xline = np.linspace(-3 * L, 3 * L, Np) # x coordinates
Yline = np.zeros_like(Xline)

# The head due to the reginal flow, with h=0 at x=0
h = q / kD * Xline # not the sign

ax = newfig("The head (change) along the line thorugh the well and its mirror.", "x [m]", "head change [m]")

for show_time in show_times:
    t = (show_time - tstart) / np.timedelta64(1, 'D')

    s = h # np.zeros_line(Xline)

    for k in wells:
        well = wells[k]
        dx_, dy_ = Xline - well['x'], Yline - well['y']
        r2 = dx_ ** 2 + dy_ ** 2
        r2[r2 < well['r0']] = well['r0']
        u = r2 * Sy / (4 * kD * t)
        s += well['sign'] * Q / (4 *np.pi * kD) * exp1(u)

    ax.plot(Xline, s, label=f'time ={show_time}')
    ax.legend()


I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
4 Inflow from the river.
This is in fact the same as the question above. The main difference is that we only need the y-values at x=0. Therefore our coordinates will be those along the river, i.e. a long vector of zeros for the x and regular values for the y coordinate.
Can you write Python code for it?

# Inflow at the river, i.e. X=0
q = -1 # m2/d

# Generate the 2 grids of coordinates: (using np.meshgrid)
Np = 101
# For contours
a = 6
Yriv = np.linspace(- a * L, a * L, Np + 1)
Xriv = np.zeros_like(Yriv)

ax = newfig(f'River inflow date m2/d, with a fixed base flow q={q:.2f} m2/d', '$q_{in}$ [m2/d]', 'y [m]')
for show_time in show_times:

    qx = np.zeros_like(Yriv) + q
    qy = np.zeros_like(Yriv)

    t = (show_time - tstart) / np.timedelta64(1, 'D')

    for k in wells:
        well = wells[k]

        dx_, dy_ = Xriv - well['x'], Yriv - well['y']
        r2 = dx_ ** 2 + dy_ ** 2
        u = r2 * Sy / (4 * kD * t)
        qx += well['sign'] * Q / (2 * np.pi) * np.exp(-u) * dx_ / r2
        qy += well['sign'] * Q / (2 * np.pi) * np.exp(-u) * dy_ / r2

    ax.plot(qx, Yriv, label=f'date={show_time}, t={t:3g} d')
ax.legend(loc='lower right')
plt.show()


I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
As can be seen, the difference between the situation at t=356 d and t=3650 d is very small. Hence, we can assume that the situation at t=3650 d is practically steady-state.
It is convenient and later on much more compact to set up plots more generally with a function.
2.  Make a picture of the head contours in top view after 1 month, 1 year and 10 years.
Can you write Python code for it?

# Generate the 2 grids of coordinates: (using np.meshgrid)
Np = 101
# For contours
Xcline = np.linspace(- 2 * L, 2 * L, Np + 1)
Ycline = np.linspace(- 2 * L, 2 * L, Np + 1)

# For quiver
Xqline = np.linspace(- 2 * L, 2 * L, Np//4 + 1)
Yqline = np.linspace(- 2 * L, 2 * L, Np//4 + 1)

Xc, Yc = np.meshgrid(Xcline, Ycline)
Xq, Yq = np.meshgrid(Xqline, Yqline)

aday = np.timedelta64(1, 'D')
titles  = [f'Contours date = {show_time}, t={(show_time - tstart)/aday:4g} d' for show_time in show_times[-3:]]
xlabels = ['x [m]' for _ in show_times[-3:]]
ylabels = ['y [m]' for _ in show_times[-3:]]

axs = newfigs((1, 3), titles, xlabels, ylabels, sharex=True, sharey=True, figsize=(12, 5), fontsize='small')
for ax, show_time in zip(axs.ravel(), show_times[-3:]):

    qx = np.zeros_like(Xq)
    qy = np.zeros_like(Xq)

    s = q / kD * Xc * .0
    t = (show_time - tstart) / np.timedelta64(1, 'D')

    for k in wells:
        well = wells[k]

        dx_, dy_ = Xc - well['x'], Yc - well['y']
        r2 = dx_ ** 2 + dy_ ** 2
        r2[r2 < well['r0']] = well['r0']

        u = r2 * Sy  / (4 * kD * t)
        s += well['sign'] * Q / (4 * np.pi * kD) * exp1(u)

        dxq, dyq = Xq - well['x'], Yq - well['y']
        rq2 = dxq ** 2 + dyq ** 2
        uq = rq2 * Sy / (4 * kD * t)
        qx += well['sign'] * Q / (2 * np.pi) * np.exp(-uq) * dxq / rq2
        qy += well['sign'] * Q / (2 * np.pi) * np.exp(-uq) * dyq / rq2

    cs = ax.contour(Xc, Yc, s, levels = 50)
    ax.clabel(cs)

    ax.quiver(Xq, Yq, qx, qy)

plt.show()