I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
The RadarDisplay
This is the most commonly used class designed for surface-based scanning radar
Plot a NEXRAD file
Can you write Python code for it?

nexf = "data/KILN20140429_231254_V06"
nexr = pyart.io.read(nexf)
nexd = pyart.graph.RadarDisplay(nexr)

nexr.fields.keys()

fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(16, 12))
nexd.plot('reflectivity', sweep=1, cmap='pyart_NWSRef', vmin=0., vmax=55., mask_outside=False, ax=ax[0, 0])
nexd.plot_range_rings([50, 100], ax=ax[0, 0])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 0])

nexd.plot('velocity', sweep=1, cmap='pyart_NWSVel', vmin=-30, vmax=30., mask_outside=False, ax=ax[0, 1])
nexd.plot_range_rings([50, 100], ax=ax[0, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 1])

nexd.plot('cross_correlation_ratio', sweep=0, cmap='pyart_BrBu12', vmin=0.85, vmax=1., mask_outside=False, ax=ax[1, 0])
nexd.plot_range_rings([50, 100], ax=ax[0, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[0, 1])

nexd.plot('differential_reflectivity', sweep=0, cmap='pyart_BuDOr12', vmin=-2, vmax=2., mask_outside=False, ax=ax[1, 1])
nexd.plot_range_rings([50, 100], ax=ax[1, 1])
nexd.set_limits((-150., 150.), (-150., 150.), ax=ax[1, 1])


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
The following function will not work with the original VM for the Short course. To use, install s3fs package (via conda or pip). It is a file system interface for AWS S3 buckets and provides a nice interface similar to unix/ftp command line arguments.
Can you write Python code for it?

def open_nexrad_file(filename, io='radx'):
    """
    Open file using pyart nexrad archive method.

    Parameters
    ----------
    filename: str
        Radar filename to open.
    io: str
        Py-ART open method. If radx then file is opened via Radx
        otherwise via native Py-ART function.
        Using Radx will handle split-cut sweeps
    """
    filename, zipped = try_file_gunzip(filename)
    if io.lower() == 'radx':
        radar = pyart.aux_io.read_radx(filename)
    else:
        radar = pyart.io.read_nexrad_archive(filename)
    if zipped:
        os.system('gzip {}'.format(filename))
    return radar

def get_latest_file(radar_id, bucket='noaa-nexrad-level2', engine='s3fs', io='radx'):
    """Return latest NEXRAD data file name."""
    try:
        import s3fs
        import tempfile

        s3conn = s3fs.S3FileSystem(anon=True)
        latest_year = os.path.join(
            bucket, os.path.basename(s3conn.ls(bucket)[-1]))
        latest_month = os.path.join(
            latest_year, os.path.basename(s3conn.ls(latest_year)[-1]))
        latest_day = os.path.join(
            latest_month, os.path.basename(s3conn.ls(latest_month)[-1]))
        s3key = s3conn.ls(os.path.join(latest_day, radar_id))[-1]

        path, filename = os.path.split(s3key)
        with tempfile.TemporaryFile() as temp88d:
            s3fs.get(s3key, temp88d)
        return open_nexrad_file(temp88d, io=io)
    except:
        print("Missing s3fs package, please install via conda or pip.")


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
There are many keyword values we can employe to refine the plot
Keywords exist for title, labels, colorbar, along with others.
In addition, there are many methods that can be employed. For example, pull out a constructed RHI at a given azimuth.
Can you write Python code for it?

nexd.plot_azimuth_to_rhi('reflectivity', 305., cmap='pyart_NWSRef', vmin=0., vmax=55.)
nexd.set_limits((0., 150.), (0., 15.))


I am working on the file "5b_PyART_visualization.ipynb".
The first task is:
Use what you have learned!
Using all that you have learned, make a two panel plot of reflectivity and doppler velocity using the data file from an RHI of NOXP data/noxp_rhi_140610232635.RAWHJFH. Use Cartopy to overlay the plots on a map of Austrailia and play around with differing colormaps and axes limits!
Solution
The following cell loads a solution to the above problem.
Can you write Python code for it?


I am working on the file "5c_Simple_proccessing_and_adding_a_field.ipynb".
The first task is:
Data available by FigShare Here: http://figshare.com/articles/Data_for_AMS_Short_Course_on_Open_Source_Radar_Software/1537461
Download and unpack into the data subdirectory of this repository
Can you write Python code for it?

print radar.fields.keys()

print radar.fields['reflectivity'].keys()

display = pyart.graph.RadarMapDisplay(radar)
f = plt.figure(figsize = [17,4])
plt.subplot(1, 3, 1)
display.plot_ppi_map('differential_reflectivity', max_lat = 26.5, min_lat =25.4, min_lon = -81., max_lon = -79.5,
                     vmin = -7, vmax = 7, lat_lines = np.arange(20,28,.2), lon_lines = np.arange(-82, -79, .5),
                     resolution = 'l')
plt.subplot(1, 3, 2)
display.plot_ppi_map('reflectivity', max_lat = 26.5, min_lat =25.4, min_lon = -81., max_lon = -79.5,
                     vmin = -8, vmax = 64, lat_lines = np.arange(20,28,.2), lon_lines = np.arange(-82, -79, .5),
                     resolution = 'l')
plt.subplot(1, 3, 3)
display.plot_ppi_map('velocity', sweep = 1, max_lat = 26.5, min_lat =25.4, min_lon = -81., max_lon = -79.5,
                     vmin = -15, vmax = 15, lat_lines = np.arange(20,28,.2), lon_lines = np.arange(-82, -79, .5),
                     resolution = 'l')

zdr = radar.fields['differential_reflectivity']['data']
smooth_zdr = np.zeros_like(zdr)
for i in range(smooth_zdr.shape[0]):
    smooth_zdr[i,:] = \
            pyart.correct.phase_proc.smooth_and_trim(zdr[i,:], 8)

radar.add_field_like('differential_reflectivity',
                     'differential_reflectivity_smooth',
                     smooth_zdr, replace_existing = True)

display = pyart.graph.RadarMapDisplay(radar)
f = plt.figure(figsize = [17,4])
plt.subplot(1, 3, 1)
display.plot_ppi_map('differential_reflectivity_smooth', max_lat = 26.5, min_lat =25.4, min_lon = -81., max_lon = -79.5,
                     vmin = -7, vmax = 7, lat_lines = np.arange(20,28,.2), lon_lines = np.arange(-82, -79, .5),
                     resolution = 'i')
plt.subplot(1, 3, 2)
display.plot_ppi_map('reflectivity', max_lat = 26.5, min_lat =25.4, min_lon = -81., max_lon = -79.5,
                     vmin = -8, vmax = 64, lat_lines = np.arange(20,28,.2), lon_lines = np.arange(-82, -79, .5),
                     resolution = 'i')
plt.subplot(1,...


I am working on the file "6 - Regular Expressions.ipynb".
The first task is:
(More square-bracket features) You can also use a dash to indicate a range, so [a-z] matches all lowercase letters. To use a dash without indicating a range, put the dash last, e.g. [abc-]. An up-hat (^) at the start of a square-bracket set inverts it, so [^ab] means any char except 'a' or 'b'.
Group Extraction
The "group" feature of a regular expression allows you to pick out parts of the matching text. Suppose for the emails problem that we want to extract the username and host separately. To do this, add parenthesis ( ) around the username and host in the pattern, like this: r'([\w.-]+)@([\w.-]+)'. In this case, the parenthesis do not change what the pattern will match, instead they establish logical "groups" inside of the match text. On a successful search, match.group(1) is the match text corresponding to the 1st left parenthesis, and match.group(2) is the text corresponding to the 2nd left parenthesis. The plain match.group() is still the whole match text as usual.
Can you write Python code for it?

str = 'purple alice-b@google.com monkey dishwasher'
match = re.search('([\w.-]+)@([\w.-]+)', str)
if match:
    print match.group()   ## 'alice-b@google.com' (the whole match)
    print match.group(1)  ## 'alice-b' (the username, group 1)
    print match.group(2)  ## 'google.com' (the host, group 2)


I am working on the file "6 - Regular Expressions.ipynb".
The first task is:
Emails Example
Suppose you want to find the email address inside the string 'xyz alice-b@google.com purple monkey'. We'll use this as a running example to demonstrate more regular expression features. Here's an attempt using the pattern r'\w+@\w+':
Can you write Python code for it?

str = 'purple alice-b@google.com monkey dishwasher'
match = re.search(r'\w+@\w+', str)
test(match)


I am working on the file "6 - Regular Expressions.ipynb".
The first task is:
Repetition
Things get more interesting when you use + and * to specify repetition in the pattern

+ - 1 or more occurrences of the pattern to its left, e.g. 'i+' = one or more i's
* - 0 or more occurrences of the pattern to its left
? - match 0 or 1 occurrences of the pattern to its left

Leftmost & Largest
First the search finds the leftmost match for the pattern, and second it tries to use up as much of the string as possible -- i.e. + and * go as far as possible (the + and * are said to be "greedy").
Can you write Python code for it?

## i+ = one or more i's, as many as possible.
match = re.search(r'pi+', 'piiig')# =>  found, match.group() == "piii"
test(match)

## Finds the first/leftmost solution, and within it drives the +
## as far as possible (aka 'leftmost and largest').
## In this example, note that it does not get to the second set of i's.
match = re.search(r'i+', 'piigiiii')# =>  found, match.group() == "ii"
test(match)

## \s* = zero or more whitespace chars
## Here look for 3 digits, possibly separated by whitespace.
match = re.search(r'\d\s*\d\s*\d', 'xx1 2   3xx')
test(match)

match = re.search(r'\d\s*\d\s*\d', 'xx12  3xx')
test(match)

match = re.search(r'\d\s*\d\s*\d', 'xx123xx')
test(match)

## ^ = matches the start of string, so this fails:
match = re.search(r'^b\w+', 'foobar')
test(match)

## but without the ^ it succeeds:
match = re.search(r'b\w+', 'foobar')
test(match)


I am working on the file "6 - Regular Expressions.ipynb".
The first task is:
findall With Files
For files, you may be in the habit of writing a loop to iterate over the lines of the file, and you could then call findall() on each line. Instead, let findall() do the iteration for you -- much better! Just feed the whole file text into findall() and let it return a list of all the matches in a single step (recall that f.read() returns the whole text of a file in a single string):
Can you write Python code for it?

# Open file
f = open('data/poem.txt', 'r')
# Feed the file text into findall(); it returns a list of all the found strings
strings = re.findall(r'light\w*', f.read())
strings


I am working on the file "6 - Regular Expressions.ipynb".
The first task is:
findall and Groups
The parenthesis ( ) group mechanism can be combined with findall(). If the pattern includes 2 or more parenthesis groups, then instead of returning a list of strings, findall() returns a list of tuples. Each tuple represents one match of the pattern, and inside the tuple is the group(1), group(2) .. data. So if 2 parenthesis groups are added to the email pattern, then findall() returns a list of tuples, each length 2 containing the username and host, e.g. ('alice', 'google.com').
Can you write Python code for it?

str = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher'
tuples = re.findall(r'([\w\.-]+)@([\w\.-]+)', str)
print tuples

for tuple in tuples:
    print tuple[0]  ## username
    print tuple[1]  ## host


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Answer
We first write a scoring function for clustering so that we can use for GridSearchCV.
Take a look at use_scorer under scikit learn.
Can you write Python code for it?

## Note: We do not guarantee that there is a one-to-one correspondence, and therefore the toy result is different.
##       See Explanation for more information
def clustering_accuracy_score(y_true, y_pred):
    n_labels = len(list(set(y_true)))
    n_clusters = len(list(set(y_pred)))
    Pre = np.zeros((n_clusters, n_labels))
    Rec = np.zeros((n_clusters, n_labels))
    F = np.zeros((n_clusters, n_labels))
    w = np.zeros((n_clusters))
    F_i = np.zeros((n_clusters))
    P = np.zeros((n_labels))
    C = np.zeros((n_clusters))

    for i in range(n_clusters):
        C[i] = sum(y_pred == i)
    for j in range(n_labels):
        P[j] = sum(y_true == j)

    for i in range(n_clusters):
        F_i_max = 0
        for j in range(n_labels):
            if (C[i]):
                Pre[i][j] = sum(y_pred[y_true == j] == i) / C[i]
            if (P[j]):
                Rec[i][j] = sum(y_true[y_pred == i] == j) / P[j]
            if (Pre[i][j]+Rec[i][j]):
                F[i][j] = 2*Pre[i][j]*Rec[i][j]/(Pre[i][j]+Rec[i][j])
            F_i_max = max(F_i_max, F[i][j])
        F_i[i] = F_i_max
        w[i] = sum(y_pred == i) / len(y_pred)
    return F_i.dot(w)

# toy case demonstrating the clustering accuracy
# this is just a reference to illustrate what this score function is trying to achieve
# feel free to design your own as long as you can justify

# ground truth class label for samples
toy_y_true = np.array([0, 0, 0, 1, 1, 2])

# clustering id for samples
toy_y_pred_true = np.array([1, 1, 1, 2, 2, 0])
toy_y_pred_bad1 = np.array([0, 0, 1, 1, 1, 2])
toy_y_pred_bad2 = np.array([2, 2, 1, 0, 0, 0])

toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_true)
print('accuracy', toy_accuracy, ', should be 1')

toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_bad1)
print('accuracy', toy_accuracy, ', should be', 5.0/6.0)

toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_bad2)
print('accuracy', toy_accuracy, ', should be', 4.0/6.0, ', this will be explained in the...


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Data sets: training versus test
Can you write Python code for it?

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1)

num_training = y_train.shape[0]
num_test = y_test.shape[0]
print('training: ' + str(num_training) + ', test: ' + str(num_test))

import numpy as np

# check to see if the data are well distributed among digits
for y in [y_train, y_test]:
    print(np.bincount(y))


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Explanation
I adopt a modified version of F-value selection, that is, for each cluster, select the best label class with highest F-score. This accuracy calculating method supports the condition that number of clusters not equal to number of labels, which supports GridSearchCV on number of clusters.
Formula: Let $C[i]$ denotes cluster $i$ and $P[j]$ denotes label $j$. Then for each $(i, j)$, we have

$$
\begin{align}
\text{Precision}[i][j] & = \frac{\left|C[i]\cap P[j]\right|}{|C[i]|} \
\text{Recall}[i][j] & = \frac{\left|C[i]\cap P[j]\right|}{|P[j]|} \
\text{F-value}[i][j] & = \frac{ 2 \times \text{Precision}[i][j] \times \text{Recall}[i][j]}{\text{Precision}[i][j] + \text{Recall}[i][j]}.
\end{align}
$$
Then for each cluster, we search the best F-value for each label, that is,
$$\text{F-value}[i] = \max\limits_{j} \text{F-value}[i][j].$$
We also store the weight $w$ for each cluster, that is,
$$w[i] = \frac{|C[i]|}{n}$$
Hence the final score is simply the dot product of $\text{F-value}$ and $w$, which is the weighted F-score.
Again, since this accuracy calculating method supports the condition that number of clusters not equal to number of labels, we do not guarantee that there is a bijection between clusters and labels, therefore there are cases that some labels have no corresponding clusters, as the second toy example shows.
Build a pipeline with standard scaler, PCA, and clustering.
Can you write Python code for it?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import KernelPCA
from sklearn.cluster import KMeans
from sklearn.metrics import make_scorer
from scipy.stats import mode

pipe = Pipeline([('scl', StandardScaler()),
                 ('pca', KernelPCA()),
                 ('km', KMeans(random_state=1))])

# map cluster number to acutal label
def cluster_mapping(y_true, y_pred):
    mapping = {}
    n_labels = len(list(set(y_true)))
    n_clusters = len(list(set(y_pred)))
    Pre = np.zeros((n_clusters, n_labels))
    Rec = np.zeros((n_clusters, n_labels))
    F = np.zeros((n_clusters, n_labels))
    P = np.zeros((n_labels))
    C = np.zeros((n_clusters))

    for i in range(n_clusters):
        C[i] = sum(y_pred == i)
    for j in range(n_labels):
        P[j] = sum(y_true == j)

    for i in range(n_clusters):
        F_i_max = 0
        F_i_max_label = 0
        for j in range(n_labels):
            if (C[i]):
                Pre[i][j] = sum(y_pred[y_true == j] == i) / C[i]
            if (P[j]):
                Rec[i][j] = sum(y_true[y_pred == i] == j) / P[j]
            if (Pre[i][j]+Rec[i][j]):
                F[i][j] = 2*Pre[i][j]*Rec[i][j]/(Pre[i][j]+Rec[i][j])
            if (F_i_max < F[i][j]):
                F_i_max_label = j
                F_i_max = F[i][j]
        mapping[i] = F_i_max_label
    return mapping


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Explanation
Since the accuracy is 84.4%, which means more than 1 digit will be incorrectly clustered in a group of 10 digits, the error is still considered to be high (compared with using neural networks or other methods).
The mis-clustered samples, as we can observe from the picture above, are generally two kinds:
1. Hard to differentiate: e.g. No. 14, No. 21 and No, 22. It is even difficult for a human to tell which classes they should belong to.
2. Blurred to much: e.g. No. 4, No. 18 and No. 19. Those can be differentiated by a human but seems too vague to be correctly clustered.
Furthermore, we can find that digit '5' tends to be mis-clustered to digit '9' (e.g. No. 5, No. 17 and No. 18), and digit '9' tends to be mis-clustered to digit '1' and '7' (e.g. No. 1, No. 2, No. 8, No. 19, No. 23 and No. 24). The reason may be that the distance between those digit clusters are not far, and a digit lying around the border can be easily mis-clustered. For example, when the digit is blurred/vague.<br>
Also, we can find from No. 10, No. 14, No. 15 that the digit '1' in the dataset are sometimes twisted, tilted or blurred. It's even hard for a human to tell whether it is digit '1' or not. From the examples, we can see that they tend to be clustered as digit '6' or '2' with respect to their shape and level of clarity.
Tiny image classification
We will use the CIFAR-10 dataset for image object recognition.
The dataset consists of 50000 training samples and 10000 test samples in 10 different classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck; see the link above for more information).
The goal is to maximize the accuracy of your classifier on the test dataset after being optimized via the training dataset.
You can use any learning models (supervised or unsupervised) or optimization methods (e.g. search methods for hyper-parameters).
The only requirement is that your code can run inside an ipynb file, as usual.
Please provide a description of your method, in addition to the code.
Your answer...
Can you write Python code for it?

# Functions to build a user-defined wide resnet for cifa-10
# Author: Somshubra Majumdar https://github.com/titu1994/Wide-Residual-Networks
# Modified By: Gao Chang, HKU

from keras.models import Model
from keras.layers import Input, merge, Activation, Dropout, Flatten, Dense
from keras.layers.convolutional import Convolution2D, MaxPooling2D, AveragePooling2D
from keras.layers.normalization import BatchNormalization
from keras import backend as K

def initial_conv(input):
    x = Convolution2D(16, 3, 3, border_mode='same')(input)

    channel_axis = 1 if K.image_dim_ordering() == "th" else -1

    x = BatchNormalization(axis=channel_axis)(x)
    x = Activation('relu')(x)
    return x

def conv1_block(input, k=1, dropout=0.0):
    init = input

    channel_axis = 1 if K.image_dim_ordering() == "th" else -1

    # Check if input number of filters is same as 16 * k, else create convolution2d for this input
    if K.image_dim_ordering() == "th":
        if init._keras_shape[1] != 16 * k:
            init = Convolution2D(16 * k, 1, 1, activation='linear', border_mode='same')(init)
    else:
        if init._keras_shape[-1] != 16 * k:
            init = Convolution2D(16 * k, 1, 1, activation='linear', border_mode='same')(init)

    x = Convolution2D(16 * k, 3, 3, border_mode='same')(input)
    x = BatchNormalization(axis=channel_axis)(x)
    x = Activation('relu')(x)

    if dropout > 0.0: x = Dropout(dropout)(x)

    x = Convolution2D(16 * k, 3, 3, border_mode='same')(x)
    x = BatchNormalization(axis=channel_axis)(x)
    x = Activation('relu')(x)

    m = merge([init, x], mode='sum')
    return m

def conv2_block(input, k=1, dropout=0.0):
    init = input

    channel_axis = 1 if K.image_dim_ordering() == "th" else -1

    # Check if input number of filters is same as 32 * k, else create convolution2d for this input
    if K.image_dim_ordering() == "th":
        if init._keras_shape[1] != 32 * k:
            init = Convolution2D(32 * k, 1, 1, activation='linear', border_mode='same')(init)
    else:
        if...


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Load data
Can you write Python code for it?

import numpy as np

from sklearn.datasets import load_digits
digits = load_digits()

X = digits.data # data in pixels
y = digits.target # digit labels

print(X.shape)
print(y.shape)

print(np.unique(y))


I am working on the file "6 Cluster and CNN.ipynb".
The first task is:
Visualize data
Can you write Python code for it?

import matplotlib.pyplot as plt
import pylab as pl

num_rows = 4
num_cols = 5

fig, ax = plt.subplots(nrows=num_rows, ncols=num_cols, sharex=True, sharey=True)
ax = ax.flatten()
for index in range(num_rows*num_cols):
    img = digits.images[index]
    label = digits.target[index]
    ax[index].imshow(img, cmap='Greys', interpolation='nearest')
    ax[index].set_title('digit ' + str(label))

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Data from each firm in each period
Can you write Python code for it?

if firms: print(firms_df.describe())
else: print('no data for each firm in each period')


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Managing parameter list
Can you write Python code for it?

ctitle=""
if len(par_df.columns)==2: ctitle=par_df.columns[0]
if len(par_df.columns)==3: ctitle=par_df.columns[1]
if len(ts_df.columns) == 6:
    parList=par_df[ctitle].tolist()
    valList=par_df["Values"].tolist()
if len(ts_df.columns) == 8:
    parList=par_df["Parameter internal names"].tolist()
    valList=par_df["Values"].tolist()
# both parList are generated by the 'print' of parameters.py


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Modified parameters
Can you write Python code for it?


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Parameters
Can you write Python code for it?


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Structural infos, data collected at the beginning of each period
Can you write Python code for it?


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
The origin of the partial_corr source is https://gist.github.com/fabianp/9396204419c7b638d38f
At http://en.wikipedia.org/wiki/Partial_correlation#Using_linear_regression we have the explanation of the need of augmenting the data matrix with a 1 to allow for a constant term in the regression.
Can you write Python code for it?

wn.filterwarnings(action="ignore") # to eliminate a warning about
                                   #LAPACK lib

np.set_printoptions(precision=2,suppress=True)
ts=ts_df.values
ts_int = np.hstack((np.ones((ts.shape[0],1)), ts))

out1=partial_corr.partial_corr(ts_int)[1:, 1:]
out1

ts=ts_df.drop(columns="plannedP.").values
ts_int = np.hstack((np.ones((ts.shape[0],1)), ts))

out2=partial_corr.partial_corr(ts_int)[1:, 1:]
out2

ts=ts_df.drop(columns="totalProd.").values
ts_int = np.hstack((np.ones((ts.shape[0],1)), ts))

out3=partial_corr.partial_corr(ts_int)[1:, 1:]
out3

ts2_df=ts_df
if len(ts_df.columns) == 6:
    ts2_df.columns = \
    ['unempl','totalProfit','totalProd','plannedP','price','wage']
if len(ts_df.columns) == 8:
    ts2_df.columns = \
    ['unempl','totalProfit','totalProd','plannedP','cQ','hPSd','price','wage']

result = sm.ols(formula="totalProfit ~ price + wage + totalProd + unempl", \
                data=ts2_df).fit()

print (result.summary())


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
Time series, data collected at the end of each period
Can you write Python code for it?

if len(ts_df.columns) == 6:
    ts_df.columns = \
    ['unempl.','totalProfit','totalProd.','plannedP.','price','wage']
    # to have shorter names
if len(ts_df.columns) == 8:
    ts_df.columns = \
    ['unempl.','totalProfit','totalProd.','plannedP.', 'cQ','hPSd','price','wage']
    # to have shorter names

ts_df

ts_df.describe()

ts_df.corr(method="pearson").style.format("{:.2}")


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
dictionay of values
d_val
it comes from the file *_par.csv coming from the 'print' of parameters.py
NB the different versions of the model have different parameters output sequences; the main difference is about the 6 time series case and the 8 time series case in file *_ts.csv, emerging above
[zip() function take iterables (can be zero or more), makes
iterator that aggregates elements based on the iterables passed,
and returns an iterator of tuples. zip() function take iterables
(can be zero or more), makes iterator that aggregates elements
based on the iterables passed, and returns an iterator of tuples]
Can you write Python code for it?


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
levels of gray
https://en.wikipedia.org/wiki/Shades_of_gray
Can you write Python code for it?

myPlot = ts_df.plot(figsize=(11,8),secondary_y=['hPriceSd', 'price','wage'],marker="*",
color=["OrangeRed","LawnGreen","Blue","Violet","lightblue","Pink","Gray","Brown"])
myPlot.set_ylabel('unemployed, totalProfit, totalProduction, plannedProduction, consumptionQ')
myPlot.right_ax.set_ylabel('hPriceSd, price, wage')
myPlot.legend(loc='upper left') #, bbox_to_anchor=(-0.35, 0.5)
myPlot.axes.right_ax.legend(loc='lower right') #, bbox_to_anchor=(1.1, 0.5)

myPlot = ts_df.plot(figsize=(11,8),secondary_y=['hPriceSd', 'price','wage'],marker="",
color=["lightgray","Black","Black","Black","Gray","lightgray","lightgray","lightgray"],
style=['-', '--', '-.', ':','-', '--', '-.'],
linewidth=1.)
myPlot.set_ylabel('unemployed, totalProfit, totalProduction, plannedProduction, consumptionQ')
myPlot.right_ax.set_ylabel('hPriceSd, price, wage')
myPlot.legend(loc='upper left') #, bbox_to_anchor=(-0.35, 0.5)
myPlot.axes.right_ax.legend(loc='lower right') #, bbox_to_anchor=(1.1, 0.5)

myPlot = ts_df.plot(figsize=(11,8),secondary_y=['hPriceSd', 'price','wage'],marker="",
color=["silver","Black","Black","Black","Gray","slategray","slategray","slategray"],
style=['-', '--', '-.', ':','-', '--', '-.'],
linewidth=2.)
myPlot.set_ylabel('unemployed, totalProfit, totalProduction, plannedProduction, consumptionQ')
myPlot.right_ax.set_ylabel('hPriceSd, price, wage')
myPlot.legend(loc='upper left') #, bbox_to_anchor=(-0.35, 0.5)
myPlot.axes.right_ax.legend(loc='lower right') #, bbox_to_anchor=(1.1, 0.5)

str_df.plot(figsize=(11,8),secondary_y='workers',marker="*",color=["r","b"])

str_df.plot(figsize=(11,8),secondary_y='workers',marker="*",color=["black",
            "lightgrey"])

str_df.plot(figsize=(11,8),linewidth=2.0,secondary_y='workers',marker="*",color=["black",
            "gray"])


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
select rows (time steps) in the database
activating the cell below before running the whole program
[a:b] => from a+1 to b

[:b]  => fron init to b
[a:]  => fron a+1 to end
Can you write Python code for it?


I am working on the file "6 objectSwarmObserverAgents_AESOP_turtleLib_NetworkX/oligopoly/readingCsvOutput_par_corr_BWter.ipynb".
The first task is:
the parameter checkResConsUnsoldProd (not affecting the model, but only working on its output) appears 20180829 in 28ter experiment; in the first commit, of 20180830, the name is checkResCons, but quite immediately became checkResConsUnsoldProd; the commit of 20181013 signals the we have the output from parameters.py (the experiment 80 is of 20181009, so without that output); all the experiments from 28ter to 80 have internally checkResConsUnsoldProd set to True
so from >= 20180829 to <= 20181009 the val of checkResConsUnsoldProd is True
1535414400 is equivalent to 08/28/2018 @ 12:00am (UTC)
1539129600 is equivalent to 10/10/2018 @ 12:00am (UTC)
Can you write Python code for it?

import platform
def creation_date(path_to_file):
    """
    Try to get the date that a file was created, falling back to when it was
    last modified if that isn't possible.
    See http://stackoverflow.com/a/39501288/1709587 for explanation.
    """
    if platform.system() == 'Windows':
        return os.path.getctime(path_to_file)
    else: #MacOs
        stat = os.stat(path_to_file)
        try:
            return stat.st_birthtime
        except AttributeError:
            # We're probably on Linux. No easy way to get creation dates here,
            # so we'll settle for when its content was last modified.
            return stat.st_mtime

#converter https://www.unixtimestamp.com

fileTime=creation_date("./"+nameFilePar)

if fileTime >= 1535414400 and fileTime <= 1539129600:
    row[8]='True'

#row

#for i in range(1,len(row)-1):
#    print(row[i],"& ",end='')
#print(row[-1])

for i in range(1,26):
    print(row[i],"& ",end='')
print(row[26])

for i in range(27,len(row)-1):
    print(row[i],"& ",end='')
if '[' in row[-1]: row[-1]=row[-1][1:5] # [1:5] is to avoid the [ ] output
print(row[-1])


I am working on the file "6-Pandas.ipynb".
The first task is:
Cargando los datos y explorándolos
Trabajaremos sobre un fichero de datos metereológicos de la Consejeria Agricultura Pesca y Desarrollo Rural Andalucía.
http://www.juntadeandalucia.es
Can you write Python code for it?

# Vemos qué pinta tiene el fichero
!head ./tabernas_meteo_data.txt


I am working on the file "6-Pandas.ipynb".
The first task is:
Las fechas también se pueden parsear de manera manual con el argumento:
date_parser=lambda x: pd.datetime.strptime(x, '%d-%m-%y'),  # Parseo manual
Can you write Python code for it?

# Pedomos información general del dataset
data.info()

# Descripción estadística
data.describe()

# Una vez convertido en un objeto fecha se pueden obtener cosas como:
data.index.dayofweek


I am working on the file "6-Pandas.ipynb".
The first task is:
Plotting
Líneas
Can you write Python code for it?

# Pintar la temperatura máx, min, med
data.plot(y=["TMAX", "TMIN", "TMED"])
plt.title('Temperaturas')


I am working on the file "6-Pandas.ipynb".
The first task is:
Podemos agrupar nuestros datos utilizando groupby:
Can you write Python code for it?

# Agruparemos por año y día: creemos dos columnas nuevas
data['year'] = data.index.year
data['month'] = data.index.month

# Creamos la agrupación
monthly = data.groupby(by=['year', 'month'])

# Podemos ver los grupos que se han creado
monthly.groups.keys()

# Accedemos a un grupo
monthly.get_group((2016,3)).head()

# O hacemos una agregación de los datos:
monthly_mean = monthly.mean()
monthly_mean.head(24)


I am working on the file "6-Pandas.ipynb".
The first task is:
Por último, pandas proporciona métodos para calcular magnitudes como medias móviles usando el método rolling:
Can you write Python code for it?

# Calcular la media de la columna TMAX
monthly.TMAX.mean().head(15)

# Media trimensual centrada
monthly_mean.TMAX.rolling(3, center=True).mean().head(15)


I am working on the file "6-Pandas.ipynb".
The first task is:
Puedo incluso hacer secciones basadas en fechas:
Can you write Python code for it?


I am working on the file "6-Pandas.ipynb".
The first task is:
Tenemos que hacer los siguientes cambios:

Separar los campos por un número arbitrario de espacios en blanco.
Saltar las primeras líneas.
Dar nombres nuevos a las columnas.
Descartar la columna del día del año (podemos calcularla luego).
Parsear las fechas en el formato correcto.
Can you write Python code for it?

data = pd.read_csv(
    "./tabernas_meteo_data.txt",
    delim_whitespace=True,   # delimitado por espacios en blanco
    usecols=(0, 2, 3, 4, 5),  # columnas que queremos usar
    skiprows=2,  # saltar las dos primeras líneas
    names=['DATE', 'TMAX', 'TMIN', 'TMED', 'PRECIP'],
    parse_dates=['DATE'],
#     date_parser=lambda x: pd.datetime.strptime(x, '%d-%m-%y'),  # Parseo manual
    dayfirst=True,  # ¡Importante
    index_col=["DATE"]  # Si queremos indexar por fechas
)

# Ordenando de más antigua a más moderna
data.sort_index(inplace=True)

# Mostrando sólo las primeras o las últimas líneas
data.head()

# Comprobamos los tipos de datos de la columnas
data.dtypes


I am working on the file "6-Pandas.ipynb".
The first task is:
Vemos que los datos no están en formato CSV, aunque sí tienen algo de estructura. Si intentamos cargarlos con pandas no tendremos mucho éxito:
Can you write Python code for it?

# Tratamos de cargarlo en pandas
pd.read_csv("./tabernas_meteo_data.txt").head(5)


I am working on the file "6-Pandas.ipynb".
The first task is:
Visualizaciones especiales
Can you write Python code for it?

# scatter_matrix
from pandas.tools.plotting import scatter_matrix
axes = scatter_matrix(data.loc[:, "TMAX":"TMED"])

# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = './css/aeropython.css'
HTML(open(css_file, "r").read())


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
6.1.3. Zadaci iz poglavlja "Rad s nizovima znakova" (1.9. poglavlje)
Zadatak 18. Ispišite poruku na ekran 'Rezultat dijenjenja iz 14. zadatka je 7.5, a cijeli dio tog izraza je 7.'. Brojčane vrijednosti ispišite iz objekta odnosno varijable broj1. Za ispis druge vrijednosti trebate funkcijom promijeniti tip podatka.
Can you write Python code for it?

print 'Rezultat dijenjenja iz 14. zadatka je '+str(broj1/2.)+', a cijeli dio tog izraza je '+str(int(broj1/2))+'.'


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
6.1.5. Zadaci iz poglavlja "Kontrola tijeka" (1.11. poglavlje)
Zadatak 38. U varijablu ime pohranite svoje ime. Ispišite poruku na ekran ovisno o tome sadrži li ime paran ili neparan broj znakova (npr. “Ime Maja sadrži paran broj znakova”).
Can you write Python code for it?

ime='Nikola'
if len(ime)%2==0:
    print 'Ime '+ime+' sadrži paran broj znakova.'
else:
    print 'Ime '+ime+' sadrži neparan broj znakova.'


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
6.2. Zadaci iz poglavlja "Rad s datotekama" (2. poglavlje)
Zadatak 48. Pročitajte sadržaj datoteke datoteka.txt. Izračunajte duljinu teksta.
Can you write Python code for it?

dat=open('datoteka.txt','r').read()

print dat
print len(dat)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
6.3. Zadaci iz poglavlja "Regularni izrazi" (3. poglavlje)
Zadatke rješavajte na datotekama hr.txt i en.txt.
Zadatak 64. Regularnim izrazom izračunajte broj pojava samoglasnika u svakoj datoteci.
Can you write Python code for it?

import re

hr=open('hr.txt').read().decode('utf8').lower()
en=open('en.txt').read().decode('utf8').lower()

print len(re.findall(r'[aeiou]',hr))
print len(re.findall(r'[aeiou]',en))


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
6.4. Zadaci iz poglavlja "Rječnici" (4. poglavlje)
Zadatke rješavajte na datotekama hr.txt i en.txt.
Zadatak 69. Stvorite prazni rječnik, pridružite mu više parova (ključ, vrijednost) različitih tipova podataka. Ispišite sve ključeve te potom sve vrijednosti iz stvorenog rječnika.
Can you write Python code for it?

rjecnik={'a':6788,'b':'neki tekst',999:1.5,0.1:'jedan'}
print rjecnik


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Proučite sadržaj datoteke primjer.txt (najbolje pomoću programa Notepad++). predstavlja uzorak rezultata označavanja podataka studenata na kolegiju "Odabrana poglavlja obrade prirodnog jezika". Drugi stupac datoteke predstavlja identifikator korisnika, treći stupac o tome je li točan odgovor unaprijed znan (1 je istina, 0 laž) te četvrti podatak je li taj odgovor točan (1 je ponovno istina, 0 laž). Sljedeće zadatke rješavajte na ovoj datoteci.
Zadatak 53. Izračunajte broj odgovora u uzorku. Broj odgovora jednak je broju redova u datoteci.
Can you write Python code for it?

broj_odgovora=0

for red in open('primjer.txt'):
    broj_odgovora+=1

print broj_odgovora


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 12. U varijablu niz5 pohranite niz znakova 'prvi red\ndrugi red\n3. red\n'.
Can you write Python code for it?

niz5='prvi red\ndrugi red\n3. red\n'
print niz5


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 14. Podijelite vrijednost varijable broj1 sa 2 te ispišite novonastalu realnu vrijednost.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 16. Provjerite je li izraz (6+5)%3 jednak izrazu 1+5/3.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 17. Provjerite je li izraz 2**3 različit od izraza 116/14.5.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 19. U jednu varijablu pohranite svoju godinu rođenja, a u drugu varijablu prošlu godinu. Ispišite poruku 'Prošle ste godine navršili godina: x'. Na mjestu znaka 'x' trebate izračunati broj godina pomoću varijabli. Na kraju promijenite vrijednost varijable rođenja te ponovite ispis.
Can you write Python code for it?

g_rodjenja=1995
g_prosla=2016
print 'Prošle ste godine navršili godina: '+str(g_prosla-g_rodjenja)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 2. U varijablu broj1 pohranite cjelobrojni tip podatka vrijednosti 15 i ispišite ju.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 21. Provjerite sadrži li varijabla niz5 paran broj znakova.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 23. Provjerite nalazi li se podniz 'jo' u varijabli niz1.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 25. U varijabli niz1 zamijenite znak 'k'sa znakom 'K'.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 26. Iz varijable niz5 uklonite bjeline s početka i kraja niza.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 28. Pohranite vrijednost iz prethodnog zadatka u varijablu lista. Stvorite novi niz iz varijable lista spajanjem elemenata prelaskom u novi red \n.
Can you write Python code for it?

lista=niz1.split(' ')
novi_niz='\n'.join(lista)
print novi_niz


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 29. Iz varijable niz4 stvorite listu odvajanjem elemenata tabulatorom '\t'.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 30. U varijablu lista1 pohranite listu ['jedan','dva','tri'].
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 31. U varijablu lista2 pohranite listu s četiri elementa različitih tipova podataka.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 32. U varijablu lista3 pohranite praznu listu.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 33. Iz varijable lista1 indeksirajte prvo slovo posljednjeg elementa.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 34. U varijablu lista4 spojite varijable lista1 i lista2 te izračunajte duljinu novonastale liste.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 35. Iz varijable lista2 promijenite predzadnji element liste.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 36. U varijablu lista1 dodajte elemente 4 te 'pet'.
Can you write Python code for it?

lista1.append(4)
lista1.append('pet')
print lista1


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 40. Prođite kroz sva slova imena i izračunajte broj samoglasnika.
Can you write Python code for it?

ime='Emanuela'

broj_samoglasnika=0

for znak in ime.lower():
    if znak in 'aeiou':
        broj_samoglasnika+=1
print 'Ime '+ime+' sadrži broj samoglasnika: '+str(broj_samoglasnika)+'.'


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 43. Izračunajte sumu parnih brojčanih vrijednosti.
Can you write Python code for it?

lista=[3,4,5,6]

suma_parnih=0

for element in lista:
    if element%2==0:
        suma_parnih+=element

print suma_parnih


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 44. Izračunajte sumu vrijednosti djeljivih s 3.
Can you write Python code for it?

lista=[3,4,5,6]

suma_3=0

for element in lista:
    if element%3==0:
        suma_3+=element

print suma_3


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 45. Definirajte listu nizova znakova. Stvorite novi niz znakova koji sadrži prvi znak svakog elementa liste. Na primjer, iz liste ['dan','mjesec','godina'], niz znakova je 'dmg'.
Can you write Python code for it?

lista=['dan','mjesec','godina']
novi_niz=''

for element in lista:
    novi_niz+=element[0]

print novi_niz


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 46. Stvorite novi niz znakova koji sadrži:
    1. prvi znak elementa liste ako je duljina elementa parna, odnosno
    2. posljednji znak elementa liste ako je duljina elementa neparna. Na primjer, iz liste ['dan','mjesec','godina'], niz znakova je 'nmg'.
Can you write Python code for it?

lista=['dan','mjesec','godina']
novi_niz=''

for element in lista:
    if len(element)%2==0:
        novi_niz+=element[0]
    else:
        novi_niz+=element[-1]

print novi_niz


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 51. Ispišite posljednji znak svakog reda u datoteci datoteka.txt.
Can you write Python code for it?

for red in open('datoteka.txt'):
    print red.strip()[-1]


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 52. Zapišite proizvoljan tekst od više redaka bez dijakritičkih znakova u neku datoteku (Python kodom, dakle funkcijom open() itd.) te potom pročitajte taj tekst iz datoteke i izračunajte njegovu duljinu.
Can you write Python code for it?

d=open('moj_tekst.txt','w')
d.write('neki proizvoljan tekst\nbez dijakritickih znakova\n')
d.write('treba izracunati\nduljinu')
d.close()

print len(open('moj_tekst.txt').read())


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 55. Izračunajte broj točnih odgovora za koje je odgovor unaprijed znan. Iteriranjem kroz datoteku, brojač povećajte za 1 ako treći stupac ima vrijednost 1 i ako četvrti stupac ima vrijednost 1.
Can you write Python code for it?

tocni_odgovori=0

for red in open('primjer.txt'):
    if red.strip().split('\t')[2]=='1' and red.strip().split('\t')[3]=='1':
        tocni_odgovori+=1

print tocni_odgovori


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 56. Izračunajte postotak točnih odgovora. Iteriranjem kroz datoteku, izračunajte broj svih odgovora (redovi u datoteci) i broj točnih odgovora (četvrti stupac ima vrijednost 1). Ispišite postotak točnih odgovora.
Can you write Python code for it?

broj_odgovora=0
tocni_odgovori=0

for red in open('primjer.txt'):
    broj_odgovora+=1
    if red.strip().split('\t')[3]=='1':
        tocni_odgovori+=1

print float(tocni_odgovori)/broj_odgovora*100


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 57. Izračunajte postotak točnih odgovora korisnika s identifikatorom 9 s obzirom na njegove sve odgovore.
Can you write Python code for it?

broj_odgovora_9=0
tocni_odgovori_9=0

for red in open('primjer.txt'):
    if red.strip().split('\t')[1]=='9':
        broj_odgovora_9+=1
        if red.strip().split('\t')[3]=='1':
            tocni_odgovori_9+=1

print float(tocni_odgovori_9)/broj_odgovora_9*100


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 59. Dalje radite nad datotekama hr.txt i en.txt. Izračunajte broj redaka u svakoj od datoteka iteriranjem kroz svaki red.
Can you write Python code for it?

hr_br_redova=0
en_br_redova=0

for red in open('hr.txt'):
    hr_br_redova+=1
for red in open('en.txt'):
    en_br_redova+=1

print 'broj redova u hr je '+str(hr_br_redova)+', a u en je '+str(en_br_redova)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 6. U varijablu niz2 pohranite niz znakova '5.05' i ispišite ju.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 60. Izračunajte broj znakova u svakoj od datoteka. Pazite da ispravno dekodirate tekst prije izračuna njegove duljine.
Can you write Python code for it?

hr_tekst=open('hr.txt').read().decode('utf8')
en_tekst=open('en.txt').read().decode('utf8')

hr_br_znakova=len(hr_tekst)
en_br_znakova=len(en_tekst)

print 'broj znakova u hr je '+str(hr_br_znakova)+', a u en je '+str(en_br_znakova)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 61. Izračunajte broj znakova razmaka i prelazaka u novi red te pretpostavite broj riječi u svakom dokumentu. Izračun broja traženih znakova provedite iteriranjem kroz svaki znak.
Can you write Python code for it?

hr_br_rijeci=0
en_br_rijeci=0

for znak in hr_tekst:
    if znak in ' \n':
        hr_br_rijeci+=1

for znak in en_tekst:
    if znak in ' \n':
        en_br_rijeci+=1

print 'procjena broja riječi u hr je '+str(hr_br_rijeci)+', a u en je '+str(en_br_rijeci)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 62. Izračunajte procjenu prosječne duljine riječi u svakom uzorku ((broj_znakova-broj_rijeci)/broj_rijeci).
Can you write Python code for it?

print 'procjena prosječne duljine riječi u hr je '+str(float(hr_br_znakova-hr_br_rijeci)/hr_br_rijeci)
print 'procjena prosječne duljine riječi u en je '+str(float(en_br_znakova-en_br_rijeci)/en_br_rijeci)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 63. Izračunajte broj pojava svakog samoglasnika u svakom od tekstnih uzoraka. Izvršite cijeli izračun jednom iteracijom po datoteci kroz svaki znak pojedinog niza znakova.
Can you write Python code for it?

hr_br_a=0
hr_br_e=0
hr_br_i=0
hr_br_o=0
hr_br_u=0

en_br_a=0
en_br_e=0
en_br_i=0
en_br_o=0
en_br_u=0

for znak in hr_tekst.lower():
    if znak=='a':
        hr_br_a+=1
    elif znak=='e':
        hr_br_e+=1
    elif znak=='i':
        hr_br_i+=1
    elif znak=='o':
        hr_br_o+=1
    elif znak=='u':
        hr_br_u+=1

print 'hr a '+str(hr_br_a)
print 'hr e '+str(hr_br_e)
print 'hr i '+str(hr_br_i)
print 'hr o '+str(hr_br_o)
print 'hr u '+str(hr_br_u)

hr_br_sam=float(hr_br_a+hr_br_e+hr_br_i+hr_br_o+hr_br_u)

print 'hr a '+str(hr_br_a/hr_br_sam)
print 'hr e '+str(hr_br_e/hr_br_sam)
print 'hr i '+str(hr_br_i/hr_br_sam)
print 'hr o '+str(hr_br_o/hr_br_sam)
print 'hr u '+str(hr_br_u/hr_br_sam)

for znak in en_tekst.lower():
    if znak=='a':
        en_br_a+=1
    elif znak=='e':
        en_br_e+=1
    elif znak=='i':
        en_br_i+=1
    elif znak=='o':
        en_br_o+=1
    elif znak=='u':
        en_br_u+=1

print 'en a '+str(en_br_a)
print 'en e '+str(en_br_e)
print 'en i '+str(en_br_i)
print 'en o '+str(en_br_o)
print 'en u '+str(en_br_u)

en_br_sam=float(en_br_a+en_br_e+en_br_i+en_br_o+en_br_u)

print 'en a '+str(en_br_a/en_br_sam)
print 'en e '+str(en_br_e/en_br_sam)
print 'en i '+str(en_br_i/en_br_sam)
print 'en o '+str(en_br_o/en_br_sam)
print 'en u '+str(en_br_u/en_br_sam)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 65. Regularnim izrazom izračunajte broj pojava jednog ili više samoglasnika u svakoj datoteci.
Can you write Python code for it?

import re

hr=open('hr.txt').read().decode('utf8').lower()
en=open('en.txt').read().decode('utf8').lower()

print len(re.findall(r'[aeiou]+',hr))
print len(re.findall(r'[aeiou]+',en))


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 66. Regularnim izrazom izračunajte broj pojava znamenki u svakoj datoteci.
Can you write Python code for it?

import re

hr=open('hr.txt').read().decode('utf8').lower()
en=open('en.txt').read().decode('utf8').lower()

print len(re.findall(r'\d',hr))
print len(re.findall(r'\d',en))


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 67. Regularnim izrazom izračunajte broj pojava jedne ili više znamenki u svakoj datoteci.
Can you write Python code for it?

import re

hr=open('hr.txt').read().decode('utf8').lower()
en=open('en.txt').read().decode('utf8').lower()

print len(re.findall(r'\d+',hr))
print len(re.findall(r'\d+',en))


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 7. U varijablu niz3 pohranite prazan niz znakova i ispišite ju.
Can you write Python code for it?


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 72. Ispišite na ekran dvadeset najfrekventnijih znakova iz svakog od rječnika iz prethodnog zadatka s pripadajućim frekvencijama. Usporedite distribucije.
Can you write Python code for it?

hr_sort_znak=sorted(hr_frek_znak.items(),key=lambda x:-x[1])
print hr_sort_znak[:20]

en_sort_znak=sorted(en_frek_znak.items(),key=lambda x:-x[1])
print en_sort_znak[:20]


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 76. U svakoj od datoteka izračunajte tzv. omjer broja različnica (različitih riječi u tekstu) i pojavnica (riječi u tekstu). Broj različnica možete dobiti računanjem duljine rječnika izračunatog u prethodnom zadatku. U kojem je jeziku taj omjer veći? Zašto?
Can you write Python code for it?

print float(len(hr_fd_rijeci))/len(hr_rijeci)
print float(len(en_fd_rijeci))/len(en_rijeci)


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 82. Napišite funkciju opojavnici() koja kao argument prima Unicode niz znakova, a vraća listu pojavncia. Isprobajte funkciju u glavnom programu.
Can you write Python code for it?

def opojavnici(niz):
    import re
    return re.findall(r'\w+',niz,re.UNICODE)

if __name__=='__main__':
    print opojavnici('Ovo je neki niz znakova.')


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 83. U glavnom programu uključite modul funkcije. Pomoću funkcije frekvencijska_distribucija() izračunajte frekvencijsku distribuciju znakova pretvorenih u mala slova iz datoteka hr.txt i en.txt. Ispišite sortirane distribucije te uočite razlike u frekvenciji znakova svakog jezika.
Can you write Python code for it?

import funkcije

hr=open('hr.txt').read().decode('utf8').lower()
en=open('en.txt').read().decode('utf8').lower()

hr_frek=funkcije.frekvencijska_distribucija(hr)
en_frek=funkcije.frekvencijska_distribucija(en)

print funkcije.sortiraj_distribuciju(hr_frek)[:20]
print funkcije.sortiraj_distribuciju(en_frek)[:20]


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 84. Izračunajte frekvencijsku distribuciju pojavnica pretvorenih u mala slova iz datoteka hr.txt i en.txt. Ispišite prvih 50 zapisa sortirane distribucije. O kakvim se riječima radi?
Can you write Python code for it?

hr_pojavnice=[]
hr_pojavnice=funkcije.opojavnici(hr)

en_pojavnice=[]
en_pojavnice=funkcije.opojavnici(en)

hr_sortirano=funkcije.sortiraj_distribuciju((funkcije.frekvencijska_distribucija((funkcije.opojavnici(hr)))))
en_sortirano=funkcije.sortiraj_distribuciju((funkcije.frekvencijska_distribucija((funkcije.opojavnici(en)))))

print hr_sortirano[:50]
print en_sortirano[:50]


I am working on the file "6-Zadaci_rjesenja.ipynb".
The first task is:
Zadatak 85. Izračunajte na temelju svake sortirane distribucije pojavnica iz prošlog zadatka koliki postotak pojavnica u tekstu tvori 20% najčešćih razčičnica. Izračunajte koliko riječi čini 20% različnica te broj pretvorite u cijeli broj. To će biti broj puta koliko će se vršiti iteracija. Iteriranjem kroz sortiranu frekvencijsku distribuciju pojavnica (za već utvrđeni broj puta), zbrojite vrijednosti. Za rezultat, dobiveni broj podijelite sa sveukupnim brojem pojavnica.
Can you write Python code for it?

hr_br20=int(len(hr_sortirano)*0.2)
en_br20=int(len(en_sortirano)*0.2)

hr_br20_pojava=0
en_br20_pojava=0

for par in hr_sortirano[:hr_br20]:
	hr_br20_pojava+=par[1]
hr_rez=float(hr_br20_pojava)/len(hr_pojavnice)

for par in en_sortirano[:en_br20]:
	en_br20_pojava+=par[1]
en_rez=float(en_br20_pojava)/len(en_pojavnice)

print hr_rez
print en_rez


I am working on the file "6-clustering/example-clustering-states.ipynb".
The first task is:
Data visualization
Can you write Python code for it?

#Count the number of states assigned to each cluster
df.cluster.value_counts().plot.bar()

#Let's take a look at how the algorithm clustered states
df.groupby(df.cluster).mean().transpose().plot.bar()


I am working on the file "6-clustering/example-clustering-states.ipynb".
The first task is:
The column names in our data are coded, since we are interested in the meaning of each column we are using, we need to replace the code for the actual meaning. The following cell achieves that.
Can you write Python code for it?

#Rename columns to use their meaning instead their codename
col_names = dict((k, v) for k,v in county_facts_columns.itertuples(index=False, name=None))
df.rename(columns=col_names, inplace=True)
df.head()


I am working on the file "6-clustering/example-clustering-states.ipynb".
The first task is:
We can se that cluster 0 has a much higher proportion of hispanics, cluster 3 of asians and clister 1 african americanas.
Can you write Python code for it?

#Let's see which states are in each cluster
df[df.cluster==0]

df[df.cluster==3]

df[df.cluster==2]


I am working on the file "6-clustering/example-clustering-states.ipynb".
The first task is:
While we can cluster using all columns, let's subset them for now. Feel free to modify this piece of code and experiment with different column combinations!
Can you write Python code for it?

#Select columns that have to do with etnicity proportion
df = df.filter(regex='RHI*')


I am working on the file "6-errors.ipynb".
The first task is:
IndexError - meaning we tried to access a list index that did not exist

File Errors
* errors dealing with reading and writing files
Can you write Python code for it?


I am working on the file "6-errors.ipynb".
The first task is:
Item errors

errors dealing with containers
trying to access an item in a list that doesn't exist
Can you write Python code for it?

letters = ['a', 'b', 'c']
print("Letter #1 is " + letters[0])
print("Letter #2 is " + letters[1])
print("Letter #3 is " + letters[2])
print("Letter #4 is " + letters[3])


I am working on the file "6-errors.ipynb".
The first task is:
Two levels in this traceback:
the first shows code from teh cell above with an arrow pointing to line 2 (favorite_ice_cream()).
the second shows somce code in the other function (favorit_ice_cream, located in the file error_01.py on line 7.
This is where the actual error happened.

LONG tracebacks

sometimes tracebacks are very long (20 levels deep)
this can make it seem like something terrible has happened, but just means your program called many functions before the error

Most times you can just go to the bottom on the trace

What error did we get above?

Last line tells us the type - IndexError and a message "list index out of range"

if you encoutner an error and don't know what it means, read the trace closely

that way if you fix the and then encounter a new one, you can tell that the error change
sometimes just knowing where the error occured is enough to fix it

 Don't the error
* look at the official documentation on errors on docs.python.org (google it)
* it might not be there - it might be a custom error
Syntax Errors

when you forget a colon at the end of a line, add one space too man when indenting uder an if statement, forget parenthesis -- you'll get a syntax errror
this means python  couldn't figure out how to read your program
this is similar to forgetting punctuation in English:

this text is difficult to read there is no punctuation there is also no capitalization why is this hard because you have to figure out where each sentence ends you also have to figure out where each sentence begins to some extent it might be ambiguous if there should be a sentence break or not

if python can't figure out how to read a program, it will give up and inform you
Can you write Python code for it?

def some_function()
    msg = "hello, world"
    print(msg)
      return msg


I am working on the file "6-errors.ipynb".
The first task is:
ah, we actually had two errors in that code
now we see an 'IndentationError' and the arrow pointing to the 'return' statement

this tells us or indentation is off on that line

SyntaxError and IndentationError indicate a problem with syntax in your program

IndentationError is more specific - it always means there is aproblem with your indentation in the code

Note on TABS and Spaces
* both white spaces
* hard to visually tell def.
* some editors don't show the diff
Variable Name Errors

very common errors
occurs when you try and use a variable that doesn't exist
Can you write Python code for it?


I am working on the file "6-errors.ipynb".
The first task is:
lastly you might have made  a typo
let's say we fixed the above by adding the line Count = 0
you still get an error
remember variables are case sensitive, so the var count is different from Count
Can you write Python code for it?

Count = 0
for number in range(10):
    count = count + number
print("The count is:" + str(count))


I am working on the file "6-errors.ipynb".
The first task is:
or forgot to create the variable before using it
Can you write Python code for it?

for number in range(10):
    count = count + number
print("The count is:" + str(count))


I am working on the file "6-errors.ipynb".
The first task is:
usually tell us the var name doesn't exist
Why does this occur?
maybe you forgot to quote a string
Can you write Python code for it?


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Application to Image Denoising
Let's put our convolutional autoencoder to work on an image denoising problem. It's simple: we will train the autoencoder to map noisy digits images to clean digits images.
Here's how we will generate synthetic noisy digits: we just apply a gaussian noise matrix and clip the images between 0 and 1.
Can you write Python code for it?

from keras.datasets import mnist
import numpy as np

(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = np.reshape(x_train, (len(x_train), 28, 28, 1))  # adapt this if using `channels_first` image data format
x_test = np.reshape(x_test, (len(x_test), 28, 28, 1))  # adapt this if using `channels_first` image data format

noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape)
x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape)

x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Because our latent space is two-dimensional, there are a few cool visualizations that can be done at this point.
One is to look at the neighborhoods of different classes on the latent 2D plane:
Can you write Python code for it?

x_test_encoded = encoder.predict(x_test, batch_size=batch_size)

plt.figure(figsize=(6, 6))
plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test)
plt.colorbar()
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Convolutional AutoEncoder
Since our inputs are images, it makes sense to use convolutional neural networks (convnets) as encoders and decoders.
In practical settings, autoencoders applied to images are always convolutional autoencoders --they simply perform much better.
The encoder will consist in a stack of Conv2D and MaxPooling2D layers (max pooling being used for spatial down-sampling), while the decoder will consist in a stack of Conv2D and UpSampling2D layers.
Can you write Python code for it?

from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model
from keras import backend as K

input_img = Input(shape=(28, 28, 1))  # adapt this if using `channels_first` image data format

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

conv_autoencoder = Model(input_img, decoded)
conv_autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

from keras import backend as K

if K.image_data_format() == 'channels_last':
    shape_ord = (28, 28, 1)
else:
    shape_ord = (1, 28, 28)

(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

x_train = np.reshape(x_train, ((x_train.shape[0],) + shape_ord))
x_test = np.reshape(x_test, ((x_test.shape[0],) + shape_ord))

x_train.shape

from keras.callbacks import TensorBoard

batch_size=128
steps_per_epoch = np.int(np.floor(x_train.shape[0] / batch_size))
conv_autoencoder.fit(x_train, x_train, epochs=50, batch_size=128,
                     shuffle=True, validation_data=(x_test, x_test),
                     callbacks=[TensorBoard(log_dir='./tf_autoencoder_logs')])

decoded_imgs = conv_autoencoder.predict(x_test)

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i+1)
    plt.imshow(x_test[i].reshape(28, 28))
   ...


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Decoder Network
Finally, we can map these sampled latent points back to reconstructed inputs:
Can you write Python code for it?

decoder_h = Dense(intermediate_dim, activation='relu')
decoder_mean = Dense(original_dim, activation='sigmoid')
h_decoded = decoder_h(z)
x_decoded_mean = decoder_mean(h_decoded)


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Here's how the noisy digits look like:
Can you write Python code for it?

n = 10
plt.figure(figsize=(20, 2))
for i in range(n):
    ax = plt.subplot(1, n, i+1)
    plt.imshow(x_test_noisy[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Let's Visualise the VAE Model
Can you write Python code for it?

from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(vae).create(prog='dot', format='svg'))

## Exercise: Let's Do the Same for `encoder` and `generator` Model(s)


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Let's train the AutoEncoder for 100 epochs
Can you write Python code for it?

autoencoder.fit(x_train_noisy, x_train,
                epochs=100,
                batch_size=128,
                shuffle=True,
                validation_data=(x_test_noisy, x_test),
                callbacks=[TensorBoard(log_dir='/tmp/autoencoder_denoise',
                                       histogram_freq=0, write_graph=False)])


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Now Let's Take a look....
Can you write Python code for it?

decoded_imgs = autoencoder.predict(x_test_noisy)

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i+1)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(2, n, i + n + 1)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Pretraining encoders
One of the powerful tools of auto-encoders is using the encoder to generate meaningful representation from the feature vectors.
Can you write Python code for it?

# Use the encoder to pretrain a classifier


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Sample generation with Autoencoder
Can you write Python code for it?

encoded_imgs = np.random.rand(10,32)
decoded_imgs = decoder.predict(encoded_imgs)

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # generation
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Testing the Autoencoder
Can you write Python code for it?

from matplotlib import pyplot as plt

%matplotlib inline

encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Traing on MNIST Digits
Can you write Python code for it?

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

vae.fit(x_train, x_train,
        shuffle=True,
        epochs=epochs,
        batch_size=batch_size,
        validation_data=(x_test, x_test))


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
VAE on MNIST
We train the model using the end-to-end model, with a custom loss function: the sum of a reconstruction term, and the KL divergence regularization term.
Can you write Python code for it?

from keras.objectives import binary_crossentropy

def vae_loss(x, x_decoded_mean):
    xent_loss = binary_crossentropy(x, x_decoded_mean)
    kl_loss = - 0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)
    return xent_loss + kl_loss

vae.compile(optimizer='rmsprop', loss=vae_loss)


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
Variational AutoEncoder
(Reference https://blog.keras.io/building-autoencoders-in-keras.html)
Variational autoencoders are a slightly more modern and interesting take on autoencoding.
What is a variational autoencoder ?
It's a type of autoencoder with added constraints on the encoded representations being learned.
More precisely, it is an autoencoder that learns a latent variable model for its input data.
So instead of letting your neural network learn an arbitrary function, you are learning the parameters of a probability distribution modeling your data.
If you sample points from this distribution, you can generate new input data samples:
a VAE is a "generative model".
How does a variational autoencoder work?
First, an encoder network turns the input samples $x$ into two parameters in a latent space, which we will note $z_{\mu}$ and $z_{log_{\sigma}}$.
Then, we randomly sample similar points $z$ from the latent normal distribution that is assumed to generate the data, via $z = z_{\mu} + \exp(z_{log_{\sigma}}) * \epsilon$, where $\epsilon$ is a random normal tensor.
Finally, a decoder network maps these latent space points back to the original input data.
The parameters of the model are trained via two loss functions:

a reconstruction loss forcing the decoded samples to match the initial inputs (just like in our previous autoencoders);
and the KL divergence between the learned latent distribution and the prior distribution, acting as a regularization term.

You could actually get rid of this latter term entirely, although it does help in learning well-formed latent spaces and reducing overfitting to the training data.
Encoder Network
Can you write Python code for it?

batch_size = 100
original_dim = 784
latent_dim = 2
intermediate_dim = 256
epochs = 50
epsilon_std = 1.0

x = Input(batch_shape=(batch_size, original_dim))
h = Dense(intermediate_dim, activation='relu')(x)
z_mean = Dense(latent_dim)(h)
z_log_sigma = Dense(latent_dim)(h)


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
We can use these parameters to sample new similar points from the latent space:
Can you write Python code for it?

from keras.layers.core import Lambda
from keras import backend as K

def sampling(args):
    z_mean, z_log_sigma = args
    epsilon = K.random_normal(shape=(batch_size, latent_dim),
                              mean=0., stddev=epsilon_std)
    return z_mean + K.exp(z_log_sigma) * epsilon

# note that "output_shape" isn't necessary with the TensorFlow backend
# so you could write `Lambda(sampling)([z_mean, z_log_sigma])`
z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_sigma])


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
We coudl also have a look at the 128-dimensional encoded middle representation
Can you write Python code for it?

conv_encoder = Model(input_img, encoded)
encoded_imgs = conv_encoder.predict(x_test)

n = 10
plt.figure(figsize=(20, 8))
for i in range(n):
    ax = plt.subplot(1, n, i+1)
    plt.imshow(encoded_imgs[i].reshape(4, 4 * 8).T)
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "6. AutoEncoders and Embeddings/6.1. AutoEncoders and Embeddings.ipynb".
The first task is:
What we've done so far allows us to instantiate 3 models:

an end-to-end autoencoder mapping inputs to reconstructions
an encoder mapping inputs to the latent space
a generator that can take points on the latent space and will output the corresponding reconstructed samples.
Can you write Python code for it?

# end-to-end autoencoder
vae = Model(x, x_decoded_mean)

# encoder, from inputs to latent space
encoder = Model(x, z_mean)

# generator, from latent space to reconstructed inputs
decoder_input = Input(shape=(latent_dim,))
_h_decoded = decoder_h(decoder_input)
_x_decoded_mean = decoder_mean(_h_decoded)
generator = Model(decoder_input, _x_decoded_mean)


I am working on the file "6. Community Detection.ipynb".
The first task is:
1. Load the Network
Here, we use the Zachary's karate club network $K$ to illustrate some concepts on network structures. This network was first presented in [1].
[1] W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of Anthropological Research 33, 452-473 (1977).
Can you write Python code for it?


I am working on the file "6. Community Detection.ipynb".
The first task is:
2. Network Structure and Properties

Clique
Connected Components
Edge Betweenness
Can you write Python code for it?

print "Number of nodes: ", K.size()
print "Number of edges: ", K.order()

pos = nx.fruchterman_reingold_layout(K);

plt.figure(figsize=(8,8));
plt.axis("off");
nx.draw_networkx_nodes(K, pos, node_size=300, node_color="black");
nx.draw_networkx_edges(K, pos, alpha=0.500);
nx.draw_networkx_labels(K, pos, font_color="white");
plt.show();


I am working on the file "6. Community Detection.ipynb".
The first task is:
4. Agglomerative Method: Hierarchical Clustering
In this method, we start with a distance matrix $D$, which is a matrix that provides the distance between two nodes $i$ and $j$ in network $G$. Using $D$, we then perform hierarchical clustering in a recursive manner, group similar nodes at each step. There are various ways to perform the similarity grouping using agglomerative procedures such as single, complete, and average, to name a few. In the example below, we implement the average/UPGMA linkage, which is the one implemented in the Ravasz Algorithm published in Science in 2002.
In the following cells, we perform hierarchical clustering and visualize the order in which the nodes are grouped together in communities.
Can you write Python code for it?

__author__ = """\n""".join(['Erika Fille Legara <legareft@ihpc.a-star.edu.sg>',
                            'Maksim Tsvetovat <maksim@tsvetovat.org',
                            'Drew Conway <drew.conway@nyu.edu>',
                            'Aric Hagberg <hagberg@lanl.gov>'])

'''
The original code was written by Drew Conway and Aric Hagberg and was later
modified by Maksim Tsvetovat for his book titled “Social Network Analysis for Startups”.
EF Legara slightly modified Tsvetovat's version to address some issues
on the ordering of keys within the path_length dictionary. Moreover,
in this code, a different agglomerative procedure is implemented.
'''

def create_hc(G, t):
    ## Set-up the distance matrix D
    labels=G.nodes()    # keep node labels
    path_length=nx.all_pairs_shortest_path_length(G)
    distances=np.zeros((len(G),len(G)))
    for u,p in path_length.items():
        for v,d in p.items():
            distances[G.nodes().index(u)][G.nodes().index(v)] = d
            distances[G.nodes().index(v)][G.nodes().index(u)] = d
            if u==v: distances[G.nodes().index(u)][G.nodes().index(u)]=0

    # Create hierarchical cluster (HC)
    # There are various other routines for agglomerative clustering,
    # but here we create the HCs using the complete/max/farthest point linkage
    Y = distance.squareform(distances) ## the upper triangular of the distance matrix
    Z = hierarchy.average(Y)

    # This partition selection (t) is arbitrary, for illustrive purposes
    membership=list(hierarchy.fcluster(Z,t=t))

    # Create collection of lists for blockmodel
    partition = defaultdict(list)
    for n,p in zip(list(range(len(G))),membership):
        partition[p].append(labels[n])

    return Z, membership, partition

Z, membership, partition = create_hc(K, t=1.15)
partition.items()

partition = {}
i = 0
for i in range(len(membership)):
    partition[i]=membership[i]

plt.figure(figsize=(10,10))
plt.axis('off')

nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu,...


I am working on the file "6. Community Detection.ipynb".
The first task is:
5. Modularity-Based Community Detection: The Louvain Method
The community detection algorithm that we implement in this notebook is a modularity-based algorithm. The modularity $M_c$ quantifies how good a "community" or partition is, and is given by
\begin{equation}
M_c = \sum _{c=1} ^{n_c} \left[ \frac{L_c}{L} - \left (\frac{k_c}{2L} \right)^2\right]
\end{equation}
where $n_c$ is the number of communiies, $L_c$ is the number of links within the community, $k_c$ is the total degree of nodes belonging to the community, and $L$ is the total number of links in the network. The illustration below that is taken from the slide deck shows how modularity is calculated.

The higher the network modularity is, the more optimal the partitionioning. And, if you only have a single community, $M=0$.

In this recipe, we implement a modularity-maximization algorithm called the Louvain algorithm. We go back to Zachary's karate club and find the best partitions.
Can you write Python code for it?


I am working on the file "6. Community Detection.ipynb".
The first task is:
6. More Real-World Networks
Sample 1: The Marvel Universe
There are two types of Marvel Universe networks that is available on the web, the Comic Network and the Hero Network. The Comic Network connects the characters to the commic issues they appeared in; on ther hand, the Hero Network represents the connections between characters as they appear in the same comic issue (weighted). In this notebook, we look at the Hero Network.
Can you write Python code for it?

import unicodecsv as csv

M = nx.Graph(name="Marvel Universe")
reader = csv.reader(open("./Module 6 Datasets/hero-network.csv", 'rU'))
for row in reader:
    M.add_edge(*row)


I am working on the file "6. Community Detection.ipynb".
The first task is:
Cliques
Cliques are complete subgraphs; that is, subgraphs where the nodes are all directly connected to each other. For example, using the Karate Club network $K$, the cliques are listed below. Note that any two connected nodes in an undirected network are considered to form a clique.
Can you write Python code for it?


I am working on the file "6. Community Detection.ipynb".
The first task is:
Finding Communities
Can you write Python code for it?

partition = community.best_partition(M)

len(set(partition.values()))


I am working on the file "6. Community Detection.ipynb".
The first task is:
Finding Communities
To perform the community detection algorithm, the directed graph needs to be made into an undirected one.
Can you write Python code for it?

PR = P.to_undirected()
PR = nx.Graph(PR)

mypalette = ["blue","red","green", "yellow", "orange", "violet", "grey", "grey","grey"]

pos = nx.spring_layout(PR)
#colors = [mypalette[PR.node[i]['value']] for i in range(1,len(PR.nodes()))]
colors = [mypalette[PR.node[i]['value']] for i in PR.nodes()]
nx.draw(PR, pos, node_color=colors, node_size=10)
plt.show()


I am working on the file "6. Community Detection.ipynb".
The first task is:
Iteratively remove edges with the highest edge betweenness. For purpose of illustration, we first make a copy of the karate club network $K$. We then perform $i=10$ iterations of edge removal.
Can you write Python code for it?

K2 = K.copy()

def remove_top_ebet(K2):
    ebet = nx.edge_betweenness_centrality(K2)
    sorted_ebet = sorted(ebet.items(), key=operator.itemgetter(1), reverse=True)
    edge_to_remove = sorted_ebet[0]
    K2.remove_edge(*edge_to_remove[0])

    return K2

for i in range(10):
    K2=remove_top_ebet(K2)

nx.draw(K2, pos)
nx.draw_networkx_labels(K2, pos)
plt.show()

for i in range(5):
    K2=remove_top_ebet(K2)

nx.draw(K2, pos)
nx.draw_networkx_labels(K2, pos)
plt.show()


I am working on the file "6. Community Detection.ipynb".
The first task is:
Now, let's redraw the network; this time, highlighting the different communities by node color.
Can you write Python code for it?

plt.figure(figsize=(10,10))
plt.axis('off')

nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu, node_color=partition.values())
nx.draw_networkx_edges(K, pos, alpha=0.5)
nx.draw_networkx_labels(K, pos)
plt.show()


I am working on the file "6. Community Detection.ipynb".
The first task is:
Sample 2: Word Co-occurrence Network
In this example, we look at the word co-occurrence dataset used in the paper titled "News Framing of Population and Family Planning Issues via Syntactic Network Analysis" by E.F. Legara et al. Using a community detection algorithm, we looked for word patterns (how the words were 'arranged' or connected in the sentences of each news article) to evaluate how the media framed the population issue in the Philippines through the use of the different labels to refer to it in public discourse.
Can you write Python code for it?

F = nx.read_gml("./Module 6 Datasets/word-net.gml", label="label")

# nx.write_gexf(F, "framing.gexf")

partition = community.best_partition(F, weight="weight")


I am working on the file "6. Community Detection.ipynb".
The first task is:
Some Network Properties
Can you write Python code for it?

P.nodes(data=True)[0:2]

P.order(), P.size()

pos = nx.spring_layout(P)
nx.draw(P, pos, node_size=30)
plt.show()


I am working on the file "6. Community Detection.ipynb".
The first task is:
The algorithm for finding "communities" based on edge betweenness removal was presented by Girvan-Newmann in their 2002 PNAS paper. Based on the method implemented, the algorithm is classified as a dividive procedure (as opposed to agglomerative, which is discussed in the next section. A code snippet taken here is shown below. <img src="./figs/girvan-newman.png">
Connected Components
Connected components are subgraphs where all the nodes are connected to each via a path. Unlike cliques, they don't have to be directly connected. Moreover, the Giant Component refers to the network's largest component.
Can you write Python code for it?

print "Number of components: ", len(list(nx.connected_components(K2))),"\n"
print list(nx.connected_components(K2))


I am working on the file "6. Data cleaning (matplotlib+pandas). Star Wars Survey /Star+Wars+movies.ipynb".
The first task is:
Now that we've cleaned up the ranking columns, we can find the highest-ranked movie more quickly
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.bar(range(6), star_wars[star_wars.columns[9:15]].mean())


I am working on the file "6. Data cleaning (matplotlib+pandas). Star Wars Survey /Star+Wars+movies.ipynb".
The first task is:
Some columns are currently string types, because the main values they contain are Yes and No. We can make the data a bit easier to analyze down the road by converting each column to a Boolean having only the values True, False, and NaN
Can you write Python code for it?

bool_type = {
    'Yes': True,
    'No': False
}
star_wars['Have you seen any of the 6 films in the Star Wars franchise?'] = star_wars['Have you seen any of the 6 films in the Star Wars franchise?'].map(bool_type)
star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'] = star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'].map(bool_type)

star_wars['Do you consider yourself to be a fan of the Star Wars film franchise?'].head()


I am working on the file "6. Data cleaning (matplotlib+pandas). Star Wars Survey /Star+Wars+movies.ipynb".
The first task is:
The 5th movies (Episode V The Empire Strikes Back) has a highest rating (in this survey 1=best, 6=worst). "Episode III  Revenge of the Sith" has a worst rate.
Can you write Python code for it?


I am working on the file "6. Data cleaning (matplotlib+pandas). Star Wars Survey /Star+Wars+movies.ipynb".
The first task is:
We can figure out how many people have seen each movie just by taking the sum of the column. Earliest movies is more popular - this corresponds to ranking above (earlier have better ranking).
Can you write Python code for it?

males = star_wars[star_wars["Gender"] == "Male"]
females = star_wars[star_wars["Gender"] == "Female"]

## Redo the two previous analyses (find the most viewed movie and the highest-ranked movie) separately for each group

## find highest-ranked movie (lower is better)
plt.bar(range(6), females[females.columns[9:15]].mean())
plt.show()
plt.bar(range(6), males[males.columns[9:15]].mean())
plt.show()

## find most viewed movie (higher is better)
plt.bar(range(6), females[females.columns[3:9]].mean())
plt.show()
plt.bar(range(6), males[males.columns[3:9]].mean())
plt.show()


I am working on the file "6. Data cleaning (matplotlib+pandas). Star Wars Survey /Star+Wars+movies.ipynb".
The first task is:
We need to specify an encoding because the data set has some characters that aren't in Python's default utf-8 encoding. You can read more about character encodings on developer Joel Spolsky's blog.
Can you write Python code for it?

star_wars.head(10)

star_wars.columns

star_wars = star_wars[pd.notnull(star_wars['RespondentID'])]
star_wars.head()


I am working on the file "6. Functions/.ipynb_checkpoints/Functions-checkpoint.ipynb".
The first task is:
Ahora veamos que pasa cuando llamamos a la función
Can you write Python code for it?


I am working on the file "6. Functions/.ipynb_checkpoints/Functions-checkpoint.ipynb".
The first task is:
Esto no significa que la función que acabo de escribir sea definitiva y no pueda yo modificarla para sacar las potencias con otros números. Como veremos a continuación, la función puede tomar cualquier número. Sólo tenemos que hacerlo explícito esta vez...
Can you write Python code for it?

exponente(4, 0.5)

exponente(5, -1)

exponente(0.5, 2)


I am working on the file "6. Functions/.ipynb_checkpoints/Functions-checkpoint.ipynb".
The first task is:
Pausa para dudas
3 ..
2..
1
Muy bien! ahora te toca a tí :)
Ejercicio 2
Crea una función que dibuje una barra de carga con un porcentaje. Digamos que queremos que se dibuje el 35% entonces el resultado de correr la función sería:
[#######-------------] 35%
Ahora prueba tu función con estos valores de porcentage:
    * 12.5%
    * 167 %
    * -20 *
Ejercicio 3
Escribe una función que te diga cuantas consonantes hay en una palabra. Ejemplo: La palabra "carroza" tiene 4 consonantes
Argumentos predeterminados
Hay ocasiones en las cuales los argumentos para una función que vamos a crear los vamos a ocuparemos cotidianamente o simplemente tienen más sentido y para no tener que escribirlos cada vez que llamamos a la función lo que podemos hacer es definirlos desde el momento en el que estamos creando una función
Vamos a asumir que yo quiero hacer una función que eleve a la potencia n un número x. Digamos que de acuerdo a mi experiencia, la mayoría de las personas quiere saber el cuadrado de 4. Lo que hago entonces es una función que tenga como argumentos predeterminados el 4 y el 2... Veamos el ejemplo
Can you write Python code for it?

def exponente(numero=4, exponente=2):
    '''Toma un número y lo eleva a la potencia de otro'''
    resultado = numero**exponente
    return resultado


I am working on the file "6. Functions/Functions.ipynb".
The first task is:
Ahora prueba tu función con estos valores de porcentage:
    * 12.5%
    * 167 %
    * -20 *
Ejercicio 3
Escribe una función que te diga cuantas consonantes hay en una palabra. Ejemplo: La palabra "carroza" tiene 4 consonantes
Argumentos predeterminados
Hay ocasiones en las cuales los argumentos para una función que vamos a crear los vamos a ocuparemos cotidianamente o simplemente tienen más sentido y para no tener que escribirlos cada vez que llamamos a la función lo que podemos hacer es definirlos desde el momento en el que estamos creando una función
Vamos a asumir que yo quiero hacer una función que eleve a la potencia n un número x. Digamos que de acuerdo a mi experiencia, la mayoría de las personas quiere saber el cuadrado de 4. Lo que hago entonces es una función que tenga como argumentos predeterminados el 4 y el 2... Veamos el ejemplo
Can you write Python code for it?


I am working on the file "6. Functions/Functions.ipynb".
The first task is:
Ahora veamos que pasa cuando llamamos a la función
Can you write Python code for it?


I am working on the file "6. Functions/Functions.ipynb".
The first task is:
Ejercicio 1.
Cada una va a escoger una función de las que ya están incluídas en python y la va a explicar a sus compañeras
Creando tus propias funciones en python
Ya que estás más familiarizada con las funciones pre hechas en python te podrás dar cuenta de que no siempre van a tener una que necesites, entonces, ¿cómo puedo hacer mis propias funciones?
En python la forma de hacerlo es la siguiente:
Primero tienes que hacerle claro a python que el bloque de código (o pequeño programa) que vas a escribir forma va a ser una función para esto se escribe def que es la abreviatura para definir.
Después tienes que inventar un nombre para tu función. En teoría puedes llamarlo como quieras, sin embargo, es de buena práctica en python llamar a tus funciones de forma tal que cuando las leas después de meses o años puedas claramente recordar que es lo que hacen.
Después de escribir def y el nombre de la función viene algo crucial para crear funciones, basándote en la estructura de las que ya vienen incluídas en python, qué crees que sea...
... Exactamente!! los argumentos!!
Esta parte es crucial porque de aquí vas a sacar la información necesaria para poder dar un resultado. Veremos esto más adelante.
Después viene el bloque de código que deseas ejecutar y esto puede constar de operaciones complejas y transformación de los datos.
Finalmente, para que quede claro para python lo que te debe devolver al final de la función necesitas escribir un return seguido de lo que será el resultado de la función.
La estructura para definir funciones queda de la siguiente manera:
def nombre_función(argumento 1, argumento 2, ... , argumento n):
    operación 1
operación 2

resultado = operación 1 + operación 2

return resultado

Hagamos una pequeña función como ejemplo.
Can you write Python code for it?

def cuadrado(numero):
    '''Función que da como resultado el cuadrado de un número
    necesitas un número como argumento'''
    resultado = numero**2
    return resultado

# Probemos la función

cuadrado(8)

cuadrado(8.0)

cuadrado(-8)


I am working on the file "6. Functions/Functions.ipynb".
The first task is:
Pausa para dudas
3 ..
2..
1
Muy bien! ahora te toca a tí :)
Ejercicio 2
Crea una función que dibuje una barra de carga con un porcentaje. Digamos que queremos que se dibuje el 35% entonces el resultado de correr la función sería:
[#######-------------] 35%
Can you write Python code for it?

def barras(porcentaje):
    gatos = (porcentaje*20)//100
    guiones = 20 - gatos
    print('['+'#'* gatos + '-' * guiones + ']'+str(porcentaje)+'%')

barras(167)


I am working on the file "6. Network Statistical Inference (Instructor).ipynb".
The first task is:
From visualizing these two distributions, it is clear that they look very different. How do we quantify this difference, and statistically test whether the protein-protein graph could have arisen under an Erdos-Renyi model?
One thing we might observe is that the variance, that is the "spread" around the mean, differs between the E-R model compared to our data. Therefore, we can compare variance of the data to the distribtion of variances under an E-R model.
This is essentially following the logic of statistical inference by 'hacking' (not to be confused with the statistical bad practice of p-hacking).
Exercise
Fill in the skeleton code below to simulate 100 E-R graphs.
Can you write Python code for it?

# 1. Generate 100 E-R graph degree centrality variance measurements and store them.
# Takes ~50 seconds or so.
n_sims = 100
er_vars = np.zeros(n_sims)  # variances for n simulaed E-R graphs.
for i in range(n_sims):
    erG = nx.erdos_renyi_graph(n=len(G.nodes()), p=nx.density(G))
    erG_deg_centralities = list(nx.degree_centrality(erG).values())
    er_vars[i] = np.var(erG_deg_centralities)

# 2. Compute the test statistic that is going to be used for the hypothesis test.
# Hint: numpy has a "var" function implemented that computes the variance of a distribution of data.
ppG_var = np.var(ppG_deg_centralities)

# Do a quick visual check
n, bins, patches = plt.hist(er_vars)
plt.vlines(ppG_var, ymin=0, ymax=max(n), color='red', lw=2)


I am working on the file "6. Network Statistical Inference (Instructor).ipynb".
The first task is:
How are protein-protein networks formed? Are they formed by an Erdos-Renyi process, or something else?

In the G(n, p) model, a graph is constructed by connecting nodes randomly. Each edge is included in the graph with probability p independent from every other edge.

If protein-protein networks are formed by an E-R process, then we would expect that properties of the protein-protein graph would look statistically similar to those of an actual E-R graph.
Exercise
Make a histogram of the degree centralities for the protein-protein interaction graph, and the E-R graph.
Can you write Python code for it?

ppG_deg_centralities = list(nx.degree_centrality(G).values())
n, bins, patches = plt.hist(ppG_deg_centralities)


I am working on the file "6. Network Statistical Inference (Instructor).ipynb".
The first task is:
Introduction
In this notebook, we will walk through a hacker's approach to statistical thinking, as applied to network analysis.
Statistics in a Nutshell
All of statistics can be broken down into two activities:

Descriptively summarizing data. (a.k.a. "descriptive statistics")
Figuring out whether something happened by random chance. (a.k.a. "inferential statistics")

Descriptive Statistics

Centrality measures: mean, median, mode
Variance measures: inter-quartile range (IQR), variance and standard deviation

Inferential Statistics

Models of Randomness (see below)
Hypothesis Testing
Fitting Statistical Models

Load Data
Let's load a protein-protein interaction network dataset.

This undirected network contains protein interactions contained in yeast. Research showed that proteins with a high degree were more important for the surivial of the yeast than others. A node represents a protein and an edge represents a metabolic interaction between two proteins. The network contains loops.
Can you write Python code for it?

# Read in the data.
# Note from above that we have to skip the first two rows, and that there's no header column,and that the edges are
# delimited by spaces in between the nodes. Hence the syntax below:
G = cf.load_propro_network()


I am working on the file "6. Network Statistical Inference (Instructor).ipynb".
The first task is:
Visually, it should be quite evident that the protein-protein graph did not come from an E-R distribution. Statistically, we can also use the hypothesis test procedure to quantitatively test this, using our simulated E-R data.
Can you write Python code for it?

# Conduct the hypothesis test.
ppG_var > np.percentile(er_vars, 99)  # we can only use the 99th percentile, because there are only 100 data points.


I am working on the file "6. Network Statistical Inference (Student).ipynb".
The first task is:
Another way to do this is to use the 2-sample Kolmogorov-Smirnov test implemented in the scipy.stats module. From the docs:

This tests whether 2 samples are drawn from the same distribution. Note
that, like in the case of the one-sample K-S test, the distribution is
assumed to be continuous.
This is the two-sided test, one-sided tests are not implemented.
The test uses the two-sided asymptotic Kolmogorov-Smirnov distribution.
If the K-S statistic is small or the p-value is high, then we cannot
reject the hypothesis that the distributions of the two samples
are the same.

As an example to convince yourself that this test works, run the synthetic examples below.
Can you write Python code for it?

# Scenario 1: Data come from the same distributions.
# Notice the size of the p-value.
dist1 = npr.random(size=(100))
dist2 = npr.random(size=(100))

ks_2samp(dist1, dist2)
# Note how the p-value, which ranges between 0 and 1, is likely to be greater than a commonly-accepted
# threshold of 0.05

# Scenario 2: Data come from different distributions.
# Note the size of the KS statistic, and the p-value.

dist1 = norm(3, 1).rvs(100)
dist2 = norm(5, 1).rvs(100)

ks_2samp(dist1, dist2)
# Note how the p-value is likely to be less than 0.05, and even more stringent cut-offs of 0.01 or 0.001.


I am working on the file "6. Network Statistical Inference (Student).ipynb".
The first task is:
How are protein-protein networks formed? Are they formed by an Erdos-Renyi process, or something else?

In the G(n, p) model, a graph is constructed by connecting nodes randomly. Each edge is included in the graph with probability p independent from every other edge.

If protein-protein networks are formed by an E-R process, then we would expect that properties of the protein-protein graph would look statistically similar to those of an actual E-R graph.
Exercise
Make a histogram of the degree centralities for the protein-protein interaction graph, and the E-R graph.
The construction of an E-R graph requires a value for n and p.
A reasonable number for n is the number of nodes in our protein-protein graph.
A reasonable value for p might be the density of the protein-protein graph.
Can you write Python code for it?