I am working on the file "00_matrix_factorization.ipynb".
The first task is:
Dealing with missing values
A regular SVD can not be computed for a matrix with missing values, therefore these need to be filled by some values, e.g. the average of the matrix. SimEc on the other hand can compute the backpropagation error considering only the available entries of the matrix, which means it can easily handle about 50% missing values without seeing major performance decreases and consistently outperforms the regular SVD (results would probably be even better with more careful hyperparameter tuning, e.g. using some regularization).
Can you write Python code for it?

X = np.eye(n_input)
missing_targets = [0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
e_dims =  [2, 10, 50, 100, 250, 500]
for e_dim in e_dims:
    np.random.seed(15)
    mses = []
    mses_svd = []
    mse_svd = msqe(A, np.dot(U[:,:e_dim], np.dot(S[:e_dim,:e_dim], Vh[:e_dim,:])))
    for m in missing_targets:
        print(m)
        A_noisy = A.copy()
        A_noisy[np.random.rand(*A_noisy.shape)<=m] = -100
        model = SimilarityEncoder(n_input, e_dim, n_output, mask_value=-100, l2_reg_emb=0.00001,
                                  l2_reg_out=0. if m < 0.7 else 0.00001, opt=keras.optimizers.Adamax(lr=0.025 if e_dim < 50 else 0.01))
        model.fit(X, A_noisy, epochs=60)
        mse = msqe(A, model.predict(X))
        mses.append(mse)
        A_noisy[A_noisy == -100] = np.mean(A)
        U_n, s_n, Vh_n = np.linalg.svd(A_noisy, full_matrices=True)
        S_n = np.zeros((n_input, n_output))
        S_n[:n_input, :n_input] = np.diag(s_n)
        mses_svd.append(msqe(A, np.dot(U_n[:,:e_dim], np.dot(S_n[:e_dim,:e_dim], Vh_n[:e_dim,:]))))
    print(mses)
    plt.figure();
    plt.plot([0, missing_targets[-1]], [mse_svd, mse_svd], '--', linewidth=0.5, label='SVD noise free');
    plt.plot(missing_targets, mses_svd, '-o', markersize=3, label='SVD');
    plt.plot(missing_targets, mses, '-o', markersize=3, label='SimEc SVD');
    plt.legend(loc=0);
    plt.title('Matrix factorization of A (%i embedding dim)' % e_dim);
    plt.xticks(missing_targets, missing_targets);
    plt.xlabel('Fraction of Missing Entries');
    plt.ylabel('Mean Squared Error');


I am working on the file "00_matrix_factorization.ipynb".
The first task is:
Eigendecomposition of a square symmetric matrix
Can you write Python code for it?

# get 2 square symmetric matrices as AA^T and A^TA
S1 = np.dot(A, A.T)
S2 = np.dot(A.T, A)
# make sure their range of values is still sort of reasonable (for >100 we should probably rescale)
np.max(S1), np.min(S1), np.max(S2), np.min(S2)

# the corresponding SVD eigenvalues and -vectors should work here as well
for e_dim in [2, 10, 25, 50, 75, 100, 250, 400, 500]:
    print("mse with %3i e_dim: %11.8f" % (e_dim, msqe(S1, np.dot(U[:,:e_dim], np.dot(S[:e_dim,:e_dim]**2, U.T[:e_dim,:])))))
S_out = np.zeros((n_output, n_output))
S_out[:n_input, :n_input] = S[:n_input, :n_input]
for e_dim in [2, 10, 25, 50, 75, 100, 250, 500, 700]:
    print("mse with %3i e_dim: %11.8f" % (e_dim, msqe(S2, np.dot(Vh.T[:,:e_dim], np.dot(S_out[:e_dim,:e_dim]**2, Vh[:e_dim,:])))))

# factorize the similarity matrix S1 with a simec
X = np.eye(n_input)
mses1 = []
mses2 = []
e_dims =  [2,   10,  25,  50,  75,  100,  250,  400,  500,  750,  1000]
l_rates = [0.1, 0.1, 0.1, 0.1, 0.1, 0.05, 0.05, 0.01, 0.01, 0.01, 0.01]
for i, e_dim in enumerate(e_dims):
    model = SimilarityEncoder(n_input, e_dim, S1.shape[1], s_ll_reg=1., S_ll=S1, opt=keras.optimizers.Adamax(lr=l_rates[i]))
    model.fit(X, S1, epochs=100)
    mse = msqe(S1, model.predict(X))
    mses1.append(mse)
    print("mse with %4i e_dim: %11.8f" % (e_dim, mse))
    # scalar product of emedding should also approximate S1
    Y = model.transform(X)
    mse = msqe(S1, np.dot(Y, Y.T))
    mses2.append(mse)
    print("mse with %4i e_dim: %11.8f" % (e_dim, mse))
for i, e_dim in enumerate(e_dims):
    print("mse with %4i e_dim: %11.8f (%11.8f)" % (e_dim, mses1[i], mses2[i]))

# factorize the similarity matrix S2 with a simec
X = np.eye(n_output)
mses1 = []
mses2 = []
e_dims =  [2,   10,  25,  50,  75,  100,  250,  400,  500,  750,  1000]
l_rates = [0.1, 0.1, 0.1, 0.1, 0.1, 0.05, 0.05, 0.01, 0.01, 0.01, 0.01]
for i, e_dim in enumerate(e_dims):
    model = SimilarityEncoder(n_output, e_dim, S2.shape[1], s_ll_reg=1., S_ll=S2, opt=keras.optimizers.Adamax(lr=l_rates[i]))
    model.fit(X, S2,...


I am working on the file "00_matrix_factorization.ipynb".
The first task is:
SVD of a random matrix
Can you write Python code for it?

# mean squared error of approximation decreases with more embedding dim
for e_dim in [2, 10, 25, 50, 75, 100, 250, 400, 500]:
    print("mse with %3i e_dim: %.8f" % (e_dim, msqe(A, np.dot(U[:,:e_dim], np.dot(S[:e_dim,:e_dim], Vh[:e_dim,:])))))

# factorize the matrix with a simec
X = np.eye(n_input)
mses = []
e_dims =  [2, 10, 25, 50, 75, 100, 250, 400, 500, 750, 1000]
l_rates = {400: 0.004, 500: 0.0034, 750: 0.0032, 1000: 0.003}
for e_dim in e_dims:
    model = SimilarityEncoder(n_input, e_dim, n_output, orth_reg=0.001 if e_dim > 500 else 0., opt=keras.optimizers.Adamax(lr=0.005 if e_dim <= 250 else l_rates[e_dim]))
    model.fit(X, A, epochs=50)
    mse = msqe(A, model.predict(X))
    mses.append(mse)
    print("mse with %4i e_dim: %.8f" % (e_dim, mse))
for i, e_dim in enumerate(e_dims):
    print("mse with %4i e_dim: %.8f" % (e_dim, mses[i]))

# factorize the matrix with a simec - transpose works just as well
# this time we use the embedding we learned before as the weights of the last layer
# to get the mapping function for the other side of the equation (which is of course kind of useless
# here since we don't actually map feature vectors but only identiy vectors)
X1 = np.eye(n_input)
X2 = np.eye(n_output)
mses1 = []
mses2 = []
mses3 = []
e_dims =  [2, 10, 25, 50, 75, 100, 250, 400, 500, 750, 1000]
for e_dim in e_dims:
    model = SimilarityEncoder(n_input, e_dim, n_output, opt=keras.optimizers.Adamax(lr=0.005 if e_dim <= 250 else l_rates[e_dim]))
    model.fit(X1, A, epochs=50)
    mse = msqe(A, model.predict(X1))
    mses1.append(mse)
    print("mse with %4i e_dim: %.8f" % (e_dim, mse))
    Y1 = model.transform(X1)
    model = SimilarityEncoder(n_output, e_dim, n_input, W_ll=Y1.T, wll_frozen=True, opt=keras.optimizers.Adamax(lr=0.005 if e_dim <= 250 else l_rates[e_dim]))
    model.fit(X2, A.T, epochs=50)
    mse = msqe(A.T, model.predict(X2))
    mses2.append(mse)
    print("mse with %4i e_dim: %.8f" % (e_dim, mse))
    Y2 = model.transform(X2)
    # the dot product of both embeddings should also...


I am working on the file "00_overview.ipynb".
The first task is:

Can you write Python code for it?


I am working on the file "00_overview.ipynb".
The first task is:
"Cython is a compiled language that generates CPython extension modules. These extension modules can then be loaded and used by regular Python code using the import statement.
It works by producing a standard Python module. However, the behavior differs from standard Python in that the module code, originally written in Python, is translated into C. While the resulting code is fast, ..."
, Wikipedia

Documentation
Video Tutorial with Slides: Cython: Blend the Best of Python and C++, SciPy 2015
Book  Cython, A Guide for Python Programmers, O'Reilly Media (2015)
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('gMvkiQ-gOW8')


I am working on the file "00_overview.ipynb".
The first task is:
"HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.", HDF Group

Documentation
Quickstart
Video Introduction: HDF5 is Eating the World, SyiPy (2015)
Book: Python and HDF5, O'Reilly Media (2013)
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('nddj5OA8LJo')


I am working on the file "00_overview.ipynb".
The first task is:
"NumPy is an extension to the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large library of high-level mathematical functions to operate on these arrays.
Using NumPy in Python gives functionality comparable to MATLAB ...", Wikipedia
* Documentation
* Quickstart
* Video Tutorial with Slides: Introduction to NumPy, SciPy 2015
* Book: Numpy Beginner's Guide - Third Edition, Packt Publishing (2015)
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("1zmV8lZsHF4")


I am working on the file "00_overview.ipynb".
The first task is:
"SciPy contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering.", Wikipedia

Documentation
Book: Learning SciPy for Numerical and Scientific Computing Second Edition, Packt Publishing (2015)

"Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.", Wikpedia

Documentation
Quickstart
Video Tutorial with Notebooks: Analyzing and Manipulating Data with Pandas, SciPy (2015
)* Book: Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, O'Reilly and Associates (2012)
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('0CFFTJUZ2dc')


I am working on the file "00_overview.ipynb".
The first task is:
matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy.
There is also a procedural "pylab" interface ..., designed to closely resemble that of MATLAB.

Documentation: Guides, FAQ, Resources, API, ...
Gallery: Unterschiedlichste Plots mit Code Beispielen
Quickstart
Video Tutorial with Notebooks: Anatomy of matplotlib, SciPy (2015)
Buch: Mastering matplotlib, Packt Publishing (2015)
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("MKucn8NtVeI")


I am working on the file "00_overview.ipynb".
The first task is:
Examples
Typical imports
Can you write Python code for it?

# Import necassary packages
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit

# Set up matpltolib for nice plotting in the notebook
%matplotlib notebook
plt.style.use('seaborn-notebook')


I am working on the file "00_overview.ipynb".
The first task is:
Simple Plot
Can you write Python code for it?

x = np.linspace(0, 4 * np.pi, 200)
rad = x / np.pi

plt.figure(figsize=(12, 3))
line, = plt.plot(rad, 2 * np.sin(x))
plt.ylim(-5, 5)
plt.grid()

plt.tight_layout()


I am working on the file "00_strStr.ipynb".
The first task is:
Given a list of numbers that may has duplicate numbers, return all possible subsets
Can you write Python code for it?

class Solution:
    """
    @param S: A set of numbers.
    @return: A list of lists. All valid subsets.
    """
    def subsetsWithDup(self, S):
        # write your code here
        res = [[]]
        S.sort()
        for i in range(len(S)):
            if i == 0 or S[i] != S[i-1]:
                l = len(res)
            for j in range(len(res)-l, len(res)):
                res.append(res[j] + [S[i]])
        return res

Solution().subsetsWithDup([1,2,2])


I am working on the file "00_strStr.ipynb".
The first task is:
Implement strStr function in O(n + m) time.
strStr return the first index of the target string in a source string. The length of the target string is m and the length of the source string is n.
If target does not exist in source, just return -1.
2. Subsets
Given a set of distinct integers, return all possible subsets.
Can you write Python code for it?

class Solution:
    """
    @param S: The set of numbers.
    @return: A list of lists. See example.
    """
    def subsets(self, S):
        # write your code here
        if len(S) == 0:
            return [[]]
        S.sort()
        res = self.subsets(S[:-1])
        ans = []
        for s in res:
            ans.append(s)
            ans.append(s + [S[-1]])
        return  ans

Solution().subsets([1,2])

class Solution:
    """
    @param S: The set of numbers.
    @return: A list of lists. See example.
    """
    def subsets(self, S):
        # write your code here

        self.results = []
        self.subset = []
        S.sort()
        self.search(S, [], 0)
        return self.results

    def search(self, S, subset, index):
        if index == len(S):
            self.results.append(subset)
            return

        self.search(S, subset + [S[index]], index + 1)
        self.search(S, subset, index + 1)

class Solution:
    """
    @param S: The set of numbers.
    @return: A list of lists. See example.
    """
    def subsets(self, S):
        # write your code here

        res = [[]]
        S.sort()
        for i in range(len(S)):
            for j in range(len(res)):
                res.append(res[j] + [S[i]])
        return res


I am working on the file "00_tensorboard.ipynb".
The first task is:
Linear Regression
Can you write Python code for it?

# Create mock data, solve an sklearn linear model
Nsamp = 50
Nfeatures = 1
xarr = np.linspace(-0.5, 0.5, Nsamp)
np.random.seed(83749)
beta_0 = -2.0
beta_1 = 4.3
yarr = (beta_0 + beta_1 * xarr) + (np.random.normal(size=Nsamp) * 0.5)

mdl = linear_model.LinearRegression(fit_intercept=False)
mdl = mdl.fit(np.c_[np.ones(Nsamp), xarr], yarr)
mdl.coef_

fig, ax = plt.subplots(figsize=(5,5))
plt.scatter(xarr, yarr, s=10, color='blue')
plt.plot(xarr, mdl.coef_[0] + mdl.coef_[1] * xarr, color='red')

logdir='/tmp/tensorflow/00_tensorboard/logs'
if os.path.exists(logdir):
    print('removing logdir')
    os.system('rm -rf {}'.format(logdir))

with tf.Graph().as_default() as graph:

    Nfeatures=1
    X = tf.placeholder(tf.float32, [None, Nfeatures], name='X')
    y = tf.placeholder(tf.float32, [None, 1], name='y')

    with tf.name_scope('logit'):
        W = tf.Variable(tf.random_normal([Nfeatures, 1]), name='W')
        b = tf.Variable(tf.zeros([1]), name='b')
        z = tf.matmul(X, W) + b

        tf.summary.histogram('weights', W)
        tf.summary.histogram('biases', b)
        tf.summary.histogram('z', z)

    with tf.name_scope('mse'):
        loss = tf.reduce_mean(tf.squared_difference(z, y), name='loss')
        tf.summary.scalar('loss', loss)

    with tf.name_scope('train'):
        learning_rate = 0.1
        train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

    merged_summary = tf.summary.merge_all()

feed_dict = {X: xarr.reshape(Nsamp, 1), y: yarr.reshape(Nsamp,1)}
with tf.Session(graph=graph) as sess:

    sess.run(tf.global_variables_initializer())
    file_writer = tf.summary.FileWriter(logdir)
    file_writer.add_graph(graph)

    for i in range(300):
        train_step_out = sess.run(train_step, feed_dict=feed_dict)
        loss_out = sess.run(loss, feed_dict=feed_dict)
        print(i, loss_out)
        s = sess.run(merged_summary, feed_dict=feed_dict)
        file_writer.add_summary(s, i)

    W_out =...


I am working on the file "00_tensorboard.ipynb".
The first task is:
Very simple graph (scalars)
tensorboard --logdir=/tmp/tensorflow/00_tensorboard/logs_01_scalars
Can you write Python code for it?

logdir='/tmp/tensorflow/00_tensorboard/logs_01_scalars'
if os.path.exists(logdir):
    os.system('rm -rf {}'.format(logdir))

with tf.Graph().as_default() as graph:
    a = tf.constant(1, name='a')
    b = tf.constant(2, name='b')
    c = tf.add(a, b, name='c')
    d = tf.multiply(a, c, name='d')

with tf.Session(graph=graph) as sess:
    file_writer = tf.summary.FileWriter(logdir, graph)
    d_out = sess.run(d)


I am working on the file "00_uvod.ipynb".
The first task is:
Polje za diskusijo
Naslednje kode ni treba poganjati, saj je namenjena le temu, da se na koncu zvezka prikaže polje za diskusijo. Komentarji so dobrodošli.
Can you write Python code for it?

import disqus
%reload_ext disqus
%disqus matpy


I am working on the file "01 - Intro to Bookworm.ipynb".
The first task is:
The fisrt thing we'll do is load in a book and a list of its characters. These operations are both pretty simple. The book is loaded in as one long string from a .txt file. Character lists are stored in a .csv, with all potential names for a character stored on each row. They're loaded in as tuples of names in a list of characters.
Can you write Python code for it?

book = load_book('data/raw/ij.txt', lower=True)
characters = load_characters('data/raw/characters_ij.csv')


I am working on the file "01 - Intro to Bookworm.ipynb".
The first task is:
Then we split the book down into sections. Bookworm works by looking for coocurrence of characters in these sections of the text as a proxy for their connectedness. It's a very simple trick which works stupidly well.
There are a few ways we can break down the book into sections:
- get_sentence_sequences() uses NLTK's standard .tokenize() function to split the book into sentences.
- get_word_sequences() uses NLTK's word_tokenize() to split the book into words, of which it will then select ordered lists of length n (default 40).
- get_character_sequences() uses python builtins to split it into substrings of length n (default 200).
Fundamentally, they all return a list of strings which each cover a very small section of the novel. For simplicity's sake we're going to use the sentence-wise splitter.
Can you write Python code for it?


I am working on the file "01 - Intro to Bookworm.ipynb".
The first task is:
Yep... Compare the results we've generated to the ones in the diagram at the top of the notebook.
Same code, different book
Lets run the whole thing for an entirely different book and see whether we get similarly positive results. This time, Harry Potter and The Philosopher's Stone - chosen because you're more likely to have some contextual knowledge of who's who and what's what in that book.
Can you write Python code for it?

book = load_book('data/raw/hp_philosophers_stone.txt', lower=True)
characters = load_characters('data/raw/characters_hp.csv')
sequences = get_sentence_sequences(book)

df = find_connections(sequences, characters)
cooccurence = calculate_cooccurence(df).to_sparse()

characters[:5]

print_five_closest(('harry ', ' potter '))

print_five_closest(('voldemort ', ' lord ', ' you-know-who '))

print_five_closest(('crabbe ',))

print_five_closest(('fred ',))


I am working on the file "01 - Intro to Bookworm.ipynb".
The first task is:
calculate_cooccurence() does this computation and then wipes out any interaction of a character with themselves. For the table above, this would give us:
|             | character 1 | character 2 | character 3 |
|-------------|-------------|-------------|-------------|
| character 1 | 0           | 1           | 1           |
| character 2 | 1           | 0           | 0           |
| character 3 | 1           | 0           | 0           |
showing that character 1 has interacted with character 2 and character 3, but character 2 and character 3 haven't interacted. Note the symmetry across the diagonal...
The cooccurence matrix we're referring to here is also known as an adjacency matrix - I might use the terms interchangably from here on.
The example table above is miniscule in comparison to the dozens of characters who might turn up in a reasonably sized novel, and the hundreds or thousands of opportunities they have to interact with one another. The coocurence matrix in reality is likely to contain much larger numbers between characters who regularly appear in the same sentences. Unless we're working with a really tiny, incestuous network, this coocurence matrix is also probably going to be pretty sparse. For that reason it'll often make sense to store it as a sparse matrix:
Can you write Python code for it?


I am working on the file "01 - Introduction to Scikit-learn.ipynb".
The first task is:
1) Instantiate an object and set the parameters
Can you write Python code for it?


I am working on the file "01 - Introduction to Scikit-learn.ipynb".
The first task is:
And again
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=50)

rf.fit(X_train, y_train)

rf.score(X_test, y_test)

#%load from github

import numpy as np
import pylab as pl
from matplotlib.colors import ListedColormap
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.lda import LDA
from sklearn.qda import QDA

h = .02  # step size in the mesh

names = ["Nearest Neighbors", "Linear SVM", "RBF SVM", "Decision Tree",
         "Random Forest", "AdaBoost", "Naive Bayes", "LDA", "QDA"]
classifiers = [
    KNeighborsClassifier(3),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    DecisionTreeClassifier(max_depth=5),
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
    AdaBoostClassifier(),
    GaussianNB(),
    LDA(),
    QDA()]

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,
                           random_state=1, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(noise=0.3, random_state=0),
            make_circles(noise=0.2, factor=0.5, random_state=1),
            linearly_separable
            ]

figure = pl.figure(figsize=(27, 9))
i = 1
# iterate over datasets
for ds in datasets:
    # preprocess dataset, split into training and test part
    X, y = ds
    X = StandardScaler().fit_transform(X)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4)

    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
          ...


I am working on the file "01 - Introduction to Scikit-learn.ipynb".
The first task is:
Exercises
Load the iris dataset from the sklearn.datasets module using the load_iris function.
Split it into training and test set using train_test_split.
Then train an evaluate a classifier of your choice.
Can you write Python code for it?


I am working on the file "01 - Introduction.ipynb".
The first task is:
How many exceedances of the limit values?
Can you write Python code for it?

exceedances = data > 200
exceedances = exceedances.groupby(exceedances.index.year).sum()
ax = exceedances.loc[2005:].plot(kind='bar')
ax.axhline(18, color='k', linestyle='--')


I am working on the file "01 - Introduction.ipynb".
The first task is:
Let's start with a showcase
Case study: air quality in Europe
AirBase (The European Air quality dataBase): hourly measurements of all air quality monitoring stations from Europe
Starting from these hourly data for different stations:
Can you write Python code for it?

data = pd.read_csv('data/airbase_data.csv', index_col=0, parse_dates=True, na_values='-9999')

data


I am working on the file "01 - Introduction.ipynb".
The first task is:
What is the difference in diurnal profile between weekdays and weekend?
Can you write Python code for it?

data['weekday'] = data.index.weekday
data['weekend'] = data['weekday'].isin([5, 6])
data_weekend = data.groupby(['weekend', data.index.hour])['FR04012'].mean().unstack(level=0)
data_weekend.plot()


I am working on the file "01 - Multimedia.ipynb".
The first task is:
An image can also be displayed from raw data or a URL.
Can you write Python code for it?


I am working on the file "01 - Multimedia.ipynb".
The first task is:
External sites
You can even embed an entire page from another site in an iframe; for example this is today's Wikipedia
page for mobile users:
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://ipython.org', width='100%', height=350)


I am working on the file "01 - Multimedia.ipynb".
The first task is:
HTML
Python objects can declare HTML representations that will be displayed in the Notebook. If you have some HTML you want to display, simply use the HTML class.
Can you write Python code for it?

from IPython.display import HTML

s = """

Header 1
Header 2

row 1, cell 1
row 1, cell 2

row 2, cell 1
row 2, cell 2

"""

h = HTML(s)

display(h)


I am working on the file "01 - Multimedia.ipynb".
The first task is:
Or you can enter LaTeX directly with the %%latex cell magic:
Can you write Python code for it?

%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}


I am working on the file "01 - Multimedia.ipynb".
The first task is:
Pass a string of JavaScript source code to the JavaScript object and then display it.
Can you write Python code for it?

js = Javascript('alert("hi")');

display(js)


I am working on the file "01 - Multimedia.ipynb".
The first task is:
Returning an Image object from an expression will automatically display it:
Can you write Python code for it?


I am working on the file "01 - Multimedia.ipynb".
The first task is:
The same thing can be accomplished using the %%javascript cell magic:
Can you write Python code for it?


I am working on the file "01 - Multimedia.ipynb".
The first task is:
Using the nascent video capabilities of modern browsers, you may also be able to display local
videos.  At the moment this doesn't work very well in all browsers, so it may or may not work for you;
we will continue testing this and looking for ways to make it more robust.
The following cell loads a local file called  animation.m4v, encodes the raw video as base64 for http
transport, and uses the HTML5 video tag to load it. On Chrome 15 it works correctly, displaying a control bar at the bottom with a play/pause button and a location slider.
Can you write Python code for it?

from IPython.display import HTML
from base64 import b64encode
video = open("./animation.m4v", "rb").read()
video_encoded = b64encode(video).decode('ascii')
video_tag = '<video controls alt="test" src="data:video/x-m4v;base64,{0}">'.format(video_encoded)
HTML(data=video_tag)


I am working on the file "01 - Multimedia.ipynb".
The first task is:
Video
More exotic objects can also be displayed, as long as their representation supports the IPython display protocol.  For example, videos hosted externally on YouTube are easy to load:
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('sjfsUzECqK0')


I am working on the file "01 - Multimedia.ipynb".
The first task is:
With the Latex class, you have to include the delimiters yourself.  This allows you to use other LaTeX modes such as eqnarray:
Can you write Python code for it?

from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")


I am working on the file "01 - Multimedia.ipynb".
The first task is:
You can also use the %%html cell magic to accomplish the same thing.
Can you write Python code for it?

%%html

Header 1
Header 2

row 1, cell 1
row 1, cell 2

row 2, cell 1
row 2, cell 2


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
(above : showing dataframe samples of file MID4.xls)
concat of the 9 dataframes into one (concat by row) and NA are being replaced by "unknown"
Can you write Python code for it?

mids_df_concat = pd.concat(mids_df, axis=1)
mids_df_concat.fillna(value="unknown", inplace=True)

mids_df_concat


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
1. Describe the type and the value range of each attribute. Indicate and transform the attributes that can be Categorical.
Data description

pclass: is an Integer between 1 and 3 that represents in which class that passenger was in. It can be categorical because we cannot say that 3rd class is twice as far as 2nd class from 1st class.
survived: is an Integer that tells us whether the passenger survived (1) or died (0). Even though it should be categorical, it is easier to let it non-categorical as it is then possible to sum the values to get the total number of people who survived.
name: is a String that contains the name of the passenger. The title could be extracted from this attribute and be transformed into a categorical attribut but we don't use it here.
sex: is String that is either 'female' or 'male'. It contains the information about the sex of the passenger. It can be categorical.
age: is a Double that represents the age in year of the passenger. It could be transformed into ranges and which could then be considered as categorical.
sibsp: is an Integer that contains the number of siblings and spouse also embarked aboard the Titanic.
parch: is an Integer that contains the number of children and parents also embarked aboard the Titanic.
ticket: is a String that contains the ticket number of the passenger.
fare: is an Integer that represents the price that the passenger paid for its ticket.
cabin: is a String that contains the deck letter and the cabin number of the passenger.
embarked: is a character {S, Q, C} that represent the Port where the passenger embarked.
boat: is the id of the boat that the passenger took after the Titanic crashed into the iceberg.
body: is the number attributed to a dead body when it was found.
home.dest:  represents two informations: before the / is the home of the person and after the / is the destination of the person.
Can you write Python code for it?

data['pclass'] = data.pclass.astype('category')
data['sex'] = data.sex.astype('category')
data['embarked'] = data.embarked.astype('category')


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
2. Plot histograms for the travel class, embarkation port, sex and age attributes. For the latter one, use discrete decade intervals.
Can you write Python code for it?

sns.countplot(x="pclass", data=data, palette="Greens_d")

sns.countplot(x="embarked", data=data, palette="Greens_d")

sns.countplot(x="sex", data=data, palette="Greens_d")


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
3. Calculate the proportion of passengers by cabin floor. Present your results in a pie chart.
Let's first plot a pie for every values we have for the cabin floors.
Can you write Python code for it?


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
5. Calculate the proportion of the passengers that survived by travel class and sex. Present your results in a single histogram.
Seaborn allows us to plot such a graph. The black bars show the variance.
Can you write Python code for it?

sns.barplot(x="sex", y="survived", hue="pclass", data=data);


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
6. Create 2 equally populated age categories and calculate survival proportions by age category, travel class and sex. Present your results in a DataFrame with unique index.
To split the datas in two equally populated age categories, we use the median and group the data on both side of the median. Obviously some datas will reside in the median and hence we will not get two perfectly equally populated categories but we assume that considering the number of datas we have, we condiser a small delta is acceptable
Can you write Python code for it?

df = data[['pclass', 'sex', 'age', 'survived']][data['age'].isnull() != True]
median_age = df[['age']].median()
df['age_cat'] = np.where(df[['age']] > median_age, '>{}'.format(median_age[0]), '<{}'.format(median_age[0]))
df = df[['pclass', 'sex', 'age_cat', 'survived']]
df.groupby(['age_cat', 'pclass', 'sex']).agg(np.mean)


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
As we can see above, around 3/4 of the data is missing for this attribute. We need to ignore the missing values to have a better view of the pie chart. Note that the cabin floor T might seem to be a typo, we will assume it refers to the tank top that one can find on a titanic deckplan and hence keep the value.
Can you write Python code for it?


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
Assumption

We will assume that for each row the total sum is contained in the column named "Total". Therefore if there is a difference between the value in the column "Total" and the sum of the values in the other columns, the ground truth resides in the value in the column "Total".
Based on this assumption, only the columns "Info" and the "Total" will be used.
Can you write Python code for it?

ebola_df = ebola_df[['Total', 'Info']]

# displaying some rows
ebola_df[:5]


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
Data reading
Using the paths we extracted above, we read and then merge the data into 3 Dataframes, one for each country. Finally we add a column to each Dataframe containing the name of the country the data comes from
Can you write Python code for it?

guinea_files = all_ebola_files[0]
guinea_df = pd.concat([pd.read_csv(path) for path in guinea_files], axis=0).rename(
    columns={'Date': 'Date', 'Description': 'Info', 'Totals': 'Total'})
guinea_df['Country'] = 'Guinea'
#'Total deaths of confirmed' | 'Total cases of confirmed'

liberia_files = all_ebola_files[1]
liberia_df = pd.concat([pd.read_csv(path) for path in liberia_files], axis=0).rename(
    columns={'Date': 'Date', 'Variable': 'Info', 'National': 'Total'})
liberia_df['Country'] = 'Liberia'
# 'Total death/s in confirmed cases' | 'Total confirmed cases'

sl_files = all_ebola_files[2]
sl_df = pd.concat([pd.read_csv(path) for path in sl_files], axis=0).rename(
    columns={'date': 'Date', 'variable': 'Info', 'National': 'Total'})
sl_df['Country'] = 'Sierra Leone'
#'death_confirmed' | 'cum_confirmed'  || totals


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
For each of the following questions state clearly your assumptions and discuss your findings:
1. Describe the type and the value range of each attribute. Indicate and transform the attributes that can be Categorical.
2. Plot histograms for the travel class, embarkation port, sex and age attributes. For the latter one, use discrete decade intervals.
3. Calculate the proportion of passengers by cabin floor. Present your results in a pie chart.
4. For each travel class, calculate the proportion of the passengers that survived. Present your results in pie charts.
5. Calculate the proportion of the passengers that survived by travel class and sex. Present your results in a single histogram.
6. Create 2 equally populated age categories and calculate survival proportions by age category, travel class and sex. Present your results in a DataFrame with unique index.
Data loading
Can you write Python code for it?

data = pd.read_excel(DATA_FOLDER+'/titanic.xls')

data.head(5)

data.describe(include='all')


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
For the discrete interval, we decide to divide the ages in 9 intervals: 1 for each decades from 0 to 90 years old. Thanks to the use of panda cut function we do not have to bother about NAs.
Can you write Python code for it?

sns.countplot(x=pd.cut(data.age, [0,10,20,30,40,50,60,70,80,90], right=False), palette="Greens_d")


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
Since data is sometimes missing, we then extract the intervals for each country where we have data about the number of cases, and where we have the number of deads.
For each of those intervals:
- we get the length in days for each month in the interval.
- we create pairs for each month in the interval, where the first item is the earliest day of that month contained in the interval, and the second one, the latest one.
Can you write Python code for it?

def build_interval_by_month(start, end):
    assert(start.year == end.year) # works within the same year only
    assert(start.month < end.month ) # can't go backwards or same month

    interval = []

    # corner case #1 : start.day is not the first of the month
    interval.append([
        datetime.datetime(start.year, start.month, start.day),
        datetime.datetime(start.year, start.month, days_in_month(start.year, start.month))
    ])
    for month_idx in range(start.month + 1, end.month):
        interval.append([
            datetime.datetime(start.year, month_idx, 1),
            datetime.datetime(start.year, month_idx, days_in_month(start.year, month_idx))
        ])
    # corner case #2 : end.day in not necessary the last of the month
    interval.append([
        datetime.datetime(end.year, end.month, 1),
        datetime.datetime(end.year, end.month, end.day)
    ])

    return [[date-day_offset for date in dates]  for dates in interval]

intervals_of_interest = {}

for country in countries:
    intervals_of_interest[country] = {}
    for info in infos:
        agg_data = ebola_infos_df.loc[country].groupby(['Date_index']).agg(sum)
        agg_data_greater_zero = agg_data[agg_data[info]>0]
        start = agg_data_greater_zero.index[0]
        end = agg_data_greater_zero.index[-1]
        intervals_of_interest[country][info] = build_interval_by_month(start, end)


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
Thanks to the sort, we can see that the metadata is at the end of the list. Hence we extract it using this info.
separating data file paths (0 to 8) and metadata file path (last = 9)
Can you write Python code for it?

metadata_file = task2_files[-1]
mids_files = task2_files[:-1]


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
The values in the date column are not all in the same format, therefore we need to uniformize their format.<br/>
Then, we set the index of the Dataframe into a combination of the country and the date.
Can you write Python code for it?

# unify dates
ebola_df['Date'] = pd.to_datetime(ebola_df['Date'])

# build index
ebola_df.index = [ebola_df['Country'], ebola_df['Date']]
ebola_df.index.rename(['Country', 'Date_index'], inplace=True)

# displaying some
ebola_df[:5]


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
To train our models, we created some new features to complement the only one we have. The primary feature being the date in seconds of the record since a previously decided day offset (beginning of the year) divided by 100'000.
Can you write Python code for it?

def get_features(dates):
    X = pd.DataFrame(
        data=[date.total_seconds() for date in dates],
        index=range(len(dates)),
        columns=['Date_index']
    )

    X["Date_index"] = X["Date_index"] / 100000.
    X["log"] = X["Date_index"].apply(np.log)
    X["Date1/2"] = X["Date_index"]**(1/2.)
    X["Date^2"] = X["Date_index"]**2
    return X

# note : it also plots the predictions
def get_model(country, info):
    agg_data = ebola_infos_df.loc[country].groupby(['Date_index']).agg(sum)
    agg_data_greater_zero = agg_data[agg_data[info] > 0]

    delta = pd.DataFrame(agg_data_greater_zero[info].index)["Date_index"] - day_offset

    X = get_features(delta.tolist())

    reg1 = linear_model.RANSACRegressor(random_state=1024)
    reg2 = ensemble.ExtraTreesRegressor(random_state=1411)

    reg1.fit(X, agg_data_greater_zero[info])
    inlier_mask = reg1.inlier_mask_
    outlier_mask = np.logical_not(inlier_mask)
    x_train, x_test, y_train, y_test = train_test_split(X[inlier_mask], agg_data_greater_zero[info][inlier_mask], test_size=0.0)
    reg2.fit(x_train, y_train)

    pred_df = pd.DataFrame(reg2.predict(X)).rename(columns={0:"Prediction"})
    pred_df["Real Values (zeroes filtered)"] = agg_data_greater_zero[info].values
    pred_df["Date"] = agg_data_greater_zero.index
    pred_df.plot(x="Date", title=country+' - '+info)
    return reg2

def plot_info_per_month(intervals, plot_name):
    intervals_df = pd.DataFrame(
        data=[interval[0] for interval in intervals],
        index=[interval[1] for interval in intervals],
    )
    intervals_df.index.name = "Months"

    intervals_df.plot(kind="bar", title=plot_name, legend=False)
    return intervals_df

pred = []
final_df = pd.DataFrame()
for country in countries:
    for info in infos:
        model = get_model(country, info)
        intervals = []
        for interval in intervals_of_interest[country][info]:
            features = get_features(interval)
            pred.append(model.predict(features))
           ...


I am working on the file "01 - Pandas and Data Wrangling/Homework 1.ipynb".
The first task is:
We then merge the three Dataframes into one and replace missing values by 0.
Can you write Python code for it?

ebola_df = pd.concat(
    [
        guinea_df,
        liberia_df,
        sl_df
    ],
    axis=0
)

# replace missing values by 0
ebola_df.fillna('0', inplace=True)


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
Higher Dimensional Data Visualization
We've already seen that attributes of the plot can allow us to increase the amount of information we add; however we typically say that only 7 attributes can be embedded into a plot:

space (possibly in three dimensions)
color
size
shape
time (slider)

Pandas does provide some interesting visualizations for higher dimensional work, though, including SPLOMs, Radviz, and Parallel coordinates
Can you write Python code for it?

# Load the data
import os

DATA = os.path.join("data", "wheat", "seeds_dataset.txt")

FEATURES  = [
    "area",
    "perimeter",
    "compactness",
    "length",
    "width",
    "asymmetry",
    "groove",
    "label"
]

LABEL_MAP = {
    1: "Kama",
    2: "Rosa",
    3: "Canadian",
}

# Read the data into a DataFrame
df = pd.read_csv(DATA, sep='\s+', header=None, names=FEATURES)

# Convert class labels into text
for k,v in LABEL_MAP.items():
    df.ix[df.label == k, 'label'] = v

# Describe the dataset
print(df.describe())

# Determine the shape of the data
print("{} instances with {} features\n".format(*df.shape))

# Determine the frequency of each class
print(df.groupby('label')['label'].count())

from pandas.tools.plotting import scatter_matrix
scatter_matrix(df, alpha=0.2, figsize=(9,9), diagonal='kde')

from pandas.tools.plotting import parallel_coordinates
plt.figure(figsize=(9,9))
parallel_coordinates(df, 'label')

from pandas.tools.plotting import radviz
plt.figure(figsize=(9,9))
radviz(df, 'label')


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
Seaborn
Seaborn is a statistical visualization library, meaning that it embeds both models and data into visualizations, and is intended for visual statistical analysis. It is currently free and open source, written by Michael Waskom while he was at Stanford university.
Similar to Pandas, Seaborn wraps matplotlib, and even wraps data frames - but provides higher level abstractions for a bunch of different plots and configurations.
Can you write Python code for it?

%matplotlib inline
import os
import pandas as pd
import seaborn as sns

IRIS = os.path.join("data", "iris.csv")
data = pd.read_csv(IRIS)

sns.set_style('darkgrid')
sns.set_palette('deep')
sns.set_context('notebook')

sns.pairplot(data, hue='class', diag_kind="kde", size=3)

sns.distplot(data['sepal width'], rug=True)

sns.jointplot("petal length", "petal width", data=data, kind='reg', size=6)

sns.boxplot(x='petal length', data=data)

sns.boxplot(data=data)

sns.violinplot(data=data)

sns.swarmplot(data=data)

ax = sns.boxplot(data=data)
ax = sns.swarmplot(data=data)

sns.lmplot(x="sepal width", y="sepal length", hue="class", data=data)

sns.lmplot(x="sepal width", y="sepal length", col="class", data=data)

sns.barplot(data=data)

import numpy as np
sns.barplot(data=data, estimator=np.median)


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
The default style is currently 'ggplot' -- though this is going to be updated soon. You can set the style of the graphs, or even provide your own CSS style sheet with the plt.style.use function:
Can you write Python code for it?

# plt.style.use('fivethirtyeight')

# Note that I'm going to use temporary styling so I don't mess up the notebook!
with plt.style.context(('fivethirtyeight')):
    plt.plot(X, Y1)
    plt.plot(X, Y2)

# To see the available styles:
for style in plt.style.available: print("- {}".format(style))


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
The kind argument allows you to specify a wide variety of plots for data:

‘bar’ or ‘barh’ for bar plots
‘hist’ for histogram
‘box’ for boxplot
‘kde’ or 'density' for density plots
‘area’ for area plots
‘scatter’ for scatter plots
‘hexbin’ for hexagonal bin plots
‘pie’ for pie plots
Can you write Python code for it?

df2 = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df2.plot(kind='bar')

df2.plot.bar(stacked=True)

df2.plot(kind='area')

df2.plot.area(stacked=False)

df3 = pd.DataFrame(np.random.rand(25, 4), columns=['a', 'b', 'c', 'd'])
ax = df3.plot.scatter(x='a', y='b', color='r', label="B")
df3.plot.scatter(x='a', y='c', color='c', ax=ax, label="C")
df3.plot.scatter(x='a', y='d', color='g', ax=ax, label="D")

# Add new dimensions such as color and size based on other attributes in the data frame
# This creates a bubble plot, points sized based on the 'C' attribute and colors based on 'D'.
df3.plot.scatter(x='a', y='b', c=df3['d'], s=df3['c']*200);


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
The plt.plot function does a lot of work on our behalf: it initializes the figure, creates a subplot with axes, then computes and draws two Line2D objects. Because we're in notebook mode, the current figure is displayed in an interactive fashion and can be explored until "shutdown".
Key points:

There is a global figure that is drawn on
The objects that are returned from these functions can be directly manipulated
Outside of a notebook nothing will be rendered until plt.show

Just note, to get the same functionality in a Python script you'll have to tell pyplot to render the figure, either to an interactive backend like Tk or to an image, either raster or SVG:
plt.show()
plt.savefig('myfig.pdf')

The goal of pyplot has always been to give as simple plotting functions as possible, so that figures can be drawn without much effort. In practice, it is easy to get simple graphs plotted, but much tougher to configure them as needed.
Let's now look at all the steps it would take to draw this from scratch, which also demonstrate the configuration control you have over the figure.
Can you write Python code for it?

# Create a new figure of size 8x6 points, using 72 dots per inch
plt.figure(figsize=(8,6), dpi=72)

# Create a new subplot from a 1x1 grid
plt.subplot(111)

# Create the data to plot
X = np.linspace(-10, 10, 255)
Y1 = 2*X ** 2 + 10
Y2 = 3*X ** 2 + 50

# Plot the first quadratic using a blue color with a continuous line of 1px
plt.plot(X, Y1, color='blue', linewidth=1.0, linestyle='-')

# Plot the second quadratic using a green color with a continuous line of 1px
plt.plot(X, Y2, color='green', linewidth=1.0, linestyle='-')

# Set the X limits
plt.xlim(-10, 10)

# Set the X ticks
plt.xticks(np.linspace(-10, 10, 9, endpoint=True))

# Set the Y limits
plt.ylim(0, 350)

# Set the Y ticks
plt.yticks(np.linspace(0, 350, 5, endpoint=True))

# Save the figure to disk
plt.savefig("figures/quadratics.png")

# Create the data to plot
# This data will be referenced for the next plots below
# For Jupyter notebooks, pay attention to variables!

X = np.linspace(-10, 10, 255)
Y1 = 2*X ** 2 + 10
Y2 = 3*X ** 2 + 50


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
We can add a legend and a title:
Can you write Python code for it?

plt.figure(figsize=(9,6))
plt.plot(X, Y1, color="b", linewidth=2.5, linestyle="-", label="Y1")
plt.plot(X, Y2, color="r", linewidth=2.5, linestyle="-", label="Y2")

plt.xlim(X.min()*1.1, X.max()*1.1)
plt.ylim(Y1.min()*-1.1, Y2.max()*1.1)

plt.title("Two Quadratic Curves")
plt.legend(loc='best')


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
We can also change the x and y limits to put some space into out graph:
Can you write Python code for it?

plt.figure(figsize=(9,6))
plt.plot(X, Y1, color="b", linewidth=2.5, linestyle="-")
plt.plot(X, Y2, color="r", linewidth=2.5, linestyle="-")

plt.xlim(X.min()*1.1, X.max()*1.1)
plt.ylim(Y1.min()*-1.1, Y2.max()*1.1)


I am working on the file "01 - Visual Data Exploration.ipynb".
The first task is:
We'll look at each of these steps in detail in the next few boxes.
Colors and style
We can directly pass colors and style to each of the drawing functions in the pyplot API. The arguments for color and linestyle can either be full words, e.g. "blue" or "dashed" or they can be shortcodes, for example 'b' or '--'.
The color cycle in matplotlib determines which colors will be used for each new element drawn to the graph. The cycle is keyed to the short codes: 'bgrmyck' which stands for:
blue green red maroon yellow cyan key

A quick visualization of these colors is as follows:
Can you write Python code for it?

from matplotlib.colors import ListedColormap

colors = 'bgrmyck'
fig, ax = plt.subplots(1, 1, figsize=(7, 1))
ax.imshow(np.arange(7).reshape(1,7), cmap=ListedColormap(list(colors)), interpolation="nearest", aspect="auto")
ax.set_xticks(np.arange(7) - .5)
ax.set_yticks([-0.5,0.5])
ax.set_xticklabels([])
ax.set_yticklabels([])


I am working on the file "01 Basic Data Checking.ipynb".
The first task is:
Check Subtype Edges
11 February 2015 11:05:AM
Just to make sure that the graph is done correctly, I will make sure that the edges between two nodes have the same HA or NA (reassortant) or both (full complement).
Can you write Python code for it?

# Check full edges
for sc, sk, d in G.edges(data=True):
    if d['edge_type'] == 'full_complement':
        sc_subtype = G.node[sc]['subtype']
        sk_subtype = G.node[sk]['subtype']

        mixed = ['mixed', 'Mixed']
        if sc_subtype != sk_subtype and sc_subtype not in mixed and sk_subtype not in mixed:
            print(sc_subtype, sk_subtype, sc, sk) # nothing should be printed.

# Define a function to remove all in_edges from a node.
def remove_in_edges(G, node):
    for sc, sk in G.in_edges(node):
        if (sc, sk) in G.edges():
            G.remove_edge(sc, sk)

# Check reassortant edges
for sc, sk, d in G.edges(data=True):
    if d['edge_type'] == 'reassortant':
        sc_subtype = G.node[sc]['subtype']
        sk_subtype = G.node[sk]['subtype']

        mixed = ['mixed', 'Mixed']

        if sc_subtype not in mixed and sk_subtype not in mixed:
            sc_ha = sc_subtype.split('N')[0].split('H')[1]
            sk_ha = sk_subtype.split('N')[0].split('H')[1]

            sc_na = sc_subtype.split('N')[1]
            sk_na = sk_subtype.split('N')[1]

            if 4 in d['segments'].keys() and sc_ha != sk_ha:
                print(sc, sk, sc_subtype, sk_subtype, d['segments'][4]) # nothing should be printed
                remove_in_edges(G, sk)

            if 6 in d['segments'].keys() and sc_na != sk_na:
                print(sc, sk, sc_na, sk_na, d['segments'][6]) # nothing should be printed
                remove_in_edges(G, sk)


I am working on the file "01 Basic Data Checking.ipynb".
The first task is:
Check to make sure that there are no zero-pwi edges
Can you write Python code for it?

for sc, sk, d in G.edges(data=True):
    if d['pwi'] == 0:
        print(sc, sk, d)


I am working on the file "01 Basic Data Checking.ipynb".
The first task is:
Check to see if a particular field is missing in the nodes metadata.
Can you write Python code for it?

for n, d in G.nodes(data=True):
    if 'collection_date' not in d.keys():
        print(n)


I am working on the file "01 Coin Flip Simulation.ipynb".
The first task is:
Hypothesis Test
Power Calculation
We have estimated p_hidden by conducting a pilot experiment of one Binomial(n=50, p=p_hidden). The estimated value of p_hidden is p_hat. Let's assume p_hat is correct for a moment.
What is the necessary sample size for us to have such that we can be confident that we won't:

Falsely reject the null hypothesis (Type I Error: False Positive)? In other words, the null hypothesis should not be rejected, and we were wrong about our alternate hypothesis.
Falsely fail to reject the null hypothesis (Type II Error: False Negative)? In other words, we were correct about our hypothesis, but our data were not good enough to reject the null hypothesis.

The place where most of our intuition fails is about here. We have to be comfortable with being wrong simply by chance. The only question is how comfortable are we being wrong?
Alphabet(a)

alpha: the probability of falsely rejecting the null hypothesis.
beta: the probability of falsely failing to reject the null hypothesis.
power: 1 - beta. The probability of not falsely failing to reject the null hypothesis.

We have to define this a priori, at some probability
Can you write Python code for it?

alpha = 0.05
beta = 0.05

p_hidden_est = binom.rvs(n=1000, p=p_hidden, size=50)
plt.hist(p_hidden_est)
plt.hist(proportions)


I am working on the file "01 Coin Flip Simulation.ipynb".
The first task is:
Problem Setup
We have a coin. We don't know whether it is biased or not (i.e. the probability of one side is greater than 0.5, and the probability of the other side is less than 0.5). How do we figure this out?
We can flip the coin many times, and measure the number of times heads comes up out of all flips.
This is a power calculation: How many times would we need to flip the coin in order to conclude that it is indeed rigged?
We can compare our measured proportion of heads to a known model of randomness.
This is hypothesis testing: How probable is it that we would have observed the number of flips in our coin, assuming that the coin was unbiased?
A Biased Coin
The coin has a biased probability that is hidden. We are going to use statistical tests to estimate its probability of tossing a heads. Evaluate the cell, but don't peek at the value of p_hidden!
Can you write Python code for it?


I am working on the file "01 Hello World.ipynb".
The first task is:
Play around with what you have learned . . .
Can you write Python code for it?

import math   # now I'm just playing around, not following Socratica


I am working on the file "01 Hello World.ipynb".
The first task is:
Pythagorean Theorem:  $c = \sqrt{a^2 + b^2}$
Can you write Python code for it?

def pythag(a,b):
    return math.sqrt(a**2 + b**2)       # defined a python function

pythag(3,4)                             # used my function

math.sin(math.pi / 2)                   # playing with math.sin and math.pi

5 / 2

5 // 2


I am working on the file "01 Introducing the IPython Notebook.ipynb".
The first task is:
Cell Magics

A cell magic takes its entire cell as an argument.  Although there are a number of useful cell magics, you may find %%timeit to be useful for exploring code performance.
Can you write Python code for it?

%%timeit

import numpy as np
np.sum(np.random.rand(1000))


I am working on the file "01 Introducing the IPython Notebook.ipynb".
The first task is:
Execute Code as Python 2
Can you write Python code for it?


I am working on the file "01 Introducing the IPython Notebook.ipynb".
The first task is:
Interacting with the Command Line
IPython supports one final trick, the ability to interact directly with your  shell by using the ! operator.
Can you write Python code for it?


I am working on the file "01 Introducing the IPython Notebook.ipynb".
The first task is:
Modal editor
Starting with IPython 2.0, the IPython Notebook has a modal user interface. This means that the keyboard does different things depending on which mode the Notebook is in. There are two modes: edit mode and command mode.
Edit mode
Edit mode is indicated by a green cell border and a prompt showing in the editor area:

When a cell is in edit mode, you can type into the cell, like a normal text editor.

Enter edit mode by pressing `enter` or using the mouse to click on a cell's editor area.

While in edit mode, tab-completion works for variables the kernel knows about from executing previous cells.

Command mode
Command mode is indicated by a grey cell border:

When you are in command mode, you are able to edit the notebook as a whole, but not type into individual cells. Most importantly, in command mode, the keyboard is mapped to a set of shortcuts that let you perform notebook and cell actions efficiently. For example, if you are in command mode and you press c, you will copy the current cell - no modifier is needed.

Don't try to type into a cell in command mode; unexpected things will happen!

Enter command mode by pressing `esc` or using the mouse to click *outside* a cell's editor area.

Mouse navigation
All navigation and actions in the Notebook are available using the mouse through the menubar and toolbar, which are both above the main Notebook area:

The first idea of mouse based navigation is that cells can be selected by clicking on them. The currently selected cell gets a grey or green border depending on whether the notebook is in edit or command mode. If you click inside a cell's editor area, you will enter edit mode. If...
Can you write Python code for it?


I am working on the file "01 Introducing the IPython Notebook.ipynb".
The first task is:
Other HTML
<strong> Be Bold! </strong>
Mathematical Equations
Courtesy of MathJax, you can beautifully render mathematical expressions, both inline:
$e^{i\pi} + 1 = 0$, and displayed:
$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$
Equation Environments
You can also use a number of equation environments, such as align:
\begin{align}
  x &= 4 \
y+z &= x
\end{align}
A full list of available TeX and LaTeX commands is maintained by Dr. Carol Burns.
Other Useful MathJax Notes

inline math is demarcated by $ $, or \( \)
displayed math is demarcated by $$ $$ or \[ \]
displayed math environments can also be directly demarcated by \begin and \end
\newcommand and \def are supported, within areas MathJax processes (such as in a \[ \] block)
equation numbering is not officially supported, but it can be indirectly enabled

A Note about Notebook Security
By default, a notebook downloaded to a new computer is untrusted

HTML and Javascript in Markdown cells is now never executed
HTML and Javascript code outputs must be explicitly re-executed
Some of these restrictions can be mitigrated through shared accounts (Sage MathCloud) and secrets

More information on notebook security is in the IPython Notebook documentation
Magics
IPython kernels execute a superset of the Python language.  The extension functions, commonly referred to as magics, come in two variants.
Line Magics

A line magic looks like a command line call.  The most important of these is %matplotlib inline, which embeds all matplotlib plot output as images in the notebook itself.
Can you write Python code for it?


I am working on the file "01 MNIST example.ipynb".
The first task is:
At the beginning of the fitting procedure, neon propagates train_set through the model to set the input and output shapes of each layer. Each layer has a configure() method that determines the appropriate layer shapes, and an allocate() method to set up the needed buffers for holding the forward propagation information.
During the training, neon sends batches of the training data through the model, calling each layers’ fprop() and bprop() methods to compute the gradients and update the weights.
Using the trained model
Now that the model is successfully trained, we can use the trained model to classify a novel image, measure performance, and visualize the weights and training results.
Get outputs
Given a set of images such as those contained in the iterable test_set, we can fetch the ouput of the final model layer via
Can you write Python code for it?


I am working on the file "01 MNIST example.ipynb".
The first task is:
Callbacks
Neon provides an API for calling operations during the model fit (see Callbacks). Here we set up the default callback, which is displaying a progress bar for each epoch.
Can you write Python code for it?

from neon.callbacks.callbacks import Callbacks

callbacks = Callbacks(mlp, eval_set=valid_set)


I am working on the file "01 MNIST example.ipynb".
The first task is:
During training, neon iterates over the training examples to compute the gradients. The train_iter and valid_iter handle sending data to the model for training and validation, respectively.
For small datasets like MNIST, this step may seem trivial. However, for large datasets that cannot fit into memory (e.g. ImageNet or Sports-1M), the data has to be efficiently loaded and fed to the optimizer in batches. This requires more advanced iterators described in Loading data.
Model specification
Training a deep learning model in Neon requires
* Specifying the dataset
* Building a model from a list of layers
* Cost function
* Learning rule.
Here we guide you through each item in turn.
Initializing weights
Neon supports many ways of initializing weight matrices. In this tutorial, we initialize the weights using a Gaussian distribution with zero mean and 0.01 standard deviation.
Can you write Python code for it?

from neon.initializers import Gaussian

init_norm = Gaussian(loc=0.0, scale=0.01)


I am working on the file "01 MNIST example.ipynb".
The first task is:
Inference
Now let's download a new digit image from the web, and use our trained model to recognize the digit. We first download the image and scale it to the 28x28 pixels that our model expects.
Can you write Python code for it?

import urllib
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# download image
url = "http://datawrangling.s3.amazonaws.com/sample_digit.png"
urllib.urlretrieve(url, filename="data/digit.jpg")

# scale to 28x28 pixels
img = Image.open("data/digit.jpg")
img.thumbnail((28, 28))

digit = np.asarray(img, dtype=np.float32)[:, :, 0]

# reshape to a single feature vector
digit = digit.reshape(784, 1)

# store digit into a GPU tensor
x_new = be.zeros((28*28, 128), dtype=np.float32)
x_new[:, 0] = digit

# forward pass through the model
outputs = mlp.fprop(x_new)
outputs = outputs.get()[:, 0]

# examine the output of the model for this image
print "Model final layer was: {}".format(outputs)
print "The most probable guess is digit: {}".format(np.argmax(outputs))
plt.figure(2)
plt.imshow(img)


I am working on the file "01 MNIST example.ipynb".
The first task is:
Putting it all together
We are ready to put all the ingredients together and run our model!
Can you write Python code for it?

mlp.fit(train_set, optimizer=optimizer, num_epochs=10, cost=cost,
        callbacks=callbacks)


I am working on the file "01 MNIST example.ipynb".
The first task is:
The MNIST dataset can be found on Yann LeCunn’s website. We have included an easy function that downloads the MNIST dataset into your ~/nervana/data/ directory, loads it into memory, and returns ArrayIterator objects for the training and validation set.
Can you write Python code for it?

from neon.data import MNIST

mnist = MNIST(path='data/')
train_set = mnist.train_iter
valid_set = mnist.valid_iter


I am working on the file "01 MNIST example.ipynb".
The first task is:
We initialize the weights in each layer with the init_norm defined previously. Neon supports many other layer types (convolutional, pooling, recurrent, etc.) that will be described in subsequent examples. We then construct the model via
Can you write Python code for it?

# initialize model object
from neon.models import Model

mlp = Model(layers=layers)


I am working on the file "01 Python Refresher.ipynb".
The first task is:
2. Python "classes" are objects that have methods.
Can you write Python code for it?

from datetime import datetime

class Watch(object):
    def __init__(self):
        self.stopwatch_timer = 0

    def current_time(self):
        return str(datetime.now())

w = Watch()
# Write the funciton call that tells the current time on the watch.


I am working on the file "01 Python Refresher.ipynb".
The first task is:
3. Numbers can be represented numerically or as strings
Can you write Python code for it?

x = (3 == '3')  # change this line to make x evaluate to True
x


I am working on the file "01 Python Refresher.ipynb".
The first task is:
4. Dictionaries are really useful
Can you write Python code for it?

from datetime import datetime  # usually, I would opt to place all the import statements at the top.

useful_info = dict()

# Add the key-value pairs to the useful_info dictionary:
# - 'instructor_name':'Eric Ma'
# - 'date_today':datetime.today()


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Aha! It seems that the feature at index 15 is categorical, and that's why the rows that follow it don't have a value under NAME OF DATA ELEMENT. Just for now, let's get rid of those NAN rows.
Can you write Python code for it?

data_dict_no_nan_names = data_dict.dropna(subset=['NAME OF DATA ELEMENT'])
data_dict_no_nan_names[10:20]


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Analyze Data
First, let's see a little bit of the data
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Delete features that are meaningless
There are features that are meaningless for the problem we are trying to solve. We need to drop these features, but we need a criterion to eliminate them. The criterion that we are going to employ is to eliminate the features that are unique for every entry and don't add information to the problem, for example if we have a unique ID for every institution, this ID doesn't add information to the problem.
Also, we need to take in account that there area features that may be unique for every entry, but DOES add relevant information. For example, the tuition fees may be unique and add information.
Let's find the ratio of the number of unique values over number of examples:
Can you write Python code for it?

# Create a list to save the features that are above a certain threshold
features_with_high_ratio = []
# Create a list to save the features in all_data but not in the dict
features_not_in_dict = []

#Calculate the ratio
for feature in all_data_no_na_columns.columns.values:
    # Get the row in the dict wich have VARIABLE NAME == feature
    row_in_dict = data_dict_no_nan_names[data_dict_no_nan_names['VARIABLE NAME'] == feature]
    # Get the data type of the row
    data_type_series = row_in_dict['API data type']

    #Check if exists in the dict
    if data_type_series.size > 0:
        # Get the data type
        data_type = data_type_series.values[0]
        # float features (numeric features) are not taken in account
        if data_type == 'integer' or data_type == 'string' or data_type ==  'autocomplete':
            column = all_data_no_na_columns[feature]
            column_no_na = column.dropna()
            r = column_no_na.unique().size / column_no_na.size
            if r > 0.8:
                features_with_high_ratio.append(feature)
                print(str(feature) + ": " + str(r))
    #The feature is not in the dict
    else:
        features_not_in_dict.append(feature)

print ("\nFeatures in data but not in the dictionary:" + str(features_not_in_dict))


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Filling missing data
We need to fill the mising data. To do this we need to know if the feature is numeric or categorical. Let's use the dictionary to get that info.
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Find information about the features
Let's find more about the data
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Let's find out how many features have each data type
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Let's save the data in a file
Can you write Python code for it?

all_data_no_nan.to_csv('datasets/CollegeScorecardDataCleaned.csv', index = False)


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Let's see what are these features about:
Can you write Python code for it?

high_ratio_features = pd.DataFrame()
for feature in features_with_high_ratio:
    high_ratio_features = high_ratio_features.append(data_dict_no_nan_names[data_dict_no_nan_names['VARIABLE NAME'] == feature])
high_ratio_features


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Lets find wich features are date features
Can you write Python code for it?

features_with_date = []
for column in all_data_replaced_with_nan:
    if all_data_replaced_with_nan[column].dtypes == 'object':
        if all_data_replaced_with_nan[column].str.match('[0-9]{2}/[0-9]{2}/[0-9]{4}').any():
            features_with_date.append(column)

features_with_date

data_dict_no_nan_names[data_dict_no_nan_names['VARIABLE NAME'] == 'SEPAR_DT_MDN']


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Lets get the info of the new dict
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Now les's explore the autocomplete data type:
Can you write Python code for it?

data_dict_no_nan_names[data_dict_no_nan_names['API data type'] == 'autocomplete']


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Now we have gotten rid of the object dtype
Eliminate features with high number of NaN values
We already deleted features with that had all of their value as NaN, but now we will eliminate features with a high percentage of NaN values (more than 90%)
Can you write Python code for it?

high_nan_features = []
for feature in all_data_no_objects:
    size = all_data_no_objects[feature].size
    number_of_valid = all_data_no_objects[feature].count()
    number_of_nan =  size - number_of_valid
    ratio = number_of_nan / size
    if ratio > 0.9:
        high_nan_features.append(feature)
print (len(high_nan_features))

all_data_no_high_nan = all_data_no_objects.drop(high_nan_features, axis = 1)

all_data_no_high_nan.info()


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Now we will transfore all the object features to numeric
Can you write Python code for it?

all_data_no_objects = all_data_no_dates.copy()
for feature in all_data_no_dates:
    if all_data_no_dates[feature].dtypes == 'object':
        #Make all data numeric
        all_data_no_objects[feature] = pd.to_numeric(all_data_no_dates[feature])

all_data_no_objects.info()


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
So UNITID, OPEID, OPEID6, INSTNM, INSTURL, NPCURL and ALIAS are features that have to do with the identity of the institution, so they don't add relevant information to the problem, therfore they will be eliminated. (flag_e)
The ZIP code could be useful if it is used to group the schools to some sort of category about it's location. We are not going to to this so we are going to eliminate it as well.
Can you write Python code for it?

all_data_no_id_cols = all_data_no_na_columns.drop(['UNITID', 'OPEID', 'OPEID6', 'INSTNM', 'INSTURL', 'NPCURL', 'ALIAS', 'ZIP'], axis = 1)

all_data_no_id_cols.head()


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
So in reality, there are 1206 float features, 521 integers, and 7 string features. (For now we assume that the autocomplete type is string). This numbers differ a lot from our previus analisys, in which we had 443 float features, 13 integer features and 1287 features that are strings.
Also, we cannot asume that all features of type integer are categorical, for example the ZIP code feature is integer but is not a categorical feature.
Let's find more about the autocomplete features:
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
So this feature has to do with the state where the school is located. Let's explore the ACCREDAGENCY feature:
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
There are 7703 examples and 1743 features.
There are 443 float features that may be numeric, 13 integer features that may be categorical, and 1287 features that are strings, but may be numbers but data was not entered correctly (for example, if there was not data for a given feature, someone could have written "blank"). Given the high number of non numerical features, we need to explore them more. Luckly, there is a dictionary provided with the data, so we can explore it a little bit to learn about the data (The original file was converted do CSV)
Can you write Python code for it?

data_dict = pd.read_csv('datasets/CollegeScorecardDataDictionary.csv')

data_dict.head()

data_dict.tail()

data_dict.info()


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
We already dropped INSTURL and NPCURL. Let's explore the STABBR feature
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
We are interested primarly in the NAME OF DATA ELEMENT, VARIABLE NAME and API data type. They seem complete. Let's see howe many data types there are
Can you write Python code for it?


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
We can see that after the name of a categorical feature, there is at least one item with value NaN. Let's use this to get a list of categorical features
Can you write Python code for it?

categorical_features = []
is_null = data_dict['NAME OF DATA ELEMENT'].isnull()
for i in range(len(is_null) - 1):
    if not is_null[i] and is_null[i+1]:
        categorical_features.append(data_dict['VARIABLE NAME'][i])


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
We can see that these autocomplete features can be treated as strings.
Delete features that have all their values NaN
Can you write Python code for it?

all_data_no_na_columns = all_data.dropna(axis=1, how='all')


I am working on the file "01 StudentLoans - Cleaning.ipynb".
The first task is:
Work on the string and autocmplet data
Can you write Python code for it?

data_dict_no_nan_names[data_dict_no_nan_names['API data type'] == 'string']


I am working on the file "01 data structures and algorithms/01.01 unpack sequence into separate variables.ipynb".
The first task is:
如果变量个数和序列元素的个数不匹配，会产生一个异常。
代码示例：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.01 unpack sequence into separate variables.ipynb".
The first task is:
有时候，你可能只想解压一部分，丢弃其他的值。对于这种情况 Python 并没有提供特殊的语法。 但是你可以使用任意变量名去占位，到时候丢掉这些变量就行了。
代码示例：
Can you write Python code for it?

data = ["ACME", 50, 91.1, (2012, 12, 21)]
_, shares, price, _ = data

shares

price


I am working on the file "01 data structures and algorithms/01.02 unpack elements from iterables.ipynb".
The first task is:
值得注意的是上面解压出的 phone_numbers 变量永远都是列表类型，不管解压的电话号码数量是多少(包括 0 个)。 所以，任何使用到 phone_numbers 变量的代码就不需要做多余的类型检查去确认它是否是列表类型了。
星号表达式也能用在列表的开始部分。比如，你有一个公司前 8 个月销售数据的序列， 但是你想看下最近一个月数据和前面 7 个月的平均值的对比。你可以这样做：
``` python
*trailing_qtrs, current_qtr = sales_record
trailing_avg = sum(trailing_qtrs) / len(trailing_qtrs)
return avg_comparison(trailing_avg, current_qtr)
```
下面是在 Python 解释器中执行的结果：
Can you write Python code for it?

*trailing, current = [10, 8, 7, 1, 9, 5, 10, 3]
trailing

current


I am working on the file "01 data structures and algorithms/01.02 unpack elements from iterables.ipynb".
The first task is:
在很多函数式语言中，星号解压语法跟列表处理有许多相似之处。比如，如果你有一个列表， 你可以很容易的将它分割成前后两部分：
Can you write Python code for it?

items = [1, 10, 7, 4, 5, 9]
head, *tail = items

head

tail


I am working on the file "01 data structures and algorithms/01.03 keep last n items.ipynb".
The first task is:
尽管你也可以手动在一个列表上实现这一的操作(比如增加、删除等等)。但是这里的队列方案会更加优雅并且运行得更快些。
更一般的， deque 类可以被用在任何你只需要一个简单队列数据结构的场合。 如果你不设置最大队列大小，那么就会得到一个无限大小队列，你可以在队列的两端执行添加和弹出元素的操作。
代码示例：
Can you write Python code for it?

q = deque()
q.append(1)
q.append(2)
q.append(3)

q

q.appendleft(4)
q

q.pop()

q

q.popleft()


I am working on the file "01 data structures and algorithms/01.04 find largest or smallest n items.ipynb".
The first task is:
上面代码在对每个元素进行对比的时候，会以 price 的值进行比较。
讨论
如果你想在一个集合中查找最小或最大的 N 个元素，并且 N 小于集合元素数量，那么这些函数提供了很好的性能。 因为在底层实现里面，首先会先将集合数据进行堆排序后放入一个列表中：
Can you write Python code for it?

nums = [1, 8, 2, 23, 7, -4, 18, 23, 42, 37, 2]

import heapq
heapq.heapify(nums)
nums


I am working on the file "01 data structures and algorithms/01.04 find largest or smallest n items.ipynb".
The first task is:
堆数据结构最重要的特征是 heap[0] 永远是最小的元素。并且剩余的元素可以很容易的通过调用 heapq.heappop() 方法得到， 该方法会先将第一个元素弹出来，然后用下一个最小的元素来取代被弹出元素(这种操作时间复杂度仅仅是 O(log N) ，N 是堆大小)。 比如，如果想要查找最小的 3 个元素，你可以这样做：
Can you write Python code for it?

heapq.heappop(nums)

heapq.heappop(nums)

heapq.heappop(nums)


I am working on the file "01 data structures and algorithms/01.05 implement a priority queue.ipynb".
The first task is:
下面是它的使用方式：
Can you write Python code for it?

class Item():
    def __init__(self, name):
        self.name = name

    def __repr__(self):
        return 'Item({!r})'.format(self.name)

q = PriorityQuene()
q.push(Item("foo"), 1)
q.push(Item("bar"), 5)
q.push(Item("spam"), 4)
q.push(Item("grok"), 1)

q.pop()

q.pop()

q.pop()

q.pop()


I am working on the file "01 data structures and algorithms/01.05 implement a priority queue.ipynb".
The first task is:
仔细观察可以发现，第一个 pop() 操作返回优先级最高的元素。 另外注意到如果两个有着相同优先级的元素( foo 和 grok )，pop 操作按照它们被插入到队列的顺序返回的。
讨论
这一小节我们主要关注 heapq 模块的使用。 函数 heapq.heappush() 和 heapq.heappop() 分别在队列 queue 上插入和删除第一个元素， 并且队列 _queue 保证第一个元素拥有最小优先级( 1.4 节已经讨论过这个问题)。 heappop() 函数总是返回”最小的”的元素，这就是保证队列 pop 操作返回正确元素的关键。 另外，由于 push 和 pop 操作时间复杂度为 O(log N) ，其中 N 是堆的大小，因此就算是 N 很大的时候它们运行速度也依旧很快。
在上面代码中，队列包含了一个 (-priority, index, item) 的元组。 优先级为负数的目的是使得元素按照优先级从高到低排序。 这个跟普通的按优先级从低到高排序的堆排序恰巧相反。
index 变量的作用是保证同等优先级元素的正确排序。 通过保存一个不断增加的 index 下标变量，可以确保元素按照它们插入的顺序排序。 而且， index 变量也在相同优先级元素比较的时候起到重要作用。
为了阐明这些，先假定 Item 实例是不支持排序的：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.05 implement a priority queue.ipynb".
The first task is:
如果你使用元组 (priority, item) ，只要两个元素的优先级不同就能比较。 但是如果两个元素优先级一样的话，那么比较操作就会跟之前一样出错：
Can you write Python code for it?

a = (1, Item("foo"))
b = (5, Item("bar"))
a < b

c = (1, Item("grok"))
a < c


I am working on the file "01 data structures and algorithms/01.05 implement a priority queue.ipynb".
The first task is:
通过引入另外的 index 变量组成三元组 (priority, index, item) ，就能很好的避免上面的错误， 因为不可能有两个元素有相同的 index 值。 Python 在做元组比较时候，如果前面的比较以及可以确定结果了， 后面的比较操作就不会发生了：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.06 map keys to multiple values in dict.ipynb".
The first task is:
选择使用列表还是集合取决于你的实际需求。如果你想保持元素的插入顺序就应该使用列表， 如果想去掉重复元素就使用集合（并且不关心元素的顺序问题）。
你可以很方便的使用 collections 模块中的 defaultdict 来构造这样的字典。 defaultdict 的一个特征是它会自动初始化每个 key 刚开始对应的值，所以你只需要关注添加元素操作了。比如：
Can you write Python code for it?

from collections import defaultdict

d = defaultdict(list)
d["a"].append(1)
d["a"].append(2)
d["b"].append(4)

d = defaultdict(set)
d["a"].add(1)
d["a"].add(2)
d["b"].add(4)


I am working on the file "01 data structures and algorithms/01.07 keep dict in order.ipynb".
The first task is:
当你想要构建一个将来需要序列化或编码成其他格式的映射的时候， OrderedDict 是非常有用的。 比如，你想精确控制以 JSON 编码后字段的顺序，你可以先使用 OrderedDict 来构建这样的数据：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
不幸的是，通常这个结果同样也不是你想要的。 你可能还想要知道对应的键的信息(比如那种股票价格是最低的？)。
你可以在 min() 和 max() 函数中提供 key 函数参数来获取最小值或最大值对应的键的信息。比如：
Can you write Python code for it?

min(prices, key = lambda k:prices[k])

max(prices, key = lambda k:prices[k])


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
为了对字典值执行计算操作，通常需要使用 zip() 函数先将键和值反转过来。 比如，下面是查找最小和最大股票价格和股票值的代码：
Can you write Python code for it?

min_price = min(zip(prices.values(), prices.keys()))
print("min_price is:", min_price)

max_price = max(zip(prices.values(), prices.keys()))
print("max_price is:", max_price)


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
但是，如果还想要得到最小值，你又得执行一次查找操作。比如：
Can you write Python code for it?

min_value = prices[min(prices, key = lambda k:prices[k])]
min_value


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
前面的 zip() 函数方案通过将字典”反转”为(值，键)元组序列来解决了上述问题。 当比较两个元组的时候，值会先进行比较，然后才是键。 这样的话你就能通过一条简单的语句就能很轻松的实现在字典上的求最值和排序操作了。
需要注意的是在计算操作中使用到了(值，键)对。当多个实体拥有相同的值的时候，键会决定返回结果。 比如，在执行 min() 和 max() 操作的时候，如果恰巧最小或最大值有重复的，那么拥有最小或最大键的实体会返回：
Can you write Python code for it?

prices = {"AAA" : 45.23, "ZZZ" : 45.23}
min(zip(prices.values(), prices.keys()))

max(zip(prices.values(), prices.keys()))


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
执行这些计算的时候，需要注意的是 zip() 函数创建的是一个只能访问一次的迭代器。 比如，下面的代码就会产生错误：
Can you write Python code for it?

price_and_names = zip(prices.values(), prices.keys())
print(min(price_and_names)) # OK

print(max(price_and_names)) # ValueError


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
类似的，可以使用 zip() 和 sorted() 函数来排列字典数据：
Can you write Python code for it?

prices_sorted = sorted(zip(prices.values(), prices.keys()))
print("prices_sorted is:", prices_sorted)


I am working on the file "01 data structures and algorithms/01.08 calculating with dict.ipynb".
The first task is:
讨论
如果你在一个字典上执行普通的数学运算，你会发现它们仅仅作用于键，而不是值。比如：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.09 find commonalities in dicts.ipynb".
The first task is:
为了寻找两个字典的相同点，可以简单的在两字典的 keys() 或者 items() 方法返回结果上执行集合操作。比如：
Can you write Python code for it?

# Find keys in common
a.keys() & b.keys()

# Find keys in a that are not in b
a.keys() - b.keys()

# Find (key,value) pairs in common
a.items() & b.items()


I am working on the file "01 data structures and algorithms/01.09 find commonalities in dicts.ipynb".
The first task is:
这些操作也可以用于修改或者过滤字典元素。 比如，假如你想以现有字典构造一个排除几个指定键的新字典。 下面利用字典推导来实现这样的需求：
Can you write Python code for it?

# Make a new dictionary with certain keys removed
c = {key:a[key] for key in a.keys() - {"z", "w"}}

c


I am working on the file "01 data structures and algorithms/01.10 remove duplicates from seq order.ipynb".
The first task is:
下面是使用上述函数的例子：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.10 remove duplicates from seq order.ipynb".
The first task is:
这里的 key 参数指定了一个函数，将序列元素转换成 hashable 类型。下面是它的用法示例：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.11 naming slice.ipynb".
The first task is:
与其那样写，为什么不想这样命名切片呢：
Can you write Python code for it?

SHARES = slice(20, 23)
PRICE = slice(31, 37)
const = int(record[SHARES]) * float(record[PRICE])


I am working on the file "01 data structures and algorithms/01.11 naming slice.ipynb".
The first task is:
另外，你还能通过调用切片的 indices(size) 方法将它映射到一个确定大小的序列上， 这个方法返回一个三元组 (start, stop, step) ，所有值都会被合适的缩小以满足边界限制， 从而使用的时候避免出现 IndexError 异常。比如：
Can you write Python code for it?

s = "Hello World"
a.indices(len(s))

for i in range(*a.indices(len(s))):
    print(s[i])


I am working on the file "01 data structures and algorithms/01.11 naming slice.ipynb".
The first task is:
如果你有一个切片对象 a，你可以分别调用它的 a.start , a.stop , a.step 属性来获取更多的信息。比如：
Can you write Python code for it?

a = slice(5, 50, 2)
a.start

a.stop

a.step


I am working on the file "01 data structures and algorithms/01.11 naming slice.ipynb".
The first task is:
第二种版本中，你避免了大量无法理解的硬编码下标，使得你的代码更加清晰可读了。
讨论
一般来讲，代码中如果出现大量的硬编码下标值会使得可读性和可维护性大大降低。 比如，如果你回过来看看一年前你写的代码，你会摸着脑袋想那时候自己到底想干嘛啊。 这里的解决方案是一个很简单的方法让你更加清晰的表达代码到底要做什么。
内置的 slice() 函数创建了一个切片对象，可以被用在任何切片允许使用的地方。比如：
Can you write Python code for it?

items = [0, 1, 2, 3, 4, 5, 6]
a = slice(2, 4)
items[2:4]

items[a]

items[a] = [10, 11]
items

del items[a]
items


I am working on the file "01 data structures and algorithms/01.12 determine most freqently items in seq.ipynb".
The first task is:
Counter 实例一个鲜为人知的特性是它们可以很容易的跟数学运算操作相结合。比如：
Can you write Python code for it?

a = Counter(words)
a

b = Counter(morewords)
b

c = a + b
c

d = a - b
d


I am working on the file "01 data structures and algorithms/01.12 determine most freqently items in seq.ipynb".
The first task is:
如果你想手动增加计数，可以简单的用加法：
Can you write Python code for it?

morewords = ['why','are','you','not','looking','in','my','eyes']
for word in morewords:
    word_counts[word] += 1

word_counts["eyes"]


I am working on the file "01 data structures and algorithms/01.12 determine most freqently items in seq.ipynb".
The first task is:
或者你可以使用 update() 方法：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.13 sort list of dicts by key.ipynb".
The first task is:
代码的输出如上：
itemgetter() 函数也支持多个 keys ，比如下面的代码:
Can you write Python code for it?

rows_by_lfname = sorted(rows, key = itemgetter("lname", "fname"))
print(rows_by_lfname)


I am working on the file "01 data structures and algorithms/01.13 sort list of dicts by key.ipynb".
The first task is:
根据任意的字典字段来排序输入结果行是很容易实现的，代码示例：
Can you write Python code for it?

from operator import itemgetter
rows_by_fname = sorted(rows, key = itemgetter("fname"))
print(rows_by_fname)

rows_by_uid = sorted(rows, key = itemgetter("uid"))
print(rows_by_uid)


I am working on the file "01 data structures and algorithms/01.14 sort objects without compare support.ipynb".
The first task is:
另外一种方式是使用 operator.attrgetter() 来代替 lambda 函数：
Can you write Python code for it?

from operator import attrgetter
users = [User(23), User(3), User(99)]
sorted(users, key = attrgetter("user_id"))


I am working on the file "01 data structures and algorithms/01.14 sort objects without compare support.ipynb".
The first task is:
讨论
选择使用 lambda 函数或者是 attrgetter() 可能取决于个人喜好。 但是， attrgetter() 函数通常会运行的快点，并且还能同时允许多个字段进行比较。 这个跟 operator.itemgetter() 函数作用于字典类型很类似(参考 1.13 小节)。 例如，如果 User 实例还有一个 first_name 和 last_name 属性，那么可以向下面这样排序：
``` python
by_name = sorted(users, key = attrgetter("last_name", "first_name"))
```
同样需要注意的是，这一小节用到的技术同样适用于像 min() 和 max() 之类的函数。比如：
Can you write Python code for it?

min(users, key = attrgetter("user_id"))

max(users, key = attrgetter("user_id"))


I am working on the file "01 data structures and algorithms/01.16 filter sequence elements.ipynb".
The first task is:
filter() 函数创建了一个迭代器，因此如果你想得到一个列表的话，就得像示例那样使用 list() 去转换。
讨论
列表推导和生成器表达式通常情况下是过滤数据最简单的方式。 其实它们还能在过滤的时候转换数据。比如：
Can you write Python code for it?

mylist = [1, 4, -5, 10, -7, 2, 3, -1]

import math
[math.sqrt(n) for n in mylist if n > 0]


I am working on the file "01 data structures and algorithms/01.16 filter sequence elements.ipynb".
The first task is:
现在你想将那些对应 count 值大于 5 的地址全部输出，那么你可以这样做：
Can you write Python code for it?

from itertools import compress
more5 = [n > 5 for n in counts]
more5

list(compress(addresses, more5))


I am working on the file "01 data structures and algorithms/01.16 filter sequence elements.ipynb".
The first task is:
过滤操作的一个变种就是将不符合条件的值用新的值代替，而不是丢弃它们。 比如，在一列数据中你可能不仅想找到正数，而且还想将不是正数的数替换成指定的数。 通过将过滤条件放到条件表达式中去，可以很容易的解决这个问题，就像这样：
Can you write Python code for it?

clip_neg = [n if n > 0 else 0 for n in mylist]
clip_neg

clip_pos = [n if n < 0 else 0 for n in mylist]
clip_pos


I am working on the file "01 data structures and algorithms/01.17 extract subset of dict.ipynb".
The first task is:
讨论
大多数情况下字典推导能做到的，通过创建一个元组序列然后把它传给 dict() 函数也能实现。比如：
Can you write Python code for it?

p1 = dict((key, value) for key, value in prices.items() if value > 200)
p1


I am working on the file "01 data structures and algorithms/01.18 map names to sequence elements.ipynb".
The first task is:
_replace() 方法还有一个很有用的特性就是当你的命名元组拥有可选或者缺失字段时候， 它是一个非常方便的填充数据的方法。 你可以先创建一个包含缺省值的原型元组，然后使用 _replace() 方法创建新的值被更新过的实例。比如：
Can you write Python code for it?

from collections import namedtuple

Stock = namedtuple("Stock", ["name", "shares", "price", "date", "time"])

# Create a prototype instance
stock_prototype = Stock("", 0, 0.0, None, None)

# Function to convert a dictionary to a Stock
def dict_to_stock(s):
    return stock_prototype._replace(**s)


I am working on the file "01 data structures and algorithms/01.18 map names to sequence elements.ipynb".
The first task is:
下面是它的使用方法：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.18 map names to sequence elements.ipynb".
The first task is:
如果你真的需要改变然后的属性，那么可以使用命名元组实例的 _replace() 方法， 它会创建一个全新的命名元组并将对应的字段用新的值取代。比如：
Can you write Python code for it?


I am working on the file "01 data structures and algorithms/01.18 map names to sequence elements.ipynb".
The first task is:
讨论
命名元组另一个用途就是作为字典的替代，因为字典存储需要更多的内存空间。 如果你需要构建一个非常大的包含字典的数据结构，那么使用命名元组会更加高效。 但是需要注意的是，不像字典那样，一个命名元组是不可更改的。比如：
Can you write Python code for it?

s = Stock("ACME", 100, 123.45)
s

s.shares

s.shares = 75


I am working on the file "01 data structures and algorithms/01.19 transform and reduce data same time.ipynb".
The first task is:
讨论
上面的示例向你演示了当生成器表达式作为一个单独参数传递给函数时候的巧妙语法(你并不需要多加一个括号)。 比如，下面这些语句是等效的：
Can you write Python code for it?

s = sum((x * x for x in nums)) # 显示的传递一个生成器表达式对象
s = sum(x * x for x in nums) # 更加优雅的实现方式，省略了括号


I am working on the file "01 data structures and algorithms/01.20 combine multiple map to single map.ipynb".
The first task is:
ChainMap 对于编程语言中的作用范围变量(比如 globals , locals 等)是非常有用的。 事实上，有一些方法可以使它变得简单：
Can you write Python code for it?

values = ChainMap()
values["x"] = 1

# Add a new mapping
values = values.new_child()

values["x"] = 2

# Add a new mapping
values = values.new_child()
values["x"] = 3

values

values["x"]

# Discard last mapping
values = values.parents
values["x"]

values = values.parents
values["x"]

values


I am working on the file "01 data structures and algorithms/01.20 combine multiple map to single map.ipynb".
The first task is:
现在假设你必须在两个字典中执行查找操作(比如先从 a 中找，如果找不到再在 b 中找)。 一个非常简单的解决方案就是使用 collections 模块中的 ChainMap 类。比如：
Can you write Python code for it?

from collections import ChainMap
c = ChainMap(a, b)
print(c["x"])

print(c["y"])

print(c["z"])


I am working on the file "01 data structures and algorithms/01.20 combine multiple map to single map.ipynb".
The first task is:
讨论
一个 ChainMap 接受多个字典并将它们在逻辑上变为一个字典。 然后，这些字典并不是真的合并在一起了， ChainMap 类只是在内部创建了一个容纳这些字典的列表 并重新定义了一些常见的字典操作来遍历这个列表。大部分字典操作都是可以正常使用的，比如：
Can you write Python code for it?

len(c)

list(c.keys())

list(c.values())


I am working on the file "01 data structures and algorithms/01.20 combine multiple map to single map.ipynb".
The first task is:
这样也能行得通，但是它需要你创建一个完全不同的字典对象(或者是破坏现有字典结构)。 同时，如果原字典做了更新，这种改变不会反应到新的合并字典中去。比如：
Can you write Python code for it?


I am working on the file "01+Beautifulsoup.ipynb".
The first task is:
Definieren wir eine Variable damit.
Can you write Python code for it?

table = arrivals_soup.find('div', {'class': 'cblock modules-flights-flightlist modules-flights'})

type(table)

table.text


I am working on the file "01+Beautifulsoup.ipynb".
The first task is:
Developer Tools im Browser
Suchen wir die Daten, der uns wirklich interessiert. Beginnen bei den Developers Tools.
Find und Findall
Can you write Python code for it?

arrivals_soup.find('tbody')

arrivals_soup.find('div', {'class': 'cblock modules-flights-flightlist modules-flights'})


I am working on the file "01+Beautifulsoup.ipynb".
The first task is:
Gehen wir die zweite Liste durch?
Can you write Python code for it?

fluege1 = [] #das ändern

for elem in row1:
    ga_zeit = elem.find('td', {'class': 'first'}).text
    herkunft = elem.find('td', {'class': 'first'}).find_next_sibling('td').text
    airline = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .text.replace('\n', '').replace('\t','')
    nummer = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td').text \
                                      .replace('\t','').replace('\n', '')
    a_zeit = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td').text \
                                      .replace('\t','').replace('\n', '')
    typ = elem.find('td', {'class': 'last'}).text

    mini_dict = {'Geplante Ankunft': ga_zeit,
                 'Ankunft': a_zeit,
                 'Ankunft aus': herkunft,
                 'Flugnummer': nummer,
                 'Passagier/Cargo': typ}

    fluege1.append(mini_dict) #und hier


I am working on the file "01+Beautifulsoup.ipynb".
The first task is:
Packen wir das alles in einen Loop?
Can you write Python code for it?

fluege = []

for elem in row0:
    ga_zeit = elem.find('td', {'class': 'first'}).text
    herkunft = elem.find('td', {'class': 'first'}).find_next_sibling('td').text
    airline = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .text.replace('\n', '').replace('\t','')
    nummer = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td').text \
                                      .replace('\t','').replace('\n', '')
    a_zeit = elem.find('td', {'class': 'first'}).find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td') \
                                      .find_next_sibling('td').text \
                                      .replace('\t','').replace('\n', '')
    typ = elem.find('td', {'class': 'last'}).text

    mini_dict = {'Geplante Ankunft': ga_zeit,
                 'Ankunft': a_zeit,
                 'Ankunft aus': herkunft,
                 'Flugnummer': nummer,
                 'Passagier/Cargo': typ}

    fluege.append(mini_dict)


I am working on the file "01+Beautifulsoup.ipynb".
The first task is:
Verbinden wir beide Listen
Can you write Python code for it?


I am working on the file "01--intro-py.ipynb".
The first task is:
By the way, here's a neat trick to output typeset mathematics (via $\LaTeX$), taken from this StackOverflow post.

Try hitting "Play" to see what it outputs. Note: This trick only works inside an IPython notebook.
Can you write Python code for it?

from IPython.display import Math
Math("\\sum_{i=1}^{%d} i = %d" % (n, s_computed))


I am working on the file "01--intro-py.ipynb".
The first task is:
Concept 2: Lists
Beyond scalar values, you can also operate on collections of values.
There are many kinds of collections in Python. The simplest is the basic list, which is simply a sequence of values.
You can define small lists directly using a special kind of expression, in which the values (themselves expressions) appear as a comma-delimited sequence within square brackets:

A list of integers: [3, 1, 4, 1, 5, 9]
A list of strings: ["my", "momma", "told", "me", "you", "better", "shop", "around"]
A mixed list: [72, "hello", square, square (5)]

Q: Try evaluating the above expressions.
Q: What value does the following expression produce?
Can you write Python code for it?


I am working on the file "01--intro-py.ipynb".
The first task is:
Concept 3: Sum the sequence
Compute the sum of the sequence X created above.
Can you write Python code for it?

s_analytical = n * (n+1) / 2
s_computed = sum (X)
assert s_computed == s_analytical
print ("Sum of {1, 2, ..., %d} = %d" % (n, s_computed))


I am working on the file "01--intro-py.ipynb".
The first task is:
Concept 4: Correctness and efficiency
It's probably obvious that one of your main goals is to write correct programs. But another important goal, especially if you want to scale to large data sets, is efficiency.
Often, efficiency is possible through the use of the right patterns or idioms. Just like idioms in natural language, programming language idioms are shortcuts that have a particular meaning.
Consider two different ways to carry out the sum of n elements: using the sum() function, as above, compared to using an explicit loop. You should observe that one is much faster than the other. Why?
Can you write Python code for it?

import time

t_start = time.time () # Start timer, which returns a time stamp in seconds
s_sum = sum (X)
t_sum = time.time () - t_start
assert s_sum+1 == s_analytical

t_start = time.time ()
s_loop = 0
for x_i in X:
    s_loop = s_loop + x_i
t_loop = time.time () - t_start
assert s_loop == s_analytical

print ("Time using sum: %g seconds" % t_sum)
print ("Time using loop: %g seconds" % t_loop)