I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
In fig 8.3, observe that Balance must exhaust the budget of at least one of the advertisers, say $A_2$.
If the revenue of Balance is at least $3/4$th the revenue of the optimum algorithm, we need to show $y \geq x$.
There are two cases that the queries that are assigned to $A_1$ by the optimum algorithm are assigned to $A_1$ or $A_2$ by Balance:

Suppose at least half of these queries are assigned by Balance to $A_1$. Then $y \geq B/2$, so surely $y \geq x$.

Suppose more than half of these queries are assigned by Balance to $A_2$.
   Why dose Balance assgin them to $A_2$, instead of $A_1$ like the optimum algorithm? Because $A_2$ must have had at least as great a budget available as $A_1$.
   Since more than half of the $B$ queries that the optimum algorithm assigns to $A_1$ are assigned to $A_2$ by Balance, so the remaining budget of $A_2$ was less than $B/2$.
   Thus, the remaining budget of $A_1$ was laso less than $B/2$. We know that $x < B/2$.
   It follows that $y > x$, since $x + y = B$.

We conclude that $y \geq x$ in either case, so the competitve ratio of the Balance Algorithm is $3/4$.
8.4.6 The Balance Algorithm with Many Bidders
The worst case for Balance is as follows:

There are $N$ advertisers, $A_1, A_2, \dotsc, A_N$.

Each advertiser has a budge $B = N!$.

There are $N$ queries $q_1, q_2, \dotsc, q_N$.

Advertiser $A_i$ bids on queries $q_1, q_2, \dotsc, q_i$ and no other queries.

The query sequence consists of $N$ rounds. The $i$th round consists of $B$ occurrences of query $q_i$ and nothing else.

The optimum off-line algorithm assigns the $B$ queries $q_i$ in the $i$th round to $A_i$ for all $i$. Its total revenue is $NB$.
However, for the Balance Algorithm,
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
Lower-numbered advertisers cannot bid at first, and the budgets of hte higher-numbered advertisers will be exhausted eventually. All advertisers will end at $j$ round where
$$B(\frac{1}{N} + \frac{1}{N-1} + \dotsb + \frac{1}{N-j+1}) \geq B$$
Solving this equation for $j$, we get $$j = N(1 - \frac{1}{e})$$
Thus, the approxiamte revenue obtained by the Balance Algorithm is $BN(1 - \frac{1}{e})$. Therefore, the competitive ration is $1 - \frac{1}{e}$.
8.4.7 The Generalized Balance Algorithm
With arbitrary bids and budgets Balance fails to weight the sizes of the bids properly. In order to make Balance work in more general situations, we need to make two modifications:

bias the choice of ad in favor of higher bids.

use the fraction of the budgets remaining.

We calculate $\Phi_i = x_i (1 - e^{-f_i})$, where $x_i$ is the bid of $A_i$ for the query, and $f_i$ is the fraction fo the unspent budget of $A_i$. The algorithm assignes the query to $\text{argmax} \Phi_i$.
The competitive ration is $1 - \frac{1}{e}$.
8.4.8 Final Observations About the Adwords Problem

click-through rate.
   multiply the bid by the click-through rate when computing the $\Phi_i$'s.

historical frequency of queries.
   If $A_i$ has a budget sufficiently small, then we maintain $\Phi_i$ as long as we can expect that there will be enough queries remaining in the month to give $A_i$ its full budget of ads.

Exercises  for Section 8.4
#maybe
8.5 Adwords Implementation
8.5.1 Matching Bids and Search Queries
If a search query occurs having exactly that set of words in some order, then the bid is said to match the query, and it becomes a candidate for selection.
Storing all sets of words representing a bid in lexicographic(alphabetic) order, and use it as the hash-key for the bid.
Can you write Python code for it?

class advertiser:
    def __init__(self, name, bids):
        self.name = name
        self.bids = bids
    def get_info(self):
        return self.name, self.bids

advertisers = [
    advertiser('David', ['Google', 'email', 'product']),
    advertiser('Jim', ['SNS', 'Facebook', 'product']),
    advertiser('Sun', ['product', 'Google', 'email']),
]

bids_hash_table = dict()
for ad in advertisers:
    v, k = ad.get_info()
    k = [x.lower() for x in k]
    k = ' '.join(sorted(k))
    if k not in bids_hash_table:
        bids_hash_table[k] = [v]
    else:
        bids_hash_table[k].append(v)

bids_hash_table

queries = [
    ('EMAIL', 'google', 'Product'),
    ('google', 'facebook', 'Product')
]

def handle_query(query):
    q = [x.lower() for x in query]
    q = ' '.join(sorted(q))
    print(q)
    try:
        print('Found: {}'.format(bids_hash_table[q]))
    except KeyError:
        print('No bids')

for query in queries:
    handle_query(query)
    print()


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
The bids are stored in a hash-table, whose hash key is the first word of the bid, in the order explained above.
There is another hash table, whose job is to contain copies of those bids that have been partially matched. If the status is $i$, then the hash-key for this hash table is the $(i + 1)$st word.
To process a document:
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Clustering/note.ipynb".
The first task is:
7.3.4 The Algorithm of Bradley, Fayyad, and Reina
designed to cluster data in a high-dimensional Euclidean space.
strong assumption about the shape of clusters:

normally distributed about a centriod.

the dimensions must be independent.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Clustering/note.ipynb".
The first task is:
The points of the data file are read in chunks.
The main-memory data other than the chunk consists of three types of objects:

The Discard Set:
   simple summaries of the clusters themselves.

The Compressed Set:
   summaries of the points that have been found close to one another, but not close to any cluster. Each represented set of points is called a minicluster.

The Retained Set:
   remaining points.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Dimensionality_Reduction/note.ipynb".
The first task is:
11 Dimensionality Reduction
11.1 Eigenvalues and Eigenvectors of Symmetric Matrices
11.1.1 Definitions
$M e = \lambda e$, where $\lambda$ is an eigenvalue and $e$ is the corresponding eigenvector.
to make the eigenvector unique, we require that:

every eigenvector is a unit vector.

the first nonzero component of an eigenvector is positive.

11.1.2 Computing Eigenvalues and Eigenvectors
To find principal eigenvector, we use power iteration method:
start with any unit vector $v$, and compute $M^i v$ iteratively until it converges. When $M$ is a stochastic matrix, the limiting vector is the principal eigenvector, and its corresponding eigenvalue is 1.
Another method $O(n^3)$: algebra solution.
11.1.3 Finding Eigenpairs by Power Iteration
idea:
Start by computing the pricipal eigenvector, then modify the matrix to, in effect, remove the principal eigenvector.

To find the pricipal eigenpair
power iteration:
Start with any nonzero vector $x_0$ and then iterate:
$$x_{k+1} := \frac{M x_k}{\| M x_k\|}$$
$x$ is the pricipal eigenvector when it reaches convergence, and $\lambda_1 = x^T M x$.
Attention: Since power iteration will introduce small errors, inaccuracies accumulate when we try to compute all eigenpairs.

To find the second eigenpair
we create $M^* = M - \lambda_1 x x^T$, then use power iteration again.
proof: the second eigenpair of $M^*$ is also that of $M$.
略

11.1.4 The Matrix of Eigenvectors
the eigenvectors of a symmetric matrix are orthonormal
11.1.5 Exercises for Section 11.1
11.2 Pricipal-Component Analysis
idea:
treat the set of tuples as a matrix $M$ and find the eigenvectors for $M M^T$ or $M^T M$.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Dimensionality_Reduction/note.ipynb".
The first task is:
11.2.2 Using Eigenvectors for Dimensionality Reduction
Since $M^T M e = \lambda e = e \lambda$,
Let $E$ be the matrix whose columns are the eigenvectors, ordered as largest eigenvalue first. Define the matrix $L$ to have the eigenvalues of $M^T M$ along the diagonal, largest first, and 0's in all other entries.
$M^T M E = E L$
Let $E_k$ be the first $k$ columns of $E$. Then $M E_k$ is a $k$-dimensional representation of $M$.
11.2.3 The Matrix of Distances
\begin{align}
    M^T M e &= \lambda e \
    M M^T (M e) &= M \lambda e = \lambda (M e)
\end{align}
the eigenvalues of $M M^T$ are the eigenvalues of $M^T M$ plus additional 0's, and their eigenvectors are shared.
11.2.4 Exercises for Section 11.2
11.3 Singular-Value Decomposition
11.3.1 Definition of SVD
Let $M$ be an $m \times n$ matrix, and let the rank of $M$ be $r$.
$$M = U \Sigma V^T$$

$U$ is an $m \times r$ column-orthnormal matrix (each of its columns is a unit vector and the dot product of any two columns is 0).

$V$ is an $n \times r$ column-orthnormal maxtrix.

$\Sigma$ is a diagonal matrix. The elements of $\Sigma$ are called the singular values of $M$.
Can you write Python code for it?

show_image('fig11_5.png')

show_image('fig11_7.png', figsize=(8, 10))


I am working on the file "Mining_of_Massive_Datasets/Dimensionality_Reduction/note.ipynb".
The first task is:
11.3.2 Interpretation of SVD
viewing the $r$ columns of $U$, $\Sigma$, and $V$ as representing concepts that are hidden in the original matrix $M$.
In Fig 11.7:

concepts: "science fiction" and "romance".

$U$ connects peopel to concepts.

$V$ connects movies to concepts.

$\Sigma$ give the strength of each of the concepts.

11.3.3 Dimensionality Reduction Using SVD
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Dimensionality_Reduction/note.ipynb".
The first task is:
Eliminating Duplicate Rows and Columns
it is possible that a single row or column is selected more than once, how to deal with it?

let it go.

or, merge same rows and/or columns
  $W$ will not be a sqaure matrix, then we need transpose the result to get $\Sigma^+$.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
#exercise 3.2.3
k-shingles 最大数量：所有提取的 shingle 均不同
故从左向右依次取片，有：
the largest number = n - (k - 1) = n - k + 1
3.3 Similarity-Preserving Summaries of Sets
Matrix Representation of Sets:
(r,c) = 1: the element for row r is a member of the set for column c.
(r,c) = 0: otherwise

matrices are almost always sparse
Can you write Python code for it?

#Example 3.6

logger.setLevel(logging.WARNING)

my_database = ['a', 'b', 'c', 'd', 'e']
my_features_dict = {
    'S1':['a', 'd'],
    'S2':['c'],
    'S3':['b', 'd', 'e'],
    'S4':['a', 'c', 'd']
    }

def matrix_representation_create(database, features_dict):
    """create the matrix representation of one database.

    """
    matrix_ = np.zeros((len(database), len(features_dict)), dtype=np.int)
    matrix = pd.DataFrame(matrix_, index=database, columns=sorted(features_dict.keys()))

    for feature_name, values in features_dict.iteritems():
        for value in values:
            matrix.loc[value, feature_name] = 1

    return matrix

my_matrix = matrix_representation_create(my_database, my_features_dict)
my_matrix


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
#exercise 3.3.4
不是很明白意思，大概想法是：
从 $n$ 中抽取 $m$ 个元素集合 $S$ 和 $T$，则从 $S$ 和 $T$ 中各抽一次，元素相同的概率是多少？
#exercise 3.3.5
元素相同的概率是 0
3.4 Locality-Sensitive Hashing for Documents
minhash 降低了维度，但计算耗时问题仍然存在：
令样本集有 $k$ 个元素，两两比较次数是 $C_k^2 = O(k^2)$。
若 $k$ 很大，则比较次数成乘方倍增长。
解决思路：
1. 并行处理
2. 只比较可能相似的样本
   often we want only the most similar pairs or all paris that are above some lower bound in similarity.
   ----> Locality-sensitive hasing (LSH) or near-neighbor search.
LSH:
"hash" items several times, in such a way: $$P[\text{similar items in the same bucket}] >> P[\text{dissimilar ...}]$$
candidate pair: any pair hashed to the same bucket.
false postive: dissimilar items in the candidate pair.
false negtive: similar items NOT in the cnadidate pair.
if we have minhash signatures for the items, an effective way to choose the hashings is:
1. divide the signature matrix into $b$ bands consisting of $r$ rows each.
2. for each band $b_i$, hash each vector in $b_i$
   hash function $h$: same vectors -> same bucket
也就是说，将两个样本向量分割，只要有一个节点哈希到一起，就认为是候选对。
注意，统计的是每个节点各自哈希的情况，所以每个节点使用相同的哈希函数，但使用独立的哈希表存储中间结果。如下图：
matrix | --> | hash table
:-------:|:------:|:------------:
band 1 | h(v) | hash_talbe_1
band 2 | h(v) | hash_talbe_2
....   | h(v) | ....
Analysis the probabilty of two pairs to become a candidate pairs:
Given the pairs $s_a$ and $s_b$ have Jaccard similarity $s$, we use $b$ bands of $r$ row each to split their signature matrix $M_a$ and $M_b$.
1. $\forall \, i^{th} \, \text{row}, \, prob_{rowY} = P[M_a[i,:] == M_b[i,:]] = \text{Jaccard similarity} = s$
2. the possibility that one band($r$ row) is the same is: $$prob_{bandY} = prob_{rowY}^r = s^r$$
   .......................................not the same is: $$prob_{bandN} = 1 - prob_{bandY} = 1 - s^r$$
3. $s_a$ and $s_b$ are totally different: $$prob_{pairN} = prob_{bandN}^b = (1 - s^r)^b$$
   $s_a$ and $b$ at least agree in one band: $$prob{pairY} = 1 - prob_{pairN} = 1 - (1 - s^r)^b$$
threshold: the value of $s$ at which the probability of becoming a candidate...
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
2) $h_i(x)$ should be statistically independent, because we need estimate the possbility by using the product rule for independent events.
   eg: $s^r$ in $1 - (1 - s^r)^b$ for Jaccard similarity.
3) efficient in two ways:
+ identify candidate pairs in time much less than the time it takes to look at all pairs.
+ They must be combinable to build functions that are better at avoiding false positives and negatives, and the combined functions must also take time that is much less than the number of pairs.
当我们使用 “band, then hash" 这种方法来组合哈希函数达到概率 $1 - (1 - s^r)^b$ 变成 "S" 曲线（其实是倒”S"型，如上图），
所以我们需要找到的哈希方法，它应该有将距离接近的候选对更高概率匹配的特性，即 $P-d(x,y)$ 是倒 ”S"。这就引出了接下的内容：local sensitive.
$(d_1, d_2, p_1, p_2)$-sensitive：
1. if $d(x, y) \leq d_1$, then $P[f(x) = f(y)] \geq p_1$.
2. if $d(x, y) \geq d_2$, then $P[f(x) = f(y)] \leq p_2$.
eg: For Jaccard distance, the family of minhash functinos is a $(d_1, d_2, 1-d_1, 1-d_2)$-sensitive family for any $d_1$ and $d_2$ where $0 \leq d_1 < d_2 \leq 1$.
Can you write Python code for it?

#minhash for Jaccard distance
d = 1 - s
d_1, d_2 = 0.4, 0.6

plt.plot(d[d<d_1], p(s, r, b)[d<d_1], d[d>d_2], p(s, r, b)[d>d_2])
plt.xlabel('d(x, y)')
plt.ylabel('Probility of being a candidate')


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
2) 再按照行号重新对行排序，生成新矩阵 $\bar{M}$
但第 2 步排序仍然需要操作矩阵，见函数 minhash_by_rows 中的语句 matrix_p.sort_index(inplace=True)。
Can you write Python code for it?

#Fig 3.4

df_matrix = my_matrix
df_h_rows = pd.DataFrame(np.array(h_rows).T)
df_h_rows.columns = ['h{}'.format(x+1) for x in df_h_rows.columns]

print('Hash functions computed for the matrix:\n{}\n'.format(pd.concat([df_matrix, df_h_rows], axis=1)))
print('signature matrix\n(SIG):\n{}'.format(my_minhash_res))


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
3.8 Applications of Locality-Sensitive Hashing
1. Entity Resolution
to match data records that refer to the same real-world entity.
找到同一对象在不同数据库里的数据
eg: A 和 B 公司对账（记录方式不尽相同）
1. 相似度比较：
   使用"姓名“，”电话“，”地址信息“，三个分别用编辑距离计算，评分（d=0-1, score=100-0)，累积总值就是相似度评分 $s$。
   + certain publicly available tables were used to reduce the penalty in appropriate situations, like "Bill" and "William" are treated as 1 letter.

全部比较太耗时，所以用 LSH 先找出潜在配对。
   对三组分别哈希，用 AND- 构建。
   实际中的方法是直接排序：名字，电话，地址均按升序排列，相似的就聚焦在一起。

评价匹配正确率 $f$。
   使用其它未使用的记录来做为评据。
   如这里使用两个数据库配对项目的创建时间差 $t$ 做为度量：

$\bar{t} = E[\text{random_pick}(t)]$: 随机配对的时间差期望
$t_p = E[t_{xy}: argmax_{x,y} s(x,y)]$: 最大 $s$ 数据集的时间差期望（假设全是正确匹配，$f=1$）。

假设现在用 $s_i$ 做阈值，得到的匹配对算出均值 $t_i$，其正确匹配率是 $f_i$，
      则有 $t_i = t_p * f_i + \bar{t} * (1 - f_i)$
      解得：
      $$f_i = \frac{\bar{t} - t_i}{\bar{t} - t_p}$$
即，我们可以用结果中的其他数据项，组成评价匹配正确率的函数。

2. Matching Fingerprints
特征描述： minutiae 构建指纹的特征向量。
两类问题：
    1. 一对多：找出某个指纹的匹配
    2. 多对多：找出潜在相同的匹配
假设随机抽取一个指纹的特征向量的一位，其中有 minutiae 的概率是 $P[x] 20\%$。
   同一来源的两个指纹在特征向量的一位，一个有 minutiae 时，另一个也有的概率是 $P[Y|X] = 80\%$。
定义哈希函数 $f$ 为：
   抽取特征向量的三位，若两个指纹的每一位均含有 minutiae，则做为候选对。
分析：随机对候选的概率是 $0.2^6 = 0.000064$, 同一来源候选的概率是 $0.2^3 \times 0.8^3 = 0.004096$，
我们生成 1024 个 $f_i$ 哈希函数，使用 OR- 构建，再用 AND- 构建，结果见下图。
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
Minhashing
target: sets of shingles are large =====> much smaller representations called "signatures"
minhash
1. pick a permutation of the rows
2. h(columu c) = the row which has the first "1"
Can you write Python code for it?

#Example 3.7

logger.setLevel(logging.WARNING)

def minhash(matrix, row_orders):
    """calculate the minhash value of matrix according to element permutation in Fig 3.3.

    """
    hash_fun_names = ['h{}'.format(i) for i in range(1, len(row_orders)+1)]
    hash_table = pd.DataFrame(np.zeros((len(row_orders), matrix.shape[1])), index=hash_fun_names, columns=matrix.columns)

    for row_order, hash_fun_name in zip(row_orders, hash_fun_names):
        matrix_p = matrix.loc[row_order,:]
        logging.debug('after permutation: \n{}'.format(matrix_p))

        for c in matrix_p.columns:
            first_one_index = next((i for i, x in enumerate(matrix_p.loc[:,c]) if x), None)
            hash_table.loc[hash_fun_name, c] = row_order[first_one_index]

    return hash_table

minhash(my_matrix, [['b', 'e', 'a', 'd', 'c']])


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
exercise
Can you write Python code for it?

#exercise 3.2.1

data = "The most effective way to represent documents as sets, for the purpose of iden- tifying lexically similar documents is to construct from the document the set of short strings that appear within it. "

print('first 10 3-shingles:{}'.format(k_shingle_extrac(data,3)[0:10]))

#exercise 3.2.2
logger.setLevel(logging.WARNING)

def stopwords_shingles_extract(document, k):
    """extract the stopwords-shingle.

    stropwords-shingle = stop-words + k words
    """
    stop_words_list = ['the', 'you', 'to', 'as', 'for', 'of', 'is', 'that', 'it', 'from']

    document = to_alphanumeric_string(document)
    document = document.split()
    logging.debug('split:{}'.format(document))

    shingles = []
    k = k + 1 #len(shingle) = 1 stop-word + k words
    while(document):
        try:
            logging.debug('check:{}'.format(document[0]))
            if document[0] in stop_words_list:
                shingle = ' '.join(document[0:k])
                logging.debug('hit: {}'.format(shingle))

                if shingle not in shingles:
                    shingles.append(shingle)

        except IndexError:
            logging.debug('Index Error: no of char:{}, k: {}'.format(len(document), k))
            k = len(document)
            continue

        document = document[1:]

    return shingles

print('stop-shingles:{}'.format(stopwords_shingles_extract(data, 2)))


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
exercise 3.3
Can you write Python code for it?

#exercise 3.3.1

#generate 120 permutations
h_rows = list(itertools.permutations(range(5),5))
my_minhash_res = minhash_by_rows(my_matrix, h_rows)

for comb in list(itertools.combinations(range(4),2)):
    s_a, s_b = 'S{}'.format(comb[0]+1), 'S{}'.format(comb[1]+1)
    print('{}-{}'.format(s_a, s_b)),
    #calc Jaccard similarity
    print('true SIM:{:.3f}'.format(Jaccard_similarity_calc(set(my_features_dict[s_a]), set(my_features_dict[s_b])))),
    #calc the fraction of the 120 permutations in which the value is same
    print('fraction:{:.3f}'.format(sum(my_minhash_res.loc[:,s_a] == my_minhash_res.loc[:,s_b])/120))

#exercise 3.3.2
h_funcs[2:4] = [
    add_hash_func(2, 4, 5),
    add_hash_func(3, -1, 5)
]

minhash_signatures_calc(df_matrix, h_funcs, nagging=True)

#exercise 3.3.3

my_database = range(6)
my_features_dict = {
    'S1':[2, 5],
    'S2':[0, 1],
    'S3':[3, 4],
    'S4':[0, 2, 4]
    }

df_fig_3_5 = matrix_representation_create(my_database, my_features_dict)
print('Fig 3.5:\n{}\n'.format(df_fig_3_5))

#(a)
h_funcs = [
    add_hash_func(2, 1, 6),
    add_hash_func(3, 2, 6),
    add_hash_func(5, 2, 6)
]
df_matrix = df_fig_3_5
my_minhash_res = minhash_signatures_calc(df_matrix, h_funcs, nagging=True)
my_minhash_res

#(b) h_3 is a true permutation.

#(c)

for comb in list(itertools.combinations(range(4),2)):
    s_a, s_b = 'S{}'.format(comb[0]+1), 'S{}'.format(comb[1]+1)
    print('{}-{}'.format(s_a, s_b)),

    set_a, set_b = set(my_minhash_res.iloc[:,comb[0]]), set(my_minhash_res.iloc[:,comb[1]])
    print('minhash:{:.3f}'.format(Jaccard_similarity_calc(set_a, set_b))),
    print('true SIM:{:.3f}'.format(Jaccard_similarity_calc(set(my_features_dict[s_a]), set(my_features_dict[s_b]))))


I am working on the file "Mining_of_Massive_Datasets/Finding_Similar_Items/note.ipynb".
The first task is:
对 $M$ 中的所有行做一次随机组合得到 $\bar{M}$，非常耗时。
解决方法是不直接操作矩阵 $M$，而是分两步操作：
1) 选择合适的哈希函数生成行号序列来模拟随机排列结果，如上个代码区块中 h_rows 即可用 $h_1(x) = (x + 1) \% 5$ 和 $h_2(x) = (3x + 1) \% 5$ 生成对应的行号
Can you write Python code for it?

def add_hash_func(a, b, c):
    return lambda x: (a*x + b) % c

h_funcs = [
    add_hash_func(1, 1, 5),
    add_hash_func(3, 1, 5)
    ]

h_rows = []
for h_func in h_funcs:
    h_rows.append(map(h_func, range(5)))

print('h_rows:{}'.format(h_rows))


I am working on the file "Mining_of_Massive_Datasets/Frequent_Itemsets/note.ipynb".
The first task is:
6 Frequent Itemsets
6.1 The Market-Basket Model
Each Basket consists of a set of items(an itemset)

The number of items in a basket is small.

The number of baskets is usually very large.

Basket are sets, and in priciple items can appear only once.

Definition of Frequent Itemses
a set of items that appears in many baskets is said to be "frequent".
support: if ${I}$ is a set of items, the support of ${I}$ is the number of baskets for which I is a subset.
Assume $s$ is the support threshold, then we say ${I}$ is frequent if its support is $s$ or more.
Can you write Python code for it?

logger.setLevel('WARN')

data_raw = [
    ['Cat', 'and', 'dog', 'bites'],
    ['Yahoo', 'news', 'claims', 'a', 'cat', 'mated', 'with', 'a', 'dog', 'and', 'produced', 'viable', 'offspring'],
    ['Cat', 'killer', 'likely', 'is', 'a', 'big', 'dog'],
    ['Professional', 'free', 'advice', 'on', 'dog', 'training', 'puppy', 'training'],
    ['Cat', 'and', 'kitten', 'training', 'and', 'behavior'],
    ['Dog', '&', 'Cat', 'provides', 'dog', 'training', 'in', 'Eugene', 'Oregon'],
    ['Dog', 'and', 'cat', 'is', 'a', 'slang', 'term', 'used', 'by', 'police', 'officers', 'for', 'a', 'male-female', 'relationship'],
    ['Shop', 'for', 'your', 'show', 'dog', 'grooming', 'and', 'pet', 'supplies']
]

data = [set(map(str.lower, x)) for x in data_raw]
data = pd.Series(data)

def calc_occurrence_of_doubletons(df, data):
    logger.info('df: \n{}\n'.format(df))
    for i_ in df.index:
        for c_ in df.columns:
            if not np.isnan(df.loc[i_,c_]):
                key = {i_, c_}
                ind_ = [ind+1 for ind, is_in in enumerate(data.apply(lambda x: key.issubset(x))) if is_in]
                df.loc[i_,c_] = ','.join([str(x) for x in ind_])
    return df

mask = [
    [1, 1, 1, 1],
    [1, 1, 1, np.nan],
    [1, 1, np.nan, np.nan],
    [1, np.nan, np.nan, np.nan]
]
df = pd.DataFrame(mask, index=['dog', 'cat', 'and', 'a'], columns=['training', 'a', 'and', 'cat'])
calc_occurrence_of_doubletons(df, data)


I am working on the file "Mining_of_Massive_Datasets/Frequent_Itemsets/note.ipynb".
The first task is:
6.3.2 The Multistage Algorithm
It improves upon PCA by using several successive hash tables to reduce further the number of candidate pairs.

1st pass is the same as of PCY.

2nd pass:
   We hash ${i,j}$ if and only if:

$i$ and $j$ are both frequent.
${i,j}$ hashed to a frequent bucket of $B_1$ on the 1st pass.

Then summarized as a bitmap $B_2$.

$C_2$ pairs ${i,j}$:
$i$ and $j$ are both frequent items.
${i,j}$ hashed to a frequent bucket in $B_1$.
${i,j}$ hashed to a frequent bucket in $B_2$ as well.

Attention:
Each pass must store the bitmaps, eventually, there is not enough space left to count if used too many stages.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Large-Scale_Machine_Learning/note.ipynb".
The first task is:
12 Large-Scale Machine Learning
All algorithms for analysis of data are designed to produce a useful summary of the data, from which decisions are made.
"machine learning" not only summarize our data; they are perceived as learning a model or classifier from the data, and thus discover something about data that will be seen in the future.
12.1 The Machine-Learning Model
Training sets:
+ feature vector: $x$
+ label: $y$
  - real number, regression
  - boolean value, binary classification
  - finite set, multiclass classification
  - infinite set, eg: parse tree
12.1.3 Approaches to Machine Learning

Decision trees
   suitable for binary and multiclass classification
   small features

Perceptrons $\sum w x \geq \theta$
   binary classification, very large features

Neural nets
   binary or multiclass classification

Instance-based learning
   use the entire training set to represent the function $f$.
   eg: k-nearest-neighbor

Support-vector machines
   accureate on unseen data

12.1.4 Machine-Learning Architecture

Training and Testing

Batch Versus On-Line Learning
   on-line:

very large training sets
adapt to changes as times goes

active learning

Feature Selection

Creating a Training Set
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Large-Scale_Machine_Learning/note.ipynb".
The first task is:
12.2 Perceptrons
The output of the perceptron is:
+ $+1$, if $w x > 0$
+ $-1$, if $w x < 0$
+ wrong, if $w x = 0$
The weight vector $w$ defines a hyperplane of dimension $d-1$ where $w x = 0$.
A perceptron classifier works only for data that is linearly separable.
12.2.1 Training a Perceptron with Zero Threshold

Initialize $w = 0$.

Pick a learning-rate parameter $\eta > 0$.

Consider each training example $t = (x, y)$ in turn.

$y' = w x$.
if $y'$ and $y$ have the same sign, then do nothing.
otherwise, replace $w$ by $w + \eta y x$.
    That is, adjust $w$ slightly in the direction of $x$.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Large-Scale_Machine_Learning/note.ipynb".
The first task is:
12.2.2 Convergence of Perceptrons
data point are not linearly separable $\to$ loop infinitely.
some common tests for termination:
1. Terminate after a fixed number of rounds.

Terminate when the number of misclassified training points stops changing.

Terminate when the number of errors on the test set stops changing.

Another technique that will aid convergence is to lower the traing rate as the number of rounds increases. eg: $\eta = \eta / (1 + ct)$.
12.2.3 The Winnow Algorithm
Winnow assumes that the feature vectors consist of 0's and 1's, and the labels are $+1$ or $-1$. And Winnow produce only positive weithts $w$.
idea: there is a positive threshold $\theta$.

if $w x > \theta$ and $y = +1$, or $w x  \theta$ and $y = -1$, then do nothing.

if $w x \leq \theta$, but $y = +1$, then the weights for the components where $x$ has 1 are too low as a group, increase them by a factor, say 2.

if $w x \geq \theta$, but $y = -1$, then the weights for the components where $x$ has 1 are too high as a group, decreae them by a factor, say 2.

12.2.4 Allowing the Threshold to Vary
At the cost of adding another dimension to the feature vectors, we can treate $\theta$ as one of the components of $w$.

$w' = [w \theta]$.

$x' = [x -1]$.

We allow a $-1$ in $x'$ for $\theta$ if we treat it in the manner opposite to the way we treat components that are 1.

12.2.5 Multiclass Perceptrons
one VS all
12.2.6 Transforming the Training Set
turn to linear separable set.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Large-Scale_Machine_Learning/note.ipynb".
The first task is:
12.2.7 Problems With Perceptrons

The biggest problem is that sometimes the data is inherently not separable by a hyperplane.

transforming $\to$ overfitting.

many different hyperplanes that will separate the points.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Large-Scale_Machine_Learning/note.ipynb".
The first task is:
12.2.8 Parallel Implementation of Perceptrons
training examples are used with the same $w$.
map: calculate $w$ independently.
reduce: sum all $w$.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
2. modify the process of moving
"taxation"
5.1.5 Spider Traps and Taxation
spider traps:
a set of nodes with no dead ends but no arcs out.
they cause the PageRank calculation to place all the weights within the spider traps.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
5.1.3 Structure of the Web
Some structures in reality violate the assumptions needed for the Markov-process iteration to converge to a limit.
Can you write Python code for it?

plt.figure(figsize = (10,10))
plt.imshow(plt.imread('./res/fig_5_2.png'))


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
5.2.2 PageRank Iteration Using MapReduce
estimate: $$v' = \beta M v + (1 - \beta) \frac{e}{n}$$
if $v$ is much too large to fit in main memory, we could use the method of triping:
break $M $into stripes and break $v$ into corresponding horizontal stripes.
5.2.3 Use of Combiners to Consolidate the Result Vector
There are two reasons the method of Section 5.2.2 might not be adequate:

We might wish to add term for $v'_i$.

We might execute the iteration step at a single machine or a collection of machines instead.

Hence, we need to implement a combiner in conjunction with a Map task.
thrashing:
repeatly read some components of $v'$.
An alternative strategy is based on partitioning $M$ into $k^2$ blocks.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
Although C gets more than half of the PageRank for itself, the effect has been limited.
Note that for a random surfer, there are three path to move:

follow a link.

teleport to a random page. $\gets$ taxation

goes nowhere. $\gets$ dead ends

Since there will always be some fraction of a surfer operating on the Web, so even if there are dead ends, the sum of the ocmponents of $v$ may be less than 1, but it will never reacher 0.
5.1.6 Using PageRank in a Search Engine

find the qualified pages, which have at least one of the search terms in the query.

calculate a score for those pages, including PageRank.

5.1.7 Exercises
5.1.1
略
5.1.2
略
5.1.3
$n$ nodes: 1/n
the additional node: n * 1/n * 1/n = 1/n
5.1.4
todo
5.1.5
略
5.1.6
the first node: 1
the left nodes: 1/2
5.1.7
root: 1
height = 1:  1/3
height = 2:  1/3 * 1/2
height = k:  $\frac{1}{3} \times (\frac{1}{2})^{k-1}, k > 1$
5.2 Efficient Computation of PageRank
PageRank: matrix-vector multiplicaton $\to$ MapReduce.
However, we must deal with two issues:

$M$ is very sparse.
   list its nonzero elements.

we may wish to use a combiner to reduce the amount of data (Map $\to$ Reduce).
   striping approach.

5.2.1 Representing Transition Matrices
The proper way to represent any sparse matrix is to list the locations and values of the nonzero entries.
The space needed is linear in the number of nonzero entries.
Represent a column by:
    1. one integer for the out-degree,
    2. one integer for rowname per nonzero entry.
Can you write Python code for it?

matrix_5_1

import string

df_M = pd.DataFrame(matrix_5_1, index=list(string.uppercase[0:4]), columns=list(string.uppercase[0:4]))
df_M

def compact_representation_of_sparse_matrix(df):
    """It is introduced in Example 5.7"""

    degree = df.apply(np.count_nonzero, axis=0)

    dest = df.apply(np.nonzero, axis=0)
    dest = dest.apply(lambda x: x[0])

    return pd.concat([degree, dest], axis=1, keys=['Degree', 'Destinations'])

compact_representation_of_sparse_matrix(df_M)


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
Each task gets $M_{ij}$ and $v_j$.
Thus, $v$ is transmitted over the network $k$ times, but $M$ is sent only once.
The advantage of this approach is:
We can keep both the $v_j$ and $v'i$ in main memory as we process $M{ij}$.
5.2.4 Representing Blocks of the Transition Matrix
for each columns of $M_{ij}$, we need list its out-degree and rows of nonzero entry.
And it might lead to that the out-degree will be repeated as many times as the out-degree itself.
Can you write Python code for it?

plt.figure(figsize=(10,10))
plt.imshow(plt.imread('./res/fig_5_14.png'))

#todo


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
Links from Accessible Pages: comments contained spam link in blog or news sites.
5.4.2 Analysis of a Spam Farm
Given: $n$ pages in total, $m$ support pages, $y$ is the PageRank of target page $t$.
then:
the PageRank of each support page:
$$\frac{\beta y}{m} + \frac{1 - \beta}{n}$$
support $x$ is the contribute of all accessible pages for target page.
we can get the PageRank $p$ of targe page:
\begin{align}
    p &= x + \beta (\frac{\beta y}{m} + \frac{1 - \beta}{n}) m  + \frac{1 - \beta}{n} \
      &\approx x + \beta (\frac{\beta y}{m} + \frac{1 - \beta}{n}) m  \
      &= x + \beta^2 y + \beta (1-\beta) \frac{m}{n}
\end{align}
so we can solve $y$:
$$y = \frac{x}{1-\beta^2} + c \frac{m}{n}$$
where $c = \frac{\beta}{1+\beta}$.
Can you write Python code for it?

beta = 0.85
x_coe = 1 / (1 - beta**2)
c = beta / (1+beta)

print('y = {} x + {} m/n'.format(x_coe, c))


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
Markov processes:
It is known that the distribution of the surfer approaches a limiting distribution $v$ that satisfies $v = Mv$, provided two conditions are met:

The graph is trongly connnected.
   namely, it is possible to get from any node to any other node.

There are no dead ends.

eigenvalue and eigenvector: $Mx = \lambda x$
$v = x : \lambda = 1 \text{ and } \sum_j x_{ij} = 1 \text{ for every } i$.
$v$ is a stochastic vector, and $M$ is a stochastic/Markov matrix.
Iteration $O(M^2)$ is better than Gaussian elimination $O(M^3)$.
in practice, for the Web itself, 50-75 itreations are sufficient to converge to within the error limits of double-precision arithmetic.
Can you write Python code for it?

# eg 5.2
v_ = dist_after_surfing(matrix_5_1, v, 10)
v_

v_ = dist_after_surfing(matrix_5_1, v, 50)
v_

v_ = dist_after_surfing(matrix_5_1, v, 75)
v_


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
PageRank $v$ simulate random surfers:

start at a random page of all $n$.
   $v_i^0 = \frac{1}{n} \quad i = 1, 2, \dotsc, n$.

randomly choose next page linked.
   $v^{k+1} = M v^{k}$
   give us the distribution of the surfer after $k+1$ stpes.
Can you write Python code for it?

# eg. 5.1
matrix_5_1 = np.array([
                [0, 1/3, 1/3, 1/3],
                [1/2, 0, 0, 1/2],
                [1, 0, 0, 0],
                [0, 1/2, 1/2, 0]
             ]).T

matrix_5_1

n = matrix_5_1.shape[1]

v = np.ones((n,1)) / n
v

def dist_after_surfing(M, v=None, steps=1):
    if v is None:
        n = M.shape[1]
        v = np.ones((n,1)) / n

    for __ in xrange(steps):
        v = M.dot(v)

    return v

steps = 1
v_ = dist_after_surfing(matrix_5_1, v, steps)
v_


I am working on the file "Mining_of_Massive_Datasets/Link_Analysis/note.ipynb".
The first task is:
Two Solutions
1. Drop the dead end
recursive deletion of dead ends, and solve the remaining graph $G'$.
   then we restore $G$ from $G'$, recursivly.
   $$ e = \sum \frac{v_p}{k_p} $$
   where $e \in (G - G')$, $p \in G'$ and is the predecessor of $e$, $k$ is the number of successors of $p$ in $G$.
Can you write Python code for it?

# eg 5.4
plt.imshow(plt.imread('./res/fig_5_4.png'))

M_G = np.array([
        [0, 1/3, 1/3, 1/3, 0],
        [1/2, 0, 0, 1/2, 0],
        [0, 0, 0, 0, 1],
        [0, 1/2, 1/2, 0, 0],
        [0, 0, 0, 0, 0]
    ]).T
M_G

from sklearn.preprocessing import normalize

index = [0, 1, 3]
M = M_G.take(index, axis=0).take(index, axis=1)
M = normalize(M, norm='l1', axis=0)
M

n = M.shape[1]
v = np.ones((n,1)) / n

v_ = dist_after_surfing(M, v, 50)
v_

# the PageRank of C, whose predecessors are A and D.
1/3 * v_[0] + 1/2 * v_[2]

# the PageRank of D, which is the only one successor of C.
# hence, it is the same with C.
_


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
2 MapReduce and the New Software Stack
"big-data" analysis:

manage immense amounts of data quickly.

data is extremely regular $\to$ exploit parallelism.

new software stack:
"distributed file system" $\to$ MapReduce
When designing MapReduce algorithms, we often find that the greatest cost is in the communication.
2.1 Distributed File Systems
commodity hardware + network
2.1.1 Physical Organization of Compute Nodes
cluster computing: the new parallel-computing architecture.
racks: compute nodes are stored on racks.
communication:

The nodes on a single rack are connected by a network, typically gigabit Ethernet. And racks are connected by another level of network or a switch.

The bandwidth of inter-rack communication is somewhat greater than the intrarack Ethernet.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
2.4.3 Pregel
solution B: backup entire stats of each task checkpoints, so recovery the backup point if fail.
checkpoints is triggered at fixed supersteps.
2.4.4 Exercises for Section 2.4
Ex 2.4.1
for the prob of a taks, success is $(1-p)^t$, fail is $(1 - (1 - p)^t)$ . So the expected execution time of a task is $(1-p)^t t  + (1 - (1-p)^t) 10 t = 10t - 9t(1-p)^t$.
Thus, total expected time is $n(10t - 9t (1-p)^t) = nt(10 - 9 (1-p)^t)$.
Can you write Python code for it?

def calc_prob(p, t):
    return 10 - 9 * ((1 - p)**t)

p = np.linspace(0, 1, 100)

y1 = calc_prob(p, 10)
y2 = calc_prob(p, 100)
plt.plot(p, y1, 'b', p, y2, 'g')


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
Each Map task is assigned a chunk from one of the stripes of the matrix and gets the entire corresponding stripe of the vector.
particular application (PageRank calculation) has an additional constraint that the result vector should be partitioned in the same way as the input vector.
We shall see there that the best strategy involves partitioning the matrix $\mathbf{M}$ into square blocks, rahter than stripes.
2.3.3 Relational-Algebra Operations
There are many operations on data that can be described easily in terms of the common database-query primitives.
a relation is a table with column headers called attributes.
Rows of the relation are called tuples.
The set of attributes of a relation is called its schema.
$R(A_1, A_2, \dotsc, A_n)$: the relation name is $R$ and its attributes are $A_1, A_2, \dotsc, A_n$.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
Ex 2.4.2
suppose that supersteps should be $n$, the time of one execting a superstep is $t$.
2.5 The Communication Cost Model
for many applications, the bottleneck is moving data among tasks.
2.5.1 Communication-Cost for Task Networks
The communication cost of a task is the size of the input to the task. we shall often use the number of tuples as a measure size, rather than bytes.
The communication cost of an algorithm is the sum of the communication cost of all the tasks implementing that algorithm. We shall focus on the communication cost as the way to measure the efficiency of an algorithm, since the exceptions, where execution time of tasks dominates, are rare in practice.
We count only input size, and not output size.
2.5.2 Wall-Clock Time
Besides communication cost, we must also be aware of the importance of wall-clock time, the time it takes a prallel algorithm to finish.
The algorithms shall have the property that the work is divided fairly among the tasks.
2.5.3 Multiway Joins
example: $R(A, B) \bowtie S(B, C) \bowtie T(C, D)$.
Suppose that the relation $R, S, T$ have sizes $r, s$, and $t$, respectively. and for simplicity, suppose $p$ is the probability that any two tuples in each relations agree on the item they share.
Solution 1: general theory
$\operatorname{join} \left ( \operatorname{join} \left ( R(A, B) \bowtie S(B, C) \right ) \bowtie T(C, D) \right )$
or exchange the sequence of join.

1st MapReduce
   $\operatorname{join} \left ( R(A, B) \bowtie S(B, C) \right )$
   $O(t + prs)$

2nd MapReduce
   $O(r + s + t + prs)$

Solution 2: use a single MapReduce job that joins the three relations at once.
Assume:

We plan to use $k$ reducers for the job.

$b, c$ represents the number of buckets into which we shall hash $B-$ and $C-$values, respectively.

$h(B) \to b$.

$g(C) \to c$.

we require $b c = k$.
So, the reducer corresponding to bucket pair $(i, j)$ is responsible for joining the tuples $R(u, v), S(v, w)$, and $T(w, x)$ whenever $h(v) = i$ and $g(w) = j$.

$S(v, w)$ to reducer...
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
relational algebra: several standard operations on relations.

Selection $\sigma{C} (R)$: select tuples that satisfy $C$
Map: produce $(t, t)$ if $t$ satisfies $C$ where $t \in R$.

Reduce: simple passes each key-value pair to the output.

Projection $\pi_{S} (R)$: produces subset $S$ of the attributed

Map: output $(t', t')$ where $t'$ is subset of $t$.

Reduce: eliminate duplicates, turns $(t', [t', t', \dotsc, t'])$ into $(t', t')$.
     associative and commutative $\to$ combiner

Union, Intersection, and Difference

Union:

Map: turn $t$ into $(t, t)$.
Reduce: Produce $(t, t)$ for input $(t, [t])$ or $(t, [t, t])$.

Intersection:

Map: turn $t$ into $(t, t)$.
Reduce: produce $(t, t)$ if input $(t, [t, t])$.

Difference $S - R$:

Map: produce $(t, R)$ where $t \in R$, or $(t, S)$ where $t \in S$.
Reduce: produce $(t, t)$ if input $(t, [R])$.

Natural Join $R \bowtie S$

Map: produce $(b, (R, a))$ for $(a, b) \in R$, or $(b, (S, c))$ for $(b, c) \in S$.

Reduce: $(a, b, c)$ when input $(b, [(R, a), (S, c)])$.

Grouping and Aggregation $\gamma_{X} (R)$:
   where $X$ consists of:

a grouping attribute,
an expression $\theta(A)$

Let $R(A, B, C)$, for $\gamma_{A, \theta(B)} (R)$:
   + Map: produce $(a, b)$ for each tuple $(a, b, c)$.
   + Reduce: apply the aggregation operator $\theta$ to the list $[b_1, b_2, \dotsc, b_n]$ of B-values associated with key $a$.
2.3.9 Matrix Multiplication
$\mathbf{M} \times \mathbf{N}$
grouping and aggregation
two MapReduce step

1st MapReduce
Map: $(j, (M, i, m_{ij}))$ and $(j, (N, k, n_{jk}))$.

Reduce: $((i, k), m_{ij} n_{jk})$ for its associated values of each key $j$.

2nd MapReduce

Map: identity
Reduce: for each key $(i, k)$, produce the sum of the list of values associated with this key.

one MapReduce step

Map:
for each $m_{ij}$, produce all pairs $((i, k), (M, j, m_{ij}))$ for $k = 1, 2, \dotsc$.

for each $n_{jk}$, produce all pairs $((i, k), (N, j, n_{jk}))$ for $i = 1, 2, \dotsc$.

Reduce:
  Each key $(i, k)$ will have an associated list with all...
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/MapReduce_and_the_New_Software_Stack/note.ipynb".
The first task is:
the solution for components failure (loss of node or rack):

Files must be stored redundantly.

Computations must be divided into tasks.

2.1.2 Large-Scale File-System Organization
DFS: distributed file system

Google File System (GFS)

Hadoop Distributed File System (HDFS)

CloudStore

It is typically used as follows:

Files can be enormous, possibly a terabyte in size.

Files are rarely updated.

manage:

Files are divided into chunks.
   Chunks are replicated at different compute nodes of different racks.

master node or name node: another small file to find the chunks of a file.
   master node is iteself replicated, and a directory for the file systme as a whole knows where to find its copies.
   The directory itself can be replicated, and all participants using the DFS know where the directory copies are.

2.2 MapReduce
All you need to write are two functions, called Map and Reduce.
a MapReduce computation executes as follows:

Map function: Map tasks turn the chunk given into a sequence of key-value pairs.

The key-value pairs from each Map task are collected by a master controller and sorted by key.
   divide by key: all pairs with same key $\to$ same Reduce task.

Reduce function: The Reduce tasks work on one key at a time, and combine all the values associated with that key in some way.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
#exercise
10.2 Clustering of Social-Network Graphs
clustering of the graph as a way to identify communities.
10.2.1 Distance Measures for Social-Network Graphs
distance measure:
\begin{align}
    d(x,y) =
    \begin{cases}
         0 \text{ or } 1 \text{ or } 1 & \text{if edge} (x,y) {exists} \
         1 \text{ or } \infty \text{ or } 1.5 & \text{otherwise}
    \end{cases}
\end{align}
10.2.2 Applying Standard Clustering Methods
They are generally unsuitable for the problem of clustering social-network graphs.
10.2.3  Betweenness: define
betweenness of an edge $(a,b)$:
the number of pairs of nodes $x$ and $y$ such that $(a,b)$ lies on the shortest path between $x$ and $y$. And it's credicted with the fraction of all shortest paths between $x$ and $y$ if existed.
A high score indicates that $(a,b)$ runs between two different communities, namely, $a$ and $b$ do no belong to the same community.
10.2.4 The Girvan-Newman Algorithm: calculate betweenness
The algorithm is aimed to calculate the number of shortest paths going through each eage.
1) Starting at a node $X$, perform a breadth-first search (BFS) of the graph to label levels of each node.
   + DAG edges: edges between levels.

If there is a DAG edge $(Y,Z)$, where $Y$ is at the level about $Z$, then we shall call $Y$ a parent of $Z$ and $Z$ a child of $Y$.

2) to label each node by the number of shortest paths that reach it from the root.
   + Start by labeling the root 1.
   + Then, from the top down, label each node $Y$ by the sum of the labes of its parents.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10 Mining Social-Network Graphs

how to identify "communities"?
   communities: strong connections, usually overlap.

explore efficient algorithms for discovering other properities of graphs.

10.1 Social Networks as Graphs
10.1.1 What is a Social Network?
The essential characteristics of a social network are:

There is a collection of entities that participate in the network.

There is at least one relationship between entities of the network.

all-or-nothing: friends of Facebook.

discrete degree: friends, family as in Google plus.

real number: the times of talking.

There is an assumption of nonrandomness or locality.
   If A is related to both B and C, then there is a higher probability than average that B and C are related.

10.1.2 Social Networks as Graphs
social graph: The entities are the nodes, and an edge connects two nodes if the nodes are realted by the relationship that characterizes the network.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10.3 Direct Discovery of Communities
10.3.1 Finding Cliques
NP-complete: finding a large clique (a sef of nodes with edges between any two of them).
10.3.2 Complete Bipartite Graphs
A complete bipartite graph $K_{s,t}$ consists of $s$ nodes on one side and $t$ nodes on the other side, with all $st$ possible edges between the nodes of one side and the other present.
Idea:

While it is not possible to guarantee that a graph with many edges necessarily has a large clique, it is possible to guarantee that a bipartite graph with many edges has a large complete bipartite subgraph.

We can regard a complete bipartite subgraph as the nucleus of a community, and add to it nodes with many edges to existing members of the community.

If its nodes is consisted of two or more types, construct bipartite graphs directly.

If all nodes have the same type, divide the node into two equal groups at random.

10.3.3 Finding Complete Bipartite Subgraphs
It's possible to view the problem of finding instance of $K_{s,t}$ within $G$ as one of finding frequent itemsets.

"items" - left side.

"baskets" - right side.
    The members of the basket for node $v$ are the nodes of the left side to which $v$ is connected.

Let the support threshold be $s$, the number of nodes that the instance of $K_{s,t}$ has on the right side.

the problem of find $K_{s,t}$ $\to$ finding frequent itemsets $F$ of size $t$.
10.3.4 Why Complete Bipartite Graphs Must Exist
Assume:

the graph $G$ has $n$ nodes on the left and another $n$ nodes on the right.

let $d$ be the average degree of all nodes.

the degree of the $i$th node on the right is $d_i$.

Proof:

The total contribution of the $n$ nodes on the right is $\sum_i \binom{d_i}{t} \geq n \binom{d}{t}$ .

The number of itemsets of size $t$ is $\binom{n}{t}$.

Thus, the average count of an itemset of size $t$ is $n \binom{d}{t} / \binom{n}{t}$ ???

\begin{align}
    n \binom{d}{t} / \binom{n}{t} &= n \frac{d!}{(d-t)! t!} \frac{t! (n-t)!}{n!} \
        &= n \frac{d (d-1)...
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10.4.2 Normalized Cuts
A proper definition of a "good" cut must belance the size of the cut itself against the difference in the sizes of the sets that the cut creates.
The normalized cut value for $S$ and $T$ is
\begin{equation}
    \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(S)} + \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(T)}
\end{equation}
where $\operatorname{Vol}(S)$ is the number of edges with at least one end in $S$, and $\operatorname{Cut}(S,T)$ is the number of edges connected between $S$ and $T$.
10.4.3 Some Matrices That Describe Graphs

adjacent matrix:
   \begin{equation}
       A_{i,j} = \begin{cases}
                     1, & \text{if node $i$ and $j$ is connected} \
                     0, & \text{otherwise}
                 \end{cases}
   \end{equation}

degree matrix:
   $D_{i,i}$ is the degree of the $i$th node.

Laplacian matrix:
   $L = D - A$
   Notice that each row and column sums to zero.

10.4.4 Eigenvalues of the Laplacian Matrix
The smallest eignevalues and their eigenvectors reveal the information we desire.

The smallest eignevalues for every Laplacian matrix is 0, and its corresponding eigenvectors is $\mathbf{1}$ ones matrix.

the second-smallest eigenvalues of $L$ is the minimum of $x^T L x$, and the minimum is taken under the constraints:

$\displaystyle \sum_{i=1}^n x_i^2 = 1$
$x$ is orthogonal to the eigenvector associated with the smallest eigenvalue.
     $$x^T \mathbf{1} = \displaystyle \sum_{i=1}^n x_i = 0$$

In all:
$x^T L x = \sum_{i,j} (x_i - x_j)^2$
As a consequence, $x$ must have some positive and some negative components $\to$ two sets/groups.
10.4.5 Alternative Partitioning Methods

We could set the threshold at some point other than zero.

We may also want to a partition into more than two components:

split repeatedly as far as desired.
use several of the eigenvectors to partition the graph.

Attention:
while each eigenvector tries to produce a minimum-sized cut, the fact that successive eigenvectors have to satisfy more and more...
Can you write Python code for it?

# Exercises for Section 10.4
## Ex 10.4.1
### (a)

edges = [
    ('A', 'B'), ('A', 'C'), ('B', 'C'),
    ('B', 'H'), ('C', 'D'), ('H', 'I'),
    ('H', 'G'), ('D', 'E'), ('D', 'F'),
    ('I', 'G'), ('G', 'E'), ('E', 'F')
]

G = nx.Graph()
G.add_edges_from(edges)

nx.draw(G)

A = nx.adjacency_matrix(G).todense()
A

D = np.diag(np.ravel(A.sum(axis=1)))
D

L = D - A
L

### Ex 10.4.2
w, v = np.linalg.eig(L)

eig_min = np.argmin(w)
w = np.delete(w, eig_min)
v =np.delete(v, eig_min, axis=0)

eig_2nd_min = np.argmin(w)
eigv_2nd_min = np.ravel(v[eig_2nd_min])

setA = set([n for v, n in zip(eigv_2nd_min, G.nodes()) if v >= 0])
setB = set(G.nodes()) - setA
print(setA, setB)


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10.5 Finding Overlapping Communities
Communities are in practice rarely disjoint.
Assume:
the probability that two individuals are connected by an edge increases as they become members of more communities in common.
10.5.1 The Nature of Communities
We expect edges to be dense with any community, but we expect edges to be even denser in the intersection of two communities, three communities, and so on.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10.5.2 Maximum-Likelihood Estimation
We assume that the value of the parameters that gives the largest value of the likelihood is the correct model for the observed artifact.
$$\operatorname{argmax}_{\theta} P[f(\theta)]) $$
prior probabilities:
$$\operatorname{argmax}{\theta} P[f(\theta)]) = \operatorname{argmax}{\theta} P[f(\theta) | \theta] \, P[\theta] $$
10.5.3 The Affiliation-Graph Model
affiliation-graph model: generate social graphs from communities.
community-affiliation graphs:

Given: $C$ communities, $N$ nodes(individuals).

Question: $n_i \in C_k$ ?

Model:

Parameter: the memberships in the communities, $C_k = {n_i}$.
Parameter: $P_{ck}$ is the probability that two members of community $C_k$ are connected by an edge, $P_{ck} = P[(u,v) \in E \, | \, u \in C_k, v \in C_k]$.

we compute the likelihood that a given graph with the proper number of nodes is generated by this mechanism.
membership: Y/N

Parameter: membership
  define membership: $n_i \in C_k$.
  0 / 1, Yes / No, decrete variable $\to$ brute search

Parameter: $P_{ck}$
\begin{align}
    &P_{u,v} = 1 - \displaystyle \prod_{C_k in M} (1 - P_{ck})  \quad \text{where } M = {C_i: u \in C_i, v \in C_i}\
    &P[f(P_{ck}, C_k, E)] = \displaystyle \prod_{(u,v) \in E} P_{u,v} \, \prod_{(u,v) \notin E} (1 - P_{u,v})
\end{align}

Goal:
   find $\operatorname{argmax}{C_k} P[f(P{ck}, C_k, E)]$.

membership: "strength"
avoiding the use of discrete membership changes. This improvement allows us to use standard methods.
"strenght of membership":
the stronger the membership of two individuals in the same community, the more likely it is that this community will cause them to have an edge between them.
In the improved model,

Parameter: membership
   define membership: strength, $F_{xC} \in \mathbb{R}_{\ge 0}$

Parameter: $P_{ck}$
   $$P_C(u,v) = 1 - e^{- F_{u,C} F_{v,C}}$$
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
10.6 Simrank
The purpose of simrank is to measure the similarity between nodes of the same type, and it does so by seeing where random walkers on the graph wind up when starting at a particular node $\to$ limited in the size of graphs.
10.6.1 Random Walkers on a Social Graph
Random walkers:
A walker at a node $N$ of an undirected graph will move with equal probability to any of the neighbors of $N$.
10.6.2 Random Walks with Restart
Let $M$ be the transition matrix of the graph $G$: the entry in row $i$ and column $j$ of $M$ is $1/k$ if node $j$ of $G$ has degree $k$, and one of the adjacent nodes is $i$.
$\beta$ is the probability that the walker continues at random, so $1 - \beta$ is the probability the walker will teleport to the initial node $N$.
$v'$ is the probability the walker is at each of the nodes at the next round:
$$v' = \beta M v + (1 - \beta) e_N$$
Can you write Python code for it?

plt.imshow(plt.imread('./res/fig10_22.png'))

edges = [
    ('Picture 1', 'Sky'),
    ('Picture 1', 'Tree'),
    ('Picture 2', 'Sky'),
    ('Picture 3', 'Sky'),
    ('Picture 3', 'Tree')
]

G=nx.Graph()
G.add_edges_from(edges)
nx.draw(G)

nodelist = ['Picture 1', 'Picture 2', 'Picture 3', 'Sky', 'Tree']
adj = nx.adjacency_matrix(G, nodelist=nodelist).todense()

adj = pd.DataFrame(adj, index=nodelist, columns=nodelist)
adj

M = adj.apply(lambda x: x / sum(x), axis=0)
M

beta = 0.8
e_0 = np.identity(M.shape[0])[0]
e_0 = pd.DataFrame({'e_0': e_0}, index=nodelist)
e_0

def random_walk(v: pd.DataFrame, beta: float, M: pd.DataFrame, e_n: pd.DataFrame) -> pd.DataFrame:
    return beta * (M.dot(v)) + (1 - beta) * e_n

v = [e_0]
iter_time = 50

v_ = v[0]
for k in range(iter_time):
    v_ = random_walk(v_, beta, M, e_0)
    v.append(v_)

index = list(range(iter_time+1))
pd.concat(v, axis=1).T.reset_index(drop=True).T


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
4) After performing the credit calculation with each node as the root,
   1. we sum the credits for each edge.

And then divive the credit for each edge by 2, as each shortest path will have been discovered twice.

Then we get the the true betweenness.

10.2.5 Using Betweenness to Find Communities
It's a process of edge removal:

Start with the graph and all its edges;

then remove edges with the highest betweenness, until the graph has broken into a suitable number of connected components.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
Is Fig 10.1 typical of a social network, in the sense that it exhibits locality of relationships?
Suppose X, Y and Z are nodes of Fig 10.1, with edges between X and Y and also between X and Z.

What would we expect the probability of an edge between Y and Z to be?
   The graph has 9 edges out of the $C_7^2 = 21$ pairs of nodes.

Since we already have (X,Y) and (X,Z), the probability of an edge (Y,Z) is $(9-2)/(21-2) = 0.368$.  If the graph were large enough, that probability would be very close to the fraction of the pairs of nodes that have edges between them, i.e., $9/21 = 0.429$.

Then, we must compute the probability that $(Y,Z)$ exists in Fig 10.1, given that edges $(X,Y)$ and $(Y,Z)$ exist.
   We count the pairs of nodes that could be $Y$ and $Z$, without worrying about which node is $Y$ and which is $Z$.

If $X$ is $A$, $(B,C)$ contributes one positive example. All details are in the table following. In all, the fraction of times the third edge exists in thus $9 / 16 = 0.563 > 0.368$.
| node | pos | neg |
|------|-----|-----|
| A | 1 | 0|
| C | 1 | 0|
| E | 1 | 0|
| G | 1 | 0|
| F | 2 | 1|
| B | 1 | 2|
| D | 2 | 4|
|sum| 9 | 7| 16 |
10.1.3 Varieties of Social Networks

Telephone Networks

Email Networks

Collaboration Networks

Wikipedia: articles and editors.

published research papers and authors.

Other example

information networks (patents)
infrastruture networks (roads)
biological networks (genes)
product co-purchasing networks (Groupon)

10.1.4 Graphs with Several Node Types
There are other social phenomena that involve entities of different types. The natural way to represent such information is as a $K$-partite graph for some $k > 1$. In general, a $k$-partite graph consists of $k$ disjoint sets of nodes, with no edges between nodes of the same set.
Eg:
deli.cio.cu: there are three different kinds of entites: users, tags and pages.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Mining_Social_Network_Graphs/note.ipynb".
The first task is:
Speeding up the Betweenness Calculation:
If the large is large, we can pick a subset of the nodes at random and use these as the root of breadth-first searches, we can get an approximation to the betweenness of each edge that will serve in most applications.
cons:
It is not possible to place an individual in two different communities, and everyone is assigned to a community.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Recommendation_Systems/note.ipynb".
The first task is:
9.3.2 The Duality of Similarity

We can use information about users to recommend items, whereas even if we find pairs of similar items, it takes an additional step in order to recommend items to users.

find $n$ similar users $\to$ recommend item $I$ to user $U$.
     normalize the utility matrix first.
     \begin{align}
         M[U,I] &= Ave(M[U,:]) + Ave(M[0:n,I] - Ave(M[0:n,I])) \
                 &\approx Ave(M[U,:]) + Std(M[0:n,I])
     \end{align}

find $m$ similar items $\to$ recommend item $I$ to user $U$.
     $$M[U,I] = Ave(M[U,0:m])$$

in order to recommend items to user $U$, we need to find all or most of entry in $M[U,:]$.
tradeoff:

user-item: find similar users, directly get all predict values of all potent items.
    item-item: find similar items, we need calculate all items one by one (additional step) to fill $M[U,:]$.

item-item similarity often provides more reliable information due to the simplicity of items (genre).

precompute preferred items for each user.
     utility matrix evolves slowly $\implies$ compute it infrequently and assume that it remains fixed between recomputations.

Items tend to be classifiable in simple terms (eg: genre), whereas the individuals are complex.

9.3.3 Clustering Users and Items
Hierachical approach is prefered:

leave many cluster unmerged at first.

cluster items, and average the corresponding value in utility matrix.

cluster users, and average as well.

repeat several times if we like.

Predict $M[U,I]:

$U \in C$, and $I \in D$.

predict:

\begin{equation}
       M[U,I] = \begin{cases}
           M_{revised}[C,D] & \quad \text{if existed.}  \
           \text{estimate using similar users/items} & \quad \text{otherwise}
       \end{cases}
   \end{equation}
Can you write Python code for it?

# Fig 9.8

raw_data = [
    [4, 5, np.nan, 5, 1, np.nan, 3, 2],
    [np.nan, 3, 4, 3, 1, 2, 1, np.nan],
    [2, np.nan, 1, 3, np.nan, 4, 5, 3]
]

import string
M_uti = pd.DataFrame(raw_data, index=list(string.uppercase[:3]), columns=list(string.lowercase[:8]))
M_uti

logger.setLevel('WARN')
# exercise 9.3.1
from scipy.spatial.distance import jaccard, cosine
from itertools import combinations

def calc_distance_among_matrix(M, func_dis):
    for c in list(combinations(M.index, 2)):
        logger.info('c: {}'.format(c))
        u, v = M.loc[c[0]], M.loc[c[1]]
        logger.info('\n u:{},\n v:{}\n'.format(u.values,v.values))
        print('{} {}: {}'.format(c, func_dis.__name__, func_dis(u,v)))

# (a)
calc_distance_among_matrix(M_uti.notnull(), jaccard)

# (b)
calc_distance_among_matrix(M_uti.fillna(value=0), cosine)

# (c)
M_tmp = M_uti.copy()
M_tmp[M_uti < 3] = 0
M_tmp[M_uti >= 3] = 1
calc_distance_among_matrix(M_tmp, jaccard)

# (d)
calc_distance_among_matrix(M_tmp.fillna(value=0), cosine)

# (e)
M_uti_nor = M_uti.apply(lambda x: x - np.mean(x), axis=1)
M_uti_nor

# (f)
calc_distance_among_matrix(M_uti_nor.fillna(value=0), cosine)

# exercise 9.3.2
#todo


I am working on the file "Mining_of_Massive_Datasets/Recommendation_Systems/note.ipynb".
The first task is:
9.4 Dimensionality Reduction
UV-decomposition: $$M = U \times V$$
measure: RMSE (root-mean-square error)
Building a Complete UV-Decomposition Algorithm

Preprocessing of the matrix $M$.
normalization:
subtract: average rating of user $i$, then average rating of item $j$.
subtract: first item, then user.

subtract: half of average of item and half of average of user.

Initializing $U$ and $V$.
   choice: gives the elements of $UV$ the average of the nonblank elements of $M$.
   $\implies$ the element of $U$ and $V$ should be $\sqrt{a/d}$,
   where $a$ is the average nonblank element of $M$, $d$ is the lengths of the short sides of $U$ and $V$.

local minima contains global minima:
   1. vary the initial values of $U$ and $V$:
      perturb the value $\sqrt{a/d}$ randomly.
   2. vary the way we seek the optimum.

Performing the Optimization.
   different optimization path:
   choose a permutation of the elements and follow that order for every round.

Gradient Descent $\to$ stochastic gradient descent.

Converging to a Minimum.
   track the amount of improvement in the RMSE obtained.

stop condition:
   1. stop when that improvement in one round falls below a threshold.
   2. stop when the maximum improvement during a round is below a threshold.
Avoiding Overfitting
solutions:

optimized by only moving the value of a component a fraction of the way from its current value toward its optimized value.

Stop before the process has converged.

Take several different $UV$ decompositions, and average their predicts.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Recommendation_Systems/note.ipynb".
The first task is:
In practice, the matrix would be even sparser, with the typical user rating only a tiny fraction of all avalibale items.
the goal of a recommendation system is: to predict the blanks in the utility matrix.
+ slightly difference in many application:
  - predict every blank entry $<$ discover some potential entries in each row.
  - find all items with the highest expected ratings $<$ find a large subset of those.
The Long Tail
physical institutions | online institutions
---- | -----
provide only the most popular items | provide the entire range of items
the long tail force online institutions to recommend items to individual users:

It's no possible to present all avaliable items to the user.

Neither can we expect users to have heared of each of the items they might like.

Applications of Recommendation Systems

Product Recommendations

Movie Recommendations

News Articles

Populating the Utility Matrix
how to discovery the value users place on items:

We can ask users to rate items.
   cons: users are unwilling to do, and so samples are biased by very little fraction of peoples.

We can make inferences from users' behavior.
   eg: items purchased/viewed/rated.

9.2 Content-Based Recommendations
9.2.1 Item Profiles
a record representing important characteristics of items.
Discovering Features

for Documents
   idea: find the identification of words that characterize the topic of a document.
   namely, we expect a sets of words to express the subjects or main ideas of the document.
eliminate stop words.
compute the TF.DIF score for each reamining word in the document.
take as the features of a document the $n$ words with the highest TF.DIF scores.

to measure the similarity of two documents, the distance measures we could use are:
   1. Jaccard distance
   2. cosine distance
      cosine distance of vectors is not affected by components in which both vectors have 0.

for Images
   invite users to tag the items.
   cons: users are unwilling to do...
Can you write Python code for it?

# example 9.4

users_name = ['U', 'V']
items_name = ['F{}'.format(x) for x in range(4)]
features_name = ['Julia Roberts', 'others']

# utility matrix
M_uti = pd.DataFrame([
                        [3, 4, 5, 0],
                        [6, 2, 3, 5]
                     ],
                     index=users_name,
                     columns=items_name
                    )

M_uti

# item profile
M_item = pd.DataFrame(index=items_name, columns=features_name)

M_item.loc[:, features_name[0]] = 1
M_item = M_item.fillna(value=0)

M_item

M_uti.apply(lambda x: x - np.mean(x), axis=1)

M_user = M_uti.fillna(value=0).dot(M_item) / 4 #average = sum/len
M_user


I am working on the file "Misc experiments/ErgodicityProblem.ipynb".
The first task is:
Gain 50%, lose 30%, 50/50
Can you write Python code for it?

# run 1 simulation
start_cash = 10000
savings = start_cash
n_trades = 1000

history = []

for i in range(n_trades):

    if np.random.randint(100) % 2 == 0:
        savings = savings * 1.5
    else:
        savings = savings * .7

    history.append(savings)

history = np.asarray(history)
time_to_max = np.argmax(history)
time_to_min = 0
max_value = np.max(history)

for i in range(len(history)):
    if history[i] > 1: time_to_min += 1
    else: break
print('Gain 50%, lose 30% odds 50/50')
print( 'Starting Savings $%d \nTime to max %d trades, value %.0f, Time to $1 %d trades' % (start_cash, time_to_max, max_value, time_to_min))

# plot

plt.figure(figsize=(20, 10))
plt.plot(history)
plt.show()


I am working on the file "Misc experiments/ErgodicityProblem.ipynb".
The first task is:
Gain 60%, lose 40% 50/50
Can you write Python code for it?

# run 1 simulation
start_cash = 10000
savings = start_cash
n_trades = 1000

history = []

for i in range(n_trades):

    if np.random.randint(100) % 2 == 0:
        savings = savings * 1.6
    else:
        savings = savings * .6

    history.append(savings)

history = np.asarray(history)
time_to_max = np.argmax(history)
time_to_min = 0
max_value = np.max(history)

for i in range(len(history)):
    if history[i] > 1: time_to_min += 1
    else: break
print('Gain 60%, lose 40% odds 50/50')
print( 'Starting Savings $%d \nTime to max %d trades, value %.0f, Time to $1 %d trades' % (start_cash, time_to_max, max_value, time_to_min))

# plot

plt.figure(figsize=(20, 10))
plt.plot(history)
plt.show()


I am working on the file "Misc experiments/ErgodicityProblem.ipynb".
The first task is:
Gain 70% lose 40%, 50/50
Can you write Python code for it?

# run 1 simulation
start_cash = 10000
savings = start_cash
n_trades = 1000

history = []

for i in range(n_trades):

    if np.random.randint(100) % 2 == 0:
        savings = savings * 1.7
    else:
        savings = savings * .6

    history.append(savings)

history = np.asarray(history)
time_to_max = np.argmax(history)
time_to_min = 0
max_value = np.max(history)

for i in range(len(history)):
    if history[i] > 1: time_to_min += 1
    else: break
print('Gain 70%, lose 40% odds 50/50')
print( 'Starting Savings $%d \nTime to max %d trades, value %.0f, Time to $1 %d trades' % (start_cash, time_to_max, max_value, time_to_min))

# plot

plt.figure(figsize=(20, 10))
plt.plot(history)
plt.show()


I am working on the file "Misc experiments/ErgodicityProblem.ipynb".
The first task is:
try 1000 runs, maybe that one was just a fluke
Can you write Python code for it?

savings = start_cash
n_simulations = 10
n_trades = 1000

history = []
simulations = []

for i in range(n_simulations):

    savings = start_cash
    history = []

    for j in range(n_trades):

        if np.random.randint(100) % 2 == 0:
            savings = savings * 1.5
        else:
            savings = savings * .6
        history.append(savings)

    simulations.append(history)

print(len(simulations))
print(len(simulations[0]), len(simulations[4]))

# quit while your ahead

time_to_max = []
time_to_min = []
best = []

for i in range(n_simulations):
    history = np.asarray(simulations[i])
    time_to_bottom = 0

    time_to_max.append(np.argmax(history))
    best.append(np.max(history))

    for j in range(len(history)):
        if history[j] > 1: time_to_bottom += 1
    time_to_min.append(time_to_bottom)

avg_best = sum(best)/len(best)
avg_time_to_max = sum(time_to_max)/len(time_to_max)
avg_time_to_min = sum(time_to_min)/len(time_to_min)
best_best = max(best)

print('Avg peak $%.0f, best peak $%0.f\nAvg time to peak %d trades, Avg time to $1 %d trades' %(avg_best, best_best, avg_time_to_max, avg_time_to_min))

# plot
colors = plt.cm.rainbow(np.linspace(0, 1, n_simulations))
plt.figure(figsize=(20, 10))
plt.ylim(0, avg_best)

for i in range(n_simulations):

    plt.plot(simulations[i], c=colors[i])
plt.show()


I am working on the file "Misc experiments/ErgodicityProblem.ipynb".
The first task is:
try increasing the odds to gain 60%, lose 40% 50/50
Can you write Python code for it?

# change the odds....

# run 1 simulation
start_cash = 10000
savings = start_cash
n_trades = 1000

history = []

for i in range(n_trades):

    if np.random.randint(100) % 2 == 0:
        savings = savings * 1.5
    else:
        savings = savings * .6

    history.append(savings)

history = np.asarray(history)
time_to_max = np.argmax(history)
time_to_min = 0
max_value = np.max(history)

for i in range(len(history)):
    if history[i] > 1: time_to_min += 1
    else: break
print('Gain 60%, lose 40% odds 50/50')
print( 'Starting Savings $%d \nTime to max %d trades, value %.0f, Time to $1 %d trades' % (start_cash, time_to_max, max_value, time_to_min))

# plot

plt.figure(figsize=(20, 10))
plt.plot(history)
plt.show()


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:

Can you write Python code for it?

df.replace(' ?',np.nan,inplace=True)

df.loc[26,:]

df.shape

#We can also specify to only drop rows that are complete missing all data
df.dropna(inplace=True)

df.shape


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Decision Tree accuracy and time elapsed caculation
Can you write Python code for it?

t0=time()
print ("DecisionTree")

dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)
# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)

clf_dt=dt.fit(X_train,y_train)

print ("Acurracy: ", clf_dt.score(X_test,y_test))
t1=time()
print ("time elapsed: ", t1-t0)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
It is worth remembering the goal of this data mining activity: predicting wealth.
Using a combination of good features and feature selection, we can achieve 83
percent accuracy using just three features of a person!
Principal Components Analysis
Can you write Python code for it?

from sklearn.decomposition import PCA
pca = PCA(n_components=5)
Xd = pca.fit_transform(X)

np.set_printoptions(precision=3, suppress=True)
pca.explained_variance_ratio_


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
KNN accuracy and time elapsed caculation
Can you write Python code for it?

t6=time()
print ("KNN")
# knn = KNeighborsClassifier(n_neighbors=3)
knn = KNeighborsClassifier()
clf_knn=knn.fit(X_train, y_train)
print ("Acurracy: ", clf_knn.score(X_test,y_test) )
t7=time()
print ("time elapsed: ", t7-t6)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Naive Bayes accuracy and time elapsed caculation
Can you write Python code for it?

t4=time()
print ("NaiveBayes")
nb = BernoulliNB()
clf_nb=nb.fit(X_train,y_train)
print ("Acurracy: ", clf_nb.score(X_test,y_test))
t5=time()
print ("time elapsed: ", t5-t4)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Now all of these features are categorical classes, but most scikit-learn classifiers (in particular the Decision Trees we plan to use), expect real-valued attributes. We can easily convert sex  to a binary value (0=female,1=male). We will use the LabelEncoder class from scikit-learn:
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
enc = LabelEncoder()
label_encoder = enc.fit(df['Sex'])
print ("Categorical classes:", label_encoder.classes_)
integer_classes = label_encoder.transform(label_encoder.classes_)
print ("Integer classes:", integer_classes)
t = label_encoder.transform(df['Sex'])
df['Sex'] = t
print ('Feature names:',feature_names)
print ('Features:', df['Sex'][10:16])


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Now, we have to convert other categorical features. Since we have three different classes, we cannot convert to binary values (and using 0/1/2 values would imply an order, something we do not want). We use pandas get_dummies method for it or we can also use the OneHotEncoder to get three different attributes:
Can you write Python code for it?

categorical_features = []
for col in df.columns:
    if df[col].dtype != 'int64':
        categorical_features.append(col)
categorical_features

df.shape[0]

onehot_df = pd.get_dummies(df)

onehot_df.head()

onehot_df.columns

"""
from sklearn.preprocessing import OneHotEncoder

enc = LabelEncoder()

for f in categorical_features:

    label_encoder = enc.fit(df[f])
    print "Categorical classes:", label_encoder.classes_

    integer_classes = label_encoder.transform(label_encoder.classes_).reshape(
        len(pd.Series(df[f].values.ravel()).unique()), 1)
    print "Integer classes:", integer_classes

    enc = OneHotEncoder()
    one_hot_encoder = enc.fit(integer_classes)

    # First, convert clases to 0-(N-1) integers using label_encoder
    num_of_rows = df.shape[0]
    t = label_encoder.transform(df[f]).reshape(num_of_rows, 1)

    # Second, create a sparse matrix with three columns, each one indicating if the instance belongs to the class
    new_features = one_hot_encoder.transform(t)

    # Add the new features to df
    df = np.concatenate([df, new_features.toarray()], axis = 1)

    #Eliminate converted columns
    df.drop(f, axis=1, inplace=True)

    # Update feature names
    feature_names = ['age', 'sex', 'first_class', 'second_class', 'third_class']
    # Convert to numerical values
    df = df.astype(float)
    titanic_y = titanic_y.astype(float)
"""

features_list = list(onehot_df.columns)

features_list.remove('Earnings-Raw')

X = onehot_df[features_list]
y= onehot_df['Earnings-Raw']


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Random Forest accuracy and time elapsed caculation
Can you write Python code for it?

t2=time()
print ("RandomForest")
rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)
clf_rf = rf.fit(X_train,y_train)
print ("Acurracy: ", clf_rf.score(X_test,y_test))
t3=time()
print ("time elapsed: ", t3-t2)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Receiver Operating Characteristic (ROC) curve
Can you write Python code for it?

roc_auc_score(y_test,rf.predict(X_test))

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc

predictions = rf.predict_proba(X_test)

false_positive_rate, recall, thresholds = roc_curve(y_test, predictions[:, 1])
roc_auc = auc(false_positive_rate, recall)
plt.title('Receiver Operating Characteristic')
plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f' % roc_auc)
plt.legend(loc='lower right')

plt.plot([0, 1], [0, 1], 'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.ylabel('Recall')
plt.xlabel('Fall-out')
plt.show()


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Running fit_transform will call fit and then transform with the same dataset.
The result will create a new dataset, choosing only the best three features.
Let's look at the code:
Can you write Python code for it?

Xt_chi2 = transformer.fit_transform(X, y)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The Pearson value could be between -1 and 1. A value of 1 implies a
perfect correlation between two variables, while a value of -1 implies a
perfect negative correlation, that is, high values in one variable give low
values in the other and vice versa. Such features are really useful to have,
but would be discarded. For this reason, we have stored the absolute
value in the scores array, rather than the original signed value.
Can you write Python code for it?

transformer = SelectKBest(score_func=multivariate_pearsonr, k=3)
Xt_pearson = transformer.fit_transform(X, y)
print(transformer.scores_)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The adult file itself contains two blank lines at the end of the file. By default, pandas
will interpret the penultimate new line to be an empty (but valid) row. To remove
this, we remove any line with invalid numbers (the use of inplace just makes sure
the same Dataframe is affected, rather than creating a new one):
Can you write Python code for it?

df.dropna(how='all', inplace=True)

feature_names = df.columns


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The highest values are for the first, third, and fourth columns Correlates to the Age,
Capital-Gain, and Capital-Loss features. Based on a univariate feature selection,
these are the best features to choose.
We could also implement other correlations, such as the Pearson's correlation
coefficient. This is implemented in SciPy, a library used for scientific computing
(scikit-learn uses it as a base).
Can you write Python code for it?


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The preceding function almost fits the interface needed to be used in scikit-learn's
univariate transformers. The function needs to accept two arrays (x and y in our
example) as parameters and returns two arrays, the scores for each feature and the
corresponding p-values. The chi2 function we used earlier only uses the required
interface, which allowed us to just pass it directly to SelectKBest.
The pearsonr function in SciPy accepts two arrays; however, the X array it accepts is
only one dimension. We will write a wrapper function that allows us to use this for
multivariate arrays like the one we have. Let's look at the code:
Can you write Python code for it?

def multivariate_pearsonr(X, y):
    scores, pvalues = [], []
    for column in range(X.shape[1]):
        cur_score, cur_p = pearsonr(X[:,column], y)
        scores.append(abs(cur_score))
        pvalues.append(cur_p)
    return (np.array(scores), np.array(pvalues))


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The resulting matrix now only contains three features. We can also get the scores
for each column, allowing us to find out which features were used. Let's look at
the code:
Can you write Python code for it?


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
The results show each of the feature names that are stored inside an Index object
from pandas.
The Adult dataset contains several categorical features, with Work-Class being one
example. While we could argue that some values are of higher rank than others (for
instance, a person with a job is likely to have a better income than a person without),
it doesn't make sense for all values. For example, a person working for the state
government is not more or less likely to have a higher income than someone working
in the private sector.
We can view the unique values for this feature in the dataset using the
unique() function:
Can you write Python code for it?


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
There are some missing values in the preceding dataset, but they won't affect our
computations in this example.
Selecting the best individual features
If we have a number of features, the problem of finding the best subset is a difficult
task.
The scikit-learn package has a number of transformers for performing univariate
feature selection. They include SelectKBest, which returns the k best performing
features, and SelectPercentile, which returns the top r% of features. In both cases,
there are a number of methods of computing the quality of a feature.
There are many different methods to compute how effectively a single feature
correlates with a class value. A commonly used method is the chi-squared (χ2) test.
Other methods include mutual information and entropy.
We can observe single-feature tests in action using our Adult dataset. First, we
extract a dataset and class values from our pandas DataFrame. We get a selection of
the features:
Can you write Python code for it?

X = df[["Age", "Education-Num", "Capital-gain", "Capital-loss", "Hours-per-week"]].values


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
This returns a different set of features! The features chosen this way are the first,
second, and fifth columns: the Age, Education, and Hours-per-week worked. This
shows that there is not a definitive answer to what the best features are— it depends
on the metric.
We can see which feature set is better by running them through a classifier. Keep in
mind that the results only indicate which subset is better for a particular classifier
and/or feature combination—there is rarely a case in data mining where one method
is strictly better than another in all cases! Let's look at the code:
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
from sklearn.cross_validation import cross_val_score
clf = DecisionTreeClassifier(random_state=14)
scores_chi2 = cross_val_score(clf, Xt_chi2, y, scoring='accuracy')
scores_pearson = cross_val_score(clf, Xt_pearson, y, scoring='accuracy')

print("Chi2 performance: {0:.3f}".format(scores_chi2.mean()))
print("Pearson performance: {0:.3f}".format(scores_pearson.mean()))


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
Tuning Models using GridSearch
Can you write Python code for it?

pipeline2 = Pipeline([
('clf', RandomForestClassifier(criterion='entropy'))
])

parameters = {
    'clf__n_estimators': (5, 25, 50, 100),
    'clf__max_depth': (5, 25 , 50),
    'clf__min_samples_split': (2, 5, 10),
    'clf__min_samples_leaf': (1, 2, 3)
}

grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)

grid_search.fit(X_train, y_train)

print ('Best score: %0.3f' % grid_search.best_score_)

print ('Best parameters set:')
best_parameters = grid_search.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):
    print ('\t%s: %r' % (param_name, best_parameters[param_name]))

predictions = grid_search.predict(X_test)
print ('Accuracy:', accuracy_score(y_test, predictions))
print (classification_report(y_test, predictions))


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
cross validation for KNN
Can you write Python code for it?

tt6=time()
print ("cross result========")
scores = cross_validation.cross_val_score(knn, X, y, cv=5)
print (scores)
print (scores.mean())
tt7=time()
print ("time elapsed: ", tt7-tt6)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
cross-validation for NB
Can you write Python code for it?

tt4=time()
print ("cross result========")
scores = cross_validation.cross_val_score(nb, X, y, cv=5)
print (scores)
print (scores.mean())
tt5=time()
print ("time elapsed: ", tt5-tt4)


I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
feature_importances_
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import ExtraTreesClassifier

# Build a classification task using 3 informative features

# Build a forest and compute the feature importances
forest = ExtraTreesClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

importances[indices[:5]]

features = features_list
for f in range(5):
    print("%d. feature %d - %s (%f)" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))

best_features = []
for i in indices[:5]:
    best_features.append(features[i])

# Plot the top 5 feature importances of the forest
plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.title("Feature importances")
plt.bar(range(5), importances[indices][:5],
       color="r",  yerr=std[indices][:5], align="center")
plt.xticks(range(5), best_features)
plt.xlim([-1, 5])
plt.show()


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
After we have installed GraphViz on our computer, we can convert the tree.dot file into a PNG file by executing the following command from the command line in the location where we saved the tree.dot file:

dot -Tpng tree.dot -o tree.png

Random Forest accuracy and time elapsed caculation
Can you write Python code for it?


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
Decision Tree accuracy and time elapsed caculation
Can you write Python code for it?

t0=time()
print ("DecisionTree")

dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)
# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)

clf_dt=dt.fit(X_train,y_train)

print ("Acurracy: ", clf_dt.score(X_test,y_test))
t1=time()
print ("time elapsed: ", t1-t0)


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
Feature importances with forests of trees
This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability.
Can you write Python code for it?

features = index

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import ExtraTreesClassifier

# Build a classification task using 3 informative features

# Build a forest and compute the feature importances
forest = ExtraTreesClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(X.shape[1]):
    print("%d. feature %d - %s (%f) " % (f + 1, indices[f], features[indices[f]], importances[indices[f]]))

# Plot the feature importances of the forest
plt.figure(num=None, figsize=(14, 10), dpi=80, facecolor='w', edgecolor='k')
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices],
       color="r", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices)
plt.xlim([-1, X.shape[1]])
plt.show()

importances[indices[:5]]

for f in range(5):
    print("%d. feature %d - %s (%f)" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))

best_features = []
for i in indices[:5]:
    best_features.append(features[i])

# Plot the top 5 feature importances of the forest
plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.title("Feature importances")
plt.bar(range(5), importances[indices][:5],
       color="r",  yerr=std[indices][:5], align="center")
plt.xticks(range(5), best_features)
plt.xlim([-1, 5])
plt.show()


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
KNN accuracy and time elapsed caculation
Can you write Python code for it?

t6=time()
print( "KNN")
# knn = KNeighborsClassifier(n_neighbors=3)
knn = KNeighborsClassifier(n_neighbors=3)
clf_knn=knn.fit(X_train, y_train)
print ("Acurracy: ", clf_knn.score(X_test,y_test) )
t7=time()
print ("time elapsed: ", t7-t6)


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
Naive Bayes accuracy and time elapsed caculation
Can you write Python code for it?

t4=time()
print ("NaiveBayes")
nb = BernoulliNB()
clf_nb=nb.fit(X_train,y_train)
print ("Acurracy: ", clf_nb.score(X_test,y_test))
t5=time()
print ("time elapsed: ", t5-t4)


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
Tuning Models using GridSearch
Can you write Python code for it?


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
cross validation for DT
Can you write Python code for it?

tt0=time()
print ("cross result========")
scores = cross_validation.cross_val_score(dt, X, y, cv=5)
print (scores)
print (scores.mean())
tt1=time()
print ("time elapsed: ", tt1-tt0)


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
cross validation for RF
Can you write Python code for it?

tt2=time()
print ("cross result========")
scores = cross_validation.cross_val_score(rf, X, y, cv=5)
print (scores)
print (scores.mean())
tt3=time()
print ("time elapsed: ", tt3-tt2)


I am working on the file "Miscellaneous/Connect-4 Classification.ipynb".
The first task is:
cross validation for SVM
Can you write Python code for it?

tt7=time()
print ("cross result========")
scores = cross_validation.cross_val_score(svc,X,y, cv=5)
print (scores)
print (scores.mean())
tt8=time()
print ("time elapsed: ", tt7-tt6)


I am working on the file "Miscellaneous/Gesture-Phase-Detection.ipynb".
The first task is:

Can you write Python code for it?

X = df_1[features]
y = df_1['phase']
# split dataset to 60% training and 40% testing
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)


pca = PCA(n_components=2)
X = pca.fit_transform(X)

c = []
from matplotlib.pyplot import cm
n=6
color=iter(cm.rainbow(np.linspace(0,1,n)))
for i in range(n):
    c.append(next(color))

n = 5
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8,6))

km = KMeans(n_clusters= n , random_state=0)
y_km = km.fit_predict(X)

for i in range(n):
    ax1.scatter(X[y_km==i,0], X[y_km==i,1], c=c[i], marker='o', s=40, label='cluster{}'.format(i))
ax1.set_title('K-means clustering')

ac = AgglomerativeClustering(n_clusters=n, affinity='euclidean', linkage='complete')
y_ac = ac.fit_predict(X)
for i in range(n):
    ax2.scatter(X[y_ac==i,0], X[y_ac==i,1], c=c[i], marker='o', s=40, label='cluster{}'.format(i))
ax2.set_title('Agglomerative clustering')

# Put a legend below current axis
plt.legend()

plt.tight_layout()
#plt.savefig('./figures/kmeans_and_ac.png', dpi=300)
plt.show()


I am working on the file "Miscellaneous/Gesture-Phase-Detection.ipynb".
The first task is:
Bagging -- Building an ensemble of classifiers from bootstrap samples
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=None)

bag = BaggingClassifier(base_estimator=tree,
                        n_estimators=500,
                        max_samples=1.0,
                        max_features=1.0,
                        bootstrap=True,
                        bootstrap_features=False,
                        n_jobs=1,
                        random_state=1)

from sklearn.metrics import accuracy_score

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

bag = bag.fit(X_train, y_train)
y_train_pred = bag.predict(X_train)
y_test_pred = bag.predict(X_test)

bag_train = accuracy_score(y_train, y_train_pred)
bag_test = accuracy_score(y_test, y_test_pred)
print('Bagging train/test accuracies %.3f/%.3f'
      % (bag_train, bag_test))


I am working on the file "Miscellaneous/Gesture-Phase-Detection.ipynb".
The first task is:
Classification
Can you write Python code for it?

import os
from sklearn.tree import DecisionTreeClassifier, export_graphviz
import pandas as pd
import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn import cross_validation, metrics
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from time import time
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score , classification_report
from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report

X = df_1[features]
y = df_1['phase']
# split dataset to 60% training and 40% testing
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)

print (X_train.shape, y_train.shape,X_test.shape, y_test.shape)


I am working on the file "Miscellaneous/Gesture-Phase-Detection.ipynb".
The first task is:
K Means
Can you write Python code for it?

from sklearn import cluster
clf = cluster.KMeans(init='k-means++', n_clusters=5, random_state=5)
clf.fit(X_train)
print clf.labels_.shape
print clf.labels_

# Predict clusters on testing data
y_pred = clf.predict(X_test)

from sklearn import metrics
print "Addjusted rand score:{:.2}".format(metrics.adjusted_rand_score(y_test, y_pred))
print "Homogeneity score:{:.2} ".format(metrics.homogeneity_score(y_test, y_pred))
print "Completeness score: {:.2} ".format(metrics.completeness_score(y_test, y_pred))
print "Confusion matrix"
print metrics.confusion_matrix(y_test, y_pred)


I am working on the file "Miscellaneous/Gesture-Phase-Detection.ipynb".
The first task is:
Leveraging weak learners via adaptive boosting
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=1)

ada = AdaBoostClassifier(base_estimator=tree,
                         n_estimators=500,
                         learning_rate=0.1,
                         random_state=0)

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

ada = ada.fit(X_train, y_train)
y_train_pred = ada.predict(X_train)
y_test_pred = ada.predict(X_test)

ada_train = accuracy_score(y_train, y_train_pred)
ada_test = accuracy_score(y_test, y_test_pred)
print('AdaBoost train/test accuracies %.3f/%.3f'
      % (ada_train, ada_test))