I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
We assume different possible forms for the maintenance cost function:
$$\text{A linear form: } \qquad \text{MF}(s, \theta) = \theta_1 s \
\text{A quadratic form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}  s + \theta_{1 2}s^2 \
\text{An exponential form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}\exp(\theta_{1 2}s) \
\text{A logarithmic form: } \qquad \text{MF}(s, \theta) = \theta_{1 1}\log(\theta_{1 2}s)$$
Can you write Python code for it?

def lin_cost(s, params):
    try:
        theta1_1, = params
        return s*theta1_1
    except ValueError:
        raise ValueError
        print("Wrong number of parameters specified: expected 2, got {}".format(len(params)))

def quad_cost(s, params):
    try:
        theta1_1, theta1_2 = params
        return s*theta1_1 + (s**2)*theta1_2
    except ValueError:
        raise ValueError
        print("Wrong number of parameters specified: expected 2, got {}".format(len(params)))

def exp_cost(s, params):
    try:
        theta1_1, = params
        return np.exp(s*theta1_1)
    except ValueError:
        raise ValueError
        print("Wrong number of parameters specified: expected 1, got {}".format(len(params)))

def log_cost(s, params):
    try:
        theta1_1, theta1_2 = params
        return np.log(theta1_1 + s*theta1_2)
    except ValueError:
        raise ValueError
        print("Wrong number of parameters specified: expected 2, got {}".format(len(params)))


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
1 - Fit the training data using k-means clustering (without labels) using a number a clusters determined by the label values
Can you write Python code for it?

from sklearn.cluster import KMeans

km = KMeans(n_clusters=3)
km.fit(iris_train.iloc[:, :4])
print(km.labels_)
print(km.cluster_centers_)


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
2 - Create a script that will simulate this data as a feed
- number of defects will only be analyzed every 1000 observations
- each 1000 will count as one block
- each of these blocks will be classified as anomalous or not (as a whole)
- use a threshold of 0.01
Can you write Python code for it?

threshold = 0.01
n = 1000
p = 5/1000
n_blocks = int(np.ceil(len(errors) / 1000))

from scipy.stats import binom
distr = binom(p=p, n=n)

blocks = []
is_out = []
for i in np.arange(n_blocks):
    curr_count = errors[n*i:n*(i+1)].sum()[0]
    blocks.append(curr_count)
    is_out.append(distr.pmf(curr_count) < threshold)

print(is_out[:10])
print(blocks[:10])


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
3 - Create the following functions
- on that returns an estimated multivariate normal distribution for a given window of data
- another that determines if a single observation is an outlier and takes these params:
    - a single observation
    - the estimated distribution object
    - probability threshold: below which observation is outlier
Can you write Python code for it?

def estd_multi(data):
    cov = np.cov(data.values.T)
    mu = np.mean(data).values
    return multivariate_normal(mean=mu, cov=cov)

def is_outlier(obs, distr, threshold):
    prob = distr.pdf(obs)
    if prob < threshold:
        return 1, prob
    else:
        return 0, prob

multi = estd_multi(Xtrain.iloc[:100, ])
print(is_outlier(Xtrain.iloc[152, ], multi, threshold=0.01))
print(is_outlier(Xtrain.iloc[1523, ], multi, threshold=0.01))


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
3 - Use these percentiles to determine outliers in the test data and create a 3d plot using the first three attributes and highlight outliers
Can you write Python code for it?

pred = km.predict(iris_test.iloc[:, :4])
centers = np.array([list(km.cluster_centers_[lab]) for lab in pred])
distances = np.sqrt(np.sum((iris_test.iloc[:, :4] - centers)**2, axis=1))
is_out = [distances[i] > percentiles[lab] for i, lab in enumerate(pred)]

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(16, 10))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(iris_test.iloc[:, 0],
           iris_test.iloc[:, 1],
           iris_test.iloc[:, 2],
           color=['red' if out else 'blue' for out in is_out]);


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
5 - Create a single plot that has
- scatter plot of the test data highlighting detected outliers
- contour plot overlaid showing how the distribution pdf varies
Can you write Python code for it?

past = Xtest.iloc[:1000, ]
is_out, _ = live_feed(Xtest.iloc[1000:, ], past, 1000)

np.append(np.repeat(False, 1000), (np.array(is_out)))[1000:1100]

plt.figure(figsize=(16, 10))
plt.scatter(Xtest.iloc[np.append(np.repeat(False, 1000), (np.array(is_out)))==0, 0],
            Xtest.iloc[np.append(np.repeat(False, 1000), (np.array(is_out)))==0, 1], marker='o')
plt.scatter(Xtest.iloc[np.append(np.repeat(False, 1000), (np.array(is_out)))==1, 0],
            Xtest.iloc[np.append(np.repeat(False, 1000), (np.array(is_out)))==1, 1], marker='x')

# I'm drawing just 10 examples of the distribution in order to make the plot readable...
x = np.linspace(min(Xtest.iloc[:, 0])-0.5, max(Xtest.iloc[:, 0])+0.5, 1000)
y = np.linspace(min(Xtest.iloc[:, 1])-0.5, max(Xtest.iloc[:, 1])+0.5, 1000)
x, y = np.meshgrid(x, y)

# for i in range(10):

#     cov = np.cov(Xtest.iloc[i*2550:i*2550+10, ].values.T)
#     mu = np.mean(Xtest.iloc[i*2550:i*2550+10, ]).values

#     mult = multivariate_normal(mean=mu, cov=cov)
#     z = mult.pdf(np.array([x.ravel(), y.ravel()]).T).reshape(x.shape)

#     plt.contour(x, y, z, levels=[0.0001]);

cov = np.cov(Xtest.values.T)
mu = np.mean(Xtest).values

mult = multivariate_normal(mean=mu, cov=cov)
z = mult.pdf(np.array([x.ravel(), y.ravel()]).T).reshape(x.shape)

plt.contour(x, y, z);


I am working on the file "Data Mining/Anomaly Detection/anomaly_detection_problems.ipynb".
The first task is:
Anomaly Detection Using Cluster Analysis
This is a very common dataset (slightly modified) of three varieties of the iris flower; it will work well for clustering.
Can you write Python code for it?

iris_train = pd.read_csv('./data/iris_train.csv')
iris_test = pd.read_csv('./data/iris_test.csv')

iris_train.head()

iris_train.describe()


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
1. Check that the training data is suitable for a multivariate modeling approach

i.e. check that the data appears to be from a multivariate gaussian distribution
you don't need to get too fancy with tests here
Can you write Python code for it?

data_tr.describe()

from pandas.plotting import scatter_matrix

scatter_matrix(data_tr, hist_kwds={'bins':25}, figsize=(10,7));


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
1. Fit the training data using k-means clustering (without labels) using a number a clusters determined by the label values
Can you write Python code for it?

Xtr = train.iloc[:, :-1]
ytr = train.iloc[:, -1]

Xts = test.iloc[:, :-1]
yts = test.iloc[:, -1]

ytr.value_counts()

km = KM(n_clusters=3)
km.fit(Xtr)


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
1. Given the description and goal, what distribution should be used to model this data?
Poisson Distribution.
2. Create a script that will simulate this data as a feed and label anomalies

number of errors will only be analyzed every 1000 observations
each 1000 will count as one block
each of these blocks will be classified as anomalous or not (as a whole)
use a threshold of 0.01
Can you write Python code for it?

1e3*data.mean(), data.reshape(1000,-1).sum(axis=0).mean()

anomalies = []
fullset = []
thresh = 0.01
mu = 1e3*data.mean()
rv = st.poisson(mu=mu)
n = 0
for i,v in enumerate(data):
    if i%1e3 == 0:
        p = rv.pmf(n)
        fullset.append((i//1e3, n))
        if p < thresh:
            anomalies.append((i//1e3, n, p))
        n = 0
    if v == 1:
        n += 1
print('Stream Complete')

anomalies = DF(anomalies, columns=['block','n_errors','p-value']).astype({'block': int})
fullset = DF(fullset, columns=['block','n_errors']).astype({'block': int})

anomalies

fullset.head()


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
2. Check for any drift in the training data mean and std over time

if there is drift, determine what span of time (window) should used for modeling incoming data
Can you write Python code for it?


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
3. Create a plot with the following properties

x-axis: block number
y-axis: number of errors in the block
highlight blocks labeld as anomalies
Can you write Python code for it?

# fullset.n_errors.plot(figsize=(16,8), style='o')
outliers = fullset[fullset.block.isin(anomalies.block)].n_errors
inliers = fullset[~fullset.index.isin(anomalies.block)].n_errors

outliers.plot(figsize=(16,8), style='ro')
inliers.plot(figsize=(16,8), style='x');


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
3. Create the following functions

on that returns an estimated multivariate normal distribution for a given window of data
another that determines if a single observation is an outlier and takes these params:
a single observation
the estimated distribution object
probability threshold: below which observation is outlier
Can you write Python code for it?

# estimate and return multivariate gaussian distribution
def multi_gaus_dist(window):
    means = window.mean()
    cov = window.cov()
    dist = st.multivariate_normal(means, cov)
    return dist

# check if obs is outlier
def isoutlier(obs, dist, thresh=0.0001):
    p = dist.pdf(obs)
    if p < thresh:
        return True
    else:
        return False

dist = multi_gaus_dist(data_tr)
sum([isoutlier(point[1:], dist) for point in data.itertuples()])


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
3. Use these percentiles to determine outliers in the test data and create a 3d plot using the first three attributes and highlight outliers
Can you write Python code for it?

# outliers are those where distance from all centers is greater than respective percentiles
outlier = (distances_ts > quants).all(axis=1)
outliers = Xts[outlier]
inliers = Xts[~Xts.index.isin(outliers.index)]

outliers

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(1, figsize=(14, 8))
# create 3d axis object to use and set rotation for good view
ax = Axes3D(fig, elev=-150, azim=3)
ax.scatter(outliers['1'], outliers['2'], outliers['3'], c='r')
# map label numbers to color values
colors = yts[yts.index.isin(inliers.index)].map({0:'b', 1:'g', 2:'k'})
ax.scatter(inliers['1'], inliers['2'], inliers['3'], c=colors);


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
4. Create a function to simulate a live feed (a stream) of the test data

returns a boolean list indicating if each observation was an outlier
computes these one-by-one as they come in through simulated feed
each observation should be based on last N observations
N is the windows size determined in (2)

use the last N observations in the test data as a starting point
use a threshold of 0.0001
Can you write Python code for it?

combined_df = pd.concat([data_tr.iloc[-1000:], data], axis=0)

anomaly = []
for i in range(1000, len(combined_df)):
    window = combined_df.iloc[i-1000:i]
    dist = multi_gaus_dist(window)
    obs = combined_df.iloc[i]
    anomaly.append(isoutlier(obs, dist, thresh=0.00001))

outliers = data[anomaly]
inliers = data[~np.array(anomaly)]

outliers.shape, inliers.shape


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
5. Create a single plot that has

scatter plot of the test data highlighting detected outliers
contour plot overlaid showing how the distribution pdf varies
Can you write Python code for it?

mu = data.mean()
cov = data.cov()
mvd = st.multivariate_normal(mu, cov)
mesh = np.mgrid[data.X1.min()-1:data.X1.max()+1:.01,
               data.X2.min()-1:data.X2.max()+1:.01]
pos = np.empty(mesh[0].shape + (2,))
pos[:, :, 0] = mesh[0]
pos[:, :, 1] = mesh[1]

plt.figure(figsize=(14,10))
ax = plt.contour(mesh[0], mesh[1], mvd.pdf(pos), 10, cmap='bone')
plt.clabel(ax, inline=1, fontsize=12)

temp = data.copy()
temp['anomaly'] = anomaly
temp.loc[:, 'anomaly'] = temp.anomaly.map({True:'red', False:'#cccccc'})
plt.scatter(temp['X1'], temp['X2'], c=temp['anomaly'], alpha=0.5)
plt.xlabel('X1')
plt.ylabel('X2');


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
6. How do you think the threshold could be optimized?
One way to optimize the threshold would be to use historical data to label observations that, in hindsight, qualify as anomalies. With these labels you can use CV to tune the threshold until you get the best value on some metric (e.g. F1-score).
Detect Anomalous Number Of Defects
This dataset contains chronological data for a computer system. The data simply indicates if an error ocurred at a given time with a 1, otherwise a 0 is output. This system has a fixed expectation value of 5 errors per thousand. The goal is to detect if there were an abnormal number of errors for any given time-window.

Given the description and goal, what distribution should be used to model this data?
Create a script that will simulate this data as a feed
number of defects will only be analyzed every 1000 observations
each 1000 will count as one block
each of these blocks will be classified as anomalous or not (as a whole)
use a threshold of 0.01

Create a plot with the following properties
x-axis: block number
y-axis: number of errors in the block
highlight blocks labeld as anomalies
Can you write Python code for it?


I am working on the file "Data Mining/Anomaly Detection/solution/anomaly_detection_walkthrough.ipynb".
The first task is:
Anomaly Detection Using Cluster Analysis
This is a very common dataset (slightly modified) of three varieties of the iris flower; it will work well for clustering.

Fit the training data using k-means clustering (without labels) using a number a clusters determined by the label values
Get the 98th percentile for distances from cluster centers for the training data
one value for each label/class

Use these percentiles to determine outliers in the test data and create a 3d plot using the first three attributes and highlight outliers
Can you write Python code for it?

# read data
train = pd.read_csv('iris_train.csv')
test = pd.read_csv('iris_test.csv')
# import k-means
from sklearn.cluster import KMeans as KM


I am working on the file "Data Mining/Association Rules/assoc_mining_problems.ipynb".
The first task is:
2. For a bit of EDA:

create a script that plots the proportions of single items sold
plot the counts for the different itemset sizes
Can you write Python code for it?

items_counts = {}
for tran in transactions:
    for item in tran:
        if item in items_counts:
            items_counts[item] += 1
        else:
            items_counts[item] = 1

plt.bar(items_counts.keys(), items_counts.values());

itemsets_sizes = {}
for tran in transactions:
    tran = frozenset(tran)
    if tran in itemsets_sizes:
        itemsets_sizes[tran] += 1
    else:
        itemsets_sizes[tran] = 1

fig, ax = plt.subplots(figsize=(16, 8))
ax.bar(itemsets_sizes.keys(), itemsets_sizes.values())
ax.set_xticklabels([list(itemset) for itemset in itemsets_sizes.keys()], rotation='vertical');


I am working on the file "Data Mining/Association Rules/assoc_mining_problems.ipynb".
The first task is:
3. Create a function to compute support for a given itemset from a transaction
Can you write Python code for it?

def support_count(trans, itemset):
    count = 0
    for tran in trans:
        if set(itemset).issubset(set(tran)):
            count += 1
    return count

print(support_count(transactions, ['A']))
print(support_count(transactions, ['A', 'C']))
print(support_count(transactions, ['F']))

def support(trans, itemset):
    return support_count(trans, itemset) / len(trans)

print(support(transactions, ['A']))
print(support(transactions, ['A', 'C']))
print(support(transactions, ['F']))


I am working on the file "Data Mining/Association Rules/assoc_mining_problems.ipynb".
The first task is:
4. Write an apriori function that handles the pruning and generation of frequent itemsets

should return a dict of frequent itemsets as keys, and their support as values
must have a method for setting a minimum support value to determine frequent itemsets
hint: use sets and frozensets <- dict keys cant be mutable objects
Can you write Python code for it?

def generate_supsets(items, sets):
    sups = {}
    for s in sets:
        for item in items:
            sup = frozenset(s.union(item))
            if sup != s and sup not in sups:
                sups[sup] = 1
    return list(sups.keys())

def apriori(trans, minsupp):
    frequent_itemsets = {}
    # get items list
    items = list(set([item for tran in trans for item in tran]))
    # initialize list of itemsets to check
    curr_sets = items
    # iterate till max itemset length
    for i in range(len(items)):
        # print(i, curr_sets)
        # initialize candidates itemsets for generation of supsets
        next_sets = []
        # initialize current iteration unfrequent itemsets list
        unfrequent = []
        for s in curr_sets:
            supp = support(trans, s)
            # print(s, supp)
            # if we are over minsupp add itemset to frequent list and to supsets generation candidates
            if supp >= minsupp:
                frequent_itemsets[frozenset(s)] = supp
                next_sets.append(frozenset(s))
            #else add to unfrequent list
            else:
                unfrequent.append(frozenset(s))
        # if this is the first iteration update items list in order to optimize supsets generation
        if i == 0:
            items = [item for item in items if item not in unfrequent]
        # generate supsets and exclude those that contain an unfrequent itemset
        curr_sets = generate_supsets(items, next_sets)
        for unfr in unfrequent:
            curr_sets = [s for s in curr_sets if not unfr.issubset(s)]
        # print(next_sets)
        # print(unfrequent)
        # print(curr_sets)
        if len(curr_sets) == 0:
            break
    return frequent_itemsets

apriori(transactions, 0.1)


I am working on the file "Data Mining/Association Rules/assoc_mining_problems.ipynb".
The first task is:
5. Create a function that uses the frequent itemsets to generate association rules

must have a min confidence level as a parameter
do not keep rules that fall under this threshold

return dict of rules with string for keys
string key will take a form like: "{'A', 'B'} -> {'C'}"
values must be a dict containing metric:value pairs where metrics include support, confidence, and lift

return dataframe of rules with the following columns
antecedent, consequent, support, confidence, lift

Note: the support is for the entire set: antecedent union consequent
Can you write Python code for it?

def calc_metrics(trans, X, Y):
    supp = support(trans, X.union(Y))
    supp_X = support(trans, X)
    supp_Y = support(trans, Y)
    conf = supp / supp_X
    lift = supp / (supp_X*supp_Y)
    return conf, lift, supp

def generate_subsets(items, sets):
    subs = {}
    for s in sets:
        for item in items:
            sup = frozenset(s.difference(item))
            if sup != s and sup not in subs:
                subs[sup] = 1
    return list(subs.keys())

def association_rules(trans, frequent, minconf):
    rules = {}
    # get items list
    items = list(frequent)
    # initialize list of antecedents to check
    curr_antecedents = generate_subsets(items, [frequent])
    # iterate till itemset length - 1
    for i in range(len(frequent)-1):
        # print(i, curr_rules)
        # initialize candidates itemsets for generation of subsets
        next_antecedents = []
        # initialize current iteration unfrequent itemsets list
        unconfident = []
        for ant in curr_antecedents:
            cons = set(items).difference(ant)
            conf, lift, supp = calc_metrics(trans, ant, cons)
            # print(ant, conf)
            # if we are over minconf add rule to rules list and to subsets generation candidates
            if conf >= minconf:
                rule_ant = ', '.join('{}'.format(a) for a in (list(ant)))
                rule_cons = ', '.join('{}'.format(c) for c in (list(cons)))
                rule = '{{{}}}->{{{}}}'.format(rule_ant, rule_cons)
                metrics = {}
                metrics['confidence'] = conf
                metrics['lift'] = lift
                metrics['support'] = supp
                rules[rule] = metrics
                next_antecedents.append(frozenset(ant))
            #else add to unconfident list
            else:
                unconfident.append(frozenset(ant))
        # generate subsets and exclude those that are contained in an unconfident rule
        curr_antecedents = generate_subsets(items, next_antecedents)
        for uncf in...


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
10. Determine the Degree, Closeness, and Betweenness measures of centrality for this network
Can you write Python code for it?

# degree = number of adjacent nodes
g2.degree

# closeness centrality or closeness = reciprocal of the sum of the lengths of the shortest paths between
# the node and all other nodes divided by N-1 - the greater the more central
nx.closeness_centrality(g2)

# betweenness centrality or betweenness = sum of the number of shortest paths from all nodes different from v that
# pass through v divided by the number of shortest paths from all nodes different from v
nx.betweenness_centrality(g2)


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
11. Based on the measures above, which actors have the greatest control over the flow of information? Why do some actors have betweenness measures of zero?
Actors 0, 1 and 2 have the greatest control: they are more and more easily connected to others and, most of all, the betweenness tells us that most of the shortest paths from one person to another go through them.
Some actors have betweenness measures of 0 because no shortest path needs them: this may be because they have a low number of connection that are already connected to others, i.e. they are peripheral nodes of the graph and they don't know someone which isn't known by someone else.
12. Create a copy of the last network and do the following:

remove node 0
add and edge between 10 and 14 and remove the edge between 1 and 2
draw the graph
determine the eccentricity of node 1
find all cliques containing node 1
compute the density of the graph
remove node 1 and compute the density of the resulting graph
Can you write Python code for it?

g2_copy = g2.copy()
g2_copy.remove_node(0)
g2_copy.add_edge(10, 14)
g2_copy.remove_edge(1, 2)
draw_with_labels(g2_copy)

# eccentricity = greatest geodesic distance between v and any other node
nx.eccentricity(g2_copy, 1)

# for mc in nx.find_cliques(g2_copy):
#     if 1 in mc:
#         print(mc)
nx.cliques_containing_node(g2_copy, 1)

# density = how much the number of edges is close to the maximum possible (complete graph)
nx.density(g2_copy)

g2_copy.remove_node(1)
nx.density(g2_copy)


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
13. Create a directed graph from edges_3.pkl and do the following:

create the adjacency matrix
compute the indegree and outdegree centrality for all actors in this network using the adjacency matrix
compare your results to the in and out_degree_centrality methods
Can you write Python code for it?

edgelist = nx.read_gpickle('edges_3.pkl')
g3 = nx.DiGraph()
g3.add_edges_from(edgelist)
g3.nodes

draw_with_labels(g3)

# create the adjacency matrix
adj = nx.adjacency_matrix(g3)
adj = adj.todense()
# node 1 comes before node 0
adj

# in-out degree centrality = fraction of number of nodes incoming-outcoming edges are connected to
n_edges = len(g3.edges)
insum = np.sum(adj, axis=0).tolist()[0]
outsum = np.sum(adj, axis=1).reshape(1, -1).tolist()[0]
# remember that node 1 comes before node 0
insum[0], insum[1] = insum[1], insum[0]
outsum[0], outsum[1] = outsum[1], outsum[0]

[d / n_edges for d in insum]

[d / n_edges for d in outsum]

# compare your results to the in and out_degree_centrality methods
nx.in_degree_centrality(g3)

nx.out_degree_centrality(g3)


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
2. Create a function that will draw a given graph that has a layout type parameter and labels the nodes. Now draw the graph created in the last problem using circular layout.
Can you write Python code for it?

def draw_with_labels(graph, layout_type=None, **kwargs):
    plt.figure(figsize=(16, 10))
    if layout_type == None:
        nx.draw(graph, with_labels=True)
    elif layout_type == 'circular':
        nx.draw_circular(graph, with_labels=True)
    elif layout_type == 'random':
        nx.draw_random(graph, with_labels=True)
    elif layout_type == 'spectral':
        nx.draw_spectral(graph, with_labels=True)
    elif layout_type == 'shell':
        nx.draw_shell(graph, with_labels=True, **kwargs)

draw_with_labels(g, 'circular')

draw_with_labels(g, 'shell', nlist=[range(4, 7), range(4)])


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
8. Identify and remove a cutpoint from this graph and re-draw it
A cutpoint (whatever this means mathematically, I thing nothing though, it should be a cut set) could be the edge between 5 and 6:
Can you write Python code for it?

cut = [(5, 6)]
g1.remove_edges_from(cut)
draw_with_labels(g1)


I am working on the file "Data Mining/Network Analysis/network_analysis_problems.ipynb".
The first task is:
9. Use edges_2 to create a graph. List any subgraphs that are maximal cliques
Can you write Python code for it?

edgelist = nx.read_gpickle('edges_2.pkl')
g2 = nx.Graph()
g2.add_edges_from(edgelist)
g2.nodes

draw_with_labels(g2)

# clique = subgraph such that every two nodes are adjacent (i.e. it is a complete graph if considered by itself)
# maximal clique = a clique which isn't contained in any other clique
for mc in nx.find_cliques(g2):
    print(mc)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
1. Create a complete graph with 7 nodes and verify that it is complete by looking at the edges. Do this manually and using a built-in method.
Remember, a complete graph is an undirected graph that has every pair of nodes connected by a unique edge. In other words, every node is adjacent to every other.
Can you write Python code for it?

# manually
from itertools import combinations
complete_edges = combinations(range(7), 2)
G_complete = nx.Graph(complete_edges)

G_complete.edges()

# built-in method
G_complete = nx.complete_graph(7)
G_complete.edges()


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
10. Determine the Degree, Closeness, and Betweenness measures of centrality for this network
Can you write Python code for it?

degree = nx.degree_centrality(G)
closeness = nx.closeness_centrality(G)
betweenness = nx.betweenness_centrality(G)

Series(degree)

Series(closeness)

Series(betweenness)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
11. Based on the measures above, which actors have the greatest control over the flow of information? Why do some actors have betweenness measures of zero?
When considering control over flow of information, we look at the betweenness measure. In this case, actors 0, 1, and 2 have the greatest control over information flow. The actors with betweenness measures of zero have no geodesics through them connecting other pairs of nodes. For example, there is a path from 0 to 4 that goes through 9, but that has a length of 2 and 0 connects to 4 directly.
12. Create a copy of the last network and do the following:

remove node 0
add and edge between 10 and 14 and remove the edge between 1 and 2
draw the graph
determine the eccentricity of node 1
find all cliques containing node 1
compute the density of the graph
remove node 1 and compute the density of the resulting graph
Can you write Python code for it?

H = G.copy()
H.remove_node(0)
H.add_edge(10,14)
H.remove_edge(1,2)
# draw(H, graphviz_layout)
draw(H, nx.circular_layout)

# eccentricity of node 1
nx.eccentricity(H, 1)

# find cliques containing node 1
nx.cliques_containing_node(H, 1)

# density
nx.density(H)

# remove node 1
H.remove_node(1)
nx.density(H)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
13. Create a directed graph from edges_3.pkl and do the following:

create the adjacency matrix
compute the indegree and outdegree centrality for all actors in this network using the adjacency matrix
compare your results to the in and out_degree_centrality methods
Can you write Python code for it?

with open('edges_3.pkl', 'rb') as f:
    edges = pickle.load(f)

G = nx.DiGraph(edges)
draw(G, graphviz_layout)

# adjacency matrix
adj_matrix = nx.to_numpy_matrix(G)
DF(adj_matrix)

# indegree and outdegree
indegree = adj_matrix.sum(axis=0) / (len(adj_matrix)-1)
outdegree = adj_matrix.sum(axis=1) / (len(adj_matrix)-1)

in_method = nx.in_degree_centrality(G)
out_method = nx.out_degree_centrality(G)

# indegree comparison
(Series(np.array(indegree).flatten()) == Series(in_method)).all()

# outdegree comparison
(Series(np.array(outdegree).flatten()) == Series(out_method)).all()

Series(in_method).sort_values(ascending=False)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
3. Create a graph with 10 nodes and 3 components. Draw this graph.
Recall that a component is a connected subgraph.
Can you write Python code for it?

from networkx.drawing.nx_agraph import graphviz_layout

edges_1 = [(0,1), (0,2)]
edges_2 = [(3,4), (3,5), (4,5), (4,6)]
edges_3 = [(7,8), (7,9)]

G = nx.Graph(edges_1 + edges_2 + edges_3)

draw(G, graphviz_layout)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
4. Create a simple connected digraph with 4 nodes and a diameter of 2.
Can you write Python code for it?

# create in and out edge lists
out_edges = [(0,1), (0,2), (1,2), (2,3)]
# create the empty digraph
G = nx.DiGraph(out_edges)

draw(G, graphviz_layout)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
6. Create the adjacnency matrix from the edge data in edges_1.pkl
Can you write Python code for it?

import pickle
with open('../edges_1.pkl', 'rb') as f:
    edges = pickle.load(f)

G = nx.Graph(edges)

# lets see what it looks like
draw(G, graphviz_layout)

adj_matrix = nx.to_numpy_matrix(G)
DF(adj_matrix)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
9. Use edges_2 to create a graph. List any subgraphs that are maximal cliques
Can you write Python code for it?

# G = nx.moebius_kantor_graph()
G = nx.dorogovtsev_goltsev_mendes_graph(3)

with open('../edges_2.pkl', 'rb') as f:
    edges = pickle.load(f)

G = nx.Graph(edges)

# draw(G, graphviz_layout)
draw(G, nx.circular_layout)

list(nx.find_cliques(G))


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
And we can add other attributes
Can you write Python code for it?


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
Edge properties
We can add weights or other properties to edges in a graph in different ways. The first is to add properties at creation time by passing triples instead of doubles for each edge. The third value will be the edge property.
Can you write Python code for it?

G = nx.Graph()
G.add_nodes_from([1,2,3])
G.add_weighted_edges_from([(1,2,3.14), (2,3,6.5)])

# calling edges() alone will not return weights
print(G.edges(), '\n')

# we need to use the data parameter to get triples
print(G.edges(data='weight'), '\n')

# we can also get data for individual edges
print(G.get_edge_data(1,2))


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
Notice that removing nodes automatically removed the related edges for us. One last basic inspection method is to get a list of neighbors (adjacent nodes) for a specific node in a graph.
Can you write Python code for it?

H = nx.path_graph(7)
# get the neighbors for node 5
H.neighbors(5)


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
Notice that the edge list is empty, since we haven't added any edges yet. Also, because the number of edges in a graph can become very large, there is an iterator method for returning edges: edges_iter().
Can you write Python code for it?

# add a single edge between 3 and 5
G.add_edge(3,5)
# add multiple edges using list of tuples
edge_list = [(1,2),(2,3),(2,5)]
G.add_edges_from(edge_list)

G.edges()


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
Subscript notation for accessing edges
We can use subscript notiation on a graph object to easily get edge data. Access edge data for a node by entering that node as the subscript. This will return a dict with connected nodes as keys, and their respective edge weights as values.
Can you write Python code for it?

# get edge data for node 2
print(G[2], '\n')

# subscript further to get only the weight for edge between 2 and 3
print(G[2][3])


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
We can also add nodes and edges using objects called nbunch and ebunch. These objects are any iterables or generators of nodes or edge tuples. We will do this below using a couple different methods.
Can you write Python code for it?

# generate a graph of linearly connected nodes
# this is a graph of a single path with 5 nodes and 4 edges
H = nx.path_graph(5)
# a look at the nodes and edges produced
H.nodes(), H.edges()

# Create a graph using nbunch and ebunch from the graph H
G = nx.Graph()
G.add_nodes_from(H)
# we have to specify edges
G.add_edges_from(H.edges())

G.nodes(), G.edges()

# now add edges to a graph using an iterator instead of an iterable list
# this is another example of ebunch, and node iterators work too
G = nx.Graph()
G.add_nodes_from([1,2,3])
# create edge generator connecting all possible node pairs
from itertools import combinations
edge_generator = combinations([1,2,3], 2)
# to show this is a generator and not a list
print('not a list: ', edge_generator)

# now lets add the edges using the iterator
G.add_edges_from(edge_generator)
G.edges()


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
We can also remove nodes or edges using similar methods, just replacing 'add' with 'remove':
Can you write Python code for it?

H.nodes(), H.edges()

H.remove_nodes_from([0,4])
H.nodes(), H.edges()


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
While we have been using numbers to represent nodes, we can use any hashable object as a node. For example, this means that lists, sets and arrays can't be nodes, but frozensets can:
Can you write Python code for it?

G = nx.Graph()
# G.add_node([0,1]) <-- raises error
# G.add_node({0,1}) <-- raises error
G.add_node(frozenset([0,1]))  # this works
G.nodes()


I am working on the file "Data Mining/Network Analysis/solution/network_analysis_walkthrough.ipynb".
The first task is:
Working with Graphs using Networkx Intro
When creating a graph object, it can either be empty (default) or you can pass data as an argument. The data can take multiple forms:

an edge list
a numpy matrix or 2D ndarray
a Networkx graph object
a scipy sparse matrix
a PyGraphviz graph

Let's start by creating an empty graph, and then add nodes.
Can you write Python code for it?

# isntantiate a graph object
G = nx.Graph()

# add a single node
G.add_node(1)
# add multiple nodes from a list
G.add_nodes_from([2,3,5])

# return lists of nodes and edges in the graph
G.nodes(), G.edges()


I am working on the file "Data Mining/Recommender Systems/Recommender-Engine_exercise.ipynb".
The first task is:
Cosine Similarity
Building a recommendation engine can be thought of as "filling in the holes" in the sparse matrix you made above.  For example, take a look at the MovieLense data.  You'll see that that matrix is mostly zeros.  Our task here is to predict what a given user will rate a given movie depending on the users tastes.  To determine a users taste, we can use cosine similarity which is given by $$s_u^{cos}(u_k,u_a)
 = \frac{ u_k \cdot u_a }{ \left \| u_k \right \| \left \| u_a \right \| }
 = \frac{ \sum x_{k,m}x_{a,m} }{ \sqrt{\sum x_{k,m}^2\sum x_{a,m}^2} }$$
for users $u_a$ and $u_k$ on ratings given by $x_{a,m}$ and $x_{b,m}$.  This is just the cosine of the angle between the two vectors.  Likewise, this can also be calculated for the similarity between two items, $i_a$ and $i_b$, given by $$s_u^{cos}(i_m,i_b)
 = \frac{ i_m \cdot i_b }{ \left \| i_m \right \| \left \| i_b \right \| }
 = \frac{ \sum x_{a,m} x_{a,b} }{ \sqrt{ \sum x_{a,m}^2 \sum x_{a,b}^2 } }$$
Then, the similarity between two users is given by $$\hat{x}{k,m} = \bar{x}{k} + \frac{\sum\limits_{u_a} s_u^{cos}(u_k,u_a) (x_{a,m})}{\sum\limits_{u_a}|s_u^{cos}(u_k,u_a)|}$$ and for items given by $$\hat{x}{k,m} = \frac{\sum\limits{i_b} s_u^{cos}(i_m,i_b) (x_{k,b}) }{\sum\limits_{i_b}|s_u^{cos}(i_m,i_b)|}$$
Use these ideas to construct a class cos_engine which can be used to recommend movies for a given user.  Be sure to also test your algorithm, reporting its accuracy.  Bonus: Use adjusted cosine similiarity.
Can you write Python code for it?

class cos_engine():
    def __init__(self, k, how='user'):
        self.k = k
        self.how = how

    def _cos_sim(self, x, y):
        num = np.sum(x * y)
        den = np.sum(x**2) * np.sum(y**2)
        return num / np.sqrt(den)

    def _user_based_filtering(self, user, X):
        sim = X.apply(lambda row: self._cos_sim(X.loc[user, :].values, row.values), axis=1)
        sim = sim.drop(user, axis=0)
        rec = sim.sort_values(ascending=False).head(self.k)
        return rec

    def _item_based_filtering(self, item, X):
        Xt = X.T
        sim = Xt.apply(lambda row: self._cos_sim(Xt.loc[item, :].values, row.values), axis=1)
        sim = sim.drop(item, axis=0)
        rec = sim.sort_values(ascending=False).head(self.k)
        return rec

    def fit(self, X):
        if self.how == 'user':
            Xfilled = []
            for user in X.index.values:
                ubf = self._user_based_filtering(user, X)
                weighted_sum = np.sum(X.loc[ubf.index, :].values * ubf.values.reshape(-1, 1), axis=0)
                Xfilled.append(weighted_sum / np.sum(np.abs(ubf.values)))
            self.Xfilled = pd.DataFrame(np.array(Xfilled), index=X.index, columns=X.columns)
        elif self.how == 'item':
            Xtfilled = []
            for item in X.columns.values:
                ibf = self._item_based_filtering(item, X)
                Xt = X.T
                weighted_sum = np.sum(Xt.loc[ibf.index, :].values * ibf.values.reshape(-1, 1), axis=0)
                Xtfilled.append(weighted_sum / np.sum(np.abs(ibf.values)))
            self.Xfilled = pd.DataFrame(np.array(Xtfilled).T, index=X.index, columns=X.columns)
        else:
            print('Use either "user" or "item" for the parameter "how".')

    def predict(self, user, X):
        try:
            ratings = self.Xfilled.loc[user, :]
            rated_items = X.loc[user, :] > 0
            ratings = ratings[~rated_items]
            return ratings.sort_values(ascending=False)
        except...


I am working on the file "Data Mining/Recommender Systems/solution/Recommender-Engine_solution.ipynb".
The first task is:
Before building any recommendation engines, we'll have to get the data into a useful form.  Do this by first splitting the data into testing and training sets, and then by constructing two new dataframes whose columns are each unique movie and rows are each unique user, filling in 0 for missing values.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

# First split into train and test sets
data_train_raw, data_test_raw = train_test_split(data_movie_raw, train_size = 0.8)

# Turning to pivot tables
data_train = data_train_raw.pivot_table(index = 'user_id', columns = 'item_id', values = 'rating').fillna(0)
data_test = data_test_raw.pivot_table(index = 'user_id', columns = 'item_id', values = 'rating').fillna(0)

# Print the firest few rows
data_train.head()


I am working on the file "Data Mining/Recommender Systems/solution/Recommender-Engine_solution.ipynb".
The first task is:
Now split the data into a training and test set, using a ratio 80/20 for train/test.
Cosine Similarity
Building a recommendation engine can be thought of as "filling in the holes" in the sparse matrix you made above.  For example, take a look at the MovieLense data.  You'll see that that matrix is mostly zeros.  Our task here is to predict what a given user will rate a given movie depending on the users tastes.  To determine a users taste, we can use cosine similarity which is given by $$s_u^{cos}(u_k,u_a)
 = \frac{ u_k \cdot u_a }{ \left \| u_k \right \| \left \| u_a \right \| }
 = \frac{ \sum x_{k,m}x_{a,m} }{ \sqrt{\sum x_{k,m}^2\sum x_{a,m}^2} }$$
for users $u_a$ and $u_k$ on ratings given by $x_{a,m}$ and $x_{b,m}$.  This is just the cosine of the angle between the two vectors.  Likewise, this can also be calculated for the similarity between two items, $i_a$ and $i_b$, given by $$s_u^{cos}(i_m,i_b)
 = \frac{ i_m \cdot i_b }{ \left \| i_m \right \| \left \| i_b \right \| }
 = \frac{ \sum x_{a,m} x_{a,b} }{ \sqrt{ \sum x_{a,m}^2 \sum x_{a,b}^2 } }$$
Then, the similarity between two users is given by $$\hat{x}{k,m} = \bar{x}{k} + \frac{\sum\limits_{u_a} s_u^{cos}(u_k,u_a) (x_{a,m})}{\sum\limits_{u_a}|s_u^{cos}(u_k,u_a)|}$$ and for items given by $$\hat{x}{k,m} = \frac{\sum\limits{i_b} s_u^{cos}(i_m,i_b) (x_{k,b}) }{\sum\limits_{i_b}|s_u^{cos}(i_m,i_b)|}$$
Use these ideas to construct a class cos_engine which can be used to recommend movies for a given user.  Be sure to also test your algorithm, reporting its accuracy.  Bonus: Use adjusted cosine similiarity.
Can you write Python code for it?

# Libraries
import pandas as pd
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class cos_engine:
    def __init__(self, data_all):
        """Constructor for cos_engine class

        Args:
            data_all: Raw dataset containing all movies to build
                    a list of movies already seen by each user.
        """
        # Create copy of data
        self.data_all = data_all.copy()

        # Now build a list of movies each of you has seen
        self.seen = []
        for user in data_all.user_id.unique():
            cur_seen = {}
            cur_seen["user"] = user
            cur_seen["seen"] = self.data_all[data_all.user_id == user].item_id
            self.seen.append(cur_seen)

    def fit(self, data_train):
        """Performs cosine similarity on a sparse matrix data_train

        Args:
            data_train: A pandas data frame data to estimate cosine similarity
        """
        # Create a copy of the dataframe
        self.data_train = data_train.copy()

        # Save the indices and column names
        self.users = self.data_train.index
        self.items = self.data_train.columns

        # Compute mean vectors
        self.user_means = self.data_train.replace(0, np.nan).mean(axis = 1)
        self.item_means = self.data_train.T.replace(0, np.nan).mean(axis = 1)

        # Get similarity matrices and compute sums for normalization
        # For non adjusted cosine similarity, neglect subtracting the means.
        self.data_train_adj = (self.data_train.replace(0, np.nan).T - self.user_means).fillna(0).T
        self.user_cos = cosine_similarity(self.data_train_adj)
        self.item_cos = cosine_similarity(self.data_train_adj.T)

        self.user_cos_sum = np.abs(self.user_cos).sum(axis = 1)
        self.item_cos_sum = np.abs(self.item_cos).sum(axis = 1)

        self.user_cos_sum = self.user_cos_sum.reshape(self.user_cos_sum.shape[0], 1)
       ...


I am working on the file "Data Munging Practice - Cylinder Bands Data Set.ipynb".
The first task is:
Automating Data Extraction
Can you write Python code for it?

import os
import urllib

root_url = "https://archive.ics.uci.edu/ml/machine-learning-databases/"
full_url = root_url + "cylinder-bands/bands.data"
dataset_location = os.path.join("datasets","cylinder-bands")
filename = 'cylinder-bands.csv'

def fetch_data(filename, full_url=full_url, dataset_location=dataset_location):
    if not os.path.isdir(dataset_location):
        os.makedirs(dataset_location)
    file_path = os.path.join(dataset_location, filename)
    urllib.request.urlretrieve(full_url, file_path)

fetch_data("cylinder-bands.csv")

def load_data(filename, dataset_location=dataset_location):
    csv_path = os.path.join(dataset_location, filename)
    return pd.read_csv(csv_path, header=None)

cylinder_bands = load_data("cylinder-bands.csv")
cylinder_bands.head()

attributes = [
    'timestamp',
    'cylinder number',
    'customer',
    'job number',
    'grain screened',
    'ink color',
    'proof on ctd ink',
    'blade mfg',
    'cylinder divisional',
    'paper type',
    'ink type',
    'direct stream',
    'solvent type',
    'type on cylinder',
    'press type',
    'press',
    'unit number',
    'cylinder size',
    'paper mill location',
    'plating tank',
    'proof cut',
    'viscosity',
    'caliper',
    'ink temperature',
    'humidity',
    'roughness',
    'blade pressure',
    'varnish pct',
    'press speed',
    'ink pct',
    'solvent pct',
    'ESA Voltage',
    'ESA Amperage',
    'wax',
    'hardener',
    'roller durometer',
    'current density',
    'anode space ratio',
    'chrome content',
    'band type',
]

len(attributes)

cylinder_bands.columns = attributes

cylinder_bands.head()


I am working on the file "Data Munging Practice - Cylinder Bands Data Set.ipynb".
The first task is:
Preprocessing Pipelines
Can you write Python code for it?

from category_encoders import OrdinalEncoder, OneHotEncoder

from sklearn.base import BaseEstimator, TransformerMixin

# Create a class to select numerical or categorical columns
# since Scikit-Learn doesn't handle DataFrames yet
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names].values


I am working on the file "Data Munging Practice - Cylinder Bands Data Set.ipynb".
The first task is:
Reformatting Data Types
Can you write Python code for it?

cylinder_bands.info()

dt_attributes = [
    'timestamp',
]

cat_attributes = [
    'cylinder number',
    'customer',
    'job number',
    'grain screened',
    'ink color',
    'proof on ctd ink',
    'blade mfg',
    'cylinder divisional',
    'paper type',
    'ink type',
    'direct stream',
    'solvent type',
    'type on cylinder',
    'press type',
    'press',
    'unit number',
    'cylinder size',
    'paper mill location',
    'plating tank',
]

num_attributes = [
    'proof cut',
    'viscosity',
    'caliper',
    'ink temperature',
    'humidity',
    'roughness',
    'blade pressure',
    'varnish pct',
    'press speed',
    'ink pct',
    'solvent pct',
    'ESA Voltage',
    'ESA Amperage',
    'wax',
    'hardener',
    'roller durometer',
    'current density',
    'anode space ratio',
    'chrome content',
]

def contextual_convert(df, dt_attributes=dt_attributes, num_attributes=num_attributes, cat_attributes=cat_attributes):
    attributes = dt_attributes + num_attributes + cat_attributes
    for att in attributes:
        if att in dt_attributes:
            df[att] = pd.to_datetime(df[att], format="%Y%m%d", errors='coerce')
        elif att in num_attributes:
            df[att] = pd.to_numeric(df[att], errors='coerce')
        elif att in cat_attributes:
            df[att] = df[att].astype('category')
        else:
            pass

contextual_convert(cylinder_bands)

cylinder_bands[dt_attributes].info()

cylinder_bands[cat_attributes].info()

cylinder_bands[num_attributes].info()

cylinder_bands['customer'].str.upper()

cylinder_bands['customer'].value_counts()

def capitalize_columns(df, cols):
    for col in cols:
        df[col] = df[col].str.upper()

cylinder_bands['customer'] = cylinder_bands['customer'].str.upper()

capitalize_columns(cylinder_bands, ['customer', 'cylinder number'])


I am working on the file "Data Munging Practice - Cylinder Bands Data Set.ipynb".
The first task is:
Scratch Work
Can you write Python code for it?

cylinder_bands['cylinder size'].value_counts()

cylinder_bands['plating tank'].value_counts()

cylinder_bands[cylinder_bands['cylinder size'] == "0.7"]

raw_cylinder_bands = load_data(filename)

raw_cylinder_bands.iloc[522]


I am working on the file "Data Mungining Done.ipynb".
The first task is:
If we want to look at covariates, we need a new approach.
We'll use Cox proprtional hazards, a very popular regression model.
To fit in python we use the module lifelines:
http://lifelines.readthedocs.io/en/latest/
Can you write Python code for it?


I am working on the file "Data Mungining Done.ipynb".
The first task is:
Model selection
Difficult to do with classic tools (here)
Problem:
1 - Calculate the BMA coefficient values

2 - Try running with different priors
Can you write Python code for it?

#### BMA Coefficient values

#### Different priors


I am working on the file "Data Mungining Done.ipynb".
The first task is:
Once we've fit the data, we need to do something useful with it.  Try to do the following things:
1 - Plot the baseline survival function

2 - Predict the functions for a particular set of features

3 - Plot the survival function for two different set of features

4 - For your results in part 3 caculate how much more likely a death event is for one than the other for a given period of time
Can you write Python code for it?

#### Plot baseline hazard function

#### Predict

#### Plot survival functions for different covariates

#### Plot some odds


I am working on the file "Data Mungining Done.ipynb".
The first task is:
Problems:
4 - Try adjusting the number of samples for burning and thinnning
5 - Try adjusting the prior and see how it affects the estimate
Can you write Python code for it?

#### Adjust burn and thin, both paramters of the mcmc sample function

#### Narrow and broaden prior


I am working on the file "Data Mungining Done.ipynb".
The first task is:
Problems:
7 - Try testing whether the median is greater than a different values
Can you write Python code for it?


I am working on the file "Data Mungining Done.ipynb".
The first task is:
The first step in any data analysis is acquiring and munging the data
Our starting data set can be found here:
   http://jakecoltman.com in the pyData post
It is designed to be roughly similar to the output from DCM's path to conversion
Download the file and transform it into something with the columns:
id,lifetime,age,male,event,search,brand
where lifetime is the total time that we observed someone not convert for and event should be 1 if we see a conversion and 0 if we don't.  Note that all values should be converted into ints
It is useful to note that end_date = datetime.datetime(2016, 5, 3, 20, 36, 8, 92165)
Can you write Python code for it?

running_id = 0
output = [[0]]
with open("E:/output.txt") as file_open:
    for row in file_open.read().split("\n"):
        cols = row.split(",")
        if cols[0] == output[-1][0]:
            output[-1].append(cols[1])
            output[-1].append(True)
        else:
            output.append(cols)
    output = output[1:]

for row in output:
    if len(row) == 6:
        row += [datetime(2016, 5, 3, 20, 36, 8, 92165), False]
output = output[1:-1]

def convert_to_days(dt):
    day_diff = dt / np.timedelta64(1, 'D')
    if day_diff == 0:
        return 23.0
    else:
        return day_diff

df = pd.DataFrame(output, columns=["id", "advert_time", "male","age","search","brand","conversion_time","event"])
df["lifetime"] = pd.to_datetime(df["conversion_time"]) - pd.to_datetime(df["advert_time"])
df["lifetime"] = df["lifetime"].apply(convert_to_days)
df["male"] = df["male"].astype(int)
df["search"] = df["search"].astype(int)
df["brand"] = df["brand"].astype(int)
df["age"] = df["age"].astype(int)
df["event"] = df["event"].astype(int)
df = df.drop('advert_time', 1)
df = df.drop('conversion_time', 1)
df = df.set_index("id")
df = df.dropna(thresh=2)
df.median()

###Parametric Bayes
#Shout out to Cam Davidson-Pilon

## Example fully worked model using toy data
## Adapted from http://blog.yhat.com/posts/estimating-user-lifetimes-with-pymc.html
## Note that we've made some corrections

N = 2500

##Generate some random data
lifetime = pm.rweibull( 2, 5, size = N )
birth = pm.runiform(0, 10, N)
censor = ((birth + lifetime) >= 10)
lifetime_ = lifetime.copy()
lifetime_[censor] = 10 - birth[censor]

alpha = pm.Uniform('alpha', 0, 20)
beta = pm.Uniform('beta', 0, 20)

@pm.observed
def survival(value=lifetime_, alpha = alpha, beta = beta ):
    return sum( (1-censor)*(log( alpha/beta) + (alpha-1)*log(value/beta)) - (value/beta)**(alpha))

mcmc = pm.MCMC([alpha, beta, survival ] )
mcmc.sample(50000, 30000)

pm.Matplot.plot(mcmc)
mcmc.trace("alpha")[:]


I am working on the file "Data Plots New.ipynb".
The first task is:
Plot the data in 2D
Can you write Python code for it?

%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt

plt.figure(1,figsize=(12,8))
plt.xlabel('Print Time (mins)')
plt.ylabel('Filament (m)')
plt.title('Prints (Combined)')

plt.figure(2,figsize=(12,8))
plt.xlabel('Print Time (mins)')
plt.ylabel('Filament (m)')
plt.title('Prints (Failed Focus)')

plt.figure(3,figsize=(10,8))
plt.xlabel('Print Time (mins)')
plt.ylabel('Filament (m)')
plt.title('Prints (Success Only)')

for data in matrix:
    filament = [data[0][:,8]]
    time = [data[0][:,9]]
    satisfaction = [data[:,[10]]]
    result = [data[:,[11]]]
    result_success = [[1]]
    if result[0] == result_success:
        plt.figure(1)
        plt.scatter(time, filament, c="blue", alpha=0.5,)
        plt.figure(2)
        plt.scatter(time, filament, c="blue", alpha=0.05,)
        plt.figure(3)
        plt.scatter(time, filament, c="blue", alpha=1,)
    else:
        plt.figure(1)
        plt.scatter(time, filament, c="red", alpha=0.5,)
        plt.figure(2)
        plt.scatter(time, filament, c="red", alpha=1,)

plt.tight_layout()
plt.show()


I am working on the file "Data Plots New.ipynb".
The first task is:
Prepare the individual data axis
Can you write Python code for it?

filament = matrix[:,[8]]
time = matrix[:,[9]]
satisfaction = matrix[:,[10]]
result = matrix[:,[11]]


I am working on the file "Data Plots New.ipynb".
The first task is:
Process the data
We will output the data in the form of a numpy matrix using our preprocessing functions.
The first row in the matrix (first data entry) and total number of samples will be displayed.
Can you write Python code for it?

matrix = obtain_data_matrix(data)
samples = len(matrix)
print("Number of samples: " + str(samples))


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
0) Dependencies
Can you write Python code for it?

################################################################################
######################### CHANGE BQ PROJECT NAME BELOW #########################
################################################################################

project_name = '' #add proj name

# Google credentials authentication libraries
from google.colab import auth
auth.authenticate_user()

# data processing libraries
import numpy as np
from numpy.core.numeric import NaN
import datetime
import pandas as pd
import pandas_gbq

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
!pip install boruta #boruta for feature selection
from boruta import BorutaPy
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier

# modeling and metrics
from scipy.optimize import least_squares
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant
import statsmodels.api as sm

import itertools
from scipy.stats.stats import pearsonr

# Visualization
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = [10, 5] #change size of plot
import seaborn as sns

# BigQuery Magics
'''
BigQuery magics are used to run BigQuery SQL queries in a python environment.
These queries can also be run in the BigQuery UI
'''

from google.cloud import bigquery
from google.cloud.bigquery import magics

magics.context.project = project_name  #update your project name

client = bigquery.Client(project=magics.context.project)


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
2) Data Cleaning
2.1) Check for missing data and impute
Check the amount of of missing values (% of total column) in the data and sort by
highest to lowest.
Can you write Python code for it?

missing_values = 100*df.isnull().sum()/len(df)
missing_values.sort_values(ascending = False)


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
5) Feature Creation
5.1) Check for Seasonality and add Flag
View the target variable as a time series plot and identify periods where data peaks.
In this specific example, conversions spike on Mondays, so we will create
a specific flag (0/1 variable) for that day and use it as a control feature in the feature set.
Can you write Python code for it?

plt.plot_date(df.index, target_variable, linestyle = '-')

featureset_df['Is_Monday'] = (df.index.get_level_values(0).weekday == 0).astype(int)
featureset_df['Is_Q2Q3'] = (df.index.get_level_values(0).month == 4).astype(int) | (df.index.get_level_values(0).month == 5).astype(int) | (df.index.get_level_values(0).month == 6).astype(int) | (df.index.get_level_values(0).month == 7).astype(int) | (df.index.get_level_values(0).month == 8).astype(int) | (df.index.get_level_values(0).month == 9).astype(int)


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
5.2) Lag, Carryover, Diminishing Returns
We'll need to transform the raw data by applying lag, carryover, and diminishing returns to have it most accuratley predict the target variable.
First, split the df into two different dataframes:

Features that don't need to be transformed

Examples are:
date
target variable
control variables (seasonality, promotions, etc.)

Features that do need to be transformed

Paid media tactics
Any other feature where there is some sort of delayed response with the target variable

Starting points for lags:
- If you are using daily data, the lag should at default be 14.
- If you are using weekly data, the lag should at default be 5.
Others can and should be tested to determine the best lag length for your specific data.
Can you write Python code for it?

# Variables that do not need to be transformed

untransformed_df = pd.concat([target_variable, featureset_df[['Is_Monday','Is_Q2Q3']]], axis = 1) #Target variable + controls

# Variables that do need to be transformed

#exclude dummies/controls that do not need to be transformed
#transformed_df = featureset_df[['feature1', 'feature2',...]]

'''
Note: In this example case, almost all of the features in the featureset_df are media features.
As more dummy variables or other control variables are added, the user will need to
specify which columns should be transformed
'''
transformed_df = featureset_df.loc[:,~featureset_df.columns.isin(['Is_Monday','Is_Q2Q3'])]

#This function creates the different combinations of Lag, Decay, and Curve
def transformation(dataframe,x):
    lag = []
    for i in range(0, 7, 1):
        data = dataframe[x].shift(i).to_frame()
        data.columns = [col_name+'Lag'+str(i)for col_name in data.columns]
        # store DataFrame in list
        lag.append(data)
    # see pd.concat documentation for more info
    lag = pd.concat(lag,axis=1)
    lag=lag.fillna(0)
    Alpha = []
    for i in np.linspace(0.6,1.0,num=5):
        data = pow(lag,i)
        data.columns = [col_name+'Alpha'+str(i)for col_name in data.columns]
    # store DataFrame in list
        Alpha.append(data)
    # see pd.concat documentation for more info
    Alpha = pd.concat(Alpha,axis=1)
    Decay=[]
    #j = 0
    for percent in np.linspace(0.6,1.0,5):
        data = Alpha.copy()
        data.columns = [col_name+'Decay'+str(percent)for col_name in data.columns]
        for i in range(0,len(Alpha)):
            for j in range(0,len(Alpha.columns)):
            #data = data + data.shift(1)*(1-i)
                if(i == 0):
                    data.iloc[i, j] = data.iloc[i, j]*percent
                else:
                    data.iloc[i, j] = data.iloc[i - 1, j] *(1-percent) + data.iloc[i,j] * percent
        Decay.append(data)
        j = j + 1
    Decay = pd.concat(Decay,axis=1)

    return Decay


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
6) Feature Selection
For feature selection we employ the Boruta algorithm.(More information here)
This algorithm will tell you the rank of each feature and whether or not to keep a varaible in the model (i.e. Keep  = True/False). The goal of RBA is to optimize across all paid digital media tactics, therefore select the top ranking feature for each group of features (whether or not the algorithm tells you to keep the feature).
Can you write Python code for it?

# Specifiying the target and x variables
y = target_variable
x = transformed_df #update with transformed features

# define random forest classifier
forest = RandomForestRegressor(n_jobs=-1, max_depth=5)
forest.fit(x, y)

# define Boruta feature selection method
feat_selector = BorutaPy(forest, n_estimators='auto', verbose=2, random_state=1)

# find all relevant features
feat_selector.fit(np.array(x), np.array(y))

# check selected features
feat_selector.support_

# check ranking of features
feat_selector.ranking_

# call transform() on X to filter it down to selected features
X_filtered = feat_selector.transform(np.array(x))

#Select the top ranking variable for each group of variables.
feature_ranks = list(zip(x.columns,
                         feat_selector.ranking_,
                         feat_selector.support_))

# iterate through and print out the results
for feat in feature_ranks:
    print('{:<25}, Rank: {},  Keep: {}'.format(feat[0], feat[1], feat[2]))


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
7) Feature Scaling
7.1) Feature Scaling
The default method of standardization utilizes Standard Scaler, which takes in
input data and transforms so that the output has mean 0 and standard deviation of 1
across all features.
Alternative methods of feature scaling include square-root transformation,
de-meaning, natural log transformations, Min-Max Scalers, or normalization
Can you write Python code for it?

scaler = StandardScaler()
standardized_transform = scaler.fit_transform(selected_featureset_df)
selected_featureset_df = pd.DataFrame(standardized_transform, columns = selected_featureset_df.columns)


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
Drop the highest VIF features and print the high collinearity columns in a list
Can you write Python code for it?

high_collinearity_columns = vif_df.feature[vif_df['vif'] >= 10].to_list()

high_collinearity_columns


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
If there are any NAs in the data that should be zeros, replace those data
points with zero.
Can you write Python code for it?


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
Immediately remove variables that won't be used in the model. Here, includes columns like geo which is consistent across the dataset, and aggregated media such as total clicks across DSPs.
Can you write Python code for it?


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
Option to review visuals of the data. After the data is standardized the distributions may take on a more normal shape.
Can you write Python code for it?

'''
for i in range(0,len(X_transform_stand.columns)):
  plt.figure()
  sns.kdeplot(X_transform_stand[X_transform_stand.columns[i]], label = X_transform_stand.columns[i], shade = True)
'''

selected_featureset_df.head()


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
Run VIF analysis and flag values greater than 10.
Industry best practice flags values above 10 as an extreme violation of regression model assumptions. (Reference)
Can you write Python code for it?

vif = add_constant(selected_featureset_df)

# loop to calculate the VIF for each X
vif = pd.Series([variance_inflation_factor(vif.values, i)
      for i in range(vif.shape[1])],
      index=vif.columns)

# processing to output VIF results as a dataframe
vif_df=vif.to_frame().reset_index()

vif_df.columns = ['feature', 'vif']
vif_df=vif_df.replace([np.inf], np.nan) # replace inf calculations as missing and zero fill
vif_df=vif_df.fillna(0).sort_values(by="vif", ascending=False)

vif_df.reset_index(inplace = True)
vif_df


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
Trim the start of your dataset to correspond with the max lag
(if max lag is 4 weeks, trim the first 4 weeks off of the data)
Can you write Python code for it?

max_lag = 5
final_df = final_df[max_lag:]
final_df.reset_index(inplace = True)
final_df.drop(columns = 'index',inplace = True)

final_df.head()

################################################################################
######################### CHANGE BQ PROJECT NAME BELOW #########################
################################################################################

destination_project_id = "" #@param
destination_dataset = "RBA_demo" #@param
destination_table = "cleaned_data" #@param
dataset_table = destination_dataset+"."+destination_table

final_df.to_gbq(dataset_table,
                 destination_project_id,
                 chunksize=None,
                 if_exists='replace'
                 )


I am working on the file "Data Pre-Processing & Feature Selection.ipynb".
The first task is:
WARNING: This section of the code is estimated to take 10-30 minutes to complete
depending on weekly vs daily data and the number of features in the model.
Can you write Python code for it?

#Loop through each column and apply the transformation

columns = transformed_df.columns
sales = target_variable
all_data = []
for col in columns:
  newdf = transformation(transformed_df, col)
  corr_df = pd.concat([sales, newdf], axis=1)
  corr = corr_df.corr().sort_values(kpi_col,ascending=False)
  new_vals= corr.iloc[1:4 , 0:1].index.tolist()
  data = newdf[new_vals]
  all_data.append(data)
transformed_df = pd.concat(all_data,axis=1)


I am working on the file "Data Prep.ipynb".
The first task is:
Current project:
you have unlabeled cans of paint.  You want to be able to assign color names to them.
My model currently just dumps all the cans of paint into a bucket, mixes it up, and labels each paint can that gross dark brown purlply color that comes out when a child mixes all the paints together.
Can you write Python code for it?

#Testing kmeans cluster centroids

topic1 = [word for word, score in wordvecs.most_similar(positive=['engine'], topn=10)]
topic2 = [word for word, score in wordvecs.most_similar(positive=['chair'], topn=10)]
topic3 = [word for word, score in wordvecs.most_similar(positive=['succulent'], topn=10)]
topic4 = [word for word, score in wordvecs.most_similar(positive=['panda'], topn=10)]
topic5 = [word for word, score in wordvecs.most_similar(positive=['cancer'], topn=10)]

topic1

arrs = []
for doc in [topic1, topic2, topic3, topic4, topic5]:
    for word in doc:
        arrs.append(wordvecs[word])
arrs = np.array(arrs)

def is_pos_def(x):
    return np.all(np.linalg.eigvals(x) > 0)

mean = np.mean(arrs, axis=0)

is_pos_def(np.shape(arrs)[0] * (arrs-mean).dot((arrs-mean).T) + 0.001)

k = 5
km = KMeans(n_clusters=k)
km.fit(arrs)

for i in range(k):
    print wordvecs.most_similar(positive=[km.cluster_centers_[i]]), '\n'


I am working on the file "Data Prep.ipynb".
The first task is:
Making More Sythetic Data
Can you write Python code for it?

%pprint on
topic1 = ' '.join([word for word, score in wordvecs.most_similar(positive=['car'], topn=10)])
topic2 = ' '.join([word for word, score in wordvecs.most_similar(positive=['science'], topn=10)])
topic3 = ' '.join([word for word, score in wordvecs.most_similar(positive=['construction'], topn=10)])
print topic1, topic2, topic3

docs = [topic1, topic2, topic3]

f = '/Users/michael/Documents/GaussianLDA/clean20news.txt'
# np.savetxt(f, docs)
with open(f, 'w') as fi:
    for doc in docs:
        try:
            fi.write("%s\n" % doc)
        except UnicodeEncodeError: continue


I am working on the file "Data Prep.ipynb".
The first task is:
Making mixed topic documents
75/25/1 mix between the three sythetic topic vocabs
Can you write Python code for it?

import pprint

%pprint

for _ in range(10):
    print stat.dirichlet.rvs([0.3]*5)[0], "\n"

import scipy.stats as stat

iterations = 100
corpus = []
for t in range(iterations):
    doc_len = 100
#     dirdist = np.array([0.90, 0.1])
    dirdist = stat.dirichlet.rvs([.2]*5, random_state=np.random.randint(low=0, high=10000)) # topic distribution in a doc
    num_words = dirdist * doc_len # how many words to take from each topic, approx
#     print dirdist
    # print num_words[0]
    BoW = [topic1, topic3,topic4, topic2, topic5]
    doc = []
    for bag, count in zip(BoW, num_words[0]):

        num = int(np.round(count))
        for x in range(num):
            doc.append(random.choice(bag)) # randomly choosing words from topic **UNIFORM
    random.shuffle(doc)
    doc = ' '.join(doc)

    corpus.append(doc)

# corpus

# with open('/Users/michael/Documents/GaussianLDA/cleandocs.txt', 'w') as f:
#     for doc in corpus:
#         f.write(doc + "\n")

with open('/Users/michael/Documents/GaussianLDA/cleandocs.txt', 'r') as f:
    corpus = f.read().splitlines()

corpus[1]


I am working on the file "Data Prep.ipynb".
The first task is:
Testing to make sure we can read it well
Can you write Python code for it?

# f = '/Users/michael/Documents/GaussianLDA/clean20news.txt'
# with open(f, 'r') as fi:
#     text = fi.read().splitlines()
#     fi.close()
# print len(text)

a, b, c = [0,0], [0,0], [0,0]

for x, y in zip(txt, data['target']):
    if y == 0: a[0] += len(x); a[1] += 1
    if y == 1: b[0] += len(x); b[1] += 1
    if y == 2: c[0] += len(x); c[1] += 1

print a[0] / a[1], a[1]
print b[0] / b[1], b[1]
print c[0] / c[1], c[1]

# mansour_topic = np.loadtxt("/Users/michael/Documents/GaussianLDA/output/1iter4topic2Topic Mean.txt")
# das_topic = np.loadtxt("/Users/michael/Documents/Gaussian_LDA-master/output/Aliaswiki50D20T3.txt")


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.2.2  Missing Value Treatment
We can do the following with missing values
3.2.2.1  Drop missing values
3.2.2.2 Fill missing values with test statistic
3.2.2.3 impute  missing value with a machine learning algorithm
3.2.2.1 dropping missing values
Can you write Python code for it?


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.2.2.3.1  For imputing all missing values
Can you write Python code for it?

#This will look for all columns where we have NaN value and replace the NaN value with specified test statistic.
#for mode we specify strategy='most_frequent'

from sklearn.preprocessing import Imputer
imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(df[['y','z']])
df[['y','z']]= imp.transform(df[['y','z']])

#assert that there are no missing values.
#When it encounters an assert statement, Python evaluates the accompanying expression, which is hopefully true.
#If the expression is false, Python raises an AssertionError exception.
assert pd.notnull(df).all().all()

df.apply(lambda x: sum(x.isnull().values), axis = 0) # For columns


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.2.2.3.2 In addition this has the following way to impute missing value using back fill or forward fill
Can you write Python code for it?

# filling missing value with test statistic
#for back fill
df.fillna(method='bfill')
#for forward-fill
df.fillna(method='ffill')
#one can also specify an axis to propagate (1 is for rows and 0 is for columns)
df.fillna(method='bfill', axis=1).head()


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.2.2.3.3 predict missing value with a machine learning algorithm
Example- Split data into sets with missing values and without missing values, name the missing set X_text and the one without missing values X_train and take y (variable or feature where there is missing values) off the second set, naming it y_train.
Use one of classification methods to predict y_pred.Add it to X_test as your y_test column. Then combine sets together
or use fancyimpute
Can you write Python code for it?


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.3.2.1.1 How to deal with missing values that are not NaN. Example Replace values of zero with NaN in the column named column_nam
Can you write Python code for it?

# Example- Replace values of zero with NaN in the column named column_name
#df[‘column_name’].replace(0, np.nan, inplace= True)


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.5 Manual Cleaning, Construct and Preprocessing Data
Using Re to clean data below
Here re.sub replaces the $ and , patterns with nothing ("") from each value of the list  .
In python str converts an object to string
. In the next step, int converts the object to numeric values (integer)
Can you write Python code for it?

#MATCHING
import re
names=["Ajay","V ijay","Ra jay ", " Jayesh"]

for name in names:
    print (re.search(r'(jay)',name))
for name in names:
    print (bool(re.search(r'(jay)',name)))

numlist=["$10000","$20,000","30,000",40000,"50000   "]
for i,value in enumerate(numlist):
    numlist[i]=re.sub(r"([$,])","",str(value))
    numlist[i]=int(numlist[i])

numlist


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.6 DIMENSIONALITY REDUCTION -t-SNE
t-distributed Stochastic Neighbor Embedding.
t-SNE  is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results.
It is highly recommended to use another dimensionality reduction method (e.g. PCA for dense data or TruncatedSVD for sparse data) to reduce the number of dimensions to a reasonable amount (e.g. 50) if the number of features is very high.
The disadvantages to using t-SNE are roughly:
t-SNE is computationally expensive, and can take several hours on million-sample datasets where PCA will finish in seconds or minutes
The Barnes-Hut t-SNE method is limited to two or three dimensional embeddings.
The algorithm is stochastic and multiple restarts with different seeds can yield different embeddings. However, it is perfectly legitimate to pick the embedding with the least error.
Global structure is not explicitly preserved. This is problem is mitigated by initializing points with PCA (using init=’pca’).
Can you write Python code for it?

%whos DataFrame

df[features].head()

data_scaled[features].head()

df['price'].head()

data_scaled['price'].head()

data_scaled.values


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
3.7 Integrate Data
Merged Data
3.8 Format Data
Reformatted data
Can you write Python code for it?

df.columns

a=list(df.columns)
a

##### list to tuple
x=tuple(a,)
x

x=23.5
##### to int
b=int(x)
b

###### to string
c=str(b)
c

#### convert dataframe to numpy array
df.values

from datetime import datetime

datetime_object = datetime.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')
datetime_object


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
Looking at variables  as continuous (numeric) , categorical (discrete) and string variables

Note some numeric variables can be categoric. (eg 3,4,8 cylinder cars)

We use values_count.unique to find number of unique values_count for categoric and describe for numeric variables as above in continuous and categorical variales

Categorical values can be made discrete using get_dummy method where each level of categorical variable can be made 1/0 binary variable

Some categoric data can be string

You can also modify the lists of variables below manually
Can you write Python code for it?

#https://docs.scipy.org/doc/numpy/reference/arrays.scalars.html
Image("C:\\Users\\ajaohri\\Desktop\\ODSP\\img\\dtype-hierarchy.png")

df=pd.get_dummies(df, drop_first=True,sparse=True) #Sparse = True takes care of memory error

df.head()


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
RESTARTING
Taking a fresh dataset for demonstrating missing value imputation, since previous dataset has all missing values replaced
Can you write Python code for it?

df2=pd.read_csv('C:\\Users\\ajaohri\\Desktop\\ODSP\\data\\BigDiamonds.csv')

df2.columns
df2 = df2.drop("Unnamed: 0", axis=1)

df2 = df2.drop("measurements", axis=1)

df2.apply(lambda x: sum(x.isnull().values), axis = 0) # For columns

df2=df2.dropna()

colormap = plt.cm.RdBu
plt.figure(figsize=(32,10))
plt.title('Pearson Correlation of Features', y=1.05, size=15)
sns.heatmap(df2.corr(),linewidths=0.1,vmax=1.0,
            square=True, cmap=colormap, linecolor='white', annot=True)
# Note that the categorical features have been neglected in the
# correlation matrix.


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
Taking a smaller  dataset for TSNE since it takes a long time
Can you write Python code for it?

df2=pd.read_csv('https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv')

df2.info()

df2 = df2.drop("Unnamed: 0", axis=1)

df2=pd.get_dummies(df2,drop_first=True)

from sklearn import preprocessing
data_scaled = pd.DataFrame(preprocessing.scale(df2),columns = df2.columns)

%whos

import numpy as np
from sklearn.manifold import TSNE
X = data_scaled.values
X1=TSNE(n_components=2, init='pca',n_iter=250)
X_embedded =X1.fit_transform(X)
X_embedded.shape

X_embedded

x1=X_embedded[:,0]
y1=X_embedded[:,1]

x1

y1

df2.columns

n_sne = 32
rndperm = np.random.permutation(df2.shape[0])

tsne_results = X1.fit_transform(df2.loc[rndperm[:n_sne],df2.columns].values)

tsne_results

df_tsne = df2.loc[rndperm[:n_sne],:].copy()
df_tsne['x-tsne'] = tsne_results[:,0]
df_tsne['y-tsne'] = tsne_results[:,1]

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn
seaborn.set(rc={'axes.facecolor':'gray', 'figure.facecolor':'white'})

fig, ax = plt.subplots()
plt.gcf().set_size_inches(15 ,12)

ax = sns.scatterplot(x='x-tsne', y='y-tsne', data=df_tsne)

from ggplot import *
chart = ggplot( df_tsne, aes(x='x-tsne', y='y-tsne') ) \
        + geom_point(size=70,alpha=0.1) \
        + ggtitle("tSNE dimensions    ")
chart


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
The curse of dimensionality
For an estimator to be effective, you need the distance between neighboring points to be less than some value , which depends on the problem. In one dimension, this requires on average  points. In the context of the above -NN example, if the data is described by just one feature with values ranging from 0 to 1 and with  training observations, then new data will be no further away than . Therefore, the nearest neighbor decision rule will be efficient as soon as  is small compared to the scale of between-class feature variations.
If the number of features is , you now require  points. Let’s say that we require 10 points in one dimension: now  points are required in  dimensions to pave the  space. As  becomes large, the number of training points required for a good estimator grows exponentially.
For example, if each point is just a single number (8 bytes), then an effective -NN estimator in a paltry  dimensions would require more training data than the current estimated size of the entire internet (±1000 Exabytes or so).
This is called the curse of dimensionality and is a core problem that machine learning addresses
3.4 DIMENSIONALITY REDUCTION -SVD
Dimensionality reduction using truncated SVD (aka LSA). The Singular-Value Decomposition (SVD), is a matrix decomposition method for reducing a matrix to its constituent parts in order to make certain subsequent matrix calculations simpler.
This transformer performs linear dimensionality reduction by means of truncated singular value decomposition (SVD). Contrary to PCA, this estimator does not center the data before computing the singular value decomposition. This means it can work with scipy.sparse matrices efficiently.
TruncatedSVD is very similar to PCA, but differs in that it works on sample matrices  directly instead of their covariance matrices. When the columnwise (per-feature) means of  are subtracted from the feature values, truncated SVD on the resulting matrix is equivalent to PCA
Can you write Python code for it?

mean_vec = np.mean(x, axis=0)
cov_mat = (x - mean_vec).T.dot((x - mean_vec)) / (x.shape[0]-1)
print('Covariance matrix \n%s' %cov_mat)

print('NumPy covariance matrix: \n%s' %np.cov(x.T))


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
for regression we take dataset from http://bit.ly/dsdata
Can you write Python code for it?

path='C:\\Users\\ajaohri\\Desktop\\data'
os.chdir(path)
os.listdir()

%%time
%time

df=pd.read_csv('C:\\Users\\ajaohri\\Desktop\\ODSP\\data\\BigDiamonds.csv')

print("The name of variables in the input dataset are ",df.columns)

print("The type of variables in the input dataset are ",df.dtypes)

print("The first five rows of dataset are")
df.head()

print('The input dataset df','has',df.shape[0],"rows and",df.shape[1],'columns')

df = df.drop("Unnamed: 0", axis=1)

df = df.drop("measurements", axis=1)


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
np.linalg.eig Performs eigendecomposition on covariance matrix. Compute the eigenvalues and right eigenvectors of a square array.Eigendecomposition of a matrix is a type of decomposition that involves decomposing a square matrix into a set of eigenvectors and eigenvalues.
A vector is an eigenvector of a matrix if it satisfies the following equation. Av = lambdav
 This is called the eigenvalue equation, where A is the parent square matrix that we are decomposing, v is the eigenvector of the matrix, and lambda is the lowercase Greek letter and represents the eigenvalue scalar
Can you write Python code for it?

cov_mat = np.cov(x.T)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)
print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)

# Make a list of (eigenvalue, eigenvector) tuples
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eig_pairs.sort()
eig_pairs.reverse()

# Visually confirm that the list is correctly sorted by decreasing eigenvalues
print('Eigenvalues in descending order:')
for i in eig_pairs:
    print(i[0])

from numpy import array
from numpy.linalg import pinv
A=df.values
print(A)
# calculate pseudoinverse
B = pinv(A)
print(B)

# Pseudoinverse via SVD
from scipy import linalg
from numpy import array
#from numpy.linalg import svd
from numpy import zeros
from numpy import diag
# calculate svd
U, s, Vh=linalg.svd(A, full_matrices=False)
U.shape,  s.shape, Vh.shape

Sigma = diag(s)
# reconstruct matrix
B = U.dot(Sigma.dot(Vh))
print(B)

from numpy import array
from numpy.linalg import pinv
# define matrix
A = df.values
print(A)
# calculate pseudoinverse
B = pinv(A)
print(B)

from numpy import array
from sklearn.decomposition import TruncatedSVD
# define array
A =df.values
print(A)
# svd
svd = TruncatedSVD(n_components=2)
svd.fit(A)
result = svd.transform(A)
print(result)


I am working on the file "Data Preparation Dimensionality Reduction.ipynb".
The first task is:
we can see price and x,y,carat are highly corelated but carat has no null values
Can you write Python code for it?

df2.apply(lambda x: sum(x.isnull().values), axis = 0) # For columns

(df2.price).corr(df2.carat)

%whos DataFrame

from fancyimpute import KNN


I am working on the file "Data Preprocessing.ipynb".
The first task is:
Complete process for all columns
It worked!
Now we want to turn that into a process that goes through all columns in this dataframe.
Can you write Python code for it?

state_days_on_zillow_flat = pd.DataFrame(state_days_on_zillow__limited[['2011-01']], index=[state_days_on_zillow__limited.axes[0]])
state_days_on_zillow_flat['Period'] = pd.Series({state:'2011-01' for state in state_days_on_zillow__limited.axes[0]})
state_days_on_zillow_flat = state_days_on_zillow_flat.set_index('Period', append=True).rename_axis({'2011-01':'days_on_zillow'}, axis='columns')

for col_name in state_days_on_zillow__limited.axes[1][1:]:
    temp = pd.DataFrame(state_days_on_zillow__limited[[col_name]], index=[state_days_on_zillow__limited.axes[0]])
    temp['Period'] = pd.Series({state:col_name for state in state_days_on_zillow__limited.axes[0]})
    temp = temp.set_index('Period', append=True).rename_axis({col_name:'days_on_zillow'}, axis='columns')

    state_days_on_zillow_flat = state_days_on_zillow_flat.append(temp)

state_days_on_zillow_flat


I am working on the file "Data Preprocessing.ipynb".
The first task is:
Creates CSVs of text from comments made by users who have posted about anorexia or obesity.
Can you write Python code for it?

import pandas as pd
from tqdm import tqdm
from nltk.corpus import wordnet
from nltk.stem.porter import *
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import TweetTokenizer

tokenizer = TweetTokenizer()
wordnet_lemmatizer = WordNetLemmatizer()

# Create synonym sets for obesity and anorexia
def syn_set(word_list):
    syns = set()
    for word in word_list:
        for synset in wordnet.synsets(word):
            for lemma in synset.lemmas():
                syns.add(lemma.name())
    return syns

OBESITY_SYNS = syn_set(['obesity'])
ANOREXIA_SYNS = syn_set(['anorexia'])

def row_filter_fn(df, syns):
    """Returns True if the row should be included, False otherwise."""
    # Check if any synonyms can be found.
    if set([wordnet_lemmatizer.lemmatize(token.lower()) for token in tokenizer.tokenize(df)]) & syns:
        return True

    return False

csv_filename = 'data_full_preprocessed.csv'
chunksize = 10000
count = 0
obesity_data_frames = []
anorexia_data_frames = []
for chunk in tqdm(pd.read_csv(csv_filename, chunksize=chunksize)):
    obesity_df = chunk[chunk['body'].apply(row_filter_fn, syns=OBESITY_SYNS)]
    if not obesity_df.empty:
        obesity_data_frames.append(obesity_df)
    anorexia_df = chunk[chunk['body'].apply(row_filter_fn, syns=ANOREXIA_SYNS)]
    if not anorexia_df.empty:
        anorexia_data_frames.append(anorexia_df)
    count += 1
    #if count == 100: break
print('Total # chunks processed: {}.'.format(count))

# Write out to CSVs.
pd.concat(obesity_data_frames).to_csv('obesity.csv', index=False)
pd.concat(anorexia_data_frames).to_csv('anorexia.csv', index=False)


I am working on the file "Data Preprocessing.ipynb".
The first task is:
Flatten the array
Right now we have an array for a single metric. We don't really want this - we want one row per month per state, so we can join other metrics on by those indicies.
Below we test an approach to turning this array into a flatter form, which will allow us to join with other metrics.
Can you write Python code for it?

test = pd.DataFrame(state_days_on_zillow__limited[['2011-01']], index=[state_days_on_zillow__limited.axes[0]])
test['Period'] = pd.Series({state:'2011-01' for state in state_days_on_zillow__limited.axes[0]})
test = test.set_index('Period', append=True).rename_axis({'2011-01':'var_name'}, axis='columns')

test2 = pd.DataFrame(state_days_on_zillow__limited[['2011-02']], index=[state_days_on_zillow__limited.axes[0]])
test2['Period'] = pd.Series({state:'2011-02' for state in state_days_on_zillow__limited.axes[0]})
test2 = test2.set_index('Period', append=True).rename_axis({'2011-02':'var_name'}, axis='columns')

test.append(test2)


I am working on the file "Data Preprocessing.ipynb".
The first task is:
Import data
First we need to import our data from a CSV.
Can you write Python code for it?

state_data_filepath = '.\\Data\\State\\DaysOnZillow_Public_State.csv'

state_days_on_zillow = pd.read_csv(state_data_filepath)
state_days_on_zillow.head()


I am working on the file "Data Preprocessing.ipynb".
The first task is:
Limit columns
From outside analysis, I've found that the 2011-01 through 2016-01 periods are the best populated with data, so we limit ourselves to that period for now.
Below we drop all the other, unnecessary columns, with the exception of RegionName, which will become an index.
Can you write Python code for it?

state_days_on_zillow__limited = state_days_on_zillow[[col for col in state_days_on_zillow.axes[1] if ((col >= '2011-01' and col <= '2016-01') or (col == 'RegionName'))]]
state_days_on_zillow__limited = state_days_on_zillow__limited.set_index(['RegionName'])

state_days_on_zillow__limited.head()


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
2 - Fit a Nearest Neighbors model to the data, using a normalized data set, a stardardized data set, and the original.  Split into test and train sets and compute the accuracy of the classifications and comment on your results.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.model_selection import train_test_split


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
2 - Perform a multilinear regression, using the classified data, removing the NA values.  Comment on your results.
Can you write Python code for it?

lr.fit(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])
lr.score(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
3 - Which variables lend themselves to be in a regression model?  Select those variables, and then fit a regression model for each of the following imputation strategies, commenting on your results.
    - Dropping all rows with at least 1 NA
    - Dropping all rows with at least 3 NA
    - Imputing 0
    - Mean
    - Median
    - Mode
Can you write Python code for it?

# I'm dropping wind_dir and id
regr_cols = ['year', 'month', 'day', 'hour', 'dew_point', 'temperature',
             'pressure', 'wind_speed', 'hours_snow', 'hours_rain', 'pm2']
pm2_regr = pm2.loc[:, regr_cols]
# in the solution there is no year, month, day and hour
# also, he discards hours_snow and hours_rain (though they aren't binary or categorical)

# from sklearn.model_selection import train_test_split
from sklearn.preprocessing import Imputer
from sklearn.linear_model import LinearRegression

lr = LinearRegression()
# X = pm2_regr.iloc[:, :-1]
# y = pm2_regr.iloc[:, -1]
# Xtrain, Xtest, ytrain, ytest = train_test_split(pm2_regr.iloc[:, :-1], pm2_regr.iloc[:, -1], test_size=0.2, random_state=0)

#just a note to self
pm2_regr1 = pm2_regr.dropna(thresh=7) # same as dropna without thresh
# thresh is the number of non nan columns required to mantain the rows
pm2_regr1 = pm2_regr.dropna(thresh=5)
pm2_regr1.info()


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
After the fourth feature there is no improvement.
Also, the MSE seems better than all the trials before.
5 - L1 regularization can also be used for model selection.  Choose an algorithm in sklearn and repeat part (4) using model selection via regularization.
Can you write Python code for it?

# again, in solutions he uses the c_vals as before and he fits the lasso

from sklearn.linear_model import LassoCV
from sklearn.feature_selection import SelectFromModel
# fitting ridge regression
lasso = LassoCV()
c_vals = np.arange(0.1, 2.1, 0.1)
cols = Xtrain.columns

mse = []
# looping through the possible threshholds from above and storing the results in mse
for c in c_vals:
    # using SelectFromModel with the ridge scores from above
    selfrmod = SelectFromModel(lasso, threshold=str(c) + '*mean')
    # fitting the selector
    selfrmod.fit(Xtrain, ytrain)
    # transforming train and test sets
    Xtrain_k = selfrmod.transform(Xtrain)
    Xtest_k = selfrmod.transform(Xtest)
    # fitting linear regression model and printing out the k best features
    lasso.fit(Xtrain_k, ytrain)
    print('c={} features {}'.format(c, cols[selfrmod.get_support()]))
    mse.append(mean_squared_error(lasso.predict(Xtest_k), ytest))

mse

import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(figsize=(16, 8))
plt.plot(c_vals, mse)
plt.title('MSE for different thresholds')
plt.xlabel('c')
plt.ylabel('MSE');

from sklearn.linear_model import LassoCV
# fitting lasso regression
lasso = LassoCV()
lasso.fit(Xtrain, ytrain)
# storing features importance
coef = lasso.coef_

mse = []
# looping through the possible threshholds from above and storing the results in mse
for k, thresh in enumerate(sorted(coef, reverse=True)):
    # using SelectFromModel with the lasso scores from above
    selfrmod = SelectFromModel(lasso, threshold=thresh)
    # fitting the selector
    selfrmod.fit(Xtrain, ytrain)
    # transforming train and test sets
    Xtrain_k = selfrmod.transform(Xtrain)
    Xtest_k = selfrmod.transform(Xtest)
    # fitting linear regression model and printing out the k best features
    lr.fit(Xtrain_k, ytrain)
    print('Top {} features {}'.format(k+1, pd.Series(lasso.coef_, index=Xtrain.columns).\
                                      sort_values(ascending=False).\
                                     ...


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Dropping all row with at least 3 NA gets me an error because I have nans in some rows:
Can you write Python code for it?

lr.fit(pm2_regr.dropna(thresh=5).iloc[:, :-1], pm2_regr.dropna(thresh=5).iloc[:, -1])
lr.score(pm2_regr.dropna(thresh=5).iloc[:, :-1], pm2_regr.dropna(thresh=5).iloc[:, -1])


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Dropping all rows with at least 1 NA:
Can you write Python code for it?

lr.fit(pm2_regr.dropna().iloc[:, :-1], pm2_regr.dropna().iloc[:, -1])
lr.score(pm2_regr.dropna().iloc[:, :-1], pm2_regr.dropna().iloc[:, -1])


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Normalized dataset:
Can you write Python code for it?

gnb.fit(Xtrain_norm, ytrain)
print(gnb.score(Xtrain_norm, ytrain))
print(gnb.score(Xtest_norm, ytest))


mms = MinMaxScaler()
Xtrain_norm = mms.fit_transform(Xtrain)
Xtest_norm = mms.transform(Xtest)

knn.fit(Xtrain_norm, ytrain)
print(knn.score(Xtrain_norm, ytrain))
print(knn.score(Xtest_norm, ytest))


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Original dataset:
Can you write Python code for it?

Xtrain, Xtest, ytrain, ytest = train_test_split(wine.iloc[:, 1:], wine.iloc[:, 0], test_size=0.3, random_state=0)
knn = KNeighborsClassifier()
knn.fit(Xtrain, ytrain)
print(knn.score(Xtrain, ytrain))
print(knn.score(Xtest, ytest))


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Standardized dataset:
Can you write Python code for it?

gnb.fit(Xtrain_std, ytrain)
print(gnb.score(Xtrain_std, ytrain))
print(gnb.score(Xtest_std, ytest))


ssc = StandardScaler()
Xtrain_std = ssc.fit_transform(Xtrain)
Xtest_std = ssc.transform(Xtest)

knn.fit(Xtrain_std, ytrain)
print(knn.score(Xtrain_std, ytrain))
print(knn.score(Xtest_std, ytest))