I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn:

Histogram: Plot a histogram of all the predicted prices
Scatter Plot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with statsmodels before).
Can you write Python code for it?

# your turn
# Plot a histogram of all the predicted prices
plt.hist(lm.predict(X))
plt.title("Predicted Prices")
plt.xlabel("Predicted Prices")
plt.ylabel("Frequency")
plt.show()

# Let's plot the true prices compared to the predicted prices to see they disagree
plt.scatter(bos.PRICE, lm.predict(X))
plt.xlabel("Housing Price")
plt.ylabel("Predicted Housing Price")
plt.title("Relationship between Predicted and Actual Price")


# your turn

plt.hist(lm.predict(X))
plt.title("Predicted Prices")
plt.xlabel("Price")
plt.ylabel("Frequency")
plt.show()


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn:  Let's build a linear regression model using our new training data sets.

Fit a linear regression model to the training set
Predict the output on the test set
Can you write Python code for it?

# your turn
lm = LinearRegression()
lm.fit(X_train[['RM']], Y_train)

print np.mean((Y_train - lm.predict(X_train[['RM']])) ** 2)

plt.scatter(X_train.RM, Y_train)
plt.xlabel("No. of Rooms")
plt.ylabel("Housing Price")
plt.title("Relationship between No. of Rooms and Price")

plt.plot(X_train.RM, lm.predict(X_train[['RM']]), color='blue', linewidth=3)
plt.show()


# your turn
# Fit a linear regression model to the training set
lm.fit(X_train, Y_train)
lm.predict(X_test)


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn: Create scatter plots between RM and PRICE, and PTRATIO and PRICE. What do you notice?
Can you write Python code for it?

#your turn: scatter plot between *RM* and *PRICE*

plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("Average Number of Rooms per Dwelling")
plt.ylabel("Housing Price")
plt.title("Relationship between No. of Rooms and Price");

#your turn: scatter plot between *PTRATIO* and *PRICE*

plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-Teacher Ratio by town")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRatio and Price");


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn: Do you think this linear regression model generalizes well on the test data?
->  No, the scatter points are not close to zero so the model needs improvements.  Check the features to see highly correlated predictors and remove one of them or check the parameters of the model and do fine-tuning.
K-fold Cross-validation as an extension of this idea

 A simple extension of the Test/train split is called K-fold cross-validation.

 Here's the procedure:

 randomly assign your $n$ samples to one of $K$ groups. They'll each have about $n/k$ samples

 For each group $k$:

 Fit the model (e.g. run regression) on all data excluding the $k^{th}$ group

 Use the model to predict the outcomes in group $k$

 Calculate your prediction error for each observation in $k^{th}$ group (e.g. $(Y_i - \hat{Y}_i)^2$ for regression, $\mathbb{1}(Y_i = \hat{Y}_i)$ for logistic regression).

 Calculate the average prediction error across all samples $Err_{CV} = \frac{1}{n}\sum_{i=1}^n (Y_i - \hat{Y}_i)^2$

Luckily you don't have to do this entire process all by hand (for loops, etc.) every single time, sci-kit learn has a very nice implementation of this, have a look at the documentation.
Your turn (extra credit): Implement K-Fold cross-validation using the procedure above and Boston Housing data set using $K=4$. How does the average prediction error compare to the train-test split above?
Can you write Python code for it?

from sklearn import cross_validation, linear_model

# If the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used.
# In all other cases, KFold is used
scores = cross_validation.cross_val_score(lm, X, bos.PRICE, scoring='mean_squared_error', cv=4)

# This will print metric for evaluation
print ('(MSE) Using k-fold: ', np.mean(scores))
print ('The K-fold cross-validation is not performaing well compared to the previous train-test split above')


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn: How would you change the model to not fit an intercept term? Would you recommend not having an intercept?
Estimated intercept and coefficients
Let's look at the estimated coefficients from the linear model using 1m.intercept_ and lm.coef_.
After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:
$$ \hat{\beta}0, \hat{\beta}_1, \ldots, \hat{\beta}{13} $$
Can you write Python code for it?

print ('Estimated intercept coefficient:', lm.intercept_)

print ('Number of coefficients:', len(lm.coef_))

# The coefficients
pd.DataFrame(list(zip(X.columns, lm.coef_)), columns = ['features', 'estimatedCoefficients'])


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn: Plot separate histograms and one for RM, one for PTRATIO. Any interesting observations?
Can you write Python code for it?

#your turn

plt.hist(bos.PTRATIO)
plt.title("PTRATIO")
plt.xlabel("Pupil-Teacher Ratio")
plt.ylabel("Frequency")
plt.show()


I am working on the file "Mini_Project_Linear_Regression.ipynb".
The first task is:
Your turn: What are some other numeric variables of interest? Plot scatter plots with these variables and PRICE.
Can you write Python code for it?

#your turn: create some other scatter plots


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:

ANSWERS to Checkup Exercise Set IV

* Instead of writing your own loops above to iterate over the model parameters, can you use GridSearchCV to find the best model over the training set?
* Does it give you the same best value of `C`?
No, C=1 for gridsearchcv while C=.1 for previous cv
* How does this model you've obtained perform on the test set?
It performed that same way with previous model at 0.92

## A Walkthrough of the Math Behind Logistic Regression

### Setting up Some Demo Code

Let's first set some code up for classification that we will need for further discussion on the math. We first set up a function `cv_optimize` which takes a classifier `clf`, a grid of hyperparameters (such as a complexity parameter or regularization parameter) implemented as a dictionary `parameters`, a training set (as a samples x features array) `Xtrain`, and a set of labels `ytrain`. The code takes the traning set, splits it into `n_folds` parts, sets up `n_folds` folds, and carries out a cross-validation by splitting the training set into a training and validation section for each foldfor us. It prints the best value of the parameters, and retuens the best classifier to us.
Can you write Python code for it?

def cv_optimize(clf, parameters, Xtrain, ytrain, n_folds=5):
    gs = sklearn.model_selection.GridSearchCV(clf, param_grid=parameters, cv=n_folds)
    gs.fit(Xtrain, ytrain)
    print("BEST PARAMS", gs.best_params_)
    best = gs.best_estimator_
    return best


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:

Checkup Exercise Set II

Exercise: Implement the following search procedure to find a good model

 You are given a list of possible values of `C` below

 For each C:

 Create a logistic regression model with that value of C

 Find the average score for this model using the `cv_score` function **only on the training set** `(Xlr, ylr)`

 Pick the C with the highest average score

Your goal is to find the best model parameters based *only* on the training set, without showing the model test set at all (which is why the test set is also called a *hold-out* set).

Can you write Python code for it?

# your turn
# the grid of parameters to search over
Cs = [0.001, 0.1, 1, 10, 100]
best_score = 0

for i in Cs:
    clf = LogisticRegression(C=i)
    score = cv_score(clf, Xlr, ylr)
    if score > best_score:
        best_score = score
        print('C = ', i, 'score: ', score)


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:

Checkup Exercise Set III

**Exercise:** Now you want to estimate how this model will predict on unseen data in the following way:

 Use the C you obtained from the procedure earlier and train a Logistic Regression on the training data

 Calculate the accuracy on the test data

You may notice that this particular value of `C` may or may not do as well as simply running the default model on a random train-test split.

 Do you think that's a problem?

 Why do we need to do this whole cross-validation and grid search stuff anyway?

Can you write Python code for it?

# your turn
clf = LogisticRegression(C=0.1)
score = cv_score(clf, Xtestlr, ytestlr)
print(score)


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:
Logistic Regression: The Math
We could approach classification as linear regression, there the class, 0 or 1, is the target variable $y$. But this ignores the fact that our output $y$ is discrete valued, and futhermore, the $y$ predicted by linear regression will in general take on values less than 0 and greater than 1. Additionally, the residuals from the linear regression model will not be normally distributed. This violation means we should not use linear regression.
But what if we could change the form of our hypotheses $h(x)$ instead?
The idea behind logistic regression is very simple. We want to draw a line in feature space that divides the '1' samples from the '0' samples, just like in the diagram above. In other words, we wish to find the "regression" line which divides the samples. Now, a line has the form $w_1 x_1 + w_2 x_2 + w_0 = 0$ in 2-dimensions. On one side of this line we have
$$w_1 x_1 + w_2 x_2 + w_0 \ge 0,$$
and on the other side we have
$$w_1 x_1 + w_2 x_2 + w_0 < 0.$$
Our classification rule then becomes:
\begin{eqnarray}
y = 1 &\mbox{if}& \v{w}\cdot\v{x} \ge 0\
y = 0 &\mbox{if}& \v{w}\cdot\v{x} < 0
\end{eqnarray}
where $\v{x}$ is the vector ${1,x_1, x_2,...,x_n}$ where we have also generalized to more than 2 features.
What hypotheses $h$ can we use to achieve this? One way to do so is to use the sigmoid function:
$$h(z) = \frac{1}{1 + e^{-z}}.$$
Notice that at $z=0$ this function has the value 0.5. If $z > 0$, $h > 0.5$ and as $z \to \infty$, $h \to 1$. If $z < 0$, $h < 0.5$ and as $z \to -\infty$, $h \to 0$. As long as we identify any value of $y > 0.5$ as 1, and any $y < 0.5$ as 0, we can achieve what we wished above.
This function is plotted below:
Can you write Python code for it?

h = lambda z: 1. / (1 + np.exp(-z))
zs=np.arange(-5, 5, 0.1)
plt.plot(zs, h(zs), alpha=0.5);


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:
Training and Test Datasets
When fitting models, we would like to ensure two things:

We have found the best model (in terms of model parameters).
The model is highly likely to generalize i.e. perform well on unseen data.

Purpose of splitting data into Training/testing sets

 We built our model with the requirement that the model fit the data well.

 As a side-effect, the model will fit THIS dataset well. What about new data?

 We wanted the model for predictions, right?

 One simple solution, leave out some data (for testing) and train the model on the rest

 This also leads directly to the idea of cross-validation, next section.

First, we try a basic Logistic Regression:

Split the data into a training and test (hold-out) set
Train on the training set, and test for accuracy on the testing set
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Split the data into a training and test set.
Xlr, Xtestlr, ylr, ytestlr = train_test_split(dflog[['Height','Weight']].values,
                                              (dflog.Gender == "Male").values,random_state=5)

clf = LogisticRegression()
# Fit the model on the trainng data.
clf.fit(Xlr, ylr)
# Print the accuracy from the testing data.
print(accuracy_score(clf.predict(Xtestlr), ytestlr))


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:
Tuning the Model
The model has some hyperparameters we can tune for hopefully better performance. For tuning the parameters of your model, you will use a mix of cross-validation and grid search. In Logistic Regression, the most important parameter to tune is the regularization parameter C. Note that the regularization parameter is not always part of the logistic regression model.
The regularization parameter is used to control for unlikely high regression coefficients, and in other cases can be used when data is sparse, as a method of feature selection.
You will now implement some code to perform model tuning and selecting the regularization parameter $C$.
We use the following cv_score function to perform K-fold cross-validation and apply a scoring function to each test fold. In this incarnation we use accuracy score as the default scoring function.
Can you write Python code for it?

from sklearn.model_selection import KFold
from sklearn.metrics import accuracy_score

def cv_score(clf, x, y, score_func=accuracy_score):
    result = 0
    nfold = 5
    for train, test in KFold(nfold).split(x): # split data into train/test groups, 5 times
        clf.fit(x[train], y[train]) # fit
        result += score_func(clf.predict(x[test]), y[test], normalize=True) # evaluate score function on held-out data
    return result / nfold # average


I am working on the file "Mini_Project_Logistic_Regression.ipynb".
The first task is:
We then use this best classifier to fit the entire training set. This is done inside the do_classify function which takes a dataframe indf as input. It takes the columns in the list featurenames as the features used to train the classifier. The column targetname sets the target. The classification is done by setting those samples for which targetname has value target1val to the value 1, and all others to 0. We split the dataframe into 80% training and 20% testing by default, standardizing the dataset if desired. (Standardizing a data set involves scaling the data so that it has 0 mean and is described in units of its standard deviation. We then train the model on the training set using cross-validation. Having obtained the best classifier using cv_optimize, we retrain on the entire training set and calculate the training and testing accuracy, which we print. We return the split data and the trained classifier.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

def do_classify(clf, parameters, indf, featurenames, targetname, target1val, standardize=False, train_size=0.8):
    subdf=indf[featurenames]
    if standardize:
        subdfstd=(subdf - subdf.mean())/subdf.std()
    else:
        subdfstd=subdf
    X=subdfstd.values
    y=(indf[targetname].values==target1val)*1
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, train_size=train_size)
    clf = cv_optimize(clf, parameters, Xtrain, ytrain)
    clf=clf.fit(Xtrain, ytrain)
    training_accuracy = clf.score(Xtrain, ytrain)
    test_accuracy = clf.score(Xtest, ytest)
    print("Accuracy on training data: {:0.2f}".format(training_accuracy))
    print("Accuracy on test data:     {:0.2f}".format(test_accuracy))
    return clf, Xtrain, ytrain, Xtest, ytest


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:

 good words         P(fresh | word)

      touching 0.96

      delight 0.95

      delightful 0.95

     brilliantly 0.94

 energetic 0.94

       superb 0.94

      ensemble 0.93

      childhood 0.93

     engrossing 0.93

     absorbing 0.93

  Bad words         P(fresh | word)

              sorry 0.13

            plodding 0.13

          dull 0.11

           bland 0.11

     disappointing 0.10

           forced 0.10

    uninspired 0.08

      pointless 0.07

   unfortunately 0.07

           stupid 0.06

Exercise Set VI

Exercise: Why does this method work? What does the probability for each row in the identity matrix represent

The above exercise is an example of feature selection. There are many other feature selection methods. A list of feature selection methods available in sklearn is here. The most common feature selection technique for text mining is the chi-squared $\left( \chi^2 \right)$ method.
Prediction Errors
We can see mis-predictions as well.
Can you write Python code for it?

x, y = make_xy(critics, vectorizer)

prob = clf.predict_proba(x)[:, 0]
predict = clf.predict(x)

bad_rotten = np.argsort(prob[y == 0])[:5]
bad_fresh = np.argsort(prob[y == 1])[-5:]

print("Mis-predicted Rotten quotes")
print('---------------------------')
for row in bad_rotten:
    print(critics[y == 0].quote.iloc[row])
    print("")

print("Mis-predicted Fresh quotes")
print('--------------------------')
for row in bad_fresh:
    print(critics[y == 1].quote.iloc[row])
    print("")


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set I

Exercise/Answers:

 Look at the histogram above. Tell a story about the average ratings per critic.
 The average fresh ratings per critic is around 0.6 with a minimum ratings of 0.35 and max of 0.81

 What shape does the distribution look like?
 The shape looks like a normal distribution or bell shape

 What is interesting about the distribution? What might explain these interesting things?

The Vector Space Model and a Search Engine
All the diagrams here are snipped from Introduction to Information Retrieval by Manning et. al. which is a great resource on text processing. For additional information on text mining and natural language processing, see Foundations of Statistical Natural Language Processing by Manning and Schutze.
Also check out Python packages nltk, spaCy, pattern, and their associated resources. Also see word2vec.
Let us define the vector derived from document $d$ by $\bar V(d)$. What does this mean? Each document is treated as a vector containing information about the words contained in it. Each vector has the same length and each entry "slot" in the vector contains some kind of data about the words that appear in the document such as presence/absence (1/0), count (an integer) or some other statistic. Each vector has the same length because each document shared the same vocabulary across the full collection of documents -- this collection is called a corpus.
To define the vocabulary, we take a union of all words we have seen in all documents. We then just associate an array index with them. So "hello" may be at index 5 and "world" at index 99.
Suppose we have the following corpus:
A Fox one day spied a beautiful bunch of ripe grapes hanging from a vine trained along the branches of a tree. The grapes seemed ready to burst with juice, and the Fox's mouth watered as he gazed longingly at them.
Suppose we treat each sentence as a document $d$. The vocabulary (often called...
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

text = ['Hop on pop', 'Hop off pop', 'Hop Hop hop']
print("Original text is\n{}".format('\n'.join(text)))

vectorizer = CountVectorizer(min_df=0)

# call `fit` to build the vocabulary
vectorizer.fit(text)

# call `transform` to convert text to a bag of words
x = vectorizer.transform(text)

# CountVectorizer uses a sparse array to save memory, but it's easier in this assignment to
# convert back to a "normal" numpy array
x = x.toarray()

print("")
print("Transformed text vector is \n{}".format(x))

# `get_feature_names` tracks which word is associated with each column of the transformed x
print("")
print("Words for each feature:")
print(vectorizer.get_feature_names())

# Notice that the bag of words treatment doesn't preserve information about the *order* of words,
# just their frequency

def make_xy(critics, vectorizer=None):
    #Your code here
    if vectorizer is None:
        vectorizer = CountVectorizer()
    X = vectorizer.fit_transform(critics.quote)
    X = X.tocsc()  # some versions of sklearn return COO format
    y = (critics.fresh == 'fresh').values.astype(np.int)
    return X, y
X, y = make_xy(critics)


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set V: Working with the Best Parameters

Exercise: Using the best value of  `alpha` you just found, calculate the accuracy on the training and test sets. Is this classifier better? Why (not)?

 ANSWER: Yes, it is a better classifier since it improves the accuracy on test data from 72 (`alpha`= .1) to 74 percent (`alpha` = 1)

Can you write Python code for it?

vectorizer = CountVectorizer(min_df=best_min_df)
X, y = make_xy(critics, vectorizer)
xtrain=X[mask]
ytrain=y[mask]
xtest=X[~mask]
ytest=y[~mask]

clf = MultinomialNB(alpha=best_alpha).fit(xtrain, ytrain)

#your turn. Print the accuracy on the test and training dataset
training_accuracy = clf.score(xtrain, ytrain)
test_accuracy = clf.score(xtest, ytest)

print("Accuracy on training data: {:2f}".format(training_accuracy))
print("Accuracy on test data:     {:2f}".format(test_accuracy))

from sklearn.metrics import confusion_matrix
print(confusion_matrix(ytest, clf.predict(xtest)))
print(xtest.shape)


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set VII: Predicting the Freshness for a New Review

Exercise:

 Using your best trained classifier, predict the freshness of the following sentence: *'This movie is not remarkable, touching, or superb in any way'*

 Is the result what you'd expect? Why (not)?
 The predicted result is "Fresh" which is not I expect. The word 'Not' is not taken into account thus the analysis mistakenly predicted it as "Fresh" based on the words remarkable, touching and superb which have a high probability of being a good review.  The solution is to take the analysis into a bi-gram level which will take pair each words together and come up with an analysis based on consecutive pair of words.  This will in effect see that the review is rotten since "not remarkable" will be taken as a negative review.

Can you write Python code for it?

#your turn
# Predicting the Freshness for a New Review
docs_new = ['This movie is not remarkable, touching, or superb in any way']
X_new = vectorizer.transform(docs_new)
X_new = X_new.tocsc()
str = "Fresh" if clf.predict(X_new) == 1 else "Rotten"
print('"', docs_new[0], '"==> ', "", str)


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Aside: TF-IDF Weighting for Term Importance
TF-IDF stands for
Term-Frequency X Inverse Document Frequency.
In the standard CountVectorizer model above, we used just the term frequency in a document of words in our vocabulary. In TF-IDF, we weight this term frequency by the inverse of its popularity in all documents. For example, if the word "movie" showed up in all the documents, it would not have much predictive value. It could actually be considered a stopword. By weighing its counts by 1 divided by its overall frequency, we downweight it. We can then use this TF-IDF weighted features as inputs to any classifier. TF-IDF is essentially a measure of term importance, and of how discriminative a word is in a corpus. There are a variety of nuances involved in computing TF-IDF, mainly involving where to add the smoothing term to avoid division by 0, or log of 0 errors. The formula for TF-IDF in scikit-learn differs from that of most textbooks:
$$\mbox{TF-IDF}(t, d) = \mbox{TF}(t, d)\times \mbox{IDF}(t) = n_{td} \log{\left( \frac{\vert D \vert}{\vert d : t \in d \vert} + 1 \right)}$$
where $n_{td}$ is the number of times term $t$ occurs in document $d$, $\vert D \vert$ is the number of documents, and $\vert d : t \in d \vert$ is the number of documents that contain $t$
Can you write Python code for it?

# http://scikit-learn.org/dev/modules/feature_extraction.html#text-feature-extraction
# http://scikit-learn.org/dev/modules/classes.html#text-feature-extraction-ref
from sklearn.feature_extraction.text import TfidfVectorizer
tfidfvectorizer = TfidfVectorizer(min_df=1, stop_words='english')
Xtfidf=tfidfvectorizer.fit_transform(critics.quote)


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Explore
Can you write Python code for it?

n_reviews = len(critics)
n_movies = critics.rtid.unique().size
n_critics = critics.critic.unique().size

print("Number of reviews: {:d}".format(n_reviews))
print("Number of critics: {:d}".format(n_critics))
print("Number of movies:  {:d}".format(n_movies))

df = critics.copy()
df['fresh'] = df.fresh == 'fresh'
grp = df.groupby('critic')
counts = grp.critic.count()  # number of reviews by each critic
means = grp.fresh.mean()     # average freshness for each critic

means[counts > 100].hist(bins=10, edgecolor='w', lw=1)
plt.xlabel("Average Rating per critic")
plt.ylabel("Number of Critics")
plt.yticks([0, 2, 4, 6, 8, 10]);


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Interpretation
What are the strongly predictive features?
We use a neat trick to identify strongly predictive features (i.e. words).

first, create a data set such that each row has exactly one feature. This is represented by the identity matrix.
use the trained classifier to make predictions on this matrix
sort the rows by predicted probabilities, and pick the top and bottom $K$ rows
Can you write Python code for it?

words = np.array(vectorizer.get_feature_names())

x = np.matrix(np.identity(xtest.shape[1]), copy=False)
probs = clf.predict_log_proba(x)[:, 0]
ind = np.argsort(probs)

good_words = words[ind[:10]]
bad_words = words[ind[-10:]]

good_prob = probs[ind[:10]]
bad_prob = probs[ind[-10:]]

print("Good words\t     P(fresh | word)")
for w, p in list(zip(good_words, good_prob)):
    print("{:>20}".format(w), "{:.2f}".format(1 - np.exp(p)))

print("Bad words\t     P(fresh | word)")
for w, p in list(zip(bad_words, bad_prob)):
    print("{:>20}".format(w), "{:.2f}".format(1 - np.exp(p)))


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Naive Bayes
From Bayes' Theorem, we have that
$$P(c \vert f) = \frac{P(c \cap f)}{P(f)}$$
where $c$ represents a class or category, and $f$ represents a feature vector, such as $\bar V(d)$ as above. We are computing the probability that a document (or whatever we are classifying) belongs to category c given the features in the document. $P(f)$ is really just a normalization constant, so the literature usually writes Bayes' Theorem in context of Naive Bayes as
$$P(c \vert f) \propto P(f \vert c) P(c) $$
$P(c)$ is called the prior and is simply the probability of seeing class $c$. But what is $P(f \vert c)$? This is the probability that we see feature set $f$ given that this document is actually in class $c$. This is called the likelihood and comes from the data. One of the major assumptions of the Naive Bayes model is that the features are conditionally independent given the class. While the presence of a particular discriminative word may uniquely identify the document as being part of class $c$ and thus violate general feature independence, conditional independence means that the presence of that term is independent of all the other words that appear within that class. This is a very important distinction. Recall that if two events are independent, then:
$$P(A \cap B) = P(A) \cdot P(B)$$
Thus, conditional independence implies
$$P(f \vert c)  = \prod_i P(f_i | c) $$
where $f_i$ is an individual feature (a word in this example).
To make a classification, we then choose the class $c$ such that $P(c \vert f)$ is maximal.
There is a small caveat when computing these probabilities. For floating point underflow we change the product into a sum by going into log space. This is called the LogSumExp trick. So:
$$\log P(f \vert c)  = \sum_i \log P(f_i \vert c) $$
There is another caveat. What if we see a term that didn't exist in the training data? This means that $P(f_i \vert c) = 0$ for that term, and thus $P(f \vert c)  = \prod_i P(f_i | c) = 0$, which doesn't help us at all. Instead of using zeros, we add a small...
Can you write Python code for it?

# your turn
# split the data set into a training and test set
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=5)

clf = MultinomialNB()
clf.fit(X_train, y_train)

print('accuracy score on training set: ', clf.score(X_train, y_train))
print('accuracy score on test set: ', clf.score(X_test, clf.predict(X_test)))
print('Noticed that the accuracy on test set is 100%.')
print('The model perfectly predicted if the movie will be rated as fresh based on the reviews')
print('This is a very good classifier')


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Table of Contents

Rotten Tomatoes Dataset
Explore

The Vector Space Model and a Search Engine
In Code

Naive Bayes
Multinomial Naive Bayes and Other Likelihood Functions
Picking Hyperparameters for Naive Bayes and Text Maintenance

Interpretation

Rotten Tomatoes Dataset
Can you write Python code for it?

critics = pd.read_csv('./critics.csv')
#let's drop rows with missing quotes
critics = critics[~critics.quote.isnull()]
critics.head()


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
The parameter $\alpha$ is chosen to be a small value that simply avoids having zeros in the probability computations. This value can sometimes be chosen arbitrarily with domain expertise, but we will use K-fold cross validation. In K-fold cross-validation, we divide the data into $K$ non-overlapping parts. We train on $K-1$ of the folds and test on the remaining fold. We then iterate, so that each fold serves as the test fold exactly once. The function cv_score performs the K-fold cross-validation algorithm for us, but we need to pass a function that measures the performance of the algorithm on each fold.
Can you write Python code for it?

from sklearn.model_selection import KFold
def cv_score(clf, X, y, scorefunc):
    result = 0.
    nfold = 5
    for train, test in KFold(nfold).split(X): # split data into train/test groups, 5 times
        clf.fit(X[train], y[train]) # fit the classifier, passed is as clf.
        result += scorefunc(clf, X[test], y[test]) # evaluate score function on held-out data
    return result / nfold # average


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Try adding supplemental features -- information about genre, director, cast, etc.
Can you write Python code for it?

# Create a random forest classifier. By convention, clf means 'classifier'
#clf = RandomForestClassifier(n_jobs=2)

# Train the classifier to take the training features and learn how they relate
# to the training y (the species)
#clf.fit(train[features], y)

critics.head()


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
Use TF-IDF weighting instead of word counts.
Can you write Python code for it?

# http://scikit-learn.org/dev/modules/feature_extraction.html#text-feature-extraction
# http://scikit-learn.org/dev/modules/classes.html#text-feature-extraction-ref
from sklearn.feature_extraction.text import TfidfVectorizer
tfidfvectorizer = TfidfVectorizer(min_df=1, stop_words='english')
Xtfidf=tfidfvectorizer.fit_transform(critics.quote)
X = Xtfidf.tocsc()  # some versions of sklearn return COO format
y = (critics.fresh == 'fresh').values.astype(np.int)

xtrain=X[mask]
ytrain=y[mask]
xtest=X[~mask]
ytest=y[~mask]

clf = MultinomialNB(alpha=best_alpha).fit(xtrain, ytrain)

#your turn. Print the accuracy on the test and training dataset
training_accuracy = clf.score(xtrain, ytrain)
test_accuracy = clf.score(xtest, ytest)

print("Accuracy on training data: {:2f}".format(training_accuracy))
print("Accuracy on test data:     {:2f}".format(test_accuracy))


I am working on the file "Mini_Project_Naive_Bayes.ipynb".
The first task is:
We use the log-likelihood as the score here in scorefunc. The higher the log-likelihood, the better. Indeed, what we do in cv_score above is to implement the cross-validation part of GridSearchCV.
The custom scoring function scorefunc allows us to use different metrics depending on the decision risk we care about (precision, accuracy, profit etc.) directly on the validation set. You will often find people using roc_auc, precision, recall, or F1-score as the scoring function.
Can you write Python code for it?

def log_likelihood(clf, x, y):
    prob = clf.predict_log_proba(x)
    rotten = y == 0
    fresh = ~rotten
    return prob[rotten, 0].sum() + prob[fresh, 1].sum()


I am working on the file "Mini_tutoriel_pour_la_resolution_de_programmes_lineaires_avec_Python__ENS_Rennes_2021.ipynb".
The first task is:
Et voilà, pas plus compliqué !
Vous pourrez observer que le résultat de l'appel à cette fonction (si tout marche bien) est un objet qui encapsule plusieurs choses :

la valeur de la solution, result.x ;
le nombre d'itérations, result.nit ;
l'état des slacks variables (cf cours sur le simplexe), result.slack ;
évaluation de l'état de l'optimisation, result.success, et result.message ;
et même la valeur de la fonction objectif à ce point solution (c'est utile pour gagner du temps, si cette fonction objectif est très couteuse, par exemple quand on apprend un gros réseau de neurone, avec d'autres solveurs).
Can you write Python code for it?


I am working on the file "Mini_tutoriel_pour_la_resolution_de_programmes_lineaires_avec_Python__ENS_Rennes_2021.ipynb".
The first task is:
On trouve donc la solution commerciale optimale pour Charlie le chocolatier : 300 pyramides, et 100 pyramides de luxe.
Exercice 1 : sur ce problème, cherchez quelles contraintes (inégalités) sont saturées (devenues des égalités)
Indice : lire le tableau result.slack et l'interpréter.
Can you write Python code for it?

# TODO
print(result)

print("Variables slack:")
print(result.slack)


I am working on the file "Mini_tutoriel_pour_la_resolution_de_programmes_lineaires_avec_Python__ENS_Rennes_2021.ipynb".
The first task is:
Problème exemple du cours sur le simplexe
Attention, généralement les solveurs cherchent à minimiser la fonction de coût, pas à la maximiser !
C'est un piège classique, on rentre le problème, le solveur répond [0,0,..,0] comme solution, et on ne sait pas ce qui n'a pas marché...
Can you write Python code for it?

# Objective Function: x_1 + 6*x_2 + 13*x_3
# Constraint 1: x_1 <= 200
# Constraint 2: x_2 <= 300
# Constraint 3: x_1 + x_2 + x_3 <= 400
# Constraint 4: x_2 + 3*x_3 <= 600
# les variables sont supposées positives par défaut
# x_1 >= 0
# x_2 >= 0
# x_3 >= 0

result = scipy.optimize.linprog(
    [-1, -6, -13],  # Cost function: -x_1 + -6*x_2 + -13*x_3 to MINIMIZE
    A_ub=[  # Coefficients for inequalities
        [1, 0, 0],  # for C1: 1*x_1 + 0*x_2 + 0*x_3 <= 200
        [0, 1, 0],  # for C2: 0*x_1 + 1*x_2 + 0*x_3 <= 300
        [1, 1, 1],  # for C3: 1*x_1 + 1*x_2 + 1*x_3 <= 400
        [0, 1, 3],  # for C4: 0*x_1 + 1*x_2 + 3*x_3 <= 600
    ],
    b_ub=[200, 300, 400, 600],  # Constraints for inequalities: 200, 300, 400, 600
    bounds=(0, None), # Bounds on x, 0 <= x_i <= +oo by default
    method="simplex",
)

print(result)

if result.success:
    print(f"X1: {round(result.x[0], 2)} chocolats simples")
    print(f"X2: {round(result.x[1], 2)} pyramides")
    print(f"X2: {round(result.x[2], 2)} pyramides de luxe")
else:
    print("No solution")


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
A função linspace(a,b,c) retorna um vetor de c elementos, iniciando em a, terminando em b, igualmente espaçados. Ela gera a sequência de valores de x que vamos passar à função para plotagem.
A função plot(a,b) plota o vetor b contra o vetor a, ambos de mesmo tamanho, na forma de pontos ${(a_1,b_1),...,(a_n,b_n)}$ se a = [a1,...,an] e b = [b1,...,bn]. O comandof(x,.5) gera um vetor de 30 elementos que corresponde à imagem do vetor x sob a função f.
Can you write Python code for it?

p1 = py.plot(x, f(x,.05))
#p2 = py.plot([0,2],[0,2])
#show()
# experimente apagar a porção "p1 = " do código acima
# experimente descomentar a segunda linha


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
Exemplo 2
Can you write Python code for it?

class Complex:
    def __init__(x, realpart, imagpart): # este "x" só pertence ao escopo deste bloco de definição de classe
        x.r = realpart
        x.i = imagpart

x = Complex(2,1) # este "x" é diferente do x utilizado na definição de classe acima
x.i,x.r


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
Exemplo 3
def f(x,c) é a sintaxe de declaração de uma função chamada
 f que recebe x e c como argumentos. Os dois pontos : denotam que há um
 bloco de comandos a partir da linha seguinte, pertencente à
 função f. Neste caso, o bloco é formado por apenas um comando:
 retorne o valor de x**2 + c, lembrando que x**2 é a sintaxe do
 Python para a função quadrado.
Can you write Python code for it?

def f(x,c):
    return x**2 + c

x = np.linspace(-2,2,30)
x


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
Exercício 2
Exemplo:
Can you write Python code for it?

x = np.ones(10)
for i in x:
    if np.sin(i) > .5:
        print(i)

# exclua este comentário


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
Exercício 3
O código usado no exemplo abaixo foi escrito com funções práticas e atalhos para facilitar a leitura e por simplicidade. Mas como você faria sem lançar mão dessas ferramentas?
Exemplo:
Can you write Python code for it?

x = np.ones(10)

def media(vetor):
    return np.sum(vetor)/len(vetor)

a = np.random.rand(30,3)
#y = np.zeros(30)

y = [media(i) for i in a] #opa! Trapaça!

py.plot(y,'b.')

# exclua este comentário


I am working on the file "Minicurso_programação_modelagem_1.2016.1.3.5.ipynb".
The first task is:
Normalmente o Python não mostra instantaneamente os gráficos gerados, então precisamos pedir para que eles sejam mostrados através da função show do módulo Matplotlib. Mas o Jupyter Notebook é configurado para exibir os gráficos da função plot automaticamente sem precisarmos invocar o show.
Exercício 1

http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html
http://docs.scipy.org/doc/numpy/reference/routines.random.html
http://docs.scipy.org/doc/numpy/reference/routines.statistics.html
http://docs.scipy.org/doc/numpy/reference/routines.math.html
Can you write Python code for it?

v1 = [1,2,3]
v2 = [v1,v1]
v3 = range(10)
v1,v2,v3

# exclua este comentário


I am working on the file "Minimal Adaptive Controller.ipynb".
The first task is:
Let's quickly examine what sorts of activity values the neurons give for different inputs.
Can you write Python code for it?

n_neurons = 100
ac = AdaptiveControl(n_inputs=1, n_outputs=1, n_neurons=n_neurons, seed=1)

inputs = np.linspace(-1, 1, 100)
rates = np.zeros((len(inputs), n_neurons))
for i, input in enumerate(inputs):
    current = ac.compute_neuron_input([input])
    activity = ac.neuron(current)
    rates[i, :] = activity
pylab.plot(inputs, rates);


I am working on the file "Minimal_Working_Example.ipynb".
The first task is:
2. Plotting C/T for specific fields and with both heating and cooling pulses
If one wants to plot $C$ instead of $C/T$, replace the "plotHCT" command with "plotHC"
Can you write Python code for it?

f, ax = plt.subplots(figsize=(5,5))

YbTiO.labels=[]  # If with these commands, one needs set labels=[] first.
for jj in range(len(YbTiO.Bfield)):
	B = round(YbTiO.Bfield[jj],-1)
	if B == 1000:
		YbTiO.plotHCT(plt,index=jj,heatingcolor='darkblue',coolingcolor='blue',
			shortpulsecolor='steelblue',Blabels=True, demag=False)
	if B == 3000:
		YbTiO.plotHCT(plt,index=jj,heatingcolor='darkred',coolingcolor='red',
			shortpulsecolor='red',Blabels=True, demag=False)
	if B == 4000:
		YbTiO.plotHCT(plt,index=jj,heatingcolor='darkgreen',coolingcolor='limegreen',
			shortpulsecolor='green',Blabels=True, demag=False)

plt.xlabel("T (K)")
plt.ylabel("C/T $(\\rm{J\> K^{-2} mol^{-1}_{Yb}})$")
plt.title('$\\rm{Yb_2Ti_2O_7}$', fontsize=18)
plt.text(0.02,0.97,'B=0.1T \n$\parallel \\langle 111 \\rangle$',horizontalalignment='left',
	verticalalignment = 'top',fontsize=14, transform=ax.transAxes)
plt.legend(handles=YbTiO.labels,numpoints=1, fontsize=14, handlelength=1.4, frameon=False)
#plt.ylim(0,37)
plt.xlim(0.1,0.8)
plt.tight_layout()


I am working on the file "Minimal_Working_Example.ipynb".
The first task is:
Alternatively, if you want to process and plot the data with an external program, the computed heat capacity traces can be saved to a comma-separated text file with the following command:
Can you write Python code for it?

YbTiO.savetraces('YTO_processedData.txt', Barray=[0.0])  #Barray can be set to all, or a select number of fields.


I am working on the file "Minimal_Working_Example.ipynb".
The first task is:
C: Plot Commands
LinePlot
Plots each heat capacity trace (cooling curves only) individually
Can you write Python code for it?

YbTiO.labels=[]
YbTiO.shortpulselabels=[]
f,ax = plt.subplots(1,1)

YbTiO.lineplot(ax,'All', demag=False)
# If you want to plot a particular set of magnetic fields, change 'All' to a list of fields
# If you want to plot heating pulses as well, add the command "plotHeatPulses=True"

# Label Axes
ax.set_ylabel("$C$ $(\\rm{J\> K^{-1} mol^{-1}_{Yb}})$")
ax.set_xlabel("T (K)")
ax.set_yscale('log')

# Show legend
short_legend=ax.legend(handles=YbTiO.shortpulselabels,labelspacing = 0,handlelength=1.4,
	fontsize=11,frameon=False, bbox_to_anchor=(0.97, 1.0),numpoints=1)
long_legend=ax.legend(handles=YbTiO.labels,labelspacing = 0,handlelength=1.4,fontsize=11,
	frameon=False, bbox_to_anchor=(1.0, 0.92))
ax.add_artist(short_legend)


I am working on the file "Minimal_Working_Example.ipynb".
The first task is:
MeshGrid
Make a 2D plot of heat capacity vs. temperature and field
Can you write Python code for it?

f, ax = plt.subplots(1,1)

# Create 2D grid
xarray = np.arange(0.0,0.9,0.004)
intens, Bedg = YbTiO.meshgrid(Tarray =xarray,Barray='All')
# Plot 2D grid of heat capacity
meshdata = ax.pcolormesh(xarray,Bedg/10000,intens,rasterized = True,
                        cmap = 'rainbow', vmin=0.0, vmax=10)

# Label Axes
ax.set_ylabel('$B_{ext}$ (T)',fontsize=14,labelpad = 4)
ax.set_ylim(-100./10000,1)
ax.set_xlabel("$T$ (K)", labelpad = 1)
ax.set_xlim(0,0.7)
# Colorscale bar
cb=plt.colorbar(meshdata)
cb.set_label('$C$ $(\\rm{J\> K^{-1} mol^{-1}_{Yb}})$', rotation = -90, labelpad = 14)


I am working on the file "Minimal_Working_Example.ipynb".
The first task is:
plotEntropy
This function integrates heat capacity to get entropy with $\int \frac{C}{T} dT = \Delta S$.
It does this by first combining the traces (using the same method as lineplotcombine, which the "smooth" parameter is for) and then integrating over each trace, extrapolating linearly from the lowest temperature data point to $T=0$, $C=0$.
Can you write Python code for it?

#***************
# Plot the entropy
#***************
f, ax = plt.subplots(1,1,figsize=(6,5))
YbTiO.plotEntropy(ax,'All',smooth=3)

# Label axes
ax.set_xlabel('T (K)')
ax.set_ylabel('Entropy $(\\rm{J\> K^{-1} mol^{-1}_{Yb}})$', labelpad = 3)
ax.set_ylim(-0.05,2)
ax.set_title('Entropy Recovered')
# Plot legend
f.subplots_adjust(left = 0.12,right = 0.80, bottom = 0.14, top = 0.92)
ax.legend(handles=YbTiO.entropylabels,labelspacing = 0.1,handlelength=1.4,fontsize=12,
	frameon=False, bbox_to_anchor=(1.3, 1.01))


I am working on the file "Minimized-Volume-Reactors -Series/Minimized-Volume-Reactors -Series.ipynb".
The first task is:
Dibujamos la gráfica
Can you write Python code for it?

def serie(reaccion, reactor_1, reactor_2, X1, X2):

    reactor1 = reactor(0,1,1, reactor_type = "CSTR")
    reactor2 = reactor(0,1,1, reactor_type = "CSTR")

    if reactor_1 == 0:
        reactor1 = reactor(0,X1,reaccion, reactor_type = "CSTR")
    elif reactor_1 == 1:
        reactor1 = reactor(0,X1,reaccion, reactor_type = "PFR")

    if reactor_2 == 0:
        reactor2 = reactor(X1,X2,reaccion, reactor_type = "CSTR")
    elif reactor_2 == 1:
        reactor2 = reactor(X1,X2,reaccion, reactor_type = "PFR")

    #representamos F0*dx/(-ra)
    n = 5000

    x = np.linspace(0,0.99,n)

    n1 = int(n/0.99*X1)
    n2 = int(n/0.99*X2)

    x1 = x[0:n1]
    x2 = x[n1:n2]

    #calculamos reaccion
    reactor0 = reactor(0,0.99,reaccion,reactor_type = "PFR")
    y0 = reactor0.representar(x)

    #calculamos el primer reactor
    y1 = reactor1.representar(x1)

    #calculamos el segundo reactor
    y2 = reactor2.representar(x2)

    #para que fill lo represente bien:
    y1[-1] = 0
    y1[0] = 0

    #evitar error
    if X2 > X1:
        y2[-1] = 0
        y2[0] = 0

    #dibujamos
    plt.figure(figsize=(12, 6))
    plt.plot(x,y0,'black')
    plt.fill(x1,y1)
    if X2 > X1:
        plt.fill(x2,y2)

    plt.ylim(0,10)

    plt.ylabel('F0/(-ra)')
    plt.xlabel('X')

    volume_reactor1 =  reactor1.volumen()
    volume_reactor2 =  reactor2.volumen()

    if volume_reactor2 < 0:
        volume_reactor2 = 0

    volume_total = volume_reactor1+volume_reactor2

    #Use 8 characters, give f decimal places
    reac1_string = "%8.2f" % volume_reactor1
    reac2_string = "%8.2f" % volume_reactor2
    volumetotal_string = "%8.2f" % volume_total

    # Adding these annotations to the graph

    plt.text(1.01, 0, #position (time, temperature)

             'Volumen del reactor 1: ' + reac1_string + \
             ' $ L $ \n'+ \

             'Volumen del reactor 2: ' + reac2_string + \
             ' $ L $...


I am working on the file "Minimos_Cuadrados/.ipynb_checkpoints/Minimos_Cuadrados-checkpoint.ipynb".
The first task is:
$$S_{y} = \sqrt{\frac{1}{n-2}\sum_{i=1}^{n}(y_{i}-a_{0}-a_{1}x_{i})^{2}}$$
$$S_{my} = \frac{S_{y}}{n^{1/2}}$$
$$S_{ma0}^{2}= \frac{S_{my}^{2}\sum x_{i}^{2}}{n\sum x_{i}^{2} - (\sum x_{i})^{2}}$$
$$ S_{ma1}^{2}= \frac{n S_{my}^{2}}{n\sum x_{i}^{2} - (\sum x_{i})^{2}}$$
Can you write Python code for it?

#desviacion estandar

Sy=0
for i in xrange(0,len(y)):
    Sy+= (y[i]-a0-a1*x[i])**2
    #print Sy
Sy*=(1/(len(y)-2.))
#print Sy
Sy=Sy**(1/2)
print 'Sy %.1f'%Sy

#error en y
raiz= np.sqrt(n)
Smy=Sy/(raiz)
print 'Smy %.1f'%Smy

#error en a0
S2_ma0=(Smy*Smy*suma_xi2)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma0

#error en a0
S2_ma1=(Smy*Smy*n)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma1


I am working on the file "Minimos_Cuadrados/.ipynb_checkpoints/Minimos_Cuadrados-checkpoint.ipynb".
The first task is:
$$a_{0}\pm S_{ma0} $$     $$ a_{1}\pm S_{ma1} $$
Can you write Python code for it?

print 'a0 ± sma0: %f ± %f'%(a0,np.sqrt(S2_ma0))
print 'a1 ± sma1: %f ± %f'%(a1,np.sqrt(S2_ma1))


I am working on the file "Minimos_Cuadrados/.ipynb_checkpoints/Minimos_Cuadrados-checkpoint.ipynb".
The first task is:
2- Aplique paso a paso el método de mínimos cuadrados de tal forma que le permita obtener la mejor curva lineal de ajuste de los datos anteriores, y determine las incertezas asociadas a los parámetros. Determine los coeficientes de correlación $ \chi ^{2} $ y  $ R ^{2} $.  Reporte correctamente los parámetros con su incertidumbre y concluya sobre la conveniencia de la regresión lineal a partir de las correlaciones obtenidas.
a)  $$a_{0}= \frac{(\sum x_{i}^{2})\sum y_{i} - (\sum x_{i})(\sum x_{i}y_{i})}{n\sum x_{i}^{2} - (\sum x_{i})^{2}}$$
Can you write Python code for it?

#n:
n=len(X1)

#suma xi cuadrado:
suma_xi2=0
for i in xrange(0,n):
    suma_xi2+=(X1[i]*X1[i])

#suma yi cuadrado:
suma_yi2=0
for i in xrange(0,n):
    suma_yi2+=(Y1[i]*Y1[i])

#suma xi simple:
suma_xi=0
for i in xrange(0,n):
    suma_xi+=(X1[i])

#suma yi simple:
suma_yi=0
for i in xrange(0,n):
    suma_yi+=(Y1[i])

#suma xi*yi:
suma_xiyi=0
for i in xrange(0,n):
    suma_xiyi+=(X1[i]*Y1[i])

a0=((suma_xi2*suma_yi)-(suma_xi*suma_xiyi))/(n*suma_xi2-(suma_xi*suma_xi))
print 'a0 =  %.1f'%a0


I am working on the file "Minimos_Cuadrados/.ipynb_checkpoints/Minimos_Cuadrados-checkpoint.ipynb".
The first task is:
c) $$y= a_{0}+a_{1}x$$
Can you write Python code for it?

x=np.linspace(X1[0],X1[-1],n)
y=(a0 +a1*x)

print
print 'grafica de los puntos con ajuste: '
fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(X1,Y1,'ro')
ax.plot(x,y,'b-')
ax.set_xlim(xmin=0.0, xmax=8)
ax.set_ylim(ymin=0.0, ymax=8)
plt.show()


I am working on the file "Minimos_Cuadrados/Minimos_Cuadrados.ipynb".
The first task is:
$$a_{0}\pm S_{ma0} $$     $$ a_{1}\pm S_{ma1} $$
Can you write Python code for it?

print 'a0 ± sma0: %f ± %f'%(a0,np.sqrt(S2_ma0))
print 'a1 ± sma1: %f ± %f'%(a1,np.sqrt(S2_ma1))
print ('y= (%.2f±%.2f)x+(%.2f±%.2f)'%(a0,np.sqrt(S2_ma0),a1,np.sqrt(S2_ma1)))


I am working on the file "Minimos_Cuadrados/Minimos_Cuadrados.ipynb".
The first task is:
2- Aplique paso a paso el método de mínimos cuadrados de tal forma que le permita obtener la mejor curva lineal de ajuste de los datos anteriores, y determine las incertezas asociadas a los parámetros. Determine los coeficientes de correlación $ \chi ^{2} $ y  $ R ^{2} $.  Reporte correctamente los parámetros con su incertidumbre y concluya sobre la conveniencia de la regresión lineal a partir de las correlaciones obtenidas.
a)  $$a_{0}= \frac{(\sum x_{i}^{2})\sum y_{i} - (\sum x_{i})(\sum x_{i}y_{i})}{n\sum x_{i}^{2} - (\sum x_{i})^{2}}$$
Can you write Python code for it?


I am working on the file "Minimos_Cuadrados/Minimos_Cuadrados.ipynb".
The first task is:
B- Ajuste de Curva.
Tabla de datos:
|    X |  Y   |
|------|------|
|1.0| 2.1|
|2.0| 4.3|
|3.0| 6.0|
|4.0| 7.8|
1 - Encontrar la función que mejor se ajuste a los datos.
Pruebe con las siguientes regresiones: LINEAL “y=ax + “b, y CUADRADA “$y=ax^{2}$”. Redefina la función cuadrada de forma que quede lineal y pueda usar todo lo que ya aplicó sobre regresión lineal.
Lineal
Can you write Python code for it?

X2=[1.0,2.0,3.0,4.0]
Y2=[2.1,4.3,6.0,7.8]

print
print 'grafica preliminar de los puntos: '
fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(X2,Y2,'o')
ax.set_xlim(xmin=0.0, xmax=10)
ax.set_ylim(ymin=0.0, ymax=10)
plt.show()

#n:
n2=len(X2)

#suma xi cuadrado:
suma_xi2=0
for i in xrange(0,n2):
    suma_xi2+=(Y2[i]*X2[i])

#suma yi cuadrado:
suma_yi2=0
for i in xrange(0,n2):
    suma_yi2+=(Y2[i]*Y2[i])

#suma xi simple:
suma_xi=0
for i in xrange(0,n2):
    suma_xi+=(X2[i])

#suma yi simple:
suma_yi=0
for i in xrange(0,n2):
    suma_yi+=(Y2[i])

#suma xi*yi:
suma_xiyi=0
for i in xrange(0,n2):
    suma_xiyi+=(X2[i]*Y2[i])

a0=((suma_xi2*suma_yi)-(suma_xi*suma_xiyi))/(n2*suma_xi2-(suma_xi*suma_xi))
print 'a0 =  %.1f'%a0

a1=((n2*suma_xiyi)-(suma_xi*suma_yi))/(n2*suma_xi2-(suma_xi*suma_xi))
print 'a1 =  %.1f'%a1

x=np.linspace(X2[0],X2[-1],n2)
y=(a0 +a1*x)

print
print 'grafica de los puntos con ajuste: '
fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(X2,Y2,'ro')
ax.plot(x,y,'b-')
ax.set_xlim(xmin=0.0, xmax=8)
ax.set_ylim(ymin=0.0, ymax=8)
plt.show()

#desviacion estandar

Sy=0
for i in xrange(0,n2):
    Sy+= (y[i]-a0-a1*x[i])**2
    #print Sy
Sy*=(1/(len(y)-2.))
#print Sy
Sy=Sy**(1/2)
print 'Sy %.1f'%Sy

#error en y
raiz= np.sqrt(n)
Smy=Sy/(raiz)
print 'Smy %.1f'%Smy

#error en a0
S2_ma0=(Smy*Smy*suma_xi2)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma0

#error en a0
S2_ma1=(Smy*Smy*n)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma1

print 'a0 ± sma0: %f ± %f'%(a0,np.sqrt(S2_ma0))
print 'a1 ± sma1: %f ± %f'%(a1,np.sqrt(S2_ma1))
print ('y= (%.2f±%.2f)x+(%.2f±%.2f)'%(a0,np.sqrt(S2_ma0),a1,np.sqrt(S2_ma1)))

#chi2

chi2=0

for i in xrange(0,n2):
    chi2=((y[i]-Y2[i])**2)/Y2[i]

print 'chi^2 = ',chi2

# r2

b=a1
bprima=a1=((n2*suma_xiyi)-(suma_xi*suma_yi))/(n2*suma_yi2-(suma_yi*suma_yi))

r2=b*bprima
print 'r^2 = ',r2


I am working on the file "Minimos_Cuadrados/Minimos_Cuadrados.ipynb".
The first task is:
c) $$y= a_{0}+a_{1}x$$
Can you write Python code for it?


I am working on the file "Minimos_Cuadrados/Minimos_Cuadrados.ipynb".
The first task is:
cuadratica
linealización, realizada a partir de la ecuacion dada para linealización potencial en: https://aarrietaj.files.wordpress.com/2012/02/linealizacion.pdf
Can you write Python code for it?

logX2=np.log(X2)
logY2=np.log(Y2)

print
print 'grafica preliminar de los puntos: '
fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(logX2,logY2,'o')
ax.set_xlim(xmin=0.0, xmax=2)
ax.set_ylim(ymin=0.0, ymax=4)
plt.show()

#n:
n2=len(logX2)

#suma xi cuadrado:
suma_xi2=0
for i in xrange(0,n2):
    suma_xi2+=(logY2[i]*logX2[i])

#suma yi cuadrado:
suma_yi2=0
for i in xrange(0,n2):
    suma_yi2+=(logY2[i]*logY2[i])

#suma xi simple:
suma_xi=0
for i in xrange(0,n2):
    suma_xi+=(logX2[i])

#suma yi simple:
suma_yi=0
for i in xrange(0,n2):
    suma_yi+=(logY2[i])

#suma xi*yi:
suma_xiyi=0
for i in xrange(0,n2):
    suma_xiyi+=(logX2[i]*logY2[i])

a0=((suma_xi2*suma_yi)-(suma_xi*suma_xiyi))/(n2*suma_xi2-(suma_xi*suma_xi))
print 'a0 =  %.1f'%a0

a1=((n2*suma_xiyi)-(suma_xi*suma_yi))/(n2*suma_xi2-(suma_xi*suma_xi))
print 'a1 =  %.1f'%a1

#desviacion estandar

Sy=0
for i in xrange(0,len(y)):
    Sy+= (y[i]-a0-a1*x[i])**2
    #print Sy
Sy*=(1/(len(y)-2.))
#print Sy
Sy=Sy**(1/2)
print 'Sy %.1f'%Sy

#error en y
raiz= np.sqrt(n)
Smy=Sy/(raiz)
print 'Smy %.1f'%Smy

#error en a0
S2_ma0=(Smy*Smy*suma_xi2)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma0

#error en a0
S2_ma1=(Smy*Smy*n)/(n*suma_xi2-(suma_xi*suma_xi))
print 'S2_ma0 %f'%S2_ma1

print 'a0 ± sma0: %f ± %f'%(a0,np.sqrt(S2_ma0))
print 'a1 ± sma1: %f ± %f'%(a1,np.sqrt(S2_ma1))
print ('y= (%.2f±%.2f)x+(%.2f±%.2f)'%(a0,np.sqrt(S2_ma0),a1,np.sqrt(S2_ma1)))

x=np.linspace(logX2[0],logX2[-1],n2)
y=(a0 +a1*x)

print
print 'grafica de los puntos con ajuste: '
fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(logX2,logY2,'ro')
ax.plot(x,y,'b-')
ax.set_xlim(xmin=0.0, xmax=2)
ax.set_ylim(ymin=0.0, ymax=5)

#chi2

chi2=0

for i in xrange(0,n2):
    chi2=((y[i]-Y2[i])**2)/Y2[i]

print 'chi^2 = ',chi2

# r2

b=a1
bprima=a1=((n2*suma_xiyi)-(suma_xi*suma_yi))/(n2*suma_yi2-(suma_yi*suma_yi))

r2=b*bprima
print 'r^2 = ',r2


I am working on the file "Mining massive datasets/Data stream mining.ipynb".
The first task is:
Question 2
We wish to estimate the surprise number (2nd moment) of a data stream, using the method of AMS. It happens that our stream consists of ten different values, which we'll call 1, 2,..., 10, that cycle repeatedly. That is, at timestamps 1 through 10, the element of the stream equals the timestamp, at timestamps 11 through 20, the element is the timestamp minus 10, and so on. It is now timestamp 75, and a 5 has just been read from the stream. As a start, you should calculate the surprise number for this time.
For our estimate of the surprise number, we shall choose three timestamps at random, and estimate the surprise number from each, using the AMS approach (length of the stream times 2m-1, where m is the number of occurrences of the element of the stream at that timestamp, considering all times from that timestamp on, to the current time). Then, our estimate will be the median of the three resulting values.
You should discover the simple rules that determine the estimate derived from any given timestamp and from any set of three timestamps. Then, identify the set of three "random" timestamps that give the closest estimate from {20, 49, 53}, {17, 43, 51}, {25, 34, 47}, {37, 46, 55}.
Can you write Python code for it?

import json
import pprint

def surprise(seq):
    timestamp = {}
    for n in seq:
        timestamp[n] = timestamp.get(n, 0) + 1
    pprint.pprint(timestamp)
    return sum([v*v for k,v in timestamp.items()])

triples = [[20, 49, 53],
           [17, 43, 51],
           [25, 34, 47],
           [37, 46, 55]]

def median(triple):
    return sorted(triple)[len(triple)>>1]

def estimate(r, s):
    v = [sum([v == s[x-1] and p>=(x-1) for p, v in enumerate(s)]) for x in r]
    return [len(s) * (2*m - 1) for m in v]

def main():
    s = [(i % 10) + 1 for i in range(0, 75)]
    print surprise(s)

    for a in triples:
        print '%s => %d' % (json.dumps(a), median(estimate(a, s)))

    return 0

if __name__ == '__main__':
    main()


I am working on the file "Mining massive datasets/Data stream mining.ipynb".
The first task is:
Question 3
We wish to use the Flagolet-Martin lgorithm to count the number of distinct elements in a stream. Suppose that there ten possible elements, 1, 2,..., 10, that could appear in the stream, but only four of them have actually appeared. To make our estimate of the count of distinct elements, we hash each element to a 4-bit binary number. The element x is hashed to 3x + 7 (modulo 11). For example, element 8 hashes to 3x8+7 = 31, which is 9 modulo 11 (i.e., the remainder of 31/11 is 9). Thus, the 4-bit string for element 8 is 1001.
A set of four of the elements 1 through 10 could give an estimate that is exact (if the estimate is 4), or too high, or too low. You should figure out under what circumstances a set of four elements falls into each of those categories. Then, identify the set of four elements that gives the exactly correct estimate from {3, 4, 8, 10}, {1, 2, 3, 9}, {4, 5, 6, 7}, { 3, 7, 8, 10}.
Can you write Python code for it?

import json

streams = [[3, 4, 8, 10],
           [1, 2, 3, 9],
           [4, 5, 6, 7],
           [3, 7, 8, 10]]

def mod(x):
    return (3*x + 7) % 11

def bit(x):
    m = [0xf, 0x7, 0x3, 0x1]
    for i in range(0, 4):
        if m[i] & x == 0:
            return 4-i
    return 0

def main():
    for stream in streams:
        r = max([bit(mod(x)) for x in stream])
        print '%s = %d' % (json.dumps(stream), r*r)

if __name__ == '__main__':
    main()


I am working on the file "Mining massive datasets/Dimensionality Reduction.ipynb".
The first task is:
Question 2
This is a table giving the profile of three items:
|A |1 |0 |1 |0 |1 |2 |
|-|-|-|-|-|-|-|
|B |1 |1 |0 |0 |1 |6 |
|C |0 |1 |0 |1 |0 |2 |
The first five attributes are Boolean, and the last is an integer "rating." Assume that the scale factor for the rating is α. Compute, as a function of α, the cosine distances between each pair of profiles. For each of α = 0, 0.5, 1, and 2, determine the cosine of the angle between each pair of vectors.
Can you write Python code for it?

import numpy as np
import math

def vectorLength(v):
    return math.sqrt(sum([vi**2 for vi in v]))

def vectorProducts(v1,v2):
    return np.dot(v1,v2)/(vectorLength(v1)*vectorLength(v2))

def cosineAngle(v, alpha):
    va = [vi for vi in v]
    va[-1] *= alpha
    return va

A = [1,0,1,0,1,2]
B = [1,1,0,0,1,6]
C = [0,1,0,1,0,2]

alphaVec = [0,0.5,1,2]

for alpha in alphaVec:

    AA = cosineAngle(A, alpha)
    AB = cosineAngle(B, alpha)
    AC = cosineAngle(C, alpha)

    cosAB = vectorProducts(AA,AB)
    cosAC = vectorProducts(AA,AC)
    cosBC = vectorProducts(AB,AC)

    print "Alpha: %s" % alpha
    print "cos(A,B): %s" % (cosAB)
    print "cos(A,C): %s" % (cosAC)
    print "cos(B,C): %s" % (cosBC)
    print "\n",


I am working on the file "Mining massive datasets/Dimensionality Reduction.ipynb".
The first task is:
Question 3
In this question, all columns will be written in their transposed form, as rows, to make the typography simpler. Matrix M has three rows and two columns, and the columns form an orthonormal basis. One of the columns is [2/7,3/7,6/7]. There are many options for the second column [x,y,z]. Write down those constraints on x, y, and z. Then, identi fy in the list below the one column that could be [x,y,z]. All components are computed to three decimal places, so the constraints may be satisfied only to a close approximation.
Can you write Python code for it?


I am working on the file "Mining massive datasets/Dimensionality Reduction.ipynb".
The first task is:
Question 5
Identify the vector that is orthogonal to the vector [1,2,3].
Can you write Python code for it?


I am working on the file "Mining massive datasets/Dimensionality Reduction.ipynb".
The first task is:
Question 6
Consider the diagonal matrix M =
|1 |0 |0 |
|-|-|-|
|0 |2 |0 |
|0 |0 |0 |
Compute its Moore-Penrose pseudoinverse,
Can you write Python code for it?

import numpy as np
M = np.array([[1,0,0],[0,2,0],[0,0,0]])
print M

MInv = np.linalg.pinv(M)
print MInv


I am working on the file "Mining massive datasets/Locality Sensitive Hashing.ipynb".
The first task is:
Question 4
Find the set of 2-shingles for the "document": ABRACADABRA and also for the "document": BRICABRAC. Answer the following questions:

How many 2-shingles does ABRACADABRA have?
How many 2-shingles does BRICABRAC have?
How many 2-shingles do they have in common?
What is the Jaccard similarity between the two documents"?
Can you write Python code for it?

str1 = "ABRACADABRA"
str2 = "BRICABRAC"

def getShingles(str):
    shingles = []
    doc_shingles = []
    for i in range(len(str)-1):
        shingles.append(str[i]+str[i+1])
    return set(shingles)

def getCommon(list1,list2):
    s1 = set(list1)
    s2 = set(list2)
    return len(s1.intersection(s2))

def getTotal(list1,list2):
    s1 = set(list1)
    s2 = set(list2)
    return len(s1.union(s2))

print "Set of 2-shinlges in ABRACADABRA =",  getShingles(str1)
print "Set of 2-shinlges in BRICABRAC =", getShingles(str2)
print
print "Number of 2-shinlges in ABRACADABRA = ", len(getShingles(str1))
print "Number of 2-shinlges in BRICABRAC = ", len(getShingles(str2))
print
print "Number of Common Shingles = ", getCommon(getShingles(str1),getShingles(str2))
print "Total Number of Shingles = ", getTotal(getShingles(str1),getShingles(str2))


I am working on the file "Mining massive datasets/Locality Sensitive Hashing.ipynb".
The first task is:
Question 5
How many distinct 3-shingles are there in the string "hello world" (excluding the quotes)?
Can you write Python code for it?

str1 = "hello world"

def shingle(str):
    s = set()
    for i,c in enumerate(str):
        if i < len(str) - 2:
            shing = c+str[i+1]+str[i+2]
            print shing
            s.add(shing)
    print
    print s
    return s

shingle1 = shingle(str1)
print
print('Number of 3-shinlges in "hello world" = '+str(len(shingle1)));


I am working on the file "Mining massive datasets/Locality Sensitive Hashing.ipynb".
The first task is:
Question 8
There are 8 strings that represent sets: s1 = abcef; s2 = acdeg; s3 = bcdefg; s4 = adfg; s5 = bcdfgh; s6 = bceg; s7 = cdfg; s8 = abcd. The upper limit on Jaccard distance is 0.2, and we index the strings based on symbols appearing in the prefix. For each of s1, s3, and s6, determine how many other strings that string will be compared with, if it is used as the probe string.
Can you write Python code for it?

# The 8 strings that represent sets:
s1 = "abcef"
s2 = "acdeg"
s3 = "bcdefg"
s4 = "adfg"
s5 = "bcdfgh"
s6 = "bceg"
s7 = "cdfg"
s8 = "abcd"

# Upper limit on Jaccard distance is 0.2
# We index a string of length L on the symbols appearing in its prefix of length floor(0.2L+1)
# Strings of length 5 and 6 are indexed on their first 2 symbols
# Strings of length 4 are indexed on their first symbol
Jaccard = 0.2

def prefxlen(str, Jaccard):
    return int(np.floor(len(str)*Jaccard + 1))

s = [s1, s2, s3, s4, s5, s6, s7, s8]
length = map(lambda s: prefxlen(s, Jaccard), s)
print "Prefix lenth of s1 to s8:", length

# We want to find the strings in index a, b and c
def index(str):
    indx = {}
    for s in str:
        for l in range(prefxlen(s, Jaccard)):
            if s[l] in indx:
                indx[s[l]].append(s)
            else:
                indx[s[l]] = [s]
    return indx

indx = index(s)
print "strings in a, b and c:", indx

# Number of strings compared with each of s1, s3 and s6 when it is used as the probe string
def count(indx, s):
    counts = set(sum([indx[s[l]] for l in range(prefxlen(s, Jaccard))], []))
    counts.remove(s)
    return (s, len(counts), counts)

print "String, number of strings for comparisons, set of strings for comparisons:"
for s in [s1, s3, s6]:
    print count(indx, s)


I am working on the file "Mining massive datasets/MapReduce SVM.ipynb".
The first task is:
Question 2
Use the matrix-vector multiplication and apply the Map function to this matrix and vector:
| 1 | 2 | 3 | 4 | | 1 |
|---|---|---|---| |---|
| 5 | 6 | 7 | 8 | | 2 |
| 9 | 10 | 11 | 12 | | 3 |
| 13 | 14 | 15 | 16 | | 4 |
Then, identify the key-value pairs that are output of Map.
Can you write Python code for it?

import numpy as np
import itertools

M = np.array([[1, 2, 3, 4],
              [5, 6, 7, 8],
              [9, 10, 11, 12],
              [13, 14, 15, 16],])

v = np.array([1, 2, 3, 4])

def mr(M, v):
    t = []
    mr, mc = M.shape
    for i in range(mc):
        for j in range(mr):
            t.append((i, M[i, j] * v[j]))

    t = sorted(t, key=lambda x:x[0])
    for x in t:
        print (x[0]+1, x[1])

    r = np.zeros((mr, 1))
    for key, vals in itertools.groupby(t, key=lambda x:x[0]):
        vals = [x[1] for x in vals]
        r[key] = sum(vals)
        print '%s, %s' % (key, sum(vals))
    return r.transpose()

#print np.dot(M, v.transpose())
print mr(M, v)


I am working on the file "Mining massive datasets/MapReduce SVM.ipynb".
The first task is:
Question 3
Suppose we have the following relations:
Can you write Python code for it?

from IPython.display import Image
Image(filename='relations.jpeg')


I am working on the file "Mining massive datasets/MapReduce SVM.ipynb".
The first task is:
Question 6
Below we see a set of 20 points and a decision tree for classifying the points.
Can you write Python code for it?


I am working on the file "Mining massive datasets/MapReduce SVM.ipynb".
The first task is:
To be precise, the 20 points represent (Age,Salary) pairs of people who do or do not buy gold jewelry. Age (appreviated A in the decision tree) is the x-axis, and Salary (S in the tree) is the y-axis. Those that do are represented by gold points, and those that do not by green points. The 10 points of gold-jewelry buyers are:
(28,145), (38,115), (43,83), (50,130), (50,90), (50,60), (50,30), (55,118), (63,88), and (65,140).
The 10 points of those that do not buy gold jewelry are:
(23,40), (25,125), (29,97), (33,22), (35,63), (42,57), (44, 105), (55,63), (55,20), and (64,37).
Some of these points are correctly classified by the decision tree and some are not. Determine the classification of each point, and then indicate the points that are misclassified.
Can you write Python code for it?


I am working on the file "Mining massive datasets/MapReduce SVM.ipynb".
The first task is:
We propose to use the diagonal line with slope +1 and intercept +2 as a decision boundary, with positive examples above and negative examples below. However, like any linear boundary for this training set, some examples are misclassified. We can measure the goodness of the boundary by computing all the slack variables that exceed 0, and then using them in one of several objective functions. In this problem, we shall only concern ourselves with computing the slack variables, not an objective function.
To be specific, suppose the boundary is written in the form w.x+b=0, where w = (-1,1) and b = -2. Note that we can scale the three numbers involved as we wish, and so doing changes the margin around the boundary. However, we want to consider this specific boundary and margin. Determine the slack for each of the 16 points.
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
Question 2
Consider three Web pages with the following links:
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
Question 3
Consider three Web pages with the following links:
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
Question 4
Consider the link graph below. First, construct the L, the link matrix, as discussed in the HITS algorithm. Then do the following:

Start by assuming the hubbiness of each node is 1; that is, the vector h is (the transpose of) [1,1,1,1].
Compute an estimate of the authority vector $a=L^Th$.
Normalize a by dividing all values so the largest value is 1.
Compute an estimate of the hubbiness vector h=La.
Normalize h by dividing all values so the largest value is 1.
Repeat steps 2-5.

Now, identify the final estimates.
Can you write Python code for it?

Image(filename='pagerank4.jpg')

import numpy as np

# Function to normalize all values so that the largest value is 1
def norm(Matrix):
    return Matrix/float(Matrix.max())

def estimate(L,h):
    # To estimate of the authority vector a = LTh
    #a = L.T*h
    a = np.dot(L.T, h)
    # Normalize a by dividing all values so the largest value is 1
    a = norm(a)
    # To estimate of the hubbiness vector h = La
    #h = L*a
    h = np.dot(L, a)
    # Normalize h by dividing all values so the largest value is 1
    h = norm(h)

    return a,h

# The vector h is (the transpose of) [1,1,1,1]
h = np.matrix([1,1,1,1]).T
# The link graph: 1->2; 1->3; 2->1; 3->4; 4->3
L = np.matrix([[0,1,1,0],
               [1,0,0,0],
               [0,0,0,1],
               [0,0,1,0]])

# After step 1
a,h = estimate(L,h)
print "After step 1:"
print "authority:", np.round(a.T, decimals=3)
print "hubbiness:", np.round(h.T, decimals=3)

# After step 2 (repeat of step 1)
a,h = estimate(L,h)
print "Final estimate:"
print "authority:", np.round(a.T, decimals=3)
print "hubbiness:", np.round(h.T, decimals=3)


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
Question 5
Compute the Topic-Specific PageRank for the following link topology. Assume that pages selected for the teleport set are nodes 1 and 2 and that in the teleport set, the weight assigned for node 1 is twice that of node 2. Assume further that the teleport probability, (1 - beta), is 0.3. Which of the following statements is correct?
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
Suppose we compute PageRank with β=0.85. Write the equations for the PageRanks a, b, and c of the three pages A, B, and C, respectively. Then, identify the correct equations representing a, b and c.
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
TSPR(1) = 0.3576
TSPR(2) = 0.2252
TSPR(3) = 0.2454
TSPR(4) = 0.1718

Question 6
The spam-farm architecture suffers from the problem that the target page has many links --- one to each supporting page. To avoid that problem, the spammer could use the architecture shown below:
Can you write Python code for it?


I am working on the file "Mining massive datasets/Pagerank exercise.ipynb".
The first task is:
There, k "second-tier" nodes act as intermediaries. The target page t has only to link to the k second-tier pages, and each of those pages links to m/k of the m supporting pages. Each of the supporting pages links only to t (although most of these links are not shown). Suppose the taxation parameter is β = 0.85, and x is the amount of PageRank supplied from outside to the target page. Let n be the total number of pages in the Web. Finally, let y be the PageRank of target page t. If we compute the formula for y in terms of k, m, and n, we get a formula with the form
y = ax + bm/n + ck/n
Note: To arrive at this form, it is necessary at the last step to drop a low-order term that is a fraction of 1/n. Determine coefficients a, b, and c, remembering that β is fixed at 0.85. Then, identify the value of these coefficients.
Can you write Python code for it?

import numpy as np
import math

beta = 0.85
a = 1.0/ (1 - np.power(beta, 3))
b = beta / (1.0 + beta + np.power(beta, 2))
c = np.power(beta, 2)/ (1.0 + beta + np.power(beta, 2))
print 'a = %f , b = %f , c = %f' % (a, b, c)


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Bloom Filter
Can you write Python code for it?


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Calculate psi

x: A has bid x for this query
f: Fraction f of the budget of A is currently unspent
Can you write Python code for it?

psi = lambda x, f: x * (1 - e ** (-f))

xs = [1, 2, 3]
fs = [0.9, 0.5, 0.6]

print "If a query arrives that is bidded on by A and B"
for i in [0, 1]:
    print psi(xs[i], fs[i])

print "If a query arrives that is bidded on by A and C"
for i in [0, 2]:
    print psi(xs[i], fs[i])

print "If a query arrives that is bidded on by A and B and C"
for i in [0, 1, 2]:
    print psi(xs[i], fs[i])


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Flajolet-Martin
Can you write Python code for it?

# Decimal number to binary number
def dec2bin(dec):
    return bin(dec)[2:]

# Count the number of trailing zeros
def counttrailingzero(b):
    cnt = 0
    for i in range(len(b))[::-1]:
        if b[i] == '0':
            cnt += 1
        else:
            return cnt
    return cnt

# Given R = max r(a), estimate number of distinct elements
def distinctelements(r):
    return 2 ** r

print counttrailingzero(dec2bin(10))
print counttrailingzero(dec2bin(12))
print counttrailingzero(dec2bin(4))
print counttrailingzero(dec2bin(16))
print counttrailingzero(dec2bin(1))


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Generate M from adjacency list
Can you write Python code for it?

def getM(adjList):
    size = len(adjList)
    M = np.zeros([size, size])
    for u in adjList:
        vs = adjList[u]
        n = len(vs)
        for v in vs:
            M[v, u] = 1.0 / n
    return M


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Generate adjacency list from nodes and edges
Can you write Python code for it?

def getAdjList(nodes, edges):
    nodeMap = {nodes[i] : i for i in range(len(nodes))}
    adjList = {i : [] for i in range(len(nodes))}
    for u, v in edges:
        adjList[nodeMap[u]].append(nodeMap[v])
    return adjList


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Generate adjacency list from nodes and edges
```

nodes = ['yahoo', 'amazon', 'microsoft']
edges = [('yahoo', 'yahoo'), ('yahoo', 'microsoft'), ('yahoo', 'amazon'), ('amazon', 'microsoft'), ('amazon', 'yahoo'), ('microsoft', 'amazon')]
getAdjList(nodes, edges)
{0: [0, 2, 1], 1: [2, 0], 2: [1]}
```
Can you write Python code for it?


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Genereate A from adjacency list
```

adjList = {0: [0, 2, 1], 1: [2, 0], 2: [1]}
A = getA(adjList)
```
Can you write Python code for it?

def getA(adjList):
    N = len(adjList)
    A = np.zeros([N, N])
    for u in adjList:
        vs = adjList[u]
        for v in vs:
            A[u, v] = 1
    return A


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

andor = lambda x, r, b: 1 - (1 - x ** r) ** b
orand = lambda x, r, b: (1 - (1 - x) ** b) ** r
cascade = lambda x, r, b: orand(andor(x, r, b), r, b)

print andor(0.2, 3, 4)

def plot():
    # Variable Initialization
    k = 2
    r = k ** 2
    b = k ** 2

    # AND-OR Construction
    x1 = np.arange(0, 1, 0.01)
    y1 =  andor(x1, r, b)

    # OR-AND Construction
    x2 = np.arange(0, 1, 0.01)
    y2 = orand(x2, r, b)

    # Cascade Construction
    x3 = np.arange(0, 1, 0.01)
    y3 = cascade(x3, k, k)

    # Show plot
    plt.plot(x1, y1, '-r', x2, y2, '-g', x3, y3, '-b')
    plt.grid(True)
    plt.legend(('and-or', 'or-and', 'cascade'))
    #plt.savefig('lsh.pdf')

plot()


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Hits algorithm
Can you write Python code for it?

import numpy as np
from copy import deepcopy


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Jaccard similarity
Can you write Python code for it?

from scipy.spatial.distance import jaccard
from scipy.spatial.distance import cosine

def jaccard_sim(u, v):
    return 1 - jaccard(u, v)

def cosine_sim(u, v):
    return 1 - cosine(u, v)

print jaccard_sim([1,0,0,1,1], [0,1,1,1,0])

print cosine_sim([1,0,1], [0,1,1])

print cosine_sim([5.22, 1.42], [4.06, 6.39])


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Min-hashing algorithm
Can you write Python code for it?

import numpy as np

class MinHashing:
    def __init__(self, mat, hashfunc):
        self.matrix = mat
        self.m = len(mat)
        self.n = len(mat[0])
        self.hashfunc = hashfunc
        self.k = len(self.hashfunc)
    def minhash(self):
        self.sig = np.ones((self.k, self.n)) * 2 ** 10
        for j in range(self.m):
            for c in range(self.n):
                if self.matrix[j][c] == 1:
                    for i in range(self.k):
                        if self.hashfunc[i](j) < self.sig[i][c]:
                            self.sig[i][c] = self.hashfunc[i](j)
    def show(self):
        print self.sig

if __name__ == '__main__':
    hashfunc = [lambda x: (3 * x + 2) % 7, lambda x: (x - 1) % 7]
    mat = [[0,1],[1,0],[0,1],[0,0],[1,1],[1,1],[1,0]]
    mh = MinHashing(mat, hashfunc)
    mh.minhash()
    mh.show()

    # 2009 final
    mat = [[0,0,1],[1,1,1],[0,1,1],[1,0,0],[0,1,0]]
    hashfunc = [lambda x: x + 1, lambda x: (x - 1) % 5 + 1, lambda x: (x - 2) % 5 + 1, lambda x : (x - 3) % 5 + 1, lambda x: (x - 4) % 5 + 1]
    mh = MinHashing(mat, hashfunc)
    mh.minhash()
    mh.show()


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
PageRank
nodes = ['y', 'a', 'm']
edges = [('y', 'y'), ('a', 'm'), ('m', 'm'), ('a', 'y'), ('y', 'a')]
adjList = getAdjList(nodes, edges)
M = getM(adjList)
print pageRank(M, 0.8)
Can you write Python code for it?

def pageRank(M, beta=0.8, epsilon=10**-6, numiter=1000):
    N = len(M)
    const = np.ones(N) * (1 - beta) / N
    rold = np.ones(N) * 1.0 / N
    rnew = np.zeros(N)
    for i in range(numiter):
        rnew = beta * M.dot(rold)
        rnew += const
        if np.sum(np.abs(rold - rnew)) < epsilon:
            break
        rold = rnew
    return rnew


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Random hyperplane
Can you write Python code for it?


I am working on the file "Mining massive datasets/algorithms.ipynb".
The first task is:
Topic-specific pagerank
```

nodes = [1, 2, 3, 4]
edges = [(1, 2), (1, 3), (3, 4), (4, 3), (2, 1)]
adjList = getAdjList(nodes, edges)
M = getM(adjList)
print topicSpecific(M, {0: 1}, 0.8)
print topicSpecific(M, {0: 1}, 0.9)
print topicSpecific(M, {0: 1}, 0.7)
```
Can you write Python code for it?

def topicSpecific(M, S, beta=0.8, epsilon=10**-6, numiter=1000):
    N = len(M)
    rold = np.ones(N) * 1.0 / N
    const = np.zeros(N)
    for i in S:
        const[i] = (1 - beta) * S[i]
    for i in range(numiter):
        rnew = M.dot(rold) * beta
        rnew += const
        if np.sum(np.abs(rold - rnew)) < epsilon:
            break
        rold = rnew
    return rnew

def main():
    nodes = ['y', 'a', 'm']
    edges = [('y', 'y'), ('a', 'm'), ('m', 'm'), ('a', 'y'), ('y', 'a')]
    adjList = getAdjList(nodes, edges)
    M = getM(adjList)
    print pageRank(M, 0.8)

def test():
    nodes = [1,2,3,4,5,6]
    edges = [(1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 1), (2, 1), (2, 4), (2, 6), (2, 3), (2, 5), (3, 2), (3, 6), (3, 5), (4, 1), (4, 6), (5, 2), (5, 3), (6, 1)]
    adjList = getAdjList(nodes, edges)
    M = getM(adjList)
    print M
    print pageRank(M, 0.8)

if __name__ == '__main__':
    test()


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Count all items
Can you write Python code for it?

data = open("browsing.txt","r")
for basket in data:
    #print basket
    transactions += 1
    for item in set(basket.split()):
        #print item
        if item not in mappings:
            mappings.append(item)
            itemCounts.append(1)
        else:
            indexItem = mappings.index(item)
            counter = itemCounts[indexItem]
            counter += 1
            itemCounts[indexItem] = counter

data.close()


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Final sort rules and get top 15 desc
Can you write Python code for it?

cp = sorted(pairRules.iteritems(), key = operator.itemgetter(1))
cp.reverse()
cp5 = [ "%s-->%s  %s" % (mappings[rule[0][0]],mappings[rule[0][1]],rule[1])\
                                   for rule in cp[0:15] ]
print 'Top 15 pairs by confidence:'
print "\n".join(cp5)

ct = sorted(tripleRules.iteritems(), key = operator.itemgetter(1))
ct.reverse()
ct5 = [ "{%s,%s}-->%s  %s" % (mappings[rule[0][0][0]],   \
                              mappings[rule[0][0][1]], \
                              mappings[rule[0][1][0]], \
                              rule[1])\
                            for rule in ct[0:15] ]
print 'Top 15 triples by confidence:'
print "\n".join(ct5)


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Generate candidate triples by frequentPairs JOIN frequentPairs
Can you write Python code for it?

candidateTriples = {}
allCandidateTriples = []
for fcPair in frequentPairs:
    for jp in [joinPair for joinPair in frequentPairs \
              if joinPair[0] == fcPair[1]]:
        allCandidateTriples.append( (fcPair[0],fcPair[1],jp[1]) )


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Generating Rules for confidence
Can you write Python code for it?

def confidence(I,J):
    # Calculate P(IJ)
    PIJ = 0

    IJ = set(I).union(set(J))

    if len(IJ) == 2:
        PIJ = candidatePairs[tuple(sorted(IJ))]
    elif len(IJ) == 3:
        PIJ = candidateTriples[tuple(sorted(IJ))]

    #Calculate P(I)
    PI = 0
    if len(I) == 1:
        PI = itemCounts[I[0]]
    elif len(I) == 2:
        PI = candidatePairs[tuple(sorted(I))]
    if PIJ > PI:
        print I, J, IJ
        print PIJ, PI, PIJ / PI

    return PIJ / PI


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Generating Rules for lift
Can you write Python code for it?

def lift(J,conf):
    if isinstance(J, tuple):
        suppJ = itemCounts[J[0]]
    else:
        suppJ = itemCounts[J]

    SJ = suppJ / transactions
    return conf / SJ

liftedPairRules = { k:lift(k[1],v) for k,v in pairRules.iteritems()}
lp = sorted(liftedPairRules.iteritems(), key = operator.itemgetter(1))
lp.reverse()
lp5 = [ "%s-->%s  %s" % (mappings[rule[0][0]],mappings[rule[0][1]],rule[1])\
                                   for rule in lp[0:15] ]
print 'Top 15 pairs by lift:'
print "\n".join(lp5)

liftedTripleRules = { k:lift(k[1],v) for k,v in tripleRules.iteritems()}
lt = sorted(liftedTripleRules.iteritems(), key = operator.itemgetter(1))
lt.reverse()

lt5 = [ "{%s,%s}-->%s  %s" % (mappings[rule[0][0][0]],   \
                              mappings[rule[0][0][1]], \
                              mappings[rule[0][1][0]], \
                              rule[1])\
                            for rule in lt[0:15] ]
print 'Top 15 triples by lift:'
print "\n".join(lt5)


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Get all candidate pairs (all combination pairs of frequent items).
Can you write Python code for it?

candidatePairs = {}
for pair in itertools.combinations(sorted(frequentItems),2):
    candidatePairs[pair] = 0


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Get count for candidate triples
Can you write Python code for it?

data = open("browsing.txt","r")

for basket in data:
    items = sorted([mappings.index(item) for item in set(basket.split())])

    fPair = []
    for triple in itertools.combinations(items,3):
        if triple in candidateTriples:
            tripleCount = candidateTriples[triple]
            tripleCount = tripleCount +1
            candidateTriples[triple] = tripleCount

data.close()


I am working on the file "Mining massive datasets/association.ipynb".
The first task is:
Get frequent items
Can you write Python code for it?

frequentItems = [mappings.index(item) for item in mappings \
                 if itemCounts[mappings.index(item)] > support]


I am working on the file "Mining massive datasets/dense networks.ipynb".
The first task is:
Community Plots
Can you write Python code for it?

rhoViii = pylab.loadtxt('rhoVeciii')
eViii = pylab.loadtxt('eVeciii')
sTSii = pylab.loadtxt('sizeTSiii')

f3 = pylab.figure(figsize=(9,18))
p1 = f3.add_subplot(311)
p1.plot(rhoViii[:,0], rhoViii[:,1])
p1.set_xlabel("Community #")
p1.set_ylabel("rho(S~)")

p2 = f3.add_subplot(312)
p2.plot(eViii[:,0], eViii[:,1])
p2.set_xlabel("Community #")
p2.set_ylabel("|S~|")

p3 = f3.add_subplot(313)
p3.plot(sTSii[:,0], sTSii[:,1])
p3.set_xlabel("Community #")
p3.set_ylabel("|E(S~)|")


I am working on the file "Mining massive datasets/dense networks.ipynb".
The first task is:
Part 1
The number of iterations needed to find a dense subgraph depends upon the value of $\epsilon$. Show how many iterations it takes to calculate the first dense subgraph when $\epsilon$ = {0.1, 0.5, 1, 2}
Can you write Python code for it?

epsVec = [0.1, 0.5, 1, 2];
iterVec = {}
for i in range(len(epsVec)):
    tS, numIter = findDenseCommunity(V, edgesfile, epsVec[i]);
    iterVec[epsVec[i]] = numIter;
print(iterVec)


I am working on the file "Mining massive datasets/dense networks.ipynb".
The first task is:
Part 3
The algorithm above only describes how to find one dense component (or community). It is also possible to find multiple components by running the above algorithm to find the first dense component and then deleting all vertices (and edges) belonging to that component. To find the next dense component, run the same algorithm on the modified graph. Plot separate graphs showing $\rho(\tilde{S_j})$, |$E[\tilde{S_j}]$| and |$\tilde{S_j}$| as a function of $j$ where $j$ is the current community that has been found. You can stop if 20 communities have already been found. Have $\epsilon$ = 0.05.
Note: the simulation can take over an hour because you have to run the algorithm to find communities 20 times.
Can you write Python code for it?

eps = 0.05;
numCommunities = 20;

def findSeveralDenseCommunities(V, edgesfile, eps, numCommunities):
    rhoVec = [];
    sizeTS = [];
    eVec = [];
    for i in range(numCommunities):
        print(str(i));
        tS, numIter = findDenseCommunity(V, edgesfile, eps);
        sizeTS.append(len(tS));
        eVec.append(countEdges(edgesfile, tS));
        rhoVec.append(computeDensitySet(tS, edgesfile));
        V = V - tS;
    return rhoVec, eVec, sizeTS;

rhoVec, eVec, sizeTS = findSeveralDenseCommunities(V, edgesfile, eps, numCommunities);
writeVectorToFile(rhoVec, 'rhoVeciii');
writeVectorToFile(eVec, 'eVeciii');
writeVectorToFile(sizeTS, 'sizeTSiii');


I am working on the file "Mining massive datasets/dense networks.ipynb".
The first task is:
Single Iteration Plots
Can you write Python code for it?

rhoVii = pylab.loadtxt('rhoVecii')
eVii = pylab.loadtxt('eVecii')
sizSii = pylab.loadtxt('sizeSii')

f2 = pylab.figure(figsize=(9,18))
p1 = f2.add_subplot(311)
p1.plot(rhoVii[:,0], rhoVii[:,1])
p1.set_xlabel("Iteration #")
p1.set_ylabel("rho(S)")

p2 = f2.add_subplot(312)
p2.plot(eVii[:,0], eVii[:,1])
p2.set_xlabel("Iteration #")
p2.set_ylabel("|S|")

p3 = f2.add_subplot(313)
p3.plot(sizSii[:,0], sizSii[:,1])
p3.set_xlabel("Iteration #")
p3.set_ylabel("|E(S)|")


I am working on the file "Mining massive datasets/latent features.ipynb".
The first task is:
Part C
We define the training error as:
$$E_{tr} = \sum\limits_{(i,u) \in \ ratings} (R_{iu} - q_i \cdot p^T_u)^2$$
and the test error as:
$$E_{te} = \sum\limits_{(i,u) \in \ test ratings} (R_{iu} - q_i \cdot p^T_u)^2$$
Note that these equations do not include the regularization penalty that $E$ had from parts a and b. Plot $E_{te}$ and $E_{tr}$ as a function of $k  \in {1,2,...10}$ for the following values of $\lambda$:

$\lambda$ = 0.0 (Part C.i)
$\lambda$ = 0.2 (Part C.ii)

For $\eta$ use the value found in Part B. Based on the output, identify the valid statements.
Ans:

Regularization decreases the test error for $k \ge$ 5
Regularization increases the training error for all (or almost all) $k$
Regularization decreases overfitting

Part C.i
Can you write Python code for it?

K = [i+1 for i in xrange(10)]

e_tot = [0 for i in K]
e_trn = [0 for i in K]
e_tst = [0 for i in K]
count = 0
eta = 0.03
lam = 0.0

for j in K:
    print "k = ",j
    P = create_random_matrix(u, j)
    Q = create_random_matrix(m, j)
    X, Y, E = stochastic_gradient_descent(P,Q,TRAIN_FILE,iter,lam,eta)
    e_tot[count] = E.pop()
    e_trn[count]= t_error(X,Y,TRAIN_FILE)
    e_tst[count]= t_error(X,Y,TEST_FILE)
    print "Error[ Total :%s Training: %s Test: %s]" % \
                (e_tot[count],e_trn[count],e_tst[count])
    count = count + 1

x = pylab.arange(0,len(K))

f1 = pylab.figure(figsize=(9,9))
p1 = f1.add_subplot(211)
p1.plot(x, e_trn)
p1.set_xlabel("Iteration")
p1.set_ylabel("Training Error")
p1.set_title("No regularization : Training Error vs. Iterations")

p2 = f1.add_subplot(212)
p2.plot(x, e_tst)
p2.set_xlabel("Iteration")
p2.set_ylabel("Test Error")
p2.set_title("No regularization : Test Error vs. Iterations")


I am working on the file "Mining massive datasets/recommendation systems.ipynb".
The first task is:
Compute the matrices Q
Can you write Python code for it?

def calcQ(R):
    _, n = R.shape
    Q = np.zeros([n, n])
    for i in range(n):
        Q[i, i] = np.sum(R[:, i])
    return Q


I am working on the file "Mining massive datasets/recommendation systems.ipynb".
The first task is:
Load Alex
Can you write Python code for it?

def loadAlex():
    with open('alex.txt', 'r') as f:
        alex = map(int, f.readline().strip().split())
    return np.array(alex)

def recommend():
    print "load ratings"
    R = np.matrix(loadRatings())
    m, n = R.shape
    print "calc P"
    P = calcP(R)
    print "calc Q"
    Q = calcQ(R)

    Ps = P
    for i in range(m):
        Ps[i, i] = np.sqrt(1.0 / P[i, i])
    Qs = Q
    for i in range(n):
        Qs[i, i] = np.sqrt(1.0 / Q[i, i])

    print "Compute gamma for the user-user collaborative filtering"
    Guu = Ps * R * R.T * Ps * R
    print np.array(Guu[499, :100]).flatten().argsort()[::-1]

    print "Compute gamma for the item-item collaborative filtering"
    Gii = R * Qs * R.T * R * Qs
    print np.array(Gii[499, :100]).flatten().argsort()[::-1]

def plot(uu, ii, alex):
    uuplot = []
    for i in range(19):
        cnt = 0
        for s in uu[:(i+1)]:
            if alex[s] == 1:
                cnt += 1
        uuplot.append(float(cnt) / (i + 1))
    print uuplot
    iiplot = []
    for i in range(19):
        cnt = 0
        for s in ii[:(i+1)]:
            if alex[s] == 1:
                cnt += 1
        iiplot.append(float(cnt) / (i + 1))
    print iiplot

    import matplotlib.pyplot as plt
    plt.plot(range(1, 20), uuplot, 'bs-', range(1, 20), iiplot, 'r^-')
    plt.xlabel('K')
    plt.ylabel('Precision')
    plt.grid(True)
    plt.legend(('User-user', 'Item-item'))

if __name__ == '__main__':
    # recommend()
    uu = map(int, '96 74 45 60  9 68 82  5 72 62 64 59 20 90 97 35 46 76 65 25'.split())
    ii = map(int, '96 74 60 45 82  9 68  5 72 20 62 59 64 90 97 35 25 65 76  2'.split())
    alex = loadAlex()
    plot(uu, ii, alex)


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
8.2 On-Line Algorithms
8.2.1 On-Line and Off-Line Algorithms
Off-Line: The algorithm can access all the data in any order, and produces its answer at the end.
On-Line: The algorithm must decide about each stream element knowing nothing at all of the future.
Since we don't know the future, an on-line algorithm cannot always do as well as an off-line algorithm.
8.2.2 Greedy Algorithms
Greedy: make their decision in response to each input element by maximizing some function of the input element and the past.
might be not optimal.
8.2.3 The Competitive Ratio
an on-line algorithm need not give as good a result as the best off-line algorithm for the same problem:
particular on-line algorithm >= $C \times$ the optimum off-line algorithm, where $C \in (0,1)$ and is called the competitive ratio for the on-line algorithm.
The competitive ratio for an algorithm may depend on what kind of data is allowd to be input to the algorithm.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
8.3 The Matching Problem
bipartite graphs:
graphs with two sets of nodes - left and right - with all edges connecting a node in the left set node to a node in the right set.
Can you write Python code for it?


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
8.3.2 The Greedy Algorithm for Maximal Matching
Off-line algorithm for finding a maximal matching: $O(n^2)$ for an $n$-node graph.
On-line greedy algorithm:
We consider the edges in whatever order they are given.
When we consider $(x,y)$, add this edge to the matching if neither $x$ nor $y$ are ends of any edge selected for the matching so far. Otherwise, skip $(x,y)$.
Can you write Python code for it?

bipartite_graph = [('1', 'a'), ('1', 'c'), ('2', 'b'), ('3', 'b'), ('3', 'd'), ('4', 'a')]
bipartite_graph

logger.setLevel('WARN')

def greedy_maximal_matching(connections):
    maximal_matches = np.array([connections[0]])
    logger.debug('maximal_matches: \n{}'.format(maximal_matches))
    for c in connections[1:]:
        logger.debug('c: {}'.format(c))
        if (c[0] not in maximal_matches[:,0]) and (c[1] not in maximal_matches[:,1]):
            maximal_matches = np.append(maximal_matches, [c], axis=0)
            logger.debug('maximal_matches: \n{}'.format(maximal_matches))

    return maximal_matches

from random import sample
connections = sample(bipartite_graph, len(bipartite_graph))
print('connections: \n{}'.format(connections))
greedy_maximal_matching(bipartite_graph)


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
8.3.3 Competitive Ratio for Greedy Matching
conclusion: The competitive ratio is 1/2 exactly.
The proof is as follows:
<= 1/2
The competitive ratio for the greedy matching cannot be more than 1/2, as shown in Example 8.6.
>= 1/2
The competitive ration is no more than 1/2.
Proof:
Suppose $M$ is bipartitle graph, $M_o$ is a maximal matching, and $M_g$ is the matching of the greedy algorithm.
Let $L = {M_o.l - M_g.l}$, and $R = {r \ | (l,r) \in M; l \in L}$

Lemma (0): $R \subset M_g.r$

Suppose $\forall r \in R, r\notin M_g.r$,
   becase $\exists l \in L, (l,r) \in M$, so $(l,r) \in M_g$. $\implies$ conradiction.

Lemma (1): $|M_o| \leq |M_g| + |L|$

$|M_o| = |M_o.l| = |M_o.l \cap M_g.l| + |L| \leq |M_g| + |L|$

Lemma (2): $|L| \leq |R|$

according to the definition of $R$, one-vs-many might exist.

Lemma (3): $|R| \leq |M_g|$

according to Lemma (0).
Combine Lemma (2) and Lemma (3), we get $|L| \leq |M_g|$. And together with Lemma (1), gives us $|M_o| \leq 2|M_g|$, namely,$$|M_g| \geq \frac{1}{2}|M_o|$$.
Exercises for Section 8.3
8.3.1

$j$ and $k$ cannot be the same for any $i$.

The number of node in $a$ linked to $b_j$ is no more than 2.

Proof:
   because $i = 0, 1, \dotsc, n-1$,
   so $j \in [0, 2, \dotsc, 2n-2] \text{ mod } n$.
hence $j$ is no more than $2n = 2*n$, namely, there are only two node in $a$ can link to any $b_j$.

The number of node in $a$ linked to $b_k$ is no more than 2.
   Proof is similar with (2).

In all, there are only two node in $a$ can link to any node in $b$. So assign $b$ to $a$ one by one, the peferct matching always exists.
8.3.2
Because any node in $b$ has only two links, and also any node in $a$ has only two links. And for any $j$, there has one $k$ paired. Namely, two node of $a$ is full linked to two node of $b$.
num: $n$.
8.3.3
(1) depend on the order of edges.
Can you write Python code for it?

#(2)
from itertools import permutations

stat = []
for connections in permutations(bipartite_graph, len(bipartite_graph)):
    stat.append(greedy_maximal_matching(connections).shape[0])

pd.Series(stat).value_counts()


I am working on the file "Mining_of_Massive_Datasets/Advertising_on_the_Web/note.ipynb".
The first task is:
8.5.2 More Complex Matching Problems

Hard: Matches adwords bids to emails.
   a bid on a set of words $S$ matches an email if all the words in $S$ appear anywhere in the email.

Easy: Matching single words or consecutive sequences of words in a long article
   On-line news sites often push certain news or articles to users who subscribed by keywords or phrases.

8.5.3 A Matching Algorithm for Documents and Bids
match many "bids" against many "documents".
A bid is a (typically small) set of words.
A document is a larger set of words, such as email, tweet, or news article.
We assume there may be hundrends of documents per second arriving, and there are many bids, perhaps on the order of a hundred million or a billion.
representing a bid by its words listed in some order

status: It is an integer indicating how many of the first words on the list have been matched by the current document.

ordering words rarest-first.

We might identify the $n$ most common words are sorted by frequency, and they occupy the end of the list, with the most frequent words at the very end.
All words not among the $n$ most frequent can be assumed equally infrequent and ordered lexicographically.
Can you write Python code for it?

n_common_words = {'the': 0.9, 'and': 0.8, 'twas': 0.3}

def construct_document(doc):
    doc = doc.replace(',','').lower().split(' ')
    com = set(doc).intersection(set(n_common_words.keys()))
    diff = set(doc).difference(set(n_common_words.keys()))

    freq = [n_common_words[x] for x in com]

    freq_sec = [x for (y,x) in sorted(zip(freq, com))]
    rare_sec = sorted(diff)
    sec = ' '.join(rare_sec + freq_sec)
    print(sec)

doc = 'Twas brilling, and the slithy toves'
construct_document(doc)