I am working on the file "Scikit - 02 Visualization.ipynb".
The first task is:
Bar plot using matplotlib visualization
Can you write Python code for it?

plt.figure(figsize = (10, 4))
df.year.value_counts().sort_index().plot.bar()


I am working on the file "Scikit - 02 Visualization.ipynb".
The first task is:
Classificaition plot
Can you write Python code for it?

iris = pd.read_csv("https://raw.githubusercontent.com/abulbasar/data/master/iris.csv")
iris.head()

fig, ax = plt.subplots()
x1, x2 = "SepalLengthCm", "PetalLengthCm"
cmap = sns.color_palette("husl", n_colors=3)
for i, c in enumerate(iris.Species.unique()):
    iris[iris.Species == c].plot.scatter(x1, x2, color = cmap[i], label = c, ax = ax)
plt.legend()


I am working on the file "Scikit - 02 Visualization.ipynb".
The first task is:
Scatter plot using pandas dataframe plot function
Can you write Python code for it?

df.plot.scatter(x= "weight", y = "mpg")
plt.title("Mpg vs Weight, correlation: %.2f" % corr)


I am working on the file "Scikit - 02 Visualization.ipynb".
The first task is:
Show the boxplot of MPG by year
Can you write Python code for it?

plt.figure(figsize=(10, 5))
sns.boxplot("year", "mpg", data = df)


I am working on the file "Scikit - 02 Visualization.ipynb".
The first task is:
Visualization for a single continuous variable
Can you write Python code for it?

plt.hist(df["mpg"], bins = 30)
plt.title("Histogram plot of mpg")
plt.xlabel("mpg")
plt.ylabel("Frequency")

plt.boxplot(df["mpg"])
plt.title("Boxplot of mpg\n ")
plt.ylabel("mpg")

#plt.figure(figsize = (10, 6))
plt.subplot(2, 1, 1)
n, bins, patches = plt.hist(df["mpg"], bins = 50, normed = True)
plt.title("Histogram plot of mpg")
plt.xlabel("MPG")

pdf = normpdf(bins, df["mpg"].mean(), df["mpg"].std())
plt.plot(bins, pdf, color = "red")

plt.subplot(2, 1, 2)
plt.boxplot(df["mpg"], vert=False)
plt.title("Boxplot of mpg")
plt.tight_layout()
plt.xlabel("MPG")

normpdf(bins, df["mpg"].mean(), df["mpg"].std())

# using pandas plot function
plt.figure(figsize = (10, 6))
df.mpg.plot.hist(bins = 50, normed = True)
plt.title("Histogram plot of mpg")
plt.xlabel("mpg")


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
A simpliest prediction model could have been the average. Let's how the model did overall against one feature.
Can you write Python code for it?

X_test_inv = scaler.inverse_transform(X_test)
plt.scatter(X_test_inv[:, 0], y_test, alpha = 0.3, c = "blue", label = "Actual")
plt.scatter(X_test_inv[:, 0], y_test_pred, c = "red", label = "Predicted")

plt.xlabel("R&D Spend")
plt.ylabel("Profit")
plt.title("Profit Actual vs Estimate")
plt.legend()

np.mean((y_test_pred - y_test) ** 2)

y_train_pred = lr.predict(X_train)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Avg Profit is highest in state of Florida and least in California.
Let's create the y vector containing the outcome column.
Can you write Python code for it?


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
By looking at the cofficients, we can conclude that R&D Spend has the higest influence on the outcome variable.
Predict the outcome based on the model
Can you write Python code for it?

y_test_pred = lr.predict(X_test)

output = pd.DataFrame({"actual": y_test, "prediction": y_test_pred})
output["error"] = output.actual - output.prediction
output


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Compare the root mean squared error (RMSE) of test dataset against the training.
Can you write Python code for it?

print("Test rmse: ", sqrt(mean_squared_error(y_test, y_test_pred)),
      "\nTraining rmse:", sqrt(mean_squared_error(y_train, y_train_pred)))


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Create dummy variables for categorical feature.
Can you write Python code for it?

df_features = df.iloc[:, 0:4]
df_dummied = pd.get_dummies(df_features, columns=["State"], drop_first=True)
df_dummied.sample(10)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Encapsulate the steps in a pipeline
Can you write Python code for it?

from sklearn.pipeline import Pipeline

pipeline = Pipeline(steps = [
    ("poly", PolynomialFeatures(degree=2, include_bias=False)),
    ("scaler", StandardScaler()),
    ("lasso", Lasso(alpha=0.03, max_iter=10000, normalize=False, random_state=1))
])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 1)
pipeline.fit(X_train, y_train)
rmse(y_test, pipeline.predict(X_test))


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
From this graph, which alpha values should we select. That question can be answered by looking which alpha values gives the best performance (rmse for example). lassocv function does that for us, or we can use model tuning techniques using grid search - that will be explained later.
Xgboost
Can you write Python code for it?

poly = PolynomialFeatures(degree=2)

X = df.iloc[:, 0:4].values
X_poly = poly.fit_transform(X)
X_poly_train, X_poly_test, y_train, y_test = train_test_split(X_poly, y, test_size = 0.3, random_state = 100)
X_poly_train_std = scaler.fit_transform(X_poly_train)
X_poly_test_std = scaler.transform(X_poly_test)

gbm = xgb.XGBRegressor(max_depth=10, learning_rate=0.1, n_estimators=100,
                     objective='reg:linear', booster='gbtree',
                     reg_alpha=0.01, reg_lambda=1, random_state=0)
gbm.fit(X_poly_train_std, y_train)
print("rmse:", rmse(y_test, gbm.predict(X_poly_test_std)))

param = {'silent':1,
         'objective':'reg:linear',
         'booster':'gbtree',
         'alpha': 0.01,
         'lambda': 1
        }

dtrain = xgb.DMatrix(X_poly_train_std, label=y_train)
dtest = xgb.DMatrix(X_poly_test_std, label=y_test)
watchlist  = [(dtrain,'eval'), (dtest, 'train')]
num_round = 100
bst = xgb.train(param, dtrain, num_round, watchlist, verbose_eval=False)
print("rmse:", rmse(y_test, bst.predict(dtest)))

plt.figure(figsize=(8, 10))
xgb.plot_importance(bst)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Let's see the distribution of the Profit using a histogram plot and see if there is any outliers in the data using bosplot.
Can you write Python code for it?

plt.figure(figsize = (8, 6))
plt.subplot(2, 1, 1)
df.Profit.plot.hist(bins = 10, normed = True)
df.Profit.plot.kde(title = "Historgram of Profit")

plt.subplot(2, 1, 2)
df.Profit.plot.box(vert = False, title = "Boxplot of Profit")

plt.tight_layout()


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Look at the cofficients values. Many of the features are not zero making the model parsimonious hence more robust - that is less prone to overfitting.
Let's plot how coefficient reached 0 values by varying the alpha valuess.
Can you write Python code for it?

coefs = []
alphas = 10 ** np.linspace(-5, 5, 20)
for alpha in alphas:
    lasso = Lasso(alpha=alpha, max_iter=10000, tol=1e-5,random_state=100)
    lasso.fit(X_poly_std, y)
    coefs.append(lasso.coef_)

plt.plot(alphas, coefs)
plt.xscale("log")
plt.xlabel("Alpha (penalty term on the coefficients)")
plt.ylabel("Coefficients of the features")


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Polynomial regression generally sufferes from overfitting. Let's regularize the model using Lasso.
Can you write Python code for it?

lasso = Lasso(alpha=0.03, max_iter=10000, normalize=False, random_state=100)
lasso.fit(X_poly_train_std, y_train)
print("Train rmse: ", rmse(y_train, lasso.predict(X_poly_train_std)))
print("Test rmse: ", rmse(y_test, lasso.predict(X_poly_test_std)))
print(lasso.intercept_, lasso.coef_)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Profit has one outlier. We can try to take log scale to remove the outlier value before doing any prediction. But for now, let ignore the outlier.
Let's plot association between each pair of columns.
Can you write Python code for it?


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
R2 can be viewed as (1 - mse/variance(y))
Significance Scores for feature selection
Can you write Python code for it?

from sklearn.feature_selection import f_regression

_, p_vals = f_regression(X_train, y_train)
p_vals

pd.DataFrame({"feature": df_dummied.columns, "p_value": p_vals})


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
Split the X and y into training and test sets.
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X_std, y,
                                                    test_size = 0.3, random_state = 100)

print("Training set: ", X_train.shape, y_train.shape)

print("Test set: ", X_test.shape, y_test.shape)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
State column has been replaced by two additional column - one for Florida and one NY. First value in the categorical values CA has been dropped to avoid collinearity issue.
Now, let's create X feature matrix and y outcome vector.
Can you write Python code for it?


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
There are 50 observations and 5 columns. 4 columns - R&D Spend, Administration and Marketing Spend, and Profile are numeric and one is categorical - State. There is 2 null values in columns R&D Spend feature and and 3 in Marketing Spend.
Replace the null values with median for respective state.
Can you write Python code for it?

df_null_idx = df[df.isnull().sum(axis = 1) > 0].index
df.iloc[df_null_idx]

median_values = df.groupby("State")[["R&D Spend", "Marketing Spend"]].median()
median_values

df["R&D Spend"] = df.apply(lambda row: median_values.loc[row["State"], "R&D Spend"]  if np.isnan(row["R&D Spend"]) else row["R&D Spend"], axis = 1 )
df["Marketing Spend"] = df.apply(lambda row: median_values.loc[row["State"], "Marketing Spend"]  if np.isnan(row["Marketing Spend"]) else row["Marketing Spend"], axis = 1 )
df.iloc[df_null_idx]

# Check if there are any more null values.
df.isnull().sum()


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
p-value indicates the significant scores for each feature. p-value < 0.05 indicates, the corresponding feature is statistically significant. We can rebuild the model excluding the non-significant features one by one until all remaining features are significant.
Power Plant Dataset
Let's look at another dataset
Can you write Python code for it?

df = pd.read_csv("/data/Combined_Cycle_Power_Plant.csv")
df.head()

X = df.iloc[:, 0:4].values
y = df.PE.values

sns.pairplot(df)

scaler = StandardScaler()
X_std = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size = 0.3, random_state = 1)

def rmse(y_true, y_pred):
    return sqrt(mean_squared_error(y_true, y_pred))

lr = LinearRegression(normalize=False)
lr.fit(X_train, y_train)
y_train_pred = lr.predict(X_train)
y_test_pred = lr.predict(X_test)
rmse(y_test, y_test_pred)

from scipy import stats

residuals = y_test - y_test_pred

plt.figure(figsize=(15, 6))
plt.subplot(1, 2, 1)
plt.scatter(y_test, residuals)
plt.xlabel("y_test")
plt.ylabel("Residuals")
plt.hlines([0], xmin = 420, xmax = 500, linestyles = "dashed")

plt.subplot(1, 2, 2)
stats.probplot(residuals, plot=plt)


I am working on the file "Scikit - 03 Linear Regression.ipynb".
The first task is:
r2 score can have a max value 1, negative values of R2 means suboptimal model
Can you write Python code for it?

r2_score(y_test, y_test_pred), r2_score(y_train, y_train_pred)


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Above we see the impact of different gamma and C values. Higher the gamma value or C values, each observation seems to loose territory of influence and hence tend to create a more overfit model. But we see the accuracy score is already matched with some of the best we have got so far. Let retrain the model using all features and tune the model using parameter grid.
Can you write Python code for it?

svc = SVC(C=10, kernel="linear", random_state=345)
svc.fit(X_train, y_train)
print("accuracy", svc.score(X_test, y_test))
plot_decision_regions(X_std, y, svc)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")
plt.title("SVM classifier using gamma = 1")

svc = SVC(C=10, kernel="poly", degree=2, random_state=345)
svc.fit(X_train, y_train)
print("accuracy", svc.score(X_test, y_test))
plot_decision_regions(X_std, y, svc)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")
plt.title("SVM classifier using gamma = 1")

SVC().get_params()

%%time
param_grid = {
    "C": 10 ** np.linspace(-2, 2, 10),
    "gamma": 10 ** np.linspace(-1, 2, 10),
    "kernel": ["linear", "rbf", "sigmoid"]
}
grid_search = GridSearchCV(cv = 5, estimator=SVC(),
                           param_grid=param_grid, scoring="accuracy", verbose=True)
grid_search.fit(X_std, y)
print("Best score(CV): ", grid_search.best_score_,
      "\n Best parameters: ", grid_search.best_params_)

np.set_printoptions(suppress=True)
10 ** np.linspace(-5, 5, 10)


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
After tuning we got a better result 0.9733.
Random Forest Classifier
Can you write Python code for it?

X = iris.iloc[:, [2,0]].values
y = iris.Species.values
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
pd.DataFrame(X_std).head()
le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size = 0.30, random_state = 100)
print(X_train.shape)

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(max_depth=4, random_state=123)
forest.fit(X_train, y_train)
print("Accuracy:", forest.score(X_test, y_test))
plot_decision_regions(X_std, y, forest)
plt.xlabel("Petal Length (standarized)")
plt.ylabel("Sepal Length (standarized)")
plt.legend(loc = "upper left")


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Decision Tree Classifier
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier

X = iris.iloc[:, [2,0]].values
y = iris.Species.values
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
pd.DataFrame(X_std).head()
le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 100)
print(X_train.shape)

tree = DecisionTreeClassifier(max_depth=4)
tree.fit(X_train, y_train)

plot_decision_regions(X, y, tree)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")

tree.score(X_test, y_test)


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Distribution of the three classes (Species) in this problem is euqally distributed. Accuracy would a good measure of the performance for model evaluation.
Can you write Python code for it?

iris.info()

species = iris.Species.unique()
colors = sns.color_palette("hls", 3)
for i, v in enumerate(species):
    df = iris[iris.Species == v]
    plt.scatter(df["PetalLength"],df["SepalLength"], color = colors[i], label = v)
plt.legend(loc = "upper left")
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")

y = np.where(iris.Species == "Iris-virginica", 1, 0)


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Model Tuning using Grid Search Techqniue
Find which parameters are available to tune.
Can you write Python code for it?

LogisticRegression().get_params()

from sklearn.model_selection import GridSearchCV

param_grid ={"C": 10 ** np.linspace(-5, 5, 100)}
gs = GridSearchCV(cv=5, estimator = lr, scoring="accuracy", param_grid= param_grid)
gs.fit(X_train, y_train)

best = gs.best_estimator_
print("Best estimator score: ", best.score(X_test, y_test))
print(best.coef_[0])

gs.best_params_


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Multi Class classification
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler, LabelEncoder

X = iris.iloc[:, 0:4].values
y = iris.Species.values
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
pd.DataFrame(X_std).head()

le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size = 0.3, random_state = 100)

lr = LogisticRegression(max_iter=100, random_state=100)

param_grid = [
    {"C": 10 ** np.linspace(-5, 5, 100)}
]
gs = GridSearchCV(cv=5, estimator = lr, scoring="accuracy", param_grid= param_grid)
gs.fit(X_train, y_train)
gs.best_params_

lr = gs.best_estimator_
lr.intercept_, lr.coef_

coeffs = pd.DataFrame(np.hstack([lr.intercept_.reshape(-1, 1), lr.coef_]))
coeffs.columns = ["intercept", *iris.columns[0:4]]
coeffs

accuracy_score(y_test, lr.predict(X_test))


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Nonlinear Decision Boundary using Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import PolynomialFeatures
from mlxtend.plotting.decision_regions import plot_decision_regions
from sklearn.pipeline import Pipeline

y = np.where(iris.Species == "Iris-versicolor", 1, 0)
X = iris.loc[:, ["PetalLength","SepalLength"]].values
print(X.shape, y.shape)
lr = LogisticRegression()
poly = PolynomialFeatures(degree=2)

pipeline = Pipeline([("poly", poly), ("lr", lr)])

pipeline.fit(X, y)

plot_decision_regions(X, y, pipeline)
plt.xlabel("PetalLength")
plt.ylabel("SepalLength")

print("Acuracy score: ", lr.score(poly.transform(X), y))


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
Regularization of model using complexity parameter
Can you write Python code for it?

params = 10 ** np.linspace(-5, 5, 100)
means, stds = [], []
coefs = []

for p in params:
    lr = LogisticRegression(C = p)
    scores = cross_val_score(cv=5, scoring="accuracy", estimator=lr, X=X_train, y= y_train)
    means.append(scores.mean())
    stds.append(scores.std())
    lr.fit(X_train, y_train)
    coefs.append(lr.coef_[0])

means = np.array(means)
stds = np.array(stds)

plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.plot(params, means)
plt.fill_between(params, means + stds, means - stds, alpha = 0.2)
plt.xscale("log")
plt.xlabel("C")
plt.ylabel("accuracy")
plt.title("Impact of complexity parameter (C)\n on accuracy score")

plt.subplot(1, 2, 2)
plt.plot(params, coefs)
plt.xlabel("C")
plt.ylabel("Coefficient")
plt.xscale("log")
plt.title("Impact of complexity parameter (C)\n on feature coefficients")

plt.tight_layout()

lr = LogisticRegression(C = 10)
lr.fit(X_train, y_train)
lr.coef_[0]

lr = LogisticRegression(C = 1e-4)
lr.fit(X_train, y_train)
lr.coef_[0]


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
SVM Classifier
Can you write Python code for it?

from sklearn.svm import SVC

X = iris.iloc[:, [2,0]].values
y = iris.Species.values
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
pd.DataFrame(X_std).head()
le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X_std, y, test_size = 0.30, random_state = 100)
print(X_train.shape)

svc = SVC(gamma=1, C =1, kernel="rbf", random_state=345)
svc.fit(X_train, y_train)
print("accuracy", svc.score(X_test, y_test))
plot_decision_regions(X_std, y, svc)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")
plt.title("SVM classifier using gamma = 1")

svc = SVC(gamma=10, C = 1, kernel="rbf", random_state=345)
svc.fit(X_train, y_train)
print("accuracy", svc.score(X_test, y_test))
plot_decision_regions(X_std, y, svc)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")
plt.title("SVM classifier using gamma = 10, C = 1")

svc = SVC(gamma=1, C = 10, kernel="rbf", random_state=345)
svc.fit(X_train, y_train)
print("accuracy", svc.score(X_test, y_test))
plot_decision_regions(X_std, y, svc)
plt.xlabel("Petal Length")
plt.ylabel("Sepal Length")
plt.legend(loc = "upper left")
plt.title("SVM classifier using gamma = 1, C = 10")


I am working on the file "Scikit - 05 Classification.ipynb".
The first task is:
We got above the accuracy of the tuned model using logistic regression 95.56. We can run tuning over tree max_depth and other parameters to tune the model.
Can you write Python code for it?

tuning_grid = {"max_depth": np.arange(1, 10)}
tree = DecisionTreeClassifier()
grid_search = GridSearchCV(cv=5, estimator=tree, param_grid=tuning_grid, scoring="accuracy")
grid_search.fit(X_std, y)
grid_search.best_score_, grid_search.best_params_


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
As we see above the accuracy is the range of 0.70. Vader model performed better for the positive sentiment compared to negative sentiment. Let's now use statistical model using TFIDF which generally perform better.
Sentiment Analysis using statistical model using TFIDF
Can you write Python code for it?

def preprocess(text):

    # Remove html tags
    text = BeautifulSoup(text.lower(), "html5lib").text

    # Replace the occurrences of multiple consecutive non-word ccharacters
    # with a single space (" ")
    text = re.sub(r"[\W]+", " ", text)
    return text

preprocess(v)

%%time
# Apply the preprocessing logic to all comments
comments["content"] = comments["content"].apply(preprocess)

comments_train = comments[comments["label"] == "train"]
comments_train.sample(10)

comments_test = comments[comments["label"] == "test"]
comments_test.sample(10)

X_train = comments_train["content"].values
y_train = np.where(comments_train.sentiment == "pos", 1, 0)

X_test = comments_test["content"].values
y_test = np.where(comments_test.sentiment == "pos", 1, 0)

# http://snowball.tartarus.org/algorithms/porter/stemmer.html
# http://www.nltk.org/howto/stem.html

from nltk.stem.snowball import SnowballStemmer
from nltk.stem.porter import PorterStemmer
print(SnowballStemmer.languages)

porter = PorterStemmer()
snowball = SnowballStemmer("english")
lemmatizer = nltk.wordnet.WordNetLemmatizer()

values = []

for s in nltk.word_tokenize("""
            revival
            allowance
            inference
            relational
            runner
            runs
            ran
            has
            having
            generously
            wasn't
            leaves
            swimming
            relative
            relating
            """):
    values.append((s, porter.stem(s)
          , snowball.stem(s), lemmatizer.lemmatize(s, "v")))

pd.DataFrame(values, columns = ["original", "porter", "snowball", "lemmatizer"])

stopwords = nltk.corpus.stopwords.words("english")
print(len(stopwords), stopwords)


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Build Pipeline for classificaiton Model
Can you write Python code for it?

%%time
tfidf =feature_extraction.text.TfidfVectorizer(
              tokenizer=my_tokenizer
            , stop_words = stopwords
            , ngram_range=(1, 2)
        )

pipe = pipeline.Pipeline([
    ("tfidf", tfidf),
    ("est", linear_model.LogisticRegression(C = 1.0, random_state = 1
                            , n_jobs = 8, solver="saga"))
])
pipe.fit(X_train, y_train)

import pickle

with open("/tmp/model.pkl", "wb") as f:
    pickle.dump(pipe, f)

!ls -lh /tmp/model.pkl

with open("/tmp/model.pkl", "rb") as f:
    model = pickle.load(f)

doc1 = """when we started watching this series on
cable i had no idea how addictive it would be
even when you hate a character you hold back because
they are so beautifully developed you can almost
understand why they react to frustration fear greed
or temptation the way they do it s almost as if the
viewer is experiencing one of christopher s learning
curves i can t understand why adriana would put up with
christopher s abuse of her verbally physically and
emotionally but i just have to read the newspaper to
see how many women can and do tolerate such behavior
carmella has a dream house endless supply of expensive
things but i m sure she would give it up for a loving
and faithful husband or maybe not that s why i watch
it doesn t matter how many times you watch an episode
you can find something you missed the first five times
we even watch episodes out of sequence watch season 1
on late night with commercials but all the language a e
with language censored reruns on the movie network whenever
they re on we re there we ve been totally spoiled now i also
love the malaprop s an albacore around my neck is my favorite of
johnny boy when these jewels have entered our family vocabulary
it is a sign that i should get a life i will when the series
ends and i have collected all the dvd s and put the collection
in my will"""
doc1 = preprocess(doc1)

model.predict_proba(np.array([doc1]))[:, 1]


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Byte size of the training doc sparse doc
Can you write Python code for it?

print(X_train_tfidf.data.nbytes / (1024.0 ** 3), "GB")


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Hashing Vectorizer
Convert a collection of text documents to a matrix of deterministic hash token (murmur3) occurrences
It turns a collection of text documents into a scipy.sparse matrix holding token occurrence counts (or binary occurrence information), possibly normalized as token frequencies if norm=’l1’ or projected on the euclidean unit sphere if norm=’l2’.
Advantages
- it is very low memory scalable to large datasets as there is no need to store a vocabulary dictionary in memory
- it is fast to pickle and un-pickle as it holds no state besides the constructor parameters
- it can be used in a streaming (partial fit) or parallel pipeline as there is no state computed during fit.
Disadvantages
- there is no way to compute the inverse transform (from feature indices to string feature names) which can be a problem when trying to introspect which features are most important to a model.
- there can be collisions: distinct tokens can be mapped to the same feature index. However in practice this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems).
- no IDF weighting as this would render the transformer stateful.
Can you write Python code for it?

hashing_vectorizer = feature_extraction.text.HashingVectorizer(n_features=2 ** 3
                                            , tokenizer=my_tokenizer, ngram_range=(1, 2))

corpus = ["Today is Wednesday"
          , "Delhi weather is hot today."
          , "Delhi roads are not busy in the morning"]

doc_term_matrix = hashing_vectorizer.fit_transform(corpus)

pd.DataFrame(doc_term_matrix.toarray()) # Each cell is normalized (l2) row-wise

%%time

n_features = int(X_train_tfidf.shape[1] * 0.8)

hashing_vectorizer = feature_extraction.text.HashingVectorizer(n_features=n_features
                                         , tokenizer=my_tokenizer, ngram_range=(1, 2))
X_train_hash = hashing_vectorizer.fit_transform(X_train)
X_test_hash = hashing_vectorizer.transform(X_test)

X_train_hash

X_train_hash.shape, X_test_hash.shape

print(X_train_hash.data.nbytes / (1024.0 ** 3), "GB")

%%time
lr = linear_model.LogisticRegression(C = 1.0, random_state = 1,
                            solver = "liblinear")
lr.fit(X_train_hash, y_train)
y_train_pred = lr.predict(X_train_hash)
y_test_pred = lr.predict(X_test_hash)
print("Training accuracy: ", metrics.accuracy_score(y_train, y_train_pred))
print("Test accuracy: ", metrics.accuracy_score(y_test, y_test_pred))

print(metrics.classification_report(y_test, y_test_pred))


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Important terms for a document
Can you write Python code for it?

vocab_by_term = tfidf.vocabulary_
vocab_by_idx = dict({(vocab_by_term[term], term)
                                 for term in vocab_by_term})

str(vocab_by_term)[:100]

str(vocab_by_idx)[:100]

idx = 5
print("Content:\n", X_train[idx])
row = X_train_tfidf[idx]
terms = [(vocab_by_idx[row.indices[i]], row.data[i])
             for i, term in enumerate(row.indices)]
pd.Series(dict(terms)).sort_values(ascending = False)

idx = 50
row = X_train_tfidf[idx]
terms = [(vocab_by_idx[row.indices[i]], row.data[i])
                 for i, term in enumerate(row.indices)]
top_terms= list(pd.Series(dict(terms))\
                .sort_values(ascending = False)[:50].index)
wc = WordCloud(background_color="white",
    width=500, height=500, max_words=50).generate("+".join(top_terms))
plt.figure(figsize=(10, 10))
plt.imshow(wc)
plt.axis("off");


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Lets drop the following words from stopwords since they are likely good indicator of sentiment.
Can you write Python code for it?

stopwords.remove("no")
stopwords.remove("nor")
stopwords.remove("not")

sentence = """Financial Services revenues increased $0.5 billion, or 5%, primarily due to
lower impairments and volume growth, partially offset by lower gains."""

stemmer = SnowballStemmer("english")
#stemmer = PorterStemmer()
def my_tokenizer(s):
    terms = nltk.word_tokenize(s.lower())
    #terms = re.split("\s", s.lower())
    #terms = [re.sub(r"[\.!]", "", v) for v in terms if len(v)>2]
    #terms = [v for v in terms if len(v)>2]
    terms = [v for v in terms if v not in stopwords]
    terms = [stemmer.stem(w) for w in terms]
    #terms = [term for term in terms if len(term) > 2]
    return terms
print(my_tokenizer(sentence))

tfidf = feature_extraction.text.TfidfVectorizer(tokenizer=my_tokenizer, max_df = 0.95, min_df=0.0001
                                                , ngram_range=(1, 2))

corpus = ["Today is Wednesday"
          , "Delhi weather is hot today."
          , "Delhi roads are not busy in the morning"]

doc_term_matrix = tfidf.fit_transform(corpus)

# returns term and index in the feature matrix
print("Vocabulary: ", tfidf.vocabulary_)

columns = [None] * len(tfidf.vocabulary_)
for term in tfidf.vocabulary_:
    columns[tfidf.vocabulary_[term]] = term
columns
scores = pd.DataFrame(doc_term_matrix.toarray()
                      , columns= columns)
scores

X_train_tfidf = tfidf.fit_transform(X_train)
X_test_tfidf = tfidf.transform(X_test)

X_test_tfidf.shape, y_test.shape, X_train_tfidf.shape, y_train.shape


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Now let's see how to create text classifier using nltk and scikit learn.
Can you write Python code for it?

# The following line does not work on Windows system
!head -n 1 /data/imdb-comments.json

data = []
with open("/data/imdb-comments.json", "r", encoding="utf8") as f:
    for l in f.readlines():
        data.append(json.loads(l))

comments = pd.DataFrame.from_dict(data)
comments.sample(10)

comments.info()

comments.label.value_counts()

comments.groupby(["label", "sentiment"]).content.count().unstack()

np.random.seed(1)
v = list(comments["content"].sample(1))[0]
v

comments.head()

comments["content"].values[0]


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Run the following lines when you run this notebook first time on your system.
Can you write Python code for it?

import nltk
nltk.download("punkt")
nltk.download("stopwords")
nltk.download("wordnet")
nltk.download('averaged_perceptron_tagger')
nltk.download("vader_lexicon")

print(nltk.__version__)


I am working on the file "Scikit - 06 Text Processing.ipynb".
The first task is:
Vader Sentiment Analysis
Can you write Python code for it?

from nltk.sentiment.vader import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()

sia.polarity_scores(comments["content"].values[0])

def sentiment_score(text):
    return sia.polarity_scores(text)["compound"]

sentiment_score(comments["content"].values[0])

%%time
comments["vader_score"] = comments["content"].apply(lambda text: sentiment_score(text))

comments["vader_sentiment"] = np.where(comments["vader_score"]>0, "pos", "neg")

comments.head()

comments.vader_sentiment.value_counts()

print(metrics.classification_report(comments["sentiment"], comments["vader_sentiment"]))


I am working on the file "Scikit - 07 Dimensionality Reduction.ipynb".
The first task is:
Kernel PCA for non linear transformation
Can you write Python code for it?

kpca = KernelPCA(n_components = 2, kernel = "sigmoid", gamma = 10)
kpca.fit(X_train_std)
X_train_kpca = kpca.transform(X_train_std)
X_test_kpca = kpca.transform(X_test_std)

X_combined_kpca = np.vstack((X_train_kpca, X_test_kpca))
y_combined = np.hstack((y_train, y_test))

lr = LogisticRegression(C = 1000, random_state = 0, penalty = "l2")
lr.fit(X_train_kpca, y_train)
test_pred= lr.predict(X_test_kpca)
score = accuracy_score(test_pred, y_test)
print("Accuracy score on kpca data: %.4f" % score)

plt.figure(figsize = (15, 10))
plot_decision_regions(X_combined_kpca, y_combined, lr)
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("Decision region for the wine classification on PCA hyperplane")


I am working on the file "Scikit - 07 Dimensionality Reduction.ipynb".
The first task is:
Let's see the class distribution
Can you write Python code for it?

wine.Class.value_counts().sort_index().plot(kind = "bar")

X, y = wine.iloc[:, 1:].values, wine.iloc[:, 0].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
print("Train X dimension: %s, Test X dimension: %s" % (X_train.shape, X_test.shape))

scaler = StandardScaler()
X_train_std = scaler.fit_transform(X_train)
X_test_std = scaler.transform(X_test)

X_train_cov = np.cov(X_train_std.T)
# cov function considers: each row of m represents a variable,
# and each column a single observation of all those variables.
# resulting output will be d x d matrix, where d is no of the rows.
print("Dim of X_train: ", X_train_std.shape)
print("Dim of X_train_cov: ", X_train_cov.shape)

eigen_vals, eigen_vectors = np.linalg.eig(X_train_cov)
print("Eigen values: %s" % (eigen_vals))
print("Eigen vectors, shape: ", eigen_vectors.shape)

explained_variance = eigen_vals / eigen_vals.sum()
plt.bar(range(len(explained_variance)), explained_variance, label = "Explained variance ratio")
plt.step(range(len(explained_variance)), np.cumsum(explained_variance)
         , label = "Commulative explained variance ratio")
plt.xlim(-1, 14)
plt.ylim(0, 1.5)
plt.xlabel("Principle Components")
plt.ylabel("Explained Variance Ratio")
plt.legend(loc = "best")

k = 2
W = eigen_vectors[:, :k]
W

X_train_pca = X_train_std.dot(W)
X_test_pca = X_test_std.dot(W)

X_combined_pca = np.vstack((X_train_pca, X_test_pca))
y_combined = np.hstack((y_train, y_test))

markers = ["o", "x", "v"]
colors = ["red", "blue", "green"]
for idx, cl in enumerate(np.unique(y_train)):
    X_train_filtered = X_train_pca[y_train == cl, :]
    plt.scatter(X_train_filtered[:, 0], X_train_filtered[:, 1], marker = markers[idx], color = colors[idx])

plt.xlabel("PC1")
plt.ylabel("PC2")

lr = LogisticRegression(C = 1000, random_state = 123, penalty = "l2")
lr.fit(X_train_pca, y_train)
score = lr.score(X_test_pca, y_test)
print("Accuracy score on pca data: %.4f" % score)

plt.figure(figsize = (15,...


I am working on the file "Scikit - 07 Dimensionality Reduction.ipynb".
The first task is:
PCA using SciKit (Linear Transformation)
Can you write Python code for it?

pca = PCA(n_components = 2)
X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)

X_combined_pca = np.vstack((X_train_pca, X_test_pca))
y_combined = np.hstack((y_train, y_test))

lr = LogisticRegression(C = 1000, random_state = 0, penalty = "l2")
lr.fit(X_train_pca, y_train)
test_pred= lr.predict(X_test_pca)
score = accuracy_score(test_pred, y_test)
print("Accuracy score on pca data: %.4f" % score)

plt.figure(figsize = (15, 10))
plot_decision_regions(X_combined_pca, y_combined, lr)
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("Decision region for the wine classification on PCA hyperplane")

X_train_pca[:10, :]


I am working on the file "Scikit - 08 Clustering.ipynb".
The first task is:
Applying clustering to grouplens movies dataset based on genre
Can you write Python code for it?

movies = pd.read_csv("/data/movielens/movies.csv", index_col="movieId")
movies.head()

movies.sample(10)

movies = movies[~movies["genres"].str.contains("\(no genres listed\)")]
movies.sample(10)

genres = set()
movies["genres"].apply(lambda g: genres.update(g.split(r"|")))
genres = list(genres)
genres.sort()
print(genres, len(genres))

def to_vector(g):
    indices = np.array([genres.index(v) for v in g.split(r"|")])
    l = np.zeros(len(genres))
    l[indices] = 1
    return l

genres_idx = movies["genres"].apply(to_vector)
genres_idx.head(10)

X = np.array(genres_idx.tolist())
print("X.shape: ", X.shape)


I am working on the file "Scikit - 08 Clustering.ipynb".
The first task is:
Find number of clusters from Dendogram
Can you write Python code for it?

plt.figure(figsize = (15, 10))
row_clusters = linkage(X, method="complete", metric="euclidean")
f = dendrogram(row_clusters)


I am working on the file "Scikit - 08 Clustering.ipynb".
The first task is:
Normalize the data
Can you write Python code for it?

scaler = StandardScaler()
X_std = scaler.fit_transform(X)

plt.figure(figsize = (15, 10))
row_clusters = linkage(X_std, method="complete", metric="euclidean")
f = dendrogram(row_clusters, p = 5, truncate_mode="level")


I am working on the file "Scikit - 08 Clustering.ipynb".
The first task is:
There is not visual indication of clusters from 2 PCA components which is consistent with the finding that explained variance with 2 components is only 2%
Can you write Python code for it?

find_elbow(X_std, 40)

knn = KMeans(n_clusters=8, max_iter=300, random_state=0)
y_pred = knn.fit_predict(X_std)


I am working on the file "Scikit - 08 Clustering.ipynb".
The first task is:
To visualize the clusters lets apply PCA with 2 components.
Can you write Python code for it?

from sklearn.decomposition import KernelPCA, PCA

pca = PCA(random_state=0)
X_pca = pca.fit_transform(X_std)

ratios = pca.explained_variance_ratio_
plt.bar(range(len(ratios)), ratios)
plt.step(range(len(ratios)), np.cumsum(ratios),
         label = "Cumsum of Explained variance ratio")
plt.title("Explained variance")
plt.ylabel("Explained Variance Ratio")
plt.xlabel("Number of PCA components")


I am working on the file "Scikit - 10 Image Classification (MNIST dataset).ipynb".
The first task is:
Applying SGD classifier
Can you write Python code for it?

inits = np.random.randn(10, 784)
inits = inits / np.std(inits, axis=1).reshape(10, -1)

%%time
est = linear_model.SGDClassifier(n_jobs=4, tol=1e-5, eta0 = 0.15,
                                 learning_rate = "invscaling",
                                 alpha = 0.01, max_iter= 100)
est.fit(X_train_std, y_train, inits)
print("accuracy", est.score(X_test_std, y_test), "iterations:", est.n_iter_)

fig, _ = plt.subplots(3, 4, figsize = (15, 10))
for i, ax in enumerate(fig.axes):
    if i < est.coef_.shape[0]:
        ax.imshow(est.coef_[i, :].reshape(28, 28), cmap = "bwr", interpolation="nearest")
    else:
        ax.remove()

pd.DataFrame(est.coef_[0, :].reshape(28, 28))


I am working on the file "Scikit - 10 Image Classification (MNIST dataset).ipynb".
The first task is:
Applying logistic regression classifier
Can you write Python code for it?

%%time
lr = linear_model.LogisticRegression()
lr.fit(X_train_std, y_train)
print("accuracy:", lr.score(X_test_std, y_test))


I am working on the file "Scikit - 10 Image Classification (MNIST dataset).ipynb".
The first task is:
Chisquare test on class frequencies
Can you write Python code for it?


I am working on the file "Scikit - 10 Image Classification (MNIST dataset).ipynb".
The first task is:
Dataset description
Datasource: http://yann.lecun.com/exdb/mnist/
The training dataset consists of 60,000 training digits and the test set contains 10,000 samples, respectively. The images in the MNIST dataset consist of  pixels, and each pixel is represented by a gray scale intensity value. Here, we unroll the  pixels into 1D row vectors, which represent the rows in our image array (784 per row or image). The second array (labels) returned by the load_mnist function contains the corresponding target variable, the class labels (integers 0-9) of the handwritten digits.
Csv version of the files are available in the following links.
CSV training set http://www.pjreddie.com/media/files/mnist_train.csv
CSV test set http://www.pjreddie.com/media/files/mnist_test.csv
Can you write Python code for it?

training = pd.read_csv("/data/MNIST/mnist_train.csv", header = None)
testing = pd.read_csv("/data/MNIST/mnist_test.csv", header = None)

X_train, y_train = training.iloc[:, 1:].values, training.iloc[:, 0].values
X_test, y_test = testing.iloc[:, 1:].values, testing.iloc[:, 0].values

print("Shape of X_train: ", X_train.shape, "shape of y_train: ", y_train.shape)
print("Shape of X_test: ", X_test.shape, "shape of y_test: ", y_test.shape)


I am working on the file "Scikit - 10 Image Classification (MNIST dataset).ipynb".
The first task is:
Distribution of class frequencies
Can you write Python code for it?

label_counts = pd.DataFrame({
    "train": pd.Series(y_train).value_counts().sort_index(),
    "test": pd.Series(y_test).value_counts().sort_index()
})
(label_counts / label_counts.sum()).plot.bar()

plt.xlabel("Class")
plt.ylabel("Frequency (normed)")


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Effect of learning rate
Can you write Python code for it?

params = 10 ** - np.linspace(0.01, 2, 10)
scores_train = []
scores_test = []

timer.reset()
for p in params:
    nn = NeuralNetwork([784,100,10], p, random_state = 0)
    for i in range(X_train.shape[0]):
        nn.fit(X_train[i], y_train_ohe[i])
    scores_train.append(nn.score(X_train, y_train))
    scores_test.append(nn.score(X_test, y_test))
    timer()

plt.plot(params, scores_test, label = "Test score")
plt.plot(params, scores_train, label = "Training score")
plt.xlabel("Learning Rate")
plt.ylabel("Accuracy")
plt.legend()
plt.title("Effect of learning rate")

print("Accuracy scores")
pd.DataFrame({"learning_rate": params, "train": scores_train, "test": scores_test})


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Effect of size (num of nodes) of the single hidden layer
Can you write Python code for it?

num_layers = 50 * (np.arange(10) + 1)
learning_rate = 0.077
scores_train, scores_test = [], []

timer.reset()
for p in num_layers:
    nn = NeuralNetwork([784, p,10], learning_rate, random_state = 0)
    indices = np.arange(X_train.shape[0])
    for i in indices:
        nn.fit(X_train[i], y_train_ohe[i])
    scores_train.append(nn.score(X_train, y_train))
    scores_test.append(nn.score(X_test, y_test))
    timer("size: %d, test score: %f, training score: %f" % (p, scores_test[-1], scores_train[-1]))

plt.plot(num_layers, scores_test, label = "Test score")
plt.plot(num_layers, scores_train, label = "Training score")
plt.xlabel("Hidden Layer Size")
plt.ylabel("Accuracy")
plt.legend(loc = "lower right")
plt.title("Effect of size (num of nodes) of the hidden layer")

print("Accuracy scores")
pd.DataFrame({"layer": num_layers, "train": scores_train, "test": scores_test})


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Effect of using multiple hidden layers
Can you write Python code for it?

num_layers = np.arange(5) + 1
learning_rate = 0.077
scores_train, scores_test = [], []

timer.reset()
for p in num_layers:
    layers = [100] * p
    layers.insert(0, 784)
    layers.append(10)

    nn = NeuralNetwork(layers, learning_rate, random_state = 0)
    indices = np.arange(X_train.shape[0])
    for i in indices:
        nn.fit(X_train[i], y_train_ohe[i])
    scores_train.append(nn.score(X_train, y_train))
    scores_test.append(nn.score(X_test, y_test))
    timer("size: %d, test score: %f, training score: %f" % (p, scores_test[-1], scores_train[-1]))

plt.plot(num_layers, scores_test, label = "Test score")
plt.plot(num_layers, scores_train, label = "Training score")
plt.xlabel("No of hidden layers")
plt.ylabel("Accuracy")
plt.legend(loc = "upper right")
plt.title("Effect of using multiple hidden layers, \nNodes per layer=100")

print("Accuracy scores")
pd.DataFrame({"layer": num_layers, "train": scores_train, "test": scores_test})


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Load full MNIST dataset.
Training set size 60,000 and test set size 10,000
Original: http://yann.lecun.com/exdb/mnist/
CSV version:
training: https://pjreddie.com/media/files/mnist_train.csv
testing: https://pjreddie.com/media/files/mnist_test.csv
Can you write Python code for it?

train = pd.read_csv("../data/MNIST/mnist_train.csv", header=None, dtype="float64")
X_train = normalize_fetures(train.iloc[:, 1:].values)
y_train = train.iloc[:, [0]].values.astype("int32")
y_train_ohe = normalize_labels(y_train)
print(y_train.shape, y_train_ohe.shape)

test = pd.read_csv("../data/MNIST/mnist_test.csv", header=None, dtype="float64")
X_test = normalize_fetures(test.iloc[:, 1:].values)
y_test = test.iloc[:, 0].values.astype("int32")


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Neural Networks Classifier
Author: Abul Basar
Can you write Python code for it?

class NeuralNetwork:

    def __init__(self, layers, learning_rate, random_state = None):
        self.layers_ = layers
        self.num_features = layers[0]
        self.num_classes = layers[-1]
        self.hidden = layers[1:-1]
        self.learning_rate = learning_rate

        if not random_state:
            np.random.seed(random_state)

        self.W_sets = []
        for i in range(len(self.layers_) - 1):
            n_prev = layers[i]
            n_next = layers[i + 1]
            m = np.random.normal(0.0, pow(n_next, -0.5), (n_next, n_prev))
            self.W_sets.append(m)

    def activation_function(self, z):
        return 1 / (1 + np.exp(-z))

    def fit(self, training, targets):
        inputs0 = inputs = np.array(training, ndmin=2).T
        assert inputs.shape[0] == self.num_features, \
                "no of features {0}, it must be {1}".format(inputs.shape[0], self.num_features)

        targets = np.array(targets, ndmin=2).T

        assert targets.shape[0] == self.num_classes, \
                "no of classes {0}, it must be {1}".format(targets.shape[0], self.num_classes)

        outputs  = []
        for i in range(len(self.layers_) - 1):
            W = self.W_sets[i]
            inputs = self.activation_function(W.dot(inputs))
            outputs.append(inputs)

        errors = [None] * (len(self.layers_) - 1)
        errors[-1] = targets - outputs[-1]
        #print("Last layer", targets.shape, outputs[-1].shape, errors[-1].shape)
        #print("Last layer", targets, outputs[-1])

        #Back propagation
        for i in range(len(self.layers_) - 1)[::-1]:
            W = self.W_sets[i]
            E = errors[i]
            O = outputs[i]
            I = outputs[i - 1] if i > 0 else inputs0
            #print("i: ", i, ", E: ", E.shape, ", O:", O.shape, ", I: ", I.shape, ",W: ", W.shape)
            W += self.learning_rate * (E * O * (1 - O)).dot(I.T)
            if i > 0:
                errors[i-1] = W.T.dot(E)
     ...


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Rotation
Can you write Python code for it?

img = scipy.ndimage.interpolation.rotate(X_train[110].reshape(28, 28), -10, reshape=False)
print(img.shape)
plt.imshow(img, interpolation=None, cmap="Greys")

epochs = np.arange(10)
learning_rate = 0.077
scores_train, scores_test = [], []
nn = NeuralNetwork([784,250,10], learning_rate, random_state = 0)
indices = np.arange(X_train.shape[0])

timer.reset()
for _ in epochs:
    np.random.shuffle(indices)
    for i in indices:
        for rotation in [-10, 0, 10]:
            img = scipy.ndimage.interpolation.rotate(X_train[i].reshape(28, 28), rotation, cval=0.01, order=1, reshape=False)
            nn.fit(img.flatten(), y_train_ohe[i])
    scores_train.append(nn.score(X_train, y_train))
    scores_test.append(nn.score(X_test, y_test))
    timer("test score: %f, training score: %f" % (scores_test[-1], scores_train[-1]))

plt.plot(epochs, scores_test, label = "Test score")
plt.plot(epochs, scores_train, label = "Training score")
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend(loc = "lower right")
plt.title("Trained with rotation (+/- 10)\n Hidden Nodes: 250, LR: 0.077")

print("Accuracy scores")
pd.DataFrame({"epochs": epochs, "train": scores_train, "test": scores_test})


I am working on the file "Scikit - 12 Neural Network using Numpy.ipynb".
The first task is:
Runt the Neural Network classifier and measure performance
Can you write Python code for it?

timer.reset()
nn = NeuralNetwork([784,100,10], 0.3, random_state=0)
for i in range(X_train.shape[0]):
    nn.fit(X_train[i], y_train_ohe[i])
timer("training time")
accuracy = nn.score(X_test, y_test)
print("Testing accuracy: ", nn.score(X_test, y_test), ", Training accuracy: ", nn.score(X_train, y_train))


I am working on the file "Scikit - 16 Time Series - PerCapitaGDP.ipynb".
The first task is:
Because the p-value is larger than 0.05, the moving average is not constant over time and the null hypothesis of the Dickey-Fuller test cannot be rejected. This shows that the weekly time series is not stationary. Before you can apply ARIMA models for forecasting, you need to transform this time series into a stationary time series.
Can you write Python code for it?


I am working on the file "Scikit - 16 Time Series - PerCapitaGDP.ipynb".
The first task is:
Find optimal parameters and build an ARIMA (auto regressive integrated moving average) model
To apply an ARIMA model to your time series, you need to find optimal values for the following three model parameters (p,d,q):
 - The number of autoregressive (AR) terms (p): AR terms are just lags of the dependent variable, the euro rate, in this case. So, if p=2, it means that predictors of x(t) will be x(t-1) and x(t-2).
 - The number of moving average (MA) terms (q): MA terms are lagged forecast errors in the prediction equation. For instance, if q=2, the predictors for x(t) will be e(t-1) and e(t-2) where e(i) is the difference between the moving average at i-th instant and the actual value.
 - The number of differences (d): These are the number of non-seasonal differences. In your case, d=1, as you are modeling using the first order differenced time series.
There are two ways to determine the number of AR and MA terms. The first is to use the arma_order_select_ic function in Python. The second uses plots of the autocorrelation function (ACF) and partial autocorrelation function (PACF).
This article describes in detail the purpose of the ACF and PACF plots.
Can you write Python code for it?

#ACF and PACF plots

plt.figure(figsize=(15, 6))
def plot_acf_pacf(ts):
    lag_acf = acf(ts, nlags=10)
    lag_pacf = pacf(ts, nlags=10, method='ols')

    #Plot ACF:
    plt.subplot(121)
    plt.plot(lag_acf)
    plt.axhline(y=0,linestyle='--',color='gray')
    plt.axhline(y=-7.96/np.sqrt(len(ts)),linestyle='--',color='gray')
    plt.axhline(y=7.96/np.sqrt(len(ts)),linestyle='--',color='gray')
    plt.title('Autocorrelation Function')

    #Plot PACF:
    plt.subplot(122)
    plt.plot(lag_pacf)
    plt.axhline(y=0,linestyle='--',color='gray')
    plt.axhline(y=-7.96/np.sqrt(len(ts)),linestyle='--',color='gray')
    plt.axhline(y=7.96/np.sqrt(len(ts)),linestyle='--',color='gray')
    plt.title('Partial Autocorrelation Function')
    plt.tight_layout()

usa_diff = usa - usa.shift(periods=1)
usa_diff.dropna(inplace=True)
plot_acf_pacf(usa_diff)


I am working on the file "Scikit - 16 Time Series - PerCapitaGDP.ipynb".
The first task is:
Making Predictions
Can you write Python code for it?

from sklearn.metrics import mean_squared_error

size = int(len(usa) - 5)
train, test = usa[0:size], usa[size:len(usa)]
history = [x for x in train]
predictions = list()

print('Printing Predicted vs Expected Values...')
print('\n')
for t in range(len(test)):
    model = ARIMA(history, order=(2,1,0))
    model_fit = model.fit(disp=0)
    output = model_fit.forecast()
    yhat = output[0]
    predictions.append(float(yhat))
    obs = test[t]
    history.append(obs)
    print('predicted=%f, expected=%f' % (yhat, obs))

error = mean_squared_error(test, predictions)

print('\n')
print('Printing Mean Squared Error of Predictions...')
print('Test MSE: %.6f' % error)

predictions_series = pd.Series(predictions, index = test.index)

#rolling one-step out-of-sample
fig, ax = plt.subplots()
ax.set(title='GDP Per Capita forecasting', xlabel='Date', ylabel='Per Capita GDP')
ax.plot(usa, marker = '.', color = "red", label='observed')
ax.plot(predictions_series, color = "green", label='forecast')
legend = ax.legend(loc='upper left')
legend.get_frame().set_facecolor('w')


I am working on the file "Scikit - 16 Time Series - PerCapitaGDP.ipynb".
The first task is:
Now that you have stationarized your time series, you could go on and model residuals (fit lines between values in the plot). However, as the patterns for the trend and seasonality information extracted from the series that are plotted after decomposition are still not consistent and cannot be scaled back to the original values, you cannot use this approach to create reliable forecasts.
In case your time series does show a strong and consistent seasonal trend, here is a good article that describes the use of the SARIMAX model in Python.
Remove trend and seasonality with differencing
Can you write Python code for it?

usa_diff = usa - usa.shift(periods=1)
usa_diff.dropna(inplace=True)
plt.plot(usa_diff)
test_stationary(usa_diff)


I am working on the file "Scikit - 16 Time Series - PerCapitaGDP.ipynb".
The first task is:
Scale predictions
Now that the model is returning the results you want to see, you can scale the model predictions back to the original scale. For this, you will remove the first order differencing and take exponent to restore the predictions back to their original scale.
The lower the root mean square error (RMSE) and the closer it is to 0, the better are the model predictions in being closer to actual values.
Can you write Python code for it?

predictions_ARIMA_diff = pd.Series(results_ARIMA.fittedvalues, copy=True)
predictions_ARIMA_diff.head()

predictions_ARIMA_diff_cumsum = predictions_ARIMA_diff.cumsum()
predictions_ARIMA = pd.Series(usa.ix[0], index=usa.index)
predictions_ARIMA = predictions_ARIMA.add(predictions_ARIMA_diff_cumsum, fill_value=0)

plt.plot(usa, label = "Actual")
plt.plot(predictions_ARIMA, label = "ARIMA fitted")
plt.legend()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
2216 of the 2919 houses have same year for for GarageYrBlt and YearBuilt. Lets replace any of the NA’s for GarageYrBlt with the year from YearBuilt.
Can you write Python code for it?

df.query("GarageYrBlt != GarageYrBlt").shape

idx = df.query("GarageYrBlt != GarageYrBlt").index
df.loc[idx, "GarageYrBlt"] = df.loc[idx, "YearBuilt"]
(df.GarageYrBlt == df.YearBuilt).value_counts()

pd.isnull(df.GarageYrBlt).value_counts()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
All but one house has missing values for both columns. For houses with NA’s on both columns we can fill 0 for the area and None for the type since they likely do not have a masonry veneer. For the house with a MasVnrArea of 198 but NA for MasVnrType we can record the median areas for each type and see which type is closest to 198.
Can you write Python code for it?

df.groupby("MasVnrType").MasVnrArea.agg([len, np.median])

df.loc[2610, "MasVnrType"] = "Stone"
replace_with(df, "MasVnrType", "None")
replace_with(df, "MasVnrArea", 0)

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
All the houses that have missing values did not record having any fireplaces. We can replace the NA’s with ‘None’ since these houses don’t have any fireplaces at all.
Can you write Python code for it?

replace_with(df, "FireplaceQu", "None")

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Alley: Type of alley access
There are 2721 missing values for Alley and only 2 potential options - Grvl and Pave. We can fill ‘None’ for any of the houses with NA’s as these houses must not have any type of alley access.
Can you write Python code for it?

replace_with(df, "Alley", "None")
find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Almost all of the missing values for each categoric basement feature comes from houses with 0 on each features corresponding to area. We can fill in these values with ‘None’ since these houses certainly don’t have basements. Rows 949, 1488 and 2349 are the only missing values from BsmtExposure, we can fill this with No as that is the most frequent value and these houses most likely don’t have any exposure for their basements. The rest of the basement columns corresponding to area will be filled with 0 since they likely don’t have a basement and the categoric missing values will be filled with NoBsmt.
Can you write Python code for it?

df.loc[[949, 1488, 2349], 'BsmtExposure'] = 'No'

df[basement_columns].info()

for c in ["BsmtQual", "BsmtCond", "BsmtExposure", "BsmtFinType1", "BsmtFinType2"]:
    replace_with(df, c, "None")

for c in ["BsmtFinSF1", "BsmtFinSF2", "BsmtUnfSF",
          "TotalBsmtSF", "BsmtFullBath", "BsmtHalfBath"]:
    replace_with(df, c, 0.0)

df[basement_columns].info()

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
At the lower values there is no obvious anamoly.
That leaves 6 garage features in our dataset and 4 of them have at least 157 missing values while GarageArea and GarageCars both only have 1, thus we can assume this particular house does not have a garage at all. For the rest of the houses we can check to see that if the NA’s recorded also have 0 GarageArea and 0 GarageCars. If they do we can fill in their missing values with ‘None’ since having 0 area and 0 cars in their garage will imply that they do not have any at all.
Can you write Python code for it?


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Basement
Can you write Python code for it?

basement_columns = find_columns(df, "bsmt")
df[basement_columns].info()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Close 6% of data points across entire dataset is NA. Find out which columns have null in them
Can you write Python code for it?

def find_null_columns():
    null_columns = df.isnull().sum().sort_values(ascending = False)
    null_columns = null_columns[null_columns > 0]
    print("Total number of records: ", df.shape[0])
    print("No of columns with null values: ", len(null_columns))
    print("\nMissing values per column:")
    print(null_columns)

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Congratulations! There is no more missing values
Let's add back the SalePrice column to the dataset. Note for the testing data, there is not SalePrice. So for those, the field will be null.
Can you write Python code for it?

df.head()

df["SalesPrice"] = np.nan
df.loc[0:training.shape[0], "SalesPrice"] = SalesPrice
df.head()

df.to_csv("data/kaggle-house-prices/data_combined_cleaned.csv", index = False)


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Fence: Fence quality
We can replace any missing vlues for Fence and MiscFeature with ‘None’ as they probably don’t have this feature with their property.
Can you write Python code for it?

df.Fence.value_counts()

replace_with(df, "Fence", "None")

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Find most common values for each column
Can you write Python code for it?

def mostFrequent(col):
    counts = col.value_counts()
    counts = counts.sort_values(ascending = False)
    return  counts.index[0]

df2.apply(mostFrequent, axis = 0)

df.loc[2126, "GarageQual"] = "TA"
df.loc[2126, "GarageFinish"] = "Unf"
df.loc[2126, "GarageCond"] = "TA"


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Fireplace

Fireplaces: Number of fireplaces
FireplaceQu: Fireplace quality. FireplaceQu denotes the fireplace quality.

We can check to see if any of the missing values for FireplaceQu come from houses that recorded having at least 1 fireplace.
Can you write Python code for it?

fire_columns = find_columns(df, "fire")
fire_columns

df.groupby(["FireplaceQu", "Fireplaces"]).Id.count().unstack()

df[df.FireplaceQu.isnull()][fire_columns]


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
From the documetation below are the description of these fields.
- BsmtQual: Height of the basement
- BsmtCond: General condition of the basement
- BsmtExposure: Walkout or garden level basement walls
- BsmtFinType1: Quality of basement finished area
- BsmtFinSF1: Type 1 finished square feet
- BsmtFinType2: Quality of second finished area (if present)
- BsmtFinSF2: Type 2 finished square feet
- BsmtUnfSF: Unfinished square feet of basement area
- TotalBsmtSF: Total square feet of basement area
- BsmtFullBath: Basement full bathrooms
- BsmtHalfBath: Basement half bathrooms
There are 11 basement features each with at least 1 missing value. We can take a look at the subset of just these columns from our data.
Can you write Python code for it?


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Functional
There is no field that could help us replace the NA value for Functional. So let replace NA with most freq value.
Can you write Python code for it?

functional_columns = find_columns(df, "func")
functional_columns

df.Functional.value_counts()

replace_with(df, "Functional", "Typ")

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Garage Fields
Find garage fields
Can you write Python code for it?

garage_cols = find_columns(df, "garage")
df[garage_cols].info()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Let's see values at lower end to find any anamolies
Can you write Python code for it?

print(df.GarageYrBlt.sort_values(ascending = True)[:10])
print(df.YearBuilt.sort_values(ascending = True)[:10])


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Lets see what the deal is with GarageYrBlt. It seems reasonable that most houses would build a garage when the house itself was built. We can check this by seeing how many houses were built the same year their garage was built.
Can you write Python code for it?


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Loaded both datasets and combined them into a single dataset for data preprocessing. When we are ready for modeling prepearion, we will split them into 2 datasets.
Amount of NA values
Can you write Python code for it?

df.isnull().sum().sum() / (df.shape[0] * df.shape[1])


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
LotFrontage: Linear feet of street connected to property
There are 486 missing values for LotFrontage, which is quite a lot of values to fill and we can’t just replace these with 0. We’re given that “LotFrontage: Linear feet of street connected to property.” The area of each street connected to the house property is most likely going to have a similar area to other houses in its neighborhood. We can group by each neighborhood and take the median of each LotFrontage and fill the missing values of each LotFrontage based on what neighborhood the house comes from.
Can you write Python code for it?

lot_columns = find_columns(df, "lot")
lot_columns.append("Neighborhood")
lot_columns

neighborhood = df.groupby("Neighborhood").LotFrontage.agg([len, np.median])
neighborhood

df2 = df[lot_columns].merge(neighborhood,
                      left_on="Neighborhood", right_index=True)
df2

idx = df[df.LotFrontage.isnull()].index
df.loc[idx, "LotFrontage"] = df2.loc[idx, "median"]
df[lot_columns].merge(neighborhood,
                      left_on="Neighborhood", right_index=True)

find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
MiscFeature
Can you write Python code for it?

replace_with(df, "MiscFeature", "None")
find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Out of these, how many have >0 PoolArea?
Can you write Python code for it?

df.query("PoolArea > 0 and PoolQC != PoolQC")[['PoolArea', 'PoolQC']]


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Plot histogram of GarageYrBlt
Can you write Python code for it?

df.GarageYrBlt.plot.hist(bins = 100)
plt.title("Histogram by GarageYrBlt")
plt.xlabel("GarageYrBlt")


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Plot histogram of YearBuilt
Can you write Python code for it?

df.YearBuilt.plot.hist(bins = 100)
plt.title("Histogram by YearBuilt")
plt.xlabel("YearBuilt")


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Replace GarageYrBlt with YearBuild for record 2593
Can you write Python code for it?


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Sale Type
SaleType, Functional and Utilities have less than 3 missing values. For SaleType we can see what the SaleCondition of the house was and use a contingency table to see which SaleType and SaleCondition overlap together the most.
Can you write Python code for it?

sale_columns = find_columns(df, "sale")
sale_columns

df[df.SaleType.isnull()][sale_columns]

df.groupby(["SaleType", "SaleCondition"]).Id.count().unstack()

df.loc[2489, "SaleType"] = "WD"
find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
So, there is no more NA in the GarageYrBlt field.
Let's look at top 10 values of GarageYrBlt and top 10 YearBuilt values to to find any anamolies
Can you write Python code for it?

print(df.GarageYrBlt.sort_values(ascending = False)[:10])
print(df.YearBuilt.sort_values(ascending = False)[:10])

df[df.GarageYrBlt == 2207][["GarageYrBlt", "YearBuilt"]]


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
So, we can input Ex, Ex, Fa respectively so that PoolArea values are closer the mean values for the respective PoolQC. Replace the rest of NA values under PoolQC with "None".
Can you write Python code for it?

df.loc[2420,'PoolQC'] = "Ex"
df.loc[2503,'PoolQC'] = "Ex"
df.loc[2599,'PoolQC'] = "Fa"
idx = df.query("PoolArea == 0 and PoolQC != PoolQC").index
df.loc[idx, 'PoolQC'] = "None"
df.query("PoolArea >= 0 and PoolQC != PoolQC").shape


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
There are 34 columns that contain null values. Let's analyze each of these columns one by one in the order of null occurrences.
PoolQC
How many column names contain "pool" - likely that these are pool data related.
Can you write Python code for it?

def find_columns(df, search):
    matched = [c for c in df.columns if search.lower() in c.lower()]
    return matched

find_columns(df, "pool")


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Utilities
Utilities only has 1 value for NoSeWa and the rest AllPub. We can drop this feature from our dataset as the house with ‘NoSeWa’ is from our training set and will have won’t help with any predictive modelling
Can you write Python code for it?

df.Utilities.value_counts()

del df["Utilities"]
find_null_columns()


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
Visualization
Plot the following for train.csv
- Saleprice  - histogram or boxplot
- YearBuilt - histogram (better) or frequency plot
- Saleprice vs YearBuilt - horizontal bar chart
- Saleprice  vs GarageArea  - scatter chart
- Median Saleprice by SaleCondition and BldgType - heatmap
Can you write Python code for it?

xlims = training.SalePrice.min(), training.SalePrice.max()

plt.subplot(2, 1, 1)
training.SalePrice.plot.hist(bins = 50, xlim = xlims, sharex = True, title = "Histogram of SalePrice")
plt.subplot(2, 1, 2)
training.SalePrice.plot.box(vert = False, xlim = xlims, sharex = True, title = "Boxplot of SalePrice")
plt.tight_layout()
plt.xlabel("SalePrice")

training.YearBuilt.plot.hist(bins = 50, title = "Frequency by YearBuilt")

grouped = training.groupby("YearBuilt").SalePrice.median().sort_index()
plt.figure(figsize=(15, 5))
plt.bar(grouped.index, grouped)
plt.xlabel("YearBuilt")
plt.ylabel("Median SalePrice")
plt.title("Median SalePrice by YearBuilt")

training.plot.scatter("GarageArea", "SalePrice")
plt.title("SalePrice by GarageArea")

import seaborn as sns

grouped = training.groupby(["SaleCondition","BldgType"]).SalePrice.median().unstack()
blues = sns.color_palette("Blues", n_colors=10)
sns.heatmap(grouped, cmap = blues)
plt.title("Median Saleprice \n by SaleCondition and BldgType")


I am working on the file "Scikit - 20 Kaggle House Data Preprocessing.ipynb".
The first task is:
We can impute the values of PoolQC, by looking at PoolArea value. Find out how many diff values are there for PoolQC and their freq and mean PoolArea
Can you write Python code for it?


I am working on the file "Scikit - 21 Kaggle House price prediction (regression).ipynb".
The first task is:
Seems that Linear regression model performed very poorly. Most likely it is because model finds a lot of collinearity in the data due to the categorical columns.
Test lasso, which is more robust against multi collinearity.
Can you write Python code for it?

lasso = Lasso(random_state=1, max_iter=10000)
lasso.fit(X_train_std, y_train)
rmse(y_test, lasso.predict(X_test_std))


I am working on the file "Scikit - 21 Kaggle House price prediction (regression).ipynb".
The first task is:
This rmse score seems reasonable. Find cross validation scores.
Can you write Python code for it?

scores = cross_val_score(cv=10, estimator = lasso, scoring="neg_mean_squared_error", X=X_train_std, y = y_train)
scores = np.sqrt(-scores)
scores

from sklearn import linear_model
from sklearn import metrics
from sklearn import tree
from sklearn import ensemble
from sklearn import neighbors
import xgboost as xgb

rs = 1
estimatores = {
    #'Linear': linear_model.LinearRegression(),
    'Ridge': linear_model.Ridge(random_state=rs, max_iter=10000),
    'Lasso': linear_model.Lasso(random_state=rs, max_iter=10000),
    'ElasticNet': linear_model.ElasticNet(random_state=rs, max_iter=10000),
    'BayesRidge': linear_model.BayesianRidge(),
    'OMP': linear_model.OrthogonalMatchingPursuit(),
    'DecisionTree': tree.DecisionTreeRegressor(max_depth=10, random_state=rs),
    'RandomForest': ensemble.RandomForestRegressor(random_state=rs),
    'KNN': neighbors.KNeighborsRegressor(n_neighbors=5),
    'GradientBoostingRegressor': ensemble.GradientBoostingRegressor(n_estimators=300, max_depth=4, learning_rate=0.01, loss="ls", random_state=rs),
    'xgboost': xgb.XGBRegressor(max_depth=10)
}

errvals = {}

for k in estimatores:
    e = estimatores[k]
    e.fit(X_train_std, y_train)
    err = np.sqrt(metrics.mean_squared_error(y_test, e.predict(X_test_std)))
    errvals[k] = err

result = pd.Series.from_array(errvals).sort_values()
result.plot.barh(width = 0.8)
for y, error in enumerate(result):
    plt.text(x = 0.01, y = y - 0.1,  s = "%.3f" % error, fontweight='bold', color = "white")
plt.title("Performance comparison of algorithms")


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
Apply dendo-gram (hierarchical clustering)
Can you write Python code for it?

from scipy.cluster.hierarchy import linkage, dendrogram

plt.figure(figsize = (15, 10))
row_clusters = linkage(X_std, method="complete", metric="euclidean")
f = dendrogram(row_clusters, p = 5, truncate_mode="level")


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
As music and god are common in top 10 for each cluster, we can drop these features and retry the clustering.
Find the density of each cluster. Calculate the avg distance of a point and its closes centroid.
Can you write Python code for it?

df["cluster"] = y_pred

distances = np.zeros(len(y_pred))
for i in range(k):
    center = kmeans.cluster_centers_[i]
    distances[y_pred == i] = metrics.euclidean_distances(X_std[y_pred == i]
                                                        , center.reshape(1, -1)).squeeze()
df["distance"] = distances
df.sample(10)

df.pivot_table("distance", "cluster", "gender", aggfunc="mean")


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
Centroids of the clusters in the normal scale (using scaler.inverse_transform).
Can you write Python code for it?

centroids = pd.DataFrame(scaler.inverse_transform(kmeans.cluster_centers_), columns=features)
centroids.T


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
How many records are there for which count is greater than 20.
Can you write Python code for it?


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
Let's find the anomalies in the features depending the distance of the profile from its centroid. Using Box Whisker method, identify the outliers.
Can you write Python code for it?

def find_outliers(a):
    q1, q2, q3 = np.percentile(a, [25, 50, 75])
    iqr = q3 - q1
    lower_whisker = max(q1 - 1.5 * iqr, np.min(a))
    upper_whisker = min(q3 + 1.5 * iqr, np.max(a))
    q1, q2, q3, iqr, lower_whisker, upper_whisker
    is_outlier = (a < lower_whisker) | (a > upper_whisker)
    return is_outlier

anamolies = df[find_outliers(df.distance)]
anamolies.shape, df.shape

df.distance.plot.hist(bins = 50, log = True)


I am working on the file "Scikit - 26 Clustering SNS Data.ipynb".
The first task is:
Predict cluster for each point based on the KMeans model.
Can you write Python code for it?


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Digitize the columns for R-F-M into 5 equal buckets. To achieve this, find percentile values as bucket boundaries. These will create 5 buckets of equal sizes.
Can you write Python code for it?

quantiles = np.arange(1, 6) * 20
quantiles

rfm["r_score"] = np.digitize(rfm.recency, bins = np.percentile(rfm.recency, quantiles)
                           , right=True)

rfm["m_score"] = np.digitize(rfm.monetary, bins = np.percentile(rfm.monetary, quantiles)
                           , right=True)

rfm["f_score"] = np.digitize(rfm.frequency, bins = np.percentile(rfm.frequency, quantiles)
                           , right=True)

rfm["r_score"] = 4 - rfm["r_score"]

rfm["r_score"] = rfm["r_score"] + 1
rfm["f_score"] = rfm["f_score"] + 1
rfm["m_score"] = rfm["m_score"] + 1

rfm.head()

rfm.sample(10, random_state=123)

scaler = preprocessing.StandardScaler()
X = rfm[["r_score", "f_score", "m_score"]].values
X = scaler.fit_transform(X.astype("float32"))
X


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Drop all records having nan CustomerId
Can you write Python code for it?

df = df[~df.CustomerID.isna()]
df.info()


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Find number of unique values for each column
Can you write Python code for it?


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Find out if there are any nan in the columns
Can you write Python code for it?


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Find what could be an optimal number of clusters using elbow plot. As we see in the plot below, we can use 5 or 6 number of clusters (K) for KMeans algorithm.
Can you write Python code for it?

inertias = {}
for k in range(2, 10):
    kmeans = cluster.KMeans(n_clusters=k, random_state=1)
    kmeans.fit(X)
    inertias[k] = kmeans.inertia_

pd.Series(inertias).plot()
plt.xlabel("K (num of clusters)")
plt.ylabel("Inertia Score")

k = 5
kmeans = cluster.KMeans(n_clusters=k, random_state = 1)
rfm["cluster"] = kmeans.fit_predict(X)

rfm.cluster.value_counts()

rfm["distance"] = 0.0
for i in range(k):
    centroid = kmeans.cluster_centers_[i].reshape(1, -1)
    cluster_points = X[rfm.cluster == i]
    rfm["distance"][rfm.cluster == i] = metrics.euclidean_distances(centroid, cluster_points).flatten()
rfm.sample(20)

rfm.groupby("cluster").distance.agg(["mean", "count"])


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
For calculating rececency, use max for InvoiceDate as point of reference.
Can you write Python code for it?

last_date = df.InvoiceDate.max()
last_date


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Load data
Can you write Python code for it?

df = pd.read_excel("/data/Online Retail.xlsx")
df.head()


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Rename the columns - "recency", "frequency", "monetary"
Can you write Python code for it?

rfm.columns = ["recency", "frequency", "monetary"]
rfm.head()


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
Use describe to find range of each column
Can you write Python code for it?


I am working on the file "Scikit - 28 Clustering - Online Retail Sales Data.ipynb".
The first task is:
We see quantity column has a large negative value. Probably negative values are not valid for this analysis. Drop the records having negative values in frequency. Verify that there is not more nagative values in the columns
Can you write Python code for it?

df = df[df.Quantity>0]
df.info()

df.describe()

df.head()


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Build pipeline for numeric and categorical variables. Search over a hyper parameter space to tune the model.
Can you write Python code for it?

cat_pipe = pipeline.Pipeline([
    ('imputer', impute.SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', preprocessing.OneHotEncoder(handle_unknown='error', drop="first"))
])

num_pipe = pipeline.Pipeline([
    ('imputer', impute.SimpleImputer(strategy='median')),
    ('poly', preprocessing.PolynomialFeatures(degree=2, include_bias=False)),
    ('scaler', preprocessing.StandardScaler()),
])

preprocessing_pipe = compose.ColumnTransformer([
    ("cat", cat_pipe, cat_columns),
    ("num", num_pipe, num_columns)
])

estimator_pipe = pipeline.Pipeline([
    ("preprocessing", preprocessing_pipe),
    ("est", linear_model.ElasticNet(random_state=1))
])

param_grid = {
    "est__alpha": 0.0 + np.random.random(10) * 0.02,
    "est__l1_ratio": np.linspace(0.0001, 1, 20),
}

gsearch = model_selection.GridSearchCV(estimator_pipe, param_grid, cv = 5, verbose=1, n_jobs=8)

gsearch.fit(X, y)

print(gsearch.best_score_, gsearch.best_params_)


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Create a single record with the feature values to get the estimate.
Can you write Python code for it?

record = {"age": 18, "gender": "male", "bmi": 33.0, "smoker": "no", "children": 1, "region": "southeast"}
record


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Get the prediction for the df_input.
Can you write Python code for it?


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Identitify categorical columns and numeric columns. We will apply imputer (to replace null values) and one-hot encoding to categorical columns and imputer (to replace null values), polynomial transformation and standard scaler (z-scoring) to numeric values.
Can you write Python code for it?

cat_columns = ["gender", "smoker", "region"]
num_columns = ["age", "bmi", "children"]


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Load data
Can you write Python code for it?

df = pd.read_csv("/data/insurance.csv")
df.head()


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Reload the model from the disk. In real-use case, probably you will keep the following lines and their dependencies in a seperate script file.
Can you write Python code for it?

import pickle
import pandas as pd
with open(r"/tmp/model.pickle", "rb") as f:
    est = pickle.load(f)


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Save the model as pickle file, so that during prediction we can re-use the trained model without training the model again from scratch.
Can you write Python code for it?

with open(r"/tmp/model.pickle", "wb") as f:
    pickle.dump(gsearch, f)


I am working on the file "Scikit - 29 Build End to End pipeline.ipynb".
The first task is:
Show the few actual values vs predicted values.
Can you write Python code for it?

pd.DataFrame({"actual": y, "predict": y_pred}).sample(10)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
KNN(K=1)
Can you write Python code for it?

knn1 = KNeighborsClassifier(n_neighbors=1)
knn1.fit(X, y)
y_pred = knn1.predict(X)
print metrics.accuracy_score(y, y_pred)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()

logreg.fit(X, y)
y_pred = logreg.predict(X)
print "predicted response:\n",y_pred

len(y_pred)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
上面我们得到了训练准确度为100%的模型，貌似得到了最好的模型和参数。但我们回想一下KNN算法的原理，KNN算法寻找训练数据中的K个最近的数据，它使用指向最多的那个类别来作为预测的输出。
这下我们就明白了为什么当K=1的时候KNN算法的训练准确度为1，KNN会查找在训练数据集中的最近的观测，训练得到的模型会在相同的数据集中找到相同的观测。换句话说，KNN算法已经记住了训练数据集，因为我们使用同样的数据作为测试的数据。
小结
我们要明确训练模型的目的是得到对于未来的观测数据的良好的预测性能，如果我们只是试图最大化训练的准确率，很有可能得到的是针对训练数据的过于复杂的模型，这种模型不具有很好的泛化能力，也就是说对于未知数据的预测得不到很好的性能。
构建没有必要的过于复杂的模型被称作过拟合。这样的模型过分地拟合噪声数据，胜过去拟合信号数据。在KNN模型中，小K值的模型会生成复杂度很高的模型，因为它跟随数据中的噪声。
下面这个是过拟合的一幅图像：

这里绿色的分割线努力去学习噪声的规律，而不像黑色的分割线去学习数据信号的规律，这样对比之下，黑色的线的泛化能力应该更好。
2. 分别设置训练集和测试集

将数据集分成两部分：训练集和测试集
使用训练集对模型进行训练
使用测试集进行模型的测试并评估性能如何
Can you write Python code for it?

print X.shape
print y.shape

# 第一步：将X和y分割成训练和测试集
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)
# 这里的random_state参数根据给定的给定的整数，得到伪随机生成器的随机采样


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
上面这个图告诉我们train_test_split函数的功能，将一个数据集分成两部分。这样使用不同的数据集对模型进行分别的训练和测试，得到的测试准确率能够更好的估计模型对于未知数据的预测效果。
这里测试数据的大小没有统一的标准，一般选择数据集的20%-40%作为测试数据。
Can you write Python code for it?

print X_train.shape
print X_test.shape

print y_train.shape
print y_test.shape

# 第二步：使用训练数据训练模型
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# 第三步： 针对测试数据进行预测，并得到测试准确率
y_pred = logreg.predict(X_test)

print metrics.accuracy_score(y_test, y_pred)
# 可以尝试一下，对于上面不同数据集分割，得到的测试准确率不同


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
以上说明对于训练的数据，我们有96%的数据预测正确。这里我们使用相同的数据来训练和预测，使用的度量称其为训练准确度。
KNN(K=5)
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn5 = KNeighborsClassifier(n_neighbors=5)
knn5.fit(X, y)
y_pred = knn5.predict(X)
print metrics.accuracy_score(y, y_pred)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
使用KNN算法
Can you write Python code for it?

# K=5

knn5 = KNeighborsClassifier(n_neighbors=5)
knn5.fit(X_train, y_train)
y_pred = knn5.predict(X_test)
print metrics.accuracy_score(y_test, y_pred)

# K=1

knn1 = KNeighborsClassifier(n_neighbors=1)
knn1.fit(X_train, y_train)
y_pred = knn1.predict(X_test)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
分类准确率
现在我们需要使用一种度量方式来评价我们的模型的运行情况，我们使用正确预测的比例来作为评估的度量(evaluation metric)。
Can you write Python code for it?

from sklearn import metrics
print metrics.accuracy_score(y, y_pred)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
小结

KNN的模型复杂度主要由K的值决定，K值越小，复杂度越高
训练准确度随着模型复杂度更加复杂而升高
测试准确率在模型过于复杂和过于简单的时候都比较低

这种分割训练和测试数据集的模型评估流程的缺点是：这种方式会导致待预测数据(out-of-sample)估计准确率的方差过高。因为这种方式过于依赖训练数据和测试数据在数据集中的选择方式。一种克服这种的缺点的模型评估流程称为K折交叉验证，这种方法通过多次反复分割训练、测试集，对结果进行平均。
3. K折交叉检验
K折将数据集分成K个部分，其中K-1组数据作为构建预测函数的训练之用，剩余的一组数据作为测试之用。
Can you write Python code for it?

from sklearn.cross_validation import KFold
import numpy as np

def cv_estimate(k, kfold=5):
    cv = KFold(n = X.shape[0], n_folds=kfold)
    clf = KNeighborsClassifier(n_neighbors=k)
    score = 0
    for train, test in cv:
        clf.fit(X[train], y[train])
        score += clf.score(X[test], y[test])
        #print clf.score(X[test], y[test])
    score /= kfold
    return score

# 测试从K=1到K=25，记录测试准确率
k_range = range(1, 26)
test_accuracy = []

for k in k_range:
    test_accuracy.append(cv_estimate(k, 5))

print test_accuracy

plt.plot(k_range, test_accuracy)
plt.xlabel("Value of K for KNN")
plt.ylabel("Average Accuracy of Kfold CV")


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
我们能找到一个比较好的K值吗？
Can you write Python code for it?

# 测试从K=1到K=25，记录测试准确率
k_range = range(1, 26)
test_accuracy = []

for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    y_pred = knn.predict(X_test)
    test_accuracy.append(metrics.accuracy_score(y_test, y_pred))

%matplotlib inline
import matplotlib.pyplot as plt

plt.plot(k_range, test_accuracy)
plt.xlabel("Value of K for KNN")
plt.ylabel("Testing Accuracy")


I am working on the file "Scikit-learn/.ipynb_checkpoints/(2)choose_a_ml_model-checkpoint.ipynb".
The first task is:
模型参数的选择
在这里，我们选择K的值为11，因为在K=11时，有较高的测试准确率。
Can you write Python code for it?

# 这里我们对未知数据进行预测
knn11 = KNeighborsClassifier(n_neighbors=11)
knn11.fit(X, y)

knn11.predict([3, 5, 4, 2])


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
$y = 2.88 + 0.0466 * TV + 0.179 * Radio + 0.00345 * Newspaper$
如何解释各个特征对应的系数的意义？
- 对于给定了Radio和Newspaper的广告投入，如果在TV广告上每多投入1个单位，对应销量将增加0.0466个单位
- 更明确一点，加入其它两个媒体投入固定，在TV广告上没增加1000美元（因为单位是1000美元），销量将增加46.6（因为单位是1000）
(4)预测
Can you write Python code for it?


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
(1)平均绝对误差(Mean Absolute Error, MAE)
$\frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y_i}|$
(2)均方误差(Mean Squared Error, MSE)
$\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$
(3)均方根误差(Root Mean Squared Error, RMSE)
$\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2}$
Can you write Python code for it?

from sklearn import metrics
import numpy as np
# calculate MAE by hand
print "MAE by hand:",(10 + 0 + 20 + 10)/4.

# calculate MAE using scikit-learn
print "MAE:",metrics.mean_absolute_error(true, pred)

# calculate MSE by hand
print "MSE by hand:",(10**2 + 0**2 + 20**2 + 10**2)/4.

# calculate MSE using scikit-learn
print "MSE:",metrics.mean_squared_error(true, pred)

# calculate RMSE by hand
print "RMSE by hand:",np.sqrt((10**2 + 0**2 + 20**2 + 10**2)/4.)

# calculate RMSE using scikit-learn
print "RMSE:",np.sqrt(metrics.mean_squared_error(true, pred))


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
(2)构造训练集和测试集
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# default split is 75% for training and 25% for testing
print X_train.shape
print y_train.shape
print X_test.shape
print y_test.shape


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
(3)Scikit-learn的线性回归
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

linreg = LinearRegression()

linreg.fit(X_train, y_train)

print linreg.intercept_
print linreg.coef_

# pair the feature names with the coefficients
zip(feature_cols, linreg.coef_)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
2. 线性回归模型
优点：快速；没有调节参数；可轻易解释；可理解
缺点：相比其他复杂一些的模型，其预测准确率不是太高，因为它假设特征和响应之间存在确定的线性关系，这种假设对于非线性的关系，线性回归模型显然不能很好的对这种数据建模。
线性模型表达式：
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n$
其中
- y是响应
- $\beta_0是截距$
- $\beta_1是x1的系数，以此类推$
在这个案例中：
$y = \beta_0 + \beta_1TV + \beta_2Radio + ... + \beta_n*Newspaper$
（1）使用pandas来构建X和y

scikit-learn要求X是一个特征矩阵，y是一个NumPy向量
pandas构建在NumPy之上
因此，X可以是pandas的DataFrame，y可以是pandas的Series，scikit-learn可以理解这种结构
Can you write Python code for it?

# create a python list of feature names
feature_cols = ['TV', 'Radio', 'Newspaper']

# use the list to select a subset of the original DataFrame
X = data[feature_cols]

# equivalent command to do this in one line
X = data[['TV', 'Radio', 'Newspaper']]

# print the first 5 rows
X.head()

# check the type and shape of X
print type(X)
print X.shape

# select a Series from the DataFrame
y = data['Sales']

# equivalent command that works if there are no spaces in the column name
y = data.Sales

# print the first 5 values
y.head()

print type(y)
print y.shape


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
3. 回归问题的评价测度
对于分类问题，评价测度是准确率，但这种方法不适用于回归问题。我们使用针对连续数值的评价测度(evaluation metrics)。
下面介绍三种常用的针对回归问题的评价测度
Can you write Python code for it?

# define true and predicted response values
true = [100, 50, 30, 20]
pred = [90, 50, 50, 30]


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
4. 特征选择
在之前展示的数据中，我们看到Newspaper和销量之间的线性关系比较弱，现在我们移除这个特征，看看线性回归预测的结果的RMSE如何？
Can you write Python code for it?

feature_cols = ['TV', 'Radio']

X = data[feature_cols]
y = data.Sales

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

linreg.fit(X_train, y_train)

y_pred = linreg.predict(X_test)

print np.sqrt(metrics.mean_squared_error(y_test, y_pred))


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
seaborn的pairplot函数绘制X的每一维度和对应Y的散点图。通过设置size和aspect参数来调节显示的大小和比例。可以从图中看出，TV特征和销量是有比较强的线性关系的，而Radio和Sales线性关系弱一些，Newspaper和Sales线性关系更弱。通过加入一个参数kind='reg'，seaborn可以添加一条最佳拟合直线和95%的置信带。
Can you write Python code for it?

sns.pairplot(data, x_vars=['TV','Radio','Newspaper'], y_vars='Sales', size=7, aspect=0.8, kind='reg')


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
上面显示的结果类似一个电子表格，这个结构称为Pandas的数据帧(data frame)。
pandas的两个主要数据结构：Series和DataFrame：
- Series类似于一维数组，它有一组数据以及一组与之相关的数据标签(即索引)组成。
- DataFrame是一个表格型的数据结构，它含有一组有序的列，每列可以是不同的值类型。DataFrame既有行索引也有列索引，它可以被看做由Series组成的字典。
Can you write Python code for it?

# display the last 5 rows
data.tail()

# check the shape of the DataFrame(rows, colums)
data.shape


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
特征：
- TV：对于一个给定市场中单一产品，用于电视上的广告费用（以千为单位）
- Radio：在广播媒体上投资的广告费用
- Newspaper：用于报纸媒体的广告费用
响应：
- Sales：对应产品的销量
在这个案例中，我们通过不同的广告投入，预测产品销量。因为响应变量是一个连续的值，所以这个问题是一个回归问题。数据集一共有200个观测值，每一组观测对应一个市场的情况。
Can you write Python code for it?

import seaborn as sns

%matplotlib inline

# visualize the relationship between the features and the response using scatterplots
sns.pairplot(data, x_vars=['TV','Radio','Newspaper'], y_vars='Sales', size=7, aspect=0.8)


I am working on the file "Scikit-learn/.ipynb_checkpoints/(3)linear_regression-checkpoint.ipynb".
The first task is:
计算Sales预测的RMSE
Can you write Python code for it?


I am working on the file "Scikit-learn/.ipynb_checkpoints/(4)cross_validation-checkpoint.ipynb".
The first task is:
3. 使用交叉验证的建议

K=10是一个一般的建议
如果对于分类问题，应该使用分层抽样（stratified sampling）来生成数据，保证正负例的比例在训练集和测试集中的比例相同

4. 交叉验证的例子
4.1 用于调节参数
交叉验证的方法可以帮助我们进行调参，最终得到一组最佳的模型参数。下面的例子我们依然使用iris数据和KNN模型，通过调节参数，得到一组最佳的参数使得测试数据的准确率和泛化能力最佳。
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score

knn = KNeighborsClassifier(n_neighbors=5)
# 这里的cross_val_score将交叉验证的整个过程连接起来，不用再进行手动的分割数据
# cv参数用于规定将原始数据分成多少份
scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')
print scores

# use average accuracy as an estimate of out-of-sample accuracy
# 对十次迭代计算平均的测试准确率
print scores.mean()

# search for an optimal value of K for KNN model
k_range = range(1,31)
k_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')
    k_scores.append(scores.mean())

print k_scores

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(k_range, k_scores)
plt.xlabel("Value of K for KNN")
plt.ylabel("Cross validated accuracy")


I am working on the file "Scikit-learn/.ipynb_checkpoints/(4)cross_validation-checkpoint.ipynb".
The first task is:
上面的例子显示了偏置-方差的折中，K较小的情况时偏置较低，方差较高；K较高的情况时，偏置较高，方差较低；最佳的模型参数取在中间位置，该情况下，使得偏置和方差得以平衡，模型针对于非样本数据的泛化能力是最佳的。
4.2 用于模型选择
交叉验证也可以帮助我们进行模型选择，以下是一组例子，分别使用iris数据，KNN和logistic回归模型进行模型的比较和选择。
Can you write Python code for it?

# 10-fold cross-validation with the best KNN model
knn = KNeighborsClassifier(n_neighbors=20)
print cross_val_score(knn, X, y, cv=10, scoring='accuracy').mean()

# 10-fold cross-validation with logistic regression
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
print cross_val_score(logreg, X, y, cv=10, scoring='accuracy').mean()


I am working on the file "Scikit-learn/.ipynb_checkpoints/(4)cross_validation-checkpoint.ipynb".
The first task is:
上面的测试准确率可以看出，不同的训练集、测试集分割的方法导致其准确率不同，而交叉验证的基本思想是：将数据集进行一系列分割，生成一组不同的训练测试集，然后分别训练模型并计算测试准确率，最后对结果进行平均处理。这样来有效降低测试准确率的差异。
2. K折交叉验证

将数据集平均分割成K个等份
使用1份数据作为测试数据，其余作为训练数据
计算测试准确率
使用不同的测试集，重复2、3步骤
对测试准确率做平均，作为对未知数据预测准确率的估计
Can you write Python code for it?

# 下面代码演示了K-fold交叉验证是如何进行数据分割的
# simulate splitting a dataset of 25 observations into 5 folds
from sklearn.cross_validation import KFold
kf = KFold(25, n_folds=5, shuffle=False)

# print the contents of each training and testing set
print '{} {:^61} {}'.format('Iteration', 'Training set observations', 'Testing set observations')
for iteration, data in enumerate(kf, start=1):
    print '{:^9} {} {:^25}'.format(iteration, data[0], data[1])


I am working on the file "Scikit-learn/.ipynb_checkpoints/(4)cross_validation-checkpoint.ipynb".
The first task is:
这里要注意的是，上面的scores都是负数，为什么均方误差会出现负数的情况呢？因为这里的mean_squared_error是一种损失函数，优化的目标的使其最小化，而分类准确率是一种奖励函数，优化的目标是使其最大化。
Can you write Python code for it?

# fix the sign of MSE scores
mse_scores = -scores
print mse_scores

# convert from MSE to RMSE
rmse_scores = np.sqrt(mse_scores)
print rmse_scores

# calculate the average RMSE
print rmse_scores.mean()

# 10-fold cross-validation with two features (excluding Newspaper)
feature_cols = ['TV', 'Radio']
X = data[feature_cols]
print np.sqrt(-cross_val_score(lm, X, y, cv=10, scoring='mean_squared_error')).mean()


I am working on the file "Scikit-learn/.ipynb_checkpoints/(5)grid_search-checkpoint.ipynb".
The first task is:
4. 使用最佳参数做出预测
Can you write Python code for it?

# train your model using all data and the best known parameters
knn = KNeighborsClassifier(n_neighbors=13, weights='uniform')
knn.fit(X, y)

# make a prediction on out-of-sample data
knn.predict([3, 5, 4, 2])


I am working on the file "Scikit-learn/.ipynb_checkpoints/(5)grid_search-checkpoint.ipynb".
The first task is:
5. 使用RandomizeSearchCV来降低计算代价

RandomizeSearchCV用于解决多个参数的搜索过程中计算代价过高的问题
RandomizeSearchCV搜索参数中的一个子集，这样你可以控制计算代价
Can you write Python code for it?

from sklearn.grid_search import RandomizedSearchCV

# specify "parameter distributions" rather than a "parameter grid"
param_dist = dict(n_neighbors=k_range, weights=weight_options)

# n_iter controls the number of searches
rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10, random_state=5)
rand.fit(X, y)
rand.grid_scores_

# examine the best model
print rand.best_score_
print rand.best_params_

# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score
best_scores = []
for _ in range(20):
    rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10)
    rand.fit(X, y)
    best_scores.append(round(rand.best_score_, 3))
print best_scores


I am working on the file "Scikit-learn/.ipynb_checkpoints/(5)grid_search-checkpoint.ipynb".
The first task is:
我们可以知道，这里的grid search针对每个参数进行了10次交叉验证，并且一共对30个参数进行相同过程的交叉验证
Can you write Python code for it?

grid.fit(X, y)

# view the complete results (list of named tuples)
grid.grid_scores_

# examine the first tuple
print grid.grid_scores_[0].parameters
print grid.grid_scores_[0].cv_validation_scores
print grid.grid_scores_[0].mean_validation_score

# create a list of the mean scores only
grid_mean_scores = [result.mean_validation_score for result in grid.grid_scores_]
print grid_mean_scores

# plot the results
plt.plot(k_range, grid_mean_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')

# examine the best model
print grid.best_score_
print grid.best_params_
print grid.best_estimator_


I am working on the file "Scikit-learn/.ipynb_checkpoints/(6)classification_metrics-checkpoint.ipynb".
The first task is:
4. 基于混淆矩阵的评估度量
准确率、识别率（Classification Accuracy）：分类器正确分类的比例
Can you write Python code for it?

print (TP+TN) / float(TP+TN+FN+FP)
print metrics.accuracy_score(y_test, y_pred_class)