I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/10-Data-Capstone-Projects/02-911 Calls Data Capstone Project - Solutions.ipynb".
The first task is:
Read in the csv file as a dataframe called df
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/10-Data-Capstone-Projects/02-911 Calls Data Capstone Project - Solutions.ipynb".
The first task is:
What are the top 5 townships (twp) for 911 calls?
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/10-Data-Capstone-Projects/02-911 Calls Data Capstone Project - Solutions.ipynb".
The first task is:
You can now grab specific attributes from a Datetime object by calling them. For example:
time = df['timeStamp'].iloc[0]
time.hour

You can use Jupyter's tab method to explore the various attributes you can call. Now that the timestamp column are actually DateTime objects, use .apply() to create 3 new columns called Hour, Month, and Day of Week. You will create these columns based off of the timeStamp column, reference the solutions if you get stuck on this step.
Can you write Python code for it?

df['Hour'] = df['timeStamp'].apply(lambda time: time.hour)
df['Month'] = df['timeStamp'].apply(lambda time: time.month)
df['Day of Week'] = df['timeStamp'].apply(lambda time: time.dayofweek)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/10-Data-Capstone-Projects/02-911 Calls Data Capstone Project - Solutions.ipynb".
The first task is:
You should have noticed it was missing some Months, let's see if we can maybe fill in this information by plotting the information in another way, possibly a simple line plot that fills in the missing months, in order to do this, we'll need to do some work with pandas...
 Now create a gropuby object called byMonth, where you group the DataFrame by the month column and use the count() method for aggregation. Use the head() method on this returned DataFrame.
Can you write Python code for it?

byMonth = df.groupby('Month').count()
byMonth.head()


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Check out the Data
Can you write Python code for it?

USAhousing = pd.read_csv('USA_Housing.csv')

USAhousing.head()

USAhousing.info()

USAhousing.describe()

USAhousing.columns


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Creating and Training the Model
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

lm = LinearRegression()

lm.fit(X_train,y_train)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Interpreting the coefficients:

Holding all other features fixed, a 1 unit increase in Avg. Area Income is associated with an increase of \$21.52 .
Holding all other features fixed, a 1 unit increase in Avg. Area House Age is associated with an increase of \$164883.28 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Rooms is associated with an increase of \$122368.67 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Bedrooms is associated with an increase of \$2233.80 .
Holding all other features fixed, a 1 unit increase in Area Population is associated with an increase of \$15.15 .

Does this make sense? Probably not because I made up this data. If you want real data to repeat this sort of analysis, check out the boston dataset:
from sklearn.datasets import load_boston
boston = load_boston()
print(boston.DESCR)
boston_df = boston.data

Predictions from our Model
Let's grab predictions off our test set and see how well it did!
Can you write Python code for it?

predictions = lm.predict(X_test)

plt.scatter(y_test,predictions)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Regression Evaluation Metrics
Here are three common evaluation metrics for regression problems:
Mean Absolute Error (MAE) is the mean of the absolute value of the errors:
$$\frac 1n\sum_{i=1}^n|y_i-\hat{y}_i|$$
Mean Squared Error (MSE) is the mean of the squared errors:
$$\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2$$
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:
$$\sqrt{\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2}$$
Comparing these metrics:

MAE is the easiest to understand, because it's the average error.
MSE is more popular than MAE, because MSE "punishes" larger errors, which tends to be useful in the real world.
RMSE is even more popular than MSE, because RMSE is interpretable in the "y" units.

All of these are loss functions, because we want to minimize them.
Can you write Python code for it?

from sklearn import metrics

print('MAE:', metrics.mean_absolute_error(y_test, predictions))
print('MSE:', metrics.mean_squared_error(y_test, predictions))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, predictions)))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Train Test Split
Now let's split the data into a training set and a testing set. We will train out model on the training set and then use the test set to evaluate the model.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=101)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/01-Linear Regression with Python.ipynb".
The first task is:
Training a Linear Regression Model
Let's now begin to train out regression model! We will need to first split up our data into an X array that contains the features to train on, and a y array with the target variable, in this case the Price column. We will toss out the Address column because it only has text info that the linear regression model can't use.
X and y arrays
Can you write Python code for it?

X = USAhousing[['Avg. Area Income', 'Avg. Area House Age', 'Avg. Area Number of Rooms',
               'Avg. Area Number of Bedrooms', 'Area Population']]
y = USAhousing['Price']


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Based off this plot what looks to be the most correlated feature with Yearly Amount Spent?
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Conclusion
We still want to figure out the answer to the original question, do we focus our efforst on mobile app or website development? Or maybe that doesn't even really matter, and Membership Time is what is really important.  Let's see if we can interpret the coefficients at all to get an idea.
 Recreate the dataframe below.
Can you write Python code for it?

coeffecients = pd.DataFrame(lm.coef_,X.columns)
coeffecients.columns = ['Coeffecient']
coeffecients


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Create a scatterplot of the real test values versus the predicted values.
Can you write Python code for it?

plt.scatter(y_test,predictions)
plt.xlabel('Y Test')
plt.ylabel('Predicted Y')


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Create an instance of a LinearRegression() model named lm.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Do the same but with the Time on App column instead.
Can you write Python code for it?

sns.jointplot(x='Time on App',y='Yearly Amount Spent',data=customers)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Evaluating the Model
Let's evaluate our model performance by calculating the residual sum of squares and the explained variance score (R^2).
 Calculate the Mean Absolute Error, Mean Squared Error, and the Root Mean Squared Error. Refer to the lecture or to Wikipedia for the formulas
Can you write Python code for it?

# calculate these metrics by hand!
from sklearn import metrics

print('MAE:', metrics.mean_absolute_error(y_test, predictions))
print('MSE:', metrics.mean_squared_error(y_test, predictions))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, predictions)))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Exploratory Data Analysis
Let's explore the data!
For the rest of the exercise we'll only be using the numerical data of the csv file.

Use seaborn to create a jointplot to compare the Time on Website and Yearly Amount Spent columns. Does the correlation make sense?
Can you write Python code for it?

sns.set_palette("GnBu_d")
sns.set_style('whitegrid')

# More time on site, more money spent.
sns.jointplot(x='Time on Website',y='Yearly Amount Spent',data=customers)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Let's explore these types of relationships across the entire data set. Use pairplot to recreate the plot below.(Don't worry about the the colors)
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Predicting Test Data
Now that we have fit our model, let's evaluate its performance by predicting off the test values!
 Use lm.predict() to predict off the X_test set of the data.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Residuals
You should have gotten a very good model with a good fit. Let's quickly explore the residuals to make sure everything was okay with our data.
Plot a histogram of the residuals and make sure it looks normally distributed. Use either seaborn distplot, or just plt.hist().
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/11-Linear-Regression/03-Linear Regression Project - Solutions.ipynb".
The first task is:
Train/fit lm on the training data.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Check the head of ad_data
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Create a classification report for the model.
Can you write Python code for it?

from sklearn.metrics import classification_report

print(classification_report(y_test,predictions))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Create a jointplot of 'Daily Time Spent on Site' vs. 'Daily Internet Usage'
Can you write Python code for it?

sns.jointplot(x='Daily Time Spent on Site',y='Daily Internet Usage',data=ad_data,color='green')


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Create a jointplot showing Area Income versus Age.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Exploratory Data Analysis
Let's use seaborn to explore the data!
Try recreating the plots shown below!
 Create a histogram of the Age
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Get the Data
Read in the advertising.csv file and set it to a data frame called ad_data.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Predictions and Evaluations
 Now predict values for the testing data.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/13-Logistic-Regression/03-Logistic Regression Project - Solutions.ipynb".
The first task is:
Train and fit a logistic regression model on the training set.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

logmodel = LogisticRegression()
logmodel.fit(X_train,y_train)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/01-K Means Clustering with Python.ipynb".
The first task is:
Create some Data
Can you write Python code for it?

from sklearn.datasets import make_blobs

# Create Data
data = make_blobs(n_samples=200, n_features=2,
                           centers=4, cluster_std=1.8,random_state=101)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/01-K Means Clustering with Python.ipynb".
The first task is:
Creating the Clusters
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=4)

kmeans.fit(data[0])

kmeans.cluster_centers_

kmeans.labels_

f, (ax1, ax2) = plt.subplots(1, 2, sharey=True,figsize=(10,6))
ax1.set_title('K Means')
ax1.scatter(data[0][:,0],data[0][:,1],c=kmeans.labels_,cmap='rainbow')
ax2.set_title("Original")
ax2.scatter(data[0][:,0],data[0][:,1],c=data[1],cmap='rainbow')


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/01-K Means Clustering with Python.ipynb".
The first task is:
Visualize Data
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Check the info() and describe() methods on the data.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Create a confusion matrix and classification report to see how well the Kmeans clustering worked without being given any labels.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix,classification_report
print(confusion_matrix(df['Cluster'],kmeans.labels_))
print(classification_report(df['Cluster'],kmeans.labels_))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Create a stacked histogram showing Out of State Tuition based on the Private column. Try doing this using sns.FacetGrid. If that is too tricky, see if you can do it just by using two instances of pandas.plot(kind='hist').
Can you write Python code for it?

sns.set_style('darkgrid')
g = sns.FacetGrid(df,hue="Private",palette='coolwarm',size=6,aspect=2)
g = g.map(plt.hist,'Outstate',bins=20,alpha=0.7)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
EDA
It's time to create some data visualizations!
 Create a scatterplot of Grad.Rate versus Room.Board where the points are colored by the Private column.
Can you write Python code for it?

sns.set_style('whitegrid')
sns.lmplot('Room.Board','Grad.Rate',data=df, hue='Private',
           palette='coolwarm',size=6,aspect=1,fit_reg=False)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Evaluation
There is no perfect way to evaluate clustering if you don't have the labels, however since this is just an exercise, we do have the labels, so we take advantage of this to evaluate our clusters, keep in mind, you usually won't have this luxury in the real world.
 Create a new column for df called 'Cluster', which is a 1 for a Private school, and a 0 for a public school.
Can you write Python code for it?

def converter(cluster):
    if cluster=='Yes':
        return 1
    else:
        return 0

df['Cluster'] = df['Private'].apply(converter)

df.head()


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Fit the model to all the data except for the Private label.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
K Means Cluster Creation
Now it is time to create the Cluster labels!
 Import KMeans from SciKit Learn.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Notice how there seems to be a private school with a graduation rate of higher than 100%.What is the name of that school?
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/17-K-Means-Clustering/03-K Means Clustering Project - Solutions.ipynb".
The first task is:
Set that school's graduation rate to 100 so it makes sense. You may get a warning not an error) when doing this operation, so use dataframe operations or just re-do the histogram visualization to make sure it actually went through.
Can you write Python code for it?

df['Grad.Rate']['Cazenovia College'] = 100

df[df['Grad.Rate'] > 100]

sns.set_style('darkgrid')
g = sns.FacetGrid(df,hue="Private",palette='coolwarm',size=6,aspect=2)
g = g.map(plt.hist,'Grad.Rate',bins=20,alpha=0.7)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/01-Tensorflow Basics.ipynb".
The first task is:
Now run the session to perform the Operation:
Can you write Python code for it?

with tf.Session() as sess:
    result = sess.run(matrix_multi)
    print(result)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/01-Tensorflow Basics.ipynb".
The first task is:
Operations
You can line up multiple Tensorflow operations in to be run during a session:
Can you write Python code for it?

x = tf.constant(2)
y = tf.constant(3)

with tf.Session() as sess:
    print('Operations with Constants')
    print('Addition',sess.run(x+y))
    print('Subtraction',sess.run(x-y))
    print('Multiplication',sess.run(x*y))
    print('Division',sess.run(x/y))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/01-Tensorflow Basics.ipynb".
The first task is:
Running Sessions
Now you can create a TensorFlow Session, which is a class for running TensorFlow operations.
A Session object encapsulates the environment in which Operation
objects are executed, and Tensor objects are evaluated. For example:
Can you write Python code for it?

sess = tf.Session()

sess.run(hello)

type(sess.run(hello))

sess.run(x)

type(sess.run(x))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/01-Tensorflow Basics.ipynb".
The first task is:
Simple Constants
Let's show how to create a simple constant with Tensorflow, which TF stores as a tensor object:
Can you write Python code for it?

hello = tf.constant('Hello World')

type(hello)

x = tf.constant(100)

type(x)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/01-Tensorflow Basics.ipynb".
The first task is:
The matrix multiplication operation:
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/02-MNIST-with-Tensorflow.ipynb".
The first task is:
Alternative sources of the data just in case:

http://yann.lecun.com/exdb/mnist/
https://github.com/mrgloom/MNIST-dataset-in-different-formats
Can you write Python code for it?

type(mnist)

mnist.train.images

mnist.train.num_examples

mnist.test.num_examples

mnist.validation.num_examples


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/02-MNIST-with-Tensorflow.ipynb".
The first task is:
Create Session
Can you write Python code for it?

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    # Train the model for 1000 steps on the training set
    # Using built in batch feeder from mnist for convenience

    for step in range(1000):

        batch_x , batch_y = mnist.train.next_batch(100)

        sess.run(train,feed_dict={x:batch_x,y_true:batch_y})

    # Test the Train Model
    matches = tf.equal(tf.argmax(y,1),tf.argmax(y_true,1))

    acc = tf.reduce_mean(tf.cast(matches,tf.float32))

    print(sess.run(acc,feed_dict={x:mnist.test.images,y_true:mnist.test.labels}))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/02-MNIST-with-Tensorflow.ipynb".
The first task is:
Create the Model
Can you write Python code for it?

x = tf.placeholder(tf.float32,shape=[None,784])

# 10 because 0-9 possible numbers
W = tf.Variable(tf.zeros([784,10]))

b = tf.Variable(tf.zeros([10]))

# Create the Graph
y = tf.matmul(x,W) + b


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/02-MNIST-with-Tensorflow.ipynb".
The first task is:
Loss and Optimizer
Can you write Python code for it?

y_true = tf.placeholder(tf.float32,[None,10])

# Cross Entropy

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y_true, logits=y))

optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.5)

train = optimizer.minimize(cross_entropy)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/02-MNIST-with-Tensorflow.ipynb".
The first task is:
Visualizing the Data
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

mnist.train.images[1].shape

plt.imshow(mnist.train.images[1].reshape(28,28))

plt.imshow(mnist.train.images[1].reshape(28,28),cmap='gist_gray')

mnist.train.images[1].max()

plt.imshow(mnist.train.images[1].reshape(784,1))

plt.imshow(mnist.train.images[1].reshape(784,1),cmap='gist_gray',aspect=0.02)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/03-Tensorflow with Estimators.ipynb".
The first task is:
Estimators
Let's show you how to use the simpler Estimator interface!
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/03-Tensorflow with Estimators.ipynb".
The first task is:
Input Function
Can you write Python code for it?

# there is also a pandas_input_fn we'll see in the exercise!!
input_func = tf.estimator.inputs.pandas_input_fn(x=X_train,y=y_train,batch_size=10,num_epochs=5,shuffle=True)

classifier = tf.estimator.DNNClassifier(hidden_units=[10, 20, 10], n_classes=3,feature_columns=feat_cols)

classifier.train(input_fn=input_func,steps=50)


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/03-Tensorflow with Estimators.ipynb".
The first task is:
Model Evaluation
 Use the predict method from the classifier model to create predictions from X_test
Can you write Python code for it?

pred_fn = tf.estimator.inputs.pandas_input_fn(x=X_test,batch_size=len(X_test),shuffle=False)

note_predictions = list(classifier.predict(input_fn=pred_fn))

note_predictions[0]

final_preds  = []
for pred in note_predictions:
    final_preds.append(pred['class_ids'][0])


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Convert the scaled features to a dataframe and check the head of this dataframe to make sure the scaling worked.
Can you write Python code for it?

df_feat = pd.DataFrame(scaled_features,columns=data.columns[:-1])
df_feat.head()


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Create a Countplot of the Classes (Authentic 1 vs Fake 0)
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Create a StandardScaler() object called scaler.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Data Preparation
When using Neural Network and Deep Learning based systems, it is usually a good idea to Standardize your data, this step isn't actually necessary for our particular data set, but let's run through it for practice!
Standard Scaling
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
EDA
We'll just do a few quick plots of the data.
 Import seaborn and set matplolib inline for viewing
Can you write Python code for it?

import seaborn as sns
%matplotlib inline


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Fit scaler to the features.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Model Evaluation
 Create another pandas_input_fn that takes in the X_test data for x. Remember this one won't need any y_test info since we will be using this for the network to create its own predictions. Set shuffle=False since we don't need to shuffle for predictions.
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Now create a classification report and a Confusion Matrix. Does anything stand out to you?
Can you write Python code for it?

from sklearn.metrics import classification_report,confusion_matrix

print(confusion_matrix(y_test,final_preds))

print(classification_report(y_test,final_preds))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Now train classifier to the input function. Use steps=500. You can play around with these values if you want!
Note: Ignore any warnings you get, they won't effect your output
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Optional Comparison
 You should have noticed extremely accurate results from the DNN model. Let's compare this to a Random Forest Classifier for a reality check!
Use SciKit Learn to Create a Random Forest Classifier and compare the confusion matrix and classification report to the DNN model
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(n_estimators=200)

rfc.fit(X_train,y_train)

rfc_preds = rfc.predict(X_test)

print(classification_report(y_test,rfc_preds))

print(confusion_matrix(y_test,rfc_preds))


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Tensorflow
Can you write Python code for it?


I am working on the file "Refactored_Py_DS_ML_Bootcamp-master/22-Deep Learning/05-Tensorflow Project Exercise - Solutions.ipynb".
The first task is:
Use the .transform() method to transform the features to a scaled version.
Can you write Python code for it?


I am working on the file "References Script.ipynb".
The first task is:
Check the type distribution across Wikidata, items in our dataset and items with matching domains.
Can you write Python code for it?

dataTypes = pandas.read_csv('./Data/item_type_all_wd.csv')
domainTypes = pandas.read_csv('./Data/item_types_matchdom.csv')
itemTypes = pandas.read_csv('./Data/item_types.csv')

dataTypes.sort_values(by='shareType', ascending=False, inplace=True)
dataTypes.reset_index(drop=True, inplace=True)

tempDomainTypes = np.zeros(len(dataTypes))
tempItemTypes = np.zeros(len(dataTypes))
for i in range(0, len(dataTypes)):
    for j in range(0, len(domainTypes)):
        if dataTypes['type'][i] == domainTypes['type'][j]:
            tempDomainTypes[i] = domainTypes['shareType'][j]
    for j in range(0, len(itemTypes)):
        if dataTypes['type'][i] == itemTypes['type'][j]:
            tempItemTypes[i] = itemTypes['shareType'][j]

width = 0.28

plt.figure(figsize=(10, 6))
ax1 = plt.barh(np.arange(0, len(dataTypes) - 1) - width, np.asarray(dataTypes['shareType'][1:len(dataTypes)]), width, label = 'Wikidata', color='k', edgecolor='white', hatch='//')
ax3 = plt.barh(np.arange(0, len(dataTypes) - 1), tempItemTypes[1:len(dataTypes)], width, label = 'Items In our Dataset', color='#06b694', edgecolor='white', hatch='|')
ax3 = plt.barh(np.arange(0, len(dataTypes) - 1) + width, tempDomainTypes[1:len(dataTypes)], width, label = 'Items with Matching Domains', color='r', edgecolor='white', hatch='x')

plt.legend()
plt.grid(which='both')
plt.xlabel(r'$\%$ on the Total Number of Matches')
plt.ylim(-2 * width)
plt.yticks(np.arange(0, len(dataTypes)) + 0.45 * width, dataTypes['type'][1:len(dataTypes)])
plt.tight_layout()
plt.savefig("./Figures/MatchedTypes.pdf", format="pdf")
plt.show()


I am working on the file "References Script.ipynb".
The first task is:
Domain matches between Wikidata items and corresponding Wikipedia articles, by Wikipedia language version.
Can you write Python code for it?

domainsLanguagesSingle = pandas.read_csv('./Data/matching_domain_lang_single_ref_count.csv', header=None)
domainsLanguages = pandas.read_csv('./Data/matching_domain_unique_lang_count.csv', header=None)

domainsLanguagesSingle.sort_values(by=1, ascending=False, inplace=True)
domainsLanguagesSingle.reset_index(drop=True, inplace=True)

totalDomainsSingleOccurrences = 0
for num in domainsLanguagesSingle[1]:
    totalDomainsSingleOccurrences += num

totalDomainsOccurrences = 0
for num in domainsLanguages[1]:
    totalDomainsOccurrences += num

length = 10
tempDomainsLanguages = np.zeros(length)
for i in range(0, len(domainsLanguagesSingle[1][0:length])):
    for j in range(0, len(domainsLanguages[1])):
        if domainsLanguages[0][j] == domainsLanguagesSingle[0][i]:

            tempDomainsLanguages[i] = domainsLanguages[1][j]

width = 0.4
ax1 = plt.barh(np.arange(length), np.asarray(domainsLanguagesSingle[1][0:length] * 100 / totalDomainsSingleOccurrences), width, label = 'Total Domain Matches', color='#b60628', edgecolor='white', hatch='//')
ax2 = plt.barh(np.arange(length) + width, tempDomainsLanguages * 100 / totalDomainsOccurrences, width, label = 'Unique Domain Matches', color='#06b694', edgecolor='white', hatch='x')

plt.legend()
plt.grid(which='both')
plt.xlabel(r'$\%$ on the Total Number of Matches')
plt.ylim(-width)
plt.yticks(np.arange(length) + width, domainsLanguagesSingle[0][0:length])
plt.tight_layout()
plt.savefig("./Figures/DomainsLanguages.pdf", format="pdf")
plt.show()


I am working on the file "References Script.ipynb".
The first task is:
Scatter Plot of Top-Level Domains
Can you write Python code for it?

# plt.plot(topLevelDomainsScatter['citeTld_pc'], topLevelDomainsScatter['refTld_pc'], "o")
# plt.plot(np.log(topLevelDomainsScatter['citeTld']), np.log(topLevelDomainsScatter['refTld']), "o")

plt.plot(topLevelDomainsScatter['citeTld'], topLevelDomainsScatter['refTld'], "o", color='b')

## Find the selected n-max points; the ones that are close to the far upper-right corner in the logarithmic scale.
selected = topLevelDomainsScatter.sort_values('refTld', ascending=False)

n = 10
for index in selected[:n].index:
    # plt.text(selected['citeTld'][index], selected['refTld'][index], selected['index'][index])
    # We put this IF here in order to avoid clutter with overlapping labels in the graph.
    if selected['index'][index] in ['gov', 'pl']:
        adjust_x = 0.5 * (10 ** np.log10(selected['citeTld'][index]))
        adjust_y = 0.1 * (10 ** np.log10(selected['refTld'][index]))
        plt.annotate(selected['index'][index], (selected['citeTld'][index] - adjust_x, selected['refTld'][index] + adjust_y))
        print('Hi')
    else:
        adjust_x = 0.1 * (10 ** np.log10(selected['citeTld'][index]))
        adjust_y = 0.1 * (10 ** np.log10(selected['refTld'][index]))
        plt.annotate(selected['index'][index], (selected['citeTld'][index] + adjust_x, selected['refTld'][index] + adjust_y))

plt.grid(which='both', linewidth=0.2)
plt.xscale('log')
plt.yscale('log')
plt.xlabel('Wikipedia Citations')
plt.ylabel('Wikidata References')
plt.savefig("./Figures/ScatterPlot.pdf", format="pdf")
plt.show()


I am working on the file "References Script.ipynb".
The first task is:
Top used domains in Wikidata.
Can you write Python code for it?

totalDataOccurrences = 0
for num in dataDomains['refHost']:
    totalDataOccurrences += num

length = 10
width = 0.8
ax = plt.barh(range(length), np.asarray(dataDomains['refHost'][0:length] * 100 / totalDataOccurrences), width, align='center', color='#b60628', edgecolor='white', hatch="//")
plt.grid(which='both')
plt.xlabel(r'$\%$ on the Total Number of References')
plt.ylim(-width)
plt.yticks(range(length), dataDomains['index'][0:length])
plt.tight_layout()
plt.savefig("./Figures/TopDomainsData.pdf", format="pdf")
plt.show()


I am working on the file "Reformer_Text_Generation.ipynb".
The first task is:
As we see above, "Crime and Punishment" has just over half a million tokens with the BPE vocabulary we have selected.
Normally we would have a dataset with many examples, but for this demonstration we fit a language model on the single novel only. We don't want the model to just memorize the dataset by encoding the words in its position embeddings, so at each training iteration we will randomly select how much padding to put before the text vs. after it.
We have 8 TPU cores, so we will separately randomize the amount of padding for each core.
Can you write Python code for it?

# Set up the data pipeline.
def my_inputs(n_devices):
  while True:
    inputs = []
    mask = []
    pad_amounts = onp.random.choice(PAD_AMOUNT, n_devices)
    for i in range(n_devices):
      inputs.append(onp.pad(IDS, (pad_amounts[i], PAD_AMOUNT - pad_amounts[i]),
                            mode='constant'))
      mask.append(onp.pad(onp.ones_like(IDS, dtype=onp.float32),
                          (pad_amounts[i], PAD_AMOUNT - pad_amounts[i]),
                          mode='constant'))
    inputs = onp.stack(inputs)
    mask = onp.stack(mask)
    yield (inputs, inputs, mask)

print("(device count, tokens per device) = ",
      next(my_inputs(trax.math.device_count()))[0].shape)

# Configure hyperparameters.
gin.parse_config("""
import trax.layers
import trax.models
import trax.optimizers
import trax.supervised.inputs
import trax.supervised.trainer_lib

# Parameters that will vary between experiments:
# ==============================================================================
train.model = @trax.models.ReformerLM
# Our model will have 6 layers, alternating between the LSH attention proposed
# in the Reformer paper and local attention within a certain context window.
n_layers = 6
attn_type = [
  @TimeBinCausalAttention,
  @LSHCausalAttention,
  @TimeBinCausalAttention,
  @LSHCausalAttention,
  @TimeBinCausalAttention,
  @LSHCausalAttention,
  ]
share_qk = False  # LSHCausalAttention ignores this flag and always shares q & k
n_heads = 2
attn_kv = 64
dropout = 0.05
n_tokens = 524288

# Parameters for MultifactorSchedule:
# ==============================================================================
MultifactorSchedule.constant = 0.01
MultifactorSchedule.factors = 'constant * linear_warmup * cosine_decay'
MultifactorSchedule.warmup_steps = 100
MultifactorSchedule.steps_per_cycle = 900

# Parameters for Adam:
# ==============================================================================
Adam.weight_decay_rate=0.0
Adam.b1 = 0.86
Adam.b2 = 0.92
Adam.eps = 1e-9

# Parameters for...


I am working on the file "Reformer_Text_Generation.ipynb".
The first task is:
Setting up data and model
In this notebook, we'll be pushing the limits of just how many tokens we can fit on a single TPU device. The TPUs available in Colab have 8GB of memory per core, and 8 cores. We will set up a Reformer model that can fit a copy of "Crime and Punishment" on each of the 8 TPU cores (over 500,000 tokens per 8GB of memory).
Can you write Python code for it?

# Import a copy of "Crime and Punishment", by Fyodor Dostoevsky
with GFile('gs://trax-ml/reformer/crime-and-punishment-2554.txt') as f:
  text = f.read()

# The file read above includes metadata and licensing information.
# For training our language model, we will only use the actual novel text.
start = text.find('CRIME AND PUNISHMENT')  # skip header
start = text.find('CRIME AND PUNISHMENT', start + 1)  # skip header
start = text.find('CRIME AND PUNISHMENT', start + 1)  # skip translator preface
end = text.rfind('End of Project')  # skip extra text at the end
text = text[start:end].strip()

# Load a BPE vocabulaary with 320 types. This mostly consists of single letters
# and pairs of letters, but it has some common words and word pieces, too.
!gsutil cp gs://trax-ml/reformer/cp.320.* .

TOKENIZER = SentencePieceProcessor()
TOKENIZER.load('cp.320.model')

# Tokenize
IDS = TOKENIZER.EncodeAsIds(text)
IDS = onp.asarray(IDS, dtype=onp.int32)
PAD_AMOUNT = 512 * 1024 - len(IDS)
print("Number of tokens:", IDS.shape[0])


I am working on the file "Regex/python_regex_problems.ipynb".
The first task is:
2. Get domains (without protocols) (including extension, e.g. .com) for URLs with both http and https protocols.
Can you write Python code for it?

[m.group(1) for u in urls for m in [re.search('https?://(\w+(.\w+)+)', u)] if m]


I am working on the file "Regex/python_regex_problems.ipynb".
The first task is:
3. Below is a list of language codes. Determine how many are some form of English

English codes will start with En, en, or EN
Can you write Python code for it?

languages = ['Ar','It','it','En','En_gb','jp','en_GB','EN_IE','en-NZ','en','es','ES-es']

sum([1 for c in languages for m in [re.search('^en', c, re.IGNORECASE)] if m])


I am working on the file "Regex/python_regex_problems.ipynb".
The first task is:
4. Fix all langauge codes so that they are formatted as such:

first two letters are lower case
codes with region endings use hyphen and not underscore
the region endings are upper case
Can you write Python code for it?

[m.group(1).lower() + ('-' + m.group(4).upper() if m.group(2) else '')\
 for c in languages for m in [re.search('(^[A-Za-z]{2})((.)([A-Za-z]{2}$))?', c)] if m]


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
As we have studied for the Supergrid data structure, we have to store the differences between the pixel intensities of neighbouring pixels in the middle pixel. These are called as the CRACK EDGES.
The strength of these crack edges, help us determine if we should merge two regions or not.
In the following section, we shall compute the crack edges and store them as stated above.
First we shall iterate all the columns and store the corresponding crack edges, and then move vertically through the pixels for storing the crack edges. Next, we check how many neighbours have a crack edge and if current pixel is a crossing point of edges, by checking the adjacent pixels.
Can you write Python code for it?

def check_neighbours_crack_edge(i,j):
    pix_val = 0
    pixel_label = 0
    count = 0
    if i-1 >= 0:
        if crack_edge_data[i-1,j] != 0:
            pix_val = crack_edge_data[i-1,j]
            pixel_label = label_data[i-1,j]
        else:
            count = count + 1
    if j-1 >= 0:
        if crack_edge_data[i,j-1] != 0:
            pix_val = crack_edge_data[i,j-1]
            pixel_label = label_data[i,j-1]
        else:
            count = count + 1
    if i+1 < len(crack_edge_data):
        if crack_edge_data[i+1,j] != 0:
            pix_val = crack_edge_data[i+1,j]
            pixel_label = label_data[i+1,j]
        else:
            count = count + 1
    if j+1 < len(crack_edge_data[0]):
        if crack_edge_data[i,j+1] != 0:
            pix_val = crack_edge_data[i,j+1]
            pixel_label = label_data[i,j+1]
        else:
            count = count + 1
    if count >2:
        return 0,0
    else:
        return pix_val,pixel_label

for i in range(M):
    for j in range(N):
        if 2*j < len(crack_edge_data[0]) and 2*i+2 < len(crack_edge_data):
            diff = crack_edge_data[2*i,2*j] - crack_edge_data[2*i+2,2*j]
            if diff != 0:
                crack_edge_data[2*i+1,2*j] = 0
                label_data[2*i+1,2*j] = 0
            else:
                crack_edge_data[2*i+1,2*j] = crack_edge_data[2*i,2*j]
                label_data[2*i+1,2*j] = label_data[2*i,2*j]
        if 2*j+2 < len(crack_edge_data[0]) and 2*i < len(crack_edge_data):
            diff = crack_edge_data[2*i,2*j] - crack_edge_data[2*i,2*j+2]
            if diff != 0:
                crack_edge_data[2*i,2*j+1] = 0
                label_data[2*i,2*j+1] = 0
            else:
                crack_edge_data[2*i,2*j+1] = crack_edge_data[2*i,2*j]
                label_data[2*i,2*j+1] = label_data[2*i,2*j]
        if 2*i+1 < len(crack_edge_data) and 2*j+1 < len(crack_edge_data[0]):
            crack_edge_data[2*i+1,2*j+1],label_data[2*i+1,2*j+1] = check_neighbours_crack_edge(2*i+1,2*j+1)


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Below, we can see the segmented regions in the image. Regions have pixels with same average intensity value.
The colored plot of the segmentation, provides a much understandable visualisation, as shown below.
Can you write Python code for it?

plot_input(original_img, 'Region Segmentation - Boundary Melting')

plt.imshow(original_img)
plt.show()


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
It is often beneficial to pre-process our given raw image.
In our preprocessing step, we shall smooth the given image using a Gaussian filter, to remove any noise in the image. As our crack-edge detection and region merging is highly dependent on the difference and uniformity of pixel intensities respectively, we would like to remove extremely high frequencies in the image.
Can you write Python code for it?

gaussian_smooth_img = ndi.filters.gaussian_filter(original_img, 1.2)
plot_input(gaussian_smooth_img, 'Gaussian Smooth')


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Let us now begin with our solution, for implementing the region merging via boundary melting algorithm.
The most fundamental part of the solution is in appropriate creation of Supergrid data structure, which holds the crack edges.
Below, we have stored the original image dimensions in variables M and N.
Next we have defined a matrix 'adj_values' to store the intensities of 4 neighbors corresponding to each pixel.
'region_label' is a matrix that stores the labels of the regions to which each pixel belong. We have initialised the matrices to 0.
'avg_intensity_reg' is a matrix that stores the pixel intensity values(min,max,avg) for every region. Initially, each pixel is considered to be its own region, hence we have values corresponding to each pixel, but as the merging begins, each value in this matrix denotes the values corresponding to the region the pixel belongs.
Can you write Python code for it?

[M,N] = original_img.shape
adj_values = np.array([ {'left': 0, 'right':0,'top':0,'down':0} for x in range(M * N + 1)])
region_label = np.array([[ 0 for x in range(N)] for y in range(M)])

avg_intensity_reg = []
for i in range(M * N +1):
    avg_intensity_reg.append({'min':0,'max':0,'avg':0})


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Next, we shall initialise the Supergrid data structure, which is represented by a matrix of size twice that of the original image.
We call this matrix, crack_edge_data. We begin by storing the crack edges between two regions (pixel intensity differences), and recursively eliminate weak edges in the supergrid, between two regions as we merge them.
Can you write Python code for it?

crack_edge_data = np.zeros((2*M-1,2*N-1))
label_data = np.zeros((2*M-1,2*N-1))


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Next, we shall initialise the Supergrid data to hold the average intensities uniformly as per the region labels defined above.
We shall also keep track of the region labels in the Supergrid Data structure using the matrix label_data.
Can you write Python code for it?

for i in range(M):
    for j in range(N):
        crack_edge_data[2*i,2*j] = avg_intensity_reg[region_label[i,j]]['avg']
        label_data[2*i,2*j] = region_label[i,j]


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Next, we shall recursively iterate over each pixel and check if the label is not already set for the given pixel. If not set, we give the pixel a label and proceed to its neighbour.
Note: If a label is not set ie. it is 0, but the pixel satisfies homogenity criteria, corresponding to its neighbours, we set the label of the neighbours to this pixel.
For choosing the threshold, I iteratively generated the segmentation result for T1 in [1,2,3,4,5,6,7,8,9,10].
From the experimented results, we opined that, high values of T1(in this context) tend to be more lenient in terms of merging, and average out a lot of regions as 1.
 Output for T1 = 3
 Output for T1 = 6
 Output for T1 = 9
Lower values are more stringent, and as such create a lot of regions, which also comprise of noise. Hence, we had to choose the best threshold available for our use case.
Looking at the images above, T1=9 seems a good threshold for initial filtering, but its crack edge representation shall prove, why we have not chosen such a large threshold.
 Super grid representation for T1=9
 Super grid representation for T1=3
 Super grid representation for T1=5
After examining the crack edge data and the region segmentation results for different thresholds, we decided to choose T1 = 5.
After execution of the below code snippet, each pixel should have a label, and must belong to a region based on the average intensity of a region.
Can you write Python code for it?

def split_label_recursively(i,j,label,pixel):
    T1 = 5 # Tested for all values from 1 to 10, the higher the values the more lenient the merging, lower values create more regions and have high noise.
    if (i < 0 or i >= M or j < 0 or j >= N) or region_label[i,j] != 0 or original_img[i,j] < pixel - T1 or original_img[i,j] > pixel + T1:
        return

    pixel = original_img[i,j]

    if avg_intensity_reg[label]['min'] > pixel:
        avg_intensity_reg[label]['min'] = pixel
    elif avg_intensity_reg[label]['max'] < pixel:
        avg_intensity_reg[label]['max'] = pixel

    region_label[i,j] = label
    if adj_values[label]['left'] > i:
        adj_values[label]['left'] = i
    elif adj_values[label]['right'] < i:
        adj_values[label]['right'] = i
    if adj_values[label]['top'] > j:
        adj_values[label]['top'] = j
    elif adj_values[label]['down'] < j:
        adj_values[label]['down'] = j
    split_label_recursively(i-1,j+1,label,pixel)
    split_label_recursively(i,j+1,label,pixel)
    split_label_recursively(i+1,j+1,label,pixel)
    split_label_recursively(i+1,j,label,pixel)
    split_label_recursively(i+1,j-1,label,pixel)

label = 0
for i in range(M):
    for j in range(N):
        if region_label[i,j] == 0:
            label = label + 1
            pixel = original_img[i,j]
            adj_values[label]['left'] = i
            adj_values[label]['right'] = i
            adj_values[label]['top'] = j
            adj_values[label]['down'] = j
            split_label_recursively(i,j,label,pixel)
fin_label = label


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
The threshold above can be set by tuning for particular value which helps us obtain ideal segmentation. I experimented over various values of threshold between 0.1 to 0.9, and set this threshold by observation.
Next we overlay the obtained region edges over the original image, to view the various segmented regions in the original image provided.
Can you write Python code for it?

def superimpose_edges():
    for i in range(len(crack_edge_data)):
        for j in range(len(crack_edge_data[0])):
            if crack_edge_data[i,j] == 0:
                original_img[int((i+1)/2),int((j+1)/2)] = 255

common_perimeter, region_perimeter = update_perimeters()
elim_weak_edges(common_perimeter, region_perimeter)

superimpose_edges()


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
Thus we have made a first pass through the image pixels, and split the image into different simple regions, based on the homogenous intensity criteria for a given region.
Next let us calculate the average pixel intensity for each region, which shall be stored across every pixel in a region.
Can you write Python code for it?

for i in range(len(avg_intensity_reg)):
    avg_intensity_reg[i]['avg'] = (avg_intensity_reg[i]['max'] + avg_intensity_reg[i]['min'])/2


I am working on the file "RegionMerging-BoundaryMelting/RegionMergingByBoundaryMelting.ipynb".
The first task is:
We have defined the function for merging two regions, but when do we decide if we have to merge two adjacent regions?
Common boundaries of adjacent regions R1 and R2 are removed if:
            W/min(l1,l2) ≥ threshold

where, W is the the number of weak edges in common boundary and l1, l2 are the perimeters of R1 and R2 respectively.
We use this criteria for removing weak edges between two adjacent regions, and the threshold needs to be fixed by tuning.
In the function below, we have implemented a function to eliminate weak edges from adjacent regions R1 and R2, if the above criterion is satisfied.
Can you write Python code for it?

def elim_weak_edges(common_perimeter, region_perimeter):
    edge_rem_threshold = 0.8
    for i in range(len(crack_edge_data)):
        for j in range(len(crack_edge_data[0])):
            R1 = 0
            R2 = 0
            if crack_edge_data[i,j] == 0:
                [R1,R2] = adjacent_reg(i,j)
                if R1 != 0 and R2 != 0:
                    W = common_perimeter[int(R1),int(R2)]
                    L1 = region_perimeter[int(R1)]
                    L2 = region_perimeter[int(R2)]
                    if L1 <= L2:
                        min_perimeter = L1
                    else:
                        min_perimeter = L2
                    score = W/min_perimeter
                    if score >= edge_rem_threshold:
                        merge_reg(R1,R2)


I am working on the file "Regresión Lineal.ipynb".
The first task is:
A partir de los valores anteriores podemos construir la siguiente tabla resumen:

   Coeficiente
   Error Estándar
   estadístico t de Student
   valor - p

   Pendiente
   0.500
   0.117
   4.241
   0.002

En la tabla anterior vemos que el valor-p está lejos de y es inferior a, por ejemplo, 0.05 que sería lo que indicaría un intervalo de confianza del 95%. El intervalo de confianza al 95% de la pendiente que calculamos unas cuantas líneas más arriba indicaba el siguiente rango $[0.233, 0.767]$ por lo que parece que podríamos rechazar la hipótesis nula y aceptar la hipótesis alternativa de que existe relación entre $X$ e $Y$ con cierta confianza.

A pesar de lo dicho anteriormente, tened en cuenta que las pruebas de hipótesis deberían guiar nuestra toma de decisiones, pero no hay que tomarlos de forma estricta ni confiar ciegamente en los mismos si tenemos ciertas incertidumbres (pocos datos, modelo poco preciso,...).

Estimando la precisión del modelo (bondad del ajuste o goodness of fit)
La calidad de un ajuste por regresión lineal se realiza usando, normalmente, dos cantidades, el $RSE$ (Residual Standard Error o error estándar de los residuos, visto brevemente más arriba) y el estadístico $R^2$.
RSE
El $RSE$ es una estimación de la desviación estándar del término de error, $\epsilon$. En otras palabras, es la cantidad promedio que la respuesta se desviará de la línea real de regresión. Como vimos anteriormente, se calcula usando la siguiente fórmula:
$$RSE = \sqrt{RSS / (n - 2)}$$
donde RSS (Residual Sum of Squares) se expresa como $RSS = \sum_{i=1}^{n}{r_i}^2 = \sum_{i=1}^{n}({y_{i} - \hat{y_{i}}})^2$.
En el ejemplo anterior hemos visto que el $RSE$ valía $1.24$, esto indica que cualquier predicción que hagamos de $Y$ basándonos en $X$ tendrá un error de $1.24$ unidades. Este...
Can you write Python code for it?

# Calculamos ls suma total de los cuadrados
TSS = np.sum((Y - Y.mean())**2)
# Calculamos los residuos
residuos = Y - (ahat * X + bhat)
# Calculamos la suma cuadrática de los residuos (RSS, residual sum of squares)
RSS = np.sum(residuos**2)

R2 = 1 - RSS/TSS

print(R2)


I am working on the file "Regresión Lineal.ipynb".
The first task is:
A tener en cuenta antes de leer nada de lo que viene a continuación

Statisticians, like artists, have the bad habit of falling in love with their models
-- <cite>George Box</cite>

Regresión lineal simple con Python, versión TL;DR.
O sí, vamos a repetir la versión corta ya que, si eso es lo que buscas, así no tienes que seguir perdiendo el tiempo buscando más. Vamos a ver como se haría con numpy en unas pocas líneas.
Can you write Python code for it?

# X e Y serán el primer conjunto del cuarteto de Anscombe
# (https://en.wikipedia.org/wiki/Anscombe%27s_quartet)
X = np.array([10.0, 8.0, 13.0, 9.0, 11.0, 14.0, 6.0, 4.0, 12.0, 7.0, 5.0])
Y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])

# Calculamos los coeficientes del ajuste (a X + b)
a, b = np.polyfit(X, Y, 1)
# Calculamos el coeficiente de correlación
r = np.corrcoef(X, Y)

# Dibujamos los datos para poder visualizarlos y ver si sería lógico
# considerar el ajuste usando un modelo lineal
plt.plot(X, Y, 'o')
plt.xlim(np.min(X) -1, np.max(X) +1)
plt.ylim(np.min(Y) -1, np.max(Y) +1)
plt.plot(X, a * X + b)
plt.text(4, 10, 'r = {0:2.3f}'.format(r[0,1]))
plt.text(4, 9, 'Y = {0:2.3f} X + {1:2.3f}'.format(a, b))


I am working on the file "Regresión Lineal.ipynb".
The first task is:
En el anterior gráfico hemos mostrado las rectas teniendo en cuenta todos los extremos del rango tanto en $\hat{a}$ como en $\hat{b}$.
Test de hipótesis
Los errores estándar se pueden usar, también, para comprobar hipótesis en los coeficientes. La prueba más evidente que se nos puede ocurrir sería conocer si existe relación entre $X$ e $Y$.. Vamos a montar nuestra prueba.

La hipótesis nula, $H_{0}$, sería considerar que no hay relación entre $X$ e $Y$.
Mientras que la hipótesis alternativa, $H_{a}$, sería considerar que existe esa relación.

Matemáticamente, habría que comprobar si la pendiente de la recta es nula y por tanto la relación se queda en $Y = b + \epsilon$. Por tanto, la prueba se podría representar como:
$$H_{0}: a = 0$$
$$H_{a}: a \neq 0$$
Para comprobar la hipótesis nula tendríamos que determinar que nuestra estimación de la pendiente, $\hat{a}$, está lo sucientemente lejos de cero de forma que podemos estar seguros que $a$ no es cero. Pero, ¿cuán lejos sería suficiente? Esto, por supuesto, depende de la precisión en el cálculo de $\hat{a}$ que dependerá del error estándar, $SE(\hat{a})$. Si el error estándar es pequeño, incluso valores pequeños de la estimación de la pendiente, $\hat{a}$ mostrarían evidencias de que existe relación entre $X$ e $Y$. Sin embargo, si el error estándar es grande, necesitariamos valores grandes de $\hat{a}$ para poder asegurar que existe tal relación entre $X$ e $Y$. En la práctica, se calcula un estadístico t de Student con la siguiente forma:
$$t = \frac{\hat{a} - 0}{SE(\hat{a})}$$
El valor cero en la anterior ecuación corresponde al valor que consideremos en la hipótesis, en este caso estamos comprobando que no haya relación entre las variables $X$ e $Y$ y eso se cumple considerando que la pendiente es igual a $0$, i.e. $a = 0$.
La ecuación anterior indica el número de desviaciones estándar que nuestra estimación de la pendiente está alejada de, en este caso, $0$. El valor crítico para considerar que aceptamos o rechazamos la hipótesis vendrá dado por una...
Can you write Python code for it?

# X e Y serán el primer conjunto del cuarteto de Anscombe
# (https://en.wikipedia.org/wiki/Anscombe%27s_quartet)
X = np.array([10.0, 8.0, 13.0, 9.0, 11.0, 14.0, 6.0, 4.0, 12.0, 7.0, 5.0])
Y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])

# Calculamos los parámetros del ajuste, a y b, ax+b
ahat, bhat = np.polyfit(X, Y, deg = 1)
# Calculamos los residuos
residuos = Y - (ahat * X + bhat)
# Calculamos la suma cuadrática de los residuos (RSS, residual sum of squares)
RSS = np.sum(residuos**2)
# Calculamos el error estándar de los residuos (RSE, residual standard error)
RSE = np.sqrt(RSS/(len(X)-2))
# Error estándar de la estimación de la pendiente
SEa = np.sqrt(RSE**2 / np.sum((X - X.mean())**2))
# Calculamos el estadístico t
t = ahat / SEa
# Calculamos el p-valor
# Aquí estamos usando scipy.stats
# donde metemos los grados de libertad (len(X) - 2)
# y calculamos la CDF del valor t
# Como calculamos la CDF y solo obtenemos una de las
# colas lo multiplicamos por dos para considerar ambos extremos
p = 2 * (1 - stats.t(len(X) - 2).cdf(t))

print('Valor de la pendiente estimada, a = ', ahat)
print('RSS = ', RSS)
print('RSE = ', RSE)
print('Error estándar = ', SEa)
print('Valor de t = ', t)
print('Valor de p = ', p)


I am working on the file "Regresión Lineal.ipynb".
The first task is:
Podemos ver como el valor mínimo para los mínimos cuadrados se encuentra en ese punto si dibujamos la suma de residuos al cuadrado (RSS) en función de $a$ y $b$. En el siguiente gráfico, si se ejecuta de forma interactiva, usando interact, veremos que si modificamos el valor de la pendiente y de la ordenada en el origen el valor de RSS será superior al optimo excepto cuando $\hat{a} = 0.5, \hat{b} = 3$.
Can you write Python code for it?

a = np.linspace(0.3,0.7,100)
b = np.linspace(0,6,100)
RSS = np.empty((len(a), len(b)))
for i, ai in enumerate(a):
    for j, bj in enumerate(b):
        RSS[i, j] = np.sum((Y - bj - ai * X)**2)

def plotea(ai, bi):
    xx, yy = np.meshgrid(b, a)
    levels = np.array([10,20,30,50,80,120,180,250,400,1000,2000])
    colors = 1 - levels / levels.max()
    plt.figure(figsize = (15,7))
    plt.subplot(121)
    plt.plot(X, Y, 'o')
    plt.plot([X.min(), X.max()],
             [X.min() * ai + bi, X.max() * ai + bi])
    plt.text(5, 12, 'RSS={:5.2f}'.format(np.sum((Y - bi - ai * X)**2)))
    plt.ylim(0,14)
    plt.subplot(122)
    plt.contourf(yy, xx, RSS, levels = levels, colors = colors.astype('str'))
    CS = plt.contour(yy, xx, RSS, levels = levels, colors = 'k')
    plt.clabel(CS, inline=1, fontsize=10)
    plt.scatter(ahat, bhat, color = 'y')
    plt.scatter(ai, bi, color = 'g', s = 200, marker = 's')
    plt.grid()

interact(plotea, ai = [0.30, 0.70], bi = [0.0, 6.0])


I am working on the file "Regresión Lineal.ipynb".
The first task is:
Regresión lineal simple, versión un poco más larga...
Como ya hemos comentado, con la regresión lineal simple lo que buscamos es la relación entre un predictor, X, y un predictando, Y, asumiendo que la relación entre ambas sería aproximadamente lineal. La relación la podemos escribir así:
$$Y = a X + b$$
donde $a$ y $b$ son dos constantes (coeficientes o parámetros del modelo lineal) que definen, respectivamente, la pendiente y el término independiente (u ordenada en el origen o intercepto). Una vez que conocemos $\hat{a}$ y $\hat{b}$ (*), obtenidos a partir de los datos de entrenamiento (conjunto de datos que usamos para obtener los parámetros del modelo) podemos predecir nuevos valores y basándonos en valores x.

(*) Todos los valores estimados los pondremos con una caperuza (acento circunflexo, sombrero,...) para distinguirlos de valores teóricos que, en muchos casos, desconoceremos.

$$\hat{y} = \hat{a} x + \hat{b}$$
Para conocer $\hat{a}$ y $\hat{b}$ usamos los datos disponibles, como hemos comentado anteriormente. Tenemos los siguientes pares de datos:
$$(x_{1}, y_{1}), (x_{2}, y_{2}), (x_{3}, y_{3}),..., (x_{n}, y_{n})$$
que representan medidas de X e Y. Lo que trataremos de hacer con estos datos será obtener los parámetros del ajuste lineal, $\hat{a}$ y $\hat{b}$, de tal forma que el modelo lineal se ajuste bien a los datos, es decir, queremos encontrar una recta con una pendiente y una ordenada en el origen que minimice la distancia a todos los puntos $(x_{i}, y_{i})$
Existen varias formas de medir esta proximidad y, entre ellos, el más usado es el ajuste por mínimos cuadrados (least squares).
Ajuste por mínimos cuadrados
El método de mínimos cuadrados encontrará el valor óptimo cuando la suma de los residuos al cuadrado, $RSS=\sum_{i=1}^{n}{r_i}^2$ (RSS, Residual Sum of Squares), sea mínima.
El residuo se define como la diferencia entre el valor actual de la variable dependiente y el valor predicho por el modelo, $r_{i} = y_{i} - \hat{y}_{i}$. De esta forma...
Can you write Python code for it?

# Los datos de partida
X = np.array([10.0, 8.0, 13.0, 9.0, 11.0, 14.0, 6.0, 4.0, 12.0, 7.0, 5.0])
Y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])

# Primero calculamos la media de X e Y
xmean = X.mean()
ymean = Y.mean()

# Como b depende de a primero hemos de obtener a
ahat = np.sum((X - xmean) * (Y - ymean)) / np.sum((X - xmean)**2)
bhat = ymean - ahat * xmean

# Calculamos los valores usando numpy.polyfit
a, b = np.polyfit(X, Y, 1)

# Mostramos resultados
print('ajuste paso a paso, a, b = ', ahat, bhat)
print('ajuste con polyfit, a, b = ', a, b)
print(np.testing.assert_almost_equal(ahat, a, decimal = 7),
      np.testing.assert_almost_equal(bhat, b, decimal = 7))


I am working on the file "Regresión Lineal.ipynb".
The first task is:
Vaya, parece que los cuatro conjuntos de datos dan resultados similares para diferentes estadísticos. Los coeficientes del modelo son similares, los errores estándar también e incluso los intervalos de confianza salen parecidos.
¿Podemos afirmar categóricamente que el ajuste lineal es el más adecuado para todos los casos? Un primer paso debería ser el análisis exploratorio de los datos:
Can you write Python code for it?

plt.figure(figsize = (10,10))
i = 1
for X, Y in zip(x, y):
    result = sm.OLS(Y, sm.add_constant(X)).fit()
    print(result.summary())
    plt.subplot(2, 2, i)
    plt.plot(X, Y, 'o')
    i += 1


I am working on the file "Regresión Lineal.ipynb".
The first task is:
Vemos que $X$ sería capaz de explicar 2/3 de la variabilidad en $Y$.
¿Cuándo será bueno el valor de $R^2$? Normalmente, querremos que el valor sea lo más cercano posible a 1 pero también dependerá del problema concreto para decidir si un valor más bajo lo consideramos como adecuado.
$R^2$ es una medida de la relación lineal entre $X$ e $Y$. El coeficiente de correlación, definido como:
$$Cor(X,Y) = \frac{\sum_{i=1}^{n}({x_{i} - \bar{x}})({y_{i} - \bar{y}})}{\sqrt{\sum_{i=1}^{n}({x_{i} - \bar{x}})^2}\sqrt{\sum_{i=1}^{n}({y_{i} - \bar{y}})^2}}$$
también es una medida de la relación lineal entre $X$ e $Y$. Podríamos usar $r = Cor(X,Y)$ como medida de la bondad del ajuste en lugar de $R^2$. De hecho, en el caso concreto de la regresión lineal simple, $r^2 = R^2$. En el caso de la regresión lineal múltiple no podemos extender este hecho ya que la correlación mide la relación entre una variable y una respuesta pero no entre múltiples variables y una respuesta. El trabajo de obtener la relación entre varias variables corresponderá a $R^2$ en el caso de la regresión lineal múltiple.
Comprobemos que $r^2 = R^2$:
Can you write Python code for it?

r2 = np.corrcoef(X, Y)[0,1]**2
print(r2, R2)
print(np.testing.assert_almost_equal(r2, R2, decimal = 7))


I am working on the file "Regresión Lineal.ipynb".
The first task is:
Y usemos statsmodels para obtener más información del modelo, a parte del valor de $R^2$
Can you write Python code for it?

for X, Y in zip(x, y):
    result = sm.OLS(Y, sm.add_constant(X)).fit()
    print(result.summary().tables[1])
    print(result.rsquared)
    print('\n'*3)


I am working on the file "Regresión Lineal.ipynb".
The first task is:
¿Es nuestro modelo bueno?
Recordad la cita que hemos visto más arriba. No os enamoréis del modelo.
Regresemos de nuevo al cuarteto de Anscombe donde cuatro grupos de datos presentan el mismo valor de $R^2$ y los mismos parámetros de la recta de ajuste.
Can you write Python code for it?


I am working on the file "Regression Based Machine Learning for Algorithmic Trading/Linear Regression.ipynb".
The first task is:
Following the steps prescribed by Jake Vanderplas in his awesome text Python Data Science Handbook. He has kindly provided all his codes on github as well.
Step 1. Choose a class of model.
In this case we are using linear regression
Can you write Python code for it?

from sklearn.linear_model import LinearRegression


I am working on the file "Regression Based Machine Learning for Algorithmic Trading/Linear Regression.ipynb".
The first task is:
If you are statistically trained, you would normally dig into other information such as normality of the residuals and check for autocorrelation etc. You may also want to evaluation the parameters as well. Those are valid statistical modelling questions.
Machine Learning focus is on prediction. You will not find these information with the scikit-learn package. Do take note of this key difference between statistics and machine learning.
Step 5. Predict labels for unknown data
Can you write Python code for it?

x_test = np.linspace(0, 1)
x_test

y_pred = model.predict(x_test.reshape(-1,1))

plt.scatter(X, y)
plt.plot(x_test, y_pred);
plt.show()


I am working on the file "Regression Based Machine Learning for Algorithmic Trading/Linear Regression.ipynb".
The first task is:
Step 3. Arrange data into a features matrix and target vector
Can you write Python code for it?


I am working on the file "Regression Based Machine Learning for Algorithmic Trading/Linear Regression.ipynb".
The first task is:
Step 4. Fit the model to your data.
Can you write Python code for it?

model.fit(X, y)

model.coef_

model.intercept_


I am working on the file "Regression model - biophony.ipynb".
The first task is:
Load data
Can you write Python code for it?

data = pandas.read_csv(data_filepath)
data = data.loc[data.site<=30]


I am working on the file "Regression model - biophony.ipynb".
The first task is:
Model 1—time and site predictors
$$
\begin{align}
\text{level 1} \
y_{ts} \sim \mathcal{N}(\alpha_s + \beta_s T_t, \sigma_y^2) \
\text{level 2} \
\alpha_s \sim \mathcal{N}(\gamma_\alpha + \gamma_{\alpha s} L_s, \sigma_\alpha^2) \
\beta_s \sim \mathcal{N}(\gamma_\beta + \gamma_{\beta s} L_s, \sigma_\beta^2) \
\end{align}
$$
Can you write Python code for it?

site_predictors = [
#    'building_50m', 'pavement_50m', 'forest_50m', 'field_50m',
#    'building_100m', 'pavement_100m', 'forest_100m', 'field_100m',
#    'building_200m', 'pavement_200m', 'forest_200m', 'field_200m',
#    'building_500m', 'pavement_500m', 'forest_500m', 'field_500m',
    'd2n_50m',
    'd2n_100m',
    'd2n_200m', 'd2n_500m',
]
for predictor in site_predictors:
    with Model() as model_1:

        # intercept
        g_a = Normal('g_a', mu=0, tau=0.001)
        g_as = Normal('g_as', mu=0, tau=0.001)
        sigma_a = Uniform('sigma_a', lower=0, upper=100)
        tau_a = sigma_a**-2
        mu_a = g_a + (g_as * data_centered.groupby('site')[predictor].mean())
        a = Normal('a', mu=mu_a, tau=tau_a, shape=len(set(sites)))

        # slope
        g_b = Normal('g_b', mu=0, tau=0.001)
        g_bs = Normal('g_bs', mu=0, tau=0.001)
        sigma_b = Uniform('sigma_b', lower=0, upper=100)
        tau_b = sigma_b**-2
        mu_b = g_b + (g_bs * data_centered.groupby('site')[predictor].mean())
        b = Normal('b', mu=mu_b, tau=tau_b, shape=len(set(sites)))

        # model error (data-level)
        sigma_y = Uniform('sigma_y', lower=0, upper=100)
        tau_y = sigma_y**-2

        # expected values
        y_hat = a[sites] + (b[sites] * data_centered.week)

        # likelihood
        y_like = Normal('y_like', mu=y_hat, tau=tau_y, observed=data_centered.biophony)

        # simulated
        #y_sim = Normal('y_sim', mu=y_hat, tau=tau_y, shape=y_hat.tag.test_value.shape)

        # sample model
        backend = SQLite(path.join(trace_output_path, "model1_{}.sqlite".format(predictor)))
        model_1_samples = sample(draws=10000, step=NUTS(), random_seed=1, trace=backend)

len(model_1_samples['a'][:5000][:,1])

fig = pyplot.figure()

fig.set_figwidth(6.85)
fig.set_figheight(6.85/2)

ax_a = pyplot.subplot2grid((1, 2), (0, 0), rowspan=1, colspan=1)
ax_b = pyplot.subplot2grid((1, 2), (0, 1), rowspan=1, colspan=1, sharex=ax_a)

fig.subplots_adjust(left=0, bottom=0, right=1, top=1)

#...


I am working on the file "Regression model - biophony.ipynb".
The first task is:
Variable definitions
Can you write Python code for it?

data_filepath = "/Users/Jake/OneDrive/Documents/alpine soundscapes/data/dataset.csv"

trace_output_path = "/Users/Jake/OneDrive/Documents/alpine soundscapes/data/model traces/biophony"

seaborn_blue = seaborn.color_palette()[0]


I am working on the file "Regression model - biophony.ipynb".
The first task is:
create sites variable for PyMC3 models
Can you write Python code for it?

sites = numpy.copy(data_sorted.site.values) - 1


I am working on the file "Regression model - biophony.ipynb".
The first task is:
sort data by site and then by visit
Can you write Python code for it?

data_sorted = data.sort_values(by=['site', 'sound']).reset_index(drop=True)


I am working on the file "Regression model - sel.ipynb".
The first task is:
Model 0 - emtpy model
$$
\begin{align}
y_{ts} \sim \mathcal{N}(\alpha_s + \epsilon_t, \sigma_y^2) \
\alpha_s \sim \mathcal{N}(M + \epsilon_s, \sigma_\alpha^2) \
\end{align}
$$
Can you write Python code for it?

with Model() as model0:

    # Priors
    mu_grand = Normal('mu_grand', mu=0., tau=0.0001)
    sigma_a = Uniform('sigma_a', lower=0, upper=100)
    tau_a = sigma_a**-2

    # Random intercepts
    a = Normal('a', mu=mu_grand, tau=tau_a, shape=len(set(sites)))

    # Model error
    sigma_y = Uniform('sigma_y', lower=0, upper=100)
    tau_y = sigma_y**-2

    # Expected value
    y_hat = a[sites]

    # Data likelihood
    y_like = Normal('y_like', mu=y_hat, tau=tau_y, observed=data_centered.sel)

    # sample model
    backend = SQLite(path.join(trace_output_path, "model0.sqlite"))
    model0_samples = sample(draws=10000, step=NUTS(), random_seed=1, trace=backend)

fig, ax = pyplot.subplots()

# organize results
model0_data = pandas.DataFrame({'site': data_sorted.site.unique(),
                                 'site_name': data_sorted.site_name.unique()}).set_index('site')
model0_data['forest_200m'] = data.groupby('site')['forest_200m'].mean()
model0_data['quantiles'] = [stats.quantiles(model0_samples.a[:5000, i]) for i in range(len(set(sites)))]

# plot quantiles
for i, row in model0_data.sort_values(by='forest_200m').iterrows():
    x = row['forest_200m']
    ax.plot([x, x], [row['quantiles'][2.5], row['quantiles'][97.5]], color='black', linewidth=0.5)
    ax.plot([x, x], [row['quantiles'][25], row['quantiles'][75]], color='black', linewidth=1)
    ax.scatter([x], [row['quantiles'][50]], color='black', marker='o')

# format plot
l1 = ax.set_xlim([0, 100])
xl = ax.set_xlabel("forest land cover within 200 meters (percent area)")
yl = ax.set_ylabel("SEL (difference from grand mean)")

fig, ax = pyplot.subplots()

# organize results
model0_data = pandas.DataFrame({'site': data_sorted.site.unique(),
                                 'site_name': data_sorted.site_name.unique()}).set_index('site')
model0_data['d2n_200m'] = data.groupby('site')['d2n_200m'].mean()
model0_data['quantiles'] = [stats.quantiles(model0_samples.a[:5000, i]) for i in range(len(set(sites)))]

# plot quantiles
for i,...


I am working on the file "Regression model - sel.ipynb".
The first task is:
Model 1—time and site predictors
$$
\begin{align}
\text{level 1} \
y_{ts} \sim \mathcal{N}(\alpha_s + \beta_s T_t, \sigma_y^2) \
\text{level 2} \
\alpha_s \sim \mathcal{N}(\gamma_\alpha + \gamma_{\alpha s} L_s, \sigma_\alpha^2) \
\beta_s \sim \mathcal{N}(\gamma_\beta + \gamma_{\beta s} L_s, \sigma_\beta^2) \
\end{align}
$$
Can you write Python code for it?

site_predictors = [
#    'building_50m', 'pavement_50m', 'forest_50m', 'field_50m',
#    'building_100m', 'pavement_100m', 'forest_100m', 'field_100m',
#    'building_200m', 'pavement_200m', 'forest_200m', 'field_200m',
#    'building_500m', 'pavement_500m', 'forest_500m', 'field_500m',
    'd2n_50m', 'd2n_100m', 'd2n_200m', 'd2n_500m',
]
for predictor in site_predictors:
    with Model() as model_1:

        # intercept
        g_a = Normal('g_a', mu=0, tau=0.001)
        g_as = Normal('g_as', mu=0, tau=0.001)
        sigma_a = Uniform('sigma_a', lower=0, upper=100)
        tau_a = sigma_a**-2
        mu_a = g_a + (g_as * data_centered.groupby('site')[predictor].mean())
        a = Normal('a', mu=mu_a, tau=tau_a, shape=len(set(sites)))

        # slope
        g_b = Normal('g_b', mu=0, tau=0.001)
        g_bs = Normal('g_bs', mu=0, tau=0.001)
        sigma_b = Uniform('sigma_b', lower=0, upper=100)
        tau_b = sigma_b**-2
        mu_b = g_b + (g_bs * data_centered.groupby('site')[predictor].mean())
        b = Normal('b', mu=mu_b, tau=tau_b, shape=len(set(sites)))

        # model error (data-level)
        sigma_y = Uniform('sigma_y', lower=0, upper=100)
        tau_y = sigma_y**-2

        # expected values
        y_hat = a[sites] + (b[sites] * data_centered.week)

        # likelihood
        y_like = Normal('y_like', mu=y_hat, tau=tau_y, observed=data_centered.sel)

        # simulated
        #y_sim = Normal('y_sim', mu=y_hat, tau=tau_y, shape=y_hat.tag.test_value.shape)

        # sample model
        backend = SQLite(path.join(trace_output_path, "model1_{}.sqlite".format(predictor)))
        model_1_samples = sample(draws=10000, step=NUTS(), random_seed=1, trace=backend)

fig = pyplot.figure()

fig.set_figwidth(6.85)
fig.set_figheight(6.85/2)

ax_a = pyplot.subplot2grid((1, 2), (0, 0), rowspan=1, colspan=1)
ax_b = pyplot.subplot2grid((1, 2), (0, 1), rowspan=1, colspan=1, sharex=ax_a)

fig.subplots_adjust(left=0, bottom=0, right=1, top=1)

# organize results
model_1_data =...


I am working on the file "Regression model - sel.ipynb".
The first task is:
Model 2—environmental predictors
$$
\begin{align}
\text{level 1} \
y_{ts} \sim \mathcal{N}(\alpha_s + \beta_s T_t, \sigma_y^2) \
\text{level 2} \
\alpha_s \sim \mathcal{N}(\gamma_\alpha + \gamma_{\alpha s} L_s, \sigma_\alpha^2) \
\beta_s \sim \mathcal{N}(\gamma_\beta + \gamma_{\beta s} L_s, \sigma_\beta^2) \
\end{align}
$$
Can you write Python code for it?

measurement_predictors = [
    'temperature', 'wind_speed', 'precipitation', 'pressure',
]
for predictor in measurement_predictors:
    with Model() as model2a:

        # intercept
        g_a = Normal('g_a', mu=0, tau=0.001)
        g_as = Normal('g_as', mu=0, tau=0.001)
        sigma_a = Uniform('sigma_a', lower=0, upper=100)
        tau_a = sigma_a**-2
        mu_a = g_a + (g_as * data_centered.groupby('site')['forest_200m'].mean())
        a = Normal('a', mu=mu_a, tau=tau_a, shape=len(set(sites)))

        # time slope
        g_b = Normal('g_b', mu=0, tau=0.001)
        g_bs = Normal('g_bs', mu=0, tau=0.001)
        sigma_b = Uniform('sigma_b', lower=0, upper=100)
        tau_b = sigma_b**-2
        mu_b = g_b + (g_bs * data_centered.groupby('site')['forest_200m'].mean())
        b = Normal('b', mu=mu_b, tau=tau_b, shape=len(set(sites)))

        # temp slope
        #g_c = Normal('g_c', mu=0, tau=0.001)
        #g_cs = Normal('g_cs', mu=0, tau=0.001)
        #sigma_c = Uniform('sigma_c', lower=0, upper=100)
        #tau_c = sigma_c**-2
        #mu_c = g_c + (g_cs * data_centered.groupby('site')['forest_200m'].mean())
        #c = Normal('c', mu=mu_c, tau=tau_c, shape=len(set(sites)))
        c = Uniform('c', lower=-100, upper=100)

        # model error (data-level)
        sigma_y = Uniform('sigma_y', lower=0, upper=100)
        tau_y = sigma_y**-2

        # expected values
        y_hat = a[sites] + (b[sites] * data_centered.week) + (c * data_centered[predictor])

        # likelihood
        y_like = Normal('y_like', mu=y_hat, tau=tau_y, observed=data_centered.sel)

        # simulated
        #y_sim = Normal('y_sim', mu=y_hat, tau=tau_y, shape=y_hat.tag.test_value.shape)

        # sample model
        backend = SQLite(path.join(trace_output_path, "model2a_{0}.sqlite".format(predictor)))
        model_2_samples = sample(draws=10000, step=NUTS(), random_seed=1, trace=backend)

measurement_predictors = [
    'temperature', 'wind_speed', 'precipitation', 'pressure',
]
for predictor in...


I am working on the file "Regression model - sel.ipynb".
The first task is:
Plot settings
Can you write Python code for it?

from matplotlib import rcParams

rcParams['font.sans-serif']

rcParams['font.sans-serif'] = ['Helvetica',
                               'Arial',
                               'Bitstream Vera Sans',
                               'DejaVu Sans',
                               'Lucida Grande',
                               'Verdana',
                               'Geneva',
                               'Lucid',
                               'Avant Garde',
                               'sans-serif']


I am working on the file "Regression model - sel.ipynb".
The first task is:
Variable definitions
Can you write Python code for it?

data_filepath = "/Users/Jake/OneDrive/Documents/alpine soundscapes/data/dataset.csv"

trace_output_path = "/Users/Jake/OneDrive/Documents/alpine soundscapes/data/model traces/sel"

seaborn_blue = seaborn.color_palette()[0]


I am working on the file "Regression model - sel.ipynb".
The first task is:
transform variables (mean center)
Can you write Python code for it?

column_list = ['sel', 'sel_anthrophony', 'sel_biophony', 'biophony', 'week',
    'building_50m', 'pavement_50m', 'forest_50m', 'field_50m',
    'building_100m', 'pavement_100m', 'forest_100m', 'field_100m',
    'building_200m', 'pavement_200m', 'forest_200m', 'field_200m',
    'building_500m', 'pavement_500m', 'forest_500m', 'field_500m',
    'd2n_50m', 'd2n_100m', 'd2n_200m', 'd2n_500m',
    'temperature', 'wind_speed', 'pressure', 'bus_stop',
    'construction', 'crossing', 'cycleway', 'elevator', 'escape', 'footway',
    'living_street', 'motorway', 'motorway_link', 'path', 'pedestrian',
    'platform', 'primary_road', 'primary_link', 'proposed', 'residential',
    'rest_area', 'secondary', 'secondary_link', 'service', 'services',
    'steps', 'tertiary', 'tertiary_link', 'track', 'unclassified', 'combo']

data_centered = data_sorted.copy()
for column in column_list:
    data_centered[column] = data_sorted[column] - data_sorted[column].mean()


I am working on the file "Regression model - simulation.ipynb".
The first task is:
Model 1—time and site predictors
$$
\begin{align}
\text{level 1} \
y_{ts} \sim \mathcal{N}(\alpha_s + \beta_s T_t, \sigma_y^2) \
\text{level 2} \
\alpha_s \sim \mathcal{N}(\gamma_\alpha + \gamma_{\alpha s} L_s, \sigma_\alpha^2) \
\beta_s \sim \mathcal{N}(\gamma_\beta + \gamma_{\beta s} L_s, \sigma_\beta^2) \
\end{align}
$$
Can you write Python code for it?

with Model() as model_1:

    # intercept
    g_a = Normal('g_a', mu=0, tau=0.001)
    g_as = Normal('g_as', mu=0, tau=0.001)
    sigma_a = Uniform('sigma_a', lower=0, upper=100)
    tau_a = sigma_a**-2
    mu_a = g_a + (g_as * data_centered.groupby('site')['land_composite_200m'].mean())
    a = Normal('a', mu=mu_a, tau=tau_a, shape=len(set(sites)))

    # slope
    g_b = Normal('g_b', mu=0, tau=0.001)
    g_bs = Normal('g_bs', mu=0, tau=0.001)
    sigma_b = Uniform('sigma_b', lower=0, upper=100)
    tau_b = sigma_b**-2
    mu_b = g_b + (g_bs * data_centered.groupby('site')['land_composite_200m'].mean())
    b = Normal('b', mu=mu_b, tau=tau_b, shape=len(set(sites)))

    # model error (data-level)
    sigma_y = Uniform('sigma_y', lower=0, upper=100)
    tau_y = sigma_y**-2

    # expected values
    y_hat = a[sites] + (b[sites] * data_centered.week)

    # likelihood
    y_like = Normal('y_like', mu=y_hat, tau=tau_y, observed=data_centered.biophony)

    # simulated
    y_sim = Normal('y_sim', mu=y_hat, tau=tau_y, shape=y_hat.tag.test_value.shape)

    # sample model
    backend = SQLite(path.join(trace_output_path, "model3.sqlite"))
    model_1_samples = sample(draws=10000, step=NUTS(), random_seed=1, trace=backend)

# mean site-level parameter means
g_a = trace['g_a'].mean()
g_as = trace['g_as'].mean()
g_b = trace['g_b'].mean()
g_bs = trace['g_bs'].mean()

# calculate predicted biophony
y_hat = numpy.empty(len(data_centered))
for i, row in data_centered.iterrows():
    a = g_a + (g_as * row['land_composite_200m'])
    b = g_b + (g_bs * row['land_composite_200m'])
    y_hat[i] = a + (b * row['week'])

fig, ax = pyplot.subplots()
fig.set_figwidth(15)
fig.set_figheight(5)
#ax.scatter(data_centered['land_composite_200m'], y_hat, color='black')

# plot simulated
for s in numpy.random.randint(0, 4999, 20):
    y_sim = trace.y_sim[s, :]
    y_e = y_hat - y_sim
    ax.scatter(data_centered['biophony'], y_e, color=seaborn_blue, alpha=0.1)


I am working on the file "Regression model - simulation.ipynb".
The first task is:
Plot settings
Can you write Python code for it?


I am working on the file "Regression model - simulation.ipynb".
The first task is:
Variable definitions
Can you write Python code for it?


I am working on the file "Regression model - simulation.ipynb".
The first task is:
create composite land cover variable
Can you write Python code for it?

data_sorted['land_composite_50m'] = data_sorted.forest_50m - (data_sorted.building_50m + data_sorted.pavement_50m)
data_sorted['land_composite_100m'] = data_sorted.forest_100m - (data_sorted.building_100m + data_sorted.pavement_100m)
data_sorted['land_composite_200m'] = data_sorted.forest_200m - (data_sorted.building_200m + data_sorted.pavement_200m)
data_sorted['land_composite_500m'] = data_sorted.forest_500m - (data_sorted.building_500m + data_sorted.pavement_500m)


I am working on the file "Regression model - simulation.ipynb".
The first task is:
create sites variable for PyMC3 models
Can you write Python code for it?


I am working on the file "Regression model - simulation.ipynb".
The first task is:
sort data by site and then by visit
Can you write Python code for it?


I am working on the file "Regression model - simulation.ipynb".
The first task is:
transform variables (mean center)
Can you write Python code for it?

column_list = ['anthrophony', 'biophony', 'total', 'week',
    'building_50m', 'pavement_50m', 'forest_50m', 'field_50m',
    'building_100m', 'pavement_100m', 'forest_100m', 'field_100m',
    'building_200m', 'pavement_200m', 'forest_200m', 'field_200m',
    'building_500m', 'pavement_500m', 'forest_500m', 'field_500m',
    'land_composite_50m', 'land_composite_100m', 'land_composite_200m', 'land_composite_500m',
    'temperature', 'wind_speed', 'pressure', 'bus_stop',
    'construction', 'crossing', 'cycleway', 'elevator', 'escape', 'footway',
    'living_street', 'motorway', 'motorway_link', 'path', 'pedestrian',
    'platform', 'primary_road', 'primary_link', 'proposed', 'residential',
    'rest_area', 'secondary', 'secondary_link', 'service', 'services',
    'steps', 'tertiary', 'tertiary_link', 'track', 'unclassified', 'combo']

data_centered = data_sorted
for column in column_list:
    data_centered[column] = data_sorted[column] - data_sorted[column].mean()


I am working on the file "Regression.ipynb".
The first task is:
Ahora, graficamos la recta contra los datos de entrenamiento
Can you write Python code for it?

# data_1
plt.scatter(x_1, Y_1)
lower = min(x_1)
upper = max(x_1)
sample_x1 = np.linspace(lower, upper, num=len(x_1))
plt.plot(sample_x1, f1(sample_x1), color='r')


I am working on the file "Regression.ipynb".
The first task is:
As usual, let's create separate training and test data.
Can you write Python code for it?

# Shuffle the data, but make sure that the features and accompanying labels stay in sync.
np.random.seed(0)
shuffle = np.random.permutation(np.arange(X.shape[0]))
X, Y = X[shuffle], Y[shuffle]

# Split into train and test.
train_data, train_labels = X[:350], Y[:350]
test_data, test_labels = X[350:], Y[350:]


I am working on the file "Regression.ipynb".
The first task is:
Bayesian Regression
If we take a Bayesian approach to regression, then we write the posterior pdf for the model parameters as:
$$p(\theta|{x_i, y_i},I) \propto p({x_i,y_i} | \theta, I) \, p(\theta, I),$$
where $I$ describes the error on the dependent variable.
The likelihood for a single data point is given by
$$p(y_i|x_i,{\theta}, I) = e(y_i|y)$$
with $e(y_i|y)$ being the probability of getting $y_i$ given the true value of $y$ (i.e. the error distribution). If the error distribution is Gaussian then,
$$p(y_i|x_i,{\theta}, I) = {1 \over \sigma_i \sqrt{2\pi}} \, \exp{\left({-[y_i-f(x_i|{\theta})]^2 \over 2 \sigma_i^2}\right)}.$$
As discussed earlier in the course, when we add points we multiply the likelihood together to improve the constraints.
2-D Linear Regression
Let's start with the simplest case: a linear model with independent variable, $x$, and dependent variable, $y$:
$$y_i = \theta_0 + \theta_1 x_i + \epsilon_i,$$
where $\theta_0$ and $\theta_1$ are the coefficients of the model that we are trying to estimate and $\epsilon_i$ is an additive noise term.
If the uncertainties can be characterized by $\epsilon_i = \mathscr{N}(0,\sigma_i)$, then, following from above, the likelihood is
$$p({y_i}|{x_i},{\theta}, I) = \prod_{i=1}^N \left(\frac{-(y_i- (\theta_0 + \theta_1x_{i}))^2}{  2\sigma_i^2}\right).$$
For linear regression the log likelihood (assuming a flat/uninformative prior) is then
$$\ln \mathcal(L) \equiv \ln(p({\theta}|{x_i, y_i},I))  \propto \sum_{i=1}^N \left(\frac{-(y_i- (\theta_0 + \theta_1x_{i}))^2}{  2\sigma_i^2}\right).$$
Maximizing this expression is the same as minimizing the least squares.
For so-called "homoscedastic" errors (errors that are the same for all points, or rather that the error on the model is the same), this minimization yields to
$$\theta_1 = \frac{\sum_i^N x_i y_i - \bar{x}\bar{y}}{\sum_i^N(x_i-\overline{x})^2},$$
and
$$\theta_0 = \overline{y} - \theta_1\overline{x},$$
where $\overline{x}$ and $\overline{y}$ are the mean values.
The estimate of the variance and the...
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from scipy.stats import lognorm

from astroML.cosmology import Cosmology
from astroML.datasets import generate_mu_z
from astroML.linear_model import LinearRegression, PolynomialRegression, BasisFunctionRegression, NadarayaWatson

#------------------------------------------------------------
# Generate data: redshift, distance modulus and error on the distance modulus
z_sample, mu_sample, dmu = generate_mu_z(100, random_state=0)

cosmo = Cosmology()
z = np.linspace(0.01, 2, 1000) # "x" values
mu_true = np.asarray(map(cosmo.mu, z))  # Ground truth y values

n_constraints = 2

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(6, 6))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.1, top=0.95, hspace=0.05, wspace=0.05)

#fit data using the design matrix formalism
C = np.identity(len(z_sample))*(dmu*dmu)
M = np.column_stack((np.ones(len(z_sample)),z_sample))
A = np.dot(np.dot(M.transpose(),np.linalg.pinv(C)),M)
B = np.dot(np.dot(M.transpose(),np.linalg.pinv(C)),mu_sample)

theta = np.dot(np.linalg.pinv(A),B)
mu_out = theta[0] + theta[1]*z

#fit data using standard package
LRmodel = LinearRegression()
LRmodel.fit(z_sample[:, None], mu_sample, dmu)
mu_fit = LRmodel.predict(z[:, None])
mu_sample_fit = LRmodel.predict(z_sample[:, None])
chi2_dof = (np.sum(((mu_sample_fit - mu_sample)/dmu)**2)/(len(mu_sample) - n_constraints))

#plot the data
ax = fig.add_subplot(111)
ax.plot(z, mu_fit, '-k')
ax.plot(z, mu_true, '--', c='gray')
ax.errorbar(z_sample, mu_sample, dmu, fmt='.k', ecolor='gray', lw=1)

ax.text(0.5, 0.05, r"$\chi^2_{\rm dof} = %.2f$" % chi2_dof,
    ha='center', va='bottom', transform=ax.transAxes, fontsize=14)

ax.set_xlim(0.01, 1.8)
ax.set_ylim(36.01, 48)
ax.text(0.05, 0.95, 'Linear regression', ha='left', va='top',
                transform=ax.transAxes)

ax.set_ylabel(r'$\mu$')
ax.set_xlabel(r'$z$')

ax.plot(z, mu_out, '-k', color='red')

plt.show()


I am working on the file "Regression.ipynb".
The first task is:
Dataset
Cargamos los cuatro conjuntos de datos como arreglos numpy. Los datos reales corresponden a mediciones verificadas con algún instrumento. En la práctica esto no se realiza frecuentemente, por lo que es necesario usar estadística para realizar predicciones.
Obsérvese que a las X's se les añade una columna con unos dado que asumimos $x_0 \equiv 1$. Esto con el fin de hacer regresión lineal más adelante.
Can you write Python code for it?

X_1 = np.array([[1,x] for x, y in data_1])
Y_1 = np.array([y for x, y in data_1])

X_2 = np.array([[1,x] for x, y in data_2])
Y_2 = np.array([y for x, y in data_2])

X_3 = np.array([[1,x] for x, y in data_3])
Y_3 = np.array([y for x, y in data_3])

X_4 = np.array([[1,x] for x, y in data_4])
Y_4 = np.array([y for x, y in data_4])

X_real = np.array([[1,x] for x, y in data_real])
Y_real = np.array([y for x, y in data_real])


I am working on the file "Regression.ipynb".
The first task is:
Ejercicios:

Ajustar un regresor lineal para data_2, data_3 y data_4 usando el módulo correspondiente de Scikit-Learn.
Graficar las rectas contra sus respectivos conjuntos de datos. data_3 y data_4 deberán de aparecer en el mismo plano con distinción de color.

Regresión polinómica y regularizaciones
Can you write Python code for it?

# Polynomial regression
x_1 = np.expand_dims(x_1, axis=1)
poly = PolynomialFeatures(3)
features = poly.fit_transform(x_1)

features.shape

lin_reg = LinearRegression()
lin_reg.fit(features, Y_1)

sample_x1 = np.expand_dims(sample_x1, axis=1)

sample_x1_poly = poly.fit_transform(sample_x1)
y1_poly = lin_reg.predict(sample_x1_poly)
plt.scatter(x_1, Y_1)
plt.plot(sample_x1, y1_poly, color='g')


I am working on the file "Regression.ipynb".
The first task is:
Example

Fitting the model: estimating $w = [w_0, w_1]$

As there is noise, we can't hope to fit our model exactly

Define the error for each observation

$$e_i = y_i - f(x_i; w)$$
squared Euclidian norm. The constant $1/2$ is useful for a cosmetical simplification.

Total Error (sum over all data points)

$$
E(w) = \frac{1}{2} \sum_i (y_i - f(x_i; w))^2
$$

We can minimize the total error by adjusting $w_0$ and $w_1$

Visualization of the error surface
A good approach for low dimensional problems is the visualization of the error surface. We evaluate her exhaustively the error for each possible  setting of the parameter $w$.
Can you write Python code for it?

from itertools import product

BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

# Setup the vandermonde matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

left = -5
right = 15
bottom = -4
top = 6
step = 0.05
W0 = np.arange(left,right, step)
W1 = np.arange(bottom,top, step)

ErrSurf = np.zeros((len(W1),len(W0)))

for i,j in product(range(len(W1)), range(len(W0))):
    e = y - A*np.matrix([W0[j], W1[i]]).T
    ErrSurf[i,j] = e.T*e/2

plt.figure(figsize=(10,10))
plt.imshow(ErrSurf, interpolation='nearest',
           vmin=0, vmax=1000,origin='lower',
           extent=(left,right,bottom,top),cmap='jet')
plt.xlabel('w0')
plt.ylabel('w1')
plt.title('Error Surface')
plt.colorbar(orientation='horizontal')
plt.show()


I am working on the file "Regression.ipynb".
The first task is:
Example Dataset
Number of registered cars in Turkey, as a function of years.
|$i$|$y_i$|$x_i$|
|-------------|
|Index|Number of Cars (In Millions)|Year|
Can you write Python code for it?

%matplotlib inline

import scipy as sc
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pylab as plt

df_arac = pd.read_csv(u'data/arac.csv',sep=';')
df_arac[['Year','Car']]
#df_arac


I am working on the file "Regression.ipynb".
The first task is:
Fitting a polynomial
Often, fitting a line between inputs and outputs is not enough. A good candidate is fitting a higher order polynomial.
Parabola
\begin{eqnarray}
\left(
\begin{array}{c}
y_0 \ y_1 \ \vdots \ y_{N-1}
\end{array}
\right)
\approx
\left(
\begin{array}{ccc}
1 & x_0 & x_0^2 \ 1 & x_1 & x_1^2 \ \vdots \ 1 & x_{N-1} & x_{N-1}^2
\end{array}
\right)
\left(
\begin{array}{c}
 w_0 \ w_1  \ w_2
\end{array}
\right)
\end{eqnarray}
Polynomial of order $K$
\begin{eqnarray}
\left(
\begin{array}{c}
y_0 \ y_1 \ \vdots \ y_{N-1}
\end{array}
\right)
\approx
\left(
\begin{array}{ccccc}
1 & x_0 & x_0^2 & \dots & x_0^K \ 1 & x_1 & x_1^2 & \dots & x_1^K\ \vdots \ 1 & x_{N-1} & x_{N-1}^2 & \dots & x_{N-1}^K
\end{array}
\right)
\left(
\begin{array}{c}
 w_0 \ w_1  \ w_2 \ \vdots \ w_K
\end{array}
\right)
\end{eqnarray}
The model is identical
\begin{eqnarray}
y \approx A w
\end{eqnarray}

$A = A(x)$: Design Matrix
$w$: Model Parameters
$y$: Observations

Design matrices for polynomial fitting are also known as Vandermonde matrices.
Can you write Python code for it?

# Setup the vandermonde matrix
N = len(x)
degree = 9
#A = np.hstack((np.power(x,0), np.power(x,1), np.power(x,2)))
A = np.hstack((np.power(x,i) for i in range(degree+1)))
xx = np.matrix(np.linspace(np.asscalar(x[0])-10,np.asscalar(x[-1])+10,30)).T
A2 = np.hstack((np.power(xx,i) for i in range(degree+1)))

#plt.imshow(A, interpolation='nearest')
# Solve the least squares problem
w_ls,E,rank,sigma = np.linalg.lstsq(A, y)

f = A2*w_ls
plt.plot(x+BaseYear, y, 'o-')
plt.plot(xx+BaseYear, f, 'r')

plt.xlabel('Years')
plt.ylabel('Number of Cars')

plt.gca().set_ylim((-2,12))
plt.gca().set_xlim((1950,2025))

plt.show()

x = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5])
N = len(x)
x = x.reshape((N,1))
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]).reshape((N,1))
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74]).reshape((N,1))
#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73]).reshape((N,1))

def fit_and_plot_poly(degree):

    #A = np.hstack((np.power(x,0), np.power(x,1), np.power(x,2)))
    A = np.hstack((np.power(x,i) for i in range(degree+1)))
    # Setup the vandermonde matrix
    xx = np.matrix(np.linspace(np.asscalar(min(x))-1,np.asscalar(max(x))+1,300)).T
    A2 = np.hstack((np.power(xx,i) for i in range(degree+1)))

    #plt.imshow(A, interpolation='nearest')
    # Solve the least squares problem
    w_ls,E,rank,sigma = np.linalg.lstsq(A, y)
    f = A2*w_ls
    plt.plot(x, y, 'o')
    plt.plot(xx, f, 'r')

    plt.xlabel('x')
    plt.ylabel('y')

    plt.gca().set_ylim((0,20))
    #plt.gca().set_xlim((1950,2025))
    plt.title(E)
    plt.show()

fit_and_plot_poly(9)

interact(fit_and_plot_poly, degree=(0,12))


I am working on the file "Regression.ipynb".
The first task is:
Gráficos
Ejercicios:
- Graficar data_2.
- Graficar data_3 y data_4 en un mismo plano y con distinto color.
Can you write Python code for it?

# data_1
x_1 = np.array([x for _, x in X_1])
plt.scatter(x_1, Y_1)


I am working on the file "Regression.ipynb".
The first task is:
Least absolute shrinkage and selection (LASSO) regularization
An alternative to Ridge Regression is LASSO, which implies the following contraint
$$(Y - M \theta)^T(Y- M \theta) + \lambda |\theta|.$$
This is equivalent to least-squares minimization with
$$ |\theta | < s,$$
that is, the penalty is on the absolute values of the regression coefficients, which is also illustrated in Ivezic, Figure 8.3 as shown above.
It not only weights the regression coefficients, it also imposes sparsity on the regression
model (i.e. the penalty preferentially selects regions of likelihood space that coincide with one of the vertices within the region defined by the regularization).
This has the effect of setting one (or more) of the model attributes to zero.
Scikit-Learn's LASSO is implemented as follows:
Can you write Python code for it?

import numpy as np
from sklearn.linear_model import Lasso
XX = np.random.random((100,10))
yy = np.dot(XX, np.random.random(10))
model = Lasso(alpha = 0.05)
model.fit(XX,yy)
y_pred = model.predict(XX)


I am working on the file "Regression.ipynb".
The first task is:
Let's compare our results to sklearn's regression as well as the algebraic solution to "ordinary least squares". Try increasing the number of iterations above to see whether we get closer.
Can you write Python code for it?

def OLS(X, Y):
    # Add the intercept.
    X = np.c_[np.ones(X.shape[0]), X]

    # We use np.linalg.inv() to compute a matrix inverse.
    return np.dot(inv(np.dot(X.T, X)), np.dot(X.T, Y))

ols_solution = OLS(train_data[:,0:1], train_labels)

lr = LinearRegression(fit_intercept=True)
lr.fit(train_data[:,0:1], train_labels)

print 'Our estimated theta:     %.4f + %.4f*CRIM' %(theta[0], theta[1])
print 'OLS estimated theta:     %.4f + %.4f*CRIM' %(ols_solution[0], ols_solution[1])
print 'sklearn estimated theta: %.4f + %.4f*CRIM' %(lr.intercept_, lr.coef_[0])