I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Matrix_Computations.ipynb".
The first task is:
Basic matrix arithmetics like
Matrix addition
Matrix subtraction
Matrix multiplication
Scalar product
Cross product
and other operations on matrices

The arithemtic standard Operators
+
-
*
/
**
%

are applied on the elements, this means that the arrays have to have the same size.
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Matrix_Computations.ipynb".
The first task is:
If we want to perform matrix multiplication with two numpy arrays (ndarray), we have to use the dot product:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Matrix_Computations.ipynb".
The first task is:
In mathematics, the dot product is an algebraic operation that takes two coordinate vectors of equal size and returns a single number. The result is calculated by multiplying corresponding entries and adding up those products. The name "dot product" stems from the fact that the centered dot "·" is often used to designate this operation.
Can you write Python code for it?

x = np.array([1,2,3])
y = np.array([-7,8,9])
dot = np.dot(x,y)
np.dot(x,y)


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Matrix_Computations.ipynb".
The first task is:
In the following practical example, we come to talk about the sweet things of life.
Let's assume there are four people, and we call them Lucas, Mia, Leon and Hannah. Each of them has bought chocolates out of a choice of three. The brand are A, B and C, not very marketable, we have to admit. Lucas bought 100 g of brand A, 175 g of brand B and 210 of C. Mia choose 90 g of A, 160 g of B and 150 g of C. Leon bought 200 g of A, 50 of B and 100 g of C. Hannah apparently didn't like brand B, because she hadn't bought any of those. But she she seems to be a real fan of brand C, because she bought 310 g of them. Furthermore she bought 120 g of A.
So, what's the price in Euro of these chocolates: A costs 2.98 per 100 g, B costs 3.90 and C only 1.99 CAD.
If we have to calculate how much each of them had to pay, we can use Python, NumPy and Matrix multiplication:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Nonlinear_Equations.ipynb".
The first task is:
Check solution.
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Array Indexing and Slicing
Numpy ndarrays offer numbered indexing and slicing syntax that mirrors the syntax for Python lists:
Can you write Python code for it?

d_array = np.array([1,2,3,4,5,6])

d_array[2] # Get the item at index 2

d_array[4:]       # Get a slice from index 3 to the end

d_array[::-1]     # shortcut to reverse the array


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Array Math Operations
Numpy arrays can perform mathematical operations easily using math operators like +, -, / and *
Can you write Python code for it?

dd_array + 10    # Add 10 to each element

dd_array - 10    # Subtract 10 from each element

dd_array * 2      # Multiply each element by 2

dd_array ** 2      # Square each element


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
If an ndarray has more than one dimension, separate indexes for each dimension with a comma:
Can you write Python code for it?

# Create a new two dimensional array
dd_array = np.array([d_array, d_array + 10, d_array + 100])

print(dd_array)

# Get the element on row 2, and column 3
dd_array[2, 3]

# Slice elements starting at row 1, and column 3
dd_array[1:, 4:]

#Reverse the array in both dimensions (rotation)
dd_array[::-1, ::-1]


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Numpy also provides math functions for ndarrays such as:
Can you write Python code for it?

np.mean(dd_array) # The mean of all the elements in an array

np.std(dd_array)   #Get the standard deviation all the elements in an array

 np.sum(dd_array,
       axis=1)        # Get the row sums for the elements of an array

np.sum(dd_array,
       axis=0)        # Get the column sums

np.sqrt(dd_array) # Take the square root of each element in the array


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Numpy has several functions for creating arrays, such as:
np.identity(), to create a square 2d array with 1's across the diagonal and 0's everywhere else
Can you write Python code for it?

np.identity(n = 3)      # n is the size of the square 2-d array


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
One can also use the basic math operators on two arrays with the same shape. The basic math operators function in an element-wise fashion, returning an array with the same shape as the original.
Can you write Python code for it?

array3 = np.array([[1,2],[3,4]])

array3 + array3

array3 - array3

array3 * array3


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Reshaping Arrays
Rnp.reshape() reshapes an array into a new one with the same data but different structure:
Can you write Python code for it?

np.reshape(dd_array,        # Array to reshape
           newshape=(2,9))  # Dimensions of the new array


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
T np.dot() returns the dot product of two arrays.
Can you write Python code for it?

np.dot(dd_array[0,0:],  # Slice row 0
       dd_array[1,0:])  # Slice row 1


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
The output above shows that array2 is a 2-dimensional array with 2 rows and 5 columns. We can check the size (total number of items) of an array with the size attribute and the type of the data it contains with the dtype attribute:
Can you write Python code for it?

print("array2 has", array2.size ,"items of type", array2.dtype)             # Python 3

print ("array2 has %d items of type %s" % (array2.size, array2.dtype))      # Python 2 and 3


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Numpy_Arrays.ipynb".
The first task is:
Unravel a multi-dimensional into 1 dimension with np.ravel():
Can you write Python code for it?

np.ravel(dd_array,          # Array to reshape
         order='C')         # Unravel by rows

np.ravel(dd_array,
         order='F')         # Unravel by columns

dd_array.flatten()       #flatten a multi-dimensional array into 1 dimension and return a copy of the result

dd_array.T  #get the transpose

np.flipud(dd_array) #Flip an array vertically

np.fliplr(dd_array) #Flip an array horizontally

np.rot90(dd_array,    # Rotate the array 90 degrees counter-clockwise
         k=1)         # Number of 90 degree rotations

np.roll(dd_array,   # Shift elements in an array along a given dimension
        shift = 2,        # Shift elements 2 positions
        axis = 1)         # In each row

np.roll(dd_array,   #Leave the axis argument empty to shift across all dimensions
        shift = 2)

#Join arrays along an axis

array_to_join = np.array([[10,20,30],[40,50,60],[70,80,90]])

np.concatenate((dd_array,array_to_join),      # Arrays to join
               axis=1)                        # Axis to join upon


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
How do we interpret the coefficient of IsLarge? For a given amount of TV/Radio/Newspaper ad spending, a large market is associated with an average increase in Sales of 51.55 widgets (as compared to sales in a Small market).
If we reverse the 0/1 encoding and created the feature 'IsSmall', the coefficient would be the same in absolute value, but negative instead of positive. All that changes is the interpretation of the coefficient.
Handling Categorical Predictors with More than Two Categories
Let's create a new feature called Area, and randomly assign observations to be rural, suburban, or urban:
Can you write Python code for it?

# set a seed for reproducibility
np.random.seed(123456)

# assign roughly one third of observations to each group
nums = np.random.rand(len(data))
mask_suburban = (nums > 0.33) & (nums < 0.66)
mask_urban = nums > 0.66
data['Area'] = 'rural'
data.loc[mask_suburban, 'Area'] = 'suburban'
data.loc[mask_urban, 'Area'] = 'urban'
data.head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
How do we interpret the coefficients? For a given amount of Radio and Newspaper ad spending, an increase of a unit ($1000 dollars) in TV ad spending is associated with an increase in Sales of 45.765 widgets.
Other information is available in the model summary output:
Can you write Python code for it?

# print a summary of the fitted model
fitted_multi_model.summary()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
If the 95% confidence interval includes zero, the p-value for that coefficient will be greater than 0.05. If the 95% confidence interval does not include zero, the p-value will be less than 0.05. Thus, a p-value less than 0.05 is one way to decide whether there is likely a relationship between the feature and the response. (Again, using 0.05 as the cutoff is just a convention.)
In this case, the p-value for TV is far less than 0.05, and so we believe that there is a relationship between TV ads and Sales.
Note that we generally ignore the p-value for the intercept.
How Well Does the Model Fit the data?
The most common way to evaluate the overall fit of a linear model to the available data is by calculating the R-squared (a.k.a, "coefficient of determination") value.
R-squared has several interpretations:
(1) R-squared ×100 percent of the variation in the dependent variable ($y$) is reduced by taking into account predictor $x$
(2) R-squared is the proportion of variance in the observed data that is "explained" by the model.
R-squared is between 0 and 1, and, generally speaking, higher is considered to be better because more variance is accounted for ("explained") by the model.
Note, however, that R-squared does not indicate whether a regression model is actually good. You can have a low R-squared value for a good model, or a high R-squared value for a model that does not fit the data!
One should evaluate the adequacy of a model by looking at R-squared values as well as residual (i.e., observed value - fitted value) plots, other model statistics, and subject area knowledge.
The R-squared value for our simple linear regression model is:
Can you write Python code for it?

# print the R-squared value for the model
fitted_model.rsquared


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
Is that a "good" R-squared value? One cannot generally assess that. What a "good" R-squared value is depends on the domain and therefore R-squared is most useful as a tool for comparing different models.
Multiple Linear Regression
Simple linear regression can be extended to include multiple explanatory variables:
$y = \beta_0 + \beta_1x_1 + ... + \beta_nx_n$
Each $x$ represents a different predictor/feature, and each predictor has its own coefficient. In our case:
$y = \beta_0 + \beta_1 \times TV + \beta_2 \times Radio + \beta_3 \times Newspaper$
Let's use Statsmodels to estimate these coefficients:
Can you write Python code for it?

# create a model with all three features
multi_model = sf.ols(formula='Sales ~ TV + Radio + Newspaper', data=data)
fitted_multi_model = multi_model.fit()

# print the coefficients
print(fitted_multi_model.params)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
Plotting the Least Squares Line
Let's make predictions for the smallest and largest observed values of money spent on TV ads, and then use the predicted values to plot the least squares line:
Can you write Python code for it?

# create a DataFrame with the minimum and maximum values of TV ad money
New_TV_money = pd.DataFrame({'TV': [data.TV.min(), data.TV.max()]})
print(New_TV_money.head())

# make predictions for those x values and store them
sales_predictions = fitted_model.predict(New_TV_money)
print(sales_predictions)

# plot the observed data
data.plot(kind='scatter', x='TV', y='Sales')

# plot the least squares line
plt.plot(New_TV_money, sales_predictions, c='red', linewidth=2)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
Questions
How can the company selling the product decide on how to spend its advertising money in the future? We first need to answer the following question: "Based on this data, does there apear to be a relationship between ads and sales?"
If yes,
1. Which ad types contribute to sales?
2. How strong is the relationship between each ad type and sales?
4. What is the effect of each ad type of sales?
5. Given ad spending in a particular market, can sales be predicted?
We will use Linear Regression to try and asnwer these questions.
Simple Linear Regression
Simple linear regression is an approach for modeling the relatrionship between a dependent variable (a "response") and an explanatory variable, also known as a "predictor" or "feature". The relationship is modeled as a linear function $y = \beta_0 + \beta_1x$ whose parameters are estimated from the available data.
In the equation above:
- $y$ is called the response, regressand, endogenous variable, dependent variable, etc.
- $x$ is the feature, regressor, exogenous variable, explanatory variables, predictor, etc.
- $\beta_0$ is known as the intercept
- $\beta_1$ is the regression coefficient, effect, etc.
Together, $\beta_0$ and $\beta_1$ are called paramaters, model/regression coefficients, or effects. To create a model, we must discover/learn/estimate the values of these coefficients.
Estimating/Learning Model/Regression Coefficients
Regression coefficients are estimated using a variety of methods. The least squares method, which finds the line which minimizes the sum of squared residuals (or "sum of squared errors") is among the most oftenly used.
In the pictures below:
- The blue dots are the observed values of x and y.
- The red line is the least squares line.
- The residuals are the distances between the observed values and the least squares line.

$\beta_0$ is the intercept of the least squares line (the value of $y$ when $x$=0)
$\beta_1$ is the slope of the least squares line, i.e. the ratio of the vertical change (in $y$) and the horizontal change...
Can you write Python code for it?

import statsmodels.formula.api as sf

#create a model with Sales as dependent variable and TV as explanatory variable
model = sf.ols('Sales ~ TV', data)

#fit the model to the data
fitted_model = model.fit()

# print the coefficients
print(fitted_model.params)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
R-squared will always increase as you add more features to the model, even if they are unrelated to the response. Thus, selecting the model with the highest R-squared is not a reliable approach for choosing the best linear model.
There is alternative to R-squared called adjusted R-squared that penalizes model complexity (to control for overfitting), but this approach has its own set of issues.
Is there a better approach to feature selection? Cross-validation, which provides a more reliable estimate of out-of-sample error, and thus is better at choosing which model will better generalize to out-of-sample data. Cross-validation can be applied to any type of model, not just linear models.
Linear Regression in scikit-learn
The work done using Statsmodels can also be using scikit-learn:
Can you write Python code for it?

# create a DataFrame
feature_cols = ['TV', 'Radio', 'Newspaper']
X = data[feature_cols]
y = data.Sales

from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print(lm.intercept_)
print(lm.coef_)

# pair the feature names with the coefficients
print(zip(feature_cols, lm.coef_))

# predict for a new observation
lm.predict([[100, 25, 25]])

# calculate the R-squared
lm.score(X, y)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
Since we only have a single sample of data, and not the entire population the "true" value of the regression coefficient is either within this interval or it isn't, but there is no way to actually know.
We estimate the regression coefficient using the data we have, and then we characterize the uncertainty about that estimate by giving a confidence interval, an interval that will "probably" contain the  value coefficient. Note that there is no probability associated with the true value of the regression coefficient being in the given confidence interval!
Also note that using 95% confidence intervals is simply a convention. One can create 90% confidence intervals (narrower intervals), 99% confidence intervals (wider intervals), etc.
Hypothesis Testing and p-values
Closely related to confidence intervals is hypothesis testing. Generally speaking, you start with a null hypothesis and an alternative hypothesis (that is opposite the null). Then, you check whether the data supports rejecting the null hypothesis or failing to reject the null hypothesis.
(Note that "failing to reject" the null is not the same as "accepting" the null hypothesis. The alternative hypothesis may indeed be true, except that you just don't have enough data to show that.)
As it relates to model coefficients, here is the conventional hypothesis test:
- null hypothesis: There is no relationship between TV ads and Sales (and thus $\beta_1$ equals zero)
- alternative hypothesis: There is a relationship between TV ads and Sales (and thus $\beta_1$ is not equal to zero)
How do we test this hypothesis? Intuitively, we reject the null (and thus believe the alternative) if the 95% confidence interval does not include zero. Conversely, the p-value represents the probability that the coefficient is actually zero:
Can you write Python code for it?

# print the p-values for the model coefficients
fitted_model.pvalues


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
TV and Radio have significant p-values, whereas Newspaper does not. Thus we reject the null hypothesis for TV and Radio (that there is no association between those features and Sales), and fail to reject the null hypothesis for Newspaper.
TV and Radio ad spending are both positively associated with Sales, whereas Newspaper ad spending is slightly negatively associated with Sales.
This model has a higher R-squared (0.897) than the previous model, which means that this model provides a better fit to the data than a model that only includes TV.

Feature Selection
How do I decide which features to include in a linear model?
- Try different models and check whether the R-squared value goes up when you add new predictors.
What are the drawbacks to this approach?
- Linear models rely upon a lot of assumptions (such as the predictors/features being independent), and if those assumptions are violated (which they usually are), R-squared are less reliable.
- R-squared is susceptible to overfitting, and thus there is no guarantee that a model with a high R-squared value will generalize well to new data. For example:
Can you write Python code for it?

# only include TV and Radio in the model
model1 = sf.ols(formula='Sales ~ TV + Radio', data=data).fit()
print(model1.rsquared)

# add Newspaper to the model (which we believe has no association with Sales)
model2 = sf.ols(formula='Sales ~ TV + Radio + Newspaper', data=data).fit()
print(model2.rsquared)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
The multiple linear regression including the IsLarge predictor:
Can you write Python code for it?

# create X and y
feature_cols = ['TV', 'Radio', 'Newspaper', 'IsLarge']
X = data[feature_cols]
y = data.Sales

# instantiate, fit
lm = LinearRegression()
lm.fit(X, y)

# print coefficients
list(zip(feature_cols, lm.coef_))


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
The predicted Sales in that market are of 9.409444 * 1000 =~ 9409 widgets
Using Statsmodels:
Can you write Python code for it?

# create a DataFrame to use with the Statsmodels formula interface
New_TV_spending = pd.DataFrame({'TV': [50]})

#check the newly created DataFrame
New_TV_spending.head()

# use the model created above to predict the sales to be generated by the new TV ad money
sales = fitted_model.predict(New_TV_spending)
print(sales)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
There are 200 observations, corresponding to 200 markets.
We can try to discover if there is any relationship between the money spend on a specific type of ad, in a given market, and the sales in that market by plotting the sales figures against each category of advertising expenditure.
Can you write Python code for it?

fig, axs = plt.subplots(1, 3, sharey=True)
data.plot(kind='scatter', x='TV', y='Sales', ax=axs[0], figsize=(16, 8))
data.plot(kind='scatter', x='Radio', y='Sales', ax=axs[1])
data.plot(kind='scatter', x='Newspaper', y='Sales', ax=axs[2])


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Linear_Regression.ipynb".
The first task is:
When using scikit-learn, we need to represent all data numerically. For example, if the feature we want to represent has only two categories, we create a dummy variable that represents the categories as a binary value:
Can you write Python code for it?

# create a new Series called IsLarge
data['IsLarge'] = data.Size.map({'small':0, 'large':1})
data.head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Add a field FamilySize that aggregates the information in the fields indicating the presence of a partner (Parch) or a relative (SibSp).
Can you write Python code for it?

training_data['FamilySize'] = training_data['SibSp'] + training_data['Parch']
test_data['FamilySize'] = test_data['SibSp'] + test_data['Parch']

training_data.head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let us create a function that visualizes the accuracy of the models we are building. It plots as a continuous line the mean values of the scores of the chosen estimator for two data sets, and a coloured band around the mean line, i.e., the interval (mean - standard deviation, mean + standard deviation).
plot_learning_curve() uses in turn the function sklearn.learning_curve.learning_curve(), which determines cross-validated training and test scores for different training set sizes. An (optional) cross-validation generator splits the given dataset k times in training and test data. (The default is 3-fold cross validation.) Subsets of the training set with varying sizes will be used to train the estimator and a score for each training subset size and the test set will be computed. The scores are averaged over all k runs for each training subset size.
Can you write Python code for it?

def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None, n_jobs=1,\
                        train_sizes=np.linspace(.1, 1.0, 5), scoring='accuracy'):

    plt.figure(figsize=(10,6))
    plt.title(title)

    if ylim is not None:
        plt.ylim(*ylim)

    plt.xlabel("Training examples")
    plt.ylabel(scoring)

    train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, scoring=scoring, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)

    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\
                     train_scores_mean + train_scores_std, alpha=0.1, \
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")

    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g", label="Cross-validation score")
    plt.legend(loc="best")

    return plt

# import machine learning modules
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer, accuracy_score
from sklearn.ensemble import RandomForestClassifier
try:
    from sklearn.model_selection import train_test_split
except:
    from sklearn.cross_validation import train_test_split
try:
    from sklearn.model_selection import GridSearchCV
except:
    from sklearn.grid_search import GridSearchCV
try:
    from sklearn.model_selection import learning_curve
except:
    from sklearn.learning_curve import learning_curve


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let us fill in the missing values with the most common fare, $8.05
Can you write Python code for it?

test_data.set_value(test_data.Fare.isnull(), 'Fare', 8.05)
test_data.loc[test_data.Fare.isnull()]


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let us select just the features of interest. We are dropping features like Name, SibSp and Sex, whose information is either no longer needed or is accounted for in the columns that we have added.
Can you write Python code for it?

training_data.drop(labels=['PassengerId', 'Name', 'Cabin', 'Ticket', 'SibSp', 'Parch', 'Sex'], axis=1, inplace=True)
test_data.drop(labels=['Name', 'Cabin', 'Ticket', 'SibSp', 'Parch', 'Sex'], axis=1, inplace=True)

training_data.head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let us turn our attention to cross-validation. We are using the sklearn.model_selection.KFold(), a K-folds cross-validator which provides training and validation/testing indices to split data into training and validation (or testing) sets. KFold() splits a dataset into k consecutive folds with each fold used once as a validation while the k - 1 remaining folds form the training set.
Can you write Python code for it?

from sklearn.cross_validation import KFold

def run_kfold(clf):

    #run KFold with 10 folds instead of the default 3
    #on the 891 records in the training_data
    kf = KFold(891, n_folds=10)

    outcomes = []
    fold = 0

    for train_index, test_index in kf:
        fold += 1
        X_train, X_test = X.values[train_index], X.values[test_index]
        y_train, y_test = y.values[train_index], y.values[test_index]

        clf.fit(X_train, y_train)
        predictions = clf.predict(X_test)

        accuracy = accuracy_score(y_test, predictions)
        outcomes.append(accuracy)
        print("Fold {0} accuracy: {1}".format(fold, accuracy))

    mean_outcome = np.mean(outcomes)
    print("Mean Accuracy: {0}".format(mean_outcome))

run_kfold(clf)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let's look at the survival chances depending on the port of embarkation
Can you write Python code for it?

# plot
#sns.factorplot('Embarked','Survived', data=training_data,size=4,aspect=3)

fig, (axis1,axis2,axis3) = plt.subplots(1,3,figsize=(15,5))

sns.countplot(x='Embarked', data=training_data, ax=axis1)
sns.countplot(x='Survived', hue="Embarked", data=training_data, order=[1,0], ax=axis2)

# group by embarked, and get the mean for survived passengers for each value in Embarked
embark_perc = training_data[["Embarked", "Survived"]].groupby(['Embarked'],as_index=False).mean()
sns.barplot(x='Embarked', y='Survived', data=embark_perc,order=['S','C','Q'],ax=axis3)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Let's visualize a histogram of the fares paid by the 3rd class passengers who embarked in Southampton.
Can you write Python code for it?

fig = plt.figure(figsize=(8, 5))
ax = fig.add_subplot(111)

test_data[(test_data.Pclass==3)&(test_data.Embarked=='S')].Fare.hist(bins=100, ax=ax)
plt.xlabel('Fare')
plt.ylabel('Frequency')
plt.title('Histogram of Fare, Plcass 3 and Embarked S')

print ("The top 5 most common fares")
test_data[(test_data.Pclass==3)&(test_data.Embarked=='S')].Fare.value_counts().head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
Lets look at other variables that may indicate where passengers embarked the ship.
Can you write Python code for it?

training_data.loc[training_data.Ticket == '113572']

print( 'C == ' + str( len(training_data.loc[training_data.Pclass == 1].loc[training_data.Fare > 75].loc[training_data.Fare < 85].loc[training_data.Embarked == 'C']) ) )
print( 'S == ' + str( len(training_data.loc[training_data.Pclass == 1].loc[training_data.Fare > 75].loc[training_data.Fare < 85].loc[training_data.Embarked == 'S']) ) )

training_data = training_data.set_value(training_data.Embarked.isnull(), 'Embarked', 'C')

training_data.loc[training_data.Embarked.isnull()]


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
The <b>Survived</b> column is our target/dependent/reponse variable, 1 means the passenger survived, 0 means that the passenger died.
Several other variables describe the passengers:
- PassengerId: and id given to each traveler on the boat.
- Pclass: the passenger class. It has three possible values: 1,2,3.
- Name
- Sex
- Age
- SibSp: number of siblings and spouses traveling with the passenger
- Parch: number of parents and children traveling with the passenger
- The ticket number
- The ticket fare
- The cabin number
- The port of embarkation. It has three possible values S,C,Q. (C = Cherbourg; Q = Queenstown; S = Southampton)
Let us check which records in the training data are missing information for the Embarked field.
Can you write Python code for it?

# check the missing data for the Embarked field
training_data[training_data.Embarked.isnull()]


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
The gender of pasanger is an important factor in surviving the accident. So is a pasenger's age.  Let us introduce a new feature to take into account the gender and age of passengers.
Can you write Python code for it?

def get_person(passenger):
    age,sex = passenger
    return 'child' if age < 16 else sex

training_data['Person'] = training_data[['Age','Sex']].apply(get_person,axis=1)
test_data['Person']    = test_data[['Age','Sex']].apply(get_person,axis=1)

training_data.head()

training_data.info()
print("\n------------------------------------\n")
test_data.info()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
The names have a prefix that, in some cases, is indicative of the social status, which may have been be an important factor in surviving the accident.
Braund, Mr. Owen Harris
Heikkinen, Miss. Laina
Oliva y Ocana, Dona. Fermina
Peter, Master. Michael J
Extracting the passenger titles and storring them in an additional column called <b>Title</b>.
Can you write Python code for it?

Title_Dictionary = {
                    "Capt":       "Officer",
                    "Col":        "Officer",
                    "Major":      "Officer",
                    "Jonkheer":   "Nobel",
                    "Don":        "Nobel",
                    "Sir" :       "Nobel",
                    "Dr":         "Officer",
                    "Rev":        "Officer",
                    "the Countess":"Nobel",
                    "Dona":       "Nobel",
                    "Mme":        "Mrs",
                    "Mlle":       "Miss",
                    "Ms":         "Mrs",
                    "Mr" :        "Mr",
                    "Mrs" :       "Mrs",
                    "Miss" :      "Miss",
                    "Master" :    "Master",
                    "Lady" :      "Nobel"
                    }

training_data['Title'] = training_data['Name'].apply(lambda x: Title_Dictionary[x.split(',')[1].split('.')[0].strip()])
test_data['Title'] = test_data['Name'].apply(lambda x: Title_Dictionary[x.split(',')[1].split('.')[0].strip()])

training_data.head(10)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
We use the information available on passengers to build a statistical model for survivorship, which, given a "new" passenger will predict whether or not he survived. There is a wide variety of models to use, from logistic regression to decision trees and more sophisticated ones such as random forests.
First, let us use Pandas' get_dummies function to encode some of the features with discrete values, i.e., Person, Embarked, Title and Pclass and add those dummy variables as columns to the DataFrame object that stores the training data.
Can you write Python code for it?

dummies_person_train = pd.get_dummies(training_data['Person'],prefix='Person')
dummies_embarked_train = pd.get_dummies(training_data['Embarked'], prefix= 'Embarked')
dummies_title_train = pd.get_dummies(training_data['Title'], prefix= 'Title')
dummies_pclass_train = pd.get_dummies(training_data['Pclass'], prefix= 'Pclass')

training_data = pd.concat([training_data, dummies_person_train, dummies_embarked_train, dummies_title_train, dummies_pclass_train], axis=1)
training_data = training_data.drop(['Person','Embarked','Title', 'Pclass'], axis=1)

training_data.head()

dummies_person_test   = pd.get_dummies(test_data['Person'],prefix='Person')
dummies_embarked_test = pd.get_dummies(test_data['Embarked'], prefix= 'Embarked')
dummies_title_test    = pd.get_dummies(test_data['Title'], prefix= 'Title')
dummies_pclass_test   = pd.get_dummies(test_data['Pclass'], prefix= 'Pclass')

test_data = pd.concat([test_data, dummies_person_test, dummies_embarked_test, dummies_title_test, dummies_pclass_test], axis=1)
test_data = test_data.drop(['Person','Embarked','Title', 'Pclass'], axis=1)

test_data.head()


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Logistic_Regression-Titanic.ipynb".
The first task is:
We will use GridSearchCV, which exhaustively considers all parameter combinations, to find the best model for the data. A search consists of:
- an estimator (regressor or classifier such as RandomForestClassifier(), or LogisticRegression());
- a parameter space;
- a method for searching or sampling candidates;
- a cross-validation scheme;
- a score function, such as accurracy_score()
Can you write Python code for it?

# Choose the type of classifier.
clf = RandomForestClassifier()

# Choose some parameter combinations to try
parameters = {'n_estimators': [4, 6, 9],
              'max_features': ['log2', 'sqrt','auto'],
              'criterion': ['entropy', 'gini'],
              'max_depth': [2, 3, 5, 10],
              'min_samples_split': [2, 3, 5],
              'min_samples_leaf': [1,5,8]
             }

# make_scorer returns a callable object that scores an estimator’s output.
#We are using accuracy_score for comparing different parameter combinations.
acc_scorer = make_scorer(accuracy_score)

# Run the grid search for the Random Forest classifier
grid_obj = GridSearchCV(clf, parameters, scoring=acc_scorer)
grid_obj = grid_obj.fit(X_train, y_train)

# Set our classifier, clf, to the have the best combination of parameters
clf = grid_obj.best_estimator_

# Fit the selected classifier to the training data
clf.fit(X_train, y_train)

predictions = clf.predict(X_test)
print(accuracy_score(y_test, predictions))
plot_learning_curve(clf, 'Random Forest', X, y, cv=4);


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Preparing Numeric Data.ipynb".
The first task is:
A positive correlation implies that when one variable goes up the other tends to go up as well, while negative correlations indicate an inverse relationship.
In the mtcar dataset, the number of cylinders a car has (cyl) and its weight (wt) have fairly strong negative correlations to gas mileage (mpg.), i.e., heavier cars and cars with more cylinders tend to get lower gas mileage.
A scatter plot matrix can help  visualize this.  pandas' scatter_matrix() function accomplishes this:
Can you write Python code for it?

from pandas.tools.plotting import scatter_matrix

scatter_matrix(mtcars.ix[:,0:6],   # Make a scatter matrix of 6 columns
               figsize=(10, 10),   # Set plot size
               diagonal='kde')    # Show distribution estimates on diagonal


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Preparing Numeric Data.ipynb".
The first task is:
Highly Correlated Variables
In predictive modeling, each variable used to construct a model would ideally represent some unique feature of the data. In reality, variables often exhibit collinearity, and variables with strong correlations can interfere with the modeling process. We can check the pairwise correlations between numeric variables using the df.corr() function:
Can you write Python code for it?

mtcars.ix[:,0:6].corr()   # Check the pairwise correlations of 6 variables


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Preparing Numeric Data.ipynb".
The first task is:
Note that the values are not exactly the same as those calculated "manually", likely due to scikit-learn's implementation of centering and scaling.
Dealing With Skewed Data
The distribution of the data can have a significant impact on analysis and modeling, as many techniques assume, or require that the data follows a particular distribution, e.g., Gaussian. Some data sets exhibit significant asymmetry (skewness). To illustrate, let's generate a few distributions. Let us look at a few examples.
Can you write Python code for it?

normally_distributed = np.random.normal(size=10000)  # Generate normal data*

normally_distributed = pd.DataFrame(normally_distributed) # Convert to DF

normally_distributed.hist(figsize=(8,8),                  # Plot histogram
                          bins=30)

skewed = np.random.exponential(scale=2,      # Generate skewed data
                               size= 10000)

skewed = pd.DataFrame(skewed)                # Convert to DF

skewed.hist(figsize=(8,8),                   # Plot histogram
            bins=50)


I am working on the file "Data Science UA - September 2017/Lecture 05 - Modeling Techniques and Regression/Preparing Numeric Data.ipynb".
The first task is:
Notice that in zero-centered data, negative values represent original values that were below average and positive numbers represent values that were above average.
To put all values on a common scale, we can divide all values in a column by that column's standard deviation.
Can you write Python code for it?

# Get column standard deviations
column_deviations = centered_mtcars.std(axis=0)

centered_and_scaled_mtcars = centered_mtcars/column_deviations
print(centered_and_scaled_mtcars.describe())


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Bank_Customer_Arrival_Model.ipynb".
The first task is:
Next we initialize two arrays to keep track of the customers who arrive and who depart:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Bank_Customer_Arrival_Model.ipynb".
The first task is:
We simulate the arrivals and departures in a bank branch using a while loop
Can you write Python code for it?

while t < closing_time or n >= 0:

        # case 1 - within business hours, a customer arrives before any customer leaves the queue
        if time_arrive <= time_depart and time_arrive <= closing_time:

            t = time_arrive      # move time along to the time of the new arrival
            num_arrivals += 1    # increase the number of customers with the additional arrival
            n += 1               # we have an additional customer, increase the size of the waiting line by 1

            # generate time of next arrival
            time_arrive = random.expovariate(lambd_in) + t

            #append the new customer to the arrival list
            arrivals.append(t)
            print("Arrival ", num_arrivals, "at time ", t)

            # generate time of departure
            if n == 1:
                Y = random.expovariate(lambd_out)
                time_depart = t + Y

            '''
            print('Arrivals', arrivals)
            print('Departures', departures)
            '''

        # case 2 - within business hours, a customer departs before the next arrival
        elif time_depart < time_arrive and time_depart <= closing_time:

            # advance time to the next departure time
            t = time_depart

            # one more person served -> increase the count of clients who have been served
            num_departures += 1

            #update the departure list
            departures.append(t)
            print("Departure ", num_departures, "at time ", t)

            # one less person in line -> decrease the size of the waiting line
            n -= 1

            # if the queue is empt -> set the time of the next departure to infinity
            if n == 0:
                time_depart = float('inf')

            # if the queue isn't empty, generate the next time of departure
           ...


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Marketing_Campaign_Simulation.ipynb".
The first task is:
As a result we get a target group of clients to contact. The target group has middle ranges of probability values, namely 0.4 to 0.7 (smallest ${\rm min_probability}$ and largest ${\rm max_probability}$ among three best results) as clients with small probability values would not buy the product even if they get a phone call, while clients with high probability fo sale already decided to buy a product and the phone call would not help.
Can you write Python code for it?

vis = pd.DataFrame(result, columns=['Minimal probability', 'Maximum probability', 'No of sales', 'No of calls', 'Profit'])
print scale_color_gradient
ggplot(vis, aes(x='Minimal probability', y='Maximum probability', color='Profit')) + \
    geom_point(size=50) + \
    scale_color_gradient(low='blue', high='red')


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Alternative way to compute percentiles
5th percentile, median, 95th percentile:
Can you write Python code for it?

sortedScen = sorted(v1)   # Sort scenarios
sortedScen[5-1], sortedScen[50-1], sortedScen[95-1]


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Between 1977 and 2014, S&P 500 returned 9.38% per year on average with a standard deviation of 16.15%
Simulate 30 columns of 100 observations each of single period returns:
Can you write Python code for it?

r_speriod30  = random.normal(SnP500_mean_ret, SnP500_std_ret, (Ns, 30))
r_speriod30


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Compute and plot $v_{30}$
Can you write Python code for it?

v30 = prod(1 + r_speriod30 , 1) * v0

hist, bins = histogram(v30)
positions = (bins[:-1] + bins[1:]) / 2
width = (bins[1] - bins[0]) * 0.8
plt.bar(positions, hist, width=width)
plt.xlabel('portfolio value after 30 years')
plt.ylabel('frequency')
plt.show()


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Compute annual return for bonds:
Can you write Python code for it?

TBill_Adata['returns'] = TBill_Adata['DTB3'] / 100
TBill_Adata = TBill_Adata.dropna()
print TBill_Adata['returns']


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Compute annual return of S&P 500 index:
Can you write Python code for it?

SnP500_Adata[['Adj Close']] = SnP500_Adata[['Adj Close']].apply(pd.to_numeric, errors='ignore')
SnP500_Adata['returns'] = SnP500_Adata['Adj Close'] / SnP500_Adata['Adj Close'].shift(1) -1
SnP500_Adata = SnP500_Adata.dropna()
print SnP500_Adata['returns']


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Compute average annual return and standard deviation of return for bonds:
Can you write Python code for it?

TBill_mean_ret = float(TBill_Adata[['returns']].mean())
TBill_std_ret = float(TBill_Adata[['returns']].std())
print ("T-bill average return = %g%%, st. dev  = %g%%") % (100*TBill_mean_ret, 100*TBill_std_ret)


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Compute covariance matrix:
Can you write Python code for it?

covMat = cov(array(SnP500_Adata[['returns']]),array(TBill_Adata[['returns']]),rowvar=0)
covMat


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Mean:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Minimum, maximum:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Persentiles
5th percentile, median, 95th percentile:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Set start date, end date and data source ('Yahoo Finance', 'Google Finance', etc.). Download S&P 500 index data from Yahoo Finance.
Can you write Python code for it?

start_date = datetime.date(1976,1,1)
end_date = datetime.date(2017,1,1)
# Download S&P 500 index data
try:
    SnP500_Ddata = web.DataReader('^GSPC','yahoo',start_date,end_date)
except:
    SnP500_Ddata = pd.read_csv("http://analytics.romanko.ca/data/SP500_hist.csv")
    SnP500_Ddata.index = pd.to_datetime(SnP500_Ddata.Date)
SnP500_Ddata.head()


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Simulated paths over time:
Can you write Python code for it?

for scenario in r_speriod30:
    y = [prod(1 + scenario[0:i]) * v0 for i in range(0,31)]
    plt.plot(range(0,31), y)
plt.xlabel('time step')
plt.ylabel('portfolio value')
plt.show()


# Plot simulated paths over time
for res in v1:
    plt.plot((0,1), (v0, res))
plt.xlabel('time step')
plt.ylabel('portfolio value')
plt.show()


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Simulation Example 3
Download US Treasury bill data from Federal Reserve:
Can you write Python code for it?

# Download 3-month T-bill rates from Federal Reserve
start_date_b = datetime.date(1977,1,1)
end_date_b = datetime.date(2017,1,1)
TBill_Ddata = web.DataReader('DTB3','fred',start_date_b,end_date_b)
TBill_Ddata.head()


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Simulation Example 4
Compare two portfolios
Can you write Python code for it?

# Compute portfolios by iterating through different combinations of weights
v30comp = []
for w in arange(0.2, 1.01, 0.2):
    v30comp += [w * v0 * stockRet + (1 - w) * v0 * bondsRet]

# Plot a histogram of the distribution of
# differences in outcomes for v30
# (Stratery 4 - Strategy 2)
v30d = v30comp[3] - v30comp[1]

hist, bins = histogram(v30d, bins = 50)
positions = (bins[:-1] + bins[1:]) / 2
width = (bins[1] - bins[0]) * 0.8
plt.bar(positions, hist, width=width)
plt.show()

# Compute number of elements in v30d that are > 0 and < 0 and compare
pos_count = (v30d > 0).sum()
neg_count = (v30d <= 0).sum()

print u"""Strategy 4 was better in %d cases
Strategy 2 was better in %d cases
Difference = %d""" % (pos_count, neg_count, pos_count - neg_count)


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Standard deviation:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Transform daily data into annual data:
Can you write Python code for it?

# Create a time-series of annual data points from daily data
SnP500_Adata = SnP500_Ddata.resample('A').last()
SnP500_Adata[['Volume','Adj Close']].tail()


# Create a time-series of annual data points from daily data
TBill_Adata = TBill_Ddata.resample('A').last()
TBill_Adata[['DTB3']].tail()


I am working on the file "Data Science UA - September 2017/Lecture 07 - Simulation Modeling/Portfolio_Simulation_Modeling.ipynb".
The first task is:
Value of investment at the end of year 1:
$v_1 = v_0 + r_{0,1}\cdot v_0 = (1 + r_{0,1})\cdot v_0$
Can you write Python code for it?

# Distribution of value at the end of year 1
v1 = (r01 + 1) * v0
v1


I am working on the file "Data Science UA - September 2017/Lecture 09 - Data Mining and Machine Learning/Decision_Trees.ipynb".
The first task is:
Let’s manually go through these steps on the interviewee data set. The data set has both True and False labels, and we have four attributes we can split on. So our first step will be to find the partition with the least entropy. Function partition_by() does the partitioning and function partition_entropy_by() uses it to compute entropy.
Can you write Python code for it?

inputs = [
    ({'level':'Senior','lang':'Java','tweets':'no','phd':'no'},   False),
    ({'level':'Senior','lang':'Java','tweets':'no','phd':'yes'},  False),
    ({'level':'Mid','lang':'Python','tweets':'no','phd':'no'},     True),
    ({'level':'Junior','lang':'Python','tweets':'no','phd':'no'},  True),
    ({'level':'Junior','lang':'R','tweets':'yes','phd':'no'},      True),
    ({'level':'Junior','lang':'R','tweets':'yes','phd':'yes'},    False),
    ({'level':'Mid','lang':'R','tweets':'yes','phd':'yes'},        True),
    ({'level':'Senior','lang':'Python','tweets':'no','phd':'no'}, False),
    ({'level':'Senior','lang':'R','tweets':'yes','phd':'no'},      True),
    ({'level':'Junior','lang':'Python','tweets':'yes','phd':'no'}, True),
    ({'level':'Senior','lang':'Python','tweets':'yes','phd':'yes'},True),
    ({'level':'Mid','lang':'Python','tweets':'no','phd':'yes'},    True),
    ({'level':'Mid','lang':'Java','tweets':'yes','phd':'no'},      True),
    ({'level':'Junior','lang':'Python','tweets':'no','phd':'yes'},False)
]


I am working on the file "Data Science UA - September 2017/Lecture 09 - Data Mining and Machine Learning/Decision_Trees.ipynb".
The first task is:
Our tree will consist of decision nodes (which ask a question and direct us differently depending on the answer) and leaf nodes (which give us a prediction). We will build it using the relatively simple ID3 algorithm, which operates in the following manner. Let’s say we’re given some labeled data, and a list of attributes to consider branching on.
- If the data all have the same label, then create a leaf node that predicts that label and then stop.
- If the list of attributes is empty (i.e., there are no more possible questions to ask), then create a leaf node that predicts the most common label and then stop.
- Otherwise, try partitioning the data by each of the attributes.
- Choose the partition with the lowest partition entropy.
- Recur on each partitioned subset using the remaining attributes.
This is what’s known as a “greedy” algorithm because, at each step, it chooses the most immediately best option. Given a data set, there may be a better tree with a worse-looking first move. If so, this algorithm won’t find it. Nonetheless, it is relatively easy to understand and implement, which makes it a good place to begin exploring decision trees.
Can you write Python code for it?

def entropy(class_probabilities):
    """given a list of class probabilities, compute the entropy"""
    return sum(-p * math.log(p, 2) for p in class_probabilities if p)

def class_probabilities(labels):
    total_count = len(labels)
    return [count / total_count
            for count in Counter(labels).values()]

def data_entropy(labeled_data):
    labels = [label for _, label in labeled_data]
    probabilities = class_probabilities(labels)
    return entropy(probabilities)

def partition_entropy(subsets):
    """find the entropy from this partition of data into subsets"""
    total_count = sum(len(subset) for subset in subsets)

    return sum( data_entropy(subset) * len(subset) / total_count
                for subset in subsets )

def group_by(items, key_fn):
    """returns a defaultdict(list), where each input item
    is in the list whose key is key_fn(item)"""
    groups = defaultdict(list)
    for item in items:
        key = key_fn(item)
        groups[key].append(item)
    return groups

def partition_by(inputs, attribute):
    """returns a dict of inputs partitioned by the attribute
    each input is a pair (attribute_dict, label)"""
    return group_by(inputs, lambda x: x[0][attribute])

def partition_entropy_by(inputs,attribute):
    """computes the entropy corresponding to the given partition"""
    partitions = partition_by(inputs, attribute)
    return partition_entropy(partitions.values())

def classify(tree, input):
    """classify the input using the given decision tree"""

    # if this is a leaf node, return its value
    if tree in [True, False]:
        return tree

    # otherwise find the correct subtree
    attribute, subtree_dict = tree

    subtree_key = input.get(attribute)  # None if input is missing attribute

    if subtree_key not in subtree_dict: # if no subtree for key,
        subtree_key = None              # we'll use the None subtree

    subtree = subtree_dict[subtree_key] # choose the appropriate subtree
    return...


I am working on the file "Data Science UA - September 2017/Lecture 09 - Data Mining and Machine Learning/Decision_Trees.ipynb".
The first task is:
The lowest entropy comes from splitting on level, so we’ll need to make a subtree for each possible level value. Every Mid candidate is labeled True, which means that the Mid subtree is simply a leaf node predicting True. For Senior candidates, we have a mix of Trues and Falses, so we need to split again:
Can you write Python code for it?

senior_inputs = [(input, label)
                 for input, label in inputs if input["level"] == "Senior"]

for key in ['lang', 'tweets', 'phd']:
    print key, partition_entropy_by(senior_inputs, key)


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
Advanced Visualizations
The map above shows a network of roads which spans 40 different intersections (labeled 0 through 39).
The show_map function which generated this map also takes a few optional parameters which might be useful for visualizaing the output of the search algorithm you will write.

start - The "start" node for the search algorithm.
goal  - The "goal" node.
path  - An array of integers which corresponds to a valid sequence of intersection visits on the map.
Can you write Python code for it?

# run this code, note the effect of including the optional
# parameters in the function call.
show_map(map_40, start=5, goal=34, path=[5,16,37,12,34])


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
Map Basics
Can you write Python code for it?

map_10 = load_map('map-10.pickle')
show_map(map_10)


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
Roads
The roads property is a list where, if i is an intersection, roads[i] contains a list of the intersections that intersection i connects to.
Can you write Python code for it?

# this shows that intersection 0 connects to intersections 7, 6, and 5
map_10.roads[0]

# This shows the full connectivity of the map
map_10.roads

# map_40 is a bigger map than map_10
map_40 = load_map('map-40.pickle')
show_map(map_40)


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
Testing your Code
If the code below produces no errors, your algorithm is behaving correctly. You are almost ready to submit! Before you submit, go through the following submission checklist:
Submission Checklist

Does my code pass all tests?
Does my code implement A* search and not some other search algorithm?
Do I use an admissible heuristic to direct search efforts towards the goal?
Do I use data structures which avoid unnecessarily slow lookups?

When you can answer "yes" to all of these questions, submit by pressing the Submit button in the lower right!
Can you write Python code for it?

from test import test

test(shortest_path)


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
The map above (run the code cell if you don't see it) shows a disconnected network of 10 intersections. The two intersections on the left are connected to each other but they are not connected to the rest of the road network. On the graph above, the edge between 2 nodes(intersections) represents a literal straight road not just an abstract connection of 2 cities.
These Map objects have two properties you will want to use to implement A* search: intersections and roads
Intersections
The intersections are represented as a dictionary.
In this example, there are 10 intersections, each identified by an x,y coordinate. The coordinates are listed below. You can hover over each dot in the map above to see the intersection number.
Can you write Python code for it?


I am working on the file "Data Structures and Algorithms Nanodegree/P4 - Route Planner/project_notebook.ipynb".
The first task is:
Writing your algorithm
You should open the file student_code.py in another tab and work on your algorithm there. Do that by selecting File > Open and then selecting the appropriate file.
The algorithm you write will be responsible for generating a path like the one passed into show_map above. In fact, when called with the same map, start and goal, as above you algorithm should produce the path [5, 16, 37, 12, 34]
```bash

shortest_path(map_40, 5, 34)
[5, 16, 37, 12, 34]
```
Can you write Python code for it?

path = shortest_path(map_40, 5, 34)
if path == [5, 16, 37, 12, 34]:
    print("great! Your code works for these inputs!")
else:
    print("something is off, your code produced the following:")
    print(path)


I am working on the file "Data Visualization/Challenge - Data Visualization.ipynb".
The first task is:
4: Box Plot - Audience And Critic Ratings
Let's use box plots to better understand the distributions of ratings by critics versus ratings by the audience.
Use the Pandas Dataframe method plot to generate boxplots for the Critic Rating and Audience Rating columns.
Can you write Python code for it?

fig = plt.figure()
normal_movies.boxplot(['Critic Rating', 'Audience Rating'])
plt.show()


I am working on the file "Data Visualization/Challenge - Data Visualization.ipynb".
The first task is:
6: Box Plots - Profitable Vs Unprofitable Movies
Many Hollywood movies aren't profitable and it's interesting to understand the role of ratings in a movie's profitability. You first need to separate the movies into those were that profitable and those that weren't.
We've created a new Boolean column called Profitable with the following specification:
False if the value for Profitability is less than or equal to 1.0. <br/>
True if the value for Profitability is greater than or equal to 1.0.
Can you write Python code for it?

def is_profitable(row):
    if row["Profitability"] <= 1.0:
        return False
    return True
normal_movies["Profitable"] = normal_movies.apply(is_profitable, axis=1)

fig = plt.figure(figsize=(8,6))
ax1 = fig.add_subplot(1, 2, 1)
sns.boxplot(x=normal_movies['Profitable'], y=normal_movies['Audience Rating'], ax=ax1)
ax2 = fig.add_subplot(1, 2, 2)
sns.boxplot(x=normal_movies['Profitable'], y=normal_movies['Critic Rating'], ax=ax2)
plt.show()


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Based on the number of pokemons, we want to assign the county to a color class. For example, if number > 50, use color1, if 40 < number <= 50, use color 2, etc.
Remember that we saved the svg in the soup object. Now that we have changed the svg to fill with colors, we can just write it out as a new file.
Can you write Python code for it?

with open ('svg_colored.svg', 'w') as g:
    g.write(soup.prettify())


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Choropleth map
Let's make a choropleth map with Pokemon statistics. The color of a county should correspond to the number of Pokemons found there. You can download the data from Canvas (pokemon.csv). The data is a subset of the pokemon data from Kaggle.
We'll also need an SVG map. You can download it from Wikipedia.
If you open the SVG with a text editor, you'll see many <path> tags. Each of these is a county. We want to change their style tags, namely the fill color. We want the darkness of fill to correspond to the number of Pokemons in each county.
In the SVG, there is also an id tag for each path, which is actually something called a FIPS code. FIPS stands for Federal Information Processing Standard. Every county has a unique FIPS code, and it’s how we are going to associate each path with our pokemon data.
For this we first need to do some data cleaning.
Can you write Python code for it?

pokemon = pd.read_csv('pokemon.csv')
pokemon.head()


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Load it with BeautifulSoup
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Now we can access it as a dictionary and get the county's FIPS code.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Now we can turn to our SVG file. We want to find the paths for each county: there are over 3000 counties, so we'll need a nice way. For this, we can use the BeautifulSoup package. This is a package specialized at parsing XMLs. SVGs are essentially XML files, so can be handled in the same way as handling HTML and other XML files.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
Read in the svg
Can you write Python code for it?

svg = open('USA_Counties_with_FIPS_and_names.svg', 'r').read()


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
The data only has the latitude and longitude data. To convert this to an FIPS code, we need some reverse-geocoding. The Federal Communications Commission provides an API for such tasks.
The API works through an HTTP request, so we can use Python's urllib library to handle it. For example:
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
The result comes as a json object, so we need to parse it with Python's json decoder.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
We can do this to all data in the dataframe.  Pandas's  apply is a very nice feature that you may want to use, it allows you to write a function and apply it to the dataframe.
Can you write Python code for it?

# TODO: create a column in the dataframe called 'FIPS' for the FIPS codes.
# You should have the dataframe look like the following.
# Note that looking up all the lat-lon pairs may take some time.
def get_fips(row):
    res = urlopen("http://data.fcc.gov/api/block/find?format=json&latitude="+str(row['latitude'])+"&longitude="+str(row['longitude'])).read().decode('utf-8')
    return json.loads(res)['County']['FIPS']
pokemon['FIPS'] = pokemon.apply(get_fips, axis=1)

pokemon.head()


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
We should also decide on the colors. colorbrew provides some nice palattes. Pick one of the sequential colors and make the hexadecimal encodings into a list.
Can you write Python code for it?

colors = ['#fef0d9', '#fdd49e', '#fdbb84','#fc8d59','#e34a33','#b30000']

# TODO: substitute the above with a palatte of your choice.
colors = ['#f0f9e8','#bae4bc','#7bccc4','#43a2ca','#0868ac']


I am working on the file "Data Visualization/Lab 12/w12_lab_Vipul_Munot.ipynb".
The first task is:
We want to color the counties by the number of pokemons appearing in them, so now all we need is a table with the counties' FIPS and number of pokemons in them.
Can you write Python code for it?

pokemon_density = pd.DataFrame(pokemon.groupby('FIPS').size().reset_index())
pokemon_density.columns = ['FIPS', 'Count']

pokemon_density.head()


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Code for Q1
Can you write Python code for it?

import pandas as pd
imdb = pd.read_csv('imdb.csv', delimiter='\t')

imdb.head()

min(imdb['Year'])

max(imdb['Year'])

from collections import Counter
Counter(imdb["Year"])


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Code for Q2
Can you write Python code for it?

# implement below
imdb['Rating'].mean()

imdb['Votes'].mean()


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Code for Q3
Can you write Python code for it?

# implement below
import warnings
warnings.filterwarnings('ignore')
imdb.sort_index(by=['Rating'], ascending=[False]).head()

imdb.sort_index(by=['Votes'], ascending=[False]).head()


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Python automates the job above by using Counter.
Can you write Python code for it?

from collections import Counter

movie_counter = Counter()
movie_counter[1972] +=1
print(movie_counter[1972])
print(movie_counter[1970])


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Q2: What is the average ratings/votes?
We can store the ratings/votes column as a list and then calculate various basic statistics (mean, median, etc.). To do this, we can use the NumPy library and call the function numpy.mean and numpy.median. For example,
Can you write Python code for it?

import numpy as np

alist = [1,3,6,2,5,2]
print(np.mean(alist))
print(np.median(alist))


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
Q3: What are the 5 movies that have the highest ratings/votes?
Store the movie titles and ratings information as a dictonary:

key: movie title
value: movie rating

Then, we can sort the dictionary based on its values, which will return a list of tuples. Note to print only the top 5 movies.
Can you write Python code for it?

import operator

dt = {1971: 2, 1975: 10, 1962: 1, 1980: 50, 1981: 55}
sorted_x_by_val = sorted(dt.items(), key=operator.itemgetter(1), reverse=True )
print(sorted_x_by_val)
for elem in sorted_x_by_val:
    print(elem[0],elem[1])


I am working on the file "Data Visualization/Lab 2/lab02_Munot_Vipul.ipynb".
The first task is:
and you have convenient functions like min() and max() for calculating the min and max value of a list or iterable.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Exercise
Calculate the following basic characteristics of ratings of movies only in 1994: 10th percentile, median, mean, 90th percentile.

http://docs.scipy.org/doc/numpy/reference/generated/numpy.percentile.html
http://pandas.pydata.org/pandas-docs/stable/text.html

Write your code in the cell below
Can you write Python code for it?

# implement here
df[(df['Year']==1994)]['Rating'].describe([.1,0.9])

df['Rating'].median()


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Exercise
Can you plot the histogram of ratings of the movies between 2000 and 2014?
Can you write Python code for it?

# implement here
plt.hist(df[(df['Year']>2000) & (df['Year']<2014)]['Rating'],bins = 10)


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Exercise
Let's try to make some style changes to the plot:

change the color from blue to whatever you want
http://matplotlib.org/users/pyplot_tutorial.html#working-with-text
http://matplotlib.org/api/colors_api.html
add labels of x and y axis
change the number of bins to 20
Can you write Python code for it?

# implement here
plt.hist(df[(df['Year']>2000) & (df['Year']<2014)]['Rating'],bins = 20,facecolor='g')
plt.xlabel('bins')
plt.ylabel('# Ratings')
plt.title('Histogram of Rating Distribution for years 2000-2014')
plt.grid(True)


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Exercise
Read the document about the function and make the following changes: http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.distplot.html

change the number of bins to 10;
not to show kde;
Can you write Python code for it?

# implement here
sns.distplot(df['Rating'],bins = 10,kde=False)
plt.xlabel('bins')
plt.ylabel('# Ratings')
plt.title('Histogram of Rating Distribution for years 2000-2014')


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Finally, if we want to know the top 10 movies in 1990s, we can use the sort() function:
Can you write Python code for it?

sorted_by_rating = movie_nineties.sort('Rating', ascending=False)
sorted_by_rating[:10]


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
It is very easy to answer the question of the number of movies per year. The value_counts() function counts how many times each data value (year) appears.
Can you write Python code for it?

print( min(df['Year']), df['Year'].min(), max(df['Year']), df['Year'].max() )
year_nummovies = df["Year"].value_counts()
year_nummovies.head()


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Let's look at the first few rows to get some sense of the data.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Matplotlib magic
Jupyter notebook provides several magic commands. These are the commands that you can use only in the notebook (not in IDLE for instance). One of the greatest magic command is matplotlib inline, which displays plots within the notebook instead of creating figure files.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Q1: Revisting W2 lab
Let's revisit last week's exercise with pandas. It's very easy to read CSV files with pandas, using the panda.read_csv() function. This function has many many options and it may be worthwhile to take a look at available options. Things that you need to be careful are:

delimiter or sep: the data file may use ',', tab, or any weird character to separate fields. You can't read data properly if this option is incorrect.
header: some data files have "header" row that contains the names of the columns. If you read it as data or use the first row as the header, you'll have problems.
na_values or na_filter: often the dataset is incomplete and contains missing data (NA, NaN (not a number), etc.). It's very important to take care of them properly.

You don't need to create dictionaries and other data structures. Pandas just imports the whole table into a data structure called DataFrame. You can do all kinds of interesting manipulation with the DataFrame.
Can you write Python code for it?

df = pd.read_csv('imdb.csv', delimiter='\t')


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Q2: Basic plotting with pandas
Pandas provides some easy ways to draw plots by using matplotlib. Dataframe object has several plotting functions. For instance,
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Q4: Basic plotting with Seaborn
Seaborn sits on the top of matplotlib and makes it easier to draw statistical plots. Most plots that you create with Seaborn can be created with matplotlib. It just typically requires a lot more work.
Be sure seaborn has been installed on your computer, otherwise run
conda install seaborn
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
Then, we can do the calculation
Can you write Python code for it?

print( movie_nineties['Rating'].median(), movie_nineties['Votes'].median() )


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
There are many ways to import matplotlib, but the most common way is:
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
To calculate average ratings and votes
Can you write Python code for it?

print( np.mean(df['Rating']), np.mean(df['Votes']) )


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
To get the median ratings of movies in 1990s, we first select only movies in that decade
Can you write Python code for it?

geq = df['Year'] >= 1990
leq = df['Year'] <= 1999
movie_nineties = df[geq & leq]

movie_nineties.head()


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
You can extract one column by using dictionary-like expression
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
You can swap the order of rows and columns.
But, when you deal with large datasets, You may want to stick to this principle:

Try to reduce the size of the dataset you are handling as soon as possible, and as much as possible.

For instance, if you have a billion rows with three columns, getting the small row slice (df[:10]) and working with this small slice can be much better than getting the column slice (df['Year']) and working with this slice (still contains billion items).
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 3/lab03_Munot_Vipul.ipynb".
The first task is:
or you can even do
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
But here we only have x (the generated data). We can set the y values to 0. The np.zeros_like(data) function creates a numpy array (list) that have the same dimension as the argument.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Can you draw boxplots of log-transformed movie votes for different decade?
Can you write Python code for it?

# TODO
ax = sns.boxplot(x=decade, y = log_votes)
ax.figure.set_size_inches(12, 8)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Can you draw boxplots of movie votes for different decade?
Can you write Python code for it?

# TODO
ax = sns.boxplot(x=decade, y=df['Votes'])
ax.figure.set_size_inches(12, 8)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Generate 10 normally distributed numbers with mean 5 and sigma 3:
Can you write Python code for it?

data = generate_many_numbers(N=10)
print(data)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Let's also make the symbol transparent. Here is a useful Google query, and the documentation of scatter() also helps.
Can you write Python code for it?

data = generate_many_numbers(N=200)

# From the last question
# zittered_ypos = ??

# TODO: implement this
# plt.figure(figsize=(10,1))
# plt.scatter( ?? )
# plt.gca().axes.get_yaxis().set_visible(False)
# TODO: implement this
zittered_ypos = np.random.rand(200)
plt.figure(figsize=(10,1))
plt.scatter(data, zittered_ypos, s = 50, alpha = 0.35)
plt.gca().axes.get_yaxis().set_visible(False)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Looks good! Can you increase or decrease the number of bins? Find the documentation here.
Can you write Python code for it?

# TODO: try different number of bins
movie_df['Rating'].hist(bins = 30)

movie_df['Rating'].hist(bins = 20)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Lots and lots of points
Let's use real data. Load the IMDb dataset that we used before.
Can you write Python code for it?

movie_df = pd.read_csv('imdb.csv', delimiter='\t')
movie_df.head()


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Now let's plot the generated 1-D data.
Can you write Python code for it?

plt.figure(figsize=(10,1)) # set figure size, width = 10, height = 1
plt.scatter(data, np.zeros_like(data), s=50) # set size of symbols to 50. Change it and see what happens.
plt.gca().axes.get_yaxis().set_visible(False) # set y axis invisible


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Ok, I think we can see all data points. But what if we have more numbers?
Can you write Python code for it?

# TODO: generate 100 numbers and plot them in the same way.
data = np.random.rand(100)

plt.figure(figsize=(10,1))
plt.scatter(data, np.zeros_like(data), s = 50)
plt.gca().axes.get_yaxis().set_visible(False)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
One easy way to transform a particular year to the decade (e.g., 1874 -> 1870): divide by 10 and multiply it by 10 again.
In Python 3, the // operator is used for integer division.
Can you write Python code for it?

print(1874//10)
print(1874//10*10)
decade = (df['Year']//10) * 10
decade.head()

ax = sns.boxplot(x=decade, y=df['Rating'])
ax.figure.set_size_inches(12, 8)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Q1 1-D Scatter Plot
Using fake data
Remember that, if you want to play with visualization tools, you can use not only the real data, but also fake data. Actually it is a nice way to experiment because you can control every aspect of data. Let's create some random numbers.
The function np.random.randn() generates a sample with size $N$ from the standard normal distribution.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Q2 Histogram
There are too many data points! Let's try histogram. Actually pandas supports plotting through matplotlib and you can directly visualize dataframes and series.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Q3 Boxplot
Now let's try boxplot. We can use pandas' plotting functions. The usages of boxplot is here.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
The following small function generates $N$ normally distributed numbers:
Can you write Python code for it?

def generate_many_numbers(N=10, mean=5, sigma=3):
    return mean + sigma * np.random.randn(N)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
The most immediate method to visualize 1-D data is just plotting it. Here we can use the scatter() function to draw a scatter plot. The most basic usage of this function is to provide x and y.
Can you write Python code for it?

x = np.arange(1,11)
y = x + 5
print(x)
print(y)
plt.scatter(x, y)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
Try to plot the 'Rating' information using 1D scatter plot. Does it work?
Can you write Python code for it?

# TODO: plot 'rating'

rating = movie_df['Rating'].values
plt.figure(figsize=(10,1))
plt.scatter(rating, np.zeros_like(rating), s = 50)
plt.gca().axes.get_yaxis().set_visible(False)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
We can also easily draw a series of boxplots grouped by categories. For example, let's do the boxplots of movie ratings for different decades.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
We can use transparency as well as empty symbols.

Increase the number of points to 1,000
Set the symbol empty and edgecolor red (a useful query)
Can you write Python code for it?

# TODO: implement this
# data = ??
# zittered_ypos = ??

# TODO: implement this
# plt.figure(figsize=(10,1))
# plt.scatter( ?? )
# plt.gca().axes.get_yaxis().set_visible(False)

data = np.random.rand(1000)
zittered_ypos = np.random.rand(1000)
plt.figure(figsize=(10,1))
plt.scatter(data, zittered_ypos, s = 50, c = 'white', edgecolors='r')
plt.gca().axes.get_yaxis().set_visible(False)


I am working on the file "Data Visualization/Lab 5/w05_Vipul_Munot.ipynb".
The first task is:
What do you see? Can you actually see the "box"? The number of votes span a very wide range, from 1 to more than 1.4 million. One way to deal with this is to make a log-transformation of votes, which can be done with the numpy.log() function.
Can you write Python code for it?

log_votes = np.log(df['Votes'])
log_votes.head()


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Actually, n_raw contains the values of histograms, i.e., the number of movies in each of the 10 bins. Thus, the sum of the elements in n_raw should be equal to the total number of movies:
Can you write Python code for it?

# TODO: test whether the sum of the numbers in n_raw is equal to the number of movies.
sum(n_raw)==len(movie_df)


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
As you can see, the default histogram does not normalize with binwidth and simply shows the counts! This can be very misleading if you are working with variable bin width. One simple way to fix this is using the option normed.
Can you write Python code for it?

# TODO: fix it with normed option.
plt.hist(val, weights=weight, bins = bins, normed = True)


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Boxplot
Let's look at the example data that we looked at during the class.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Do you notice weird patterns that emerge from bins=40? Can you guess why do you see such patterns? What are the peaks and what are the empty bars? What do they tell you about choosing the binsize in histograms?
Now, let's try to apply several algorithms for finding the number of bins.
Can you write Python code for it?

N = len(movie_df['Rating'])

# TODO: plot three histograms based on three formulae

plt.figure(figsize=(12,4))

# Sqrt
nbins = int(np.sqrt(N))

plt.subplot(1,3,1)
plt.hist(movie_df['Rating'], bins = nbins)
plt.title("SQRT, {0} bins".format(nbins))

# Sturge's formula
plt.subplot(1,3,2)
nbins = int(np.ceil(np.log2(N) + 1))
plt.hist(movie_df['Rating'], bins = nbins)
plt.title("Sturge's, {0} bins".format(nbins))

# Freedman-Diaconis
plt.subplot(1,3,3)
data = movie_df['Rating'].order()
iqr = np.percentile(data, 75) - np.percentile(data, 25)
width = 2*iqr/np.power(N, 1/3)
nbins = int((max(data) - min(data)) / width)
plt.hist(movie_df['Rating'], bins = nbins)
plt.title("Freedman-Diaconis, {0} bins".format(nbins))


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Draw a histogram focused on the range [0, 10] to just see how many datapoints are there.
Can you write Python code for it?

# TODO: set the xlim to [0, 10] adjust ylim and bins so that
# we can see how many datapoints are there for each # of votes.


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Draw histogram using this data. Useful query: Google search: matplotlib histogram pre-counted
Can you write Python code for it?

# TODO: draw a histogram with pre-counted data.
#plt.xlabel("Hours")
val, weight = zip(*[(k, v) for k,v in data.items()])
plt.hist(val, weights=weight, bins = bins)
plt.xlabel("Hours")


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Histogram
Let's revisit the table from the class
| Hours | Frequency |
|-------|-----------|
| 0-1   | 4,300     |
| 1-3   | 6,900     |
| 3-5   | 4,900     |
| 5-10  | 2,000     |
| 10-24 | 2,100     |
You can draw a histogram by just providing bins and counts instead of a list of numbers. So, let's do that for convenience.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
In this case, the edges of the 10 bins do not change. But now n represents the heights of the bins. Can you verify that matplotlib has correctly normed the heights of the bins?
Hint: the area of each bin should be equal to the fraction of movies in that bin.
Can you write Python code for it?

# TODO: verify that it is properly normalized.
normalizeList = []
for i in range(len(bins)):
    try:
        Moviesbins = movie_df[(movie_df['Rating'] >= bins[i]) & (movie_df['Rating'] <= bins[i+1])]
        normalizeList.append(round(len(Moviesbins)/len(movie_df), 4))
    except IndexError:
        pass
print("Bin widths", normalizeList)
print("Data from histogram", n)


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Investigating the anomalies in the histogram
Let's investigate the anormalies in the histogram.
Can you write Python code for it?

# TODO: draw the histogram with 120 bins
n, bins, patches = plt.hist(movie_df['Rating'], bins = 120)
plt.title("Histogram with bins 120")
plt.xlabel("Rating")
plt.ylabel("Frequency")


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Noticed that the width of each bin is the same? This is equal-width binning. We can calculate the width as:
Can you write Python code for it?

min_rating = min(movie_df['Rating'])
max_rating = max(movie_df['Rating'])
print(min_rating, max_rating)
print( (max_rating-min_rating) / 10 )


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Ok, the minimum number of votes is 5 not 1. IMDB may only keep the rating information for movies with at least 5 votes. This may explain why the most frequent ratings are like 6.4 and 6.6. Let's plot the histogram with only the rows with 5 votes. Set the binsize 30.
Can you write Python code for it?

# TODO: plot the histogram only with ratings that have the minimum number of votes.
df = movie_df[movie_df['Votes'] == 5]
plt.hist(df['Rating'], bins = 30)
plt.xlabel("Rating")
plt.ylabel("Frequency")
plt.title("Histogram of rating with min no of votes")


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Ok. They doesn't necessarilly cover the integer values. Let's see the minimum number of votes.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
Plot the histogram of movie ratings using the plt.hist() function.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
The second returned object (bins_raw) is a list containing the edges of the 10 bins: the first bin is [1.0,1.89], the second [1.89,2.78], and so on. We can calculate the width of each bin.
Can you write Python code for it?

# TODO: calculate the width of each bin and print them.
for i in range(len(bins_raw)-1):
    print (bins_raw[i+1] - bins_raw[i])


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
What does the argument in plt.subplot(1,2,1) mean?
http://stackoverflow.com/questions/3584805/in-matplotlib-what-does-the-argument-mean-in-fig-add-subplot111
Ok, so create 8 subplots (2 rows and 4 columns) with the given binsizes.
Can you write Python code for it?

binsizes = [2, 3, 5, 10, 30, 40, 60, 100 ]

plt.figure(1, figsize=(18,8))

for i, bins in enumerate(binsizes):
    # TODO: use subplot and hist() function to draw 8 plots
    plt.subplot(2, 4, i + 1)
    movie_df['Rating'].hist(bins = bins)
    plt.title("Bin size " + str(bins))


I am working on the file "Data Visualization/Lab 6/w06_Vipul_Munot.ipynb".
The first task is:
What's going on? The number of votes is heavily skewed and most datapoints are at the left end.
Can you write Python code for it?

# TODO: plot the same thing but limit the xrange (xlim) to [0, 100].


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
But we only have one statistic. How can we know if this correctly represents the mean of the actual data? We need to compute a confidence interval. This is when we can use bootstrapping.
First, Let's create a function that does the resampling with replacement. It should create a list of the same length as the sample(50 in this case), in which each element is taken randomly from the sample. In this way, some elements may appear more than once, and some none. Then we calculate the mean value of this list.
Can you write Python code for it?

def bootstrap_resample(rating_list):
    resampled_list = []
    #todo: write the function that returns the mean of resampled list.
    for i in range(50):
        resampled_list.append(random.choice(rating_list))
    return np.mean(resampled_list)


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
Now we can compute the confidence interval. Say we want the 90% confidence, then we only need to pick out the .95 and .05 critical values.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
That is, we need to pick the 50th and 950th largest values from the list. We can name it x_a and x_b.
Can you write Python code for it?

#todo: sort the list by ascending and pick out the 50th and 950th value.
sampled_means.sort()
x_a = sampled_means[49]
x_b = sampled_means[949]
print (x_a,x_b)


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
The confidence interval will then be: [x - (x - x_a), x + (x_b - x)].
Can you write Python code for it?

#todo: calculate the confidence interval.
#Does the mean of the original data fall within this interval? Show your statistics.
print([x - (x - x_a), x + (x_b - x)])

np.mean(movie_df['Rating'])


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
The main advantange of CDF is that we can directly observe percentiles from the plot. Given the number of movies we have, can you estimate the following statistics by observing the plot? Compare your estimation to the precise results calculated from movie_df.

The numer of movies with rating <= 7
The median rating of movies
The rating which 90% of movies are under or equal to
Can you write Python code for it?

#TODO: provide your estimations.
#1. 0.65 * len(ratings) = 203457.15.
#2. 6.5.
#3. 8.2.

#TODO: calculate the statistics from movie_df.
seven = movie_df['Rating'][movie_df['Rating'] <= 7].values
print(len(seven))
print(np.median(movie_df['Rating']))
print(np.percentile(movie_df['Rating'], [90])[0])


I am working on the file "Data Visualization/Lab 7/w07_lab_Vipul_Munot.ipynb".
The first task is:
We don't usually just do this once: the typical minimal resample number is 1000. We can create a new list to keep this 1000 mean values.
Can you write Python code for it?

sampled_means = []

#todo: call the function 1000 times and populate the list with its returned values.
for i in range(1000):
    mean = bootstrap_resample(movie_df_sample['Rating'].values)
    sampled_means.append(mean)


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Actually, the dataset is included in seaborn and we can load it.
Can you write Python code for it?

df = sns.load_dataset("anscombe")
df.head()


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
All four datasets are in this single data frame and the 'dataset' indicator is one of the columns. This is a form often called tidy data, which is easy to manipulate and plot. In tidy data, each row is an observation and columns are the properties of the observation. Seaborn makes use of the tidy form.
We can show the linear regression results for each eadataset. Here is the example:
Can you write Python code for it?

sns.lmplot(x="x", y="y", col="dataset", hue="dataset", data=df,
           col_wrap=2, ci=None, palette="muted", size=4,
           scatter_kws={"s": 50, "alpha": 1})


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
As we can see, most votes fall in the first bin, and we cannot see the values from the second bin.
How about plotting on the log scale?
Can you write Python code for it?

# TODO: change the y scale to log
plt.hist(movie_df['Votes'])
plt.yscale('log')


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Can you plot the histogram and KDE of the log of movie votes?
Can you write Python code for it?

# TODO: implement this using pandas
logs = np.log(movie_df['Votes'])
logs.hist(bins=10, normed=True)
logs.plot(kind='kde')
plt.xlim(0, 25)

# TODO: implement this using seaborn
sns.distplot(logs, bins=10)


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Change the number of bins to 1000.
Can you write Python code for it?

# TODO: set the bin number to 1000
plt.hist(movie_df['Votes'], bins=1000)
plt.yscale('log')


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Hexbin
There are too many data points. We need to bin them, which can be done by using the jointplot() and setting the kind parameter.
Can you write Python code for it?

# TODO: draw a joint plot with hexbins and two histograms for each marginal distribution
sns.jointplot(np.log(subset['Votes']), subset['Rating'], kind='hexbin')


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
KDE
Import the IMDb data.
Can you write Python code for it?

movie_df = pd.read_csv('imdb.csv', delimiter='\t')
movie_df.head()


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
KDE
We can also do 2D KDE using seaborn's kdeplot() function.
Can you write Python code for it?

sns.kdeplot(np.log(subset['Votes']), subset['Rating'], cmap="Reds", shade=True, shade_lowest=False)


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Log-bin
Let's first see what the histogram looks like if we do not use the log scale.
Can you write Python code for it?

# TODO: plot the histogram of movie votes
movie_df = pd.read_csv('imdb.csv', delimiter='\t')
plt.hist(movie_df['Votes'])