I am working on the file "P13Interactive Widgets.ipynb".
The first task is:
multiple instances remain in sync!
Can you write Python code for it?

w.close()

w = IntSlider()
display(w)

w.value

w.value = 100

w.keys

Text(value='Hello World!')

Text(value='Hello World!', disabled=True)

from traitlets import link
a = FloatText()
b = FloatSlider()
display(a,b)

mylink = link((a, 'value'), (b, 'value'))

mylink.unlink()

print(widgets.Button.on_click.__doc__)

from IPython.display import display
button = widgets.Button(description="Click Me!")
display(button)

def on_button_clicked(b):
    print("Button clicked.")

button.on_click(on_button_clicked)

text = widgets.Text()
display(text)

def handle_submit(sender):
    print(text.value)

text.on_submit(handle_submit)

print(widgets.Widget.on_trait_change.__doc__)

print(widgets.obse)

int_range = widgets.IntSlider()
display(int_range)

def on_value_change(name, value):
    print(value)

int_range.on_trait_change(on_value_change, 'value')

import traitlets

# Create Caption
caption = widgets.Label(value = 'The values of slider1 and slider2 are synchronized')

# Create IntSlider
slider1 = widgets.IntSlider(description='Slider 1')
slider2 =  widgets.IntSlider(description='Slider 2')

# Use trailets to link
l = traitlets.link((slider1, 'value'), (slider2, 'value'))

# Display!
display(caption, slider1, slider2)

# Create Caption
caption = widgets.Latex(value = 'Changes in source values are reflected in target1')

# Create Sliders
source = widgets.IntSlider(description='Source')
target1 = widgets.IntSlider(description='Target 1')

# Use dlink
dl = traitlets.dlink((source, 'value'), (target1, 'value'))
display(caption, source, target1)

# May get an error depending on order of cells being run!
l.unlink()
dl.unlink()


I am working on the file "P14 Lorenz CustomWidgets.ipynb".
The first task is:
Using IPython's interactive function, we can explore how the trajectories behave as we change the various parameters.
Can you write Python code for it?


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Analyzing Model Performance
In this third section of the project, you'll take a look at several models' learning and testing error rates on various subsets of training data. Additionally, you'll investigate one particular algorithm with an increasing max_depth parameter on the full training set to observe how model complexity affects learning and testing errors. Graphing your model's performance based on varying criteria can be beneficial in the analysis process, such as visualizing behavior that may not have been apparent from the results alone.
Can you write Python code for it?

learning_curves(X_train, y_train, X_test, y_test)


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Answer:
The predicted value of the client's home given the client features is $20968. This is within the inter quartile range of the dataset. It's also within 1 standard deviation from the mean. This leads us to believe that the predicted value is not an extreamity . The client features correspond to near average housing price in boston.
Question 12 (Final Question):
In a few sentences, discuss whether you would use this model or not to predict the selling price of future clients' homes in the Greater Boston area.
Can you write Python code for it?


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Question 1
As a reminder, you can view a description of the Boston Housing dataset here, where you can find the different features under Attribute Information. The MEDV attribute relates to the values stored in our housing_prices variable, so we do not consider that a feature of the data.
Of the features available for each data point, choose three that you feel are significant and give a brief description for each of what they measure.
Remember, you can double click the text box below to add your answer!
Answer:
RM - Average Number of rooms, generally a a measure of how big the house is.
PTRATIO - Pupil-teacher ratio by tow school district. Lower ration should impy better more individual student attention.
NOX - The amount of Nitric Oxide in parts per 10 million in the zone, one of the measures of how polluted the zone is.
Question 2
Using your client's feature set CLIENT_FEATURES, which values correspond with the features you've chosen above?
Hint:  Run the code block below to see the client's data.
Can you write Python code for it?


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Question 4
Which performance metric below did you find was most appropriate for predicting housing prices and analyzing the total error. Why?
- Accuracy
- Precision
- Recall
- F1 Score
- Mean Squared Error (MSE)
- Mean Absolute Error (MAE)
Answer:
This project is about the prediction of housing prices and is most certainly a problem of regression. i.e. we intend to build a model that can successfully predict the value of a continuous variable viz. housing_prices, given the CLIENT_FEATURES. Regression Models are assessed by measuring the error between the observed values and the predicted. Although the measures of this error vary all regression models must have the small and unbaised error.
Among these measures of error the Mean Squared Error(RMSE) has certain properties that can be leveraged for the purpose of this project.
These include:
1. MSE is an absolute measure of fit and not a relative measure.
2. MSE is in square units of the response variable.
3. MSE penalizes and amplifies larger errors more than small errors.
Step 4 (Final Step)
In the code block below, you will need to implement code so that the fit_model function does the following:
- Create a scoring function using the same performance metric as in Step 2. See the sklearn make_scorer documentation.
- Build a GridSearchCV object using regressor, parameters, and scoring_function. See the sklearn documentation on GridSearchCV.
When building the scoring function and GridSearchCV object, be sure that you read the parameters documentation thoroughly. It is not always the case that a default parameter for a function is the appropriate setting for the problem you are working on.
Since you are using sklearn functions, remember to include the necessary import statements below as well!
Ensure that you have executed the code block once you are done. You'll know if the fit_model function is working if the statement "Successfully fit a model to the data!" is printed.
Can you write Python code for it?

# Put any import statements you need for this code block
from sklearn.metrics import make_scorer
from sklearn.grid_search import GridSearchCV

def fit_model(X, y):
    """ Tunes a decision tree regressor model using GridSearchCV on the input data X
        and target labels y and returns this optimal model. """

    # Create a decision tree regressor object
    regressor = DecisionTreeRegressor(random_state=42)

    # Set up the parameters we wish to tune
    parameters = {'max_depth':[1,2,3,4,5,6,7,8,9,10]}

    # Make an appropriate scoring function
    scoring_function = make_scorer(performance_metric,greater_is_better=False)

    # Make the GridSearchCV object
    reg = GridSearchCV(regressor,parameters,scoring=scoring_function,n_jobs=4)

    # Fit the learner to the data to obtain the optimal model with tuned parameters
    reg.fit(X, y)

    # Return the optimal model
    return reg

# Test fit_model on entire dataset
try:
    reg = fit_model(housing_features, housing_prices)
    print "Successfully fit a model!"
except:
    print "Something went wrong with fitting a model."


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Question 5
What is the grid search algorithm and when is it applicable?
Answer:
Any predictive model built has a set of hyperparameters(the constants, ex. max_depth, max_nodes etc) that results in the best scoring metric acheivable by the predictor given a set of training parameters. Since, the optimal set of hyperparameters is a subset of the cross-product of all the hyperparameter sets, manually finding the optimal hyperparameters can be time-consuming, tedious and is often an iterative process.
The Grid Search algorithm solves the above complication  by forming a grid of hyperparameter sets with each point in the grid representing a unique subset of the cross product. The algorithm takes the following inputs; a predictor instance, the sets of various hyperparameters, a scoring metric. Using these inputs the algorithm exhaustively searches for the set of optimal hyperparametes in the grid by applying the hyperparameter set at each point to the predictor instance and measuring the scoring metric. This ensures that we have a predictor that is tuned to use the optimal set of parameters in ur desired set of hyper parameters.
Question 6
What is cross-validation, and how is it performed on a model? Why would cross-validation be helpful when using grid search?
Answer:
As discussed in the test-train split section, when building predictive models one splits the given dataset into training and testing subsets to validate the models predictions. This leads to the following dilemma:
While one one hand, we wish to maximize the training dataset to build a predictive model that best suits the given data on the other, we wish to maximize the testing dataset to better assert the performance of the model. We are  thus forced to compromise on the size of either and chose an optimal split between the training and the testing dataset. How do we maximize the training and the testing samples such that we are able to maximize both the testing and the training dataset? This is where cross-validation help us. The technique...
Can you write Python code for it?

def learning_curves(X_train, y_train, X_test, y_test):
    """ Calculates the performance of several models with varying sizes of training data.
        The learning and testing error rates for each model are then plotted. """

    print "Creating learning curve graphs for max_depths of 1, 3, 6, and 10. . ."

    # Create the figure window
    fig = pl.figure(figsize=(10,8))

    # We will vary the training set size so that we have 50 different sizes
    sizes = np.round(np.linspace(1, len(X_train), 50))
    train_err = np.zeros(len(sizes))
    test_err = np.zeros(len(sizes))

    # Create four different models based on max_depth
    for k, depth in enumerate([1,3,6,10]):

        for i, s in enumerate(sizes):

            # Setup a decision tree regressor so that it learns a tree with max_depth = depth
            regressor = DecisionTreeRegressor(max_depth = depth)

            # Fit the learner to the training data
            regressor.fit(X_train[:s], y_train[:s])

            # Find the performance on the training set
            train_err[i] = performance_metric(y_train[:s], regressor.predict(X_train[:s]))

            # Find the performance on the testing set
            test_err[i] = performance_metric(y_test, regressor.predict(X_test))

        # Subplot the learning curve graph
        ax = fig.add_subplot(2, 2, k+1)
        ax.plot(sizes, test_err, lw = 2, label = 'Testing Error')
        ax.plot(sizes, train_err, lw = 2, label = 'Training Error')
        ax.legend()
        ax.set_title('max_depth = %s'%(depth))
        ax.set_xlabel('Number of Data Points in Training Set')
        ax.set_ylabel('Total Error')
        ax.set_xlim([0, len(X_train)])

    # Visual aesthetics
    fig.suptitle('Decision Tree Regressor Learning Performances', fontsize=18, y=1.03)
    fig.tight_layout()
    fig.show()

def model_complexity(X_train, y_train, X_test, y_test):
    """ Calculates the performance of the model as model complexity increases.
        The...


I am working on the file "P1:_boston_housing/boston_housing.ipynb".
The first task is:
Statistical Analysis and Data Exploration
In this first section of the project, you will quickly investigate a few basic statistics about the dataset you are working with. In addition, you'll look at the client's feature set in CLIENT_FEATURES and see how this particular sample relates to the features of the dataset. Familiarizing yourself with the data through an explorative process is a fundamental practice to help you better understand your results.
Step 1
In the code block below, use the imported numpy library to calculate the requested statistics. You will need to replace each None you find with the appropriate numpy coding for the proper statistic to be printed. Be sure to execute the code block each time to test if your implementation is working successfully. The print statements will show the statistics you calculate!
Can you write Python code for it?

# Number of houses in the dataset
total_houses = housing_features.shape[0]

# Number of features in the dataset
total_features = housing_features.shape[1]

# Minimum housing value in the dataset
minimum_price = housing_prices.min()

# Maximum housing value in the dataset
maximum_price = housing_prices.max()

# Mean house value of the dataset
mean_price = housing_prices.mean()

# Median house value of the dataset
median_price = np.median(housing_prices)

# Standard deviation of housing values of the dataset
std_dev = housing_prices.std()

# Show the calculated statistics
print "Boston Housing dataset statistics (in $1000's):\n"
print "Total number of houses:", total_houses
print "Total number of features:", total_features
print "Minimum house price:", minimum_price
print "Maximum house price:", maximum_price
print "Mean house price: {0:.3f}".format(mean_price)
print "Median house price:", median_price
print "Standard deviation of house price: {0:.3f}".format(std_dev)


I am working on the file "P1_Predicting_Boston_Housing_Prices/P1_boston_housing.ipynb".
The first task is:
Answer:

For the CLIENT_FEATURES dataset:

"CRIM" = 11.95
"RM" = 5.609
"LSTAT" = 12.13

Evaluating Model Performance
In this second section of the project, you will begin to develop the tools necessary for a model to make a prediction. Being able to accurately evaluate each model's performance through the use of these tools helps to greatly reinforce the confidence in your predictions.
Step 2
In the code block below, you will need to implement code so that the shuffle_split_data function does the following:
- Randomly shuffle the input data X and target labels (housing values) y.
- Split the data into training and testing subsets, holding 30% of the data for testing.
If you use any functions not already acessible from the imported libraries above, remember to include your import statement below as well!
Ensure that you have executed the code block once you are done. You'll know the shuffle_split_data function is working if the statement "Successfully shuffled and split the data!" is printed.
Can you write Python code for it?

# Put any import statements you need for this code block here

def shuffle_split_data(X, y):
    """ Shuffles and splits data into 70% training and 30% testing subsets,
        then returns the training and testing subsets. """

    # create dataframe on X
    X_df = pd.DataFrame(X)

    # Shuffle the data
    np.random.seed(123)
    indices_shuffled = np.random.permutation(X_df.index)
    X_df = X_df.reindex(indices_shuffled)
    y = y[indices_shuffled]

    # Split the data
    percent_train = 0.7
    index_split = np.floor(X_df.shape[0] * percent_train).astype(int)

    X_train = X_df.values[:index_split, :]
    y_train = y[:index_split]
    X_test = X_df.values[index_split:, :]
    y_test = y[index_split:]

    # Return the training and testing data subsets
    return X_train, y_train, X_test, y_test

# Test shuffle_split_data
try:
    X_train, y_train, X_test, y_test = shuffle_split_data(housing_features, housing_prices)
    print "Successfully shuffled and split the data!"
except:
    print "Something went wrong with shuffling and splitting the data."


I am working on the file "P1_Predicting_Boston_Housing_Prices/P1_boston_housing.ipynb".
The first task is:
Question 4
Which performance metric below did you find was most appropriate for predicting housing prices and analyzing the total error. Why?
- Accuracy
- Precision
- Recall
- F1 Score
- Mean Squared Error (MSE)
- Mean Absolute Error (MAE)
Answer:

Since predicting the housing price is a regression problem, thus the relevant metrics are MSE and MAE. Both metrics are valid measure of total error. In this case, MSE is preferable because samples further away from model predictions are assigned quadratic weight and regressors minimizing MSE would lead to a unique solution as opposed to many possible solutions with MAE. Inccidentally, sklearn's Decision Tree Regressor used in subsequent problems also utilize MSE as the criterion.
Accuracy, precision, recall and F1 score would be useful when evaluating models performing classification tasks.

Step 4 (Final Step)
In the code block below, you will need to implement code so that the fit_model function does the following:
- Create a scoring function using the same performance metric as in Step 2. See the sklearn make_scorer documentation.
- Build a GridSearchCV object using regressor, parameters, and scoring_function. See the sklearn documentation on GridSearchCV.
When building the scoring function and GridSearchCV object, be sure that you read the parameters documentation thoroughly. It is not always the case that a default parameter for a function is the appropriate setting for the problem you are working on.
Since you are using sklearn functions, remember to include the necessary import statements below as well!
Ensure that you have executed the code block once you are done. You'll know the fit_model function is working if the statement "Successfully fit a model to the data!" is printed.
Can you write Python code for it?

# Put any import statements you need for this code block
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import make_scorer

def fit_model(X, y):
    """ Tunes a decision tree regressor model using GridSearchCV on the input data X
        and target labels y and returns this optimal model. """

    # Create a decision tree regressor object
    regressor = DecisionTreeRegressor(random_state=0)

    # Set up the parameters we wish to tune
    parameters = {'max_depth': (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15),
                  'min_samples_split': (2, 3, 4, 5, 6, 7, 8, 9, 10),
                  'max_features': ['auto', 'sqrt', 'log2']}

    # Make an appropriate scoring function
    scoring_function = make_scorer(score_func=performance_metric, greater_is_better=False)

    # Make the GridSearchCV object
    reg = GridSearchCV(estimator=regressor,
                       param_grid=parameters,
                       scoring=scoring_function)

    # Fit the learner to the data to obtain the optimal model with tuned parameters
    reg.fit(X, y)

    # Return the optimal model
    return reg.best_estimator_

reg = fit_model(housing_features, housing_prices)
# Test fit_model on entire dataset
try:
    reg = fit_model(housing_features, housing_prices)
    print "Successfully fit a model!"
except:
    print "Something went wrong with fitting a model."


I am working on the file "P1_Predicting_Boston_Housing_Prices/P1_boston_housing.ipynb".
The first task is:
Question 5
What is the grid search algorithm and when is it applicable?
Answer:

Grid search refers to systematically working through multiple combinations of hyperparameters in order to achieve optimal model performance according to specified model evaluation metrics. It's applicable when the model requires hyperparameter inputs from users, such as the 'max_depth' parameter in Decision Tree Regressor.

Question 6
What is cross-validation, and how is it performed on a model? Why would cross-validation be helpful when using grid search?
Answer:

Cross-validation is a strategy to assess how the reporting metrics such as MSE or accuracy would generalize onto unseen testing set. A popular choice is k-fold cross-validation, in which
1. the training set is divided into k subsets,
2. k-1 subsets are used for model training while the remaining subset is used as an internal testing sets to generate performance metrics,
3. repeat step number two k times with different subsets and report the average performance metrics for the chosen hyperparameters.
During grid search, there's chance that the chosen hyperparameters would lead to minimal errors on the training set but fail to generalize to unseen data. Cross-validation can make the reported performance metrics more generalizable to the testing sets, thus help select the hyperparameters that decrease model overfitting.

Checkpoint!
You have now successfully completed your last code implementation section. Pat yourself on the back! All of your functions written above will be executed in the remaining sections below, and questions will be asked about various results for you to analyze. To prepare the Analysis and Prediction sections, you will need to intialize the two functions below. Remember, there's no need to implement any more code, so sit back and execute the code blocks! Some code comments are provided if you find yourself interested in the functionality.
Can you write Python code for it?

def learning_curves(X_train, y_train, X_test, y_test):
    """ Calculates the performance of several models with varying sizes of training data.
        The learning and testing error rates for each model are then plotted. """

    print "Creating learning curve graphs for max_depths of 1, 3, 6, and 10. . ."

    # Create the figure window
    fig = pl.figure(figsize=(10,8))

    # We will vary the training set size so that we have 50 different sizes
    sizes = np.rint(np.linspace(1, len(X_train), 50)).astype(int)
    train_err = np.zeros(len(sizes))
    test_err = np.zeros(len(sizes))

    # Create four different models based on max_depth
    for k, depth in enumerate([1,3,6,10]):

        for i, s in enumerate(sizes):

            # Setup a decision tree regressor so that it learns a tree with max_depth = depth
            regressor = DecisionTreeRegressor(max_depth = depth)

            # Fit the learner to the training data
            regressor.fit(X_train[:s], y_train[:s])

            # Find the performance on the training set
            train_err[i] = performance_metric(y_train[:s], regressor.predict(X_train[:s]))

            # Find the performance on the testing set
            test_err[i] = performance_metric(y_test, regressor.predict(X_test))

        # Subplot the learning curve graph
        ax = fig.add_subplot(2, 2, k+1)
        ax.plot(sizes, test_err, lw = 2, label = 'Testing Error')
        ax.plot(sizes, train_err, lw = 2, label = 'Training Error')
        ax.legend()
        ax.set_title('max_depth = %s'%(depth))
        ax.set_xlabel('Number of Data Points in Training Set')
        ax.set_ylabel('Total Error')
        ax.set_xlim([0, len(X_train)])

    # Visual aesthetics
    fig.suptitle('Decision Tree Regressor Learning Performances', fontsize=18, y=1.03)
    fig.tight_layout()
    # fig.show()

def model_complexity(X_train, y_train, X_test, y_test):
    """ Calculates the performance of the model as model complexity...


I am working on the file "P1_Predicting_Boston_Housing_Prices/P1_boston_housing.ipynb".
The first task is:
Question 7
Choose one of the learning curve graphs that are created above. What is the max depth for the chosen model? As the size of the training set increases, what happens to the training error? What happens to the testing error?
Answer:

For the upper left graph, the "max depth" is 1. As the training set increases, the training error rises from 0, fluctuates a bit in the beginning but eventually settles down to a relatively stable value. For the testing sets, the error starts out high and settles to a relatively stable value as number of data points increase in the training set.

Question 8
Look at the learning curve graphs for the model with a max depth of 1 and a max depth of 10. When the model is using the full training set, does it suffer from high bias or high variance when the max depth is 1? What about when the max depth is 10?
Answer:

When max_depth=1, the model suffers from high bias. The error levels for both training and testing sets are high even though the errors are similar, indicating underfitting.
When max_dpeth=10, the model suffers from high variance because of the discrepancy in training and testing errors, indicating overfitting.
Can you write Python code for it?

model_complexity(X_train, y_train, X_test, y_test)


I am working on the file "P1_Predicting_Boston_Housing_Prices/P1_boston_housing.ipynb".
The first task is:
Statistical Analysis and Data Exploration
In this first section of the project, you will quickly investigate a few basic statistics about the dataset you are working with. In addition, you'll look at the client's feature set in CLIENT_FEATURES and see how this particular sample relates to the features of the dataset. Familiarizing yourself with the data through an explorative process is a fundamental practice to help you better understand your results.
Step 1
In the code block below, use the imported numpy library to calculate the requested statistics. You will need to replace each None you find with the appropriate numpy coding for the proper statistic to be printed. Be sure to execute the code block each time to test if your implementation is working successfully. The print statements will show the statistics you calculate!
Can you write Python code for it?

# X -> features; y -> target (house value)
X = housing_features
y = housing_prices

# Number of houses in the dataset
total_houses = X.shape[0]

# Number of features in the dataset
total_features = X.shape[1]

# Minimum housing value in the dataset
minimum_price = np.min(y)

# Maximum housing value in the dataset
maximum_price = np.max(y)

# Mean house value of the dataset
mean_price = np.mean(y)

# Median house value of the dataset
median_price = np.median(y)

# Standard deviation of housing values of the dataset
std_dev = np.std(y, ddof=1)

# Show the calculated statistics
print "Boston Housing dataset statistics (in $1000's):\n"
print "Total number of houses:", total_houses
print "Total number of features:", total_features
print "Minimum house price:", minimum_price
print "Maximum house price:", maximum_price
print "Mean house price: {0:.3f}".format(mean_price)
print "Median house price:", median_price
print "Standard deviation of house price: {0:.3f}".format(std_dev)


I am working on the file "P1_SSG.ipynb".
The first task is:
Build a Lane Finding Pipeline
Build the pipeline and run your solution on all test_images. Make copies into the test_images_output directory, and you can use the images in your writeup report.
Try tuning the various parameters, especially the low and high Canny thresholds as well as the Hough lines parameters.
Can you write Python code for it?

# TODO: Build your pipeline that will draw lane lines on the test_images
# then save them to the test_images directory.
def image_pipe(img):
    gr_img = grayscale(img)
    ga_img = gaussian_blur(gr_img, 11)
    edges = canny(ga_img, 40, 50)
    imshape = img.shape
    vertices = np.array([[(0,imshape[0]),(460, 310), (500, 310), (imshape[1],imshape[0])]], dtype=np.int32)
    masked_img = region_of_interest(edges, vertices)
#     parameters of Hough grid
    rho = 2            #distance resolution
    theta = np.pi/180  #angular resolution
    threshold = 30     #min votes
    min_line_len = 40  #min line len
    max_line_gap = 20  #maximum gap between lines
    line_image = hough_lines(masked_img, rho, theta, threshold, min_line_len, max_line_gap)

    result = weighted_img(line_image, img)
    return result

img_path = "test_images/"
tst_imgs = os.listdir(img_path)
for tst_img in tst_imgs:
    img = mpimg.imread(img_path + tst_img)
    img = image_pipe(img)
    mpimg.imsave(img_path + 'processed_' + tst_img[:-3] + '.png', img)


I am working on the file "P1_SSG.ipynb".
The first task is:
Improve the draw_lines() function
At this point, if you were successful with making the pipeline and tuning parameters, you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform. As mentioned previously, try to average and/or extrapolate the line segments you've detected to map out the full extent of the lane lines. You can see an example of the result you're going for in the video "P1_example.mp4".
Go back and modify your draw_lines function accordingly and try re-running your pipeline. The new output should draw a single, solid line over the left lane line and a single, solid line over the right lane line. The lines should start from the bottom of the image and extend out to the top of the region of interest.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?

yellow_output = 'test_videos_output/solidYellowLeft.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))


I am working on the file "P1_SSG.ipynb".
The first task is:
Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.
Can you write Python code for it?

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))


I am working on the file "P1_SSG.ipynb".
The first task is:
Read in an Image
Can you write Python code for it?

#reading in an image
image = mpimg.imread('test_images/solidWhiteRight.jpg')

#printing out some stats and plotting
print('This image is:', type(image), 'with dimensions:', image.shape)
plt.imshow(image)  # if you wanted to show a single color channel image called 'gray', for example, call as plt.imshow(gray, cmap='gray')


I am working on the file "P1_SSG.ipynb".
The first task is:
Writeup and Submission
If you're satisfied with your video outputs, it's time to make the report writeup in a pdf or markdown file. Once you have this Ipython notebook ready along with the writeup, it's time to submit for review! Here is a link to the writeup template file.
Optional Challenge
Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
Can you write Python code for it?

challenge_output = 'test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,5)
clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Analyzing Model Performance
In this third section of the project, you'll take a look at several models' learning and testing error rates on various subsets of training data. Additionally, you'll investigate one particular algorithm with an increasing max_depth parameter on the full training set to observe how model complexity affects learning and testing errors. Graphing your model's performance based on varying criteria can be beneficial in the analysis process, such as visualizing behavior that may not have been apparent from the results alone.
Can you write Python code for it?


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Answer:  The recommended prices are as in the print-out:
 - Predicted selling price for Client 1's home: \$403,025.00
 - Predicted selling price for Client 2's home: \$237,478.72
 - Predicted selling price for Client 3's home: \$931,636.36
All houses fall within the maximum and minimum values, and are within about 2 standard deviations of the mean. Also, the correlations between price and number of rooms, poverty level, and student-teacher ratio all are as expected.
Furthermore, as can be seen from the plots below, the predicted values fall nicely inside the clear patters of housing prices, for each of the featues:
Can you write Python code for it?

import matplotlib.pyplot as plt

client_data = np.array(client_data)
client_predictions = reg.predict(client_data)

plt.scatter(features["RM"], prices, alpha=0.3)
plt.scatter(client_data[:,0], client_predictions, alpha=1, c="black")
plt.xlabel("Number of rooms")
plt.ylabel("Market price")
plt.show()

plt.scatter(features["LSTAT"], prices, alpha=0.3)
plt.scatter(client_data[:,1], client_predictions, alpha=1, c="black")
plt.xlabel("Neighborhood poverty")
plt.ylabel("Market price")
plt.show()

plt.scatter(features["PTRATIO"], prices, alpha=0.3)
plt.scatter(client_data[:,2], client_predictions, alpha=1, c="black")
plt.xlabel("Student-teacher ratio")
plt.ylabel("Market price")
plt.show()


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Answer:  max_depth = 4. This is consistent with my discussion in question 6.
Question 10 - Predicting Selling Prices
Imagine that you were a real estate agent in the Boston area looking to use this model to help price homes owned by your clients that they wish to sell. You have collected the following information from three of your clients:
| Feature | Client 1 | Client 2 | Client 3 |
| :---: | :---: | :---: | :---: |
| Total number of rooms in home | 5 rooms | 4 rooms | 8 rooms |
| Neighborhood poverty level (as %) | 17% | 32% | 3% |
| Student-teacher ratio of nearby schools | 15-to-1 | 22-to-1 | 12-to-1 |
What price would you recommend each client sell his/her home at? Do these prices seem reasonable given the values for the respective features?
Hint: Use the statistics you calculated in the Data Exploration section to help justify your response.
Run the code block below to have your optimized model make predictions for each client's home.
Can you write Python code for it?

# Produce a matrix for client data
client_data = [[5, 17, 15], # Client 1
               [4, 32, 22], # Client 2
               [8, 3, 12]]  # Client 3

# Show predictions
for i, price in enumerate(reg.predict(client_data)):
    print "Predicted selling price for Client {}'s home: ${:,.2f}".format(i+1, price)


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Data Exploration
In this first section of this project, you will make a cursory investigation about the Boston housing data and provide your observations. Familiarizing yourself with the data through an explorative process is a fundamental practice to help you better understand and justify your results.
Since the main goal of this project is to construct a working model which has the capability of predicting the value of houses, we will need to separate the dataset into features and the target variable. The features, 'RM', 'LSTAT', and 'PTRATIO', give us quantitative information about each data point. The target variable, 'MEDV', will be the variable we seek to predict. These are stored in features and prices, respectively.
Implementation: Calculate Statistics
For your very first coding implementation, you will calculate descriptive statistics about the Boston housing prices. Since numpy has already been imported for you, use this library to perform the necessary calculations. These statistics will be extremely important later on to analyze various prediction results from the constructed model.
In the code cell below, you will need to implement the following:
- Calculate the minimum, maximum, mean, median, and standard deviation of 'MEDV', which is stored in prices.
  - Store each calculation in their respective variable.
Can you write Python code for it?

# TODO: Minimum price of the data
minimum_price = np.min(prices)

# TODO: Maximum price of the data
maximum_price = np.max(prices)

# TODO: Mean price of the data
mean_price = np.mean(prices)

# TODO: Median price of the data
median_price = np.median(prices)

# TODO: Standard deviation of prices of the data
std_price = np.std(prices)

# Show the calculated statistics
print "Statistics for Boston housing dataset:\n"
print "Minimum price: ${:,.2f}".format(minimum_price)
print "Maximum price: ${:,.2f}".format(maximum_price)
print "Mean price: ${:,.2f}".format(mean_price)
print "Median price ${:,.2f}".format(median_price)
print "Standard deviation of prices: ${:,.2f}".format(std_price)


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
From the plots above we can see that especially for the "LSTAT" variable our predictions are very well placed within the trends. With regards to the number of rooms, the house priced at \$403,025 is a little above the trend, but it is below the trend when looking at the student-teacher ratio; it is likely the two effects are balancing out and yield the current output of approximately \$400,000.
Sensitivity
An optimal model is not necessarily a robust model. Sometimes, a model is either too complex or too simple to sufficiently generalize to new data. Sometimes, a model could use a learning algorithm that is not appropriate for the structure of the data given. Other times, the data itself could be too noisy or contain too few samples to allow a model to adequately capture the target variable — i.e., the model is underfitted. Run the code cell below to run the fit_model function ten times with different training and testing sets to see how the prediction for a specific client changes with the data it's trained on.
Can you write Python code for it?

vs.PredictTrials(features, prices, fit_model, client_data)


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 1
As a reminder, you can view a description of the Boston Housing dataset here, where you can find the different features under Attribute Information. The MEDV attribute relates to the values stored in our housing_prices variable, so we do not consider that a feature of the data.
Of the features available for each data point, choose three that you feel are significant and give a brief description for each of what they measure.
Remember, you can double click the text box below to add your answer!
Answer:
1. CRIM: per capita crime rate by town, indicating surrounding security environment.
2. RM: average number of rooms per dwelling, indicating the level of the house.
3. PTRATIO: pupil-teacher ratio by town, indicating the primary education quality.
Question 2
Using your client's feature set CLIENT_FEATURES, which values correspond with the features you've chosen above?
Hint:  Run the code block below to see the client's data.
Can you write Python code for it?


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 2 - Goodness of Fit
Assume that a dataset contains five data points and a model made the following predictions for the target variable:
| True Value | Prediction |
| :-------------: | :--------: |
| 3.0 | 2.5 |
| -0.5 | 0.0 |
| 2.0 | 2.1 |
| 7.0 | 7.8 |
| 4.2 | 5.3 |
Would you consider this model to have successfully captured the variation of the target variable? Why or why not?
Run the code cell below to use the performance_metric function and calculate this model's coefficient of determination.
Can you write Python code for it?

# Calculate the performance of this model
score = performance_metric([3, -0.5, 2, 7, 4.2], [2.5, 0.0, 2.1, 7.8, 5.3])
print "Model has a coefficient of determination, R^2, of {:.3f}.".format(score)


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 4 - Learning the Data
Choose one of the graphs above and state the maximum depth for the model. What happens to the score of the training curve as more training points are added? What about the testing curve? Would having more training points benefit the model?
Hint: Are the learning curves converging to particular scores?
Answer:  Looking at the graph for max_depth=3, we see that the more training points we have, the lower the score on the training set. This is simply due to the fact that it is hard to achieve "pure" end nodes in the tree with more data; conversely, when we have little data it's easy to tune the splits of the tree such that all end-nodes are all labeled with a single label. This is clearly an overfitting strategy.
As we obtain more data, we are forced to tune the splits of the decision tree more carefully, in order to account for an ever more varied dataset. Therefore, the testing score improves with more data, since the model is now able to better capture more "general" situations than when it overfit.
More training points will no longer benefit the model; it has more or less maximized the amount of variance it is able to capture.
Complexity Curves
The following code cell produces a graph for a decision tree model that has been trained and validated on the training data using different maximum depths. The graph produces two complexity curves — one for training and one for validation. Similar to the learning curves, the shaded regions of both the complexity curves denote the uncertainty in those curves, and the model is scored on both the training and validation sets using the performance_metric function.
Run the code cell below and use this graph to answer the following two questions.
Can you write Python code for it?


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 5
What is the grid search algorithm and when is it applicable?
Answer:
The grid search algorithm exhaustively generates candidates from a grid of parameter values specified with the param_grid parameter. All the possible combinations of parameter values are evaluated and the best combination is retained.
When an objective metric or score function is chosen and the parameter space is reasonably small, it's applicable.
Question 6
What is cross-validation, and how is it performed on a model? Why would cross-validation be helpful when using grid search?
Answer:
Cross-validation is a model validation technique for assessing how the results of a statistical analysis will generalized to an independent data set.
It is recommended to shuffle the input samples prior to cross-validation in order to randomize any localized patterns found in the input samples. One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.
Though cross validation with grid search would increase training time but would increase the model's robustness, and the hyper-parameters are optimized properly without being tied to a specific training or test sets. So it can derive a more accurate estimate of model prediction performance.
Checkpoint!
You have now successfully completed your last code implementation section. Pat yourself on the back! All of your functions written above will be executed in the remaining sections below, and questions will be asked about various results for you to analyze. To prepare the Analysis and Prediction sections, you will need to intialize the two functions below. Remember, there's no need to implement any more code, so sit back and execute the code...
Can you write Python code for it?

def learning_curves(X_train, y_train, X_test, y_test):
    """ Calculates the performance of several models with varying sizes of training data.
        The learning and testing error rates for each model are then plotted. """

    print "Creating learning curve graphs for max_depths of 1, 3, 6, and 10. . ."

    # Create the figure window
    fig = pl.figure(figsize=(10,8))

    # We will vary the training set size so that we have 50 different sizes
    sizes = np.rint(np.linspace(1, len(X_train), 50)).astype(int)
    train_err = np.zeros(len(sizes))
    test_err = np.zeros(len(sizes))

    # Create four different models based on max_depth
    for k, depth in enumerate([1,3,6,10]):

        for i, s in enumerate(sizes):

            # Setup a decision tree regressor so that it learns a tree with max_depth = depth
            regressor = DecisionTreeRegressor(max_depth = depth)

            # Fit the learner to the training data
            regressor.fit(X_train[:s], y_train[:s])

            # Find the performance on the training set
            train_err[i] = performance_metric(y_train[:s], regressor.predict(X_train[:s]))

            # Find the performance on the testing set
            test_err[i] = performance_metric(y_test, regressor.predict(X_test))

        # Subplot the learning curve graph
        ax = fig.add_subplot(2, 2, k+1)
        ax.plot(sizes, test_err, lw = 2, label = 'Testing Error')
        ax.plot(sizes, train_err, lw = 2, label = 'Training Error')
        ax.legend()
        ax.set_title('max_depth = %s'%(depth))
        ax.set_xlabel('Number of Data Points in Training Set')
        ax.set_ylabel('Total Error')
        ax.set_xlim([0, len(X_train)])

    # Visual aesthetics
    fig.suptitle('Decision Tree Regressor Learning Performances', fontsize=18, y=1.03)
    fig.tight_layout()
    fig.show()

def model_complexity(X_train, y_train, X_test, y_test):
    """ Calculates the performance of the model as model complexity increases.
 ...


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 7
Choose one of the learning curve graphs that are created above. What is the max depth for the chosen model? As the size of the training set increases, what happens to the training error? What happens to the testing error?
Answer:
The third graph is chosen, and max_depth = 6.
As the size of the training set increases, the training error increases, but the testing error decreases dramatically when the number of the data points < 50 and then levels off.
Question 8
Look at the learning curve graphs for the model with a max depth of 1 and a max depth of 10. When the model is using the full training set, does it suffer from high bias or high variance when the max depth is 1? What about when the max depth is 10?
Answer:
When the model is using the full training set, for the max depth is 1, two error curves converge into a high error value, which means it suffers from high bias; for the max depth is 10, the training curve remains a quite low value but the difference between the training and hihg testing curves is a huge gap, which indicating that it suffers from high variance.
Can you write Python code for it?


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Question 9
From the model complexity graph above, describe the training and testing errors as the max depth increases. Based on your interpretation of the graph, which max depth results in a model that best generalizes the dataset? Why?
Answer:
As the max depth increases, the training and testing errors decreases which means lower bias, but the difference between training and testing errors increases which indicates higher variance, i.e. Bias-Variance Tradeoff.
So the optimum max depth would be ~5, where the testing error is low and the model remains its simplicity.
Model Prediction
In this final section of the project, you will make a prediction on the client's feature set using an optimized model from fit_model. When applying grid search along with cross-validation to optimize your model, it would typically be performed and validated on a training set and subsequently evaluated on a dedicated test set. In this project, the optimization below is performed on the entire dataset (as opposed to the training set you made above) due to the many outliers in the data. Using the entire dataset for training provides for a less volatile prediction at the expense of not testing your model's performance.
To answer the following questions, it is recommended that you run the code blocks several times and use the median or mean value of the results.
Question 10
Using grid search on the entire dataset, what is the optimal max_depth parameter for your model? How does this result compare to your intial intuition?
Hint:  Run the code block below to see the max depth produced by your optimized model.
Can you write Python code for it?

print "Final model has an optimal max_depth parameter of", reg.get_params()['max_depth']


I am working on the file "P1_boston_housing/boston_housing.ipynb".
The first task is:
Statistical Analysis and Data Exploration
In this first section of the project, you will quickly investigate a few basic statistics about the dataset you are working with. In addition, you'll look at the client's feature set in CLIENT_FEATURES and see how this particular sample relates to the features of the dataset. Familiarizing yourself with the data through an explorative process is a fundamental practice to help you better understand your results.
Step 1
In the code block below, use the imported numpy library to calculate the requested statistics. You will need to replace each None you find with the appropriate numpy coding for the proper statistic to be printed. Be sure to execute the code block each time to test if your implementation is working successfully. The print statements will show the statistics you calculate!
Can you write Python code for it?

# Number of houses in the dataset
total_houses = housing_features.shape[0]

# Number of features in the dataset
total_features = housing_features.shape[1]

# Minimum housing value in the dataset
minimum_price = housing_prices.min()

# Maximum housing value in the dataset
maximum_price = housing_prices.max()

# Mean house value of the dataset
mean_price = housing_prices.mean()

# Median house value of the dataset
median_price = np.median(housing_prices)

# Standard deviation of housing values of the dataset
std_dev = housing_prices.std()

# Show the calculated statistics
print "Boston Housing dataset statistics (in $1000's):\n"
print "Total number of houses:", total_houses
print "Total number of features:", total_features
print "Minimum house price:", minimum_price
print "Maximum house price:", maximum_price
print "Mean house price: {0:.3f}".format(mean_price)
print "Median house price:", median_price
print "Standard deviation of house price: {0:.3f}".format(std_dev)


I am working on the file "P2 Statistical Significance/.ipynb_checkpoints/P2_Analyzing_the_NYC_Subway_Dataset-checkpoint.ipynb".
The first task is:
2.1 What approach did you use to compute the coefficients theta and produce prediction for ENTRIESn_hourly in your regression model:
OLS using Statsmodels.
2.2 What features (input variables) did you use in your model? Did you use any dummy variables as part of your features?
Input Variables:
* temperature
* precipitation
* mean precipitation
* mean temperature
* pressure
* mean pressure
* wind speed
* mean wind spead
Dummy Variables:
* Unit
* Weather Conditions
* Hour of day
* Day of Week
2.3 Why did you select these features in your model? We are looking for specific reasons that lead you to believe that the selected features will contribute to the predictive power of your model.
Features were selected in the beginning solely by intuition. For any features with categorical values, dummy values were used.
By investigating the values in the weather conditions column, I found that the indicators for fog, rain were not necessary since the weather condition column already captured that in one of the categories.
The hours feature was later converted into a categorical input since I suspected that a coefficient could not capture its effect on the the fluctuations in the number of hourly entries.
It was the same suspicion for adding the day_week feature as a dummy variable too, rendering the weekday indicator obsolete.
2.4 What are the parameters (also known as "coefficients" or "weights") of the non-dummy features in your linear regression model?
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/.ipynb_checkpoints/P2_Analyzing_the_NYC_Subway_Dataset-checkpoint.ipynb".
The first task is:
2.6 What does this R2 value mean for the goodness of fit for your regression model? Do you think this linear model to predict ridership is appropriate for this dataset, given this R2  value?
Our $r^2$ value tells us that {{r_squared_pct}}% of the variation of hourly entries for the NYC subway is explained by the features provided. It indicates that there is a relationship between the features listed above and NYC ridership.
Can you write Python code for it?

def plot_residuals(turnstile_weather, predictions):
    turnstile_weather["Hourly Entries Residuals"] = turnstile_weather["ENTRIESn_hourly"] - predictions
    cropped = turnstile_weather[(turnstile_weather["Hourly Entries Residuals"] > -5000) & (turnstile_weather["Hourly Entries Residuals"] < 5000)]
    plt = gp.ggplot(gp.aes(x="Hourly Entries Residuals"), data=cropped) + gp.geom_histogram(binwidth=100)
    return plt

print(plot_residuals(df, predictions))
entries_mean = round(df['ENTRIESn_hourly'].mean(),2)


I am working on the file "P2 Statistical Significance/.ipynb_checkpoints/P2_Analyzing_the_NYC_Subway_Dataset-checkpoint.ipynb".
The first task is:
3.2 One visualization can be more freeform. You should feel free to implement something that we discussed in class (e.g., scatter plots, line plots) or attempt to implement something more advanced if you'd like. Some suggestions are:

Ridership by time-of-day
Ridership by day-of-week
Can you write Python code for it?

grouped = df[["hour","ENTRIESn_hourly"]].groupby("hour")
ridership_hour = grouped.aggregate(np.mean)
ridership_hour = ridership_hour.reset_index()
gp.ggplot(ridership_hour, gp.aes('hour', weight="ENTRIESn_hourly")) + \
    gp.geom_bar(binwidth=4) + \
    gp.scale_y_continuous(labels='comma') + \
    gp.scale_x_continuous(limits=(0,20)) + \
    gp.ggtitle('Average Ridership by Hour of the Day')

def convertDays(x):
    if x == 0:
        return "Mon"
    elif x == 1:
        return "Tues"
    elif x == 2:
        return "Wed"
    elif x == 3:
        return "Thurs"
    elif x == 4:
        return "Fri"
    elif x == 5:
        return "Sat"
    elif x == 6:
        return "Sun"

grouped = df[["day_week","ENTRIESn_hourly"]].groupby("day_week")
ridership_day = grouped.aggregate(np.mean)
ridership_day = ridership_day.reset_index()

ridership_day["day_week"] = ridership_day["day_week"].map(convertDays)

gp.ggplot(ridership_day, gp.aes(x="day_week", y="ENTRIESn_hourly")) +\
    gp.geom_bar(stat='identity') +\
    gp.ggtitle('Average Ridership by Day of Week')


I am working on the file "P2 Statistical Significance/.ipynb_checkpoints/P2_Analyzing_the_NYC_Subway_Dataset-checkpoint.ipynb".
The first task is:
Analyzing the NYC Subway Dataset
Section 0. References
Mann-Whitney U
* https://storage.googleapis.com/supplemental_media/udacityu/ud359-nd/Mann-Whitney%20U%20Test.pdf
* https://statistics.laerd.com/premium-sample/mwut/mann-whitney-test-in-spss-2.php
Two-tailed vs One-tailed
* http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests.htm
Section 1. Statistical Test
1.1 Which statistical test did you use to analyze the NYC subway data? Did you use a one-tail or a two-tail P value? What is the null hypothesis? What is your p-critical value?
The Mann-Whitney U-Test was used to analyze the NYC subway data. A two-tail P value was used because due to the uncertainty of whether rain increases or decreases the number of people using the subway. The Null Hypothesis used to compare between the two populations, rainy days and non rainy days is as follows:
$H_0$: The distribution of hourly entries are the same for the two populations.
$H_A$: The distribution of hourly entries are different for the two populations.
Justification for this hypothesis will be explained in the next question. A two tailed p-critical value of .05 will be used.
1.2 Why is this statistical test applicable to the dataset? In particular, consider the assumptions that the test is making about the distribution of ridership in the two samples.
In order to choose an appropriate statistical test for the dataset, the shape of the distributions needed to be graphically shown:
Can you write Python code for it?

df = pd.read_csv("./turnstile_weather_v2.csv")

df_rain = df[df['rain'] == 1]
df_no_rain = df[df['rain'] == 0]

df_rain_entries = df_rain.ENTRIESn_hourly
df_no_rain_entries = df_no_rain.ENTRIESn_hourly

gp.ggplot(gp.aes(x="ENTRIESn_hourly", fill='rain',color='rain'), data=df) +\
    gp.geom_histogram(alpha=0.6, binwidth=1000)


I am working on the file "P2 Statistical Significance/.ipynb_checkpoints/P2_Analyzing_the_NYC_Subway_Dataset-checkpoint.ipynb".
The first task is:
By looking at the histogram we see that the datasets seem to not follow a normal distribution. We can verify this statistically with the Shapiro Wilk's Test.
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
2.1 What approach did you use to compute the coefficients theta and produce prediction for ENTRIESn_hourly in your regression model:
OLS using Statsmodels.
2.2 What features (input variables) did you use in your model? Did you use any dummy variables as part of your features?
Input Variables:
* temperature
* precipitation
* mean precipitation
* mean temperature
* pressure
* mean pressure
* wind speed
* mean wind spead
Dummy Variables:
* Unit
* Weather Conditions
* Hour of day
* Day of Week
2.3 Why did you select these features in your model? We are looking for specific reasons that lead you to believe that the selected features will contribute to the predictive power of your model.
Features were selected in the beginning solely by intuition. For any features with categorical values, dummy values were used.
By investigating the values in the weather conditions column, I found that the indicators for fog, rain were not necessary since the weather condition column already captured that in one of the categories.
The hours feature was later converted into a categorical input since I suspected that a coefficient could not capture its effect on the the fluctuations in the number of hourly entries.
It was the same suspicion for adding the day_week feature as a dummy variable too, rendering the weekday indicator obsolete.
2.4 What are the parameters (also known as "coefficients" or "weights") of the non-dummy features in your linear regression model?
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
2.5 What is your model’s $R^2$ (coefficients of determination) value?
Can you write Python code for it?

r_squared = round(results.rsquared,4)
r_squared_pct = r_squared * 100
r_squared


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
3.2 One visualization can be more freeform. You should feel free to implement something that we discussed in class (e.g., scatter plots, line plots) or attempt to implement something more advanced if you'd like. Some suggestions are:

Ridership by time-of-day
Ridership by day-of-week
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
Analyzing the NYC Subway Dataset
Section 0. References
Mann-Whitney U
* https://storage.googleapis.com/supplemental_media/udacityu/ud359-nd/Mann-Whitney%20U%20Test.pdf
* https://statistics.laerd.com/premium-sample/mwut/mann-whitney-test-in-spss-2.php
Two-tailed vs One-tailed
* http://www.ats.ucla.edu/stat/mult_pkg/faq/general/tail_tests.htm
Section 1. Statistical Test
1.1 Which statistical test did you use to analyze the NYC subway data? Did you use a one-tail or a two-tail P value? What is the null hypothesis? What is your p-critical value?
The Mann-Whitney U-Test was used to analyze the NYC subway data. A two-tail P value was used because due to the uncertainty of whether rain increases or decreases the number of people using the subway. The Null Hypothesis used to compare between the two populations, rainy days and non rainy days is as follows:
$H_0$: The distribution of hourly entries are the same for the two populations.
$H_A$: The distribution of hourly entries are different for the two populations.
Justification for this hypothesis will be explained in the next question. A two tailed p-critical value of .05 will be used.
1.2 Why is this statistical test applicable to the dataset? In particular, consider the assumptions that the test is making about the distribution of ridership in the two samples.
In order to choose an appropriate statistical test for the dataset, the shape of the distributions needed to be graphically shown:
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
By looking at the histogram we see that the datasets seem to not follow a normal distribution. We can verify this statistically with the Shapiro Wilk's Test.
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
Conditions non-dummy variable:
Can you write Python code for it?


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
Mean Hourly Entries while raining: {{with_rain_mean}}
Mean Hourly Entries while not raining: {{without_rain_mean}}
U Statistic: {{U}}
p-value: {{p}}
1.4 What is the significance and interpretation of these results?
Since our p-value of {{p}} is less than 0.05, we can reject the Null Hypothesis saying that the the two distributions are identical. We can conclude that rain does in fact play a factor for hourly entries in the NYC subway. Since the mean hourly entries while raining is greater than the mean hourly entries while not raining, we can also say that rain does in fact increase NYC Subway ridership.
Section 2. Linear Regression
Can you write Python code for it?

def linear_regression(features, values):
    features = sm.add_constant(features)
    model = sm.OLS(values, features)
    return model.fit()

def predictions(dataframe):
    features = dataframe[['tempi', 'precipi', 'meanprecipi', 'pressurei',
                          'meanpressurei', 'wspdi', 'meanwspdi']]
    dummy_units = pd.get_dummies(dataframe['UNIT'], prefix='unit')
    features = features.join(dummy_units)
    dummy_units = pd.get_dummies(dataframe['conds'], prefix='cond')
    features = features.join(dummy_units)
    dummy_units = pd.get_dummies(dataframe['hour'], prefix='hour')
    features = features.join(dummy_units)
    dummy_units = pd.get_dummies(dataframe['day_week'], prefix='day')
    features = features.join(dummy_units)

    # Values
    values = dataframe['ENTRIESn_hourly']

    # Perform linear regression
    results = linear_regression(features, values)
    intercept, params = results.params[0], results.params[1:]

    predictions = intercept + np.dot(features, params)
    return results, predictions

results, predictions = predictions(df)


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
Section 4. Conclusion
Please address the following questions in detail. Your answers should be 1-2 paragraphs long.
4.1 From your analysis and interpretation of the data, do more people ride the NYC subway when it is raining or when it is not raining?
Yes, I believe taht more people ride the NYC subway when it is raining.
4.2 What analyses lead you to this conclusion? You should use results from both your statistical tests and your linear regression to support your analysis.
The Mann-Whitney U test performed on the two datasets, ridership with rain and ridership without rain, showed that the datasets did not come from the same distribution. Along with a higher mean of ridership while it was raining, it can be concluded that people do ride the NYC subway more.
In addition to the Mann-Whitney U test, a linear regression was performed to see what other factors play a role on NYC ridership. With the coefficients calculated we can compare the coefficients related to non-rainy days and rainy days. As suspected the coefficient for the cond_Rain feature was much larger than cond_Clear feature.
Section 5. Reflection
Please address the following questions in detail. Your answers should be 1-2 paragraphs long.
5.1 Please discuss potential shortcomings of the methods of your analysis, including:
The dataset provided does not account for how much of a factor holidays and special events play in subway ridership. These should either be included as dummy variables or be taken out to take account of outliers.
5.2 (Optional) Do you have any other insight about the dataset that you would like to share with us?
From making the 'conds' column into a dummy variable, further insight can be seen in how more descriptive weather conditions play a role in subway ridership. By looking at the fog and mist condition's coefficient, it seems like they play a bigger role than rain in subway ridership.
Can you write Python code for it?

def css_styling():
    styles = open("../css/custom.css", "r").read()
    return disp.HTML(styles)
css_styling()


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
We can make a safe assumption that the populations are not normal after analyzing it visually and statistically.
In 1.1, the hypotheses are comparing the distributions of the two populations because we are making an assumption that the distributions are similarily shaped, as shown in the histogram above.
1.3 What results did you get from this statistical test? These should include the following numerical values: p-values, as well as the means for each of the two samples under test.
Can you write Python code for it?

with_rain_mean = round(df_rain_entries.mean(),2)
without_rain_mean = round(df_no_rain_entries.mean(),2)

U,p = st.mannwhitneyu(df_rain_entries, df_no_rain_entries)
p = round(p*2,7)


I am working on the file "P2 Statistical Significance/P2_Analyzing_the_NYC_Subway_Dataset.ipynb".
The first task is:
We can take a look at what the $r^2$ value represents by looking at the residuals. Majority of the residuals lie between -2000 to 2000, which is not good considering the mean is {{entries_mean}} for subway ridership. Therefore I think that this linear model is not enough to predict subway ridership.
Section 3. Visualization
Please include two visualizations that show the relationships between two or more variables in the NYC subway data. Remember to add appropriate titles and axes labels to your plots. Also, please add a short description below each figure commenting on the key insights depicted in the figure.
3.1 One visualization should contain two histograms: one of  ENTRIESn_hourly for rainy days and one of ENTRIESn_hourly for non-rainy days.

You can combine the two histograms in a single plot or you can use two separate plots.
If you decide to use to two separate plots for the two histograms, please ensure that the x-axis limits for both of the plots are identical. It is much easier to compare the two in that case.
For the histograms, you should have intervals representing the volume of ridership (value of ENTRIESn_hourly) on the x-axis and the frequency of occurrence on the y-axis. For example, each interval (along the x-axis), the height of the bar for this interval will represent the number of records (rows in our data) that have ENTRIESn_hourly that falls in this interval.
Remember to increase the number of bins in the histogram (by having larger number of bars). The default bin width is not sufficient to capture the variability in the two samples.
Can you write Python code for it?

gp.ggplot(df, gp.aes(x="ENTRIESn_hourly", fill="rain", color="rain")) + gp.geom_histogram()


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
A passenger from Class 1 is about 2.5x times more likely to survive than a passenger in Class 3.
Social-economic standing was a factor in survival rate of passengers.

Class 1: 62.96%
Class 2: 47.28%
Class 3: 24.24%
Can you write Python code for it?

survivals = pd.crosstab([data_s.Pclass, data_s.Sex], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=True)
survive=data.groupby(['Sex','Pclass'])
plt.ylabel("Passenger count")
plt.title('Histogram of passenger count grouped by sex and Class')

#survive.Survived.sum().plot(kind='barh')
survive.mean()


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
Data Description
From a sample of the RMS Titanic data, we can see the various features present for each passenger on the ship:
- Survived: Outcome of survival (0 = No; 1 = Yes)
- Pclass: Socio-economic class (1 = Upper class; 2 = Middle class; 3 = Lower class)
- Name: Name of passenger
- Sex: Sex of the passenger
- Age: Age of the passenger (Some entries contain NaN)
- SibSp: Number of siblings and spouses of the passenger aboard
- Parch: Number of parents and children of the passenger aboard
- Ticket: Ticket number of the passenger
- Fare: Fare paid by the passenger
- Cabin Cabin number of the passenger (Some entries contain NaN)
- Embarked: Port of embarkation of the passenger (C = Cherbourg; Q = Queenstown; S = Southampton)
Variable Notes
pclass: A proxy for socio-economic status (SES)
- 1st = Upper
- 2nd = Middle
- 3rd = Lower
age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5
sibsp: The dataset defines family relations in this way...
Sibling = brother, sister, stepbrother, stepsister
Spouse = husband, wife (mistresses and fiancés were ignored)
parch: The dataset defines family relations in this way...
Parent = mother, father
Child = daughter, son, stepdaughter, stepson
Some children travelled only with a nanny, therefore parch=0 for them.
Can you write Python code for it?

data =data_titanic

# Show the dataset
display(data.head())
data.info()


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
From the above info(),We can see columns Age, Cabin and Embarked have missing values.
Handling the missing values:

Ignore the rows with missing data,
Exclude the variable at all or we might substite it with  mean or median.

Age  80% of the data is available,which seems a important variable so not to exclude.
Port of embarkation doesn't seem interesting.
cabin 23% of the data so decided to exclude.
PassengerId,Name,fare doesnt seem to contribute to any survival investigation
Can you write Python code for it?

#exculding some coloumns
del data['Ticket']
del data['Cabin']
del data['Embarked']
del data['Name']
del data['PassengerId']
del data['Fare']

data.describe(include='all')

# Calculate number of missing values
data.isnull().sum()

null_female = data[pd.isnull(data['Age'])]['Sex'] == 'female'
null_male = data[pd.isnull(data['Age'])]['Sex'] == 'male'

print "Total missing age for female:",null_female.sum()

print "Total missing age for male:",null_male.sum()


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
From the above plot  we can clearly see individuals of different class distibuted for various ages. And the red line shows the average of age for each class
Can you write Python code for it?

 sns.factorplot( 'Sex' , 'Survived', data = data, kind = 'bar')
plt.title('Histogram of Survival rate grouped by Sex')
print "From the plot we can clearly see the survival rate of female is very high"

## GENDER
survivals = pd.crosstab([ data_s.Sex], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=False)
plt.ylabel("Passenger count")
plt.title('Histogram of Passenger count grouped by Sex and survived')

survival = data_s.groupby('Sex')['Survived']
survival.mean()

#PCLASS

survivals = pd.crosstab([data_s.Pclass], data_s.Survived.astype(bool))
survivals.plot(kind='bar', stacked=True)
plt.ylabel("Passenger count")
plt.title('Histogram of Passenger count grouped by Class')

survival=data.groupby(['Pclass'])
survival.mean()


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
From the above plot we can see how female individuals are given 1st preference and based on class.
Social-economic standing was a factor in survival rate of passengers by gender

Class 1 - female survival rate: 96.81%

Class 1 - male survival rate: 36.89%

Class 2 - female survival rate: 92.11%

Class 2 - male survival rate: 15.74%

Class 3 - female survival rate: 50.0%

Class 3 - male survival rate: 13.54%
Can you write Python code for it?

#Age
sns.factorplot(x="Survived", y="Age", hue='Sex', data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["Expired","Survived"])
plt.title('Boxplot of Age grouped by sex and Survival')
# survive_A=data.groupby(['Sex','Age'])


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
From the above plot we can see that gender played an important role in survival of each
individaul
Female Survival rate : 74.2%
Male Survival rate: 18.8%
Can you write Python code for it?

sns.factorplot(x="Pclass", y="Age", data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["1","2","3"])
plt.title('Boxplot of Age grouped by sex')
print "From the below plot we can see the average age is decreasing from calss 1 to class 3"


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
From the above statistics
- Youngest to survive: 0.42
- Youngest to die: 1.0
- Oldest to survive: 80.0
- Oldest to die: 74.0
Can you write Python code for it?

#  Seriously i dont understand why age is 0.42
data_s[data_s['Age'] < 1]


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
Hypothesis
To fill the missing data with mean, i will decide with t test by being sure that passengers in these 2 samples are likely to have the similar survival rate.

H0:means of the samples populatins are equal
H1: population means are different

If the resulted p value is going to be less than the critical value (with alpha level 0.05),
I should reject the null hypothesis and conclude that population means are different not by chance
(Ignoring the data of missing data which almost 20% of the data should be neglected).
I'm using the existing in scipy.stats function to perform t test for independent variables:
Can you write Python code for it?

from scipy.stats import ttest_ind
ttest_ind(notnull_age['Survived'], null_age['Survived'])


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
These are percentage of survivors for Group_age
- Adult : 36.0%
- Child : 50.35%
- Senior: 36.66%
Women and children have preference First to lifeboats?
Can you write Python code for it?

def group(d,v):
    if (d == 'female') and (v >= 18):
        return 'Woman'
    elif v < 18:
        return 'child'
    elif (d == 'male') and (v >= 18):
        return  'Man'

data['Category'] = data.apply(lambda row:group(row['Sex'], row['Age']), axis=1)
data.head(5)

survival_a = pd.crosstab([data.Category], data_s.Survived.astype(bool))
survival_a.plot(kind='bar', stacked=True)
plt.ylabel("Passenger count")
plt.title('Survival by Age category')
data.groupby(['Category']).mean()["Survived"]


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
These must be new borns and all survived
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns

# Set style for all graphs
#sns.set_style("light")
#sns.set_style("whitegrid")
sns.set_style("ticks", {"xtick.major.size": 8, "ytick.major.size": 8})

def plot(a,i):
    fig=plt.figure() #Plots in matplotlib reside within a figure object, use plt.figure to create new figure
    #Create one or more subplots using add_subplot, because you can't create blank figure
    ax = fig.add_subplot(1,1,1)
    #Variable
    ax.hist(data[a],bins = i) # Here you can play with number of bins
    plt.title(a + ' distribution')
    plt.xlabel(a)
    plt.ylabel('Passenger Count')
    plt.show()

plot("Age",30)
print "The above distribution of Age seems a little bit deviating from normal distribution"
print
plot("SibSp",8)
print "The above distribution of SibSp seems a right-skewed distribution"
plot("Parch",6)
print "The above distribution of Age seems a right-skewed distribution"

sns.factorplot(x="Sex", y="Age", data=data_s, kind="box", size=7, aspect=.8)\
.set_xticklabels(["Male","Female"])
plt.title('Boxplot of Age grouped by sex')
print "From the below plot we can see there are more elderly men than women and average age for men is higher than women"


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
lets decide should we remove missing age rows or fill the missing values with the mean, I'm going first to split the sample data into 2 samples  with missing age and with age and perform a t test
Can you write Python code for it?

notnull_age = data[pd.notnull(data['Age'])]
null_age = data[pd.isnull(data['Age'])]


I am working on the file "P2 investigate_dataset/P2.ipynb".
The first task is:
p value is than 0.05 which results in rejecting H0 ,so there is a significant difference in mean .So, I'm going to substitute the missing values with the mean age.
Can you write Python code for it?

print "Age median values by Age and Sex:"
#we are grouping by gender and class and taking median of age so we can replace with corrresponding values instead of NaN
print data.groupby(['Sex','Pclass'], as_index=False).median().loc[:, ['Sex','Pclass', 'Age']]
print "Age values for 5 first persons in dataset:"
print data.loc[data['Age'].isnull(),['Age','Sex','Pclass']].head(5)
# apply transformation: Age missing values are filled with regard to Pclass and Sex:
data.loc[:, 'Age'] = data.groupby(['Sex','Pclass']).transform(lambda x: x.fillna(x.median()))
print data.loc[[5,17,19,26,28],['Age','Sex','Pclass']].head(5)
data['Age'] = data['Age'].fillna(data['Age'].mean())

data.describe(include='all')


I am working on the file "P2 titanic.ipynb".
The first task is:
At first sight, it looks like having paid for a cabin affected the probability of survival. The problem is most cabins where occupied by fist class passengers:
Can you write Python code for it?

print titanic[['PassengerId', 'Has_Cabin', 'Pclass']].groupby(['Has_Cabin', 'Pclass']).count()


I am working on the file "P2 titanic.ipynb".
The first task is:
Because the Survived column can only contain 1 for yes and 0 for no, we can count all the survivors of each sex by grouping by sex and adding the Survived column.
Can you write Python code for it?

sex_surv_count['Survived'] = sex_surv[['Survived', 'Sex']].groupby('Sex').sum()
print sex_surv_count

totals = pd.Series(sex_surv_count.sum(), name='total')
sex_surv_count = sex_surv_count.append(totals)
sex_surv_count['SurvPerc'] = sex_surv_count.Survived * 100 / sex_surv_count.PassengerId
print sex_surv_count

sex_surv_count['SurvPerc'].plot(kind='bar', rot=0, title='Percentage of survivors by sex')


I am working on the file "P2 titanic.ipynb".
The first task is:
It looks like the survival rate is very biased towards the female sex: nearly three quarters of women survived, compared to a bit less than one fifth of the men aboard the Titanic. The total survival rate is low because there were more men than women aboard.
Now we will analyze age. We choose to drop the passengers for whom we do not know the age. Because age is a continuous variable, we also have to group the passengers in age bins in order to have meaningful statistics:
Can you write Python code for it?

age_surv = sex_age_surv[['PassengerId', 'Age', 'Survived']].dropna()
print age_surv.Age.max()
age_surv['bin'] = pd.cut(age_surv['Age'],np.arange(0,90,10), right=False)
print age_surv.head()


I am working on the file "P2 titanic.ipynb".
The first task is:
Now we can calculate the percentage of each class per port:
Can you write Python code for it?

passengers_by_port_class['Percent'] = passengers_by_port_class.PassengerId * 100 / passengers_by_port_class.Total
print passengers_by_port_class


I am working on the file "P2 titanic.ipynb".
The first task is:
So, a fairer comparison could be between the 40 first class passengers that did not pay for a cabin and the rest of the same class:
Can you write Python code for it?

cabin_surv = titanic[['PassengerId', 'Has_Cabin', 'Survived']][titanic['Pclass'] == 1].copy()
cabin_surv_count = cabin_surv[['PassengerId', 'Has_Cabin']].groupby('Has_Cabin').count()
cabin_surv_count['Survived'] = cabin_surv[['Survived', 'Has_Cabin']].groupby('Has_Cabin').sum()
cabin_surv_count['SurvPerc'] = cabin_surv_count['Survived'] * 100 / cabin_surv_count['PassengerId']
print cabin_surv_count

ax = cabin_surv_count.SurvPerc.plot(kind='bar', rot=0, title='Percentage of first class survivors by cabin rental')
ax.set_xlabel('Had a cabin')


I am working on the file "P2 titanic.ipynb".
The first task is:
The graph shows that for children below 10 both female and male survival rates are very similar. However, for the rest of age groups, the female survival rates are high, with the minimum around 70%, and male survival rates are low, with the maximum around 20%.
The data suggests that the "women and children first" code of conduct was indeed followed.
Did having a cabin influence survivability?
We first add a Has_Cabin boolean column to our dataset, in order to ease following data wrangling operations.
Can you write Python code for it?

titanic['Has_Cabin'] = titanic.Cabin.notnull()
cabin_surv = titanic[['PassengerId', 'Has_Cabin', 'Survived']].copy()


I am working on the file "P2 titanic.ipynb".
The first task is:
The graph shows that more than 90% of the passengers that embarked from Queenstown in Ireland where third class passengers, in contrast with the other ports, where third class made up for 40-55% of the total number of passengers.
The Maddison Project Database shows that Ireland had an estimated GDP per capita of 2.736 (in 1990 International Geary-Khamis Dollars) by 1913 (no data for 1912), while the UK had a GDP of 4.762 and France's was 3.514 by 1912.
Our passenger data coincides with the Maddison Project in the case of Ireland: it was a poor country compared to Europe's average by the year the Titanic sank. However, our data cannot explain why 1st class percentage in Cherbourg doubles Southampton's, being the UK a richer country than France during that time. Maybe the Cherbourg region was wealthier than the average of France, or Southampton was specially poor, but we do not have the data to check this.
Of course, there are other problems. For instance, we do not know if Titanic passengers make a representative sample of the demography of these regions.
How did sex and age affect survivability? How did these factors work together?
Can you write Python code for it?

sex_age_surv = titanic[['PassengerId', 'Sex', 'Age', 'Survived']]
sex_age_surv_count = sex_age_surv.count()
print sex_age_surv_count


I am working on the file "P2 titanic.ipynb".
The first task is:
The passengers embarked from three different ports in three separate regions: Southampton in England, Queenstown in Ireland (that was part of UK) and Cherbourg in France. Can we infere wealth differences between these regions based on passenger data from this dataset?
Using the fare column for this question is complicated, because some passengers paid for their companions (family members, maids, nurses...), and prices varied greatly even for the same class.
Thus, using the passenger class is a more straightforward way. For each passenger, we need the class and the port they embarked from. Unfortunately, we do not always know the port, so we will drop these rows:
Can you write Python code for it?

class_port = titanic[['PassengerId', 'Pclass', 'Embarked']]
print class_port.isnull().any()
print
class_port = class_port.dropna()
print class_port.isnull().any()


I am working on the file "P2 titanic.ipynb".
The first task is:
We do not have the age for every passenger, so we will have to take this into account when analysing the data. The rest of the data is complete.
First, we will analyze the effect of sex alone on survivability. We can use a bar chart with the percentage of survivors for the total sample and for every sex, like we did in the last question for the class and port.
Can you write Python code for it?

sex_surv = sex_age_surv[['PassengerId', 'Sex', 'Survived']]
sex_surv_count = sex_surv[['PassengerId', 'Sex']].groupby('Sex').count()
print sex_surv_count


I am working on the file "P2 titanic.ipynb".
The first task is:
We need the number of passengers for each port of embarkment and class:
Can you write Python code for it?

passengers_by_port_class = class_port.groupby(['Embarked', 'Pclass'], as_index=False).count()
print passengers_by_port_class


I am working on the file "P2 titanic.ipynb".
The first task is:
and it looks like the passenger class also influenced survivability:
Can you write Python code for it?

class_surv = titanic[['PassengerId', 'Pclass']].copy()
class_surv_count = class_surv.groupby('Pclass').count()
class_surv_count['Survived'] = titanic[['Survived', 'Pclass']].groupby('Pclass').sum()
print class_surv_count


I am working on the file "P2 titanic.ipynb".
The first task is:
and the total number of passengers for each port:
Can you write Python code for it?

passengers_by_port_class['Total'] = passengers_by_port_class.groupby('Embarked')['PassengerId'].transform(sum)
print passengers_by_port_class


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Adding new column to dataframe
Can you write Python code for it?

# add "Passenger" column represent who alive or died
titanic['Passenger'] = map(had_alive, titanic['Survived'])

# add "Gender" column these map Sex from text to number for future analysis
titanic['Gender'] = titanic['Sex'].map({'female' : 0, 'male' : 1}).astype(int)

# add "Who" column, to categorize passenger by group of male, female
# and children as who under 15 as a child,
titanic['Who'] = titanic[['Age','Sex']].apply(had_child, axis=1)

# add "Port" column these map each embarktation from text to number for future analysis
titanic['Port'] = titanic['Embarked'].dropna().map({'C' : 0, 'Q' : 1, 'S': 2 }).astype(int)

# add "Class" column represen clss of each passenger
titanic['Class'] = map(had_class, titanic['Pclass'])

# add "Family" column to represent passenger who travel alone or with their family
titanic['Family'] = map(had_family, titanic['Parch'] + titanic['SibSp'])

# drop "Parch" & "SibSp"
titanic = remove_column(titanic, ['Parch', 'SibSp'])

titanic.head()


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Cheap fare most sold in Queenstown, look seem the economics of this town inwealthy
5.) Dose passenger class related with cabin position
Can you write Python code for it?

# create a new datafram that rmove missing cabin
titanic_no_missing = titanic.dropna()

titanic_no_missing.info()

# grab only letter for the cabin position
for i, x in titanic_no_missing['Cabin'].iteritems():
    titanic_no_missing.loc[i,('Cabin')] = x[0]

#print titanic_no_missing['Cabin']

# Quick preview of the cabin position
titanic_no_missing.head()

# get total cabin passenger
cabincount = float(len(titanic_no_missing))

# quick look cabin row

passenger_by_class_who = titanic_no_missing.groupby('Cabin')
print passenger_by_class_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Titanic passenger cabin')

# show the counts of passengers
ax = sns.countplot(x = 'Cabin',data = titanic_no_missing, palette='summer')

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height, '%1.2f'%((height*100)/cabincount))

# look at cabin row T was out of group, so redefine cabin data again
titanic_no_missing = titanic_no_missing[titanic_no_missing['Cabin'] != 'T']

# plot cabin position

passenger_by_class_who = titanic_no_missing.groupby('Cabin')
print passenger_by_class_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Titanic passenger cabin')

# show the counts of passengers
ax = sns.countplot(x = 'Cabin',data = titanic_no_missing, palette='summer')

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height, '%1.2f'%((height*100)/cabincount))

# check position by passengers class
passenger_by_class_who = titanic_no_missing.groupby(['Class', 'Cabin'])
print passenger_by_class_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Passenger cabin by Class')
# show the counts of passengers
ax = sns.countplot(x = 'Class', hue = 'Cabin', data = titanic_no_missing)

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    if math.isnan(height):
        continue

    else:
        ax.text(p.get_x(), height, '%1.2f'%((height*100)/cabincount))
       ...


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Exploration Phase
Can you write Python code for it?

# import library for the analysis and visualization
import math
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pylab import rcParams
from pandas.tools.plotting import scatter_matrix

sns.set(style="white")

%matplotlib inline


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
First class fare was expensive than other class.
Can you write Python code for it?

# check passengers fare by embarked
ax = sns.FacetGrid(titanic, hue = 'Embarked', aspect = 3)
ax.map(sns.kdeplot,'Fare',shade = True)
oldest = titanic['Fare'].max()
ax.set(xlim = (0,oldest))
ax.add_legend()
ax = sns.plt.title('Ticket fare by port of embarked')

# check passengers fare by embarked
ax = sns.FacetGrid(titanic, hue = 'Who', aspect = 3)
ax.map(sns.kdeplot,'Fare',shade = True)
oldest = titanic['Fare'].max()
ax.set(xlim = (0,oldest))
ax.add_legend()
ax = sns.plt.title('Ticket fare by passenger')


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Fixed missing "Age" column
Can you write Python code for it?

# find all the unique values for "Age"
print(titanic['Age'].unique())

# fill missing "Age" with mean
titanic['Age'] = titanic['Age'].fillna(titanic['Age'].median())

# inspect dataset
#show_info(titanic, True)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Fixed missing "Cabin" column
Can you write Python code for it?

# find all the unique values for "Age"
print titanic['Cabin'].unique()

# we see the "Cabin" show 687 missing rows,
# assume passenger who held same ticket number should seat in same cabin

# loop through rows in datafram and fill Cabin
for i, row in titanic.iterrows():
    if pd.isnull(row['Cabin']):
        continue
    else:
        for j, row in titanic.iterrows():
            if pd.isnull(titanic.loc[j,'Cabin']):
                if titanic.loc[j,'Ticket'] == titanic.loc[i,'Ticket']:
                    titanic.loc[j,'Cabin'] = titanic.loc[i,'Cabin']
                    break

titanic.head()


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
From data, the passengers on 1st class almost on cabin row A, B, C and D,
Almost passenger from Queentown were on cabin row E and F
6.) Who are the passengers travel alone or travel with family?
Can you write Python code for it?

# check to make sure it worked
titanic['Family'].count()

# plot cabin position

passenger_by_class_who = titanic.groupby('Family')
print passenger_by_class_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Titanic passenger family')

# show the counts of passengers
ax = sns.countplot(x = 'Family',data = titanic, palette='summer')

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height + 5, '%1.2f'%((height*100)/passengercount))


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
From the result, look seek, higher fare may little bit increased survival chance
Now, how if passenger who travel with family will increase chance of survival?
Can you write Python code for it?

# how about survival rate if relate with family or alone
ax = sns.factorplot(x = 'Family', y = 'Survived',
                   data = titanic, size = 5, aspect = .6)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
From this result, most alone passengers were male.
Passenger who from Southampton were travel alone
7.) What is the  factors helped some passengers survive?
Can you write Python code for it?

# check survial rate
passenger_by_class_who = titanic.groupby('Passenger')
print passenger_by_class_who.count()['Survived']

# set plot title
ax = sns.plt.title('Titanic survival passenger')
# show the counts of passengers
ax = sns.countplot(x = 'Passenger', data = titanic)

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height + 2, '%1.2f'%((height*100)/passengercount))


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
In Queentown, almost passengers that boarded there were third class.
4.) Who are purchase expensive ticket?
Can you write Python code for it?

# quick check clss of embarked
group_by_ses_embark = titanic.groupby(['Embarked'])
print group_by_ses_embark['Fare'].agg([np.sum, np.max, np.min, np.mean, np.std])

# plot distribution of ticket price
ax = sns.FacetGrid(titanic, col = 'Class', size = 5, aspect = 0.7)
(ax.map(sns.boxplot, 'Embarked', 'Fare')
 .despine(left = True));

# check passengers fare for class
ax = sns.FacetGrid(titanic, hue = 'Class', aspect = 3)
ax.map(sns.kdeplot,'Fare', shade = True)
oldest = titanic['Fare'].max()
ax.set(xlim = (0,oldest))
ax.add_legend()
# set title
ax = sns.plt.title('Ticket fare by class')


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
In the third class males were double of female and most child were in this class
2.) How old of the passengers?
Can you write Python code for it?

# show passenger age
passenger_by_age_who = titanic.groupby(['Who'])
print passenger_by_age_who['Age'].mean()

# quick look the distribution
ax = sns.distplot(titanic['Age'], hist = True)
# set plot title
ax = sns.plt.title('Titanic passengers by Age')

# check age for passenger by type

# faceplot by passenger type
ax = sns.FacetGrid(titanic, hue = 'Who', aspect = 3)
ax.map(sns.kdeplot,'Age',shade = True)
oldest = titanic['Age'].max()
ax.set(xlim = (0, oldest))
ax.add_legend();
# set title
ax = sns.plt.title('Titanic passengers Age by person')


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Look at passenger gender, whom male and female aged were not difference
Can you write Python code for it?

# check age for passenger by class

# faceplot by passenger type
ax = sns.FacetGrid(titanic, hue = 'Class', aspect = 3)
ax.map(sns.kdeplot,'Age',shade = True)
oldest = titanic['Age'].max()
ax.set(xlim=(0,oldest))
ax.add_legend();
# set title
ax = sns.plt.title('Titanic passengers Age by Class')


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Look like higher fare will increase chance of survival rate, this maybe related with 1st class passenger have higher survived
Can you write Python code for it?

# check survived rate relate fare by gender
ax = sns.lmplot('Fare', 'Survived', hue = 'Sex',  col = 'Class',
               data = titanic, palette = 'winter',
               x_bins = farerange)

# check survived rate relate fare by gender
ax = sns.lmplot('Fare', 'Survived', hue = 'Sex',  col = 'Embarked',
               data = titanic, palette = 'winter',
               x_bins = farerange)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Look like that being a male in any age will decreases a chances of survival.
From the result is show that being a male in any age will lowest a chances of survival.
Now investigate what about ticket fare relate survival rate ?
Can you write Python code for it?

# set range of price
farerange = [0,250,500,750,1000]

# plot data of fare versus survival
ax = sns.lmplot('Fare', 'Survived', data = titanic,
               palette = 'winter', x_bins = farerange)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Look seem a few more passenger died than those who survived.
Let's take a look which variables had an effect on their survival rate
Can you write Python code for it?

# plot survival probability against several variables
ax = sns.factorplot(x = 'Class', y = 'Survived',
                   hue = 'Who', col = 'Embarked',
                   data = titanic, size = 5, aspect = .6)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Looks like there is a formal trend that an older passenger less survival rate.
Can you write Python code for it?

# how about survival rate if relate class and age
# use use linear plot on age versus survival by each class
ax = sns.lmplot('Age', 'Survived', hue ='Class',
               data = titanic, palette = 'winter',
               x_bins = age_range)