I am working on the file "BlogPosts/Modern_TensorFlow/modern-tensorflow.ipynb".
The first task is:
model_fn
Inputs (features, labels, mode, params); returns EstimatorSpec objects.
Can you write Python code for it?

def model_fn(features, labels, mode, params):
    # Args features and labels are the same as returned by the dataset
    tokens, sequence_length = features

    # For Serving (ignore this)
    if isinstance(features, dict):
        tokens = features['tokens']
        sequence_length = features['sequence_length']

    # 1. Define the graph
    vocab_size = params['vocab_size']
    dim = params['dim']
    embeddings = tf.get_variable('embeddings', shape=[vocab_size, dim])
    token_embeddings = tf.nn.embedding_lookup(embeddings, tokens)
    lstm_cell = tf.contrib.rnn.LSTMCell(20)
    lstm_output, _ = tf.nn.dynamic_rnn(
        lstm_cell, token_embeddings, dtype=tf.float32)

    logits = tf.layers.dense(lstm_output, 3)
    preds = tf.argmax(logits, axis=-1)

    # 2. Define EstimatorSpecs for PREDICT
    if mode == tf.estimator.ModeKeys.PREDICT:
        # Predictions is any nested object (dict is convenient)
        predictions = {'logits': logits, 'preds': preds}
        # export_outputs is for serving (ignore this)
        export_outputs = {
            'predictions': tf.estimator.export.PredictOutput(predictions)}
        return tf.estimator.EstimatorSpec(mode, predictions=predictions,
                                          export_outputs=export_outputs)
    else:
        # 3. Define loss and metrics
        # Define weights to account for padding
        weights = tf.sequence_mask(sequence_length)
        loss = tf.losses.sparse_softmax_cross_entropy(
            logits=logits, labels=labels, weights=weights)
        metrics = {
            'accuracy': tf.metrics.accuracy(labels=labels, predictions=preds),
        }
        # For Tensorboard
        for k, v in metrics.items():
            # v[1] is the update op of the metrics object
            tf.summary.scalar(k, v[1])

        # 4. Define EstimatorSpecs for EVAL
        # Having an eval mode and metrics in Tensorflow allows you to use
        # built-in early stopping (see later)
        if mode == tf.estimator.ModeKeys.EVAL:
   ...


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
Here's a look at all the series we're working with:
<img src="fig/all_series.png" width=600>
Detection
Now let's find outliers.
Can you write Python code for it?


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
Here's the run with AnomalyDetection:
Can you write Python code for it?


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
In the call to make_time_series, we specify a date range, a unit for the time step (here, h for hours), and a list giving each function along with its noise rate and its outlier percentage.
Can you write Python code for it?


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
Series A
Here's a visualization of Series A with red dots marking the points identified as outliers. Series A only has a little noise, and that makes it easier for us to find outliers because they stand out more — almost every point above 1 and below –1 is considered an outlier in this series. More importantly, we're also able to find outliers that are solidly in the middle of the range we expect. Each of the red circles between –1 and 1 is an outlier because it breaks the seasonal pattern determined when using STL, even though it wouldn't raise any flags if taken out of its sequential context.

Series B
Series B is a little more complicated than Series A because it has a downward trend on top of seasonality. However, STL has no problem identifying the overall pattern. The story is similar to Series A because the noise term still isn't too strong, but we do start to see a few points that could have been considered outliers but were missed because of the increase in noise; the Potential outlier labeled in the plot is one such point.

Series C
For Series C, as with Series B, the upward trend doesn't cause any problems. However, unlike Series A and B, the increase in noise in C causes far fewer outliers to be found; only the largest, most obvious points are identified. Taking even a quick look shows some suspect points like the ones labeled Extreme outlier(s). These seem like they should be spotted as outliers, but the noise in the series makes the method more conservative.

Benchmarks
At the top of this post, we said our implementation performed outlier detection "very quickly" — now we'll quantify that.
To benchmark our work we compare outlierDetection to a similar package called AnomalyDetection, released by Twitter. This is an R package using similar (but not identical) techniques as our own, where the biggest differences are that AnomalyDetection has more features and...
Can you write Python code for it?


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
TL;DR: We describe a method of finding outliers in time series data by combining two distinct techniques, STL decomposition and sequential Grubbs' tests. In the end, we arrive at a method that is flexible and identifies points of interest very quickly. Additionally, we offer a new function, comparable to scikit-learn's dataset.make_*, for generating time series datasets with varying functional forms, noise, and number of outliers.

Introduction
STL Decomposition
Grubbs' test
Sequential Grubbs' tests
Data
Detection
Benchmarks
Conclusion

Introduction
Outliers are often nuisances that we remove to avoid skew in estimates and help ensure well-behaved models. However, in business and many other contexts,  outliers convey unique information, so simply identifying them can yield powerful insights and offer guidance on which parts of the data need more delving into.
In general, there are two parts to outlier detection. The first is identifying a pattern in the data, and the second is locating the points that don't fit that pattern. In traditional cases, finding a pattern usually means inferring distributions that fit the data best, and identifying outliers means finding the points that have a low probability of belonging to any of the distributions.
This framework also makes sense for time series, but instead of estimating distributions to find patterns, we need to use methods that respect the sequence structure of the data. In this post, we'll describe a method of outlier detection that combines two steps: (1) using STL decomposition to define a time series pattern, and (2) applying sequential Grubbs' tests to spot points that don't fit that pattern.
The code supporting this post is available as roam_outliers. That code creates a Python bridge to our R package outlierDetection, which is also available. Our hope is that this facilitates further exploration of our proposed method.
Can you write Python code for it?


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
The function np_to_df is a helper functions for plotting and passing data between Python and R.
Can you write Python code for it?

wide_df = np_to_df(dt, ys, cols=['A','B','C'])


I am working on the file "BlogPosts/Outlier_detection/Outlier_detection_post.ipynb".
The first task is:
This creates a large time series of ~86K observations:
Can you write Python code for it?

wide_df = np_to_df(dt, ys, ['A'])
print("(rows, cols): {}".format(wide_df.shape))


I am working on the file "BlogPosts/Translation_scaling_invariance_regression/Translation_and_scaling_invariance_in_regression_models.ipynb".
The first task is:
Ignoring the intercept
It follows from this result that we can ignore $\mu$ in the regression model. Why is this? We know for sure that we can mean-center $X$ without changing the model. If $X$ is mean-centered, then the optimal value of $\mu$ is the mean of $Y$, written $\bar{Y}$. (This requires just a bit of calculus: ${\bf{1}}^{\top}X = 0$ when you've mean-centered $X$.) We might as well mean-center $Y$ as well, leaving $\mu = 0$, so we can ignore it from now on.
Scaling does not affect unregularized regression
Formal argument
Let's now look at scaling in the case of unregularized regression.  We've ignored $\mu$ and there is no $R$ either for an unregularized model, so we have the following minimization problem:
\begin{equation}
\min_\VEC{\beta} \; \|Y - X\VEC{\beta} \|_2^2
\end{equation}
Let's rescale the features by dividing by the standard deviation of each column (we'll denote these by $\sigma_1, \sigma_2, \ldots $)
\begin{align}
\begin{split}
\text{for } i=1, \ldots, n:& \
Z_{i1} &= \frac{X_{i1}}{\sigma_1}\
Z_{i2} &= \frac{X_{i2}}{\sigma_2}\
&\text{etc.}
\end{split}
\end{align}
which we can succinctly write as
\begin{equation}
Z = XD
\end{equation}
where $D$ is a diagonal matrix with $D_{ii} = \sigma_i^{-1}$.
The objective function for regression with $Z$ is
\begin{align}
\|Y - Z\VEC{\theta} \|_2^2
= \|Y - XD\VEC{\theta} \|^2_2
\end{align}
Now set $\VEC{\beta} = D\VEC{\theta}$ and we're back to the original objective for $X$. That means the minimizers are related simply by $\hat{\VEC{\beta}} = D\hat{\VEC{\theta}}$. Just like with mean-centering, this result actually applies to any rescaling, not just dividing by the standard deviations.
Predictions stay the same
What about predictions? Suppose we introduce a test point $\VEC{x}^\star$ and its corresponding $\VEC{z}$-value, which is $\VEC{z}^\star = D^T \VEC{x}^\star $. To make predictions with the original (not unit variance) model we use $\VEC{x}^{\star\top}\hat{\VEC{\beta}}$ and for the unit variance model, we use...
Can you write Python code for it?

Z = X / X.std()
compare_normalizations(
    X, Z, y,
    title1="Before rescaling",
    title2="After rescaling")


I am working on the file "BlogPosts/Translation_scaling_invariance_regression/Translation_and_scaling_invariance_in_regression_models.ipynb".
The first task is:
It should be completely unsurprising that the lower scatterplot now matches the upper scatterplot. The more notable thing is that the regression line, which was fitted in $z$-space and then transformed, is the same as the line fitted in $x$-space to start with. Thus, we see clearly that scaling doesn't affect the unregularized model.
When we look at regularized regression, these lines will not match.
Scaling affects regularized regression
Formal argument
Let's try to redo the proof for regularized regression (that nothing changes) and see where it breaks.
Recall the objective
\begin{equation}
\min_\VEC{\beta} \; \|Y - X\VEC{\beta} \|_2^2 + \lambda R(\VEC{\beta})
\end{equation}
and the rescaling
\begin{equation}
Z = XD
\end{equation}
We'll consider a general $D$ from now on (so need not have $D_{ii} = \sigma_i^{-1}$).
For regression with $Z$ the objective with our new coefficient variable $\VEC{\theta}$ is
\begin{align}
\|Y - Z\VEC{\theta}\|_2^2 + \lambda R(\VEC{\theta})
&= \|Y - XD\VEC{\theta} \|_2^2 + \lambda R(\VEC{\theta})
\end{align}
Now set $\VEC{\beta} = D\VEC{\theta}$
\begin{align}
\|Y - XD\VEC{\theta} \|_2^2 + \lambda R(\VEC{\theta})
&= \|Y - X\VEC{\beta} \|_2^2 + \lambda R(D^{-1}\VEC{\beta}) \
&\neq \|Y - X\VEC{\beta} \|_2^2 + \lambda R(\VEC{\beta})
\end{align}
Unsurprisingly, the problem is in the regularization term. Instead of regularizing $\VEC{\beta}$ we end up regularizing $D^{-1}\VEC{\beta}$. That means it's unlikely the two problems will have the same minimizers. Although we set $\VEC{\beta} = D\VEC{\theta}$, we can't conclude that $\hat{\VEC{\beta}} = D\hat{\VEC{\theta}}$.
Quantifying the change
So we know that in regularized regression, the models might not be equivalent. Can we be more precise? Let's examine one particular example of regularized regression, the Lasso \cite{tibshirani1996}. For the Lasso we have
\begin{align}
R(\VEC{\beta})&= \sum_{i=1}^p |\beta_i| \
R(D^{-1}\VEC{\beta}) &= \sum_{i=1}^p \frac{|\beta_i|}{D_{ii}} \label{eq:lasso}
\end{align}
This tells us that we will...
Can you write Python code for it?

from sklearn.linear_model import Lasso

compare_transformed_normalizations(
    X, y,
    transform=lambda x : x/3,
    model=Lasso(),
    title1="Trained using original data",
    title2="Trained using scaled data")


I am working on the file "BlogPosts/Translation_scaling_invariance_regression/Translation_and_scaling_invariance_in_regression_models.ipynb".
The first task is:
Notice that we have a different intercept, but the same slope. The predictions from these two models will be identical. For instance, the prediction at $x = 10$ is around $2.5$ on the upper graph. The corresponding $z$ is $z = 10 - \bar{x} \approx 6$ with precisely the same prediction on the lower graph. The following code works through this in detail:
Can you write Python code for it?

# The case of `x`:
x_star = 10
lr = LinearRegression()
lr.fit(X.reshape(-1,1), y)
x_prediction = lr.predict(x_star)[0]
print("The prediction at x = {} is {:.4}".format(
        x_star, x_prediction))

# The case of `z`:
z_star = x_star - empirical_mean
lr.fit(Z.reshape(-1,1), y)
z_prediction = lr.predict(z_star)[0]
print("The prediction at z = {} - {:.4} = {:.4} is {:.4}".format(
        x_star, empirical_mean, z_star, z_prediction))


I am working on the file "Blurred Image Comparison.ipynb".
The first task is:
We'll start with this image:
Can you write Python code for it?

imgpath = 'images/original/image.bmp'
blurredpath = 'images/image_blurred.bmp'

img = Image.open(imgpath)
blurred = img.copy().filter(ImageFilter.BLUR)

blurred.save(blurredpath)


I am working on the file "Bokeh/03 - models.ipynb".
The first task is:
Adding a Hover Tool
Can you write Python code for it?

from bokeh.models import HoverTool

tooltips = """

    @Name
    (@Abbrev)

    @Time{0.00}
    @Year

@{MetersBack}{0.00} meters behind

"""

xdr = Range1d(start=sprint.MetersBack.max()+2, end=0)  # +2 is for padding
ydr = DataRange1d(range_padding=0.05)

plot = Plot(x_range=xdr, y_range=ydr, **plot_options)
medal = plot.add_glyph(source, medal_glyph)  # we need this renderer to configure the hover tool
plot.add_glyph(source, athlete_glyph)
plot.add_glyph(no_olympics_glyph)

xticker = SingleIntervalTicker(interval=5, num_minor_ticks=0)
xaxis = LinearAxis(ticker=xticker, axis_line_color=None, major_tick_line_color=None,
                   axis_label="Meters behind 2012 Bolt", axis_label_text_font_size="10pt",
                   axis_label_text_font_style="bold")
plot.add_layout(xaxis, "below")

xgrid = Grid(dimension=0, ticker=xaxis.ticker, grid_line_dash="dashed")
plot.add_layout(xgrid)

yticker = SingleIntervalTicker(interval=12, num_minor_ticks=0)
yaxis = LinearAxis(ticker=yticker, major_tick_in=-5, major_tick_out=10)
plot.add_layout(yaxis, "right")

hover = HoverTool(tooltips=tooltips, renderers=[medal])
plot.add_tools(hover)

show(plot)

from bubble_plot import get_1964_data

def get_plot():
    return Plot(
        x_range=Range1d(1, 9), y_range=Range1d(20, 100),
        title="", plot_width=800, plot_height=400,
        outline_line_color=None, toolbar_location=None,
    )

df = get_1964_data()
df.head()

# EXERCISE: Add Circles to the plot from the data in `df`.
# With `fertility` for the x coordinates, `life` for the y coordinates.

plot = get_plot()

# EXERCISE: Color the circles by region_color & change the size of the color by population

# EXERCISE: Add axes and grid lines

# EXERCISE: Manually add a legend...


I am working on the file "Bokeh/03 - models.ipynb".
The first task is:
Next we import some of the Bokeh models that need to be assembled to make a plot. At a minimum, we need to start with Plot, the glyphs (Circle and Text) we want to display, as well as ColumnDataSource to hold the data and range obejcts to set the plot bounds.
Can you write Python code for it?

from bokeh.io import output_notebook, show
from bokeh.models.glyphs import Circle, Text
from bokeh.models import ColumnDataSource, Range1d, DataRange1d, Plot

output_notebook()


I am working on the file "Bokeh/03 - models.ipynb".
The first task is:
Setting up Data
Can you write Python code for it?

abbrev_to_country = {
    "USA": "United States",
    "GBR": "Britain",
    "JAM": "Jamaica",
    "CAN": "Canada",
    "TRI": "Trinidad and Tobago",
    "AUS": "Australia",
    "GER": "Germany",
    "CUB": "Cuba",
    "NAM": "Namibia",
    "URS": "Soviet Union",
    "BAR": "Barbados",
    "BUL": "Bulgaria",
    "HUN": "Hungary",
    "NED": "Netherlands",
    "NZL": "New Zealand",
    "PAN": "Panama",
    "POR": "Portugal",
    "RSA": "South Africa",
    "EUA": "United Team of Germany",
}

gold_fill   = "#efcf6d"
gold_line   = "#c8a850"
silver_fill = "#cccccc"
silver_line = "#b0b0b1"
bronze_fill = "#c59e8a"
bronze_line = "#98715d"

fill_color = { "gold": gold_fill, "silver": silver_fill, "bronze": bronze_fill }
line_color = { "gold": gold_line, "silver": silver_line, "bronze": bronze_line }

def selected_name(name, medal, year):
    return name if medal == "gold" and year in [1988, 1968, 1936, 1896] else None

t0 = sprint.Time[0]

sprint["Abbrev"]       = sprint.Country
sprint["Country"]      = sprint.Abbrev.map(lambda abbr: abbrev_to_country[abbr])
sprint["Medal"]        = sprint.Medal.map(lambda medal: medal.lower())
sprint["Speed"]        = 100.0/sprint.Time
sprint["MetersBack"]   = 100.0*(1.0 - t0/sprint.Time)
sprint["MedalFill"]    = sprint.Medal.map(lambda medal: fill_color[medal])
sprint["MedalLine"]    = sprint.Medal.map(lambda medal: line_color[medal])
sprint["SelectedName"] = sprint[["Name", "Medal", "Year"]].apply(tuple, axis=1).map(lambda args: selected_name(*args))

source = ColumnDataSource(sprint)


I am working on the file "Bokeh_plotting_of_latest_mooring_station_data.ipynb".
The first task is:
Define data sources
We define some basic data handling functions here. These will just enable us to e.g. access the nefCDF variable data as numpy array (significantly faster) or convert times to a joint base.
Can you write Python code for it?

def get_data_array(data_array):
    if type(data_array.__array__()) is np.ma.masked_array:
        return data_array.__array__().data
    else:
        return data_array.__array__()

def get_qc_variable_name(variable):
    try:
        qc_variable_name = variable.ancillary_variables
    except AttributeError:
        # print "No QC variable found for " + variable.name
        qc_variable_name = None
    return qc_variable_name

def get_pandas_timestamp_series(datetime_array):
    out = pd.Series(np.zeros(len(datetime_array)))
    counter = 0
    for i in datetime_array:
        out[counter] = pd.tslib.Timestamp(i)
        counter += 1
    return out

def days_to_seconds(days):
    return int(days) * 24 * 60 * 60

def get_str_time(x): return str(x)

def totimestamp(dt, epoch=datetime.datetime(1970,1,1)):
    td = dt - epoch
    # return td.total_seconds()
    return (td.microseconds + (td.seconds + td.days * 86400) * 10**6) / 10**6


I am working on the file "Bokeh_plotting_of_latest_mooring_station_data.ipynb".
The first task is:
Get latest data
Here, we will call our defined methods. Also, we will define the output filename and the desired timespan of the plotting.
Can you write Python code for it?

station_names, station_links = get_mooring_stations('http://thredds.socib.es/thredds/catalog/mooring/weather_station/catalog.html')
# get latest x days

days = 2
html_file = 'bokeh_latest_data.html'

seconds = days_to_seconds(days)
dt = datetime.datetime.now()
desired_start_time = calendar.timegm(dt.utctimetuple()) - seconds
output_file(html_file)
draw_data(station_links, desired_start_time, station_names)


I am working on the file "Bokeh_plotting_of_latest_mooring_station_data.ipynb".
The first task is:
Here, we define the bokeh plotting parameters. Also, we create a javascript callback to automatically adjust the y-axis according to the current zoom-extend.
Can you write Python code for it?

def draw_data(links, desired_start_time, station_names):
    global VARIABLES_OF_INTEREST
    counter = 0
    output_stations = []
    for station in links:
        root = Dataset(station)
        time = get_data_array(root.variables["time"])
        idx = time >= desired_start_time
        if not np.any(idx):
            counter += 1
            continue
        variables = root.get_variables_by_attributes(standard_name=lambda n: n in VARIABLES_OF_INTEREST)
        time = time[idx]
        subplot =  []
        variable_names = []
        for v in variables:
            try:
                qc_data = get_data_array(root.variables[get_qc_variable_name(v)])
                qc_data = qc_data[idx]
                bad_idx = get_data_array(qc_data) != 1
            except KeyError:
                print "No QC found for " + v.name
            v_name = v.name
            variable_names.append(v_name)
            v = get_data_array(v)
            v = v[idx]
            conv_time = get_pandas_timestamp_series([datetime.datetime.fromtimestamp(ts) for ts in time])
            subplot.append(get_bokeh_grid_figure(v, qc_data, conv_time, station_names[counter]))

        sub_counter = 0
        my_tabs = []
        for sp in subplot:
            my_tabs.append(Panel(child=sp, title=variable_names[sub_counter]))
            sub_counter += 1
        p = Tabs(tabs=my_tabs)
        output_stations.append(p)
        counter += 1
    amount_stations = len(output_stations)
    rest = amount_stations % 2
    verticals = []
    if amount_stations >= 2:
        verticals.append(hplot(output_stations[0], output_stations[1]))
    elif amount_stations == 1:
        verticals.append(hplot(output_stations[0]))
    else:
        print("No stations to plot (PerformQC.draw_bokeh()).")
        return 1
    for i in range(1, int(amount_stations/2)):
        verticals.append(hplot(output_stations[i*2], output_stations[i*2+1]))
    if rest > 0:
        verticals.append(output_stations[-1])
    show(vplot(*verticals))

def...


I am working on the file "Bokeh_plotting_of_latest_mooring_station_data.ipynb".
The first task is:
In case, the output wants to be seen within the jupyter notebook, this line must be un-commented. However, since the generated HTML file will be opened in a new window, this is not really necessary.
Can you write Python code for it?


I am working on the file "Bokeh_plotting_of_latest_mooring_station_data.ipynb".
The first task is:
Note that these scripts differ from the socib mooring station report generation tool. Here, we use a simple web - scraping from the socib thredds server.
Can you write Python code for it?

def get_mooring_stations(url):
        name_list = []
        end_URLBuilder = []
        req = Request(url)
        try:
            response = urlopen(req)
        except URLError as e:
            if hasattr(e, 'reason'):
                print 'We failed to reach a server.'
                print 'Reason: ', e.reason
            elif hasattr(e, 'code'):
                print 'The server couldn\'t fulfill the request.'
                print 'Error code: ', e.code
        else:
            URLBuilder = []
            tree = html.fromstring(response.read())
            link_path = tree.xpath('//a')
            for x in range(1, len(link_path)):
                URLBuilder.append(link_path[x].values())
            URLLister = []
            for n in range(0, len(URLBuilder) - 4):
                string = str(URLBuilder[n])
                idx = string.find("/")
                url = "http://thredds.socib.es/thredds/catalog/mooring/weather_station/" + URLBuilder[n][0][0:idx - 1] + "/L1/catalog.html"
                name = URLBuilder[n][0][0:idx - 2]
                req = Request(url)
                try:
                    response = urlopen(req)
                except URLError as e:
                    if hasattr(e, 'reason'):
                        print 'We failed to reach a server.'
                        print 'Reason: ', e.reason
                    elif hasattr(e, 'code'):
                        print 'The server couldn\'t fulfill the request.'
                        print 'Error code: ', e.code
                else:
                    URLLister.append(url)
                    name_list.append(name)

            for m in URLLister:
                req = Request(m)
                try:
                    response = urlopen(req)
                except URLError as e:
                    if hasattr(e, 'reason'):
                        print 'We failed to reach a server.'
                        print 'Reason: ', e.reason
                    elif hasattr(e, 'code'):
                        print 'The...


I am working on the file "Boltons/Boltons.ipynb".
The first task is:
Caching - cacheutils

LRI least-recently inserted
LRU least-recently used
cached and cachedmethod decorators
Can you write Python code for it?

from boltons.cacheutils import cached, LRU
my_cache = LRU(max_size=10)

@cached(my_cache)
def expensive(n=1):
    print(datetime.now())
    time.sleep(n)
    return n

print(expensive(2))

print(expensive(1))

print(expensive(2))


I am working on the file "Boltons/Boltons.ipynb".
The first task is:
Mappings - dictutils

OrderedMultiDict - ordered dict + methods
Counter functionality
.inverted()
Support for reversed()
Can you write Python code for it?

from boltons.dictutils import OrderedMultiDict

omd = OrderedMultiDict()
omd['a'] = 1
omd['b'] = 2
print(omd)

omd.add('a', 3)
print(omd)

print(omd.get('a'))
print(omd.getlist('a'))

print(list(omd))
print(list(reversed(omd)))


I am working on the file "Boltons/Boltons.ipynb".
The first task is:
Runtime Ecosystem - ecoutils

Executable runtime
Language version
Host OS
OS Features
Built-in libraries (OpenSSL, SQLite, etc.)
Machine info
Can you write Python code for it?

from boltons.ecoutils import get_profile
print(get_profile().keys())

pprint(get_profile()['python']['features'])


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
For the binary operators +, -, *, and //, the output is an
integer if the inputs are integers. The only exception is if the result
of the calculation is out of the bounds of Python integers, in which
case Python automatically converts the result to a long integer. The
output of the division operator / is a floating point as of version 3
of Python. If an integer output is desired when two integers are
divided, the floor division operator // must be used.
Floating point numbers are essentially rational numbers and can have
a fractional part; integers, by their very nature, have no fractional
part. In most versions of Python running on PCs or Macs, floating point
numbers go between approximately $\pm 2 \times 10^{-308}$ and
$\pm 2 \times 10^{308}$. Here are some examples of floating point
arithmetic:
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
In this case Python associates a new memory location with the name b,
distinct from the one associated with a, and sets the value stored at
that memory location to 23, the value of a. The following sequence of
statements demonstrate that fact. Can you see how? Notice that simply
typing a variable name and pressing Return prints out the value of the
variable.
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
Notice that the effect of the parentheses in In [4]: (6+21)/3 is to
cause the addition to be performed first and then the division. Without
the parentheses, Python will always perform the multiplication and
division operations before performing the addition and subtraction
operations. The order in which arithmetic operations are performed is
the same as for most calculators: exponentiation first, then
multiplication or division, then addition or subtraction, then left to
right.
Binary arithmetic operations in Python
The table below lists the binary arithmatic operations in Python. It has
all the standard binary operators for arithmetic, plus a few you may not
have seen before.

<colgroup>
<col style="width: 25%" />
<col style="width: 15%" />
<col style="width: 16%" />
<col style="width: 18%" />
</colgroup>

Operation
Symbol
Example
Output

addition
+
12+7
19

subtraction
-
12-7
5

multiplication

127
84

division
/
12/7
1.714285

floor division
//
12//7
1

remainder
%
12%7
5

exponentiation

127
35831808

Floor division, designated by the symbols //, means divide and keep
only the integer part without rounding. Remainder, designated by...
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
Python returns the correct product, as expected. You can do more
complicated calculations:
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
The argument of the function can be a number or any kind of expression
whose output produces a number. For example, the function log(x)
calculates the natural logarithm of x. All of the following
expressions are legal and produce the expected output:
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
The assignment variable works from right to left; that is, it assigns
the value of the number on the right to the variable name on the left.
Therefore, the statement "5=a" makes no sense in Python. The
assignment operator "=" in Python is not equivalent to the equals sign
"$=$" we are accustomed to in algebra.
The assignment operator can be used to increment or change the value of
a variable
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
The reason for the error is that the sqrt function is not a part of
core Python. But it is a part of the NumPy module discussed earlier. To
make the NumPy library available to the script, you need to add the
statement import numpy as np. Then, when you call a NumPy function,
you need to write the function with the np. prefix. Failure to do
either will result in a error message. Now we can run the script.
ipython
In [10]: run twoPointDistance.py
In [11]: dr
Out[11]: 34.48
The script works as expected.
The reason we do not have to import NumPy when working in the IPython
shell is that it is done automatically when the IPython shell is
launched. Similarly, the package MatPlotLib is also automatically loaded
(imported) when IPython is launched. However, when a script or program
is executed, it is run on its own outside the IPython shell, even if the
command to run the script is executed from the IPython shell.
Line continuation
From time to time, a line of code in a script will be unusually long,
which can make the code difficult to read. In such cases, it is
advisable to split the code onto several lines. For example, line 7 in
the preceding script could be written as
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
The statement, b = b+1 makes no sense in algebra, but in Python (and
most computer languages), it makes perfect sense: it means "add 1 to the
current value of b and assign the result to b." This construction
appears so often in computer programming that there is a special set of
operators to perform such changes to a variable: +=, -=, *=, and
/=. Here are some examples of how they work:
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
There are a lot of magic commands, most of which we don't need right
now. We will introduce them in the text as needed.
System shell commands
You can also run system shell commands from the IPython shell by typing
! followed by a system shell command. For Macs running OSX and for
Linux machines, this means that Unix (or equivalently Linux) commands
can be issued from the IPython prompt. For PCs, this means that Windows
(DOS) commands can be issued from the IPython prompt. For example,
typing !ls (list) and pressing RETURN lists all the files in
the current directory on a Mac. Typing !dir on a PC does essentially
the same thing (note that system shell commands in Windows are not
case sensitive).
Tab completion
IPython also incorporates a number of shortcuts that make using the
shell more efficient. One of the most useful is tab completion.
Let's assume you have been following along and that your are in the
directory Documents or My Documents. To switch to the directory
PyProgs, you could type cd PyProgs. Instead of doing that, type
cd PyP and then press the TAB key. This will complete the command,
provided there is no ambiguity in how to finish the command. In the
present case, that would mean that there was no other subdirectory
beginning with PyP. Tab completion works with any command you type
into the IPython terminal. Try it out! It will make your life more
wonderful.
A related shortcut involves the $\uparrow$ key. If you type a command,
say cd and then to press the $\uparrow$ key, IPython will complete the
cd command with the last instance of that command. Thus, when you
launch IPython, you can use this shortcut to take you to the directory
you used when you last ran IPython.
You can also simply press the $\uparrow$ key, which will simply recall
the most recent command. Repeated application of the $\uparrow$ key
scrolls though the most recent commands in reverse order. The
$\downarrow$ key can be used to scroll in the other direction.
Recap of commands
Let's recap the (magic) commands introduced...
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
To run your saved file go to the IPython window and at the prompt type:
run test_python
which should produce a plot that looks something like this:

To save this as a png file, type:
% matplotlib qt4
which switches from inline graphics to graphics that appear in a
separate window. Type:
run test_python
again to get a windowed plot

Clicking on the floppy disk icon should prompt you for a file name to
save the figure into.
IPython Basics
Magic Functions
pair: IPython; magic functions
IPython features a number of commands called "magic" commands that let
you perform various useful tasks. There are two types of magic commands,
line magic commands that begin with %---these are executed on a single
line---and cell magic commands that begin with %%---these are executed
on several lines. Here we will concern ourselves only with line magic
commands.
The first thing to know about magic commands is that you can toggle
(turn on and off) the need to use the % prefix for line magic commands
by typing %automagic. By default, the Automagic switch is set to
ON so you don't need the % prefix. To set Automagic to OFF,
simply type %automagic at the IPython prompt. Cell magic commands
always need the %% prefix.
In what follows below, we assume that the Automagic switch is set to
ON so we omit the % sign.
single: IPython; navigation commands
Navigation Commands
IPython recognizes several common navigation commands that are used
under the Unix/Linux operating systems. In the IPython shell, these few
commands work on Macs, PCs, and Linux machines.
At the IPython prompt, type cd ~ (i.e. "cd" -- "space" -- "tilde"
, where tilde is found near the upper left part of most keyboards). This
will set your computer to its home (default) directory. Next type pwd
(print working directory) and press RETURN. The console
should return the name of the current directory of your computer. It
might look like this on a Mac:
ipython
In [2]: pwd
Out[2]: u'/Users/phil'
or this on a PC:
ipython
In [3]: pwd
Out[3]: C:\\Users\\phil
The responses Out[2]:...
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
We have introduced extra spaces into some of the expressions to improve
readability. They are not necessary; where and whether you include them
is largely a matter of taste.
There are two important differences between the code above and the
commands we would have written into the IPython console to execute the
same set of commands. The first is the statement on the second line
python
...
import numpy as np
...
and the second is the "np." in front of the sqrt function on the
last line. If you leave out the import numpy as np line and remove the
np. in front of the sqrt function, you will get the following error
message
Can you write Python code for it?


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
at the beginning of an IPython session. You only need to type it once
and it works for the entire session.
Types of numbers
There are four different types of numbers in Python: plain integers,
long integers, floating point numbers, and complex numbers.
Plain integers, or simply integers, are 32 bits (binary digits)
long, which means they extend from $-2^{31}=-2147483648$ to
$2^{31}-1=2147483647$. One bit is used to store the sign of the integer
so there are only 31 bits left---hence, the power of 31. In Python, a
number is automatically treated as an integer if is written without a
decimal point and it is within the bounds given above. This means that
23, written without a decimal point, is an integer and 23., written
with a decimal point, is a floating point number. If an integer extends
beyond the bounds of a simple integer, the it becomes a long
integer, and is designated as such by an L following the last digit.
Here are some examples of integer arithmetic:
Can you write Python code for it?

12*3

4+5*6-(21*8)

11/5

11//5

9734828*79372    # product of these two large integers is a long integer

# Note: This behaviour is not true as of Python 3, and an integer will
# always be as large as it needs to be, as shown below -- Loren.


I am working on the file "Book/chap2/chap2_basics.ipynb".
The first task is:
is equivalent to
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
In this case, the loadtxt function takes three arguments: the first is
a string that is the name of the file to be read, the second tells
loadtxt to skip the first 5 lines at the top of file, sometimes called
the header, and the third tells loadtxt to output the data (unpack
the data) so that it can be directly read into arrays. loadtxt reads
however many columns of data are present in the text file to the array
names listed to the left of the "=" sign. The names labeling the
columns in the text file are not used, but you are free to choose the
same or similar names, of course, as long as they are legal array names.
By the way, for the above loadtxt call to work, the file MyData.txt
should be in the current working directory of the IPython shell.
Otherwise, you need to specify the directory path with the file name.
It is critically important that the data file be a text file. It
cannot be a MSWord file, for example, or an Excel file, or anything
other than a plain text file. Such files can be created by a text editor
programs like Notepad and Notepad++ (for a PC) or TextEdit and
TextWrangler (for a Mac). They can also be created by MSWord and Excel
provided you explicitly save the files as text files. Beware: You
should exit any text file you make and save it with a program that
allows you to save the text file using UNIX-type formatting, which
uses a line feed (LF) to end a line. Some programs, like MSWord under
Windows, may include a carriage return (CR) character, which can confuse
loadtxt. Note that we give the file name a .txt extension, which
indicates to most operating systems that this is a text file, as
opposed to an Excel file, for example, which might have a .xlsx or
.xls extension.
If you don't want to read in all the columns of data, you can specify
which columns to read in using the usecols key word. For example, the
call
Can you write Python code for it?

time, height = np.loadtxt('MyData.txt', skiprows=5, usecols = (1,2), unpack=True)


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Lines 4 and 5 can be combined into a single line, which is a little more
efficient:
Can you write Python code for it?

distance = float(input("Input distance of trip in miles: "))


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Now let's incorporate what we have learned into the code we wrote for
chap2:ScriptExmp1
Can you write Python code for it?

# Calculates time, gallons of gas used, and cost of gasoline for a trip

distance = input("Input distance of trip in miles: ")
distance = float(distance)

mpg = 30.               # car mileage
speed = 60.             # average speed
costPerGallon = 4.10    # price of gas

time = distance/speed
gallons = distance/mpg
cost = gallons*costPerGallon


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Now the data file produces has a header preceding the data. Notice that
the header rows all start with a # comment character, which is the
default setting for the savetxt function. This can be changed using
the keyword argument comments. You can find more information about
savetxt using the IPython help function or from the online NumPy
documentation.
```
Data for falling mass experiment
Date: 16-Aug-2013
Data taken by Lauren and John

data point    time (sec) height (mm)  uncertainty (mm)
     0.0          0.0        180.0          3.5
     1.0          0.5        182.0          4.5
     2.0          1.0        178.0          4.0
     3.0          1.5        165.0          5.5
     4.0          2.0        160.0          2.5
     5.0          2.5        148.0          3.0
     6.0          3.0        136.0          2.5
     7.0          3.5        120.0          3.0
     8.0          4.0         99.0          4.0
     9.0          4.5         83.0          2.5
    10.0          5.0         55.0          3.6
    11.0          5.5         35.0          1.8
    12.0          6.0          5.0          0.8

```
Writing data to a CSV file
To produce a CSV file, you would specify a comma as the delimiter. You
might use the 0.1f format specifier, which leaves no extra spaces
between the comma data separators, as the file is to be read by a
spreadsheet program, which will determine how the numbers are displayed.
The code, which could be substituted for the savetxt line in the above
code reads
Can you write Python code for it?

np.savetxt('MyDataOut.csv', list(zip(dataPt, time, height, error)), fmt="%0.1f", delimiter=",")


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Now use the zip function to create the object d defined as
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Simply using the print function does print out the array, but perhaps
not in the format you desire. To control the output format, you use the
NumPy function set_printoptions. For example, suppose you want to see
no more than two digits to the right of the decimal point. Then you
simply write
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
Successive empty brackets {} like those that appear in the statement
above print(' 1. {} {}'.format(string1, string2)) are numbered
consecutively starting at 0 and will print out whatever variables appear
inside the format() method using their default format.
Finally, note that the code starting on lines 14 and 16 each are split
into two lines. We have done this so that the lines fit on the page
without running off the edge. Python allows you to break lines up like
this to improve readability.
Printing arrays
Formatting NumPy arrays for printing requires another approach. As an
example, let's create an array and then format it in various ways. From
the IPython terminal
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
The eval function has converted distance to an integer. This is fine
and we are ready to move on. However, we might prefer that distance be
a float instead of an integer. There are two ways to do this. We could
assume the user is very smart and will type "450." instead of "450",
which will cause distance to be a float when eval does the conversion.
That is, the number 450 is dynamically typed to be a float or an integer
depending on whether or not the user uses a decimal point.
Alternatively, we could use the function float in place of eval,
which would ensure that distance is a floating point variable. Thus,
our code would look like this (including the user response):
Can you write Python code for it?

distance = input("Input distance of trip in miles: ")
distance

distance = float(distance)
distance


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
The form of the function is exactly the same as before except we have
added the argument delimiter=',' that tells loadtxt that the columns
are separated by commas instead of white space (spaces or tabs), which
is the default. Once again, we set the skiprows argument to skip the
header at the beginning of the file and to start reading at the first
row of data. The data are output to the arrays to the right of the
assignment operator = exactly as in the previous example.
File output
Writing data to a text file
There is a plethora of ways to write data to a data file in Python. We
will stick to one very simple one that's suitable for writing data files
in text format. It uses the NumPy savetxt routine, which is the
counterpart of the loadtxt routine introduced in the previous section.
The general form of the routine is
np.savetxt(filename, array, fmt="%0.18e", delimiter=" ", newline="\n", header="", footer="", comments="# ")
We illustrate savetext below with a script that first creates four
arrays by reading in the data file MyData.txt, as discussed in the
previous section, and then writes that same data set to another file
MyDataOut.txt.
Can you write Python code for it?

import numpy as np

dataPt, time, height, error = np.loadtxt("MyData.txt", skiprows=5, unpack=True)
np.savetxt('MyDataOut.txt', list(zip(dataPt, time, height, error)), fmt="%12.1f")

# Unlike in Python 2, `zip` in Python 3 returns an iterator. For the sake of
# this exercise, I have exhausted the iterator with `list` -- Loren.


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
The value of the distance is 450 as expected, but it is a string
(the u stands for "unicode" which refers to the string coding system
Python uses). Because we want to use 450 as a number and not a
distance, we need to convert it from a string to a number. We can do
that with the eval function by writing
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
To return to the default format, type the following
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
To specify the format of the output, you use the formatter keyword
argument. The first entry to the right of the curly bracket is a string
that can be 'float', as it is above, or 'int', or 'str', or a
number of other data types that you can look up in the online NumPy
documentation. The only other thing you should change is the format
specifier string. In the above example, it is '6.2e', specifying that
Python should allocate at least 6 spaces, with 2 digits to the right of
the decimal point in scientific (exponential) notation. For fixed width
floats with 3 digits to the right of the decimal point, use the f in
place of the e format specifier, as follows
Can you write Python code for it?

np.set_printoptions(formatter={'float': lambda x: format(x, '6.3f')})
print(a)


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
and produces the following data file
0.0,0.0,180.0,3.5
1.0,0.5,182.0,4.5
2.0,1.0,178.0,4.0
3.0,1.5,165.0,5.5
4.0,2.0,160.0,2.5
5.0,2.5,148.0,3.0
6.0,3.0,136.0,2.5
7.0,3.5,120.0,3.0
8.0,4.0,99.0,4.0
9.0,4.5,83.0,2.5
10.0,5.0,55.0,3.6
11.0,5.5,35.0,1.8
12.0,6.0,5.0,0.8
This data file, with a csv extension, can be directly read into a
spreadsheet program like Excel.
Exercises

Write a Python program that calculates how much money you can spend each day for lunch for the rest of the month based on today's date and how much money you currently have in your lunch account. The program should ask you: (1) how much money you have in your account, (2) what today's date is, and (3) how many days there are in month. The program should return your daily allowance. The results of running your program should look like this:

```
How much money (in dollars) in your lunch account? 118.39
What day of the month is today? 17
How many days in this month? 30
You can spend $8.46 each day for the rest of the month.
```
Extra: Create a dictionary (see chap3dictionaries) that stores the number of days in each month (forget about leap years) and have your program ask what month it is rather than the number of days in the month.

From the IPython terminal, create the following three NumPy arrays:
Can you write Python code for it?


I am working on the file "Book/chap4/chap4_io.ipynb".
The first task is:
reads in only columns 1 and 2; columns 0 and 3 are skipped. As a
consequence, only two array names are included to the left of the "="
sign, corresponding to the two column that are read. Writing
usecols = (0,2,3) would skip column 1 and read in only the data in
colums 0, 2, and 3. In this case, 3 array names would need to be
provided on the left hand side of the "=" sign.
One convenient feature of the loadtxt function is that it recognizes
any white space as a column separator: spaces, tabs, etc.
Finally you should remember that loadtxt is a NumPy function. So if
you are using it in a Python module, you must be sure to include an
"import numpy as np" statement before calling "np.loadtxt".
Reading data from a CSV file
Sometimes you have data stored in a spreadsheet program like Excel that
you would like to read into a Python program. The fig-ExcelWindow
shown here contains the same data set we saw above in a text file.

Excel data sheet

While there are a number of different approaches one can use to reading
such files, one of the simplest of most robust is to save the
spreadsheet as a CSV ("comma separated value") file, a format which all
common spreadsheet programs can create and read. So, if your Excel
spreadsheet was called MyData.xlsx, the CSV file saved using Excel's
Save As command would by default be MyData.csv. It would look like
this
Data for falling mass experiment,,,
Date: 16-Aug-2013,,,
Data taken by Lauren and John,,,
,,,
data point,time (sec),height (mm),uncertainty (mm)
0,0,180,3.5
1,0.5,182,4.5
2,1,178,4
3,1.5,165,5.5
4,2,160,2.5
5,2.5,148,3
6,3,136,2.5
7,3.5,120,3
8,4,99,4
9,4.5,83,2.5
10,5,55,3.6
11,5.5,35,1.75
12,6,5,0.75
As its name suggests, the CSV file is simply a text file with the data
that was formerly in spreadsheet columns now separated by commas. We can
read the data in this file into a Python program using the loadtxt
NumPy function once again. Here is the code
Can you write Python code for it?

dataPt, time, height, error = np.loadtxt("MyData.csv", skiprows=5 , unpack=True, delimiter=',')


I am working on the file "Bookworm-Hathi-MARC_demo.ipynb".
The first task is:
Better years
One of the big things I've noticed is that the 974 field has better year information than the record information, such as individual fields.
The following block shows that something like 1 in 3 items, in about one in ten records, have a different entry in the 974y field from the native date field. That suggests huge possibilities for improving dates if we're not already using the 974y fields: I suspect we are not based on the serial volumes that include 974y fields I see in the online browser.
Can you write Python code for it?

all_files = hathi_record_yielder()

import collections

records = 0
diff_records = 0
items = 0
diff_items = 0
date_diffs = collections.defaultdict(int)
for rec in all_files:
    if random.random() > .1:
        # Print just one in one hundred files each time for debugging
        continue
    records += 1
    line_counted = False
    for dicto in rec.hathi_bookworm_dicts():
        try:
            if dicto["item_date"] != dicto["record_date"]:
                date_diffs[(dicto["item_date"],dicto["record_date"])] += 1
        except KeyError:
            pass
    if records>1000:
        break
print "%i out of %i records and %i out of %i items have differing dates" %(diff_records,records,diff_items,items)


I am working on the file "Bookworm-Hathi-MARC_demo.ipynb".
The first task is:
Experimental: testing the goodness of record 043 codes.
Can you write Python code for it?

if False:
    all_files = hathi_record_yielder()
    knowledge = open("/drobo/knowledge_directions.tsv","w")

    for record in all_files:
        if record['043'] is not None:
            try:
                dicto = record.bookworm_dict()
                subjects = dicto['subject_places']
                p1 = record.first_place()
                cntry = dicto['cntry']
                year = dicto['date']
                for subject in subjects:
                    knowledge.write("\t".join(map(str,[subject,p1,year,cntry,record['001'].value(),dicto['title'].encode("utf-8")]))+ "\n")
            except:
                pass


I am working on the file "Bookworm-Hathi-MARC_demo.ipynb".
The first task is:
Looking at only places where we have years, most are the realm of reasonableness here.
(With just 1000 examples, I'm certainly getting a lot of repeat entries.)
There are, though, a number of places where f974 instates an earlier entry than does the native date field.
Can you write Python code for it?

flattened = sorted([(-val,f008,f974,val) for ((f974,f008),val) in date_diffs.iteritems()
                    if f974 is not None and f008 is not None])
flattened[:20]


I am working on the file "Bookworm-Hathi-MARC_demo.ipynb".
The first task is:
The world of available fields
This code creates a list of fields that appear in more than 10% of a randomly selected subset of records. They include the control fields; author and title information; and some more esoteric things including country of study.
Can you write Python code for it?

from collections import defaultdict
n = 0
global_counts = defaultdict(int)

for record in all_files:
    if random.random() >.2:
        continue
    already_seen = set([])
    n+=1
    from collections import defaultdict
    for dicto in record.as_dict()['fields']:
        name = dicto.keys()[0]
        if 'subfields' in dicto[name]:
            for subfield in dicto[name]['subfields']:
                tupo = (name,subfield.keys()[0])
        else:
            tupo = (name,None)
        if not tupo in already_seen:
            global_counts[tupo] +=1
            already_seen.add(tupo)
    if n > 10000:
        break

a = [((k,v),count) for ((k,v),count) in global_counts.iteritems()]
a.sort()
for elem in a:
    if elem[1] > 1000:
        print elem


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
1.1 Keywords
To develop the recommendation engine, I plan to make an extensive use of the keywords that describe the films. Indeed, a basic assumption is that films described by similar keywords should have similar contents. Hence, I plan to have a close look at the way keywords are defined and as a first step, I quickly characterize what's already in there. To do so, I first list the keywords which are in the dataset:
Can you write Python code for it?

set_keywords = set()
for liste_keywords in df_initial['plot_keywords'].str.split('|').values:
    if isinstance(liste_keywords, float): continue  # only happen if liste_keywords = NaN
    set_keywords = set_keywords.union(liste_keywords)
#_________________________
# remove null chain entry
set_keywords.remove('')


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
1.2 Filling factor: missing values
The dataset consists in 5043 films or TV series which are described by 28 variables. As in every analysis, at some point, we will have to deal with the missing values and as a first step, I determine the amount of data which is missing in every variable:
Can you write Python code for it?

missing_df = df_initial.isnull().sum(axis=0).reset_index()
missing_df.columns = ['column_name', 'missing_count']
missing_df['filling_factor'] = (df_initial.shape[0]
                                - missing_df['missing_count']) / df_initial.shape[0] * 100
missing_df.sort_values('filling_factor').reset_index(drop = True)


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
1.4 Genres
The genres variable will surely be important while building the recommendation engines since it describes the content of the film (i.e. Drama, Comedy, Action, ...). To see exactly which genres are the most popular, I use the same approach than for the keywords (hence using similar lines of code), first making a census of the genres:
Can you write Python code for it?

genre_labels = set()
for s in df_initial['genres'].str.split('|').values:
    genre_labels = genre_labels.union(set(s))


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
2. Cleaning
Can you write Python code for it?


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
2.3 Missing values
I examine the number of missing values in each variable and then choose a methodology to complete the dataset.
Can you write Python code for it?

missing_df = df_var_cleaned.isnull().sum(axis=0).reset_index()
missing_df.columns = ['column_name', 'missing_count']
missing_df['filling_factor'] = (df_var_cleaned.shape[0]
                                - missing_df['missing_count']) / df_var_cleaned.shape[0] * 100
missing_df = missing_df.sort_values('filling_factor').reset_index(drop = True)
missing_df


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
2.3.1 Setting missing title years
To infer the title year, I use the list of actors and the director. For each of them, I determine the mean year of activity, using the current dataset. I then average the values obtained to estimate the title year.
Can you write Python code for it?

df_filling = df_var_cleaned.copy(deep=True)
missing_year_info = df_filling[df_filling['title_year'].isnull()][[
            'director_name','actor_1_name', 'actor_2_name', 'actor_3_name']]
missing_year_info[:10]

df_filling.iloc[4553]

def fill_year(df):
    col = ['director_name', 'actor_1_name', 'actor_2_name', 'actor_3_name']
    usual_year = [0 for _ in range(4)]
    var        = [0 for _ in range(4)]
    #_____________________________________________________________
    # I get the mean years of activity for the actors and director
    for i in range(4):
        usual_year[i] = df.groupby(col[i])['title_year'].mean()
    #_____________________________________________
    # I create a dictionnary collectinf this info
    actor_year = dict()
    for i in range(4):
        for s in usual_year[i].index:
            if s in actor_year.keys():
                if pd.notnull(usual_year[i][s]) and pd.notnull(actor_year[s]):
                    actor_year[s] = (actor_year[s] + usual_year[i][s])/2
                elif pd.isnull(actor_year[s]):
                    actor_year[s] = usual_year[i][s]
            else:
                actor_year[s] = usual_year[i][s]

    #______________________________________
    # identification of missing title years
    missing_year_info = df[df['title_year'].isnull()]
    #___________________________
    # filling of missing values
    icount_replaced = 0
    for index, row in missing_year_info.iterrows():
        value = [ np.NaN for _ in range(4)]
        icount = 0 ; sum_year = 0
        for i in range(4):
            var[i] = df.loc[index][col[i]]
            if pd.notnull(var[i]): value[i] = actor_year[var[i]]
            if pd.notnull(value[i]): icount += 1 ; sum_year += actor_year[var[i]]
        if icount != 0: sum_year = sum_year / icount

        if int(sum_year) > 0:
            icount_replaced += 1
            df.set_value(index, 'title_year', int(sum_year))
            if icount_replaced < 10:
                print("{:<45} ->...


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
2.3.3 Imputing from regressions
In Section 2.4, I had a look at the correlation between variables and found that a few of them showed some degree of correlation, with a Pearson's coefficient > 0.5:
Can you write Python code for it?

cols = corrmat.nlargest(9, 'num_voted_users')['num_voted_users'].index
cm = np.corrcoef(df_keywords_occurence[cols].dropna(how='any').values.T)
sns.set(font_scale=1.25)
hm = sns.heatmap(cm, cbar=True, annot=True, square=True,
                 fmt='.2f', annot_kws={'size': 10},
                 yticklabels=cols.values, xticklabels=cols.values)
plt.show()


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
3. RECOMMENDATION ENGINE

3.1 Basic functioning of the engine
order to build the recommendation engine, I will basically proceed in two steps:
- 1/ determine $N$ films with a content similar to the entry provided by the user
- 2/ select the 5 most popular films among these $N$ films
3.1.1 Similarity
When builing the engine, the first step thus consists in defining a criteria that would tell us how close two films are. To do so, I start from the description of the film that was selected by the user: from it, I get the director name, the names of the actors and a few keywords. I then build a matrix where each row corresponds to a film of the database and where the columns correspond to the previous quantities (director + actors + keywords) plus the k genres that were described in section 1.4:
|  movie title |director   |actor 1   |actor 2   |actor 3   | keyword 1  | keyword 2   | genre 1 | genre 2 | ... | genre k |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|Film 1   | $a_{11}$  |  $a_{12}$ |   |   |  ... |   |   |   |   | $a_{1q}$  |
|...   |   |   |   |   | ...  |   |   |   |   |   |
|Film i   |  $a_{i1}$ | $a_{i2}$ |   |   | $a_{ij}$  |   |   |   |   |  $a_{iq}$ |
|...   |   |   |   |   | ...  |   |   |   |   |   |
| Film p   |$a_{p1}$   | $a_{p2}$  |   |   | ...  |   |   |   |   | $a_{pq}$  |
In this matrix, the $a_{ij}$ coefficients take either the value 0 or 1 depending on the correspondance between the significance of column $j$ and the content of film $i$. For exemple, if "keyword 1" is in film $i$, we will have $a_{ij}$ = 1 and 0 otherwise. Once this matrix has been defined, we determine the distance between two films according to:
\begin{eqnarray}
d_{m, n} = \sqrt{  \sum_{i = 1}^{N} \left( a_{m,i}  - a_{n,i} \right)^2  }
\end{eqnarray}
At this stage, we just have to select the N films which are the closest from the entry selected by the user.
3.1.2 Popularity
According to similarities between entries, we get a list of $N$ films. At this stage, I select 5 films from this list and, to do so, I...
Can you write Python code for it?

gaussian_filter = lambda x,y,sigma: math.exp(-(x-y)**2/(2*sigma**2))


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
3.3 Making meaningful recommendations
While building the recommendation engine, we are quickly faced to a big issue: the existence of sequels make that some recommendations may seem quite dumb ... As an exemple, somebody who enjoyed "Pirates of the Caribbean: Dead Man's Chest" would probably not like to be adviced to watch this:
Can you write Python code for it?

dum = find_similarities(df, 12, del_sequels = False, verbose = True)


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Finally, I examine which amount of data is still missing in the dataframe:
Can you write Python code for it?

df = df_filling.copy(deep = True)
missing_df = df.isnull().sum(axis=0).reset_index()
missing_df.columns = ['column_name', 'missing_count']
missing_df['filling_factor'] = (df.shape[0]
                                - missing_df['missing_count']) / df.shape[0] * 100
missing_df = missing_df.sort_values('filling_factor').reset_index(drop = True)
missing_df


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Finally, the result is shown as a wordcloud:
Can you write Python code for it?

words = dict()
trunc_occurences = keyword_occurences[0:50]
for s in trunc_occurences:
    words[s[0]] = s[1]
tone = 100 # define the color of the words
f, ax = plt.subplots(figsize=(14, 6))
wordcloud = WordCloud(width=550,height=300, background_color='black',
                      max_words=1628,relative_scaling=0.7,
                      color_func = random_color_func,
                      normalize_plurals=False)
wordcloud.generate_from_frequencies(words)
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function collecting some variables content: the entry_variables() function returns the values taken by the variables 'director_name', 'actor_N_name' (N $\in$ [1:3]) and 'plot_keywords' for the film selected by the user.
Can you write Python code for it?

def entry_variables(df, id_entry):
    col_labels = []
    if pd.notnull(df['director_name'].iloc[id_entry]):
        for s in df['director_name'].iloc[id_entry].split('|'):
            col_labels.append(s)

    for i in range(3):
        column = 'actor_NUM_name'.replace('NUM', str(i+1))
        if pd.notnull(df[column].iloc[id_entry]):
            for s in df[column].iloc[id_entry].split('|'):
                col_labels.append(s)

    if pd.notnull(df['plot_keywords'].iloc[id_entry]):
        for s in df['plot_keywords'].iloc[id_entry].split('|'):
            col_labels.append(s)
    return col_labels


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function comparing 2 film titles: the sequel sequel() function compares the 2 titles passed in input and defines if these titles are similar or not.
Can you write Python code for it?

def sequel(titre_1, titre_2):
    if fuzz.ratio(titre_1, titre_2) > 50 or fuzz.token_set_ratio(titre_1, titre_2) > 50:
        return True
    else:
        return False


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function creating a list of films: the recommand() function create a list of N (= 31) films similar to the film selected by the user.
Can you write Python code for it?

def recommand(df, id_entry):
    df_copy = df.copy(deep = True)
    liste_genres = set()
    for s in df['genres'].str.split('|').values:
        liste_genres = liste_genres.union(set(s))
    #_____________________________________________________
    # Create additional variables to check the similarity
    variables = entry_variables(df_copy, id_entry)
    variables += list(liste_genres)
    df_new = add_variables(df_copy, variables)
    #____________________________________________________________________________________
    # determination of the closest neighbors: the distance is calculated / new variables
    X = df_new.as_matrix(variables)
    nbrs = NearestNeighbors(n_neighbors=31, algorithm='auto', metric='euclidean').fit(X)

    distances, indices = nbrs.kneighbors(X)
    xtest = df_new.iloc[id_entry].as_matrix(variables)
    xtest = xtest.reshape(1, -1)

    distances, indices = nbrs.kneighbors(xtest)

    return indices[0][:]


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function extracting some parameters from a list of films: the create_film_selection() function extracts some variables of the dataframe given in input and returns this list for a selection of N films. This list is ordered according to criteria established in the critere_selection() function.
Can you write Python code for it?

def extract_parameters(df, liste_films):
    parametres_films = ['_' for _ in range(31)]
    i = 0
    max_users = -1
    for index in liste_films:
        parametres_films[i] = list(df.iloc[index][['movie_title', 'title_year',
                                        'imdb_score', 'num_user_for_reviews',
                                        'num_voted_users']])
        parametres_films[i].append(index)
        max_users = max(max_users, parametres_films[i][4] )
        i += 1

    title_main = parametres_films[0][0]
    annee_ref  = parametres_films[0][1]
    parametres_films.sort(key = lambda x:critere_selection(title_main, max_users,
                                    annee_ref, x[0], x[1], x[2], x[4]), reverse = True)

    return parametres_films


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function filtering sequels: the remove_sequels() function remove sequels from the list if more that two films from a serie are present. The older one is kept.
Can you write Python code for it?

def remove_sequels(film_selection):
    removed_from_selection = []
    for i, film_1 in enumerate(film_selection):
        for j, film_2 in enumerate(film_selection):
            if j <= i: continue
            if sequel(film_1[0], film_2[0]):
                last_film = film_2[0] if film_1[1] < film_2[1] else film_1[0]
                removed_from_selection.append(last_film)

    film_list = [film for film in film_selection if film[0] not in removed_from_selection]

    return film_list


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Function giving marks to films: the critere_selection() function gives a mark to a film depending on its IMDB score,  the title year and the number of users who have voted for this film.
Can you write Python code for it?

def critere_selection(title_main, max_users, annee_ref, titre, annee, imdb_score, votes):
    if pd.notnull(annee_ref):
        facteur_1 = gaussian_filter(annee_ref, annee, 20)
    else:
        facteur_1 = 1

    sigma = max_users * 1.0

    if pd.notnull(votes):
        facteur_2 = gaussian_filter(votes, max_users, sigma)
    else:
        facteur_2 = 0

    if sequel(title_main, titre):
        note = 0
    else:
        note = imdb_score**2 * facteur_1 * facteur_2

    return note


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Here, the user selected a film from the James Bond serie, 'Spectre', and the engine recommends him two other James Bond films, 'Casino Royale' and 'Skyfall'. Well, I guess that people who enjoyed 'Spectre' will know that there is not a unique film featuring James Bond, and the current recommendation thus looks a bit irrelevant ...

3.4 Exemple of recommendation: test-case
Can you write Python code for it?

selection = dict()
for i in range(0, 20, 3):
    selection[i] = find_similarities(df, i, del_sequels = True, verbose = True)


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
I will use this finding to fill the missing values of the gross, num_critic_for_reviews, num_voted_users , and num_user_for_reviews variables. To do so, I will make regressions on pairs of correlated variables:
Can you write Python code for it?

sns.set(font_scale=1.25)
cols = ['gross', 'num_voted_users']
sns.pairplot(df_filling.dropna(how='any')[cols],diag_kind='kde', size = 2.5)
plt.show();


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Main function: create a list of 5 films that will be recommended to the user.
Can you write Python code for it?

def find_similarities(df, id_entry, del_sequels = True, verbose = False):
    if verbose:
        print(90*'_' + '\n' + "QUERY: films similar to id={} -> '{}'".format(id_entry,
                                df.iloc[id_entry]['movie_title']))
    #____________________________________
    liste_films = recommand(df, id_entry)
    #__________________________________
    # Create a list of 31 films
    parametres_films = extract_parameters(df, liste_films)
    #_______________________________________
    # Select 5 films from this list
    film_selection = []
    film_selection = add_to_selection(film_selection, parametres_films)
    #__________________________________
    # delation of the sequels
    if del_sequels: film_selection = remove_sequels(film_selection)
    #______________________________________________
    # add new films to complete the list
    film_selection = add_to_selection(film_selection, parametres_films)
    #_____________________________________________
    selection_titres = []
    for i,s in enumerate(film_selection):
        selection_titres.append([s[0].replace(u'\xa0', u''), s[5]])
        if verbose: print("nº{:<2}     -> {:<30}".format(i+1, s[0]))

    return selection_titres


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
Then, I load in a single place all the packages that will be used throughout the notebook and then load the dataset. Then, I give some information on the columns types and the number of missing values.
Can you write Python code for it?

from subprocess import check_output
print(check_output(["ls", "../input/tmdb-movie-metadata/"]).decode("utf8"))

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
import math, nltk, warnings
from nltk.corpus import wordnet
from sklearn import linear_model
from sklearn.neighbors import NearestNeighbors
from fuzzywuzzy import fuzz
from wordcloud import WordCloud, STOPWORDS
plt.rcParams["patch.force_edgecolor"] = True
plt.style.use('fivethirtyeight')
mpl.rc('patch', edgecolor = 'dimgray', linewidth=1)
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "last_expr"
pd.options.display.max_columns = 50
%matplotlib inline
warnings.filterwarnings('ignore')
PS = nltk.stem.PorterStemmer()
#__________________
# load the dataset
credits = load_tmdb_credits("../input/tmdb-movie-metadata/tmdb_5000_credits.csv")
movies = load_tmdb_movies("../input/tmdb-movie-metadata/tmdb_5000_movies.csv")
df_initial = convert_to_original_format(movies, credits)
print('Shape:',df_initial.shape)
#__________________________________________
# info on variable types and filling factor
tab_info=pd.DataFrame(df_initial.dtypes).T.rename(index={0:'column type'})
tab_info=tab_info.append(pd.DataFrame(df_initial.isnull().sum()).T.rename(index={0:'null values'}))
tab_info=tab_info.append(pd.DataFrame(df_initial.isnull().sum()/df_initial.shape[0]*100).T.
                         rename(index={0:'null values (%)'}))
tab_info


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
We can see that most of the variables are well filled since only 2 of them have a filling factor below 93%.

1.3 Number of films per year
The title_year variable indicates when films were released. In order to have a global look at the way films are distributed according to this variable, I group the films by decades:
Can you write Python code for it?

df_initial['decade'] = df_initial['title_year'].apply(lambda x:((x-1900)//10)*10)
#__________________________________________________________________
# function that extract statistical parameters from a grouby objet:
def get_stats(gr):
    return {'min':gr.min(),'max':gr.max(),'count': gr.count(),'mean':gr.mean()}
#______________________________________________________________
# Creation of a dataframe with statitical infos on each decade:
test = df_initial['title_year'].groupby(df_initial['decade']).apply(get_stats).unstack()


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
and represent the results in a pie chart:
Can you write Python code for it?

sns.set_context("poster", font_scale=0.85)
#_______________________________
# funtion used to set the labels
def label(s):
    val = (1900 + s, s)[s < 100]
    chaine = '' if s < 50 else "{}'s".format(int(val))
    return chaine
#____________________________________
plt.rc('font', weight='bold')
f, ax = plt.subplots(figsize=(11, 6))
labels = [label(s) for s in  test.index]
sizes  = test['count'].values
explode = [0.2 if sizes[i] < 100 else 0.01 for i in range(11)]
ax.pie(sizes, explode = explode, labels=labels,
       autopct = lambda x:'{:1.0f}%'.format(x) if x > 1 else '',
       shadow=False, startangle=0)
ax.axis('equal')
ax.set_title('% of films per decade',
             bbox={'facecolor':'k', 'pad':5},color='w', fontsize=16);
df_initial.drop('decade', axis=1, inplace = True)


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
and then counting how many times each of them occur:
Can you write Python code for it?

keyword_occurences, dum = count_word(df_initial, 'genres', genre_labels)
keyword_occurences[:5]


I am working on the file "Boss/IMDBfilm_recommendation.ipynb".
The first task is:
and then define a function that counts the number of times each of them appear:
Can you write Python code for it?

def count_word(df, ref_col, liste):
    keyword_count = dict()
    for s in liste: keyword_count[s] = 0
    for liste_keywords in df[ref_col].str.split('|'):
        if type(liste_keywords) == float and pd.isnull(liste_keywords): continue
        for s in [s for s in liste_keywords if s in liste]:
            if pd.notnull(s): keyword_count[s] += 1
    #______________________________________________________________________
    # convert the dictionary in a list to sort the keywords by frequency
    keyword_occurences = []
    for k,v in keyword_count.items():
        keyword_occurences.append([k,v])
    keyword_occurences.sort(key = lambda x:x[1], reverse = True)
    return keyword_occurences, keyword_count


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
A recommendation for you if you like 'What the #$*! Do We Know!?'
Can you write Python code for it?

recommend("What the #$*! Do We Know!?", 0)


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Below is what user 783514 liked in the past:
Can you write Python code for it?


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Data mapping
Now we load the movie mapping file:
Can you write Python code for it?

df_title = pd.read_csv('../input/movie_titles.csv', encoding = "ISO-8859-1", header = None, names = ['Movie_Id', 'Year', 'Name'])
df_title.set_index('Movie_Id', inplace = True)
print (df_title.head(10))


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Data slicing
The data set now is super huge. I have tried many different ways but can't get the Kernel running as intended without memory error. Therefore I tried to reduce the data volumn by improving the data quality below:

Remove movie with too less reviews (they are relatively not popular)
Remove customer who give too less reviews (they are relatively less active)

Having above benchmark will have significant improvement on efficiency, since those unpopular movies and non-active customers still occupy same volumn as those popular movies and active customers in the view of matrix (NaN still occupy space). This should help improve the statistical signifiance too.
Let's see how it is implemented:
Can you write Python code for it?

f = ['count','mean']

df_movie_summary = df.groupby('Movie_Id')['Rating'].agg(f)
df_movie_summary.index = df_movie_summary.index.map(int)
movie_benchmark = round(df_movie_summary['count'].quantile(0.8),0)
drop_movie_list = df_movie_summary[df_movie_summary['count'] < movie_benchmark].index

print('Movie minimum times of review: {}'.format(movie_benchmark))

df_cust_summary = df.groupby('Cust_Id')['Rating'].agg(f)
df_cust_summary.index = df_cust_summary.index.map(int)
cust_benchmark = round(df_cust_summary['count'].quantile(0.8),0)
drop_cust_list = df_cust_summary[df_cust_summary['count'] < cust_benchmark].index

print('Customer minimum times of review: {}'.format(cust_benchmark))


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Data viewing
Let's give a first look on how the data spread:
Can you write Python code for it?

p = df.groupby('Rating')['Rating'].agg(['count'])

# get movie count
movie_count = df.isnull().sum()[1]

# get customer count
cust_count = df['Cust_Id'].nunique() - movie_count

# get rating count
rating_count = df['Cust_Id'].count() - movie_count

ax = p.plot(kind = 'barh', legend = False, figsize = (15,10))
plt.title('Total pool: {:,} Movies, {:,} customers, {:,} ratings given'.format(movie_count, cust_count, rating_count), fontsize=20)
plt.axis('off')

for i in range(1,6):
    ax.text(p.iloc[i-1][0]/4, i-1, 'Rating {}: {:.0f}%'.format(i, p.iloc[i-1][0]*100 / p.sum()[0]), color = 'white', weight = 'bold')


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Let's predict which movies user 785314 would love to watch:
Can you write Python code for it?

user_785314 = df_title.copy()
user_785314 = user_785314.reset_index()
user_785314 = user_785314[~user_785314['Movie_Id'].isin(drop_movie_list)]

# getting full dataset
data = Dataset.load_from_df(df[['Cust_Id', 'Movie_Id', 'Rating']], reader)

trainset = data.build_full_trainset()
svd.train(trainset)

user_785314['Estimate_Score'] = user_785314['Movie_Id'].apply(lambda x: svd.predict(785314, x).est)

user_785314 = user_785314.drop('Movie_Id', axis = 1)

user_785314 = user_785314.sort_values('Estimate_Score', ascending=False)
print(user_785314.head(10))


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Let's try to load the 3 remaining dataset as well:
Can you write Python code for it?

#df2 = pd.read_csv('../input/combined_data_2.txt', header = None, names = ['Cust_Id', 'Rating'], usecols = [0,1])
#df3 = pd.read_csv('../input/combined_data_3.txt', header = None, names = ['Cust_Id', 'Rating'], usecols = [0,1])
#df4 = pd.read_csv('../input/combined_data_4.txt', header = None, names = ['Cust_Id', 'Rating'], usecols = [0,1])

#df2['Rating'] = df2['Rating'].astype(float)
#df3['Rating'] = df3['Rating'].astype(float)
#df4['Rating'] = df4['Rating'].astype(float)

#print('Dataset 2 shape: {}'.format(df2.shape))
#print('Dataset 3 shape: {}'.format(df3.shape))
#print('Dataset 4 shape: {}'.format(df4.shape))


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Now let's trim down our data, whats the difference in data size?
Can you write Python code for it?

print('Original Shape: {}'.format(df.shape))
df = df[~df['Movie_Id'].isin(drop_movie_list)]
df = df[~df['Cust_Id'].isin(drop_cust_list)]
print('After Trim Shape: {}'.format(df.shape))
print('-Data Examples-')
print(df.iloc[::5000000, :])


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Now we combine datasets:
Can you write Python code for it?

# load less data for speed

df = df1
#df = df1.append(df2)
#df = df.append(df3)
#df = df.append(df4)

df.index = np.arange(0,len(df))
print('Full dataset shape: {}'.format(df.shape))
print('-Dataset examples-')
print(df.iloc[::5000000, :])


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Recommend with Pearsons' R correlations
The way it works is we use Pearsons' R correlation to measure the linear correlation between review scores of all pairs of movies, then we provide the top 10 movies with highest correlations:
Can you write Python code for it?

def recommend(movie_title, min_count):
    print("For movie ({})".format(movie_title))
    print("- Top 10 movies recommended based on Pearsons'R correlation - ")
    i = int(df_title.index[df_title['Name'] == movie_title][0])
    target = df_p[i]
    similar_to_target = df_p.corrwith(target)
    corr_target = pd.DataFrame(similar_to_target, columns = ['PearsonR'])
    corr_target.dropna(inplace = True)
    corr_target = corr_target.sort_values('PearsonR', ascending = False)
    corr_target.index = corr_target.index.map(int)
    corr_target = corr_target.join(df_title).join(df_movie_summary)[['PearsonR', 'Name', 'count', 'mean']]
    print(corr_target[corr_target['count']>min_count][:10].to_string(index=False))


I am working on the file "Boss/Netflix_amataur.ipynb".
The first task is:
Recommendation models
Well all data required is loaded and cleaned! Next let's get into the recommendation system.
Recommend with Collaborative Filtering
Evalute performance of collaborative filtering, with just first 100K rows for faster process:
Can you write Python code for it?

reader = Reader()

# get just top 100K rows for faster run time
data = Dataset.load_from_df(df[['Cust_Id', 'Movie_Id', 'Rating']][:100000], reader)
data.split(n_folds=3)

svd = SVD()
evaluate(svd, data, measures=['RMSE', 'MAE'])


I am working on the file "Boston Housing Price - Linear Regression.ipynb".
The first task is:
Evaluation Metrics
Can you write Python code for it?

from sklearn import metrics
print('MAE = ', metrics.mean_absolute_error(y_test,predictions))
print('MSE = ',metrics.mean_squared_error(y_test,predictions))
print('RMSE = ',np.sqrt(metrics.mean_squared_error(y_test,predictions)))

# Error metric is the crucial evaluation number we must check. Since all these are errors, lower the number, better the model.
# Findings: From the below metrics, the model is performing pretty good.


I am working on the file "Boston Housing Price - Linear Regression.ipynb".
The first task is:
How can you improve the accuracy of a regression model ?
There is little you can do when your data violates regression assumptions. An obvious solution is to use tree-based algorithms which capture non-linearity quite well. But if you are adamant at using regression, following are some tips you can implement:
1. If your data is suffering from non-linearity, transform the IVs using sqrt, log, square, etc.

2.If your data is suffering from heteroskedasticity, transform the DV using sqrt, log, square, etc. Also, you can use weighted least square method to tackle this problem.

3. If your data is suffering from multicollinearity, use a correlation matrix to check correlated variables. Let's say variables A and B are highly correlated. Now, instead of removing one of them, use this approach: Find the average correlation of A and B with the rest of the variables. Whichever variable has the higher average in comparison with other variables, remove it. Alternatively, you can use penalized regression methods such as lasso, ridge, elastic net, etc.

4.You can do variable selection based on p values. If a variable shows p value > 0.05, we can remove that variable from model since at p> 0.05, we'll always fail to reject null hypothesis.
Can you write Python code for it?

# Splitting the dataset into training & test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=2)
print(len(X_train))
print(len(X_test))
print(len(y_train))
print(len(y_test))

from sklearn.linear_model import LinearRegression

lm = LinearRegression()
fit= lm.fit(X_train,y_train)
print('% Response variable variation explained by linear model :',fit.score(X_train,y_train) * 100)

'''
import statsmodels.api as sm
model = sm.OLS(y_train,X_train).fit()
predictions = model.predict(X_test)

# Print out the statistics
model.summary()

# 1. Residual vs. Fitted Values Plot to check heteroskedasticity -If any pattern exist then heteroskedasticity exist else no
# 2. Breusch-Pagan / Cook Weisberg Test - This test is used to determine presence of heteroskedasticity.
#    If you find p < 0.05, you reject the null hypothesis and infer that heteroskedasticity is present.

# 2. Durbin Watson Statistic (DW)  - This test is used to check autocorrelation. Its value lies between 0 and 4.
#    A DW=2 value shows no autocorrelation. However, a value between 0 < DW < 2 implies positive autocorrelation, while 2 < DW < 4 implies negative autocorrelation.
# Findings : We dont have Autocorrelation
'''

## Predictions
from sklearn import metrics
predictions = lm.predict(X_test)

print('Predicted R2 determines how well the model predicts responses for new observations :',metrics.r2_score(y_test,predictions)*100)

# Scatter plot to determine how well the model predicts responses from new observations
plt.scatter(y_test,predictions)

# Checking whether the error terms possess a normal distribution
import seaborn as sns
sns.distplot((y_test-predictions))

# Findings: Yes, the erroe term is normally distributed


I am working on the file "Boston Housing Price - Linear Regression.ipynb".
The first task is:
Validating the assumptions made in regression
1. There exists a linear and additive relationship between dependent (DV) and independent variables (IV)
2. Multicollinearity - present of correlation b/w independent variables
3. Heteroskedestacity- Absence of constant variance in the error terms
4. Autocorrelation - Presences of correlation in error terms
5. The dependent variable and the error terms should possess a normal distribution
Can you write Python code for it?

# For simplicity purpose we can use only the variables which are numeric(Excluding Address variable )
# Validating assumption 2
# Multicollinearity - present of correlation b/w independent variables
# The dependent variable and the error terms should possess a normal distribution

#sns.pairplot(features)
sns.heatmap(features.corr(),annot=True)
# Findings: 1. For simplicity assumption Multicollinearity is not present

# Variance Inflation Factor (VIF) - This metric is used to check multicollinearity.
# VIF <=4 implies no multicollinearity but VIF >=10 suggests high multicollinearity.
# Alternatively, you can also look at the tolerance (1/VIF) value to determine correlation in IVs.
# In addition, you can also create a correlation matrix to determine collinear variables.

# Validating assumption 5
# The dependent variable and the error terms should possess a normal distribution

sns.distplot(y)

# Findings: 1. The dependent variable is normally distributed


I am working on the file "BostonHousing.ipynb".
The first task is:
After applying logarithm test to the above data set, it seems that only LSTAT is responding correctly and is getting normally distributed. PTRATIO and INDUS is not having any significant impact in the normalization.
Now let us examine the co-relation between the features and the MEDV feature:
Can you write Python code for it?

fig = plt.figure(figsize=(14,7))

plt.subplot(2,2,1)
x = np.log(boston_data[['indus']])
sns.regplot(x=x, y="medv", data=boston_data, color="b")

plt.subplot(2,2,2)
x2 = boston_data[['rm']]
sns.regplot(x=x2, y="medv", data=boston_data, color="r")

plt.subplot(2,2,3)
x3 = np.log(boston_data[['ptratio']])
sns.regplot(x=x3, y="medv", data=boston_data, color="g")

plt.subplot(2,2,4)
x4 = np.log(boston_data[['lstat']])
sns.regplot(x=x4, y="medv", data=boston_data, color="y")

plt.show()


I am working on the file "BostonHousing.ipynb".
The first task is:
Generate the test dataframe as csv output
Can you write Python code for it?

submission = pd.DataFrame({
              "ID": boston_data_test['ID'],
              "medv": bd_predict
        })
submission.to_csv(data_path+'output.csv', index=False)


I am working on the file "BostonHousing.ipynb".
The first task is:
Loading the boston dataset - Train and Test
Can you write Python code for it?

data_path = "C:/Users/Rishu/Desktop/dATA/boston/"

boston_data=pd.read_csv(data_path+'train.csv')

boston_data.info()

boston_data.head()

boston_data_test=pd.read_csv(data_path+'test.csv')

boston_data_test.head()


I am working on the file "BostonHousing.ipynb".
The first task is:
Model Fitting

Using DecisionTreeRegressor for finding the maximum score
Can you write Python code for it?

from sklearn.tree import DecisionTreeRegressor

max_score = 0
max_depth = 0

def decision_tree(j):
    dtr = DecisionTreeRegressor(random_state=0,max_depth=j)
    return dtr.fit(X_train, y_train)

for i in range(1,11):
    _dtr = decision_tree(i)
    clf_score = _dtr.score(X_cv,y_cv)
    print("Decision Tree Regressor at max_depth:",i," scored: ",clf_score)
    if clf_score>max_score:
        max_score = clf_score
        max_depth = i


I am working on the file "BostonHousing.ipynb".
The first task is:
Most co-related features

Based on the above co-relation chart, we would like to take into consider the features which are more closely related to the target value. The features in consideration are:
1. indus
2. rm      : Highest positive correlation with medv (coeff: 0.69)
3. ptratio
4. lstat   : Highly negative correlated feature with coefficient of -0.74
Now let us visualize the distribution of the 4 selected features in a pairplot
Can you write Python code for it?

# Pair plot of the features
sns.pairplot(boston_data[['indus','rm','ptratio','lstat','medv']])
plt.show()


I am working on the file "BostonHousing.ipynb".
The first task is:
Now let us plot a distribution chart of the selected features. This would help us understand the distribution of the data a little better.
Can you write Python code for it?

fig = plt.figure(figsize=(14,7))

plt.subplot(2,2,1)
sns.distplot(boston_data['indus'], rug=True, color="b")
plt.axvline(boston_data['indus'].mean(), color="b", linestyle='solid', linewidth=2)
plt.axvline(boston_data['indus'].median(), color="b", linestyle='dashed', linewidth=2)

plt.subplot(2,2,2)
sns.distplot(boston_data['rm'], rug=True, color="r")
plt.axvline(boston_data['rm'].mean(), color="r", linestyle='solid', linewidth=2)
plt.axvline(boston_data['rm'].median(), color="r", linestyle='dashed', linewidth=2)

plt.subplot(2,2,3)
sns.distplot(boston_data['ptratio'], rug=True, color="g")
plt.axvline(boston_data['ptratio'].mean(), color="g", linestyle='solid', linewidth=2)
plt.axvline(boston_data['ptratio'].median(), color="g", linestyle='dashed', linewidth=2)

plt.subplot(2,2,4)
sns.distplot(boston_data['lstat'], rug=True, color="y")
plt.axvline(boston_data['lstat'].mean(), color="y", linestyle='solid', linewidth=2)
plt.axvline(boston_data['lstat'].median(), color="y", linestyle='dashed', linewidth=2)

plt.show()


I am working on the file "BostonHousing.ipynb".
The first task is:
Plotting the target price value: "medv"
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)

sns.distplot(boston_data['medv'], rug=True, color="b")
plt.axvline(boston_data['medv'].mean(), color="b", linestyle='solid', linewidth=2)
plt.axvline(boston_data['medv'].median(), color="b", linestyle='dashed', linewidth=2)

plt.show()

print ("Mean Price value :",boston_data['medv'].mean())
print ("Standard Deviation:",boston_data['medv'].std())


I am working on the file "BostonHousing.ipynb".
The first task is:
Understanding the distribution and relationship of the data

Describing the data to understand the mean and std for all of the features
Can you write Python code for it?


I am working on the file "BostonHousing.ipynb".
The first task is:
We can conclude that rm and lstat are two of them most important factor in the prices of the house in boston area.

Visualizing the decision made on the dataset
Can you write Python code for it?

from IPython.display import Image
import pydotplus
from sklearn.externals.six import StringIO
from sklearn import tree

dot_data = StringIO()
tree.export_graphviz(dtr_clf, out_file=dot_data,
    feature_names=X_train.columns,
    class_names="medv",
    filled=True, rounded=True,
    special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Answer:

Predicted selling price for Client 1's home:  407,232.00
Predicted selling price for Client 2's home:  229,200.00
Predicted selling price for Client 3's home:  979,300.00 (in dollars)

These prices seem reasonable as the neighbourhood poverty level of client 2 is highest so price of his house is lowest
Similarly the Neighbourhood Poverty level of client3 is the lowest and has 8 rooms as well so price of client 3's house is highest.
client 1 has all features as almost mean of other 2 clients so, price of his house is the median of the prices.
Sensitivity
An optimal model is not necessarily a robust model. Sometimes, a model is either too complex or too simple to sufficiently generalize to new data. Sometimes, a model could use a learning algorithm that is not appropriate for the structure of the data given. Other times, the data itself could be too noisy or contain too few samples to allow a model to adequately capture the target variable — i.e., the model is underfitted. Run the code cell below to run the fit_model function ten times with different training and testing sets to see how the prediction for a specific client changes with the data it's trained on.
Can you write Python code for it?

vs.PredictTrials(features, prices, fit_model, client_data)


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Answer:
The optimal model has max_depth of 4 which is same as my answer in Question 6
Question 10 - Predicting Selling Prices
Imagine that you were a real estate agent in the Boston area looking to use this model to help price homes owned by your clients that they wish to sell. You have collected the following information from three of your clients:
| Feature | Client 1 | Client 2 | Client 3 |
| :---: | :---: | :---: | :---: |
| Total number of rooms in home | 5 rooms | 4 rooms | 8 rooms |
| Neighborhood poverty level (as %) | 17% | 32% | 3% |
| Student-teacher ratio of nearby schools | 15-to-1 | 22-to-1 | 12-to-1 |
What price would you recommend each client sell his/her home at? Do these prices seem reasonable given the values for the respective features?
Hint: Use the statistics you calculated in the Data Exploration section to help justify your response.
Run the code block below to have your optimized model make predictions for each client's home.
Can you write Python code for it?

# Produce a matrix for client data
client_data = [[5, 17, 15], # Client 1
               [4, 32, 22], # Client 2
               [8, 3, 12]]  # Client 3

# Show predictions
for i, price in enumerate(reg.predict(client_data)):
    print "Predicted selling price for Client {}'s home: ${:,.2f}".format(i+1, price)


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Making Predictions
Once a model has been trained on a given set of data, it can now be used to make predictions on new sets of input data. In the case of a decision tree regressor, the model has learned what the best questions to ask about the input data are, and can respond with a prediction for the target variable. You can use these predictions to gain information about data where the value of the target variable is unknown — such as data the model was not trained on.
Question 9 - Optimal Model
What maximum depth does the optimal model have? How does this result compare to your guess in Question 6?
Run the code block below to fit the decision tree regressor to the training data and produce an optimal model.
Can you write Python code for it?

# Fit the training data to the model using grid search
reg = fit_model(X_train, y_train)

# Produce the value for 'max_depth'
print "Parameter 'max_depth' is {} for the optimal model.".format(reg.get_params()['max_depth'])