I am working on the file "Archive/Actors.ipynb".
The first task is:
To indicate that an actor requires one GPU, we pass in num_gpus=1 to ray.remote. Note that in order for this to work, Ray must have been started with some GPUs, e.g., via ray.init(num_gpus=2). Otherwise, when you try to instantiate the GPU version with NeuralNetOnGPU.remote(), an exception will be thrown saying that there aren’t enough GPUs in the system.
When the actor is created, it will have access to a list of the IDs of the GPUs that it is allowed to use via ray. get_gpu_ids(). This is a list of integers, like [], or [1], or [2, 5, 6]. Since we passed in ray. remote(num_gpus=1), this list will have length one.
We can put this all together as follows.
Can you write Python code for it?

# Restart the kernel before running this to initialize Ray with specified
# number of GPUs.
import os
import ray
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
ray.init(num_gpus=8)
def construct_network():
    x = tf.placeholder(tf.float32, [None, 784])
    y_ = tf.placeholder(tf.float32, [None, 10])
    W = tf.Variable(tf.zeros([784, 10]))
    b = tf.Variable(tf.zeros([10]))
    y = tf.nn.softmax(tf.matmul(x, W) + b)
    cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
    train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
    correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    return x, y_, train_step, accuracy
@ray.remote(num_gpus=1)
class NeuralNetOnGPU(object):
    def __init__(self, mnist_data):
        self.mnist = mnist_data
        # Set an environment variable to tell TensorFlow which GPUs to use. Note
        # that this must be done before the call to tf.Session.
        os.environ["CUDA_VISIBLE_DEVICES"] = ",".join([str(i) for i in
                                                       ray.get_gpu_ids()])

        with tf.Graph().as_default():
            with tf.device("/gpu:0"):
                # Allow this to run on CPUs if there aren't any GPUs.
                self.x, self.y_, self.train_step, self.accuracy = construct_network()
                config = tf.ConfigProto(allow_soft_placement=True)
                self.sess = tf.Session(config=config)
                # Initialize the network.
                init = tf.global_variables_initializer()
                self.sess.run(init)
    def train(self, num_steps):
        for _ in range(num_steps):
            batch_xs, batch_ys = self.mnist.train.next_batch(100)
            self.sess.run(self.train_step, feed_dict={self.x: batch_xs, self.y_: batch_ys})
    def get_accuracy(self):
        return self.sess.run(self.accuracy, feed_dict={self.x:...


I am working on the file "Archive/Actors.ipynb".
The first task is:
We can then define an actor for this network as follows.
Can you write Python code for it?

import os

# Define an actor that runs on GPUs. If there are no GPUs, then simply use
# ray.remote without any arguments and no parentheses.
@ray.remote(num_gpus=1)

class NeuralNetOnGPU(object):
    def __init__(self):
        # Set an environment variable to tell TensorFlow which GPUs to use.
        # Note that this must be done before the call to tf.Session.
        os.environ["CUDA_VISIBLE_DEVICES"] = ",".join([str(i) for i in
                                                       ray.get_gpu_ids()])
        with tf.Graph().as_default():
            with tf.device("/gpu:0"):
                self.x, self.y_, self.train_step, self.accuray = construct
                _network()
                # Allow this to run on CPUs if there aren't any GPUs
                self.sess = tf.Session(config=config)
                # Initialize the network.
                init = tf.global_variables_initializer()
                self.sess.run(init)


I am working on the file "Archive/Actors.ipynb".
The first task is:
We can then instantiate an actor and schedule a task on that actor as follows.
Can you write Python code for it?

pong = GymEnvironment.remote("Pong-v0")
pong.step.remote(0)  # Take action 0 in the simulator.


I am working on the file "Archive/Actors.ipynb".
The first task is:
When a1.increment.remote() is called, the following events happens.

A task is created.

The task is assigned directly to the local scheduler responsible for the actor by the driver's local scheduler. Thus, this scheduling procedure bypasses the global scheduler.

An object ID is returned.

We can then call ray.get on the object ID to retrieve the actual value.
Similarly, the call to a2.increment.remote() generates a task that is scheduled on the second Counter actor. Since these two tasks run on different actors, they can be executed in parallel (note that only actor methods will be scheduled on actor workers, regular remote functions will not be).
On the other hand, methods called on the same Counter actor are executed serially in the order that they are called. They can thus share state with one another, as shown below.
Can you write Python code for it?

# Create ten Counter actors.
counters = [Counter.remote() for _ in range(10)]

# Increment each Counter once and get the results. These tasks all happen
# in PARALLEL.
results = ray.get([c.increment.remote() for c in counters])
print(results)  # prints [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

# Increment the first Counter five times. These tasks are executed SERIALLY
# and share state.
results = ray.get([counters[0].increment.remote() for _ in range(5)])
print(results)  # prints [2, 3, 4, 5, 6]


I am working on the file "Archive/Actors.ipynb".
The first task is:
When an actor is instantiated, the following events happen.

A node in the cluster is chosen and a worker process is created on that node (by the local scheduler on that node) for the purpose of running methods called on the actor.

A Counter object is created on that worker and the Counter constructor is run.

Using an actor
We can schedule tasks on the actor by calling its methods.
Can you write Python code for it?

a1.increment.remote()  # ray.get returns 1
a2.increment.remote()  # ray.get returns 1


I am working on the file "Archive/Analysis/pressure-reservoir-notebook.ipynb".
The first task is:
Pressure Drop calculations
Collecting K-values of fittings, connections, etc...
Can you write Python code for it?


I am working on the file "Archive/Hyperparameter Optimization.ipynb".
The first task is:
Basic random search
Something that works surprisingly well is to try random values for the hyperparameters. For example, we can write a function that randomly generates hyperparameter configurations.
Can you write Python code for it?

def generate_hyperparameters():
    # Randomly choose values for the hyperparameters.
    return {"learning_rate": 10 ** np.random.uniform(-5, 5),
           "batch_size": np.random.randint(1, 100),
           "dropout": np.random.uniform(0, 1),
           "stddev": 10 ** np.random.uniform(-5, 5)}


I am working on the file "Archive/Hyperparameter Optimization.ipynb".
The first task is:
Here’s a different variant that uses the same principles. Divide each training session into a series of shorter training sessions. Whenever a short session finishes, if it still looks promising, then continue running it. If it isn’t doing well, then terminate it and start a new experiment.
Can you write Python code for it?

import numpy as np

def is_promising(model):
    # Return true if the model is doing well and false otherwise. In
    # practice, this function will want more information than just the
    # model.
    return np.random.choice([True, False])

# Start 10 experiments.
remaining_ids = []
for _ in range(10):
    experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters,
                                                         model=None)
    remaining_ids.append(experiment_id)

accuracies = []
for i in range(100):
    # Whenever a segment of an experiment finishes, decide if it looks
    # promising or not.
    ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
    experiment_id = ready_ids[0]
    current_accuracy, current_model = ray.get(experiment_id)
    accuracies.append(current_accuracy)

    if is_promising(experiment_id):
        # Continue running the experiment.
        experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters,
                                                             model=current_model)
    else:
        # Start a new experiment.
        experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters)

    remaining_ids.append(experiment_id)

print(accuracies)


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
Asynchronous Computation in Ray
Ray enables arbitrary Python functions to be executed asynchronously. This is done by designating a Python function as a $\bf \text{remote function}$.
For example, a normal Python function looks like this.
Can you write Python code for it?


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
Expressing dependencies between tasks
Programmers can express dependencies between tasks by passing the object ID output of one task as an argument to another task. For example, we can launch three tasks as follows, each of which depends on the previous task.
Can you write Python code for it?

@ray.remote
def f(x):
    return x + 1

x = f.remote(0)
y = f.remote(x)
z = f.remote(y)
ray.get(z)  # 3


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
If the remote object corresponding to the object ID x_id has not been created yet, the command ray.get(x_id) will wait until the remote object has been created.
A very common use case of ray.get is to get a list of object IDs. In this case, you can call ray. get(object_ids) where object_ids is a list of object IDs.
Can you write Python code for it?

result_ids = [ray.put(i) for i in range(10)]
ray.get(result_ids) # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
Remote functions
Whereas calling add1(1,2) returns 3 and causes the Python interpreter to block until the computation has finished, calling add2.remote(1, 2) immediately returns an object ID and creates a task. The task will be scheduled by the system and executed asynchronously (potentially on a different machine). When the task finishes executing, its return value will be stored in the object store.
Can you write Python code for it?

x_id = add2.remote(1, 2)
ray.get(x_id)  # 3


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
Remote functions $\bf \text{never}$ return actual values, they always return object IDs.
When the remote function is actually executed, it operates on $\bf \text{Python objects}$. That is, if the remote function was called with any object IDs, the system will retrieve the corresponding objects from the object store.
Note that a remote function can return multiple object IDs.
Can you write Python code for it?

@ray.remote(num_return_vals=3)
def return_multiple():
    return 1, 2, 3

a_id, b_id, c_id = return_multiple.remote()


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
The command ray.put(x) would be run by a worker process or by the driver process (the driver process is the one running your script). It takes a Python object and copies it to the local object store (here local means on the same node). Once the object has been stored in the object store, its value cannot be changed.
In addition, ray.put(x) returns an object ID, which is essentially an ID that can be used to refer to the newly created remote object. If we save the object ID in a variable with x_id = ray.put(x), then we can pass x_id into remote functions, and those remote functions will operate on the corresponding remote object.
The command ray.get(x_id) takes an object ID and creates a Python object from the corresponding remote object. For some objects like arrays, we can use shared memory and avoid copying the object. For other objects, this copies the object from the object store to the worker process’s heap. If the remote object corresponding to the object ID x_id does not live on the same node as the worker that calls ray.get(x_id), then the remote object will first be transferred from an object store that has it to the object store that needs it.
Can you write Python code for it?

x_id = ray.put("example")
ray.get(x_id)   # "example"


I am working on the file "Archive/Ray Project Tutorial Walkthrough.ipynb".
The first task is:
There is a sharp distinction between $\it \text{submitting a task}$ and $\it \text{executing the task}$. When a remote function is called, the task of executing that function is $\bf \text{submitted to a local scheduler}$, and object IDs for the outputs of the task are immediately returned. However, the task will not be executed until the system actually schedules the task on a worker. Task execution is not done lazily. The system moves the input data to the task, and the task will execute $\bf \text{as soon as}$ its input dependencies are available and there are enough resources for the computation.
$\bf \text{When a task is submitted, each argument may be passed in by value or by object ID}$. For example, these lines have the same behavior.
Can you write Python code for it?

add2.remote(1, 2)
add2.remote(1, ray.put(2))
add2.remote(ray.put(1), ray.put(2))


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
By default, Ray will use psutil.cpu_count() to determine the number of CPUs, and by default the number of GPUs will be zero.
Instead of thinking about the number of "worker" processes on each node, we prefer to think in terms of the quantities of CPU and GPU resources on each node and to provide the illusion of an infinite pool of workers. Tasks will be assigned to workers based on the availability of resources so as to avoid contention and not based on the number of available worker processes.
Connecting to an existing cluster
Once a Ray cluster has been started, the only thing you need in order to connect to it is the address of the Redis server in the cluster. In this case, your script will not start up or shut down any processes. The cluster and all of its processes may be shared between multiple scripts and multiple users. To do this, you simply need to know the address of the cluser's Redis server. This can be done with a command like the following.
Can you write Python code for it?


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
If there are GPUs available on the machine, you should specify this with the num_gpus argument. Similarly, you can also specify the number of CPUs with num_cpus.
Can you write Python code for it?


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
If you want a remote function to return multiple object IDs, you can do that by passing the num_return_vals argument into the remote decorator.
Can you write Python code for it?

@ray.remote(num_return_vals=2)
def f():
    return 1,2

x_id, y_id = f.remote()
ray.get(x_id) #1
#ray.get(y_id) #2


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
In this case, you cannot specify num_cpus or num_gpus in ray.init because that information is passed into the cluster when the cluster is started, not when your script is started.
ray.init (redis_address=None, node_ip_address=None, object_id_seed=None, num_workers=None, driver_mode=0, redirect_output=False, num_cpus=None, num_gpus=None, num_custom_resource=None, num_redis_shards=None, plasma_directory=None, huge_pages=False)
Parameters
* redis_address (str) - The address of the Redis server to connect to. If this address is not provided, then this command will start Redis, a global scheduler, a local scheduler, a plasma store, a plasma manager, and some workers. It will also kill these processes when Python exists.

object_id_seed (int) - Used to seed the deterministic generation of object IDs. The same value can be used across multiple runs of the same job in order to generate the object IDs in a consistent manner. However, the same ID should not be used for different jobs.

num_workers (int) - The number of workers to start. This is only provided if redis_address is not provided.

driver_mode (bool) - The mode in which to start the driver. This should be one of ray.SCRIPT_MODE, ray.PYTHON_MODE, and ray.SILENT_MODE.

redirect_output (bool) - True if stdout and stderr for all the processes should be redirected to files and false otherwise.

num_cpus (int) - Number of cpus the user wishes all local schedulers to be configured with.

num_gpus (int) - Number of gpus the user wishes all local schedulers to be configured with.

num_custom_resource (int) - The quantity of a user-defined custom resource that the local scheduler should be configured with. This flag is experimental and is subject to changes in the future.

num_redis_shards - The number of Redis shards to start in addition to the primary Redis shard.

plasma_directory - A directory where the Plasma memory mapped files will be created.

huge_pages - Boolean flag indicating whether to start the Object Store with hugetlbfs support. Requires...
Can you write Python code for it?

@ray.remote
def f(x):
    return x+1

x_id = f.remote(0)
ray.get(x_id)  # 1

#y_id = f.remote(x_id)
#ray.get(y_id)  # 2


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
It is easy to use this construct to create an infinite loop in which multiple tasks are executing, and whenever one task finishes, a new one is launched.
Can you write Python code for it?

@ray.remote
def f():
    return 10

# Start 5 tasks.
remaining_ids = [f.remote() for i in range(5)]
#print(ray.get(remaining_ids))
'''
The following few lines is for testing the behavior of the wait method
without specifying the number of returning object IDs. Comment it before
running the for loop in the original documentation.
'''
# # Whenever one task finishes, start a new one.
# ready_ids, remaining_ids, = ray.wait(remaining_ids, num_returns=3)
# # Get the available object and do something with it.
# print(ray.get(ready_ids))
# # Print out the remaining ids
# print(ray.get(remaining_ids))

'''
Observation: when the number of object IDs to be returned is not specified,
the wait method will automatically set it to 1.
'''
for _ in range(10):
    # actually this only works when num_returns = 1, otherwise it will
    # result in a dead kernel since finally the number of remaining IDs
    # will be 0
    ready_ids, remaining_ids = ray.wait(remaining_ids)
    #print(ray.get(remaining_ids))
    # Get the available object and do something with it.
    print(ray.get(ready_ids))
    # Start a new task. The number of remaining IDs will be consistent.
    remaining_ids.append(f.remote())


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
Starting Ray
There are two main ways in which Ray can be used. First, you can start all of the relevant Ray processes and shut them all down within the scope of a single script. Second, you can connect to and use an existing Ray cluster.
Starting and stopping a cluster within a script
One use case is to start all of the relevant Ray processes when you call ray.init and shut them down when the script exits. These processes include local and global schedulers, an object manager, a redis server, and more.
Note: this approach is limited to a single machine.
This can be done as follows.
Can you write Python code for it?


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
The main reason to use ray.put is that you want to pass the same large object into a number of tasks. By first doing ray.put and then passing the resulting object ID into each of the tasks, the large object is copied into the object store only once, whereas when we directly pass the object in, it is copied multiple times, which is not efficient.
Can you write Python code for it?

import numpy as np

@ray.remote
def f(x):
    pass

x = np.zeros(10 ** 6)

# Alternative 1: Here, x is copied into the object store 10 times.
[f.remote(x) for _ in range(10)]

# Alternative 2: Here, x is copied into the object store once.
x_id = ray.put(x)
[f.remote(x_id) for _ in range(10)]


I am working on the file "Archive/The Ray API.ipynb".
The first task is:
ray.remote (*args , **kwargs)
This decorator is used to define remote functions and to define actors.
Parameters
   * num_return_vals (int) - The number of object IDs that a call to this function should return.

num_cpus (int) - The number of CPUs needed to execute this function.

num_gpus (int) - The number of GPUs needed to execute this function.

num_custom_resource (int) - The quantity of a user-defined custom resource that is needed to execute this function. This flag is experimental and is subject to changes in the future.

max_calls (int) - The maximum number of tasks of this kind that can be run on a worker before the worker needs to be restarted.

checkpoint_interval (int) - The number of tasks to run between checkpoints of the actor state.

Getting Values from Object IDs
Object IDs can be converted into objects by calling ray.get on the object ID. Note that ray.get accepts either a single object ID or a list of object IDs.
Can you write Python code for it?

@ray.remote
def f():
    return {'key1': ['value']}

# Get one object ID.
ray.get(f.remote())  # {'key1': ['value']}

# Get a list of object IDs.
ray.get([f.remote() for _ in range(2)])  # [{'key1': ['value']}, {'key1': ['value']}]


I am working on the file "Are Models from a KFold CV similar enough to be IID?.ipynb".
The first task is:
Hmmm...
Ok, it is not perfect and we do have some outliers, but those distributions are damn close, given: we did use 250 folds in our cross validation, lets try with 3 of those to see what happens
Can you write Python code for it?

# Cross validate 3 models
cv_results = cross_validate(LinearRegression(), x_train, y_train, cv = 3,
                            return_estimator=True, scoring='neg_root_mean_squared_error')
A, B, C = cv_results["estimator"][:3]

A_scores_3 = mean_squared_error_sample(A, x_val, y_val)
B_scores_3 = mean_squared_error_sample(B, x_val, y_val)
C_scores_3 = mean_squared_error_sample(C, x_val, y_val)

# Plot the histogram
fig, ax = plt.subplots(dpi=100)
ax.hist([A_scores_3, B_scores_3, C_scores_3])
ax.set(title="Error Metric Distribution for Models A and B",
      xlabel="Score",
      ylabel="Frequecy")
ax.legend(["Model A", "Model B", "Model C"]);


I am working on the file "Are Models from a KFold CV similar enough to be IID?.ipynb".
The first task is:
Interesting...
As we can see, for Linear Regression models as the number K of Folds increase, the average distance of each model from the "center" model decreases, we interpret this as the models becoming more and more similar the more in-sample data they have to learn from.
Lets try something else, lets try to see how the metric distribution from two different models trained on the same CV iteration compare.
Can you write Python code for it?

# This function will sample the MSE for a model iteration number of times
def mean_squared_error_sample(model, x, true_y, samples=50, iterations=200):
    result = []
    for i in range(iterations):
        idx = np.random.choice(np.arange(len(x)), samples, replace=False)
        data = x.iloc[idx]
        values = true_y[idx]
        result.append(mean_squared_error(values, model.predict(data)))

    return np.array(result)

# First we split 30% of the data to use as Out-of-sample later
from sklearn.model_selection import train_test_split
x_train, x_val, y_train, y_val = train_test_split(x,y,test_size=0.3)
# Cross validate a bunch of models
cv_results = cross_validate(LinearRegression(), x_train, y_train, cv = 250,
                            return_estimator=True, scoring='neg_root_mean_squared_error')
A, B = cv_results["estimator"][:2]
# We are using this metric
from sklearn.metrics import mean_squared_error
A_scores_250 = mean_squared_error_sample(A, x_val, y_val)
B_scores_250 = mean_squared_error_sample(B, x_val, y_val)

# Plot the histogram
fig, ax = plt.subplots(dpi=100)
ax.hist([A_scores_250, B_scores_250])
ax.set(title="Error Metric Distribution for Models A and B",
      xlabel="Score",
      ylabel="Frequecy")
ax.legend(["Model A", "Model B"]);


I am working on the file "Are Models from a KFold CV similar enough to be IID?.ipynb".
The first task is:
Oh wow...
With only 3 folds the distributions look very different fro mone another, anyway, lets see what Mr. Kruskal and Mr. Wallis have to say about that
Can you write Python code for it?

from scipy.stats import kruskal

t_3, p_3 = kruskal(A_scores_3, B_scores_3, C_scores_3)
t_250, p_250 = kruskal(A_scores_250, B_scores_250)
print(f"3-Fold T-statistic is {t_3:.4f} and p-value is {p_3:.4e}")
print(f"250-Fold T-statistic is {t_250:.4f} and p-value is {p_250:.4e}")


I am working on the file "Are Models from a KFold CV similar enough to be IID?.ipynb".
The first task is:
Ok, those p-values won't allow us to reject $H_0$, which should mean that, as far as the scores go, each of our models is sampling from distributions with the same median.
Note: since we are generating a lot of models (mostly for the 250 fold test, we must correct our p-value to account for multiple testing, a simples correction, called the Bonferroni correction, tells us to divide our significance level $\alpha$ by the number of tests we are performing, sure, we are only computing one of them, but given model A and 249 other models, at $\alpha=0.05$ whe can expect to see around $249*0.05 \tilde= 13$ false positives, applying the Bonferroni correction, and assuming we will test A against each other model, our $\alpha$ is then $0.05/249 = 0.0002$
That settles it for models that come from the same CV procedure, but what about models that are trained with a different number of folds?
Can you write Python code for it?

# Here we do the same thing we did at the beggining
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_validate
mse_means = np.array([])
mse_std = np.array([])
avg_scaled_distances = []
folds = np.arange(50, len(x_train), 50)
for f in folds:
    # Cross validate a bunch of models
    results = cross_validate(LinearRegression(normalize=True), x_train, y_train, cv = f,
                             return_estimator= True, scoring="neg_root_mean_squared_error")
    coef = np.array([estimator.coef_ for estimator in results['estimator']])
    # Lets compute the "similarity" of our models
    avg_scaled_distances.append(np.mean(scaled_distances_from_centroid(coef)))

    # Lets also take a look at how our metric behaves as the number of folds increase, this will take time
    sampled_mse = np.array([mean_squared_error_sample(model,
                                                      x_val,
                                                      y_val,
                                                      iterations=100) for model in results['estimator']])
    # Compute mean
    mse_means = np.append(mse_means, sampled_mse.mean())
    # And standard deviation
    mse_std = np.append(mse_std, sampled_mse.std())

fig, ax = plt.subplots(dpi=100)
ax2 = ax.twinx()

ax.set(title="Distances from Parameter Vector to Centroid",
      xlabel="Number of Folds")
ax.set_ylabel("Avg. Distance to Centroid", color="cornflowerblue")
ax.set_ylim(0, 0.04)
ax.scatter(folds, avg_scaled_distances)

ax2.plot(folds, mse_means, 'k--', alpha=0.7, label="Avg MSE")
ax2.plot(folds, mse_means+mse_std, 'y--', alpha=0.7, label= "MSE Stdev.")
ax2.plot(folds, mse_means-mse_std, 'y--', alpha=0.7)
ax2.legend()
ax2.set_ylabel("MSE")
ax2.set_ylim(0, 50);


I am working on the file "Are Models from a KFold CV similar enough to be IID?.ipynb".
The first task is:
What kind of sorcery is this?
A simple and intuitive way to define similarity is to think of how close two things are to a reference point,
since "closeness" is a transitive property, that is: if A is close to C and B is close to C it is fair to say that A is close to B, we can try and compute how far our models are from a given point.
For our experiment we will be considering Linear Regression models, each model is uniquely defined by its coefficients and intercepts, as an array of numbers, those coefficients define a point in a n-dimensional space, we can then for a set of models (and a set of points) compute the average distance from the centroid, if a bunch of models is very similar, the average distance from the centroid will be smaller as the points that represent the coefficient vectors are clustered together.
As a safety measure we will normalize the distances to avoid the "curse of high dimensions"
Can you write Python code for it?

# This function computes the scaled distances from the centroid for a set of points
def scaled_distances_from_centroid(points):
    scaled_coef = points/np.max(np.abs(coef))
    scaled_centroid = scaled_coef.mean(axis=0)
    return np.array([np.linalg.norm(point-scaled_centroid) for point in scaled_coef])

# Import our Model and Cross Validation utility
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_validate
# Lets train 9 different sets, each one with increasingly more models
folds = np.arange(50, 500, 50)
# And store the avg of the (scaled) distances in an array
avg_scaled_distances = []
for f in folds:
    results = cross_validate(LinearRegression(normalize=True), x, y, cv = f,
                             return_estimator= True, scoring="neg_root_mean_squared_error")
    coef = np.array([estimator.coef_ for estimator in results['estimator']])
    avg_scaled_distances.append(np.mean(scaled_distances_from_centroid(coef)))

fig, ax = plt.subplots(dpi=100)
ax.set(title="Distances from Parameter Vector to Centroid",
      xlabel="Number of Folds",
      ylabel="Avg. Distance to Centroid")
ax.set_ylim(0, 0.04)
ax.scatter(folds, avg_scaled_distances);


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
Before closing the session we close our file:
Can you write Python code for it?


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
Building the initial PSTH and stablizing them
For the citronellal response of Neuron 1 we do:
Can you write Python code for it?

target_mean = 3
n_stim_citron = len(e060817citron[0])
n1citron = np.sort(np.concatenate(e060817citron[0]))
left = -5+citron_onset
right = 6+citron_onset
n1citron = n1citron[np.logical_and(left <= n1citron,n1citron <= right)]-citron_onset
bin_width_citron = np.ceil(target_mean/n_stim_citron/e060817_spont_nu[0]*1000)/1000
n1citron_bin = np.arange(-5,6+bin_width_citron,bin_width_citron)
n1citron_counts, n1citron_bin = np.histogram(n1citron,n1citron_bin)
n1citron_stab = 2*np.sqrt(n1citron_counts+0.25)
n1citron_x = n1citron_bin[:-1]+bin_width_citron/2


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
Choosing the initial bin size
The bin width should be chosen large enough to have a few events per bin most of the time. We set this width such that an expected number of 3 events per bin was obtained using the spontaneous frequency estimated from 60 seconds long recordings without stimulus presentation—more specifically, the width was set to the smallest millisecond larger or equal to the targeted count divided by the product of the spontaneous frequency and the number of trials.
For the neurons of the data set we get the following spontaneous discharge rates:
Can you write Python code for it?

e060817_spont_nu = [len(f["e060817/Neuron"+str(i)+"/spont"])/60
                    for i in range(1,4)]
print("The spontaneous discharge rates are:")
for i in range(len(e060817_spont_nu)):
    print("  Neuron {0}: {1:.2f} (Hz)".format(i+1,e060817_spont_nu[i]))


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
For the even and odd stimuli terpineol responses we do:
Can you write Python code for it?

n1terpi_even = np.sort(np.concatenate(e060817terpi[0][0:20:2]))
n1terpi_even = n1terpi_even[np.logical_and(left <= n1terpi_even,n1terpi_even <= right)]-terpi_onset
bin_width_terpi_even = np.ceil(target_mean/n_stim_terpi/e060817_spont_nu[0]*1000)/1000/2
n1terpi_even_bin = np.arange(-5,6+bin_width_terpi_even,bin_width_terpi_even)
n1terpi_even_counts, n1terpi_even_bin = np.histogram(n1terpi_even,n1terpi_even_bin)
n1terpi_even_stab = 2*np.sqrt(n1terpi_even_counts+0.25)

n1terpi_odd = np.sort(np.concatenate(e060817terpi[0][1:20:2]))
n1terpi_odd = n1terpi_odd[np.logical_and(left <= n1terpi_odd,n1terpi_odd <= right)]-terpi_onset
bin_width_terpi_odd = np.ceil(target_mean/n_stim_terpi/e060817_spont_nu[0]*1000)/1000/2
n1terpi_odd_bin = np.arange(-5,6+bin_width_terpi_odd,bin_width_terpi_odd)
n1terpi_odd_counts, n1terpi_odd_bin = np.histogram(n1terpi_odd,n1terpi_odd_bin)
n1terpi_odd_stab = 2*np.sqrt(n1terpi_odd_counts+0.25)
n1terpi_odd_x = n1terpi_odd_bin[:-1]+bin_width_terpi_odd/2


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
Making the raster plots
We make our raster plots with a short function raster_plot that we define with:
Can you write Python code for it?

def raster_plot(train_list,
                stim_onset=None,
                color = 'black'):
    """Create a raster plot.

    Parameters
    ----------
    train_list: a list of spike trains (1d vector with strictly
      increasing elements).
    stim_onst: a number giving the time of stimulus onset. If
      specificied, the time are realigned such that the stimulus
      comes at 0.
    color: the color of the ticks representing the spikes.

    Side effect:
    A raster plot is created.
    """
    import numpy as np
    import matplotlib.pyplot as plt
    if stim_onset is None:
        stim_onset = 0
    for idx,trial in enumerate(train_list):
        plt.vlines(trial-stim_onset,
                   idx+0.6,idx+1.4,
                   color=color)
    plt.ylim(0.5,len(train_list))


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
The file is opened with:
Can you write Python code for it?


I am working on the file "AreTwoPSTHsIdentical.ipynb".
The first task is:
We just have to make the graph after computing the cumsum of the differences:
Can you write Python code for it?

n1_diff_y = (n1terpi_stab-n1citron_stab)/np.sqrt(2)
X1 = np.arange(1,len(n1terpi_x)+1)/len(n1terpi_x)
Y1 = np.cumsum(n1_diff_y)/np.sqrt(len(n1_diff_y))
n1_diff_terpi_y = (n1terpi_even_stab-n1terpi_odd_stab)/np.sqrt(2)
X2 = np.arange(1,len(n1terpi_odd_x)+1)/len(n1terpi_odd_x)
Y2 = np.cumsum(n1_diff_terpi_y)/np.sqrt(len(n1_diff_terpi_y))
xx = np.linspace(0,1,201)
fig = plt.figure(figsize=(14,8))
plt.plot(xx,c95(xx),color='grey',lw=2,linestyle='dashed')
plt.plot(xx,-c95(xx),color='grey',lw=2,linestyle='dashed')
plt.plot(xx,c99(xx),color='grey',lw=2)
plt.plot(xx,-c99(xx),color='grey',lw=2)
plt.plot(X2,Y2,color='grey',lw=2)
plt.plot(X1,Y1,color='black',lw=2)
plt.xlabel("Normalized time",fontdict={'fontsize':20})
plt.ylabel("$S_k(t)$",fontdict={'fontsize':20})


I am working on the file "Array/Pascal's Triangle.ipynb".
The first task is:
Corner Cases
There are a number of corner cases which should be considered, for example passing in a negative integer, or passing a string for example, but the typical corner case with pascals triangle is an integer overflow when dealing with large numbers. In python however integers have arbitrary size, and expand as required, for example:
Can you write Python code for it?


I am working on the file "Array/Pascal's Triangle.ipynb".
The first task is:
Optimised Solution
However it is possible to do this in O(n) space and O(n/2) time complexity:
Can you write Python code for it?

def pascals_triangle_2(k):

    k = k - 1

    row = [1]

    # only need to calculate half of the row, since the triangle is
    # symmetric
    for n in xrange(k / 2):
        row.append(row[n] * (k - n) / (n + 1))

    # middle element is repeated only for odd values of k
    r = list(reversed(row))
    r = r[1:] if k % 2 == 0 else r

    return row + r

for k in xrange(1, 10):
    print pascals_triangle_2(k)

%%timeit
pascals_triangle_2(40)


I am working on the file "Array/Pascal's Triangle.ipynb".
The first task is:
With very large numbers in a python solution you will eventually run out of memory, for example:
Can you write Python code for it?

import sys
result2 = pascals_triangle_2(sys.maxint - 1)


I am working on the file "Array/Rotate Array.ipynb".
The first task is:
However, although the space complexity is now O(1), the time complexity is O(n * k). It would be good to find a solution that has O(1) space complexity and O(n) time complexity.
Reverse Rotate
Another way of rotating the array is to split the array into two sub arrays at the point of rotation. Each subarray is reversed, before rotating the entire array. This solution achieves O(1) space complexity and O(n).
Can you write Python code for it?

def rotate_reverse_inplace(input_array, order):
    length = len(input_array)
    order = -order % length
    split_location = length - order
    input_array[:split_location] = reversed(input_array[:split_location])
    input_array[split_location:] = reversed(input_array[split_location:])
    input_array.reverse()
    return input_array

example_array2 = list(example_array)
rotate_reverse_inplace(example_array2, k)

%%timeit
rotate_reverse_inplace(example_array, k)


I am working on the file "Array/Rotate Array.ipynb".
The first task is:
Simple Solution
The simplest solution splits the array at the point to rotate, and constructs a new array using the two parts.
Can you write Python code for it?

def rotate_simple(input_array, order):
    order %= len(input_array)
    return input_array[order:] + input_array[:order]

rotate_simple(example_array, k)

%%timeit
rotate_simple(example_array, k)


I am working on the file "Array/Rotate Array.ipynb".
The first task is:
This solution has both time and space complexity of O(n).
Bubble Rotate
However if we want to rotate a large array in place (without creating a new array), the solution above is inefficient. The time complexity is O(n), dependant only on the size of the input array, but the space complexity is also O(n) since we need to create a new array with the rotated elements. By applying a similar algorithm to bubble sort we can perform the rotation in place.
Can you write Python code for it?

def rotate_bubble_inplace(input_array, order):
    order %= len(input_array)
    for i in range(order):
        for j in range(len(input_array)):
            input_array[j], input_array[j - 1] = input_array[j - 1], input_array[j]
    return input_array

example_array2 = list(example_array)
rotate_bubble_inplace(example_array2, k)

%%timeit
rotate_bubble_inplace(example_array, k)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Hence confirmed the missing values are imputed. Note: This is the most primitive forms of imputation. Other sophisticated techniques include modeling the missing values using some variables or using grouped averages (mean/mode/median). We'll take up the later next.
More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html#pandas.DataFrame.fillna
5. Pivot Table
Pandas can be used to create MS Excel style pivot tables. For isntance, in this case a very key column is "LoanAmount" which has missing values. We can impute it using mean amount of each 'Gender', 'Married' and 'Self_Employed' group. The mean 'LoanAmount' of each group can be determined as:
Can you write Python code for it?

#Determine pivot table
impute_grps = data.pivot_table(values=["LoanAmount"], index=["Gender","Married","Self_Employed"], aggfunc=np.mean)
print impute_grps


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Here we see that Credit_History is a nominal variable but appearing as float. A good way to tackle this issue is to create a csv file with column names and types. This way we can make a generic function to read the file and assign column data types. For instance, in this case I've defined a csv file datatypes.csv (download).
Can you write Python code for it?

#Load the file:
colTypes = pd.read_csv('datatypes.csv')
print colTypes


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pivot_table.html#pandas.DataFrame.pivot_table
6. MultiIndexing
If you notice the output of step #4, it has a peculiar property. Each index is made of a combination of 3 values! This is called MultuIndexing and it can be used to perform some operations really fast if used wisely. Continuing the example from #4, we have the values for each group but not they have to be imputed.
This can be done using the various techniques learned till now.
Can you write Python code for it?

#iterate only through rows with missing LoanAmount
for i,row in data.loc[data['LoanAmount'].isnull(),:].iterrows():
    ind = tuple([row['Gender'],row['Married'],row['Self_Employed']])
    data.loc[i,'LoanAmount'] = impute_grps.loc[ind].values[0]

#Now check the #missing values again to confirm:
print data.apply(num_missing, axis=0)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
More: http://pandas.pydata.org/pandas-docs/stable/indexing.html
2. Iterating over rows of a dataframe
Although rarely used, but at times you may need to iterate through all rows. One common problem we face is nominal variables with numeric categories are treated as numerical by defauly by python. Also, there might be some numeric variable where some characters are entered in one of the rows and it'll be considered categorical by default. So it's generally a good idea to manually define the column types.
Can you write Python code for it?


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
More: http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.cut.html
12. Coding nominal data
Very often we come accross a case where we've to modify the categories of a nominal variable. This can be due to various reasons:
    1. Some algorithms (like Logistic Regression) require all inputs to be numeric. So nominal varialbes are mostly coded as 0---(n-1)
    2. Sometimes the same category might be represented in 2 ways. For eg temperature might be recorded as "High", "Medium", "Low", "H", "low". Here, both "High" and "H" refer to same category. Similarly, in "Low" and "low" there is only a difference of case.
    3. Some categories might have very low frequencies and its generally a good idea to combine them
Here I've defined a generic function which takes in input as a dictionary and codes the values using 'replace' function in Pandas
Can you write Python code for it?

#Define a generic function using Pandas replace function
def coding(col, codeDict):
    colCoded = pd.Series(col, copy=True)
    for key, value in codeDict.items():
        colCoded.replace(key, value, inplace=True)
    return colCoded

#Coding LoanStatus as Y=1, N=0:
print 'Before Coding:'
print pd.value_counts(data["Loan_Status"])
data["Loan_Status_Coded"] = coding(data["Loan_Status"], {'N':0,'Y':1})
print '\nAfter Coding:'
print pd.value_counts(data["Loan_Status_Coded"])


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Now it is clearly evident that people with a credit histpry have much higher chances of getting a loan as 80% people with credit history got a loan as compared to only 9% without credit histoty.
But that's not it. It tells an iteresting story. Since I know that having a credit history is super important, what if I predict loan status to be Y for ones with credit history and N otherwise. Surprisingly, we'll be right 82+378=460 times out of 614 which is a whopping 75%! I won't blame you if you're wondering why the hell do we need statistical models. But trust me, increasing the accuracy by even 0.001% beyond this is a challenging ask.
Note: 75% is on train set. The test set will be slightly different but close.
Also, I hope this gives some intuition into why even a 0.05% increase in accuracy can result in jump of 500 ranks on a Kaggle leaderboard.
More: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.crosstab.html
8. Merge DataFrames
Merging dataframes becomes essential when we have information coming from different sources which is to be collated. Consider a hypothetical case where the average property rates (INR per sq meters) is available for different property types. Let's define a dataframe as:
Can you write Python code for it?

prop_rates = pd.DataFrame([1000, 5000, 12000], index=['Rural','Semiurban','Urban'],columns=['rates'])
prop_rates


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Now the credit history column is modified to 'object' type which is used for representing nominal variables in Pandas.
More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iterrows.html#pandas.DataFrame.iterrows
3. Apply Function
It is one of the most handy functions when it comes to playing with the data, specially creating new varaibles. As the name suggests, apply returns some value after passing each row/column of a dataframe through some function. The function can be both default or user-defined.
For instance, here it can be used to find the #missing values in each row and column.
Can you write Python code for it?

#Create a new function:
def num_missing(x):
    return sum(x.isnull())

#Applying per column:
print "Missing values per column:"
print data.apply(num_missing, axis=0)  #axis=0 defines that function is to be applied on each column

#Applying per row:
print "\nMissing values per row:"
print data.apply(num_missing, axis=1).head()  #axis=1 defines that function is to be applied on each column


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Now we can fill the missing values and check using technique #3.
Can you write Python code for it?

#Impute the values:
data['Gender'].fillna(mode(data['Gender']).mode[0], inplace=True)
data['Married'].fillna(mode(data['Married']).mode[0], inplace=True)
data['Self_Employed'].fillna(mode(data['Self_Employed']).mode[0], inplace=True)

#Now check the #missing values again to confirm:
print data.apply(num_missing, axis=0)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
Now we can merge this information with the original dataframe as:
Can you write Python code for it?

data_merged = data.merge(right=prop_rates, how='inner',left_on='Property_Area',right_index=True, sort=False)
data_merged.pivot_table(values='Credit_History',index=['Property_Area','rates'], aggfunc=len)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
The pivot table validates sucessful merge operation. Note that the 'values' argument is irrelevant here because we are simply counting the values.
More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html#pandas.DataFrame.merge
9. Sorting DataFrames
Pandas allows easy sorting based on multiple columns. This can be done as:
Can you write Python code for it?

data_sorted = data.sort_values(['ApplicantIncome','CoapplicantIncome'], ascending=False)
data_sorted[['ApplicantIncome','CoapplicantIncome']].head(10)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
These are absolute numbers but percentages can be more intuitive in making some quick insights. We can do this using the apply function:
Can you write Python code for it?

def percConvert(ser):
    return ser/float(ser[-1])
pd.crosstab(data["Credit_History"],data["Loan_Status"],margins=True).apply(percConvert, axis=1)


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
This returns both mode and count. Remember that mode can be an array as there can be multiple values with high frequency. We will take the first one by default always using:
Can you write Python code for it?


I am working on the file "Articles/12_Useful_Pandas_Techniques_in_Python_for_Data_Manipulation/PythonTipsNTricks.ipynb".
The first task is:
This shows that income is not a big deciding factor on its own as there is no appreciable difference between the people who received and were denied the loan.
More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html#pandas.DataFrame.hist |
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.boxplot.html#pandas.DataFrame.boxplot
11. Cut function for binning
Sometimes numerical values make more sense if clustered together. For example, if we're trying to model traffic (#cars on road) with time of the day (minutes). The exact minute of an hour might not be that relevant for predicting traffic as compared to actual period of the day like "Morning", "Afternoon", "Evening", "Night", "Late Night". Modeling traffic this way will be more intuitive and will avoid overfitting minute details which practically make little sense.
Here I've defined a simple function which can be re-used for binning any variable fairly easily.
Can you write Python code for it?

#Binning:
def binning(col, cut_points, labels=None):
    #Define min and max values:
    minval = col.min()
    maxval = col.max()

    #create list by adding min and max to cut_points
    break_points = [minval] + cut_points + [maxval]

    #if no labels provided, use default labels 0 ... (n-1)
    if not labels:
        labels = range(len(cut_points)+1)

    #Binning using cut function of pandas
    colBin = pd.cut(col,bins=break_points,labels=labels,include_lowest=True)
    return colBin

#Binning age:
cut_points = [90,140,190]
labels = ["low","medium","high","very high"]
data["LoanAmount_Bin"] = binning(data["LoanAmount"], cut_points, labels)
print pd.value_counts(data["LoanAmount_Bin"], sort=False)


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
2. Data Cleaning
Imputation
Can you write Python code for it?

#Determine the average weight per item:
item_avg_weight = data.pivot_table(values='Item_Weight', index='Item_Identifier')

#Get a boolean variable specifying missing Item_Weight values
miss_bool = data['Item_Weight'].isnull()

#Impute data and check #missing values before and after imputation to confirm
print 'Orignal #missing: %d'% sum(miss_bool)
data.loc[miss_bool,'Item_Weight'] = data.loc[miss_bool,'Item_Identifier'].apply(lambda x: item_avg_weight[x])
print 'Final #missing: %d'% sum(data['Item_Weight'].isnull())

#Import mode function:
from scipy.stats import mode

#Determing the mode for each
outlet_size_mode = data.pivot_table(values='Outlet_Size', columns='Outlet_Type',aggfunc=(lambda x:mode(x).mode[0]) )
print 'Mode for each Outlet_Type:'
print outlet_size_mode

#Get a boolean variable specifying missing Item_Weight values
miss_bool = data['Outlet_Size'].isnull()

#Impute data and check #missing values before and after imputation to confirm
print '\nOrignal #missing: %d'% sum(miss_bool)
data.loc[miss_bool,'Outlet_Size'] = data.loc[miss_bool,'Outlet_Type'].apply(lambda x: outlet_size_mode[x])
print sum(data['Outlet_Size'].isnull())


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
2. Feature Engineering:
Step1: Consider combining categories in Outlet_Type
Can you write Python code for it?

#Check the mean sales by type:
data.pivot_table(values='Item_Outlet_Sales',index='Outlet_Type')


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
Loading data:
The files can be downloaded from: http://datahack.analyticsvidhya.com/contest/practice-problem-bigmart-sales-prediction
Can you write Python code for it?

#Read files:
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

#Combine test and train into one file
train['source']='train'
test['source']='test'
data = pd.concat([train, test],ignore_index=True)
print train.shape, test.shape, data.shape

#Check missing values:
data.apply(lambda x: sum(x.isnull()))

#Numerical data summary:
data.describe()

#Number of unique values in each:
data.apply(lambda x: len(x.unique()))

#Filter categorical variables
categorical_columns = [x for x in data.dtypes.index if data.dtypes[x]=='object']
#Exclude ID cols and source:
categorical_columns = [x for x in categorical_columns if x not in ['Item_Identifier','Outlet_Identifier','source']]
#Print frequency of categories
for col in categorical_columns:
    print '\nFrequency of Categories for varible %s'%col
    print data[col].value_counts()


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
Step 3: Create a broad category of Type of Item
Can you write Python code for it?

#Item type combine:
data['Item_Identifier'].value_counts()
data['Item_Type_Combined'] = data['Item_Identifier'].apply(lambda x: x[0:2])
data['Item_Type_Combined'] = data['Item_Type_Combined'].map({'FD':'Food',
                                                             'NC':'Non-Consumable',
                                                             'DR':'Drinks'})
data['Item_Type_Combined'].value_counts()


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
Step 4: Determine the years of operation of a store
Can you write Python code for it?

#Years:
data['Outlet_Years'] = 2013 - data['Outlet_Establishment_Year']
data['Outlet_Years'].describe()


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
Step 5: Modify categories of Item_Fat_Content
Can you write Python code for it?

#Change categories of low fat:
print 'Original Categories:'
print data['Item_Fat_Content'].value_counts()

print '\nModified Categories:'
data['Item_Fat_Content'] = data['Item_Fat_Content'].replace({'LF':'Low Fat',
                                                             'reg':'Regular',
                                                             'low fat':'Low Fat'})
print data['Item_Fat_Content'].value_counts()

#Mark non-consumables as separate category in low_fat:
data.loc[data['Item_Type_Combined']=="Non-Consumable",'Item_Fat_Content'] = "Non-Edible"
data['Item_Fat_Content'].value_counts()


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/exploration_and_feature_engineering.ipynb".
The first task is:
Step 6: Numerical and One-Hot Coding of Categorical variables
Can you write Python code for it?

#Import library:
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
#New variable for outlet
data['Outlet'] = le.fit_transform(data['Outlet_Identifier'])
var_mod = ['Item_Fat_Content','Outlet_Location_Type','Outlet_Size','Item_Type_Combined','Outlet_Type','Outlet']
le = LabelEncoder()
for i in var_mod:
    data[i] = le.fit_transform(data[i])

#One Hot Coding:
data = pd.get_dummies(data, columns=['Item_Fat_Content','Outlet_Location_Type','Outlet_Size','Outlet_Type',
                              'Item_Type_Combined','Outlet'])

data.dtypes

data[['Item_Fat_Content_0','Item_Fat_Content_1','Item_Fat_Content_2']].head(10)


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Baseline models:
Can you write Python code for it?

#Mean based:
mean_sales = train['Item_Outlet_Sales'].mean()

#Define a dataframe with IDs for submission:
base1 = test[['Item_Identifier','Outlet_Identifier']]
base1['Item_Outlet_Sales'] = mean_sales

#Export submission file
base1.to_csv("alg0.csv",index=False)


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Decision Tree Model:
Can you write Python code for it?

from sklearn.tree import DecisionTreeRegressor

predictors = [x for x in train.columns if x not in [target]+IDcol]
alg3 = DecisionTreeRegressor(max_depth=15, min_samples_leaf=100)
modelfit(alg3, train, test, predictors, target, IDcol, 'alg3.csv')
coef3 = pd.Series(alg3.feature_importances_, predictors).sort_values(ascending=False)
coef3.plot(kind='bar', title='Feature Importances')

predictors = ['Item_MRP','Outlet_Type_0','Outlet_5','Outlet_Years']
alg4 = DecisionTreeRegressor(max_depth=8, min_samples_leaf=150)
modelfit(alg4, train, test, predictors, target, IDcol, 'alg4.csv')
coef4 = pd.Series(alg4.feature_importances_, predictors).sort_values(ascending=False)
coef4.plot(kind='bar', title='Feature Importances')


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Function to fit and generate submission file:
Can you write Python code for it?

#Define target and ID columns:
target = 'Item_Outlet_Sales'
IDcol = ['Item_Identifier','Outlet_Identifier']
from sklearn import cross_validation, metrics
def modelfit(alg, dtrain, dtest, predictors, target, IDcol, filename):
    #Fit the algorithm on the data
    alg.fit(dtrain[predictors], dtrain[target])

    #Predict training set:
    dtrain_predictions = alg.predict(dtrain[predictors])

    #Perform cross-validation:
    cv_score = cross_validation.cross_val_score(alg, dtrain[predictors], dtrain[target], cv=20, scoring='mean_squared_error')
    cv_score = np.sqrt(np.abs(cv_score))

    #Print model report:
    print "\nModel Report"
    print "RMSE : %.4g" % np.sqrt(metrics.mean_squared_error(dtrain[target].values, dtrain_predictions))
    print "CV Score : Mean - %.4g | Std - %.4g | Min - %.4g | Max - %.4g" % (np.mean(cv_score),np.std(cv_score),np.min(cv_score),np.max(cv_score))

    #Predict on testing data:
    dtest[target] = alg.predict(dtest[predictors])

    #Export submission file:
    IDcol.append(target)
    submission = pd.DataFrame({ x: dtest[x] for x in IDcol})
    submission.to_csv(filename, index=False)


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Linear Regression Model:
Can you write Python code for it?

from sklearn.linear_model import LinearRegression, Ridge

predictors = [x for x in train.columns if x not in [target]+IDcol]
# print predictors
alg1 = LinearRegression(normalize=True)
modelfit(alg1, train, test, predictors, target, IDcol, 'alg1.csv')
coef1 = pd.Series(alg1.coef_, predictors).sort_values()
coef1.plot(kind='bar', title='Model Coefficients')


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Random Forest Model:
Note: random forest models are not 100% replicable. So the outputs might differ very slightly but should be around the ballpark.
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor

predictors = [x for x in train.columns if x not in [target]+IDcol]
alg5 = RandomForestRegressor(n_estimators=200,max_depth=5, min_samples_leaf=100,n_jobs=4)
modelfit(alg5, train, test, predictors, target, IDcol, 'alg5.csv')
coef5 = pd.Series(alg5.feature_importances_, predictors).sort_values(ascending=False)
coef5.plot(kind='bar', title='Feature Importances')

predictors = [x for x in train.columns if x not in [target]+IDcol]
alg6 = RandomForestRegressor(n_estimators=400,max_depth=6, min_samples_leaf=100,n_jobs=4)
modelfit(alg6, train, test, predictors, target, IDcol, 'alg6.csv')
coef6 = pd.Series(alg6.feature_importances_, predictors).sort_values(ascending=False)
coef6.plot(kind='bar', title='Feature Importances')


I am working on the file "Articles/Getting_Started_with_BigMart_Sales(AV_Datahacks)/model_building.ipynb".
The first task is:
Ridge Regression Model:
Can you write Python code for it?

predictors = [x for x in train.columns if x not in [target]+IDcol]
alg2 = Ridge(alpha=0.05,normalize=True)
modelfit(alg2, train, test, predictors, target, IDcol, 'alg2.csv')
coef2 = pd.Series(alg2.coef_, predictors).sort_values()
coef2.plot(kind='bar', title='Model Coefficients')


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
1/10th learning rate
Can you write Python code for it?

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_2 = GradientBoostingClassifier(learning_rate=0.01, n_estimators=600,max_depth=9, min_samples_split=1200,
                                         min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7)
modelfit(gbm_tuned_2, train, test, predictors)


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
Baseline Model
Since here the criteria is AUC, simply predicting the most prominent class would give an AUC of 0.5 always. Another way of getting a baseline model is to use the algorithm without tuning, i.e. with default parameters.
Can you write Python code for it?

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm0 = GradientBoostingClassifier(random_state=10)
modelfit(gbm0, train, test, predictors,printOOB=False)


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
GBM Models:
There 2 types of parameters here:
1. Tree-specific parameters
  * min_samples_split
  * min_samples_leaf
  * max_depth
  * min_leaf_nodes
  * max_features
  * loss function
2. Boosting specific paramters
  * n_estimators
  * learning_rate
  * subsample
Approach for tackling the problem

Decide a relatively higher value for learning rate and tune the number of estimators requried for that.
Tune the tree specific parameters for that learning rate
Tune subsample
Lower learning rate as much as possible computationally and increase the number of estimators accordingly.

Step 1- Find the number of estimators for a high learning rate
We will use the following benchmarks for parameters:
1. min_samples_split = 500 : ~0.5-1% of total values. Since this is imbalanced class problem, we'll take small value
2. min_samples_leaf = 50 : Just using for preventing overfitting. will be tuned later.
3. max_depth = 8 : since high number of observations and predictors, choose relatively high value
4. max_features = 'sqrt' : general thumbrule to start with
5. subsample = 0.8 : typically used value (will be tuned later)
0.1 is assumed to be a good learning rate to start with. Let's try to find the optimum number of estimators requried for this.
Can you write Python code for it?

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
param_test1 = {'n_estimators':range(20,81,10)}
gsearch1 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, min_samples_split=500,
                                  min_samples_leaf=50,max_depth=8,max_features='sqrt', subsample=0.8,random_state=10),
                       param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(train[predictors],train[target])

gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
Load Data:
The data has gone through following pre-processing:
1. City variable dropped because of too many categories
2. DOB converted to Age | DOB dropped
3. EMI_Loan_Submitted_Missing created which is 1 if EMI_Loan_Submitted was missing else 0 | EMI_Loan_Submitted dropped
4. EmployerName dropped because of too many categories
5. Existing_EMI imputed with 0 (median) - 111 values were missing
6. Interest_Rate_Missing created which is 1 if Interest_Rate was missing else 0 | Interest_Rate dropped
7. Lead_Creation_Date dropped because made little intuitive impact on outcome
8. Loan_Amount_Applied, Loan_Tenure_Applied imputed with missing
9. Loan_Amount_Submitted_Missing created which is 1 if Loan_Amount_Submitted was missing else 0 | Loan_Amount_Submitted dropped
10. Loan_Tenure_Submitted_Missing created which is 1 if Loan_Tenure_Submitted was missing else 0 | Loan_Tenure_Submitted dropped
11. LoggedIn, Salary_Account removed
12. Processing_Fee_Missing created which is 1 if Processing_Fee was missing else 0 | Processing_Fee dropped
13. Source - top 2 kept as is and all others combined into different category
14. Numerical and One-Hot-Coding performed
Can you write Python code for it?

train = pd.read_csv('train_modified.csv')

target='Disbursed'
IDcol = 'ID'

train['Disbursed'].value_counts()


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
Since we reached the maximum of min_sales_split, we should check higher values as well. Also, we can tune min_samples_leaf with it now as max_depth is fixed. One might argue that max depth might change for higher value but if you observe the output closely, a max_depth of 9 had a better model for most of cases.
So lets perform a grid search on them:
Can you write Python code for it?

#Grid seach on subsample and max_features
param_test3 = {'min_samples_split':range(1000,2100,200), 'min_samples_leaf':range(30,71,10)}
gsearch3 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60,max_depth=9,
                                                    max_features='sqrt', subsample=0.8, random_state=10),
                       param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch3.fit(train[predictors],train[target])

gsearch3.grid_scores_, gsearch3.best_params_, gsearch3.best_score_

modelfit(gsearch3.best_estimator_, train, test, predictors)


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
So we got 60 as the optimal estimators for the 0.1 learning rate. Note that 60 is a reasonable value and can be used as it is. But it might not be the same in all cases. Other situations:
1. If the value is around 20, you might want to try lowering the learning rate to 0.05 and re-run grid search
2. If the values are too high ~100, tuning the other parameters will take long time and you can try a higher learning rate
Step 2- Tune tree-specific parameters
Now, lets move onto tuning the tree parameters. We will do this in 2 stages:
1. Tune max_depth and num_samples_split
2. Tune min_samples_leaf
3. Tune max_features
Can you write Python code for it?

#Grid seach on subsample and max_features
param_test2 = {'max_depth':range(5,16,2), 'min_samples_split':range(200,1001,200)}
gsearch2 = GridSearchCV(estimator = GradientBoostingClassifier(learning_rate=0.1, n_estimators=60,
                                                max_features='sqrt', subsample=0.8, random_state=10),
                       param_grid = param_test2, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch2.fit(train[predictors],train[target])

gsearch2.grid_scores_, gsearch2.best_params_, gsearch2.best_score_


I am working on the file "Articles/Parameter_Tuning_GBM_with_Example/GBM model.ipynb".
The first task is:
With all tuned lets try reducing the learning rate and proportionally increasing the number of estimators to get more robust results:
Can you write Python code for it?

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
gbm_tuned_1 = GradientBoostingClassifier(learning_rate=0.05, n_estimators=120,max_depth=9, min_samples_split=1200,
                                         min_samples_leaf=60, subsample=0.85, random_state=10, max_features=7)
modelfit(gbm_tuned_1, train, test, predictors)


I am working on the file "Articles/Parameter_Tuning_XGBoost_with_Example/XGBoost models.ipynb".
The first task is:
Load Data:
The data has gone through following pre-processing:
1. City variable dropped because of too many categories
2. DOB converted to Age | DOB dropped
3. EMI_Loan_Submitted_Missing created which is 1 if EMI_Loan_Submitted was missing else 0 | EMI_Loan_Submitted dropped
4. EmployerName dropped because of too many categories
5. Existing_EMI imputed with 0 (median) - 111 values were missing
6. Interest_Rate_Missing created which is 1 if Interest_Rate was missing else 0 | Interest_Rate dropped
7. Lead_Creation_Date dropped because made little intuitive impact on outcome
8. Loan_Amount_Applied, Loan_Tenure_Applied imputed with missing
9. Loan_Amount_Submitted_Missing created which is 1 if Loan_Amount_Submitted was missing else 0 | Loan_Amount_Submitted dropped
10. Loan_Tenure_Submitted_Missing created which is 1 if Loan_Tenure_Submitted was missing else 0 | Loan_Tenure_Submitted dropped
11. LoggedIn, Salary_Account removed
12. Processing_Fee_Missing created which is 1 if Processing_Fee was missing else 0 | Processing_Fee dropped
13. Source - top 2 kept as is and all others combined into different category
14. Numerical and One-Hot-Coding performed
Can you write Python code for it?

train = pd.read_csv('train_modified.csv')
test = pd.read_csv('test_modified.csv')

train.shape, test.shape

target='Disbursed'
IDcol = 'ID'

train['Disbursed'].value_counts()


I am working on the file "Articles/Parameter_Tuning_XGBoost_with_Example/XGBoost models.ipynb".
The first task is:
Step 1- Find the number of estimators for a high learning rate
Can you write Python code for it?

predictors = [x for x in train.columns if x not in [target, IDcol]]
xgb1 = XGBClassifier(
        learning_rate =0.1,
        n_estimators=1000,
        max_depth=5,
        min_child_weight=1,
        gamma=0,
        subsample=0.8,
        colsample_bytree=0.8,
        objective= 'binary:logistic',
        nthread=4,
        scale_pos_weight=1,
        seed=27)
modelfit(xgb1, train, test, predictors)

#Grid seach on subsample and max_features
#Choose all predictors except target & IDcols
param_test1 = {
    'max_depth':range(3,10,2),
    'min_child_weight':range(1,6,2)
}
gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=140, max_depth=5,
                                        min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27),
                       param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(train[predictors],train[target])

gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_

#Grid seach on subsample and max_features
#Choose all predictors except target & IDcols
param_test2 = {
    'max_depth':[4,5,6],
    'min_child_weight':[4,5,6]
}
gsearch2 = GridSearchCV(estimator = XGBClassifier( learning_rate=0.1, n_estimators=140, max_depth=5,
                                        min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.8,
                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
                       param_grid = param_test2, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch2.fit(train[predictors],train[target])

gsearch2.grid_scores_, gsearch2.best_params_, gsearch2.best_score_

#Grid seach on subsample and max_features
#Choose all predictors except target & IDcols
param_test2b = {
    'min_child_weight':[6,8,10,12]
}
gsearch2b = GridSearchCV(estimator = XGBClassifier( learning_rate=0.1, n_estimators=140, max_depth=4,
    ...


I am working on the file "Articles/Parameter_Tuning_XGBoost_with_Example/XGBoost models.ipynb".
The first task is:
Tune subsample and colsample_bytree
Can you write Python code for it?

#Grid seach on subsample and max_features
#Choose all predictors except target & IDcols
param_test4 = {
    'subsample':[i/10.0 for i in range(6,10)],
    'colsample_bytree':[i/10.0 for i in range(6,10)]
}
gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=177, max_depth=4,
                                        min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,
                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
                       param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch4.fit(train[predictors],train[target])

gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_


I am working on the file "Articles/Parameter_Tuning_XGBoost_with_Example/XGBoost models.ipynb".
The first task is:
tune subsample:
Can you write Python code for it?

#Grid seach on subsample and max_features
#Choose all predictors except target & IDcols
param_test5 = {
    'subsample':[i/100.0 for i in range(75,90,5)],
    'colsample_bytree':[i/100.0 for i in range(75,90,5)]
}
gsearch5 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=177, max_depth=4,
                                        min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,
                                        objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
                       param_grid = param_test5, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch5.fit(train[predictors],train[target])

gsearch5.grid_scores_, gsearch5.best_params_, gsearch5.best_score_


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Additional examples (mentioned as exercise for users)
Eg: Prime numbers:
Can you write Python code for it?

#FOR:
def eg4_for(N):
    non_primes = []
    for i in range(2,int(N**0.5)+1):
        for j in range(i,N,i):
#             print j
            non_primes.append(j)
    primes = []
    for i in range(2,N):
        if i not in non_primes:
            primes.append(i)
    return primes
print eg4_for(100)
%timeit eg4_for(100)

#LC:
def eg4_lc(N):
    non_primes = [ j for i in range(2,int(N**0.5)+1) for j in range(i,N,i)]
    return [ i for i in range(2,N) if i not in non_primes]
print eg4_lc(100)
%timeit eg4_lc(100)


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Data Science Examples:
Example 4: Reading list of list:
Can you write Python code for it?

import pandas as pd
data = pd.read_csv("skills.csv")
print data

#Split text with the separator ';'
data['skills_list'] = data['skills'].apply(lambda x: x.split(';'))
print data['skills_list']

#Initialize the set
skills_unq = set()
#Update each entry into set. Since it takes only unique value, duplicates will be ignored automatically.
skills_unq.update( (sport for l in data['skills_list'] for sport in l) )
print skills_unq

#Convert set to list:
skills_unq = list(skills_unq)
sport_matrix = [ [1 if skill in row else 0 for skill in skills_unq] for row in data['skills_list']  ]
sport_matrix

data = pd.concat([data, pd.DataFrame(sport_matrix,columns=skills_unq)],axis=1)
print data


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Eg1: Matrix Flatten:
Can you write Python code for it?

# matrix = [ range(0,5), range(5,10), range(10,15) ]
# print matrix

def eg1_for(matrix):
    flat = []
    for row in matrix:
        for x in row:
            flat.append(x)
    return flat

def eg1_lc(matrix):
    return [x for row in matrix for x in row ]

matrix = [ range(0,5), range(5,10), range(10,15) ]
print "Original Matrix: " + str(matrix)
print "FOR-loop result: " + str(eg1_for(matrix))
print "LC result      : " + str(eg1_lc(matrix))

%timeit eg1_for(matrix)

%timeit eg1_lc(matrix)


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Eg6: Filtering columns:
Can you write Python code for it?

cols = ['a','b','c','d','a_transform','b_transform','c_transform','d_power2','d_power3','d_power4','d_power5','temp1','temp2']
#Here a,b,c,d are original variables; transform are transformation, power are for polynomial reg, temp are intermediate

#Select only variables with 'transform':
col_set1 = [x for x in cols if x.endswith('transform')]
col_set2 = [x for x in cols if 'power' in x]
col_set3 = [x for x in cols if (x.endswith('transform')) | ('power' in x)]
col_set4 = [x for x in cols if x not in ['temp1','temp2']]

print 'Set1: ', col_set1
print 'Set2: ', col_set2
print 'Set3: ', col_set3
print 'Set4: ', col_set4


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Eg: Find all possible triangles:
We are given an integer N and we have to find all possible triangles with unique lengths that can be formed using side lengths <=N. Let's compare the for and LC cases:
Can you write Python code for it?

def tri_for(N):
    L=[]
    for i in range(1,N-2):
        for j in range(i+1,N-1):
            for k in range(j+1, N):
                if (i+j<k) | (i+k<j) | (j+k<i):
                    L.append((i,j,k))
    return L

def tri_lc(N):
    return [(i,j,k) for i in range(1,N-2) for j in range(i+1,N-1) for k in range(j+1,N) if ((i+j<k) | (i+k<j) | (j+k<i))]
#     [ (i,j,k) for i in range(1,N-2) for j in range(i+1,N-1) for k in range(j+1,N) ]

print tri_for(10)
%timeit tri_for(10)

print tri_lc(10)
%timeit tri_lc(10)


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Here we can see that in this case LC is ~30% faster than for-loop and ~45% faster than map function.
Let's check for the second example:
Can you write Python code for it?

%timeit square_even_for(range(1,11))

%timeit square_even_map(range(1,11))

%timeit square_even_lc(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Here we have used the Python temporary function lambda. This can be replaced with a standard Python function or a user-defined function declared earlier.
A simple example
Can you write Python code for it?

#Method 1: For-Loop
def square_for(arr):
    result = []
    for i in arr:
        result.append(i**2)
    return result
print square_for(range(1,11))

#Method 2: Map Function
def square_map(arr):
    return map(lambda x: x**2, arr)
print square_map(range(1,11))

#Method 3: List comprehension:
def square_lc(arr):
    return [i**2 for i in arr]
print square_lc(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Here we see a similar trend as before. So till the point of iterating and making slight modifications, for-loop is clear winner.
LC is close to for-loop but again map takes around twice as much time. Note that here the difference between time will also depend on the complexity of the function being applied to each element.
Another intuition for higher time of map and LC can be that in both cases, it is compulsory to store information and we are actually performing all 3 steps for LC and map. So let's check runtime of for-loop with step 3:
Can you write Python code for it?

def store_for(arr):
    result=[]
    for i in arr:
        result.append(i*2)
    return result
%timeit store_for(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Here we see that for-loop is fasters. This is because in a for-loop, we need not return an element and just move onto next iteration using "pass".
In both LC and map, returning an element is necessary. The codes here return None. But still map takes more than twice the time. Intuitively, we can think that map involves a definite function call at each iteration which can be the reason behind the extra time.
Now, lets perform a simple operation of multiplying the number by 2 but we need not store the result:
Can you write Python code for it?

#Method 1: For-loop:
def x2_for(arr):
    for i in arr:
        i*2
%timeit x2_for(range(1,11))

#Method 2: Map
def x2_map(arr):
    map(lambda x: x*2,arr)
%timeit x2_map(range(1,11))

#Method 3: LC
def x2_lc(arr):
    [i*2 for i in arr]
%timeit x2_lc(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
In this case, LC is ~20% faster than for-loop and ~65% faster than map function.
Now this is something incredible. Not only is LC more elegant but also faster than its counterparts. Yes, even I want to get into advanced applications of LC. But hang on! I am not convinced. Why is LC faster? Will it faster in all scenarios or are these special cases? Let's try to find out!
Why is LC fast?
I would not doubt your intellectual skills at this point if you are still wondering why is LC faster. After all it's following the same process:
1. Iterating over the list
2. Modifying each element
3. Storing the result
Let's try to inspect each element one by one. Let's simply call a function that does nothing and check for iteration times:
Can you write Python code for it?

#Method 1: For-loop:
def empty_for(arr):
    for i in arr:
        pass
%timeit empty_for(range(1,11))

#Method 2: Map
def empty_map(arr):
    map(lambda x: None,arr)
%timeit empty_map(range(1,11))

#Method 3: LC
def empty_lc(arr):
    [None for i in arr]
%timeit empty_lc(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
It is clearly evident that with the slight increase in complexity, both for and map routines became bulkier and less readable. However, the LC routine is still concise and required a minor modification.
Before going into more complex examples, let's try to appreciate another advantage of using LC - lower computational time!
Comparing run-times:
Let us compare the time taken for each of the above functions to run. We'll be using the %timeit magic function of iPython notebook to determine the runtime. Alternatively, you can use the time or timeit modules.
Now you will be able to appreciate the importance of writing each code fragment as a function. Also, we shall focus on the relative run times and not the absolute values because it is subject to the machine specs. FYI, I am using a Dell XPS 14Z system with following specs:
2nd Gen i5 (2.5GHz) | 4GB RAM | 64-bit OS | Windows 7 Home Premium
Let's compare the time for first example:
Can you write Python code for it?

%timeit square_for(range(1,11))

%timeit square_map(range(1,11))

%timeit square_lc(range(1,11))


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
The Time Advantage
Map Function Review:
Used to apply a function to each element of a list or any other iterable.
Syntax: map(function, iterable)
For example, we can multiply each element of a list of integers with the next number.
Can you write Python code for it?

arr = range(10) #contains [0,1,...,9]
map(lambda x: x*(x+1), arr)


I am working on the file "Articles/Python_List_Comprehension/List and Dictionary Comprehension-final.ipynb".
The first task is:
Though the three techniques produce the same result, we can see that LC is the most elegant and readable technique. You might argue that even the map function is not bad in this case. But map has its own limitations which are not evident in this example.
Taking a step forward
Let's include a catch here. What if we want the square of only even numbers in the list? The three functions would look like:
Can you write Python code for it?

#Method 1: For-Loop
def square_even_for(arr):
    result = []
    for i in arr:
        if i%2 == 0:
            result.append(i**2)
    return result
print square_even_for(range(1,11))

#Method 2: Map Function
def square_even_map(arr):
    return filter(lambda x: x is not None,map(lambda x: x**2 if x%2==0 else None, arr))
print square_even_map(range(1,11))

#Method 3: List comprehension:
def square_even_lc(arr):
    return [i**2 for i in arr if i%2==0]
print square_even_lc(range(1,11))


I am working on the file "Articles/Ridge_Lasso_Regression/Ridge_Lasso_Regression.ipynb".
The first task is:
Creating the data:
Here I am going to take a sine function with angles between 60 and 3000 degrees. The corresponding sine value is modifier with some random noise. The aim is to get as close to the sine function as possible using regression models.
Can you write Python code for it?

x = np.array([i*np.pi/180 for i in range(60,300,4)])
np.random.seed(10)  #Setting seed for reproducability
y = np.sin(x) + np.random.normal(0,0.15,len(x))
data = pd.DataFrame(np.column_stack([x,y]),columns=['x','y'])
plt.plot(data['x'],data['y'],'.')

print data.shape


I am working on the file "Articles/Ridge_Lasso_Regression/Ridge_Lasso_Regression.ipynb".
The first task is:
Determining overfitting:
Create 15 powers:
Can you write Python code for it?

#Create powers upto 15:
for i in range(2,16):
    colname = 'x_%d'%i
    data[colname] = data['x']**i
print data.head()


I am working on the file "Articles/Ridge_Lasso_Regression/Ridge_Lasso_Regression.ipynb".
The first task is:
Fit a Linear regression model on the 15 powers:
Can you write Python code for it?

#Initialize a dataframe to store the results:
col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
ind = ['model_pow_%d'%i for i in range(1,16)]
coef_matrix_simple = pd.DataFrame(index=ind, columns=col)

#Define the powers for which a plot is required:
models_to_plot = {1:231,3:232,6:233,9:234,12:235,15:236}

#Iterate through all powers and assimilate results
for i in range(1,16):
    coef_matrix_simple.iloc[i-1,0:i+2] = linear_regression(data, power=i, models_to_plot=models_to_plot)

#Set the display format to be scientific for ease of analysis
pd.options.display.float_format = '{:,.2g}'.format
coef_matrix_simple


I am working on the file "Articles/Ridge_Lasso_Regression/Ridge_Lasso_Regression.ipynb".
The first task is:
Lasso Modeling:
Can you write Python code for it?

from sklearn.linear_model import Lasso
def lasso_regression(data, predictors, alpha, models_to_plot={}):
    #Fit the model
    lassoreg = Lasso(alpha=alpha,normalize=True, max_iter=1e6)
    lassoreg.fit(data[predictors],data['y'])
    y_pred = lassoreg.predict(data[predictors])

    #Check if a plot is to be made for the entered alpha
    if alpha in models_to_plot:
        plt.subplot(models_to_plot[alpha])
        plt.tight_layout()
        plt.plot(data['x'],y_pred)
        plt.plot(data['x'],data['y'],'.')
        plt.title('Plot for alpha: %.3g'%alpha)

    #Return the result in pre-defined format
    rss = sum((y_pred-data['y'])**2)
    ret = [rss]
    ret.extend([lassoreg.intercept_])
    ret.extend(lassoreg.coef_)
    return ret

predictors=['x']
predictors.extend(['x_%d'%i for i in range(2,16)])

alpha_lasso = [1e-15, 1e-10, 1e-8, 1e-5,1e-4, 1e-3,1e-2, 1, 5, 10]

col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
ind = ['alpha_%.2g'%alpha_lasso[i] for i in range(0,10)]
coef_matrix_lasso = pd.DataFrame(index=ind, columns=col)

models_to_plot = {1e-10:231, 1e-5:232,1e-4:233, 1e-3:234, 1e-2:235, 1:236}
for i in range(10):
    coef_matrix_lasso.iloc[i,] = lasso_regression(data, predictors, alpha_lasso[i], models_to_plot)

#Set the display format to be scientific for ease of analysis
pd.options.display.float_format = '{:,.2g}'.format
coef_matrix_lasso

coef_matrix_lasso.apply(lambda x: sum(x.values==0),axis=1)


I am working on the file "Articles/Ridge_Lasso_Regression/Ridge_Lasso_Regression.ipynb".
The first task is:
Though RSS is going down, but the coefficients are increasing in magnitude.
Ridge Modeling:
Can you write Python code for it?

from sklearn.linear_model import Ridge
def ridge_regression(data, predictors, alpha, models_to_plot={}):
    #Fit the model
    ridgereg = Ridge(alpha=alpha,normalize=True)
    ridgereg.fit(data[predictors],data['y'])
    y_pred = ridgereg.predict(data[predictors])

    #Check if a plot is to be made for the entered alpha
    if alpha in models_to_plot:
        plt.subplot(models_to_plot[alpha])
        plt.tight_layout()
        plt.plot(data['x'],y_pred)
        plt.plot(data['x'],data['y'],'.')
        plt.title('Plot for alpha: %.3g'%alpha)

    #Return the result in pre-defined format
    rss = sum((y_pred-data['y'])**2)
    ret = [rss]
    ret.extend([ridgereg.intercept_])
    ret.extend(ridgereg.coef_)
    return ret

# predictors=['x']
# predictors.extend(['x_%d'%i for i in range(2,16)])
# alp = 1e5
# print ridge_regression(data, predictors, alpha=alp, models_to_plot={alp:111})

    predictors=['x']
    predictors.extend(['x_%d'%i for i in range(2,16)])

    alpha_ridge = [1e-15, 1e-10, 1e-8, 1e-4, 1e-3,1e-2, 1, 5, 10, 20]

    col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
    ind = ['alpha_%.2g'%alpha_ridge[i] for i in range(0,10)]
    coef_matrix_ridge = pd.DataFrame(index=ind, columns=col)

    models_to_plot = {1e-15:231, 1e-10:232, 1e-4:233, 1e-3:234, 1e-2:235, 5:236}
    for i in range(10):
        coef_matrix_ridge.iloc[i,] = ridge_regression(data, predictors, alpha_ridge[i], models_to_plot)

#Set the display format to be scientific for ease of analysis
pd.options.display.float_format = '{:,.2g}'.format
coef_matrix_ridge

coef_matrix_ridge.apply(lambda x: sum(x.values==0),axis=1)


I am working on the file "Artificial Neural Network Example.ipynb".
The first task is:
The mean squared error (MSE) provides measure of the distance between the actual value and what is estimated by the neural network.
Can you write Python code for it?

def mean_squared_error(X):
    return np.power(X, 2).mean(axis=None)


I am working on the file "Artificial Neural Network Example.ipynb".
The first task is:
The nn_predict function takes the theta values calculated by nn_train to make predictions about the data in X.
Can you write Python code for it?

def nn_predict(X, theta1, theta2):
    a2 = sigmoid(np.insert(X, 0, 1, axis=1).dot(theta1))
    return sigmoid(np.insert(a2, 0, 1, axis=1).dot(theta2))


I am working on the file "Artificial Neural Network Example.ipynb".
The first task is:
The nn_train function trains an artificial neural network with a single hidden layer. Each column in X is a feature and each row in X is a single training observation. The y value contains the classifications for each observation. For  multi-classification problems, y will have more than one column. After training, this function returns the calculated theta values (weights) that can be used for predictions.
The training will end when the desired error or maximum iterations is reached whichever comes first.
Can you write Python code for it?

def nn_train(X, y, desired_error = 0.001, max_iterations = 100000, hidden_nodes = 5):

    m = X.shape[0]
    input_nodes = X.shape[1]
    output_nodes = y.shape[1]

    a1 = np.insert(X, 0, 1, axis=1)
    theta1 = theta_init(input_nodes, hidden_nodes)
    theta2 = theta_init(hidden_nodes, output_nodes)

    for x in range(0, max_iterations):
        # Feedforward
        a2 = np.insert(sigmoid(a1.dot(theta1)), 0, 1, axis=1)
        a3 = sigmoid(a2.dot(theta2))

        # Calculate error using backpropagation
        a3_delta = np.subtract(y, a3)
        mse = mean_squared_error(a3_delta)
        if mse <= desired_error:
            print "Achieved requested MSE %f at iteration %d" % (mse, x)
            break
        a2_error = a3_delta.dot(theta2.T)
        a2_delta = np.multiply(a2_error, sigmoid_derivative(a2))

        # Update thetas to reduce the error on the next iteration
        theta2 += np.divide(a2.T.dot(a3_delta), m)
        theta1 += np.delete(np.divide(a1.T.dot(a2_delta), m), 0, 1)

    return (theta1, theta2)


I am working on the file "Artificial Neural Network Example.ipynb".
The first task is:
The theta_init function is used to initialize the thetas (weights) in the network. It returns a random matrix with values in the range of [-epsilon, epsilon].
Can you write Python code for it?

def theta_init(in_size, out_size, epsilon = 0.12):
    return np.random.rand(in_size + 1, out_size) * 2 * epsilon - epsilon


I am working on the file "Artificial Neuron/Artificial Neuron.ipynb".
The first task is:
Artificial Neuron
Artificial neuron has the structure shown in the figure below. Essentially, a neuron takes the weighted linear combination of inputs and feeds its value into activation function. The latter should be differentiable, so it is possbile to compute its gradient needed for the learning.
<img height='500' width='500' src='https://upload.wikimedia.org/wikipedia/commons/6/60/ArtificialNeuronModel_english.png'/>
We are going to use sigmoid activation function.
Can you write Python code for it?

# plots sigmoid function
xs = np.linspace(-10, 10)
plt.figure(figsize=(10, 7))
plt.plot(xs, sp.special.expit(xs))
plt.vlines(0, 0 , 1, linestyles='dashed')
plt.hlines(0.5, -10, 10, linestyles='dashed')
plt.hlines(1, -10, 10, linestyles='dashed')
plt.hlines(0, -10, 10, linestyles='dashed')
plt.ylim(-0.1, 1.1)


I am working on the file "Artificial Neuron/Artificial Neuron.ipynb".
The first task is:
Now it's time to plot decision boundary. All points above the line are classified as '1' and as '0' below the line. We can see that the obtained line perfectly divides two sets of points belonging of different classes.
Can you write Python code for it?

plt.figure(figsize=(10, 7))
plt.plot(np.linspace(-5, 5), -1 * (np.linspace(-5, 5)*W[1]/W[2] + W[0]/W[2]))
plt.scatter(class1[0], class1[1], color = 'b', marker='x', label='0')
plt.scatter(class2[0], class2[1], color = 'r', marker = 'o', label='1')
plt.xlim(0, 5)
plt.ylim(0, 5)
plt.legend()


I am working on the file "Artificial Neuron/Artificial Neuron.ipynb".
The first task is:
Now we can plot the decision boundary. It looks exactly the same as for artifical neuron implemented by ourselves.
Can you write Python code for it?

plt.figure(figsize=(10, 7))
plt.plot(np.linspace(-5, 5), -1 * (np.linspace(-5, 5)*W.eval()[1]/W.eval()[2] + W.eval()[0]/W.eval()[2]))
plt.scatter(class1[0], class1[1], color = 'b', marker='x', label='0')
plt.scatter(class2[0], class2[1], color = 'r', marker = 'o', label='1')
plt.xlim(0, 5)
plt.ylim(0, 5)
plt.legend()


I am working on the file "Artificial Neuron/Artificial Neuron.ipynb".
The first task is:
TensorFlow session should be closed after we're done with computations.
Can you write Python code for it?


I am working on the file "Ass2.5/END3_MNIST_addnumber_ass2_5.ipynb".
The first task is:
Make the train and test datasets
Can you write Python code for it?

train_set = MNIST_add_dataset(trainset=True)
test_set = MNIST_add_dataset(trainset=False)


I am working on the file "Ass2.5/END3_MNIST_addnumber_ass2_5.ipynb".
The first task is:
Model
The model consists of 3 convolutional blocks, followed by 3 linear blocks, with ReLU activation in between. The MNIST image is fed to the 1st Conv block, and the ranbdom number (one hot) to the first linear block (alongwith the output of the 3rd Conv block, concatenated).
Can you write Python code for it?

class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    # conv layer 1
    self.conv1 = nn.Sequential(

        nn.Conv2d(1,16,5), # 16x24x24
        nn.ReLU(),
        nn.MaxPool2d(2,2)  # 16x12x12
    )
    self.conv2 = nn.Sequential(
        nn.Conv2d(16,32,5), # 32x8x8
        nn.ReLU(),
        nn.MaxPool2d(2,2)  # 32x4x4
    )
    self.conv3 = nn.Sequential(
        nn.Conv2d(32,10,3), # 10x2x2
        nn.MaxPool2d(2,2)   # 10x1x1
    )
    self.relu = nn.ReLU()
    self.fc1 = nn.Linear(10+10, 60) # adding random number one hot to the 1x10 MNIST output
    self.fc2 = nn.Linear(60, 30)
    self.fc3 = nn.Linear(30, 15) # 10 for MNIST 1-hot coding, and 5 for binary repr of sum of digits

  def forward(self, image, number):
    #print("0 ",image.shape)
    x = self.conv1(image)
    #print("1 ",x.shape)
    x = self.conv2(x)
    #print("2 ",x.shape)
    x = self.conv3(x)
    #print("3 ",x.shape)
    x = x.view(-1,10)
    #print("after ",x.shape)
    # concatenate the number
    x = torch.cat((x,number),1)
    x = self.fc1(x)
    x = self.relu(x)
    x = self.fc2(x)
    x = self.relu(x)
    x = self.fc3(x)
    x = x.view(-1,15)
    #print("In forward x shape ",x.shape)

    # The first 10 outputs should be the onehot encoding of the MNIST digit
    # using a Log softmax (with NLL Loss) for this
    o1 = F.log_softmax(x[:,:10])
    #print("In forward o1 shape ",o1.shape)

    # for the 5 digit sum outout - as it is a multi-label classification, I am using a Sigmoid and not a softmax as there
    # will be multiple 1's in the output
    # used Hardsigmoid as it has a more sharp curve
    sig = nn.Hardsigmoid()
    o2 = sig(x[:,10:])
    #print("In forward o2 shape ",o2.shape)
    return torch.cat((o1,o2),1)

model = Model().to(device)
print(model)
# random test
model.forward(torch.rand((1, 1, 28, 28)).to(device), torch.rand((1, 10)).to(device))

def train(model, device, train_loader, optimizer, epoch, losses):
    print(f"EPOCH - {epoch}")
   ...