I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
3. Training the model
Training
During training we are going to compute the forward pass first to get the value of the loss function.
After that we are doing the backward pass and computing all gradients the loss wrt parameters at each layer with backpropagation.
Can you write Python code for it?

# this functions trains the model
def train(sess, x, y, y_pred, loss, checkpoint_dir):
    num_epochs = 30
    batch_size = 10
    log_dir = 'local/logs'
    utils.clear_dir(log_dir)
    utils.clear_dir(checkpoint_dir)

    learning_rate = 1e-3
    decay_power = 1.0

    global_step = tf.Variable(0, trainable=False)
    decay_steps = num_epochs * train_data.num_batches
    # usually SGD learning rate is decreased over time which enables us
    # to better fine-tune the parameters when close to solution
    lr = tf.train.polynomial_decay(learning_rate, global_step, decay_steps,
                                   end_learning_rate=0, power=decay_power)

    train_step = tf.train.AdamOptimizer(lr).minimize(loss, global_step=global_step)

    saver = tf.train.Saver()

    summary_all = tf.summary.merge_all()
    train_writer = tf.summary.FileWriter(join(log_dir, 'train'), sess.graph)

    tf.global_variables_initializer().run(session=sess)

    step = 0
    best_iou = 0
    best_epoch = 0
    exp_start_time = time.time()
    for epoch in range(1, num_epochs+1):
        # confusion_mat = np.zeros((num_classes, num_classes), dtype=np.uint64)
        print('\nTraining phase:')
        for x_np, y_np, names in train_data:
            start_time = time.time()
            loss_np, summary, _ = sess.run([loss, summary_all, train_step],
                feed_dict={x: x_np, y: y_np})
            train_writer.add_summary(summary, step)
            duration = time.time() - start_time
            # confusion_mat += batch_conf_mat.astype(np.uint64)
            if step % 20 == 0:
#             if step % 2 == 0:
                string = '%s: epoch %d / %d, iter %05d, loss = %.2f  (%.1f images/sec)' % \
                (utils.get_expired_time(exp_start_time), epoch, num_epochs, step, loss_np, batch_size / duration)
                print(string)
            step += 1
        # utils.print_metrics(confusion_mat, 'Train')
        # add this later
        iou = validate(sess, val_data, x, y, y_pred, loss, draw_steps=5)
        if iou...


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Dataset
The Dataset class implements an iterator which returns the next batch data in each iteration. Data is already normalized to have zero mean and unit variance. The iteration is terminated when we reach the end of the dataset (one epoch).
Can you write Python code for it?

batch_size = 10
num_classes = Dataset.num_classes
# create the Dataset for training and validation
train_data = Dataset('train', batch_size)
val_data = Dataset('val', batch_size, shuffle=False)
# downsample = 2
# train_data = Dataset('train', batch_size, downsample)
# val_data = Dataset('val', batch_size, downsample, shuffle=False)

print('Train shape:', train_data.x.shape)
print('Validation shape:', val_data.x.shape)

#print('mean = ', train_data.x.mean((0,1,2)))
#print('std = ', train_data.x.std((0,1,2)))


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Inputs
First, we will create input placeholders for Tensorflow computational graph of the model. For a supervised learning model, we need to declare placeholders which will hold input images (x) and target labels (y) of the mini-batches as we feed them to the network.
Can you write Python code for it?

# store the input image dimensions
height = train_data.height
width = train_data.width
channels = train_data.channels

# create placeholders for inputs
def build_inputs():
    with tf.name_scope('data'):
        x = tf.placeholder(tf.float32, shape=(None, height, width, channels), name='rgb_images')
        y = tf.placeholder(tf.int32, shape=(None, height, width), name='labels')
    return x, y


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Loss
Now we are going to implement the build_loss function which will create nodes for loss computation and return the final tf.Tensor representing the scalar loss value.
Because segmentation is just classification on a pixel level we can again use the cross entropy loss function \(L\) between the target one-hot distribution \( \mathbf{y} \) and the predicted distribution from a softmax layer \( \mathbf{s} \). But compared to the image classification here we need to define the loss at each pixel. Below are the equations describing the loss for just one example (one pixel in our case).
$$
L = - \sum_{i=1}^{C} y_i log(s_j(\mathbf{x})) \
s_i(\mathbf{x}) = \frac{e^{x_i}}{\sum_{j=1}^{C} e^{x_j}} \
$$
Can you write Python code for it?

# this funcions takes logits and targets (y) and builds the loss subgraph
def build_loss(logits, y):
  with tf.name_scope('loss'):
    # vectorize the image
    y = tf.reshape(y, shape=[-1])
    logits = tf.reshape(logits, [-1, num_classes])

    # gather all labels with valid ID
    mask = y < num_classes
    y = tf.boolean_mask(y, mask)
    logits = tf.boolean_mask(logits, mask)

    # define softmax and cross entropy loss
    y_one_hot = tf.one_hot(y, num_classes)
    xent = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_one_hot)

    # take the mean because we don't want the loss to depend on the number of pixels in batch
    xent = tf.reduce_mean(xent)
    tf.summary.scalar('cross_entropy', xent)
    return xent


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Model
Now we can define the computational graph. Here we will heavily use tf.layers high level API which handles tf.Variable creation for us. The main difference here compared to the classification model is that the network is going to be fully convolutional without any fully connected layers. Brief sketch of the model we are going to define is given below.
conv3x3(32) -> 4 x (pool2x2 -> conv3x3(64) -> conv3x3(64)) -> conv1x1(7) -> resize_bilinear -> softmax() -> Loss
Can you write Python code for it?

# helper function which applies conv2d + ReLU with filter size k
def conv(x, num_maps, k=3):
    x = tf.layers.conv2d(x, num_maps, k, padding='same')
    x = tf.nn.relu(x)
    return x

# helper function for 2x2 max pooling with stride=2
def pool(x):
    return tf.layers.max_pooling2d(x, pool_size=2, strides=2, padding='same')

# this functions takes the input placeholder and the number of classes, builds the model and returns the logits
def build_model(x, num_classes):
    input_size = x.get_shape().as_list()[1:3]
    block_sizes = [64, 64, 64, 64]
    x = conv(x, 32, k=3)
    for i, size in enumerate(block_sizes):
        with tf.name_scope('block'+str(i)):
            x = pool(x)
            x = conv(x, size)
            x = conv(x, size)
    print(x)
    with tf.name_scope('logits'):
        x = tf.layers.conv2d(x, num_classes, 1, padding='same')
        # ask why no relu
        x = tf.image.resize_bilinear(x, input_size, name='upsample_logits')
    return x


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Putting it all together
Now we can use all the building blocks from above and construct the whole forward pass Tensorflow graph in just a couple of lines.
Can you write Python code for it?

# create inputs
x, y = build_inputs()
# create model
logits = build_model(x, num_classes)
# create loss
loss = build_loss(logits, y)
# we are going to need argmax predictions for IoU
y_pred = tf.argmax(logits, axis=3, output_type=tf.int32)


I am working on the file "Day-2/segmentation/semantic_segmentation_solved.ipynb".
The first task is:
Tensorboard
$ tensorboard --logdir=local/logs/
4. Restoring the pretrained network
Can you write Python code for it?

# restore the best checkpoint
checkpoint_path = 'local/pretrained1/model.ckpt'
saver = tf.train.Saver()
saver.restore(sess, checkpoint_path)
validate(sess, val_data, x, y, y_pred, loss, draw_steps=10)


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
3.2 Building the model
After having our data prepared and convenience functions split_data and get_batch for handling the data during the training of our model, we can finally start building the model using the TensorFlow library. We will break the model building into five parts:
* building input placeholders for x, y and dropout
* building multi-layer RNN with stacked LSTM cells
* building softmax output layer
* computation for training loss
* building the optimizer for the model parameters
Inputs
First, we will create our input placeholders for Tensorflow computational graph of the model. As we are building supervised learning model, we need to declare placeholders for inputs (x) and targets (y). We also need to one-hot encode the input and target tokens, remember we are getting them as encoded characters. Here, we will also declare scalar placeholder keep_prob for output keep probablity for dropout.
New functions used here:
- tf.name_scope
- tf.one_hot
Exercise: Define placeholders for inputs and targets.
Can you write Python code for it?

def build_inputs(batch_size, num_steps, num_classes):
    ''' Define placeholders for inputs, targets, and dropout.

        Arguments
        ---------
        batch_size: Batch size, number of sequences per batch
        num_steps: Number of sequence steps in a batch
        num_classes: Number of classes (target values)

    '''

    with tf.name_scope('inputs'):
        # EXERCISE: Declare placeholder for inputs and one-hot encode inputs

    with tf.name_scope('targets'):
        # EXERCISE: Declare placeholder for targets and one-hot encode targets

    # Keep probability placeholder for drop out layers
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')

    return inputs, x_one_hot, targets, y_one_hot, keep_prob


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
3.4 Testing model - sampling from the model
Can you write Python code for it?

from IPython.core.display import display, HTML

def pick_top_n(preds, vocab_size, top_n=5):
    p = np.squeeze(preds)
    p[np.argsort(p)[:-top_n]] = 0
    p = p / np.sum(p)
    c = np.random.choice(vocab_size, 1, p=p)[0]
    return c

def sample_model(checkpoint, n_samples, lstm_size, vocab_size, num_layers=2, prime="The "):
    samples = [c for c in prime]
    model = CharRNN(len(vocab), lstm_size=lstm_size, num_layers=num_layers, sampling=True)
    saver = tf.train.Saver()

    states = []
    with tf.Session() as sess:
        saver.restore(sess, checkpoint)
        new_state = sess.run(model.initial_state)

        for c in prime:
            x = np.zeros((1, 1))
            x[0,0] = vocab_to_int[c]
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state = sess.run([model.prediction, model.final_state],
                                         feed_dict=feed)

            states.append(new_state)

        c = pick_top_n(preds, len(vocab))
        samples.append(int_to_vocab[c])
        states.append(new_state)

        for i in range(n_samples):
            x[0,0] = c
            feed = {model.inputs: x,
                    model.keep_prob: 1.,
                    model.initial_state: new_state}
            preds, new_state = sess.run([model.prediction, model.final_state],
                                         feed_dict=feed)

            c = pick_top_n(preds, len(vocab))
            samples.append(int_to_vocab[c])
            states.append(new_state)

    return (''.join(samples), states)


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
And see how they are encoded as integers:
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Build the network
Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use tf.nn.dynamic_rnn. This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as final_state so we can pass it to the first LSTM cell in the the next mini-batch run. For tf.nn.dynamic_rnn, we pass in the cell and initial state we get from build_lstm, as well as our input sequences.
Exercise: Fill in CharRNN class to run each sequence step through the RNN and collect the outputs using tf.nn.dynamic_rnn.
Can you write Python code for it?

class CharRNN:

    def __init__(self, num_classes, batch_size=64, num_steps=50,
                       lstm_size=128, num_layers=2, learning_rate=0.001,
                       grad_clip=5, sampling=False):

        if sampling == True:
            # When we will use the network for sampling later, we will pass in one character at a time
            batch_size, num_steps = 1, 1
        else:
            batch_size, num_steps = batch_size, num_steps

        tf.reset_default_graph()

        # Build the input placeholder tensors, and one-hot encode the input and target tokens
        self.inputs, x_one_hot, self.targets, y_one_hot, self.keep_prob = \
        build_inputs(batch_size, num_steps, num_classes)

        # Build the LSTM cell
        cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, self.keep_prob)

        with tf.name_scope("RNN_forward"):

            # EXERCISE: Run each sequence step through the RNN and collect the outputs

        self.final_state = state

        # Get softmax predictions and logits
        self.prediction, self.logits = build_output(outputs, lstm_size, num_classes)

        # Loss and optimizer (with gradient clipping)
        self.loss = build_loss(self.logits, y_one_hot, lstm_size)
        self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)

        self.summary_merged = tf.summary.merge_all()


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Building RNN Output Layer
Here we will create the output layer. We need to connect the output of the RNN cells to a fully connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character. The output 3D tensor with size $(batch_size \times num_steps \times lstm_size)$ has to be reshaped to $((batch_size \times num_steps) \times  lstm_size)$, so we can do the matrix multiplication with the softmax weights.
The output is calculated using softmax function
$$
    P(y=c\text{ } | \text{ }\mathbf{x}) = \frac{e^{\mathbf{x}^T\mathbf{w}c+b_c}}{\sum{k=1}^{|C|}e^{\mathbf{x}^T\mathbf{w}_k+b_k}}
    ,\
$$
where $\mathbf{x}\in\mathbb{R}^{512}$ is output of the last hidden layer, and $\mathbf{W}\in\mathbb{R}^{512\times 92}$ and $\mathbf{b}\in\mathbb{R}^{92}$ are the model parameters.
Exercise: Fill in build_output function by defining logits and softmax function.
Can you write Python code for it?

def build_output(lstm_output, in_size, out_size):
    ''' Build a softmax layer, return the softmax output and logits.

        Arguments
        ---------

        lstm_output: Output tensor of previous layer
        in_size: Size of the input tensor
        out_size: Size of the softmax layer

    '''

    # Reshape output so it is a bunch of rows, one row for each step for each sequence.
    # That is, the shape should be batch_size*num_steps rows by lstm_size columns.
    with tf.name_scope('sequence_reshape'):
        seq_output = tf.concat(lstm_output, axis=1, name='seq_output')
        x = tf.reshape(seq_output, [-1, in_size], name='graph_output')

    # Connect the RNN outputs to a softmax layer
    with tf.name_scope('logits'):

        # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch
        # of rows of logit outputs, one for each step and sequence

        # EXERCISE: Define W and b and multiply inputs with weights and add bias

        # Tensorboard
        tf.summary.histogram('h_softmax_w', softmax_w)
        tf.summary.histogram('h_softmax_b', softmax_b)

    with tf.name_scope('predictions'):

        # EXERCISE: Use softmax to get the probabilities for predicted characters

        # Tensorboard
        tf.summary.histogram('h_predictions', predictions)

    return predictions, logits


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Fill in build_lstm function by stacking layers using tf.contrib.rnn.MultiRNNCell.
Can you write Python code for it?

def build_lstm(lstm_size, num_layers, batch_size, keep_prob):
    ''' Build Multi-RNN cell.

        Arguments
        ---------
        lstm_size: Size of the hidden layers in the LSTM cells
        num_layers: Number of LSTM layers
        batch_size: Batch size
        keep_prob: Dropout keep probability

    '''

    # EXERCISE: Stack up multiple LSTM layers

    with tf.name_scope("RNN_init_state"):
        initial_state = cell.zero_state(batch_size, tf.float32)

    return cell, initial_state


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Generate example integer array. Use function split_data to split example_arr into train and validation sets.
Can you write Python code for it?

# example array generation

batch_size = 5
num_steps = 3
split_frac = 0.9

# split example array into train and validation sets


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Load checkpoint trump_tb_20_i3880_l512_1.327.ckpt from assets/checkpoints/ssds/ and generate some sample text using function sample_model. The use utility function make_colored_text to color each character by cell activations in certain layer.
Can you write Python code for it?

checkpoint = 'assets/checkpoints/ssds/trump_tb_20_i3880_l512_1.327.ckpt'

# sample model from the loaded checkpoint


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Train the model again starting from the initial state for a few iterations and save a checkpoint, then load it and generate text using sample_model function.
Can you write Python code for it?

# load generated checkpoint

# generate text using sample_model function


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Use the for loop to iterate through all train and validation batches.
Can you write Python code for it?

# iterate through all train and validation batches


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Exercise: Use utility funtion make_colored_text and Jupyter widget HTML to visualize cell activations for the text above. Here are some examples of interesting visualizations.
layer_id = 0
cell_id:
- position in tweet - 4
- short urls - 10, 50, 130, 160, 163, 164, 183, 218, 230
- separate fixed and variable part of short url - 80, 152
- just variable part of short url - 75, 84, 118, 273, 380
- position in short url - 22*, 112, 206, 386
- urls and references - 115, 403, 483
layer_id = 1
cell_id:
- just variable part of short url - 21, 107, 250, 300, 420
- beginning of a word - 22, 112
- urls and references - 51, 273, 438
- position in short url - 202, 326
- quotation marks - 252
- position in a sentence - 413
Can you write Python code for it?

# Position in a tweet
HTML(make_colored_text(samp, states, cell_id=4, layer_id=0))

# Beggining of a word
HTML(make_colored_text(samp, states, cell_id=22, layer_id=1))


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Hyperparameters
Here we declare the hyperparameters for the network.

batch_size - Number of sequences running through the network in one pass.
num_steps - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
lstm_size - The number of units in the hidden layers.
num_layers - Number of hidden LSTM layers to use
learning_rate - Learning rate for training
keep_prob - The dropout keep probability when training. If you're network is overfitting, try decreasing this.

Here's some good advice from Andrej Karpathy on training the network https://github.com/karpathy/char-rnn#tips-and-tricks.
Can you write Python code for it?

batch_size = 100        # Sequences per batch
num_steps = 100         # Number of sequence steps per batch
lstm_size = 512         # Size of hidden layers in LSTMs
num_layers = 2          # Number of LSTM layers
learning_rate = 0.001   # Learning rate
keep_prob = 0.5         # Dropout keep probability


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Making training and validation mini-batches
Neural networks are trained by approximating the gradient of loss function with respect to the neuron weights, by looking at only a small subset of the data, also known as a mini-batch. Here is where we will make our mini-batches for training and validation. Now we need to split up the data into batches, as well as into training and validation sets.
For the test we will observe how the network generates new text, thus we will not be using test set. We will feed a character into the network and sample a next one from the distribution over characters likely to come next. We feed the sampled character right back to get next character. Repeating this process character by character will generate new text, hopefully indistinguishable from Donald Trump's Twitter tweets.
Can you write Python code for it?

def split_data(arr, batch_size, num_steps, split_frac=0.9):
    """
    Split data into batches and training and validation sets.

    Arguments
    ---------
    arr: Array of encoded characters as integers
    batch_size: Number of sequences per batch
    num_steps: Length of the sequence in a batch
    split_frac: Fraction of batches to keep in the training set

    Returns train_x, train_y, val_x, val_y
    """

    slice_size = batch_size * num_steps
    n_batches = int(len(arr) / slice_size)

    # Drop the last few characters to make only full batches
    x = arr[: n_batches*slice_size]

    # The targets are the same as the inputs, except shifted one character over.
    # number of batches covers full size of arr (no characters dropped)
    if(len(arr) == n_batches*slice_size):
        # for the last target character use first input character
        y = np.roll(x, -1)
    else:
        # for the last target characher use first dropped character
        y = arr[1: n_batches*slice_size + 1]

    # Split the data into batch_size slices and then stack slices
    x = np.stack(np.split(x, batch_size))
    y = np.stack(np.split(y, batch_size))

    # Now x and y are arrays with dimensions batch_size x (n_batches x num_steps)

    # Split into training and validation sets, keep the first split_frac batches for training
    split_idx = int(n_batches*split_frac)
    train_x, train_y= x[:, :split_idx*num_steps], y[:, :split_idx*num_steps]
    val_x, val_y = x[:, split_idx*num_steps:], y[:, split_idx*num_steps:]

    return train_x, train_y, val_x, val_y


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Multi-layer LSTM Cell
We first implement build_cell function where we create the LSTM cell we will use in the hidden layer. We will use this cell as a building block for the multi-layer RNN. Afterwards, we implement the build_lstm function to create multiple LSTM cells stacked on each other using build_cell function. We can stack up the LSTM cells into layers with tf.contrib.rnn.MultiRNNCell.
Exercise: Fill in build_cell function for building LSTM cell using:

tf.contrib.rnn.BasicLSTMCell
contrib.rnn.DropoutWrapper.
Can you write Python code for it?

def build_cell(lstm_size, keep_prob):
    ''' Build LSTM cell.

        Arguments
        ---------
        lstm_size: Size of the hidden layers in the LSTM cells
        keep_prob: Dropout keep probability

    '''

    # EXERCISE: Use a basic LSTM cell

    # EXERCISE: Add dropout to the cell

    return drop


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Next, we will create a generator function to get batches from the arrays made by split_data. This will provide us with the functionality to iterate over batches, which we can feed to our network model. The arrays are of dimension (batch_size, n_batches*num_steps). Each batch is a sliding window on these arrays with size batch_size X num_steps.
Can you write Python code for it?

def get_batch(arrs, num_steps):
    batch_size, slice_size = arrs[0].shape

    n_batches = int(slice_size/num_steps)
    for b in range(n_batches):
        yield [x[:, b*num_steps: (b+1)*num_steps] for x in arrs]


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Optimizer
Here we build the optimizer. Traditional RNNs face vanishing gradient problem. LSTMs fix the vanishing problem, but the gradients can still grow without bound. To fix this we can clip the gradients larger than some threshold. That is, if a gradient is larger than the prespecified threshold, we set it to the threshold value. This will ensure the gradients never grow too large. Then we use an AdamOptimizer for the learning step.
Exercise: Fill in the function build_optimizer:

Calculate and clip gradients using functions tf.gradients and tf.clip_by_global_norm
Define Adam optimizer using tf.train.AdamOptimizer
Apply gradients to trainable variables using function tf.train.Optimizer.apply_gradients
Can you write Python code for it?

def build_optimizer(loss, learning_rate, grad_clip):
    ''' Build optmizer for training, using gradient clipping.

        Arguments:
        loss: Network loss
        learning_rate: Learning rate for optimizer
        grad_clip: Clipping ratio

    '''

    # Optimizer for training, using gradient clipping to control exploding gradients
    with tf.name_scope('optimizer'):
        tvars = tf.trainable_variables()

        # EXERCISE: Calculate and clip gradients

        # EXERCISE: Use Adam optimizer

        # EXERCISE: Apply gradients to trainable variables

    return optimizer


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Run tensorboard from command line by issuing command (e.g. from root repository directory):
tensorboard --logdir=Day-3/assets/logs/
3.3 Training model
This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by save_every_n) we calculate the validation loss and save a checkpoint.
Please download provided trump_tb_20_i3880_l512_1.327.ckpt
checkpoint and place it in assets/checkpoints/ssds direcory in the repository.
Exercise: Fill in the code below:

Through all train batches run session and save loss.
Through all validation batches run session and append validation loss.
Can you write Python code for it?

epochs = 1 #20
save_every_n = 10 #200
train_x, train_y, val_x, val_y = split_data(encoded_chars, batch_size, num_steps)

model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
                lstm_size=lstm_size, num_layers=num_layers,
                learning_rate=learning_rate)

saver = tf.train.Saver(max_to_keep=100)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    # Tensorboard
    train_writer = tf.summary.FileWriter('assets/logs/2/train', sess.graph)
    test_writer = tf.summary.FileWriter('assets/logs/2/test')

    #############################################################
    # Use the line below to load a checkpoint and resume training
    saver.restore(sess, 'assets/checkpoints/ssds/trump_tb_20_i3880_l512_1.327.ckpt')
    #############################################################

    n_batches = int(train_x.shape[1]/num_steps)
    iterations = n_batches * epochs

    # Train network
    for e in range(epochs):

        new_state = sess.run(model.initial_state)
        loss = 0

        for b, (x, y) in enumerate(get_batch([train_x, train_y], num_steps), 1):
            start = time.time()

            # EXERCISE: Run session and save loss for train batches

            end = time.time()
            iteration = e*n_batches + b
            print('Epoch {}/{} '.format(e+1, epochs),
                  'Iteration {}/{}'.format(iteration, iterations),
                  'Training loss: {:.4f}'.format(loss/b),
                  '{:.4f} sec/batch'.format((end-start)))

            # Tensorboard
            train_writer.add_summary(summary, iteration)

            if (iteration%save_every_n == 0) or (iteration == iterations):
                # Check performance, notice dropout has been set to 1
                val_loss = []
                new_state = sess.run(model.initial_state)

                # EXERCISE: Same as above, run session and append...


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Saved checkpoints
Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Use the code below to generate html file that contains colorings of the text above from all 512 cells.
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs.ipynb".
The first task is:
Write out the graph for TensorBoard
Can you write Python code for it?

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    file_writer = tf.summary.FileWriter('assets/logs/1', sess.graph)

    file_writer.close()


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
3.1 Data preparation
Loading and encoding text
We will train our language model on a complete collection of Donald Trump's tweets obtained from Trump Twitter Archive, which we already downloaded and made available in PATH-TO-REPOSITORY/Day-3/assets/data/trump_tweets_ascii.txt. First, we will load the text file and encode its characters as integers.
Can you write Python code for it?

with open('assets/data/trump_tweets_ascii.txt', 'r') as f:
    text=f.read()

# get set of characters contained in the loaded text file
vocab = sorted(set(text))

# encoding characters as integers
vocab_to_int = {c: i for i, c in enumerate(vocab)}
encoded_chars = np.array([vocab_to_int[c] for c in text], dtype=np.int32)

# make dict for decoding intergers to corresponding characters
int_to_vocab = dict(enumerate(vocab))

print('Text size: {}'.format(len(encoded_chars)))
print('Vocabulary size: {}'.format(len(vocab)))
print('*******************************')
print('Number of tweets: {}'.format(len(text.split('\n'))))
print('Median size of a tweet: {}'.format(np.percentile([len(t) for t in text.split('\n')], 50)))


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
3.5 Visualization of memory cell activations
Can you write Python code for it?

from IPython.core.display import display, HTML
from utils import save_lstm_vis, make_colored_text


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
And see how they are encoded as integers:
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Building RNN Output Layer
Here we will create the output layer. We need to connect the output of the RNN cells to a fully connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character. The output 3D tensor with size $(batch_size \times num_steps \times lstm_size)$ has to be reshaped to $((batch_size \times num_steps) \times  lstm_size)$, so we can do the matrix multiplication with the softmax weights.
The output is calculated using softmax function
$$
    P(y=c\text{ } | \text{ }\mathbf{x}) = \frac{e^{\mathbf{x}^T\mathbf{w}c+b_c}}{\sum{k=1}^{|C|}e^{\mathbf{x}^T\mathbf{w}_k+b_k}}
    ,\
$$
where $\mathbf{x}\in\mathbb{R}^{512}$ is output of the last hidden layer, and $\mathbf{W}\in\mathbb{R}^{512\times 92}$ and $\mathbf{b}\in\mathbb{R}^{92}$ are the model parameters.
Exercise: Fill in build_output function by defining logits and softmax function.
Can you write Python code for it?

def build_output(lstm_output, in_size, out_size):
    ''' Build a softmax layer, return the softmax output and logits.

        Arguments
        ---------

        lstm_output: Output tensor of previous layer
        in_size: Size of the input tensor
        out_size: Size of the softmax layer

    '''

    # Reshape output so it is a bunch of rows, one row for each step for each sequence.
    # That is, the shape should be batch_size*num_steps rows by lstm_size columns.
    with tf.name_scope('sequence_reshape'):
        seq_output = tf.concat(lstm_output, axis=1, name='seq_output')
        x = tf.reshape(seq_output, [-1, in_size], name='graph_output')

    # Connect the RNN outputs to a softmax layer
    with tf.name_scope('logits'):

        # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch
        # of rows of logit outputs, one for each step and sequence

        # EXERCISE: Define W and b and multiply inputs with weights and add bias
        softmax_w = tf.Variable(tf.truncated_normal((in_size, out_size), stddev=0.1), name='softmax_w')
        softmax_b = tf.Variable(tf.zeros(out_size), name='softmax_b')
        logits = tf.matmul(x, softmax_w) + softmax_b

        # Tensorboard
        tf.summary.histogram('h_softmax_w', softmax_w)
        tf.summary.histogram('h_softmax_b', softmax_b)

    with tf.name_scope('predictions'):

        # EXERCISE: Use softmax to get the probabilities for predicted characters
        predictions = tf.nn.softmax(logits, name='predictions')

        # Tensorboard
        tf.summary.histogram('h_predictions', predictions)

    return predictions, logits


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Create new instance of CharRNN class using parameters defined above. Print trainable variables in the default graph using tensorflow function trainable_variables. Does the number of parameters correspond to what we expect? Hint: Number of parameters in first hidden layer of LSTM is equal to:
$4 \times \big[N_{units} \times (N_{inputs}+1) + N_{units}^{2}\big]$,
where $N_{units}$ is the number of units in hidden layer (lstm_size) and $N_{inputs}$ is the length of the vocabulary.
Can you write Python code for it?

model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,
                lstm_size=lstm_size, num_layers=num_layers,
                learning_rate=learning_rate)

tf.trainable_variables()


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Generate example integer array. Use function split_data to split example_arr into train and validation sets.
Can you write Python code for it?

example_arr = np.arange(63)
print(np.array2string(example_arr, max_line_width=100, separator=', '))

batch_size = 5
num_steps = 3
split_frac = 0.9

train_x, train_y, val_x, val_y = split_data(example_arr, batch_size, num_steps, split_frac)

train_x

train_y


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Load checkpoint trump_tb_20_i3880_l512_1.327.ckpt from assets/checkpoints/ssds/ and generate some sample text using function sample_model. The use utility function make_colored_text to color each character by cell activations in certain layer.
Can you write Python code for it?

checkpoint = 'assets/checkpoints/ssds/trump_tb_20_i3880_l512_1.327.ckpt'
samp, states = sample_model(checkpoint, 1000, lstm_size, len(vocab), prime="Obama (")
print(samp)


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Train the model again starting from the initial state for a few iterations and save a checkpoint, then load it and generate text using sample_model function.
Can you write Python code for it?

checkpoint = 'assets/checkpoints/trump/EARLY_EPOCH_CHECKPOINT'
samp, _ = sample_model(checkpoint, 1000, lstm_size, len(vocab), prime="Obama (")
print(samp)


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Use the for loop to iterate through all train batches.
Can you write Python code for it?

for b, (x, y) in enumerate(get_batch([train_x, train_y], num_steps), 1):
    print('\nBatch {}:'.format(b))
    print(np.stack([x,y]))

for b, (x, y) in enumerate(get_batch([val_x, val_y], num_steps), 1):
    print('\nBatch {}:'.format(b))
    print(np.stack([x,y]))


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Exercise: Use utility funtion make_colored_text and Jupyter widget HTML to visualize cell activations for the text above. Here are some examples of interesting visualizations.
layer_id = 0
cell_id:
- position in tweet - 4
- short urls - 10, 50, 130, 160, 163, 164, 183, 218, 230
- separate fixed and variable part of short url - 80, 152
- just variable part of short url - 75, 84, 118, 273, 380
- position in short url - 22*, 112, 206, 386
- urls and references - 115, 403, 483
layer_id = 1
cell_id:
- just variable part of short url - 21, 107, 250, 300, 420
- beginning of a word - 22, 112
- urls and references - 51, 273, 438
- position in short url - 202, 326
- quotation marks - 252
- position in a sentence - 413
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Hyperparameters
Here we declare the hyperparameters for the network.

batch_size - Number of sequences running through the network in one pass.
num_steps - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
lstm_size - The number of units in the hidden layers.
num_layers - Number of hidden LSTM layers to use
learning_rate - Learning rate for training
keep_prob - The dropout keep probability when training. If you're network is overfitting, try decreasing this.

Here's some good advice from Andrej Karpathy on training the network https://github.com/karpathy/char-rnn#tips-and-tricks.
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
In the above output, we can see that trump_tweets_ascii.txt contains in total 2 167 951 characters. Tweets contain 92 unique characters which will form a vocabulary for a language model.
Lets see first 300 characters of the provided text:
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Making training and validation mini-batches
Neural networks are trained by approximating the gradient of loss function with respect to the neuron weights, by looking at only a small subset of the data, also known as a mini-batch. Here is where we will make our mini-batches for training and validation. Now we need to split up the data into batches, as well as into training and validation sets.
For the test we will observe how the network generates new text, thus we will not be using test set. We will feed a character into the network and sample a next one from the distribution over characters likely to come next. We feed the sampled character right back to get next character. Repeating this process character by character will generate new text, hopefully indistinguishable from Donald Trump's Twitter tweets.
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Multi-layer LSTM Cell
We first implement build_cell function where we create the LSTM cell we will use in the hidden layer. We will use this cell as a building block for the multi-layer RNN. Afterwards, we implement the build_lstm function to create multiple LSTM cells stacked on each other using build_cell function. We can stack up the LSTM cells into layers with tf.contrib.rnn.MultiRNNCell.
Exercise: Fill in build_cell function for building LSTM cell using:

tf.contrib.rnn.BasicLSTMCell
contrib.rnn.DropoutWrapper.
Can you write Python code for it?

def build_cell(lstm_size, keep_prob):
    ''' Build LSTM cell.

        Arguments
        ---------
        lstm_size: Size of the hidden layers in the LSTM cells
        keep_prob: Dropout keep probability

    '''

    # EXERCISE: Use a basic LSTM cell
    lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)

    # EXERCISE: Add dropout to the cell
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)

    return drop


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Next, we will create a generator function to get batches from the arrays made by split_data. This will provide us with the functionality to iterate over batches, which we can feed to our network model. The arrays are of dimension (batch_size, n_batches*num_steps). Each batch is a sliding window on these arrays with size batch_size X num_steps.
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Saved checkpoints
Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Training loss
Next we need to calculate the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First, we need to reshape the one-hot targets so it is a 2D tensor with size $((batch_size \times num_steps) \times  num_classes)$, which match logits. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $num_classes$ units. Then we run the logits and targets through tf.nn.softmax_cross_entropy_with_logits and find the mean to get the loss.
Exercise: Fill in build loss function:

Reshape one-hot encoded targets to match logits
Define loss and cost function using tf.nn.softmax_cross_entropy_with_logits and tf.reduce_mean.
Can you write Python code for it?

def build_loss(logits, y_one_hot, lstm_size):
    ''' Calculate the loss from the logits and the targets.

        Arguments
        ---------
        logits: Logits from final fully connected layer
        y_one_hot: one hot encoding of target
        lstm_size: Number of LSTM hidden units
    '''

    # Softmax cross entropy loss
    with tf.name_scope('loss'):

        # EXERCISE: Reshape one-hot encoded targets to match logits (one row per batch_size per step)
        # then define loss and cost function
        y_reshaped = tf.reshape(y_one_hot, logits.get_shape(), name='y_reshaped')
        loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped, name='loss')
        cost = tf.reduce_mean(loss, name='cost')

        # Tensorboard
        tf.summary.scalar('s_cost', cost)

    return cost


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Use the code below to generate html file that contains colorings of the text above from all 512 cells.
Can you write Python code for it?


I am working on the file "Day-3/3_SSDS_2017_CharLSTMs_Solved.ipynb".
The first task is:
Write out the graph for TensorBoard
Can you write Python code for it?


I am working on the file "Day-5/Day 5 - Vanilla GAN Solved.ipynb".
The first task is:
5.3 The generator network
Can you write Python code for it?

class Generator:
    """The generator network

    the generator network takes as input a vector z of dimension input_dim, and transforms it
    to a vector of size output_dim. The network has one hidden layer of size hidden_dim.

    We will define the following methods:

    __init__: initializes all variables by using tf.get_variable(...)
                and stores them to the class, as well a list in self.theta
    forward: defines the forward pass of the network - how do the variables
                interact with respect to the inputs
    """

    def __init__(self, input_dim, hidden_dim, output_dim):
        """Constructor for the generator network. In the constructor, we will
        just initialize all the variables in the network.

        Args:
            input_dim: The dimension of the input data vector (z).
            hidden_dim: The dimension of the hidden layer of the neural network (h)
            output_dim: The dimension of the output layer (equivalent to the size of the image)

        """

        with tf.variable_scope("generator"):
            self.W1 = tf.get_variable(name="W1",
                                      shape=[input_dim, hidden_dim],
                                      initializer=tf.contrib.layers.xavier_initializer())
            self.b1 = tf.get_variable(name="b1",
                                      shape=[hidden_dim],
                                      initializer=tf.zeros_initializer())

            self.W2 = tf.get_variable(name="W2",
                                      shape=[hidden_dim, output_dim],
                                      initializer=tf.contrib.layers.xavier_initializer())
            self.b2 = tf.get_variable(name="b2",
                                      shape=[output_dim],
                                      initializer=tf.zeros_initializer())

            self.theta = [self.W1, self.W2, self.b1, self.b2]

    def forward(self, z):
        """The...


I am working on the file "Day-5/Day 5 - Vanilla GAN Solved.ipynb".
The first task is:
5.4 The basic network for the discriminator
Can you write Python code for it?

class Discriminator:
    """The discriminator network

    the discriminator network takes as input a vector x of dimension input_dim, and transforms it
    to a vector of size output_dim. The network has one hidden layer of size hidden_dim.

    You will define the following methods:

    __init__: initializes all variables by using tf.get_variable(...)
                and stores them to the class, as well a list in self.theta
    forward: defines the forward pass of the network - how do the variables
                interact with respect to the inputs
    """

    def __init__(self, input_dim, hidden_dim, output_dim):

        with tf.variable_scope("discriminator"):

            self.W1 = tf.get_variable(name="W1",
                                      shape=[input_dim, hidden_dim],
                                      initializer=tf.contrib.layers.xavier_initializer())
            self.b1 = tf.get_variable(name="b1", shape=[hidden_dim],
                                      initializer=tf.zeros_initializer())

            self.W2 = tf.get_variable(name="W2",
                                      shape=[hidden_dim, output_dim],
                                      initializer=tf.contrib.layers.xavier_initializer())
            self.b2 = tf.get_variable(name="b2",
                                      shape=[output_dim],
                                      initializer=tf.zeros_initializer())

            self.theta = [self.W1, self.W2, self.b1, self.b2]

    def forward(self, x):
        """The forward pass of the network -- here we will define the logic of how we combine
        the variables through multiplication and activation functions in order to get the
        output.

        """
        h1 = tf.nn.relu(tf.matmul(x, self.W1) + self.b1)
        logit = tf.matmul(h1, self.W2) + self.b2
        prob = tf.nn.sigmoid(logit)

        return prob, logit


I am working on the file "Day-5/Day 5 - Vanilla GAN Solved.ipynb".
The first task is:
5.6 Check the implementation of the classes
Can you write Python code for it?

d = Discriminator(image_dim, discriminator_hidden_dim, discriminator_output_dim)
for param in d.theta:
    print (param)

g = Generator(random_sample_dim, generator_hidden_dim, generator_output_dim)
for param in g.theta:
    print (param)


I am working on the file "Day-5/Day 5 - Vanilla GAN Solved.ipynb".
The first task is:
Intermezzo: Xavier initialization of weights
Glorot, X., & Bengio, Y. (2010, March). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (pp. 249-256).
Implemented in tensorflow, as part of the standard library: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/xavier_initializer
1. Idea:

If the weights in a network are initialized to too small values, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.
If the weights in a network are initialized to too large, then the signal grows as it passes through each layer until it’s too massive to be useful.

2. Goal:

We need initial weight values that are just right for the signal not to explode or vanish during the forward pass

3. Math

Trivial

4. Solution

$v = \frac{2}{n_{in} + n_{out}}$

In the case of a Gaussian distribution, we set the variance to $v$.
In the case of a uniform distribution, we set the interval  to $\pm v$ (the default distr. in tensorflow is the uniform).
http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
5.5 Define the model parameters
We will take a brief break to set the values for the parameters of the model. Since we know the dataset we are working with, as well as the shape of the generator and discriminator networks, your task is to fill in the values of the following variables.
Can you write Python code for it?

image_dim = 784 # The dimension of the input image vector to the discrminator
discriminator_hidden_dim = 128 # The dimension of the hidden layer of the discriminator
discriminator_output_dim = 1 # The dimension of the output layer of the discriminator

random_sample_dim = 100 # The dimension of the random noise vector z
generator_hidden_dim = 128 # The dimension of the hidden layer of the generator
generator_output_dim = 784 # The dimension of the output layer of the generator


I am working on the file "Day-5/Day 5 - Vanilla GAN Solved.ipynb".
The first task is:
Intermezzo: sigmoid cross entropy with logits
We defined the loss of the model as the log of the probability, but we are not using a $log$ function or the model probablities anywhere?
Enter sigmoid cross entropy with logits: https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits

From the tensorflow documentation

Putting it all together
Can you write Python code for it?

X = tf.placeholder(tf.float32, name="input", shape=[None, image_dim])
Z = tf.placeholder(tf.float32, name="latent_sample", shape=[None, random_sample_dim])

G_sample, D_loss, G_loss = gan_model_loss(X, Z, d, g)

with tf.variable_scope('optim'):
    D_solver = tf.train.AdamOptimizer(name='discriminator').minimize(D_loss, var_list=d.theta)
    G_solver = tf.train.AdamOptimizer(name='generator').minimize(G_loss, var_list=g.theta)

saver = tf.train.Saver()

# Some runtime parameters predefined for you
minibatch_size = 128 # The size of the minibatch

num_epoch = 500 # For how many epochs do we run the training
plot_every_epochs = 5 # After this many epochs we will save & display samples of generated images
print_every_batches = 1000 # After this many minibatches we will print the losses

restore = False
checkpoint = 'fc_2layer_e100_2.170.ckpt'
model = 'gan'
model_save_folder = os.path.join('data', 'chkp', model)
print ("Model checkpoints will be saved to:", model_save_folder)
image_save_folder = os.path.join('data', 'model_output', model)
print ("Image samples will be saved to:", image_save_folder)

minibatch_counter = 0
epoch_counter = 0

d_losses = []
g_losses = []

with tf.device("/gpu:0"), tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    if restore:
        saver.restore(sess, os.path.join(model_save_folder, checkpoint))
        print("Restored model:", checkpoint, "from:", model_save_folder)

    while epoch_counter < num_epoch:

        new_epoch, X_mb = mnist.next_batch(minibatch_size)

        _, D_loss_curr = sess.run([D_solver, D_loss],
                                  feed_dict={
                                      X: X_mb,
                                      Z: sample_Z(minibatch_size, random_sample_dim)
                                    })

        _, G_loss_curr = sess.run([G_solver, G_loss],
                                  feed_dict={
                                      Z: sample_Z(minibatch_size,...


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
5.4 The basic network for the discriminator
Can you write Python code for it?

class Discriminator:
    """The discriminator network

    the discriminator network takes as input a vector x of dimension input_dim, and transforms it
    to a vector of size output_dim. The network has one hidden layer of size hidden_dim.

    You will define the following methods:

    __init__: initializes all variables by using tf.get_variable(...)
                and stores them to the class, as well a list in self.theta
    forward: defines the forward pass of the network - how do the variables
                interact with respect to the inputs
    """

    def __init__(self, input_dim, hidden_dim, output_dim):

        with tf.variable_scope("discriminator"):
            pass

    def forward(self, x):
        """The forward pass of the network -- here we will define the logic of how we combine
        the variables through multiplication and activation functions in order to get the
        output.

        Along with the probabilities, also return the unnormalized probabilities
        (the values in the output layer before being passed through the sigmoid function)
        """
        pass


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
5.6 Check the implementation of the classes
Can you write Python code for it?


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
Drawing samples from the latent space
Can you write Python code for it?

def sample_Z(m, n):
    pass

plt.imshow(sample_Z(16, 100), cmap='gray')


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
Goals:

Implement the model from "Generative Adversarial Networks" by Goodfellow et al. (1284 citations since 2014.)

Understand how the model learns to generate realistic images

In ~two hours.
5.1 Downloading the datasets and previewing data
Can you write Python code for it?

data_folder = 'data'; dataset = 'mnist'  # the folder in which the dataset is going to be stored

download_folder = util.download_mnist(data_folder, dataset)
images, labels = util.load_mnist(download_folder)

print("Folder:", download_folder)
print("Image shape:", images.shape) # greyscale, so the last dimension (color channel) = 1
print("Label shape:", labels.shape) # one-hot encoded

show_n_images = 25
sample_images, mode = util.get_sample_images(images, n=show_n_images)
mnist_sample = util.images_square_grid(sample_images, mode)
plt.imshow(mnist_sample, cmap='gray')

sample = images[3]*50 #
sample = sample.reshape((28, 28))
print(np.array2string(sample.astype(int), max_line_width=100, separator=',', precision=0))

plt.imshow(sample, cmap='gray')


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
Intermezzo: Xavier initialization of weights
Glorot, X., & Bengio, Y. (2010, March). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (pp. 249-256).
Implemented in tensorflow, as part of the standard library: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/xavier_initializer
1. Idea:

If the weights in a network are initialized to too small values, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.
If the weights in a network are initialized to too large, then the signal grows as it passes through each layer until it’s too massive to be useful.

2. Goal:

We need initial weight values that are just right for the signal not to explode or vanish during the forward pass

3. Math

Trivial

4. Solution

$v = \frac{2}{n_{in} + n_{out}}$

In the case of a Gaussian distribution, we set the variance to $v$.
In the case of a uniform distribution, we set the interval  to $\pm v$ (the default distr. in tensorflow is the uniform).
http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
5.5 Define the model parameters
We will take a brief break to set the values for the parameters of the model. Since we know the dataset we are working with, as well as the shape of the generator and discriminator networks, your task is to fill in the values of the following variables.
Can you write Python code for it?

image_dim = # The dimension of the input image vector to the discrminator
discriminator_hidden_dim = # The dimension of the hidden layer of the discriminator
discriminator_output_dim = # The dimension of the output layer of the discriminator

random_sample_dim =  # The dimension of the random noise vector z
generator_hidden_dim = # The dimension of the hidden layer of the generator
generator_output_dim = # The dimension of the output layer of the generator


I am working on the file "Day-5/Day 5 - Vanilla GAN.ipynb".
The first task is:
What are we going to do with the data?

We have $70000$ images of hand-written digits generated from some distribution $X \sim P_{real}$
We have $70000$ labels $y_i \in {0,..., 9}$ indicating which digit is written on the image $x_i$

Problem: Imagine that the number of images we have is not enough - a common issue in computer vision and machine learning.

We can pay experts to create new images
Expensive
Slow
Realiable
We can generate new images ourselves
Cheap
Fast
Unreliable?

Problem: Not every image that we generate is going to be perfect (or even close to perfect). Therefore, we need some method to determine which images are realistic.

We can pay experts to determine which images are good enough
Expensive
Slow
Reliable
We can train a model to determine which images are good enough
Cheap
Fast
Unreliable?

Formalization

$X \sim P_{real}$ : existing images of shape $s$
$Z \sim P_z$ : a $k$-dimensional random vector
$G(z; \theta_G): Z \to \hat{X}$ : the generator, a function that transforms the random vector $z$ into an image of shape $s$
$D(x, \theta_D): X \to (Real, Fake)$ : the discriminator a function that given an image of shape $s$ decides if the image is real or fake

Details
The existing images $X$ in our setup are images from the mnist dataset. We will arbitrarily decide that vectors $z$ will be sampled from a uniform distribution, and $G$ and $D$ will both be 'deep' neural networks.
For simplicity, and since we are using the mnist dataset, both $G$ and $D$ will be multi-layer perceptrons (and not deep convolutional networks) with one hidden layer. The generated images $G(z) \sim P_{fake}$ as well as real images $x \sim P_{real}$ will be passed on to the discriminator, which will classify them into $(Real, Fake)$.

Figure 1.  General adversarial network architecture

Discriminator
 The goal  of the discriminator is to successfully recognize which image is sampled from the true distribution, and which image is sampled...
Can you write Python code for it?

class Generator:
    """The generator network

    the generator network takes as input a vector z of dimension input_dim, and transforms it
    to a vector of size output_dim. The network has one hidden layer of size hidden_dim.

    We will define the following methods:

    __init__: initializes all variables by using tf.get_variable(...)
                and stores them to the class, as well a list in self.theta
    forward: defines the forward pass of the network - how do the variables
                interact with respect to the inputs
    """

    def __init__(self, input_dim, hidden_dim, output_dim):
        """Constructor for the generator network. In the constructor, we will
        just initialize all the variables in the network.

        Args:
            input_dim: The dimension of the input data vector (z).
            hidden_dim: The dimension of the hidden layer of the neural network (h)
            output_dim: The dimension of the output layer (equivalent to the size of the image)

        """

        with tf.variable_scope("generator"):
            pass

    def forward(self, z):
        """The forward pass of the network -- here we will define the logic of how we combine
        the variables through multiplication and activation functions in order to get the
        output.

        """
        pass


I am working on the file "Day01_ComputerBasics/notebooks/01 - Data Science.ipynb".
The first task is:
Read the full story:
http://lucafoschini.com/notebooks/Agile%20Data%20Science%20Meetup.slides.html#/
Source (Another feature of Notebooks: Make slides out of them!):
https://github.com/LucaFoschini/lucafoschini.github.io/blob/master/notebooks/Agile%20Data%20Science%20Meetup.ipynb
Getting started
This section focuses on the 'Hacking Skills' part of the Venn diagram -- we're going to get you up and running on Linux, using git and GitHub, and starting to work on Jupyter notebooks. This is all stuff which will be used for future sections in this bootcamp, so take notes!
First up: opening the terminal. Either search for a program named 'terminal' on your computer (more on that in the links below), or use 'Applications (top left corner of your screen) -> System Tools -> Terminal'
The terminal is a powerful and efficient replacement for the file explorer:
Can you write Python code for it?


I am working on the file "Day01_ComputerBasics/notebooks/03 - Version Control.ipynb".
The first task is:
If that hasn't convinced you, here are some other benefits:
http://stackoverflow.com/questions/1408450/why-should-i-use-version-control
Replace 'code' in the first answer with 'essay', 'thesis', 'homework' -- all stuff that a version control system such as git and GitHub can help you with!
Git for Scientists: A Tutorial (by John McDonnell)
http://nyuccl.org/pages/GitTutorial/
Go through the tutorial. You can either follow along from the terminal in the command line, or from within this very notebook using the %%bash magic:
Can you write Python code for it?


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Bubble sort implemented in  parakeet
Similar to Numba,  parakeet is a Python compiler that optimizes the runtime of numerical computations based on the NumPy data types, such as NumPy arrays.
The usage is also similar to Numba where we just have to put the jit decorator on top of the function we want to optimize.
Can you write Python code for it?

from parakeet import jit as para_jit
@para_jit
def parakeet_bubblesort(np_ary):
    """ The parakeet implementation of Bubblesort on NumPy arrays."""
    length = np_ary.shape[0]
    swapped = 1
    for i in xrange(0, length):
        if swapped:
            swapped = 0
            for ele in xrange(0, length-i-1):
                if np_ary[ele] > np_ary[ele + 1]:
                    temp = np_ary[ele + 1]
                    np_ary[ele + 1] = np_ary[ele]
                    np_ary[ele] = temp
                    swapped = 1
    return np_ary


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Bubble sort implemented in Cython
Maybe we can speed things up a little bit via Cython's C-extensions for Python. Cython is basically a hybrid between C and Python and can be pictured as compiled Python code with type declarations.
Since we are working in an IPython notebook here, we can make use of the very convenient IPython magic: It will take care of the conversion to C code, the compilation, and eventually the loading of the function.
Note that the static type declarations that we add via cdef are note required for Cython to work, but it will speed things up tremendously.
Can you write Python code for it?

%load_ext cythonmagic

%%cython
import numpy as np
cimport numpy as np
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef cython_bubblesort(inp_ary):
    """ The Cython implementation of Bubblesort with NumPy memoryview."""
    cdef unsigned long length, i, swapped, ele, temp
    cdef long[:] np_ary = inp_ary
    length = np_ary.shape[0]
    swapped = 1
    for i in xrange(0, length):
        if swapped:
            swapped = 0
            for ele in xrange(0, length-i-1):
                if np_ary[ele] > np_ary[ele + 1]:
                    temp = np_ary[ele + 1]
                    np_ary[ele + 1] = np_ary[ele]
                    np_ary[ele] = temp
                    swapped = 1
    return inp_ary


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Bubble sort implemented in Numba
Numba is using the LLVM compiler infrastructure for compiling Python code to machine code. Its strength is to work with NumPy arrays to speed-up the code. If you want to read more about Numba, please see refer to the original website and documentation.
Can you write Python code for it?

from numba import jit as numba_jit
@numba_jit
def numba_bubblesort(np_ary):
    """ The NumPy implementation of Bubblesort on NumPy arrays."""
    length = np_ary.shape[0]
    swapped = 1
    for i in xrange(0, length):
        if swapped:
            swapped = 0
            for ele in xrange(0, length-i-1):
                if np_ary[ele] > np_ary[ele + 1]:
                    temp = np_ary[ele + 1]
                    np_ary[ele + 1] = np_ary[ele]
                    np_ary[ele] = temp
                    swapped = 1
    return np_ary


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Results
Can you write Python code for it?

title = 'Performance of Bubblesort in Python, Cython, parakeet, and Numba'

labels = {'python_bubblesort':'(C)Python Bubblesort - Python lists',
          'python_bubblesort_ary':'(C)Python Bubblesort - NumPy arrays',
          'cython_bubblesort': 'Cython Bubblesort - NumPy arrays',
          'numba_bubblesort': 'Numba Bubblesort - NumPy arrays',
          'parakeet_bubblesort': 'parakeet Bubblesort - NumPy arrays'
          }

ranked_by_time = sorted([(time[1][-1],time[0]) for time in timings.items()], reverse=True)

print_sysinfo()
plot(timings, title, [l for t,l in ranked_by_time], labels, orders_n)
summary_table(ranked_by_time)


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Setting up the plots
Can you write Python code for it?

import platform
import multiprocessing
from cython import __version__ as cython__version__
from llvm import __version__ as llvm__version__
from numba import __version__ as numba__version__
from parakeet import __version__ as parakeet__version__

def print_sysinfo():

    print '\nPython version  :', platform.python_version()
    print 'compiler        :', platform.python_compiler()
    print 'Cython version  :', cython__version__
    print 'NumPy version   :', np.__version__
    print 'Numba version   :', numba__version__
    print 'llvm version    :', llvm__version__
    print 'parakeet version:', parakeet__version__

    print '\nsystem     :', platform.system()
    print 'release    :', platform.release()
    print 'machine    :', platform.machine()
    print 'processor  :', platform.processor()
    print 'CPU count  :', multiprocessing.cpu_count()
    print 'interpreter:', platform.architecture()[0]
    print '\n\n'

%matplotlib inline

import matplotlib.pyplot as plt

def plot(timings, title, ranked_labels, labels, orders_n):
    plt.rcParams.update({'font.size': 12})

    fig = plt.figure(figsize=(11,10))
    for lb in ranked_labels:
        plt.plot(orders_n, timings[lb], alpha=0.5, label=labels[lb],
                 marker='o', lw=3)
    plt.xlabel('sample size n (items in the list)')
    plt.ylabel('time per computation in milliseconds')
    plt.xlim([min(orders_n) / 10, max(orders_n)* 10])
    plt.legend(loc=2)
    plt.grid()
    plt.xscale('log')
    plt.yscale('log')
    plt.title(title)
    plt.show()

import prettytable

def summary_table(ranked_labels):
    fit_table = prettytable.PrettyTable(['n=%s' %orders_n[-1],
                                         'bubblesort function' ,
                                         'time in millisec.',
                                         'rel. performance gain'])
    fit_table.align['bubblesort function'] = 'l'
    for entry in ranked_labels:
        fit_table.add_row(['', labels[entry[1]], round(entry[0]*100, 3),
                          ...


I am working on the file "Day02_EverythingData/notebooks/05 - Theory and Practice.ipynb".
The first task is:

Timing
Can you write Python code for it?

import timeit
import copy
import numpy as np

funcs = ['python_bubblesort',
         'python_bubblesort_ary',
         'cython_bubblesort',
         'numba_bubblesort'
         ]

orders_n = [10**n for n in range(1, 6)]
timings = {f:[] for f in funcs}

for n in orders_n:
    l = [np.random.randint(n) for num in range(n)]
    for f in funcs:
        l_copy = copy.deepcopy(l)
        if f != 'python_bubblesort':
            l_copy = np.asarray(l_copy)
        timings[f].append(min(timeit.Timer('%s(l_copy)' %f,
                      'from __main__ import %s, l_copy' %f)
                              .repeat(repeat=3, number=10)))


I am working on the file "Day05_GraphAlgorithms1/notebooks/02 - Visualization.ipynb".
The first task is:
Now, we're going to display this graph in the notebook with d3.js. The first step is to bring this graph to Javascript. We choose here to export the graph in JSON. Note that d3.js generally expects each edge to be an object with a source and a target. Also, we specify which side each member has taken (club attribute).
Can you write Python code for it?

from networkx.readwrite import json_graph
data = json_graph.node_link_data(g)
with open('graph.json', 'w') as f:
    json.dump(data, f, indent=4)


I am working on the file "Day05_GraphAlgorithms1/notebooks/02 - Visualization.ipynb".
The first task is:
The last step is trickier. We write the Javascript code to load the graph from the JSON file, and display it with d3.js. Knowing the basics of d3.js is required here (see the documentation of d3.js). We also give detailled explanations in the code comments below. (http://d3js.org)
Can you write Python code for it?

%%javascript
// We load the d3.js library from the Web.
require.config({paths: {d3: "http://d3js.org/d3.v3.min"}});
require(["d3"], function(d3) {
    // The code in this block is executed when the
    // d3.js library has been loaded.

    // First, we specify the size of the canvas containing
    // the visualization (size of the <div> element).
    var width = 300,
        height = 300;

    // We create a color scale.
    var color = d3.scale.category10();

    // We create a force-directed dynamic graph layout.
    var force = d3.layout.force()
        .charge(-120)
        .linkDistance(30)
        .size([width, height]);

    // In the <div> element, we create a <svg> graphic
    // that will contain our interactive visualization.
    var svg = d3.select("#d3-example").select("svg")
    if (svg.empty()) {
        svg = d3.select("#d3-example").append("svg")
                    .attr("width", width)
                    .attr("height", height);
    }

    // We load the JSON file.
    d3.json("graph.json", function(error, graph) {
        // In this block, the file has been loaded
        // and the 'graph' object contains our graph.

        // We load the nodes and links in the force-directed
        // graph.
        force.nodes(graph.nodes)
            .links(graph.links)
            .start();

        // We create a <line> SVG element for each link
        // in the graph.
        var link = svg.selectAll(".link")
            .data(graph.links)
            .enter().append("line")
            .attr("class", "link");

        // We create a <circle> SVG element for each node
        // in the graph, and we specify a few attributes.
        var node = svg.selectAll(".node")
            .data(graph.nodes)
            .enter().append("circle")
            .attr("class", "node")
            .attr("r", 5)  // radius
            .style("fill", function(d) {
                // The node color depends on the club.
                return color(d.club);
            })
           ...


I am working on the file "Day05_GraphAlgorithms1/notebooks/02 - Visualization.ipynb".
The first task is:
The next step is to create an HTML object that will contain the visualization. Here, we create a <div> element in the notebook. We also specify a few CSS styles for nodes and links (also called edges).
Can you write Python code for it?


I am working on the file "Day05_GraphAlgorithms1/notebooks/02 - Visualization.ipynb".
The first task is:
We load a famous social graph published in 1977, called Zachary's Karate club graph. This graph represents the friendships between members of a Karate Club. The club's president and the instructor were involved in a dispute, resulting in a split-up of this group. Here, we simply display the graph with matplotlib (using networkx.draw()).
Can you write Python code for it?

g = nx.karate_club_graph()
plt.figure(figsize=(6,4));
nx.draw(g)


I am working on the file "Day06_GraphAlgorithms2/notebooks/Fun with graphs 1.ipynb".
The first task is:
<h1> Graph Traversal BFS, DFS<h1>
Can you write Python code for it?

T = nx.dfs_tree(rgraph1,0)
print "DFS Tree edges : ", T.edges()

T = nx.bfs_tree(rgraph1, 0)
print "BFS Tree edges : ", T.edges()


I am working on the file "Day07_Stats/notebooks/histograms_and_univariate_exploration.ipynb".
The first task is:
<img src="Normal_Distribution_PDF.svg">
Quantiles
Going back to the cat example, we might be interested in asking questions like how many cat hearts weight more than 10 grams, or how many weight less than 6 grams. A histogram can give you rough answers to these sorts of questions. We can also work directly with the data and find what are called quantiles.
Can you write Python code for it?

print 'Percent of heart rates greater than 10', (catsData.Hwt > 10).mean()


I am working on the file "Day07_Stats/notebooks/histograms_and_univariate_exploration.ipynb".
The first task is:
The histogram tells us a few things:

The heart weights range from 6 to 20 grams.
10 grams seems to be the most common heart rate, i.e. the mode of random variable.
The average or Expected value of the random variable also seems to be around 10 grams.
The histogram leads us to believe that there's a good chance the cat heart rates are normally distributed.

The Connections Between Histograms and PDFs
We can generate values (or samples) from the normal distribution with a specific mean and variance using numpy. Here we generate 500 samples from a normal distribution with $\mu=-2$ and $\sigma^2=0.5$. We store these samples as a vector in a variable called normal_samples. We then plot a histogram of these samples.
Can you write Python code for it?

import numpy as np
normal_samples = np.random.normal(loc=-2, scale=0.5, size=500)

%matplotlib inline
plt.hist(normal_samples)


I am working on the file "Day1.ipynb".
The first task is:
Demo for extracting the quantities and vaccines names for a given county_city combo
Can you write Python code for it?

na = np.array([vaccinuri[jud_or=="Prahova_Busteni"], cantitati[jud_or=="Prahova_Busteni"]]).T

print(na)

print (np.sum(na[:,-1]))


I am working on the file "Day1.ipynb".
The first task is:
Identify the most common/popular/abundant vaccine
Can you write Python code for it?

max_vac = np.max(na[:,-1])
vaccine_name = na[na[:,-1]==max_vac][0][0]
print(vaccine_name)


I am working on the file "Day1.ipynb".
The first task is:
Now for the least common/popular/abundant vaccine
Can you write Python code for it?

min_vac = np.min(na[:,-1])
vaccine_name = na[na[:,-1]==min_vac][0][0]
print(vaccine_name)


I am working on the file "Day1.ipynb".
The first task is:
Prepare filter columns that allow us to select any county/town combo, or county/town/vaccine name combo
Can you write Python code for it?

jud_or = judete + "_"+ orase
jud_or_vac = judete + "_"+ orase + "_"+ vaccinuri

print(np.sum(cantitati[jud_or=="Prahova_Busteni"]))
print(np.sum(cantitati[jud_or_vac=="Bucuresti_Sector 2_Hexacima"]))


I am working on the file "Day1.ipynb".
The first task is:
Test printing a filter
Can you write Python code for it?


I am working on the file "Day1/03_machine_learning_intro.ipynb".
The first task is:
Each value we are predicting is the response (also known as: target, outcome, label, dependent variable)
Classification is supervised learning in which the response is categorical
Regression is supervised learning in which the response is ordered and continuous

Requirements for working with data in scikit-learn

Features and response are separate objects
Features and response should be numeric
Features and response should be NumPy arrays
Features and response should have specific shapes
Can you write Python code for it?

# check the types of the features and response
print type(iris.data)
print type(iris.target)

# check the shape of the features (first dimension = number of observations, second dimensions = number of features)
print iris.data.shape

# check the shape of the response (single dimension matching the number of observations)
print iris.target.shape

# store feature matrix in "X"
X = iris.data

# store response vector in "y"
y = iris.target


I am working on the file "Day1/03_machine_learning_intro.ipynb".
The first task is:
Machine learning terminology

Each row is an observation (also known as: sample, example, instance, record)
Each column is a feature (also known as: predictor, attribute, independent variable, input, regressor, covariate)
Can you write Python code for it?

# print the names of the four features
print iris.feature_names

# print integers representing the species of each observation
print iris.target

# print the encoding scheme for species: 0 = setosa, 1 = versicolor, 2 = virginica
print iris.target_names


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
Returns a NumPy array
Can predict for multiple observations at once
Can you write Python code for it?


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
Step 2: "Instantiate" the "estimator"

"Estimator" is scikit-learn's term for model
"Instantiate" means "make an instance of"
Can you write Python code for it?


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
Step 4: Predict the response for a new observation

New observations are called "out-of-sample" data
Uses the information it learned during the model training process
Can you write Python code for it?


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
Using a different classification model
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response for new observations
logreg.predict(X_new)


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
Using a different value for K
Can you write Python code for it?

# instantiate the model (using the value K=5)
knn = KNeighborsClassifier(n_neighbors=5)

# fit the model with data
knn.fit(X, y)

# predict the response for new observations
knn.predict(X_new)


I am working on the file "Day1/04_model_training.ipynb".
The first task is:
scikit-learn 4-step modeling pattern
Step 1: Import the class you plan to use
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier


I am working on the file "Day1/05_model_evaluation_ta.ipynb".
The first task is:
Generating an Optimal KNN classifier
Look back at 04_model_training and see how high an accuracy you can achieve for different values of n_neighbors.  Try to understand why different values do better than others in terms of the pictures we saw in 04_model_training.
You can change feature1 and feature2 in the cell below to visualize different projections of the data.
Can you write Python code for it?

feature1 = 1 # feature on x axis
feature2 = 3 # feature on y axis

data = X
f1vals = X[:,feature1]
f2vals = X[:,feature2]

import numpy as np
targets = dict(zip(range(3), iris.target_names))
features = dict(zip(range(4), iris.feature_names))
%matplotlib inline
import matplotlib.pyplot as plt
colors = ['g', 'r', 'b']
fig = plt.figure(figsize=(8,8))
ax = plt.subplot()
for species in targets.keys():
    f1 = f1vals[np.where(y==species)]
    f2 = f2vals[np.where(y==species)]
    ax.scatter(f1, f2, c=colors[species], label=targets[species], s=40)
    ax.set(xlabel=features[feature1], ylabel=features[feature2])
    ax.legend()


I am working on the file "Day1/05_model_evaluation_ta.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response values for the observations in X
y_pred = logreg.predict(X)
print(y_pred)
print("{0} predictions".format(len(y_pred)))


I am working on the file "Day1/05_model_evaluation_tts.ipynb".
The first task is:
Can you find an even better value for K?
Can you write Python code for it?

# try K=1 through K=25 and record testing accuracy
k_range = range(1, 26)
scores = [] # calculate accuracies for each value of K!

#Now we plot:

import matplotlib.pyplot as plt
# allow plots to appear within the notebook
%matplotlib inline

plt.plot(k_range, scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Testing Accuracy')


I am working on the file "Day1/05_model_evaluation_tts.ipynb".
The first task is:
Repeat for KNN with K=1:
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Day1/05_model_evaluation_tts.ipynb".
The first task is:
Repeat for KNN with K=5:
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Day1/05_model_evaluation_tts.ipynb".
The first task is:
What did this accomplish?

Model can be trained and tested on different data
Response values are known for the testing set, and thus predictions can be evaluated
Testing accuracy is a better estimate than training accuracy of out-of-sample performance
Can you write Python code for it?

# STEP 1: split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)

# print the shapes of the new X objects
print X_train.shape
print X_test.shape

# print the shapes of the new y objects
print y_train.shape
print y_test.shape

from sklearn.linear_model import LogisticRegression
# STEP 2: train the model on the training set
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# STEP 3: make predictions on the testing set
y_pred = logreg.predict(X_test)

from sklearn import metrics
# compare actual response values (y_test) with predicted response values (y_pred)
print metrics.accuracy_score(y_test, y_pred)


I am working on the file "Day1/06_cross_validation.ipynb".
The first task is:
Improvements to cross-validation
Repeated cross-validation

Repeat cross-validation multiple times (with different random splits of the data) and average the results
More reliable estimate of out-of-sample performance by reducing the variance associated with a single trial of cross-validation

Creating a hold-out set

"Hold out" a portion of the data before beginning the model building process
Locate the best model using cross-validation on the remaining data, and test it using the hold-out set
More reliable estimate of out-of-sample performance since hold-out set is truly out-of-sample

Splitting a Second Time? Train/Cross-Validation/Test Split
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(k_range, k_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')


I am working on the file "Day1/06_cross_validation.ipynb".
The first task is:
Question: What if we created a bunch of train/test splits, calculated the testing accuracy for each, and averaged the results together?
Answer: That's the essense of K-fold cross-validation!
Steps for K-fold cross-validation

Split the dataset into K equal partitions (or "folds").
Use fold 1 as the testing set and the union of the other folds as the training set.
Calculate testing accuracy.
Repeat steps 2 and 3 K times, using a different fold as the testing set each time.
Use the average testing accuracy as the estimate of out-of-sample accuracy.

Diagram of 5-fold cross-validation:
Can you write Python code for it?

# simulate splitting a dataset of 25 observations into 5 folds
from sklearn.cross_validation import KFold
kf = KFold(25, n_folds=5, shuffle=False)

# print the contents of each training and testing set
print '{} {:^61} {}'.format('Iteration', 'Training set observations', 'Testing set observations')
for iteration, data in enumerate(kf, start=1):
    print '{:^9} {} {:^25}'.format(iteration, data[0], data[1])


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Can we do better than this? Let us add in non linear features
Can you write Python code for it?

#since we are interested in the Volume of trees
#it's nature to add in the square of Girth into our features

#add in a new feature
X["GirthSquare"]=trees["Girth"]**2.

feature_cols=["Girth", "Height","GirthSquare"]

# fit with LinearRegression
lm=LinearRegression()
lm.fit(X,Y)
#print out result
zip(feature_cols, lm.coef_)

Ypredict=lm.predict(X)
#print "MSE",np.sqrt(metrics.mean_squared_error(Y, Ypredict))
from matplotlib import pyplot
pyplot.plot(X["Girth"],Ypredict)
pyplot.scatter(X["Girth"],Y)

#We can keep on trying even higher order non liearn features
X["GirthCube"]=trees["Girth"]**3.
X["GirthFouth"]=trees["Girth"]**4.
print X.shape
feature_cols=["Girth", "Height","GirthSquare","GirthCube","GirthFouth"]
# fit with LinearRegression
lm=LinearRegression()
lm.fit(X,Y)
#print out result
zip(feature_cols, lm.coef_)

Ypredict=lm.predict(X)
#print "MSE",np.sqrt(metrics.mean_squared_error(Y, Ypredict))
#print type(X)
from matplotlib import pyplot
pyplot.plot(X["Girth"],Ypredict)
pyplot.scatter(X["Girth"],Y)


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Example: Advertising Data
Let's take a look at some data, ask some questions about that data, and then use linear regression to answer those questions!
Can you write Python code for it?

# read data into a DataFrame
data = pd.read_csv('data/Advertising.csv', index_col=0)
data.head()


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
How do we interpret the Size_large coefficient? For a given amount of TV/Radio/Newspaper ad spending, being a large market is associated with an average increase in Sales of 57.42 widgets (as compared to a small market, which is called the baseline level).
What if we had reversed the 0/1 coding and created the feature 'Size_small' instead? The coefficient would be the same, except it would be negative instead of positive. As such, your choice of category for the baseline does not matter, all that changes is your interpretation of the coefficient.
Handling Categorical Features with More than Two Categories
Let's create a new feature called Area, and randomly assign observations to be rural, suburban, or urban:
Can you write Python code for it?

# set a seed for reproducibility
np.random.seed(123456)

# assign roughly one third of observations to each group
nums = np.random.rand(len(data))
mask_suburban = (nums > 0.33) & (nums < 0.66)
mask_urban = nums > 0.66
data['Area'] = 'rural'
data.loc[mask_suburban, 'Area'] = 'suburban'
data.loc[mask_urban, 'Area'] = 'urban'
data.head()


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
How do we interpret these coefficients? For a given amount of Radio and Newspaper ad spending, an increase of $1000 in TV ad spending is associated with an increase in Sales of 45.765 widgets.
A lot of the information we have been reviewing piece-by-piece is available in the Statsmodels model summary output:
Feature Selection
How do I decide which features to include in a linear model?
-the answer will be in the next session.
Handling Categorical Features with Two Categories
Up to now, all of our features have been numeric. What if one of our features was categorical?
Let's create a new feature called Size, and randomly assign observations to be small or large:
Can you write Python code for it?

# set a seed for reproducibility
np.random.seed(12345)

# create a Series of booleans in which roughly half are True
nums = np.random.rand(len(data))
mask_large = nums > 0.5

# initially set Size to small, then change roughly half to be large
data['Size'] = 'small'
data.loc[mask_large, 'Size'] = 'large'
data.head()


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Interpreting Model Coefficients
How do we interpret the TV coefficient ($\beta_1$)?
- A "unit" increase in TV ad spending is associated with a 0.047537 "unit" increase in Sales.
- Or more clearly: An additional $1,000 spent on TV ads is associated with an increase in sales of 47.537 widgets.
Note that if an increase in TV ad spending was associated with a decrease in sales, $\beta_1$ would be negative.
Using the Model for Prediction
Let's say that there was a new market where the TV advertising spend was $50,000. What would we predict for the Sales in that market?
$$y = \beta_0 + \beta_1x$$
$$y = 7.032594 + 0.047537 \times 50$$
Can you write Python code for it?

# manually calculate the prediction
7.032594 + 0.047537*50

### SCIKIT-LEARN ###

# predict for a new observation
lm.predict(50)


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Let's examine the result of the fitting.
Can you write Python code for it?

Ypredict=lm.predict(X)
print "MSE",np.sqrt(metrics.mean_squared_error(Y, Ypredict))
#print type(X)
from matplotlib import pyplot
pyplot.plot(X["Girth"],Ypredict)
pyplot.scatter(X["Girth"],Y)


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Let's redo the multiple linear regression and include the Size_large feature:
Can you write Python code for it?

# create X and y
feature_cols = ['TV', 'Radio', 'Newspaper', 'Size_large']
X = data[feature_cols]
y = data.Sales

# instantiate, fit
lm = LinearRegression()
lm.fit(X, y)

# print coefficients
zip(feature_cols, lm.coef_)


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
Questions About the Advertising Data
Let's pretend you work for the company that manufactures and markets this widget. The company might ask you the following: On the basis of this data, how should we spend our advertising money in the future?
This general question might lead you to more specific questions:
1. Is there a relationship between ads and sales?
2. How strong is that relationship?
3. Which ad types contribute to sales?
4. What is the effect of each ad type of sales?
5. Given ad spending in a particular market, can sales be predicted?
We will explore these questions below!
Simple Linear Regression
Simple linear regression is an approach for predicting a quantitative response using a single feature (or "predictor" or "input variable"). It takes the following form:
$y = \beta_0 + \beta_1x$
What does each term represent?
- $y$ is the response
- $x$ is the feature
- $\beta_0$ is the intercept
- $\beta_1$ is the coefficient for x
Together, $\beta_0$ and $\beta_1$ are called the model coefficients. To create your model, you must "learn" the values of these coefficients. And once we've learned these coefficients, we can use the model to predict Sales!
Estimating ("Learning") Model Coefficients
Generally speaking, coefficients are estimated using the least squares criterion, which means we are find the line (mathematically) which minimizes the sum of squared residuals (or "sum of squared errors"):

What elements are present in the diagram?
- The black dots are the observed values of x and y.
- The blue line is our least squares line.
- The red lines are the residuals, which are the distances between the observed values and the least squares line.
How do the model coefficients relate to the least squares line?
- $\beta_0$ is the intercept (the value of $y$ when $x$=0)
- $\beta_1$ is the slope (the change in $y$ divided by change in $x$)
Here is a graphical depiction of those calculations:

Let's estimate the model coefficients for the advertising data:
Can you write Python code for it?

### SCIKIT-LEARN ###

# create X and y
feature_cols = ['TV']
X = data[feature_cols]
y = data.Sales

# instantiate and fit
lm = LinearRegression()
lm.fit(X, y)

# print the coefficients
print lm.intercept_
print lm.coef_


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
The dataset trees have two features Girth and Height. we want to use them to predict the Volume of the trees.
Can you write Python code for it?

#set up features and aimed result
feature_cols=["Girth", "Height"]
X=trees[feature_cols]
Y=trees.Volume
# fit with LinearRegression
lm=LinearRegression()
lm.fit(X,Y)
#print out result
zip(feature_cols, lm.coef_)


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
We have to represent Area numerically, but we can't simply code it as 0=rural, 1=suburban, 2=urban because that would imply an ordered relationship between suburban and urban, and thus urban is somehow "twice" the suburban category. Note that if you do have ordered categories (i.e., strongly disagree, disagree, neutral, agree, strongly agree), you can use a single dummy variable and represent the categories numerically (such as 1, 2, 3, 4, 5).
Anyway, our Area feature is unordered, so we have to create additional dummy variables. Let's explore how to do this using pandas:
Can you write Python code for it?

# create three dummy variables using get_dummies
x = pd.get_dummies(data.Area, prefix='Area')

x.tail()

data = pd.concat([data, x], axis=1)

data.tail()


I am working on the file "Day1/07_linear_regression.ipynb".
The first task is:
What are the features?
- TV: advertising dollars spent on TV for a single product in a given market (in thousands of dollars)
- Radio: advertising dollars spent on Radio
- Newspaper: advertising dollars spent on Newspaper
What is the response?
- Sales: sales of a single product in a given market (in thousands of widgets)
Can you write Python code for it?

# print the shape of the DataFrame
data.shape


I am working on the file "Day1/08_featureselection_cv.ipynb".
The first task is:
MSE is more popular than MAE because MSE "punishes" larger errors. But, RMSE is even more popular than MSE because RMSE is interpretable in the "y" units.
Can you write Python code for it?

# The MSE scores can be calculated by:
scores = cross_val_score(lm, X, y, cv=10, scoring='mean_squared_error')
print scores

# fix the sign of MSE scores
mse_scores = -scores
print mse_scores

# convert from MSE to RMSE
rmse_scores = np.sqrt(mse_scores)
print rmse_scores

# calculate the average RMSE
print rmse_scores.mean()

# 10-fold cross-validation with two features (excluding Newspaper)
feature_cols = ['TV', 'Radio']
X = data[feature_cols]
print np.sqrt(-cross_val_score(lm, X, y, cv=10, scoring='mean_squared_error')).mean()


I am working on the file "Day1/Lab_1_Pandas.ipynb".
The first task is:
Conversión data frame para trabajar en sklearn
Uno de los probemas que se pueden presentar cuando se trabajar con bases de datos heterogéneas es cómo tratar las variables categorícas que tiene más de dos niveles.
A menudo, muchas características son categóricas:
["from Europe", "from US", "from Asia"], ["uses Firefox", "uses Chrome", "uses Safari", "uses Internet Explorer"].
EStas variables se pueden procesar utilizando un esquema de conversión llamado one-hot encoding. De esta forma, se pasará a tener tantas variables como diferentes niveles tuviese la variable categórica. Cada nueva variable (llamada dummy variable) será binaria, 1 o 0, indicando si para ese caso en particular se cumple o no la condición indicada por el nivel de la variable categórica.
Así, para el caso de una variable categórica que indicase el navegador utilizado se tendrían cuatro nuevas dummy variables.
* uses Firefox: [1,0], donde 1 indica que se utiliza y cero en caso contrario.
* uses Chrome: [1,0]
* uses Safari: [1,0]
* uses IE: [1,0]
Este tipo de codificación es la más recomendable para el caso de sklearn, que no trabaja con variables categóricas.
Desde pandas se pueden crear este tipo de variables utilizando el método get_dummies, o bien desde sklearn utilizando OneHotEncoder
Ejercicio 7
En este ejercicio se pide convertir la variable sexo en dos variable dummies
Can you write Python code for it?

diabetes_dummy = pd.get_dummies(diabetes)
diabetes_dummy


I am working on the file "Day1/Lab_1_Pandas.ipynb".
The first task is:
Ejercicio 3
A continuación, una buena práctica es intentar resumir toda la información posible de los datos. Habitualmente nos interesa saber la media y desviación estándar, posiblemente quartiles de cada una de las variables. Esto nos permitirá, por una lado, tener una idea de cómo son las ditribuciones de cada una de las variables y por otra, nos permitirá verificar si existen datos anómalos (outliers).
Existe en pandas una función que permite describir todos los datos de forma simultánea. Busque en la documentación la función y haga la llamada correspondiente.
Pregunta Identifique el valor media de la variable S5.
Pregunta Qué encuentra extraño en el resumen de la variable sex
Can you write Python code for it?

#resumen de los datos
diabetes.describe()