I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Now you can use these layers to implement one_step_attention(). In order to propagate a Keras tensor object X through one of these layers, use layer(X) (or layer([X,Y]) if it requires multiple inputs.), e.g. densor(X) will propagate X through the Dense(1) layer defined above.
Can you write Python code for it?

# GRADED FUNCTION: one_step_attention

def one_step_attention(a, s_prev):
    """
    Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights
    "alphas" and the hidden states "a" of the Bi-LSTM.

    Arguments:
    a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)
    s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)

    Returns:
    context -- context vector, input of the next (post-attetion) LSTM cell
    """

    ### START CODE HERE ###
    # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states "a" (≈ 1 line)
    s_prev = repeator(s_prev)
    # Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)
    concat = concatenator([a, s_prev])
    # Use densor1 to propagate concat through a small fully-connected neural network to compute the "intermediate energies" variable e. (≈1 lines)
    e = densor1(concat)
    # Use densor2 to propagate e through a small fully-connected neural network to compute the "energies" variable energies. (≈1 lines)
    energies = densor2(e)
    # Use "activator" on "energies" to compute the attention weights "alphas" (≈ 1 line)
    alphas = activator(energies)
    # Use dotor together with "alphas" and "a" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)
    context = dotor([alphas,a])
    ### END CODE HERE ###

    return context


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Run the following cell to create your model.
Can you write Python code for it?

model = model(Tx, Ty, n_a, n_s, len(human_vocab), len(machine_vocab))


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:

Thus, dense_2_acc_8: 0.89 means that you are predicting the 7th character of the output correctly 89% of the time in the current batch of data.

We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)
Can you write Python code for it?


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character.
3 - Visualizing Attention (Optional / Ungraded)
Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can  visualize what part of the output is looking at what part of the input.
Consider the task of translating "Saturday 9 May 2018" to "2018-05-09". If we visualize the computed $\alpha^{\langle t, t' \rangle}$ we get this:

 Figure 8: Full Attention Map

Notice how the output ignores the "Saturday" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's "18" in order to generate "2018."
3.1 - Getting the activations from the network
Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\alpha^{\langle t, t' \rangle}$.
To figure out where the attention values are located, let's start by printing a summary of the model .
Can you write Python code for it?


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You can now see the results on new examples.
Can you write Python code for it?

EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']
for example in EXAMPLES:

    source = string_to_int(example, Tx, human_vocab)
    source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1)
    prediction = model.predict([source, s0, c0])
    prediction = np.argmax(prediction, axis = -1)
    output = [inv_machine_vocab[int(i)] for i in prediction]

    print("source:", example)
    print("output:", ''.join(output))


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You now have:
- X: a processed version of the human readable dates in the training set, where each character is replaced by an index mapped to the character via human_vocab. Each date is further padded to $T_x$ values with a special character (< pad >). X.shape = (m, Tx)
- Y: a processed version of the machine readable dates in the training set, where each character is replaced by the index it is mapped to in machine_vocab. You should have Y.shape = (m, Ty).
- Xoh: one-hot version of X, the "1" entry's index is mapped to the character thanks to human_vocab. Xoh.shape = (m, Tx, len(human_vocab))
- Yoh: one-hot version of Y, the "1" entry's index is mapped to the character thanks to machine_vocab. Yoh.shape = (m, Tx, len(machine_vocab)). Here, len(machine_vocab) = 11 since there are 11 characters ('-' as well as 0-9).
Lets also look at some examples of preprocessed training examples. Feel free to play with index in the cell below to navigate the dataset and see how source/target dates are preprocessed.
Can you write Python code for it?

index = 0
print("Source date:", dataset[index][0])
print("Target date:", dataset[index][1])
print()
print("Source after preprocessing (indices):", X[index])
print("Target after preprocessing (indices):", Y[index])
print()
print("Source after preprocessing (one-hot):", Xoh[index])
print("Target after preprocessing (one-hot):", Yoh[index])


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You will be able to check the expected output of one_step_attention() after you've coded the model() function.
Exercise: Implement model() as explained in figure 2 and the text above. Again, we have defined global layers that will share weights to be used in model().
Can you write Python code for it?

n_a = 32
n_s = 64
post_activation_LSTM_cell = LSTM(n_s, return_state = True)
output_layer = Dense(len(machine_vocab), activation=softmax)


I am working on the file "Sequence Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You've loaded:
- dataset: a list of tuples of (human readable date, machine readable date)
- human_vocab: a python dictionary mapping all characters used in the human readable dates to an integer-valued index
- machine_vocab: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent with human_vocab.
- inv_machine_vocab: the inverse dictionary of machine_vocab, mapping from indices back to characters.
Let's preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since "YYYY-MM-DD" is 10 characters long).
Can you write Python code for it?

Tx = 30
Ty = 10
X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)

print("X.shape:", X.shape)
print("Y.shape:", Y.shape)
print("Xoh.shape:", Xoh.shape)
print("Yoh.shape:", Yoh.shape)


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
Do you notice anything surprising? It is astonishing how these results reflect certain unhealthy gender stereotypes. For example, "computer" is closer to "man" while "literature" is closer to "woman". Ouch!
We'll see below how to reduce the bias of these vectors, using an algorithm due to Boliukbasi et al., 2016. Note that some word pairs such as "actor"/"actress" or "grandmother"/"grandfather" should remain gender specific, while other words such as "receptionist" or "technology" should be neutralized, i.e. not be gender-related. You will have to treat these two type of words differently when debiasing.
3.1 - Neutralize bias for non-gender specific words
The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which we'll call $g_{\perp}$. In linear algebra, we say that the 49 dimensional $g_{\perp}$ is perpendicular (or "othogonal") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$.
Even though $g_{\perp}$ is 49 dimensional, given the limitations of what we can draw on a screen, we illustrate it using a 1 dimensional axis below.

 Figure 2: The word vector for "receptionist" represented before and after applying the neutralize operation.

Exercise: Implement neutralize() to remove the bias of words such as "receptionist" or "scientist". Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$:
$$e^{bias_component} = \frac{e*g}{||g||_2^2} * g\tag{2}$$
$$e^{debiased} = e - e^{bias_component}\tag{3}$$
If you are an expert in linear algebra, you may recognize $e^{bias_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert in...
Can you write Python code for it?

def neutralize(word, g, word_to_vec_map):
    """
    Removes the bias of "word" by projecting it on the space orthogonal to the bias axis.
    This function ensures that gender neutral words are zero in the gender subspace.

    Arguments:
        word -- string indicating the word to debias
        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)
        word_to_vec_map -- dictionary mapping words to their corresponding vectors.

    Returns:
        e_debiased -- neutralized word vector representation of the input "word"
    """

    ### START CODE HERE ###
    # Select word vector representation of "word". Use word_to_vec_map. (≈ 1 line)
    e = word_to_vec_map[word]

    # Compute e_biascomponent using the formula give above. (≈ 1 line)
    e_biascomponent = (e*g)/ np.sqrt(np.sum(g**2))**2

    # Neutralize e by substracting e_biascomponent from it
    # e_debiased should be equal to its orthogonal projection. (≈ 1 line)
    e_debiased = e-e_biascomponent
    ### END CODE HERE ###

    return e_debiased

e = "receptionist"
print("cosine similarity between " + e + " and g, before neutralizing: ", cosine_similarity(word_to_vec_map["receptionist"], g))

e_debiased = neutralize("receptionist", g, word_to_vec_map)
print("cosine similarity between " + e + " and g, after neutralizing: ", cosine_similarity(e_debiased, g))


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
Expected Output:

            **cosine_similarity(father, mother)** =

         0.890903844289

            **cosine_similarity(ball, crocodile)** =

         0.274392462614

            **cosine_similarity(france - paris, rome - italy)** =

         -0.675147930817

After you get the correct expected output, please feel free to modify the inputs and measure the cosine similarity between other pairs of words! Playing around the cosine similarity of other inputs will give you a better sense of how word vectors behave.
2 - Word analogy task
In the word analogy task, we complete the sentence "a is to b as c is to ____". An example is  'man is to woman as king is to queen' . In detail, we are trying to find a word d, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner: $e_b - e_a \approx e_d - e_c$. We will measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity.
Exercise: Complete the code below to be able to perform word analogies!
Can you write Python code for it?

# GRADED FUNCTION: complete_analogy

def complete_analogy(word_a, word_b, word_c, word_to_vec_map):
    """
    Performs the word analogy task as explained above: a is to b as c is to ____.

    Arguments:
    word_a -- a word, string
    word_b -- a word, string
    word_c -- a word, string
    word_to_vec_map -- dictionary that maps words to their corresponding vectors.

    Returns:
    best_word --  the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity
    """

    # convert words to lower case
    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()

    ### START CODE HERE ###
    # Get the word embeddings v_a, v_b and v_c (≈1-3 lines)
    e_a, e_b, e_c = word_to_vec_map[word_a],  word_to_vec_map[word_b],  word_to_vec_map[word_c]
    ### END CODE HERE ###

    words = word_to_vec_map.keys()
    max_cosine_sim = -100              # Initialize max_cosine_sim to a large negative number
    best_word = None                   # Initialize best_word with None, it will help keep track of the word to output

    # loop over the whole word vector set
    for w in words:
        # to avoid best_word being one of the input words, pass on them.
        if w in [word_a, word_b, word_c] :
            continue

        ### START CODE HERE ###
        # Compute cosine similarity between the combined_vector and the current word (≈1 line)
        cosine_sim = cosine_similarity(e_b-e_a, word_to_vec_map[w]-e_c)

        # If the cosine_sim is more than the max_cosine_sim seen so far,
            # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)
        if cosine_sim > max_cosine_sim:
            max_cosine_sim = cosine_sim
            best_word = w
        ### END CODE HERE ###

    return best_word


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
Expected Output: The second result is essentially 0, up to numerical roundof (on the order of $10^{-17}$).

            **cosine similarity between receptionist and g, before neutralizing:** :

         0.330779417506

            **cosine similarity between receptionist and g, after neutralizing:** :

         -3.26732746085e-17

3.2 - Equalization algorithm for gender-specific words
Next, lets see how debiasing can also be applied to word pairs such as "actress" and "actor." Equalization is applied to pairs of words that you might want to have differ only through the gender property. As a concrete example, suppose that "actress" is closer to "babysit" than "actor." By applying neutralizing to "babysit" we can reduce the gender-stereotype associated with babysitting. But this still does not guarantee that "actor" and "actress" are equidistant from "babysit." The equalization algorithm takes care of this.
The key idea behind equalization is to make sure that a particular pair of words are equi-distant from the 49-dimensional $g_\perp$. The equalization step also ensures that the two equalized steps are now the same distance from $e_{receptionist}^{debiased}$, or from any other work that has been neutralized. In pictures, this is how equalization works:

The derivation of the linear algebra to do this is a bit more complex. (See Bolukbasi et al., 2016 for details.) But the key equations are:
$$ \mu = \frac{e_{w1} + e_{w2}}{2}\tag{4}$$
$$ \mu_{B} = \frac {\mu *  \text{bias_axis}}{||\text{bias_axis}||_2} + ||\text{bias_axis}||_2 *\text{bias_axis}
\tag{5}$$
$$\mu_{\perp} = \mu - \mu_{B} \tag{6}$$
$$e_{w1B} = \sqrt{ |{1 - ||\mu_{\perp} ||^2_2} |} * \frac{(e_{\text{w1}} - \mu_{\perp}) - \mu_B} {|(e_{w1} - \mu_{\perp}) - \mu_B)|} \tag{7}$$
$$e_{w2B} = \sqrt{ |{1 - ||\mu_{\perp} ||^2_2} |} *...
Can you write Python code for it?

def equalize(pair, bias_axis, word_to_vec_map):
    """
    Debias gender specific words by following the equalize method described in the figure above.

    Arguments:
    pair -- pair of strings of gender specific words to debias, e.g. ("actress", "actor")
    bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender
    word_to_vec_map -- dictionary mapping words to their corresponding vectors

    Returns
    e_1 -- word vector corresponding to the first word
    e_2 -- word vector corresponding to the second word
    """

    ### START CODE HERE ###
    # Step 1: Select word vector representation of "word". Use word_to_vec_map. (≈ 2 lines)
    w1, w2 = None
    e_w1, e_w2 = None

    # Step 2: Compute the mean of e_w1 and e_w2 (≈ 1 line)
    mu = None

    # Step 3: Compute the projections of mu over the bias axis and the orthogonal axis (≈ 2 lines)
    mu_B = None
    mu_orth = None

    # Step 4: Set e1_orth and e2_orth to be equal to mu_orth (≈2 lines)
    e1_orth = None
    e2_orth = None

    # Step 5: Adjust the Bias part of u1 and u2 using the formulas given in the figure above (≈2 lines)
    e_w1B = None
    e_w2B = None

    # Step 6: Debias by equalizing u1 and u2 to the sum of their projections (≈2 lines)
    e1 = None
    e2 = None
    ### END CODE HERE ###

    return e1, e2

print("cosine similarities before equalizing:")
print("cosine_similarity(word_to_vec_map[\"man\"], gender) = ", cosine_similarity(word_to_vec_map["man"], g))
print("cosine_similarity(word_to_vec_map[\"woman\"], gender) = ", cosine_similarity(word_to_vec_map["woman"], g))
print()
e1, e2 = equalize(("man", "woman"), g, word_to_vec_map)
print("cosine similarities after equalizing:")
print("cosine_similarity(e1, gender) = ", cosine_similarity(e1, g))
print("cosine_similarity(e2, gender) = ", cosine_similarity(e2, g))


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
Now, you will consider the cosine similarity of different words with $g$. Consider what a positive value of similarity means vs a negative cosine similarity.
Can you write Python code for it?

print ('List of names and their similarities with constructed vector:')

# girls and boys name
name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']

for w in name_list:
    print (w, cosine_similarity(word_to_vec_map[w], g))


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
Run the cell below to test your code, this may take 1-2 minutes.
Can you write Python code for it?

triads_to_try = [('italy', 'italian', 'spain'), ('india', 'delhi', 'japan'), ('man', 'woman', 'boy'), ('small', 'smaller', 'large')]
for triad in triads_to_try:
    print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad,word_to_vec_map)))


I am working on the file "Sequence Models/Operations+on+word+vectors+-+v1.ipynb".
The first task is:
You've loaded:
- words: set of words in the vocabulary.
- word_to_vec_map: dictionary mapping words to their GloVe vector representation.
You've seen that one-hot vectors do not do a good job cpaturing what words are similar. GloVe vectors provide much more useful information about the meaning of individual words. Lets now see how you can use GloVe vectors to decide how similar two words are.
1 - Cosine similarity
To measure how similar two words are, we need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows:
$$\text{CosineSimilarity(u, v)} = \frac {u . v} {||u||_2 ||v||_2} = cos(\theta) \tag{1}$$
where $u.v$ is the dot product (or inner product) of two vectors, $||u||_2$ is the norm (or length) of the vector $u$, and $\theta$ is the angle between $u$ and $v$. This similarity depends on the angle between $u$ and $v$. If $u$ and $v$ are very similar, their cosine similarity will be close to 1; if they are dissimilar, the cosine similarity will take a smaller value.

 Figure 1: The cosine of the angle between two vectors is a measure of how similar they are

Exercise: Implement the function cosine_similarity() to evaluate similarity between word vectors.
Reminder: The norm of $u$ is defined as $ ||u||2 = \sqrt{\sum{i=1}^{n} u_i^2}$
Can you write Python code for it?

# GRADED FUNCTION: cosine_similarity

def cosine_similarity(u, v):
    """
    Cosine similarity reflects the degree of similariy between u and v

    Arguments:
        u -- a word vector of shape (n,)
        v -- a word vector of shape (n,)

    Returns:
        cosine_similarity -- the cosine similarity between u and v defined by the formula above.
    """

    distance = 0.0

    ### START CODE HERE ###
    # Compute the dot product between u and v (≈1 line)
    dot = np.dot(u, v)
    # Compute the L2 norm of u (≈1 line)
    norm_u = np.sqrt(np.sum(u**2))

    # Compute the L2 norm of v (≈1 line)
    norm_v = np.sqrt(np.sum(v**2))
    # Compute the cosine similarity defined by formula (1) (≈1 line)
    cosine_similarity = dot/(norm_u*norm_v)
    ### END CODE HERE ###

    return cosine_similarity

father = word_to_vec_map["father"]
mother = word_to_vec_map["mother"]
ball = word_to_vec_map["ball"]
crocodile = word_to_vec_map["crocodile"]
france = word_to_vec_map["france"]
italy = word_to_vec_map["italy"]
paris = word_to_vec_map["paris"]
rome = word_to_vec_map["rome"]

print("cosine_similarity(father, mother) = ", cosine_similarity(father, mother))
print("cosine_similarity(ball, crocodile) = ",cosine_similarity(ball, crocodile))
print("cosine_similarity(france - paris, rome - italy) = ",cosine_similarity(france - paris, rome - italy))


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Decoder During Inference

Reuse the weights the biases from the training decoder using tf.variable_scope("decoding", reuse=True)
Build the inference decoder using tf.contrib.seq2seq.simple_decoder_fn_inference and tf.contrib.seq2seq.dynamic_rnn_decoder.
The output function is applied to the output in this step
Can you write Python code for it?

with tf.variable_scope("decoding", reuse=True) as decoding_scope:
    # Inference Decoder
    infer_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_inference(
        output_fn, enc_state, dec_embeddings, target_letter_to_int['<s>'], target_letter_to_int['<\s>'],
        sequence_length - 1, target_vocab_size)
    inference_logits, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(dec_cell, infer_decoder_fn, scope=decoding_scope)


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Decoder During Training

Build the training decoder using tf.contrib.seq2seq.simple_decoder_fn_train and tf.contrib.seq2seq.dynamic_rnn_decoder.
Apply the output layer to the output of the training decoder
Can you write Python code for it?

with tf.variable_scope("decoding") as decoding_scope:
    # Training Decoder
    train_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_train(enc_state)
    train_pred, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(
        dec_cell, train_decoder_fn, dec_embed_input, sequence_length, scope=decoding_scope)

    # Apply output function
    train_logits =  output_fn(train_pred)


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Decoding

Embed the decoding input
Build the decoding RNNs
Build the output layer in the decoding scope, so the weight and bias can be shared between the training and inference decoders.
Can you write Python code for it?

target_vocab_size = len(target_letter_to_int)

# Decoder Embedding
dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)

# Decoder RNNs
dec_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size)] * num_layers)

with tf.variable_scope("decoding") as decoding_scope:
    # Output Layer
    output_fn = lambda x: tf.contrib.layers.fully_connected(x, target_vocab_size, None, scope=decoding_scope)


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Hyperparameters
Can you write Python code for it?

# Number of Epochs
epochs = 60
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 50
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 13
decoding_embedding_size = 13
# Learning Rate
learning_rate = 0.001


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Preprocess
To do anything useful with it, we'll need to turn the characters into a list of integers:
Can you write Python code for it?

def extract_character_vocab(data):
    special_words = ['<pad>', '<unk>', '<s>',  '<\s>']

    set_words = set([character for line in data.split('\n') for character in line])
    int_to_vocab = {word_i: word for word_i, word in enumerate(special_words + list(set_words))}
    vocab_to_int = {word: word_i for word_i, word in int_to_vocab.items()}

    return int_to_vocab, vocab_to_int

# Build int2letter and letter2int dicts
source_int_to_letter, source_letter_to_int = extract_character_vocab(source_sentences)
target_int_to_letter, target_letter_to_int = extract_character_vocab(target_sentences)

# Convert characters to ids
source_letter_ids = [[source_letter_to_int.get(letter, source_letter_to_int['<unk>']) for letter in line] for line in source_sentences.split('\n')]
target_letter_ids = [[target_letter_to_int.get(letter, target_letter_to_int['<unk>']) for letter in line] for line in target_sentences.split('\n')]

print("Example source sequence")
print(source_letter_ids[:3])
print("\n")
print("Example target sequence")
print(target_letter_ids[:3])


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Process Decoding Input
Can you write Python code for it?

import numpy as np

# Process the input we'll feed to the decoder
ending = tf.strided_slice(targets, [0, 0], [batch_size, -1], [1, 1])
dec_input = tf.concat([tf.fill([batch_size, 1], target_letter_to_int['<s>']), ending], 1)

demonstration_outputs = np.reshape(range(batch_size * sequence_length), (batch_size, sequence_length))

sess = tf.InteractiveSession()
print("Targets")
print(demonstration_outputs[:2])
print("\n")
print("Processed Decoding Input")
print(sess.run(dec_input, {targets: demonstration_outputs})[:2])


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
Train
We're now ready to train our model. If you run into OOM (out of memory) issues during training, try to decrease the batch_size.
Can you write Python code for it?

import numpy as np

train_source = source_ids[batch_size:]
train_target = target_ids[batch_size:]

valid_source = source_ids[:batch_size]
valid_target = target_ids[:batch_size]

sess.run(tf.global_variables_initializer())

for epoch_i in range(epochs):
    for batch_i, (source_batch, target_batch) in enumerate(
            helper.batch_data(train_source, train_target, batch_size)):
        _, loss = sess.run(
            [train_op, cost],
            {input_data: source_batch, targets: target_batch, lr: learning_rate})
        batch_train_logits = sess.run(
            inference_logits,
            {input_data: source_batch})
        batch_valid_logits = sess.run(
            inference_logits,
            {input_data: valid_source})

        train_acc = np.mean(np.equal(target_batch, np.argmax(batch_train_logits, 2)))
        valid_acc = np.mean(np.equal(valid_target, np.argmax(batch_valid_logits, 2)))
        print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss: {:>6.3f}'
              .format(epoch_i, batch_i, len(source_ids) // batch_size, train_acc, valid_acc, loss))


I am working on the file "Sequence-to-Sequence/sequence_to_sequence_implementation.ipynb".
The first task is:
source_sentences contains the entire output sequence file as text delimited by newline symbols.  Each line corresponds to the line from source_sentences.  source_sentences contains a sorted characters of the line.
Can you write Python code for it?


I am working on the file "Sequence_Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb".
The first task is:
1 - Problem Statement
1.1 - Dataset and Preprocessing
Run the following cell to read the dataset of dinosaur names, create a list of unique characters (such as a-z), and compute the dataset and vocabulary size.
Can you write Python code for it?

data = open('dinos.txt', 'r').read()
data= data.lower()
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))


I am working on the file "Sequence_Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb".
The first task is:
Conclusion
You can see that your algorithm has started to generate plausible dinosaur names towards the end of the training. At first, it was generating random characters, but towards the end you could see dinosaur names with cool endings. Feel free to run the algorithm even longer and play with hyperparameters to see if you can get even better results. Our implemetation generated some really cool names like maconucon, marloralus and macingsersaurus. Your model hopefully also learned that dinosaur names tend to end in saurus, don, aura, tor, etc.
If your model generates some non-cool names, don't blame the model entirely--not all actual dinosaur names sound cool. (For example, dromaeosauroides is an actual dinosaur name and is in the training set.) But this model should give you a set of candidates from which you can pick the coolest!
This assignment had used a relatively small dataset, so that you could train an RNN quickly on a CPU. Training a model of the english language requires a much bigger dataset, and usually needs much more computation, and could run for many hours on GPUs. We ran our dinosaur name for quite some time, and so far our favoriate name is the great, undefeatable, and fierce: Mangosaurus!

4 - Writing like Shakespeare
The rest of this notebook is optional and is not graded, but we hope you'll do it anyway since it's quite fun and informative.
A similar (but more complicated) task is to generate Shakespeare poems. Instead of learning from a dataset of Dinosaur names you can use a collection of Shakespearian poems. Using LSTM cells, you can learn longer term dependencies that span many characters in the text--e.g., where a character appearing somewhere a sequence can influence what should be a different character much much later in ths sequence. These long term dependencies were less important with dinosaur names, since the names were quite short.
<img src="images/shakespeare.jpg"...
Can you write Python code for it?

from __future__ import print_function
from keras.callbacks import LambdaCallback
from keras.models import Model, load_model, Sequential
from keras.layers import Dense, Activation, Dropout, Input, Masking
from keras.layers import LSTM
from keras.utils.data_utils import get_file
from keras.preprocessing.sequence import pad_sequences
from shakespeare_utils import *
import sys
import io


I am working on the file "Sequence_Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb".
The first task is:
Expected output:

    **gradients["dWaa"][1][2] **

    10.0

    **gradients["dWax"][3][1]**

    -10.0

    **gradients["dWya"][1][2]**

0.29713815361

    **gradients["db"][4]**

[ 10.]

    **gradients["dby"][1]**

[ 8.45833407]

2.2 - Sampling
Now assume that your model is trained. You would like to generate new text (characters). The process of generation is explained in the picture below:

 Figure 3: In this picture, we assume the model is already trained. We pass in $x^{\langle 1\rangle} = \vec{0}$ at the first time step, and have the network then sample one character at a time.

Exercise: Implement the sample function below to sample characters. You need to carry out 4 steps:

Step 1: Pass the network the first "dummy" input $x^{\langle 1 \rangle} = \vec{0}$ (the vector of zeros). This is the default input before we've generated any characters. We also set $a^{\langle 0 \rangle} = \vec{0}$

Step 2: Run one step of forward propagation to get $a^{\langle 1 \rangle}$ and $\hat{y}^{\langle 1 \rangle}$. Here are the equations:

$$ a^{\langle t+1 \rangle} = \tanh(W_{ax}  x^{\langle t \rangle } + W_{aa} a^{\langle t \rangle } + b)\tag{1}$$
$$ z^{\langle t + 1 \rangle } = W_{ya}  a^{\langle t + 1 \rangle } + b_y \tag{2}$$
$$ \hat{y}^{\langle t+1 \rangle } = softmax(z^{\langle t + 1 \rangle })\tag{3}$$
Note that $\hat{y}^{\langle t+1 \rangle }$ is a (softmax) probability vector (its entries are between 0 and 1 and sum to 1). $\hat{y}^{\langle t+1 \rangle}_i$ represents the probability that the character indexed by "i" is the next character.  We have provided a softmax() function that you can use.

Step 3: Carry out sampling: Pick the next...
Can you write Python code for it?

# GRADED FUNCTION: sample

def sample(parameters, char_to_ix, seed):
    """
    Sample a sequence of characters according to a sequence of probability distributions output of the RNN

    Arguments:
    parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b.
    char_to_ix -- python dictionary mapping each character to an index.
    seed -- used for grading purposes. Do not worry about it.

    Returns:
    indices -- a list of length n containing the indices of the sampled characters.
    """

    # Retrieve parameters and relevant shapes from "parameters" dictionary
    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']
    vocab_size = by.shape[0]
    n_a = Waa.shape[1]

    ### START CODE HERE ###
    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)
    x = np.zeros((vocab_size,1))
    # Step 1': Initialize a_prev as zeros (≈1 line)
    a_prev = np.zeros((n_a,1))

    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)
    indices = []

    # Idx is a flag to detect a newline character, we initialize it to -1
    idx = -1

    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append
    # its index to "indices". We'll stop if we reach 50 characters (which should be very unlikely with a well
    # trained model), which helps debugging and prevents entering an infinite loop.
    counter = 0
    newline_character = char_to_ix['\n']

    while (idx != newline_character and counter != 50):

        # Step 2: Forward propagate x using the equations (1), (2) and (3)
        a = np.tanh(np.dot(Wax,x)+np.dot(Waa,a_prev)+b)
        z = np.dot(Wya,a)+by
        y = softmax(z)

        # for grading purposes
        np.random.seed(counter+seed)

        # Step 3: Sample the index of a...


I am working on the file "Sequence_Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb".
The first task is:
Expected output:

    **list of sampled indices:**

    [12, 17, 24, 14, 13, 9, 10, 22, 24, 6, 13, 11, 12, 6, 21, 15, 21, 14, 3, 2, 1, 21, 18, 24,

    7, 25, 6, 25, 18, 10, 16, 2, 3, 8, 15, 12, 11, 7, 1, 12, 10, 2, 7, 7, 11, 5, 6, 12, 25, 0, 0]

    **list of sampled characters:**

    ['l', 'q', 'x', 'n', 'm', 'i', 'j', 'v', 'x', 'f', 'm', 'k', 'l', 'f', 'u', 'o',

    'u', 'n', 'c', 'b', 'a', 'u', 'r', 'x', 'g', 'y', 'f', 'y', 'r', 'j', 'p', 'b', 'c', 'h', 'o',

    'l', 'k', 'g', 'a', 'l', 'j', 'b', 'g', 'g', 'k', 'e', 'f', 'l', 'y', '\n', '\n']

3 - Building the language model
It is time to build the character-level language model for text generation.
3.1 - Gradient descent
In this section you will implement a function performing one step of stochastic gradient descent (with clipped gradients). You will go through the training examples one at a time, so the optimization algorithm will be stochastic gradient descent. As a reminder, here are the steps of a common optimization loop for an RNN:

Forward propagate through the RNN to compute the loss
Backward propagate through time to compute the gradients of the loss with respect to the parameters
Clip the gradients if necessary
Update your parameters using gradient descent

Exercise: Implement this optimization process (one step of stochastic gradient descent).
We provide you with the following functions:
```python
def rnn_forward(X, Y, a_prev, parameters):
    """ Performs the forward propagation through the RNN and computes the cross-entropy loss.
    It returns the loss' value as well as a "cache" storing values to be used in the backpropagation."""
    ....
    return loss, cache
def rnn_backward(X, Y, parameters, cache):
    """ Performs the backward propagation through time to compute the gradients of the loss with respect
    to the parameters. It returns also all the hidden states."""
    ...
    return gradients,...
Can you write Python code for it?

# GRADED FUNCTION: optimize

def optimize(X, Y, a_prev, parameters, learning_rate = 0.01):
    """
    Execute one step of the optimization to train the model.

    Arguments:
    X -- list of integers, where each integer is a number that maps to a character in the vocabulary.
    Y -- list of integers, exactly the same as X but shifted one index to the left.
    a_prev -- previous hidden state.
    parameters -- python dictionary containing:
                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
                        b --  Bias, numpy array of shape (n_a, 1)
                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
    learning_rate -- learning rate for the model.

    Returns:
    loss -- value of the loss function (cross-entropy)
    gradients -- python dictionary containing:
                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)
                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)
                        dWya -- Gradients of hidden-to-output weights, of shape (n_y, n_a)
                        db -- Gradients of bias vector, of shape (n_a, 1)
                        dby -- Gradients of output bias vector, of shape (n_y, 1)
    a[len(X)-1] -- the last hidden state, of shape (n_a, 1)
    """

    ### START CODE HERE ###

    # Forward propagate through time (≈1 line)
    loss, cache = rnn_forward(X, Y, a_prev, parameters)

    # Backpropagate through time (≈1 line)
    gradients, a = rnn_backward(X, Y, parameters, cache)

    # Clip your gradients between -5 (min) and 5 (max) (≈1 line)
    gradients = clip(gradients, 5)

    # Update parameters (≈1 line)
    parameters =...


I am working on the file "Sequence_Models/Dinosaurus+Island+--+Character+level+language+model+final+-+v3.ipynb".
The first task is:
To save you some time, we have already trained a model for ~1000 epochs on a collection of Shakespearian poems called "The Sonnets".
Let's train the model for one more epoch. When it finishes training for an epoch---this will also take a few minutes---you can run generate_output, which will prompt asking you for an input (<40 characters). The poem will start with your sentence, and our RNN-Shakespeare will complete the rest of the poem for you! For example, try "Forsooth this maketh no sense " (don't enter the quotation marks). Depending on whether you include the space at the end, your results might also differ--try it both ways, and try other inputs as well.
Can you write Python code for it?

print_callback = LambdaCallback(on_epoch_end=on_epoch_end)

model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])

# Run this cell to try with different inputs without having to re-train the model
generate_output()


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
1 - Baseline model: Emojifier-V1
1.1 - Dataset EMOJISET
Let's start by building a simple baseline classifier.
You have a tiny dataset (X, Y) where:
- X contains 127 sentences (strings)
- Y contains a integer label between 0 and 4 corresponding to an emoji for each sentence

 Figure 1: EMOJISET - a classification problem with 5 classes. A few examples of sentences are given here.

Let's load the dataset using the code below. We split the dataset between training (127 examples) and testing (56 examples).
Can you write Python code for it?

X_train, Y_train = read_csv('data/train_emoji.csv')
X_test, Y_test = read_csv('data/tesss.csv')

maxLen = len(max(X_train, key=len).split())


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
1.2 - Overview of the Emojifier-V1
In this part, you are going to implement a baseline model called "Emojifier-v1".

 Figure 2: Baseline model (Emojifier-V1).

The input of the model is a string corresponding to a sentence (e.g. "I love you). In the code, the output will be a probability vector of shape (1,5), that you then pass in an argmax layer to extract the index of the most likely emoji output.
To get our labels into a format suitable for training a softmax classifier, lets convert $Y$ from its current shape  current shape $(m, 1)$ into a "one-hot representation" $(m, 5)$, where each row is a one-hot vector giving the label of one example, You can do so using this next code snipper. Here, Y_oh stands for "Y-one-hot" in the variable names Y_oh_train and Y_oh_test:
Can you write Python code for it?

Y_oh_train = convert_to_one_hot(Y_train, C = 5)
Y_oh_test = convert_to_one_hot(Y_test, C = 5)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
2.1 - Overview of the model
Here is the Emojifier-v2 you will implement:

 Figure 3: Emojifier-V2. A 2-layer LSTM sequence classifier.

2.2 Keras and mini-batching
In this exercise, we want to train Keras using mini-batches. However, most deep learning frameworks require that all sequences in the same mini-batch have the same length. This is what allows vectorization to work: If you had a 3-word sentence and a 4-word sentence, then the computations needed for them are different (one takes 3 steps of an LSTM, one takes 4 steps) so it's just not possible to do them both at the same time.
The common solution to this is to use padding. Specifically, set a maximum sequence length, and pad all sequences to the same length. For example, of the maximum sequence length is 20, we could pad every sentence with "0"s so that each input sentence is of length 20. Thus, a sentence "i love you" would be represented as $(e_{i}, e_{love}, e_{you}, \vec{0}, \vec{0}, \ldots, \vec{0})$. In this example, any sentences longer than 20 words would have to be truncated. One simple way to choose the maximum sequence length is to just pick the length of the longest sentence in the training set.
2.3 - The Embedding layer
In Keras, the embedding matrix is represented as a "layer", and maps positive integers (indices corresponding to words) into dense vectors of fixed size (the embedding vectors). It can be trained or initialized with a pretrained embedding. In this part, you will learn how to create an Embedding() layer in Keras, initialize it with the GloVe 50-dimensional vectors loaded earlier in the notebook. Because our training set is quite small, we will not update the word embeddings but will instead leave their values fixed. But in the code below, we'll show you how Keras allows you to either train or leave fixed this layer.
The Embedding() layer takes an integer matrix of size (batch size, max input length) as input....
Can you write Python code for it?

# GRADED FUNCTION: sentences_to_indices

def sentences_to_indices(X, word_to_index, max_len):
    """
    Converts an array of sentences (strings) into an array of indices corresponding to words in the sentences.
    The output shape should be such that it can be given to `Embedding()` (described in Figure 4).

    Arguments:
    X -- array of sentences (strings), of shape (m, 1)
    word_to_index -- a dictionary containing the each word mapped to its index
    max_len -- maximum number of words in a sentence. You can assume every sentence in X is no longer than this.

    Returns:
    X_indices -- array of indices corresponding to words in the sentences from X, of shape (m, max_len)
    """

    m = X.shape[0]                                   # number of training examples

    ### START CODE HERE ###
    # Initialize X_indices as a numpy matrix of zeros and the correct shape (≈ 1 line)
    X_indices = np.zeros((m,max_len))

    for i in range(m):                               # loop over training examples

        # Convert the ith training sentence in lower case and split is into words. You should get a list of words.
        sentence_words = [w.lower() for w in X[i].split()]

        # Initialize j to 0
        j = 0

        # Loop over the words of sentence_words
        for w in sentence_words:
            # Set the (i,j)th entry of X_indices to the index of the correct word.
            X_indices[i, j] = word_to_index[w]
            # Increment j to j + 1
            j = j+1

    ### END CODE HERE ###

    return X_indices


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
<font color='blue'>
What you should remember from this part:
- Even with a 127 training examples, you can get a reasonably good model for Emojifying. This is due to the generalization power word vectors gives you.
- Emojify-V1 will perform poorly on sentences such as "This movie is not good and not enjoyable" because it doesn't understand combinations of words--it just averages all the words' embedding vectors together, without paying attention to the ordering of words. You will build a better algorithm in the next part.
2 - Emojifier-V2: Using LSTMs in Keras:
Let's build an LSTM model that takes as input word sequences. This model will be able to take word ordering into account. Emojifier-V2 will continue to use pre-trained word embeddings to represent words, but will feed them into an LSTM, whose job it is to predict the most appropriate emoji.
Run the following cell to load the Keras packages.
Can you write Python code for it?

import numpy as np
np.random.seed(0)
from keras.models import Model
from keras.layers import Dense, Input, Dropout, LSTM, Activation
from keras.layers.embeddings import Embedding
from keras.preprocessing import sequence
from keras.initializers import glorot_uniform
np.random.seed(1)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Amazing! Because adore has a similar embedding as love, the algorithm has generalized correctly even to a word it has never seen before. Words such as heart, dear, beloved or adore have embedding vectors similar to love, and so might work too---feel free to modify the inputs above and try out a variety of input sentences. How well does it work?
Note though that it doesn't get "not feeling happy" correct. This algorithm ignores word ordering, so is not good at understanding phrases like "not happy."
Printing the confusion matrix can also help understand which classes are more difficult for your model. A confusion matrix shows how often an example whose label is one class ("actual" class) is mislabeled by the algorithm with a different class ("predicted" class).
Can you write Python code for it?

print(Y_test.shape)
print('           '+ label_to_emoji(0)+ '    ' + label_to_emoji(1) + '    ' +  label_to_emoji(2)+ '    ' + label_to_emoji(3)+'   ' + label_to_emoji(4))
print(pd.crosstab(Y_test, pred_test.reshape(56,), rownames=['Actual'], colnames=['Predicted'], margins=True))
plot_confusion_matrix(Y_test, pred_test)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics your are want to use. Compile your model using categorical_crossentropy loss, adam optimizer and ['accuracy'] metrics:
Can you write Python code for it?


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Exercise: Implement sentence_to_avg(). You will need to carry out two steps:
1. Convert every sentence to lower-case, then split the sentence into a list of words. X.lower() and X.split() might be useful.
2. For each word in the sentence, access its GloVe representation. Then, average all these values.
Can you write Python code for it?

# GRADED FUNCTION: sentence_to_avg

def sentence_to_avg(sentence, word_to_vec_map):
    """
    Converts a sentence (string) into a list of words (strings). Extracts the GloVe representation of each word
    and averages its value into a single vector encoding the meaning of the sentence.

    Arguments:
    sentence -- string, one training example from X
    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation

    Returns:
    avg -- average vector encoding information about the sentence, numpy-array of shape (50,)
    """

    ### START CODE HERE ###
    # Step 1: Split sentence into list of lower case words (≈ 1 line)
    words = sentence.lower().split()

    # Initialize the average word vector, should have the same shape as your word vectors.
    avg = np.zeros((50,))

    # Step 2: average the word vectors. You can loop over the words in the list "words".
    for w in words:
        avg += word_to_vec_map[w]

    avg = avg/len(words)

    ### END CODE HERE ###

    return avg

avg = sentence_to_avg("Morrocan couscous is my favorite dish", word_to_vec_map)
print("avg = ", avg)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Expected Output (on a subset of iterations):

            **Epoch: 0**

           cost = 1.95204988128

           Accuracy: 0.348484848485

            **Epoch: 100**

           cost = 0.0797181872601

           Accuracy: 0.931818181818

            **Epoch: 200**

           cost = 0.0445636924368

           Accuracy: 0.954545454545

            **Epoch: 300**

           cost = 0.0343226737879

           Accuracy: 0.969696969697

Great! Your model has pretty high accuracy on the training set. Lets now see how it does on the test set.
1.4 - Examining test set performance
Can you write Python code for it?

print("Training set:")
pred_train = predict(X_train, Y_train, W, b, word_to_vec_map)
print('Test set:')
pred_test = predict(X_test, Y_test, W, b, word_to_vec_map)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Expected Output:

            **X1 =**

           ['funny lol' 'lets play football' 'food is ready for you']

            **X1_indices =**

           [[ 155345.  225122.       0.       0.       0.]

            [ 220930.  286375.  151266.       0.       0.]

            [ 151204.  192973.  302254.  151349.  394475.]]

Let's build the Embedding() layer in Keras, using pre-trained word vectors. After this layer is built, you will pass the output of sentences_to_indices() to it as an input, and the Embedding() layer will return the word embeddings for a sentence.
Exercise: Implement pretrained_embedding_layer(). You will need to carry out the following steps:
1. Initialize the embedding matrix as a numpy array of zeroes with the correct shape.
2. Fill in the embedding matrix with all the word embeddings extracted from word_to_vec_map.
3. Define Keras embedding layer. Use Embedding(). Be sure to make this layer non-trainable, by setting trainable = False when calling Embedding(). If you were to set trainable = True, then it will allow the optimization algorithm to modify the values of the word embeddings.
4. Set the embedding weights to be equal to the embedding matrix
Can you write Python code for it?

# GRADED FUNCTION: pretrained_embedding_layer

def pretrained_embedding_layer(word_to_vec_map, word_to_index):
    """
    Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.

    Arguments:
    word_to_vec_map -- dictionary mapping words to their GloVe vector representation.
    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)

    Returns:
    embedding_layer -- pretrained layer Keras instance
    """

    vocab_len = len(word_to_index) + 1                  # adding 1 to fit Keras embedding (requirement)
    emb_dim = word_to_vec_map["cucumber"].shape[0]      # define dimensionality of your GloVe word vectors (= 50)

    ### START CODE HERE ###
    # Initialize the embedding matrix as a numpy array of zeros of shape (vocab_len, dimensions of word vectors = emb_dim)
    emb_matrix = np.zeros((vocab_len, emb_dim))

    # Set each row "index" of the embedding matrix to be the word vector representation of the "index"th word of the vocabulary
    for word, index in word_to_index.items():
        emb_matrix[index, :] = word_to_vec_map[word]

    # Define Keras embedding layer with the correct output/input sizes, make it trainable. Use Embedding(...). Make sure to set trainable=False.
    embedding_layer = Embedding(vocab_len, emb_dim, trainable=False)
    ### END CODE HERE ###

    # Build the embedding layer, it is required before setting the weights of the embedding layer. Do not modify the "None".
    embedding_layer.build((None,))

    # Set the weights of the embedding layer to the embedding matrix. Your layer is now pretrained.
    embedding_layer.set_weights([emb_matrix])

    return embedding_layer

embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)
print("weights[0][1][3] =", embedding_layer.get_weights()[0][1][3])


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Expected Output:

            **avg= **

           [-0.008005    0.56370833 -0.50427333  0.258865    0.55131103  0.03104983
 -0.21013718  0.16893933 -0.09590267  0.141784   -0.15708967  0.18525867
  0.6495785   0.38371117  0.21102167  0.11301667  0.02613967  0.26037767
  0.05820667 -0.01578167 -0.12078833 -0.02471267  0.4128455   0.5152061
  0.38756167 -0.898661   -0.535145    0.33501167  0.68806933 -0.2156265
  1.797155    0.10476933 -0.36775333  0.750785    0.10282583  0.348925
 -0.27262833  0.66768    -0.10706167 -0.283635    0.59580117  0.28747333
 -0.3366635   0.23393817  0.34349183  0.178405    0.1166155  -0.076433
  0.1445417   0.09808667]

Model
You now have all the pieces to finish implementing the model() function. After using sentence_to_avg() you need to pass the average through forward propagation, compute the cost, and then backpropagate to update the softmax's parameters.
Exercise: Implement the model() function described in Figure (2). Assuming here that $Yoh$ ("Y one hot") is the one-hot encoding of the output labels, the equations you need to implement in the forward pass and to compute the cross-entropy cost are:
$$ z^{(i)} = W . avg^{(i)} + b$$
$$ a^{(i)} = softmax(z^{(i)})$$
$$ \mathcal{L}^{(i)} = - \sum_{k = 0}^{n_y - 1} Yoh^{(i)}_k * log(a^{(i)}_k)$$
It is possible to come up with a more efficient vectorized implementation. But since we are using a for-loop to convert the sentences one at a time into the avg^{(i)} representation anyway, let's not bother this time.
We provided you a function softmax().
Can you write Python code for it?

# GRADED FUNCTION: model

def model(X, Y, word_to_vec_map, learning_rate = 0.01, num_iterations = 400):
    """
    Model to train word vector representations in numpy.

    Arguments:
    X -- input data, numpy array of sentences as strings, of shape (m, 1)
    Y -- labels, numpy array of integers between 0 and 7, numpy-array of shape (m, 1)
    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation
    learning_rate -- learning_rate for the stochastic gradient descent algorithm
    num_iterations -- number of iterations

    Returns:
    pred -- vector of predictions, numpy-array of shape (m, 1)
    W -- weight matrix of the softmax layer, of shape (n_y, n_h)
    b -- bias of the softmax layer, of shape (n_y,)
    """

    np.random.seed(1)

    # Define number of training examples
    m = Y.shape[0]                          # number of training examples
    n_y = 5                                 # number of classes
    n_h = 50                                # dimensions of the GloVe vectors

    # Initialize parameters using Xavier initialization
    W = np.random.randn(n_y, n_h) / np.sqrt(n_h)
    b = np.zeros((n_y,))

    # Convert Y to Y_onehot with n_y classes
    Y_oh = convert_to_one_hot(Y, C = n_y)

    # Optimization loop
    for t in range(num_iterations):                       # Loop over the number of iterations
        for i in range(m):                                # Loop over the training examples

            ### START CODE HERE ### (≈ 4 lines of code)
            # Average the word vectors of the words from the i'th training example
            avg = sentence_to_avg(X[i], word_to_vec_map)

            # Forward propagate the avg through the softmax layer
            z = np.dot(W, avg)+b
            a = softmax(z)

            # Compute cost using the i'th training label's one hot representation and "A" (the output of the softmax)
            cost = -np.sum(np.multiply(Y_oh, np.log(a)))
      ...


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Expected Output:

            **weights[0][1][3] =**

           -0.3403

2.3 Building the Emojifier-V2
Lets now build the Emojifier-V2 model. You will do so using the embedding layer you have built, and feed its output to an LSTM network.

 Figure 3: Emojifier-v2. A 2-layer LSTM sequence classifier.

Exercise: Implement Emojify_V2(), which builds a Keras graph of the architecture shown in Figure 3. The model takes as input an array of sentences of shape (m, max_len, ) defined by input_shape. It should output a softmax probability vector of shape (m, C = 5). You may need Input(shape = ..., dtype = '...'), LSTM(), Dropout(), Dense(), and Activation().
Can you write Python code for it?

# GRADED FUNCTION: Emojify_V2

def Emojify_V2(input_shape, word_to_vec_map, word_to_index):
    """
    Function creating the Emojify-v2 model's graph.

    Arguments:
    input_shape -- shape of the input, usually (max_len,)
    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation
    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)

    Returns:
    model -- a model instance in Keras
    """

    ### START CODE HERE ###
    # Define sentence_indices as the input of the graph, it should be of shape input_shape and dtype 'int32' (as it contains indices).
    sentence_indices = Input(input_shape, dtype='int32')

    # Create the embedding layer pretrained with GloVe Vectors (≈1 line)
    embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)

    # Propagate sentence_indices through your embedding layer, you get back the embeddings
    embeddings = embedding_layer(sentence_indices)

    # Propagate the embeddings through an LSTM layer with 128-dimensional hidden state
    # Be careful, the returned output should be a batch of sequences.
    X = LSTM(128, return_sequences=True)(embeddings)
    # Add dropout with a probability of 0.5
    X = Dropout(0.5)(X)
    # Propagate X trough another LSTM layer with 128-dimensional hidden state
    # Be careful, the returned output should be a single hidden state, not a batch of sequences.
    X = LSTM(128, return_sequences=False)(X)
    # Add dropout with a probability of 0.5
    X = Dropout(0.5)(X)
    # Propagate X through a Dense layer with softmax activation to get back a batch of 5-dimensional vectors.
    X = Dense(5)(X)
    # Add a softmax activation
    X = Activation('softmax')(X)

    # Create Model instance which converts sentence_indices into X.
    model = Model(inputs=sentence_indices ,outputs=X)

    ### END CODE HERE ###

    return model


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Fit the Keras model on X_train_indices and Y_train_oh. We will use epochs = 50 and batch_size = 32.
Can you write Python code for it?

model.fit(X_train_indices, Y_train_oh, epochs = 50, batch_size = 32, shuffle=True)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
It's time to train your model. Your Emojifier-V2 model takes as input an array of shape (m, max_len) and outputs probability vectors of shape (m, number of classes). We thus have to convert X_train (array of sentences as strings) to X_train_indices (array of sentences as list of word indices), and Y_train (labels as indices) to Y_train_oh (labels as one-hot vectors).
Can you write Python code for it?

X_train_indices = sentences_to_indices(X_train, word_to_index, maxLen)
Y_train_oh = convert_to_one_hot(Y_train, C = 5)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Now you can try it on your own example. Write your own sentence below.
Can you write Python code for it?

# Change the sentence below to see your prediction. Make sure all the words are in the Glove embeddings.
x_test = np.array(['not feeling happy'])
X_test_indices = sentences_to_indices(x_test, word_to_index, maxLen)
print(x_test[0] +' '+  label_to_emoji(np.argmax(model.predict(X_test_indices))))


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Run the following cell to create your model and check its summary. Because all sentences in the dataset are less than 10 words, we chose max_len = 10.  You should see your architecture, it uses "20,223,927" parameters, of which 20,000,050 (the word embeddings) are non-trainable, and the remaining 223,877 are. Because our vocabulary size has 400,001 words (with valid indices from 0 to 400,000) there are 400,001*50 = 20,000,050 non-trainable parameters.
Can you write Python code for it?

model = Emojify_V2((maxLen,), word_to_vec_map, word_to_index)
model.summary()


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Run the following cell to print sentences from X_train and corresponding labels from Y_train. Change index to see different examples. Because of the font the iPython notebook uses, the heart emoji may be colored black rather than red.
Can you write Python code for it?

index = 59
print(X_train[index], label_to_emoji(Y_train[index]))


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Run the next cell to train your model and learn the softmax parameters (W,b).
Can you write Python code for it?

pred, W, b = model(X_train, Y_train, word_to_vec_map)
print(pred)


I am working on the file "Sequence_Models/Emojify+-+v2.ipynb".
The first task is:
Your model should perform close to 100% accuracy on the training set. The exact accuracy you get may be a little different. Run the following cell to evaluate your model on the test set.
Can you write Python code for it?

X_test_indices = sentences_to_indices(X_test, word_to_index, max_len = maxLen)
Y_test_oh = convert_to_one_hot(Y_test, C = 5)
loss, acc = model.evaluate(X_test_indices, Y_test_oh)
print()
print("Test accuracy = ", acc)


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
2 - Neural machine translation with attention
If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate. Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down.
The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step.
2.1 - Attention mechanism
In this part, you will implement the attention mechanism presented in the lecture videos. Here is a figure to remind you how the model works. The diagram on the left shows the attention model. The diagram on the right shows what one "Attention" step does to calculate the attention variables $\alpha^{\langle t, t' \rangle}$, which are used to compute the context variable $context^{\langle t \rangle}$ for each timestep in the output ($t=1, \ldots, T_y$).

 Figure 1: Neural machine translation with attention

Here are some properties of the model that you may notice:

There are two separate LSTMs in this model (see diagram on the left). Because the one at the bottom of the picture is a Bi-directional LSTM and comes before the attention mechanism, we will call it pre-attention Bi-LSTM. The LSTM at the top of the diagram comes after the attention mechanism, so we will call it the post-attention LSTM. The pre-attention Bi-LSTM goes through $T_x$ time steps; the post-attention LSTM goes through $T_y$ time steps.

The post-attention LSTM passes $s^{\langle t \rangle}, c^{\langle t \rangle}$ from one time step to the next. In the lecture videos, we were using only a basic RNN for the post-activation sequence model, so the state captured by the RNN output activations $s^{\langle t\rangle}$. But since we are...
Can you write Python code for it?

# Defined shared layers as global variables
repeator = RepeatVector(Tx)
concatenator = Concatenate(axis=-1)
densor1 = Dense(10, activation = "tanh")
densor2 = Dense(1, activation = "relu")
activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook
dotor = Dot(axes = 1)


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Expected Output:
Here is the summary you should see

            **Total params:**

         185,484

            **Trainable params:**

         185,484

            **Non-trainable params:**

         0

            **bidirectional_1's output shape **

         (None, 30, 128)

            **repeat_vector_1's output shape **

         (None, 30, 128)

            **concatenate_1's output shape **

         (None, 30, 256)

            **attention_weights's output shape **

         (None, 30, 1)

            **dot_1's output shape **

         (None, 1, 128)

            **dense_2's output shape **

         (None, 11)

As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics your are want to use. Compile your model using categorical_crossentropy loss, a custom Adam optimizer (learning rate = 0.005, $\beta_1 = 0.9$, $\beta_2 = 0.999$, decay = 0.01)  and ['accuracy'] metrics:
Can you write Python code for it?

### START CODE HERE ### (≈2 lines)
opt = model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=0.001, beta_1=0.9, beta_2=0.999, decay=0.1), metrics=['accuracy'])
opt
### END CODE HERE ###


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Let's get a summary of the model to check if it matches the expected output.
Can you write Python code for it?


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Let's now fit the model and run it for one epoch.
Can you write Python code for it?

model.fit([Xoh, s0, c0], outputs, epochs=1, batch_size=100)


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Navigate through the output of model.summary() above. You can see that the layer named attention_weights outputs the alphas of shape (m, 30, 1) before dot_2 computes the context vector for every time step $t = 0, \ldots, T_y-1$. Lets get the activations from this layer.
The function attention_map() pulls out the attention values from your model and plots them.
Can you write Python code for it?

attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, "Tuesday 09 Oct 1993", num = 7, n_s = 64)


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
Now you can use these layers to implement one_step_attention(). In order to propagate a Keras tensor object X through one of these layers, use layer(X) (or layer([X,Y]) if it requires multiple inputs.), e.g. densor(X) will propagate X through the Dense(1) layer defined above.
Can you write Python code for it?

# GRADED FUNCTION: one_step_attention

def one_step_attention(a, s_prev):
    """
    Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights
    "alphas" and the hidden states "a" of the Bi-LSTM.

    Arguments:
    a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)
    s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)

    Returns:
    context -- context vector, input of the next (post-attetion) LSTM cell
    """

    ### START CODE HERE ###
    # Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states "a" (≈ 1 line)
    s_prev = repeator(s_prev)
    # Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)
    concat = concatenator([s_prev,a])
    # Use densor1 to propagate concat through a small fully-connected neural network to compute the "intermediate energies" variable e. (≈1 lines)
    e = densor1(concat)
    # Use densor2 to propagate e through a small fully-connected neural network to compute the "energies" variable energies. (≈1 lines)
    energies = densor2(e)
    # Use "activator" on "energies" to compute the attention weights "alphas" (≈ 1 line)
    alphas = activator(energies)
    # Use dotor together with "alphas" and "a" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)
    context = dotor([alphas, a])
    ### END CODE HERE ###

    return context


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
The last step is to define all your inputs and outputs to fit the model:
- You already have X of shape $(m = 10000, T_x = 30)$ containing the training examples.
- You need to create s0 and c0 to initialize your post_activation_LSTM_cell with 0s.
- Given the model() you coded, you need the "outputs" to be a list of 11 elements of shape (m, T_y). So that: outputs[i][0], ..., outputs[i][Ty] represent the true labels (characters) corresponding to the $i^{th}$ training example (X[i]). More generally, outputs[i][j] is the true label of the $j^{th}$ character in the $i^{th}$ training example.
Can you write Python code for it?

s0 = np.zeros((m, n_s))
c0 = np.zeros((m, n_s))
outputs = list(Yoh.swapaxes(0,1))


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:

Thus, dense_2_acc_8: 0.89 means that you are predicting the 7th character of the output correctly 89% of the time in the current batch of data.

We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)
Can you write Python code for it?


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character.
3 - Visualizing Attention (Optional / Ungraded)
Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can  visualize what part of the output is looking at what part of the input.
Consider the task of translating "Saturday 9 May 2018" to "2018-05-09". If we visualize the computed $\alpha^{\langle t, t' \rangle}$ we get this:

 Figure 8: Full Attention Map

Notice how the output ignores the "Saturday" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's "18" in order to generate "2018."
3.1 - Getting the activations from the network
Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\alpha^{\langle t, t' \rangle}$.
To figure out where the attention values are located, let's start by printing a summary of the model .
Can you write Python code for it?


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You will be able to check the expected output of one_step_attention() after you've coded the model() function.
Exercise: Implement model() as explained in figure 2 and the text above. Again, we have defined global layers that will share weights to be used in model().
Can you write Python code for it?

n_a = 32
n_s = 64
post_activation_LSTM_cell = LSTM(n_s, return_state = True)
output_layer = Dense(len(machine_vocab), activation=softmax)


I am working on the file "Sequence_Models/Neural+machine+translation+with+attention+-+v3.ipynb".
The first task is:
You've loaded:
- dataset: a list of tuples of (human readable date, machine readable date)
- human_vocab: a python dictionary mapping all characters used in the human readable dates to an integer-valued index
- machine_vocab: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent with human_vocab.
- inv_machine_vocab: the inverse dictionary of machine_vocab, mapping from indices back to characters.
Let's preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since "YYYY-MM-DD" is 10 characters long).
Can you write Python code for it?

Tx = 30
Ty = 10
X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)

print("X.shape:", X.shape)
print("Y.shape:", Y.shape)
print("Xoh.shape:", Xoh.shape)
print("Yoh.shape:", Yoh.shape)

from itertools import islice

def take(n, iterable):
    "Return first n items of the iterable as a list"
    return list(islice(iterable, n))
n_items = take(5, human_vocab.items())
n_items


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
Do you notice anything surprising? It is astonishing how these results reflect certain unhealthy gender stereotypes. For example, "computer" is closer to "man" while "literature" is closer to "woman". Ouch!
We'll see below how to reduce the bias of these vectors, using an algorithm due to Boliukbasi et al., 2016. Note that some word pairs such as "actor"/"actress" or "grandmother"/"grandfather" should remain gender specific, while other words such as "receptionist" or "technology" should be neutralized, i.e. not be gender-related. You will have to treat these two type of words differently when debiasing.
3.1 - Neutralize bias for non-gender specific words
The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which we'll call $g_{\perp}$. In linear algebra, we say that the 49 dimensional $g_{\perp}$ is perpendicular (or "othogonal") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$.
Even though $g_{\perp}$ is 49 dimensional, given the limitations of what we can draw on a screen, we illustrate it using a 1 dimensional axis below.

 Figure 2: The word vector for "receptionist" represented before and after applying the neutralize operation.

Exercise: Implement neutralize() to remove the bias of words such as "receptionist" or "scientist". Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$:
$$e^{bias_component} = \frac{e \cdot g}{||g||_2^2} * g\tag{2}$$
$$e^{debiased} = e - e^{bias_component}\tag{3}$$
If you are an expert in linear algebra, you may recognize $e^{bias_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert...
Can you write Python code for it?

def neutralize(word, g, word_to_vec_map):
    """
    Removes the bias of "word" by projecting it on the space orthogonal to the bias axis.
    This function ensures that gender neutral words are zero in the gender subspace.

    Arguments:
        word -- string indicating the word to debias
        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)
        word_to_vec_map -- dictionary mapping words to their corresponding vectors.

    Returns:
        e_debiased -- neutralized word vector representation of the input "word"
    """

    ### START CODE HERE ###
    # Select word vector representation of "word". Use word_to_vec_map. (≈ 1 line)
    e = None

    # Compute e_biascomponent using the formula give above. (≈ 1 line)
    e_biascomponent = None

    # Neutralize e by substracting e_biascomponent from it
    # e_debiased should be equal to its orthogonal projection. (≈ 1 line)
    e_debiased = None
    ### END CODE HERE ###

    return e_debiased

e = "receptionist"
print("cosine similarity between " + e + " and g, before neutralizing: ", cosine_similarity(word_to_vec_map["receptionist"], g))

e_debiased = neutralize("receptionist", g, word_to_vec_map)
print("cosine similarity between " + e + " and g, after neutralizing: ", cosine_similarity(e_debiased, g))


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
Expected Output:

            **cosine_similarity(father, mother)** =

         0.890903844289

            **cosine_similarity(ball, crocodile)** =

         0.274392462614

            **cosine_similarity(france - paris, rome - italy)** =

         -0.675147930817

After you get the correct expected output, please feel free to modify the inputs and measure the cosine similarity between other pairs of words! Playing around the cosine similarity of other inputs will give you a better sense of how word vectors behave.
2 - Word analogy task
In the word analogy task, we complete the sentence "a is to b as c is to ____". An example is  'man is to woman as king is to queen' . In detail, we are trying to find a word d, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner: $e_b - e_a \approx e_d - e_c$. We will measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity.
Exercise: Complete the code below to be able to perform word analogies!
Can you write Python code for it?

# GRADED FUNCTION: complete_analogy

def complete_analogy(word_a, word_b, word_c, word_to_vec_map):
    """
    Performs the word analogy task as explained above: a is to b as c is to ____.

    Arguments:
    word_a -- a word, string
    word_b -- a word, string
    word_c -- a word, string
    word_to_vec_map -- dictionary that maps words to their corresponding vectors.

    Returns:
    best_word --  the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity
    """

    # convert words to lower case
    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()

    ### START CODE HERE ###
    # Get the word embeddings v_a, v_b and v_c (≈1-3 lines)
    e_a, e_b, e_c = word_to_vec_map[word_a], word_to_vec_map[word_b], word_to_vec_map[word_c]
    ### END CODE HERE ###

    words = word_to_vec_map.keys()
    max_cosine_sim = -100              # Initialize max_cosine_sim to a large negative number
    best_word = None                   # Initialize best_word with None, it will help keep track of the word to output

    # loop over the whole word vector set
    for w in words:
        # to avoid best_word being one of the input words, pass on them.
        if w in [word_a, word_b, word_c] :
            continue

        ### START CODE HERE ###
        # Compute cosine similarity between the vector (e_b - e_a) and the vector ((w's vector representation) - e_c)  (≈1 line)
        cosine_sim = cosine_similarity(e_b-e_a, word_to_vec_map[w]-e_c)

        # If the cosine_sim is more than the max_cosine_sim seen so far,
            # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)
        if cosine_sim > max_cosine_sim:
            max_cosine_sim = cosine_sim
            best_word = w
        ### END CODE HERE ###

    return best_word


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
Expected Output:

            **italy -> italian** ::

         spain -> spanish

            **india -> delhi** ::

         japan -> tokyo

            **man -> woman ** ::

         boy -> girl

            **small -> smaller ** ::

         large -> larger

Once you get the correct expected output, please feel free to modify the input cells above to test your own analogies. Try to find some other analogy pairs that do work, but also find some where the algorithm doesn't give the right answer: For example, you can try small->smaller as big->?.
Congratulations!
You've come to the end of this assignment. Here are the main points you should remember:

Cosine similarity a good way to compare similarity between pairs of word vectors. (Though L2 distance works too.)
For NLP applications, using a pre-trained set of word vectors from the internet is often a good way to get started.

Even though you have finished the graded portions, we recommend you take a look too at the rest of this notebook.
Congratulations on finishing the graded portions of this notebook!
3 - Debiasing word vectors (OPTIONAL/UNGRADED)
In the following exercise, you will examine gender biases that can be reflected in a word embedding, and explore algorithms for reducing the bias. In addition to learning about the topic of debiasing, this exercise will also help hone your intuition about what word vectors are doing. This section involves a bit of linear algebra, though you can probably complete it even without being expert in linear algebra, and we encourage you to give it a shot. This portion of the notebook is optional and is not graded.
Lets first see how the GloVe word embeddings relate to gender. You will first compute a...
Can you write Python code for it?

g = word_to_vec_map['woman'] - word_to_vec_map['man']
print(g)


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
Now, you will consider the cosine similarity of different words with $g$. Consider what a positive value of similarity means vs a negative cosine similarity.
Can you write Python code for it?

print ('List of names and their similarities with constructed vector:')

# girls and boys name
name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']

for w in name_list:
    print (w, cosine_similarity(word_to_vec_map[w], g))


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
Run the cell below to test your code, this may take 1-2 minutes.
Can you write Python code for it?

triads_to_try = [('italy', 'italian', 'spain'), ('india', 'delhi', 'japan'), ('man', 'woman', 'boy'), ('small', 'smaller', 'large')]
for triad in triads_to_try:
    print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad,word_to_vec_map)))


I am working on the file "Sequence_Models/Operations+on+word+vectors+-+v2.ipynb".
The first task is:
You've loaded:
- words: set of words in the vocabulary.
- word_to_vec_map: dictionary mapping words to their GloVe vector representation.
You've seen that one-hot vectors do not do a good job cpaturing what words are similar. GloVe vectors provide much more useful information about the meaning of individual words. Lets now see how you can use GloVe vectors to decide how similar two words are.
1 - Cosine similarity
To measure how similar two words are, we need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows:
$$\text{CosineSimilarity(u, v)} = \frac {u . v} {||u||_2 ||v||_2} = cos(\theta) \tag{1}$$
where $u.v$ is the dot product (or inner product) of two vectors, $||u||_2$ is the norm (or length) of the vector $u$, and $\theta$ is the angle between $u$ and $v$. This similarity depends on the angle between $u$ and $v$. If $u$ and $v$ are very similar, their cosine similarity will be close to 1; if they are dissimilar, the cosine similarity will take a smaller value.

 Figure 1: The cosine of the angle between two vectors is a measure of how similar they are

Exercise: Implement the function cosine_similarity() to evaluate similarity between word vectors.
Reminder: The norm of $u$ is defined as $ ||u||2 = \sqrt{\sum{i=1}^{n} u_i^2}$
Can you write Python code for it?

# GRADED FUNCTION: cosine_similarity

def cosine_similarity(u, v):
    """
    Cosine similarity reflects the degree of similariy between u and v

    Arguments:
        u -- a word vector of shape (n,)
        v -- a word vector of shape (n,)

    Returns:
        cosine_similarity -- the cosine similarity between u and v defined by the formula above.
    """

    distance = 0.0

    ### START CODE HERE ###
    # Compute the dot product between u and v (≈1 line)
    dot = np.dot(u,v)
    # Compute the L2 norm of u (≈1 line)
    norm_u = np.sqrt(np.sum(u*u))

    # Compute the L2 norm of v (≈1 line)
    norm_v = np.sqrt(np.sum(v*v))
    # Compute the cosine similarity defined by formula (1) (≈1 line)
    cosine_similarity = dot/(norm_u*norm_v)
    ### END CODE HERE ###

    return cosine_similarity

father = word_to_vec_map["father"]
mother = word_to_vec_map["mother"]
ball = word_to_vec_map["ball"]
crocodile = word_to_vec_map["crocodile"]
france = word_to_vec_map["france"]
italy = word_to_vec_map["italy"]
paris = word_to_vec_map["paris"]
rome = word_to_vec_map["rome"]

print("cosine_similarity(father, mother) = ", cosine_similarity(father, mother))
print("cosine_similarity(ball, crocodile) = ",cosine_similarity(ball, crocodile))
print("cosine_similarity(france - paris, rome - italy) = ",cosine_similarity(france - paris, rome - italy))


I am working on the file "Sequential and Binary Search in Python.ipynb".
The first task is:
We can see this returns a boolean answer of False, indicating that the integer isn't present in the array.
Below is another example where the answer is True.
Can you write Python code for it?

# Finding a single integer in an array of integers using Python's `in`
# operator

11 in [3,5,6,9,12,11]


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
2.2 Python变量赋值与操作
多变量赋值（不推荐）：
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
Python同样拥有增量赋值方法：
Can you write Python code for it?

a = 42
a += 1
print a
a -= 1
a *= 2
print a


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
世间真理42，以及
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
但是如果需要用户在电脑上输入一些字符怎么办？反复输入和打印字符串也并不方便。
Keep your code DRY (Don't Repeat Yourself)
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
其他操作 —— Join，Split，Strip, Upper, Lower
Can you write Python code for it?

s = "a\tb\tc\td"
print s
l = s.split('\t')
print l
snew = ','.join(l)
print snew
line = '\t Blabla \t \n'
print line.strip()
print line.lstrip()
print line.rstrip()
salpha = 'Abcdefg'
print salpha.upper()
print salpha.lower()

#isupper islower isdigit isalpha


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
加号（+）用于字符串连接运算，星号（*）用来重复字符串。
Can you write Python code for it?

teststr,stringback="Clojure is","cool"
print '-'*20
print teststr+stringback
print teststr*3
print '-'*20


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
复制：
Can you write Python code for it?

c = "Hello world"
d = c[:]        #复制字符串，赋值给d
del c           #删除原字符串
print d         #d 字符串依然可用
e = d[0:4]
print e         #新构造一个截取d字符串部分所组成的串
f = e
print id(f)
print id(e)
del e
print id(f),f        #仍然可用


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
如果需要标识注释，请以# 符号开始一段语句（这与大部分脚本语言和unix-shell语言一样）。从# 开始直到一行结束的内容都是注释。
Can you write Python code for it?

#Skip these comments.
print "These lines can be run."
print '''
Whatever.
'''


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
字符串。
用print打印任意我们想要的内容：
（一个对象在REPL中直接回车和被print分别输出的是他的__repr__和__str__方法对应的字符串）
Can you write Python code for it?

print "Scala|Python|C"
print 42
print repr(42),str(42)


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
字符串复杂效果举例：
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
布尔值：
Can you write Python code for it?

print not True
print not False
print 40 < 42 <= 100
print not 40 > 30
print 40 > 30 and 40 < 42 <= 100
print 40 < 30  or 40 < 42 <=100


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
当然你也可以为多个对象指定多个变量，例如：
Can you write Python code for it?

x,y,z = "So long","and thanks for all","the fish"
print x,y,z


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
普通floor除法：
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
更多操作：

查阅dir('s')
查阅codecs
查阅re（regexp，正则表达式）

2.5 初识Python可变类型（Mutables）

字符串（string）：不可变
字节数组（bytearray）：可变
Can you write Python code for it?

s = "string"
print type(s)

s[3] = "o"

s = "String"
sba = bytearray(s)
sba[3] = "o"
print sba


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
美化打印：
Can you write Python code for it?

pystr,pystring="Clojure is","cool"
print '-'*20
print pystr,pystring
print '-'*20
print pystr+'\t'+pystring
print '-'*20

pystr = "Clojure"
pystring = "cool"
yastr = "LISP"
yastring = "wonderful "
print('Python\'s Format I : {0} is {1}'.format(pystr,pystring))
print 'Python\'s Format II: {language} is {description}'.\
format(language='Scala',description='awesome') #使用\接续换行
print 'C Style Print: %s is %s'%(yastr,yastring)


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
赋值的语法糖（Syntax Sugar）：
Can you write Python code for it?

x,y = "So long",42
x,y = y,x
print x
print y


I am working on the file "Series_0_Python_Tutorials/S0EP1_Python_Basics.ipynb".
The first task is:
超大整数，或者试试
Can you write Python code for it?

"The answer to life,the universe,and everything is ?"


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
* 告诉Python即将传入的参数Your_choice不是单独一个序列，而是把Your_choice中的每一项作为参数
从下面的字典里按值来排序，取到值最大或者值最小的那条记录：
Can you write Python code for it?

Base_Damage={'Blade Master':48,'Death King':65,'Tauren Chieftain':51}
print zip(Base_Damage.itervalues(),Base_Damage.iterkeys())
max_Damage=max(zip(Base_Damage.itervalues(),Base_Damage.iterkeys()))
min_Damage=min(zip(Base_Damage.itervalues(),Base_Damage.iterkeys()))

print max_Damage,min_Damage


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
1.2 使用中括号进行列表解析
回忆enumerate
任务：把一个list里的元素和索引号找出来，更新回原来的list中去
循环操作可变类型
Can you write Python code for it?

Lord_of_ring = ['Ainur','Dragons','Dwarves','Elves','Ents','Hobbits','Men','Orcs']

print type(enumerate(Lord_of_ring)),enumerate(Lord_of_ring)

for idx,element in enumerate(Lord_of_ring):
    Lord_of_ring[idx] ="{0}:{1}".format(idx,element)

print Lord_of_ring


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
1.3 使用小括号进行生成器
使用小括号做Comprehension返回生成器对象，占用O(1)内存空间：
Can you write Python code for it?

num=range(0,20)

simple_generator=(x**2 for x in num if x > 0)

print simple_generator

for element in simple_generator:
    print element,


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
1.4 使用大括号解析得到集合或字典
使用小括号解析并不会返回一个不可变元组而是生成器，是一个需要强记的规则；然而大括号解析式就普通了许多。
Can you write Python code for it?

x = range(10)

print { i for i in x if i%2==0 }
print { idx:i**2 for idx,i in enumerate(x) if i%2==0 }


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
3 变量之变：深浅拷贝
深浅拷贝：关系到变量的正确修改与复制
变量的属性：

身份：就像身份证（或者内存地址）那样，id()
属性：表示变量的类型，type()或者isInstance()确认
值：这个地址存的数据，通过与名字绑定的方法来读取

3.1 浅拷贝

完全切片(Slicing)操作[:]
工厂函数，比如list(),tuple()
copy中的copy
Can you write Python code for it?

import copy
unit = ['name',['Base_Damage',65.00]]
my_hero = unit[:]               #使用切片拷贝
your_hero = list(unit)          #使用工厂方法拷贝
its_hero = copy.copy(unit)      #使用浅拷贝
print [id(x) for x in unit,my_hero,your_hero,its_hero]
print [id(x[0]) for x in unit,my_hero,your_hero,its_hero]
print [id(x[1]) for x in unit,my_hero,your_hero,its_hero]


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
3.3 不想Debug太烦？

规避修改复杂可变类型内容，维护唯一拷贝
规避使用可变类型
理解Clojure

4 异常处理：Try-Except-Else-Finally
会犯错的是人，能原谅人的是……

try
except
finally

Python允许程序在运行当中检测错误。
每检测到一个错误，Python解释器就引发一个异常并报出详细的错误信息：
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
4.3 异常处理：完全体
通过try-except-else-finally来感受异常处理的完全体吧！

try 下面是可能有异常的代码块
except 下面是对异常的处理
else 下面是在并没有异常的时候执行的代码块
finally 下面是收尾工作，无论是否有异常都执行

并用另一种方式记录异常。
Can you write Python code for it?

import sys

try:
    x = 1/0
    y = range(10)[-1]
except Exception,error:
    x = 0
    y = 9
    print 'X,Y is corrected.'
    info = sys.exc_info()
else:
    print 'Catch no exceptions. Great!'
finally:
    z = x + y
    print z**2 + x**2 + y**2
    print 'Finished'

print '\n',error,'\n',info[0],'\n',info[1],'\n',info[2],info[2].tb_lineno


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
为什么相互影响了呢？
Can you write Python code for it?

print [id(x[0]) for x in unit,my_hero,your_hero,its_hero]
print [id(x[1]) for x in unit,my_hero,your_hero,its_hero]


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
多重解析
Can you write Python code for it?

print [(x+1,y+1) for x in xrange(4) for y in xrange(4)]
print [(x+1,y+1) for x in xrange(4) for y in xrange(4) if y<x]
print [(x+1,y+1) for x in xrange(4) for y in xrange(x)]


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
字典也是Iterable:
Can you write Python code for it?

language={"Scala":"Martin Odersky",\
          "Clojure":"Richy Hickey",\
          "C":"Dennis Ritchie",\
          "Standard ML":"Robin Milner"}

['{0:<12} created by {1:<15}'.format(la,ua)\
 for la,ua in language.iteritems()]


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
带默认值的参数叫keyword arguments(kargs):
Can you write Python code for it?

def triplesum_default(a=0,b=0,c=0,*args):
    return a*100+b*10+c

def ntuplesum_default(*args):
    sum = 0
    for i in args:
        sum*=10
        sum+=i
    return sum

print triplesum_default(*[1,2,3,4])
print ntuplesum_default(*[1,2,3,5])
print ntuplesum_default(1,2,3,5,6,7,8)
print triplesum_default(*[1,3])
print triplesum_default(**{'b':2,'c':3,'a':1})
print triplesum_default(**{'c':3,'a':1})


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
我们做的事情在数学定义上看来像是什么呢？
${x|x \in {0,1,2,....,9}, s.t. x\%2==0 }$
Can you write Python code for it?


I am working on the file "Series_0_Python_Tutorials/S0EP3_Coding_Pythonically.ipynb".
The first task is:
注意生成字典时使用key:value的形式就可以了。
2 花样传参：zip与星号操作

zip: 拉链函数
*: 经常和zip在一起，用于传递参数。
**: 用于传递关键字型参数

2.1 Zip(拉链)

enumerate: 返回生成器，生成器每次给出下标和Iterable的内容
sorted: 返回列表，可以进行排序
zip: 把多个长度相同的列表当成数据列组成的数据表，返回一个包含着元组的列表，每个元组是数据表中的一行
Can you write Python code for it?

war3_char = ['Orc','Humans','Undead','Night Elves']
dota_hero = ['Blade Master','Archmage','Death King','Demon Hunter']
Your_choice=zip(war3_char,dota_hero)
print Your_choice