I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
I will use the following function to track our training vocabulary, which goes through all our text and counts the occurance of the contained words.
Can you write Python code for it?

def build_vocab(sentences, verbose =  True):
    """
    :param sentences: list of list of words
    :return: dictionary of words and their count
    """
    vocab = {}
    for sentence in tqdm(sentences, disable = (not verbose)):
        for word in sentence:
            try:
                vocab[word] += 1
            except KeyError:
                vocab[word] = 1
    return vocab


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Lets load our data
Can you write Python code for it?

train = pd.read_csv("../input/train.csv")
test = pd.read_csv("../input/test.csv")
print("Train shape : ",train.shape)
print("Test shape : ",test.shape)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Next I define a function that checks the intersection between our vocabulary and the embeddings. It will output a list of out of vocabulary (oov) words that we can use to improve our preprocessing
Can you write Python code for it?

import operator

def check_coverage(vocab,embeddings_index):
    a = {}
    oov = {}
    k = 0
    i = 0
    for word in tqdm(vocab):
        try:
            a[word] = embeddings_index[word]
            k += vocab[word]
        except:

            oov[word] = vocab[word]
            i += vocab[word]
            pass

    print('Found embeddings for {:.2%} of vocab'.format(len(a) / len(vocab)))
    print('Found embeddings for  {:.2%} of all text'.format(k / (k + i)))
    sorted_x = sorted(oov.items(), key=operator.itemgetter(1))[::-1]

    return sorted_x

oov = check_coverage(vocab,embeddings_index)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Nice! Another 3% increase. Now as much as with handling the puntuation, but every bit helps. Lets check the oov words
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Nice! We were able to increase our embeddings ratio from 24% to 57% by just handling punctiation. Ok lets check on thos oov words.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Ok now we  take care of common misspellings when using american/ british vocab and replacing a few "modern" words with "social media" for this task I use a multi regex script I found some time ago on stack overflow. Additionally we will simply remove the words "a","to","and" and "of" since those have obviously been downsampled when training the GoogleNews Embeddings.
Can you write Python code for it?

def _get_mispell(mispell_dict):
    mispell_re = re.compile('(%s)' % '|'.join(mispell_dict.keys()))
    return mispell_dict, mispell_re

mispell_dict = {'colour':'color',
                'centre':'center',
                'didnt':'did not',
                'doesnt':'does not',
                'isnt':'is not',
                'shouldnt':'should not',
                'favourite':'favorite',
                'travelling':'traveling',
                'counselling':'counseling',
                'theatre':'theater',
                'cancelled':'canceled',
                'labour':'labor',
                'organisation':'organization',
                'wwii':'world war 2',
                'citicise':'criticize',
                'instagram': 'social medium',
                'whatsapp': 'social medium',
                'snapchat': 'social medium'

                }
mispellings, mispellings_re = _get_mispell(mispell_dict)

def replace_typical_misspell(text):
    def replace(match):
        return mispellings[match.group(0)]

    return mispellings_re.sub(replace, text)

train["question_text"] = train["question_text"].progress_apply(lambda x: replace_typical_misspell(x))
sentences = train["question_text"].progress_apply(lambda x: x.split())
to_remove = ['a','to','of','and']
sentences = [[word for word in sentence if not word in to_remove] for sentence in tqdm(sentences)]
vocab = build_vocab(sentences)

oov = check_coverage(vocab,embeddings_index)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
On first place there is "to". Why? Simply because "to" was removed when the GoogleNews Embeddings were trained. We will fix this later, for now we take care about the splitting of punctuation as this also seems to be a Problem. But what do we do with the punctuation then - Do we want to delete or consider as a token? I would say: It depends. If the token has an embedding, keep it, if it doesn't we don't need it anymore. So lets check:
Can you write Python code for it?

'?' in embeddings_index

'&' in embeddings_index


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
Ouch only 24% of our vocabulary will have embeddings, making 21% of our data more or less useless. So lets have a look and start improving. For this we can easily have a look at the top oov words.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
So lets populate the vocabulary and display the first 5 elements and their count. Note that now we can use progess_apply to see progress bar
Can you write Python code for it?

sentences = train["question_text"].progress_apply(lambda x: x.split()).values
vocab = build_vocab(sentences)
print({k: vocab[k] for k in list(vocab)[:5]})


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
We see that although we improved on the amount of embeddings found for all our text from 89% to 99%. Lets check the oov words again
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/how-to-preprocessing-when-using-embeddings.ipynb".
The first task is:
hmm why is "##" in there? Simply because as a reprocessing all numbers bigger tha 9 have been replaced by hashs. I.e. 15 becomes ## while 123 becomes ### or 15.80€ becomes ##.##€. So lets mimic this preprocessing step to further improve our embeddings coverage
Can you write Python code for it?

import re

def clean_numbers(x):

    x = re.sub('[0-9]{5,}', '#####', x)
    x = re.sub('[0-9]{4}', '####', x)
    x = re.sub('[0-9]{3}', '###', x)
    x = re.sub('[0-9]{2}', '##', x)
    return x

train["question_text"] = train["question_text"].progress_apply(lambda x: clean_numbers(x))
sentences = train["question_text"].progress_apply(lambda x: x.split())
vocab = build_vocab(sentences)

oov = check_coverage(vocab,embeddings_index)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
Better, but we lost a bit of information on the other embeddings.

Therer are words known that are known with upper letters and unknown without. Let us fix that :
- word.lower() takes the embedding of word if word.lower() doesn't have an embedding
Can you write Python code for it?

def add_lower(embedding, vocab):
    count = 0
    for word in vocab:
        if word in embedding and word.lower() not in embedding:
            embedding[word.lower()] = embedding[word]
            count += 1
    print(f"Added {count} words to embedding")

print("Glove : ")
add_lower(embed_glove, vocab)
print("Paragram : ")
add_lower(embed_paragram, vocab)
print("FastText : ")
add_lower(embed_fasttext, vocab)

print("Glove : ")
oov_glove = check_coverage(vocab_low, embed_glove)
print("Paragram : ")
oov_paragram = check_coverage(vocab_low, embed_paragram)
print("FastText : ")
oov_fasttext = check_coverage(vocab_low, embed_fasttext)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
FastText does not understand contractions

We use the map to replace them
Can you write Python code for it?

def clean_contractions(text, mapping):
    specials = ["’", "‘", "´", "`"]
    for s in specials:
        text = text.replace(s, "'")
    text = ' '.join([mapping[t] if t in mapping else t for t in text.split(" ")])
    return text

df['treated_question'] = df['lowered_question'].apply(lambda x: clean_contractions(x, contraction_mapping))

vocab = build_vocab(df['treated_question'])
print("Glove : ")
oov_glove = check_coverage(vocab, embed_glove)
print("Paragram : ")
oov_paragram = check_coverage(vocab, embed_paragram)
print("FastText : ")
oov_fasttext = check_coverage(vocab, embed_fasttext)


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
First faults appearing are :

Contractions
Words with punctuation in them

Let us correct that.

Contractions
Can you write Python code for it?

contraction_mapping = {"ain't": "is not", "aren't": "are not","can't": "cannot", "'cause": "because", "could've": "could have", "couldn't": "could not", "didn't": "did not",  "doesn't": "does not", "don't": "do not", "hadn't": "had not", "hasn't": "has not", "haven't": "have not", "he'd": "he would","he'll": "he will", "he's": "he is", "how'd": "how did", "how'd'y": "how do you", "how'll": "how will", "how's": "how is",  "I'd": "I would", "I'd've": "I would have", "I'll": "I will", "I'll've": "I will have","I'm": "I am", "I've": "I have", "i'd": "i would", "i'd've": "i would have", "i'll": "i will",  "i'll've": "i will have","i'm": "i am", "i've": "i have", "isn't": "is not", "it'd": "it would", "it'd've": "it would have", "it'll": "it will", "it'll've": "it will have","it's": "it is", "let's": "let us", "ma'am": "madam", "mayn't": "may not", "might've": "might have","mightn't": "might not","mightn't've": "might not have", "must've": "must have", "mustn't": "must not", "mustn't've": "must not have", "needn't": "need not", "needn't've": "need not have","o'clock": "of the clock", "oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not", "sha'n't": "shall not", "shan't've": "shall not have", "she'd": "she would", "she'd've": "she would have", "she'll": "she will", "she'll've": "she will have", "she's": "she is", "should've": "should have", "shouldn't": "should not", "shouldn't've": "should not have", "so've": "so have","so's": "so as", "this's": "this is","that'd": "that would", "that'd've": "that would have", "that's": "that is", "there'd": "there would", "there'd've": "there would have", "there's": "there is", "here's": "here is","they'd": "they would", "they'd've": "they would have", "they'll": "they will", "they'll've": "they will have", "they're": "they are", "they've": "they have", "to've": "to have", "wasn't": "was not", "we'd": "we would", "we'd've": "we would have", "we'll": "we will", "we'll've": "we will have", "we're": "we are", "we've": "we have", "weren't": "were not",...


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
Now, let us deal with special characters
Can you write Python code for it?

punct = "/-'?!.,#$%\'()*+-/:;<=>@[\\]^_`{|}~" + '""“”’' + '∞θ÷α•à−β∅³π‘₹´°£€\×™√²—–&'

def unknown_punct(embed, punct):
    unknown = ''
    for p in punct:
        if p not in embed:
            unknown += p
            unknown += ' '
    return unknown

print("Glove :")
print(unknown_punct(embed_glove, punct))
print("Paragram :")
print(unknown_punct(embed_paragram, punct))
print("FastText :")
print(unknown_punct(embed_fasttext, punct))


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
Vocabulary and Coverage functions

Again, check Dieter's work if you haven't, those are his.
Can you write Python code for it?

def build_vocab(texts):
    sentences = texts.apply(lambda x: x.split()).values
    vocab = {}
    for sentence in sentences:
        for word in sentence:
            try:
                vocab[word] += 1
            except KeyError:
                vocab[word] = 1
    return vocab

def check_coverage(vocab, embeddings_index):
    known_words = {}
    unknown_words = {}
    nb_known_words = 0
    nb_unknown_words = 0
    for word in vocab.keys():
        try:
            known_words[word] = embeddings_index[word]
            nb_known_words += vocab[word]
        except:
            unknown_words[word] = vocab[word]
            nb_unknown_words += vocab[word]
            pass

    print('Found embeddings for {:.2%} of vocab'.format(len(known_words) / len(vocab)))
    print('Found embeddings for  {:.2%} of all text'.format(nb_known_words / (nb_known_words + nb_unknown_words)))
    unknown_words = sorted(unknown_words.items(), key=operator.itemgetter(1))[::-1]

    return unknown_words


I am working on the file "DEEP LEARNING/NLP/Kaggle Quora Insincere Questions Classification/improve-your-score-with-some-text-preprocessing.ipynb".
The first task is:
What's wrong ?
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
And we can see those same characters from above, encoded as integers.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Getting the best model
To set your hyperparameters to get the best performance, you'll want to watch the training and validation losses. If your training loss is much lower than the validation loss, you're overfitting. Increase regularization (more dropout) or use a smaller network. If the training and validation losses are close, you're underfitting so you can increase the size of the network.
Hyperparameters
Here are the hyperparameters for the network.
In defining the model:
* n_hidden - The number of units in the hidden layers.
* n_layers - Number of hidden LSTM layers to use.
We assume that dropout probability and learning rate will be kept at the default, in this example.
And in training:
* batch_size - Number of sequences running through the network in one pass.
* seq_length - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.
* lr - Learning rate for training
Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to where it originally came from.

Tips and Tricks
Monitoring Validation Loss vs. Training Loss
If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:

If your training loss is much lower than validation loss then this means the network might be overfitting. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.
If your training/validation loss are about equal then your model is underfitting. Increase the size of your model (either number of layers or the raw number of...
Can you write Python code for it?

# change the name, for saving multiple files
model_name = 'rnn_20_epoch.net'

checkpoint = {'n_hidden': net.n_hidden,
              'n_layers': net.n_layers,
              'state_dict': net.state_dict(),
              'tokens': net.chars}

with open(model_name, 'wb') as f:
    torch.save(checkpoint, f)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
If you implemented get_batches correctly, the above output should look something like
```
x
 [[25  8 60 11 45 27 28 73  1  2]
 [17  7 20 73 45  8 60 45 73 60]
 [27 20 80 73  7 28 73 60 73 65]
 [17 73 45  8 27 73 66  8 46 27]
 [73 17 60 12 73  8 27 28 73 45]
 [66 64 17 17 46  7 20 73 60 20]
 [73 76 20 20 60 73  8 60 80 73]
 [47 35 43  7 20 17 24 50 37 73]]
y
 [[ 8 60 11 45 27 28 73  1  2  2]
 [ 7 20 73 45  8 60 45 73 60 45]
 [20 80 73  7 28 73 60 73 65  7]
 [73 45  8 27 73 66  8 46 27 65]
 [17 60 12 73  8 27 28 73 45 27]
 [64 17 17 46  7 20 73 60 20 80]
 [76 20 20 60 73  8 60 80 73 17]
 [35 43  7 20 17 24 50 37 73 36]]
 ``
 although the exact numbers may be different. Check to make sure the data is shifted over one step fory`.

Defining the network with PyTorch
Below is where you'll define the network.
<img src="assets/charRNN.png" width=500px>
Next, you'll use PyTorch to define the architecture of the network. We start by defining the layers and operations we want. Then, define a method for the forward pass. You've also been given a method for predicting characters.
Model Structure
In __init__ the suggested structure is as follows:
* Create and store the necessary dictionaries (this has been done for you)
* Define an LSTM layer that takes as params: an input size (the number of characters), a hidden layer size n_hidden, a number of layers n_layers, a dropout probability drop_prob, and a batch_first boolean (True, since we are batching)
* Define a dropout layer with dropout_prob
* Define a fully-connected layer with params: input size n_hidden and output size (the number of characters)
* Finally, initialize the weights (again, this has been given)
Note that some parameters have been named and given in the __init__ function, and we use them and store them by doing something like self.drop_prob = drop_prob.

LSTM Inputs/Outputs
You can create a basic LSTM layer as follows
python
self.lstm = nn.LSTM(input_size, n_hidden, n_layers,
                            dropout=drop_prob, batch_first=True)
where input_size...
Can you write Python code for it?

# check if GPU is available
train_on_gpu = torch.cuda.is_available()
if(train_on_gpu):
    print('Training on GPU!')
else:
    print('No GPU available, training on CPU; consider making n_epochs very small.')

class CharRNN(nn.Module):

    def __init__(self, tokens, n_hidden=256, n_layers=2,
                               drop_prob=0.5, lr=0.001):
        super().__init__()
        self.drop_prob = drop_prob
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.lr = lr

        # creating character dictionaries
        self.chars = tokens
        self.int2char = dict(enumerate(self.chars))
        self.char2int = {ch: ii for ii, ch in self.int2char.items()}

        ## TODO: define the LSTM
        self.lstm = nn.LSTM(len(self.chars), n_hidden, n_layers,
                            dropout=drop_prob, batch_first=True)

        ## TODO: define a dropout layer
        self.dropout = nn.Dropout(drop_prob)

        ## TODO: define the final, fully-connected output layer
        self.fc = nn.Linear(n_hidden, len(self.chars))

    def forward(self, x, hidden):
        ''' Forward pass through the network.
            These inputs are x, and the hidden/cell state `hidden`. '''

        ## TODO: Get the outputs and the new hidden state from the lstm
        r_output, hidden = self.lstm(x, hidden)

        ## TODO: pass through a dropout layer
        out = self.dropout(r_output)

        # Stack up LSTM outputs using view
        # you may need to use contiguous to reshape the output
        out = out.contiguous().view(-1, self.n_hidden)

        ## TODO: put x through the fully-connected layer
        out = self.fc(out)

        # return the final output and the hidden state
        return out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to...


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Instantiating the model
Now we can actually train the network. First we'll create the network itself, with some given hyperparameters. Then, define the mini-batches sizes, and start training!
Can you write Python code for it?

# define and print the net
n_hidden=512
n_layers=2

net = CharRNN(chars, n_hidden, n_layers)
print(net)

batch_size = 128
seq_length = 100
n_epochs = 20 # start smaller if you are just testing initial behavior

# train the model
train(net, encoded, epochs=n_epochs, batch_size=batch_size, seq_length=seq_length, lr=0.001, print_every=10)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Load in Data
Then, we'll load the Anna Karenina text file and convert it into integers for our network to use.
Can you write Python code for it?

# open text file and read in data as `text`
with open('data/anna.txt', 'r') as f:
    text = f.read()


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Making training mini-batches
To train on this data, we also want to create mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:

In this example, we'll take the encoded characters (passed in as the arr parameter) and split them into multiple sequences, given by batch_size. Each of our sequences will be seq_length long.
Creating Batches
1. The first thing we need to do is discard some of the text so we only have completely full mini-batches.
Each batch contains $N \times M$ characters, where $N$ is the batch size (the number of sequences in a batch) and $M$ is the seq_length or number of time steps in a sequence. Then, to get the total number of batches, $K$, that we can make from the array arr, you divide the length of arr by the number of characters per batch. Once you know the number of batches, you can get the total number of characters to keep from arr, $N * M * K$.
2. After that, we need to split arr into $N$ batches.
You can do this using arr.reshape(size) where size is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences in a batch, so let's make that the size of the first dimension. For the second dimension, you can use -1 as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \times (M * K)$.
3. Now that we have this array, we can iterate through it to get our mini-batches.
The idea is each batch is a $N \times M$ window on the $N \times (M * K)$ array. For each subsequent batch, the window moves over by seq_length. We also want to create both the input and target arrays. Remember that the targets are just the inputs shifted over by one character. The way I like to do this window is use range to take steps of size n_steps from $0$ to arr.shape[1], the total number of tokens in each...
Can you write Python code for it?

def get_batches(arr, batch_size, seq_length):
    '''Create a generator that returns batches of size
       batch_size x seq_length from arr.

       Arguments
       ---------
       arr: Array you want to make batches from
       batch_size: Batch size, the number of sequences per batch
       seq_length: Number of encoded chars in a sequence
    '''

    batch_size_total = batch_size * seq_length
    # total number of batches we can make
    n_batches = len(arr)//batch_size_total

    # Keep only enough characters to make full batches
    arr = arr[:n_batches * batch_size_total]
    # Reshape into batch_size rows
    arr = arr.reshape((batch_size, -1))

    # iterate through the array, one sequence at a time
    for n in range(0, arr.shape[1], seq_length):
        # The features
        x = arr[:, n:n+seq_length]
        # The targets, shifted by one
        y = np.zeros_like(x)
        try:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, n+seq_length]
        except IndexError:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, 0]
        yield x, y


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Priming and generating text
Typically you'll want to prime the network so you can build up a hidden state. Otherwise the network will start out generating characters at random. In general the first bunch of characters will be a little rough since it hasn't built up a long history of characters to predict from.
Can you write Python code for it?

def sample(net, size, prime='The', top_k=None):

    if(train_on_gpu):
        net.cuda()
    else:
        net.cpu()

    net.eval() # eval mode

    # First off, run through the prime characters
    chars = [ch for ch in prime]
    h = net.init_hidden(1)
    for ch in prime:
        char, h = predict(net, ch, h, top_k=top_k)

    chars.append(char)

    # Now pass in the previous character and get a new one
    for ii in range(size):
        char, h = predict(net, chars[-1], h, top_k=top_k)
        chars.append(char)

    return ''.join(chars)

print(sample(net, 1000, prime='Anna', top_k=5))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/Char level RNN/Character_Level_RNN.ipynb".
The first task is:
Time to train
The train function gives us the ability to set the number of epochs, the learning rate, and other parameters.
Below we're using an Adam optimizer and cross entropy loss since we are looking at character class scores as output. We calculate the loss and perform backpropagation, as usual!
A couple of details about training:

Within the batch loop, we detach the hidden state from its history; this time setting it equal to a new tuple variable because an LSTM has a hidden state that is a tuple of the hidden and cell states.
We use clip_grad_norm_ to help prevent exploding gradients.
Can you write Python code for it?

def train(net, data, epochs=10, batch_size=10, seq_length=50, lr=0.001, clip=5, val_frac=0.1, print_every=10):
    ''' Training a network

        Arguments
        ---------

        net: CharRNN network
        data: text data to train the network
        epochs: Number of epochs to train
        batch_size: Number of mini-sequences per mini-batch, aka batch size
        seq_length: Number of character steps per mini-batch
        lr: learning rate
        clip: gradient clipping
        val_frac: Fraction of data to hold out for validation
        print_every: Number of steps for printing training and validation loss

    '''
    net.train()

    opt = torch.optim.Adam(net.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    # create training and validation data
    val_idx = int(len(data)*(1-val_frac))
    data, val_data = data[:val_idx], data[val_idx:]

    if(train_on_gpu):
        net.cuda()

    counter = 0
    n_chars = len(net.chars)
    for e in range(epochs):
        # initialize hidden state
        h = net.init_hidden(batch_size)

        for x, y in get_batches(data, batch_size, seq_length):
            counter += 1

            # One-hot encode our data and make them Torch tensors
            x = one_hot_encode(x, n_chars)
            inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

            if(train_on_gpu):
                inputs, targets = inputs.cuda(), targets.cuda()

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            h = tuple([each.data for each in h])

            # zero accumulated gradients
            net.zero_grad()

            # get the output from the model
            output, h = net(inputs, h)

            # calculate the loss and perform backprop
            loss = criterion(output, targets.view(batch_size*seq_length))
            loss.backward()
            #...


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
Implement an RNN using PyTorch's Module class. You may choose to use a GRU or an LSTM. To complete the RNN, you'll have to implement the following functions for the class:
 - __init__ - The initialize function.
 - init_hidden - The initialization function for an LSTM/GRU hidden state
 - forward - Forward propagation function.
The initialize function should create the layers of the neural network and save them to the class. The forward propagation function will use these layers to run forward propagation and generate an output and a hidden state.
The output of this model should be the last batch of word scores after a complete sequence has been processed. That is, for each input sequence of words, we only want to output the word scores for a single, most likely, next word.
Hints

Make sure to stack the outputs of the lstm to pass to your fully-connected layer, you can do this with lstm_output = lstm_output.contiguous().view(-1, self.hidden_dim)
You can get the last batch of word scores by shaping the output of the final, fully-connected layer like so:

```
reshape into (batch_size, seq_length, output_size)
output = output.view(batch_size, -1, self.output_size)
get last batch
out = output[:, -1]
```
Can you write Python code for it?

import torch.nn as nn

class RNN(nn.Module):

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5):
        """
        Initialize the PyTorch RNN Module
        :param vocab_size: The number of input dimensions of the neural network (the size of the vocabulary)
        :param output_size: The number of output dimensions of the neural network
        :param embedding_dim: The size of embeddings, should you choose to use them
        :param hidden_dim: The size of the hidden layer outputs
        :param dropout: dropout to add in between LSTM/GRU layers
        """
        super(RNN, self).__init__()
        # TODO: Implement function

        # set class variables

        # define model layers

    def forward(self, nn_input, hidden):
        """
        Forward propagation of the neural network
        :param nn_input: The input to the neural network
        :param hidden: The hidden state
        :return: Two Tensors, the output of the neural network and the latest hidden state
        """
        # TODO: Implement function

        # return one batch of output word scores and the hidden state
        return None, None

    def init_hidden(self, batch_size):
        '''
        Initialize the hidden state of an LSTM/GRU
        :param batch_size: The batch_size of the hidden state
        :return: hidden state of dims (n_layers, batch_size, hidden_dim)
        '''
        # Implement function

        # initialize hidden state with zero weights, and move to GPU if available

        return None

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_rnn(RNN, train_on_gpu)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
In this section, you'll build the components necessary to build an RNN by implementing the RNN Module and forward and backpropagation functions.
Check Access to GPU
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import torch

# Check for a GPU
train_on_gpu = torch.cuda.is_available()
if not train_on_gpu:
    print('No GPU found. Please use a GPU to train your neural network.')


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import problem_unittests as tests

int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Define forward and backpropagation
Use the RNN class you implemented to apply forward and back propagation. This function will be called, iteratively, in the training loop as follows:
loss = forward_back_prop(decoder, decoder_optimizer, criterion, inp, target)
And it should return the average loss over a batch and the hidden state returned by a call to RNN(inp, hidden). Recall that you can get this loss by computing it, as usual, and calling loss.item().
If a GPU is available, you should move your data to that GPU device, here.
Can you write Python code for it?

def forward_back_prop(rnn, optimizer, criterion, inp, target, hidden):
    """
    Forward and backward propagation on the neural network
    :param decoder: The PyTorch Module that holds the neural network
    :param decoder_optimizer: The PyTorch optimizer for the neural network
    :param criterion: The PyTorch loss function
    :param inp: A batch of input to the neural network
    :param target: The target output for the batch of input
    :return: The loss and the latest hidden state Tensor
    """

    # TODO: Implement Function

    # move data to GPU, if available

    # perform backpropagation and optimization

    # return the loss over a batch and the hidden state produced by our model
    return None, None

# Note that these tests aren't completely extensive.
# they are here to act as general checks on the expected outputs of your functions
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_forward_back_prop(RNN, forward_back_prop, train_on_gpu)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Generate a New Script
It's time to generate the text. Set gen_length to the length of TV script you want to generate and set prime_word to one of the following to start the prediction:
- "jerry"
- "elaine"
- "george"
- "kramer"
You can set the prime word to any word in our dictionary, but it's best to start with a name for generating a TV script. (You can also start with any other names you find in the original text file!)
Can you write Python code for it?

# run the cell multiple times to get different results!
gen_length = 400 # modify the length to your preference
prime_word = 'jerry' # name for starting the script

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
pad_word = helper.SPECIAL_WORDS['PADDING']
generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)
print(generated_script)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Hyperparameters
Set and train the neural network with the following parameters:
- Set sequence_length to the length of a sequence.
- Set batch_size to the batch size.
- Set num_epochs to the number of epochs to train for.
- Set learning_rate to the learning rate for an Adam optimizer.
- Set vocab_size to the number of unique tokens in our vocabulary.
- Set output_size to the desired size of the output.
- Set embedding_dim to the embedding dimension; smaller than the vocab_size.
- Set hidden_dim to the hidden dimension of your RNN.
- Set n_layers to the number of layers/cells in your RNN.
- Set show_every_n_batches to the number of batches at which the neural network should print progress.
If the network isn't getting the desired results, tweak these parameters and/or the layers in the RNN class.
Can you write Python code for it?

# Data params
# Sequence Length
sequence_length =   # of words in a sequence
# Batch Size
batch_size =

# data loader - do not change
train_loader = batch_data(int_text, sequence_length, batch_size)

# Training parameters
# Number of Epochs
num_epochs =
# Learning Rate
learning_rate =

# Model parameters
# Vocab size
vocab_size =
# Output size
output_size =
# Embedding Dimension
embedding_dim =
# Hidden Dimension
hidden_dim =
# Number of RNN Layers
n_layers =

# Show stats for every n number of batches
show_every_n_batches = 500


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Implement Pre-processing Functions
The first thing to do to any dataset is pre-processing.  Implement the following pre-processing functions below:
- Lookup Table
- Tokenize Punctuation
Lookup Table
To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:
- Dictionary to go from the words to an id, we'll call vocab_to_int
- Dictionary to go from the id to word, we'll call int_to_vocab
Return these dictionaries in the following tuple (vocab_to_int, int_to_vocab)
Can you write Python code for it?

import problem_unittests as tests

def create_lookup_tables(text):
    """
    Create lookup tables for vocabulary
    :param text: The text of tv scripts split into words
    :return: A tuple of dicts (vocab_to_int, int_to_vocab)
    """
    # TODO: Implement Function

    # return tuple
    return (None, None)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_create_lookup_tables(create_lookup_tables)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Neural Network Training
With the structure of the network complete and data ready to be fed in the neural network, it's time to train it.
Train Loop
The training loop is implemented for you in the train_decoder function. This function will train the network over all the batches for the number of epochs given. The model progress will be shown every number of batches. This number is set with the show_every_n_batches parameter. You'll set this parameter along with other parameters in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""

def train_rnn(rnn, batch_size, optimizer, criterion, n_epochs, show_every_n_batches=100):
    batch_losses = []

    rnn.train()

    print("Training for %d epoch(s)..." % n_epochs)
    for epoch_i in range(1, n_epochs + 1):

        # initialize hidden state
        hidden = rnn.init_hidden(batch_size)

        for batch_i, (inputs, labels) in enumerate(train_loader, 1):

            # make sure you iterate over completely full batches, only
            n_batches = len(train_loader.dataset)//batch_size
            if(batch_i > n_batches):
                break

            # forward, back prop
            loss, hidden = forward_back_prop(rnn, optimizer, criterion, inputs, labels, hidden)
            # record loss
            batch_losses.append(loss)

            # printing loss stats
            if batch_i % show_every_n_batches == 0:
                print('Epoch: {:>4}/{:<4}  Loss: {}\n'.format(
                    epoch_i, n_epochs, np.average(batch_losses)))
                batch_losses = []

    # returns a trained rnn
    return rnn


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Save your favorite scripts
Once you have a script that you like (or find interesting), save it to a text file!
Can you write Python code for it?

# save script to a text file
f =  open("generated_script_1.txt","w")
f.write(generated_script)
f.close()


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Next Chars pytorch/project-tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Train
In the next cell, you'll train the neural network on the pre-processed data.  If you have a hard time getting a good loss, you may consider changing your hyperparameters. In general, you may get better results with larger hidden and n_layer dimensions, but larger models take a longer time to train.

You should aim for a loss less than 3.5.

You should also experiment with different sequence lengths, which determine the size of the long range dependencies that a model can learn.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""

# create model and move to gpu if available
rnn = RNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5)
if train_on_gpu:
    rnn.cuda()

# defining loss and optimization functions for training
optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)
criterion = nn.CrossEntropyLoss()

# training the model
trained_rnn = train_rnn(rnn, batch_size, optimizer, criterion, num_epochs, show_every_n_batches)

# saving the trained model
helper.save_model('./save/trained_rnn', trained_rnn)
print('Model Trained and Saved')


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Check your work
With train, validation, and test fractions equal to 0.8, 0.1, 0.1, respectively, the final, feature data shapes should look like:
Feature Shapes:
Train set:       (20000, 200)
Validation set:     (2500, 200)
Test set:         (2500, 200)

DataLoaders and Batching
After creating training, test, and validation data, we can create DataLoaders for this data by following two steps:
1. Create a known format for accessing our data, using TensorDataset which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.
2. Create DataLoaders and batch our training, validation, and test Tensor datasets.
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
train_loader = DataLoader(train_data, batch_size=batch_size)
This is an alternative to creating a generator function for batching our data into full batches.
Can you write Python code for it?

import torch
from torch.utils.data import TensorDataset, DataLoader

# create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))
test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))

# dataloaders
batch_size = 50

# make sure the SHUFFLE your training data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)
valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)
test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)

# obtain one batch of training data
dataiter = iter(train_loader)
sample_x, sample_y = dataiter.next()

print('Sample input size: ', sample_x.size()) # batch_size, seq_length
print('Sample input: \n', sample_x)
print()
print('Sample label size: ', sample_y.size()) # batch_size
print('Sample label: \n', sample_y)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Data pre-processing
The first step when building a neural network model is getting your data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit.
You can see an example of the reviews data above. Here are the processing steps, we'll want to take:

We'll want to get rid of periods and extraneous punctuation.
Also, you might notice that the reviews are delimited with newline characters \n. To deal with those, I'm going to split the text into each review using \n as the delimiter.
Then I can combined all the reviews back together into one big string.

First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words.
Can you write Python code for it?

from string import punctuation

# get rid of punctuation
reviews = reviews.lower() # lowercase, standardize
all_text = ''.join([c for c in reviews if c not in punctuation])

# split by new lines and spaces
reviews_split = all_text.split('\n')
all_text = ' '.join(reviews_split)

# create a list of words
words = all_text.split()

words[:30]


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Encoding the labels
Our labels are "positive" or "negative". To use these labels in our network, we need to convert them to 0 and 1.

Exercise: Convert labels from positive and negative to 1 and 0, respectively, and place those in a new list, encoded_labels.
Can you write Python code for it?

# 1=positive, 0=negative label conversion
labels_split = labels.split('\n')
encoded_labels = np.array([1 if label == 'positive' else 0 for label in labels_split])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Encoding the words
The embedding lookup requires that we pass in integers to our network. The easiest way to do this is to create dictionaries that map the words in the vocabulary to integers. Then we can convert each of our reviews into integers so they can be passed into the network.

Exercise: Now you're going to encode the words with integers. Build a dictionary that maps words to integers. Later we're going to pad our input vectors with zeros, so make sure the integers start at 1, not 0.
Also, convert the reviews to integers and store the reviews in a new list called reviews_ints.
Can you write Python code for it?

# feel free to use this import
from collections import Counter

## Build a dictionary that maps words to integers
counts = Counter(words)
vocab = sorted(counts, key=counts.get, reverse=True)
vocab_to_int = {word: ii for ii, word in enumerate(vocab, 1)}

## use the dict to tokenize each review in reviews_split
## store the tokenized reviews in reviews_ints
reviews_ints = []
for review in reviews_split:
    reviews_ints.append([vocab_to_int[word] for word in review.split()])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Padding sequences
To deal with both short and very long reviews, we'll pad or truncate all our reviews to a specific length. For reviews shorter than some seq_length, we'll pad with 0s. For reviews longer than seq_length, we can truncate them to the first seq_length words. A good seq_length, in this case, is 200.

Exercise: Define a function that returns an array features that contains the padded data, of a standard size, that we'll pass to the network.
* The data should come from review_ints, since we want to feed integers to the network.
* Each row should be seq_length elements long.
* For reviews shorter than seq_length words, left pad with 0s. That is, if the review is ['best', 'movie', 'ever'], [117, 18, 128] as integers, the row will look like [0, 0, 0, ..., 0, 117, 18, 128].
* For reviews longer than seq_length, use only the first seq_length words as the feature vector.

As a small example, if the seq_length=10 and an input review is:
[117, 18, 128]
The resultant, padded sequence should be:
[0, 0, 0, 0, 0, 0, 0, 117, 18, 128]
Your final features array should be a 2D array, with as many rows as there are reviews, and as many columns as the specified seq_length.
This isn't trivial and there are a bunch of ways to do this. But, if you're going to be building your own deep learning networks, you're going to have to get used to preparing your data.
Can you write Python code for it?

def pad_features(reviews_ints, seq_length):
    ''' Return features of review_ints, where each review is padded with 0's
        or truncated to the input seq_length.
    '''

    # getting the correct rows x cols shape
    features = np.zeros((len(reviews_ints), seq_length), dtype=int)

    # for each review, I grab that review and
    for i, row in enumerate(reviews_ints):
        features[i, -len(row):] = np.array(row)[:seq_length]

    return features

# Test your implementation!

seq_length = 200

features = pad_features(reviews_ints, seq_length=seq_length)

## test statements - do not change - ##
assert len(features)==len(reviews_ints), "Your features should have as many rows as reviews."
assert len(features[0])==seq_length, "Each feature row should contain seq_length values."

# print first 10 values of the first 30 batches
print(features[:30,:10])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Removing Outliers
As an additional pre-processing step, we want to make sure that our reviews are in good shape for standard processing. That is, our network will expect a standard input text size, and so, we'll want to shape our reviews into a specific length. We'll approach this task in two main steps:

Getting rid of extremely long or short reviews; the outliers
Padding/truncating the remaining data so that we have reviews of the same length.

Before we pad our review text, we should check for reviews of extremely short or long lengths; outliers that may mess with our training.
Can you write Python code for it?

# outlier review stats
review_lens = Counter([len(x) for x in reviews_ints])
print("Zero-length reviews: {}".format(review_lens[0]))
print("Maximum review length: {}".format(max(review_lens)))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Sentiment Network with PyTorch
Below is where you'll define the network.
<img src="assets/network_diagram.png" width=40%>
The layers are as follows:
1. An embedding layer that converts our word tokens (integers) into embeddings of a specific size.
2. An LSTM layer defined by a hidden_state size and number of layers
3. A fully-connected output layer that maps the LSTM layer outputs to a desired output_size
4. A sigmoid activation layer which turns all outputs into a value 0-1; return only the last sigmoid output as the output of this network.
The Embedding Layer
We need to add an embedding layer because there are 74000+ words in our vocabulary. It is massively inefficient to one-hot encode that many classes. So, instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table. You could train an embedding layer using Word2Vec, then load it here. But, it's fine to just make a new layer, using it for only dimensionality reduction, and let the network learn the weights.
The LSTM Layer(s)
We'll create an LSTM to use in our recurrent network, which takes in an input_size, a hidden_dim, a number of layers, a dropout probability (for dropout between multiple layers), and a batch_first parameter.
Most of the time, you're network will have better performance with more layers; between 2-3. Adding more layers allows the network to learn really complex relationships.

Exercise: Complete the __init__, forward, and init_hidden functions for the SentimentRNN model class.

Note: init_hidden should initialize the hidden and cell state of an lstm layer to all zeros, and move those state to GPU, if available.
Can you write Python code for it?

# First checking if GPU is available
train_on_gpu=torch.cuda.is_available()

if(train_on_gpu):
    print('Training on GPU.')
else:
    print('No GPU available, training on CPU.')

import torch.nn as nn

class SentimentRNN(nn.Module):
    """
    The RNN model that will be used to perform Sentiment analysis.
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Initialize the model by setting up the layers.
        """
        super(SentimentRNN, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                            dropout=drop_prob, batch_first=True)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()

    def forward(self, x, hidden):
        """
        Perform a forward pass of our model on some input and hidden state.
        """
        batch_size = x.size(0)

        # embeddings and lstm_out
        embeds = self.embedding(x)
        lstm_out, hidden = self.lstm(embeds, hidden)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        # sigmoid function
        sig_out = self.sig(out)

        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1)
        sig_out = sig_out[:, -1] # get last batch of labels

        # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new...


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_RNN.ipynb".
The first task is:
Testing
There are a few ways to test your network.

Test data performance: First, we'll see how our trained model performs on all of our defined test_data, above. We'll calculate the average loss and accuracy over the test data.

Inference on user-generated data: Second, we'll see if we can input just one example review at a time (without a label), and see what the trained model predicts. Looking at new, user input data like this, and predicting an output label, is called inference.
Can you write Python code for it?

# Get test data loss and accuracy

test_losses = [] # track loss
num_correct = 0

# init hidden state
h = net.init_hidden(batch_size)

net.eval()
# iterate over test data
for inputs, labels in test_loader:

    # Creating new variables for the hidden state, otherwise
    # we'd backprop through the entire training history
    h = tuple([each.data for each in h])

    if(train_on_gpu):
        inputs, labels = inputs.cuda(), labels.cuda()

    # get predicted outputs
    output, h = net(inputs, h)

    # calculate loss
    test_loss = criterion(output.squeeze(), labels.float())
    test_losses.append(test_loss.item())

    # convert output probabilities to predicted class (0 or 1)
    pred = torch.round(output.squeeze())  # rounds to the nearest integer

    # compare predictions to true label
    correct_tensor = pred.eq(labels.float().view_as(pred))
    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
    num_correct += np.sum(correct)

# -- stats! -- ##
# avg test loss
print("Test loss: {:.3f}".format(np.mean(test_losses)))

# accuracy over all test data
test_acc = num_correct/len(test_loader.dataset)
print("Test accuracy: {:.3f}".format(test_acc))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
And run the following cell to test it's performance.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
As you can see, common words like "the" appear very often in both positive and negative reviews. Instead of finding the most common words in positive or negative reviews, what you really want are the words found in positive reviews more often than in negative reviews, and vice versa. To accomplish this, you'll need to calculate the ratios of word usage between positive and negative reviews.
TODO: Check all the words you've seen and calculate the ratio of postive to negative uses and store that ratio in pos_neg_ratios.

Hint: the positive-to-negative ratio for a given word can be calculated with positive_counts[word] / float(negative_counts[word]+1). Notice the +1 in the denominator – that ensures we don't divide by zero for words that are only seen in positive reviews.
Can you write Python code for it?

pos_neg_ratios = Counter()

# Calculate the ratios of positive and negative uses of the most common words
# Consider words to be "common" if they've been used at least 100 times
for term,cnt in list(total_counts.most_common()):
    if(cnt > 100):
        pos_neg_ratio = positive_counts[term] / float(negative_counts[term]+1)
        pos_neg_ratios[term] = pos_neg_ratio


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
End of Project 2 solution.
Watch the next video to continue with Andrew's next lesson.
Project 3: Building a Neural Network
TODO: We've included the framework of a class called SentimentNetork. Implement all of the items marked TODO in the code. These include doing the following:
- Create a basic neural network much like the networks you've seen in earlier lessons and in Project 1, with an input layer, a hidden layer, and an output layer.
- Do not add a non-linearity in the hidden layer. That is, do not use an activation function when calculating the hidden layer outputs.
- Re-use the code from earlier in this notebook to create the training data (see TODOs in the code)
- Implement the pre_process_data function to create the vocabulary for our training data generating functions
- Ensure train trains over the entire corpus
Where to Get Help if You Need it

Re-watch previous week's Udacity Lectures
Chapters 3-5 - Grokking Deep Learning - (Check inside your classroom for a discount code)
Can you write Python code for it?

import time
import sys
import numpy as np

# Encapsulate our neural network in a class
class SentimentNetwork:
    def __init__(self, reviews,labels,hidden_nodes = 10, learning_rate = 0.1):
        """Create a SentimenNetwork with the given settings
        Args:
            reviews(list) - List of reviews used for training
            labels(list) - List of POSITIVE/NEGATIVE labels associated with the given reviews
            hidden_nodes(int) - Number of nodes to create in the hidden layer
            learning_rate(float) - Learning rate to use while training

        """
        # Assign a seed to our random number generator to ensure we get
        # reproducable results during development
        np.random.seed(1)

        # process the reviews and their associated labels so that everything
        # is ready for training
        self.pre_process_data(reviews, labels)

        # Build the network to have the number of hidden nodes and the learning rate that
        # were passed into this initializer. Make the same number of input nodes as
        # there are vocabulary words and create a single output node.
        self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

    def pre_process_data(self, reviews, labels):

        # populate review_vocab with all of the words in the given reviews
        review_vocab = set()
        for review in reviews:
            for word in review.split(" "):
                review_vocab.add(word)

        # Convert the vocabulary set to a list so we can access words via indices
        self.review_vocab = list(review_vocab)

        # populate label_vocab with all of the words in the given labels.
        label_vocab = set()
        for label in labels:
            label_vocab.add(label)

        # Convert the label vocabulary set to a list so we can access labels via indices
        self.label_vocab = list(label_vocab)

        # Store the sizes of the review and label vocabularies.
       ...


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
End of Project 5 solution.
Watch the next video to continue with Andrew's next lesson.
Further Noise Reduction
Can you write Python code for it?

Image(filename='sentiment_network_sparse_2.png')

# words most frequently seen in a review with a "POSITIVE" label
pos_neg_ratios.most_common()

# words most frequently seen in a review with a "NEGATIVE" label
list(reversed(pos_neg_ratios.most_common()))[0:30]

from bokeh.models import ColumnDataSource, LabelSet
from bokeh.plotting import figure, show, output_file
from bokeh.io import output_notebook
output_notebook()

hist, edges = np.histogram(list(map(lambda x:x[1],pos_neg_ratios.most_common())), density=True, bins=100, normed=True)

p = figure(tools="pan,wheel_zoom,reset,save",
           toolbar_location="above",
           title="Word Positive/Negative Affinity Distribution")
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color="#555555")
show(p)

frequency_frequency = Counter()

for word, cnt in total_counts.most_common():
    frequency_frequency[cnt] += 1

hist, edges = np.histogram(list(map(lambda x:x[1],frequency_frequency.most_common())), density=True, bins=100, normed=True)

p = figure(tools="pan,wheel_zoom,reset,save",
           toolbar_location="above",
           title="The frequency distribution of the words in our corpus")
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color="#555555")
show(p)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Examine the ratios you've calculated for a few words:
Can you write Python code for it?

print("Pos-to-neg ratio for 'the' = {}".format(pos_neg_ratios["the"]))
print("Pos-to-neg ratio for 'amazing' = {}".format(pos_neg_ratios["amazing"]))
print("Pos-to-neg ratio for 'terrible' = {}".format(pos_neg_ratios["terrible"]))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Note: The data in reviews.txt we're using has already been preprocessed a bit and contains only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like The, the, and THE, all the same way.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Project 2: Creating the Input/Output Data
TODO: Create a set named vocab that contains every word in the vocabulary.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Project 4: Reducing Noise in Our Input Data
TODO: Attempt to reduce the noise in the input data like Andrew did in the previous video. Specifically, do the following:
* Copy the SentimentNetwork class you created earlier into the following cell.
* Modify update_input_layer so it does not count how many times each word is used, but rather just stores whether or not a word was used.
The following code is the same as the previous project, with project-specific changes marked with "New for Project 4"
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Project 5: Making our Network More Efficient
TODO: Make the SentimentNetwork class more efficient by eliminating unnecessary multiplications and additions that occur during forward and backward propagation. To do that, you can do the following:
* Copy the SentimentNetwork class from the previous project into the following cell.
* Remove the update_input_layer function - you will not need it in this version.
* Modify init_network:

You no longer need a separate input layer, so remove any mention of self.layer_0
You will be dealing with the old hidden layer more directly, so create self.layer_1, a two-dimensional matrix with shape 1 x hidden_nodes, with all values initialized to zero
Modify train:
Change the name of the input parameter training_reviews to training_reviews_raw. This will help with the next step.
At the beginning of the function, you'll want to preprocess your reviews to convert them to a list of indices (from word2index) that are actually used in the review. This is equivalent to what you saw in the video when Andrew set specific indices to 1. Your code should create a local list variable named training_reviews that should contain a list for each review in training_reviews_raw. Those lists should contain the indices for words found in the review.
Remove call to update_input_layer
Use self's  layer_1 instead of a local layer_1 object.
In the forward pass, replace the code that updates layer_1 with new logic that only adds the weights for the indices used in the review.
When updating weights_0_1, only update the individual weights that were used in the forward pass.
Modify run:
Remove call to update_input_layer
Use self's  layer_1 instead of a local layer_1 object.
Much like you did in train, you will need to pre-process the review so you can work with word indices, then update layer_1 by adding weights for the indices used in the review.

The following code is the same as the previous project, with project-specific changes marked with "New for Project 5"
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Project 6: Reducing Noise by Strategically Reducing the Vocabulary
TODO: Improve SentimentNetwork's performance by reducing more noise in the vocabulary. Specifically, do the following:
* Copy the SentimentNetwork class from the previous project into the following cell.
* Modify pre_process_data:

Add two additional parameters: min_count and polarity_cutoff
Calculate the positive-to-negative ratios of words used in the reviews. (You can use code you've written elsewhere in the notebook, but we are moving it into the class like we did with other helper code earlier.)
Andrew's solution only calculates a postive-to-negative ratio for words that occur at least 50 times. This keeps the network from attributing too much sentiment to rarer words. You can choose to add this to your solution if you would like.
Change so words are only added to the vocabulary if they occur in the vocabulary more than min_count times.
Change so words are only added to the vocabulary if the absolute value of their postive-to-negative ratio is at least polarity_cutoff
Modify __init__:
Add the same two parameters (min_count and polarity_cutoff) and use them when you call pre_process_data

The following code is the same as the previous project, with project-specific changes marked with "New for Project 6"
Can you write Python code for it?

import time
import sys
import numpy as np

# Encapsulate our neural network in a class
class SentimentNetwork:
    ## New for Project 6: added min_count and polarity_cutoff parameters
    def __init__(self, reviews,labels,min_count = 10,polarity_cutoff = 0.1,hidden_nodes = 10, learning_rate = 0.1):
        """Create a SentimenNetwork with the given settings
        Args:
            reviews(list) - List of reviews used for training
            labels(list) - List of POSITIVE/NEGATIVE labels associated with the given reviews
            min_count(int) - Words should only be added to the vocabulary
                             if they occur more than this many times
            polarity_cutoff(float) - The absolute value of a word's positive-to-negative
                                     ratio must be at least this big to be considered.
            hidden_nodes(int) - Number of nodes to create in the hidden layer
            learning_rate(float) - Learning rate to use while training

        """
        # Assign a seed to our random number generator to ensure we get
        # reproducable results during development
        np.random.seed(1)

        # process the reviews and their associated labels so that everything
        # is ready for training
        ## New for Project 6: added min_count and polarity_cutoff arguments to pre_process_data call
        self.pre_process_data(reviews, labels, polarity_cutoff, min_count)

        # Build the network to have the number of hidden nodes and the learning rate that
        # were passed into this initializer. Make the same number of input nodes as
        # there are vocabulary words and create a single output node.
        self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

    ## New for Project 6: added min_count and polarity_cutoff parameters
    def pre_process_data(self, reviews, labels, polarity_cutoff, min_count):

        ## ----------------------------------------
        ## New for Project 6: Calculate...


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell to check your vocabulary size. If everything worked correctly, it should print 74074
Can you write Python code for it?

vocab_size = len(vocab)
print(vocab_size)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell to create a SentimentNetwork that will train on all but the last 1000 reviews (we're saving those for testing). Here we use a learning rate of 0.1.
Can you write Python code for it?

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell to recreate the network and train it. Notice we've gone back to the higher learning rate of 0.1.
Can you write Python code for it?

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)
mlp.train(reviews[:-1000],labels[:-1000])

mlp.test(reviews[-1000:],labels[-1000:])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell to test the network's performance against the last 1000 reviews (the ones we held out from our training set).
We have not trained the model yet, so the results should be about 50% as it will just be guessing and there are only two possible values to choose from.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell to test updating the input layer with the first review. The indices assigned may not be the same as in the solution, but hopefully you'll see some non-zero values in layer_0.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following cell. It should display (1, 74074)
Can you write Python code for it?

layer_0.shape

from IPython.display import Image
Image(filename='sentiment_network.png')


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
Run the following two cells to list the words used in positive reviews and negative reviews, respectively, ordered from most to least commonly used.
Can you write Python code for it?

# Examine the counts of the most common words in positive reviews
positive_counts.most_common()

# Examine the counts of the most common words in negative reviews
negative_counts.most_common()


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
TODO:  Complete the implementation of update_input_layer. It should count
          how many times each word is used in the given review, and then store
          those counts at the appropriate indices inside layer_0.
Can you write Python code for it?

def update_input_layer(review):
    """ Modify the global layer_0 to represent the vector form of review.
    The element at a given index of layer_0 should represent
    how many times the given word occurs in the review.
    Args:
        review(string) - the string of the review
    Returns:
        None
    """

    global layer_0

    # clear out previous state, reset the layer to be all 0s
    layer_0 *= 0

    # count how many times each word is used in the given review and store the results in layer_0
    for word in review.split(" "):
        layer_0[0][word2index[word]] += 1


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
That most likely didn't train very well. Part of the reason may be because the learning rate is too high. Run the following cell to recreate the network with a smaller learning rate, 0.01, and then train the new network.
Can you write Python code for it?

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.01)
mlp.train(reviews[:-1000],labels[:-1000])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
That should have trained much better than the earlier attempts. Run the following cell to test your model with 1000 predictions.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
We'll create three Counter objects, one for words from postive reviews, one for words from negative reviews, and one for all the words.
Can you write Python code for it?

# Create three Counter objects to store positive, negative and total counts
positive_counts = Counter()
negative_counts = Counter()
total_counts = Counter()


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Sentiment pytorch/Sentiment_classif.ipynb".
The first task is:
layer_0 contains one entry for every word in the vocabulary, as shown in the above image. We need to make sure we know the index of each word, so run the following cell to create a lookup table that stores the index of every word.
Can you write Python code for it?

# Create a dictionary of words in the vocabulary mapped to index positions
# (to be used in layer_0)
word2index = {}
for i,word in enumerate(vocab):
    word2index[word] = i

# display the map of words to indices
word2index


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Now we're ready to fit out model! Use validation_split when not submitting.
Can you write Python code for it?

model.fit(X_t, y, batch_size=128, epochs=2, validation_split=0.35)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Read in our data and replace missing values:
Can you write Python code for it?

train = pd.read_csv(TRAIN_DATA_FILE)
test = pd.read_csv(TEST_DATA_FILE)

list_sentences_train = train["comment_text"].fillna("_na_").values
list_classes = ["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]
y = train[list_classes].values
list_sentences_test = test["comment_text"].fillna("_na_").values


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Read the glove word vectors (space delimited strings) into a dictionary from word->vector.
Can you write Python code for it?

def get_coefs(word,*arr):
    return word, np.asarray(arr, dtype='float32')
embeddings_index = dict(get_coefs(*o.strip().split()) for o in open(EMBEDDING_FILE))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Set some basic config parameters:
Can you write Python code for it?

embed_size = 50 # how big is each word vector
max_features = 20000 # how many unique words to use (i.e num rows in embedding vector)
maxlen = 100 # max number of words in a comment to use


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Simple bidirectional LSTM with two fully connected layers. We add some dropout to the LSTM since even 2 epochs is enough to overfit.
Can you write Python code for it?

inp = Input(shape=(maxlen,))
x = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)
x = Bidirectional(LSTM(50, return_sequences=True, dropout=0.1, recurrent_dropout=0.1))(x)
x = GlobalMaxPool1D()(x)
x = Dense(50, activation="relu")(x)
x = Dropout(0.1)(x)
x = Dense(6, activation="sigmoid")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

print(model.summary())


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Standard keras preprocessing, to turn each comment into a list of word indexes of equal length (with truncation or padding as needed).
Can you write Python code for it?

tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(list(list_sentences_train))
list_tokenized_train = tokenizer.texts_to_sequences(list_sentences_train)
list_tokenized_test = tokenizer.texts_to_sequences(list_sentences_test)
X_t = pad_sequences(list_tokenized_train, maxlen=maxlen)
X_te = pad_sequences(list_tokenized_test, maxlen=maxlen)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
Use these vectors to create our embedding matrix, with random initialization for words that aren't in GloVe. We'll use the same mean and stdev of embeddings the GloVe has when generating the random init.
Can you write Python code for it?

all_embs = np.stack(embeddings_index.values())
emb_mean,emb_std = all_embs.mean(), all_embs.std()
emb_mean,emb_std

word_index = tokenizer.word_index
nb_words = min(max_features, len(word_index))
embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))
for word, i in word_index.items():
    if i >= max_features: continue
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None: embedding_matrix[i] = embedding_vector


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/Toxic multiclass prediction Glove + Bidirection LSTM.ipynb".
The first task is:
We include the GloVe word vectors in our input files. To include these in your kernel, simple click 'input files' at the top of the notebook, and search 'glove' in the 'datasets' section.
Can you write Python code for it?

EMBEDDING_FILE = f'glove.6B.50d.txt'
TRAIN_DATA_FILE = f'train.csv'
TEST_DATA_FILE = f'test.csv'


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
...and let's see if our model can generate a bit more text all by itself!
Can you write Python code for it?

print(ss,"\n")
for i in range(50):
    n=res[-1].topk(2)[1]
    n = n[1] if n.data[0]==0 else n[0]
    print(TEXT.vocab.itos[n.data[0]], end=' ')
    res,*_ = m(n[0].unsqueeze(0))
print('...')


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
After building our ModelData object, it automatically fills the TEXT object with a very important attribute: TEXT.vocab. This is a vocabulary, which stores which words (or tokens) have been seen in the text, and how each word will be mapped to a unique integer id. We'll need to use this information again later, so we save it.
(Technical note: python's standard Pickle library can't handle this correctly, so at the top of this notebook we used the dill library instead and imported it as pickle).
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
As you can see below, I gradually tuned the language model in a few stages. I possibly could have trained it further (it wasn't yet overfitting), but I didn't have time to experiment more. Maybe you can see if you can train it to a better accuracy! (I used lr_find to find a good learning rate, but didn't save the output in this notebook. Feel free to try running it yourself now.)
Can you write Python code for it?

learner.fit(3e-3, 4, wds=1e-6, cycle_len=1, cycle_mult=2)

learner.save_encoder('adam1_enc')

learner.load_encoder('adam1_enc')

learner.fit(3e-3, 1, wds=1e-6, cycle_len=10)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Because we're fine-tuning a pretrained model, we'll use differential learning rates, and also increase the max gradient for clipping, to allow the SGDR to work better.
Can you write Python code for it?

m3.clip=25.
lrs=np.array([1e-4,1e-4,1e-4,1e-3,1e-2])

m3.freeze_to(-1)
m3.fit(lrs/2, 1, metrics=[accuracy])
m3.unfreeze()
m3.fit(lrs, 1, metrics=[accuracy], cycle_len=1)

m3.fit(lrs, 7, metrics=[accuracy], cycle_len=2, cycle_save_name='imdb2')

m3.load_cycle('imdb2', 4)

accuracy_np(*m3.predict_with_targs())


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
In the sentiment analysis section, we'll just need half of the language model - the encoder, so we save that part.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Language modeling
Data
The large movie view dataset contains a collection of 50,000 reviews from IMDB. The dataset contains an even number of positive and negative reviews. The authors considered only highly polarized reviews. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. Neutral reviews are not included in the dataset. The dataset is divided into training and test sets. The training set is the same 25,000 labeled reviews.
The sentiment classification task consists of predicting the polarity (positive or negative) of a given text.
However, before we try to classify sentiment, we will simply try to create a language model; that is, a model that can predict the next word in a sentence. Why? Because our model first needs to understand the structure of English, before we can expect it to recognize positive vs negative sentiment.
So our plan of attack is the same as we used for Dogs v Cats: pretrain a model to do one thing (predict the next word), and fine tune it to do something else (classify sentiment).
Unfortunately, there are no good pretrained language models available to download, so we need to create our own. To follow along with this notebook, we suggest downloading the dataset from this location on files.fast.ai.
Can you write Python code for it?

PATH='data/aclImdb/'

TRN_PATH = 'train/all/'
VAL_PATH = 'test/all/'
TRN = f'{PATH}{TRN_PATH}'
VAL = f'{PATH}{VAL_PATH}'

%ls {PATH}


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Language modeling accuracy is generally measured using the metric perplexity, which is simply exp() of the loss function we used.
Can you write Python code for it?

math.exp(4.165)

pickle.dump(TEXT, open(f'{PATH}models/TEXT.pkl','wb'))


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Note that in a LanguageModelData object there is only one item in each dataset: all the words of the text joined together.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Our LanguageModelData object will create batches with 64 columns (that's our batch size), and varying sequence lengths of around 80 tokens (that's our bptt parameter - backprop through time).
Each batch also contains the exact same data as labels, but one word later in the text - since we're trying to always predict the next word. The labels are flattened into a 1d array.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Sentiment
We'll need to the saved vocab from the language model, since we need to ensure the same words map to the same IDs.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Sounds like I'd really enjoy Zombiegeddon...
Now we'll check how many words are in the dataset.
Can you write Python code for it?

!find {TRN} -name '*.txt' | xargs cat | wc -w

!find {VAL} -name '*.txt' | xargs cat | wc -w


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
Test
We can play around with our language model a bit to check it seems to be working OK. First, let's create a short bit of text to 'prime' a set of predictions. We'll use our torchtext field to numericalize it so we can feed it to our language model.
Can you write Python code for it?

m=learner.model
ss=""". So, it wasn't quite was I was expecting, but I really liked it anyway! The best"""
s = [TEXT.preprocess(ss)]
t=TEXT.numericalize(s)
' '.join(s[0])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
This is the start of the mapping from integer IDs to unique tokens.
Can you write Python code for it?

# 'itos': 'int-to-string'
TEXT.vocab.itos[:12]

# 'stoi': 'string to int'
TEXT.vocab.stoi['the']


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
We haven't yet added methods to make it easy to test a language model, so we'll need to manually go through the steps.
Can you write Python code for it?

# Set batch size to 1
m[0].bs=1
# Turn off dropout
m.eval()
# Reset hidden state
m.reset()
# Get predictions from model
res,*_ = m(t)
# Put the batch size back to what it was
m[0].bs=bs


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
fastai can create a ModelData object directly from torchtext splits.
Can you write Python code for it?

md2 = TextData.from_splits(PATH, splits, bs)

m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl,
           dropout=0.1, dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)
m3.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
m3.load_encoder(f'adam3_10_enc')


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
fastai uses a variant of the state of the art AWD LSTM Language Model developed by Stephen Merity. A key feature of this model is that it provides excellent regularization through Dropout. There is no simple way known (yet!) to find the best values of the dropout parameters below - you just have to experiment...
However, the other parameters (alpha, beta, and clip) shouldn't generally need tuning.
Can you write Python code for it?

learner = md.get_model(opt_fn, em_sz, nh, nl,
               dropouti=0.05, dropout=0.05, wdrop=0.1, dropoute=0.02, dropouth=0.05)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip=0.3


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
fastai works closely with torchtext. We create a ModelData object for language modeling by taking advantage of LanguageModelData, passing it our torchtext field object, and the paths to our training, test, and validation sets. In this case, we don't have a separate test set, so we'll just use VAL_PATH for that too.
As well as the usual bs (batch size) parameter, we also now have bptt; this define how many words are processing at a time in each row of the mini-batch. More importantly, it defines how many 'layers' we will backprop through. Making this number higher will increase time and memory requirements, but will improve the model's ability to handle long sentences.
Can you write Python code for it?

bs=64; bptt=70

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=10)


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
sequential=False tells torchtext that a text field should be tokenized (in this case, we just want to store the 'positive' or 'negative' single label).
splits is a torchtext method that creates train, test, and validation sets. The IMDB dataset is built into torchtext, so we can take advantage of that. Take a look at lang_model-arxiv.ipynb to see how to define your own fastai/torchtext datasets.
Can you write Python code for it?

IMDB_LABEL = data.Field(sequential=False)
splits = torchtext.datasets.IMDB.splits(TEXT, IMDB_LABEL, 'data/')

t = splits[0].examples[0]

t.label, ' '.join(t.text[:16])


I am working on the file "DEEP LEARNING/NLP/LSTM RNN/imdb sentiment analysis + language modelling fastai .ipynb".
The first task is:
torchtext will handle turning this words into integer IDs for us automatically.
Can you write Python code for it?


I am working on the file "DEEP LEARNING/NLP/elmo EMBEDDINGS/Elmo_contextual_embeddings.ipynb".
The first task is:
Create a semantic search engine:
Can you write Python code for it?

#@title Sementic search
#@markdown Enter a set of words to find matching sentences. 'results_returned' can beused to modify the number of matching sentences retured. To view the code behind this cell, use the menu in the top right to unhide...
search_string = "code of ethics" #@param {type:"string"}
results_returned = "3" #@param [1, 2, 3]

from sklearn.metrics.pairwise import cosine_similarity

embeddings2 = embed(
    [search_string],
    signature="default",
    as_dict=True)["default"]

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  sess.run(tf.tables_initializer())
  search_vect = sess.run(embeddings2)

cosine_similarities = pd.Series(cosine_similarity(search_vect, x).flatten())
output =""
for i,j in cosine_similarities.nlargest(int(results_returned)).iteritems():
  output +='
 '
  for i in sentences[i].split():
    if i.lower() in search_string:
      output += " "+str(i)+""
    else:
      output += " "+str(i)
  output += "

"

output = '
Results:
'+output
display(HTML(output))
#   print(sentences[i])
#   print('\n')


I am working on the file "DEEP LEARNING/NLP/elmo EMBEDDINGS/Elmo_contextual_embeddings.ipynb".
The first task is:
Visualize the sentences using PCA and TSNE
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components=50)
y = pca.fit_transform(x)

from sklearn.manifold import TSNE

y = TSNE(n_components=2).fit_transform(y)

import plotly.plotly as py
import plotly.graph_objs as go
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot

init_notebook_mode(connected=True)

data = [
    go.Scatter(
        x=[i[0] for i in y],
        y=[i[1] for i in y],
        mode='markers',
        text=[i for i in sentences],
    marker=dict(
        size=16,
        color = [len(i) for i in sentences], #set color equal to a variable
        opacity= 0.8,
        colorscale='Viridis',
        showscale=False
    )
    )
]
layout = go.Layout()
layout = dict(
              yaxis = dict(zeroline = False),
              xaxis = dict(zeroline = False)
             )
fig = go.Figure(data=data, layout=layout)
file = plot(fig, filename='Sentence encode.html')

from google.colab import files
files.download('Sentence encode.html')


I am working on the file "DEEP LEARNING/NLP/text analyses/CookingLDA_PA.ipynb".
The first task is:
Для такого большого количества классов это неплохая точность. Вы можете попроовать обучать RandomForest на исходной матрице частот слов, имеющей значительно большую размерность, и увидеть, что accuracy увеличивается на 10-15%. Таким образом, LDA собрал не всю, но достаточно большую часть информации из выборки, в матрице низкого ранга.
LDA --- вероятностная модель
Матричное разложение, использующееся в LDA, интерпретируется как следующий процесс генерации документов.
Для документа $d$ длины $n_d$:
1. Из априорного распределения Дирихле с параметром alpha сгенерировать распределение над множеством тем: $\theta_d \sim Dirichlet(\alpha)$
1. Для каждого слова $w = 1, \dots, n_d$:
    1. Сгенерировать тему из дискретного распределения $t \sim \theta_{d}$
    1. Сгенерировать слово из дискретного распределения $w \sim \phi_{t}$.
Подробнее об этом в Википедии.
В контексте нашей задачи получается, что, используя данный генеративный процесс, можно создавать новые рецепты. Вы можете передать в функцию модель и число ингредиентов и сгенерировать рецепт :)
Can you write Python code for it?

def generate_recipe(model, num_ingredients):
    theta = np.random.dirichlet(model.alpha)
    for i in range(num_ingredients):
        t = np.random.choice(np.arange(model.num_topics), p=theta)
        topic = model.show_topic(0, topn=model.num_terms)
        topic_distr = [x[1] for x in topic]
        terms = [x[0] for x in topic]
        w = np.random.choice(terms, p=topic_distr)
        print w

generate_recipe(lda2, 5)


I am working on the file "DEEP LEARNING/NLP/text analyses/CookingLDA_PA.ipynb".
The first task is:
Интерпретация построенной модели
Вы можете рассмотреть топы ингредиентов каждой темы. Большиснтво тем сами по себе похожи на рецепты; в некоторых собираются продукты одного вида, например, свежие фрукты или разные виды сыра.
Попробуем эмпирически соотнести наши темы с национальными кухнями (cuisine). Построим матрицу A размера темы x кухни, ее элементы $a_{tc}$ - суммы p(t|d) по всем документам d, которые отнесены к кухне c. Нормируем матрицу на частоты рецептов по разным кухням, чтобы избежать дисбаланса между кухнями. Следующая функция получает на вход объект модели, объект корпуса и исходные данные и возвращает нормированную матрицу A. Ее удобно визуализировать с помощью seaborn.
Can you write Python code for it?

import pandas
import seaborn
from matplotlib import pyplot as plt
%matplotlib inline

def compute_topic_cuisine_matrix(model, corpus, recipes):
    # составляем вектор целевых признаков
    targets = list(set([recipe["cuisine"] for recipe in recipes]))
    # составляем матрицу
    tc_matrix = pandas.DataFrame(data=np.zeros((model.num_topics, len(targets))), columns=targets)
    for recipe, bow in zip(recipes, corpus):
        recipe_topic = model.get_document_topics(bow)
        for t, prob in recipe_topic:
            tc_matrix[recipe["cuisine"]][t] += prob
    # нормируем матрицу
    target_sums = pandas.DataFrame(data=np.zeros((1, len(targets))), columns=targets)
    for recipe in recipes:
        target_sums[recipe["cuisine"]] += 1
    return pandas.DataFrame(tc_matrix.values/target_sums.values, columns=tc_matrix.columns)

def plot_matrix(tc_matrix):
    plt.figure(figsize=(10, 10))
    seaborn.heatmap(tc_matrix, square=True)

# Визуализируйте матрицу
cus_matr = compute_topic_cuisine_matrix(lda2, corpus2, recipes)
plot_matrix(cus_matr)


I am working on the file "DEEP LEARNING/NLP/text analyses/CookingLDA_PA.ipynb".
The first task is:
Наша коллекция небольшая и влезает в оперативную память. Gensim может работать с такими данными и не требует их сохранения на диск в специальном формате. Для этого коллекция должна быть представлена в виде списка списков, каждый внутренний список соответствует отдельному документу и состоит из его слов. Пример коллекции из двух документов:
[["hello", "world"], ["programming", "in", "python"]]
Преобразуем наши данные в такой формат, а затем создадим объекты corpus и dictionary, с которыми будет работать модель.
Can you write Python code for it?

texts = [recipe["ingredients"] for recipe in recipes]
dictionary = corpora.Dictionary(texts)   # составляем словарь
corpus = [dictionary.doc2bow(text) for text in texts]  # составляем корпус документов

print texts[0]
print corpus[0]


I am working on the file "DEEP LEARNING/NLP/text analyses/CookingLDA_PA.ipynb".
The first task is:
Составление корпуса
Can you write Python code for it?

from gensim import corpora, models
import gensim
import numpy as np


I am working on the file "DEEP LEARNING/NLP/text analyses/CookingLDA_PA.ipynb".
The first task is:
Сравнение когерентностей
Задание 3. Постройте еще одну модель по корпусу corpus2 и словарю dictioanary2, остальные параметры оставьте такими же, как при первом построении модели. Сохраните новую модель в другую переменную (не перезаписывайте предыдущую модель). Не забудьте про фиксирование seed!
Затем воспользуйтесь методом top_topics модели, чтобы вычислить ее когерентность. Передайте в качестве аргумента соответствующий модели корпус. Метод вернет список кортежей (топ токенов, когерентность), отсортированных по убыванию последней. Вычислите среднюю по всем темам когерентность для каждой из двух моделей и передайте в функцию save_answers3.
Can you write Python code for it?

np.random.seed(76543)
# здесь код для построения модели:
lda2 = models.ldamodel.LdaModel(corpus2, num_topics=40, passes=5)

top2 = lda2.top_topics(corpus2)
top = lda.top_topics(corpus)

cog2 = []
for item in top2:
    for cog in item[0]:
        cog2.append(cog[0])

print np.mean(cog2)

cog1 = []
for item in top:
    for cog in item[0]:
        cog1.append(cog[0])
print np.mean(cog1)

def save_answers3(coherence, coherence2):
    with open("cooking_LDA_pa_task3.txt", "w") as fout:
        fout.write(" ".join(["%3f"%el for el in [coherence, coherence2]]))

save_answers3(np.mean(cog1), np.mean(cog2))