Multiclass BiLSTM issue

Smaranika_Sikdar: <p>Hi,</p>
<p>I am getting error:<br>
18 <span class="hashtag">#print</span>(output)<br>
—&gt; 19 loss = criterion(output.squeeze(), labels.float())<br>
…<br>
2516     reduction_enum = _Reduction.get_enum(reduction)<br>
2517 if target.size() != input.size():<br>
 → 2518     raise ValueError("Using a target size ({}) that is different to the input size ({}) is deprecated. "<br>
2519                      “Please ensure they have the same size.”.format(target.size(), input.size()))<br>
2521 if weight is not None:<br>
2522     new_size = _infer_size(target.size(), weight.size())</p>
<p>ValueError: Using a target size (torch.Size([128, 238])) that is different to the input size (torch.Size([128])) is deprecated. Please ensure they have the same size.</p>
<p>The whole code:</p>
<p>import os<br>
import re<br>
import json<br>
import torch<br>
import random<br>
import warnings<br>
import numpy as np<br>
get_ipython().run_line_magic(‘matplotlib’, ‘inline’)<br>
import pandas as pd<br>
import seaborn as sns<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F<br>
warnings.filterwarnings(“ignore”)<br>
from matplotlib import pyplot as plt<br>
from gensim.models import KeyedVectors<br>
from sklearn.model_selection import StratifiedKFold<br>
from sklearn.metrics import roc_auc_score, roc_curve, auc, confusion_matrix<br>
embeddingDim = 300</p>
<h1>
<a name="torchcudais_available-checks-and-returns-a-boolean-true-if-a-gpu-is-available-else-itll-return-false-1" class="anchor" href="#torchcudais_available-checks-and-returns-a-boolean-true-if-a-gpu-is-available-else-itll-return-false-1"></a>torch.cuda.is_available() checks and returns a Boolean True if a GPU is available, else it’ll return False</h1>
<p>is_cuda = torch.cuda.is_available()</p>
<h1>
<a name="if-we-have-a-gpu-available-well-set-our-device-to-gpu-well-use-this-device-variable-later-in-our-code-2" class="anchor" href="#if-we-have-a-gpu-available-well-set-our-device-to-gpu-well-use-this-device-variable-later-in-our-code-2"></a>If we have a GPU available, we’ll set our device to GPU. We’ll use this device variable later in our code.</h1>
<p>if is_cuda:<br>
print(“cuda available”)<br>
device = torch.device(“cuda”)<br>
else:<br>
device = torch.device(“cpu”)<br>
batchSize=100<br>
embeddingDim = 300</p>
<h1>
<a name="in28-3" class="anchor" href="#in28-3"></a>In[28]:</h1>
<p><span class="hashtag">#Read</span> FastText En model. If the file wiki.multi.en.vec’ does not exist, download it from</p>
<h1>
<a name="httpsdlfbaipublicfilescomarrivalvectorswikimultienvec-4" class="anchor" href="#httpsdlfbaipublicfilescomarrivalvectorswikimultienvec-4"></a><a href="https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.en.vec" rel="noopener nofollow ugc">https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.en.vec</a>
</h1>
<p>get_ipython().system(‘pip install wget’)<br>
import wget<br>
word2VecFile = os.path.join(os.curdir, ‘wiki.multi.en.vec’)</p>
<p>if os.path.exists(word2VecFile):<br>
print(‘Word2Vec file has been found and is being loaded…’)<br>
else:<br>
print(‘Word2Vec file does not exist and needs to be downloaded’)<br>
url = ‘<a href="https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.en.vec" rel="noopener nofollow ugc">https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.en.vec</a>’<br>
wget.download(url)<br>
print(‘Downloading from’, url)<br>
en_model = KeyedVectors.load_word2vec_format(‘wiki.multi.en.vec’)</p>
<p>data_test<br>
df – train data set</p>
<p>“”"<br>
Read the data into a pandas DataFrame. Only the fifth and sixth columns are required.<br>
The second column is review text and third one is review rating on a scale of 0-10.<br>
“”"<br>
df = data.iloc[:, [4,5]]<br>
testdf = data_test.iloc[:, [4,5]]<br>
“”"<br>
Add “Sentiment” column to the DataFrame based on overall rating. It will be a multiclass problem<br>
it is 5 class classification problem.<br>
“”"<br>
df[“Sentiment”] = df[‘overall’].astype(str)<br>
testdf[“Sentiment”] = testdf[‘overall’].astype(str)<br>
df.head(10)</p>
<p><span class="hashtag">#df</span>.head(5)<br>
<span class="hashtag">#testdf</span>.head(5)</p>
<p>“”"<br>
Distribution of different overall ratings<br>
“”"<br>
rating_frequency_count = df.overall.value_counts()<br>
sns.barplot(x=rating_frequency_count.index, y=rating_frequency_count.values)</p>
<h1>
<a name="in30-5" class="anchor" href="#in30-5"></a>In[30]:</h1>
<p>“”"<br>
Now let’s find out a value for maxlen. For that, we first look at the distribution of review lengths in terms of<br>
number of words. A box plot is used to visualize this distribution.<br>
“”"</p>
<p>re_wordMatcher = re.compile(r’[a-z0-9]+') <span class="hashtag">#Declare</span> regex to extract words<br>
numWords = df[“reviewText”].map(lambda x: len(re_wordMatcher.findall(x.lower())))<br>
g = sns.boxplot(numWords)</p>
<h1>
<a name="in38-6" class="anchor" href="#in38-6"></a>In[38]:</h1>
<p>df.loc[df.overall == 1.0, ‘Sentiment’] = “Bad”<br>
df.loc[df.overall == 2.0, ‘Sentiment’] = “Fair”<br>
df.loc[df.overall == 3.0, ‘Sentiment’] = “Good”<br>
df.loc[df.overall == 4.0, ‘Sentiment’] = “Very Good”<br>
df.loc[df.overall == 5.0, ‘Sentiment’] = “Excellent”</p>
<h1>
<a name="create-a-new-column-and-use-npselect-to-assign-values-to-it-using-our-lists-as-arguments-7" class="anchor" href="#create-a-new-column-and-use-npselect-to-assign-values-to-it-using-our-lists-as-arguments-7"></a>create a new column and use np.select to assign values to it using our lists as arguments</h1>
<p>df.head(10)</p>
<h1>
<a name="remove-column-name-a-8" class="anchor" href="#remove-column-name-a-8"></a>Remove column name ‘A’</h1>
<p>df.drop([‘overall’], axis=1)</p>
<h1>
<a name="in39-9" class="anchor" href="#in39-9"></a>In[39]:</h1>
<p>“”"<br>
The above plot shows that there are few very long reviews (black dots on the right) but most of the reviews are<br>
comparatively shorter than around 250 words. Specifically, let’s find the 90th quantile of the review length.<br>
“”"</p>
<p>reviewLen90 = np.quantile(numWords, 0.90)<br>
print(“90th quantile of review length:”, reviewLen90)</p>
<p>“”"<br>
Thus, 90% of reviews are of 259 words or shorter. We’ll set maxlen close to this.<br>
“”"<br>
maxlen = 238</p>
<p>labels = np.array(df[“Sentiment”])<br>
<span class="hashtag">#skf</span> = StratifiedKFold(n_splits=5)<br>
<span class="hashtag">#for</span> trn_idx, tst_idx in skf.split(labels.reshape((-1, 1)), labels):</p>
<h1>
<a name="break-10" class="anchor" href="#break-10"></a>break</h1>
<p><span class="hashtag">#train_df</span>, test_df = df.iloc[trn_idx], df.iloc[tst_idx]<br>
test_df = testdf<br>
train_df = df<br>
print(“Shape of train and validation dataframes:”, train_df.shape, test_df.shape)</p>
<p>“”"<br>
Now let us create a numpy array containing the word vectors. Later this numpy array will be used for initilaizing<br>
the embedding layer in the model.<br>
“”"</p>
<p><span class="hashtag">#vocab</span> = list(en_model.vocab.keys())<br>
vocab = list(en_model.index_to_key)<br>
print(“Vocab size in pretrained model:”, len(vocab))</p>
<h1>
<a name="check-if-the-word-and-is-present-in-the-pretrained-model-11" class="anchor" href="#check-if-the-word-and-is-present-in-the-pretrained-model-11"></a>check if the word ‘and’` is present in the pretrained model</h1>
<p>assert “and” in en_model</p>
<h1>
<a name="check-the-dimension-of-the-word-vectors-12" class="anchor" href="#check-the-dimension-of-the-word-vectors-12"></a>check the dimension of the word vectors</h1>
<p>assert embeddingDim == len(en_model[“and”])</p>
<h1>
<a name="initialize-a-numpy-matrix-which-will-store-the-word-vectors-13" class="anchor" href="#initialize-a-numpy-matrix-which-will-store-the-word-vectors-13"></a>initialize a numpy matrix which will store the word vectors</h1>
<h1>
<a name="first-row-is-for-the-padding-token-14" class="anchor" href="#first-row-is-for-the-padding-token-14"></a>first row is for the padding token</h1>
<p>pretrained_weights = np.zeros((1+len(vocab), embeddingDim))</p>
<h1>
<a name="tqdm-just-adds-a-progress-bar-15" class="anchor" href="#tqdm-just-adds-a-progress-bar-15"></a>tqdm just adds a progress bar</h1>
<p>for i, token in enumerate(vocab):<br>
pretrained_weights[i, :] = en_model[token]</p>
<h1>
<a name="map-tokens-in-the-vocab-to-ids-16" class="anchor" href="#map-tokens-in-the-vocab-to-ids-16"></a>map tokens in the vocab to ids</h1>
<p>vocab = dict(zip(vocab, range(1, len(vocab)+1)))</p>
<h1>
<a name="in40-17" class="anchor" href="#in40-17"></a>In[40]:</h1>
<p>def reviewText2Features(reviewText):<br>
“”"<br>
Function which takes review text (basically a string!) as input and returns a features matrix X of shape<br>
(maxlen, embeddingDim). This is done by splitting the review into words and then representing each word by it’s<br>
word vector obtained from the Word2Vec model. Sentences having more than maxlen words are truncated while shorter<br>
ones are zero-padded by pre-adding all zero vectors.<br>
“”"<br>
X = []</p>
<pre><code>reviewWords = re_wordMatcher.findall(reviewText.lower())

"""
Tokenize the review using the word-matching regex and get its word vector from the pretrained Word2Vec model.
Words not found in the Word2Vec model are ignored
"""
for i, word in enumerate(reviewWords):
    if word not in en_model:
        continue
    if i &gt;= maxlen:
        break
    # X.append(en_model[word])
    X.append(vocab[word])

"""
Add zero padding in the begining of the sequence if the number of words is less than maxlen.
"""
if len(X) &lt; maxlen:
    # zero_padding = [[0.]*embeddingDim]*(maxlen - len(X))
    zero_padding = [0.]*(maxlen - len(X))
    X = zero_padding + X

return X # np.array(X)
</code></pre>
<p>def row2Features(row):<br>
“”"<br>
Function which takes a datafram row as input and produces features and labels.</p>
<pre><code>Input: row | Type: pandas.core.series.Series

Output: X, y | Type: X - np.ndarray of shape (maxlen, embeddingDim) &amp; y - int where Positive = 0 &amp; Negative = 1
"""    

X = reviewText2Features(row["reviewText"])
y = reviewText2Features(row["Sentiment"])
    
return X, y
</code></pre>
<h1>
<a name="in41-18" class="anchor" href="#in41-18"></a>In[41]:</h1>
<p>“”"<br>
Now apply the above function on a sample row<br>
“”"<br>
sampleRow = df.iloc[0]<br>
reviewWords = re_wordMatcher.findall(sampleRow[“reviewText”].lower())<br>
print(“Review:”, sampleRow[“reviewText”])<br>
print(“Rating:”, sampleRow[“overall”])<br>
print(“Sentiment:”, sampleRow[“Sentiment”])<br>
print(“Review words:”, reviewWords)</p>
<p>“”"<br>
Give the sample row to the function row2Features<br>
“”"<br>
X, y = row2Features(sampleRow)<br>
print(“Dimension of X:”, len(X))<br>
print(“Label y:”, y)</p>
<h1>
<a name="in42-19" class="anchor" href="#in42-19"></a>In[42]:</h1>
<p>def shuffleArray(X, y):<br>
idx = np.arange(X.shape[0])<br>
np.random.shuffle(idx)<br>
X = X[idx, :]<br>
y = y[idx]<br>
return X, y</p>
<p>def generateModelReadyData(data, batchSize = batchSize, shuffle=False):<br>
“”"<br>
Generator function which generates features and labels in batches</p>
<pre><code>Input:
data - DataFrame where each row has review and sentiment
batchSize - No. of rows for which features will be created and returned in a batch.
Note: This is useful for running mini-batch Gradient Descent optimization when the dataset is large.

Output:
X - 3D np.ndarray of shape (batchSize, maxlen, embeddingDim)
y - 1D np. array of shape (batchSize,)        
"""

while(True):
    X = []
    y = []
    for _, row in data.iterrows():
        """Generate features and label for this row"""
        X_, y_ = row2Features(row)

        """Keep accumulating the row-wise features"""
        X.append(X_)
        y.append(y_)   

        """If number of rows processed is greater than batchSize yield the batch and trim down X &amp; y
        Note: This way we avoid running into memory issues by not bloating X and y bigger and bigger
        """
        if len(X) &gt; batchSize:
            temp_X, temp_y = np.array(X[:batchSize]), np.array(y[:batchSize])
            if shuffle:
                temp_X, temp_y = shuffleArray(temp_X, temp_y)
            
            X, y = X[batchSize:], y[batchSize:]                    
            yield temp_X, temp_y

    """Yield the remaining few rows when number of rows in data isn't a mutiple of batchSize"""
    if len(X) &gt; 0:
        temp_X, temp_y = np.array(X), np.array(y)
        if shuffle:
            temp_X, temp_y = shuffleArray(temp_X, temp_y)
        
        yield temp_X, temp_y
</code></pre>
<h1>
<a name="model-training-in-pytorch-20" class="anchor" href="#model-training-in-pytorch-20"></a>#### Model Training in Pytorch</h1>
<h1>
<a name="in43-21" class="anchor" href="#in43-21"></a>In[43]:</h1>
<p>class SentimentNet(nn.Module):<br>
def <strong>init</strong>(self, embedding_dim, hidden_dim, vocab_size, pretrained_weights):<br>
super(SentimentNet, self).<strong>init</strong>()</p>
<pre><code>    self.embedding=nn.Embedding(vocab_size, embedding_dim)
    self.embedding.weight.data.copy_(torch.from_numpy(pretrained_weights))
    
    """
    Adding a dropout layer to force some of the feature values to zero.
    Note: Dropout is a regularization technique which sets the activation of few randomly chosen neurons of
    a hidden layer to zero. It can also be applied to the input layer where some of the input features are set to zero.
    For more details refer http://jmlr.org/papers/v15/srivastava14a.html
    """
    self.sentInputDropout = nn.Dropout(0.1)
    
    """
    Now let's stack a couple of bidirectional RNNs to process the input sequence and extract features
    BiLSTM with 1 layer as hiddenlayer size. Use dropout=0.1. Use 1 dense hidden layer of size 50 before the final output layer
    """
    self.biLSTM1 = nn.LSTM(embedding_dim, hidden_dim[0], bidirectional=True, batch_first=True)
    self.biLSTMDropOut = nn.Dropout(0.1)
    self.dropout1 = nn.Dropout(0.1)
    self.dense1 = nn.Linear(2*hidden_dim[0], 50)
    self.relu1 = nn.ReLU()
    self.dropout2 = nn.Dropout(0.1)
    
    self.outputLayer = nn.Linear(50, 1)
    self.sigmoid = nn.Sigmoid()
    
    self.hidden_dim = hidden_dim
    
    
 
    
def forward(self, x):
    
    batch_len = x.shape[0]
    out = self.embedding(x)
    out = self.sentInputDropout(out)
    out, hidden = self.biLSTM1(out)
    out = self.biLSTMDropOut(out)
    out = self.dropout1(out)
    out = self.dense1(out)
    out = self.relu1(out)
    out = self.dropout2(out)
    
    out = self.outputLayer(out)
    out = self.sigmoid(out)
    out = out.view(batch_len, -1)
    out = out[:,-1]
    return out  
</code></pre>
<h1>
<a name="in47-22" class="anchor" href="#in47-22"></a>In[47]:</h1>
<p>model = SentimentNet(embeddingDim, [batchsize,100], 1+len(vocab), pretrained_weights)<br>
<span class="hashtag">#model</span> = SentimentNet(size_of_vocab, embedding_dim, num_hidden_nodes,num_output_nodes, num_layers,<br>
# bidirectional = True, dropout = dropout)<br>
model.to(device)<br>
len(vocab)</p>
<h1>
<a name="in48-23" class="anchor" href="#in48-23"></a>In[48]:</h1>
<p>lr=0.005<br>
criterion = nn.BCELoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=lr)</p>
<h1>
<a name="in50-24" class="anchor" href="#in50-24"></a>In[50]:</h1>
<p>epochs = 5</p>
<p>torch.manual_seed(0)<br>
random.seed(0)<br>
np.random.seed(0)<br>
<span class="hashtag">#torch</span>.use_deterministic_algorithms(True)</p>
<p>counter = 0<br>
print_every = 1000<br>
clip = 5<br>
valid_loss_min = np.Inf<br>
batchSize=128</p>
<p>model = model.float()<br>
model.train()</p>
<h1>
<a name="in51-25" class="anchor" href="#in51-25"></a>In[51]:</h1>
<p>for i in range(epochs):<br>
print(“Epoch:”, i+1)<br>
<span class="hashtag">#h</span> = model.init_hidden(batchSize)<br>
print(“Running a pass over the training data…”)<br>
for j, (inputs, labels) in enumerate(generateModelReadyData(train_df, batchSize=batchSize, shuffle=True)):<br>
if j &gt;= np.ceil(train_df.shape[0]/batchSize):<br>
break</p>
<pre><code>#for inputs, labels in train_loader:
    counter += 1
    #h = tuple([e.data for e in h])
    inputs, labels = torch.from_numpy(inputs), torch.from_numpy(labels)
    inputs, labels = inputs.to(device), labels.to(device)
    model.zero_grad()
    #output, h = model(inputs, h)
    output = model(inputs.long())
    #print(output.shape)
    #print(output)
    loss = criterion(output.squeeze(), labels.float())
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), clip)
    optimizer.step()        
    if (j+1) % 100 == 0:
        print("Batches completed:", j+1)

print("Batches completed:", j+1)

#val_h = model.init_hidden(batch_size)
val_losses = []
model.eval()
print("Running a pass over the test data...")
for k, (inp, lab) in enumerate(generateModelReadyData(test_df, batchSize=200, shuffle=False)):
    if k &gt;= np.ceil(test_df.shape[0]/200):
        break
#for inp, lab in val_loader:
    #val_h = tuple([each.data for each in val_h])
    inp, lab = torch.from_numpy(inp), torch.from_numpy(lab)
    inp, lab = inp.to(device), lab.to(device)
    out = model(inp.long())
    val_loss = criterion(out.squeeze(), lab.float())
    val_losses.append(val_loss.item())
    if (k+1) % 100 == 0:
        print("Batches completed:", k+1)

print("Batches completed:", k+1)

model.train()
print("Epoch: {}/{}...".format(i+1, epochs),
      "Step: {}...".format(counter),
      "Loss: {:.6f}...".format(loss.item()),
      "Val Loss: {:.6f}".format(np.mean(val_losses)))
if np.mean(val_losses) &lt;= valid_loss_min:
    torch.save(model.state_dict(), './state_dict.pt')
    print('Validation loss decreased ({:.6f} --&gt; {:.6f}).  Saving model ...'.format(valid_loss_min,np.mean(val_losses)))
    valid_loss_min = np.mean(val_losses)
</code></pre>
<h1>
<a name="in-26" class="anchor" href="#in-26"></a>In[ ]:</h1>
<p>The Dataset is: Amazon review. Considered only Review and Overall rating.</p>

vdw: <p>Here’s your problem, I think:</p>
<pre><code class="lang-auto">...
out, hidden = self.biLSTM1(out)
...
out = out.view(batch_len, -1)
...
</code></pre>
<p><code>out</code> contains the hidden states for <strong>all</strong> time steps. So when you flatten <code>out</code> you get something of which you try to fix with <code>out = out[:,-1]</code>. I would recommend using <code>hidden</code>, which contains only the last hidden state(s) for the last time step – well just 1 state since you have <code>num_layers=1</code>. As you’re using <code>bidirectional=True</code> things are a tad more tricky. Try the following code snippet:</p>
<pre><code class="lang-auto">...
out, hidden = self.biLSTM1(out)
# Get the last hidden state in case num_layers &gt; 1; 2 reflects number of directions
hidden = self.hidden[0].view(1, 2, batch_len, self.hidden_dim)[-1]
h_1, h_2 = hidden[0], hidden[1]
hidden = torch.cat((h_1, h_2), 1)  # Concatenate both states
</code></pre>

Smaranika_Sikdar: <p>I have only one hidden layer of size 100.</p>
<pre><code>def forward(self, x):
    
    batch_len = x.shape[0]
    out = self.embedding(x)
    out = self.sentInputDropout(out)
    out, hidden = self.biLSTM1(out)
    # Get the last hidden state in case num_layers &gt; 1; 2 reflects number of directions
    hidden = self.hidden[0].view(1, 2, batch_len, self.hidden_dim)[-1]
    h_1, h_2 = hidden[0], hidden[1]
    hidden = torch.cat((h_1, h_2), 1)
    out = self.biLSTMDropOut(out)
    
    out = self.dense1(out)
    out = self.relu1(out)
    out = self.dropout2(out)
    
    out = self.outputLayer(out)
    out = self.sigmoid(out)
    out = out.view(batch_len, -1)
    out = out[:,-1]
    out, hidden = self.biLSTM1(out)

    return out  
</code></pre>
<p>error:<br>
ModuleAttributeError: ‘SentimentNet’ object has no attribute ‘hidden’</p>

vdw: <aside class="quote no-group" data-username="Smaranika_Sikdar" data-post="3" data-topic="167069">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smaranika_sikdar/48/55158_2.png" class="avatar"> Smaranika_Sikdar:</div>
<blockquote>
<p><code>hidden = self.hidden[0].view(1, 2, batch_len, self.hidden_dim)[-1]</code></p>
</blockquote>
</aside>
<p>Sorry, I’ve copy-and-pasted to quickly. It needs to be:</p>
<pre><code class="lang-auto">hidden = hidden[0].view(1, 2, batch_len, self.hidden_dim)[-1]
</code></pre>
<p>You need to have a <code>self.hidden_dim=hidden_dim</code> in your <code>__init__()</code> method. I’m actually not quite sure why <code>hidden_dim</code> contains the batch size. It’s an odd practice :). For testing, you can simply add <code>self.hidden_dim=100</code> to your <code>__init__()</code> method.</p>

Smaranika_Sikdar: <p>def forward(self, x, hidden_dim=[100]):</p>
<pre><code>    batch_len = x.shape[0]
    out = self.embedding(x)
    out = self.sentInputDropout(out)
    out = self.biLSTM1(out)
    # Get the last hidden state in case num_layers &gt; 1; 2 reflects number of directions
    hidden = hidden_dim[0].view(1, 2, batch_len, self.hidden_dim)[-1]
    h_1, h_2 = hidden_dim[0], hidden_dim[1]
    hidden = torch.cat((h_1, h_2), 1)
    
    out = self.biLSTMDropOut(out)
    
    out = self.dense1(out)
    out = self.relu1(out)
    out = self.dropout2(out)
    
    out = self.outputLayer(out)
    out = self.sigmoid(out)
    out = out.view(batch_len, -1)
    out = out[:,-1]
    out, hidden = self.biLSTM1(out)

    return out 
</code></pre>
<p>Input In [80], in SentimentNet.forward(self, x, hidden_dim)<br>
40 out = self.biLSTM1(out)<br>
41 # Get the last hidden state in case num_layers &gt; 1; 2 reflects number of directions<br>
—&gt; 42 hidden = hidden_dim[0].view(1, 2, batch_len, self.hidden_dim)[-1]<br>
43 h_1, h_2 = hidden_dim[0], hidden_dim[1]<br>
44 hidden = torch.cat((h_1, h_2), 1)</p>
<p>AttributeError: ‘int’ object has no attribute ‘view’</p>

vdw: <p>Well, the details are up to you. For example, you initialize your LSTM layer with <code>hidden_dim[0]</code> where hidden_dim seems to be <code>[batchsize,100]</code> according to your code. So you set the hidden dimension of the LSTM to the batch size. Which is weird. Also my snippet didn’t say</p>
<pre><code class="lang-auto">hidden = hidden_dim[0].view(1, 2, batch_len, self.hidden_dim)[-1]
h_1, h_2 = hidden_dim[0], hidden_dim[1]
</code></pre>
<p>but</p>
<pre><code class="lang-auto">hidden = hidden[0].view(1, 2, batch_len, self.hidden_dim)[-1]
h_1, h_2 = hidden[0], hidden[1]
</code></pre>
<p>Since you’re using an LSTM, <code>hidden</code>  is a tuple <code>(h, c)</code> where <code>h</code> is the “real” hidden state and <code>c</code> is the cell state. So <code>hidden[0]</code> gets you this real hidden state that you want/need.</p>