Feeding data into LSTM with batches

username: Aref_Ghodamai1
cooked: <p>I’m trying to make a NLP NER LSTM model, and my data looks like this:<br>
input:</p>
<pre><code class="lang-auto">input = [["i","am","here"], ["you","are","not","there"]]
</code></pre>
<p>and output:</p>
<pre><code class="lang-auto">output = [[1, 2,1], [1,2,1,2]]
</code></pre>
<p>As you can see, the length of input tensors are not the same. for <code>batch_size =1</code>, but increasing batch size to any thing bigger that 1, creates this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "stage_runner.py", line 28, in &lt;module&gt;
    main()
  File "stage_runner.py", line 24, in main
    job_table[args.job](**params)
  File "/Users/arefghodamai/Desktop/Projects/key_value_extraction/src/dvc_dags/train_model.py", line 22, in train_model
    for sentence, tags in model_manager.data_loader:
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 83, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]
  File "/usr/local/Cellar/python@3.7/3.7.10_3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
</code></pre>
<p>Is there any way to make this right, besides padding?</p>

username: AbdulsalamBande
cooked: <p>Hello <a class="mention" href="/u/aref_ghodamai1">@Aref_Ghodamai1</a> . You have to know that every sentence in a batch has to have the same length. So in your case <code>["i","am","here"]</code> has a length of <code>3</code> while [“you”,“are”,“not”,“there”] has a length of <code>4</code>, and this is a problem. What you can do is to make [“i”,“am”,“here”] to  [“i”,“am”,“here”,“pad_tensor”]  or clip [“you”,“are”,“not”,“there”] to [“you”,“are”,“not”].<br>
The point is to make sure the sentences have same length. Thank you</p>

username: djlindsay
cooked: <p>Pad, my brother. Or cut N-Grams (look em up) with the lengths of the minimum length input. Here’s what I did in my code. The batch function is separate.</p>
<pre><code class="lang-auto"># create sequences of length 5 tokens
def create_seq(text, seq_len = 5):
    
    sequences = []
    words=text.split(' ')

    # if the number of tokens in 'text' is greater than 5
    if len(words) &gt; seq_len:
        for i in range(seq_len, len(words)+1):
            # select sequence of tokens
            seq = words[i-seq_len:i]
            # add to the list
            sequences.append(" ".join(seq))
    elif len(words) &lt; seq_len:
        seq=words
        for i in range(len(words),seq_len):
            seq.append("&lt;pad&gt;")
        sequences.append(" ".join(seq))

    # if the number of tokens in 'text' is equal to 5
    else:
        sequences=[text]
    return sequences

seqs5=sum([create_seq(i) for i in clean_lines],[]) # clean lines is a big list of sentences (strings)
seqs5
</code></pre>