Small LSTM slower than large LSTM on GPU

username: LittleResearcher
cooked: <p>I was experimenting the GPU inference time using different sizes of LSTM. It turns out that when the LSTM hidden size is 200 and 400, the inference time is extremely slow (even much slower than hidden size =700). Below are the code and the relationship between hidden_size and average_sequence_inference_time (in seconds).</p>
<p>Other info:<br>
GPU: 1 RTX2070<br>
torch version: 1.6.0</p>
<p>Eval setting: only 1 example per batch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1abad2e1035f3765d5a6f296628aa197007ea14c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1abad2e1035f3765d5a6f296628aa197007ea14c" title="Screen Shot 2022-02-08 at 8.50.17 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1abad2e1035f3765d5a6f296628aa197007ea14c_2_269x500.png" alt="Screen Shot 2022-02-08 at 8.50.17 AM" data-base62-sha1="3OsGiBYmUuUNEHLqeQCqmFVzPaY" width="269" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1abad2e1035f3765d5a6f296628aa197007ea14c_2_269x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1abad2e1035f3765d5a6f296628aa197007ea14c_2_403x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1abad2e1035f3765d5a6f296628aa197007ea14c_2_538x1000.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-02-08 at 8.50.17 AM</span><span class="informations">556×1032 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
import numpy as np
import time

import torch.nn as nn

class LSTMDummy(nn.Module):
    def __init__(self, embedding_size, input_size, hidden_size,
                 n_layers, dropout_p):
        super(LSTMDummy, self).__init__()

        self.embedding = nn.Embedding(embedding_size, input_size)
        self.lstm = nn.LSTM(input_size, hidden_size,
                            num_layers=n_layers,
                            dropout=dropout_p,
                            bidirectional=True,
                            batch_first=True)

    def forward(self, tensor):
        torch.cuda.synchronize()
        start_time = time.time()

        lstm_input = self.embedding(tensor)
        output = self.lstm(lstm_input)

        torch.cuda.synchronize()
        end_time = time.time()

        return end_time - start_time

class CheckGPUTensorMul:

    device = torch.device("cuda:0")
    n_exp = 10000

    sizes = [16, 25, 32, 50, 64, 100, 128, 200, 256, 300, 400, 500, 512, 600, 700, 750, 768, 800, 900, 1000, 1024, 2048]
    sizes2 = [200, 750]

    @classmethod
    def calculate_time_lstm(cls):

        n_vocab = 10000
        seq_len = 20

        input_tensors = [torch.tensor([[np.random.randint(0, n_vocab)] * seq_len], dtype=torch.int64).to(cls.device)
            for i in range(cls.n_exp)]  # n_exp = 10,000

        for size in cls.sizes:   # 16, 32, 50, 64 ...
            time_list = []

            lstm_model = LSTMDummy(
                embedding_size=n_vocab,
                input_size=100,
                hidden_size=size,
                n_layers=1,
                dropout_p=0.0
            )
            lstm_model.to(cls.device)
            lstm_model.eval()

            # Warm up
            for input_tensor in input_tensors[:100]:
                time_ = lstm_model(input_tensor)

            for input_tensor in input_tensors:
                time_ = lstm_model(input_tensor)

                time_list.append(time_)

            print("=" * 40)
            print("size:", size)
            print("avg time:", sum(time_list)/len(time_list))
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/littleresearcher">@LittleResearcher</a> It looks like some random processing bottleneck that the GPU might have experienced.</p>
<p>I tried on my GTX 1060Ti and the time taken is linear</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>hidden_size</th>
<th>average time</th>
</tr>
</thead>
<tbody>
<tr>
<td>16</td>
<td>0.0005613381147</td>
</tr>
<tr>
<td>25</td>
<td>0.0005528689623</td>
</tr>
<tr>
<td>32</td>
<td>0.0005538116217</td>
</tr>
<tr>
<td>50</td>
<td>0.0005623131752</td>
</tr>
<tr>
<td>64</td>
<td>0.0006439544201</td>
</tr>
<tr>
<td>100</td>
<td>0.0005706382751</td>
</tr>
<tr>
<td>128</td>
<td>0.0005544938326</td>
</tr>
<tr>
<td>200</td>
<td>0.0005606959343</td>
</tr>
<tr>
<td>256</td>
<td>0.0006359843731</td>
</tr>
<tr>
<td>300</td>
<td>0.0007430159807</td>
</tr>
<tr>
<td>400</td>
<td>0.0009713747263</td>
</tr>
<tr>
<td>500</td>
<td>0.001388325262</td>
</tr>
<tr>
<td>512</td>
<td>0.001418832326</td>
</tr>
<tr>
<td>600</td>
<td>0.001854181433</td>
</tr>
<tr>
<td>700</td>
<td>0.002511228943</td>
</tr>
<tr>
<td>750</td>
<td>0.00277370615</td>
</tr>
<tr>
<td>768</td>
<td>0.002898940682</td>
</tr>
<tr>
<td>800</td>
<td>0.003245669317</td>
</tr>
<tr>
<td>900</td>
<td>0.004082291055</td>
</tr>
<tr>
<td>1000</td>
<td>0.004929169559</td>
</tr>
</tbody>
</table>
</div>

username: LittleResearcher
cooked: <p>Thanks! I just did this with a Tesla P4 and the increase seems to be linear. So I think this is a GPU-specific observation.</p>