Training on RTX3090, tensor.to(device) very slow on some networks

Kingsmove: <h2>Issue description</h2>
<p>Recently our lab set up a new machine with RTX3090, I installed GPU driver 460.32, CUDA 11.2, and pytorch through</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
</code></pre>
<p>Then I tested with a seq2seq model (LSTM-&gt;LSTM) I used before, training very fast, working fine.</p>
<p>However, when I use this machine to train a TextCNN classification model, I find it is much slower than even when I did it on my laptop with GTX1660ti (cuda10.2+torch1.7.0).</p>
<p>There is no error message during training, though. To analyze where the problem is, I just added some timer in the program and found the difference.</p>
<pre><code class="lang-auto">    for step, (x_batch, y_batch) in enumerate(train_loader):
        t1=time.clock()
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        t2=time.clock()
        print('transfer data needs %s ms' % ((t2 - t1) * 1000))

        output = model(x_batch)
        #training steps...
        t3=time.clock()
        print('training step uses %s ms' % ((t3 - t2) * 1000))
</code></pre>
<p>On my laptop, the data loading time is just 0.06ms, the following training steps take about 3.5~4.5ms.<br>
<strong>However, on the 3090 machine, the data loading time needs about 102ms, while training steps only take only 2.3ms.</strong><br>
I also tried the same training operation on the old server of my lab with RTX2080 (cuda11.0 torch1.7.1), data loading time 0.03ms, training steps take 1.9ms. (P.S. I also noticed that on 2080 machine the ‘GPU-Util Compute M.’ (can be seen on nvidia-smi) was just about 54% through the entire training procedure, but the 3090 machine kept 100% from the start.)<br>
The codes put on these 3 machine are exactly the same, but why the tensor.to(device) on 3090 machine is so slow?<br>
(By the way, I also tried a 3MLP GAN network which works fine on my laptop and old 2080 server, too…But again, tensor.to(device) on 3090 is slow…)</p>
<p>The data loading methods of Seq2Seq and TextCNN are also almost the same, using</p>
<pre><code class="lang-auto">train = torch.utils.data.TensorDataset(x_train, y_train)
train_loader = DataLoader(train, batch_size=BATCH_SIZE, shuffle=True)
</code></pre>
<h2>System Info</h2>
<p>Here is the system info of that new server with RTX3090.</p>
<p>PyTorch version: 1.7.1<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.0<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.7 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: 11.2.142<br>
GPU models and configuration: GPU 0: GeForce RTX 3090<br>
Nvidia driver version: 460.32.03<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip] numpy==1.19.5<br>
[pip] numpydoc==0.9.2<br>
[pip] torch==1.7.1<br>
[pip] torchaudio==0.7.2<br>
[pip] torchvision==0.8.2+cu110<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               11.0.221             h6bb024c_0<br>
[conda] mkl                       2020.0                      166<br>
[conda] mkl-service               2.3.0            py37he904b0f_0<br>
[conda] mkl_fft                   1.0.15           py37ha843d7b_0<br>
[conda] mkl_random                1.1.0            py37hd6b4f25_0<br>
[conda] numpy                     1.18.1           py37h4f9e942_0<br>
[conda] numpy-base                1.18.1           py37hde5b4d6_1<br>
[conda] numpydoc                  0.9.2                      py_0<br>
[conda] pytorch                   1.7.1           py3.7_cuda11.0.221_cudnn8.0.5_0    pytorch<br>
[conda] torchaudio                0.7.2                    pypi_0    pypi<br>
[conda] torchvision               0.8.2+cu110              pypi_0    pypi</p>

ptrblck: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the timer.<br>
Also, note that the very first CUDA operation initializes the CUDA context etc. and is thus slower.<br>
You could use the <code>torch.utils.benchmark</code> utilities to profile specific operations, which would make sure to properly add warmup iterations and synchronizations.</p>
<p>If you are seeing a performance regression in the model execution, please post the model definition (if possible), so that we could test it with the latest cudnn release and check the performance on this GPU.<br>
That being said, note that we are currently facing an issue in the PyTorch binaries with statically linked cudnn, which removes xmma kernels as described <a href="https://github.com/pytorch/pytorch/issues/50153">here</a>, and we are still working on forcing these kernels into the binaries.</p>

Kingsmove: <p>Thank you for your reply!<br>
From a macro point of view, because I set the result to be printed every 100 steps, I just saw that on my 1660ti laptop with cuda10.2 and torch1.7.0, this output is very fast (because textcnn is not a very complicated network), and about 2 lines of output per second can be jumped out. But on 3090, I need to wait for nearly 10 seconds to see a result output.</p>
<p>Also thank you for pointing out the asynchronization problem. I am sorry but I am not so sure how to use <code>torch.utils.benchmark</code> in multiple lines code… But I revised my timing method as</p>
<pre><code class="lang-auto">    for step, (x_batch, y_batch) in enumerate(train_loader):
        torch.cuda.synchronize()
        t1=time.clock()
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        torch.cuda.synchronize()
        t2=time.clock()
        print('transfer data needs %s ms' % ((t2 - t1) * 1000))

        torch.cuda.synchronize()
        t3=time.clock()
        output = model(x_batch)
        output = output.squeeze()
        loss = loss_func(output, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        torch.cuda.synchronize()
        t4=time.clock()
        print('training step uses %s ms' % ((t4 - t3) * 1000))
</code></pre>
<p>As I usually train the network for 7 epochs, to make the network ‘warm-up’, I just record the time result of the last steps of the last epoch.<br>
The situation indeed changed! <strong>This time on the 3090 machine, the data loading time needs about 0.038ms, while training steps take around 106ms.</strong> While my laptop still needs around 0.07ms to transfer data and 4ms to do the train step…</p>
<p>Then the problem becomes why the GPU training is slow on 3090, so I used another method <code>torch.autograd.profiler.profile()</code> to see what happens:</p>
<pre><code class="lang-auto">    for step, (x_batch, y_batch) in enumerate(train_loader):
        torch.cuda.synchronize()
        t1=time.clock()
        with torch.autograd.profiler.profile(use_cuda=True) as prof:
            x_batch = x_batch.to(device)
            y_batch = y_batch.to(device)
        torch.cuda.synchronize()
        t2=time.clock()
        print('transfer data needs %s ms' % ((t2 - t1) * 1000))
        print(prof)
</code></pre>
<p>And I also use this method on training part mentioned above.<br>
Due to it that the result is too long, I will add two pictures. It seems that the <code>aten::cudnn_convolution</code> operation took the most time? Which seems to have some similarities with the problem you mentioned. (We both used a convolutional structure, I will post my model definition on the next reply.)</p>
<p>Finally, by the way, though I do not quite understand the linked binary… as it seems that the problem happens on cudnn, I just remember that I installed the cudnn library through <a href="https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#package-manager-ubuntu-install" rel="noopener nofollow ugc">package-manager-ubuntu-install</a>, maybe this can also be a factor causing this problem? I will try to copy cudnn to <code>/usr/local/cuda-11.2/</code> rather than <code>/usr/lib/x86_64-linux-gnu/</code> later.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abe32a325dbcb58b62187c7a6e922003ea532554.png" data-download-href="https://discuss.pytorch.org/uploads/default/abe32a325dbcb58b62187c7a6e922003ea532554" title="cost_of_data_transfer"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abe32a325dbcb58b62187c7a6e922003ea532554_2_690x132.png" alt="cost_of_data_transfer" data-base62-sha1="owAlrsmIGSGYqqTf9ySCpufyoDy" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abe32a325dbcb58b62187c7a6e922003ea532554_2_690x132.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abe32a325dbcb58b62187c7a6e922003ea532554_2_1035x198.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abe32a325dbcb58b62187c7a6e922003ea532554_2_1380x264.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cost_of_data_transfer</span><span class="informations">1816×349 44.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48afcf805e5b79970cac9da0eeccacb1dde6b364.png" data-download-href="https://discuss.pytorch.org/uploads/default/48afcf805e5b79970cac9da0eeccacb1dde6b364" title="cost_of_training_steps"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48afcf805e5b79970cac9da0eeccacb1dde6b364_2_324x500.png" alt="cost_of_training_steps" data-base62-sha1="an1378wg2Kdc8747JZU1Bz6qT6A" width="324" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48afcf805e5b79970cac9da0eeccacb1dde6b364_2_324x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48afcf805e5b79970cac9da0eeccacb1dde6b364_2_486x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48afcf805e5b79970cac9da0eeccacb1dde6b364_2_648x1000.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cost_of_training_steps</span><span class="informations">2121×3273 479 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Kingsmove: <p>And here is my model definition. A typical TextCNN network used for NLP intention classification.</p>
<pre><code class="lang-auto">def create_emb_layer(weights_matrix, non_trainable=False):
    # how many words in dict (matrix), embedding dim
    num_embeddings, embedding_dim = weights_matrix.shape
    emb_layer = nn.Embedding(num_embeddings, embedding_dim, padding_idx=0)
    emb_layer.weight = nn.Parameter(torch.tensor(weights_matrix, dtype=torch.float32))
    if non_trainable:
        emb_layer.weight.requires_grad = False

    return emb_layer, num_embeddings, embedding_dim

class TextCNN(nn.Module):
    def __init__(self, weights_matrix, class_num, kernel_num, kernel_sizes):
        super(TextCNN, self).__init__()
        Ci = 1
        Co = kernel_num

        self.embd, embd_num, embd_dim = create_emb_layer(weights_matrix, False)
        self.convs1 = nn.ModuleList([nn.Conv2d(Ci, Co, (k, embd_dim)) for k in kernel_sizes])
        # self.dropout = nn.Dropout(dropout)
        self.fc = nn.Linear(Co * len(kernel_sizes), class_num)

    def forward(self, x):
        x = self.embd(x) # (batch_N, token_num(word in one sent), embd_dim)
        x = x.unsqueeze(1) # (N, Ci(channel, for text only 1, token_num, embd_dim)
        x = [F.relu(conv(x)).squeeze(3) for conv in self.convs1]
        x = [F.max_pool1d(i, i.size(2)).squeeze(2) for i in x]
        x = torch.cat(x, 1) # concat results of 3 size kernel
        # x = self.dropout(x)
        logit = self.fc(x)
        return logit
</code></pre>
<p>Before training, I extracted pre-trained word embeddings from fasttext English then use it to initialize embedding layer. The hyperparameters and model initialization code is:</p>
<pre><code class="lang-auto"># Hyperparameters
LR = 0.001
EPOCH = 7
BATCH_SIZE = 16
EMBD_DIM = 300
KERNEL_NUM = 16
KERNEL_SIZES = [3, 4, 5]
CLASS_NUM = 150

embedding_matrix = np.load('Embd_matrix_8000x300.npy')
model = TextCNN(embedding_matrix, CLASS_NUM, KERNEL_NUM, KERNEL_SIZES).to(device)
</code></pre>

ptrblck: <p>Thanks for the model definition! Could you post the input shapes you are using as well, please?</p>
<aside class="quote no-group" data-username="Kingsmove" data-post="3" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e68b1a/48.png" class="avatar"> Kingsmove:</div>
<blockquote>
<p>Finally, by the way, though I do not quite understand the linked binary… as it seems that the problem happens on cudnn, I just remember that I installed the cudnn library through</p>
</blockquote>
</aside>
<p>Your local CUDA toolkit and cudnn library won’t be used, if you’ve installed the conda binaries or the pip wheels.<br>
You could indeed build from source and compare the speed. If you are seeing a speedup it might come from a different cudnn version and/or from the usage of the missing xmma kernels in the binary.</p>
<p>Once you provide the input shapes, I could check the model perf. using the latest cudnn release as well as internal versions.</p>

Kingsmove: <p>Okay, my input includes X and Y, where the X are the token-sequences that comes from actual sentences which passed through the tokenizer, the shape is (18000, 23), as there is 18000 data and the padding size is 23.<br>
For example, part of the X is:</p>
<pre><code class="lang-auto">[[   1    8 2860   45    4   73    5  124    4  520    7   44    4  156
    49  553    2    0    0    0    0    0    0]
 [   1   15    7   23   11   13    5  124    4   14  121  242   52  401
    17  401    2    0    0    0    0    0    0] ...]
</code></pre>
<p>Then, Y are the label-tensors for each token-sequence in X, the shape is (18000,). For example, part of the Y is:</p>
<pre><code class="lang-auto">[0, 0, 0, 1, 1, ...]
</code></pre>
<p>And I use</p>
<pre><code class="lang-auto">train = torch.utils.data.TensorDataset(x_train, y_train)
train_loader = DataLoader(train, batch_size=BATCH_SIZE, shuffle=True)
</code></pre>
<p>to put them together to train.</p>

Kingsmove: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could indeed build from source and compare the speed. If you are seeing a speedup it might come from a different cudnn version and/or from the usage of the missing xmma kernels in the binary.</p>
</blockquote>
</aside>
<p>Thank you for the suggestion!!<br>
I built the pytorch from source following <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">our guide</a> with cuda 11.0 and finally succeed! (P.S. I had ever tried using cuda 11.0 + conda-binaries-based pytorch, still slow, <strong>so build from source and install should be the solution</strong>.)<br>
The running time finally comes to a satisfying level! Where the data loading took 0.038ms and <strong>training steps took only 2.05ms!</strong></p>
<p>Note that at first I failed to build pytorch because a library reference not find error, as I used cuda 11.2 (though NVIDIA said the cuda 11.2 should be compatible with cuda 11.0?) So I uninstalled cuda 11.2 and installed cuda 11.0 and cudnn as normal, then successfully built pytorch from source and installed it, tested the program, whose result finally became good.<br>
Really thank you for patiently helping me solve the problem! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>By the way, it seems that the local cuda and cudnn was indeed not used, as you said:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Your local CUDA toolkit and cudnn library won’t be used, if you’ve installed the conda binaries or the pip wheels.</p>
</blockquote>
</aside>
<p>But I still do not know why I can use conda-binaries-based pytorch normally on my old lab<br>
server with RTX2080, cuda11.0?.. Maybe because the latest Ampere architecture of RTX 30xx?</p>

ptrblck: <aside class="quote no-group" data-username="Kingsmove" data-post="7" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e68b1a/48.png" class="avatar"> Kingsmove:</div>
<blockquote>
<p>The running time finally comes to a satisfying level! Where the data loading took 0.038ms and <strong>training steps took only 2.05ms!</strong></p>
</blockquote>
</aside>
<p>That’s good to hear, so I assume an update in the cudnn version as well as potential calls into the missing kernels might have accelerated the workload! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Kingsmove" data-post="7" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e68b1a/48.png" class="avatar"> Kingsmove:</div>
<blockquote>
<p>Note that at first I failed to build pytorch because a library reference not find error, as I used cuda 11.2 (though NVIDIA said the cuda 11.2 should be compatible with cuda 11.0?)</p>
</blockquote>
</aside>
<p>Could you post the error message you were seeing? As we are building it with CUDA11.2, it would be good to see what wasn’t working in your setup.</p>
<aside class="quote no-group" data-username="Kingsmove" data-post="7" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e68b1a/48.png" class="avatar"> Kingsmove:</div>
<blockquote>
<p>But I still do not know why I can use conda-binaries-based pytorch normally on my old lab<br>
server with RTX2080, cuda11.0?.. Maybe because the latest Ampere architecture of RTX 30xx?</p>
</blockquote>
</aside>
<p>The conda binaries ship with the specified CUDA runtime (as well as cudnn, NCCL etc.), which is in your case 11.0. This CUDA release supports multiple GPU architectures, where your RTX2080 (sm_75) is included.</p>

Kingsmove: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="112812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you post the error message you were seeing? As we are building it with CUDA11.2, it would be good to see what wasn’t working in your setup.</p>
</blockquote>
</aside>
<p>I am really sorry but as I deleted ‘pytorch’ folder after I failed to build pytorch with CUDA 11.2, it may be hard to find the full error message at that time (By the way, I remember I used this CUDA 11.2 with cudnn 8.1, I installed it by decompressing <strong>cudnn-11.2-linux-x64-v8.1.0.77.tgz</strong>  , copying the files with</p>
<pre><code class="lang-auto">cp cuda/lib64/* /usr/local/cuda-11.2/lib64/
cp cuda/include/* /usr/local/cuda-11.2/include/
</code></pre>
<p>Fortunately, I remember some of the details when I encountered this problem. (As I googled some of the error messages.) <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> I hope it could do some help.</p>
<ol>
<li>
<p>I remember we need about 5990 files to build, right? Then my build error happened on about [5660/5990]. At that point the setup.py seemed building ‘test cudnn library’? or say ‘building with cudnn’  part? Then the error happened.</p>
</li>
<li>
<p>The final error message is<br>
<code>subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '32']' returned non-zero exit status 1.</code><br>
Before that there were many error messages coming out like:<br>
<code>libcublas.so: undefined reference to  xxxxxx</code><br>
And for now I can find only one exact record that I have ever searched with google… Sorry for that<br>
<code>libcublas.so: undefined reference to `cublasltbiimatmulalgogetheuristic@libcublaslt.so.11'</code></p>
</li>
</ol>
<p>I searched github and our forum but they were not my situation. Then I searched the internet and found that</p>
<blockquote>
<p>One person said in his blog that he is now using CUDA10.1, but he wants to compile a program based on CUDA10.0.<br>
As a result, a similar error would appear, like <code>libnvinfer.so: undefined reference to `__cudaPushCallConfiguration@libcudart.so.10.0'</code>.<br>
So he manually modified cuda100wrapper.map, disguising <code>cudaGetDeviceCount@libcudart.so.10.1</code> as <code>cudaGetDeviceCount@libcudart.so.10.0</code>.<br>
But he said he did this because of his work needs. He mentioned that actually it could be much faster to avoid this error if you can uninstall CUDA10.1 and then install CUDA10.0.</p>
</blockquote>
<p>So I unistalled CUDA11.2 and installed CUDA11.0 and also cudnn with <strong>cudnn-11.2-linux-x64-v8.1.0.77.tgz</strong>, and this time the building procedure was successful.</p>