Forward pass gets 10000x slower after iterating for a while

Arash_Vahabpour: <p>I implemented a simple Deconv network just like pytorch’s official DCGAN tutorial.<br>
I repeatedly pass a <code>zeros</code> vector to it. The time taken slows down significantly after a while. I am wondering what the reason is and how I can resolve it.</p>
<p>Code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import time

# JUST TO MEASURE TIME
class Timer:
    def __init__(self, msg):
        self.msg = msg
        
    def __enter__(self):
        self.start = time.process_time()
        return self

    def __exit__(self, *args):
        self.end = time.process_time()
        self.interval = self.end - self.start
        
        print('{}: {:.5f}'.format(self.msg, self.interval))
        
device = torch.device("cuda")

ngf, nc, nz, batchSize = 64, 1, 6, 1&lt;&lt;16
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 4, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 4 x 4
            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 8 x 8
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 16 x 16
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 32 x 32
        )

    def forward(self, input):
        return self.main(input)
    
# Create the generator
netG = Generator().to(device)

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

netG.apply(weights_init)

# torch.backends.cudnn.benchmark=True

while True:
    with Timer('Time elapsed'):
        with torch.no_grad():
            netG(torch.zeros([batchSize, nz, 1, 1], device=device))
</code></pre>
<p>Result:</p>
<blockquote>
<p>Time elapsed: 0.02309<br>
Time elapsed: 0.00072<br>
Time elapsed: 0.00208<br>
Time elapsed: 0.00128<br>
Time elapsed: 0.00119<br>
Time elapsed: 0.00153<br>
Time elapsed: 0.00176<br>
Time elapsed: 0.00170<br>
Time elapsed: 0.00185<br>
Time elapsed: 0.00188<br>
Time elapsed: 0.00191<br>
Time elapsed: 0.00190<br>
Time elapsed: 0.00171<br>
Time elapsed: 0.00176<br>
Time elapsed: 0.00167<br>
Time elapsed: 0.00120<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00169<br>
Time elapsed: 0.00166<br>
Time elapsed: 0.00167<br>
Time elapsed: 0.00171<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00169<br>
Time elapsed: 0.00177<br>
Time elapsed: 0.00173<br>
Time elapsed: 0.00176<br>
Time elapsed: 0.00173<br>
Time elapsed: 0.00171<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00173<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00178<br>
Time elapsed: 0.00169<br>
Time elapsed: 0.00171<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00169<br>
Time elapsed: 0.00169<br>
Time elapsed: 0.00173<br>
Time elapsed: 0.00154<br>
Time elapsed: 0.00170<br>
Time elapsed: 0.00167<br>
Time elapsed: 0.00224<br>
Time elapsed: 0.00117<br>
Time elapsed: 0.00175<br>
Time elapsed: 0.00168<br>
Time elapsed: 0.00173<br>
Time elapsed: 0.00169<br>
Time elapsed: 12.62760<br>
Time elapsed: 12.71425<br>
Time elapsed: 12.71379<br>
Time elapsed: 12.71846<br>
Time elapsed: 12.71909<br>
Time elapsed: 12.71898<br>
Time elapsed: 12.72288<br>
Time elapsed: 12.72157<br>
Time elapsed: 12.72226<br>
Time elapsed: 12.72456<br>
Time elapsed: 12.72350<br>
Time elapsed: 12.72480<br>
Time elapsed: 12.72644<br>
Time elapsed: 12.72337<br>
Time elapsed: 12.72424<br>
Time elapsed: 12.72538<br>
Time elapsed: 12.72533<br>
Time elapsed: 12.72510<br>
Time elapsed: 12.72507<br>
Time elapsed: 12.72806<br>
Time elapsed: 12.72865<br>
Time elapsed: 12.72764<br>
Time elapsed: 12.72431</p>
</blockquote>
<p>My GPU: Titan RTX<br>
PyTorch version: 1.4</p>

Arash_Vahabpour: <p>This is a problem particularly when I take a very large batch size for GPU memory.</p>

ptrblck: <p>CUDA calls are asynchronous, so your current <code>Timer</code> class will most likely only time the Python overhead and the kernel launch.<br>
To properly time the kernels, you would have to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.</p>

Arash_Vahabpour: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="68796">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.cuda.synchronize()</p>
</blockquote>
</aside>
<p>Your point is correct, but regardless, the overall code was previously taking unusually long.<br>
After I wrapped my code with torch.cuda.synchronize() things got normal. I assume the problem previously was that a long buffer of these calls was somehow forming and ruining everything. Am I correct?</p>

Arash_Vahabpour: <p>Now each batch is taking 0.25 seconds, is that normal?</p>

ptrblck: <p>That seems unlikely, as you are passing 65k samples at once.<br>
I see a forward time of ~12.5s per iterations on a V100.</p>