akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataloader with custom modified randomization

Niamul_Quader: <p>Hello,</p>
<p>I am trying to implement a dataloader with the following sampling characteristics:</p>
<p>Essentially, given that there are N (e.g. N=10) data points in a dataset, I am trying to build a dataloader where the data points sampled in the second half of a batch will depend on the data points sampled in the first half of a batch. Let’s say that in the first half of the batch (batch size of 4, so half the batch size is 2), we sample datapoint 1 and datapoint 5. Let’s also assume that if we sample datapoint 1, we want to prioritize sampling datapoint 9 within that same batch. Since we have sampled datapoint 1 in the first half of the batch, we want to sample datapoint 9 and some other datapoint to complete the 4 datapoint sample, i.e.,</p>
<p>N: {1,2,3,4,5,6,7,8,9,10}<br>
Batch size=4<br>
Fist 2 elements of a batch: {1, 5}<br>
Desired 4 elements of this batch: {1,5,9,n}, where datapoint 9 is desired based on the presence of datapoint 1 in this batch, and n is some other random datapoint.</p>
<p>Can someone please advise on whether such custom dataloader can be implemented with PyTorch, and if yes, how I can implement that? Thanks a lot in advance!</p>
<p>Best,<br>
Niam</p>

ptrblck: <p>One simple approach would be to use <code>batch_size=2</code> in the <code>DataLoader</code> and add the sampling logic to <code>Dataset.__getitem__</code>.<br>
This method would get the two indices as its <code>index</code> argument, and you could use this index to sample the desired paired sample. If you want or need, you could also use a <code>BatchSampler</code>, which would pass both indices together and might make your custom sampling easier.</p>
<p>A disadvantage would be that your <code>batch_size</code> is specified as e.g. 2, while your actual <code>DataLoader</code> returns a tensor with a <code>batch_size</code> of 4.</p>
<p>Alternatively, you could also write a custom sampler.</p>

Niamul_Quader: <p>Ingenious! You are awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Two Networks speed different when I apply the two different GPU

henry_Kang: <p>Hello. Happy New year!!!</p>
<p>I am trying to calculate my suggested network speed and comparison network. Both are semantic segmentation network.<br>
My network name is A.<br>
The other name is B.<br>
The input size of image is same. There is no resize or crop.</p>
<p>From Titan RTX, the network A has 25 FPS. The B has 20 FPS.<br>
In the google colab I get the V100. When I use v100, network A &amp; B has similar speed and sometimes B has better speed. Do you have any idea why it happens?</p>
<p>Thank you.</p>

ptrblck: <p>If you are using cudnn, then different kernels might be picked due to the difference in workload (models) and GPU architectures.<br>
You could use <code>torch.backend.cudnn.benchmark = True</code> to let cudnn profile the kernels and pick the fastest one. Note that the first iteration for each new input shape will be slow, since the profiling will be executed.</p>

henry_Kang: <p>Thank you for answering.</p>
Make_grid - How to control the resolution of every image

Moran_Nechushtan: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Iv’e coded a simple photomosaic algorithm using the Cifar10 (3x32x32) dataset (looks pretty good).<br>
Iv’e notice that when I’m using different number of images in the grid,<br>
the make_grid method change the resolution of every cifar10 image. How can I control the resolution of the images in the grid? In particular how can I keep the resolution the same? (32x32)</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>make_grid</code> is not changing the resolution as seen <a href="https://github.com/pytorch/vision/blob/ae87c1e46df6fb404654935c82e15013d56b7aa8/torchvision/utils.py#L87-L105">here</a> and the shape of the result <code>grid</code> should correspond to the number of images in the columns and rows + the additional padding.<br>
I guess the library to plot this image (e.g. <code>matplotlib</code>) might rescale it, so you could check, if specifying the size in e.g. <code>plt.imshow</code> would help.</p>
Does nvlink help?

Scott_Hoang: <p>I have a machine 4x rtxes. Does install a nvlink benefit Pytorch?</p>

ptrblck: <p><code>nvlink</code> is used for p2p communication between GPUs so in particular multi-GPU setups can be accelerated. The performance depends on your actual setup.</p>

Scott_Hoang: <p>I have a setup with 4x 2080ti. My research mostly revolves around CV tasks using PyTorch’s DPP. Do you think it is a worthwhile investment?</p>

ptrblck: <p>I cannot give any recommendations to buy hardware and my best recommendation would be to check open and independent benchmarks for this or similar setups.</p>
CUDA error: an illegal memory access (when fine-tuning GroupNorm)

phaedrus: <p>I’m trying to fine-tune Resnet18, but replaced BatchNorm with GroupNorm. It works on CPU, it works when I don’t unfreeze the GroupNorm, it works with BatchNorm… but with GroupNorm unfrozen, it always fails with <code>RuntimeError: CUDA error: an illegal memory access was encountered</code>.</p>
<p>Here is a minimal code that reproduces the issue:</p>
<pre><code class="lang-auto">## Repro CUDA issue
import torch
from torchvision.models import resnet18

m = resnet18(pretrained=True)

def replace_bn(m):
    for name, child in m.named_children():
        if type(child) == torch.nn.BatchNorm2d:
            setattr(m, name, torch.nn.GroupNorm(num_groups=1, num_channels=child.num_features))
        replace_bn(child)

replace_bn(m)

# Freeze the model, except GroupNorm
for p in m.parameters():
    p.requires_grad = False

def unfreeze_norm_layers(m):
    if type(m) == torch.nn.modules.batchnorm.BatchNorm2d or type(m) == torch.nn.GroupNorm:
        for p in m.parameters():
            p.requires_grad = True
m.apply(unfreeze_norm_layers);

device = torch.device('cuda')
m.to(device)
inp = torch.randn(64, 3, 218, 178).to(device)
labels = torch.randint(0, 1000, (64,)).to(device)

opt = torch.optim.SGD(m.parameters(), lr=0.01, momentum=0.9)
opt.zero_grad()
l = torch.nn.CrossEntropyLoss()(m(inp), labels)
l.backward()
l.item()

# -&gt; RuntimeError: CUDA error: an illegal memory access was encountered
# It works if you don't replace the BatchNorm (you can unfreeze BatchNorms)
# It works if you don't unfreeze the GroupNorm
# It works on CPU

# torch version: 1.6.0+cu92
</code></pre>
<p>Any ideas what am I doing wrong?</p>

ptrblck: <p>Could you update to the latest PyTorch release and also try the CUDA10.2 or 11.0 binaries?<br>
If you are still hitting the illegal memory access, could you run the script via:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here?</p>

phaedrus: <p>Yep, with 1.7.1 cuda 10.1, it doesn’t crash.</p>
Weight Pruning on BERT

Senzeyu_Zhang: <p>Hello, I used torch.nn.utils.prune to apply weight pruning on BERT. It generates sparse matrices as expected. However, the model’s size doesn’t decrease and the inference speed doesn’t increase? How can I speed up after pruning and actually compress the model’s size? Thank you!</p>
<pre><code class="lang-auto">BERT_QA = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
for child in BERT_QA.modules():
    if isinstance(child, nn.Linear):
        prune.l1_unstructured(child, name='weight', amount=0.2)
        prune.l1_unstructured(child, name='bias', amount=0.2)
        prune.remove(child, 'weight')
        prune.remove(child,'bias') 
</code></pre>

Michela: <p>The point of PyTorch pruning, at the moment, is not necessarily to guarantee inference time speedups or memory savings. It’s more of an experimental feature to enable pruning research.</p>
<p>In any case, answers to questions similar to yours were given <a href="https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076" rel="nofollow noopener">here</a> and <a href="https://discuss.pytorch.org/t/how-to-improve-inference-time-of-pruned-model-using-torch-nn-utils-prune/78633/4">here</a>.</p>
<p><em>TL;DR</em>: You can save space by calling <code>.to_sparse()</code> which brings your sparse tensor into coordinate representation. You cannot expect any inference speedups unless you use a custom sparse matrix algebra library to power your computation. <code>torch.sparse</code> is still a work in progress for now. Otherwise, for now, you’ll just be doing the same number of operations as you did before pruning, only now with a bunch of entries equal to zero in your tensors.</p>

pintonos: <p>Hello Michela,<br>
sorry to reopen this thread.<br>
Is there any update in applying pruning and reducing inference time in doing so? If not, which library you would recommend to perform this computations in an efficient manner?</p>
<p>Thanks!</p>
<p>Best, Andreas</p>
GRU/RNN model unrolling

jackfoo: <p>I’m training a CNN+GRU that needs to work on a large sequence length. The problem is that it cannot fit in the GPU memory with such sequence lengths due to the unrolling in time. Is there a way to prevent this from happening? There is an option in Keras <code>unroll=False</code> when creating a model, but I can’t find something like this in Pytorch.</p>

miro: <p>You could try passing in half of your sequence, then detaching the output hidden state from the backprop graph, and then pass that output hidden state into the LSTM again along with the second half of the sequence.</p>