akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training faster with single gpu

safin_salih: <p>Is there a way to train faster on a single gpu, I have a 16 core cpu, and it seems like increasing the number of workers on the dataloader slows down training? Can I get any hints in this regards. I’m somewhat new in this domain. Thank you.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> gives a great overview how to handle data loading bottlenecks.<br>
In particular this section might be interesting for you:</p>
<blockquote>
<p>Beyond an optimal number (experiment!), throwing more worker processes at the IOPS barrier WILL NOT HELP, it’ll make it worse. You’ll have more processes trying to read files at the same time, and you’ll be increasing the shared memory consumption by significant amounts for additional queuing, thus increasing the paging load on the system and possibly taking you into thrashing territory that the system may never recover from</p>
</blockquote>
Objective() missing 1 required positional argument: ‘trial’

KanZa: <pre><code class="lang-auto">class Trainer():
     def __init__(self,param):
        ..........
        optimizer_target = getattr(optim, param['optimizer'])(net.parameters(), lr = param['learning_rate'])

     def train(epoch):
        .......... 

     def test(self, epoch):
        ..........

  def objective(self,train):
        params = {}

if __name__ == '__main__':
    study = optuna.create_study(direction="maximize")
    study.optimize(objective)

</code></pre>
<p>when I call this objective function in main() it gives me different errors.No matter if I call this function like this it gives me the error</p>
<pre><code class="lang-auto">study.optimize(lambda trial: Trainer.objective(trial))
</code></pre>
<p>If I design it without the class Trainer it gives me the error. Kindly suggest how to call the dictionary of objective class and calling objective function in main()</p>
<p><strong>Error</strong><br>
study.optimize(lambda trial: Trainer.objective(trial))<br>
TypeError: objective() missing 1 required positional argument: ‘trial’</p>
Largest connected component in loss function

phcerdan: <p>I would like to add a loss function that only takes into account the largest connected component of the output of my network (a segmentation). My idea is that this will led the network to be less eager to disconnect small objects.</p>
<p>Is it possible with torch operations?<br>
I already tried to detach and use numpy methods (<code>skimage.label</code>), but using numpy is not compatible with autograd.</p>
<p>Any suggestions? Thanks</p>
Uniformly scale down ResNet model size

snehjp2: <p>Hi,</p>
<p>I am attempting to uniformly scale down the model parameters of my ResNet50 by shrinking the width of the network: decreasing the size of the  <code>in_channel</code> and <code>out_channel</code> arguments of every <code>Conv2D</code> layer (excluding initial input) and the width of the <code>Linear</code> layer at the end by some prefactor (excluding the final network output).  The model is obtained via torchvision:</p>
<p><code>self.model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True)</code></p>
<p>I’m hoping there’s a way to do this without brute-force redefining all the layers within my network class.  Any tips?</p>
Reduce GPU Memory blocked by context

alinak: <p>Hi,</p>
<p>I’m currently working on a single-GPU system with limited GPU memory where multiple torch models are offered as “services” that run in separate python processes.<br>
Ideally, I would like to be able to free the GPU memory for each model on demand without killing their respective python process. However, if I only delete the models (and empty the cache) without killing the process, I’m still stuck with the memory usage of the cuda context, which is about 1.2 GB per process in my case.</p>
<p>In general, the memory usage of the context seems extremely high to me, especially when considering that the model itself only adds about 200MB to it while it’s loaded.<br>
So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?</p>
<p>I’d be grateful for advice!</p>

ptrblck: <aside class="quote no-group" data-username="alinak" data-post="1" data-topic="142587">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/4bbf92/48.png" class="avatar"> alinak:</div>
<blockquote>
<p>So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?</p>
</blockquote>
</aside>
<p>You could reduce the CUDA context size by removing kernels e.g. via dropping libraries such as MAGMA or cuDNN, if these are not used.<br>
Besides that I’m not aware of reducing the operator set in the framework itself.</p>

alinak: <p>Thanks for the reply! So do I understand correctly that this memory will always be occupied one way or another if I want to use cuda functionality, no matter if it’s via pytorch or if I’d e.g. export the models to onnx and use them from c++?</p>

ptrblck: <p>The memory will be occupied after loading the CUDA kernels.<br>
Depending on the build you are using, all libraries might be loaded directly when the context is created (e.g. in the pip wheels as libs are statically linked) or when they are used the first time (e.g. cuDNN via dynamic linking should behave like this).<br>
Regardless of this behavior, if you are using “all libraries” the context size will be the same.</p>
Problem calculating accuracy on multiclass classfication

gkrisp9: <p>Hi, I am facing a problem when I try to compute accuracy on a multiclass classification problem with the Accuracy() function from torchmetrics<br>
Here is the training function I created:</p>
<pre><code class="lang-auto">from torchmetrics import Accuracy
accuracy = Accuracy()

def training(epoch, model, train_loader, optimizer, criterion):
  "Training over an epoch"
  metric_monitor = MetricMonitor()
  
  model.train()

  for batch in train_loader:
    images = batch['t1'][tio.DATA].cuda()
    labels = batch['label'].cuda()
    output = F.softmax(model(images), dim=0)

    loss = criterion(output, labels)

    output = output.data.max(dim=1,keepdim=True)[1]

    acc = accuracy(output, labels)
 
    metric_monitor.update("Loss", loss.item())
    metric_monitor.update("Accuracy", acc)

    optimizer.zero_grad()

    loss.backward()
    optimizer.step()
  print("[Epoch: {epoch:03d}] Train      | {metric_monitor}".format(epoch=epoch, metric_monitor=metric_monitor))
  return metric_monitor.metrics['Loss']['avg'], metric_monitor.metrics['Accuracy']['avg']
</code></pre>
<p>When I try to call this function to train the model I get the following error :<br>
<strong>ValueError: If <code>preds</code> have one dimension more than <code>target</code>, <code>preds</code> should be a float tensor</strong>.<br>
Does anybody know what I am doing wrong ?</p>

anantguptadbl: <p><a class="mention" href="/u/gkrisp9">@gkrisp9</a> This is a classic case of CrossEntropy evaluation</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>
<p>When you have multi label classification, your output dimension becomes<br>
batch_size, num_of_classes</p>
<p>And your input target is batch_size</p>
<p>The solution would be to add this extra line</p>
<pre><code class="lang-auto">output = output.data.max(dim=1,keepdim=True)[1]
output = output.view(-1)
acc = accuracy(output, labels)
</code></pre>

gkrisp9: <p>Thanks again! Now I am getting this error: <strong>Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!</strong><br>
I am bit confused because as I see from my code all the labels etc are in cuda.<br>
I added this command: output.cuda() to see if that was the problem but the error appears again.</p>
<p>Update: I made a function to return accuracy and pass output and labels to cpu and it works</p>

anantguptadbl: <pre><code class="lang-auto">output = F.softmax(model(images), dim=0).cuda()
...
output = output.data.max(dim=1,keepdim=True)[1].cuda()
</code></pre>
Torchvision's ColorJitter support of uint8 images?

vadimkantorov: <p>Does ColorJitter transform support torch.uint8 tensor images?</p>
<p>I found a similar question but for torch.int16 tensor images: <a href="https://discuss.pytorch.org/t/colorjitter-transformation-for-16-bit-images/108897" class="inline-onebox">ColorJitter Transformation for 16-bit images?</a></p>

ptrblck: <p>Yes, this seems to work:</p>
<pre><code class="lang-python">transform = transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.5)
x = torch.randint(0, 256, (3, 224, 224), dtype=torch.uint8)
out = transform(x)
</code></pre>

vadimkantorov: <p>I confirm, it seems to work. I was worried about the scale of <code>brightness</code> and other parameters. Does it need to be scaled to 255 or not… Gotta read the code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>F.adjust_brightness</code> expects a <code>float</code> as seen <a href="https://github.com/pytorch/vision/blob/59c723cb45d0f8ab897cc7836d408e9fdde4b552/torchvision/transforms/functional.py#L812-L831">here</a> using this description:</p>
<blockquote>
<p>brightness_factor (float):  How much to adjust the brightness. Can be any non negative number. 0 gives a black image, 1 gives the original image while 2 increases the brightness by a factor of 2.</p>
</blockquote>
Distributed Data Parallel doesn't remove hooks

kumar.s: <p>The Docs on the DDP class say:</p>
<pre><code class="lang-auto">Forward and backward hooks defined on module and its submodules 
won’t be invoked anymore, unless the hooks are initialized in the forward() method.
</code></pre>
<p>However, they are working for me. Did I misinterpret the warning? Does it mean something else?<br>
Is there a reason I should be careful when using hooks with DDP? In my original code I use it to modify the output of some layers. But here’s a simplified minimum reproducible example:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import os


class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.model = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Conv2d(1, 1, 1))

    def forward(self, x):
        self.model(x)


def hook_fn(mod, inp, out):
    if dist.is_initialized():
        print(
            f"The forward hook is not removed in rank : {dist.get_rank()}")


def run_model(rank):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("nccl", rank=rank, world_size=2)
    model = SimpleCNN().to(rank)
    for mod in model.modules():
        if 'Conv' in mod.__class__.__name__:
            mod.register_forward_hook(hook_fn)

    ddp_mod = nn.parallel.DistributedDataParallel(model, device_ids=[rank])
    for _ in range(2):
        ddp_mod(torch.randn(1, 1, 10, 10))
    dist.destroy_process_group()


def run():
    mp.spawn(run_model, nprocs=2, join=True)


if __name__ == '__main__':
    run()
</code></pre>
<p>And the corresponding output:</p>
<pre><code class="lang-auto">The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 0
The forward hook is not removed in rank : 1
The forward hook is not removed in rank : 1
</code></pre>

kumar.s: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you help please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I think based on <a href="https://github.com/pytorch/pytorch/issues/35191#issuecomment-700702620">this comment</a> the issue would only arise if you would be using the single process - multiple devices approach (not recommended) and hooks should work fine for single process - single device (your approach and the recommended one).</p>
Voice like this

blackbirdbarber: <p>Can anyone point where to find  TTS voice like this:</p><div class="youtube-onebox lazy-video-container" data-video-id="hJQ4o0Zouf0" data-video-title="In a World full of Superheroes a Group of Kids Team up to Save The Planet From an Alien Invasion" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=hJQ4o0Zouf0" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/hJQ4o0Zouf0/maxresdefault.jpg" title="In a World full of Superheroes a Group of Kids Team up to Save The Planet From an Alien Invasion" width="690" height="388">
  </a>
</div>
Find Average rgb value of an image

KURUVILLA_ABRAHAM: <p>Hi Team,</p>
<p>How to compute most used color in a image using pytorch .</p>

anantguptadbl: <p><a class="mention" href="/u/kuruvilla_abraham">@KURUVILLA_ABRAHAM</a></p>
<p>Your question asks for average RGB value, but in the description you asked for the most used color. The following is for most used color</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
a = torch.LongTensor(plt.imread(r"temp.jpg"))
b = a[:,:, 0]* 1000000 + a[:,:,1]*1000 + a[:,:,2]
results = torch.unique(b.view(-1), return_counts=True)
results = results[0][torch.argmax(results[1])]
r = int(results/1000000)
g = int((results/1000.) - int(r*1000.) )
b = results - r*1000000 - g*1000
torch.FloatTensor([r,g,b])
</code></pre>
<p>The above is a very dirty implementation of a groupby. Torch does not have a string data type so you have to convert it to a scalar through some other trick</p>

KURUVILLA_ABRAHAM: <p>Hello anand,</p>
<p>Thanks for Answering ,</p>
<p>wanted to check why you have multiplied by  —</p>
<p>“b = a[:,:, 0]* 1000000 + a[:,:,1]*1000 + a[:,:,2]”</p>
<p>Also after uploading a image in png format</p>
<p>it is giving result as -</p>
<p>tensor([0., 0., 0.]).</p>
Calculate accuracy

gkrisp9: <p>Hi, I am a bit confused about the accuracy on multi label classification.<br>
I have this function for calculating accuracy:</p>
<pre><code class="lang-auto">def calculate_accuracy(output, target):
    "Calculates accuracy"
    output = output.data.max(dim=1,keepdim=True)[1]
    output = output == 1.0
    output = torch.flatten(output)
    target = target == 1.0
    target = torch.flatten(target)
    return torch.true_divide((target == output).sum(dim=0), output.size(0)).item() 
</code></pre>
<p>but when I run it, it gives wrong outputs. For example:</p>
<pre><code class="lang-auto">x = torch.tensor([[2],[0],[1],[3],[4],[1]])
y = torch.tensor([[2],[1],[1],[4],[3],[2]])
</code></pre>
<p>calculate_accuracy(x,y) prints 0.6667… where it clearly should print 0.333.<br>
Where have I mess up ?</p>

anantguptadbl: <p><a class="mention" href="/u/gkrisp9">@gkrisp9</a></p>
<p>The mistake that you have made is with these two lines</p>
<pre><code class="lang-auto">output = output == 1.0
target = target == 1.0
</code></pre>
<p>This is translating the data into</p>
<pre><code class="lang-auto">tensor([False, False, False, False, False, False])
tensor([False,  True,  True, False, False, False])
</code></pre>
<p>Due to this there is a match on 66% of the data and that is what is coming up in your results</p>

gkrisp9: <p>You are right! Thanks</p>

gkrisp9: <p>Now that I see it again shouldn’t that print 33% ? There are 2 ‘True’ over 6 ‘False’</p>

anantguptadbl: <p><a class="mention" href="/u/gkrisp9">@gkrisp9</a><br>
You are not just evaluating True against True. You are also evaluating False against False. Even that is a valid match as per your logic</p>
<pre><code class="lang-auto">tensor([False, False, False, False, False, False])
tensor([False,  True,  True, False, False, False])
</code></pre>
<p>You have 4 matches of False against False and 0 matches of True against True<br>
Final score 4/6</p>

gkrisp9: <p>Oh I missed that. Now I got it, thanks a lot again!</p>
Shaping and Reshaping error before and after the fixed filters

Artemisia_Jonath: <p>I get the following error when i try to run this model.<br>
RuntimeError: Given groups=1, weight of size [128, 64, 5, 5], expected input[2048, 1, 8, 8] to have 64 channels, but got 1 channels instead<br>
I am using cifar100 dataset. Batch size is 128.</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:</p>
<pre><code># functions to show an image

def imshow(img):

    img = img / 2 + 0.5  # unnormalize

    npimg = img.numpy()

    plt.imshow(np.transpose(npimg, (1, 2, 0)))

    plt.show()



class Net(nn.Module):

    def __init__(self):

        super().__init__()

        self.conv1 = nn.Conv2d(3, 64,5,1,2)

        self.conv_resh = nn.Conv2d(1, 1, 2, 2, 1)

        self.filt1 = nn.Conv2d(64, 64, 2, 2, bias=False, padding=1) 

        self.pool = nn.MaxPool2d(2, 2)

        self.conv2 = nn.Conv2d(64, 128,5,1,2)

        self.conv3 = nn.Conv2d(128,128,5,1,2)

        self.conv4 = nn.Conv2d(128, 128,5,1,2)

        self.conv5 = nn.Conv2d(128, 128,5,1,2)

        self.fc1 = nn.Linear(128, 120)

        self.fc2 = nn.Linear(120, 64)

        self.fc3 = nn.Linear(64,32)

        self.fc4 = nn.Linear(32,16)

        self.fc5 = nn.Linear(16, 100)

        h,w=32,32

    def forward(self, x):

        x = self.pool(F.relu(self.conv1(x)))

        x = x.view(-1,1,32,32)

        x = self.pool(F.relu(self.filt1(x)))

        print(x.shape)

        x = self.pool(F.relu(self.conv2(x)))

        x = self.pool(F.relu(self.conv3(x)))

        x = self.pool(F.relu(self.conv4(x)))

        x = self.pool(F.relu(self.conv5(x)))

        x = torch.flatten(x, 1)  # flatten all dimensions except batch

        x = F.relu(self.fc1(x))

        x = F.relu(self.fc2(x))

        x = F.relu(self.fc3(x))

        x = F.relu(self.fc4(x))

        x = self.fc5(x)

        return x

net = Net()

with torch.no_grad():

            net.filt1.weight = nn.Parameter(torch.tensor([[[[1., 0.],[0., 1.]]]]))

print(net)

print(net.filt1.weight)</code></pre>

anantguptadbl: <p><a class="mention" href="/u/artemisia_jonath">@Artemisia_Jonath</a></p>
<p>The following statement changes your channels from 64 to 1</p>
<aside class="quote no-group" data-username="Artemisia_Jonath" data-post="1" data-topic="142688">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/artemisia_jonath/48/46132_2.png" class="avatar"> Artemisia_Jonath:</div>
<blockquote>
<p><code>x = x.view(-1,1,32,32)</code></p>
</blockquote>
</aside>
<p>Due to this the following statement fails as it expects 64 channels</p>
<pre><code class="lang-auto">self.filt1 = nn.Conv2d(64, 64, 2, 2, bias=False, padding=1) 
</code></pre>
Setting up sparse connections without gradients

gokhan: <p>Hello everyone,</p>
<p>Hope you are having a productive day.</p>
<p>I am trying to build a model which has</p>
<ul>
<li>sparse connections between layers</li>
<li>some weights that should not be updated</li>
</ul>
<p>For example,</p>
<pre><code class="lang-auto">self.L_vc = nn.Linear(E, E, bias=False)
torch.nn.init.ones_(self.L_vc.weight)
for p in self.L_vc.parameters():
    p.requires_grad = False
prune.custom_from_mask(self.L_vc, name='weight', mask=vc_mask)
</code></pre>
<p>Here, I specify the connectivity between the neurons using the tensor <code>vc_mask</code> and preventing the weights from being updated by setting <code>p.requires_grad = False</code>. I also set <code>bias = False</code> since I just want <code>L_vc</code> to specify the connectivity just like an adjacency matrix. I am not sure if this is the correct way to do that.</p>
<p>Am I interrupting the gradients from propagating backwards by setting a constant layer like this? What should I expect if I have a layer like this in the middle of the model? Any help would be highly appreciated.</p>

anantguptadbl: <p><a class="mention" href="/u/gokhan">@gokhan</a></p>
<p>Since you have mentioned the below lines, the grads will propagate, but not get updated. This will serve your purpose</p>
<aside class="quote no-group" data-username="gokhan" data-post="1" data-topic="142544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gokhan/48/46084_2.png" class="avatar"> gokhan:</div>
<blockquote>
<pre><code class="lang-auto">for p in self.L_vc.parameters():
    p.requires_grad = False
</code></pre>
</blockquote>
</aside>
How to load data using torchmeta without any class-per-task break up?

Adarsh_Subramanian: <p>I want to use the tiered-imagenet data for an image classification problem. I want to use the whole data together and don’t want to perform few-shot learning. The tiered imagenet data is available directly on torchmeta and I would like to use it. How do I load the data without any break up of the dataset?</p>
<pre><code class="lang-auto">    dataset = torchmeta.datasets.TieredImagenet(root, num_classes_per_task=1,
    meta_train=False, meta_val=False, meta_test=False, meta_split=mode,
    transform=None, target_transform=None, dataset_transform=None,
    class_augmentations=None, download=True)
</code></pre>
<p>However, even this is creating errors. I checked the documentation and seems like the num_classes_per_task is mandatory. How do I resolve this?</p>
<p>Thanking you in advance!</p>
YoloV5x model prediction time

Sreekanth: <p>Hi,</p>
<p>We are using YoloV5x model for object detection. In out setup we have Geforce GT730 GPU and we have build the torch(V1.10) and torchvision (V0.11) from source. The model is able to detect the CUDA and is working. When we check the model prediction time, the time taken from the prediction is high.</p>
<p>In our application we will be using 4 cameras for object detection. When we tested the YoloV5 model with 2 cameras, the time taken is approximately 1.6-1.8 sec with imgsz size of 640.</p>
<p>After training the model we have saved the model with .pt extension.</p>
<p>Is there any way where we can increase the model prediction speed? Also the CPU usage is 100% when we run the model with 2 cameras.</p>
<p>Thanks,<br>
Sreekanth</p>

ptrblck: <p>You could profile your application with the native PyTorch profiler or e.g. Nsight Systems and could try to narrow down where the bottleneck is. Once isolated the improvements depend on the actual root cause. For general recommendation check the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a>.</p>

hsekki: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , can you suggest ways of optimizing model inference , I am using yolov5 models with input size of 1920 px but inference time took around 1000ms per image. Appreciate your response. Thanks.</p>

ptrblck: <p>Same as before: try to profile your workload to narrow down the bottleneck as optimizing code without actually improving the bottleneck might not yield the expected gains.</p>
How to do elementwise max between multiple tensors? getting the following error for 4 tensors

Umair_Javaid: <pre><code class="lang-auto">a = torch.ones(1,4,1,1)*2

b = torch.ones(1,4,1,1)*3

c = torch.ones(1,4,1,1)*4

#d = torch.ones(1,4,1,1)*5

max_ = torch.max(a,b,c)

print(max_)
</code></pre>
<pre><code class="lang-auto">TypeError: max() received an invalid combination of arguments - got (Tensor, Tensor, Tensor), but expected one of:
 * (Tensor input)
 * (Tensor input, name dim, bool keepdim, *, tuple of Tensors out)
 * (Tensor input, Tensor other, *, Tensor out)
 * (Tensor input, int dim, bool keepdim, *, tuple of Tensors out)
</code></pre>

albanD: <p>Hi,</p>
<p>I’m afraid the element-wise max will only work for a pair of Tensors, You will have to do them two by two <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

LawsonAbs: <p>you can use topk to get the wanted operation</p>
CPU usage too high when multiple inference using multiprocessing in torch

Gwon: <p>I have a problem with use torch.multiprocessing modules.</p>
<p>In my code,</p>
<pre><code class="lang-python">import torch.multiprocessing as mp

class MultiProcessing:
     def run_inference(self):
          # Load detection model
          # Model is YOLOR(detection model)
          model = MyModel()
          ...
          p1 = mp.Process(target=self.do_detection, args=(model,))
          p2 = mp.Process(target=self.do_detection, args=(model,))
          p1.start()
          p2.start()
          p1.join()
          p2.join()

     def do_detection(self, model):
          model = model.cuda()
          vid_cap = cv2.VideoCapture("&lt;VideoFile&gt;")
          while True:
               # Read video
               _, img = vid_cap.read()
               # Input video frame to detection model
               pred = model(img)
               # Visualize detection result
               ...
if __name__ == '__main__':
     inference = MultiProcessing()
     inference.run_inference()
</code></pre>
<pre><code class="lang-auto"># My CPU / GPU spec
CPU: i7-11700K, GPU: RTX 3070
</code></pre>
<p>When run this code, I only run two processes, CPU usage 100% but GPU usage approximately under 50%. due to CPU usage, the detection process is too slow</p>
<p>Why is it happen? and How can i solve this problem?</p>
ChecksumMismatchError for installing PyTorch

hudongyue1: <p>My system information:</p>
<pre><code class="lang-auto">Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-37-generic x86_64)

conda version 4.11.0

CUDA Version 10.1.243
</code></pre>
<p>and when I try to install PyTorch</p>
<pre><code class="lang-auto">ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/ffmpeg-4.3-hf484d3e_0.tar.bz2'.
  download saved to: /data/dyhu/anaconda3/pkgs/ffmpeg-4.3-hf484d3e_0.tar.bz2
  expected md5: 0b0bf7c3d7e146ef91de5310bbf7a230
  actual md5: 01d5d7540ae69053e5a2e27289c06961

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.11.2-py39_cu102.tar.bz2'.
  download saved to: /data/dyhu/anaconda3/pkgs/torchvision-0.11.2-py39_cu102.tar.bz2
  expected md5: 936e492460b62c385361e88527cbdc12
  actual md5: b8d622a3186183ee02b21bb605f752fb

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/pytorch-1.10.1-py3.9_cuda10.2_cudnn7.6.5_0.tar.bz2'.
  download saved to: /data/dyhu/anaconda3/pkgs/pytorch-1.10.1-py3.9_cuda10.2_cudnn7.6.5_0.tar.bz2
  expected md5: bf9986e1812d05784b0823f34ccb433e
  actual md5: c0cffd05b030cf0c194ad9e4bbb8a1ed

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/torchaudio-0.10.1-py39_cu102.tar.bz2'.
  download saved to: /data/dyhu/anaconda3/pkgs/torchaudio-0.10.1-py39_cu102.tar.bz2
  expected md5: 66f25c0cb8fa754c6cb396a11c371686
  actual md5: ff5cfdb91a2a56edc35d704af409976f

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/noarch/pytorch-mutex-1.0-cuda.tar.bz2'.
  download saved to: /data/dyhu/anaconda3/pkgs/pytorch-mutex-1.0-cuda.tar.bz2
  expected md5: a948316e36fb5b11223b3fcfa93f8358
  actual md5: 36f806af7e57fdef13f31bbb095f27d8
</code></pre>
<p>the same problem in <a href="https://github.com/pytorch/pytorch/issues/63006" rel="noopener nofollow ugc">GitHub Issue</a></p>
RuntimeError: tensor does not have a device

Ximeng: <p>Hi,</p>
<p>I got the following error:<br>
“RuntimeError: The following operation failed in the TorchScript interpreter.<br>
Traceback of TorchScript (most recent call last):<br>
RuntimeError: tensor does not have a device”</p>
<p>Some of my modules inherit jit.ScriptModule. Torch version:1.10.1</p>
<p>Thanks</p>
<p>Best regards</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce this issue, please?</p>
A model with self-attention layer throw cuda out of memory error

Roqyah_bdeen: <p>My model perform this convolution operation on an image through 6 CNN pyramid levels with different resolutions. Unfortunately, the self.attenflow layer throw cuda out of memory error although the tensors have 2 channels only. could any one explain how to solve this problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:"></p>
<pre><code class="lang-auto">class FlowEstimator(nn.Module):
   def __init__(self, ch_in, ch_out=2, bn=False):
       super(FlowEstimator, self).__init__()
       self.conv1 = conv(bn, ch_in, 128)
       self.conv2 = conv(bn, 128, 128)
       self.conv3 = conv(bn, 128 + 128, 96)
       self.conv4 = conv(bn, 128 + 96, 64)
       self.conv5 = conv(bn, 96 + 64, 32)
       self.attenflow=Cross_Attn(2,'relu')

       self.final_out = 32
       self.predict_flow = CustomConv2D(64 + 32, ch_out, kernel_size=3, stride=1,
                                        padding=1)

   def forward(self, x,context):
       x1 = self.conv1(x)    
       x2 = self.conv2(x1)
       x3 = self.conv3(torch.cat([x1, x2], dim=1))
       x4 = self.conv4(torch.cat([x2, x3], dim=1))
       x5 = self.conv5(torch.cat([x3, x4], dim=1))
       c=context[0]
       flow = self.predict_flow(torch.cat([x4, x5], dim=1))
       final_flow=self.attenflow(flow,c)
       return x5, final_flow

</code></pre>

ptrblck: <p>Could you check the memory usage of the used modules by printing either <code>torch.cuda.memory_allocated()</code> or <code>torch.cuda.memory_summary()</code>. Based on these results you could then check if the memory requirement is expected or not.</p>
RNN(LSTM, GRU) hidden states

dim314159: <p>I’ve seen 2 ways to use hidden states.<br>
First way:<br>
in class:<br>
self.rnn = nn.rnn(…)<br>
def forward(self, x, h):<br>
out, h = self.rnn(x,h)<br>
return out, h</p>
<p>In training:<br>
for … epochs:<br>
h = torch.zeros(num_layers, batch_size, hidden_size)<br>
…<br>
for … batch loop:<br>
out, h = model(x, h)<br>
…<br>
h.detach_()</p>
<p>Second way:<br>
In class:<br>
self.rnn = nn.rnn(…)<br>
def weight_init():<br>
self.h = torch.zeros(num_layers, batch_size, hidden_size)<br>
def forward(x):<br>
out, self.h = self.rnn(x, self.h)<br>
return out</p>
<p>In training:<br>
for epochs:<br>
for batch:<br>
model.weight_init()<br>
out = model(x)<br>
…</p>
<p>So, which way is correct, or what is the difference?<br>
And also in the second way, should I do weight_init for every batch or for every epoch?<br>
Thank you.</p>
BufferedShuffleDataset

dmus: <p>I see there is a BufferedShuffleDataset in the source code: <a href="https://github.com/pytorch/pytorch/commit/96540e918c4ca3f0a03866b9d281c34c65bd76a4#diff-425b66e1ff01d191679c386258a7156dfb5aacd64a8e0947b24fbdebcbee8529" class="inline-onebox" rel="noopener nofollow ugc">Add ShuffleDataset with buffer (#45290) · pytorch/pytorch@96540e9 · GitHub</a></p>
<p>However, I cannot find this back in the documentation and it seems not to be in release 1.7.1. When does this become available in the standard torch pip package?</p>

ptrblck: <p>This PR was not cherry-picked into <code>1.7.1</code> and should be already available in the nightly binaries (<a href="https://pytorch.org/docs/master/data.html#torch.utils.data.BufferedShuffleDataset">docs</a>) and the next stable minor release (<code>1.8.0</code>).</p>

Pierre-Bartet: <p>In can’t see it in recent PyTorch versions, has it been dropped ?</p>
Different unit test results when using pip and conda to install torch+cuda

jojo: <p>Hi,</p>
<p>I have torch (1.10.1), torchvision (0.11.2) and CUDA (11.3) installed in two different virtual environments (python 3.7.10). For one environment I used pip, for the other I used conda following the installation guide.</p>
<pre><code class="lang-auto"># pip

pip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

# conda

conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
</code></pre>
<p>When running the exact same code for unit testing an InceptionResnetV1 classifier from facenet_pytorch (2.5.2) using GPU (Quadro RTX 5000), I get different checksums for the pip and conda based environments.</p>
<p>Checksum is the average confidence value across all faces detected in a test image. Checksums also differ when using other models, for example yolo_v5.</p>
<p>I use same np.random.seed and torch.manual_seed.</p>
<p>Running the unit test on different computers with the same type of gpu produces the same results.</p>
<p>I’m on a Microsoft stack.</p>
<p>What causes the differences?</p>
<p>Thank you in advance</p>
Share tensor with entire batch without .repeat()?

pumplerod: <p>I am trying to train samples (some 4000 per batch) in relation to a single image for that batch.<br>
Can I do this without the need to repeat my image for each sample?  Because it is the same data shared to all training samples, this becomes a huge memory drain.</p>

pumplerod: <p>To try and be a little more clear…</p>
<p>assume I have a batch which has aprox 4000 samples, 100 features, where I will be using linear layers to process each sample, something like:</p>
<pre><code class="lang-auto">    linear_block = torch.nn.Sequential(
        torch.nn.Linear( 100, 10),
        torch.nn.ReLU( inplace=True)
    )
</code></pre>
<p>I would also like to take the entire (4000, 100) batch and pass through Conv2D layer(s), ultimately passing features into a final merge block so that each sample is provided information from the entire batch…</p>
<pre><code class="lang-auto">    batch_block = torch.nn.Sequential( 
        torch.nn.Conv2D( 1, 6, kernel_size=(3,9), stride=(1,2), pad=(1,4)),
        torch.nn.ReLU( inplace=True),
        torch.nn.Conv2D( 6, 6, kernel_size=(3,9), stride=(1,2), pad=(1,4)),
        torch.nn.ReLU( inplace=True)
    }

    merge_block = torch.nn.Sequential( 
        torch.nn.Linear( combined_output_size_of_flat_batch_and_linear_block, 5),
        torch.nn.Softmax( dim=1)
    }

forward( X):
    X_full_batch_conv_features = batch_block( X.view(1, 1, 4000, 100))
    X = linear_block( X)
    batch_flatten = X_full_batch_conv_features.flatten()
    merge = torch.cat( [X, batch_flatten], dim=1)
    return merge_block( merge)
</code></pre>
<p>I’ve been able to do this using <code>.repeat()</code> for the batch_block, copying all the data to every sample but that seems like a huge waste of memory when every sample is looking back to the same set of numbers.</p>
Does additional pre-training of BERT mean the same as fine-tuning it on MaskedLM?

s_am: <p>I want to fine-tune the BERT model on Sequence Classification after additional pre-training on MaskedLM. Can I do MaskedLM fine-tuning first then sequence classification fine-tuning on same model? Would that add 2 additional layers on top of the pre-trained model?<br>
I do not understand it clearly. I am new to this, I have tried reading some blogs and forums but I’m still not very clear.<br>
I will be thankful for any kind of clarifiation.</p>
Improve the indexing spped

rickhuang: <p>Hi, x is a 4-D tensor and y is a 2-D tensor， and i want to replace the value of x by corresponding value in y with x as index. Like the code I write below. Because x.shape[0] is so big, it will make my code be very slow, so I want to know whether some quick method to do this by some torch function. Thanks</p>
<pre><code class="lang-python">index_array = x.clone().long()
for i in range(0,x.shape[0]):
    x[i] = y[i][index_array[i]]
</code></pre>

ptrblck: <p>Direct indexing with broadcasting should work:</p>
<pre><code class="lang-python">x = torch.randint(0, 10, (10, 2, 3, 4)).float()
y = torch.randn(10, 10)

index_array = x.clone().long()
for i in range(0,x.shape[0]):
    x[i] = y[i][index_array[i]]


x2 = y[torch.arange(y.size(0))[:, None, None, None], index_array]
print((x - x2).abs().max())
# &gt; tensor(0.)
</code></pre>

rickhuang: <p>Thanks. It can work.</p>

rickhuang: <p>Sorry for disturbing you.<br>
I want to ask that do you know any methods for facilitate this indexing?<br>
Because I need to replace the huge size of activation with look-up tables, this step is the bottleneck for my training.<br>
Or broadcasting is the fastest method build in the pytorch?<br>
I don’t know whether I can use gpu or other resources to parallel implement it?<br>
Thanks.</p>

ptrblck: <p>If you are already pushing the tensors to the GPU, the GPU would also be used in the indexing operation. I’m not aware of a quick speedup. How much time does this indexing take compared to other operations?</p>

rickhuang: <p>I only add this to the activation for every layer.<br>
Then, the computing time for one epoch will increase from 5s to 44s for 10000 data.<br>
And, the storage increase much.<br>
Thanks</p>

rickhuang: <pre><code class="lang-python">class Mapping_multi(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, num_partial_sum):
        output = input.clone()
        with torch.no_grad():
            error_bias_array_real = (torch.multinomial(error_distribution, num_partial_sum, replacement=True) - 35).float().permute(1,0)
            error_array_real = error_bias_array_real + identity_array[None, : ]
            error_array = error_array_real.float()
            value_index = (output+35).long()
            output = error_array[torch.arange(num_partial_sum)[:, None, None, None], value_index]
        return output

    @staticmethod
    def backward(ctx, grad_output):
           grad_input = grad_output.clone()
        return grad_input, None
</code></pre>
<p>Here is my code, if I add this function after the convolution at every layer, it will increase much time.<br>
This function is that I want to replace the activation to error value.<br>
error array is 2-d tensor with error value, I want replace the activation according to its value with error array.</p>
Decoding ascii tensor

austinmw: <p>Hi, I have a tensor of ascii integers (for example 48 through 57, representing the MNIST integer labels of 0 through 9). What’s the fastest way to convert these ascii integers to the true integer labels?</p>

JuanFMontesinos: <p>Hmmm if you are in cpu a look-up table.<br>
If you are in GPU don’t really know. Guess a dictionary would require synchronization to the cpu. Still don’t know if the nn.ModuleDict allows to use a tensor as key.<br>
If that were the case you can use the standard mapping  aka<br>
if if ‘0’ is 45 then 3 is 48 and can do that in gpu.</p>
Error when using torch.multiprocessing module

Gwon: <p>Hi,<br>
I have a problem with use torch.multiprocessing modules.</p>
<p>In my code,</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

class MultiProcessing:
     def run_inference(self):
          # Load detection model
          model = MyModel()
          ...
          processes = []
          for process_idx in range(2):
               p = mp.Process(target=self.do_detection, args=(model, ))
               p.start()  # Error here
               processes.append(p)
          for p in processes:
               p.join()

     def do_detection(self, model):
          vid_cap = cv2.VideoCapture("&lt;VideoFile&gt;")
          while True:
               # Read video
               _, img = vid_cap.read()
               # Input video frame to detection model
               pred = model(img)
               # Visualize detection result
               ...
if __name__ == '__main__':
     inference = MultiProcessing()
     inference.run_inference()
</code></pre>
<p>Error like below</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=..\torch/csrc/generic/StorageSharing.cpp line=249 error=801 : operation not supported
Traceback (most recent call last):
  File "D:\anaconda3\envs\conda\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\anaconda3\envs\conda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "D:\anaconda3\envs\conda\lib\site-packages\torch\multiprocessing\reductions.py", line 247, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
RuntimeError: cuda runtime error (801) : operation not supported at ..\torch/csrc/generic/StorageSharing.cpp:249
Traceback (most recent call last):
  File "D:\anaconda3\envs\conda\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
</code></pre>
<p>To solve this problem, I search many solutions.<br>
Most solutions say set the <code>num_worker=0</code> in dataloader, but i’m not using dataloader.</p>
<p>How can i solve this problem?</p>