akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Return string by "torchtext.data.Field"

username: Hou_ZeYu
cooked: <p>I already read the tutorial about the <strong>torchtext</strong>.<br>
I known how to use the <strong>Field</strong> handle text and target data.</p>
<p>Now I want to customized my data loader base on <strong>Field</strong> and <strong>Iterator</strong> object. In the <strong>torchtext.data.Field</strong>, the return type should be <strong>torch.tensor</strong>.</p>
<p>Is it possible to customize so that the output is string type?<br>
Because some times, I need to check the data id for each mini-batch in my case.</p>
<p>For example, I have a data like blow.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ad433eaf01cb9e5a6e2f96c1d1071d9c5e2d4e5.png" alt="A" data-base62-sha1="m5GaItapDTrrX4B6ph4s9nN8z4N" width="192" height="367"></p>
<p>I want to create a <strong>dataloader</strong> base on <strong>torch.data.TabularDataset</strong> so that the return is a list [id26305, id17569, id11008, id27763, id12958] when batch size is 5.</p>
<p>My code is here.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/951b3d06c08c1e76c1a11dc084ce5e9601e9b4fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/951b3d06c08c1e76c1a11dc084ce5e9601e9b4fd" title="B"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/951b3d06c08c1e76c1a11dc084ce5e9601e9b4fd.png" alt="B" data-base62-sha1="lh3v464O7OboXiF5InLKrRaRdUx" width="543" height="500" data-dominant-color="2D3038"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">B</span><span class="informations">560×515 12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I try display a mini-batch for data id, I get the error blow.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72beca496769cd0fa50d71e9ceac62ca3bbb35a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/72beca496769cd0fa50d71e9ceac62ca3bbb35a1" title="C"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72beca496769cd0fa50d71e9ceac62ca3bbb35a1.png" alt="C" data-base62-sha1="gn5cg0kxG96gxuAtWecyHVhwX4d" width="453" height="500" data-dominant-color="2B3237"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">C</span><span class="informations">728×803 24.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to customized the data loader base on <strong>torchtext</strong> for this case?</p>
Computer shuts down during inference time

username: paul_shuvo
cooked: <p>My computer shuts down when running PyTorch. I have the following specs:</p>
<pre><code class="lang-auto">Ubuntu 20.04
GPU: GTX1080
CUDA 10.2
Python 3.7
PyTorch 1.6.0
</code></pre>
<p>I’m running some model using the image from the webcam or video, but the CPU heats up and then shuts down. First I thought it might be the model/code that causing this, but I’ve run several different models, but the issue persists.</p>

username: Megh_Bhalerao
cooked: <p>Are you pushing your model, inputs etc. to cuda?</p>

username: paul_shuvo
cooked: <p>Yes, I’m. it’s using ‘cuda:0’ as a device.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="paul_shuvo" data-post="1" data-topic="104132">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paul_shuvo/48/14353_2.png" class="avatar"> paul_shuvo:</div>
<blockquote>
<p>but the CPU heats up and then shuts down.</p>
</blockquote>
</aside>
<p>This sounds like a root cause and I’m not sure if this is still a PyTorch question.<br>
Could you make sure your CPU is properly cooled?</p>

username: paul_shuvo
cooked: <p>It only happens with PyTorch (I’ve tried several pytorch implementation of human pose estimation techniques). I’ve been running other ml stuff (TF, scikit-learn); they run just fine.</p>
<p>Currently, I’m trying to run <a href="https://github.com/MVIG-SJTU/AlphaPose" rel="noopener nofollow ugc">AlphaPose</a>. Same issue.</p>

username: ptrblck
cooked: <p>If you cannot provide proper cooling, you could try to artificially slow down the code e.g. by adding sleeps to the code or reducing the number of used threads (via <code>torch.set_num_threads</code> and <code>torch.set_num_interop_threads</code>).</p>
Why is there no distributed inference?

username: 0xFFFFFFFF
cooked: <p>Is there a way to enable distributed inference, instead of training? Also, is it possible to distribute the work across multiple servers each with multiple GPUs, or does it only work for a single server with multiple GPU? If any of these features are missing, will they be coming out soon?</p>
<p>Lastly, what would be the recommended environment / library to enable distributed inference on multiple servers each with multiple GPUs?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>For single server, you can use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel" rel="nofollow noopener">nn.DataParallel</a>.<br>
For multiple servers, the <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">distributed</a> package should have everything you need.</p>

username: Valiox
cooked: <p>I don’t get it. Everything in the distributed docs relates to training.</p>
<p>And DataParallel can be used for inference, sure, but for production it has little use if requests come at random times.</p>

username: seungjun
cooked: <p>So far I have only used a singler-server multi-GPU environment but in principle, DDP can be used at inference time, too.</p>
<p>What hinders using DDP at inference are the</p>
<ol>
<li>synchronization at backward</li>
<li>
<code>DistributedSampler</code> that modifies the dataloader so that the number of samples are evenly divisible by the number of GPUs.</li>
</ol>
<p>At inference, you don’t need backward computation and you don’t want to modify the evaluation data.<br>
You can use a custom dataloader for evaluation similarly <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/data/sampler.py" rel="noopener nofollow ugc">this example</a> to avoid the problems.</p>
<p>A related thread is <a href="https://discuss.pytorch.org/t/how-to-validate-in-distributeddataparallel-correctly/94267/11">here</a>.</p>

username: osalpekar
cooked: <p>For those looking for a production inference service that allows for serving requests on models in parallel, you can check out <a href="https://pytorch.org/serve/" rel="noopener nofollow ugc">TorchServe</a>.</p>
How to collect outputs from different GPUs, compute a batch level loss and do backward

username: KaiHoo
cooked: <p>This paper (SimCLR) introduced a self-supervised learning method. In this method, the InfoNCE loss is computed in a batch level with the feature similarities between different inputs. The paper also point out that a bigger batch size makes a better performance.</p>
<p>For example, I have a 8GPU machine and I can put 128 images on each GPU. If I use the trivial implementation, I will get eight 128<em>128 similarity metrics. How could I get one 1024</em>1024 similarity matrix? I was thinking about use all-reduce, but I am not sure if the gradient can be passed to all GPUs. Any methods to implement it? Thanks!</p>
<p>Edit: to make it  more clear, suppose we have B images and the feature extracted from the model is x1, x2, …, xB, the loss function takes the pairwise dot product similarity as inputs. Now I can only compute the pairwise similarity (128 times 128) on each GPU, sum up the loss from 8 GPUs and do backward. I hope to compute the pairwise similarity (1024 times 1024)  and directly do backward. How can we do this? Thanks!</p>

username: osalpekar
cooked: <p><a class="mention" href="/u/kaihoo">@KaiHoo</a> What is the operation you want to perform across these 8 128x128 matrices? Allreduce will ensure that each GPU ends up with the average (or sum) of the matrices on all the GPUs. There are numerous such collective operations that you can perform to communicate data across GPUs in the distributed package that may be useful (docs here: <a href="https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations</a>)</p>

username: KaiHoo
cooked: <p>Hello, I made my question more clear. I know the Allreduce op, but I am not sure if the gradient could pass this op? Thanks!</p>

username: osalpekar
cooked: <p>Yes, the gradients can be passed to collective operations. You can access the gradient tensors by checking the <code>.grad</code> field of each of the parameters.</p>
Only batches of spatial targets supported (3D tensors) but got targets of dimension: 4

username: moreshud
cooked: <p>I currently encountered the error. Sample code to reproduce the error:</p>
<pre><code class="lang-auto">prediction = torch.rand((2,3,5,5))
label = torch.randint(0,2,(2,3,5,5))
criterion = nn.CrossEntropyLoss()
loss = criterion(prediction, label)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/moshood/Documents/AMMI/AMMI_Research_Project/trail-1.py", line 240, in &lt;module&gt;
    loss = criterion(prediction, label)
  File "/home/moshood/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/moshood/.local/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/moshood/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/moshood/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2266, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>I read posts relating to Crossentroy with the label, not one-encoded, but in my case, the label per channel are either one or zero. I will be glad for your assistance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="moreshud" data-post="1" data-topic="104389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> moreshud:</div>
<blockquote>
<p>I read posts relating to Crossentroy with the label, not one-encoded, but in my case, the label per channel are either one or zero.</p>
</blockquote>
</aside>
<p>As you said, <code>nn.CrossEntropyLoss</code> doesn’t accept one-hot encoded targets, which seems to be the case in your workflow. Pass either class indices or use e.g. <code>nn.BCEWithLogitsLoss</code>, if you are dealing with a multi-label classification.</p>

username: moreshud
cooked: <p>You mean taking the argmax of both predictions and labels?</p>

username: moreshud
cooked: <p>I follow your approach and got the loss.</p>
<pre><code class="lang-auto">prediction = torch.rand((1,3,5,5), dtype=torch.float32)
label = torch.randint(0,2,(1,3,5,5), dtype=torch.float32)
criterion = nn.CrossEntropyLoss()
label = torch.argmax(label, 1)
print(label.shape, prediction.shape, label.dtype, prediction.dtype)
loss = criterion(prediction, label)
print(loss)
</code></pre>
<pre><code class="lang-auto">torch.Size([1, 5, 5]) torch.Size([1, 3, 5, 5]) torch.int64 torch.float32
tensor(1.1114)
</code></pre>
Libtorch C++ MPI example

username: kais
cooked: <p>Hello,<br>
Do you guys have a C++ example similar to the python sample here:<br>
<a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a><br>
From looking at the source code, I only see python support in source code; for example here …\torch\csrc\multiprocessing\init.cpp</p>
<p>Thank you.</p>

username: pietern
cooked: <p>Do you mean an example of distributed training using the C++ frontend? We don’t have one combining the two unfortunately. Also, there is not yet a <code>torch.nn.parallel.DistributedDataParallel</code> equivalent for the C++ frontend. That said, it is possible to use the distributed <em>primitives</em> from C++. See <code>torch/lib/c10d</code> for the source code.</p>

username: kais
cooked: <p>Yep, that’s what I meant. I will take a look at torch/lib/c10d and try to build one myself.<br>
Thanks for the reply.</p>

username: soumyadipghosh
cooked: <p><a class="mention" href="/u/kais">@kais</a> I am looking for an example for distributed training with C++ frontend. If you managed to build one, can you please share?</p>

username: soumyadipghosh
cooked: <p>I managed to implement a few examples using Libtorch and MPI to help others in the community. Check <a href="https://github.com/soumyadipghosh/eventgrad" rel="noopener nofollow ugc">https://github.com/soumyadipghosh/eventgrad</a></p>

username: osalpekar
cooked: <p><a class="mention" href="/u/soumyadipghosh">@soumyadipghosh</a> Thanks for contributing this to the community and for the C++/MPI example PR!</p>
<p>Just as a general note for this thread, using the c10d APIs will enable distributed data parallel training that will produce the same results as DDP. However, calling allreduce after the backward pass to synchronize gradients will likely lag in performance as compared to DDP, which overlaps computation and communication by synchronizing smaller buckets of gradients during the backward pass.</p>
Normalizing images between 0 and 1 with ToTensor() doesn't work

username: Codophile1
cooked: <p>Hi everyone,</p>
<p>I’m trying to train a segmentation network and I would like to normalize images between 0 and 1. However the ToTensor transform I use outputs very low pixel values. Here’s a code sample</p>
<pre><code class="lang-python">img_transforms = transforms.Compose([transforms.Grayscale(num_output_channels=1),
                                     transforms.Resize((100,100), interpolation=2),
                                     transforms.ToTensor()])
img = Image.open("myimg.png")
img = np.array(img).astype(float)
img*= (255.0/img.max())
img = img.astype(np.uint8)
print(img.max())
img = Image.fromarray(img)
img = img_transforms(img)
print(img.max())
</code></pre>
<p>And the given output:</p>
<pre><code class="lang-auto">255
tensor(0.1137)
</code></pre>
<p>I don’t understand why I don’t get 1 as a max value since the ToTensor function is supposed to output values between 0 and 1.</p>
<p>Could anyone shed some light on what may be occurring?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>The <code>ToTensor</code> transformation will normalize the <code>uint8</code> input image by dividing by <code>255</code> as seen <a href="https://github.com/pytorch/vision/blob/4ab46e5f7585b86fb2befdc32d22d13635868c4e/torchvision/transforms/functional.py#L112-L113">here</a>.<br>
Based on your code, I guess that the <code>Resize</code> operation might lower the max. values of <code>255</code> to a smaller one (although I wouldn’t expect to see such a decrease). Could you try to use <code>PIL.Image.NEAREST</code> as the <code>interpolation</code> method and recheck the code?</p>

username: Codophile1
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, the resize transform was not the issue here but I figured out it was actually due to the fact that my images are mainly blue and that the Grayscale transform multiply blue values by 0.114. I solved the problem by manually setting the max value to 1. I still don’t know why the ToTensor() function didn’t normalize the values between 0 and 1 after the Grayscal transform though.</p>
<p>Here is my solution:</p>
<pre><code class="lang-python">img = Image.open("myimg.png")
img = img_transforms(img)
img*= (1.0/img.max())
</code></pre>
Model doesn't seem to be learning using a pretrained VGG16 for transfer learning

username: Vedank_Pande
cooked: <p>I’m trying to use transfer learning to classify facial emotions on the FER2013 dataset from kaggle. I resized the images to 224x224 and converted them to 3 channel grayscale images. While training, there seems to be no progress at all and the loss function graph doesn’t look right either. Is there anything wrong I’m doing? (I’m relatively new to deep learning)</p>
<pre><code class="lang-auto">model = torchvision.models.vgg16(pretrained=True)

#model summary

print(model)
# freezing parameters for pretrained VGG16

for params in model.parameters():

    params.require_grad = False

#number of features in penultimate classifier layer

num_feature = model.classifier[6].in_features

model.classifier[6] = nn.Sequential(

    nn.Linear(num_feature, 256), nn.ReLU(), nn.Dropout(0.2),

    nn.Linear(256, 7))

for params in model.classifier.parameters():

    params.require_grad = True

model = model.to(device)

print(model)


criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.parameters())

epochs = 10

loss_log = 100

LOSS = []

for epoch in range(epochs):

    epoch_correct = 0

    print(f"Epoch: {epoch+1}/{epochs}")

    for idx,(x,label) in enumerate(x_train_data):

        x = x.to(device)

        label = label.to(device)

        optimizer.zero_grad()

        out = model(x)

        loss = criterion(out,label)

        if (idx+1)%loss_log == 0:

            print(f"log loss at {idx}: {loss}")

        LOSS.append(loss)

        loss.backward()

        optimizer.step()
</code></pre>
<p>Here’s the graph:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b02f48b6402319a8aa88a49f1ebd9ac4120e592.png" alt="index" data-base62-sha1="hyd3Kq60BPWWgH7YCHuDUbuCNVM" width="378" height="248"></p>

username: ptrblck
cooked: <p>Try to overfit a small dataset first, e.g. just 10 samples, by playing around with some hyperparameters (e.g. the learning rate) and make sure your model is able to do so.</p>
<p>Also, don’t append the loss directly to a <code>list</code> via:</p>
<pre><code class="lang-python">LOSS.append(loss)
</code></pre>
<p>if you don’t want to call <code>LOSS.backward()</code> at some point, as it would store the complete computation graph and would thus increase the memory usage.<br>
Use <code>LOSS.append(loss.item())</code> instead.</p>

username: Vedank_Pande
cooked: <p>Yup I’ll try that and thanks for the extra tip. I was wondering if maybe vgg16 just isn’t extracting the right features to classify facial emotions? Maybe trying a more a relevant network will work better like VGGFace2, I’ll still see what I get out of the standard vgg net though .</p>
Large reconstruction error in torch.svd() compared to svd() function in MATLAB

username: vaisakh_m
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/646a754232cdd406281f36362f3b3c61d1de9189.png" alt="image" data-base62-sha1="ekjPBxuZhikGuATMr8S2WOMLsSZ" width="581" height="193"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69e4119db31ea3ff00d12781419deecbb7aac261.png" alt="image" data-base62-sha1="f6KLIf9fVAQg1okR8fWLFsyDRTj" width="353" height="202"></p>
<p>As seen in the above two images, I’m getting different(extremely) reconstruction errors while using the torch.svd() and svd() function from MATLAB. What might be the reason? Is it solely because of the ways in which SVD is computed are different? Or, are there any factors in play here?<br>
Is there any method by which I can reduce the error while using torch.svd() ?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>MATLAB uses double precision by default, so you could also apply the same in PyTorch:</p>
<pre><code class="lang-python">W = torch.randn(1000, 1000, dtype=torch.float64)
U, S, V = torch.svd(W)
error = torch.norm(W - U@torch.diag(S)@V.t())
print(error)
&gt; tensor(3.3153e-12, dtype=torch.float64)
</code></pre>

username: vaisakh_m
cooked: <p>I completely overlooked this difference.<br>
Really appreciate your help. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/+1/5.png?v=9" title=":+1:t5:" class="emoji" alt=":+1:t5:"></p>
Matrix multiplication with transposed matrix

username: John_Price
cooked: <p>what is the best way to multiply a transposed matrix D^T with matrix D in pytorch?</p>

username: vaisakh_m
cooked: <p><a class="mention" href="/u/john_price">@John_Price</a><br>
Use any one of the methods given below:</p>
<ul>
<li>
<p>D @ D.t()</p>
</li>
<li>
<p>torch.matmul(D,D.t())</p>
</li>
<li>
<p>torch.mm(D,D.t())</p>
</li>
</ul>
Recreate array using only PyTorch

username: Harry-Garrison
cooked: <p>I am trying to recreate the following array in pure torch. How can I achieve it using nothing but PyTorch?</p>
<p><code>t_list= float('inf') * np.ones(np.append( 10, 3+ 1 + np.arange( 2)))</code></p>

username: vaisakh_m
cooked: <p><a class="mention" href="/u/harry-garrison">@Harry-Garrison</a> I hope this works for you.</p>
<pre><code class="lang-auto">t_list = float('inf') * torch.ones(torch.cat( [torch.tensor([10]), 3+1+torch.arange( 2)] ).tolist())
</code></pre>

username: Harry-Garrison
cooked: <aside class="quote no-group" data-username="vaisakh_m" data-post="2" data-topic="104396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vaisakh_m/48/31107_2.png" class="avatar"> vaisakh_m:</div>
<blockquote>
<pre><code class="lang-auto">t_list = float('inf') * torch.ones(torch.cat( [torch.tensor([10]), 3+1+torch.arange( 2)] ).tolist())
</code></pre>
</blockquote>
</aside>
<p>Excellent, thank you!</p>
Torch.nn.Module always necessary?

username: samra-irshad
cooked: <p>Hi everyone, I have a question regarding wrapping an operation/module in a CNN model. What I mostly see in codes is people wrap the modules inside custom classes that inherit from <code>torch.nn.Module</code> class. I am curious if it’s necessary to wrap a module in <code>torch.nn.Module</code> class? or it’s only necessary if we want that custom module to be registered and possibly if it requires gradient?</p>

username: Shisho_Sama
cooked: <p>It depends.<br>
If what you need to implement have a custom forward where it requires some logic (if/else, etc) go the module way.<br>
otherwise If for example all you need to do is to tie together some layers in seccession, (create some blocks like conv, bn, activation , etc) you can use functions as well and return a modulelist, nn.Sequential, etc and use that.</p>
Libtorch sparse x dense batch multiplication

username: rb11
cooked: <p>Hi,</p>
<p>Is there any way to perform broadcasting matrix product between sparse A[1x13x13] and dense B[1024x13x139]?<br>
If A is dense torch::matmul works fine, but if it’s sparse everything crashes.<br>
I’m using libtorch 1.7.0<br>
If it’s impossible right now then probably there’s some workaround?</p>
Image Captioning: How to Pack Extracted Image Features With Caption Embeddings

username: darylnak
cooked: <p>I am following an image captioning tutorial <a href="https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning" rel="noopener nofollow ugc">here</a> and am struggling to comprehend a particular line of code. As a visual, I am trying to build the following architecture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://raw.githubusercontent.com/yunjey/pytorch-tutorial/master/tutorials/03-advanced/image_captioning/png/model.png" title="" rel="noopener nofollow ugc"><img src="https://raw.githubusercontent.com/yunjey/pytorch-tutorial/master/tutorials/03-advanced/image_captioning/png/model.png" alt="" width="690" height="326"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">1695×801</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p>In reference to the code <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/image_captioning/model.py" rel="noopener nofollow ugc">here</a>, I don’t understand why the last line in this snippet is possible (lines 37-39):</p>
<pre><code class="lang-auto">embeddings = self.embed(captions)
embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
packed = pack_padded_sequence(embeddings, lengths, batch_first=True)
</code></pre>
<p>Specifically, if <code>lengths</code> corresponds to the length of each respective caption, why is it that we can<br>
concatenate <code>features.unsqueeze(1)</code> to our embeddings without adding an additional entry to <code>lengths</code>?</p>
Very Weird error,No such file or directory: '/content/dataset/4738.jpg'

username: vishak_bharadwaj
cooked: <p>I can’t figure out why/where the dataloaders are figuring there is a file called 4738.jpg. I can’t even find the source of the mistake</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/VishakBharadwaj94/Image_Similarity_AutoEncoder/blob/main/Image_Similarity.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/VishakBharadwaj94/Image_Similarity_AutoEncoder/blob/main/Image_Similarity.ipynb" target="_blank" rel="noopener nofollow ugc">VishakBharadwaj94/Image_Similarity_AutoEncoder/blob/main/Image_Similarity.ipynb</a></h4>
<pre><code class="lang-ipynb">{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Image_Similarity.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP/df/gPIWoABuWEe/KZ1al",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
</code></pre>

  This file has been truncated. <a href="https://github.com/VishakBharadwaj94/Image_Similarity_AutoEncoder/blob/main/Image_Similarity.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>The image path is defined as <code>img_loc = self.data_path/f'{idx}.jpg'</code>, so it uses the passed <code>idx</code> to the <code>__getitem__</code>. By default this variable will be in the range <code>[0, len(dataset)-1]</code>, so you should make sure that all these files exist.</p>

username: vishak_bharadwaj
cooked: <p>Wow, that was extremely silly. Thank you for helping me with the blindspot!</p>
LSTM source code question

username: Inkyung
cooked: <p>Hi</p>
<p>I am trying to make customized LSTM cell but have some problems with figuring out what the really output is.</p>
<p>From the source code, it seems like returned value of <code>output</code> and <code>permute_hidden</code> value.</p>
<p>And <code>output</code> and <code>hidden</code> values are from <code>result</code>.</p>
<p>I am wondering the what result means and where the <code>result</code> is coming from.</p>
<pre><code class="lang-auto">        if batch_sizes is None:
            result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
                              self.dropout, self.training, self.bidirectional, self.batch_first)
        else:
            result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
                              self.num_layers, self.dropout, self.training, self.bidirectional)
        output = result[0]
        hidden = result[1:]
        # xxx: isinstance check needs to be in conditional for TorchScript to compile
        if isinstance(orig_input, PackedSequence):
            output_packed = PackedSequence(output, batch_sizes, sorted_indices, unsorted_indices)
            return output_packed, self.permute_hidden(hidden, unsorted_indices)
        else:
            return output, self.permute_hidden(hidden, unsorted_indices)
</code></pre>
<p>Any help or comments would be appriciated!</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Depending on which implementation and device you are using I assume the <code>result</code> is created in one of <a href="https://github.com/pytorch/pytorch/blob/272f4db043ec2c63ecfe6d2759e7893cb842a3c3/aten/src/ATen/native/RNN.cpp">these</a> methods.</p>

username: Inkyung
cooked: <p>I am trying to understand what <code>result</code> means and how they are separated into <code>output</code> and <code>hidden</code> value!</p>
<p>Since as far as I know the <code>output</code> of the LSTM is hidden and cell states continuously, but wondering how <code>output</code> is made of.</p>
How to implement geometric sequence with a single nn.Parameter variable

username: Bob_Li
cooked: <p>I have the following definition in the init of my model<br>
<code>self.p = nn.Parameter(torch.ones(1))</code><br>
My question is how to implement a geometric sequence based on self.p and use it during <code>forward()</code>, which is like<br>
<code>[p**i for i in range(5)]</code>, I have tried <code>p_geometric = torch.tensor([self.p**i for i in range(x.size(1))]).cuda()</code>, but the weight of <code>self.p</code> did not get updated with that implementation.<br>
Thanks!</p>

username: ptrblck
cooked: <p>You shouldn’t recreate a tensor, as it would break the computation graph. Use <code>torch.cat</code> or <code>torch.stack</code> instead:</p>
<pre><code class="lang-python">p = nn.Parameter(torch.ones(1) * 2)
out = torch.cat([p**i for i in range(10)])
out.mean().backward()
print(p.grad)
&gt; tensor([409.7000])
</code></pre>

username: Bob_Li
cooked: <p>Thanks! It works~ <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Why does results with different accuracies are generate when state dicts are loaded into the model differently

username: WeiQin-C
cooked: <p>Given that I have a model and multiple sets of weights, say weights from epoch 1 to epoch 5 (w_1, w_2, w_3, w_4, w_5). I realise that different results are generated (during inference) when I load my weights into my model in two different ways:</p>
<p>My model is a CNN network with Batch Norms and ReLUs included.</p>
<p>Method 1:</p>
<pre><code class="lang-auto">model = myModel()
model = nn.DataParallel(model)
model = model.cuda()
for idx in range(1, 6):
     state_dict = torch.load(w_idx)
     model.load_state_dict(state_dict)
     model.eval()
     outputs = model(inputs)  #inference
</code></pre>
<p>Method 2:</p>
<pre><code class="lang-auto">for idx in range(1, 6):
     state_dict = torch.load(w_idx)
     ###############################
     model = myModel() 
     model = nn.DataParallel(model)
     model = model.cuda()
     ###############################
     model.load_state_dict(state_dict)
     model.eval()
     outputs = model(inputs)  #inference
</code></pre>
<p>May I ask why is that the case? Thank you</p>

username: ptrblck
cooked: <p>How large are these differences and are you also seeing them using the same approach?<br>
If so, you might be facing non-deterministic results, if you use e.g. cudnn benchmarking etc. due to the limited floating point precision.</p>
PyTorch Mobile NNAPI is slower than CPU

username: windmaple
cooked: <p>I’m following this <a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" rel="noopener nofollow ugc">tutorial</a> to try out NNAPI support in PyTorch Mobile on Pixel 3a (Android 10, Qualcomm Snapdragon 670). I compiled the speed benchmark binary off commit ID <strong>4ed7f36ed</strong></p>
<p>The commands I used are:</p>
<blockquote>
<p>adb shell /data/local/tmp/speed_benchmark_torch --pthreadpool_size=1 --model=/data/local/tmp/mobilenetv2-quant_full-nnapi.pt --use_bundled_input=0 --warmup=5 --iter=200<br>
adb shell /data/local/tmp/speed_benchmark_torch --pthreadpool_size=1 --model=/data/local/tmp/mobilenetv2-quant_full-cpu.pt --use_bundled_input=0 --warmup=5 --iter=200</p>
</blockquote>
<p>Here are the benchmark numbers I got:</p>
<p>Quant mobilenet v2 using CPU: 66.8 ms<br>
Quant mobilenet v2 using NNAPI: 171 ms</p>
<p>NNAPI not only did not accelerate the model, but slowed it down significantly. I’m really curious what I’m missing here.</p>

username: windmaple
cooked: <p>Updated my phone to Android 11 and it works now.</p>
Issues with loading csv file

username: Gelassen
cooked: <p>Hello,</p>
<p>I am a bit stuck with loading csv file. I get <code>invalid load key, '\xef'.</code> or <code>KeyError: 44</code> for code snippets below respectively</p>
<p><code>vrn = torch.load('dataset-housing-price-prediction-service/17-nov-2020-infoline-real-estate.csv')</code></p>
<pre><code class="lang-auto">import joblib

vrn = joblib.load(open('test.csv', 'rb'))
</code></pre>
<p>In web there were different proposals on the cause of the issue, but nothing in details. As you may notice I tried to vary tools to find out is this a tool related issue.</p>
<p>Likely it is an issue with file, but it is plane cvs. I attach file sample by this <a href="https://drive.google.com/file/d/1PDBCUKm45BlJJeGF_ynob3XNNqsM9N9p/view?usp=sharing" rel="noopener nofollow ugc">link</a></p>
<p>Could you please take a look on?</p>

username: granth_jain
cooked: <p>You can try to read CSV files from pandas library.<br>
Import pandas as pd<br>
Data = pd.read_csv(“path/to/file.csv”)<br>
Then you can convert it to the data type you want</p>

username: Gelassen
cooked: <p>Thank you, pandas parse it without an issue. Do you have ideas why pytorch didn’t it?</p>
Can I use the same layer instance for different convolution layers?

username: Jingnan_Jia
cooked: <p>One classic network for image classification is as follows:</p>
<pre><code class="lang-auto">class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))
</code></pre>
<p>If I want to add a new convolution layer, can I just reuse the <code>self.conv2 </code>? like the following code:</p>
<pre><code class="lang-auto">class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv2(xb))  # add a new layer
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))
</code></pre>
<p>I am afraid the new-added layer would have the same weights as the former layer.</p>

username: ptrblck
cooked: <p>Adding a new layer (with new parameters) is not the same as reusing the layer (same parameters).<br>
If you want to add a module, you would have to initialize it and use it in the <code>forward</code>.<br>
Depending on your use case, you might want to derive a new class using <code>Mnist_CNN</code> as the base class and add the new layer to it.</p>
How to combine several lstm networks into one model?

username: andy29
cooked: <p>I am trying to create three separate LSTM networks, and then merge them together into one big model. From my understanding I can create three lstm networks and then create a class for merging those networks together. Is that correct?<br>
I am kind of new to this.</p>

username: ptrblck
cooked: <p>Yes, that should be possible as you can freely concatenate the outputs of several modules and pass it to a new module.</p>

username: andy29
cooked: <p>I have tried to code this, but this is as far as I got. I am not sure how to determine the input and output dimension of concat_layer. Also, what goes in the forward function under lstms?<br>
Any help is greatly appreciated.</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):

        super(Net, self).__init__()
        self.lstm1 = torch.nn.LSTM(input_size = 1, hidden_size = 2, num_layers = 1, batch_first = True)
        self.lstm2 = torch.nn.LSTM(input_size = 1, hidden_size = 2, num_layers = 1, batch_first = True)
        self.lstm3 = torch.nn.LSTM(input_size = 1, hidden_size = 2, num_layers = 1, batch_first = True)
        self.concat_layer = torch.nn.Linear(?, ?)
        self.linear = torch.nn.Linear(,1)

    def forward(self, x):

        lstm1 = ?
        lstm2 = ?
        lstm3 = ?
        concat = torch.cat((lstm1, lstm2, lstm3), dim=1)
        output = self.linear(concat)
        return output
</code></pre>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html?highlight=lstm#torch.nn.LSTM">LSTM docs</a> explain the expected input with their shapes.</p>
<p>Once your input works, you could add a print statement in the <code>forward</code> method to check the shape of the concatenated tensor and adapt the <code>in_features</code> accordingly.</p>
How can I print out dice_coef or define other metric in the code?

username: rui_zhou
cooked: <p>I am using the following code and I got the iou_score, but I want to evaluate it by:</p>
<pre><code class="lang-auto">Precision &amp; Recall
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57f18cdc456029a3b66a6a8c012d2b26716ed5fd.png" alt="image" data-base62-sha1="cxZ3Ev17yZ5y8cbXPoFeVPkCKuh" width="399" height="133"><br>
such as this one:</p>
<pre><code>FP, FN, TP, TN = numeric_score(prediction, groundtruth)
N = FP + FN + TP + TN
accuracy = np.divide(TP + TN, N)
return accuracy * 100.0`
</code></pre>
<p>how can I add them in , or just print the Dice?</p>
<pre><code class="lang-auto">def iou_score(output, target):
    smooth = 1e-5

    if torch.is_tensor(output):
        output = torch.sigmoid(output).data.cpu().numpy()
    if torch.is_tensor(target):
        target = target.data.cpu().numpy()
    output_ = output &gt; 0.5
    target_ = target &gt; 0.5
    intersection = (output_ &amp; target_).sum()
    union = (output_ | target_).sum()

    return (intersection + smooth) / (union + smooth)


def dice_coef(output, target):
    smooth = 1e-5

    output = torch.sigmoid(output).view(-1).data.cpu().numpy()
    target = target.view(-1).data.cpu().numpy()
    intersection = (output * target).sum()

    return (2. * intersection + smooth) / \
        (output.sum() + target.sum() + smooth)

</code></pre>
<pre><code class="lang-auto"> iou = iou_score(output, target)
            avg_meter.update(iou, input.size(0))

            output = torch.sigmoid(output).cpu().numpy()

            for i in range(len(output)):
                for c in range(config['num_classes']):
                    cv2.imwrite(os.path.join('outputs', config['name'], str(c), meta['img_id'][i] + '.jpg'),
                                (output[i, c] * 255).astype('uint8'))

    print('IoU: %.4f' % avg_meter.avg)

    torch.cuda.empty_cache()
</code></pre>
<p><a href="https://github.com/4uiiurz1/pytorch-nested-unet" rel="noopener nofollow ugc">https://github.com/4uiiurz1/pytorch-nested-unet</a><br>
With reagrds, any help would be so grateful!</p>
Custom loss function RuntimeError tensor does not have a grad_fn

username: abdelhaqo
cooked: <p>Trying to utilize a custom loss function and getting error ‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’. Error occurs during loss.backward()</p>
<p>I’m aware that all computations must be done in tensors with ‘require_grad = True’. I’m having trouble implementing that as my code requires a nested for loop. Is there a way to create an empty tensor and append it? Below is my code.</p>
<pre><code class="lang-auto">def Gaussian_Kernal(x, mu, sigma):
  p = (1./(math.sqrt(2. * math.pi * (sigma**2)))) * torch.exp((-1.) * (((Variable(x)**2) - mu)/(2. * (sigma**2))))
  return p

class MEE(torch.nn.Module):
  def __init__(self):
    super(MEE,self).__init__()

  def forward(self,output, target, mu, variance):

    error = torch.subtract(Variable(output),Variable(target))
  
    error_diff = []
    for i in range(0, error.size(0)):
      for j in range(0, error.size(0)):
        error_diff.append(error[i] - error[j])

    error_diff = torch.cat(error_diff)
    torch.tensor(error_diff,requires_grad=True)

    loss = (1./(target.size(0)**2)) * torch.sum(Gaussian_Kernal(Variable(error_diff), mu, variance*(2**0.5)))

    loss = Variable(loss)

    return loss
</code></pre>

username: ptrblck
cooked: <p>You are breaking the computation graph by recreating tensors and the deprecated <code>Variables</code>.<br>
To keep the computation graph intact you would need to apply operations on tensors only, so remove:</p>
<pre><code class="lang-python">... Variable(output),Variable(target)
torch.tensor(error_diff,requires_grad=True)
loss = Variable(loss)
</code></pre>
<p>etc.</p>
Auto-encoders testing images size

username: OuisYasser
cooked: <p>Hello everyone,<br>
Let’s say I’m Training an Auto-encoder( Unet) using images of input size 256x256.<br>
For testing, I have images of X.X size so how to test the model on these data ?<br>
Thank you,</p>

username: ptrblck
cooked: <p>Since you are working on an Auto-encoder I assume you could calculate the loss using the input and output directly. Depending on the model architecture you could pass variable input shapes to the model. E.g. if you are just using conv and pooling operations, this might be possible. I assume that input with a largely different input shape could create worse results as your model wasn’t trained on these, but that’s just a guess.</p>
Torch.tensor doesnt work, provides following error: " TypeError: 'module' object is not callable"

username: Akis_Linardos
cooked: <blockquote>
<blockquote>
<blockquote>
<p>torch.tensor(<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
TypeError: ‘module’ object is not callable<br>
torch.tensor([0,2])<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
TypeError: ‘module’ object is not callable</p>
</blockquote>
</blockquote>
</blockquote>
<p>Possible reasons why that would happen?</p>

username: page
cooked: <p>i had same issue in version 0.3<br>
upgraded to version 0.4. solved for me.</p>

username: Akis_Linardos
cooked: <p>Ah yes updating to the newest version did solve the issue. Thank you!</p>

username: Raghunadh_Nittala
cooked: <p>Hi,<br>
I’m also facing the same issue. Can you please tell me, how you updated the package? I tried</p>
<blockquote>
<p>conda update pytorch</p>
</blockquote>
<p>but still its showing the same version.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Try to uninstall pytorch and torchvision first, update conda and install it again using the instructions from the <a href="http://pytorch.org">website</a>.</p>

username: Raghunadh_Nittala
cooked: <p>Ahh. That worked. Thank you.</p>

username: simpleproblem
cooked: <p>I had the same problem. Upgrading to 0.4 works, however I have to use 0.3 for other reasons. Is there any way to work around this issue but still using 0.3 version? (The reason to avoid 0.4 is that the custom function seems to be very slow, so…) Can anyone help me on this? Thank you.</p>

username: ptrblck
cooked: <p>In older version you could use:</p>
<pre><code class="lang-python">torch.FloatTensor([])
torch.FloatTensor([1., 2.])
</code></pre>
<p>Also any supported type, e.g. <code>torch.LongTensor</code>.</p>

username: simpleproblem
cooked: <p>For 0.3 version, I want to add a learnable parameter like "self.scale = nn.Parameter(torch.tensor([10.0]), requires_grad=True) in nn.Module. But it throws an error, TypeError: ‘module’ object is not callable. If I use a tensor, torch.FloatTensor([10.0]), it is not learnable, right?</p>
<p>class PreActBottleneck(nn.Module):<br>
def <strong>init</strong>(self, inplanes, planes, stride=1, downsample=None):<br>
super(PreActBottleneck, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)<br>
self.scale = nn.Parameter(torch.tensor([10.0]), requires_grad=True)</p>

username: ptrblck
cooked: <p>You could still wrap the <code>FloatTensor</code> into a <code>Parameter</code>:</p>
<pre><code>self.scale = nn.Parameter(torch.FloatTensor([10.]))</code></pre>

username: simpleproblem
cooked: <p>Thank you. It works.</p>

username: ptrblck
cooked: <p>Could you post your code so that we can have a look at it?</p>

username: Megh_Bhalerao
cooked: <p>Sorry my apologies, I was making a stupid mistake. I am not facing this issue. Thanks for your help.</p>
Doubt about using conv2d or conv3d

username: deepfailure
cooked: <p>Hello,</p>
<p>I have a doubt about using conv2d or conv3d on my problem. I have an array of shape (M, M, N) where each image is formed by M x M pixels and we have N of those.</p>
<p>My question is:</p>
<ul>
<li>
<p>Should I use 2d conv where the channels are the N value (i.e. input shape (batch size, N, M, M)</p>
</li>
<li>
<p>3d conv net where we start with one channel on a 3D image (i.e. input shape (batch size, 1, M, M, N)</p>
</li>
</ul>
<p>The N images may have shared statistics. Now, I have tried both and the 3D conv net seems to give better results but I am not sure how should I interpret that and maybe it is due to an hyperparameter issue.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>The difference between <code>nn.Conv2d</code> and <code>nn.Conv3d</code> would be how the additional <code>N</code> dimension is handled in your use case.<br>
The <code>nn.Conv2d</code> layer would interpret <code>N</code> as the channel dimension and each kernel would thus use all channels in the default setup. The sliding windows would be applied in the spatial <code>MxM</code> dimensions.<br>
On the other hand, the <code>nn.Conv3d</code> layer would use the “sliding cube” in the <code>NxMxM</code> dimensions and use a single input channel.</p>
<p>I’m not familiar with your use case and don’t know which approach would make more sense. Since the 3D layer seems to give better results, you could stick to it.</p>
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

username: James_Lee
cooked: <p>It seemed that PyTorch failed to find my GPU. How can I fix this? Many thanks!</p>
<p>Python 3.8.3 (default, Jul  2 2020, 16:21:59)<br>
[GCC 7.3.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.randn(5)<br>
tensor([ 0.9021,  0.6406,  0.1212,  0.3215, -0.4576])<br>
torch.randn(5).to(0)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>Are you able to use other CUDA applications? If not, could you check if the compute mode of your GPUs is in an exclusive mode and reset it to the default one? Also, if you’ve upgraded something on your system make sure to reboot it afterwards.</p>