akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What convolution to use (conv1d, conv2d or conv3d)

username: pamdwj
cooked: <p>The image is represented with a stack of layers which are equidistant in depth (depth layers), and instead of feeding the whole image (every pixel) i want to feed R random pixels (from 0 to HxW). So I have an array of shape (N,D,C,R,1) where N is the batch, D is the number of layers, C channel and R is the number of random pixels. I want to combine the layers in a final image (N, 1, C, R, 1). What conv is best to use for this network?</p>

username: KFrank
cooked: <p>Hi Pam!</p>
<aside class="quote no-group" data-username="pamdwj" data-post="1" data-topic="129573" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> pamdwj:</div>
<blockquote>
<p>instead of feeding the whole image (every pixel) i want to feed R random pixels (from 0 to HxW).</p>
</blockquote>
</aside>
<p>Sampling random pixels seems like a bad idea because doing so<br>
throws away the important two-dimensional structure that your image<br>
has.</p>
<p>If for memory or computation-time reasons you need to reduce the<br>
amount of data in your images, I would recommend down-sampling<br>
in a way that preserves that two-dimensional structure.  This might<br>
be as easy as using pytorch’s <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.AvgPool2d.html" rel="noopener nofollow ugc"><code>AvgPool2d</code></a> or <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.MaxPool2d.html" rel="noopener nofollow ugc"><code>MaxPool2d</code></a> (across<br>
the HxW dimensions).</p>
<p>If you do that, I would then recommend applying  <code>Conv3d</code> across the<br>
depth and the down-sampled H and W dimensions.</p>
<p>To answer your original question, if you randomly sample pixels<br>
independently from each depth layer, I don’t think any convolution<br>
would make sense – there would be no “nearest-neighbor” spatial<br>
structure left for the convolution to be applied to.</p>
<p>If instead you sample pixels from each depth layer using the <em>same</em><br>
randomly-chosen locations in HxW, you could conceivably use<br>
<code>Conv1d</code>, applied across the depth dimension (because you have<br>
preserved the nearest-neighbor structure when you move from one<br>
depth layer to the next while keeping the location in your R dimension<br>
fixed).</p>
<p>Best.</p>
<p>K. Frank</p>
How to choose the DL model parameters?

username: KKKcat
cooked: <p>Hi, I’m comparing two semantic segmentation models FCN and DeepLabV3 with the same dataset, same seed, same loss function Cross Entropy. But i’m confused about how to pick the right model weights.</p>
<ol>
<li>For each model, should I pick the mode weights with smallest test loss, or the highest evaluation metrics (mean iou, pixel acc)? Because I remember in order to prevent overfitting, we should choose the smallest test loss. However, in this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">tutorial</a>, the author picks the greatest eval acc instead of smallest test loss.</li>
</ol>
<pre><code class="lang-auto"># deep copy the model
if phase == 'val' and epoch_acc &gt; best_acc:
    best_acc = epoch_acc
    best_model_wts = copy.deepcopy(model.state_dict())
</code></pre>
<ol start="2">
<li>The FCN model has better training loss, training acc, training iou, test acc, test iou but <strong>larger test loss</strong> than DeepLabV3 model along all epochs. In such case, can I say FCN model beats the DeepLabV3 model?</li>
</ol>

username: ptrblck
cooked: <aside class="quote no-group" data-username="KKKcat" data-post="1" data-topic="129484">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/958977/48.png" class="avatar"> KKKcat:</div>
<blockquote>
<p>Because I remember in order to prevent overfitting, we should choose the smallest test loss. However, in this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html">tutorial</a>, the author picks the greatest eval acc instead of smallest test loss.</p>
</blockquote>
</aside>
<p>In the common use case you won’t see the test loss until you finalize the model training and calculate the test loss/accuracy once. If you use the test loss for e.g. early stopping this would be considered a data leak and the model would perform worse during its deployment and usage of new data.</p>

username: KKKcat
cooked: <p>Thanks for your explanation. And sorry, I may mislead you.<br>
For example, I train a model 100 epochs. for each epoch, i have a training loss and test loss (i’ve done train test split), and other eval metrics. Now i find epoch 50 has the smallest test loss, while epoch 90 has the highest test mean iou. Which model weights should I use? the one of epoch 50 or the one of epoch 90?</p>
Reduction='none' leads to different computed loss

username: ettelephonne
cooked: <p>Hi everyone. I hope my question is no too stupid as I am a beginner.</p>
<p>I use the cross entropy loss with 512*512 images and a batch size of 3. I want to compute the reduction by myself. So I do:</p>
<p>criterion_none = torch.nn.CrossEntropyLoss(weight=class_weights, reduction=‘none’)<br>
criterion_reduc = torch.nn.CrossEntropyLoss(weight=class_weights)</p>
<p>loss_none = criterion_none(preds, masks) # without reduction<br>
loss_reduc = criterion(preds, masks) # with reduction</p>
<p>The first one gives a 3<em>512</em>512 (b, H, W) array of losses and the second one gives a scalar. So far, so good, that’s what does the reduction. The documentation says that by default, this reduction is done by averaging the losses over the mini batch. So I thought that doing:</p>
<p>loss_none = torch.mean(loss_none) will give the same result.</p>
<p>loss_none = loss_reduc</p>
<p>But I have (3 examples):</p>
<p>loss torch.mean() tensor(<strong>2.8491</strong>, device=‘cuda:0’, grad_fn=) loss reduction tensor(<strong>28.0402</strong>, device=‘cuda:0’, grad_fn=)<br>
loss torch.mean() tensor(<strong>5.1461</strong>, device=‘cuda:0’, grad_fn=) loss reduction tensor(<strong>44.5213</strong>, device=‘cuda:0’, grad_fn=)<br>
loss torch.mean() tensor(<strong>2.9905</strong>, device=‘cuda:0’, grad_fn=) loss reduction tensor(<strong>26.4063</strong>, device=‘cuda:0’, grad_fn=)</p>
<p>And why the “grad_fn” is not the same?</p>
<p>Cleary I do not understand what I am doing.</p>
<p>Thank you very for any help you could provide. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html">docs</a> specify how the average is calculated, if a <code>weight</code> is specified.<br>
<a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">This post</a> gives you an example of a manual reduction vs. the internal one.</p>

username: ettelephonne
cooked: <p>Thank you very much. I just tried with:</p>
<p>criterion_none = torch.nn.CrossEntropyLoss(reduction=‘none’)<br>
criterion = torch.nn.CrossEntropyLoss()</p>
<p>And indeed, it gave me the same loss.</p>
<p>loss torch.mean() tensor(<strong>1.3486</strong>, device=‘cuda:0’, grad_fn=) loss reduction tensor(<strong>1.3486</strong>, device=‘cuda:0’, grad_fn=)</p>
<p>I read the post with the example you gave for manual calculation of loss. I am still trying to adapt this part:</p>
<p>loss_weighted_manual = loss_weighted_manual.sum() / <strong>weights[target].sum()</strong></p>
<p>for image segmentation. I guess I’ll figure it out.</p>
<p>Thank you again.</p>
Downloading from torch.hub into memory

username: thoger
cooked: <p>Hi forum!</p>
<p>Question: Does torch.hub support downloading a file from URL directly to memory without touching disk first? Such as the utility <code>download_url_to_file</code> just without the <code>_to_file</code> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>I am hitting the classic issue of AWS Lambda of “No more space on device” when it hits the space limits of what can be stored on temporary disk on a Lambda instance. Typically I would load a file into memory which is a much more loose limitation on Lambda.</p>
<p>It may be a very practical question, but highly appreciated if anyone here have some insights that I don’t.</p>
<p>Thank you in advance!</p>
What's wrong with my code?

username: hananelroe
cooked: <p>import torch<br>
import torchvision<br>
from torchvision import transforms, datasets</p>
<p>train = datasets.MNIST("", train=True, download=True,<br>
transform=transforms.Compose([transforms.ToTensor()]))</p>
<p>test = datasets.MNIST("", train=False, download=True,<br>
transform=transforms.Compose([transforms.ToTensor()]))</p>

username: arya47
cooked: <pre><code class="lang-auto">import torch
from torchvision import datasets
import torchvision.transforms as tf

train_dataset = datasets.MNIST('./data', train=True, download=True, transform=tf.ToTensor())
test_dataset = datasets.MNIST('./data', train=False, download=True, transform=tf.ToTensor())
</code></pre>

username: hananelroe
cooked: <p>it works, thank you so much!</p>

username: hananelroe
cooked: <p>okay it worked once and when i tried to run it again it stopped working</p>

username: arya47
cooked: <p>Can you share what the error was, i believe its to do with the data already existing when you run it the second time (try deleting the data dir) and running again.</p>

username: hananelroe
cooked: <p>yeah I deleted the data and the error diappeared. thanks!</p>

username: hananelroe
cooked: <p>the error keeps appearing when I try to use the data, I think it has something to do with some of the downloads failing. am I right?</p>

username: hananelroe
cooked: <p>is it possible to download the dataset simply as a ZIP or something?</p>

username: arya47
cooked: <p>There should not an error. Can you share what the error is?</p>

username: hananelroe
cooked: <p>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\torchvision\datasets\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  …\torch\csrc\utils\tensor_numpy.cpp:180.)<br>
return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)</p>

username: arya47
cooked: <p>In my opinion you can safely ignore this warning when loading the dataset.</p>
FPN Anchor Generator

username: Jeremy_Tavrisov
cooked: <p>Looking for advice on how to set the parameters for the anchor generator for Mask-RCNN on a custom dataset.</p>
<p>Given a FPN ResNet backbone there are 5 levels with each level cutting the stride in half. Each output is given a single anchor size and a list of aspect ratios. So by default on the first level of the pyramid the image is 1/4 of the size and the anchor sizes are 32x32 with various aspect ratios applied. Then the next level is 1/8 with 64x64 and so on. Is this the right interpretation?</p>
<p>My image set has very long and very skinny objects and I’m not doing so well in detecting them. The problem I see is that as it downsamples that object gets too skinny that it’s hard to see but at the lower levels the anchors are too small. The other objects are very big with 1 class being fairly small but regularly shaped (not an extreme aspect ratio)</p>
<p>What’s the ideal way to compute ideal parameters for the anchor generator. I can loop through my train set and cluster the boxes but the whole FPN thing confuses me.</p>
Truncated BPTT with seq2seq

username: facto
cooked: <p>Hi,</p>
<p>I’m currently working on an encoder-decoder architecture using soft hybrid attention. Everything is working fine but I am facing OOM issues when dealing with long sequences.<br>
The best solution I found is to use Truncated Back Propagation Through Time (TBPTT) to reduce the memory consumption by splitting the sequence to produce.<br>
So my question is, how to retain the computation graph for my encoder only, after the backward pass ?</p>
<p>More specifically, the architecture is made up of an encoder which extracts a sequence of features from the input, an attention module which computes attention weights for each feature frame at a given time step and an LSTM-based decoder that outputs class probabilities for this same time step.<br>
This can be seen as a transformer architecture but the attention is hybrid so the training cannot be parallelized for the whole sequence with teacher forcing.</p>
<p>Let’s say I want to use TBPTT with k1=k2=k.<br>
I tried to call backward() after k step, detaching all intermediate tensors, but then the computation graph for the encoder is freed and the encoder parameters will not be updated for the following steps.<br>
And if I use backward(retain_graph=True), allocated memory is increasing at each step, also leading to OOM issues.<br>
The aim is to process the sequence by sub-sequences of length k and update the weights of the whole network for each sub-sequence without recomputing the features (encoder part) each time.</p>
<p>Am I doing things right ? Is there alternatives to TBPTT in my case ?</p>
<p>To make things clearer, here is how the code looks like without TBPTT:</p>
<pre><code class="lang-python"># x: input
# y: output sequence
# hidden: LSTM hidden state of the decoder module
loss = 0
features = EncoderModule(x)
for i in range(len(y)-1):
    att_weights, hidden, pred = AttentionDecoderModule(x, att_weights, hidden, y[i])
    loss += loss_func(pred, y[i+1]
self.backward_loss(loss)
self.step_optimizer()
self.optimizer.zero_grad(set_to_none=True)
</code></pre>
<p>And I would like something like:</p>
<pre><code class="lang-python"># x: input
# y: output sequence
# hidden: LSTM hidden state of the decoder module
loss = 0
features = EncoderModule(x)
for i in range(len(y)-1):
    att_weights, hidden, pred = AttentionDecoderModule(x, att_weights, hidden, y[i])
    loss += loss_func(pred, y[i+1]
    if i % k == 0 or i == len(y)-2:
        self.backward_loss(loss)
        # keep computation graph for the encoder
        self.step_optimizer()
        self.optimizer.zero_grad(set_to_none=True)
        loss = 0
        hidden = [h.detach() for h in hidden]
        att_weights = att_weights.detach()

</code></pre>
Is there a typo in the DataParallel? Device is only set to "cuda:0"

username: Santosh_Gupta
cooked: <p><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a></p>
<p>At beginning</p>
<blockquote>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>
</blockquote>
<p>So all the models and data are still just going to the first gpu. Is this a typo, or something deeper going on?</p>

username: ptrblck
cooked: <p>This is expected, as <code>nn.DataParallel</code> will use this GPU (the default device) to store the model and create the copies of the model before copying them to the other specified devices.<br>
<a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">This blog post</a> explains the workflow in detail.<br>
Note that this communication overhead (scattering/gathering the data from/to the default device) is also the reason why we recommend to use <code>DistributedDataParallel</code> with a single process per GPU for the best performance.</p>

username: Santosh_Gupta
cooked: <p>Thanks. From the blog post, for <code>nn.DataParallel</code>, we should be using <code>loss.mean().backward()</code> instead of <code>loss.backward()</code> ?<br>
The official tutorial does not indicate a need to change the backwards pass code.</p>

username: ptrblck
cooked: <p>The loss dimension or size won’t be increased, so no changes would be necessary compared to a single GPU run (assuming your model returns an output and you are calculating the loss “outside” of the model).</p>
What's the maning of this cpde?

username: wukong
cooked: <p>I know the meaning of std and mean, but it seem strange here with img = img* std +mean, why this code write like this form?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/402dad5601981896cc793063c17a5714a502a96e.png" data-download-href="https://discuss.pytorch.org/uploads/default/402dad5601981896cc793063c17a5714a502a96e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402dad5601981896cc793063c17a5714a502a96e_2_690x407.png" alt="image" data-base62-sha1="99Kq5FFZUV2pNVOA5v7gziAMAzA" width="690" height="407" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402dad5601981896cc793063c17a5714a502a96e_2_690x407.png, https://discuss.pytorch.org/uploads/default/original/3X/4/0/402dad5601981896cc793063c17a5714a502a96e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/402dad5601981896cc793063c17a5714a502a96e.png 2x" data-dominant-color="D2D0D2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">867×512 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: samarendra109
cooked: <p>First of all this is not at all the location to ask this question. All function and methods you are using are of numpy.<br>
To answer the question I think it is doing what it seems. It is multiplying img by 0.5 and adding 0.5 to it. I think the purpose is that the data was initially standardized.</p>
<p>Standardization formula : z = (x-mean)/std</p>
<p>So by doing z*std + mean you are basically de-standardizing the data to it’s original form.</p>

username: wukong
cooked: <aside class="quote no-group" data-username="samarendra109" data-post="2" data-topic="129353">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarendra109/48/32204_2.png" class="avatar"> samarendra109:</div>
<blockquote>
<p>Standardization formula : z = (x-mean)/std</p>
</blockquote>
</aside>
<p>Thank u for your anser^_^</p>
Pytorch3D: 2D projection object boundary mask and visual quality

username: Divya_B
cooked: <p>Hi Team,</p>
<p>I am a newbie to pytorch3d exploring supercool notebooks!<br>
I have the following questions on <a href="https://github.com/facebookresearch/pytorch3d/blob/main/docs/tutorials/render_textured_meshes.ipynb" class="inline-onebox" rel="noopener nofollow ugc">pytorch3d/docs/tutorials/render_textured_meshes.ipynb at main · facebookresearch/pytorch3d · GitHub</a></p>
<ol>
<li><code>image_grid(images.cpu().numpy(), rows=4, cols=5, rgb=True)</code> previews the 3D object projection on to 2D view. Is there a generalizable way to extract accurate 2D object boundary mask?</li>
</ol>
<p>2.<code> img1 = images.cpu().numpy()[0]; plt.imshow(img1)</code> yields image of different visual quality(slightly faded and grainy) compared to the plot from <span class="hashtag-raw">#1</span> . why is there a difference?</p>
<p>Thanks!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3" title="Screen Shot 2021-08-17 at 12.31.13 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3_2_441x500.jpeg" alt="Screen Shot 2021-08-17 at 12.31.13 AM" data-base62-sha1="q7gbIyVYktN2i2mZRqlBnNBrVhF" width="441" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3_2_441x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3_2_661x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70aa5bb1d9af99d117445e6dd67bfaca36b74c3_2_882x1000.jpeg 2x" data-dominant-color="F5F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-17 at 12.31.13 AM</span><span class="informations">1122×1272 77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
No rule to make target libtorch.so

username: a_d
cooked: <p>Hello all,<br>
after running cmake CMakeLists.txt I am getting an error after running make saying -</p>
<pre><code class="lang-auto">No rule to make target /home/atharva/torch/torch/lib/libtorch.so needed by Pytorch_c__(this is the project name)
</code></pre>
<p>my cmake is -</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.16)
project(Pytorch_c__)
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_PREFIX_PATH "/home/atharva/torch/libtorch")
find_package(Torch REQUIRED)
add_executable(Pytorch_c__ main.cpp)
target_link_libraries(Pytorch_c__ PUBLIC "${TORCH_LIBRARIES}")
</code></pre>
<p>What could I be doing wrong?  I am fairly new to c++ and this is quite basic but I have been struggling with this for a couple of days now.<br>
Also where is libtorch.so? I have no folder named /torch/torch in my home directory<br>
TIA</p>

username: ptrblck
cooked: <p>Did you already download <code>libtorch</code> form <a href="https://pytorch.org/get-started/locally/">here</a> (or did you build it manually)?<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html">This tutorial</a> would give you a small working example on how to use the C++ API.</p>

username: a_d
cooked: <p>I downloaded the nightly build of libtorch and the code is simply building a printing an Identity matrix.<br>
I followed the tutorial to a T(had to make a small change in cuda.cmake though, changes cudnn.h to cudnn_version.h ,probably that does not matter though)<br>
provided the CMAKE_PREFIX_PATH and everything<br>
I downloaded and extracted it to - /home/atharva/torch/libtorch/(LIbraries here)</p>

username: a_d
cooked: <p>Nevermind I was constantly making a typo which I only realized after close inspection…<br>
I am so sorry…</p>

username: Zedzou
cooked: <p>Have you soved this problem?I am getting this error too.</p>
Tensor slice in pytorch?

username: gaoking132
cooked: <p>Is there any function that behaves similarly to the function of tf.slice in pytorch?</p>

username: jekbradbury
cooked: <p><a href="http://pytorch.org/docs/tensors.html#torch.Tensor.narrow" rel="nofollow noopener"><code>torch.Tensor.narrow</code></a></p>

username: gaoking132
cooked: <p>I just found a similar post regarding using indexing to slice a tensor. So my question is “Is index slicing same as using narrow function?”</p>
<p>Thanks</p>

username: fmassa
cooked: <p>If by index slicing you mean using <code>a[2:5]</code>, then yes, it’s equivalent to applying narrows.</p>

username: gaoking132
cooked: <p>Thanks. This is what I mean.</p>

username: Rohit_Keshari
cooked: <p>I have a question regarding tensor slices. My variables <em>vtf</em> and <em>a</em> have torch.FloatTensor of size 5x25 and torch.LongTensor of size 5x5 respectively. When I do <em>vtf[:,a[0,:]]</em>, it gives me torch.FloatTensor of size 5x5. However if I do <em>vtf[0,a[0,:]]</em>  then I received TypeError: indexing a tensor with an object of type torch.LongTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument.</p>
<p>Please let me know if I am doing something wrong</p>

username: nicozhou
cooked: <p>so, how do we slice the tensor become patches using narrow?</p>
Creating an Inverse Gamma distribution in with torch.distributions

username: ronnyb29
cooked: <p>I’m looking to define an inverse gamma distribution using torch.distributions, similar to putting:</p>
<pre><code class="lang-auto">gamma_dist = torch.distributions.Gamma(alpha, beta)
</code></pre>
<p>I see that there does exist a transforms class from which one can create TransformedDistributions (refer to <a href="https://pytorch.org/docs/stable/_modules/torch/distributions/transforms.html#PowerTransform" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/distributions/transforms.html#PowerTransform</a>), however there is no 1/distribution class. How should one go about creating the Inverse gamma?</p>

username: mwojnowicz
cooked: <p>I am a n00b to pytorch, but I was surprised that the code below fails:</p>
<pre><code class="lang-auto">import torchvision
from torch.distributions.transformed_distribution import TransformedDistribution
from torch.distributions import Gamma

def construct_inverse_gamma(alpha, beta):
	"""
	If X ~ Gamma(alpha, beta), then 1/X ~ InvGamma(alpha, beta)
	"""
	reciprocal = torchvision.transforms.Lambda(lambda x: 1/x)
	InverseGamma = TransformedDistribution(Gamma(alpha, beta), [reciprocal])
	return InverseGamma

InverseGamma=construct_inverse_gamma(1.0, 1.0)
</code></pre>
<p>The error message is:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-67-d61ee5ba5c44&gt; in &lt;module&gt;()
     11         return InverseGamma
     12 
---&gt; 13 InverseGamma=construct_inverse_gamma(1.0, 1.0)

&lt;ipython-input-67-d61ee5ba5c44&gt; in construct_inverse_gamma(alpha, beta)
      8 	"""
      9         reciprocal = torchvision.transforms.Lambda(lambda x: 1/x)
---&gt; 10         InverseGamma = TransformedDistribution(Gamma(alpha, beta), [reciprocal])
     11         return InverseGamma
     12 

/redacted_path/py2-env/lib/python2.7/site-packages/torch/distributions/transformed_distribution.pyc in __init__(self, base_distribution, transforms, validate_args)
     47         elif isinstance(transforms, list):
     48             if not all(isinstance(t, Transform) for t in transforms):
---&gt; 49                 raise ValueError("transforms must be a Transform or a list of Transforms")
     50             self.transforms = transforms
     51         else:

ValueError: transforms must be a Transform or a list of Transforms
</code></pre>

username: deep_s_pandey
cooked: <p>I need to do some experiments with inverse gamma in pytorch. I wonder if there have been any updates on this. any help will be greatly appreciated</p>

username: googlebot
cooked: <p>the person above used torchvision transform. here is ready implementation: <a href="https://github.com/pyro-ppl/pyro/blob/9f67c43deab7f33216b426843292e5c5ce73df78/pyro/distributions/inverse_gamma.py" class="inline-onebox" rel="noopener nofollow ugc">pyro/inverse_gamma.py at 9f67c43deab7f33216b426843292e5c5ce73df78 · pyro-ppl/pyro · GitHub</a></p>
attempting to deserialize object on a cuda device but torch.cuda.is_available() is false

username: amahoro12
cooked: <p>raise RuntimeError('Attempting to deserialize object on a CUDA ’<br>
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device(‘cpu’) to map your storages to the CPU.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc676607c7ce3a76a99170f8423d07297498d016.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fc676607c7ce3a76a99170f8423d07297498d016" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc676607c7ce3a76a99170f8423d07297498d016_2_690x218.jpeg" alt="Capture" data-base62-sha1="A0RQSH35W6PDuEhozGRz8337cnY" width="690" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc676607c7ce3a76a99170f8423d07297498d016_2_690x218.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc676607c7ce3a76a99170f8423d07297498d016_2_1035x327.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc676607c7ce3a76a99170f8423d07297498d016_2_1380x436.jpeg 2x" data-dominant-color="302F32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1796×569 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Manuel_Alejandro_Dia
cooked: <p>Are you sure you have access to the GPU from Pytorch?<br>
Try this short script to check it:</p>
<pre><code class="lang-auto">import torch
gpu_tensor = torch.ones((10)).cuda()
</code></pre>

username: amahoro12
cooked: <p>thank you  a lot for your help. Let me try it.</p>
Netron-Like App for pytorch

username: nicozhou
cooked: <p>Hi, Just want to know, is there any netron-like app for pytorch? Which can visualize .pth file without converting it to .onnx file??</p>
<p>Tensorflow has it and its reallly nice if pytorch have it too <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Can I resolve bottleneck problem when save images?

username: DKH
cooked: <p>When I generate images using generator network like GAN and save it,</p>
<p>it takes too long time and I think that is from IO bottleneck.</p>
<p>How can I solve this problem?</p>

username: ptrblck
cooked: <p>Assuming you’ve profiled the code already and narrowed down that it’s indeed an IO bottleneck, you might want to use a faster storage (SSD etc.) to store the data.</p>
Virtual mini-batch - can this work?

username: wiiiktor
cooked: <p>Can this trivial approach really work?: <a href="https://medium.com/@davidlmorton/increasing-mini-batch-size-without-increasing-memory-6794e10db672" class="inline-onebox" rel="noopener nofollow ugc">Increasing Mini-batch Size without Increasing Memory | by David Morton | Medium</a> I ask, because I try to train a CNN (VGG19) network with batch size of 1 and I am in a desperate need for some mini-batch normalization substitute</p>
<p>(I already implemented WS+GN described in <a href="https://arxiv.org/pdf/1903.10520.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1903.10520.pdf</a> which works quite well with batch size of 2, but I need precisely 1, to be able to make network conditionals; with batch size of even just 2 this batch would have to split in parts, which is obviously not acceptable)</p>

username: ptrblck
cooked: <p>Gradient accumulation can be used to simulate larger batch sizes. However, especially if you are using batchnorm (or potentially other norm layers with running stats) you would have to be careful about the internal running stats, as each forward pass would update them with the (noisy) batch statistics (you might need to play around with the <code>momentum</code>).<br>
Also, if you are using a single sample, the batchnorm layers might not be able to calculate the batch statistics and could yield an error.</p>
Tensor Slicing and One class Classification

username: nicozhou
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7b6c4e860d967034ef570e369f4d5d0aee8649e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b7b6c4e860d967034ef570e369f4d5d0aee8649e" title="IMG_20210817_105632"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7b6c4e860d967034ef570e369f4d5d0aee8649e_2_690x484.jpeg" alt="IMG_20210817_105632" data-base62-sha1="qdcXps9IofYiYaYNRqQUPyGuLrE" width="690" height="484" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7b6c4e860d967034ef570e369f4d5d0aee8649e_2_690x484.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7b6c4e860d967034ef570e369f4d5d0aee8649e_2_1035x726.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7b6c4e860d967034ef570e369f4d5d0aee8649e_2_1380x968.jpeg 2x" data-dominant-color="D7D5D0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20210817_105632</span><span class="informations">1920×1348 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi I wanted to implement this network, i somehow know the concept, but i do not know how to do the tensor slicing(slice feature map to patches) in pytorch.</p>
<p>And also, how do I do one class classification in pytorch? ex : i want to achieve something like what i draw behind where i could classify each patch of vector and for example I sliced 4 patches, if one patch is classified as NG, then the whole images will be considered as NG image.</p>
<p>Thank you.</p>
Logical element-wise multiplication of tensor with another tensor

username: dpetermann
cooked: <p>Suppose I have a tensor <strong>A</strong> of shape <code>(128,)</code> and two other tensors <strong>B</strong> and <strong>C</strong>, both of shape <code>(512,)</code>. I’d like to multiply each element of <strong>A</strong> with <strong>B</strong>, whenever <strong>A</strong> is smaller than 0 and with <strong>C</strong> otherwise. This should result in a tensor of shape <code>(128,512)</code>. Is there an “elegant” way of achieving this without involving for-loops ? As far as I am aware <strong>torch.where</strong> offers something very close to what I’d need but not quite.</p>
<p>Thanks in advance!<br>
Darius</p>
Compare Deep Learning model trained on two different GPUs

username: jS5t3r
cooked: <p>I don’t get the exact same results, but I don’t know that the model is trained once on another GPU.</p>
<p>I am thinking of training the model once on one GPU and then on another GPU and then evaluate and compare.</p>
<p>I think that it is important to use the same initial seed (<a href="https://pytorch.org/docs/stable/generated/torch.initial_seed.html" class="inline-onebox" rel="noopener nofollow ugc">torch.initial_seed — PyTorch 1.9.0 documentation</a>)  to make these GPUs more comparable. Can you give me a suggestion, how to initial it?</p>
<p>It looks like that now:</p>
<pre><code class="lang-auto">torch.manual_seed(10)
</code></pre>
<p>Is there anything else that I can have a better comparison of the results?</p>
RuntimeError: svd_cuda: For batch 0: U(51,51) is zero, singular U

username: Khabbab_Zakaria
cooked: <p>HI,<br>
I want to apply PCA on model outputs. The code is:</p>
<pre><code class="lang-auto">def PCA_svd(x, k, center=True):
    n = x.size()[0]
    ones = torch.ones(n).view([n,1])
    h = ((1/n) * torch.mm(ones, ones.t()))# if center else torch.zeros(n*n).view([n,n])
    H = torch.eye(n) - h
    #H = H.cuda()
    X_center = torch.mm(H.double().cuda(), x.double().cuda())
    u, s, v = torch.svd(X_center)
    components = v[:k].t()
    #explained_variance = torch.mul(s[:k], s[:k])/(n-1)
    return components
</code></pre>
<p>The model output is composed of feature vectors of shape 512. So, we can assume the model output = torch.rand(50,512) where 50 is the batchsize.<br>
Next I am using the function on these model outputs as:</p>
<pre><code class="lang-auto">feature = PCA_svd(model_output,128)
</code></pre>
<p>But, I am getting this error when I run the codes for a dataset of about 5000 images.</p>
<pre><code class="lang-auto">RuntimeError: svd_cuda: For batch 0: U(51,51) is zero, singular U.
</code></pre>
<p>Funny thing is that when I run it for a smaller dataset of lets say 3000 images, I don’t face this error.<br>
I don’t understand it. Is it a bug? Or, is there something wrong in my PCA_svd function.</p>
Custom dataloader for multiple samples in single file

username: CopyOfA
cooked: <p>I am working with multiple chemical spectra for a binary classification problem and I have multiple files with multiple spectra in each file. That is, a single .csv file contains many spectra, with each file containing the same number of spectra. For example, a single .csv file looks like this:</p>
<pre><code class="lang-auto">name             id   V1   V2   V3 ...
compound1        1    25   14   32 ...
compound2        2    21   10   12 ...
compound3        3    15   7    39 ...
</code></pre>
<p>The numerical portions of the file (<code>V1</code> through <code>VN</code>) will be used in the neural network. Each file has compounds that are all a single label, and the numerical portions of all files are the same. I have several thousand of these files. So, I’d like to create a dataloader that will accept a bunch of file paths, and return a dataloader, where each call to <code>next(iter(customDataloader))</code> yields a data tensor and a label tensor such that the data tensor is size <code>(number_of_spectra_in_file, 1, number_of_columns_in_numerical_part)</code>, and the label tensor is size <code>(number_of_spectra_in_file, 1)</code>. That is each spectra should be considered as a single sample, and if I make a data loader with <code>batch_size = 16</code>, then the corresponding data tensor should have size <code>(16*number_of_spectra_in_file, 1, number_of_columns_in_numerical_part)</code>. Here’s my first attempt:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
   def __init__(self, filePathDf, transform=None, target_transform=None):
      self.transform = transform
      self.target_transform = target_transform
      self.filePathDf = filePathDf

   def __len__(self):
      return len(self.filePathDf)

   def __getitem__(self, idx):
      data = torch.tensor(
         (
            pd.read_csv(self.filePathDf.loc[idx, 'path'])
            .drop('id', 'name')
            .to_numpy()
         ),
         dtype=torch.float
      )

      label = [self.filePathDf.loc[idx, 'label']] * data.size()[0]

      if self.transform:
         data = self.transform(data)
      if self.target_transform:
         label = self.target_transform(label)

      return data, label


</code></pre>
<p>When I run this, I get data that is not the right size.</p>
<pre><code class="lang-auto">&gt;&gt; trainDataset = CustomDataset(trainFilesDf)
&gt;&gt; trainDataloader = torch.utils.data.DataLoader(trainDataset, batch_size=10)
&gt;&gt; trainData, trainLabels = next(iter(trainDataloader))
&gt;&gt; print(trainData.size())

torch.Size([10, 7, 2020])
</code></pre>
<p>Here, there are 7 spectra per file, and the numerical portions of the data are 2020 long. The variable <code>trainLabels</code> is a list of 7 tensors, each of size 10. The variable <code>trainFilesDf</code> is a pandas dataframe with 2 columns: <code>path</code> and <code>label</code>, where each file path has a corresponding label.</p>
<p>My question: Should I just reshape the data and labels or is there a better way to formulate my <code>CustomDataset</code> class?</p>
Create filtering layers with binary weights

username: sze_wen
cooked: <p>I can’t see a way of doing it. What is the simplest way to constrain the weights of a 272 layers of 33 x 33 filters and each layer to <code>[0,1]</code> or <code>[-1,1]</code> during training.</p>

username: KFrank
cooked: <p>Hi Sze!</p>
<aside class="quote no-group quote-modified" data-username="sze_wen" data-post="1" data-topic="129459" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sze_wen/48/40875_2.png" class="avatar"> sze_wen:</div>
<blockquote>
<p>What is the simplest way to constrain the weights … during training.</p>
</blockquote>
</aside>
<p>There is really no sensible way to precisely constrain parameters<br>
during the training process.  At issue is that pytorch is based on<br>
various gradient-descent optimization techniques that evolve<br>
parameters “continuously” (of course, actually, with finite, but<br>
presumably small steps).  So there’s no “continuous” way for a<br>
parameter to have its value jump from, say, <code>0</code> to <code>1</code>.</p>
<p>Probably the best approach is to add a penalty to your loss that<br>
pushes the parameter values toward <code>0.0</code> and <code>1.0</code>,  e.g., something<br>
like:</p>
<pre data-code-wrap="python"><code class="lang-python">constraint_penalty = alpha * (weight * (weight - 1.0))**2
</code></pre>
<p>(For <code>alpha &gt; 0.0</code> this penalty is exactly zero only when each element<br>
of <code>weight</code> is either <code>0.0</code> or <code>1.0</code>, and is otherwise greater than zero.<br>
You could also use <code>.abs()</code> rather than <code>**2</code>; you might try both.)</p>
<p>Then start your training with <code>alpha = 0.0</code> so your weights can settle<br>
into the neighborhoods of values that give you good predictions.  Then<br>
keep training while increasing <code>alpha</code>.  When <code>alpha</code> becomes large<br>
enough, <code>constraint_penalty</code> will dominate, and your weights will<br>
become “frozen” at <code>0.0</code> and <code>1.0</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
Training multiple models together in coordinate block descent fashion

username: rudra_saha
cooked: <p>I am trying to train multiple models together in a block descent fashion aka calculating losses using all of the models but backpropagating the aggregated losses on only one of them at a time for updating the parameters. For the same, I have set up two forward passes. The outputs of the first forward pass are detached and only the model under consideration is kept with the computational graph in the second forward pass. Losses are being aggregated and backpropagated via this model.</p>
<p>I have the following questions:</p>
<ol>
<li>Are there more efficient ways of setting this up?</li>
</ol>
Training details and licensing information for pretrained models distributed with Torchvision

username: gspr
cooked: <p>Are the exact training details (training data + hyperparameters) for the various models distributed with Torchvision written down somewhere? In particular, I always assumed that the VGG16 model (<code>vgg16-397923af.pth</code>) that Torchvision can automatically download is just a repackaged version of the pretrained model distributed by the VGG paper authors at <a href="https://www.robots.ox.ac.uk/~vgg/research/very_deep/" rel="noopener nofollow ugc">their website</a>. That seems not to be the case.</p>
<p>Moreover, what are the licensing terms of the pretrained models distributed with Torchvision?</p>
WeightedRandomSampler with different target shape

username: duddal
cooked: <p>Hello,</p>
<p>I am training an Object Detection Network. The dataset contains 2700 images and is highly imbalanced. the following dictionary indicates the class_id and the corresponding number of samples in the training dataset.</p>
<p><code>samples = {'1': 306, '2': 60, '3': 133, '4': 14197, '6': 3886, '7': 69}</code></p>
<p>Therefore, I want to use the WeightedRandomSampler. According to <a href="https://discuss.pytorch.org/t/some-problems-with-weightedrandomsampler/23242/40">this</a> post, I created the weights:</p>
<p><code>weights = torch.from_numpy(1. / np.asarray(list(samples.values())))</code></p>
<p>Then it is mentioned that these weights should be applied to the targets. However, I am not doing classification, but Object Detection using a modified CenterNet architecture. Therefore, for each image in the dataset, my target has a shape: <code>[128, 384, 16]</code>. This target contains where the number 16 comes from <code>num_classes+4+1+4+1</code>. Not all the images have all the classes.</p>
<p>Should I just loop through the dataset and create an array that contains just the <code>class_labels</code> and apply the <code>weights</code> to this array?<br>
In this case, how do I use the WeighedRandomSampler?</p>
Import torch failed with Illegal instruction (core dumped)

username: konstantin-m
cooked: <p>Hi all,<br>
I built PyTorch library from source code. But when I run simple “import torch” it fails with  Illegal instruction (core dumped).</p>
<p>Starting program: /home/mkr/anaconda3/envs/pytorch-build/bin/python res.py<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
[Detaching after fork from child process 73595]</p>
<p>Program received signal SIGILL, Illegal instruction.<br>
0x00007fffe567393d in __static_initialization_and_destruction_0(int, int) [clone .constprop.0] ()<br>
from /home/mkr/anaconda3/envs/pytorch-build/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so</p>
<p>How can I fix it?</p>
<p>Thanks in advance</p>
Are "left-padded" sequences possible?

username: yukw777
cooked: <p>Hi everyone,</p>
<p>I have a very peculiar use case where some sequences in a batch start “later” than others. For example, consider the following batch:</p>
<p>t | 0 | 1 | 2 | 3 | 4 |<br>
0| a | b | c | d | e |<br>
1|==| g  | h | k | m|<br>
2|==|==|==| n | o|</p>
<p>Here, the 0’th sequence starts at time 0, first at time 1 and 2nd at time 3. The corresponding PackedSequence would be something like PackedSequence(data=[a, b, c, d, e, g, h, k, m, n, o], batch_sizes=[1, 2, 2, 3, 3]). Is there a good way to represent this “left-padded” batch as a PackedSequence and pass it to an RNN, e.g. LSTM and GRU? Or is the only way to left pad the batch manually and mask the outputs (which is not ideal b/c the initial hidden states wouldn’t be zeros for sequences that start later)?</p>
<p>I hope I was clear enough. If not, please let me know so I can clarify. Thanks for your help in advance!</p>

username: Wesley_Neill
cooked: <p>Do you have all of the sequences in hand at one time?</p>
<p>If so, I’d think you could do something like this in your collate function for your dataloader:</p>
<pre><code class="lang-auto">sequences.sort(key=lambda x: len(x), reverse=True)
lengths = [len(sequence) for sequence in text_list]

reversed_sequences = sequences[::-1]
padded_reversed = 
    pad_sequence(reversed_sequences, batch_first=True, padding_value=vocab["&lt;pad&gt;"])

return padded_reversed[::-1]
</code></pre>
<p>Basically reverse, pad, reverse…</p>
<p>Then you’d have to mask your network outputs appropriately when measuring loss.</p>
<p>I didn’t test the above, but I don’t see why it would not work assuming the rest of your model/training/loss was set up right.</p>

username: yukw777
cooked: <p>Thanks for the reply Wesley! Your code basically left pads the batch right? It’d work, except the shorter sequences that start later would not receive zeros as their initial hidden states (b/c the RNN would’ve already processed the padding), which PackedSequence would solve. I guess this doesn’t really matter in the end?</p>

username: Wesley_Neill
cooked: <p>Yeah, that should left-pad the batch. The shorter sequences should receive zeros.</p>
<p>As far as what would happen with the hidden state, I’m having a hard time visualizing it without actually playing with it. If I were you, I’d probably just try to run a single batch through a test RNN layer and see what comes out the other side.</p>