akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I access the values of conv2d before summation?

username: Yuv
cooked: <p>Hey,</p>
<p>I have an image with the size (1,32,32) and I’m using Conv2D with kernel size of 5x5 and stride 1, which means it gives (28,28) after the convolution.</p>
<p>I want to get the values of multiplication between the kernel weights(5x5) and the inputs(5x5 in every step of all  28x28 steps) before the method sums up all the 25 elements together.</p>
<p>I thought about unfolding the source image and using a conv2d with stride 5 - but again it sums all the 5x5 elements.</p>
<p>Thanks</p>

username: Dipam_Vasani
cooked: <p>I don’t think that’s possible, you might have to write your own mat mul function for this</p>

username: ptrblck
cooked: <p>Unfolding should work and you could use an elementwise multiplication which wouldn’t apply the reduction.</p>

username: Yuv
cooked: <p>Thank you, I did as you suggested.<br>
I have a follow up question  - most of the time I try to stick to the torch.nn.\torch.method because I know it will calculate the gradients right. Here I tried to do something a little bit unconventional like using unfold on input and then use different activation function on different patches of the unfolded image.<br>
Or for example using unfold on the hidden layer.<br>
Is there a way to make sure it calculates the gradients right?<br>
I’m familiar with .requires_grad and autograd but I’m not sure if it calculates the gradients as I would thought.<br>
Is there a guide that can help me get intuition about how the gradients are calculated in Pytorch?<br>
Do I need to look explicitly at the gradients and compare it to analytical results?</p>
<p>Thanks a lot</p>

username: ptrblck
cooked: <p><code>unfold</code> is differentiable so won’t detach the tensor from the computation graph.<br>
To make sure that’s indeed the case, you could check the <code>.grad_fn</code> of the output tensor of any operation and see if it points to a valid function (returning <code>None</code> would mean the tensor is not attached to a computation graph).<br>
To check the correctness of the gradient calculation you could use <code>gradcheck</code>.</p>

username: Yuv
cooked: <p>Thank you very much for all your help!</p>
Why values become very large after dist.all_reduce

username: Jensen-Su
cooked: <pre><code class="lang-python">&gt;&gt;&gt; print(f'{rank=}, before reduce, {loss=}')
rank=0 before reduce, loss=0.004893303848803043
rank=1 before reduce, loss=0.008418125100433826
rank=5 before reduce, loss=0.022900601848959923
rank=4 before reduce, loss=0.033665977805630645
rank=6 before reduce, loss=0.05732813761557371
rank=7 before reduce, loss=0.006465559359639883
rank=2 before reduce, loss=0.01541353389620781
rank=3 before reduce, loss=0.035168059170246124

&gt;&gt;&gt; dist.all_reduce(loss.div_(dist.get_world_size()))
&gt;&gt;&gt; print(f'{rank=}, after reduce, {loss=},  {world_size=}')
rank=0 after reduce, loss=-8.541948720476135e+27, world_size=8
rank=4 after reduce, loss=0.011374264427650382, world_size=8
rank=5 after reduce, loss=-8.541948720476135e+27, world_size=8
rank=1 after reduce, loss=-8.541948720476135e+27, world_size=8
rank=6 after reduce, loss=0.011374264427650382, world_size=8
rank=2 after reduce, loss=-8.541948720476135e+27, world_size=8
rank=7 after reduce, loss=-8.541948720476135e+27, world_size=8
rank=3 after reduce, loss=-8.541948720476135e+27, world_size=8
</code></pre>

username: eqy
cooked: <p>That’s strange, the tensors should be identical after <code>all_reduce</code>, could you please post some more details about your setup (e.g., how you are running <code>torch.distributed</code> such as the backend being used)?</p>
Pytorh 1.9 with cuda 11.0

username: talhaanwarch
cooked: <p>Can i install pytorch 1.9 or 1.10 with cuda 11.0? Will it work or give me errors?<br>
Currently i have cuda 11.0 install with tensorflow 2.4 and pytorch 1.7.1.<br>
I could not find any other cuda version which is compatible for both TF and Pytorch.</p>
<p>So if i install pytorch 1.9? will it cause any issue?</p>
<p>I am using ubuntu and dont want to install conda</p>

username: ptrblck
cooked: <p>Yes, you can build PyTorch 1.9 and 1.10 with CUDA 11.0.<br>
The current binaries ship with CUDA 10.2, 11.3, and 11.5, so you would need to build from source if you need this particular CUDA version.</p>
<aside class="quote no-group" data-username="talhaanwarch" data-post="1" data-topic="142955">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/talhaanwarch/48/26568_2.png" class="avatar"> talhaanwarch:</div>
<blockquote>
<p>So if i install pytorch 1.9? will it cause any issue?</p>
</blockquote>
</aside>
<p>PyTorch should run, but I haven’t checked the compatibility with other frameworks and same/different CUDA versions.</p>
Repeat tensor using expand

username: Lame
cooked: <p>Hi,</p>
<p>I want to expand a tensor <code>tensorA</code> with size: <code>torch.Size([17185, 17185])</code> to be <code>torch.Size([34370, 34370])</code> using <code>torch.expand()</code> rather than <code>torch.repeat()</code></p>
<p>I know how to do it with <code>torch.repeat()</code> by <code>tensorA.expand(2,2)</code> which will give me a tensor of size <code>torch.Size([34370, 34370])</code> but take much memory.</p>
<p>Any help with doing it using <code>torch.expand()</code> please?</p>

username: ptrblck
cooked: <p>This won’t be possible, as <code>expand</code> works on singleton dimensions. From the docs:</p>
<blockquote>
<p>Returns a new view of the :attr:<code>self</code> tensor with singleton dimensions expanded to a larger size. […] Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory.</p>
</blockquote>
Fast pytorch processing of Tensorflow tfrecords data

username: mishooax
cooked: <p>Hi all,</p>
<p>I want to devise an efficient way to load in data from a set of (relatively large) tfrecords files then pass said data on to my pytorch model for training and inference. We’re talking about ca. 64 GB of data in total right now - but the idea is to scale this to much larger datasets in the future. The tfrecords have been generated using the <code>tfds</code> API - one sample consists of 3 tensors packed into a dict: “X_hr”, “X_lr” (high-res and low-res inputs) and the target “Y” (this is a super-resolution problem). I can’t duplicate the data - i.e. reading the tfrecords in tensorflow then saving said data into a more pytorch-friendly format is not an option.</p>
<p>Here is how the data look like:</p>
<p><a href="https://gist.github.com/mishooax/7e12e473014f866d0d75b6e9d2a7ff9a" rel="noopener nofollow ugc">https://gist.github.com/mishooax/7e12e473014f866d0d75b6e9d2a7ff9a</a></p>
<p>What would be the best way to go about this?</p>
<p>1/ Write a custom torch.Dataset that wraps around a TFRecordsDataset object? The TFRecordsDataset would be in charge of decoding / unzipping the binary data and producing samples as needed.</p>
<p>2/ Write a custom DataLoader that accepts a TFRecordsDataset object, something similar to <a href="https://www.kaggle.com/hidehisaarai1213/g2net-read-from-tfrecord-train-with-pytorch" class="inline-onebox" rel="noopener nofollow ugc">G2Net: Read from TFRecord &amp; Train with PyTorch | Kaggle</a>? I’m not sure if this would work well with multiprocessing.</p>
<p>3/ Hook up the TFRecordsDataset directly to my Pytorch model (how?)</p>
<p>4/ Forget about pytorch and do end-to-end TF for this task?! (not a real option! :))</p>
<p>FWIW I am aware of <a href="https://github.com/vahidk/tfrecord" class="inline-onebox" rel="noopener nofollow ugc">GitHub - vahidk/tfrecord: TFRecord reader for PyTorch</a> but I can’t get it to work with my ZLIB-compressed tfrecords features <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Thanks!<br>
~M</p>
Discrepancy in extension tutorial and code: PackedTensorAccessor32 or PackedTensorAccessor?

username: LearnedLately
cooked: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">c++ extension tutorial</a> uses <code>torch::PackedTensorAccessor32</code> and states</p>
<blockquote>
<p>This is important as using the 64-bit variant (<code>PackedAccessor64</code> ) can make the kernel slower</p>
</blockquote>
<p>But the <a href="https://github.com/pytorch/extension-cpp/blob/master/cuda/lltm_cuda_kernel.cu" rel="noopener nofollow ugc">github code it links to</a> uses <code>torch::PackedTensorAccessor</code>. The code is supposed to be the same in both. Which one is correct?</p>
<p><strong>Update</strong>: based on compile warnings it looks like the tutorial is correct and the github code is out of date.</p>
Different predicted output size with pretrained model on caffe and torchvision model

username: emross3371
cooked: <p>I am working on converting a legacy torch(lua based) project to pytorch. The authors of the legacy code have used a vgg16 pretrained model (on caffe framework) for their experiments. I used the torchvision pretrained model. I wanted to pass a 28x28x3 image through said models feature extractor but I am getting different output tensor sizes when passing the image through these. The caffe model returns a 512x2x2 (CHW) tensor whereas torchvision one returns a 512x1x1. I printed the model submodules and they exactly match each other. What should I do?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/emross3371">@emross3371</a> looks like you are missing some operation pre or post VGG. If you provide the source code then we can check</p>

username: emross3371
cooked: <p><a href="https://github.com/kreimanlab/VisualSearchZeroShot/blob/master/IVSNtopdown_30_31_array.lua" class="inline-onebox" rel="noopener nofollow ugc">VisualSearchZeroShot/IVSNtopdown_30_31_array.lua at master · kreimanlab/VisualSearchZeroShot · GitHub</a> this is the code I’m trying to modify. This is the output of the pytorch sub model <a href="https://imgur.com/NJh5iC7" rel="noopener nofollow ugc">https://imgur.com/NJh5iC7</a>, and this is the output of the lua code( will post this as a reply in the thread). Funny thing is when i pass the standard 224x224x3 input the pytorch and caffe model work fine, it only outputs different sizes for the smaller sized image of 28x28x3</p>

username: emross3371
cooked: <p><a href="https://imgur.com/9MWr4sl" rel="noopener nofollow ugc">https://imgur.com/9MWr4sl</a>  - model submodules in caffe model</p>

username: emross3371
cooked: <p>As far as I can see, only simple mean subtraction preprocessing is done, pls correct me if im wrong (I am new to legacy torch)</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/emross3371">@emross3371</a> This can be explained. If you have very small images then during the later layers, you dont have enough pixels left to do complete convolution and therefore shape is reduced to 1. Every image processing model should accept a particular shape of the image. If 224 is working fine, then resize the 28x28x3 to 224x224x3 using torch Resize</p>
<p><a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize</a></p>
Jointly training of weights+input embeddings in a neural-network based model

username: Federico_Ottomano
cooked: <p>I’m wondering if I can do the followoing thing:</p>
<p>Basically I want to build a deep learning model which takes a pre-computed set of features which already encodes some knowledge on the target, and concatenates it along a learnable vector of parameters and train a neural network-based architecture which then jointly trains these embeddings concatenated to the fixed feature vector and returns a prediction.</p>
<p>I just have a perplexity: should I pretrain the embeddings <em>before</em> and using them after by concatenating with features that I already have, or can I train the embeddings <em>jointly</em> with the overall model?</p>
<p>To summarize, ny idea is the following: I want to use both the information provided by tabular features that I already have, plus I want to learn the best representations directly from data.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/federico_ottomano">@Federico_Ottomano</a></p>
<ul>
<li>
<p>If you train the parameters jointly along with the overall model, your embeddings will reflect the patterns that help distinguish the input based on the target</p>
</li>
<li>
<p>If you do it separately ( i am guessing autoencoder), then the embeddings will reflect the input features only</p>
</li>
</ul>
<p>You can try a third option<br>
Have a autoencoder setup for the input data and use the encoder output as an input where you concatenate it along with the other features and have the prediction output. So your model will jointly train the autoencoder and the classifier</p>

username: Federico_Ottomano
cooked: <p>Many thanks for your suggestion! But what should I consider as <em>input data</em> for the autoencoder? Should I just choose some random representation ? What about the final loss also? Will it be a sum of the reconstruction loss + target loss?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/federico_ottomano">@Federico_Ottomano</a> If you dont have any prior data, then you should use the first option. Third option is only when you have additional features which you cannot use in the model</p>

username: Federico_Ottomano
cooked: <p>Ok, but I’m not quite figuring out how can I ‘train’ without having some fixed representation to feed the neural network… I mean in such a case, should I use <code>nn.Embedding(num_samples, size)</code> . Could you provide a little example of code please?</p>
<p>Also, I guess that the problem in using the first option, is that we won’t allow any ‘inductive’ ability on the model, as the predictions will be constrained to the specific trainable embeddings we have provided at the beginning, i.e. if I provide a new input, we won’t know which features to assign it. Is that correct?</p>
Facilitate the customized convolution

username: ChiTseHuang
cooked: <pre><code class="lang-python">group = 4
# input = (batch_size, in_channel, width, height)
# weight = (out_channel, in_channel, kernel_size, kernel_size)
input_unfold = torch.nn.functional.unfold(input, (weight.size(2), weight.size(3)), padding=1, stride=1)
input_unfold_split = torch.split(input_unfold, group*weight.size(2)*weight.size(3), 1)
input_group = torch.stack(input_unfold_split).transpose(2, 3)
weight_view = weight.view(weight.size(0),-1).t()
weight_view_split = torch.split(weight_view, group*weight.size(2)*weight.size(3), 0)
weight_view_stack = torch.stack(weight_view_split)
weight_group = torch.unsqueeze(weight_view_stack, 1) 
output = input_group.matmul(weight_group).transpose(2,3)
# output = (in_channel / group, batch_size, out_channel, width*height)
</code></pre>
<p>I want to do the convolution with four-channel(group) summation, so the output will bigger than normal conv2d (in_channel / group) times.<br>
Anyone can facilitate or optimize my code with the same functionality?<br>
I cannot improve it for a long time, but I need to facilitate it.<br>
Also, the output can be reshaped to   (in_channel / group, batch_size, out_channel, width, height), this result is also can be accepted.<br>
Thanks.</p>
Applying attention on Image

username: juhyung
cooked: <p>I’m trying to apply attention on image.<br>
Is there any good references about that?</p>
<p>I’m confusing about attention mechanism, why they work, what’s difference attention on text and image…</p>

username: s_karki
cooked: <p>This will give you overview of attention in computer vision in general.<br>
<a href="https://iopscience.iop.org/article/10.1088/1742-6596/1693/1/012173" rel="noopener nofollow ugc">Attention in Computer vision.</a></p>
Improve LSTM Training Speed

username: blueeagle
cooked: <p>Hi everyone,</p>
<p>I have implemented a simple Many-to-One LSTM Encoder-Classifier.</p>
<p>The model takes a packed sequence as input (as my input data has variable length) and outputs the probabilities for the target classes. The input sequences are rather long (about 3000 data points).</p>
<p>I am running the training on a 16“ MacBook Pro (6-Core Into Core i7, AMD Radeon Pro 5300M 4 GB) but unfortunately it seems that the training is extremly slow (up to 45 minutes per epoch).</p>
<p>As I have never worked with recurrent neural networks (or LSTMs) before, it is hard for me to determine if this is a usual duration for the back propagation through time or if there is something wrong with my implementation.</p>
<p>Can anybody please help me?</p>
<p>Here is my model:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, input_features, output_features, n_classes):
        super(MyModel, self).__init__()
            self.encoder = nn.LSTM(input_features, output_features, 1, batch_first=True, dropout=0, bidirectional=False)
            self.cla = nn.Linear(output_features, n_classes)

    def forward(self, x: torch.nn.utils.rnn.PackedSequence) -&gt; Tuple:
        z, _ = self.encoder(x)
        z_unpacked, lens_unpacked = pad_packed_sequence(z, batch_first=True)
        last_elements = z_unpacked[torch.arange(z_unpacked.shape[0]), lens_unpacked - 1]
        y = self.cla(last_elements)
        y= F.softmax(y, dim=1)
        return y
</code></pre>

username: Carlos_Segura
cooked: <ul>
<li>
<p>What do you mean by “<em>3000 data points</em>”? is this the sequence length for a single input? Maximum sequence length?</p>
</li>
<li>
<p>About your setup, which device is available in your PyTorch installation? e.g., CPU or GPU.</p>
</li>
</ul>
<p>I have not tried your model, but it looks simple and shouldn’t be the bottleneck. However, If your input description  “… sequences are rather long (about 3000 data points).” implies “sequence-length.” Using RNN’s with sequences this long and the limited compute of your MacBook is the most likely reason it is slow.</p>
<p>Perhaps your problem here is how you’re processing your inputs, e.g., pre-processing, padding, batching. I suggest optimizing your data pipeline to fit the computing capabilities of your MacBook.</p>

username: blueeagle
cooked: <p>First of all thanks for your answer.</p>
<ol>
<li>Yes the sequence length is about 3000 for all input samples</li>
<li>I cannot run Pytorch on the GPU, so the device available is a 2,6 GHz 6-Core Intel Core i7</li>
</ol>
<p>As I never worked with packed sequences in PyTorch, I was not sure if the packing and unpacking of the sequences is the bottleneck of the training. However, as I now tested my pipeline on another dataset with much shorter sequence length (and it worked much faster), I assume that my implementation is alright.</p>
<p>Thanks for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Embedding(): argument 'indices' (position 2) must be Tensor, not builtin_function_or_method

username: Hasan_Khan
cooked: <p>Hello, can anyone explain me this error and help me to resolve it. I am trying to get the predictions for my RNN trained model. However, while generating the predictions I’m having this error.</p>
<blockquote>
<p>embedding(): argument ‘indices’ (position 2) must be Tensor, not builtin_function_or_method</p>
</blockquote>
<p>Here is the code and error snippet.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81cfceaf374a5d137fcdab85380d4a6b88309c44.png" data-download-href="https://discuss.pytorch.org/uploads/default/81cfceaf374a5d137fcdab85380d4a6b88309c44" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81cfceaf374a5d137fcdab85380d4a6b88309c44.png" alt="image" data-base62-sha1="iwmOLS9FyOHDlxJuoorujRASwJK" width="690" height="259" data-dominant-color="393A3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1406×528 34.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Variable named as <code>network</code> is the name of my training model.</p>

username: ptrblck
cooked: <p>Based on the error message the second argument (<code>prediction_loader</code>) is passed as the input to the model, which is a <code>DataLoader</code> instance and will thus fail.<br>
Pass tensors to the model by iterating the <code>DataLoader</code> instead.</p>

username: Hasan_Khan
cooked: <p>It got resolved! Thanks for the help.</p>
Out of memory when using transforms

username: Datai
cooked: <p>Hello,</p>
<p>I’m trying to do some data augmentation using transforms with torchaudio during training step.<br>
After like 10 steps I get an “Out of memory” error.</p>
<p>I’ve tried torch.no_grad() but still the same. How to do transformations without getting this error ?<br>
I’m kinda new so I’m sure that I’m missing something.</p>
<p>Thank you for help !</p>
Handling Undefined arguments

username: KanZa
cooked: <p>I only put the following code <a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/main.py" class="inline-onebox" rel="noopener nofollow ugc">Sequential-GCN-for-Active-Learning/main.py at master · razvancaramalau/Sequential-GCN-for-Active-Learning · GitHub</a><br>
in a class but the criterion argument in test() is giving me error <strong>NameError:</strong> name ‘criterion’ is not defined. Any solution for resolving this particular argument?<br>
whereas ‘criterion’ is defined here <a href="https://github.com/razvancaramalau/Sequential-GCN-for-Active-Learning/blob/master/train_test.py" class="inline-onebox" rel="noopener nofollow ugc">Sequential-GCN-for-Active-Learning/train_test.py at master · razvancaramalau/Sequential-GCN-for-Active-Learning · GitHub</a></p>
<pre><code class="lang-auto">from train_test import train, test

class Objective:
    def __init__(self):
    self.args = parser.parse_args()
    
    def __call__(self, optuna_trial):
    train(models, method, criterion, optimizers, schedulers, dataloaders, args.no_of_epochs, EPOCHL)

    results.close()
    return acc

if __name__ == '__main__':
</code></pre>

username: ptrblck
cooked: <p>In the linked code the methods accept arguments such as:</p>
<pre><code class="lang-python">def train_epoch(models, method, criterion, optimizers, dataloaders, epoch, epoch_loss):
    ...
</code></pre>
<p>and call these arguments inside the function:</p>
<pre><code class="lang-python">    ...
    target_loss = criterion(scores, labels)
</code></pre>
<p>while you are passing <code>optuna_trial</code> as the only argument (besides <code>self</code>) to <code>__call__</code>.<br>
If <code>train</code> or any arguments are not globally defined, the error will be raised.<br>
<a href="https://www.w3schools.com/python/gloss_python_function_arguments.asp">This tutorial</a> explains function arguments in Python and might be a good starter.</p>
What is a good cca, cka library for pytorch that works (ideally with GPU)?

username: Brando_Miranda
cooked: <p>I found:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/moskomule/anatome" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/11806234?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/moskomule/anatome" target="_blank" rel="noopener nofollow ugc">moskomule/anatome</a></h3>

<p>Ἀνατομή is a PyTorch library to analyze representation of neural networks - moskomule/anatome</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>but it doesn’t seem to work reliably (or is it documented).</p>
<p>Is there a standard library pytorch peeps use for cca, cka, neural net comparison analysis that actually works? (ideally on gpu)</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure if it has all you need but <a href="https://captum.ai/">https://captum.ai/</a> seems to be similar?</p>

username: Brando_Miranda
cooked: <p>that looks promising! Do they have CCA and CKA? I couldn’t find them in their library <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<hr>
<p>reddit crosspost: <a href="https://www.reddit.com/r/pytorch/comments/ksocf8/what_is_a_good_cca_cka_library_for_pytorch_that/" rel="noopener nofollow ugc">https://www.reddit.com/r/pytorch/comments/ksocf8/what_is_a_good_cca_cka_library_for_pytorch_that/</a></p>
<hr>
<p>link to captum github page: <a href="https://github.com/pytorch/captum" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/captum: Model interpretability and understanding for PyTorch</a></p>

username: aningineer
cooked: <p>Perhaps you could try this one? <a href="https://github.com/AntixK/PyTorch-Model-Compare.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - AntixK/PyTorch-Model-Compare: Compare neural networks by their feature similarity</a></p>
<p>It seems to work reasonably well, especially for the same model measured against itself which generates a nice diagonal line. However, for two different models, e.g. ResNet18 vs ResNet101, I haven’t had much luck in uncovering any distinct similarity patterns using CKA. I’m not sure if there is a bug in the CKA code or if the two networks just learn very differently.</p>
Pytorch3d install options

username: Griffin_Tabor
cooked: <p>Looking at using pytorch3d in software package I develop. Currently I use conda to install all the dependencies so it runs perfectly in Windows, Mac and Linux. Currently I depend on pytorch  and make sure to only update the version when all 3 platforms have new releases.</p>
<p>Pytorch conda support is great, <a href="https://anaconda.org/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">Pytorch :: Anaconda.org</a> , all platforms you could want binaries for are available with conda.</p>
<p>Pytorch3d which builds on-top of pytorch however is very different.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db48dd3249b767e842a5d0d07879135fc8164258.png" data-download-href="https://discuss.pytorch.org/uploads/default/db48dd3249b767e842a5d0d07879135fc8164258" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db48dd3249b767e842a5d0d07879135fc8164258_2_690x339.png" alt="image" data-base62-sha1="vhSFWACV1sAWiAUsfpRgQqcdxbi" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db48dd3249b767e842a5d0d07879135fc8164258_2_690x339.png, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db48dd3249b767e842a5d0d07879135fc8164258.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db48dd3249b767e842a5d0d07879135fc8164258.png 2x" data-dominant-color="F3F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">996×490 35.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Pytorch3D conda only supports linux, pip only supports mac. Neither supports both and neither supports windows.</p>
<p>Why doesn’t pytorch3D use the same building infrastructure and build for all environments like pytorch itself does and release the binaries through conda for them all?</p>

username: bottler
cooked: <p>If you have issues about the PyTorch3D project it will get more attention if you ask them on the pytorch3d github itself.</p>
<p>Basically, the installation story for PyTorch3D is different between operating systems. (1) The team have not invested in the Windows build and built packages are not availble. Help from outside the team on doing windows builds might be welcome, but things are a bit complicated. (2) For Linux, the build is most developed (and most users are on Linux) and a multitude of packages for different PyTorch, Python and CUDA versions is provided via conda. (3) For mac, CUDA is unavailable and so we don’t think PyTorch3D is used much. We provide a very limited range of packages on PyPI. Not having so many versions means that conda is less essential.</p>
Problem Resnet in pytorch

username: randino
cooked: <pre><code class="lang-auto">class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = nn .Conv2d(1, 10, 1)
        self.bn1 = nn.BatchNorm2d(10)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(10, 10, 1)
        self.bn2 = nn.BatchNorm2d(10)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out
class ResNet34(nn.Module):
    def __init__(self):
        super(ResNet34,self).__init__()
        
        self.block1 = nn.Sequential(
            nn.Conv2d(1,10,kernel_size=2,stride=2,padding=3,bias=False),
            nn.BatchNorm2d(10),
            nn.ReLU(True)
        )
        
        self.block2 = nn.Sequential(
            nn.MaxPool2d(1,1),
            ResidualBlock(64,64),
            ResidualBlock(64,64,2)
        )
        
        self.block3 = nn.Sequential(
            ResidualBlock(64,128),
            ResidualBlock(128,128,2)
        )
        
        self.block4 = nn.Sequential(
            ResidualBlock(128,256),
            ResidualBlock(256,256,2)
        )
        self.block5 = nn.Sequential(
            ResidualBlock(256,512),
            ResidualBlock(512,512,2)
        )
        
        self.avgpool = nn.AvgPool2d(2)
        # vowel_diacritic
        self.fc1 = nn.Linear(512,11)
        # grapheme_root
        self.fc2 = nn.Linear(512,168)
        # consonant_diacritic
        self.fc3 = nn.Linear(512,7)
        
    def forward(self,x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        x = self.block4(x)
        x = self.block5(x)
        x = self.avgpool(x)
        x = x.view(x.size(0),-1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x    

</code></pre>
<p>problem : RuntimeError: Given groups=1, weight of size [10, 1, 1, 1], expected input[26891, 10, 5, 5] to have 1 channels, but got 10 channels instead</p>

username: ptrblck
cooked: <p>In your custom <code>ResidualBlock</code> implementation you are passing <code>in_channels</code> as well as <code>out_channels</code> as arguments, but are never using them and are hard-coding the channels instead as <code>1</code> input channel and <code>10</code> output channels.<br>
Based on this, I would guess that <code>self.block2</code> will raise the shape mismatch as it would expect to get an input with <code>1</code> input channel while the incoming activation (coming from <code>self.block1</code>) would have <code>10</code> channels.<br>
Also, you are trying to set the <code>in_/out_channels</code> in <code>block2</code> to <code>64, 64</code>, which will also not work (as I previously described these values are not used currently but if they were they would also raise a shape mismatch).</p>
Object detection: TypeError: __call__() takes 2 positional arguments but 3 were given

username: Shantanu_Ghosh
cooked: <p>Hi i am creating an object detector, for that I am using the following code for creating the custom dataset:</p>
<p>class MIMICCXRDataset(Dataset):</p>
<pre><code>def __init__(self,
             split,
             args,
             radgraph_sids,
             radgraph_adj_mtx,
             mode,
             transform=None):

    self.args = args
    self.mode = mode
    self.transform = transform
 ..........

def __getitem__(self, idx):
    # 1. get the sample key, i.e., DICOM ID
    dicom_id = self.dicom_ids[idx]
    study_id = list(self.df_master_sel.loc[self.df_master_sel['dicom_id'] == dicom_id, 'study_id'])[0]
    idx_sid = list(self.arr_radgraph_sids).index(str(study_id))

    # 2. select and load image
    img_pth_old = self.df_master_sel.iloc[idx, 4]
    img_pth_components = img_pth_old.split("/")
    img_pth = os.path.join(
        self.args.image_path_ocean_shared,
        *img_pth_components[11:]
    )

    image = Image.open(img_pth).convert('RGB')
    np_res_bbox = np.array(res_bbox)

    if len(np_res_bbox.shape) == 1:
        np_res_bbox = np.reshape(np_res_bbox, (1, -1))

    for i in range(np_res_bbox.shape[0]):
        x1 = np_res_bbox[i, 0]
        y1 = np_res_bbox[i, 1]
        x2 = np_res_bbox[i, 2]
        y2 = np_res_bbox[i, 3]
        boxes.append([x1, y1, x2, y2])

    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    if disease == "pneumonia":
        labels = torch.ones((boxes.size(0),), dtype=torch.int64)
    elif disease == "pneumothorax":
        labels = torch.ones((boxes.size(0),), dtype=torch.int64)

    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    iscrowd = torch.zeros((boxes.size(0),), dtype=torch.int64)
    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["image_id"] = torch.tensor(idx)
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transform is not None:
        image, target = self.transform(image, target)
</code></pre>
<p>I am getting the error in the transforms. My transforms are as follows:<br>
train_dataset = MIMICCXRDataset(<br>
split=split,<br>
args=args,<br>
radgraph_sids=arr_radgraph_sids,<br>
radgraph_adj_mtx=arr_radgraph_adj,<br>
mode=‘bbox’,<br>
transform=transforms.Compose([<br>
transforms.Resize(args.resize),<br>
# resize smaller edge to args.resize and the<br>
# aspect ratio the same for the longer edge<br>
transforms.CenterCrop(args.resize),<br>
transforms.ToTensor(),  # convert pixel value to [0, 1]<br>
normalize<br>
])<br>
)</p>

username: ptrblck
cooked: <p>The used transformations accept a single input only while you are passing the <code>image</code> and <code>target</code> to them.<br>
To allow two inputs, you could refer to the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#putting-everything-together">Object Detection tutorial</a> and use the util. functions from <a href="https://github.com/pytorch/vision/blob/7d868aa608b94d022c42357ede40bda06af942f4/references/detection/transforms.py"><code>references/detection/transforms.py</code></a>.</p>
Contribution to the documentation

username: rnsishan
cooked: <p>I am a beginner to open source contributions and I wanted to contribute to pytorch like in the website a little tweaking to the documentation(minor changes).<br>
I tried looking for the same (the text in the website) in github of pytorch but couldn’t find the file for it.<br>
I’d like to fork it for changes.<br>
is there anyone who can help me with it?</p>

username: ptrblck
cooked: <p>As a starting point I would recommend to check the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contributing Guide</a> as well as <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">this doc</a>.<br>
Once you are familiar with the build process etc. you could then create an issue on GitHub and discuss the proposed changes.<br>
For your current request: which file/docs would you like to change?</p>
External Custom Loss function

username: Ilyes_hm
cooked: <p>Hello,</p>
<p>I want to take the output of the forward pass and detach it to generate a matrix (that is similar to the input) using an external function (not a pytorch function), then create a loss function based on this generated matrix and the input matrix.</p>
<p>does this cause a problem for the gradient?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Yes, using other libraries or explicitly detaching the tensors won’t allow Autograd to track these operations and will thus detach the tensor from the computation graph (parameters used in previous operations also won’t gradients).<br>
To use other libraries you could write a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a>.</p>

username: Ilyes_hm
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: Ilyes_hm
cooked: <p>Hi again <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I am trying to use some opencv functions like the following:</p>
<pre><code class="lang-auto">class opencv_torch(Function):
    @staticmethod
    def forward(ctx, im1, im2, **params,):
        numpy_im1 = im1.to('cpu').detach().numpy()
        result = cv2.FUNCTION(numpy_im1 , **params)
        return torch.FloatTensor(result)
    @staticmethod
    def backward(ctx, grad_output):
        return None, None
</code></pre>
<p>I want this Function to be parameter-less without any learnable parameters so it gives only the output, does this break the gradient flow?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Since you are returning <code>None</code> gradients the previous layers won’t get any valid gradients so this implementation could be seen as detaching the computation graph.</p>
Pytorch with cuda 11.2?

username: Prachi_Nagpal
cooked: <p>Is build with 11.2 cuda available? If so, please share the link.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>No, the binaries are currently built with CUDA 10.2, 11.3, and 11.5.<br>
If you want to use CUDA 11.2 you could build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>
Torchvision fasterrcnn and SSD mobilenet resizing question

username: SarthakJain
cooked: <p>Hi,</p>
<p>Quick conceptual question that I need to understand.<br>
I noticed that when training the <code>torchvision.models.detection.ssdlite320_mobilenet_v3_large()</code> and the <code>torchvision.models.detection.fasterrcnn_mobilenet_v3_large_320_fpn()</code> model, you don’t need to explicitly resize images into a specific value for the model to train. In my dataset, I have images of many different arbritary size that I did not resize and the model trained well. Therefore, I was wondering what resizing steps do the models do internally if they do any.<br>
I noticed in the FasterRCNN that there is this block when loading the model:<br>
<code>Resize(min_size=(320,), max_size=640, mode='bilinear')</code> and the torchvision documentation says the ssdlite takes images that 320 in inference. This is my progress solving the question, but I am stuck as Im not sure what speicifc size my model is resizing images in my dataset to be before they enter the object detection network.</p>
<p>Hopefully you can help with this,<br>
Thanks,<br>
Sarthak Jain</p>

username: Dipam_Vasani
cooked: <p>You can always step into the code on Github and check for yourself. For example, the first component in Fasterrcnn is a <a href="https://github.com/pytorch/vision/blob/7d868aa608b94d022c42357ede40bda06af942f4/torchvision/models/detection/faster_rcnn.py#L256" rel="noopener nofollow ugc">transform object</a> which transforms the images and targets. If you step into that, you will see the <a href="https://github.com/pytorch/vision/blob/7d868aa608b94d022c42357ede40bda06af942f4/torchvision/models/detection/transform.py#L163" rel="noopener nofollow ugc">resize logic</a></p>
Maximum image size Faster RCNN

username: loicdtx
cooked: <p>Hi,</p>
<p>I am wondering what is the maximum image size I can use with a pretrained torchvision <code>fasterrcnn_resnet50_fpn</code>. At the moment I am working with images of shape (3, 1300, 1300). I can’t afford the image to be downsampled as I would lose critical resolution, but if needed I can generate smaller images (they are artificial data). I found the <a href="https://github.com/pytorch/vision/blob/0156d58ec867590b1c78fe1bc834c7da9afdf46a/torchvision/models/detection/faster_rcnn.py#L145" rel="nofollow noopener">max_size</a> argument in the <code>FasterRCNN</code> function definition with default value set to 1333; and this argument is probably passed to <a href="https://github.com/pytorch/vision/blob/0156d58ec867590b1c78fe1bc834c7da9afdf46a/torchvision/models/detection/transform.py#L15" rel="nofollow noopener">GeneralizedRCNNTransform</a> at some point. So I’m assuming there is no downsampling taking place up to that value (1333), but I would like to be sure.</p>
<p>Thanks,<br>
Loïc</p>

username: emcp
cooked: <p>Have you had any progress on this?</p>
<p>I too am in a similar situation with FasterRCNN… except I am using ResNet 101… my prior implementation I could fill the GPU with batch_size = 1 … where as with this torchvision version I cannot seem to get above 40% VRAM utilization… which to me indicates down sampling</p>
<p>Is the Integer <code>max_size</code> input of class <code>FasterRCNN(GeneralizedRCNN):</code> changing anything for you in terms of VRAM usage ?  I see no change no matter what number I place in there.</p>
<p>my images are 4K, 3840 x 2160, and up… so they definitely would have trouble before fitting on my GPU which has only 6GBs of VRAM… now I am puzzled how I can maximize the resolution and avoid at all costs down sampling</p>

username: Dipam_Vasani
cooked: <p>If you look at the <a href="https://github.com/pytorch/vision/blob/0156d58ec867590b1c78fe1bc834c7da9afdf46a/torchvision/models/detection/transform.py#L22" rel="noopener nofollow ugc">docs</a> you will see that it does resize the input. If you look further in the <a href="https://github.com/pytorch/vision/blob/0156d58ec867590b1c78fe1bc834c7da9afdf46a/torchvision/models/detection/transform.py#L47" rel="noopener nofollow ugc">source code</a>, you can see exactly how the <a href="https://github.com/pytorch/vision/blob/0156d58ec867590b1c78fe1bc834c7da9afdf46a/torchvision/models/detection/transform.py#L78" rel="noopener nofollow ugc">resizing works</a>. But in short, if you don’t want it to resize, just set the min and max to the shape of your input</p>
Customized data augmentation

username: hamidfarhid
cooked: <p>Hi,<br>
I was looking for customized data augmentation for image data set that I have and found this:</p>
<aside class="quote quote-modified" data-post="1" data-topic="13797">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/cdc98d/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-augment-the-minority-class-only-in-an-unbalanced-dataset/13797">How to augment the minority class only in an unbalanced dataset</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have an unbalanced image dataset with the positive class being 1/10 of the entire dataset. Classification models trained on this dataset tend to be biased toward the majority class (small false negative rate and bigger false positive rate). I would like to do some augmentation only on the minority class to deal with this. The way I understand, using transforms (random rotation, etc.) when creating torchvision.dataset will augment all of the data so the imbalance remains the same. 
What is the …
  </blockquote>
</aside>

<p>This would be great to start as <a class="mention" href="/u/ptrblck">@ptrblck</a> pointed. However, I am looking to the clean method to augment data to the certain number of images, let’s say 5000 images per each class. I have various distribution of data  across all classes, from 200 to 1000, so I have to augment some classes more than others. Is there any way to do so?</p>
<p>Thanks</p>

username: Stardust
cooked: <p>First of all, we can select the classes that we have to focus on (minority classes for imbalanced dataset), then we can define a list of transformations for the selected classes and give the probability values for each individual transformations, like for classes that are very rare, we should augment them for 90 percent (for example) of the cases, and others that are in majority, augment them for only 10 percent (for example) of the cases, and so on. In this way, in the custom dataset class that we created, we can iterate on the classes and pass the predefined transformations with certain probability values, and apply on those classes/labels.</p>
LSTM always predicts the same class

username: Miguel_C
cooked: <p>I’m trying to solve an nlp classification problem with a LSTM. The code for the model is defined here:</p>
<pre><code>class LSTM(nn.Module):

  def __init__(self, hidden_size, embedding_size=66 ):

      super().__init__()
      self.lstm = nn.LSTM(embedding_size, hidden_size, batch_first = True, bidirectional = True)
      self.fc = nn.Linear(2*hidden_size,2)

  def forward(self, input_seq):
      
      output, (hidden_state, cell_state) = self.lstm(input_seq)

      hidden_state = torch.cat((hidden_state[-1,:], hidden_state[-2,:]), -1)

      logits = self.fc(hidden_state)
      
      return nn.LogSoftmax(dim=1)(logits)
</code></pre>
<p>And the function I’m using  to train this model is here:</p>
<pre><code>def train_loop(dataloader, model, loss_fn, optimizer):
    
    loss_fn = loss_fn
    size = len(dataloader.dataset)
    model.train()
    zeros = 0
    for batch, (X, y) in enumerate(dataloader):

        # Transform string into tensor
        tensor = torch.zeros(1,len(X[0]),66)
        for i in range(len(X[0])):
            tensor[0][i][ctoi[X[0][i]]] = 1

        pred = model(tensor)

        target = torch.zeros(2, dtype=torch.long)
        target[y] = 1
        
        if batch % 100 == 0:
            print(pred.squeeze(), target)
        loss = loss_fn(pred.squeeze(), target)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if pred.squeeze().argmax() == 0:
            zeros += 1

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")

    print(f'In trainning predicted {zeros} zeroes out of {size} samples')
</code></pre>
<p>The X’s are still strings, that’s why I need to convert them to tensors before running it through the model. The y’s are either a 0 or 1 (since its a binary classification problem), that I need to convert to a tensor of shape (2,) to run through the loss function.</p>
<p>For some reason I keep getting the same class predicted for every input. The classes are not even that unbalanced (~45% to 55%), and I’ve tried changing the weights of the classes in the loss function with no improvements, it either converges to predicting always a 0 or always a 1. Most of the time it it converges to predicting always a 0, which makes even less sense because what happens usually is that the class 0 has less samples than class 1.</p>