akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Prediction on validation set looks faded

Mukesh1729: <p>Predictions from the trained model on the validation set look faded.<br>
Any suggestion on how to tune would be helpful. Thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45659e00bbbcb94d3eb8eea4d15613fad05c0c06.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/45659e00bbbcb94d3eb8eea4d15613fad05c0c06" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45659e00bbbcb94d3eb8eea4d15613fad05c0c06_2_690x63.jpeg" alt="image" data-base62-sha1="9TUESmRzyNtXzkRtmIMBtTp6Bdc" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45659e00bbbcb94d3eb8eea4d15613fad05c0c06_2_690x63.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45659e00bbbcb94d3eb8eea4d15613fad05c0c06_2_1035x94.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45659e00bbbcb94d3eb8eea4d15613fad05c0c06_2_1380x126.jpeg 2x" data-dominant-color="939393"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2289×211 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mukesh1729: <p>Glad that I was able to make a transition into making a clear and good prediction when I had changed my optimizer from sgd() to Adam(). I used the sgd() optimizer in Keras and it works well, but when I used the same parameters for the optimizer and model in torch it failed to work.</p>
<p>Can someone explain why this happened? Thanks</p>
The limitation in using F.max_pool2d function

marsggbo: <p>I was going to implement the spatial pyramid pooling (SPP) layer, so I need to use F.max_pool2d function. Unfortunately, I got a problem as the following:</p>
<pre><code class="lang-python">invalid argument 2: pad should be smaller than half of kernel size, but got padW = 2, padH = 1, kW = 3, kH = 2 at /pytorch/torch/lib/THCUNN/generic/SpatialDilatedMaxPooling.cu:39
torch.Size([1, 512, 7, 9])
</code></pre>
<p>So I want to ask how to ignore this limitation — “pad should be smaller than half of kernel size”.</p>

marsggbo: <p>I’ve figured out this problem in the following way:</p>
<ul>
<li>
<p>we can first mutually padding our data</p>
</li>
<li>
<p>then update the kernel_size and stride</p>
</li>
</ul>
<p>the code to implement the SPP layer:</p>
<pre><code class="lang-python">

class SPPLayer(torch.nn.Module):

    def __init__(self, num_levels, pool_type='max_pool'):
        super(SPPLayer, self).__init__()

        self.num_levels = num_levels
        self.pool_type = pool_type
        

    def forward(self, x):
        # num:样本数量 c:通道数 h:高 w:宽
        # num: the number of samples
        # c: the number of channels
        # h: height
        # w: width
        num, c, h, w = x.size() 
#         print(x.size())
        for i in range(self.num_levels):
            level = i+1

            '''
            The equation is explained on the following site:
            http://www.cnblogs.com/marsggbo/p/8572846.html#autoid-0-0-0
            '''
            kernel_size = (math.ceil(h / level), math.ceil(w / level))
            stride = (math.floor(h / level), math.floor(w / level))
            pooling = (math.floor((kernel_size[0]*level-h+1)/2), math.floor((kernel_size[1]*level-w+1)/2))
            
            
            h_new = 2*pooling[0] + h
            w_new = 2*pooling[1] + w
            kernel_size = (math.ceil(h_new / level), math.ceil(w_new / level))
            stride = (math.floor(h_new / level), math.floor(w_new / level))
            
            zero_pad = torch.nn.ZeroPad2d((pooling[1],pooling[1],pooling[0],pooling[0]))
            x_new = zero_pad(x)
            
            # 选择池化方式 
            # choose the way of pooling
            if self.pool_type == 'max_pool':
                try:
                    tensor = F.max_pool2d(x_new, kernel_size=kernel_size, stride=stride).view(num, -1)
                except Exception as e:
                    print(str(e))
                    print(x.size())
                    print(level)
            else:
                tensor = F.avg_pool2d(x_new, kernel_size=kernel_size, stride=stride).view(num, -1)
            
            
              
            # 展开、拼接
            #flatten &amp; cat
            if (i == 0):
                x_flatten = tensor.view(num, -1)
            else:
                x_flatten = torch.cat((x_flatten, tensor.view(num, -1)), 1)
        return x_flatten
</code></pre>

kangkaixuan: <p>Thank s for your SPP codes, it really works!</p>
How to integrate outputsacross steps and then compute loss?

hobbitlzy: <p>Hi, in my training task, I expect a large batch for a better gradient estimate and the problem is the memory. I know one common method to enlarge the batch size is to use the gradient accumulation step. But that doesn’t work for me since these two are different for my loss:</p>
<ol>
<li>Compute the loss according to the output of several steps and add them together, which is gradient accumulation.</li>
<li>Integrate the outputs across several steps and then compute the loss.</li>
</ol>
<p>The wanted way to compute the loss is the second one. Therefore, I am thinking about whether can I save the results of multiple steps and then concatenate them to compute the loss. Two things I am worried about: First, it may take quite an amount of memory to save multiple graphs.  Second, even though the memory works fine, the outputs used to compute the loss comes from different graphs. So I suspect the results I get are still computing the loss separately.</p>
<p>In a word, the correct computation of my loss requires a large batch. But due to the memory, I have to run small batches several times. Is there any solution that I can integrate them so that they behave like running in a large batch?<img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=10" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>
How to Random Erasing with black box

nicozhou: <p>Hi, I want to do random erase with torchvision, and I want it to replace the erased region with black pixels, how to do that? I can only got gray. By the way, I am using this parameter</p>
<p><code>transforms.RandomErasing(p=1.0, value=0)</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d203ae46f89007a2d488221d568b51eb58dbc9f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/d203ae46f89007a2d488221d568b51eb58dbc9f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d203ae46f89007a2d488221d568b51eb58dbc9f5_2_690x168.png" alt="image" data-base62-sha1="tXS9B5dzUkCQKfIZjP9NEazqPmR" width="690" height="168" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d203ae46f89007a2d488221d568b51eb58dbc9f5_2_690x168.png, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d203ae46f89007a2d488221d568b51eb58dbc9f5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d203ae46f89007a2d488221d568b51eb58dbc9f5.png 2x" data-dominant-color="111112"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">947×231 33.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
What happens when we use torch.no_grad() in the middle of forward pass?

seyeeet: <p>I see some codes use <code>with torch.no_grad()</code> in the middle of forward pass in their model, I was wonder what will happen when we do it? does it mean no gradient for the part under the <code>with torch.no_grad()</code>? what are the things that I need to be careful about when I do it?</p>

googlebot: <p>yes, it affects newly created tensors within wrapped code block, disabling usual .requires_grad propagation</p>

seyeeet: <p>so would it makes sense to say that if I put that in the middle of my operation in the forward pass then the gradient will flow but it wont update the parameters for that specific area under the <code>with torch.no_grad()</code>?</p>

googlebot: <p>it breaks the flow, e.g.:</p>
<pre><code class="lang-auto">b=f1(a)
with no_grad():
  c=f2(b,p)
d=f3(c)
</code></pre>
<p>if p is a parameter, it disables its update, yes. but c-&gt;b flow is also disabled, and then b-&gt;a edge is never reached.<br>
so, in this situation, you’d normally use c = f2(b, p.detach()) instead; no_grad() in forward() is more for auxiliary paths, not for segments in the middle.</p>
Mitigate effect of host device synchronization (sync point)

lapayo: <p>Hi all,<br>
I am currently having the issue of increased runtime most likely due to sync points and would like to have some input from the community.<br>
Unfortunately, I cannot avoid them since I am working with sparse data and need quite a few operations like masked_select, non_zero, etc…</p>
<p>But are there ways to mitigate the effects and keep the GPU busy? I was thinking about things like running operations in parallel that are explicitly not dependent on the “to be synchronized” result.<br>
For example, I have a loop with 4 iterations, each independed and ending with a masked_select, technically those could easily run in parallel. However, due to the sync point each iteration waits for the one before. Is there something I can do to make this more efficient?</p>
<p>I also thought that different streams might be a solution, but since the python part blocks and waits, I cannot queue the remaining streams. Would that work or does pytorch synchronize the entire device and not just the current stream?</p>
<p>I am also open for more experimental ideas to solve the problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks a lot in advance!</p>
<p>Simon</p>
How could I let one module by-pass the DDP wrapper

coincheung: <p>Hi,</p>
<p>My model is defined in classical way, but it contains a self-defined operator:</p>
<pre><code class="lang-auto">class M(nn.Module):
    def __init__(self,):
        super(M, self).__init__()
        self.op = self_defined()
...
</code></pre>
<p>I am using DDP to train this model, which would sum up the grads of each gpu to compute the global grad and update the operators.</p>
<pre><code class="lang-auto">model = M()
model = nn.DistributedDataparallel(model)
...
</code></pre>
<p>However, in my case, it is my <code>self_defined</code> that is split into multi-gpu rather than the input data. Thus the grad computed within each gpu should not be summed up but gathered. Only this operator works in this way and all other operators works in the way DDP is designed.</p>
<p>How could I let <code>nn.DistributedDataParallel</code> by pass this operator ?</p>

huahuanZ: <p>DDP is designed for data parallel. AFAIK, your example is more like a model parallel, which I think is impossible to be properly wrapped up with DDP. Correct me if I’m wrong.</p>
Cannot import name 'int_classes' from 'torch._six'

Zrufy: <p>I’m using pytorch 1.9. I know that using 1.8 would solve it but I absolutely have to use 1.9.<br>
I have the same error but with “container_abcs” solved with “import collections.abc as container_abcs” the problem remains with “int_classes”. Does anyone have any solution to solve this problem besides pytorch downgrade.</p>

jwr7678: <p>Easiest solution would be to just set int_classes = int instead of importing it from _six.</p>

Mona_Jalal: <p>Confirming that I get the same error in PyTorch 1.9:</p>
<pre><code class="lang-auto">(proxy) [jalal@goku proxynca_pp]$ CUDA_VISIBLE_DEVICES=0,1 python train.py --dataset cub  --config config/cub.json --mode train --apex --seed 0
Traceback (most recent call last):
  File "train.py", line 3, in &lt;module&gt;
    import dataset
  File "/scratch3/research/code/fashion/proxynca_pp/dataset/__init__.py", line 6, in &lt;module&gt;
    from . import utils
  File "/scratch3/research/code/fashion/proxynca_pp/dataset/utils.py", line 8, in &lt;module&gt;
    from torch._six import int_classes as _int_classes
ImportError: cannot import name 'int_classes' from 'torch._six' (/scratch3/venv/proxy/lib/python3.8/site-packages/torch/_six.py)
(proxy) [jalal@goku proxynca_pp]$ pip freeze
h5py==3.4.0
numpy==1.21.2
Pillow==8.3.2
scipy==1.7.1
torch==1.9.0+cu111
torchaudio==0.9.0
torchvision==0.10.0+cu111
tqdm==4.62.2
typing-extensions==3.10.0.2
</code></pre>
<p>for this code repo: <a href="https://github.com/euwern/proxynca_pp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - euwern/proxynca_pp: The implementation of ProxyNCA++.</a></p>
<pre><code class="lang-auto">#from torch._six import int_classes as _int_classes
int_classes = int
</code></pre>
<p>^ worked for me.</p>
RuntimeError :(672x224 and 25088x1012)

saz: <p>I make mistakes every time on this for<br>
days, any comment would be helpful to me, thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52f96da776c9ee92d27c84959ae70dd21ac58343.png" data-download-href="https://discuss.pytorch.org/uploads/default/52f96da776c9ee92d27c84959ae70dd21ac58343" title="s"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52f96da776c9ee92d27c84959ae70dd21ac58343.png" alt="s" data-base62-sha1="bQ1y7rrpoE3MUPJ3NQAQ9hfm5ML" width="690" height="380" data-dominant-color="272726"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">s</span><span class="informations">986×544 27.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
19 image_path = test_dir+‘/dog/dog.1001.jpg’<br>
20<br>
—&gt; 21 predict(image_path, model)</p>
<p>4 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
1845     if has_torch_function_variadic(input, weight):<br>
1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)<br>
 → 1847     return torch._C._nn.linear(input, weight, bias)<br>
1848<br>
1849</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (672x224 and 25088x1012)</p>
What kind of datasets are used for model pre-training?

young124: <p>Hi, I got a question about the pre-trained models.</p>
<p>I am currently using PyTorch implemented and pre-trained MobileNetV2, ResNet, DenseNet.<br>
Are these models all pretrained with the ImageNet dataset?</p>
<p>Thanks for your help!</p>

ptrblck: <p>Yes, classification models are using ImageNet-12 based on <a href="https://discuss.pytorch.org/t/using-vision-models-with-the-cifar-dataset/7278/2">this answer</a>.</p>

young124: <p>Thanks for the reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Customize torchtext dataset for translation

keyu_Chen: <p>I am new to Torchtext, recently I try to train a translation model by using the opus dataset (<a href="https://opus.nlpl.eu/" rel="noopener nofollow ugc">https://opus.nlpl.eu/</a>). But Torchtext can only read CSV/JSON/TSV files, and the opus datasets are not in those three. I am struggling with this problem and barely find resources to solve this problem.</p>
<p>I turn some .txt files into .csv and try to customize the dataset, but the error occurred.<br>
Here is the code:<br>
spacy_ger = spacy.load(“de_core_news_sm”)<br>
spacy_eng = spacy.load(“en_core_web_sm”)</p>
<p>def tokenize_ger(text):<br>
return [tok.text for tok in spacy_ger.tokenizer(text)]</p>
<p>def tokenize_eng(text):<br>
return [tok.text for tok in spacy_eng.tokenizer(text)]</p>
<p>german = Field(tokenize=tokenize_ger, lower=True)</p>
<p>english = Field(tokenize=tokenize_eng, lower=True)</p>
<p>source_data, target_data = TabularDataset.splits(path=’./’,<br>
train=‘src-test.csv’,<br>
test=‘tgt-test.csv’,<br>
format=‘csv’,<br>
fields=(english,german))</p>
<p>source_data, target_data = TabularDataset.splits(path=’./’,</p>
<p>File “E:\Work\anaconda\envs\con_38\lib\site-packages\torchtext\data\dataset.py”, line 77, in splits<br>
train_data = None if train is None else cls(</p>
<p>File “E:\Work\anaconda\envs\con_38\lib\site-packages\torchtext\data\dataset.py”, line 271, in <strong>init</strong><br>
examples = [make_example(line, fields) for line in reader]</p>
<p>File “E:\Work\anaconda\envs\con_38\lib\site-packages\torchtext\data\dataset.py”, line 271, in <br>
examples = [make_example(line, fields) for line in reader]</p>
<p>File “E:\Work\anaconda\envs\con_38\lib\site-packages\torchtext\utils.py”, line 130, in unicode_csv_reader<br>
csv.field_size_limit(sys.maxsize)</p>
<p>OverflowError: Python int too large to convert to C long</p>

ptrblck: <p>You could try to use <a href="https://stackoverflow.com/a/58071155">this suggestion</a>.</p>
Can Gradient Checkpointing and Gradient Accumulation be together?

ben9004: <p>For train large batch,</p>
<p>can Gradient checkpointing and Gradient Accumulation be used together?</p>
<p>I think this should not be together because Gradient Checkpointing doesn’t utilize some of it’s layer’s computational graph and also off their requires_grad flag, so accumulation steps won’t be added at all</p>
<p>Am I wrong? please tell me the right answer!<br>
Thanks</p>

ptrblck: <p>I think you are wrong as I can’t think of a limitation why this wouldn’t work or are you seeing any issues while using both utilities?</p>
<aside class="quote no-group" data-username="ben9004" data-post="1" data-topic="131869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/848f3c/48.png" class="avatar"> ben9004:</div>
<blockquote>
<p>I think this should not be together because Gradient Checkpointing doesn’t utilize some of it’s layer’s computational graph and also off their requires_grad flag</p>
</blockquote>
</aside>
<p>That’s not entirely true. Gradient checkpointing will recompute specific operations to trade compute for memory. However, the result (compared to a non checkpointed run) should be the same.</p>
Calculate dim of fully connected layer in a CNN

dataloader: <p>Greetings,</p>
<p>I understand the underlying concept about calculating dimensions from a fully connected layer in a CNN. However, i am wondering if there is a smarter way to compute those.I mean, in a big network, it is a pain to calculate these dimensions. Please advice.</p>
<p>By fully conected layer in a cnn,i mean:</p>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
--&gt;&gt; self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)
</code></pre>

ptrblck: <p>The easy approach would be to print the shape of the incoming activations first and set the <code>in_features</code> of the first linear layer afterwards. Alternatively, you could use the <code>Lazy*</code> layers, which would set the input shape for you.</p>
How to mount colab to different google drive

aleemsidra: <p>I have mistakenly mounted the wrong google account while using the authorization link. How can now I change it to the one I intend to use? I have tried restarting the runtime, but it’s not working.</p>
<pre><code class="lang-auto">from google.colab import drive
drive.mount('/content/drive', force_remount= True)
os.chdir('/content/drive/My Drive/DFU')
Mounted at /content/drive
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
&lt;ipython-input-4-4c99a186fa70&gt; in &lt;module&gt;()
      1 from google.colab import drive
      2 drive.mount('/content/drive', force_remount= True)
----&gt; 3 os.chdir('/content/drive/My Drive/DFU')

FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/My Drive/DFU'
</code></pre>

ptrblck: <p>I would recommend to post this question to a Google Colab board (or maybe StackOverflow) as it seems to be unrelated to PyTorch and you might thus get better and faster support there.</p>
The difference between darknet and pytorch

sunshine1: <p>I would like to know the advantage and disadvantage of darket and pytorch. How to choose between them?</p>

ptrblck: <p>I would recommend to check both frameworks, take a look at their examples, play around with them, and see if one of them would fit your use case. Also, it seems that the <code>darknet</code> repository got an update ~3 years ago, so you might also want to consider the support (I don’t know, if it’s still developed or not).</p>
Gradients of tensor received from another GPU

marioo: <p>Hi,</p>
<p>I was planning to process some data on a GPU (let me call it GPU1) and then send a tensor from this GPU1 to another GPU (GPU2) using <code>torch.distributed.send</code> and <code> torch.distributed.recv</code>.</p>
<p>I was wondering, does the received tensor on GPU2 keep the gradients history from its previous life on GPU1? Is it possible to apply backpropagation ?</p>
<p>Thanks in advance for your help</p>

ptrblck: <p>Your use case sounds like model parallel, so I’m unsure if you would really need to use <code>send/recv</code> or could use <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html#basic-usage">this simple example</a>.</p>

marioo: <p>Thanks for your reply. Sorry for the incomplete explanation of the the problem. What I want is not similar to that you example. I want to code a parallel solver. So, each gpu is solving part of the problem in parallel and after some operations some of them have to share their tensors in pairs. I was wondering if the gradients of those tensor being transferred are lost and if they can be backpropagated.</p>

pritamdamania87: <p>There is no backpropagation for <code>send</code> and <code>recv</code>. You can use the RPC Framework: <a href="https://pytorch.org/docs/stable/rpc.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch 1.9.0 documentation</a> and that will allow you to backpropagate across RPC calls.</p>
<p>Alternatively, you could do this yourself via autograd functions, ex: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/nn/functional.py#L135" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at master · pytorch/pytorch · GitHub</a>. You can find docs for autograd functions here: <a href="https://pytorch.org/docs/stable/autograd.html#function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.9.0 documentation</a></p>
TTS Model not providing clear speech

Hasan_Khan: <p>I have trained my <strong>Text-to-Speech</strong> model on <code>tacotron</code> few months back for speech synthesis. At that time it was generating a clear audio with proper text recognition but now after sometime I again tested the results for the same trained model. However, this time it is not providing clear output. All it is inferencing is noise only. Please help me figure out the issue.<br>
The model file is loading correctly, so there could be no chance of it getting corrupt and if that’s so the would could be the reason as my model is saved in s3 bucket.</p>

Hasan_Khan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  please acknowledge</p>

ptrblck: <p>I don’t know what might be causing it, as you are excluding some issues I would have recommended to check, i.e. the model’s <code>state_dict</code> etc.<br>
I’m also unsure if anything else was changed between these runs (such as the PyTorch release) so would generally recommend to use the same setup as before.</p>
How to perform Matrix multiplication along dimension

Roman27: <p>I tried to figure it out myself but I just don’t get the summation right. I have an (n ,d^2,m) Tensor A and an (n ,d  , m )  tensor B. I want the following summation result C  (with size  (n ,d  , m)  )</p>
<p>C[i,j,k] = A[i,:,k]B[i,:,k][j]</p>
<p>where the last product is meant as a matrix multiplication (the second dimension has to split into 2 of size d). How do I achieve this?</p>

Roman27: <p>Nevermind got it. The following does the job:</p>
<pre><code class="lang-auto">A=A.reshape(n,d,d,m)
C=torch.einsum('bijk,bjk-&gt;bik', A,B)
</code></pre>
How to write scipy CSR matrix to checkpoint?

rajeshpiryani: <p>I have scipy matrix, I want to write this as a checkpoint.</p>
<p>I am new to Pytorch, please suggest to me the possible way to write scipy matrix to checkpoint.</p>

ptrblck: <p>Add this array as a new entry to a <code>dict</code> and save it via <code>torch.save</code>.</p>

rajeshpiryani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the answer. It will work.</p>
Plot loss and accuracy with fasterRCNN

Lelouch1: <p>Hello,</p>
<p>I followed this tutorial : <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.1.1+cu121 documentation</a> to implement a faster-RCNN object detector, however I am not satisfied with the way losses and accuracy are showed :</p>
<blockquote>
<p>num_epochs = 10</p>
<pre><code>for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    evaluate(model, data_loader_test, device=device)
</code></pre>
</blockquote>
<p>The issue is that the “evaluate” function does not return the loss but only a coco evaluator. On top of that, I dont know how to extract the accuracy given by this item “coco evaluator” in order to be able to plot the accuracy over epochs.<br>
So if you know how to access both the loss and the accuracy in eval mode it would help me a lot</p>
<p>Thanks</p>

Beatriz_Bessa: <p>Did you find out how to do it?</p>
Sequential batch processing vs paralle batch processing?

Adex: <p>In deep learning based model training, in general batch of inputs are passed. For example for training a deep learning model with [512] dimensional input feature vector, say for batch size= 4, we mainly pass [4,512] dimenional input. I am curious what are the logical significance of passing the same input after flattening the input across the batch and channel dimenions [2048]. Logically the locality structure will be destroyed but will it significanlty speed up my implementation? And can it affect the performance?</p>

googlebot: <p>no, elementwise ops work on internally flattened data already, and you can’t flatten conv/matmul/etc inputs.</p>
Autograd yields nan even if nan input is independant of output

Youyoun: <p>Hello,</p>
<p>I have a question about how autograd handles Nans,</p>
<pre><code class="lang-auto">a = torch.tensor([np.nan, 2])
param = torch.tensor([1.], requires_grad=True)

(a * param)[1].backward()
print(param.grad) # yields tensor([nan])
</code></pre>
<p>I was expecting it to return <code>tensor([2])</code> since param grad is independent of the <code>a[0]</code></p>
<p>Is this normal behavior ? Is there a way to force autograd to ignore <code>nan</code> if the grad is independent of that value ?</p>
<p>Thank you for your answer.</p>

googlebot: <p>you’re broadcasting param tensor to a.shape during multiplication, it would be independent with:<br>
param = torch.tensor([1., 1.], requires_grad=True)</p>

Youyoun: <p>Hi,</p>
<p>Thank you for your answer. Why does broadcasting create any dependence to the <code>nan</code> ?<br>
<code>param</code> is supposed to be a scalar value, but even while defining <code>param = torch.tensor(1., required_grad=True)</code>, <code>param.grad</code> is still equal to <code>nan</code>.</p>

googlebot: <aside class="quote no-group" data-username="Youyoun" data-post="3" data-topic="131611">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/youyoun/48/40540_2.png" class="avatar"> Youyoun:</div>
<blockquote>
<p>but even while defining <code>param = torch.tensor(1., required_grad=True)</code>, <code>param.grad</code> is still equal to <code>nan</code>.</p>
</blockquote>
</aside>
<p>that’s because of derivative d(a*p)/dp=a=nan, this is computed elementwise after broadcasting</p>