akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
License Question

Phantom999: <p>Hello,</p>
<p>I have a question regarding the license.  Does importing torch into my python file constitute as either binary or source code form?  If it does, I would need to include PyTorch’s license with source code I put on github, and if it does not, I would not need to include PyTorch’s license with source code I put on github, is this correct?</p>
<p>Thanks,<br>
Phantom</p>

ptrblck: <p>Since I have no idea about licenses, CC’ing <a class="mention" href="/u/smth">@smth</a> and <a class="mention" href="/u/alband">@albanD</a>.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

smth: <p>it’s a library form.</p>
<p>When you ship your program to customers and the world, if you include PyTorch binaries and source code, you will have to keep their LICENSE file and information intact (wherever they are, keep them unmodified). If you don’t include PyTorch binaries and source code when you ship your program (but you expect users to install PyTorch themselves), you don’t have to do anything.</p>

Phantom999: <p><a class="mention" href="/u/smth">@smth</a>, thank you so much for your response.  My question is if I have a python file that I put “import torch” at the beginning is that considered pytorch’s binary or source files (just the import line)?  I am wondering if it is reasonable to put a file like this up on github.  I would not be including an source files from pytorch.   Obviously, if I explicitly included either the binary form of pytorch or the pytorch source code in my repository I would need that license, but if I’m simply doing “import torch” is that considered distributing pytorch in source or binary form?</p>
<p>Thanks again for the help!</p>

smth: <p>if you have a python file that has <code>import torch</code>, it is not a binary that will come under pytorch licensing.</p>
<blockquote>
<p>but if I’m simply doing “import torch” is that considered distributing pytorch in source or binary form?</p>
</blockquote>
<p>No.</p>
<p>If you include PyTorch source code or binary forms (i.e. the files shipped with PyTorch which are .py or .so or other binaries), those files are under pytorch license.</p>

Phantom999: <p>Thank you so much for the response.  So as long as I simply just import torch, I do not need to include the pytorch license?</p>
<p>Thanks,</p>
<p>Phantom</p>
sizeof(torch::kF32)=1 and sizeof(torch::kU8)=1?

FantasyJXF: <h2>
<img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=9" title=":question:" class="emoji" alt=":question:"> Questions and Help</h2>
<ul>
<li>libtorch 1.2 CU100 or libtorch 1.4 cpu</li>
</ul>
<p>When I try to convert the the  <code>torch::Tensor</code>  to  <code>cv::Mat</code> , I use the following code:</p>
<pre><code class="lang-auto">std::memcpy((void *) linkmap.data, score_link.data_ptr(), sizeof(torch::kF32) * score_text.numel())
</code></pre>
<p>The tensor type is  <code>CPUFloatType</code>  and I use  <code>sizeof(torch::kF32)</code>  to get the tensor data, but failed, and I notices that  <code>sizeof(torch::kF32)</code>  and  <code>sizeof(torch::kU8)</code>  are all equal to  <code>1</code> , which was not usual, since  <code>sizeif(float)</code>  should be 4 and  <code>sizeof(char)</code>  should be 1.</p>
<p>I change the code  <code>sizeof(torch::kF32)</code>  to  <code>4</code>  and get the right cv::Mat.</p>
<p>My question is why  <code>sizeof(torch::kF32)=1</code>  and  <code>sizeof(torch::kU8)=1 </code> ?</p>

tom: <p>That’s because torch::kF32 is a constant indicating a type, not a C++ type. I think you want <code>elementSize(torch::kF32)</code>, which gives you the element size.</p>
<p>Best regards</p>
<p>Thomas</p>

FantasyJXF: <p><code>torch::elementSize(torch::kF32) = 4 </code> works.</p>
Torch.where breaks the gradient flow

mohammadf: <p>Hi!</p>
<p>I’m working with a dataset of wireless interference measurements, and for my calculations, I need to extract the measurements made for specific channels. The channels are chosen according to a hopping algorithm specified by the IEEE 802.15.4 standard. I’m using a neural networkto blacklist a few of the channels and exclude them from the hopping procedure. I came up with the following solution that uses <code>torch.where</code>:</p>
<pre><code class="lang-auto">datapoints, channels = interference.shape
available_channels, = torch.where(blacklisted &lt; 0.5)
channels_in_a_round = available_channels.view(-1, 1).repeat(1, 20).flatten()
round_length, = channels_in_a_round.shape
length_ratio = np.ceil(datapoints / round_length).astype('int')
assigned_channels = channels_in_a_round.repeat(length_ratio)[:datapoints]
interference_in_assigned_channels = interference[torch.arange(datapoints), assigned_channels]
# Some calculations on interference_in_assigned_channels leading to the final output of the model
</code></pre>
<p>Now, even though <code>blacklisted</code> is a tensor with <code>requires_grad=True</code> (it’s actually the sigmoid-activated output of a neural network with 16 dimensions) the gradient flow breaks and the <code>available_channels</code> tensor no longer carries this property. I’m guessing this operation (or at least the way I’m doing it) is not supported by autograd. I’d appreciate it if you could help me out. How should I modify this code so that the gradient flow stays intact?</p>

G.M: <p>According to <a href="https://pytorch.org/docs/stable/torch.html#torch.where" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.where</a>, there r 2 ways to use <code>torch.where</code>. Here, it seems that u r using the non-differential one.  <code>torch.where(condition)</code> is equivalent to <code>torch.nonzero(condition, as_tuple=True)</code>, which returns indices, and thus broke the graph.</p>
How to know if gradients are making it back to the model

jwillette: <p>I am trying many new things and losses and I always find myself questioning whether the gradients are actually making it back to the model like I want them to.</p>
<p>For instance, I am trying to output the parameters of a normal distribution (mu, and sigma) from my model and then I create a pytorch Normal distribution and do some operations to get a loss from that. This is one part of a larger multi part loss function so how can I be sure that my loss from the Normal distribution is actually making it to the model?</p>

tom: <p>By testing? torch.autograd.gradcheck  could be one way, or <code>retain_grad</code> to the beginning of the loss calculation you are concerned about.</p>
How should I make the below operation?

kangkang_li: <p>Given tensor a-&gt;[16,50,50,512]  and b-&gt;[16,50,512]</p>
<p>I want this a[:,0,0,:]=b[:,0,:]  from 0 to 49 with no for-loop</p>

tom: <p>You should be able to use diagonal for assignment:</p>
<pre><code class="lang-python">a = torch.zeros(6, 5, 5, 7) 
b = torch.randn(6, 5, 7) 
# the diagonal dimension is always last for numpy compat...
d = a.diagonal(dim1=1, dim2=2).permute(0, 2, 1)
d.copy_(b)
# now the "diagonal" of a is set to b.
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
Understanding cnn models

peter_wills: <p>Consider the code, starting with a can model based:</p>
<hr>
<h2>basemod = models.resnet34(pretrained=True)<br>
b_list = list(self.basemod.children())<br>
basemod_new = torch.nn.Sequential(*deepcopy(b_list))</h2>
<p>My take on this was that basemod and basemod_new are the same model however basemod_new fails during inference with:</p>
<p>RuntimeError: size mismatch, m1: [2048 x 1], m2: [512 x 1]</p>
<p>Can someone explain what the problem is?</p>

ptrblck: <p>They are only the same, if your basemodel was using all layers in a sequential manner without any other calls.<br>
In the resnet implementation you would lose the functional call to <a href="https://github.com/pytorch/vision/blob/b2e95657cd5f389e3973212ba7ddbdcc751a7878/torchvision/models/resnet.py#L210">flatten</a>.</p>
Can anybody help me with my lstm model for predicting trig function?

gokul_adethya: <p>I am trying to predict the value of sin(x) function using nn.LSTM layer.My model:-</p>
<pre><code class="lang-auto">class lstm_net(nn.Module):
    def __init__(self):
        super(lstm_net,self).__init__()
        self.lstm_layer1 = nn.LSTM(100,100)
        self.layer_output = nn.Linear(100,100)
        self.hidden_initialzie = (torch.zeros((1,1,100)),torch.zeros((1,1,100)))

    def forward(self,x):
        x , hidden = self.lstm_layer1(x,self.hidden_initialzie)
        x = self.layer_output(x)
        return x

</code></pre>
<p>My data generator:-</p>
<pre><code class="lang-auto">x =[i for i in range(1000) ]


def random_data_generator():
    y_temp = []
    x_temp = []
    index_start = random.choice(range(100,890))
    for i in range(100):
        x_temp.append(x[i+index_start])
    for i in range(100):
        y_temp.append((math.sin(x_temp[-1] + 0.1*i))*30)

    return torch.FloatTensor(x_temp).reshape((1,1,100)),torch.FloatTensor(y_temp).reshape(1,1,100)

</code></pre>
<p>My epoch loop:-</p>
<pre><code class="lang-auto">for i in range(1000):
    x_, y_ = random_data_generator()

    optimizer.zero_grad()
    output = my_model(x_)
    loss = loss_function(output,y_)
    loss_history.append(loss)
    loss.backward()
    optimizer.step()


print(loss_history[-1])
plt.plot(loss_history)
plt.show()

</code></pre>
<p>Plotting prediction:-</p>
<pre><code class="lang-auto">x_ , y_ = random_data_generator()
model_prediction = my_model(x_)
y = []
for i in model_prediction[0][0]:
    y.append(i)
print(loss_function(model_prediction,y_))
plt.plot(x_[0][0],y)
plt.plot(x_[0][0],y_[0][0])
plt.show() 
</code></pre>
<p>I have made a function that generates random dataset for the model to train that consist of 100 sinx values and i am trying to predict next 100 sinx values but my model seems to have a very high loss.I have added my loss and prediciton graphs.</p>
<p>My final loss :- tensor(418.2419, grad_fn=)<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/373014ad5ba0d6154bd7292502bde7a67b368486.png" alt="Figure_1" data-base62-sha1="7SdgX8KA6tDcRTmEVwU1bx81M90" width="640" height="480"> ![predict|640x480]</p>

gokul_adethya: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8baea4a54ee4274f5f61cff2eeb4dab54838153.png" alt="predict" data-base62-sha1="zumSlMhHcq09KDJwwLO1wlrHEaf" width="640" height="480"><br>
THIS IS MY PREDICTION GRAPH<br>
BLUE IS MODEL PREDICTION</p>
Difference between pytorch and tensorflow inverse Fourier

mchobanyan: <p>I am trying to replicate a call to tensorflow’s real 2D inverse Fourier transform using torch.irfft. Here is the code that I am using to test:</p>
<pre><code class="lang-auto">import numpy as np
import tensorflow as tf
import torch

x = np.random.randn(2, 8, 8)

x_tf = tf.complex(x[0], x[1])
x_tf = tf.signal.irfft2d(x_tf)
print(f'Tensorflow output: {x_tf.shape}')
# Tensorflow output: (8, 14)

x_pt = torch.from_numpy(x).permute(1, 2, 0)
x_pt = torch.irfft(x_pt, signal_ndim=2)
print(f'Pytorch output: {tuple(x_pt.shape)}')
# Pytorch output: (8, 15)
</code></pre>
<p>What is the reason for the discrepancy in the output shape between the two libraries? Is this the expected behavior?</p>
<p>My pytorch version is 1.4.0 and my tensorflow version is 2.1.0.</p>

albanD: <p>Hi,</p>
<p>I am not a specialist, but given the <a href="https://pytorch.org/docs/stable/torch.html#torch.irfft">doc</a>, it seems that if the size does not match, you can use the <code>signal_sizes=(8, 14)</code> to make them match what you expect.</p>

mchobanyan: <p>Thanks for the response!</p>
Libtorch statics library vs dynamicss library?

Kiki_Rizki_Arpiandi: <p>I have downloaded libtorch statics library from here <a href="https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-static-with-deps-1.4.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-static-with-deps-1.4.0.zip</a><br>
My question is</p>
<ol>
<li>What is the current state of statics libtorch statics library is it as stable as the dynamics one ?</li>
<li>It is advisable to use statics library instead of dynamics library?</li>
<li>Why there is no option to download the statics library from download menu at <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a> website?</li>
</ol>
<p>Thanks</p>

yf225: <p>Static linking libtorch doesn’t work well yet, please see <a href="https://github.com/pytorch/pytorch/issues/21737" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21737</a> for details.</p>

Kiki_Rizki_Arpiandi: <p>thanks for your response</p>
How to change Spatial Transformer Network to support different image sizes

vlco: <p>Following the tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html</a></p>
<p>What would I need to do to adapt the STN network to work with specific B=batch size, H=image height, W=image width, C= channels? I assume B is handled by .view(-1, …) but what about the remaining H, W, C values?</p>
<p>Thank you and my apologies if this is a bit of a beginner question! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>You mean the <code>view()</code> op in in <code>stn()</code> and <code>forward()</code>?<br>
If you change the number of input channels, height and width for your input, you would need to adapt the <code>in_features</code> for the linear layers as well or alternatively use adaptive pooling layers to get your desired output size.</p>
<p>As you can see <code>self.localization</code> returns an output of <code>[batch_size, 10, 3, 3]</code>. That are exactly the <code>in_features</code> of <code>self.fc_loc</code>.</p>
<p>The easiest way would be to add print statements into your <code>forward</code>, use your new image shapes and just print out the new shape returned from the layers before the linear layers.</p>

h4k1m0u: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="22911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>As you can see <code>self.localization</code> returns an output of <code>[batch_size, 10, 3, 3]</code> . That are exactly the <code>in_features</code> of <code>self.fc_loc</code> .</p>
</blockquote>
</aside>
<p>I was reading that tutorial and came across your comment. I can’t understand why <code>self.localization</code> returns <code>10*3*3</code> features and not <code>10</code> features (like the convolution layer before it)?</p>

ptrblck: <p>I’m not sure, if I’m missing something, but <code>self.localization</code> seems to be the first called block in the model.</p>

h4k1m0u: <p>Thanks for your response.<br>
I actually had an error when I tested this network on an image of 224x224, which after the two max-pooling layers became a 52x52 image. But I fixed it by replacing <code>10*3*3</code> with <code>10*52*52</code>.</p>
Element-wise multiplication

mahbub_sejan: <p>Hello all,</p>
<p>I want to multiply a matrix of 200*300 vector by each element of a 200 sized vector. Each element of the rows of the matrix will be multiplied by the corresponding element of the vector. Is it possible?</p>
<p>For example : [2, 3] is my vector and [[1, 2], [4,5]] is my matrix. So the output should be, [[2,4], [12,14]]</p>

Eta_C: <pre><code class="lang-python">matrix = torch.rand(2, 3)
vector = torch.rand(2)
result = (matrix.T * vector).T
</code></pre>
<p>or</p>
<pre><code class="lang-python">matrix = torch.rand(2, 3)
vector = torch.rand(2)
result = matrix * vector[:, None]
</code></pre>
How does torch.jit.tracer deal with precompiled functions?

MahdiNazemi: <p>My neural network includes a number of PyTorch layers, e.g. <code>nn.Linear</code>, <code>nn.Conv2d</code>, etc. and a number of precompiled functions, e.g. functions written in pure C/C++ or C/C++ with ATen data types.</p>
<p>When I run the tracer, I don’t see calls to these precompiled functions in the <code>tracer.code</code>. Should I expect to see them in the generated code?</p>
Help needed: Specific function call causing 20X slowdown of computation

Sai_Kiran: <p>Hi, I am looking for help with the following code:</p>
<blockquote>
<p>SRAM_rows = 256<br>
step_size = 1<br>
prob_table = torch.eye(SRAM_rows2+1).cuda()<br>
levels = torch.tensor([i-SRAM_rows for i in range(SRAM_rows2+1)]).cuda().float()<br>
cmprob = 0<br>
Expected_outputs = torch.tensor([0]*257).cuda().float()</p>
<p>def quant_XNORSRAM(x, prob_table, levels, step_size, lower_bound):<br>
x_ind = (x.type(torch.int64) - lower_bound) / step_size<br>
num_levels = len(levels)<br>
x_cdf = prob_table[x_ind, 0:num_levels-1].cumsum(dim=-1)<br>
x_rand = torch.rand(x.shape, device=‘cuda:0’)<br>
x_rand = torch.stack([x_rand] * (num_levels-1), dim=-1)<br>
x_comp = (x_rand &gt; x_cdf).type(torch.int64).sum(dim=-1)<br>
<span class="hashtag-raw">#import</span> pdb; pdb.set_trace()</p>
<pre><code>#here if cmprob is set, we add the ideal value + the noise
if(cmprob):
    y = Expected_outputs[x_ind] + levels[x_comp]
else:
    y = levels[x_comp]
return y
</code></pre>
</blockquote>
<p>Just by calling the quant_XNORSRAM function, my compute time is increased 20 times, can you please help me identify the issue causing this and how to fix it?</p>
<p>Thanks</p>

dskhudia: <p>This doesn’t seem related to quantization. Computation time increased 20x wrt what baseline?</p>

Sai_Kiran: <p>This is kind of quantization, as I am transforming X to Y using a specific quantization, the scenario is compute time increases 20 times if I use the function shown above  compared to the case where I don’t use it (no quantization)</p>

Zafar: <p>This is not related to <a href="https://pytorch.org/docs/stable/quantization.html" rel="nofollow noopener"><code>PyTorch Quantization</code></a>.<br>
To get help faster, I think it is better to add a different tag. Maybe “CUDA”?</p>
<p>Meanwhile, <a class="mention" href="/u/sai_kiran">@Sai_Kiran</a> can you explain what is the 20x slow down compared to (as Daya asked: what’s the baseline)?</p>
<p>Also, CUDA doesn’t really benefit from quantization unless you are using TensorRT (<a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/#arithmetic-instructions" rel="nofollow noopener">see this for throughputs in CUDA</a>)</p>
When I run a quantiztized model error happened

sptoyoursoul: <p>when I run a quantiztized model , an error happend , trace is as below<br>
Traceback (most recent call last):<br>
File “/home/guanx/pycharmProject/erfnet/venv/erfnet_pytorch-lidarseg/eval/eval_lidar_seg.py”, line 435, in <br>
evaluate(parser.parse_args())<br>
File “/home/guanx/pycharmProject/erfnet/venv/erfnet_pytorch-lidarseg/eval/eval_lidar_seg.py”, line 292, in evaluate<br>
outputs = model(inputs)<br>
File “/home/guanx/anaconda3/envs/python37/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
RuntimeError: Could not run ‘aten::max_pool2d_with_indices’ with arguments from the ‘QuantizedCPUTensorId’ backend. ‘aten::max_pool2d_with_indices’ is only available for these backends: [CPUTensorId, VariableTensorId].<br>
The above operation failed in interpreter.<br>
Traceback (most recent call last):<br>
File “”, line 63<br>
dilation: List[int],<br>
ceil_mode: bool):<br>
output, indices = torch.max_pool2d_with_indices(self, kernel_size, stride, padding, dilation, ceil_mode)<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
def backward(grad_output):<br>
grad_self = torch.max_pool2d_with_indices_backward(grad_output, self, kernel_size, stride, padding, dilation, ceil_mode, indices)</p>
<p>The above operation failed in interpreter.</p>
<p>who could help me to handle this error</p>

jerryzh168: <aside class="quote no-group" data-username="sptoyoursoul" data-post="1" data-topic="71430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d9b06d/48.png" class="avatar"> sptoyoursoul:</div>
<blockquote>
<p>max_pool2d_with_indices</p>
</blockquote>
</aside>
<p>this means we don’t have quantized implementation for <code>torch.max_pool2d_with_indices</code>, feel free to open an issue in PyTorch repo and we can have someone working on this. cc <a class="mention" href="/u/zafar">@Zafar</a></p>

Zafar: <p>This is dup: <a href="https://discuss.pytorch.org/t/i-try-to-run-quantizitzed-model-but-some-error-happend/71579" class="inline-onebox">I try to run quantizitzed model ,but some error happend</a></p>
Can we compute nn.functional.linear with torch.chunks of data

Sai_Kiran: <p>Hi, I want to split my input and weight tensors to x number of torch.chunks, and then be able to evaluate the chunk-wise nn.functional.linear or nn.functional.conv_2d in parallel without having to use a for loop. Can I know if this possible in pytorch?</p>

mrshenli: <p>Hey <a class="mention" href="/u/sai_kiran">@Sai_Kiran</a>, are you referring to Mesh-TensorFlow-like parallelism? We currently don’t have the API for it yet. But the split, scatter, parallel_apply, and gather can be done in the application side.</p>
<p>With parallel_apply, there won’t be a loop in the application code, but it internally uses a loop to launch multiple threads to process inputs in parallel. <a href="https://github.com/pytorch/pytorch/blob/f62a0060972d594cc1c4ab99d44267373eee4ec6/torch/nn/parallel/data_parallel.py#L138-L153" rel="nofollow noopener">Here</a> is an example usage of parallel_apply.</p>
DeepSpeed installation

murali_perumalla: <p>can someone help me installation of microsoft deepspeed library  for pytorch</p>

mrshenli: <p>Did you encounter any issue when installing that library? DeepSpeed authors would know better about the process. Will you consider routing this question to the <a href="https://github.com/microsoft/DeepSpeed" rel="nofollow noopener">DeepSpeed repo</a></p>
Multiple Processes Per GPU?

nyxynyx: <p>I am training a model that does not make full use of the GPU’s compute and memory. Training is carried out over two 2080ti GPUs using Distributed DataParallel.</p>
<p>How can we concurrently train 2 models per GPU (each using different parameters), so that we can more fully utilize the GPs?</p>
<p>The following code currently trains only 1 model across 2 GPUs.</p>
<pre><code class="lang-auto">import torch.multiprocessing
import torch.distributed
import torch.nn as nn

def train(gpu, args):
    distributed.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=args['world_size'],
        rank=args['nr']*args['gpu']+gpu

    ...

    torch.cuda.set_device(gpu)
    model.cuda(gpu)
    model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])

    # training loop
    for epoch in range(num_epochs):
        ...

if __name__ == '__main__':
    multiprocessing.spawn(train, nprocs=2, args=(args,))
</code></pre>

mrshenli: <p>One possibility is</p>
<ol>
<li>use the <a href="https://github.com/pytorch/pytorch/blob/45c45195cd6526fbb64773abd316a24bbe49f50d/torch/distributed/distributed_c10d.py#L1495" rel="nofollow noopener"><code>new_group</code></a> API in <code>torch.distributed</code> to create a different process group for two different models,</li>
<li>Create different DistributedDataParallel instances, one for each wrapper and pass the process group object explicitly to DistributedDataParallel constructor (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L187" rel="nofollow noopener">process_group arg</a>) instead of using the default one.</li>
</ol>
<p>In this way, DistributedDataParallel’s allreduce operations will not collide.</p>
Changing in place operation

azeeshan: <p>Hi,</p>
<p>I am getting an error: <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 5]], which is output 0 of SoftmaxBackward, is at version 1; expected version 0 instead.</code></p>
<p>My operation on the tensor of dimensions (1,5) is this:</p>
<pre><code class="lang-auto">probs = net(*input_var)
probs[-1] = 0.
probs = probs/torch.sum(probs)
</code></pre>
<p>To remove the in place operation, I replaced it with:</p>
<pre><code class="lang-auto">probs = net(*input_var)
probs_new = probs.clone()
probs_new[-1] = 0.
probs_new = probs_new/torch.sum(probs_new)
</code></pre>
<p>But this still gives the error. What is the correct way to avoid the in place operation?</p>

albanD: <p>Hi,</p>
<p>Do you get the exact same error? Is it still an output of Softmax?<br>
maybe something inside your net changes the output of the Softmax inplace before returning it?</p>

azeeshan: <p>Seems like I forgot to do the same at one more place that piece of code was used. It works now. Thanks.</p>
Extract members from tensor while keeping the shape as it is

gotlieb: <p>I have a torch tensor of size torch.Size([10, 1, 224, 224]), holding 10 images of size 224x224. I would like to extract single images but keep the tensor size as torch.Size([1, 1, 224, 224]). How can I achieve this?</p>

albanD: <p>Hi,</p>
<p>To do this, I usually use <code>.narrow()</code> and extract a single element. For example to get the second image: <code>data.narrow(0, 1, 1)</code></p>

gotlieb: <p>Hi <a class="mention" href="/u/alband">@albanD</a>! Thank you for the fast and correct response!</p>
Einsum Memory Efficiency

mark1: <p>I have a situation where I want to do a complicated einsum operation across 4 tensors.  I am trying to optimize for memory efficiency, and if the underlying code is breaking this up into multiple matmul operations, it produce intermediate matrices that are far bigger than desired.  My understanding is that other einsum implementations do use matmul as the underlying operation, but I have not found this info about pytorch.  Would this be efficient, and if not is there a clean way to do multiply-accumulate operations across more than 2 matrixes?</p>

albanD: <p>Hi,</p>
<p>Maybe you want to try to use <a href="https://github.com/dgasmith/opt_einsum">https://github.com/dgasmith/opt_einsum</a> to see if you can get a better formula?</p>
<p>For the exact implementation details, I’m afraid I don’t know <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

mark1: <p>Thanks for linking this! If I am understanding correctly, it looks like this library is using intermediate operations deliberately, which is good for some situations but not the memory-constrained case.  It does offer a bit more explanation than the pytorch documentation though</p>

tom: <p>Standard PyTorch einsum reduces to bmm calls in sequential order, so it’s not memory efficient if you have large intermediates.<br>
There isn’t a built-in way, but from the experience with torch.bilinear, you might get away by computing the result “row-wise” instead of in one go.</p>
<p>Best regards</p>
<p>Thomas</p>
Can we train the deep learning models "partially"?

shirui-japina: <p>Suppose the model consists of <em><strong>A</strong></em> + <em><strong>B</strong></em> + <em><strong>C</strong></em>.</p>
<ul>
<li>For sample image 1, only train the parts of <em><strong>A</strong></em> + <em><strong>C</strong></em>.</li>
<li>For sample image 2, only train the parts of <em><strong>B</strong></em> + <em><strong>C</strong></em>.</li>
</ul>
<p>How to implement this idea?</p>

Oli: <p>Hello! You can create two optimizers. Optimizer1(A+C) &amp; Optimizer2(B+C).Then call the step() on the optimizer to your liking <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">. Is sample image 1 and 2 in the same batch or what did you have in mind?</p>
8 Creators and Core Contributors Talk About Their Model Training Libraries From PyTorch Ecosystem

Jakub_Czakon: <p>Hi all,</p>
<p>We’ve created something really cool that I think this community especially will like!</p>
<p>I got <strong>creators and core contributors behind ALL of the model training libraries in PyTorch Ecosystem</strong> to talk about their projects, specifically:</p>
<ul>
<li>Philosophy</li>
<li>API structure</li>
<li>The learning curve for new users</li>
<li>Build-in features (what you get out-of-the-box)</li>
<li>Extension capabilities (simplicity of integration in research)</li>
<li>Reproducibility</li>
<li>Distributed training</li>
<li>Productionalization</li>
<li>Popularity</li>
</ul>
<p>The result is <a href="https://neptune.ai/blog/model-training-libraries-pytorch-ecosystem?utm_source=pytorch-forum&amp;utm_medium=post&amp;utm_campaign=blog-model-training-libraries-pytorch-ecosystem" rel="nofollow noopener">this blog post.</a></p>
<p>Hopefully, with first-hand info from people behind  <strong>fastai, lightning, ignite, skorch, catalyst and torchbearer</strong>, it will be easier to choose some of those libraries to improve your deep learning projects.</p>
<p>If you have any additional questions, stuff we didn’t mention that is interesting to you, drop a comment!</p>

ptrblck: <p>Thanks for sharing it here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
ConcatDataset behavior on internal lists

hawkeyedesi: <p>I have two data loaders that I combine using a ConcatDataset operation. However, I am performing a clustering operation on the dataset wherein I run some clustering on every epoch and update the clustered array that each dataset belongs</p>
<p>My question: How does the expected behavior of the list work when using concatenate. If I use the external update as so <code>concat_data.clusters = new cluster</code></p>
<p>I have a minimum complete example to explain the problem:</p>
<pre><code class="lang-auto"># --- minimum code to replicate the problem:
import torch.utils.data as data_utils
import torch
import random
class Data(data_utils.Dataset):
    def __init__(self,cluster_val):
        i = random.randint(0, 10)
        self.dataset = data_utils.TensorDataset(torch.arange(i*10, (i+1)*10))
        # Initialize with zero clusters
        self.clusters = np.ones(len(self.dataset)) * cluster_val
    def __len__(self): 
        return len(self.dataset)
    def __getitem__(self, idx):
        return self.dataset[idx], self.clusters[idx]

data_concat = data_utils.ConcatDataset([Data(0), Data(5)])

# -- Modify the cluster values externally. The below line does not work? 
data_concat.clusters = [random.randint(0, 20) for _ in range(0, len(data_concat))]

for i, (data, clusters) in enumerate(data_concat): 
    print(data, clusters)
</code></pre>
<p>I don’t get the expected behavior. Any suggestions on how to solve this problem?</p>

tom: <p><code>ConcatDataset</code> is intended for cases where you want to have the same thing per item as in the individual datas ets. You could either modify the individual data sets or go through <code>data_concat.datasets</code> (a list of datasets).</p>
<p>If you need to do operations across datasets, you’re better off creating a custom wrapper rather than using <code>ConcatDataset</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

hawkeyedesi: <p>Understood. Thanks. I will do the custom wrapper.</p>
Sclicing torch Tensor

Mohamed_Nabih: <p>Hello, I need help<br>
I have a torch tensor of size [1124823 x 13]  and I want from the center " of the tensor to take five frames from the right and five frames from the left and concatenate them is there is any function do this?<br>
" N.B."</p>

albanD: <p>Hi,</p>
<p>You can use things like <code>t[base-5:base+5]</code> where base is whatever you call the center of your Tensor <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Mohamed_Nabih: <p>sorry I can’t understand what is (t) and what is the base</p>

albanD: <p><code>t</code> would be your Tensor of size [1124823 x 13].<br>
And base is the index (as a python number) of the center. So something like <code>base = t.size(0) // 2</code>.</p>

Mohamed_Nabih: <p>Do you mean this</p>
<p>base = feat.size(0)//2<br>
x = feat[base-5: base:+5]</p>

albanD: <p>Yes if your tensor is called <code>feat</code> that will work.</p>

Mohamed_Nabih: <p>Thanks albanD<br>
But can I ask you if I want to go inside the center of each raw of the tensor take five elements from left and five from right</p>
<p>i = 0</p>
<p>j = 6</p>
<p>base = feat.szie(0)//2</p>
<p>for i in feat[i, j]:</p>
<pre><code>x = feat[base - 5: base:+5]

i += 1</code></pre>

albanD: <p>If you want to take form the second dimension, you can use the indexing syntax like: <code>x = feat[:, base-5, base+5]</code>. Or you can use the specialized function to get a subset: <code>x = feat.narrow(1, base - 5, 10)</code>. The two will give exactly the same result.</p>

Mohamed_Nabih: <p>Not excatly, I want to go form the center of each raw take five numbers from the left and five numbers from right</p>

albanD: <p>Rows are the second dimension for a 2D Tensor. So that should work.<br>
Maybe you want to share an example with a given Tensor and which values you expect to get?</p>

Mohamed_Nabih: <p>x = feat.narrow(1, base - 5, 10)</p>
<p><strong>IndexError: Dimension out of range (expected to be in range of [-13, 12], but got 562406)</strong></p>

albanD: <p>This is because the center of the second dimension is not the same as the one in the first dimension.</p>
<p>I am really confused about what you’re trying to do. I think an example of input/output that you want would help.</p>

Mohamed_Nabih: <p>The tensor size [ 1124823 x 13 ]. So I want to slice each raw from its center and take 10 elements, 5 from the right and five from left. So I need to iterate through every raw.</p>

albanD: <p>What do you mean by “raw”? row?</p>
<p>If you have a Tensor of size [ 1124823 x 13 ], a single row is a 1D Tensor of size 13.<br>
So you want to do: <code>feat.narrow(1, feat.size(1)//2 -5, 10)</code>.</p>

Mohamed_Nabih: <p>Yes, sorry for this typo it is raw.</p>
<p>But, Do I need a for loop to iterate for each raw</p>

albanD: <p>No, the operation above will give a Tensor of size <code>[ 1124823 x 10 ]</code> containing the result for every row.</p>
General doubt regarding training a FasterRCNN Model and feature extractor

jitesh: <p>I am working on object detection and I have a dataset containing images and their corresponding bounding boxes (ground-truth values).<br>
I actually have built my own feature extractor which takes an image as input and outputs a feature map(basically an encoder-decoder system where the final output of the decoder is the same as the image size and has 3 channels). Now, I want to feed this feature map as an input to a FasterRCNN model for detection. I have 2 main doubts at this point</p>
<ol>
<li>Is it okay if I skip training the feature extractor and just train the FasterRCNN for detection?</li>
<li>If I should also train the feature extractor, what will be my labels while training?</li>
</ol>
<p>I am actually a beginner in the field of Computer Vision and using Pytorch. It would be really helpful if anyone could guide me on how to approach this problem.</p>
Two optimisers in a network?

James_e: <p>Hi,</p>
<p>(I am quite new to neural networks to please forgive me if I say anything that is incorrect)</p>
<p>Q: Is it possible to put 2 different optimisers in a neural network? e.g different optimiser for certain layers</p>
<p>Thanks for your help</p>
<p>James</p>

albanD: <p>Hi,</p>
<p>Yes it is possible.<br>
You just need to give the corresponding parameter to the right optimizer.</p>

James_e: <p>Could you please explain how this works?</p>
<p>If I have two optimisers in the network, is it performing the same amount of back-propagation compared to if there was just one optimiser?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

albanD: <p>The backprop will just populate the <code>.grad</code> field of all parameters given a loss function.<br>
If you have a single loss and two optimizers. You do a single backward and then each optimizer’s step.<br>
If you have one loss per optimizer, then you want to do one backward, one step, zero the gradients, one backward, one step. To get only the gradients corresponding to each loss for the right optimizer.</p>

James_e: <p>What would be an example of the ‘one loss per optimiser’ situation?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

albanD: <p>Things like GAN, where the generator and discriminator have different losses.</p>
RNN implementation incorrect in PyTorch Tutorials?

bluesky314: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5540088f69f4aab55dc17c7d72c3fd476ecc94a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/5540088f69f4aab55dc17c7d72c3fd476ecc94a9" title="38%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5540088f69f4aab55dc17c7d72c3fd476ecc94a9_2_690x431.png" alt="38%20PM" data-base62-sha1="ca9M6VPfjTwiDBCX3lbIakX5yQ1" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5540088f69f4aab55dc17c7d72c3fd476ecc94a9_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5540088f69f4aab55dc17c7d72c3fd476ecc94a9_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/5540088f69f4aab55dc17c7d72c3fd476ecc94a9.png 2x" data-dominant-color="EAEBEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">38%20PM</span><span class="informations">1164×728 68.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The RNN implementation in <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Classifying Names with a Character-Level RNN — PyTorch Tutorials 2.2.0+cu121 documentation</a> does not pass the hidden into the output but computes the output entirely separately. Isn’t this the equation of RNNs: <img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b97edbd710ed5f1aa847bfadfc966a133afcdeb2.png" alt="42%20PM" data-base62-sha1="qsY7N0sG3SmAM8EQYIM5nyEqEuu" width="512" height="148"><br>
As described in <a href="http://blog.echen.me" rel="noopener nofollow ugc">http://blog.echen.me</a> ?</p>

Minseok: <p>Agree. When you check out simple RNN code result, it makes us wonder why the code in the RNN tutorial is different from the definition of RNN.</p>
<pre><code class="lang-auto">import torch.nn as nn
rnn = nn.RNN(10, 10, batch_first=True)

inp = torch.randn((1,3,10))

result = rnn(inp)

print(result[0].shape)
print(result[1].shape)
print(result[0][0])
print(result[1])
</code></pre>
<p>Same result of :<br>
tensor([[[-0.0253,  0.5289, -0.2582, -0.5125, -0.4577,  0.5456, -0.4042,<br>
-0.6476, -0.4210, -0.5473]]], grad_fn=)</p>
How to visualize loss per epoch while training Faster RCNN Resnet 50 model?

adam619: <p>Hi</p>
<p>I am struggling how to figure out visualizing loss per epoch while training pretrained Faster RCNN model.</p>
<p>I am training on a custom dataset. I have managed to train a Faster RCNN model.</p>
<p>I followed this<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">torchvision tutorial</a> to successfully train a model.</p>
<p>What is an convenient way to visualize loss per epoch?</p>
<p>Thank you</p>

simaiden: <p>Use <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">tensorboard</a></p>
How to casting cuda tensor types

mfcs: <p>Hey guys! Could you all help me with this issue?</p>
<p>“RuntimeError: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same”</p>
<p>I am using this function to initialize the weights:</p>
<pre><code class="lang-auto">def weights_init(model):
    if type(model) in [nn.Conv2d, nn.Linear]:
        nn.init.xavier_normal_(model.weight.data)
        nn.init.constant_(model.bias.data, 0.1)
</code></pre>
<p>And I am using this code to submit an image to the CNN layers:</p>
<pre><code class="lang-auto">for inputs, labels in images_loader:    
    inputs = inputs.to(device) 
    labels = labels.to(device) 
    inputs = inputs.unsqueeze(0)    

    print(inputs.shape)
    print(labels.shape)
    
    outputs = model(inputs)
</code></pre>
<p>Best regards,</p>
<p>Matheus Santos.</p>

tom: <p>As the error says, your inputs (likely) are double (aka float64) while your weights are float32 (which is the standard in GPU computing).</p>
<pre><code class="lang-python">inputs = inputs.to(device, dtype=torch.float32)
</code></pre>
<p>will help, but you could also see whether you want to provide the data as 32 bit floats to start with.</p>
<p>Best regards</p>
<p>Thomas</p>

mfcs: <p>Hey, thanks!<br>
Solved the problem, but I got another one:</p>
<p>“conv_transpose2d(): argument ‘stride’ must be tuple of ints, but found element of type float at pos 1”</p>
<p>Is this because the tensor elements are in float type?<br>
The configuration of my CNN layers is in this layout:</p>
<pre><code class="lang-auto"># conv1
          nn.Conv2d(in_channels = 1, out_channels = 64, kernel_size = 3, stride = 1, padding = 1, dilation = 1),
          nn.ReLU(),
          nn.Conv2d(in_channels = 64, out_channels = 64, kernel_size = 3, stride = 2, padding = 1, dilation = 1),
          nn.ReLU(),
          nn.BatchNorm2d(num_features = 64),
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji only-emoji" alt=":confused:"></p>

mfcs: <p>I solved here ahahahaha<br>
I was passing stride with float value <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Return variables with no grad from a checkpointed function

jatentaki: <p>I am developing an algorithm which is very memory hungry and I need to use <code>torch.utils.checkpoint</code> to fit it on my GPU. However, there are some training statistics which are easiest to compute alongside the stuff I am checkpointing (doing it separately would be very wasteful because they use many of the same intermediate results). However, the statistics have no gradient and as such cannot be returned from a function wrapped in <code>checkpoint</code>. How can I work around that? Example code looks like the following:</p>
<pre><code class="lang-auto">import torch
from torch.utils.checkpoint import checkpoint

def to_be_checkpointed(t):
   intermediate_result_1 = t.pow(2)
   stats_1 = intermediate_result_1.detach().mean()

   intermediate_result_2 = intermediate_result_1.sqrt()
   stats_2 = intermediate_result_2.detach().std()

   output = intermediate_result_2.sum()

   return output, stats_1, stats_2

t = torch.randn(30, 30, requires_grad=True)
out, stats_1, stats_2 = checkpoint(to_be_checkpointed, t)
out.backward()
</code></pre>
<p>and on <code>out.backward()</code> I get the following error: <code>RuntimeError: element 1 of tensors does not require grad and does not have a grad_fn</code>. Obviously, one way around this is to not detach the statistics and do it after the <code>checkpoint</code> call but</p>
<ol>
<li>It is not always possible (some statistics are straight up non-differentiable)</li>
<li>It’s wasteful I think (will the gradient still be calculated just to be discarded?)</li>
<li>It’s non-idiomatic and error prone (I’m writing this post after spending several hours hunting a bug I accidentally introduced this way)</li>
</ol>

tom: <p>Maybe manually implementing a specialized function would work well for you.<br>
The implementation of checkpointing is a bit tricky, but grabbing <a>CheckpointFunction</a> and amending it to support your stats should not be too hard.</p>
<p>Best regards</p>
<p>Thomas</p>

jatentaki: <p>Cool, this looks promising.</p>
<p>Thank you,<br>
Michał</p>
_thnn_conv2d_forward is not implemented for type torch.LongTensor

JasonChenhx: <p>Traceback (most recent call last):<br>
File “F:/pg/project/segmentation_eval/semseg/tool/test_fm.py”, line 100, in <br>
main()<br>
File “F:/pg/project/segmentation_eval/semseg/tool/test_fm.py”, line 56, in main<br>
test(args)<br>
File “F:/pg/project/segmentation_eval/semseg/tool/test_fm.py”, line 93, in test<br>
output  = model(input)<br>
File “D:\anconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “F:\pg\project\segmentation_eval\semseg\model\pspnet.py”, line 141, in forward<br>
x = self.layer0(x)<br>
File “D:\anconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “D:\anconda\envs\pytorch\lib\site-packages\torch\nn\modules\container.py”, line 92, in forward<br>
input = module(input)<br>
File “D:\anconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “D:\anconda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py”, line 320, in forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: _thnn_conv2d_forward is not implemented for type torch.LongTensor</p>

G.M: <p>It seems that one of your inputs to the Conv2d layer is in dtype <code>Long</code> while it should be some kind of float.</p>