akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
HTTP Connection Error

DarrenW: <p>Hi,</p>
<p>I’ve spend the last 2 hours trying to set up pytorch to run with a gpu but unfortunately all I am getting is HTTP 000 connection failures.  I have tried in in new environments for py3.6, 3.7 and 3.8.  I have also tried it based on both the conda and pip instructions on Get Started Locally and for the instructions listed under previous versions but every time it fails when it hits pytorch, torchvision or torchaudio.  All other packages will install until the point including cudatoolkit 10.2.  This is specific to PyTorch.</p>
<p>I am installing on a windows 10 enterprise build.</p>
<p>I have also tried other ML libraries with the same cudatoolkit and there is no issue installing those through either conda or pip so I am at a loss as to why these packages specifically fail.</p>
<p>I have previously installed the cpu version and while I ran into the issue for py 3.7, I was able to get around it by installing it for another version.</p>
<p>Any assistance would be appreciated.<br>
Thanks,<br>
dw</p>
What does `step_size` in `scheduler` depend on?

KarlSzp: <p>When I’m using torch.optim.lr_scheduler, it’s necessary to set a <code>step_size</code> to controll the frequency the lr decay by.</p>
<p>For a certain epochs(such as 1000), what is the proper/appropriate <code>step_size</code> should I send to the scheduler?</p>
<p>Is there any rules or tricks here?</p>
<p>Thanks in advance!</p>
How to get index of non-zero values for every row in a matrix pytorch

Andy1314Chen: <p>I want to return a dense tensor of the non-zero indices for each row. For example, given the tensors:</p>
<pre><code class="lang-auto">[0,1,1]
[1,0,0]
[0,0,1]
[0,1,0]
</code></pre>
<p>Should return</p>
<pre><code class="lang-auto">[1,2]
[0]
[2]
[1]
</code></pre>

phan_phan: <p>What do you want to do accomplish ?<br>
The output you gave as an example is not a tensor, as it does not have the same number of elements in each “row”.</p>
<p>You can try <code>torch.nonzero(x)</code>, or <code>x.ne(0)</code> (meaning ‘not equal’).<br>
One of the two solutions will probably suit your needs.</p>

fadetoblack: <p>One way to accomplish this is to do:</p>
<pre><code class="lang-auto"># x is a tensor
x.nonzero(as_tuple=True)[0]
</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nonzero.html#torch.nonzero" rel="noopener nofollow ugc">Docs for reference</a></p>

towardsautonomy: <p>Hello, is there a way to obtain indices of nonzero values such that they are differentiable?</p>
Copy_() operation not copying values

nalbwa: <p>Hi, at some part of my project, copy_() operation struggled me a bit,<br>
thus I tried to make a toy example as below but could not find a solution for it.</p>
<pre><code class="lang-auto">import torch

class A():
    def __init__(self):
        self.tensor = torch.zeros((5, 6), dtype=torch.float)

    def check(self, b):
        self.tensor[[2, 4]].copy_(b.tensor)
        print(self.tensor)

class B():
    def __init__(self, tensor):
        self.tensor = tensor

m = torch.ones((2, 6), dtype=torch.float)

a = A()
b = B(m)

a.check(b)
</code></pre>
<p>As written, this simple code is just to copy 1.0s to 0.0 at specific locations,<br>
and what I get is only full of zeros, which means copy_() operation did not work.</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>below code which uses assigning operation = instead of copy_() works</p>
<pre><code class="lang-auto">def check(self, b):
    self.tensor[[2, 4]] = b.tensor
    print(self.tensor)
</code></pre>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 1.],
        [0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 1.]])
</code></pre>
<p>and moreover, below code which is not involved in the use of classes works fine</p>
<pre><code class="lang-auto">c = torch.zeros((5, 6), dtype=torch.float)
d = torch.ones((2, 6), dtype=torch.float)

c[[2, 4]] = d
print(c)
</code></pre>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 1.],
        [0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 1.]])
</code></pre>
<p>I am hoping to fully understand the reason of not copying the values at the first toy example<br>
(I tried on PyTorch 1.6.0 and 1.7.0)</p>

albanD: <p>Hi,</p>
<p>The problem is that advanced indexing like <code>[[2, 4]]</code> does not always return a view into the original Tensor. It sometimes returns a different Tensor. And so when you do <code>tensor[[2, 4]].copy_(b.tensor)</code>, you actually do the copy into a Tensor that is completely independent of <code>tensor</code>.</p>
<p>For this to work, you can do <code>tensor[[2, 4]] = b.tensor</code> so that the advanced indexing will be used in “assignment mode” and pytorch will do the right thing.</p>

nalbwa: <p>Thank you for the great answer!</p>
<p>I am still a bit concerned with using the copy operations as I don’t know what the conditions for returning a different tensor and would like to seek for it later on</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Transformer vs RNN for real-time speech separation

Superklez: <p>I’ve read in <em>Attention is All You Need</em> that Transformers perform better than RNNs (Dual-Path RNN) in speech separation but had ten times the number of parameters. I’ve also read that it could better retain information from early inputs in the input sequence. However, how well does a Transformer network perform in real-time speech separation? Does the number of parameters and the way it deals with the inputs affect its ability to perform real-time speech separation?</p>
How to go about recreating / replacing a missing FC layer?

ProGamerGov: <p>So, I have a GoogleNet model that appears to be missing it’s main FC layer. I need the FC layer for calculations, and thus I need to find a way to recreate the missing layer.</p>
<p>I have a slightly modified version of the dataset (images were resized to be a square) that the model was trained on, but it’s 18 million images in total. This means that retraining a new FC layer while freezing the rest of the model’s parameters will take a long time, and thus I was wondering if there are any faster ways that I could go about this?</p>

ProGamerGov: <p>I just found the same model in TensorFlow, and I was able to just copy the FC layer weights and biases over from it.</p>
How to specific a op qconfig in "prepare_jit" qconfig_dict 

penghuic: <p>I want to use “prepare_jit” and “convert_jit” to quantize Resnet18. But I can’t specific ‘layer1.0.conv1’ to different qconfig.<br>
my code:<br>
model = models.<strong>dict</strong> <a href="https://github.com/pytorch/pytorch/issues/pretrained=True" rel="noopener nofollow ugc">'resnet18</a><br>
model = torch.jit.script(model.eval())<br>
qconfig1 = torch.quantization.QConfig(<br>
activation=torch.quantization.HistogramObserver.with_args(<br>
reduce_range=False),<br>
weight=torch.quantization.default_per_channel_weight_observer)<br>
torch.quantization.prepare_jit(model, {‘layer1.0.conv1’:qconfig1}, True)<br>
model(torch.randn(1, 3, 224, 224))<br>
torch.quantization.convert_jit(model, True, False)</p>
<p>But it will fail as below message:<br>
File “/home/xxx/python3.7/site-packages/torch/quantization/quantize_jit.py”, line 58, in _prepare_jit<br>
quant_type)<br>
RuntimeError: <strong>torch</strong>.torch.nn.modules.conv.___torch_mangle_67.Conv2d (of Python compilation unit at: 0x56088f811c00) is not compatible with the type <strong>torch</strong>.torch.nn.modules.conv.___torch_mangle_66.Conv2d (of Python compilation unit at: 0x56088f811c00) for the field ‘conv1’</p>
<p>It seems the key ‘layer1.0.conv1’ is not correct.<br>
How can I do?</p>

supriyar: <p>Is the goal here to only quantize one layer in the entire model with qconfig1? Can you try without specifying the inplace option for prepare_jit and convert_jit?<br>
i.e <code>torch.quantization.prepare_jit(model,  {‘layer1.0.conv1’:qconfig1})</code></p>
<p>cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a> for additional insight.</p>

jerryzh168: <aside class="quote no-group" data-username="penghuic" data-post="1" data-topic="115800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ee7513/48.png" class="avatar"> penghuic:</div>
<blockquote>
<p>File “/home/xxx/python3.7/site-packages/torch/quantization/quantize_jit.py”, line 58, in _prepare_jit<br>
quant_type)<br>
RuntimeError: <strong>torch</strong>.torch.nn.modules.conv.___torch_mangle_67.Conv2d (of Python compilation unit at: 0x56088f811c00) is not compatible with the type <strong>torch</strong>.torch.nn.modules.conv.___torch_mangle_66.Conv2d (of Python compilation unit at: 0x56088f811c00) for the field ‘conv1’</p>
</blockquote>
</aside>
<p>prepare_jit/convert_jit is no longer being maintained, for automatic quantization please try fx graph mode quantization: <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>if you encounter problems with symbolic tracing, you can take a look at: <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Quantization User Guide — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
Dataloader is giving me problems

David_Kubala: <p>Hey there!</p>
<p>I’m relatively new to PyTorch, and I’ve been working on a neural network to do sentence classification with a text embedding model.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto"># Load train, test, and dev data
data = MessagesDataset('./training/data_dev.csv')

# Create Model Object, specify parameters
net = LanguageClassifier()
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

# Create dataloader object to manage the data
dataloader = DataLoader(data, batch_size=1, shuffle=True, num_workers=5)

# For each epoch
for epoch in range(1):
	# Reset Gradients and loss statistics
	optimizer.zero_grad()
	
	# For each training example 
	for data, label in enumerate(dataloader):
		# Send data through neural net, calculate loss, and back propagate
		output = net(data)
		loss = criterion(output, label)
		loss.backward()
		optimizer.step()
</code></pre>
<p>And here’s my dataset object:</p>
<pre><code class="lang-auto"># Class for loading the messages dataset
class MessagesDataset(Dataset):
	def __init__(self, csv_file):
		self.messages_frame = pd.read_csv(csv_file)
		self.embedding_model = gensim.models.KeyedVectors.load('./models/model.wordvectors')
		self.sentence_length = 10
		
	def __len__(self):
		return len(self.messages_frame)

	def __getitem__(self, idx):
		if torch.is_tensor(idx):
			idx = idx.tolist()
		
		# Tokenize the message and create appropriate label
		message = []
		message_words = word_tokenize(self.messages_frame.iloc[idx, 0]) 
		for word in message_words:
			if word not in stop_words and word in self.embedding_model.wv.vocab:
				message.append(self.embedding_model.wv.vocab[word].index)
		
		label = self.messages_frame.iloc[idx, 1]
		
		if len(message) &lt; self.sentence_length:
			pad_zeroes = self.sentence_length - len(message)
			for i in range(pad_zeroes):
				message.append(0)
		elif len(message) &gt; self.sentence_length:
			message = message[:10]	
		
		sample = {'message': message, 'label': [int(label) for l in range(len(message))]}
		
		return sample
</code></pre>
<p>I’m having issues using the DataLoader to do minibatches. I’m following a tutorial online, but I keep having issues understanding the proper way to pass batched data into the neural network for training.</p>
<p>Thank you for your help in advance!</p>

Yasar_Niyazoglu: <p>hi<br>
You did batc_size 1, I don’t know if you know it, but if you are going to do it in mini groups, you should make the value of batch_size different from 1. I know it’s better that this value is a multiple of 2</p>

David_Kubala: <p>Hey Yasar!</p>
<p>I’ve tried it with many different batch sizes. It seems like the <strong>getitem</strong>() is returning a list, and my network is expecting a tensor. How dos this work with batching though?</p>
<p>Thank you!</p>

Yasar_Niyazoglu: <p>Thank you too, David. I’m new to pytorch and deep learning too. I hope you find the solution</p>
Register_hook is not called for all registered tensors

hamedB: <p>Hi<br>
The VGG19 has 16 Conv. layer. So during the forward pass I register a hook_register on the outputs (activation maps) of each conv. layer + the last max_poolin layer as well as record the activation maps in a list just after calling the register_hook. As a result, after forward pass operation I have activation maps for 17 layers. However, after calling backward on the model, only the gradients of the output of five last layer are recorded. To put it differently, the function that hook is registered on it is just called 5 times. Why it is not called for all the outputs of the layers which have a registered function? How can I know the hook_registered is exactly called for all the tensor that I want?</p>
<p>Thank you in advance.</p>
<p>This is the function which is called when I pass image through the model,</p>
<blockquote>
<p>def save_gradient(self, grad):<br>
print(‘calling gradients’)<br>
print(grad.shape)<br>
self.gradients.append(grad)</p>
</blockquote>
<pre><code>def __call__(self, x):
    outputs = []
    self.gradients = []
    for name, module in self.model._modules.items():
        if name not in self.target_layers:
            with torch.no_grad():
                x = module(x.to(self.device))
        else:
            x = module(x.to(self.device))
            x = x.requires_grad_(True)
            x.register_hook(self.save_gradient)
            outputs += [x]
    return outputs, x
</code></pre>

albanD: <p>Hi,</p>
<p>The hook are only called for the part of the model that you backpropagate through.<br>
And if you run the model in no grad mode, it will not compute gradients for that part and break the link between the input and output. So the gradient won’t flow backward past that part.</p>

hamedB: <p>Thank you <a class="mention" href="/u/alband">@albanD</a> or your response. I added the term torch.no_grad, since after the convolutional layer there is a module named Relu that passing the input through this layer throws below error:</p>
<blockquote>
<p>RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.</p>
</blockquote>
<p>How can I register a hook on the output of each convolutional layer without facing such error when the outputs are passing through Relu function.</p>
<p>Thank you very much for your time and consideration.</p>

albanD: <p>Well that errors means that you’re modifying inplace something you should not.<br>
If you don’t use no_grad at all, then this means that you’re modifying an input inplace which is not allowed. You should add a clone before that op or make that op not inplace to solve that issue.</p>

hamedB: <p>Thank you very much <a class="mention" href="/u/alband">@albanD</a>  for your time and explanations.<br>
I used the torch.clone before and after the Relu module and the problem solved.</p>
<p>Thank you.</p>
Tanh fusion with conv for quantization

amirhf: <p>Hello,</p>
<p>I am trying to quantize post training an augmented resnet model that uses tanh activation on the extracted features. I know that the model fusion currently supports conv+bn+relu combinations. But is there a way to fuse a tanh activation to its previous layer?<br>
Here is my model setup for the last layers of my augmented resnet:</p>
<pre><code class="lang-auto">class Resnet(nn.Module):
    def __init__():
        [previous layers]
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                       dilate=replace_stride_with_dilation[2])
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))

        # adding conv layer
        self.features_conv = nn.Sequential(
            nn.Conv2d(self.last_channel, hidden_feats, kernel_size=1),
            norm_layer(hidden_feats),
            nn.ReLU(inplace=True)
        )

        self.attention = nn.Sequential(
            nn.Conv2d(hidden_feats, hidden_feats, kernel_size=1),
            nn.Tanh()
        )

    def forward():
        [previous layers]
        x = self.layer4(x)

        out = self.features_conv(x)
        x_att = self.attention(out)
        out = out * x_att

        out = self.avgpool(out)
        out = torch.flatten(out, 1)

        return out
</code></pre>

Zafar: <p>At the moment the tanh fusion is not supported. <a class="mention" href="/u/raghuramank100">@raghuramank100</a> Any plans for this?</p>

supriyar: <p>We currently don’t have plans to support this. Please file a feature request on github or feel free to submit a PR to implement this and we can review it.<br>
Thanks!</p>
How to add blob like loss constraints that are differentiable in pytorch

eric_2_hamel: <p>Related to this question: <a href="https://discuss.pytorch.org/t/differentiable-way-to-check-for-existence-of-shapes/10244/3">https://discuss.pytorch.org/t/differentiable-way-to-check-for-existence-of-shapes/10244/3</a>.</p>
<p>I would also like to know more information regarding this question. It seems to have no answer.<br>
I will like to know if there is a way to add loss constraint based blob/shape characteristics while respecting the loss differentiability. Here are some example of desired constraint :</p>
<ol>
<li>penalize if number of blobs exceeds</li>
<li>penalize so that a specific class segmentation is included into another one. I thought maybe by multiplication with the thresholded segmentation map it will force inclusion not really a loss constraint however.</li>
<li>penalize if the blob is not convex.</li>
</ol>
<p>Thanks for your help.</p>
Differentiable way to check for existence of shapes

ganesha123: <p>Old question (see below for respecified version):<br>
Hi,<br>
Avid reader of this forum so first of all many thanks for countless problems fixed in the past.<br>
My problem: in my loss function, I want to use opencv’s cv2.findcontours to detect whether certain shapes are in the output image, and apply a penalty if they aren’t.<br>
Right now I can’t do that because opencv won’t take Variable objects, but if I convert into numpy (which is what it wants) then I won’t be able to call .backward on that loss and my model won’t be able to train.<br>
Is there another way to do this?<br>
Thanks for your help!</p>
<p>Edit:<br>
As SimonW pointed out my problem is misspecified.<br>
I need to check my image output for the existence of certain shapes (e.g. a square, or the letter ‘X’). I want to penalise the model in the loss function if the shapes are not there, so it needs to be differentiable.<br>
Is there a way to do this?</p>

SimonW: <p>I think the key thing is not whether opencv/numpy takes Variables. It is more because that what cv2.findcontours does is very likely a non-differentiable operation. You need to find differentiable alternative and write a custom loss. Maybe someone with more advanced vision background can help.</p>

eric_2_hamel: <p>I would also like to know more information regarding this question. It seems to have no answer.<br>
I will like to know if there is a way to add loss constraint based blob/shape characteristics while respecting the loss diffentiability. Here are some example of desired constraint :</p>
<ol>
<li>penalize if number of blobs exceeds</li>
<li>penalize so that a specific class segmentation is included into another one. I thought maybe by multiplication with the threholded segmentation map it will force inclusion not really a loss constraint however.</li>
<li>penalize if the blob is not convex.</li>
</ol>
<p>Is there has been any progress on that question?</p>
Pytorch: why a.storage() is b.storage() return False when a and b reference the same data?

vprokopev: <pre><code class="lang-auto">&gt;&gt;&gt; a = torch.arange(12).reshape(2, 6)
&gt;&gt;&gt; a
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11]])
&gt;&gt;&gt; b = a[1:3, :]
&gt;&gt;&gt; b.storage() is a.storage()
False
# but
&gt;&gt;&gt; b[0, 0] = 999
&gt;&gt;&gt; b, a # both tensors are changed
(tensor([[999,   7,   8,   9,  10,  11]]),
 tensor([[  0,   1,   2,   3,   4,   5],
         [999,   7,   8,   9,  10,  11]]))
</code></pre>
<p>What is exactly the objects that stores tensor data? How can I make check if 2 tensors share memory?</p>

albanD: <p>Hi,</p>
<p>The storage objects is not really a thing in python. We just provide these as a thin wrapper to be able to access attributes on them from python.<br>
What happens is that every time to ask for it, you get a new python objects and so the <code>is</code> is always going to be False:</p>
<pre><code class="lang-python">s = a.storage()
print(a is a.storage()) # Prints False
</code></pre>
<p>You can compare the <code>.data_ptr()</code> values in the storage to check this though. The two should be equal</p>
Differentiating with respect to input: tensor does not require grad and does not have a grad_fn, yet requires_grad is set to True

geodavic: <p>I want to differentiate the output of a model with respect to its input. Here is the code (self is a class similar to nn.Module, x is the input, which is a batch of images)</p>
<pre><code>lsm = torch.nn.LogSoftmax(dim=1)
x.requires_grad = True
self.eval()

logits = self.evaluate(x)
out_lsm = lsm(logits)

aux_sum = torch.sum(torch.max(out_lsm, axis=1)[0])
aux_sum.backward()
</code></pre>
<p>I want to find the gradient of <code>aux_sum</code> with respect to <code>x</code> , but for some reason <code>aux_sum</code> has no gradient function and I get the error <code> element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p>As a sanity check, I can do the same process but with a simpler “model”:</p>
<pre><code>x = torch.Tensor([0, 1, 2])
x.requires_grad = True

out_lsm = x ** 2

aux_sum = out_lsm.sum()
aux_sum.backward()
print(x.grad)
&gt;&gt;&gt; tensor([0., 2., 4.])
</code></pre>
<p>with no issues. Does anyone know what is causing the gradient issue in the first case, and why it’s not the same as the simplified second case? Thank you!</p>

suraj.pt: <p>Check if <code>out_lsm.requires_grad == True</code>. If not, maybe there is some operation in <code>self.evaluate</code> that is non differentiable that is causing <code>requires_grad</code> to be False</p>
<p>You can refer to this FAQ to know more about None gradients <a href="https://discuss.pytorch.org/t/why-are-my-tensors-gradients-unexpectedly-none-or-not-none/111461" class="inline-onebox">Why are my tensor's gradients unexpectedly None or not None?</a></p>

geodavic: <p>Thank you. The issue was actually a <code>torch.no_grad</code> hiding in <code>self.evaluate</code> that I didn’t account for.</p>
Saving dataloaders in C++

AntMorais: <p>Hi,<br>
I want to have a C++ source file that only has a test function (testFunction.cpp). As such, I was wondering if it is possible to read data, create a dataloader and store the dataloader in a .pt file, then load that dataloader in testFunction.cpp .<br>
Thank you,<br>
AntMorais</p>
Is there modulus operator `torch.mod`?

fixedrl: <p>For the modulus operator, is there a <code>torch.mod</code> existing ?</p>

albanD: <p>You have the <a href="http://pytorch.org/docs/master/torch.html?highlight=mod#torch.fmod">torch.fmod</a> and <a href="http://pytorch.org/docs/master/torch.html?highlight=remainder#torch.remainder">torch.remainder</a> functions.<br>
The difference between the two is how they handle negative numbers, see the docs for more details.</p>

iacob: <p>Python’s <a href="https://github.com/pytorch/pytorch/releases" rel="noopener nofollow ugc">modulo operator</a> <code>%</code> is also implemented for tensors.</p>
LSTM - question regarding hidden states

gortali: <p>Hello everyone,</p>
<p>I’m implementing an LSTM for a very particular time series problem where I basically have a dynamical system that advances in time and the LSTM analyzes this and makes some predictions.</p>
<p>My question is that I had a bug when I started working on the code which i solved adding this line:</p>
<pre><code class="lang-auto">hidden = tuple([each.data for each in hidden])
</code></pre>
<p>I don’t really understand why is this needed and what this does, and cannot find anywhere a proper explanation. If you can point me to some resource I would be very grateful.</p>
<p>Thank you in advance.</p>

googlebot: <p>this is preventing gradient backpropagation, like detach()</p>

gortali: <p>And why do we want to do that? In fact I do want to backpropagate in time to learn how the dynamics evolves…</p>

googlebot: <p>that was an answer about what your code does, I can’t answer why you are using it</p>

gortali: <p>Sure, the fact is that I see this used all over tutorials but I don’t undersand why.</p>
<p>If I need to translate a sentence like “I am going to the park”, I can’t do the detach after each word, right? But maybe I can do the detach at the end of the sentence, before translating a new one, assuming information on the previous sentence has nothing to do with the successive? Or do I need to do the detach after every backward() step that I do? The question is in general when should you do the detach.</p>

suraj.pt: <p>Generally you would not need to backprop through hidden states. Detaching hidden states from the graph reduces autograd’s memory and time consumption. Take a look at <a href="https://discuss.pytorch.org/t/time-memory-keeps-increasing-at-every-iteration/111453" class="inline-onebox">Time/Memory keeps increasing at every iteration</a></p>
Binary Segmentation. Representation of different classes

Augustas_Macys: <p>In binary segmentation, should the mask be represented as a matrix of 0s and 1s where 0 is one class (background) and 1 is another class. Or should it be represented as matrix of 0s and 255s where again 0 is one class (background) and 255 is another class. Or it does not matter?</p>

KFrank: <p>Hi Augustas!</p>
<aside class="quote no-group quote-modified" data-username="Augustas_Macys" data-post="1" data-topic="115734" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/augustas_macys/48/31617_2.png" class="avatar"> Augustas_Macys:</div>
<blockquote>
<p>should the mask be represented as a matrix of 0s and 1s?</p>
</blockquote>
</aside>
<p>The short answer is yes, the mask should be 0s and 1s (and, yes,<br>
it does matter).</p>
<p>Specifically, when you use pytorch’s <code>BCEWithLogitsLoss</code> (or its<br>
numerically-less-stable cousin <code>BCELoss</code>), the <code>target</code> (mask) you<br>
pass in should be a floating-point tensor of probabilities that range<br>
from <code>0.0</code> to <code>1.0</code>.</p>
<p>When your ground-truth <code>target</code> has complete certainty, it would<br>
have a value of <code>0.0</code> for the background – 0% probability of being<br>
a foreground pixel – and a value of <code>1.0</code> for the foreground – 100%<br>
probability of being a foreground pixel.  However, by not restricting<br>
the <code>target</code> to have only the values <code>0.0</code> and<code>1.0</code>, we admit the<br>
possibility that the <code>target</code> can be probabilistic. So, for example,<br>
a value of <code>0.75</code> would indicate that a pixel is probably a foreground<br>
pixel – 75% likely – but could also be – 25% likely – a background<br>
pixel.</p>
<p>(Using 0 and 255 – as you might have in an 8-bit black-and-white<br>
image – won’t work.  Pytorch’s <code>BCEWithLogitsLoss</code> requires <code>target</code><br>
values in the range <code>[0.0, 1.0]</code> and will give meaningless results for<br>
values outside of this range.)</p>
<p>Best.</p>
<p>K. Frank</p>
How to pad on the left with another matrix while repeating it?

ADONAI_TZEVAOT: <p>I have 3 matrices B (dim:100x100) , A(dim:10x10), C(dim:5x5). I want to pad the left side of B , i.e., concat the left side of B with 10 A’s to match the dimension and also do the same on the right but with 20 C’s instead. Please may I know an efficient way to do this?</p>

googlebot: <p>not sure if anything would work better than partial assignments (with broadcasting):</p>
<pre><code class="lang-auto">D = torch.empty(100,115)
D[:,:10].view(-1,10,10).copy_(A)
#similar for B,C
</code></pre>
Dropout in bidirectional nn.LSTM

Monne: <p>Hi,</p>
<p>i did like to know how dropout placed on bidirectional nn.LSTM, is dropout applied every bidirectional layer or still on RNN layer?</p>
<p>for example if i set 3 layer</p>
<pre><code class="lang-auto">bi_lstm = nn.LSTM(1280, 256, num_layers=3, dropout=0.5, bidirectional=True)
</code></pre>
<p>will it become 5 dropout like</p>
<pre><code class="lang-auto">LSTM_L0_Forward
Dropout_L0_Forward
LSTM_L0_Reverse
Dropout_L0_Reverse
Cat(L0_Forward, L0_Reverse)
LSTM_L1_Forward
Dropout_L1_Forward
LSTM_L1_Reverse
Dropout_L1_Reverse
Cat(L1_Forward, L1_Reverse)
LSTM_L2_Forward
Dropout_L2_Forward
LSTM_L2_Reverse
Cat(L2_Forward, L2_Reverse)
</code></pre>
<p>or 2 dropout</p>
<pre><code class="lang-auto">LSTM_L0_Forward
LSTM_L0_Reverse
Cat(L0_Forward, L0_Reverse)
Dropout_Cat_L0
LSTM_L1_Forward
LSTM_L1_Reverse
Cat(L1_Forward, L1_Reverse)
Dropout_Cat_L1
LSTM_L2_Forward
LSTM_L2_Reverse
Cat(L2_Forward, L2_Reverse)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">LSTM_L0_Forward
LSTM_L0_Reverse
Dropout_L0_Reverse
Cat(L0_Forward, L0_Reverse)
LSTM_L1_Forward
LSTM_L1_Reverse
Dropout_L1_Reverse
Cat(L1_Forward, L1_Reverse)
LSTM_L2_Forward
LSTM_L2_Reverse
Cat(L2_Forward, L2_Reverse)
</code></pre>

Monne: <p>anyone who can answer?</p>
Running mean and running stats

s_n: <p>I have observed that batch normalization parameters such as <strong>running mean</strong> and <strong>running stats</strong> get update after we do the forward pass through the model ( as in just after we do <strong>output=model(input)</strong> ).</p>
<p>So when we train the model and do the evaluation after every epoch is it recommended to put the model in eval mode and then do the inference or not?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>When using batchnorm in training mode, the running stats are always updated yes.<br>
You should be using the eval mode to use these stats and stop updating them when evaluating indeed.</p>

s_n: <p>Thanks for the reply <a class="mention" href="/u/alband">@albanD</a></p>
<p>So when we put the model in eval mode does it use the bn params: running_var, running_mean, gamma, and beta from the training phase?</p>

albanD: <p>Yes, once in eval mode, it uses the saves stats to compute the output.</p>

s_n: <p>So if I want to update some other parameters using my evaluation loss (on validation set) I should first calculate the validation loss with model in eval mode so that running mean and vars are not updated and then shift to train mode to update the additional parameters right?</p>

albanD: <p>It depends if you want to update the statistics when doing these computations or not.</p>

s_n: <p>But if we update these statistics during evaluation we will be kind of providing the information of validation set to train set which would not be recommended I believe.</p>

albanD: <p>Ho if you’re doing regular cross validation, indeed you don’t want to do that.<br>
But that might depend on your application.</p>

s_n: <p><a class="mention" href="/u/alband">@albanD</a> a quick question related to running mean var while the model is in train mode. So if my model is in train mode, does pytorch uses the running mean and var <strong>only</strong> of the current mini batch or does it calculates running mean and var based on previous batches and the current mini batch as well?</p>

albanD: <p>You might want to read the doc to get all the details: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm#torch.nn.BatchNorm2d" class="inline-onebox">BatchNorm2d — PyTorch 1.8.0 documentation</a></p>
<p>But the short version is that during training, it computes the stats on the current batch and use that to compute the output and update the running stats.<br>
During evaluation, it uses the saved running stats to compute the output.</p>
Implementing CNN-LSTM for a denoising task in pytprch

Lenah_Adam: <p>Hi, I am new to this platform but I have found it very useful in implementing my deep learning projects. I was wondering if it possible to implement a CNN-LSTM for a different task other than a classification task for example for image reconstruction or denoising?</p>
<p>Any reply and help would be much appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using main Ram and GPU Ram at the same time

Glerk: <p>Is there a way to have pytorch automatically offload portions of a large model to main RAM and load them back onto GPU ram for computations?</p>

JuanFMontesinos: <p>You can assign where each part of the model is allocated manually. Moving tons of data would be efficient for training. As far as I know it should be complicated and it’s not designed to be done by default. I don’t know if there is a library to do that but I doubt it.</p>

AlphaBetaGamma96: <p>If you’re trying to offload GPU memory to RAM perhaps you might want to have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html?highlight=checkpointing" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.checkpoint — PyTorch 1.8.0 documentation</a> . Although, it’s not exactly what you’re looking for, this might help reduce the amount of memory you need on the GPU!</p>
Stack matrices of repeating nature

ADONAI_TZEVAOT: <p>I want to stack matrices A, B, C,…,C. Here C is repeated 20 times. Please may I know an efficient way to do it?<br>
Please help</p>

iacob: <p>Use <code>repeat</code> and <code>stack</code> e.g. for 2d matrices:</p>
<pre><code class="lang-python">torch.stack([A,B,*C.unsqueeze(0).repeat(20,1,1)])
</code></pre>
How to select top-k values with different k per row

sisaman: <p>Hi, I am looking for a way to select top-k values of the input <code>N x d</code> tensor <code>A</code>, given a <code>N x 1</code> vector <code>K</code> whose i-th element indicates the value of k for the i-th row of <code>A</code>. The output could be a mask tensor with the same shape as <code>A</code>.</p>
<p>If k is constant, we can simply use <code>torch.topk</code>, but I wonder if there is any efficient way to do that with <code>K</code> as a vector?</p>
<p>I tried to find k-th maximum per each row and use it as a threshold, but it does not work as each row could have duplicate values:</p>
<pre><code class="lang-auto">kth_max = A.sort(dim=1, descending=True).values.gather(1, K - 1)
mask = A &gt;= kth_max  # This is wrong: could select more than k values per row
</code></pre>
<p>Any help is really appreciated.</p>
<p><strong>Edit</strong>: I overcome the duplicate values by adding a small random noise to elements of <code>A</code>, but I was wondering if it would be possible to solve this without adding noise?</p>
Efficient way to create a block diagonal?

ADONAI_TZEVAOT: <p>I have a matrix A of dim(3,3) . Using torch.block_diag(A,A,…A) I can create a block diagonal which has 200 blocks of A on the diagonals but this code is very inefficient as I have to carefully type A 200 times . Please can I know if there is an efficient way to achieve this?</p>

iacob: <p>You can achieve this with <code>repeat</code> before calling <code>block_diag</code>:</p>
<pre><code class="lang-python">A_200 = A.unsqueeze(0).repeat(200,1,1)
torch.block_diag(*A_200)
</code></pre>

ADONAI_TZEVAOT: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Problem about using "deepcopy(model)"

WindVChen: <p>I find a weird phenomenon when debugging programs these days !<br>
I want to use “deepcopy(model)”, but when there is any assignment of Class Variable like “self.x = XXX”, the program will break and give the error “Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment”.<br>
When I change the “self.x=XXX” to “x=XXX”, all is OK.<br>
I try to find why it happens but failed.<br>
Any helpful reply are appreciated.</p>
Kernel run on unspecified stream when inception_v3 run

111480: <p>When I inferred from the Inception_v3 model, I found on the profiler that kernel functions of volta_gcgemm_32, void fft1d_r2c_32, void fft1d_c2r_32 work in streams that I did not specify.<br>
How is this possible?</p>

ptrblck: <p>These seem to be cudnn kernels, which are called internally in cudnn and might use different streams.</p>

111480: <p>Thanks for replying.<br>
But, I didn’t find this phenomenon in other kinds of models. So only the Inception_v3 model uses cudnnn?<br>
Is there a way to make those kernels run only on the stream I specified?</p>

ptrblck: <p>It might be kernel-dependent, but if I’m right in my guess, I think you should also see it in more models.<br>
Are you seeing the same kernel name in other models and if so, are these executed in a different way?</p>

111480: <p>The cudnn kernel works on other models, but the kernel with the same name as the Inception does not work.<br>
If your prediction is correct, what should I do to operate those kernels only on the specified stream?</p>

ptrblck: <p>What do you mean by “work”? Is the same kernel called in other models?<br>
If so, how are you profiling it to determine the stream usage of this kernel?</p>

111480: <p>The same kernel as Inception_v3 (ex. void fft1d_r2c_32) is not called from other models.<br>
However, cudnn kernels with different names (ex. volta_scudnn_128x128_relu_interior_v1) are called.<br>
And the kernels called from other models run only on the specified stream.<br>
In both cases, the cudnn kernel are called, why does inception’s cudnn kernel run on unspecified streams but other models’s cudnn kernel run on only specified streams?</p>
Why do i get this error?;Non-empty 3D or 4D input tensor expected but got ndim: 4

ronnie: <p>for example if you have ;</p>
<p>input= torch.rand([0,1,1,66650])<br>
m=nn.MaxPool2d(1,110)<br>
output=m(input1)</p>
<p>why do i get the following error?</p>
<p>RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4</p>

ptrblck: <p>You are creating an empty tensor in:</p>
<pre><code class="lang-python">input = torch.rand([0,1,1,66650])
print(input)
&gt; tensor([], size=(0, 1, 1, 66650))
</code></pre>
<p>by setting the size of <code>dim0</code> to zero.<br>
As the error message suggests, a non-empty tensor would be required.</p>

ronnie: <p>thank you. but in my case i was being stupid to feed my network with more data then required number of batches while iterating. so this error kept coming.</p>
Problem with torchvision 'can not import name' error

Sunetra_Banerjee: <p>Hi,<br>
I am new to pytorch. I am using Windows 10 GPU laptop (with 10.0 Cuda). torchvision is not working in my system. Whenever I run a program, it gives DLL load error or ‘can not import name…’ error. I tried almost every solution found in the interent to solve the problem, but the problem still persists.<br>
I am using python=3.7<br>
pytorch=1.4.0.<br>
Can anybody help me in this regard?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Could you install the latest PyTorch and torchvision releases and check, if you are still facing the same issue?<br>
PyTorch <code>1.4.0</code> is quite old by now and I assume you are also using the corresponding <code>torchvision</code> version.</p>

Sunetra_Banerjee: <p>Hi, Yes I used Pytorch 1.12 and corresponding torchvision but the problem still persists.</p>

ptrblck: <p>The latest PyTorch release is <code>1.8.0</code> so do you mean <code>0.12</code>?<br>
If so, I would highly recommend to update to a newer version <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Maximum between two tensors of different shape

seloufian: <p>Hi,<br>
I have a small issue that I’m struggling with, and for which I didn’t find an adequate solution.</p>
<p>Having two two-dimensional tensors of different shape, I want to get the maximum between each element of the first tensor and all elements of the second tensor. As an example, I define two tensors:</p>
<pre><code class="lang-python">t1 = torch.randn(3, 4)
t2 = torch.randn(5, 4)
</code></pre>
<p>In the code below, I illustrate that I can accomplish this task partially (via broadcasting) using only the first line of <code>t1</code>. That is, each line of <code>t2</code> (which has 4 values) is compared with the first line of <code>t1</code> (which has also 4 values).</p>
<pre><code class="lang-python"># 't1[0]' has shape of (4,) 
# Output will have shape of (5, 4)
torch.maximum(t1[0], t2)
</code></pre>
<p>But what I want is to generalize this concept across all <code>t1</code> lines. I tried the code below, but it doesn’t work:</p>
<pre><code class="lang-python"># Doesn't work, but I want to get an output shape of (3, 5, 4)
torch.maximum(t1, t2)
</code></pre>
<p>Just to be more precise, I want a vectorized version of the code below:</p>
<pre><code class="lang-python"># 'maxs' will contain the maximum between each line of 't1' with 't2'
maxs = []

for line in t1:
  # Current maximum has shape of (5, 4)
  curr_max = torch.maximum(line, t2)
  maxs.append(curr_max)

# 'maxs' will have shape of (3, 5, 4)
maxs = torch.stack(maxs)
</code></pre>

ptrblck: <p>You could also use broadcasting on both tensors:</p>
<pre><code class="lang-python">t1 = torch.randn(3, 4)
t2 = torch.randn(5, 4)

maxs = []

for line in t1:
  # Current maximum has shape of (5, 4)
  curr_max = torch.maximum(line, t2)
  maxs.append(curr_max)

# 'maxs' will have shape of (3, 5, 4)
maxs = torch.stack(maxs)

# use broadcasting
max_broadcast = torch.maximum(t1.unsqueeze(1), t2.unsqueeze(0))
print((maxs - max_broadcast).abs().max())
&gt; tensor(0.)
</code></pre>
Training a neural network to minimize a black-box cost function

sureshj: <p>I have a black box function <code>g(p,q)</code>. It is evaluated using an external stimulator. For any given <code>p</code> , we want to find <code>q</code> that minimizes <code>g(p,q)</code>. I want to model <code>f(p)</code> that generates the <em>optimal</em> values of <code>q</code> from <code>p</code>. I want to model <code>f(p)</code> as a neural network.<br>
I am looking for directions on how to design the training process.<br>
Both <code>p</code> and <code>q</code> are vectors with continuous values.</p>
<p>The closest training process I could think of was the cross-entropy method used in reinforcement learning. But, I think there could be a better way.</p>
Importing COCO format label with segmentation mask

Anupama_Rajkumar: <p>Hello,</p>
<p>I am working on writing my own dataloader with labels provided in COCO json format. The task is to perform semantic segmentation. I would like to return the annotation list (segmentation mask) from the custom class that I have written but the mask for images can be of different size. When I load the dataset and iterate through the loader, I get the following error:<br>
“RuntimeError: each element in list of batch should be of equal size”</p>
<p>I understand that this is because the segmentation mask for each image has different number of vertices.<br>
How may I be able to solve this? is there a work-around?</p>
Concatenate two tesnors

ronnie: <p>i have two tensors, lets say, x=torch.rand([64,8,1,33325]) and x1= torch.rand([64,1,1,33325]) and i want to get an output of my torch.cat to be output.size([64,8,1,66650]) [batch_size, channels, height, width]. i am trying to do things but cannot get it done. anybody can help?</p>

tom: <p>So you want the tensor x1 to be expanded to 64 x <strong>8</strong> x 1 x 33325, right?<br>
This is a case where broadcasting isn’t automatic, but if you do <code>x1 = x1.expand(-1, 8, -1, -1, -1)</code>, the <code>torch.cat([x, x1], dim=-1)</code> should work.</p>
<p>Best regards</p>
<p>Thomas</p>

ronnie: <p>thank you. works like charm</p>
Share dataloader across trainings

davidenitti: <p>Is there a way to share the same dataloader across different trainings?<br>
let’s say I have 2 trainings of 2 different models that share the same data. instead of having 2 processes with independent dataloaders, I would like to have a the same dataloader and 2 processes that uses the same dataloader during training.<br>
is it possible?</p>

Shima_Shahfar: <p>I am sure it is possible if you can synchronize both your trainings somehow and then you can use DDP or DP, but I am not sure I understand you question completely.  Are you training your two models on different processes? Or do you just want a way to use the same dataloader for both trainings?</p>

davidenitti: <p>I know how I can use the same dataloader for both training, to achieve this I could just do the forward and backward of both models in the same loop but I was trying to understand if there was another possibility that is train 2 models on difference processes that share the same dataloader. so have 2 independent trainings (on different processes) but where the dataloader is in some way shared</p>

davidenitti: <p>do you have an example to have 2 independent trainings (on different processes) but where the dataloader is in some way shared?</p>

Shima_Shahfar: <p>I never tried what you are trying to do, maybe others can help you with how to do it.<br>
Though I believe it’s theoretically possible but it would be way more complicated than just using DDP with normal training(if you want to do so the way you initiate the new process seems to be very important).<br>
What I usually do specially in GAN training is that I define my data loader and then I’ll use a training_step function which basically receives a batch in each step. Then I can have separate functions for let’s say my discriminator_step and generator_step, or in some cases I might even have another module that needs to have a separate backward.</p>
Torch.mm and elementwise multiplication, getting unequal outputs

dm23: <p>Hi, I am a beginner in PyTorch, and recently faced an issue. Tried thinking a lot but couldn’t figure it out.</p>
<p>Consider we have two matrices<br>
<span class="hashtag">#take</span> c = 500<br>
x = torch.randn(c,c)<br>
y = torch.randn(c,c)</p>
<p>And we do this :<br>
r1 = torch.sum(x*y,axis=1)</p>
<p>Alternatively, if we instead take transpose of second matrix, and then perform a matrix multiplication and consider only diagonal elements, we should get the same answer.<br>
So, we do this :<br>
r2 = torch.diag(torch.mm(x,y.transpose(0,1)))</p>
<p>If we now compare for equality of the two :<br>
r1==r2</p>
<p>Ideally we should get all True because they are same thing, but I get some False.<br>
What exactly is going wrong? Can someone point out the mistake?</p>

tom: <p>The mistake is to compare floating point variables with <code>==</code> instead of <code>torch.allclose</code>.<br>
What you are seeing is that floating point numbers are not as nice as actual real numbers. For example, addition (of multiple values) does not commute due to numerical precision effects: Try <code>print(1e20 + 1 - 1e20)</code>.</p>
<p>One of my recommendations to get a first impression of whether you hit numerical accuracy is to do the same computation with <code>dtype=torch.double</code> tensors. If the difference (<code>(r1-r2).abs().max()</code>) goes down dramatically (say from 1e-5ish to 1e-13ish in your example), this points to numerical accuracy issues. If it stays the same, this might point to a different issue (if only an approximation somewhere in the computation).</p>
<p>Best regards</p>
<p>Thomas</p>