akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Print data of a 0-dim tensor

username: jerry0098
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I am a PyTorch fresher, I am trying my best to avoid these stupid questions :)))</p>
<p>I got a 0-dim tensor like this: tensor(69, device=‘cuda:0’)</p>
<p>I just want to print “69” instead of the whole one "tensor(69, device=‘cuda:0’ "</p>
<p>How can I do it? Can someone please to help me?, I searched a lot however I do not know the keyword of the problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: harsha_g
cooked: <pre><code class="lang-auto">a = torch.tensor(69, device='cuda:0')
print(a.item())
</code></pre>
Constrain parameters to satisfy an equation in Pytorch

username: ZeroMax
cooked: <p>Hello,</p>
<p>I’m trying to compute probabilities using parametric probability amplitudes in Pytorch for a variant of Dropout. I’ve implemented all the necessary complex arithmetic using the below code. However, my issue is that I need to constrain my tensors to satisfy the following equation: (1=(abs(x)**2)+(abs(y)**2), and I don’t know how I could do this in Pytorch.</p>
<p>My code:</p>
<pre><code class="lang-auto">class Complex(nn.Module):
    def __init__(self):
        super(Complex, self).__init__()
        self.real_param_a = nn.Parameter(torch.ones(1,1))
        self.imag_param_a = nn.Parameter(torch.ones(1,1))
    def abs_square(self):
        return (self.real_param_a ** 2) + (self.imag_param_a ** 2)

class QDropout(nn.Module):
    def __init__(self):
        super(QDropout, self).__init__()
        self.complex_a = Complex()
        self.complex_b = Complex()
    def forward(self, x):
        probability_one = self.complex_a.abs_square()
        probability_two = self.complex_b.abs_square()
</code></pre>
AttributeError: 'int' object has no attribute 'numpy' while trying to get torchsummary

username: shatadru_majumdar
cooked: <p>I am building a model in pytorch<br>
When I try to do:<br>
from torchsummary import summary<br>
summary(model.to(device),((3,224,224)),10)<br>
it gives:<br>
AttributeError                            Traceback (most recent call last)<br>
 in ()<br>
1 import numpy<br>
2 from torchsummary import summary<br>
----&gt; 3 summary(SegNet2(3).to(device),((3,224,224)),10)</p>
<p>/usr/local/lib/python3.6/dist-packages/torchsummary/torchsummary.py in summary(model, input_size, batch_size, device)<br>
100     total_input_size = abs(np.prod(input_size) * batch_size * 4. / (1024 ** 2.))<br>
101     total_output_size = abs(2. * total_output * 4. / (1024 ** 2.))  # x2 for gradients<br>
–&gt; 102     total_params_size = abs(total_params.numpy() * 4. / (1024 ** 2.))<br>
103     total_size = total_params_size + total_output_size + total_input_size<br>
104</p>
<p>AttributeError: ‘int’ object has no attribute ‘numpy’</p>
<p>Even though my model.parameters list contains 2 elements: A '‘last_conv_block.weight’<br>
param and a ‘last_conv_block.bias’ param both with requires_grad = True</p>
Torchvision.models vs utils.model_zoo

username: codeknight11
cooked: <p>I am going through someone’s project and noticed that they are downloading resnet models using torch.utils.model_zoo module.</p>
<p>I have only used torchvision.models so far<br>
What is the difference between loading pretrained models via model_zoo vs torchvision.models?</p>

username: ptrblck
cooked: <p>The <code>model_zoo</code> was moved to the <code>torch.hub</code>, if I’m not mistaken.<br>
<code>torch.hub</code> is an easy way to load pretrained models from public GitHub repositories and you can find the currently supported models <a href="https://pytorch.org/hub/research-models">here</a>.<br>
For a resnet you could either use the <code>torchvision</code> implementation or load these models from the hub, which should use the same checkpoint and architectures.<br>
One advantage of the hub is that you can easily add support for your current repository and share your model easier than trying to add your current model to e.g. <code>torchvision</code>.</p>
How do I derive weights for CrossEntropy Loss on my custom dataset?

username: sarmientoj24
cooked: <p>I have five classes + 1 background for my semantic segmentation. My mask is like the one below:<br>
It has a ton of background and classes are rarely seen on both image.<br>
<a href="https://user-images.githubusercontent.com/8830319/84572938-10825080-add0-11ea-9285-0082814edb94.png" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/8830319/84572938-10825080-add0-11ea-9285-0082814edb94.png" alt="download (2)" width="366" height="252"></a></p>
<p>I’ve seen codes using CrossEntropyLoss with designated weights but usually they are using available datasets such as CityScapes so they have the weights on them.</p>
<p>How do I derive CrossEntropyLoss weights for my dataset?</p>

username: Krish
cooked: <p>This should work:<br>
<code>(Average number of pixels per image) / (Average number of pixels of a class per image)</code></p>
<p>Haven’t worked a lot on segmentation, so take the suggestion lightly</p>

username: sarmientoj24
cooked: <p>are weights in the losses BCE and CE designated PER minibatch? or is it computed on the whole dataset and assigned before the training?</p>

username: Krish
cooked: <p>Usually, it is computed per dataset.</p>
<p>You can do per batch as well, but that would result in a very noisy training process, since the class priorities for the network might change per batch.</p>
Torch.cat() and nn.Linear()

username: gquenot
cooked: <p>In an <code>nn.module</code>, with:</p>
<pre><code class="lang-auto">        self.fc1a = nn.Linear(512+2*num_attrs, 512)
        self.fc1b = nn.Linear(512, 512)
        self.fc1c = nn.Linear(2*num_attrs, 512, bias=False)
</code></pre>
<p>I would expect both</p>
<pre><code class="lang-auto">        xa = F.relu(self.fc1a(torch.cat((x, 0.0*xb), 1)))
</code></pre>
<p>and</p>
<pre><code class="lang-auto">        xa = F.relu(self.fc1b(x)+self.fc1c(0.0*xb))
</code></pre>
<p>to give the same results as</p>
<pre><code class="lang-auto">        xa = F.relu(self.fc1b(x))
</code></pre>
<p>but only the latter gives the same result while the former consistently leads to a performance degradation. The two statements involves (currently) unused parameters in the same way. Is there something wrong with the first statement? (x and xb are respectively of size 512 and 2*num_attrs in the non-batch dimension). Pytorch version is 1.5.0 and the optimizer is adam.</p>

username: googlebot
cooked: <p>if your num_attrs is big, fc1a default weight initialization may be the issue, as its output variance is reduced comparing to fc1b</p>

username: gquenot
cooked: <p>Good point, thanks. I missed that adding channels decreased the fan-in. However, <code>num_attrs</code> is <code>312</code> so <code>math.sqrt((512+2*num_attrs)/512)</code> is just a bit less than <code>1.5</code>. Trying:</p>
<pre><code class="lang-auto">        self.fc1a = nn.Linear(512+2*num_attrs, 512)
        with torch.no_grad():
            self.fc1a.weight *= math.sqrt((512+2*num_attrs)/512)
            self.fc1a.bias *= math.sqrt((512+2*num_attrs)/512)
</code></pre>
<p>still did not give the same result as in the two other cases. I eventually figured out where the difference was coming from while trying this though (fc1b was used elsewhere and combined gradients helped so not using it here had a negative impact). Thanks.</p>
CNN default initialization understanding

username: Kaixuan_WANG
cooked: <p>Hi, I observed that the default CNN initialization has been changed.<br>
In version 1.0 and above:</p>
<pre><code class="lang-auto"> def reset_parameters(self):
        n = self.in_channels
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)
</code></pre>
<p>However, in 0.4 versions:</p>
<pre><code class="lang-auto">    def reset_parameters(self):
        n = self.in_channels
        for k in self.kernel_size:
            n *= k
        stdv = 1. / math.sqrt(n)
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            self.bias.data.uniform_(-stdv, stdv)
</code></pre>
<p>What the motivation of these changes and how to understand</p>
<pre><code class="lang-auto">init.kaiming_uniform_(self.weight, a=math.sqrt(5))
</code></pre>
<p>Why a=sqrt(5)<br>
Thanks for your time!</p>

username: harsha_g
cooked: <p>Probably <a href="https://discuss.pytorch.org/t/why-the-default-negative-slope-for-kaiming-uniform-initialization-of-convolution-and-linear-layers-is-5/29290">this</a> should help.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I think the motivation has been explained in the link <a class="mention" href="/u/harsha_g">@harsha_g</a> has referenced. But for more clarification, I have also explained how to get the values especially <code>sqrt(5)</code> in this post: <a href="https://discuss.pytorch.org/t/clarity-on-default-initialization-in-pytorch/84696/2" class="inline-onebox">Clarity on default initialization in pytorch</a></p>
<p>But if I want to summarize, both do the same thing but older versions of PyTorch have the convention used in Lua Torch and after a while that some modules in PyTorch have been constructed, they tried to achieve same concept using the new modules. For instance, in your example, both approchs sample from uniform distribution which can be achieved by <code>kaiming_uniform_</code> to which I have explained in the referenced link.</p>
<p>Bests</p>
Weighted Element Wise Summation

username: ashfaque
cooked: <p>Hi all. I have 3 different convolution blocks each with channel number 64. I would like to make an element wise summation with trainable weights for each of the convolution blocks, i.e.</p>
<p>let conv_1 , conv_2 and conv_3 be the convolution blocks.</p>
<p>conv_final = lambda_1 * conv_1 + lambda_2* conv_2 + lambda_3* conv_3 (+ here means element wise summation)</p>
<p>I want to train these lambda’s while i train my whole CNN using the loss function. Can somebody please help me in implementing this?</p>

username: googlebot
cooked: <p>Such things are hard to optimize. Consider scalar equation <code>x1*w1+x2*w2=y</code>, even restricting weights to a convex combination (0&lt;w1&lt;1, w2=1-w1), you could fit (x1,x2) for ANY w1. And this problem remains when x1,x2 are vectors in training; without some additional constraint, gradient descent will find w1 that is good for initial x1,x2 vectors and may stick to it.</p>
<p>Ensembles solve this with multiple losses, for x1=y, x2=y, x1<em>w1+x2</em>w2=y, i.e. parts must predict y independently, and ensemble is usually trained later, with contributing models possibly frozen.</p>
<p>That being said, doing weighted sum itself is pretty straightforward. Assuming global per-channel combination weights, create a parameter:</p>
<pre><code class="lang-auto">self.raw_lambda = nn.Parameter(torch.zeros(NumParts,NumChannels,1,1)) #for conv2d
</code></pre>
<p>In forward(), transform it (as you never want any lambda to become zero)</p>
<pre><code class="lang-auto">lambda = torch.softmax(self.raw_lambda,0)
</code></pre>
<p>Now, just use arithmetic ops and slices, e.g.:</p>
<pre><code class="lang-auto">conv_final = lambda[0] * conv_1 + lambda[1] * conv_2
</code></pre>
Torch.bmm Doubt

username: Krish
cooked: <p>I am trying to multiply a 2 dimensional matrix with a 4 dimensional tensor, but torch.bmm doesn’t seem to allow 4-dim tensors.<br>
This is the code I am using:</p>
<pre><code class="lang-auto">batch_size = 40
max_len = 96

logits = torch.randn(batch_size, 3)
outs = torch.randn(batch_size, 3, max_len, 2)

# Shape of desired output: (batch_size, max_len, 2)
result = torch.bmm(logits, outs)
</code></pre>
<p>The error is:</p>
<pre><code class="lang-auto">RuntimeError: Expected 3-dimensional tensor, but got 4-dimensional tensor for argument #1 'batch1' (while checking arguments for bmm)
</code></pre>
<p>Is this possible any way?</p>

username: Chirath_Pansilu
cooked: <p>No It is not possible .<a href="https://pytorch.org/docs/master/generated/torch.bmm.html#torch-bmm" rel="noopener nofollow ugc">Refer documentation</a></p>
<aside class="quote no-group">
<blockquote>
<p><code>input</code>  and  <code>mat2</code>  must be <strong>3-D tensors each containing the same number of matrices</strong>.</p>
</blockquote>
</aside>
<p>It is clearly documented there.</p>

username: Krish
cooked: <p>Yeah I read it, but is there any kind of workaround?</p>
Output of RoBERTa (huggingface transformers)

username: Romina_Baila
cooked: <p>Hello,<br>
I am trying to better understand how RoBERTa model (from huggingface transformers) works.</p>
<p>My<code> batch_size</code> is 64</p>
<p>My roberta model looks like this</p>
<pre><code class="lang-auto">roberta = RobertaModel.from_pretrained(config['model'])
roberta.config.max_position_embeddings = config['max_input_length']
print(roberta.config)
</code></pre>
<pre><code class="lang-auto">RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": "1024",
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 50265
}
</code></pre>
<p>And in my model, in forward() method I have tried:</p>
<pre><code class="lang-auto">embedded , x = self.roberta(text)
</code></pre>
<p>Now, from what I read in the documentation and source code from huggingface, the output of <code>self.roberta(text)</code> should be</p>
<blockquote>
<p>prediction_scores ( <code>torch.FloatTensor</code> of shape <code>(batch_size, sequence_length, config.vocab_size)</code> )</p>
</blockquote>
<p>(also checking the source code I came accross this:<br>
<code>outputs = (prediction_scores,) + outputs[2:]  # Add hidden states and attention if they are here</code>)<br>
From my understanding, I should get only <em>one</em> output, embedded, which should have the following shape: <code>torch.Size([64, 1024, 50265]</code>. Instead, I am getting 2 Tensors, embedded <em>and</em> x, with the following shapes:</p>
<pre><code class="lang-auto">torch.Size([64, 1024, 768])
torch.Size([64, 768])
</code></pre>
<p>I have checked and both <code>roberta.config.output_hidden_states</code> <code>roberta.config.output_attentions</code> are false.</p>
<p>So my questions are why am I getting 2 outputs, why do they look like that and what do they represent?</p>

username: Krish
cooked: <p>The first one is basically the output of the last layer of the model (can be used for token classification).<br>
The second one is the pooled output (can be used for sequence classification).</p>
Loss backward throws RuntimeError

username: ekosman
cooked: <p>Hello,</p>
<p>I have a problem that might have a very simple solution but it bugs me for a few hours now.</p>
<p>I have several models defined as:</p>
<pre><code class="lang-python">class Model(nn.Module):
	def __init__(self, base):
		super(Model, self).__init__()
		self.base = base
		self.coeff = nn.Parameter(torch.randn(len(base)), requires_grad=True)

	def forward(self, x):
		res = []
		for x_i in x:
			res.append(torch.tensor([c*f(x_i) for c, f in zip(self.coeff, self.base)], requires_grad=True).sum())
		return torch.stack(res)

</code></pre>
<p>I use a custom loss function, defined as:</p>
<pre><code class="lang-python">class RegressionLoss(nn.Module):
	def __init__(self):
		super(RegressionLoss, self).__init__()

	def forward(self, a, p, y_, y, q, w):
		return (w * a * p * torch.abs(y_ - y) ** q).mean()

</code></pre>
<p>All the models and the loss modules sit in seperate lists, i.e. I have 2 lists names models[] and loss_fn[]. I also have a list of LBFGS optimizers, named optimizers[].</p>
<p>I call the following function in a loop:</p>
<pre><code class="lang-python">	def lbfs_KHM(self, k, a_values, p_values, x, y, w, q=2):
		def closure():
			self.optimizers[k].zero_grad()
			output = self.models[k](x)
			loss = self.loss_fn[k](a_values, p_values, output, y, q, w)
			loss.backward()
			return loss

		self.optimizers[k].step(closure)
</code></pre>
<p>Altough I assumed this will create different graphs for different models and losses, I get the following error when calling loss.backward:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>

username: ekosman
cooked: <p>Well, apparently it seems like the problem occured because I used the models for calculating the tensors <code>a_values</code> and <code>p_values</code>. Calling <code>.detach()</code> on both solved the problem!</p>
Torch Multiprocessing Issues - Only works with classmethod

username: kleingeo
cooked: <p>I have been struggling with an issue regarding torch multiprocessing. Currently, I have a whole training pipeline made up of three classes: a Trainer which defines how training should be accomplished, and two model classes. The first is a base_model class which defines a lot of the boiler-plate functions (training loops, saving, loading, etc.) and the second is the model architecture class which inherits the base class as its parent. I have been trying to implement a distributed training method in the base_class, but have keep getting this error: <code>ValueError: bad value(s) in fds_to_keep</code>.</p>
<p>Edit: I have found that the problem stems from the fact that when my architecture is built I take the state_dict of the model. When state_dict is changed to <code>net.state_dict(keep_vars=True)</code>, it works. Why would this cause the issue?</p>
Sampling from Categorical distribution based on values

username: Andrei_Ungureanu
cooked: <p>Hello peeps, I am trying to implement the Reinforce algorithm for sequence-to-sequence modeling. For this, I need to get a baseline [greedy] distribution and a sampled distribution with probabilities.</p>
<p>When it comes to the sampled distribution, at each decoding step I am doing the following to sample a token and its probability:</p>
<pre><code class="lang-auto">token_1 = torch.gather(log_probs, 1, indexes)
multi_dist = Categorical(token_1)
x_t = multi_dist.sample()
log_prob = multi_dist.log_prob(x_t)
</code></pre>
<p>However, my model has a large action space, and I would like to sample a token from a limited space. Is there any way I can either sample a token based on their probabilities/values or mask the tensor I am passing to Categorical?</p>

username: FabianSchuetze
cooked: <p>Did you try passing logits = -inf to mask tokens you don’t want to sample?</p>
Post-Training Static Quantization for Last FC layer gets rubbish results

username: juanmanpr
cooked: <p>Hi,</p>
<p>I have this model that I have statically quantized in two manners: 1) feature extractor only and 2) feature extractor + classifier. I get adequate results with the former, but I get rubbish with the later.<br>
I do not know where to start looking, does anyone have any idea what could be going wrong?<br>
It seems when I quantize the classifier, there is very little variation in the classification output.<br>
Validation error before quantization for both cases is good, but it goes down to almost 0 when I quantize the classifier.</p>
<p>Thanks!</p>
Relation between num_workers and batch size in DataLoader?

username: shahnawaz
cooked: <p>I understand that num_workers in dataloader is for multi-threaded independent batching. Then why is it recommended that this value should not be more than batch size for efficiency? What num_workers has to do with the batch size?</p>

username: cskarthik7
cooked: <p>You can refer to this <a href="https://discuss.pytorch.org/t/relation-between-num-workers-batch-size-and-epoch-in-dataloader/18201">thread</a>.</p>
Difference between torch.Tensor and torch.empty?

username: ToruOwO
cooked: <p>Both of them seems to return an uninitialized tensor with specified size</p>

username: albanD
cooked: <p>Hi,</p>
<p>torch.empty only returns uninitialized Tensor, torch.Tensor has more arguments: for examples, you can give it a python list and it will build a Tensor from it.</p>

username: ToruOwO
cooked: <p>Sorry I didn’t make the question clear – what I was trying to ask is actually, are there any differences between these two APIs if we just want to create an uninitialized tensor?</p>

username: albanD
cooked: <p>Ho, in that case no there is no difference. For code clarity, you should probably use empty as it is clearer that you want and get a uninitialized Tensor.</p>

username: ToruOwO
cooked: <p>I see, thanks for answering!</p>

username: Brando_Miranda
cooked: <p>Can you clarify what “uninitialized” means?</p>
<p>Perhaps I am being pedantic, but I see the implementation returns small numbers rather than zero’s. Is the semantics of uninitialized intended to be zero?</p>
<p>I guess what I am really after is why it’s not <strong>exactly</strong> zero.</p>
<p>I printed some outputs and it was more confusing than clarifying</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.empty(3, dtype=torch.long)
tensor([     140187750138624,      140187750063648, -5764607523034234880])
&gt;&gt;&gt; torch.empty(3)
tensor([0.0000e+00, 1.4013e-45, 0.0000e+00])
</code></pre>
<p>Questions:</p>
<ol>
<li>the second one is trying to be zero. Why can’t it be exactly zero? I thought it was possible to represent zero exactly in float/double etc.</li>
<li>why is the first one nothing resembling zero? What is the intended semantics there?</li>
</ol>

username: ptrblck
cooked: <p>Uninitialized means that the memory will be allocated and the corresponding tensor will represent its values using the bits which were already in the memory block before it was allocated.<br>
This means the values should never be used without a proper initialization, as they might represent “everything” (valid values, NaN, Inf).</p>
<p>If you want to initialize the tensor with zeros, use <code>torch.zeros</code>.</p>
<p>The advantage of <code>torch.empty</code> is that it’s faster, because it will just assign the memory to the tensor without filling it with specific values. This is often used, if you need a buffer, which is guaranteed to be filled with other values later (or initialized somehow).</p>
Different behavior of Normal and MultivariateNormal for 1D

username: markaa
cooked: <p>Use torch.distributions.Normal</p>
<pre><code class="lang-auto">torch.manual_seed(10)

mu = torch.randn(1)
sigma = torch.rand(1)

z = torch.distributions.Normal(mu, sigma)

z.log_prob(torch.Tensor([1.]))
</code></pre>
<p>Output is:</p>
<blockquote>
<p>tensor([-27.3894])</p>
</blockquote>
<p>Use torch.distributions.MultivariateNormal</p>
<pre><code class="lang-auto">torch.manual_seed(10)

mu = torch.randn(1)
sigma = torch.rand(1)

z = torch.distributions.MultivariateNormal(mu, covariance_matrix=torch.diagflat(sigma))

z.log_prob(torch.Tensor([1.]))
</code></pre>
<p>Output is:</p>
<blockquote>
<p>tensor(-6.1411)</p>
</blockquote>
<p>I expected to get the same result for both cases. What’s wrong?</p>

username: ptrblck
cooked: <p>I’m no expert in distributions, but from the docs <code>covariance_matrix</code> should be a positive-definite covariance matrix, which would not be the case for:</p>
<pre><code class="lang-python">torch.diagflat(sigma)
&gt; tensor([[-1.0122]])
</code></pre>
<p>Also, shouldn’t the covariance matrix be the variance in this case (which would also make it positive definite)?</p>
<p>This should yield the same result:</p>
<pre><code class="lang-python">z = torch.distributions.MultivariateNormal(mu, covariance_matrix=torch.diagflat(sigma**2))
</code></pre>
How to get the index of a element in a Tensor whose value is True?

username: songyuc
cooked: <p>Hi guys,<br>
I meet a problem that, how to get the index of a element in a Tensor whose value is True?<br>
Such as, a Tensor like:<br>
False   False   Fasle<br>
Flase   True    False<br>
False   False   False<br>
Then I can get the index of (1,1).<br>
So how can I implement this operation?</p>
<p>Your answer and idea will be appreciated!</p>

username: ptrblck
cooked: <p><code>nonzero()</code> would return you the indices of all non-zero entries (in that case <code>True</code>):</p>
<pre><code class="lang-python">x = torch.bernoulli(torch.ones(3, 3) * 0.5).bool()
print(x)
&gt; tensor([[ True,  True, False],
        [False, False,  True],
        [ True, False, False]])
print(x.nonzero())
&gt; tensor([[0, 0],
        [0, 1],
        [1, 2],
        [2, 0]])
</code></pre>

username: songyuc
cooked: <p>Thanks sincerely for your answer and guide!</p>
How does detach() work?

username: Gautam_Bhattacharya
cooked: <p>Hello,<br>
In the GAN example, while training the D-network on fake data there is the line:<br>
output = netD(fake.detach())</p>
<p>Q. What is the detach operation doing?<br>
Q. This operation is not used in the Wasserstien GAN code. Why is it not needed in this model?<br>
Q. Is the same effect being obtained by:<br>
noisev = Variable(noise, volatile = True) # totally freeze netG</p>
<p>Thanks in advance,<br>
Gautam</p>

username: apaszke
cooked: <p>A simple search in the forums should be enough to answer this question. Here’s an <a href="https://discuss.pytorch.org/t/help-clarifying-repackage-hidden-in-word-language-model/226/2?u=apaszke">example answer</a>.</p>

username: Elias_Vansteenkiste
cooked: <p>After reading the thread you referenced, it is not completely clear for me what the answer to the second question would be. Could you further answer question 2?<br>
Thanks in advance,</p>

username: htt210
cooked: <p>Hi Elias,<br>
I think it’s just author’s preference. In Wasserstein implementation, they used<br>
<code>x_new = Variable(x_old.data, ...)</code><br>
to perform <code>detach</code> operation. In my opinion, the new variable name makes it easier to read.</p>

username: jadechip
cooked: <p>To my understanding, detach disables automatic differentiation, i.e stops keeping track of gradients.<br>
Further information: <a href="http://www.bnikolic.co.uk/blog/pytorch-detach.html" rel="nofollow noopener">http://www.bnikolic.co.uk/blog/pytorch-detach.html</a></p>

username: Himani_Gulati
cooked: <p>PyTorch keeps track of all operations that involve tensors and these operations are tracked/recorded as a directed graph. (Where edges have a direction associated with them).</p>
<blockquote>
<p>detach()<br>
creates a new view such that these operations are no more tracked i.e gradient is no longer being computed and subgraph is not going to be recorded.<br>
Hence memory is not utilized.<br>
So its helpful while working with billions of data.</p>
</blockquote>
Best affordable GPU/TPU for advance or intermediate deep learning?

username: JonathanSum
cooked: <p>I have been using google colab pro with their Tesla p100 like 16GB of ram. But I feel it is only like 60 to 70% of it. In addition, I often need to train something that takes like 24 hours or more. To me, colab pro only last about 12 hours or 16 hours. If it is unlucky, which I did not save the trained model during 50% of training, I will need to retrain everything again. Thus, I think the more realistic way is buy a GPU or TPU and build a new desktop for training, but it is not for gaming. I hope I can have a good suggestion since this cost me a lot. I can afford like 100 to 550 US dollars (350 is the best) for GPU or TPU.</p>
<p>For CPU, I will use AMD 3300X or below since CPU is not really a big deal in deep learning.</p>
<p>For Nvidia, I feel most of their GPUs are about gamming rather have an affordable TPU or deep learning GPU.</p>
<p>I have been google searching online on this question, but most of them suggest buying an RTX like 700-1000 US dollars. I am not that rich, but I hope I can do an intermediate research level or maybe training NPL like BERT(Maybe).</p>
<p>For the cloud, many people say using google could for one month or one day can cost 100 dollars.<br>
I have been training model on satellite images with 720p or above and Japanese anime coloring that really requires a good GPU. Even though the suggested GPU is not as good as google GPU Tesla P100, I can at least train it more than 24 hours.</p>
How to remove the intermediate layers of model

username: silvester
cooked: <p>Dear experienced ones:</p>
<p>I wonder how to remove the intermediate layers of a model. I know about this method below:</p>
<pre><code class="lang-auto">nn.Sequential(*list(model.children())[:-2])
</code></pre>
<p>The problem that my model has pooling layers and ReLU layers which are applied in the <code>forward()</code> method of the model and those modules would be missed using <code>children().</code></p>

username: ptrblck
cooked: <p>The cleanest approach would be to derive a custom model class from the model class as its base and override the <code>forward</code> method with your new definition.<br>
I would not recommend to try to wrap submodules into an <code>nn.Sequential</code> container for models, which might be more complicated than a few sequential layers.</p>

username: silvester
cooked: <p>Thanks for the hints.<br>
I need to make the process automated and general, no matter what the model is.<br>
Is there anyway to figure out if the model uses softmax in the <code>forward</code> method?<br>
Still it is not clear how to remove that layer while keep the operators in the <code>forward</code> exactly same as original model.</p>

username: ptrblck
cooked: <p>You could “remove” <code>nn.Module</code>s by replacing them with <code>nn.Identity</code>.<br>
However, this won’t work for functional calls in your model, since (as you already explained), these methods are not registered as modules, so you would need to override the <code>forward</code>.</p>
Screenshot to RGB

username: Lucid
cooked: <p>Hello. Any solution?<br>
When I write pil instead of pytorch (line19), it does not give an error, but the pil is slower. I want to use pytorch.</p>
<p><strong>My Code</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fda43620483af0183e2172d3d6599752b8eb62eb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fda43620483af0183e2172d3d6599752b8eb62eb" title="2020.06.12_007_"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fda43620483af0183e2172d3d6599752b8eb62eb_2_690x499.jpeg" alt="2020.06.12_007_" data-base62-sha1="AbOCwCDft9J3xObTz1zelm6k9v5" width="690" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fda43620483af0183e2172d3d6599752b8eb62eb_2_690x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fda43620483af0183e2172d3d6599752b8eb62eb_2_1035x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/d/fda43620483af0183e2172d3d6599752b8eb62eb.jpeg 2x" data-dominant-color="2E2E29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020.06.12_007_</span><span class="informations">1146×829 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Errors</strong></p>
<pre><code class="lang-auto">Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\1700x\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\1700x\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "F:\screenss.py", line 89, in main
    check4()
  File "F:\screenss.py", line 33, in check4
    rgb = im.convert('RGB')
AttributeError: 'Tensor' object has no attribute 'convert'
</code></pre>

username: ptrblck
cooked: <p>PyTorch doesn’t provide any image decoding methods and torchvision uses <code>PIL</code> by default.<br>
You could try to use OpenCV instead of <code>PIL</code> or alternatively <code>PIL-SIMD</code>, which might speedup your code.</p>
Convert .json to mask

username: fgauthier
cooked: <p>Hi, I am new to Pytorch and I was wondering if there is a way to convert a .json file in a numpy array. I want to do this because I want to convert my .json in a mask.<br>
Right now I have this</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/ded0a98e6c030ce10b38ca4de8ba5663380c76ec.png" alt="image" data-base62-sha1="vN73r1PmfIRuul734o974N9A1xq" width="675" height="361"></p>

username: ptrblck
cooked: <p>Depending how the data is stored in the <code>json</code> object, you might be able to directly create a numpy array.<br>
Your current code doesn’t seem to show the <code>json</code> file.<br>
This question doesn’t seem to be PyTorch-specific, so you might get a better answer on e.g. StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Torch.cat error

username: rnn_lstm
cooked: <p><strong>This snippet works fine.</strong><br>
aa = torch.FloatTensor(torch.randn(1,1,256))<br>
bb = torch.FloatTensor(torch.randn(1,5332,256))<br>
cc = torch.cat((aa,bb),1)<br>
cc.shape<br>
(1, 5333, 256)<br>
type(cc)<br>
&lt;class ‘torch.Tensor’&gt;<br>
type(aa)<br>
&lt;class ‘torch.Tensor’&gt;<br>
type(bb)<br>
&lt;class 'torch.Tens</p>
<p><strong>However this one  throws error</strong> :</p>
<p>embeddings = self.embed(captions)<br>
pdb.set_trace()<br>
embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)</p>
<p>Debugging the type and shapes:</p>
<p>(Pdb) embeddings.shape<br>
torch.Size([1, 5332, 256])<br>
(Pdb) features.shape<br>
torch.Size([1, 256])<br>
(Pdb) type(embeddings)<br>
&lt;class ‘torch.Tensor’&gt;<br>
(Pdb) type(features)<br>
&lt;class ‘torch.Tensor’&gt;<br>
(Pdb) embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)<br>
*** RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524580938250/work/aten/src/THC/THCCachingHostAllocator.cpp:257<br>
(Pdb)</p>
<p>Can’t understand why? Is it because one of the inputs can be a variable ?</p>

username: lx385095967
cooked: <p>Have you solve this problem?<br>
I face the same mistake like you.</p>

username: ptrblck
cooked: <p>I would recommend to run the code on the CPU first to get a better error message.<br>
If the code only raises the error using the GPU, run the script via:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here, please.</p>
Complexity of Softmax in pytorch

username: ruchitpatel
cooked: <p>How is softmax implemented in pytorch? The time complexity of the softmax function dependes on the number of classes <code>O(N)</code>. However, in my task, the number of classes are huge (in the order of hundred thousand). Should I go with the builtin softmax function in pytorch? If not, what are the other options?</p>

username: ptrblck
cooked: <p>I would check, if this method is the bottleneck for your current approach and then check alternatives.<br>
While a large number of classes might slow down the method, it isn’t necessarily the bottleneck, so your optimizations might not help.</p>
Sample balanced data

username: NearIt
cooked: <p>i am using</p>
<pre><code class="lang-auto">FashionMNIST_train_dataset = torchvision.datasets.FashionMNIST(
    root = './data/FashionMNIST',
    train = True,
    download = True,
    transform = transforms.Compose([
        transforms.ToTensor()                                 
    ])
)
</code></pre>
<p>and I want SPLIT this data to N balanced labeled data, and the rest (60000-N).<br>
how can I do that?</p>

username: ptrblck
cooked: <p>You could use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html"><code>sklearn.model_selection.StratifiedShuffleSplit</code></a> (or any other suitable splitting function) to create the split indices.<br>
Once you have the indices, you could pass your dataset together with the split indices to a <code>Subset</code> and pass it then to a <code>DataLoader</code>.</p>

username: NearIt
cooked: <p>this is what i did:</p>
<pre><code class="lang-auto">def GetIndices (data,N):
  # given data, divide it to N balanced labels samples.
  # return
  
  Nclasses = len(data.classes)
  Ndata = len(data.data)
  count = [[] for x in range(Nclasses)]   #init. count contain the idx of each label                                               
  for idx, (_,label) in enumerate(data):
    count[label].append(idx)
  # this is to check that this is balance label data 
  #for i in range(len(count)):
  #  Nsamples= len(count[i])
  #  print("label " + str(i) + " has " + str(Nsamples) + " images") 

  #divide
  NsampleToTakeFromEachLabel = math.floor(N / Nclasses)
  #print("total data len = " + str(Ndata) + ",  Nclasses = " +  str(Nclasses) + ",  N=" +str(N) + ",  NsampleToTakeFromEachLabel = " +str(NsampleToTakeFromEachLabel))
  picked_indices=[];
  not_picked_indices = [];
  for i in range(len(count)):
    picked_idx = random.sample(count[i],NsampleToTakeFromEachLabel);
    for j in range(len(count[i])):
      if count[i][j] in picked_idx:
        picked_indices.append(count[i][j])
      else :
        not_picked_indices.append(count[i][j])
  
  return picked_indices , not_picked_indices

def DivideData (train_dataset,test_dataset,N):
  picked_indices , not_picked_indices = GetIndices(train_dataset,N)

  trainloader = torch.utils.data.DataLoader(torch.utils.data.Subset(train_dataset, picked_indices))
  testloader1 = torch.utils.data.DataLoader(torch.utils.data.Subset(train_dataset, not_picked_indices))
  testloader2 = torch.utils.data.DataLoader(test_dataset, shuffle=True)

  return trainloader , testloader1 ,testloader2

trainloader , testloader1 ,testloader2 = DivideData(FashionMNIST_train_dataset,FashionMNIST_test_dataset,100) 

</code></pre>
<p>any idea how to concat testloader1  with testloader2</p>

username: ptrblck
cooked: <p>You could use <code>ConcatDataset</code> on the <code>Subset</code> and <code>test_dataset</code> and pass this concatenated dataset to a <code>DataLoader</code>.</p>
Cross Entropy Loss vs. Mutual Information and Generalized Mutual Information

username: amir_omidi
cooked: <p>Hi,<br>
I am working on a project with binary inputs and outputs and want to apply a loss function. in similar works <strong>cross entropy</strong> and <strong>mutual information</strong> and <strong>generalized mutual information</strong> are considered as cost function. (MI and GMI are not loss functions and I think some changes are applied before use). I want to know the mathematical difference between these three and how to use them as cost function in <strong>pytorch</strong>.</p>