akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Self-defined submodule parameters are not registered

username: Bohan_Zhuang
cooked: <p>class nonlinear(nn.Module):</p>
<pre><code>def __init__(self):
	super(nonlinear, self).__init__()
    self.scale_container = []
    for ii in range(10):
        self.scale_container.append(nn.Parameter(torch.FloatTensor([1.0]).cuda(), requires_grad=True))

def forward(self, x):
	output = None
    for ii in range(10):
        scale = self.scale_container[ii].expand_as(x)
        if output is None:
            output = (F.tanh(scale * x) + 1.0) / 2.0
        else: 
    	output += (F.tanh(scale * x) + 1.0) / 2.0
    return output
</code></pre>
<p>class network(nn.Module):</p>
<pre><code>def __init__(self, bitW, bitA):
    super(network, self).__init__()
def forward(self, x):
    x = nonlinear()(x)
    return x
</code></pre>
<p>CNN = network()<br>
CNN.cuda()<br>
Then CNN.parameters is None…So Why the parameters are not registered into the model…</p>

username: spruceb
cooked: <p>Because the default way parameters are registered is by assignment as a property of a module, through <code>nn.Module</code>'s <code>__setattr__</code>. If they’re in a list, <code>__setattr__</code> never gets called. Instead you need to use an <code>nn.ModuleList</code> to hold your parameters. It also has an <code>append</code> method and is iterable, so you can just replace</p>
<pre><code>self.scale_container = []
</code></pre>
<p>with</p>
<pre><code>self.scale_container = nn.ModuleList()
</code></pre>
<p>and you should be good, at least for <code>nonlinear</code>. Also you can simply iterate over it, i.e.</p>
<pre><code>for scale in self.scale_container:
</code></pre>
<p>rather than bothering with <code>range</code> and <code>ii</code>.</p>
<p>For <code>network</code>, if you’re using a module with parameters (i.e. <code>nonlinear</code>) you should assign it as a property in <code>__init__</code>, i.e. <code>self.nonlinearity = nonlinear()</code> or something, and then use <code>self.nonlinearity</code> in your <code>forward</code>. Otherwise again, the parameters don’t have any way to get registered.</p>

username: Bohan_Zhuang
cooked: <p><a class="mention" href="/u/spruceb">@spruceb</a><br>
Thx for your kindly reply,  it really helps!!</p>

username: Andreas_Bloch
cooked: <p>I think you should use <code>nn.ParameterList()</code> rather than <code>nn.ModuleList()</code></p>
Does padding affect the sign change in convolution filter output?

username: srikanthram
cooked: <p>Hey guys, I have observed a thing here that I want to discuss and clarify about.  Lets say you have a tensor <code>h**2 so h**2&gt;=0</code> also you have a kernel whose values are all non-negative. w&gt;=0. Now, when I do the convolution operation F.conv2d(h**2,w), the result is all non-negative.</p>
<p>However, when do the same operation with padding=1, i.e, <code>F.conv2d(h**2,w,padiing=1)</code>, I am getting some negative values as well which i checked by doing</p>
<p><code>assert torch.all(F.conv2d(h**2,w,padding=1)&gt;=0)</code></p>
<p>Why does padding affect the sign change?</p>
Can I manipulate nn.Module?

username: Mah
cooked: <p>I want to design a neural network in the form below:<br>
Wx + kx + b<br>
W is the neural network weight and And k is the weight of a fuzzy network, which ultimately combines this fuzzy network with the neural network.<br>
Can I manipulate nn.Module to do this?</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, num_classes):<br>
super(Net, self).<strong>init</strong>()                    # Inherited from the parent class nn.Module<br>
self.fc1 = nn.Linear(input_size, hidden_size)<br>
self.relu = nn.ReLU()<br>
self.fc2 = nn.Linear(hidden_size, num_classes)</p>
<pre><code>def forward(self, x):                              # Forward pass: stacking each layer together
    out = self.fc1(x)
    out = self.relu(out)
    out = self.fc2(out)
    return out</code></pre>
Are the optimizer parameters different from net.parameters?

username: Lewis
cooked: <p>Hi. When we define an optimizer such as:<br>
<code>optimizer = torch.optim.Adam(model.parameters(), lr=0.001, amsgrad=True)</code><br>
And then we manually change the network parameters, for example:</p>
<pre><code class="lang-auto">for param in net.parameters():
        do something
</code></pre>
<p>Then when we use <code>optimzier.step</code>, does the optimizer take the updates parameters we did in the previous step or the original parameters?</p>
How to change the behavior of an activation function in PyTorch?

username: Deeply
cooked: <p>I need to change the characteristics of the <code>sigmoid</code> function, which is usually done via a parameter called the temperature.<br>
Is there a way to do this in <code>PyTorch</code>? I noticed that the <code>sigmoid</code> has just one value as input.</p>

username: JuanFMontesinos
cooked: <p>Hi there,<br>
Sigmoid is defined as<br>
f(x) = A/(B+exp(-Cx)) in the most general case.<br>
Modifying C is equivalent to premultiply the input.<br>
Modifying A is equivalent to postmultiply the output</p>

username: Deeply
cooked: <p>I would also add another parameter, <code>beta</code>, to shift the input, that is, to move it away from the center that appears at input 0.5. Something like</p>
<p><code>pred = torch.sigmoid(alpha*pred - beta)</code></p>
<p>will do.</p>
For different inputs the ResNet structure should be modified?

username: Niki
cooked: <p>Regarding ResNet structure, in <a href="http://www.pabloruizruiz10.com/resources/CNNs/ResNet-PyTorch.html" rel="noopener nofollow ugc">http://www.pabloruizruiz10.com/resources/CNNs/ResNet-PyTorch.html</a></p>
<p>It is mentioned:</p>
<blockquote>
<p>For instance, ResNet on the paper is mainly explained for ImageNet dataset… since CIFAR10 input images are (32x32) instead of (224x224), the structure of the ResNets need to be modified.</p>
</blockquote>
<p>while I saw some Pytorch codes on GitHub for Resnet with CIFAR10, that are used the same structure as ResNet for ImageNet, I mean four-layers by “Conv64, Conv128, Conv256, Conv512”. Would you please guide me for Resnet CIFAR10, the structure of ResNet needs to be modified or not?</p>

username: JuanFMontesinos
cooked: <p>ResNet for cifar has less layers per block if i’m not wrong. I don’t remember about convolution size but maybe the first 7x7 convolution may be too big.</p>

username: Niki
cooked: <p>Thank you for your reply <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>. I saw 3x3 convolution is used in CIFAR cases. This means by changing the filter size, the same structure can be used for CIFAR10?</p>

username: JuanFMontesinos
cooked: <p>Yep, the structure is the same (wrt to what i remember).<br>
I think they were using 16 32 64 128 filters instead. It’s also remarkable the depth you use.</p>

username: Niki
cooked: <p>This means the structure is a little bit different by ResNet for ImageNet, and 3x3convolution with the number of filters 16 32 64 128 should be used for CIFAR10?  I also noticed in the “Deep Residual Learning for Image Recognition” paper the numbers of filters are considered 16 32 64.</p>

username: JuanFMontesinos
cooked: <p>The main problem is that if you use imagenet filters it will probably overfit so hard. That’s why CIFAR uses less. Anyway everything is on the original papers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> you don’t really need to discuss so much.</p>

username: Niki
cooked: <p>Thank you, <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>.</p>
Pytorch seq2seq tutorial vocab missing

username: keysmash97
cooked: <p>Hi, I’m woking with the tutorial of pytorch seq2seq and after go through the training progress then use the model to translate a random french sentence. It gives me error that word “Fruit” in sentence can not be translated. I understand that because word2index function can not convert “Fruit” to index. Are there any solution for this ? Thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3c29a649dc71ffe4ac9247e3fc09da62c659c1ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c29a649dc71ffe4ac9247e3fc09da62c659c1ac" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3c29a649dc71ffe4ac9247e3fc09da62c659c1ac.png" alt="image" data-base62-sha1="8AdSGr3PPEvHhYIUgnRDN3xJlKI" width="690" height="334" data-dominant-color="3C3C3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">767×372 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Reference: <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>

username: ptrblck
cooked: <p>If you are dealing with unknown words, you could add a specific word index for these unknown words and change the <code>indexesFromSequence</code> method a bit to yield it as the default value:</p>
<pre><code class="lang-python">def indexesFromSentence(lang, sentence):
    unknown_index = 0
    return [lang.word2index.get(word, unknown_index) for word in sentence.split(' ')]
</code></pre>
<p>Note that you would most likely have to adapt some other code parts to deal with this unknown index.</p>
How to access reserveSpace of cudnn RNN?

username: Thanh_Tuan_Dao
cooked: <p>I want to access the reserveSpace that was computed in the forward and re-use it in the backward to pass it to a cudnn function for re-use purpose. The reserveSpace is produced by calling the function:</p>
<pre><code class="lang-auto">cudnnStatus_t cudnnRNNForwardInference(
cudnnHandle_t handle,
const cudnnRNNDescriptor_t rnnDesc,
const int seqLength,
const cudnnTensorDescriptor_t *xDesc,
const void *x,
const cudnnTensorDescriptor_t hxDesc,
const void *hx,
const cudnnTensorDescriptor_t cxDesc,
const void *cx,
const cudnnFilterDescriptor_t wDesc,
const void *w,
const cudnnTensorDescriptor_t *yDesc,
void *y,
const cudnnTensorDescriptor_t hyDesc,
void *hy,
const cudnnTensorDescriptor_t cyDesc,
void *cy,
void *workspace,
size_t workSpaceSizeInBytes)
</code></pre>
<p>However, I could not find a way to do it from the python level yet.<br>
Any idea is appreciated,<br>
Thanks</p>
Training Stateful LSTM in Pytorch cause runtime error

username: 366baae1250c24d1e035
cooked: <pre><code class="lang-auto">class LSTMPricePredictor(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers):
        super().__init__()
        self.input_dim = input_size
        self.hidden_dim = hidden_size
        self.num_layers = num_layers
        
        self.init_hidden()
        
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
        self.hidden_to_output = nn.Linear(hidden_size, output_size)
    
    def init_hidden(self):
        self.hidden = (torch.zeros((self.num_layers, 1, self.hidden_dim), requires_grad=True), torch.zeros((self.num_layers, 1, self.hidden_dim)))
    
    def forward(self, inputs):
        x, (hx, cx) = inputs
        lstm_out, (hx, cx) = self.lstm(x, (hx, cx))
        #out = self.hidden_to_output(lstm_out.view(-1, self.hidden_dim))
        out = self.hidden_to_output(lstm_out)
        return out, (hx, cx)

class StatefullLSTMPricePredictor(LSTMPricePredictor):
    def forward(self, input_batches):
        linear_layers_results = []
        self.hidden[0].detach_()
        for i, input_batch in enumerate(input_batches):
            lstm_out, self.hidden = self.lstm(input_batch.unsqueeze(0), self.hidden)
            linear_layers_results.append(self.hidden_to_output(lstm_out[:, -1:, :]))
        return torch.cat(linear_layers_results, dim=0)

model = StatefullLSTMPricePredictor(4, HIDDEN_SIZE, 4, 1)

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())
for i in range(3, len(features), 3):
    criterion.zero_grad()
    out = model(features[:i])
    loss = criterion(out, labels[:i])
#     loss.backward() if i == 3 else loss.backward(retain_graph=True)
    loss.backward()
    optimizer.step()
</code></pre>
<p>Evaluating these code to train stateful LSTM cause following error:<br>
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>Can use please explain what is wrong?</p>

username: 366baae1250c24d1e035
cooked: <p>Is there any way to detect which model parameters are freed and cause such problem?</p>

username: babbage
cooked: <p>It might interest you to know that I’ve been trying to do something similar myself: <a href="https://discuss.pytorch.org/t/confusion-regarding-pytorch-lstms-compared-to-keras-stateful-lstm/44502" class="inline-onebox">Confusion regarding PyTorch LSTMs compared to Keras stateful LSTM</a></p>
<p>Although I’m not sure if just wrapping the previous hidden data in a torch.Variable ensures that stateful training works</p>

username: tom
cooked: <p>I think you need to detach both hiddens because the hiddens that are output from the LSTM will require grad.<br>
I would recommend doing so when you actually store them rather than before (but probably with <code>.detach()</code> rather than the inplace variant).</p>
<p>Best regards</p>
<p>Thomas</p>
index out of range at 

username: venusafroid
cooked: <p>Traceback (most recent call last):<br>
File “train_2.py”, line 46, in <br>
trainer.run()<br>
File “train_2.py”, line 28, in run<br>
outputs = self.model(inputs)<br>
File “/home/lili/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/lili/workspace/chinese_sentiment-master/qa_code/lstm.py”, line 25, in forward<br>
x = self.embedding(x[0])  # 经过embedding,x的维度为(batch_size, time_step, input_size=embedding_dim)<br>
File “/home/lili/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/lili/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/sparse.py”, line 118, in forward<br>
self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
File “/home/lili/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1454, in embedding<br>
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1550780889552/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191</p>
<p>When embedding, It comes this problem.I don’t know why.</p>

username: InnovArul
cooked: <p>What’s the shape of embedding layer and what is the content of x?  Note that embedding layer input should be 0-based indices.</p>
GANs importing generator model

username: ASHISH_MENON
cooked: <p>I have trained a GAN on Imagenet dataset which performs the task of filling the missing regions of an image by considering both L2 loss and Adverserial Loss using Pytorch.<br>
As a part of the testing process I loaded the above generator model and performed a forward pass of the test image through the generator network to fill the missing region.<br>
To my surprise the missing region was filled by random noise. Can I know what might be the reasons:<br>
1.)Is this the case of overfitting? Given the fact that I did try using the saved generator model on one of the trained images, and it still ends up filling the missing region with random noise.<br>
2.)Or is it a problem in the way I am loading the model?</p>
<p>-Thank You</p>
Nan in high order derivatives

username: Edu
cooked: <p>I would like to calculate high order derivatives in Pytorch, but the results are nan.<br>
Here is my code.</p>
<pre><code class="lang-auto">X = torch.tensor([[[0.,0.,0.],[1.,0.,0.],[3.,0.,0.]]],requires_grad=True)
adjacency_matrix = (X.unsqueeze(2)-X.unsqueeze(1)).norm(dim=3)
Y = torch.matmul(adjacency_matrix,adjacency_matrix)
grad1 = torch.autograd.grad(Y.sum(),X,create_graph=True)[0]
grad2 = torch.autograd.grad(grad1.sum(),X,create_graph=True)[0]
print(grad2)
</code></pre>
<p>The output grad2 is all nan. Could you please tell me how to get the correct result？</p>
Using own loss while having gradients and numpy implementation

username: MattyMatt
cooked: <p>Dear all,</p>
<p>I would like to implement my own loss in pytorch.</p>
<p>There are two things I would like to use:</p>
<ol>
<li>numpy implementation of a loss</li>
<li>numpy implementation of the derivative of my loss with respect to inputs.</li>
</ol>
<p>Is that possible?</p>
<p>Here is an example: lets say I would like to implement a simple MSE between two<br>
matrices, predictions Yhat and targets Y, of shape batch_sizex32x32x32:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/daab7622b1931de604dbb7c9aee86c265e3e0d18.png" alt="image" data-base62-sha1="vcrrt6Lgq4IGnlKvCGbPCzPBNnq" width="425" height="109"></p>
<p>where j goes over training examples in our batch D, and x goes over every<br>
value of the 32x32x32 grid</p>
<p>I can quickly implement it in numpy:</p>
<pre><code class="lang-auto">def my_mse_np(predictions, targets):                                            
    loss_np = np.mean(np.mean((X.reshape(batch_size,-1) - Y.reshape(batch_size,-1))**2, axis=1))
    return loss_np
</code></pre>
<p>and now, I also calculated the gradient (see image above)</p>
<p>and implemented it in numpy:</p>
<pre><code class="lang-auto">def my_mse_grad_np(predictions, targets)                                        
    grad_np = 2*np.mean(np.mean(X.reshape(batch_size,-1) - Y.reshape(batch_size,-1), axis=1))     
    return grad_np
</code></pre>
<p>My question is: can I use these two functions, ie:</p>
<ol>
<li>my loss in numpy</li>
<li>derivative of a loss in numpy</li>
</ol>
<p>without rewriting (1) using tensors?</p>
<p>In my case, the implementation of (1) is a bit long, and I already have<br>
the derivative, so would be much easier for me to use (1) and (2) instead of<br>
rewritting XXX lines of code of (1).</p>
Why dose the dilated convolution cause several times the training time than the original convolution?

username: MingHongL
cooked: <p>Hi,<br>
I found that under the same condition, using dilated convolution in the residual unit will cause several times the training time than the original convolution.<br>
What I did is the following:</p>
<p>Replace</p>
<pre><code class="lang-auto">class Block(nn.Module):
    def __init__(self, act=nn.ReLU(True)):
        super(Block, self).__init__()
        self.conv1 = nn.Conv2d(64,64, 5, padding=2, dilation=1)
        self.conv2 = nn.Conv2d(64,64, 5, padding=2, dilation=1)
        self.relu  = nn.ReLU(inplace=True)
    def forward(self, x):
        res = self.conv1(x)
        res2 = self.relu(res)
        res = self.conv2(res)
        res += x
        return res
</code></pre>
<p>by</p>
<pre><code class="lang-auto">class Block(nn.Module):
    def __init__(self, act=nn.ReLU(True)):
        super(Block, self).__init__()
        self.conv1 = nn.Conv2d(64,64, 3, padding=2, dilation=2)
        self.conv2 = nn.Conv2d(64,64, 3, padding=2, dilation=2)
        self.relu  = nn.ReLU(inplace=True)
    def forward(self, x):
        res = self.conv1(x)
        res2 = self.relu(res)
        res = self.conv2(res)
        res += x
        return res
</code></pre>
<p>In fact, under the same receptive filed, using the dilated convolution will reduce the amount of parameters, but why does it lead to a significant increase in training time?<br>
Could you tell me what is the reason and how to solve the problem?</p>
How could I flatten two dimensions of a tensor

username: coincheung
cooked: <p>Hi,</p>
<p>My question is this: Suppose I have a tensor <code>a = torch.randn(3, 4, 16, 16)</code>, and I want to flatten along the first two dimension to make its shape to be <code>(1, 12, 16, 16)</code>.</p>
<p>Now I can only operate like this: <code>size=[1, -1]+list(a.size()[2:]; a = a.view(size)</code> which I believe is not a pytorch way to do it. How could I do it in a smarter way?</p>

username: Tony-Y
cooked: <pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(3, 4, 16, 16)
&gt;&gt;&gt; c = torch.cat(a.unbind()).unsqueeze(0)
&gt;&gt;&gt; c.size()
torch.Size([1, 12, 16, 16])
</code></pre>

username: coincheung
cooked: <p>Thanks, is this an efficient way? Does not it involve memory copy in the process of unbind and cat?</p>

username: Tony-Y
cooked: <p>I am not sure whether efficient.</p>

username: Tony-Y
cooked: <pre><code class="lang-auto">c = a.view(1, -1, a.size(2), a.size(3))
</code></pre>
<p>This is an efficient way.</p>

username: coincheung
cooked: <p>Thanks, but this can only support 4-d tensor, and cannot be broadcasted to tensors of other dimensions. Would you please come up with some other methods?</p>

username: Tony-Y
cooked: <pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(3, 4, 16, 16, 16, 16)
&gt;&gt;&gt; c = a.view(1, -1, *(a.size()[2:]))
&gt;&gt;&gt; c.size()
torch.Size([1, 12, 16, 16, 16, 16])
</code></pre>

username: coincheung
cooked: <p>Thanks a lot, that is helpful!!</p>
Loss funciton has a mistake?

username: jack15710
cooked: <p>Dear all:<br>
when I read the code of pytorch transform_learnning tuitorials, I see the code below:</p>
<p>outputs = model(inputs)<br>
_, preds = torch.max(outputs, 1)<br>
loss = criterion(outputs, labels)</p>
<p>it seems that  mistake in the documentation of the loss funciton.<br>
i think that the loss should be like as:</p>
<p>loss = criterion(preds, labels)</p>
<p>Thanks .</p>

username: MariosOreo
cooked: <p>The argument passed to criterion should be a posibility map which is the output of network and calculated by <code>softmax</code> or <code>sigmoid</code> functions.</p>
<p><code>preds</code> you mentioned above is in sparse format.</p>

username: jack15710
cooked: <p>Thanks, I get it. It seems that I make a mistake in parameter needed by Function “criterion”.</p>
Binary classification with imbalanced dataset

username: shaun
cooked: <p>Hi,</p>
<p>I’m working on a binary text classification problem. I have a pretty imbalanced dataset. The statistics are shown below.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>train</td>
<td>positive</td>
<td>9598</td>
</tr>
<tr>
<td></td>
<td>negative</td>
<td>30988</td>
</tr>
<tr>
<td>val</td>
<td>positive</td>
<td>1200</td>
</tr>
<tr>
<td></td>
<td>negative</td>
<td>3874</td>
</tr>
<tr>
<td>test</td>
<td>positive</td>
<td>1200</td>
</tr>
<tr>
<td></td>
<td>negative</td>
<td>3874</td>
</tr>
</tbody>
</table>
</div><p>When I searched online and the forums, I came across a couple of methods to deal with this. I have a couple of questions about them. For this problem, I am using the <a href="https://pytorch.org/docs/stable/nn.html?highlight=bce#torch.nn.BCEWithLogitsLoss" rel="nofollow noopener">BCEWithLogitsLoss</a> which has a <code>weight</code> parameter and <code>pos_weight</code> parameter.</p>
<ol>
<li>What is the difference between <code>weight</code> and <code>pos_weight</code> and which one should I use for this problem?</li>
<li>For <code>pos_weight</code> the example tells to pass a ratio between the sizes of the positive and negative class. How will the function know which is the positive and the negative class when using the weight?</li>
<li>What do I pass for the <code>weight</code> parameter?</li>
<li>Will using PyTorch’s <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="nofollow noopener">WeightedRandomSampler</a> help in this case? Also, I don’t understand the example given for this. Again, what do I pass for the parameters?</li>
</ol>
<p>Please let me know if additional information is required and thanks for the help.</p>

username: JuanFMontesinos
cooked: <p>Hi, if you read docs, weight just weights batch-wise. Theoretically you have no knowledge about what does you batch contains. I would go forward pos_weight in order to balance the contribution of each class.</p>
<p><code>pos_weight</code> will be used when the specific class were chosen as GT (I  think, didn’t look at source code)</p>
<p>Sampling with assigning probabilities to each class is another different way of adressing this problem. However you don0t have to use both</p>
What is the src code for Tensor.normal_(), or what formula does it use under the hood?

username: MLSlayer
cooked: <p>Is it the standardization formula, which is basically (xi - x_mean) / std_dev ?</p>
<p>If it is, then how come it is not deterministic, for example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = t.Tensor([1,-1,0,2])
&gt;&gt;&gt; x.normal_()
tensor([-0.3429, -0.7214,  0.0883, -0.2900])
&gt;&gt;&gt; x = t.Tensor([1,-1,0,2])
&gt;&gt;&gt; x.normal_()
tensor([0.7380, 1.9640, 0.3068, 0.2396])
</code></pre>
<p>whereas if I was using this standardization then it should be</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = t.Tensor([1,-1,0,2])
&gt;&gt;&gt; x = x.numpy()
&gt;&gt;&gt; (x - x.mean()) / x.std()
array([ 0.4472136, -1.3416407, -0.4472136,  1.3416407], dtype=float32)
</code></pre>
<p>So how can we see what’s really going on here?</p>
<p>Here it says “standardization transforms data to have a mean of zero and a standard deviation of 1.” - <a href="https://www.statisticshowto.datasciencecentral.com/normalized/" rel="nofollow noopener">https://www.statisticshowto.datasciencecentral.com/normalized/</a></p>
<p>In the torch docs it says</p>
<pre><code class="lang-auto">normal_(mean=0, std=1, *, generator=None) → Tensor
</code></pre>
<p>-<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.normal_" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.normal_</a></p>
<p>so I assumed it was doing standardization but I guess not.</p>

username: SimonW
cooked: <p>Literally the line after your quote from the doc</p>
<blockquote>
<p>Fills  <code>self</code>  tensor with elements samples from the normal distribution parameterized by <a href="https://pytorch.org/docs/stable/torch.html#torch.mean"> <code>mean</code> </a> and <a href="https://pytorch.org/docs/stable/torch.html#torch.std"> <code>std</code> </a>.</p>
</blockquote>

username: MLSlayer
cooked: <p>This might be asking for a lot, but do you think you can stub out the src code for me? Or link me to an article that explains how this is done?</p>
<p>If I had to guess, it might be something like</p>
<pre><code class="lang-auto">torch.randn(len(x)) * std_dev + x_mean
</code></pre>
<p>Since torch rand uses normal distribution under the hood. However, doesn’t that imply that the elements of x is not relevant?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MLSlayer" data-post="4" data-topic="44650">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlslayer/48/11878_2.png" class="avatar"> MLSlayer:</div>
<blockquote>
<p>However, doesn’t that imply that the elements of x is not relevant?</p>
</blockquote>
</aside>
<p>Yes, <code>tensor.normal_()</code> will fill the tensor with values sampled from the normal distribution.<br>
The old values will be overwritten.</p>

username: MLSlayer
cooked: <p>Thanks a bunch, I was curious as to how the elements of x played a part, but they don’t</p>
Calling N models of the same architecture from one model

username: andrewnaguib
cooked: <p>I have an architecture that’s composed of 2 components.</p>
<ul>
<li>N encoders</li>
<li>N decoders</li>
</ul>
<pre><code class="lang-auto">    def __init__(self, N, **kwargs):
        super(MODEL, self).__init__()
        encoders = [Encoder(**kwargs)] * N
        decoders = [Decoder(**kwargs)] * N

    def forward(self, *input):
        pass
</code></pre>
<p>I’m not sure how do I call each (encoder/decoder)'s <code>forward</code> method properly (each will take <code>input</code>) .<br>
Knowing that I also use a custom loss function (will I need to override <code>backward</code> as well?).</p>
<p>Also, am I constructing the encoders properly? is there a better/convenient way?</p>

username: JuanFMontesinos
cooked: <p>Hi, there exists <code>torch.nn.</code> <code>ModuleList</code> in pytorch. You need to instantiate your model N times and append them to that list. Then, you can operate as if a python list were.</p>
<p>If you use python lists itself, you will be able to call each model but they won’t be properly registered in the parent module.</p>
Self-assessing neural networks: An experimental new type of network layer

username: Florian_Dietz
cooked: <p>I invented a new type of layer for neural networks: It gives the network the ability to critically assess the reliability of its own features, to enable more informed decision making.</p>
<p>Preliminary results: Use of this layer lead to a minor improvement on the MNIST dataset, but the improvement is too small to know for sure if it is useful.</p>
<p>Even if it turns out that this improvement is a fluke, there are a large number of future improvements that could make the algorithm more effective.</p>
<p>I only do research in my spare time, so I would like to listen to some feedback about this idea before I invest more time into it.</p>
<p>I explain the idea and offer a code sample <a href="https://floriandietz.me/neural_networks_self_assessment/" rel="nofollow noopener">here</a></p>
<p>I would appreciate it if you had a look at this and tell me what you think.</p>
<p>Also, if you have any practical problems or benchmarking testcases lying around, be sure to give it a try and report if it worked. It’s just one line of code to swap a normal linear layer with a self-assessment layer, as described in the article.</p>
Dataset - move all to GPU

username: Filipel_Filipel
cooked: <p>Hi. My Dataset is working with relatively small data, with no disc read. Tensors are moved to GPU in Dataset <strong>init</strong>() like that:</p>
<pre><code class="lang-auto"># Fill x&amp;y lists...

self.x = torch.tensor(self.x, device = device, dtype = dtype)
self.y = torch.tensor(self.y, device = device, dtype = torch.int64)
</code></pre>
<p>In tutorials preferred approach is to move every batch to GPU after retrieve from DataLoader. Is that two methods equivalent?</p>
Fix similar to numpy

username: thisthuan
cooked: <p>please help me,<br>
code in numpy</p>
<pre><code class="lang-auto">x_np= np.array([1,2,3])
def phi_func_np(x):
    return np.array([x, x**2])

print(phi_func_np(x_np))

</code></pre>
<p>result:</p>
<pre><code class="lang-auto">[[1 2 3]
 [1 4 9]]
</code></pre>
<p>But in pytorch, it’s error</p>
<pre><code class="lang-auto">x= torch.tensor([1,2,3])
def phi_func(x):
    return torch.tensor([x, x**2])

print(phi_func(x))
</code></pre>

username: JuanFMontesinos
cooked: <pre><code class="lang-auto">[x, x**2]
Out[4]:
 [tensor([1, 2, 3]), tensor([1, 4, 9])]
</code></pre>
<p>You cannot convert that into a tensor. You need to use torch.cat or torch.stack depending on how do you want to convert it.</p>

username: thisthuan
cooked: <p>Can you please explain more detail?</p>

username: thisthuan
cooked: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> thank you, I see.</p>
<pre><code class="lang-auto">x= torch.tensor([1,2,3])
def phi_func(x):
    return torch.stack([x, x**2])

print(phi_func(x))
</code></pre>

username: JuanFMontesinos
cooked: <p>Basically you may want to concatenate in the same row or to create a new one. That’s the difference between both. I don’t know what does numpy do by default</p>
RNN training speed

username: alexm
cooked: <p>Hello community, I might be mistaken but I have the impression that</p>
<ul>
<li>training RNN is very slow (and even slower on GPU)</li>
<li>
<strong>unless</strong> you can parallelize the training using batches (which would make GPU useful again)</li>
<li>
<strong>but</strong> this entails that sequences have a fixed length (you can use an End Of Sequence though)</li>
</ul>
<p>=&gt; so basically to be able to train faster you need to fix the length of the sequence which to me is a bit killing the whole flexibility of RNN being able to train on variable length sequences.</p>
<p>Are these assertions correct ? Am I missing something ?</p>
<p>Thanks a lot</p>
Missing output of workers when using nohup and &

username: somethingdarkside
cooked: <p>I am trying to run the following script to retrain the ITrackerModel: <a href="https://github.com/CSAILVision/GazeCapture/tree/master/pytorch" rel="nofollow noopener">https://github.com/CSAILVision/GazeCapture/tree/master/pytorch</a></p>
<p>Leaving the settings the way they are in the repo I am running into problems when I run main.py using <code>nohup python main.py --data_path /cluster/data/preprocessed/ --reset &amp;</code>. I cannot see the output of the individual workers in nohup.log and at first, it looks like it is stuck. I just stops showing me any output after main.py Line 158 (<code>for ... in enumerate(train_loader)</code>).</p>
<p>But when I just run the script with <code>python main.py --data_path /cluster/data/preprocessed/ --reset</code> i can see the output being printed to the console.</p>
<p>This, for example, applies to the print() defined here: <a href="https://github.com/CSAILVision/GazeCapture/blob/master/pytorch/main.py#L192-L197" rel="nofollow noopener">main.py L192-L197</a> which lies inside the <code>for ... in enumerate(train_loader)</code> where <code>train_loader</code> is a <code>DataLoader</code> instance. Hence my assumption, that this is happening because of the workers output not showing.</p>
<p>I am fairly new to pytorch (and to python as well). Any help is appreciated.</p>
<p>P.S.: screen and tmux are great for this kind of stuff but unfortunately I am not able to use them in this setup here.</p>
DCGAN Tutorial - Where is the output image size of the generator set?

username: McChicken93
cooked: <p>I’m still getting started with pytorch, and I don’t understand where the output image size of the generator in the C++ tutorial is set.<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html#the-generator-module" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html#the-generator-module</a></p>
<p>I get it to work, but I don’t understand why the Net produces a 28x28 px output (as it is supposed to do).</p>

username: McChicken93
cooked: <p>I found the answer myself, keeping it here for documentary purposes…<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1603.07285.pdf" target="_blank" rel="nofollow noopener">1603.07285.pdf</a></h3>

<p class="filesize">892.34 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How do I change env variables with cmake?

username: RuFAI
cooked: <p>Hi, I have some problems with my build. I can’t run my code in C++ due ot some linker errors (I am using libtorch), which is why I am trying to build my code with the help of cmake. The Problem is, that it does not find Cuda toolkit paths. I don’t even have Cuda on my PC becuase it has no graphics card. So I am trying to set a flag that set the requirement of Cuda to false. The final question is: Where do i set this environment variable in the Building process with cmake?</p>
<pre><code class="lang-auto">Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.17134.
The C compiler identification is MSVC 19.20.27508.1
The CXX compiler identification is MSVC 19.20.27508.1
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
Looking for pthread.h
Looking for pthread.h - not found
Found Threads: TRUE  
CUDA_TOOLKIT_ROOT_DIR not found or specified
Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) 
CMake Warning at C:/Users/.../libtorch/share/cmake/Caffe2/public/cuda.cmake:21 (message):
  Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  C:/Users/.../libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  C:/Users/.../libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:6 (find_package)


CMake Error at C:/Users/.../libtorch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries.  Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
  C:/Users/.../libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:6 (find_package)


Configuring incomplete, errors occurred!
See also "C:/Users/.../build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/.../build/CMakeFiles/CMakeError.log".
</code></pre>
How can I train different parts of a model with different optimizers?

username: erogol
cooked: <p>I have a network with a particular output layer needs to be trained independently than the rest of the network. What I’ve done so far is the following.</p>
<pre><code class="lang-auto">       optimizer = optim.Adam(model.parameters(), lr=c.lr, weight_decay=0)
       optimizer_st = optim.Adam(
             model.decoder.stopnet.parameters(), lr=c.lr, weight_decay=0)
....

       # backpass and check the grad norm for spec losses
        loss.backward(retain_graph=True)
        optimizer, current_lr = weight_decay(optimizer, c.wd)
        grad_norm, _ = check_update(model, c.grad_clip)
        optimizer.step()

        # backpass and check the grad norm for stop loss
        stop_loss.backward()
        optimizer_st, _ = weight_decay(optimizer_st, c.wd)
        grad_norm_st, _ = check_update(model.decoder.stopnet, 1.0)
        optimizer_st.step()
</code></pre>
<p>It actually works so far but it slows down the network a lot. Is there any better way to do that I am probably missing?</p>
cublas runtime error : library not initialized at ..\aten\src\THC\THCGeneral.cpp:250

username: venusafroid
cooked: <p>I have the problem:</p>
<p>Traceback (most recent call last):<br>
File “train_2.py”, line 44, in <br>
trainer.run()<br>
File “train_2.py”, line 27, in run<br>
outputs = self.model(inputs)<br>
File “C:\Users\lili\Anaconda3\envs\python3-torch\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\lili\workspace\用tensorflow进行中文自然语言处理的情感分析\chinese_sentiment-master\qa_code\qafa.py”, line 20, in forward<br>
question = self.dense1(question)<br>
File “C:\Users\lili\Anaconda3\envs\python3-torch\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\lili\Anaconda3\envs\python3-torch\lib\site-packages\torch\nn\modules\linear.py”, line 67, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “C:\Users\lili\Anaconda3\envs\python3-torch\lib\site-packages\torch\nn\functional.py”, line 1352, in linear<br>
ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())<br>
RuntimeError: cublas runtime error : library not initialized at …\aten\src\THC\THCGeneral.cpp:250</p>
<p>I am using single GPU.I have tried “sudo rm -rf ~/.nv” and it didn’t work. I am confused. Hop</p>
Gradcheck indicates analytical jacobian is a zero matrix after change Pytorch to 1.1.0

username: Melissa
cooked: <p>In Pytorch 1.1.0  Python 3.7.3<br>
I define a custom layer using autograd.Function. I’d like to check whether I defined the backward path correctly.<br>
I set input1 and input2 as Variables, also set requires_grad = True for both of them.Then I called:<br>
autograd.gradcheck(myLayer, (input1, input2)).<br>
This raises error “Jacobian mismatch for output 1 with respect to input 0”, it shows that analytical jacobian is a zero matrix (all entries are 0).<br>
However, if I change Pytorch back to 0.4.0, the analytical jacobian is the same as what I defined in backward.</p>
<p>Very confusing about what’s the difference there.<br>
Any message will help!</p>

username: tom
cooked: <p>Hi.</p>
<p>We cannot really tell you much about it unless you’d share a bit more about <code>myLayer</code>. Would you be able to narrow down a bit which function might be causing this?</p>
<p>Best regards</p>
<p>Thomas</p>
How to write custom functions which doesn't need backward?

username: clavichord93
cooked: <p>I am trying to implement a custom function which doesn’t need backward propagation. However, in the tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">here</a>, I only find the example with both forward and backward functions.</p>
<p>So how can I implement my function? Should I write a blank backward function or just not bind backward function with PYBIND11_MODULE?</p>

username: tom
cooked: <p>You can just skip the backward &amp; <code>autograd.Function</code> part.</p>
<p>Best regards</p>
<p>Thomas</p>
How to make a "list : target" Custom Dataset?

username: Hanlin
cooked: <p>Hello, I am a Pytorch beginner.<br>
I’m trying to make a custom Dataset which is not simply like “one image: one target" mode.<br>
I want to have a “5 input images : one target image” mode dataset. So how can I do that ?</p>
<p>Thanks very much !</p>

username: AnBucquet
cooked: <p>Hey,<br>
You should look at this <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a>. I think it’s what you’re looking for.<br>
But basically you need to create your own dataset class like:</p>
<pre><code class="lang-auto">class Dataset(Dataset):
    """dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with names of all images
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.names = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.names)

    def __getitem__(self, idx):
        img_name1 = os.path.join(self.root_dir,
                                self.names.iloc[idx, 0])
        image1 = io.imread(img_name1)
        img_name2 = os.path.join(self.root_dir,
                                self.names.iloc[idx, 1])
        image2 = io.imread(img_name2)
        img_name3 = os.path.join(self.root_dir,
                                self.names.iloc[idx, 2])
        image3 = io.imread(img_name3)
        target_name = os.path.join(self.root_dir,
                                self.names.iloc[idx, 3])
        target = io.imread(target_name)
        sample = {'image1': image1, 'image2': image2, 'image3': image3, 'target': target}

        if self.transform:
            sample = self.transform(sample)

        return sample
</code></pre>

username: Hanlin
cooked: <p>Nice! I never thought about using dictionary and csv file …<br>
Actually following the tutorial, I just know how to load an exist dataset and some other specific examples.</p>
<p>I will try it ! Thank you !</p>
LSTM error: Unable to classify sentences into classes

username: chrome
cooked: <p>I am trying to classify sentences into multiple classes. Each sentence is converted into a tensor after creating word2index.</p>
<p>The input tensor is a sentence. After converting each word’s index to tensor, the sentence looks like this: tensor([1407995,  937957,  936279,  904725,  682273, 1291222,  523149,  566120, 913504])</p>
<p>The issue I am getting is ValueError: Expected input batch_size (9) to match target batch_size (1).<br>
This I think is because my output(target label) is only a single label(whose index I have after creating word2index dictionary) but my input is actually the tensor value of each word in the sentence since the sentence length is also 9.<br>
The error occurs at loss = loss_cf(label_pred, target_label)<br>
where loss_cf = nn.NLLLoss()<br>
The output of the fully connected layer after the LSTM is :<br>
class_val = self.softmax(fc_layer)<br>
Also (fc_layer): Linear(in_features=64, out_features=100, bias=True)</p>
<p>There are 100 different classes.<br>
I am using an LSTM and I think the issue has to do with how I encode the target label which in this case is a single index value of the particular target label. I need help in figuring this out.</p>

username: vdw
cooked: <p>Without the full code, it’s difficult to pinpoint the exact problem. From what I can see, there are several issues that strike me as odd:</p>
<ul>
<li>The error is about the batch size not the sequence length. Your example sequences has 9 values. Do you use batches for training and what is the shape?</li>
<li>
<code>LSTM</code> has the setting <code>batch_first=True/False</code> (default: False). Depending on the chosen value, you have to make sure that your batch has the right shape. I assume that you have an <code>Embedding</code> layer as well.</li>
<li>Your values in your example tensor are very high. Do you really have a vocabulary of size 1,4 million words and more. how did you create your <code>word2index</code>. What us the shape of your embedding layer?</li>
</ul>

username: chrome
cooked: <p>That issue got solved. I was supposed to use the output of the last lstm i.e.,<code>lstm_out[-1]</code><br>
The exact code statement is <code>fully_conn(lstm_out[-1].view(-1, self.hidden_dim*2))</code><br>
Hopefully this will be of help to someone.<br>
Yes and to answer your questions, batch_first = True, batch_size=1.<br>
Word2index was created using the entire vocabulary in the X_train. Each word had a  dimension of128. I had used a pretrained embedding.</p>
Get error when backward:“RuntimeError: Tensor: invalid storage offset”?

username: voyager112
cooked: <p>When I use pytorch to do the backward grad, get error:</p>
<pre><code class="lang-auto">RuntimeError: Tensor: invalid storage offset at c:\users\administrator\downloads\new-builder\win-wheel\pytorch\aten\src\th\generic/THTensor.cpp:761

H_grad = torch.autograd.grad(outputs=prediction.sum(), inputs=database_choose, create_graph=True)[0]
Ht=H_grad[:,1].reshape(choose,1)
Htt=torch.autograd.grad(outputs=Ht.sum(), inputs=database_choose,create_graph=True)[0][:,1].reshape(choose,1)
loss.backward()
</code></pre>