akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why do we need the second parameter for add_graph() when using tensorboard?

username: entslscheia
cooked: <p>As we know, there are two parameters for <code>add_graph()</code>, a first parameter is an object of the model that we want to visualize, and the second parameter is just an input for the model. The input can be quite casual, i.e., we can simply randomly generate an input with the shape we want and feed it to <code>add_graph()</code>. So what’s the point to have the input as a parameter? It seems to me that this parameter only provides the shape information of an input tensor, but we can directly get this information from the model definition, right? And no matter what input we feed to <code>add_graph()</code>, the computation graph printed by tensorboard is always the same. I am really confused about this.</p>
Implementation of simple VAE loss

username: el_cangrejo
cooked: <p>Hello, I’m trying to implement a C++ version of this <a href="https://github.com/pytorch/examples/tree/master/vae" rel="noopener nofollow ugc">vae example</a> but I have trouble with implementing the KL divergence term. My implementation is as follows:</p>
<blockquote>
<p>// do stuff<br>
torch::Tensor mse = torch::mse_loss(recon_batch, batch.data.reshape({recon_batch.size(0), 784}));<br>
torch::Tensor kld = - 0.5 * torch::sum(1 + logvar - mu.pow(2) - logvar.exp());<br>
auto loss = mse + kld;<br>
loss.backward()<br>
// do stuff</p>
</blockquote>
<p>The loss though stops decreasing after  very few epochs, providing back reconstruction results when compared to the python example, when I’m optimizing only the based on the reconstruction loss it decreases to lower point. Any ideas on what I’m missing? Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Accessing activation values in the optimizer

username: dkaplan65
cooked: <p>Hi,</p>
<p>I am making a custom optimization function and I want to somehow get the activation vector coming into the node.</p>
<p>As far as I understand:<br>
<code>p.data</code> is the current value of the parameter <code>p</code><br>
<code>p.grad.data</code> is the current value of the gradient for parameter <code>p</code></p>
<p>How do I get the actual data coming into the parameter?</p>
<p>Thanks,<br>
David</p>
Optimizing an Autoencoder NN in pytorch , dimension problem

username: pedram1
cooked: <p>Hi all<br>
I am trying to optimize a 1D auto-encoder neural network. My batch size is 65536, and length of my signal is 94.<br>
I used this code to optimizing my parameters:<br>
/////////////////////////////////////////////////////////////////////////////////////////////<br>
for epoch in range(num_epochs):<br>
for data in y:<br>
img, _ = data<br>
img = Variable(img).cpu()<br>
# ===================forward=====================<br>
output = model(img)<br>
loss = criterion(output, img)<br>
# ===================backward====================<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
# ===================log========================<br>
print(‘epoch [{}/{}], loss:{:.4f}’<br>
.format(epoch+1, num_epochs, loss.data[0]))<br>
if epoch % 10 == 0:<br>
pic = to_img(output.cpu().data)<br>
save_image(pic, ‘./dc_img/image_{}.png’.format(epoch))</p>
<p>///////////////////////////////////////////////////////////////////////////////////////////////////////////////////</p>
<p>above y is my data set which is a tensor with dimension of [65536,1,94]. as I mentioned before, 65536 is my batch size, 1 is number of channel and 94 is length of my signal,<br>
when I run the code, this error appeared:</p>
<p>ValueError: not enough values to unpack (expected 2, got 1)</p>
<p>has anyone any idea?</p>
<p>Thanks a lot</p>

username: ptrblck
cooked: <p>It seems your <code>data</code> is not a tuple you could unpack:</p>
<pre><code class="lang-python">img, _ = data
</code></pre>
<p>Could you check the size and type of <code>data</code>?</p>

username: pedram1
cooked: <p>Hi<br>
Thank you for your answer. Actually my data set is a 1d signal that has 65536 sample (batch number). Its one channel.<br>
I edited my codes to the below, how ever I still get error.<br>
<span class="hashtag">#training</span> loop<br>
for epoch in range(num_epochs):<br>
for data in (data_pixel2.shape[0]):<br>
img = data<br>
img = Variable(img).cpu()<br>
# ===================forward=====================<br>
output = model(img)<br>
loss = criterion(output, img)<br>
# ===================backward====================<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
# ===================log========================<br>
print(‘epoch [{}/{}], loss:{:.4f}’<br>
.format(epoch+1, num_epochs, loss.data[0]))<br>
if epoch % 10 == 0:<br>
pic = to_img(output.cpu().data)<br>
save_image(pic, ‘./dc_img/image_{}.png’.format(epoch))</p>
<p>/////////////////////////////////////////////////////////////////////////////////</p>
<p>data_pixel2 is a tensor size of [65536,1,94], that the first number indicates batch number and 1 means channel, and 94 means length of my signal.<br>
Main reason of writing these codes was building an 1d autoencoder. However even I can’t run training loop <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Dataset not pulling with custom class

username: Jordan_Howell
cooked: <p>I tried coding a custom data set.  When I try to load the class into a data loader, it states that:</p>
<blockquote>
<hr>
<p>NameError                                 Traceback (most recent call last)<br>
 in <br>
16 # Generators<br>
17 training_set = roof_dataset(partition[‘train’], labels, transform = train_transforms)<br>
—&gt; 18 training_generator = data.DataLoader(training_set, **params)<br>
19<br>
20 test_set = roof_dataset(partition[‘test’], labels, transform = train_transforms)</p>
<p>NameError: name ‘data’ is not defined</p>
</blockquote>
<p>Here is the code pulling the data:</p>
<blockquote>
<h1>CUDA for PyTorch</h1>
<p>use_cuda = torch.cuda.is_available()<br>
device = torch.device(“cuda:0” if use_cuda else “cpu”)</p>
<h1>Parameters</h1>
<p>params = {‘batch_size’: 1000,<br>
‘shuffle’: True,<br>
‘num_workers’: 4}<br>
max_epochs = 100</p>
<h1>Generators</h1>
<p>training_set = roof_dataset(partition[‘train’], labels, transform = train_transforms)<br>
training_generator = data.DataLoader(training_set, **params)</p>
</blockquote>
<p>Here is the custom class.  I’m not sure where it’s going wrong;</p>
<blockquote>
<p>root_dir = ‘D:\CIS inspection images 0318\train\roof\’<br>
class roof_dataset(Dataset):</p>
<pre><code>'Characterizes a dataset for PyTorch'

def __init__(self, list_IDs, labels, transform):
    'Initialization'
    self.labels = labels
    self.list_IDs = list_IDs
    self.transform = transform

def __len__(self):
        
    'Denotes the total number of samples'
    return len(self.list_IDs)

def __getitem__(self, index):

        'Generates one sample of data'
        # Select sample
        ID = self.list_IDs[index]

        # Load data and get label
        X = torch.load(Path(root_dir) + ID)
        y = self.labels[ID]
        
        if self.transform:
            X = self.transform(X)

        return X, y
</code></pre>
</blockquote>

username: albanD
cooked: <p>From the error that you linked, it seems that you’re missing a <code>from torch.utils import data</code> to have <code>data.DataLoader</code> to work properly.</p>
Dopamine for PyTorch

username: alwynmathew
cooked: <p>Google <a href="https://github.com/google/dopamine" rel="nofollow noopener">Dopamine</a> seems so cool for fast RL prototyping. Do we have something like Dopamine for PyTorch?</p>

username: Michael_Guarino1
cooked: <p>not sure if this offers the same functionality as Dopamine but <a href="https://github.com/facebookresearch/ReAgent" rel="nofollow noopener">ReAgent</a> is helpful</p>
How to handle unknown and rare words with the Transformer model

username: arbint
cooked: <p>Hi all,<br>
I am using a generic Transformer model for translation and due to multi-headed attention, it seems like aligning target-source to replace an unknown token with source token is not trivial.</p>
<p>I was wondering if you know of any other techniques to replace unknown or rare words that work with the Transformer model? Do you know what is the standard industry practice (in addition to using BPE or SentencePiece to mitigate the problem)? Any idea of how Facebook or Google tackles this challenge?</p>
<p>Thanks</p>
Performant variable length reduction (inverse of repeat_interleave?)

username: Arka
cooked: <p>Hello,</p>
<p>I’m interested in the following problem: given a tensor of A of size M consisting of my data, and a tensor B of size N consisting of ‘lengths’, I would like to apply a reduction operation over the first B[0] elements of A, then the next B[1] elements of A, etc.</p>
<p>Example:</p>
<p>A: [3, 6, 5, 2, 6, 1, -3, 7]<br>
B: [2, 4, 1, 1]</p>
<p>Using a mean reduction, I want my output tensor to return:</p>
<p>Output: [4.5, 3.5, -3, 7]</p>
<p>There are various ways I can do this.</p>
<p>A simple for loop works but is very slow, as expected.</p>
<p>Using torch.scatter is much faster, but still feels like it is missing a potential speedup. In particular, scatter allows arbitrary ordering of indices; it doesn’t take advantage of the fact that my tensor B is going to be creating a series of ordered slices into A.</p>
<p>To illustrate this, if I try the reverse operation (e.g. broadcasting some tensor C out by repeating elements as many times as denoted by tensor B) via fancy indexing, this is much slower - particularly on the backwards pass - than using torch.repeat_interleave.</p>
<p>Note that I am performing all this on a GPU.</p>
<p>Does anyone have any ideas on how to make the above more performant? Does it require an explicit new kernel to be written, perhaps? I feel like this must be a useful feature to have in general, given that repeat_interleave exists.</p>
<p>Thanks!</p>

username: Arka
cooked: <p><a href="https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.ufunc.reduceat.html" rel="nofollow noopener">numpy.reduceat</a> is a slightly more general version of what I would like to have; its equivalent doesn’t seem to exist in PyTorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think such function exists at the moment.<br>
That being said we would be very happy to accept a PR if you want to add it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Arka
cooked: <p>Hello,</p>
<p>Thanks for your response. We don’t know much about CUDA kernels but I think we’d be willing to read up about it some more; would you have some resources you can point us to? Alternatively, would you happen to know anyone who would be able to guide us in some way?</p>

username: albanD
cooked: <p>I think the PR for repeat_interleave that you can find <a href="https://github.com/pytorch/pytorch/pull/18395" rel="nofollow noopener">here</a> is a very good example. You would need to change exactely the same files to have the binding. Change the implementation for the cpp and cuda code and add a test.<br>
Note that for the interface, I think an extra argument like <code>reduce="sum"</code> that can take value “sum” and “max” is good. Potentially add “mean” for convenience as well even though it can be done by dividing the result with the lenghts tensor.</p>
Question about Conv1d

username: pedram1
cooked: <p>Hi everyone<br>
I have a question and hope that someone could help me out.<br>
I am confused about how Conv1d(in_channels ,out_channels, kernel_size) is working? I was always thinking that the out_channels is equal to the number of channels that conv1d is using due to extract features from my data, However in Andrew N.J videos I have seen that number of channela have to be equal for both input data and filters in cov layer.<br>
I am now really confused that what actually channels mean here?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p><a href="http://cs231n.github.io/convolutional-networks/#conv" rel="nofollow noopener">CS231n</a> gives a good overview how convolutions are used for image data. Even though you are dealing with <code>Conv1d</code>, the arithmetic is the same and you are just using a temporal dimension instead of the two spatial ones (height and width).</p>
<p>Basically, each kernel has a channel dimension, which is (in the vanilla setup) equal to the number of input channels. <code>out_channels</code> would correspond to the number of filters in the conv layer.</p>

username: dejanbatanjac
cooked: <aside class="quote no-group" data-username="pedram1" data-post="1" data-topic="58325">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>Andrew N.J</p>
</blockquote>
</aside>
<p>You meant to say:<br>
<em>Andrew</em> Yan-Tak Ng (Chinese: 吳恩達; born 1976) is a Chinese-American computer scientist and statistician, focusing on machine learning and AI</p>
<p>Right?</p>

username: pedram1
cooked: <p>No<br>
I mean this guy : Andrew Ng</p>

username: pedram1
cooked: <p>Thank you Ptrblck , it was helpfull, I Liked the notes that u suggested</p>

username: dejanbatanjac
cooked: <p>Hi, <a class="mention" href="/u/pedram1">@pedram1</a>,</p>
<p>Thanks for the confirmation. You should just know that the number of <code>out_channels</code> is the number of <em>filters</em>.</p>
<p>I don’t know what Andrew Ng. wanted to point, or if your convolution 1d is for some specific architecture but if you consider Conv1d alone in PyTorch you have the option to set as many <code>out_channels</code> as you like.</p>
Want to use support vector regression in the last fullyconnected layer of my CNN model

username: Aditya_Raj_P15VSS201
cooked: <p>Hello Friends,</p>
<p>I am trying to build a CNN model</p>
<p>that takes image as input and produces a 6x1 vector of continuous values. Here is what I have wrote</p>
<p>class MultiLabelNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MultiLabelNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3,64, 5)<br>
self.pool = nn.MaxPool2d(2,2)<br>
self.conv2 = nn.Conv2d(64, 128, 5)<br>
self.conv3 = nn.Conv2d(128, 256, 5)<br>
self.conv4 = nn.Conv2d(256,320,5)<br>
self.fc1 = nn.Linear(250880,2048)<br>
self.fc2 = nn.Linear(2048, 1024)<br>
self.fc3 = nn.Linear(1024, 512)<br>
self.fc4 = nn.Linear(512, 6)<br>
def forward(self, x):<br>
x = self.conv1(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv2(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv3(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv4(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = x.view(-1, 250880)<br>
x = self.fc1(x)<br>
x = self.fc2(x)<br>
x = self.fc3(x)<br>
x = self.fc4(x)<br>
return x</p>
<p>Here self.fc4 = nn.Linear(512, 6) this is my final layer.I have run this network but the results are good but not much convincing. I want to use support vector regression instead of nn.Linear but I do not find any help. Please help me in this regard. I hope I have clearly mentioned my problem.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Using an SVM as the loss function for a cnn is not necessarily good as it will give sparse gradients and won’t improve training.<br>
You can also consider all other layers fixed, extract the feature for the last layer and use a regular svm solver to train the last layer as postprocessing of your training.<br>
Finally you can check <a href="https://arxiv.org/abs/1611.02185" rel="nofollow noopener">this</a> paper that tries to extend the use of svms for cnn.</p>

username: Aditya_Raj_P15VSS201
cooked: <p>Thanks for your response brother, I will check the paper.</p>
Deploying on customer local machines

username: Hamada_Fathy
cooked: <p>Let’s assume that i want to deploy and integrate my pytorch model on the customer local machines… How could i guarantee that the end customer won’t get my model weights??</p>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t have any specific tool for this.<br>
I don’t know what are the regular python tools that exist for this but they most likely also apply to pytorch.</p>
What is the behavior of passing model parameters with `requires_grad == False` to an optimizer?

username: viniciusarruda
cooked: <p>What is the behavior of passing all or some model parameters with <code>requires_grad == False</code> to an optimizer ?</p>
<p>If I have a model and want to fine tune it. Some layers are frozen (<code>requires_grad == False</code> and <code>.eval()</code>) with pre-trained values. Should I remove the parameters with <code>requires_grad == False</code> to pass to the optimizer ?</p>

username: ptrblck
cooked: <p>All parameters without a <code>.grad</code> attribute will be skipped as shown <a href="https://github.com/pytorch/pytorch/blob/9d925c1d6f81ad85bdc1b1aaa68efe6972c6f0ac/torch/optim/sgd.py#L90" rel="nofollow noopener">here</a>. You could avoid this step by filtering out all parameters which don’t require gradients.</p>

username: viniciusarruda
cooked: <p>So it is an option to filter out the parameters which don’t require gradients ?<br>
Since these parameters will be ignored, why should I worry about them (as they will pollute my code as it is useless)?</p>

username: ptrblck
cooked: <p>I would recommend to filter them out, as this would stick to the Python Zen <code>Explicit is better than implicit.</code></p>
<p>An unwanted side effect of passing all parameters could be that the parameters, which are frozen now, are still being updated, if they required gradients before, and if you are using an optimizer with running estimates, e.g. <code>Adam</code>.</p>
Invalid max index when MaxUnpool1d

username: Bryce_Ramgovind
cooked: <p>I am trying to build an auto-encoder but I keep stumbling upon an indexing issue when trying to unpool.</p>
<p>This is my code so far for the encoding:</p>
<pre><code class="lang-auto">from torch.autograd import Variable

test_in = Variable(torch.rand(1, 73, 480))

layer1 = nn.Conv1d(73, 256, kernel_size=3, stride=1, padding=2, bias=True)
layer2 = nn.ReLU()
layer3 = nn.AdaptiveMaxPool1d(240, return_indices=True)
drop_out = nn.Dropout(0.25)

out = drop_out(test_in)
out = layer1(out)
out = layer2(out)
out, indices = layer3(out)
</code></pre>
<p>This is the code for my decoding:</p>
<pre><code class="lang-auto">layer1_d = nn.Conv1d(256, 73, kernel_size=2, stride=1, padding=2, bias=True)
drop_out_d = nn.Dropout(0.25)
unpool_d = nn.MaxUnpool1d(3,1)
out_d = unpool_d(out, indices, output_size= (1, 256, 480))
out_d = drop_out_d(out_d)
out_d = layer1_d(out_d)
</code></pre>
<p>But I get this error on the  <code>unpool</code>  step</p>
<blockquote>
<p>RuntimeError: Found an invalid max index: 480 (output volumes are of size 480x1</p>
</blockquote>
<p>The resulting output ( <code>out</code> ) and  <code>indices</code>  from the encoding step have the following shape, respectively  <code>torch.Size([1, 256, 240])</code>  and  <code>torch.Size([1, 256, 240])</code> .</p>

username: albanD
cooked: <p>Hi,</p>
<p>This might be a small bug in the AdaptiveMaxPool module.<br>
Could you give a small code sample we can run to reproduce the issue please?</p>
Source code for conv2d

username: Saik
cooked: <p>Hello,<br>
Can someone point to the cpp code for conv2d. I would like to know how pytorch implements convolution internally.</p>
<p>I need to run convolution for inputs that differ very slightly(with the same kernel) and I hope that understanding how conv2d works internally will give me ideas on how to compute the many convolutions efficiently (for example by storing common computations).</p>
<p>Thank You!</p>

username: albanD
cooked: <p>Hi,</p>
<p>We try to use highly efficient libraries such as cudnn or mkldnn as much as possible. So in practice, their code is most likely to be the one that runs when you call conv.</p>
<p>The general implementation we have that works in all cases can be found there: <a href="https://github.com/pytorch/pytorch/blob/4f1f084d221a7ab9edbf09ab1904edde6d49848c/aten/src/THCUNN/generic/SpatialConvolutionMM.cu" rel="nofollow noopener">for gpu</a> and <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THNN/generic/SpatialConvolutionMM.c" rel="nofollow noopener">for cpu</a>.</p>
Learning problem

username: aldeka12
cooked: <p>Hello I used multihead attention in my network:</p>
<pre><code class="lang-auto">class SelfAttenion1Like(nn.Module):
    def __init__(self, num_classes):
        super(SelfAttenion1Like, self).__init__()
        self.multiHead = nn.MultiheadAttention(35, 5, 0.3)
        self.fc1 = nn.Linear(in_features = 3500, out_features = 2000)
        self.dropout1 = nn.Dropout(0.5)
        self.fc2 = nn.Linear(in_features = 2000, out_features = 12)
        self.swish = Swish()
    def forward(self, x):
        x = x.transpose(0,1).contiguous()
        x, y = self.multiHead(x,x,x)
        x = x.view(-1, 100*35)
        x = self.swish(self.fc1(x)) #also used relu, the same problem
        x = self.dropout1(x)
        x = F.softmax(self.fc2(x))
        return x
</code></pre>
<p>So, my input looks like (Batch, Timesteps, Features). For example:</p>
<pre><code class="lang-auto">input = torch.Tensor(np.random.random(size=(5, 100, 35)))
</code></pre>
<p>So, it seems to work with simple RNN and it  is actually learning, when this model do not learn.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = lr)
</code></pre>
<p>What is the problem, why model does not learn?<br>
Thanks in advance!</p>

username: mailcorahul
cooked: <p><a class="mention" href="/u/aldeka12">@aldeka12</a> Remove <code>F.softmax</code> at the end of the network. <code>nn.CrossEntropyLoss</code> internally applies softmax.</p>

username: aldeka12
cooked: <p>Hello, thanks for mentioning, did not know this! But still problem not in this, probably my dataset is not really suited for this kind of model.</p>

username: mailcorahul
cooked: <p>Alright.<br>
I am afraid I won’t be able to help you here, as I am not that much aware of RNNs.</p>
How to use batch in LSTM

username: comebeef0303
cooked: <p>Hi everyone.<br>
I am an absolute beginner of Neural Network and would like to try to use LSTM for predicting the last point of noised sin curve at first.<br>
But, I am confused about how to use batch method because the prediction accuracy used batch is worse than one not used batch.<br>
In the case of that total training data and epochs are 100 and batch is off, the accuracy reaches 100% about epochs=50, however, batch size=10, it reaches 25% about epochs=100 at best.<br>
On top of that, training data is 1000, the accuracy reaches 100% about epochs=50<br>
I think the reason why this is that optimized method is iterated for 100 cycles since training data is 1000.<br>
But, if it is true, the calculation unused batch is better than that of used batch and I am confusing.<br>
So, could you give me some advises?</p>
<p>My easy code is shown as below and “batch_size” is bigger than 1 if batch method is used.</p>
<p>class Net(nn.Module):<br>
“”“LSTM”""<br>
def <strong>init</strong>(self, inputDim, hiddenDim, outputDim):<br>
super(Net, self).<strong>init</strong>()<br>
self.rnn = nn.LSTM(input_size=inputDim,batch_first=True)<br>
self.output_layer = nn.Linear(hiddenDim, outputDim)</p>
<pre><code>def forward(self, inputs, hidden0=None):
    output, (hidden, cell) = self.rnn(inputs, hidden0)
    output = self.output_layer(output[:, -1, :])
    return output
</code></pre>
<p>def mkDataSet(data_size, data_length=50, freq=60., noise=0.02):<br>
train_x = []<br>
train_t = []</p>
<pre><code>for offset in range(data_size):
    train_x.append([[math.sin(2 * math.pi * (offset + i) / freq) +np.random.normal(loc=0.0, scale=noise)] for i in range(data_length)])
    train_t.append([math.sin(2 * math.pi * (offset + data_length) / freq)])

return train_x, train_t
</code></pre>
<p>def mkBatch(train_x, train_t, batch_size, step):<br>
batch_x = []<br>
batch_t = []</p>
<pre><code>for i in range(batch_size):
    idx = step * batch_size + i
    batch_x.append(train_x[idx])
    batch_t.append(train_t[idx])

return torch.FloatTensor(batch_x), torch.FloatTensor(batch_t)
</code></pre>
<p>def main():<br>
“”“main routine”""<br>
training_size = 100<br>
epochs_num = 100<br>
hidden_size = 5<br>
batch_size = 10<br>
train_x, train_t = mkDataSet(training_size)<br>
model = Net(1, hidden_size, 1)<br>
criterion = nn.MSELoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.01)<br>
for epoch in range(epochs_num):<br>
running_loss = 0.0<br>
training_accuracy = 0.0</p>
<pre><code>    for i in range(int(training_size / batch_size)):
        #Initialize
        optimizer.zero_grad()
        data, label = mkBatch(train_x, train_t, batch_size, i)
        output = model(data)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        training_accuracy += np.sum(
            np.abs((output.data - label.data).numpy()) &lt; 0.1)

return 0</code></pre>
Model always allocate some memories on gpu:0

username: IcarusWizard
cooked: <p>I have a workstation with 3 gpus. Whenever I run a model on the device other than gpu:0 (say gpu:1), the model still allocates some additional memories on gpu:0 (from my observation, it varies between 600M to 900M, which seems depend on the model I am training). I call it additional memories because when I run the same model on the gpu:0, this size of the memories won’t be allocated.<br>
The model can run, but the problem is a little annoying. Does anyone knows what’s going on here.</p>

username: JuanFMontesinos
cooked: <p>Initialization depends on gpu’s model, some need more some less.</p>

username: IcarusWizard
cooked: <p>I don’t get it. If the memory is used for intialization, the memory should be release as soon as the initialization process complete, right? Or, I should manually release the memory by some hack?</p>

username: JuanFMontesinos
cooked: <p>It’s something that CUDA needs to work. It is not like a 1st step which is realeased rather than loading packages do make that specific GPU to work.<br>
As I aforementioned it depends on the model but that behavior is okay.</p>

username: IcarusWizard
cooked: <p>Alright, it seems like a wired behavior for me, because the model don’t need the memory at all after initiallization. Is there any API I can use to manually release the memory?</p>

username: JuanFMontesinos
cooked: <p>It’s not a pytorch issue rather than nvidia’s. I don’t think they would waste memory since it’s a widely stable and developed librabry but your question is out of my knowledge. <a class="mention" href="/u/ptrblck">@ptrblck</a> may help you as he is from nvidia I think.</p>

username: IcarusWizard
cooked: <p>Thanks! I think maybe I should not be too worry about it.</p>

username: JuanFMontesinos
cooked: <p>It’s okay <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Being curious is an excellent way to learn, if you discover it write me back!</p>

username: ptrblck
cooked: <p>Based on the size, the CUDA context seems to be initialized on GPU0 (and I thought we got rid of this, but cannot find the issue on GitHub).</p>
<p>Anyway, if you only want to use a specific device, you execute your script via</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=1,2 python script.py
</code></pre>
<p>to mask all other devices.</p>
<p>Note that internally in your script, the GPUs will be remapped starting at index0.</p>
nn.DataParallel with GANs

username: arlindi
cooked: <p>Hello all,</p>
<p>I have a question regarding the usage of <code>DataParallel</code>:</p>
<p><code>model = nn.DataParallel(model)</code></p>
<p>When working with GANs, I use the above line where model is replaced by the Generator and Critic respectively. However, except for the opportunity to use a higher batch size as there is more memory, there is no actual speedup for the training process.<br>
In fact, it is slower then training the whole network on a single GPU.<br>
Is this expected when working with generative adversarial networks or networks in general?<br>
The GPUs that I am running on are the same, so I do not see the possibility of a bottleneck by a slower GPU.</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>Depending on the workload you are dealing with, you might see the communication overhead using <code>nn.DataParallel</code>.<br>
Could you post some information on the models and data (batch size) you are currently using?</p>

username: arlindi
cooked: <p>The network architecture that I am using is this one: <a href="https://github.com/bluestyle97/STGAN-pytorch" rel="nofollow noopener">https://github.com/bluestyle97/STGAN-pytorch</a><br>
It is a port of STGAN to PyTorch.<br>
When training on one GPU, I use a batch size of 32. When training on 2 GPUs, I use a batch size of 64, as to still keep 32 samples on each GPU.<br>
I know that the speedup will not be by a factor of 2, however, the network is slower by a factor of 4, compared to running it on a single GPU.</p>
No more hyperparameters

username: vainaijr
cooked: <p>How do I make all hyperparameters, learnable, for example, number of epochs, learning rate, normalization mean, std, image size, random crop size, make all of these learnable.</p>
<p>No hyperparameters in my training.</p>
<p>How do I make every hyperparameter a nn.Parameter()</p>
<p>So that computer figures these things out on its own.</p>

username: JuanFMontesinos
cooked: <p>It’s not possible unless you discover a way to make networks which learns to predict those parameters.</p>

username: lmbk
cooked: <p>Exactly, have a look to the state of the art posts <a href="https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a" rel="nofollow noopener">here</a>. The thing is to find the metric that will indicates if your training is good and minimize this value. You can apply these concepts to your problem.</p>
Confusion about tensor's grad when execute backward

username: wwwangzhch
cooked: <p>Hi,<br>
Today, when I tried to replicate the ddpg experiment, I found a proble that puzzled me.<br>
Let suppose “actor” and “critic” are the networks, and “state” represents the input tensor.<br>
when I use</p>
<blockquote>
<p>actions = actor(state)<br>
var_actions = torch.tensor(actions.data, requires_grad=True)<br>
q = critic(state, var_actions)<br>
q.backward(torch.ones(q.size())</p>
</blockquote>
<p>after q.backward(), I can find var_actions.grad become a tensor(which is the grad of var_actions), but when I use</p>
<blockquote>
<p>actions = actor(state)<br>
q = critic(state, actions)<br>
q.backward(torch.ones(q.size())</p>
</blockquote>
<p>after q.backward(), when I check  the grad of “actions”, the value is None</p>
<p>I can’t find the problem, it seems the only diference is whether  to create a new tensor?</p>

username: dejanbatanjac
cooked: <p>PyTorch creates a <strong>dynamic computational graph</strong> (like a tree) when calculating the gradients.<br>
The <em>leaves</em> of this tree are <strong>input tensors</strong>. For leaves, you can only get the gradients if I am right.</p>
<p>Gradients are calculated by tracing the graph from the root (output tensor) to the leaf and multiplying every gradient in the way using the <strong>chain rule</strong> .</p>
<p>I created several examples in <a href="https://discuss.pytorch.org/t/using-torch-no-grad-inside-forward-function/57969/7">here</a>.</p>
<p>In your example,</p>
<pre><code class="lang-auto">var_actions = torch.tensor(actions.data, requires_grad=True)
</code></pre>
<p>will be a leaf so this is why you are getting the gradients. In non leaf case, you will get <code>None</code>, as you confirmed.</p>
Saving the right model for testing

username: Er_Hall
cooked: <p>Hi all,</p>
<p>I don’t know if this pytorch-related forum is the right to ask this kind of question I hope is ok.</p>
<p>I have seen several implementations in deep learning that save the model with the minimum validation loss and use this model for testing but I have noticed that my training loss is even smaller, is right to use the model with the minimum training loss? maybe my question is silly bus I am newbie in this area.</p>
<p>Also, using the following command I save where? the model, in which directory</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), './state_dict.pt')
</code></pre>

username: ptrblck
cooked: <p>Usually you would save the model with the lowest validation loss, since the lowest training loss could yield to overfitting.</p>
<aside class="quote no-group" data-username="Er_Hall" data-post="1" data-topic="58297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/er_hall/48/16234_2.png" class="avatar"> Er_Hall:</div>
<blockquote>
<p>Also, using the following command I save where? the model, in which directory</p>
</blockquote>
</aside>
<p>The path (and directory) is given as the second argument. In your example, you are using the current working directory via <code>./</code></p>
Question about tensorboard_tutorial

username: Wang_Octavius
cooked: <p>As you mentioned in <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a>, when plotting the precision-recall curve, you take tensorboard_preds as input of writer.add_pr_curve. I have confusions on this, should not we use the ground truth label instead of tensorboard_preds which is the predicted output of the model?</p>
Dropout to inbuilt reset model

username: Ajinkya_Ambatwar
cooked: <p>Hi, I was working with CIFAR-10 dataset and Resnet18 model. I am using inbuilt model architecture and calling it as</p>
<pre><code class="lang-auto">net = torchvision.models.resnet18(pretrained=True) 
</code></pre>
<p>I am using ImageNet weights and training the entire model. I wanted to add dropout to the network after various layers. I don’t know how to add dropout to the inbuilt network called. And the network summary shows there’s only one FC(or linear) layer in the network in the end. So where can we add dropout in the network? As per my knowledge, isn’t it supposed to be added to the FC layers in the network?</p>
<p>Any suggestions around that?<br>
Thanks!</p>
Model size is not consisitent while saving

username: Mendel123
cooked: <p>I’m using torch.save to save my model and I find that when I save it in one file, the size is about 24 MB. while I save each parameter in a single file,the sum of the sizes is about 50MB. Why does this happen?</p>
<pre><code class="lang-auto">model = Net()
torch.save(model.state_dict(),'model.pth')
# 24 MB
</code></pre>
<pre><code class="lang-auto">model = torch.load('model.pth')
for k,v in model.items():
    torch.save(v,k+'.pth')
# sum of sizes is 50 MB
</code></pre>
<p>Thanks a lot.</p>

username: ptrblck
cooked: <p>The pickle module will most likely add some overhead for each file.<br>
How many items are you storing?</p>

username: Mendel123
cooked: <p>Thanks for you reply.It’s about twenty files as I store each parameter in one file.</p>
Not able to run prediction using Pytorch example from the website

username: alx
cooked: <p>Hi guys, so I’m using Pytorch’s prediction template to predict my model’s accuracy but when I add “<strong>print(f’{correct}/{total}’)</strong>” to see the results from batches I get:<br>
104/128<br>
208/256<br>
312/384<br>
416/512</p>
<p>Since the increment of correct result is exactly the same, I’m thinking there’s an issue somewhere. What’s wrong?</p>
<p>I added my data loader, dataset and trsfrm below:</p>
<pre><code class="lang-auto">test_trans = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(_image_size),
    transforms.ToTensor(),
    transforms.Normalize(_mean, _std),
])
</code></pre>
<pre><code class="lang-auto">test_ds = DiabetesDataset("./tmp/test", "retinopathy_solution.csv", transform=test_trans)
</code></pre>
<pre><code class="lang-auto">test_dl = torch.utils.data.DataLoader(test_ds, batch_size=128,
                                         shuffle=False, num_workers=4)
</code></pre>
<pre><code class="lang-auto">class DiabetesDataset(Dataset):
    def __init__(self, root_dir, csv_file, transform=None, loader=default_loader):
        self.root_dir = Path(root_dir) if type(root_dir) is str else root_dir
        self.image_names = list(self.root_dir.glob('*.jpeg'))
        self.transform = transform
        if csv_file:
            labels = pd.read_csv(csv_file)
            print(labels)
            self.labels = dict(zip(list(labels.image), list(labels.level)))
        else:
            self.labels = None
        self.loader = loader
    def __len__(self):
        return len(self.image_names)
      
    def __getitem__(self, index):
        image_path = self.image_names[index]
        image = self.loader(image_path)
        if self.transform:
            image = self.transform(image)

        if self.labels:
            label = self.labels[image_path.stem]
            return image, label
        else:
            return image, image_path.stem
</code></pre>
<p>Code from pytorch’s website:</p>
<pre><code class="lang-auto">dataiter = iter(test_dl)
images, labels = dataiter.next()

correct = 0
total = 0

with torch.no_grad():
    model.eval()
    for data in test_dl:
        images, labels = images.to(DEVICE), labels.to(DEVICE)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        print(f'{correct}/{total}')

print('Accuracy of the model test images: %d %%' % (100 * correct / total))
</code></pre>

username: tdchaitanya
cooked: <p>Could you please share a link to the code on Pytorch website? It looks like you are iterating over a single batch of data only and repeatedly making predicitions on the same thing.</p>
GPU 0 memory usage in DistributedDataParallel

username: Arun_Mallya
cooked: <p>The amount of memory used on GPU 0 seems to go up linearly with number of GPUs for my model.<br>
For DDP with 1 GPU, the memory usage is 8 GB, with 8 GPUs is 12 GB.<br>
I am not computing any particular values just on GPU 0, and the whole model is inside DDP.</p>
<p>Is this normal or expected?</p>
Model saving & loading trouble

username: coyote
cooked: <p>Hi all,</p>
<p>I have a sequence tagger built upon a Hugging Face’s bert based model.</p>
<pre><code class="lang-auto">class SequenceTagger(torch.nn.Module):
    def __init__(self,bert_dim,n_labels,bert_model_dir,dp=0.5):
        super(SequenceTagger,self).__init__()
        self.bert = BertModel.from_pretrained(bert_model_dir)
        self.out = torch.nn.Linear(bert_dim, n_labels)
        self.dropout = nn.Dropout(dp)
        self.log_softmax = nn.LogSoftmax(dim=2)
        self.nll_loss = nn.NLLLoss(ignore_index=-1)
</code></pre>
<p>After I trained the model, I save it as follows:</p>
<pre><code class="lang-auto">model = SequenceTagger(.....)
model = train(model)
</code></pre>
<pre><code class="lang-auto">            torch.save({
                "epoch": i,
                "model_state_dict": model.state_dict(),
                "optimizer_state_dict": optimizer.state_dict(),
                "best_avg_recall": current_avg_recall
            }, args.model_save_dir.split)
</code></pre>
<p>And then I load it as follows:</p>
<pre><code class="lang-auto">        model = SequenceTagger(bert_dim,len(self._l2ix),model_dir)        
        model.load_state_dict(torch.load(model_full_path)["model_state_dict"])
        model.to(self._device)
        model.eval()
</code></pre>
<p>It works smoothly if I do not change the directory of the saved model. However, if I copy the saved model somewhere else and use the copied path as argument to load function, I get</p>
<blockquote>
<p>RuntimeError: Error(s) in loading state_dict for SequenceTagger:<br>
Unexpected key(s) in state_dict:</p>
</blockquote>
<p>error. When I checked the official docs, I see this is the recommended way for serialization. What could be the problem ?</p>

username: ptrblck
cooked: <p>Do you get some unexpected keys in the error message or does it end right after what you’ve posted?</p>
Save BoolTensor to image

username: iPsych
cooked: <p>I have the tensor like below.</p>
<pre><code class="lang-auto">tensor([[False, False, False, ..., False, False, False].
           [False, False, False, ..., False, False, False].
           ......,
           [False, False, False, ..., False, False, False].
           [False, False, False, ..., False, False, False].
           [False, False, False, ..., False, False, False]])

torch.Size([15.480,640])
</code></pre>
<p>How can I save this tensor to 15 Black-and White image?</p>

username: ptrblck
cooked: <p>What do you mean by “save”?<br>
If you would like to mask the images at the <code>False</code> positions, you could simply multiply both tensors:</p>
<pre><code class="lang-python">mask = torch.randn(2, 4, 4) &gt; 0.
imgs = torch.randn(2, 4, 4)

masked_imgs = imgs * mask
</code></pre>
Cycle detection in computation graph during fusion

username: vdantu
cooked: <p>When nodes are merged (graph fusion pass for example), how are cycles detected/avoided? Is it taken care by moveAfterTopologicallyValid in alias analysis? What algorithm is used here? Any pointers would be highly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tom
cooked: <p>What do you mean by cycles?<br>
The graphs are acyclic and we loop until there is no more nodes to add to a fusion group…</p>
<p>Best regards</p>
<p>Thomas</p>

username: vdantu
cooked: <p>Lets consider a graph, like the following,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edda35f3b8f40cd46091976ba2e7c932bbf854fe.png" alt="image" data-base62-sha1="xW8FQd0FivT7IyOx9V8xFWIadk2" width="284" height="162"><br>
(Copied from <a href="https://cs.stackexchange.com/questions/71369/edge-contraction-in-dag" rel="noopener nofollow ugc">here</a>)</p>
<p>If we combine 1, 3, 4 red nodes above, we form a loop right? Considering 1, 3, 4 are can be grouped, what prevents the fusion from happening? If my question doesn’t make sense, please feel free to correct me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> .</p>

username: Michael_Suo
cooked: <p>Yes, <code>moveAfterTopologicallyValid</code> takes care of that. It does dependence analysis to make sure that this merging is disallowed.</p>

username: vdantu
cooked: <aside class="quote no-group" data-username="Michael_Suo" data-post="4" data-topic="58312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7993a0/48.png" class="avatar"> Michael_Suo:</div>
<blockquote>
<p>takes care of that. It does dependence analysis to make sure that this merging is disallowed.</p>
</blockquote>
</aside>
<p>Cool. Thanks for the response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">.</p>
rnnFusedPointwise no longer available in >=PyTorch 1.0

username: wasiahmad
cooked: <p>I had a codebase that runs fine in PyTorch 0.4 but whenever I try to run it using PyTorch 1.0+, it complains about the following import.</p>
<blockquote>
<p>from torch.nn._functions.thnn import rnnFusedPointwise as fusedBackend</p>
</blockquote>
<p>I use <code>fusedBackend</code> in the following code snippet.</p>
<pre><code>if input.is_cuda:
    igates = F.linear(input, w_ih)
    hgates = F.linear(hx, w_hh)
    state = fusedBackend.LSTMFused.apply
    return state(igates, hgates, cx, b_ih, b_hh)
</code></pre>
<p>Is there any way I can fix the import error for PyTorch 1.0+?</p>
Advice for debugging a TorchScript module from C++

username: markl
cooked: <p>I have trained a model from python and now I’m trying to load it from a c++ program. Unfortunately it crashes on the line <code>// crash here</code></p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;iostream&gt;
#include &lt;vector&gt;
#include "nets/nets.h"
#include "util/runfiles.h"

int main(int argc, char** argv) {
  std::cout &lt;&lt; "Nets example" &lt;&lt; std::endl;

  // Custom code that loads the module
  auto runfiles = MakeRunfiles(argv[0]);
  torch::jit::script::Module segnet3 = LoadSegNet3(*runfiles);
  std::cout &lt;&lt; "Loaded SegNet3" &lt;&lt; std::endl;

  // Make a fake image.
  torch::Tensor input = torch::randn({1, 3, 300, 300});
  std::cout &lt;&lt; "Made random input" &lt;&lt; std::endl;

  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back(input);

  // crash here
  torch::Tensor output = segnet3.forward(inputs).toTensor();
  std::cout &lt;&lt; "Output done" &lt;&lt; std::endl;
}

</code></pre>
<p>The output of the program is</p>
<pre><code class="lang-auto">Nets example
Loaded SegNet3
Made random input

</code></pre>
<p>and it exits without writing any error message.</p>
<p>What can I do to debug the situation? I am wondering if maybe my input tensor is of the wrong dimensionality. Is there a way to check the dimensions the network is expecting? And/or is there a way to print some debug information about the structure of the network?</p>

username: tom
cooked: <p>A segfault should not happen with that program even when the shapes do not match.<br>
Most often segfaults happen when people use <code>from_blob</code> and the memory-owner goes out of scope, but this is not case here.</p>
<ul>
<li>Does this also happen with a small / trivial module? If not, there might be a specific operation that it does not like. Ideally you would be able to isolate it.</li>
<li>If it does even for a trivial module, you could compile libtorch / PyTorch in Debug mode and run it with <code>gdb -ex run --args myprogram</code> to get a backtrace (with <code>bt</code> after the segfault).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: markl
cooked: <p>Thanks tom. It’s not so easy for me to use gdb since I’m on Windows, but I did happen to discover the issue. In my loading code, I had moved the model by calling <code>.to(at::kCUDA)</code> so the tensor needed to be moved too before.</p>

username: tom
cooked: <p>Great you figured that out. Is it crashing as in “segfault” or does it nominally print an error message on the console?</p>
<p>Best regards</p>
<p>Thomas</p>

username: markl
cooked: <p>There actually is no error message - it exits silently</p>
Deciphering weird train-valid loss curve

username: banikr
cooked: <p>Hello All,<br>
I am using a network combining u-net and residual architecture for regression. The network has 14 3D Conv layers and layers are kinda in sequence as following:</p>
<p>CONV&gt;BN&gt;ReLU</p>
<p>From an 86x110x78 Image, I am regressing to a vector of size 512.<br>
So the last two layers are fully connected layers.<br>
Starting with a batchsize of 1(batchsize 4 generates <code>CUDA out of memory error</code>), learning rate = 0.05.<br>
I am getting the following loss curves while training:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a8084c8f02fb2618c02ccb29493aadfcd09a952.png" alt="Figure_2" data-base62-sha1="8lx2GupgXp2BGdvgxlTKWb68Q5c" width="640" height="480"><br>
Close up look at validation loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35d6b11930204fbf4e15e9e11f8140e980cd127a.png" alt="valid2" data-base62-sha1="7Ghhq3UA6AKsz8NscZzNYQJJQOC" width="640" height="480"></p>
<p>Can anyone explain why the train loss has a drop like that? Is it overfitting right after the first epoch? shouldn’t the validation loss be larger at the beginning?<br>
n.b. not using any Dropout layers…</p>