akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
If I assign tensor A to a part of tensor B, will that part of tensor B contains the same gradient information as tensor A?

SamChen: <p>Let tensor A has gradient information. If I run “B[:A.size()[0]] = A”, does B[:A.size()[0]] contains the same gradient information as A?</p>
<p>I am working on implement bidirectional GRU layer from scratch. I want it can take PackedSequence as input x.  To deal with the variable batch_size for each step, I write the following code for reverseRNN layer:</p>
<pre><code>    for batch_size in reversed(batch_sizes):
        step_state = global_state[:batch_size]
        step_input = input[input_offset - batch_size : input_offset]
        input_offset -= batch_size
        
        out, step_state = self.cell(step_input, step_state)
        outputs = [out] + outputs
        global_state[:batch_size] = step_state
</code></pre>
<p>The <code>global_state</code> stores states’ situation for all sequences. And I generate PackedSequence using pack_sequence with the default setting.</p>
<p>My idea is if global_state can always get up-to-date gradient information from the current iteration’s step_state, the backpropagation should work correctly.</p>

albanD: <p>Hi,</p>
<p>No it won’t. It will contain the gradient information associated to how you use this new copy.</p>
<p>Why is this global_state necessary here? Why not just pass step_state to the next iteration?</p>

SamChen: <p>Because, in a batch, there is no guarantee that all sequence are the same length.</p>

albanD: <p>I’m confused about your example actually.<br>
I’m not sure to understand why there is a global state that is associated with different samples in your batch. Why would the ith entry in one batch should be associated with the ith entry in another batch (through your use of the global_state)?</p>
Global averaging nonzero vectors from 4d feature map

Hakjin_Lee: <p>I want to average nonzero vectors by image resolutiom.<br>
For example,<br>
there are feature map x(4, 81, 3, 3) and mask y(4, 1, 3, 3).<br>
I could generate masked feature by x*y.</p>
<p>Now, I want to average only masked vectors with dim 2, 3, so the dimension of output is (4, 81, 1, 1)</p>
<p>I tried it by torch.masked_index, but the result does not match to (batch, channel, -1).</p>
<p>Can I ask you some solutions?</p>
Convolutions-like sliding window operations

Sia_Rezaei: <p>I have a use-case where I need to do other operations on a sliding window.<br>
For example, regular convolution calculates the dot-product between the kernel weights and the sliding window, but now I want to calculate the euclidean distance (or sum, or …) of the kernel weights with the sliding window. Is there a nice way to do this?<br>
That is using the sliding window functionality of the convolutions layers but replacing the dot product with something else.</p>
<p>Thanks</p>

Sia_Rezaei: <p>I think I found an answer.<br>
It seems like the right way to do it is to use <code>unfold</code> and <code>fold</code> functions.<br>
With the <code>unfold</code> function one can create the same sliding window slices that a convolution operation slides over.<br>
But, one thing that remains unclear is how I can apply my own function (e.g. euclidean distance) efficiently (e.g. without a python for-loop) over those slices.</p>
Pip install torch error

Cagri_Kaplan: <p>$ python3 -V<br>
Python 3.4.3</p>
<p>Ubuntu</p>
<p>When I pip install torch, I receive this error:</p>
<p>ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cu92 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch==1.2.0+cu92</p>
<p>Does anyone know the reason?</p>
How to implement two similar network models with untied parameters?

okreshma: <p>I am trying to implement two similar network models with two LSTM layers whose parameters are not tied.</p>
<p>For eg., If I create two instances of the same model as:</p>
<p>model1 = Model()<br>
model2 = Model()<br>
Would the parameters of the above networks be tied? Kindly suggest a way to solve this.</p>

JTiC: <p>They wouldn’t!!!(for 20 characters.)</p>

okreshma: <p>Thank you <a class="mention" href="/u/jtic">@JTiC</a> .<br>
So the parameters won’t be tied, but I didn’t understand why it is specified as 20 characters.</p>

okreshma: <p>I tried comparing the parameters of the two models with the function described in the following post.</p>
<p><a href="https://discuss.pytorch.org/t/check-if-models-have-same-weights/4351/6">check if models have same weights</a></p>
<p>And I got the result as “mismatch”. The comparison of the parameters was done after inference and before training. Is this the proper method to check if the parameters are tied or not? Kindly suggest the correct approach for the same.</p>
Concurrent training and predicting

MadmanNero: <p>I wonder if it is safe to continuously train a model while using it for making predictions at the same time (in different threads) or whether it’s necessary to pause predictions while retraining the model?</p>
<p>Any opinions or references on this?</p>

Tahir: <p>technically its possible i think. During training you are making predictions too, to calculate the loss. However, whether it will give good, reliable and predictable results, thats another question.</p>
<p>You can access the model under training in a different thread and use locks while reading the data to make sure you get to use the model in a valid state while it is still being updated.</p>
One error about the utils.py's code

HUSTHY: <p>today,when I use python 3.7, pytorch 1.2 and torchtext 0.4.0, a error occur.<br>
OverflowError: Python int too large to convert to C long.<br>
reason is about : maxInt = sys.maxsize &gt; somevalue, in utils.py code is like this:<br>
maxInt = sys.maxsize<br>
while True:<br>
# decrease the maxInt value by factor 10<br>
# as long as the OverflowError occurs.<br>
try:<br>
csv.field_size_limit(maxInt)<br>
break<br>
except OverflowError:<br>
maxInt = int(maxInt / 10)</p>
<pre><code>csv.field_size_limit(sys.maxsize)
</code></pre>
<p>I correct this codes  as follows:<br>
maxInt = sys.maxsize<br>
while True:<br>
# decrease the maxInt value by factor 10<br>
# as long as the OverflowError occurs.<br>
try:<br>
csv.field_size_limit(maxInt)<br>
break<br>
except OverflowError:<br>
maxInt = int(maxInt / 10)</p>
<pre><code># csv.field_size_limit(sys.maxsize)
csv.field_size_limit(maxInt)
</code></pre>
<p>then, the erroe is disapper.!<br>
<a href="/uploads/short-url/5tgbwEREKPTngk7VnclMJszfv4m.png">codeerror|690x419</a></p>

ptrblck: <p>Is this error related to PyTorch or some other package to read csv files?</p>

HUSTHY: <p>This error related to torchtext package to red csv files.<br>
The function：<br>
train, valid, test = data.TabularDataset.splits(path=’…/CSVdata’,<br>
train=‘tr_cut_csv.csv’,<br>
validation=‘va_cvs.csv’,<br>
test=‘te_cut_csv.csv’,<br>
format=‘csv’,<br>
skip_header=True,<br>
fields=[(‘text’, TEXT),(‘label’, LABEL)]<br>
)<br>
when I debug this function, it will jump to  this functon<br>
def unicode_csv_reader(unicode_csv_data, **kwargs) which in the torchtext  utils.py.<br>
Above all, this error occurs in my computer. and I think the logisitc is not good in the code:<br>
maxInt = sys.maxsize<br>
while True:<br>
# decrease the maxInt value by factor 10<br>
# as long as the OverflowError occurs.<br>
try:<br>
csv.field_size_limit(maxInt)<br>
break<br>
except OverflowError:<br>
maxInt = int(maxInt / 10)</p>
<pre><code>csv.field_size_limit(sys.maxsize)
</code></pre>
<p>it had better to be changed for :<br>
maxInt = sys.maxsize<br>
while True:<br>
# decrease the maxInt value by factor 10<br>
# as long as the OverflowError occurs.<br>
try:<br>
csv.field_size_limit(maxInt)<br>
break<br>
except OverflowError:<br>
maxInt = int(maxInt / 10)<br>
csv.field_size_limit(maxInt)</p>
<p>also,this error is not about pytorch. sorry!</p>
Cuda runtime error (30)

Kim_KA: <p>File “/home/ka/.local/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py”, line 179, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:50</p>
<p>pytorch 1.2.0<br>
cuda 9.2<br>
python 3.6</p>
<p>Thanks in advance</p>

ptrblck: <p>Could you answer these questions to make debugging a bit easier:</p>
<ul>
<li>Which GPU and NVIDIA driver are you using?</li>
<li>Was PyTorch working before?</li>
<li>What does <code>nvidia-smi</code> output?</li>
</ul>

Kim_KA: <p>GeForce GTX 1080</p>
<p>Yes earlier it used to work.</p>
<p>NVIDIA-SMI 396.54<br>
Driver Version: 396.54</p>
How to access input/output activations of a layer given its parameters names?

MahdiNazemi: <p>I was wondering if it is possible to get the input and output activations of a layer given its parameters names.</p>
<p>For example, assume a weight tensor is called <code>module.fc3.weights</code>. Can I access the inputs and outputs of the layer which contains the said weight tensor?</p>
<p>I only need to do this once for a pertained neural network and therefore, good performance is not a concern.</p>

ptrblck: <p>You could use <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">forward hooks</a> and use the parameter name to register them.<br>
Let me know, if that would work for you.</p>

MahdiNazemi: <p>Thank you for the answer. This should solve the problem. I just need to find a method to iterate all layers within the neural network and add this hook automatically.</p>
<p>I have two follow up questions:</p>
<ul>
<li>If the activation function is defined in a container like sequential, e.g. nn.ReLU(), does the <code>output</code> consider such activation function or not? In other words, is the output before or after activation function is applied?</li>
<li>Would I be able to combine multiple layers into one? For example, if a layer is followed by batch normalization, can I get the output after batch normalization is applied?</li>
</ul>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-register-forward-hooks-for-each-module/43347/2">This approach</a> might work to register hooks for all modules.</p>
<ol>
<li>
<p>If you are using out of place activations, the <code>output</code> will non-linearity will be applied on the input and will return the output you could clone into your <code>dict</code>. However, if you are using <code>inplace=True</code>, note that also the input will be manipulated in-place.</p>
</li>
<li>
<p>Yes, you can pass any module (containing other submodules) and register the hook to this outer output.</p>
</li>
</ol>
Input data type for conv1d

Driss_EL_Alaoui: <p>can i use the tf-idf matrix as input to the conventional auto-encoder composed of conv1d layers?<br>
here is the form of my df-idf data</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tfidf.shape
&gt;&gt;&gt;(3047,500)
</code></pre>
<p>I use a batch of size 64  , and i reshape the input  by this line</p>
<pre><code class="lang-auto">x = x.reshape(x.shape[0], x.shape[1],1)
</code></pre>
<p>the input is become of this form</p>
<pre><code class="lang-auto">input[64, 500, 1]
</code></pre>
<p>my class are as follows</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, input_size, intermediate_size, encoding_size):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv1d(500, 100, 1),
            nn.BatchNorm1d(100),
            nn.ReLU(True),
            nn.MaxPool1d(kernel_size=1, stride=1),
            nn.Dropout(0.2),
          
            nn.Conv1d(100, 50, 1),
            nn.BatchNorm1d(50),
            nn.ReLU(True),
            nn.MaxPool1d(kernel_size=1, stride=1),
            nn.Dropout(0.2))

    def forward(self, x):
        x = x.reshape(x.shape[0], x.shape[1],1)
        x = self.encoder(x)
        return x


class Decoder(nn.Module):
    def __init__(self, output_size, intermediate_size, encoding_size):
        super().__init__()
        self.decoder = nn.Sequential(
            nn.ConvTranspose1d(50,100, 1),
           
            nn.BatchNorm1d(100),
            nn.ReLU(True),
            nn.Dropout(0.2),
            
            nn.ConvTranspose1d(100, 500, 1),
            nn.BatchNorm1d(500),
            nn.Sigmoid())

    def forward(self, x):
        #x = x.view(1, 1, -1)
        x = self.decoder(x)
        return x

</code></pre>
<p>but it shows me this error</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (500) must match the size of tensor b (64) at non-singleton dimension 1
</code></pre>
<p>Sincerely, I do not know what’s my problem here? ? can you help me please ?</p>

gioperin: <p>Can you tell us what is tensor a and what is tensor b and which are their full shapes?</p>
Using one loss function in two loss evaluation circumstances

Yupjun: <p>I want to build a model that evaluates two loss functions with the same l1 metric.</p>
<p>which way is a more pythonic method(or Pytorch method?)</p>
<pre><code class="lang-auto"># status 1
criterion1 = nn.L1Loss()
criterion2 = nn.L1Loss()

criterion1(input_1,output_1)
criterion2(input_2,output_2)
loss.backward()
</code></pre>
<pre><code class="lang-auto"># status 2
criterion1 = nn.L1Loss()

criterion1(input_1,output_1)
criterion1(input_2,output_2)
loss.backward()
</code></pre>

gioperin: <p>Both solutions are correct and wrong at the same time, because you are not defining the variable loss. If the two criteria are exactly the same one, you can create just one object and do:</p>
<pre><code>criterion = nn.L1Loss()

loss = criterion(input_1,output_1)
loss += criterion(input_2,output_2)

loss.backward()</code></pre>
MSELoss input/target

slavavs: <blockquote>
<blockquote>
<blockquote>
<p>loss = nn.MSELoss()<br>
input = torch.randn(3, 5, requires_grad=True)<br>
target = torch.randn(3, 5)<br>
output = loss(input, target)<br>
output.backward()</p>
</blockquote>
</blockquote>
</blockquote>
<p>If I do this? <code>output = loss(target, input)</code><br>
Swap target and input.<br>
Will this be a mistake?</p>

gioperin: <p>I would say that it depends on the loss: for the MSE loss it is probably the same because mse(a,b) = mse(b,a), as you are doing the square of the error. The same holds for the mae criterion, because you are taking the absolute value. But be careful: this does not hold for measures such as the kullback leibler divergence, for which kll(a,b) != kll(b,a).</p>
Question about Faster R-CNN ResNet_FPN backbone

8b97994524ae8234eb32: <p>I was wondering the function resnet_fpn_backbone defined in <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/detection/backbone_utils.py</a>, why freeze layers except layer2,layer3 and layer4? Why the first conv7x7 and layer1 also need to freeze?<br>
Thanks!</p>
Torch.nn model cannot pad a 1D tensor

john90: <p>I have a resnet that uses convolutions and nn.ReflectionPad1D. The data that I have is 1D (1045 long) with 2 channels (real and imaginary). All of the padding layers are defined using an integer, so the built-in functions convert that into a paired tuple, i.e. equal padding on both sides of the 1D vector. When I check the dimensions of the input using “input.dim()”, it of course equals 2: [1045, 2].</p>
<p>Here is the relevant portion of the error message:</p>
<blockquote>
<p>File “/home/john/Documents/Research/GAN/modules/loss_networks.py”, line 112, in setup<br>
input = autograd.Variable(gen(input).data)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/john/Documents/Research/GAN/modules/generator_networks.py”, line 63, in forward<br>
return self.model(input)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 91, in forward<br>
input = module(input)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/modules/padding.py”, line 163, in forward<br>
return F.pad(input, self.padding, ‘reflect’)<br>
File “/home/john/.local/lib/python3.6/site-packages/torch/nn/functional.py”, line 2181, in pad<br>
raise NotImplementedError(“Only 3D, 4D, 5D padding with non-constant padding are supported for now”)<br>
NotImplementedError: Only 3D, 4D, 5D padding with non-constant padding are supported for now</p>
<p>Process finished with exit code 1</p>
</blockquote>
<p>I looked at the source code and it says that 1D padding using reflection/replicaiton requires a 3D input tensor. I thought the point of nn.Reflection1DPad was to pad a 1D tensor, so I don’t understand why it requires a 3D input.</p>
<p>Can someone please explain what I am doing wrong? If you need anymore information (code, etc.), I’ll be happy to provide whatever you need.</p>
<p>Thank you in advance.</p>

liyz15: <p>Check the doc, <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.ConstantPad1d" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.ConstantPad1d</a>, input should be like <code>(N, C, W_{int})</code>. The first dimension should be batch size, in you case <code>N=1</code> so just reshape the input.</p>
Using DataParallel with functional API

hristo-vrigazov: <p>Let’s say I have the following example (modified from [Data parallel tutorial])(<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#simple-model" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#simple-model</a> )</p>
<pre><code class="lang-auto">class Model(nn.Module):
    # Our model

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)
        self.f = torch.ones(1)

    def forward(self, input):
        output = self.fc(input) * self.f
        print(self.f)
        print("\tIn Model: input size", input.size(),
              "output size", output.size())

        return output
</code></pre>
<p>wrapped in a <code>nn.DataParallel</code>.<br>
This results in this error when doing a forward pass, because of</p>
<pre><code class="lang-auto">RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #2 'other'
</code></pre>
<p>If I try to call <code>.cuda()</code> on the <code>f</code> field, it goes on the first cuda device, and then the forward pass does not work because they are on different devices:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    # Our model

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)
        self.f = torch.ones(1).cuda()
        

    def forward(self, input):
        output = self.fc(input) * self.f
        print(self.f)
        print("\tIn Model: input size", input.size(),
              "output size", output.size())

        return output
</code></pre>
<p>Log:</p>
<pre><code class="lang-auto">RuntimeError: arguments are located on different GPUs at 
</code></pre>
<p>So how do I use data parallel with the functional API?</p>

ptrblck: <p>Most likely <code>self.f</code> is not pushed to the right device, since it is neither registered as an <code>nn.Parameter</code> nor as a buffer (using <code>self.register_buffer</code>).<br>
Use the former case, if <code>self.f</code> should require gradients and the latter if not.</p>
<p>The manual <code>cuda()</code> call inside your <code>__init__</code> method won’t work, as the model will be sent to each GPU which was passed as <code>device_ids</code>.<br>
If you want to manually push the tensor (not necessary in your use case), you could use:</p>
<pre><code class="lang-python">output = self.fc(input) * self.f.to(input.device)
</code></pre>
<p>in your <code>forward</code>.</p>

hristo-vrigazov: <p>Thanks! That is exactly what the problem was. I am new to PyTorch and I had no idea I had to register it, but it definitely makes sense though, how would PyTorch know otherwise.</p>
Backwards hook dimensions unexpected

DDong: <p>Consider the following code:<br>
<code><br>
def hook(module, gradIn, gradOut):<br>
print(gradOut)<br>
print(gradIn)<br>
test_model = nn.Linear(3,1)<br>
test_model.register_backward_hook(hook)<br>
test_inp = torch.randn(3, requires_grad=True)<br>
out = test_model(test_inp)<br>
out.sum().backward()<br>
print(out)<br>
print(test_inp)<br>
</code><br>
The output is:<br>
<code><br>
(tensor([1.]),)<br>
(tensor([1.]), tensor([1.]))<br>
tensor([0.7014], grad_fn=)<br>
tensor([-0.0393, -1.4974,  1.3676], requires_grad=True)<br>
</code><br>
Why does the second output line have 2 elements when there are 3 inputs?</p>

liyz15: <p>This is a known issue: <a href="https://github.com/pytorch/pytorch/issues/12331" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/12331</a></p>
<p>Try use  <code>register_hook</code>  on each input independently.</p>
<pre><code class="lang-auto">def hook(grad):
    print(grad)

test_model = nn.Linear(3,1)
test_inp = torch.randn(3, requires_grad=True)
test_inp.register_hook(hook)
out = test_model(test_inp)
out.sum().backward()
</code></pre>
Difference in Model Performance when using Validation set/ Testing set

Animesh_Kumar_Paul: <p>I have implemented a PyTorch NN code for classification and regression.</p>
<p><strong>Classification:</strong><br>
a) Use stratifiedKfolds for cross-validation (K=10- means 10 fold-cross validation)<br>
I divided the data: as follows:<br>
Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.</p>
<p>b) Loss function = CrossEntropy<br>
c) Optimization = SGD<br>
d) Early Stopping where waittime = 100 epochs.</p>
<blockquote>
<p>Problem is:<br>
Baseline Accuracy = 51%<br>
Accuracy on Training set = 100%<br>
Accuracy on validation set = <strong>90%</strong><br>
Accuracy on testing set = <strong>72%</strong></p>
</blockquote>
<p>I don’t understand what are the reasons behind the <strong>huge performance difference in Testing data/ Validation data</strong>?</p>
<p>How can I solve this problem?</p>
<p><strong>Regression:</strong><br>
a) use the same network structure<br>
b) loss function = MSELoss<br>
c) Optimization = SGD<br>
d) Early Stopping where waittime = 100 epochs.<br>
e) Use K-fold for cross-validation.<br>
I divided the data: as follows:<br>
Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.</p>
<p><strong>&gt; Problem is:</strong><br>
Baseline MSE= 14.0<br>
Accuracy on Training set = 0.0012<br>
Accuracy on validation set = <strong>6.45</strong><br>
Accuracy on testing set = <strong>17.12</strong></p>
<p>I don’t understand what are the reasons behind the huge performance difference in Testing data/ Validation data?</p>
<p>How can I solve these problems? or Is this an obvious thing for NN/ depend on particular dataset?</p>

ptrblck: <p>You might see some generalization error in your first example.<br>
<a href="https://www.deeplearning.ai/content/uploads/2018/09/Ng-MLY01-12.pdf" rel="nofollow noopener">Chapter 40</a> of Ng’s Machine Learning Yearning might give you some more information of <a href="https://en.wikipedia.org/wiki/Generalization_error" rel="nofollow noopener">this</a> Wikipedia article.</p>
Cutoffs for Adaptive Softmax

tnguyen: <p>Are there any guidelines/articles as how to choose the cutoffs for adaptive softmax?</p>
<p>The class is here: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html</a>.</p>
<p>Thanks in advance.</p>
Is there a parser for strings of Sequential Models?

pinocchio: <p>I have a set of strings of sequential models e.g.</p>
<pre><code class="lang-auto">'Sequential( (conv1): Conv2d(3, 4, kernel_size=(3, 3), stride=(1, 1))
  (relu1): ReLU() (conv2): Conv2d(4, 2, kernel_size=(3, 3), stride=(1, 1)) (relu2):
  ReLU() (Flatten): Flatten() (fc): Linear(in_features=1568, out_features=10, bias=True)
  )'
</code></pre>
<p>I want to be able to parse it to extract the layers, the hyper parameters and sometimes even make a new model with that setting.</p>
<p>Is there an easy way to do this in PyTorch?</p>

jackparrySG: <p>There’s always regex! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>It looks as though you could split the string on whitespace + open bracket to have separate strings for each layer. Then write functions to read each type of layer. Not a particularly quick or clean solution though…</p>
<p>I don’t work with sequential models much, but is this basically the syntax you’d use to define one? If so, to make a new model with this setting you could just run mymodel = eval(mystring)</p>
Broken API in libtorch 1.2

dambo: <p>Hi,<br>
My old code which worked perfectly:<br>
<code>std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(s_model_name);</code><br>
Fails with libtorch 1.2. If you upgrade you have to change the code to:<br>
<code>torch::jit::script::Module module = torch::jit::load(s_model_name, torch::kCUDA);</code></p>
<p>You also have to change this:<br>
<code>torch::Tensor out_tensor = module-&gt;forward({input_tensor}).toTensor();</code></p>
<p>To this:<br>
<code>torch::Tensor out_tensor = module.forward({input_tensor}).toTensor();</code></p>
<p>It seems at FB QA exists only for PyTorch but not libtorch.</p>

MadmanNero: <p>In the documentation of the PyTorch C++ API it states</p>
<blockquote>
<p>At the moment, the C++ API should be considered “beta” stability; we may make major breaking changes to the backend in order to improve the API, or in service of providing the Python interface to PyTorch, which is our most stable and best supported interface.</p>
</blockquote>
<p>Guess that applies here…</p>

yf225: <p>Yes the PyTorch C++ API is still considered “beta” stability. Specifically, we changed <code>torch::jit::load(s_model_name)</code> to return <code>torch::jit::script::Module</code> instead of <code>std::shared_ptr&lt;torch::jit::script::Module&gt;</code> because <code>torch::jit::script::Module</code> is now a reference type, and this change is documented in the release note ("[jit] script::Module is now a reference type" in <a href="https://github.com/pytorch/pytorch/releases/tag/v1.2.0" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases/tag/v1.2.0</a>).</p>
Model.eval() Error -- 'bool' object is not callable

sban: <p>Hi,<br>
I am getting getting the following error when I try to evaluate my model after training:</p>
<p>In SegModel:  False                                                                                                                                                                                                ------Epoch 51 | Step 71 | Train Loss 0.1662866771221161 | Validation Loss 0.48616326600313187 -----------<br>
model loaded<br>
Traceback (most recent call last):<br>
File “tester.py”, line 61, in                                                                                                                                                                                model.eval()                                                                                                                                                                                                     File “/data/sbanerjee/anaconda_env/SV/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1009, in eval                                                                                                    return self.train(False)<br>
File “/data/sbanerjee/anaconda_env/SV/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 998, in train                                                                                                    module.train(mode)<br>
<strong>TypeError: ‘bool’ object is not callable</strong></p>
<p><strong>Here’s my code snippet:</strong></p>
<p>model = SegModel(train=False)<br>
if torch.cuda.device_count() &gt; 1:<br>
model = nn.DataParallel(model)<br>
model.to(device)</p>
<p>load_path = ‘path/to/saved/model/model_best_0816.pth.tar’<br>
checkpoint = torch.load(load_path)<br>
epoch = checkpoint[‘epoch’]<br>
train_step = checkpoint[‘train step’]<br>
train_loss = checkpoint[‘train_loss’]<br>
val_loss = checkpoint[‘val_loss’]<br>
print ("------Epoch {} | Step {} | Train Loss {} | Validation Loss {} -----------".format(epoch, train_step, train_loss, val_loss))<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
print (“model loaded”)<br>
model.eval()</p>
<p>Any help would be greatly appreciated</p>

ptrblck: <p>It looks like you have replaced the internal <code>self.train()</code> method inside your model with a bool value:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, train):
        super(MyModel, self).__init__()
        self.bn = nn.BatchNorm2d(3)
        self.train = train

    def forward(self, x):
        x = self.bn(x)
        return x
</code></pre>
<p>Note that <code>self.train()</code> is defined as a method in all <code>nn.Module</code>s, so you should use another name. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

sban: <p>Oh okay, thanks a ton <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Integrating PackedSequence with a custom recurrent model

pozimek: <p>I’m working on integrating <a href="https://github.com/kevinzakka/recurrent-visual-attention/" rel="nofollow noopener">an existing implementation of a recurrent visual attention model</a> with a video dataset (EPIC Kitchens), and I’m struggling to get this model to run with more than 1 video in a batch at a time.</p>
<p><em>Background:</em> the original model I am trying to overhaul is as recurrent model that focuses on different parts of MNIST images to classify them. For each image the model takes a ‘glimpse’ of the input image, which is essentially a cropped out image patch, and combines it with the glimpse location to produce the hidden state RNN vector. The hidden state is then used to choose the location of next image patch, or to make a prediction about the image class. Essentially the network behaves more human-like by focusing on different parts of the image based on what it has seen so far.</p>
<p>I’ve successfully made the network run on videos from EPIC Kitchens, with each video frame receiving one glimpse, but the training is incredibly slow. This is because of very slow data loading. I expected training to be quite slow given the video format and the fact that I don’t have an SSD (will get one at some point), but it’s <em>too</em> slow. I need to run experiments fairly rapidly to upgrade the architecture for the requirements of EPIC Kitchens. I figured it’s largely due to having set batch size to 1, and here is where my current problem comes in. By default PyTorch won’t accept batches of variable length video clips, and it will throw an error at data loading. Luckily, there’s a class called <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.PackedSequence" rel="nofollow noopener">PackedSequence</a> for ‘packing’ variable length sequences by padding them with 0s and all of PyTorch RNN modules ignore padding so as to be more computationally efficient.</p>
<p>That’s all neat, but I don’t use the Pytorch’s RNN modules since my model is custom and I don’t understand how to integrate PackedSequence into my model. I’m looking at the source code for modules.rnn and it’s not helping at all (in the definition of forward() for the RNNBase class what does <em>hx</em> even stand for?). I’m also worried about VRAM usage since the longest video clip in the subset I’m working on is an order of magnitude longer than the average clip and it could cause the batch to crash training.</p>
<p>Can anyone hint me at how to integrate PackedSequence into my model? If that’s not possible or viable, can anyone guide me towards a more efficient data loading paradigm/code for my problem? I’m beginning to doubt the batch loading paradigm as it’s very ill suited for my problem.</p>
How to shuffle data in Dataloader

MadmanNero: <p>As I see it there is a parameter <em>shuffle</em> in the python constructor for a dataloader but not in the C++ version. How can I shuffle my samples in C++?</p>

Haydnspass: <p>Note that by default (at least in Libtorch 1.1) the dataset will be shuffled.<br>
You can change that behaviour by specifying a sampler (from torch::data::samplers::[your sampler of choice].</p>
<p>To construct a dataloader with a sequential sampler:</p>
<pre><code class="lang-auto">auto data_loader = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(std::move(dataset), batch_size);
</code></pre>

MadmanNero: <p>Thanks for your reply. I guess using a RandomSampler is just what I need in my case.<br>
When you say the dataset will be shuffled does that mean once upon creation?</p>

Roulbac: <p>I had done something similar before (can’t remember why though).<br>
I had made my own loader inheriting from dataloader who returned an iterator that I also made myself, inheriting from the normally returned iterator.<br>
At that iterator’s next() method, I caught the stopiteration exception and then reshuffled indices for the next iteration over the entire dataset.<br>
Plus that, add a shuffling in the initialization and you’re good to go.</p>
<p>Something like this in Python</p>
<pre><code class="lang-auto">
class _DataLoaderIterWrapper(_DataLoaderIter):
    def __next__(self):
        try:
            super().__next__()
        except StopIteration:
            # DO STUFF
            raise StopIteration


class DataLoaderWrapper(DataLoader):
    def __iter__(self):
        return _DataLoaderIterWrapper(self)
</code></pre>
Full Log Poisson Loss for PyTorch

DimitrisNightmare: <p>Hello, I need a function that computes the full log poisson loss, but can’t find anything useful for PyTorch.<br>
The TF equivalent would be <code>tf.nn.log_poisson_loss(targets, log_input, compute_full_loss=True)</code>. Is there anything already implemented for PyTorch?<br>
Thanks!</p>

DimitrisNightmare: <p>I implemented it like in TF as such:</p>
<pre><code class="lang-auto">def log_poisson_loss(targets, log_input, compute_full_loss):
    if targets.size() != log_input.size():
        raise ValueError(
            "log_input and targets must have the same shape (%s vs %s)" %
            (log_input.size(), targets.size()))

    result = torch.exp(log_input) - log_input * targets
    if compute_full_loss:
        point_five = 0.5
        two_pi = 2 * math.pi

        stirling_approx = (targets * torch.log(targets)) - targets 
                       + (point_five * torch.log(two_pi * targets))
        zeros = torch.zeros_like(targets, dtype=targets.dtype)
        ones = torch.ones_like(targets, dtype=targets.dtype)
        cond = (targets &gt;= zeros) &amp; (targets &lt;= ones)
        result += torch.where(cond, zeros, stirling_approx)
    return result
</code></pre>
Blas version the pytorch uses

amsword: <p>Is there any way to check which Blas lib pytorch uses in the runtime? Is it MKL or OpenBlas?</p>
<p>I checked the loaded dll, and find both MKL and openblas lib there.</p>

LeviViana: <pre><code class="lang-python">import torch
print(*torch.__config__.show().split("\n"), sep="\n")
</code></pre>
<p>PS: this works for PyTorch 1.1 or earlier.</p>
Current way to compile from source

mittagessen: <p>Greetings,</p>
<p>I have to compile pytorch from source (with cuda 8.0 and cuDnn 7) because our cluster ships with an ancient ScientificLinux (glibc 2.12) and I was wondering what the most up to date way to compile from source is? Is running <code>setup.py</code> still appropriate or is <code>build_anaconda.sh</code> recommended?</p>
<p>In addition, there are hard-coded paths for gcc/g++ in nccl and possibly other third party dependencies that interfere with the location of the compilers on the system (/usr/bin/cc is too old for nvcc and the actual default cc returned by <code>which</code> is at some other location). Is there a nice way to fix this? Unfortunately, it isn’t easily possible to create a clean build environment just for building pytorch.</p>

richard: <p><code>python setup.py install</code> is still recommended. I’m not sure what <code>build_anaconda.sh</code> is.</p>
<p>You can specify cc and gcc paths like the following:<br>
<code>CC=clang CXX=clang++ python setup.py install</code> (replace clang and clang++ with paths to your compilers)</p>

mittagessen: <p><code>build_anaconda.sh</code> seems to be the original build script of caffe2 but supposedly builds a combined caffe2+pytorch wheel with the <code>-integrated</code> flag.</p>
<p>I thought setting CC/CXX on the top level did nothing but apparently that was just caused by some non-purged previous Makefiles.</p>

Ahmed_m: <p>Hello,<br>
Is there a way to build from source via <code>setup.py</code> for only my userspace on Linux? (i.e. to avoid any <code>permission denied</code> issues while building the source).</p>
Passing input to the Linear model

conjuring: <p>I have a basic doubt regarding the syntax of the code while passing the input to the Linear model. Following is the code that I got from documentation.</p>
<blockquote>
<p>m = nn.Linear(20, 30)                                       //Line1<br>
input = torch.randn(128, 20)//Line2<br>
output = m(input)              //Line3</p>
</blockquote>
<p>Now my doubt is:<br>
in Line1,  m is the object that we created by sending the parameters (20, 30) to the <strong>init</strong> of the nn.Linear class. Now, in Line3 while passing the inputs how are we able to pass the inputs by calling m(inputs), as m is an object? Instead, what I think should have been the correct procedure for passing the inputs is:</p>
<blockquote>
<p>output = m.forward(input)</p>
</blockquote>
<p>Please clarify their approach as its a bit unclear to me and this is something that they have used throughout the documentation.</p>

ptrblck: <p>If you call the model directly, the internal <a href="https://github.com/pytorch/pytorch/blob/6ce6939be9b01965cc830f5d819010ac97bf2a1f/torch/nn/modules/module.py#L537" rel="nofollow noopener"><code>__call__</code></a> method will be called, which will register hooks etc. and call <code>forward</code> itself.<br>
You should therefore <strong>not</strong> call <code>forward</code> manually, but let the module do it.</p>
Indexing and updating with tensors

danish037: <p>Suppose I have a memory which I want to update in the following manner:</p>
<pre><code class="lang-auto">memory = torch.zeros(4)
indices = torch.LongTensor([1, 2, 3, 1])
values = torch.FloatTensor([1, 10, 100, 1000])
memory[indices] += values
print (memory)
&gt; tensor([   0., 1000.,   10.,  100.])
</code></pre>
<p>Here, since there is a repetition in the <code>indices</code> the corresponding <code>values</code> do not get added and only the last one gets added. I can see why that can happen due to simultaneous updates. <strong>Is there an easy way to add all the <code>values</code> corresponding to repeated <code>indices</code></strong></p>
<p>Basically, I want the answer to be<br>
<code>tensor([ 0., 1001., 10., 100.])</code></p>

ptrblck: <p><code>memory.index_add_(0, indices, values)</code> should work.</p>
Model overfitts for explicit batch sizes

Taxaria: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"><br>
I made a model for time series forecasting. The model works very well. I tried different batch sizes 32, 50, 64, 100, 128 and 256.<br>
I got the best result for a batch size of 50. With a batch size of 32 the model still works, but it takes too long until the error converges.<br>
I tried multiple experiments with a batch size of 64. With that batch size the model is not able to learn. The MAE is very high, the training error decreases, but the validation error is very high. It seems, that the model is overfitting for a batch size of 64.<br>
So I made further experiments with a batch size of 100, 128 and 256.<br>
100 works of, but not as good as 50. 128 shows the same results as 64. 256 shows good results in training and validation error, but not in the MAE.<br>
Do you have any ideas why 64 and 128 do not work?<br>
I know that models tend to overfit, when the batch size is too large, but this reason makes no sense for me in this case.<br>
Thanks for your help!</p>
Torch.lu pivots are not permutations sometimes

vadimkantorov: <p>Is it okay if they are not permutation (I can still recover the permutation matix via <code>torch.lu_unpack</code>)? <a class="mention" href="/u/vishwakftw">@vishwakftw</a>, do you know if it’s fine? The final result is still correct.</p>
<pre><code class="lang-python">import torch                               
                                           
a = torch.Tensor([                         
  [-1.0357,  1.7709, -0.3924,  0.0632],    
  [-0.0573, -1.9293, -0.7089, -0.8931],    
  [-2.0286,  0.4177, -1.6017,  0.2191],    
  [ 0.9666, -0.7376,  0.6699,  1.1101]]    
)                                                        
q = a.qr().Q                               
LU, P = a.lu()                             
print(P) # tensor([3, 2, 3, 4], dtype=torch.int32)
</code></pre>
How to change ReLu(inplace = True) to ReLu(inplace = False) in torch.models?

Tomas_Batrla: <p>I would like to retrain models from torch.models, but they have inplace operation included. How can I change it to False?</p>

Tomas_Batrla: <p>Ok, I simply imported code from pytorch.models into new class and changed it there</p>
Differentiable equal

yqian: <p>Hi all, I use <em>torch.eq</em> operation in my network recently. And I find there is no grad_fn for this function, which means the loss cannot backward. Therefore, I want to define a differentiable equal function by myself. Here is my code.</p>
<pre><code class="lang-auto">class Eq(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input, value):
        mask = torch.eq(input, value)
        idx = mask.nonzero()
        ctx._input_shape = input.shape
        ctx._input_dtype = input.dtype
        ctx._input_device = input.device
        ctx.save_for_backward(idx)
        return mask.float()

    @staticmethod
    def backward(ctx, grad_output):
        idx, = ctx.saved_tensors
        grad_input = torch.zeros(ctx._input_shape, device=ctx._input_device, dtype=ctx._input_dtype)
        grad_input[idx[:,0], idx[:,1], idx[:,2]] = grad_output[idx[:,0], idx[:,1], idx[:,2]]
        return grad_input, None
</code></pre>
<p>However, I am not sure where it is right or wrong. And the funtion <em>gradcheck</em> seems not applicable to this situation. Can anyone help me solve this problem?<br>
Thanks,</p>

yqian: <p>Well, after careful consideration, equal operation cannot back propogate loss. Trying to define such a function equals banging my head against a brick wall.</p>
How come a dataloader with 10 workers is slower than single process?

hunter7z: <p>So I have a text file of about 3M training samples, with each line representing a sample. First I tried a single process to read samples line by line in a mini-batch style, then feed them to my model. Then I found that the GPU usage is just around 20%, so I guess the reading process is too slow. After I implemented a parallel version using DataLoader and expect it to have a prominent speedup, it turned out to be more than 2x slower than the single process version.<br>
These are my code:</p>
<pre><code class="lang-auto"># an iterable object of a training data file
class SampleFile(object):

    def __init__(self, filePath):
        self.filePath = filePath

    def __iter__(self):
        with open(self.filePath) as file:
            while True:
                line = file.readline().rstrip("\n")
                if not line:
                    break
                sample = self.parseSample(line)
                yield sample

# DataSet object with (maybe) multiple data files
class TrainingSet(IterableDataset):

    def __init__(self, dataFilePath, workerNum):
        super(TrainingSet).__init__()
        self.dataFilePath = dataFilePath
        self.workerNum = workerNum

    def __iter__(self):
        workerInfo = torch.utils.data.get_worker_info()
        if workerInfo is None:
             # single process, just read the whole data
            return iter(SampleFile(self.dataFilePath))
        else:
            # read splitted data file generated by Linux `split` command
            workerNumLen = len(str(self.workerNum-1)) 
            suffix = (workerNumLen-len(str(workerInfo.id))) * '0' + str(workerInfo.id)
            partFilePath = self.dataFilePath + suffix
            return iter(SampleFile(partFilePath))
</code></pre>
<p>Any one could give me a hint? Thanks</p>
Where can I get the information about {Tensor}.index_fill_() function?

DonghunP: <p>I saw the code using {Tensor}.index_fill_()<a href="https://github.com/amdegroot/ssd.pytorch/blob/5b0b77faa955c1917b0c710d770739ba8fbff9b7/layers/box_utils.py#L102" rel="nofollow noopener">link</a>. But It’s hard to find the information about the function.</p>
<p>Can anyone explain this function?</p>
<p>Thanks ahead.</p>

SimonW: <p><a href="https://pytorch.org/docs/stable/tensors.html?highlight=index_fill_#torch.Tensor.index_fill_" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html?highlight=index_fill_#torch.Tensor.index_fill_</a></p>
I want to change the Keras model to Pytorch. Please help me

111135: <p>Hello, I’m studying hard PyTorch.<br>
But I have a problem that I can’t solve myself.<br>
######################################################################<br>
sequence_Input = keras.layers.Input(shape=(None, num_x_signals,), dtype=‘float32’, name=‘sequence_Input’)<br>
RNN_Flow = Bidirectional(GRU(250, return_sequences=True, activation=‘relu’))(sequence_Input)<br>
RNN_Flow = Bidirectional(GRU(250, return_sequences=True, activation=‘tanh’))(RNN_Flow)</p>
<p>additional_Input = keras.layers.Input(shape=(None, num_add_signals,), dtype=‘float32’, name=‘addtional_input’)</p>
<p>NEW_Flow = keras.layers.Concatenate(axis=2)([RNN_Flow, additional_Input])<br>
output_Main = Dense(100, activation = ‘relu’)(NEW_Flow)<br>
output_Main = Dense(100, activation = ‘tanh’)(output_Main)<br>
output_Main = Dense(num_y_signals, activation=‘sigmoid’, name=‘output_Main’)(output_Main)</p>
<p>model = keras.models.Model(inputs=[sequence_Input, additional_Input], outputs=output_Main)<br>
#######################################################################</p>
<p>I want to change the Keras model to Pytorch. Please help me.<br>
I really need your help</p>
What is the shared embedding means

Aiman_Mutasem-bellh: <p>Dear frinds,</p>
<p>What is the shared embedding means?  and what is the diffrent between shared embadding, and the regular embadding in the sequance to sequance model?</p>
<p>I’m developing Grammar Error Correction (GEC) model using Pytorch. The develping model is a convolutional neural network, like Statistical Machine Translation (SMT). I have read the shared embedding will improve GEC model results, but I did’t find a good artical explain what is the shared embedding technequne and how to implement.</p>