akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
All stochastic elements in NN are off, still getting different results

username: barakb
cooked: <p>Hi, my network is giving me different accuracies each time I’m running , before touching anything the difference was around 0.2 top1 scores.<br>
Those are the elements I changed/added:<br>
1.</p>
<blockquote>
<p>torch.manual_seed(60)<br>
torch.cuda.manual_seed(60)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
(There is no use in numpy so didn’t add a seed for numpy)</p>
</blockquote>
<p>2.No Augmentations:</p>
<blockquote>
<pre><code>train_transform=transforms.Compose([
        #transforms.RandomHorizontalFlip(),
        #transforms.RandomCrop(32, 4),
        transforms.ToTensor(),
        normalize,
    ])
</code></pre>
</blockquote>
<ol start="3">
<li>No data shuffle.</li>
<li>Saved the model initialization and loaded the same init for each run.</li>
</ol>
<p>Now the difference is around 0.09.<br>
What did I forget?  what other stochastic ingredients do we have?<br>
Thanks.</p>

username: ptrblck
cooked: <p>Make sure to call <code>model.eval()</code> before running the validation loop.<br>
This makes sure to disable dropout layers and use the running stats of batchnorm layers instead of the current batch mean and stddev.</p>

username: barakb
cooked: <p>It’s on of course.<br>
Thanks.</p>
Do PyTorch Conv1d's require a fixed size input?

username: shamoons
cooked: <p>In keras / Tensorflow, I believe you have to specify the size of the sequence, so it works well for fixed sequences. But with the PyTorch implementation, it seems that it can handle arbitrary sized inputs. Is that correct?</p>

username: ptrblck
cooked: <p>Since convolutions use a sliding window approach (in theory), you would need to specify the number of input channels. Defining the sequence length is not necessary, as the kernel can just be applied on whatever temporal shape is used.</p>
<p>I’m not sure about TF and would be surprised, if you have to set the sequence length beforehand.</p>
[Beginner] DCGAN output uint8 => int16

username: gabe145
cooked: <p>Hello!</p>
<p>I’m trying to generate new terrain data by using a custom datasets containing TIF files from SRTM Mission which contains Earth elevation data in int16.</p>
<p>I’ve followed the PyTorch DCGAN Tutorial and it works! I get an output of 64x64 images.</p>
<p>My problem is, how can I indicate to my GAN that my input images are in int16 and how to get an output in int16 too.</p>
<p>Thanks for your answers!<br>
Regards</p>

username: ptrblck
cooked: <p>Modules usually accept floating point dtypes, so you won’t be able to convert the layers to <code>torch.int16</code>:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1).to('cuda').to(torch.int16)
&gt; TypeError: nn.Module.to only accepts floating point dtypes, but got desired dtype=torch.int16
</code></pre>
<p>If you need a specific range in the output images (based on int16), you could try to clamp the output.</p>
Average pooling problem

username: Muhammad_Izaz
cooked: <p>i want to replace my fully connected layers with average pooling or adaptive pooling layers</p>
<p>class ConvNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ConvNet, self).<strong>init</strong>()<br>
self.layer1 = nn.Sequential(<br>
nn.Conv1d(1, 32, kernel_size=3, stride=1 , padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2,))<br>
self.layer2 = nn.Sequential(<br>
nn.Conv1d(32,64, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2 ))<br>
self.drop_out = nn.Dropout()<br>
self.fc1 = nn.Linear(64*25, 1000)<br>
self.drop_out = nn.Dropout()<br>
self.fc2 = nn.Linear(1000, 2)</p>
<h1>this step is how the data flow through these layers in forward pass</h1>
<pre><code>def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    #out = self.layer3(out)
    out = out.reshape(-1, 64*25)
    out = self.drop_out(out)  
    out = self.fc1(out)
    out = self.drop_out(out)
    out = self.fc2(out)
    return out</code></pre>

username: ptrblck
cooked: <p>I’m not sure, if I understand the question correctly, but you could just replace the linear layers assigned to <code>self.fcX</code> with your desired layers.<br>
Since the expected input shape would be different, you might need to remove the <code>.reshape</code> operation.</p>
When does cuda operator synchronize in pytorch

username: lichao
cooked: <p>Hello, I am a user of pytorch (1.0.1). I would like to ask a question about the cuda operator. This question has troubled me for a long time and I have not found an answer.</p>
<p>Many cuda operators, such as “aten/src/ATen/native/cuda/Activation.cu: prelu_cuda”. In this function, “at::cuda ::getCurrentCUDAStream(curDevice)” is called, but “cudaStreamSynchronize(stream)” is not called, I know that cuda’s operators are asynchronous, but should reasonably call “cudaStreamSynchronize(stream)” in a suitable place, I only found “cudaStreamSynchronize(stream)” in “aten/src/ATen/native/cuda/Copy.cu” , But this happens during the process of copying cpu to cuda or cuda to cpu. If running a network, the CPU releases the memory allocated for cuda in advance, such as calling cudaFree, but cuda is not calculated at this time, the release of these cuda resources is triggered by the release of the temporary Tensor in the network (if I am wrong please correct me, thank you), this will cause serious problems, right? So I want to know how the pytorch guarantees that cuda is already calculated when the temporary Tensor is released, I think that “cudaStreamSynchronize” should be called once before the intermediate Tensor is released. I don’t know if you will reply to me, but I’m still looking forward to your help to answer this question that has troubled me for a long time.</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lichao" data-post="1" data-topic="73963">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lichao/48/22082_2.png" class="avatar"> lichao:</div>
<blockquote>
<p>but should reasonably call “cudaStreamSynchronize(stream)” in a suitable place</p>
</blockquote>
</aside>
<p>That is correct and synchronizations should only be called, if necessary.</p>
<aside class="quote no-group" data-username="lichao" data-post="1" data-topic="73963">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lichao/48/22082_2.png" class="avatar"> lichao:</div>
<blockquote>
<p>the memory allocated for cuda in advance, such as calling cudaFree, but cuda is not calculated at this time, the release of these cuda resources is triggered by the release of the temporary Tensor in the network (if I am wrong please correct me, thank you), this will cause serious problems, right? So I want to know how the pytorch guarantees that cuda is already calculated when the temporary Tensor is released, I think that “cudaStreamSynchronize” should be called once before the intermediate Tensor is released.</p>
</blockquote>
</aside>
<p>If I’m not mistaken, the synchronization happens <a href="https://github.com/pytorch/pytorch/blob/31cc3111430f5f7390c06549616398ba734f83ed/c10/cuda/CUDACachingAllocator.cpp#L740">here</a>.<br>
Are you seeing any issues with using CUDA or is this more of a general question?</p>
Linking python programs with cpp in pytorch

username: pinakinathc
cooked: <p>hi newbie here.<br>
i want to learn more about pytorch. so i was looking for <em>good_first_issues</em><br>
i found torch.abs behaves differently than numpy.abs in issue 33567</p>
<p>on looking at the codebase, i think i need to understand torch._C</p>
<p>so my question is: how does one link python programs with cpp in pytorch?</p>

username: ptrblck
cooked: <p>I would recommend to take a look at <a class="mention" href="/u/tom">@tom</a>’s blog post (+ video) about <a href="https://lernapparat.de/fixing-your-first-pytorch-bug/">fixing your first PyTorch bug</a>, which should be a good starter.</p>
<p>That being said, feel free to ask in the bug for some guidance for the specific issue and also make sure to communicate with <code>wDaniec</code> who already tries to fix it (if that’s not your account <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
What does __constants__ = ['reduction'] mean in class L1Loss()

username: Deepak
cooked: <p>Can anybody please tell me what does this statement <code>__constants__ = ['reduction']</code> mean, and how does it work ? Below is the class definition of L1Loss.</p>
<pre><code class="lang-auto">class L1Loss(_Loss):
    __constants__ = ['reduction']

    def __init__(self, size_average=None, reduce=None, reduction='mean'):
        super(L1Loss, self).__init__(size_average, reduce, reduction)

    def forward(self, input, target):
        return F.l1_loss(input, target, reduction=self.reduction)
</code></pre>

username: ptrblck
cooked: <p><code>__constant__</code> is used for the JIT to define this value as a constant value to allow more potential optimizations as explained <a href="https://pytorch.org/docs/stable/jit.html#constants">here</a>.<br>
Alternatively, you could use the new API with <code>Final</code>.</p>
Disable silent broadcasting?

username: stevethesteve
cooked: <p>Is there a way to make python throw a warning when doing something like<br>
torch.randn(3) * torch.randn(3,1) ?<br>
Too many times have I had silent bugs in my code because of implicit broadcasts I was not aware of…</p>

username: ptrblck
cooked: <p>I don’t think there is a way to do so currently and you would most likely be spammed by these warning, as broadcasting is used in the backend as well.</p>
Gradients over Transformation operations

username: AnupKumarGupta
cooked: <p>I am applying some transformations on an Image. Is it possible to backpropagate over the operations using the autograd ?</p>
<pre><code class="lang-auto">data_transforms = transforms.Compose([
            transforms.ToPILImage(),
            transforms.Grayscale(),
            transforms.CenterCrop(88),
            transforms.ToTensor()
        ])
</code></pre>
<p>I am fairly new to PyTorch. excuse me if I am asking something too obvious.</p>

username: JuanFMontesinos
cooked: <p>Those operations are defined on PIL library. Gradients are tracked if and only if you use pytorch tensors and pytorch operators.<br>
If you reimplement them in terms of pytorch operators you can do so.</p>

username: ptrblck
cooked: <p>I think <a href="https://github.com/kornia/kornia">kornia</a> is able to differentiate through (some) operations.<br>
CC <a class="mention" href="/u/edgarriba">@edgarriba</a>, who is one of the main authors of this lib.</p>
Remove certain weights from a trained NN

username: smu226
cooked: <p>Hello! I have a trained feed forward NN with a given number of inputs and I want to remove all the weights associated to one of the inputs (including the input node itself). Basically I want to keep everything, but have an input 1 dimension lower than before. What is the most effective way to do so? Thank you!</p>

username: ptrblck
cooked: <p>You could slice the weight parameter and reassign it to the <code>.weight</code> attribute.<br>
E.g. if you would like to keep the first 9 input features out of 10:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 5)
with torch.no_grad():
    lin.weight = nn.Parameter(lin.weight[:, :9])
    
x = torch.randn(2, 9)
output = lin(x)
</code></pre>
Recursive Equality

username: joschnei
cooked: <p>If I have two tensors, I can check for equality between the tensors using <code>torch.all(a == b)</code> or <code>torch.all(torch.eq(a, b))</code> or something like that. The tensor == operator returns a tensor of bools, so you need to aggregate across the bools afterward.</p>
<p>If I have a datastructure whose values are tensors (like a list or a dict), how do I compare equality of these datastructures? The structures themselves are not aware of torch, and so don’t know that, when they recursively check equality of their members, they need to do something special for comparing tensors? I’d really like to avoid subclassing the builtin datatypes if possible.</p>
<p>As an aside, does anyone know why it was decided to implement == this way, instead of having == return a bool and having a different elementwise equality?</p>

username: ptrblck
cooked: <p>You could use e.g .<code>map</code> to compare the paired elements in both containers.<br>
Also, note that the <code>==</code> operation is usually not recommended for floating point numbers, to you should use <code>torch.allclose</code>, if applicable:</p>
<pre data-code-wrap="python"><code class="lang-python">lista = [torch.randn(1) for _ in range(5)]
listb = [a+1e-6 for a in lista]

all(map(lambda x, y: torch.allclose(x, y), lista, listb))
&gt; True
</code></pre>
<aside class="quote no-group" data-username="joschnei" data-post="1" data-topic="73907">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f0a364/48.png" class="avatar"> joschnei:</div>
<blockquote>
<p>As an aside, does anyone know why it was decided to implement == this way, instead of having == return a bool and having a different elementwise equality?</p>
</blockquote>
</aside>
<p>Most likely to stick to the numpy API, which yields the same output.</p>
Mapping list of label

username: _joker
cooked: <p>How do I do the same thing for a list of labels?</p>
<p>label = [2,3,5,7,1,0…]</p>
<pre><code class="lang-auto">label = 3
NumClass = 10
NumRows  = 100

mask    =  torch.zeros(100, 64)
ones     =  torch.ones(1, 64)
ElementsPerClass = NumRows//NumClass
mask [ ElementsPerClass*label : ElementsPerClass*(label+1) ] = ones
</code></pre>
Real time Hand digit recognition with OpenCV

username: kritical
cooked: <p>Hi,<br>
I’ve trained a CNN with two layers and got 98% test accuracy. However, I wanted to try to do real time digit recognition and tried to incorporate OpenCV.</p>
<p>This implementation produces terrible accuracy and usually predicts the number incorrectly. Can somebody point out what it is that I am doing incorrectly?<br>
This is the code I’ve written so far. <a href="https://pastebin.com/tt8qStjj" rel="nofollow noopener">https://pastebin.com/tt8qStjj</a></p>
<p>I know this is not a question directly related to pytorch but I thought I might find some help on these forums.</p>

username: simaiden
cooked: <p>If you train a network in some dataset does not mean at all that will have good performance in real world. Did you trained with mnist?</p>

username: kritical
cooked: <p>Yes, that’s right I was using mnist. Do you suggest I use another dataset?</p>

username: kritical
cooked: <p>Or could it be my implementation of OpenCV is incorrect?</p>

username: simaiden
cooked: <p>make sure you are doing the same image transformation in training and test. The opencv implementation seems ok</p>
PyTorch: Remove one LSTM layer from nn.lstm module in a pre-trained network

username: miranda
cooked: <p>I have a 2-layer bidirectional LSTM, and I need to remove the second layer of the pre-trained LSTM. My model is very simple and looks like this:</p>
<blockquote>
<p>num_layers = 2<br>
class BiRNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, num_layers, num_classes):<br>
super(BiRNN, self).<strong>init</strong>()<br>
self.hidden_size = hidden_size<br>
self.num_layers = num_layers<br>
self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=True)<br>
self.fc = nn.Linear(hidden_size<em>2, num_classes)<br>
def forward(self, x):<br>
# Set initial states<br>
h0 = torch.zeros(self.num_layers</em>2, x.size(0), self.hidden_size).to(device)<br>
c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)</p>
<pre><code>    # Forward propagate LSTM
    lstm_out, _ = self.lstm(x, (h0, c0))  # lstm_out: tensor of shape (batch_size, seq_length, hidden_size*2)

    # Decode the hidden state of the last time step
    out = self.fc(lstm_out[:, -1, :])
    return out
</code></pre>
<h1>initialize the NN</h1>
<p>model = BiRNN(input_size, hidden_size, num_layers, num_classes).to(device)</p>
</blockquote>
<p>I train the network, and now I should remove the second layer of the LSTM for some analysis. I checked this post <a href="https://discuss.pytorch.org/t/can-i-remove-a-layer-from-a-pre-trained-model-while-loading-the-model-weights/57899">https://discuss.pytorch.org/t/can-i-remove-a-layer-from-a-pre-trained-model-while-loading-the-model-weights/57899</a>. But in that post, the goal is to remove the fully connected layer. I tried to use <code>model.childern</code>, but both layers of LSTM are counted as one child. The same is true with <code>model.modules</code>. I also read about <code>nn.Identity</code>, but not sure how to replace the second layer of LSTM by <code>nn.Identity</code>. Is there any way to remove the second layer of the model in PyTorch.<br>
Sorry if the question is trivial, I am new to PyTorch.<br>
Thanks!</p>
<p><code>&gt; Blockquote</code></p>
Sharing parameters between two instances of rnn

username: marco_zaror
cooked: <p>Hi all,</p>
<p>I am trying create two rnn models with shared parameters between them (being more specific, I need that both models has the same x2h and h2h matrices). Also, I need to optimize them jointly, so when the matrix is updated for one is updated for both.</p>
<p>It is possible to do so with the nn.RNN model?</p>
JIT Compile with Shape Parameters Passed as Templates

username: Mostafa_Elhoushi
cooked: <p>I have been looking into PyTorch’s <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">C++ CUDA/CPU Extension Tutorial</a> and at PyTorch’s <code>torch.jit.load(..)</code> function.</p>
<p>My understanding is that a CUDA kernel that is compiled for a specific shape will have a better performance than a generic CUDA kernel. (e.g. check the discussion in this <a href="https://groups.google.com/d/msg/xla-dev/i1QZBjxyCuQ/ITNJDhXgAQAJ" rel="nofollow noopener">link</a>)</p>
<p>Having said that, my question is: it possible to create a CUDA kernel with shapes passed as a template (and hence it will be compiled for a specific shape), e.g., something like:</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

#include &lt;vector&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;ATen/Config.h&gt;

template&lt;c10::ArrayRef&lt;long int&gt; weight_size,  c10::ArrayRef&lt;long int&gt; padding,
    c10::ArrayRef&lt;long int&gt; stride, c10::ArrayRef&lt;long int&gt; dilation&gt;
at::Tensor conv2d_forward(
    const at::Tensor&amp; input,
    const at::Tensor&amp; weight,
    int64_t groups,
    bool benchmark,
    bool deterministic) {

  // TODO: define kernel here
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("conv2d_forward", &amp;conv2d_forward, "Conv2d forward");
}
</code></pre>
<p>and then when JITTing pass the shape and related parameters:</p>
<pre><code class="lang-auto">from torch.utils.cpp_extension import load
conv2d_ours = load(name="conv2d_forward", sources=["conv2d_forward.cpp"], params=[ "weight_size" = weight_size, "padding"=padding, "stride"=stride, "dilation"=dilation], verbose=True)
</code></pre>
<p>Ideally, before invoking the convolution op, we look at the shape parameters and check if there has been a previously JITed version for the same parameters. If there exists, then we just invoked that pre-compiled kernel, else JIT compile the kernel and invoke it.</p>
<p>My understanding is that high-level compilers, like XLA and TVM, JIT compile kernels to specific shapes to enhance performance. It will be great if we can do something similar in PyTorch.</p>
Hyper parameters optimization or grid search technique

username: Muhammad_Izaz
cooked: <p>i want to to optimize hyper parameters for CNN model, please if any body know about it.<br>
my model:<br>
class ConvNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ConvNet, self).<strong>init</strong>()<br>
self.layer1 = nn.Sequential(<br>
nn.Conv1d(1, 32, kernel_size=3, stride=1 , padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2,))<br>
self.layer2 = nn.Sequential(<br>
nn.Conv1d(32,64, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2 ))<br>
self.drop_out = nn.Dropout()<br>
self.fc1 = nn.Linear(64*25, 1000)<br>
self.fc2 = nn.Linear(1000, 2)</p>
<h1>this step is how the data flow through these layers in forward pass</h1>
<pre><code>def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    #out = self.layer3(out)
    out = out.reshape(-1, 64*25)
    out = self.drop_out(out)  
    out = self.fc1(out)
    out = self.fc2(out)
    return out</code></pre>
Model weights not updated

username: Eugene_Lee
cooked: <p>Hi there. I’m trying to update the weights of multiple models at once where the optimizer is set as:<br>
<code>optimizer = torch.optim.SGD(list(netA.parameters()) + list(netB.parameters())+ list(netC.parameters()), args.lr)</code><br>
For the weight update, I performed <code>loss.backward()</code> and checked that all the parameter’s <code>some_param.weight.grad</code> is not <code>None</code>. I ran <code>optimizer.step()</code> and always find that only the parameters of <code>netA</code> are updated but not the rest. Any idea on what am I missing here? Thanks!</p>

username: Eugene_Lee
cooked: <p>I found the problem. The other models are actually based on the first order gradient of the first model, hence the gradient becomes very small and are attenuated.</p>
DCGAN for generating sinusoid

username: kerwin1
cooked: <p>So there are a lot of examples for DCGANs for generating pictures and other 2-dimensoinal things, but there are just a few for generating time-series. I’m not super experienced with pytorch and GAN in general, but I understand all their basics. At the moment, I want to build a DCGAN which can eventually generate sinusoidal waves (because this looks like the easiest example of mimic a general time-series).<br>
I looked at the general dcgan example and tried to make a similair dcgan with 1d convolutional layers in order to train them on sinusoidal waves. But unfortunately I didn’t succeed yet in making this GAN.<br>
So I was wondering if there are any clear notebooks in which they are making a GAN to generate a simple time-series like a sinus wave?</p>
Noun phrases encoding

username: sandeep1
cooked: <p>How to encode all the noun phrases of a sentence to create a feature for a classification problem?</p>
<p>sentence- This makes a lot of the questions asked in this paper extremely relevant to the field<br>
Noun phrases: [‘a lot’, ‘the questions’, ‘this paper’, ‘the field’]</p>
<p>I have to create a feature like<br>
sentence  feature<br>
sentence1 feature_vector1<br>
sentence2 feature_vector2</p>
<p>How to encode all the noun phrases of a sentence to create a feature for a classification problem?</p>
Feature engineering of a review

username: sandeep1
cooked: <p>I am trying to classify review of paper into these classes<br>
CLASSES:-<br>
Clarity (Writing and Composition)<br>
Appropriateness<br>
Novelty/Originality<br>
Substance/Volume<br>
Significance/Impact<br>
Meaningful Comparison<br>
Presentation &amp; Formatting<br>
Recommendation<br>
Empirical/Theoretical Soundness<br>
Can’t Decide</p>
<p>For this i am doing feature engineering<br>
One feature is am using is noun and adjective of the sentences.<br>
Example:<br>
sentence-  This makes a lot of the questions asked in this paper extremely relevant to the field<br>
Noun phrases: [‘a lot’, ‘the questions’, ‘this paper’, ‘the field’]</p>
<p>What more features can i use for this classification problem?</p>
Replace BN layers in model

username: vojtooo
cooked: <p>Hi,</p>
<p>I am using two models, DenseNet and EfficientNet (B7) and would like to replace batch norm with group norm (<a href="https://arxiv.org/pdf/1803.08494.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1803.08494.pdf</a>) or weight standardization (<a href="https://arxiv.org/pdf/1903.10520.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1903.10520.pdf</a>).</p>
<p>I have found several implementations of these norm techniques, however, I could not find how to edit the model (both of them are from libraries and I cannot edit the source code directly). Is there some way, how to replace all BN layers by GN od WS in PyTorch?</p>
<p>Thanks<br>
Best regards</p>
Sharing partial weights for different Conv2d layers

username: mengyue
cooked: <p>I wonder is it possible to let two different Conv2d layers (in different number of channels) share (partial of) the weights.</p>
<p>I know it possible to implement via <code>torch.nn.functional.conv2d</code> like the following:</p>
<pre><code class="lang-auto">import torch

class DNet(torch.nn.Module):
    def __init__(self):
        super(DNet, self).__init__()
        self.W0 = torch.nn.Parameter(torch.randn(64, 3, 7, 7), requires_grad=True)
        self.avgpool = torch.nn.AdaptiveAvgPool2d((1, 1))
        self.fc0 = torch.nn.Linear(64, 1000)
        self.fc1 = torch.nn.Linear(16, 1000)

    def forward(self, input, **kwargs):
        signal = kwargs["signal"]
        if signal == 0:
            x = torch.nn.functional.conv2d(input, self.W0, None, 1)
            fc=self.fc0
        else:
            x = torch.nn.functional.conv2d(input, self.W0[:16], None, 1)
            fc = self.fc1
        return fc(torch.flatten(self.avgpool(x), 1))

model = DNet()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
model.train()
for i in range(42):
    pred = model(torch.randn(1, 3, 224, 224), signal=i % 2)
    loss = torch.nn.CrossEntropyLoss()(pred, torch.tensor([666]))
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print("iter:%04d pred:%d loss:%.4f" % (i, torch.argmax(pred.detach()), loss.item()))
</code></pre>
<p>but it is not elegant when using ResNet, because we need to change the entire <code>__init__</code> function. It is much easier to just share the filter weights, but I don’t know why the following can’t work:</p>
<pre><code class="lang-auto">conv1.weight=conv0.weight[:16]
# or `conv1.weight=torch.nn.Parameter(conv0.weight[:16])`. In this way "conv1" just creates another set of parameters which won't affect the one in "conv0"
</code></pre>
<p>I know it will work if we simply do <code>conv1.weight=conv0.weight</code>, but I don’t know why we cannot do slicing while still keeping them point to the same data source. Any suggestion/explanation? Thanks.</p>
Does TorchScript C++ use online JIT?

username: yetanadur
cooked: <p>This is just to clarify my understanding of JIT.</p>
<p>When we talk about PyTorch JIT - is the translator that generates TorchScript IR a JIT itself ? And when we execute the TorchScript (using libtorch C++ library) is there an another JIT that is interpreting TorchScript IR and generating code that calls libtorch APIs and dispatches the calls ? If so, should we expect this online JIT to have a runtime overhead in terms of cpu cycles and memory ?</p>
<p>Thanks,<br>
Yetanadur</p>

username: James_Reed
cooked: <p>Hi <a class="mention" href="/u/yetanadur">@yetanadur</a>,</p>
<p>No, IR generation is a fully ahead-of-time process. JIT compilation is used to recover information that is potentially dynamic, such as tensor shapes, that can be used to make better optimization decisions during runtime. This all happens entirely after IR generation, though. Please let us know if you have further questions.</p>
<p>James</p>

username: yetanadur
cooked: <p>Thanks James.</p>
<p>Where can I find the entry point function (in source code) for JIT ? And which function has the code to jump to the JIT’d (optimized) code ? Basically want to step thru with a debugger.</p>
<p>Thanks a bunch,<br>
Yetanadur</p>

username: eellison
cooked: <p>This is true for 1.5, but isn’t true on master, and won’t be true for the upcoming release.</p>
How can I scale up a Conv1D?

username: shamoons
cooked: <p>If I have a <code>Conv1d</code> layers:</p>
<pre><code class="lang-auto">        self.conv1 = torch.nn.Conv1d(
            in_channels=feature_dim,
            out_channels=feature_dim,
            kernel_size=kernel_sizes[0],
            stride=1
        )
            # padding=kernel_sizes[0] // 2)

        self.conv2 = torch.nn.Conv1d(
            in_channels=feature_dim,
            out_channels=feature_dim,
            kernel_size=kernel_sizes[1],
            stride=1
        )
            # padding=kernel_sizes[1] // 2)

        self.conv3 = torch.nn.Conv1d(
            in_channels=feature_dim,
            out_channels=feature_dim,
            kernel_size=kernel_sizes[2],
            stride=1
        )
            # padding=kernel_sizes[2] // 2)

        self.conv4 = torch.nn.Conv1d(
            in_channels=feature_dim,
            out_channels=feature_dim,
            kernel_size=kernel_sizes[3],
            stride=1,
            padding=2)
</code></pre>
<p>My sizes look like:</p>
<pre><code class="lang-auto">inp torch.Size([8, 161, 196])
out1 torch.Size([8, 161, 186])
out2 torch.Size([8, 161, 178])
out3 torch.Size([8, 161, 172])
out4 torch.Size([8, 161, 188])
</code></pre>
<p>But I want <code>conv4</code> to increase the from 172 back to 196. I assume I can control that with padding, but I’m not sure how.</p>

username: ptrblck
cooked: <p>It’s hard to tell which padding value you need, as the kernel sizes are not defined in the code snippet, however you could try increasing the padding until the temporal size matches your expectation.</p>
<p>I would also recommend to check, if adding padding in the intermediate layers wouldn’t be the better option to avoid the reduction in the temp. dimension in the first place instead of filling the result with zeros.</p>
Managing gradients on variable length sequences

username: pyhh
cooked: <p>What is the best way to train an LSTM on data with variable length sequences, where some of those sequences are too long to perform backpropogation in full?</p>
<p><strong>My situation:</strong></p>
<p>I am training an LSTM on many thousands of independent sequences. The individual sequences can vary in length from 2 to 10000, but most are short (&lt;100). I am using pad_packed_sequence() and pack_padded_sequence to process the data in minibatches. The long sequences (circa 10000 long) are causing issues when I perform full backpropogation, so I am looking for smarter alternatives.</p>
<p><strong>What I have tried so far:</strong></p>
<p>First, I tried splitting the sequences into chunks of 128 and feeding them through separately. This works ok, but essentially resets the hidden and cell state after every 128 steps, resulting in lost information that should be carried forward. The training loop looks something like this:</p>
<pre><code class="lang-auto">for epoch in n_epochs:
    for batch in data_loader:
        input, target = batch
        output = network.forward(input)
        loss = loss_function(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>To stop the loss of information, I tried processing each sequence in one go but performing backpropogration after every 128 steps. To do this, I manually manage the hidden and cell states, detaching them after each forward/backward pass. I also drop sequences from the minibatch once they have been completely processed. This approach also works but is dreadfully slow, possibly because the minibatch is shrinking as I drop samples. The training loops looks something like this:</p>
<pre><code class="lang-auto">for epoch in n_epochs:
    for batch in data_loader:
        (hidden_state, cell_state) = network.init_hidden()
        while(True):
            (input, target), (hidden_state, cell_state) = get_next_chunk_and_filter(batch, hidden_state, cell_state)
            if len(input) == 0:
                break
            output, (hidden_state, cell_state) = network.forward(input, (hidden_state, cell_state))
            loss = loss_function(output, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            (hidden_state, cell_state) = (hidden_state.detach(), cell_state.detach())
</code></pre>
<p><strong>Other ideas:</strong></p>
<p>I am now considering replacing samples in the minibatch once they have been processed, rather than removing them entirely. This seems like it would make the training more efficient, but would also require some very messy juggling of samples, hidden states and cell states. I think it would work, but isn’t very elegant.</p>
<p>Is there a better way to do this? This must be a really common problem?</p>
<p>Thanks!</p>
Using autograd::grad function in C++

username: babababadukeduke
cooked: <p>I am trying to make a custom C++ extension using <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">this</a> tutorial, but I am getting an error when I am trying to compile the code. I am basically trying to call the grad function in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/autograd.h" rel="nofollow noopener">here</a>. I am getting the following error…</p>
<p>error: ‘grad’ is not a member of ‘torch::autograd’<br>
return torch::autograd::grad(outputs, inputs, grad_outputs, retain_graph, create_graph, allow_unused);</p>
<p>Any help will be appreciate! Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Which version of pytorch/libtorch are you using?</p>

username: babababadukeduke
cooked: <p>Currently I have pytorch 1.2.0 installed in my machine. I installed it using pip</p>

username: albanD
cooked: <p>I’m not sure this function was already added back then. You should try to upgrade to the latest version.</p>

username: babababadukeduke
cooked: <p>Thank you. Updating pytorch worked.</p>
Model gradients can't be changed

username: Imran_Rashid
cooked: <p>I am trying to modify my gradients by multiplying them by a constant. As a test, I multiply the gradients by zero, so I expect that the losses should stay constant.</p>
<p>However it looks like the model still updates the weights of the network even though I have tried to force the weights gradients to be zero. Why aren’t the gradients being updated?</p>
<pre><code class="lang-auto">GRADIENT_MULTIPLIER = 0.
model.train()
for epoch in range(100):
    for ind, (input_data,labels) in enumerate(train_iterator):
        optimizer.zero_grad()

        logits = model(input_data,labels)
        loss = model.loss(logits, labels)

        loss.backward()
        
        for p in model.parameters():
            p.grad *= GRADIENT_MULTIPLIER

        global_norm = np.sqrt(sum([torch.sum(w.grad**2).item() for w in model.parameters()]))
        print(f"global_norm2 : {global_norm}")
        
        optimizer.step()

        print(f"loss : {loss.data}")
</code></pre>
<p>Outputs are:</p>
<pre><code class="lang-auto">global_norm : 0.0
loss : 0.6851892471313477
global_norm : 0.0
loss : 0.6985365748405457
global_norm : 0.0
loss : 0.6622101664543152
global_norm : 0.0
loss : 0.45273280143737793
global_norm : 0.0
loss : 0.8967741131782532
global_norm : 0.0
loss : 0.28941503167152405
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>What is your optimizer? Note that some optimizers will still change the weights even for 0 gradients (for example when weight decay is used).</p>

username: Imran_Rashid
cooked: <p>I’m using Adam for my optimizer. Specifically: <code>optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)</code></p>

username: albanD
cooked: <p>For Adam, if any “momentum-like” term is non-zero, then you will update the weights even with a gradient of 0. So this is expected.</p>
Implementing perceptual loss

username: py23
cooked: <p>Hello all,<br>
I am trying to implement the perceptual loss function, however, I am getting a contrast issue when using it,<br>
I have tried this:<br>
<code>loss = torch.norm((output-target),2)</code><br>
where output and target are the features generated from the vgg19 network</p>
<p>and this</p>
<pre><code class="lang-auto">function  = nn.MSELoss()
loss = function(output,target)
</code></pre>
<p>and also this:</p>
<pre><code class="lang-auto">a = output - target
b = a.pow(2)
loss = torch.norm(b,2)
</code></pre>
<p>However, none of these are working, I have normalised the training data to [0,1] and used the imagenet mean and std values,  can you tell me where I am going wrong?</p>
<p>Thanks for reading</p>

username: ptrblck
cooked: <p>Could you explain the contrast issue a bit?<br>
I assume you are working on generating images and the contrast seems to be off?</p>

username: py23
cooked: <p>Yes I am trying to generate some upsampled images, however, everything in the generated photo is much darker (features are sharper though), so say if the sky was white in one photo, in the generated photo it would be dark grey</p>

username: py23
cooked: <p>Any solution would be appreciated</p>
Indexing with topk

username: razvanc92
cooked: <p>Hello, I’m having a tensor X of shape (Batch, Nodes, Features), and another one scores of shape (Batch, Node_score). I would like to take biggest x nodes with the highest score from each batch.</p>
<p>I managed to make it work without a batch, but I do not know how to extend it to take batches into account. Here is the version without:</p>
<pre><code class="lang-auto">X = torch.from_numpy(np.array([[1, 2], [3, 4], [5, 6] ]))
scores = torch.from_numpy(np.array([10, 2, 15]))
values, idx = torch.topk(scores, 2, dim=-1)
new_X = X[idx, :]
</code></pre>
<p>Thank you in advance.</p>
Compile PyTorch from source with SSE4.2 instructions

username: Paulo_Mann
cooked: <p>Is there a way to compile PyTorch from the source code for the SSE4.2 set of instructions? I need to compile it on PC A (possibly any set of instructions) to work on PC B (only SSE4.2) in production (both Linux buster); also, I can’t build it in the production PC: for security reasons I can’t <code>pip install</code> or <code>apt-get</code> anything. Is there any way? Or I don’t even need to worry about building it for SSE4.2??</p>
Conditional weight updates

username: Emilio
cooked: <p>Hi,</p>
<p>I’m working on an experiment where I train a model using CNN backbone which is applied on the input image and on a set of learnable prototypes. During the training process I want to update the weights of the CNN when it is applied on the image. When it is applied to the prototype I want to freeze the weights of the CNN but do update the prototypes.</p>
<p>I tried to set <code>torch.no_grad</code> on the CNN when it is applied to the prototypes but this resulted in the prototypes not getting updated as well.</p>
<p>Does anyone know a method to perform this construct?</p>
<p>Here is a part of the code used in the experiment.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F


class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()

        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.conv3 = nn.Conv2d(64, 64, 3, 1)
        self.conv4 = nn.Conv2d(64, 128, 3, 1)

    def forward(self, x):
        x = F.relu(self.conv2(F.relu(self.conv1(x))))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv4(F.relu(self.conv3(x))))
        x = F.max_pool2d(x, 2, 2)
        return x


class Model(nn.Module):
    def __init__(self, n_prototypes):
        super(Model, self).__init__()

        self.backbone = CNN()

        self.prototype = nn.Parameter(
            torch.rand((n_prototypes,) + (28, 28))
        )

    def forward(self, x):

        x = self.backbone(x)  # update CNN kernels
        y = self.backbone(self.prototype)  # update prototypes but not CNN kernels

        return x, y
</code></pre>
<p>Thanks in advance.</p>

username: kimjh069
cooked: <p>You may consider to use different optimizers.<br>
It is a way to train generative adversarial networks (training generator network once, discriminator netowork once and so on.)<br>
for example,</p>
<pre><code class="lang-auto">model_1 = CNN()
model_2 = Prototype()
optimizer_1 = torch.optim.Adam(
        params=model_1.parameters(),
        lr=lr,
    )
optimizer_2 = torch.optim.Adam(
        params=model_2.parameters(),
        lr=lr,
    )

# during training

if prototype: 
 optimizer_2.step()
else:
 optimizer_1.step()
</code></pre>
<p>Is this what you are looking for?</p>
<p>Besides, if you use <code>torch.no_grad</code>, I think it might disconnect gradient chains when loss.backward() is called.<br>
You also can look into the gradients after loss.backward() is called.</p>
<pre><code class="lang-auto">loss.backward()
print(model.backbone.conv1.grad) # print(model.some_layer.grad)
</code></pre>

username: Emilio
cooked: <p>It gets in the right direction but <code>x</code> and <code>y</code> are also used in another module so I maybe also need a optimizer for that part, making it more complex.</p>
<p>Is there maybe a way to set the kernel gradients with respect to the prototypes to zero before they are applied?</p>
Regression prediciton out of range

username: arnooo
cooked: <p>HI there,<br>
I’m trying to use transfer learning with VGG16 and imagenet weights for a regression problem. I’m changing the last layer for a linear layer with output size 1 (see code below).</p>
<p>However, the prediction of my network are super weird. My training values are property prices, hence &gt;&gt;0 and my network predicts only values &lt;0.</p>
<p>Any thoughts on this ?</p>
<pre><code class="lang-auto"># Need to change dataloader to bring class model
model = models.vgg16_bn(pretrained=True)
criterion = nn.MSELoss() #nn.CrossEntropyLoss() #nn.MSELoss()
optimizer = optim.Adam(model.parameters())

# Freeze training for all layers
for param in model.features.parameters():
    param.require_grad = False

# Newly created modules have require_grad=True by default
num_features = model.classifier[6].in_features
features = list(model.classifier.children())[:-1] # Remove last layer
features.extend([nn.Linear(num_features, 1)]) # Number of output necessary
model.classifier = nn.Sequential(*features) # Replace the model classifier

if use_gpu:
    model.cuda()
num_epochs = 20
dir_ = 'Weigths/'
vgg16 = train_model(model, criterion, optimizer, num_epochs=num_epochs, classification=False)
</code></pre>
<p>I’ve checked out the dataloader and its feeding the right label …</p>
<p>Thanks !</p>

username: yash1994
cooked: <p>Hi,</p>
<p>You need to normalize your target (property prices) values, i guess your loss function must be exploding. Neural network models are incapable of generating that kind of large values. Another way is to convert this problem to classification task (i.e.  class 1 (0 &lt; property price =&lt; 500),  class 2 (500 &lt; property price =&lt; 1000)… something like that.).</p>
Samplers vs Shuffling

username: nategr_03
cooked: <p>Hi,</p>
<p>I’m new to PyTorch and was wondering how I should shuffle my training dataset. I’ve seen some examples that use a <strong>RandomSampler</strong>, as follows:</p>
<pre><code class="lang-auto">train_data = TensorDataset(train_inputs, train_masks, train_labels)
train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
</code></pre>
<p>What if I did not use a sampler at all and instead set the <strong>shuffle</strong> parameter to True, as follows:</p>
<pre><code class="lang-auto">train_data = TensorDataset(train_inputs, train_masks, train_labels)
train_dataloader = DataLoader(train_data, shuffle=True, batch_size=batch_size)
</code></pre>
<p>Is this the same thing?</p>
<p>Also, using a <strong>SequentialSampler</strong> or setting <strong>shuffle</strong> parameter to False, are they the same thing?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>If you use <code>shuffle = True</code> the <code>DataLoader</code> will initialize a <code>RandomSampler</code> for you, otherwise it’ll use <code>SequentialSampler</code> as seen in <a href="https://github.com/pytorch/pytorch/blob/463f7920bd45b522b4675663656873fa9689cab0/torch/utils/data/dataloader.py#L212-L215">these lines of code</a>.</p>

username: nategr_03
cooked: <p>Thanks for the clarification!</p>
How to speed up following operation (for loop creating different sized matrices)

username: divinho
cooked: <p>I’ve got the following mechanism for adding positional awareness to representations, it works well but  is very slow.</p>
<p>The fundamental issue is that each sample in the batch has a different real length and I therefore have to loop over and create different embeddings for each of them.</p>
<pre><code class="lang-auto">class PositionSymmetricLayer(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.embedding = nn.Parameter(torch.linspace(0., 1., dim//2), requires_grad=False)
        self.dim = dim

    def forward(self, x, durs):
        N = x.size(0)
        T = x.size(1)
        pos_emb = torch.zeros((N, T, self.dim), dtype=torch.float32).cuda()
        for i, dur in enumerate(durs):
            factor = torch.linspace(0, 1, dur).unsqueeze(1).cuda()
            pos = (torch.cat((self.embedding - factor, -self.embedding + factor,), dim=-1) &gt;= 0).float()
            pos_emb[i, -dur:] = pos
        x = torch.cat((x, pos_emb,), dim=-1)
        return x
</code></pre>
<p>To help understand what this does you can see in the following what <code>pos</code> is given a <code>dur</code> of 3 and a <code>dim</code> of 8:</p>
<pre><code class="lang-auto">tensor([[1., 1., 1., 1., 1., 0., 0., 0.],
        [0., 0., 1., 1., 1., 1., 0., 0.],
        [0., 0., 0., 1., 1., 1., 1., 1.]])
</code></pre>
<p>Is there something I can do to speed the for loop up?</p>

username: divinho
cooked: <p>Got a large reduction by sorting the samples by duration and batchifying some operations.</p>