akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Making augmentation transform dependent on epoch number

username: Stefano_Savian
cooked: <p>Dear all,</p>
<p>I have a doubt on data augmentation using dataloader. I want to apply a transformation with a certain probability <em>p</em> which can decreases either as the number of epochs progress, or when the loss function is higher than a certain threshold. However, due to the concurrent nature of the dataloader workers, I cannot figure out how to apply this to the transorm class, as I don’t know how I could get the epoch or loss as inputs to my function. So my question is</p>
<ol>
<li>Is it possible to apply augmentation with the probability changing on the epoch number or loss function?</li>
<li>How could it be implemented?</li>
</ol>
<p>below is the pseudo code of my transform function.</p>
<p>class custom_transform(object):</p>
<pre><code>def __init__(self, size,p):
    if isinstance(size, numbers.Number):
        self.size = (int(size), int(size))
    else:
        self.size = size
    self.p = p
def __call__(self, inputs, target):
    if random.random() &lt; self.p:
        GENERATE TRANSFORMED inputs, target

        inputs, target
    else:
        return inputs, target
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>  your ideas are very welcome!</p>
<p>Many Thanks,<br>
Stefano</p>

username: albanD
cooked: <p>Hey,</p>
<p>I can’t think of a builtin way to do this. But you should be able to have a global variable shared by all the workers that contain this p that they will read and will be set by the main training process?<br>
Maybe a 1 element Tensor will work as it will be shared across the processes.</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/alband">@albanD</a>’s idea sounds good.</p>
<p>If your workflow would allow you to change the probability after each epoch finished, you could also directly manipulate the <code>loader.dataset</code>. At the beginning of each epoch, the workers of the <code>DataLoader</code> will be respawned and will thus use the changed dataset.</p>
OOM for self.attention mechanism

username: rubentorrado
cooked: <p>We are running a self-attention mechanism with a conv1d network. We have gotten oom error when we multiply the quey tensor with the key tensor. The size of the query and key tensors are [16, 4, 22778]<br>
t_1 = torch.bmm(proj_query, proj_key)</p>
<p>We have tried converting our tensors to half-precision; however, we still get the same error OOM:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 15.46 GiB (GPU 0; 14.76 GiB total capacity; 53.88 MiB already allocated; 13.97 GiB free; 80.00 MiB reserved in total by PyTorch)</p>
<p>Do you have any recommendations to fix this issue?</p>

username: ptrblck
cooked: <p>The shape of both tensors shouldn’t be equal, as this would return a shape mismatch error for the matrix multiplication.<br>
Could you recheck the shapes and post it here, please?</p>
Regarding an entirely binary neural network

username: vainaijr
cooked: <p>Is there any problem with an entirely binary neural network, that is the input to it is in binary, all its parameters in binary, what would be the substitute for gradient descent be in such a case, to keep everything in binary?</p>

username: KFrank
cooked: <p>Hello vainaijr!</p>
<aside class="quote no-group" data-username="vainaijr" data-post="1" data-topic="84409" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vainaijr/48/49167_2.png" class="avatar"> vainaijr:</div>
<blockquote>
<p>Is there any problem with an entirely binary neural network</p>
</blockquote>
</aside>
<p>There is no conceptual problem with such a network, although I<br>
don’t see what advantage one would have (other than, perhaps,<br>
economy of storage).</p>
<p>I suppose one might view a binary network as being a better<br>
approximation to some particular biological or physical system.</p>
<aside class="quote no-group">
<blockquote>
<p>what would be the substitute for gradient descent be in such a case,</p>
</blockquote>
</aside>
<p>There’s the rub.  This is a binary (special case of an integer)<br>
optimization problem.  (See <em>integer programming.</em>)  This is known<br>
to be hard.  There are various (inexact) optimization algorithms you<br>
could use – branch-and-bound, simulated annealing – but training<br>
will be expensive, and likely prohibitively so for networks of any<br>
significant size.</p>
<p>Differentiability and gradient descent are your friend.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: vainaijr
cooked: <p>hello, thanks for your reply, but is there a substitute to convolutional neural network, in the case of a binary neural network, and if I do not want filters as parameters in my neural network, that keep updating for every image, is some work done on it?</p>
<p>I think the whole notion of gradient descent would be simplified by the use of a logic gate such as NAND or XOR or XNOR, because in the case of a binary neural network, one needs to take care of flipping bits from 0 to 1, or 1 to 0, or let them stay as they were, that is 1 to 1, or 0 to 0.</p>
.reshape or .view() tensor

username: heejae
cooked: <p>for example,<br>
how to convert 1x32 1d torch tensor [1,2,3,4,5,6,7,8,9,10,11,12, …, 30,31,32] to (2x2)x(4x2) 2d array [[[1,2],[3,4]],[[5,6],[7,8]],[[9,10],[11,12]], … [[29,30],[31,32]]] without using for loop. this doesn’t work with [tensor].view(2<em>2, 4</em>2).</p>

username: alpha_omega_alpha
cooked: <p>But <code>[[[1,2],[3,4]],[[5,6],[7,8]],[[9,10],[11,12]], … [[29,30],[31,32]]]</code> is not a 2D array.</p>
<p>Try: <code>[tensor].view(8, 2, 2)</code> for that.</p>
Real-time optical flow?

username: Federico_Vasile
cooked: <p>Hi everyone,<br>
I’m facing with online action detection task(i.e. I have an input video stream and I want to classify each frame as it comes). I see in literature that optical flow gives a lot of help in capturing motion information from video, but as far as I known this is always computed as a pre-processing step (so it is not suitable for real-time applications).<br>
Is there any algorithm that is capable of computing a real-time optical flow?</p>
<p>PS: In any case, suggestion or workarounds are welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Basically, my actual model is a single-stream-CNN (with RGB input) as feature extractor followed by an LSTM that performs the prediction, I’m convinced that using instead a two-stream-CNN (with optical flow input for the second stream) will increase the accuracy.</p>
How to get the same size output of conv_transpose2d

username: dhiyu
cooked: <p>First I have a feature map, then I used <code>nn.Conv2d(in_channel, out_channel, 3, stride=2, padding=1, bias=True)</code> to generate a intermediary feature map. Next, fed it to <code>nn.ConvTranspose2d(conv_channels, conv_channels, 3, stride=2, padding=1, output_padding=1)</code> and get the final output.<br>
My question is how to control the parameters padding and output_padding in order to get the same size final output with the initial feature map. When I used padding=1 and output_padding=1, It only worked as the initial feature map had even number size. While I used  feature map with odd number size, It’s wrong.<br>
Could anyone help me please?<br>
Thanks~</p>

username: ptrblck
cooked: <p><a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html#transposed-convolution-arithmetic">This convolution arithmetic doc</a> gives you some general information about convolutions as well as transposed convolutions and also about the relationship of their parameters, which might be useful for your use case.<br>
Depending on your input shape, you might need to change more parameters than the padding, such as the kernel size.</p>
What's the difference between csrc/serialization and csrc/generic/serialization?

username: tgsmdww
cooked: <p>I want to make save/load support new device. When I test cpu save , it will call THPStorage_(writeFile),  which call generic/serialization.cpp’s THPStorage_(writeFileRaw). So I don’t know what csrc/serialization.cpp does.</p>
<p>In addition, can anyone tell me how cuda support save? Any issues is OK. I find it has so many thing to do…</p>
Deploying action recognition model

username: braindotai
cooked: <p>If performance is the priority here, would it be okay to deploy a heavy model for action recognition natively on mobile (using pytorch mobile)?</p>
<p>Or should I go with something like rest api, hosted on other remote server (amazon, azure, etc).</p>

username: Shisho_Sama
cooked: <p>Having it on remote is more logical unless you have very bad internet connection.<br>
its not customary to place the actual model on the device, as it puts everything into risk.</p>
Different samples use different loss functions

username: shirui-japina
cooked: <p>How to implement it?</p>

username: ptrblck
cooked: <p>You could index the samples and apply the loss functions separately.<br>
Feel free to post more information in case you get stuck. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to prepare the custom data for image classification model

username: suri_g
cooked: <p>Hi, I am newbie to Pytorch, I am   trying to build a multi label image  classification  model  for some dance images, but I have two file one is  images  file and another  one containing   train csv file with images ids  and target labels, please help me , how to prepare the data for cnn model.</p>
<p>here my data:</p>
<pre><code class="lang-auto">train.head()

|Image|target|
|0|96.jpg|manipuri|
|1|163.jpg|bharatanatyam|
|2|450.jpg|odissi|
|3|219.jpg|kathakali|
|4|455.jpg|odissi|
</code></pre>
<p>Here I was tried some code , but it not working.</p>
<pre><code class="lang-auto">lb=LabelEncoder()
train['encoded_labels']= lb.fit_transform(train['target'])  
class MyDataset(Dataset):
    def __init__(self , csv_file , img_dir , transforms=None ):
    
        self.df = pd.read_csv(csv_file)
        self.img_dir = img_dir
        self.transforms = transforms
    
    def __getitem__(self,idx):
        d = self.df.iloc[idx.item()]
        image = Image.open(self.img_dir/d.image).convert("RGB")
        label = torch.tensor(self.img_data.loc[index, 'encoded_labels'])
    
        if self.transforms is not None:
            image = self.transforms(image)
        return image,label
    def __len__(self):
        return len(self.df)
transform = transforms.Compose([transforms.Resize((224,224)) , 
                               transforms.ToTensor(),
                               transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
                               ])

dataset = MyDataset("dance//dataset//train.csv" , Path("dance//dataset//train") , transform)

batch_size=32
valid_no = int(len(dataset)*0.12) 
trainset ,valset  = random_split( dataset , [len(dataset) -valid_no  ,valid_no])
print(f"trainset len {len(trainset)} valset len {len(valset)}")
dataloader = {"train":DataLoader(trainset , shuffle=True , batch_size=batch_size),
              "val": DataLoader(valset , shuffle=True , batch_size=batch_size)}
</code></pre>

username: ptrblck
cooked: <p>What is currently not working in your code? If you are seeing an error message, could you please post it, so that we could have a look?</p>
How to augment gamma correction randomly?

username: Tuyen_Vo_Quang
cooked: <p>The document <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.functional.adjust_gamma" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.functional.adjust_gamma</a> us how to add gamma correction, but it seems to use a constant gain for the transformation. Is there any way to adjust randommly?</p>

username: ptrblck
cooked: <p>You could write a custom transformation using the functional API es described <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#functional-transforms">here</a>.<br>
You could pass the desired ranges for <code>gamma</code> and <code>gain</code>, sample these values randomly in the <code>__call__</code> method, and apply them on your images.</p>

username: aunsid
cooked: <p>class RandomGammaCorrection(object):<br>
“”"<br>
Apply Gamma Correction to the images<br>
“”"</p>
<pre><code>def __init__(self, gamma = None):
    self.gamma = gamma
    
    
def __call__(self,image):
    
    if self.gamma == None:
        # more chances of selecting 0 (original image)
        gammas = [0,0,0,0.5,1,1.5]
        self.gamma = random.choice(gammas)
    
    print(self.gamma)
    if self.gamma == 0:
        return image
    
    else:
        return TF.adjust_gamma(image, self.gamma, gain=1)</code></pre>
FairSeq fails to utilize multiple GPUs but says "Training on X devices"

username: Enamex
cooked: <p>Invocation:</p>
<pre><code class="lang-auto">python $FAIRSEQ/train.py "$DATABIN" \
    --max-epoch 10 --max-tokens 6000 --update-freq 1 \
    --ddp-backend=no_c10d --memory-efficient-fp16 \
    --lang-pairs "$SRC-enu" \
    # arch stuff
    --user-dir ./fairseq-modules/ \
    --save-dir "$SAVE/checkpoints" \
    --num-workers 8 \
    --tensorboard-logdir "$SAVE"
</code></pre>
<p>But when monitoring the GPUs one is at 100% and all the others are 0%.</p>
<p>I’m on a shared machine. So the devices I’m getting don’t have consecutive device IDs. But this’s done before the container image can see them. So it should be seeing only the devices assigned, and with sensible IDs.</p>
<p>CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7</p>
<p>Thank you!</p>
Mesh R-CNN on android

username: sujay
cooked: <p>Is it possible to use the pretrained Mesh R-CNN model in the android app using the existing pytorch android deployment pipeline? (for on-device inference)</p>
<p>Mesh R-CNN model from the page <a href="https://github.com/facebookresearch/meshrcnn/blob/master/INSTRUCTIONS_PIX3D.md" rel="nofollow noopener">https://github.com/facebookresearch/meshrcnn/blob/master/INSTRUCTIONS_PIX3D.md</a></p>
<p><a class="mention" href="/u/zdevito">@zdevito</a> <a class="mention" href="/u/michael_suo">@Michael_Suo</a><br>
<a class="mention" href="/u/jcjohns">@jcjohns</a></p>
How to convert a polar tensor into cartesian?

username: Yudong_Li
cooked: <p>[batch, channel, r, theta] is the input tensor. I want to convert it into [batch, channel, x, y], which x, and y are in cartesian coordinates. How am I able to implement this?</p>

username: chetan_patil
cooked: <p>If input is something like <code>[batch, channel , 2]</code> where the first entries are <code>r</code> and the second entries are <code>theta</code>,<br>
then you can use this.</p>
<pre><code class="lang-auto">#batch = 4, channels = 3
polar = torch.randn(4,3,2)

r = polar[:,:,0]
theta = polar[:,:,1]

x = r * torch.cos(theta)
y = r * torch.sin(theta)

cartesian = torch.stack([x,y], axis = -1) # resulting in (4,3,2)
</code></pre>
<p>If the input is something like <code>[batch, channel, r, theta]</code>, then you’d have an output Tensor of size <code>[batch, channel, 2, r, theta]</code> ie to say, for every <code>r</code> and <code>theta</code>, you will have two values which are <code>(x,y)</code>.</p>
What is multi scale structural similarity?

username: braindotai
cooked: <p>Hi there, I trying to evaluate a gan I’ve trained, so I ended up reading about “Multi scale structural similarity”. This is also used massively in number of gan papers.<br>
So it’d be helpful if someone in short can explain intuition behind it, and how do we calculate it.</p>
<p><a href="https://github.com/VainF/pytorch-msssim/" rel="nofollow noopener">Here</a> and <a href="https://github.com/jorge-pessoa/pytorch-msssim" rel="nofollow noopener">here</a> are implementations of MS-SS in pytorch.</p>
What is use of batch normalization and evolving normalization-activation layer

username: harshini_sewani
cooked: <p>Can someone explain to me what is the use of both?</p>

username: ptrblck
cooked: <p>Could you explain or post a link to a reference for “evolving normalization-activation layer”, please?</p>

username: harshini_sewani
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/2004.02967" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://arxiv.org/abs/2004.02967" target="_blank" rel="nofollow noopener">Evolving Normalization-Activation Layers</a></h3>

<p>Normalization layers and activation functions are fundamental components in
deep networks and typically co-locate with each other. Here we propose to
design them using an automated approach. Instead of designing them separately,
we unify them into a...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Increase num_workers stops my model from learning

username: Valerio_Biscione
cooked: <p>Hello. I generally run my experiments with <code>num_workers</code> set to 1 in the <code>DataLoader</code>. Recently I got a slightly more demanding dataset to generate and I thought of speeding things up by setting <code>num_workers</code> to 8. Strangely, this stopped my model to learn completely. Why is this the case? My datasets are generated on the fly, if that’s relevant.</p>
<p>And, more generally, how does <code>num_workers</code> work exactly? Could you point me to some docs? Each different worker works on a different batch and then they are collated together?</p>
<p>And, maybe related: since I write code on windows, all my code is wrapped around</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    freeze_support()
    .... 
</code></pre>
<p>would this afftect performance when ran on a non-windows machine?<br>
Thanks</p>

username: ebarsoum
cooked: <p>Are you running that on Windows or Linux? What you mean by stop learning?</p>

username: Valerio_Biscione
cooked: <p>Linux. The loss is flat</p>
Passing ScriptModule to C++ in 1.6

username: rjeli
cooked: <p>Hi,</p>
<p>In 1.5, I could create a pybind11 extension with a function that takes a jitted module, like:</p>
<pre><code class="lang-auto">// c++
void go(torch::jit::Module&amp; m) {
    auto out = m.forward(/* inputs */).toTensor().sum();
    std::cout &lt;&lt; out &lt;&lt; std::endl;
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("go", &amp;go);
}

// python
ext = cpp_extension.load(name='ext_cpp', sources=['ext.cc'])

class MyMod(nn.Module):
    def forward(self, x):
        return x*x

m = torch.jit.script(MyMod())
ext.go(m._c)
</code></pre>
<p>And everything Just Worked. Now in 1.6 I can’t figure out what it wants the arg type to be. I’ve tried Module, Object, IValue, everything I can think of or see in the code. Interestingly <code>_run_emit_module_hook</code> is called with the <code>RecursiveScriptModule</code>'s inner cpp_module, and it’s defined as taking a <code>torch::jit::Module</code>, so I thought that would work, but no dice.</p>
<p>Any help? <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>
<p>…or if this is an xy problem, what’s the recommended way to pass a module to an extension? Grab the custom class from the registry? or just <code>save_to_buffer</code> and <code>load</code> from istringstream?</p>
Conv3d error on dimensions of a 5d tensor

username: costa98
cooked: <p>I have a tensor shaped ([5, 1, 3, 126, 126]), which represents a video (5 frames each 126x126 rgb).<br>
I need to forward it into a</p>
<pre><code>self.resnet = nn.Sequential(
        nn.Conv3d(5,5,1), 
        nn.UpsamplingBilinear2d(size=None, scale_factor=0.5)  
    )
</code></pre>
<p>but i get</p>
<pre><code>RuntimeError: Given groups=1, weight of size [5, 5, 1, 1, 1], expected input[5, 1, 3, 126, 126] to have 5 channels, but got 1 channels instead
</code></pre>
<p>I think that I have probably misunderstood how the conv3d works but I can’t really understand why the expected dimensions are so different from the ones that my 5d tensor has at that moment</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>A 5D tensor is like <code>[batch, channel, depth, height, width]</code> and <code>nn.Conv3d(in_channel, out_channel, kernel_size)</code> expects to have <code>in_channel = channel(in your 5d tensor)</code> but you have set <code>in_channel=5</code> while your 5D tensor has 1 channel.</p>
<p>You need to use <code>nn.Conv3d(1, 5, 1)</code></p>
<p>Bests</p>
ToTensor and Normalize transforms

username: Wesley_Neill
cooked: <p>After converting a PIL or NumPy image using <code>ToTensor</code>, I usually see <code>Normalize</code> called. The arguments are usually tuples of <code>0.5</code> or the values calculated from ImageNet.</p>
<ul>
<li>
<p>First question: If the input range is [0,1] then the output would be [-1,1], which I’ve read is good for training a network. So that must mean that <code>ToTensor</code> changes the range of the input from [0, 255] to [0,1]. Is that correct?</p>
</li>
<li>
<p>Second question is, why is [-1, 1] any better than [0, 1] when it comes to pixel value ranges?</p>
</li>
<li>
<p>Third question is when would I want to tweak the mean and standard deviation? For example, I am working on the style transfer exercise of Udacity’s intro to PyTorch class. I noticed that when they load a new image for style transfer, they use the mean and standard from ImageNet. Why was this choice made?</p>
</li>
</ul>

username: Nikronic
cooked: <p>Hi,</p>
<p>About <code>ToTensor</code> you are right, it standardizes input. Please see this <a href="https://discuss.pytorch.org/t/does-pytorch-automatically-normalizes-image-to-0-1/40022">post</a>.</p>
<p>About your questions:</p>
<ol>
<li>
<p>If you use <code>mean=[0.5,...], std=[0.5,...]</code> then a standardized input will be in <code>[-1, 1]</code> but this exactly true for ImanetNet mean and std. You can verify this by calculating <code>(x - mean(x)) / std</code>.</p>
</li>
<li>
<p>This question is little bit tricky, but if I want to summarize my answer, it has almost similar effect when you do not standardize input. As you might know, when images are in range of [0-255] or in case of other data, when a feature has a bigger range than the other features, neural network tend to change based on that range so that particular feature will dominate other features. About [0, 1], still data is biased toward positive side of numbers. If you plot [-1, 1] data, all are gathered around mean like a circle which helps to have a better learning. Actually, it has more to do with statistics which you can find many good lectures about it.</p>
</li>
<li>
<p>In summary, when your dataset is far from ImageNet, you need to define your own custom mean and std, for instance in medical image processing.<br>
The idea is that a subset of a set tends to have similar statistics to the original set and that is why sampling works. Now, if your dataset is similar to ImageNet, then it has similar mean and std to ImageNet because it is like you sampled ImageNet to get your own. In your example, probably, your style transfer network is working with images with similar structure to ImageNet.</p>
</li>
</ol>
<p>Bests</p>
Adding a loss for when output is not on a unit sphere

username: wouterM
cooked: <p>Hi all,</p>
<p>I am trying to get an autoencoder to work on my L2 normalized dataset. For now I use the MSE loss which works oke but I noticed that a lot of the reconstructed vectors are not nearly on the unit sphere, which they should be since I know every input vector is on there. Is there any simple solution to penalize or enforce the decoder to generate unit sized vectors? I tried calculating the L2 norm of each item in a batch and dividing the output by that but I am not sure this will give the correct results for the computation graph.</p>
<p>Thanks!</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="wouterM" data-post="1" data-topic="88700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8c91f0/48.png" class="avatar"> wouterM:</div>
<blockquote>
<p>I tried calculating the L2 norm of each item in a batch and dividing the output by that but I am not sure this will give the correct results for the computation graph</p>
</blockquote>
</aside>
<p>This should be fine.</p>

username: KFrank
cooked: <p>Hi Wouter!</p>
<aside class="quote no-group" data-username="wouterM" data-post="1" data-topic="88700" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8c91f0/48.png" class="avatar"> wouterM:</div>
<blockquote>
<p>Is there any simple solution to penalize or enforce the decoder to generate unit sized vectors?</p>
</blockquote>
</aside>
<p>To answer this specific question, yes, you can add a loss term<br>
that will push your predictions onto the unit sphere.</p>
<p>When the (L2) norm or your prediction is <code>1</code>, the prediction is on<br>
the unit sphere.  So just add unit-sphere mismatch term:</p>
<pre><code class="lang-auto">sphere_loss = fac * ((preds**2).sum (dim = 1) - 1.0).abs().mean()
loss += sphere_loss
</code></pre>
<p>(You could also take the square of the mismatch, rather than<br>
<code>abs()</code>.  That would give you a softer penalty, but I don’t see<br>
any reason to prefer using the square.)</p>
<p>As a practical matter, there might be some benefit to doing this,<br>
but it shouldn’t be necessary.  As you train, your predictions will<br>
tend to drift outward from the unit sphere (barring some other<br>
regularization), but nothing is actively* pushing them away.</p>
<p>As an aside, Alex is right that normalizing your predictions is<br>
the proper way to go (with or without the <code>sphere_loss</code> term).<br>
Doing so won’t break autograd, or otherwise cause problems.</p>
<p>*)  When you take a <em>finite</em> step tangent to the unti sphere, you<br>
will move – to second order – outward off the unit sphere.</p>
<p>Best.</p>
<p>K. Frank</p>
Question regarding backpropagation implementation of pytorch

username: Riddle_Aaron
cooked: <p>While implementing neural models by myself without aid of pytorch module, I become curious about how backpropagation is implemented in pytorch.<br>
In pytorch, explicit implementation of backward function is not needed unlike forward function.<br>
However, I can’t understand how pytorch automatically understands the structure of a net so that backpropagation can be carried out.</p>
<p>For example, let’s see the picture below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b67850f317f66bd653ce16b8880f7a8444f9c58e.png" alt="pic" data-base62-sha1="q2cFUcAWeVjlTZaDA2jFVt4kHcW" width="281" height="264"><br>
It can be implemented like this.</p>
<pre><code class="lang-auto">maxpool1 = nn.MaxPool2d((7, 1), stride=(1, 1))
maxpool2 = nn.MaxPool2d((7, 1), stride=(1, 1))
m1 = maxpool1.forward(x1)
m2 = maxpool2.forward(x2)
c = m1 + m2
</code></pre>
<p>pytorch must know that there is an add layer to convey dL to the maxpool Layer, but c is calculated by just adding two, without a certain layer class.<br>
(Like this)</p>
<pre><code class="lang-auto">add = nn.some_add_layer(2) # 2 inputs
c = add.forward(m1, m2)
</code></pre>
<p>Then how does pytorch know that there is an add layer?</p>

username: Haider_Ali_Shuvo
cooked: <p>Pytorch builds a computational Graph for everything you do from your Data to finding out loss.Typical Pytorch training looks like below -<br>
for epoch in range(num_epochs):<br>
for data , labels in train_loader:<br>
## Whatever you do from here to calculating loss Pytorch will build computational<br>
Graph for it.<br>
#<span class="hashtag">#loss</span>.backward()<br>
#<span class="hashtag">#Optimizer</span>.step()<br>
#<span class="hashtag">#Optimizer</span>.zero_grad.</p>
<p>Now to be specific , You dont need to use only nn Modules to have this computational graph , You can do simple addition,matrix multiply or whatever and pytorch will still be able to build graph for it and when you call loss.backward() it will be able to do backpropagation.</p>

username: Riddle_Aaron
cooked: <p>It seems that every computational process is stored in each tensor variable as a “grad_fn” attribute. Am I right?</p>

username: Haider_Ali_Shuvo
cooked: <p>Yes  , Unless you do some computation inside torch.no_grad() pytorch will keep track of your each steps and associate a way to go backward and finally will build a computational graph .</p>
Is it possible to do backward on multiple modules in parallel?

username: rahul003
cooked: <p>I understand that backward passes are serialized for a particular model with multiple loss tensors for example. But if I have independent modules can I run backward pass in parallel?</p>

username: albanD
cooked: <p>Hi,</p>
<p>No you cannot do that at the moment I’m afraid.<br>
Can you share more about your use case and why it would be interesting for you?</p>
Backwards pass using Pytorch/XLA

username: almeetb
cooked: <p>Hello, sorry if this is repeat question but how does Pytorch autograd engine fit with dispatching operations to the XLA backend. I understand that functions will save outputs in forward passes to use in the backwards pass but I don’t think the XLA backend makes use of these functions or sets the grad_fn</p>

username: albanD
cooked: <p>Hi,</p>
<p>The autograd should work just the same as with regular CPU Tensors.<br>
Do you have a code sample where there is discrepancy?</p>

username: almeetb
cooked: <p>No discrepancy, I was just trying to get a better understanding of how the lazy execution flow works with backprop.</p>

username: albanD
cooked: <p>My basic understanding is that the lazy execution is completely hidden from pytorch by the xla binding.<br>
So pytorch thinks it is synchronous. And only when we access the values on the CPU that xla actually compute stuff.</p>
<p>So it should have no impact <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Is inference thread-safe?

username: pizuzadan
cooked: <p>I know that code below is thread-safe (Many modules, many threads.):</p>
<pre><code class="lang-auto">void foo(const std::vector&lt;std::shared_ptr&lt;torch::jit::script::Module&gt;&gt;&amp; modules, int idx)
{
	torch::NoGradGuard no_grad;
	torch::Tensor X = torch::rand({ 1, 10, 1, 300 });
	auto out = modules[idx]-&gt;forward({ X });
}

void first_inference()
{
	using namespace torch::jit;
	const std::string model_path{ "model.pth" };

	std::vector&lt;std::shared_ptr&lt;torch::jit::script::Module&gt;&gt; modules;
	modules.reserve(10);
	for (int i = 0; i &lt; 10; i++) {
		modules.push_back(std::make_shared&lt;script::Module&gt;(load(model_path)));
	}

	std::vector&lt;std::thread&gt; vec;
	for (int i = 0; i &lt; 10; i++) {
		std::thread t(foo, std::ref(modules), i);
		vec.push_back(std::move(t));
	}

	for (auto&amp; t : vec)
		t.join();
}
</code></pre>
<p>What about the following code (One module, many threads.)? Is it thread safe?<br>
I checked that the forward doesn’t have a global lock. The program doesn’t crash and has the correct results.</p>
<pre><code class="lang-auto">void bar(const std::shared_ptr&lt;torch::jit::script::Module&gt; module)
{
	torch::NoGradGuard no_grad;
	torch::Tensor X = torch::rand({ 1, 10, 1, 300 });
	auto out = module-&gt;forward({ X });
}

void second_inference()
{
	using namespace torch::jit;
	const std::string model_path{ "model.pth" };

	std::shared_ptr&lt;script::Module&gt; module{ std::make_shared&lt;script::Module&gt;(load(model_path)) };

	std::vector&lt;std::thread&gt; vec;
	for (int i = 0; i &lt; 10; i++) {
		std::thread t(bar, std::ref(module));
		vec.push_back(std::move(t));
	}

	for (auto&amp; t : vec)
		t.join();
}
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>In general, all the objects in pytorch are thread safe to read. But are not to write into.<br>
If your Module doesn’t write into shared structure, then it should work just fine yes.</p>

username: pizuzadan
cooked: <p>I think my torchscript model doesn’t change the internal state of the module. It is used only for forward and doesn’t calculate gradients.</p>
<p>Does this mean that the inference will work correctly for one module in many threads, if I load a model, for example, torchscript resnet using an official pytorch url?</p>
<p>Is this true for the GPU?</p>

username: albanD
cooked: <p>Having multiple modules or using the GPU doesn’t change anything there yes.</p>
<p>For the jit I’m less sure. But I would expect inference to have the same rules as regular inference.</p>

username: Michael_Suo
cooked: <p>yes, as long as you are not mutating elements of the module state in your forward pass, inference is thread safe in TorchScript</p>
Accessing and saving grad_input and grad_outpu

username: steveo
cooked: <p>Hi All,</p>
<p>Is there a simple way to access and save the grad_input and grad_output as defined in the backward component of a custom torch.autograd.Function? I would like to do this for all the layers and I was wondering if there was something similar to</p>
<pre><code class="lang-auto">net.conv11.weight.grad
net.conv21.bias.grad

</code></pre>
<p>for the grad_input and grad_output.</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>In theory, that is what the <code>register_backward_hook()</code> on nn.Module is supposed to give you.<br>
Unfortunately, they are broken right now and we will hopefully be able to fix them later this year.</p>
<p>The way you can do this right now is to use <code>register_forward_hook()</code> on your nn.Module to access both the input and outputs and use <code>register_hook()</code> on these Tensors to access their gradients.</p>

username: steveo
cooked: <p>Hi,</p>
<p>Here is the solution I’ve come up with for appending the grad_input after every loss.backward()</p>
<pre><code class="lang-auto">grad_input_list = []
def forward_hook_grad_saver(self, input, output):
     input[0].register_hook(input_gradient_appender)
def input_gradient_appender(grad):
     grad_input_list.append(grad)

model.linear2.register_forward_hook(forward_hook_grad_saver)

</code></pre>
<p>I think I’m having the same issue as posted here<br>
<a href="https://discuss.pytorch.org/t/memory-leak-when-using-forward-hook-and-backward-hook-simultaneously/27555/4" class="inline-onebox">Memory leak when using forward hook and backward hook simultaneously</a> or something similar since my GPU runs out of memory.</p>
<p>Am I using the hook incorrectly here or does my hook mess with my computational graph? Thanks I don’t have enough background or knowledge to deeply understand the solution in the other post or whether it even applies here. Thanks!</p>
CosineAnnealingWarmRestarts with float multiplier

username: leoniloris
cooked: <p>Hello folks,<br>
The learning rate scheduler <code>CosineAnnealingWarmRestarts</code> does not accept a <code>T_mult</code> as a float number.<br>
Is there a reason why? Is there a workaruond? It would be very useful to have it as a float to have more control over the rate at which the period changes.</p>
Does it make sense to call "loss_per_sample.backward()" while training a NN?

username: amitoz
cooked: <p>One way to train a NN is</p>
<pre><code class="lang-auto">for epoch in range(1,30): 
     pred  = model(whole_input_data)
     loss = myLoss(pred, whole_target_data)
     opt.zero_grad()
     loss.backward()
     opt.step()
</code></pre>
<p>If the <code>whole_input_data</code> is very large, one resolves to training in mini-batches like this:</p>
<pre><code class="lang-auto">for epoch in range(1,300):
     for batch_x, batch_y in loader:
           pred = model(batch_x)
           loss_batch = myLoss(pred, batch_y)
           opt.zero_grad()
           loss.backward()
           opt.step()
</code></pre>
<p>The downside of above is one has to increase the number of epochs.</p>
<p>Does it make sense to call <code>backward()</code> method over loss_per_sample like following to train a NN;</p>
<pre><code class="lang-auto">for epoch in range(1, 3000):
     for x_per_sample, y_per_sample in zip(X_train, y_train):
           pred = model(x_per_sample)
           loss_per_sample = myLoss(pred, y_per_sample)
           opt.zero_grad()
           loss_per_sample.backward()
           opt.step()
</code></pre>
<p>Will it train the NN?</p>

username: Nikronic
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="amitoz" data-post="1" data-topic="88577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amitoz/48/26151_2.png" class="avatar"> amitoz:</div>
<blockquote>
<p>The downside of above is one has to increase the number of epochs.</p>
</blockquote>
</aside>
<p>Main benefit of batching is that when dataset is too big, you cannot compute backward and forward due to different hardware or math issues, so every batch works as a representation of your whole data so that is why you see more fluctuation in your loss.</p>
<p>About third scenario, you can see it as a case with batch_size=1. An intuitive idea is that you still have disadvantage of increasing epochs. Another issue is that 1 sample cannot represent enough info for training so probably your model cannot fit as by viewing a new sample after 100 epochs , the learned features about first epoch may have been vanished.<br>
Actually, this situation needs to be examined mathematically, as it can be expressed by state of optimizer, the concept of loss function and finding optima.</p>
<p>Bests</p>

username: amitoz
cooked: <p>Hey buddy,</p>
<p>Thanks for your reply. From the research I did, it would still train the neural network but as expected it will take even more epochs and with strong fluctuations in loss decay. In the book <a href="https://pytorch.org/deep-learning-with-pytorch" rel="nofollow noopener">DL with pytorch</a>, they breifly touch this topic in section 7.2.4</p>
<p>Best</p>

username: Nikronic
cooked: <p>Yes exactly, that makes more sense.<br>
Actually, I am just waiting for the day that I finish my university exams to be able to read that book THROUGHLY!</p>
<p>Thank yor for sharing</p>
Model improvements

username: alameer
cooked: <p>Hi I am training the following model to count cell in microscopy images. I am keen to hear suggestions from people on how to improve the model. Many Thanks in advance.</p>
<pre><code class="lang-auto">import torch.nn as nn

class Convnet(nn.Module):
    """
    A custom convnet for convolving 1080x1080 fluorescence micrograph images.
    """

    def __init__(self):
        super(Convnet,self).__init__()
        self.main = nn.Sequential(

            nn.Conv2d(4,16,10,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(16,32,7,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(32,64,3,stride=3,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(64,96,3,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(96,128,3,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(128,192,3,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(192,256,3,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(256,256,3,stride=2,padding=0),
            nn.LeakyReLU(0.2,inplace=False),
            nn.Conv2d(256,1,1,stride=1,padding=0)
        )

    def forward(self, x):
        # x: B x 2 x 1024 x 1024
        return self.main(x) # B x 1 x 9 x 9
</code></pre>
Torch.jit.is_scripting and handle_torch_function

username: f10w
cooked: <p>Hello,</p>
<p>I observed in PyTorch’s source code some functions that follow the pattern below (for example <a href="https://github.com/pytorch/pytorch/blob/5e03a1e9261e86d3ee62481ef40d98ca5f09646c/torch/nn/functional.py#L3915" rel="nofollow noopener">this one</a>):</p>
<pre><code class="lang-auto">def my_function(a, b, c, d=None, e=None):
    if not torch.jit.is_scripting():
        tens_ops = (a, b, c)
        if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
            return handle_torch_function(my_function , a, b, c, d=d, e=e)
   # something else here
</code></pre>
<p>Could anybody please tell me what’s going on there? In particular, what do <code>torch.jit.is_scripting</code> and <code>handle_torch_function</code> do?</p>
<p>And <strong>more importantly</strong>: Where can I learn those things in detail so that I can myself write similar custom functions and stop asking stupid questions like these? <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> It seems that these are about <a href="https://pytorch.org/docs/stable/jit.html?highlight=torch.jit#module-torch.jit" rel="nofollow noopener">TorchScript</a>, but I wanted to ask just in case, before diving into that.</p>
<p>Thank you very much in advance for your help!</p>

username: ptrblck
cooked: <p><code>torch.jit.is_scripting()</code> can be used as a guard for Python-only methods, which are not exportable.<br>
Since the JIT cannot export all arbitrary Python objects, you could thus use different “paths” in your code for the Python execution and the scripting of the mdoel.<br>
<code>handle_torch_function</code> sounds like a custom function definition.</p>
<aside class="quote no-group" data-username="f10w" data-post="1" data-topic="88451">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/f10w/48/1352_2.png" class="avatar"> f10w:</div>
<blockquote>
<p>Where can I learn those things in detail so that I can myself write similar custom functions and stop asking stupid questions like these?</p>
</blockquote>
</aside>
<p>As these features are currently being developed, the documentation and tutorials might not be up to date, so please ask these kind of questions here. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p><code>torch.jit.is_scripting</code> can be found in the current <a href="https://pytorch.org/docs/master/jit_language_reference.html#torch.jit.is_scripting">master docs</a>, but might also be updated before the next release.</p>

username: f10w
cooked: <p>Thanks a lot, <a class="mention" href="/u/ptrblck">@ptrblck</a>! Your answer is very helpful, as always.</p>
Cublas Runtime Error on GPU

username: ChrisLiu2
cooked: <p>This is the environment I’m running in:</p>
<ol>
<li>CUDA 10.1</li>
<li>Python 3.7</li>
<li>Titan X</li>
<li>Pytorch 1.5.1</li>
</ol>
<p>The network used to work normally, but after I added a method in my model to compute graph normalizations, this error starts to happen.</p>
<p>The error tracing with CUDA_LAUNCH_BLOCKING=1 gives:’</p>
<pre><code class="lang-auto">Traceback (most recent call last):----------------------------------| 0.1% Training epoch 0;
  File "main.py", line 201, in &lt;module&gt;
    cross_validation_with_val_set(model, params)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/train_eval.py", line 110, in cross_validation_with_val_set
    epoch_index=epoch-1, params=params, writer=writer)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/train_eval.py", line 209, in train_test_eval
    train_acc, train_loss = train(model, loaders['train'], opt, params)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/train_eval.py", line 250, in train
    return run_windowed_model(model, loader, opt, params)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/train_eval.py", line 343, in run_windowed_model
    out = model(x_curr)
  File "/afs/ece.cmu.edu/usr/xujinl/anaconda3/envs/CSD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/modules/nets/linear.py", line 12, in forward
    x = self.MLP(x)
  File "/afs/ece.cmu.edu/usr/xujinl/anaconda3/envs/CSD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/afs/ece.cmu.edu/usr/xujinl/CSD/modules/nets/linear.py", line 38, in forward
    x = F.dropout(F.relu(self.bn1(self.lin1(x))),p=self.dropout,training=self.training)
  File "/afs/ece.cmu.edu/usr/xujinl/anaconda3/envs/CSD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/afs/ece.cmu.edu/usr/xujinl/anaconda3/envs/CSD/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/afs/ece.cmu.edu/usr/xujinl/anaconda3/envs/CSD/lib/python3.7/site-packages/torch/nn/functional.py", line 1610, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
</code></pre>
<p>What might be possible causes? Thank youl.</p>

username: ptrblck
cooked: <p>Is your model running fine on the CPU?<br>
This should give you a better stack trace than the current one.</p>
<p>If it’s running fine of the CPU, could you check, if you might be running out of memory and reduce the batch size if possible, since sometimes library errors might mask an actual OOM issue?</p>

username: ChrisLiu2
cooked: <p>Thanks for the reply. It was indeed a memory issue.</p>
How to compute derivatives of outputs w.r.t. parts of inputs in a differentiable manner, i.e. specific columns of Jacobian matrix

username: Leslie
cooked: <p>I am trying to compute the Jacobian matrix in a differentiable manner using <strong>torch.autograd.functional.jacobian with create_graph=True</strong>, but the dimensions of outputs and inputs are too large and the GPU running out of memory.</p>
<p>I found I just need some specific columns of the Jacobian matrix, ie the derivative of outputs w.r.t. some specific parts of inputs. If I can calculate that part of derivative in a differentiable manner, that would save lots of memory.</p>
<p>So I tried the following:</p>
<pre><code class="lang-auto">x = torch.tensor([1., 2., 3., 4., 5., 6.], requires_grad=True)
y = x
d = torch.autograd.grad(y[0], x[0],  retain_graph=True, create_graph=True, allow_unused=True)
print(d)

output: 
(None,)
</code></pre>
<p>But just got None.</p>
<p>Is there any way to compute derivatives of outputs w.r.t. parts of inputs in a differentiable manner?<br>
Or any other methods of computing specific columns of the Jacobian matrix in a differentiable manner?</p>

username: Leslie
cooked: <p>I’m not sure why the above code cannot get the answer.<br>
But I figure out another way:</p>
<pre><code class="lang-auto">x0 = torch.tensor([1., 2., 3.], requires_grad=True)
x1 = torch.tensor([4., 5., 6.], requires_grad=False)
x = torch.stack([x0, x1], dim=0)
y = x
d = torch.autograd.grad(y[0, 0], x0,  retain_graph=True, create_graph=True)
print(d)

output:
(tensor([1., 0., 0.]),)
</code></pre>
<p>But in this way, I have to take out the part of input <strong>x</strong> I need to calculate derivatives and then stack it with other parts of the input.</p>
<p>I used this method to calculate one column of the Jacobian matrix in a differentiable manner, but still out of memory. Any ideas for saving memory? The output size is (170, 2) and one part of inputs is (1, 768).</p>
Create a new device for FPGA and .to(device)

username: tataetae
cooked: <p>I am a part of a research group trying to accelerate certain layers on FPGA. In this case, we need to defined a new .to(device) and define our new device with a function that handles data movement between the accelerator and CPU. I have created a c++ extension for this but we are not so sure how could we define a new .to(device) without any in tree change. Is there any resource or example on that? I also need to have a custom malloc function to allocate the tensor as well. Thank you very much in advance!</p>