akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to process in parallel for parallel operations in nn.Module forward pass?

username: tlim
cooked: <p>Is there a way to write PyTorch nn.Module forward code so that we can pre-set certain operations to happen in parallel? Basically, make the <code>for loops</code> into “smart” <code>for loops</code>.</p>
<p>For example, in quite a number of scenarios like parallel convolutions, we are constantly using a <code>for loop</code> to <strong>sequentially</strong> go through each operation before proceeding to the next branch. Minimal code example:</p>
<pre><code class="lang-auto">.
.
.
class ParallelConv(nn.Module):
      def __init__(self):
            self.operations = nn.ModuleList([nn.Conv2d(....) for i in range(number_of_branches)])

      def forward(self, x):
            # Input 'x' is a list of tensors of individual branches/stream
            out = []
             for i, operation in enumerate(self.operations):
                  out.append(operations(x[i]))
             return out # returning also a list of tensor from each individual branch.

</code></pre>
<p>I do not think PyTorch will be able to optimise it during runtime to execute all the convolutions in parallel right?  How can we go about doing it ourselves then?</p>
<p>The same question for <code>torch jit</code> / <code>TorchScript</code>, when we trace the input through the modules set up, it is actually sequentially going through each branch operation before proceeding to the next. Can <code>TorchScript</code> identify that the inputs are separated into individual branches during compilation?</p>
<p>Advice is greatly appreciated.</p>

username: eqy
cooked: <p>Realistically, unless the operations can be executed concurrently on <em>multiple</em> devices, I wouldn’t expect any speedup when executing eagerly. In fact, there can be slowdowns attributed to multiple kernels contending for shared hardware resources.</p>
<p>If yo have multiple devices, you might check if running operations on different devices (e.g., in a model-parallel fashion) would speed things up, although this might require additional overhead in communicating the inputs across the devices first.</p>

username: OverLordGoldDragon
cooked: <p>This is also relevant for impl. 4D convs via 3D, the for loop is an unnecessary sequential to a fully parallelizable operation. Any pointers?</p>
Filter torch tensor of multidimensional array

username: bkbilly
cooked: <p>I have a model that responds with a tensor object and I want to filter it so that I can get a new tensor.<br>
My goal is to filter by the 6th argument so that I can get only the values that are zero (0).<br>
How would I do this efficiently?</p>
<p>I have an example of such an object. I don’t want to create a new object every time because it takes a while to create it as a cuda tensor.</p>
<pre><code class="lang-python">import torch
myobj = torch.tensor([
   [8.32500e+02, 1.72969e+02, 9.27000e+02, 3.81375e+02, 6.51855e-01, 0.00000e+01], 
   [5.01750e+02, 1.21500e+02, 8.13000e+02, 3.91875e+02, 2.75879e-01, 6.10000e+01], 
   [7.90500e+02, 2.60625e+02, 8.53500e+02, 3.55875e+02, 2.71729e-01, 0.00000e+01], 
   [0.00000e+00, 2.12812e+01, 1.26562e+02, 3.68250e+02, 2.68066e-01, 7.20000e+01]
])
myobj = myobj.to('cuda')</code></pre>

username: JLinden
cooked: <p>If I understood your question correctly, you want to filter <code>myobj</code> and in your example only return the first and third rows. You can do this with</p>
<pre><code class="lang-auto">filtered_myobj = myobj[myobj[:, 5] == 0]
</code></pre>

username: bkbilly
cooked: <p>Thanks! It works perfectly!<br>
I needed the ones that are equal with 0 so I used it like that:</p>
<pre><code class="lang-python">filtered_myobj = myobj[myobj[:, 5] == 0]
</code></pre>

username: JLinden
cooked: <p>Of course! Sorry about that. I’ve updated my answer so no one else who reads this gets confused by the accepted solution.</p>
Flatten batch input through linear layer in model

username: lanka
cooked: <p>hi,<br>
i have input tensor [10,1,74,74] here 10 is batch size,<br>
so i need flatten these vector and get [10,1024] output through fc layer(self.fc = nn.Linear(5476,1024)) like this , how can i do that.(74*74=5476)</p>
<p>input= [10,1,74,74]<br>
out=[10,1024]</p>

username: chrudimer
cooked: <p>Hi, if you want to flatten a tensor you could use the .view() function. So in your case</p>
<pre><code class="lang-auto">import torch

input_ = torch.ones((10, 1, 74, 74))

print(input_.shape)
# torch.Size([10, 1, 74, 74])

flattened_input = input_.view(10, -1)

print(flattened_input.shape)
# torch.Size([10, 5476])

linear_layer = torch.nn.Linear(5476, 1024).to("cuda")
out = linear_layer(flattened_input.to("cuda"))

print(out.shape)
# torch.Size([10, 1024])
</code></pre>
Is there any way to set the requires_grad as false of particular weights instead of whole layer?

username: divya
cooked: <p>I want to reset the gradients of certain weights. This code works at the layer level:</p>
<pre><code class="lang-auto">net.conv1.weight.requires_grad=False
</code></pre>
<p>What I want to do is, set the requires_grad to false of certain weights. Currently I am doing it in a very crude way, of manually setting the gradients to 0 after .backward() as below:</p>
<pre><code class="lang-auto">loss.backward()
net.conv1.weight.grad[i,j,k,l]=0
</code></pre>

username: KFrank
cooked: <p>Hi Divya!</p>
<aside class="quote no-group" data-username="divya" data-post="1" data-topic="144495" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/13edae/48.png" class="avatar"> divya:</div>
<blockquote>
<p>What I want to do is, set the requires_grad to false of certain weights.</p>
</blockquote>
</aside>
<p>No, you cannot turn off <code>requires_grad = True</code> just for certain elements<br>
of a pytorch tensor.  <code>requires_grad</code> applies to the entire tensor as a whole.</p>
<p>How best to do this depends on your actual goal.</p>
<p>If what you want is to “freeze” certain elements of your weight tensor – that<br>
is, have those elements not change when you call <code>opt.step()</code> – then it is<br>
safer to save the old values, call <code>opt.step()</code> (which might modify them),<br>
and then restore the saved values (rather than setting their gradients to<br>
zero before calling <code>opt.step()</code>).</p>
<p>This is because if you are using weight decay (or potentially momentum),<br>
<code>opt.step()</code> can modify elements for which the gradient is zero.</p>
<p>Best.</p>
<p>K. Frank</p>
Deeplabv3_mobilenet_large with 4 input channels

username: JLinden
cooked: <p>I’m trying to train DeepLabV3 with a MobileNetV3 backbone on a dataset with four channels. What is the preferred way of modifying the backbone to perform this? There does not seem to be a parameter for setting the number of input channels. I can not find any as straight forward approach as when using ResNet as the backbone:</p><aside class="quote" data-post="1" data-topic="123089">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/59ef9b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/train-deeplabv3-on-4-channels-images/123089">Train Deeplabv3 on 4 channels images</a> 
  </div>
  <blockquote>
    Hi there, i want to train deeplabV3 on my own Dataset with 4 channels images. but i didn’t find any PyTorch implementation of deeplabV3 where i could change parameters and input channels number of the model to fit my (4channels) images . 
How can i modify deeplabV3 to adapt it to my dataset?
  </blockquote>
</aside>

<p>The following solution works, but it does not feel right to change a protected attribute.</p>
<pre><code class="lang-auto">from functools import partial
from torch import nn
from torchvision import models
from torchvision.ops.misc import ConvNormActivation

model = models.segmentation.deeplabv3_mobilenet_v3_large(pretrained=False, num_classes=2)
model.backbone._modules['0'] = ConvNormActivation(4,
                                              16,
                                              kernel_size=3,
                                              stride=2,
                                              norm_layer=partial(nn.BatchNorm2d, eps=0.001, momentum=0.01),
                                              activation_layer=nn.Hardswish)
</code></pre>

username: JLinden
cooked: <p>I realized that <code>model.backbone</code> has a getter for the items in <code>_modules</code>, so I changed the code to:</p>
<pre><code class="lang-auto">model.backbone['0'] = ConvNormActivation(4,
                                         16,
                                         kernel_size=3,
                                         stride=2,
                                         norm_layer=partial(nn.BatchNorm2d, eps=0.001, momentum=0.01),
                                         activation_layer=nn.Hardswish)
</code></pre>
Updating few channels of the input

username: ashesh-0
cooked: <p>I’m trying to optimize specific channels of the input. So, my network weights are fixed and I just want to update select channels of the input. I was able to narrow down the issue to following bits of code. In my understanding following two pieces of code should be identical:</p>
<pre><code class="lang-auto">parameters = []
for i in range(nTot):
    parameters.append(inpt[i][:,:channel ])
    parameters.append(inpt[i][:,channel:])
optimizer = optim.Adam(parameters, lr=model.lr, weight_decay=0)

</code></pre>
<p>and</p>
<pre><code class="lang-auto">parameters = []
for i in range(nTot):
    parameters.append(inpt[i])
optimizer = optim.Adam(parameters, lr=model.lr, weight_decay=0)

</code></pre>
<p>Here <code>inpt</code> is a list of tensors and <code>channel</code> is an integer. I find that in 2nd case, my inputs gets updated but in first case, it does not. If this works then I’ll modify the first code and will then be successful in updating only few channels of the input. Many thanks !!</p>

username: ptrblck
cooked: <p>Your first approach won’t work as you are trying to optimize a non-leaf tensor by slicing it and this error is raised:</p>
<pre><code class="lang-python">ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>If you want to optimize parts of a parameter, either zero out the gradients of the frozen part or create separate parameters and use e.g. <code>torch.cat</code> or <code>torch.stack</code> to create the large parameter before applying it in your model.</p>

username: ashesh-0
cooked: <p>Thanks a lot for your reply. I was able to do this with something similar to second approach suggest by you !</p>
UnpicklingError: invalid load key, '\x0a'

username: Eddie1997
cooked: <p>Hello,</p>
<p>I met a problem that I trained my model in the lab using a Linux machine, after saving the checkpoint I took it home to use my Windows machine to load it and I got an error:</p>
<p>File “C:\Users\Eddie\anaconda3\envs\Single\lib\site-packages\torch\serialization.py”, line 777, in _legacy_load<br>
magic_number = pickle_module.load(f, **pickle_load_args)</p>
<p>UnpicklingError: invalid load key, ‘\x0a’.</p>
<p>Can anyone help me with this issue? I guess it might be the problem of pytorch version, on the Windows machine I installed the latest Pytorch v1.9.0 and cuda 11.3, but the Linux machine is like v1.8.1 or even older.</p>
<p>Thanks a lot!<br>
Eddie</p>

username: ptrblck
cooked: <p>Could you compare the file sizes on both machines and make sure that the file was properly copied?</p>

username: Eddie1997
cooked: <p>Hi</p>
<p>Sorry I just saw it, it has been solved. My mistake, I did not copy the trained models properly, sorry about that.</p>
<p>Thank you very much!<br>
Jianyu</p>
Cpp_extension TypeError:dist must be a Distribution instance

username: MartinZhang
cooked: <pre><code class="lang-auto">import os
import torch
from torch.utils.ffi import create_extension


sources = ['src/dcn_v2.c']
headers = ['src/dcn_v2.h']
defines = []
with_cuda = False

extra_objects = []
if torch.cuda.is_available():
    print('Including CUDA code.')
    sources += ['src/dcn_v2_cuda.c']
    headers += ['src/dcn_v2_cuda.h']
    defines += [('WITH_CUDA', None)]
    extra_objects += ['src/cuda/dcn_v2_im2col_cuda.cu.o']
    extra_objects += ['src/cuda/dcn_v2_psroi_pooling_cuda.cu.o']
    with_cuda = True
else:
    raise ValueError('CUDA is not available')

extra_compile_args = ['-fopenmp', '-std=c99']

this_file = os.path.dirname(os.path.realpath(__file__))
print(this_file)
sources = [os.path.join(this_file, fname) for fname in sources]
headers = [os.path.join(this_file, fname) for fname in headers]
extra_objects = [os.path.join(this_file, fname) for fname in extra_objects]

ffi = create_extension(
    '_ext.dcn_v2',
    headers=headers,
    sources=sources,
    define_macros=defines,
    relative_to=__file__,
    with_cuda=with_cuda,
    extra_objects=extra_objects,
    extra_compile_args=extra_compile_args
)

if __name__ == '__main__':
    ffi.build()
</code></pre>
<p>Import Error: torchutils.ffi is deprecated. Please use cpp extensions instread.</p>
<p>So, I make it</p>
<pre><code class="lang-auto">import os
import torch
from torch.utils.ffi import create_extension
from torch.utils.cpp_extension import BuildExtension


sources = ['src/dcn_v2.c']
headers = ['src/dcn_v2.h']
defines = []
with_cuda = False

extra_objects = []
if torch.cuda.is_available():
    print('Including CUDA code.')
    sources += ['src/dcn_v2_cuda.c']
    headers += ['src/dcn_v2_cuda.h']
    defines += [('WITH_CUDA', None)]
    extra_objects += ['src/cuda/dcn_v2_im2col_cuda.cu.o']
    extra_objects += ['src/cuda/dcn_v2_psroi_pooling_cuda.cu.o']
    with_cuda = True
else:
    raise ValueError('CUDA is not available')

extra_compile_args = ['-fopenmp', '-std=c99']

this_file = os.path.dirname(os.path.realpath(__file__))
print(this_file)
sources = [os.path.join(this_file, fname) for fname in sources]
headers = [os.path.join(this_file, fname) for fname in headers]
extra_objects = [os.path.join(this_file, fname) for fname in extra_objects]

ffi = BuildExtension(
    '_ext.dcn_v2',
    headers=headers,
    sources=sources,
    define_macros=defines,
    relative_to=__file__,
    with_cuda=with_cuda,
    extra_objects=extra_objects,
    extra_compile_args=extra_compile_args
)

if __name__ == '__main__':
    ffi.build()
</code></pre>
<p>TypeError: dist must be a Distribution instance</p>
<p>please, help me thanks.</p>

username: pranay_ar
cooked: <p>Hey, I am facing the same error too. Were you able to solve this?</p>
Most efficient way to calculate ROC-AUC in Trainings loop

username: JanoschMenke
cooked: <p>Lets say I am training a Binary Classification and I want to calculate the ROC-AUC.<br>
Whats the fastest, most efficient way to calculate the ROC_AUC see during training.<br>
Assuming I am using the <code>BCEWithLogitsLoss</code>.</p>
Pytorch version of PC and Mobile differs

username: mmz211
cooked: <p>Hi,</p>
<p>Our AI team train model under pytorch 1.10.1.<br>
and pytorch version used on Android platform is 1.9.0</p>
<pre><code class="lang-auto">    implementation 'org.pytorch:pytorch_android_lite:1.9.0'
    implementation 'org.pytorch:pytorch_android_torchvision:1.9.0'
</code></pre>
<p>will it cause problems? do I need to update versions on Android?</p>
<p>best</p>

username: Linbin
cooked: <p>It’s not guaranteed to work. Old model can work with new pytorch version but not vice versa.</p>

username: mmz211
cooked: <p>Thanks a lot. a lot. at least 20 chars</p>
How to keyboardinterrupt training in jupyter nb?

username: jayz
cooked: <p>Hello,</p>
<p>when trying to interrupt the training of my models in jupyternb, nothing happens, due to the multiprocessing in the background. My question is: is there any handy method that allows to propagate the keyboardinterrupt to all employed background processes, such that restarting the notebook is not required, anymore? Just restarting wouldnt even be so bad, but, in my case (maybe also for others), the kernel dies permanently, when I interrupt training by restarting the notebook. So I have to restart the entire notebook application every time. Do this ten times in half an hour, and you will be writing this post here <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Thanks!</p>
<p>Best, JZ</p>
Can't find linalg_lu_factor functions

username: turbotage
cooked: <p>In the c++ <a href="https://pytorch.org/cppdocs/api/file_build_aten_src_ATen_Functions.h.html#file-build-aten-src-aten-functions-h" rel="noopener nofollow ugc">documentation</a> there are many alternatives for lu factorizations. However, the only lu factorization function I can find in my project is <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a2287ab9932135bb3a0d13007410fdf83.html#exhale-function-namespaceat-1a2287ab9932135bb3a0d13007410fdf83" rel="noopener nofollow ugc">at::_lu_with_info</a>. Maybe there is some extra header I have to include. I have looked inside <strong>/include/ATen/Functions.h</strong> and the only function I find there is again <strong>at::_lu_with_info</strong>. Im looking specifically for <strong>linalg_lu_factor_ex_out</strong>, but all similar functions are also gone. I downloaded 1.10.2 with cuda 11.3</p>
TracerWarning error when calling SummaryWriter add_graph

username: ppytorch
cooked: <p>When I call <code>SummaryWriter().add_graph(model, input)</code>, I get the following error:</p>
<blockquote>
<p>TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:<br>
With rtol=1e-05 and atol=1e-05, found 32 element(s) (out of 32) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 0.013833858072757721 (-0.04951992258429527 vs. -0.03568606451153755), which occurred at index (1, 0).</p>
</blockquote>
<p>I Initially suspected that the dropout layers may be the cause, but setting <code>model.train(False)</code> before calling <code>add_graph</code> doesn’t resolve the issue.</p>
<p>Pytorch 1.9.1<br>
TensorBoard 2.7.0</p>
PyTorch DDP - Large amount of time on optimizer.step on some batches randomly

username: Ritwick_Chaudhry
cooked: <p>I’m using PyTorch DDP to train a ResNet101 over 4 p3.16xlarge EC2 instances, each with 8 GPUs. What I notice is that the training goes well till some epochs, and then it slows down suddenly, because of some batches taking a huge amount of time (~3000 seconds). I profiled, and found out that the slow down always happens due to the optimizer.step() operation. The 4 instances are in the same availability zone, same subnet and same cluster placement group. And sometimes, the training is slow (spikes in some batches) right from the start. This is not a data issue as the slowdown is not due to the batch loading time, and because the slowdown is not in all epochs. Any leads on resolving this would be greatly appreciated.</p>

username: ptrblck
cooked: <p>How did you profile the code and narrowed down the <code>optimizer.step()</code> method?<br>
Are you sure the actual <code>step()</code> call is causing the slowdown or would the <code>step()</code> function rather have to wait for other nodes to finish its backward pass before it can start updating the parameters?</p>

username: Ritwick_Chaudhry
cooked: <p>Thanks for your response! I just used <code>time.time()</code> before and after the statement and then logged it onto tensorboard. It could include time for when the <code>step()</code> function blocked. Also, if I use <code>AdamW</code> instead of <code>SGD</code> it’s the <code>backward()</code> instead of <code>step()</code>.</p>
Weight decay on parametrized weight

username: grudloff
cooked: <p>If a weight is parametrized, and one uses weight decay on the optimizer. This weight decay is applied over the original weight or the parametrized weight?</p>

username: grudloff
cooked: <p>I found the answer by myself. I will share my findings in case someone else stumbles with the same doubt.</p>
<p>Optimizer’s <code>weight_decay</code> param is used by scaling the parameters and adding them accordingly to the selected optimizer. Usually, they are added to the gradients as <code>$g += \lambda*\theta$</code>, in the case of AdamW a proportional term is added directly to the param update step(see <a href="https://paperswithcode.com/method/adamw#:~:text=AdamW%20is%20a%20stochastic%20optimization,decay%20from%20the%20gradient%20update." rel="noopener nofollow ugc">AdamW Explained</a>)</p>
<p>Since the actual parameter is the “original” parameter and not the parametrized one (as can be see from <code>model.named_parameters()</code>), then the weight decay is applied over the original parameter and not the parametrized one.</p>
<p>Here is a snippet from <code>sgd</code>'s implementation:</p>
<pre><code class="lang-python">        if weight_decay != 0:
            d_p = d_p.add(param, alpha=weight_decay)
</code></pre>
<p>As can be seen, the update is done over <code>param</code>, given to the optimizer’s constructor on creation.</p>
How to merge tensor batches!

username: Mohamed_Nagy
cooked: <p>Hello,</p>
<p>I have a tensor with size <code>(16, 3, 280, 280)</code>, and I want to merge the batches to be added as channels. So, the last size is <code>(48, 280, 280)</code>.  Is there a way to do it (ex: built-in function in pytorch)? I found function called <code>stack</code> but it accepts tuple of tensors, and I don’t want to iterate over the batch size. I would rather prefer if there is another way to do the process in bulk.</p>

username: Yuv
cooked: <p>how about using <a href="https://pytorch.org/docs/stable/generated/torch.reshape.html" rel="noopener nofollow ugc">reshape</a>?</p>
<p>For example :</p>
<pre><code class="lang-auto">x = torch.rand(16, 3, 280, 280)
x = x.reshape(48, 280, 280)
</code></pre>

username: Mohamed_Nagy
cooked: <p>I thought it might change the order of some values in inside, so I didn’t try it but It works fine. Thank you.</p>
Define a dict of dataloaders

username: Torcione
cooked: <p>Hi everyone,<br>
I have necessity, starting from a dataset, to divide it in subgroups and define for each of them a corresponding dataloader; is it possible to group them in a dictionary of dataloaders?</p>

username: ptrblck
cooked: <p>Yes, you should be able to create a <code>dict</code> containing <code>DataLoader</code> instances.</p>
Extracting Intermediate Layer Output

username: mrityu
cooked: <p>Hey,<br>
Please refer to the below custom model class’s forward method:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.layer1(x)
    temp = self.layer2(x)
    x = self.layer2(x)
    x = self.out(x)
    return x, temp
</code></pre>
<p>I am trying to extract an intermediate layer’s output using this(layer-2 output). I am aware of the forward hooks and that hooks are the best way for this task, but out of curiosity, I am asking this. Will the above code snippet will end up doing something unexpected?</p>
<p>My <code>train</code> function looks like this:</p>
<pre><code class="lang-auto">def train(X, Y):
   model = MyModel()
   output, intermediate_output = model(X)
   loss = some_criterion(output, Y)
   loss.backward()
   .
   .
   ....
</code></pre>
<p>To be precise I want to ask that will there be any change in convergence point(assuming everything idle) because of the tweaked forward method?</p>
<p>The original forward method is given below for reference:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.out(x)
    return x
</code></pre>
<p>As per my current understanding of PyTorch, I don’t think there should be any change in the model’s convergence point i.e. model should behave the same way it was doing before the forward method was tweaked. Am I missing something here?</p>

username: ptrblck
cooked: <p>Depending on the implementation of <code>self.layer2</code> your new code snippet might change the behavior e.g. if batchnorm layers are used. In this case you would update the running stats twice since two forward passes are used in the new code.</p>
NNAPI Serializer: dealing with unsupported operation transpose

username: jeeheh
cooked: <p>Hi,</p>
<p>I am trying to convert a model (namely shufflenet) for android using NNAPI but am running into some issues related to unsupported operations - namely transpose and contiguous.</p>
<p>How can I change my model so that it will convert?</p>
<pre><code class="lang-auto">class ChannelShuffle(nn.Module):
    def __init__(self, groups):
        super().__init__()
        self.groups = groups
    
    def forward(self, x):
        N,C,H,W = x.size() 
        g = self.groups
        x = x.reshape(N, g, -1, H, W)
        x = torch.transpose(x, 1, 2).contiguous() # this is the offending line
        x = x.reshape(N,C,H,W)
        
        return x
</code></pre>
<p>Thank you,<br>
Jeeheh</p>

username: axitkhurana
cooked: <p>Hi Jeeheh, at this time we don’t support transpose in NNAPI: <a href="https://github.com/pytorch/pytorch/blob/master/torch/backends/_nnapi/serializer.py#L784" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/backends/_nnapi/serializer.py#L784</a></p>
<p>We’re open to contributions to add more op support</p>
Manually changing the delta in the middle of backward

username: Ibn
cooked: <p>I don’t understand how to change manually the delta error of a specific layer in the middle of the backward process, and I need that the backpropgation will continue with the changed value.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99d282f7f84760c1304f58c95b96cf23438e83ca.png" alt="delta_bp" data-base62-sha1="lWM4sK4yCMIPyvSsQQsP9xDiq3E" width="262" height="46"></p>
<p>Thanks a lot</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use hooks for that. In particular, you can use <code>register_hook(fn)</code> on any Tensor in the middle of your computation to inspect and optionally change the gradient flowing for it.</p>
How to calculate class weights for token level classification problem?

username: TARUN_BHATIA
cooked: <p>For each of my sentence the 0 labels are very less as compared to the 1’s (for token level classification). I use batches and calculate loss(CrossEntropy) after each batch. How should i create the class weights vector and use it in the loss calculation. Please suggest !</p>

username: ptrblck
cooked: <p>You could initialize the weights as e.g. the class frequency which would then add more weight to the rare classes.</p>

username: TARUN_BHATIA
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , so it would be like class_freq_zeros = 1 / &lt;no of 0 tokens in whole batch of 8 sentences&gt;, and similarly for 1’s ?</p>

username: ptrblck
cooked: <p>Yes, a per-batch weighting would work, but you should also consider checking the class distribution of the entire dataset and set the weights once before training to see which approach would be better.</p>