A large number of indexing operations result in very slow back propagation in pytorch

Question:
when I use the following code to train my model in pytorch, I find the backward operation is extremely slow.
def sampling_prob(prob, label, num_neg):
    label = label - 1
    batch, pred_len, l_m, = prob.shape[0], prob.shape[1], prob.shape[2] - 1
    init_label = np.zeros_like(label)
    init_prob = torch.zeros(size=(batch, pred_len, num_neg + batch))  # torch.Size([32, 12, 42])

    for step in range(pred_len):
        _label = np.linspace(0, batch - 1, batch)
        _prob = torch.zeros(size=(batch, num_neg + batch))

        batch_step_label = label[:, step]

        random_ig = random.sample(range(1, l_m + 1), num_neg)
        while len([lab for lab in batch_step_label if lab in random_ig]) != 0:
            random_ig = random.sample(range(1, l_m + 1), num_neg)

        global global_seed
        random.seed(global_seed)
        global_seed += 1

        for k in range(batch):
            for i in range(num_neg + batch):
                if i < batch:
                    _prob[k, i] = prob[k, step, batch_step_label[i]]
                else:
                    _prob[k, i] = prob[k, step, random_ig[i - batch]]

        init_label[:, step] = _label
        init_prob[:, step, :] = _prob

prob = self.model(train_input, train_m1, self.mat2s, train_m2t, train_len)
prob_sample, label_sample = sampling_prob(prob, train_label, self.num_neg)
loss_train = F.cross_entropy(prob_sample, label_sample)
loss_train.backward()
optimizer.step()
optimizer.zero_grad()

Indeed, I want to sample some data from the the output of the model, i.e.,prob, to calculate the loss and optimize the parameters. But the speed of the backward operation is extremely slow. I am wondering this is because a lot of indexing operations. What would be the reason for that?  How can I solve this problem?

Context:
This records all _prob tensor “versions”, the framework doesn’t know that writes are non-overlapping and intermediate states are discardable. Then you have a long chain of autograd nodes to be processed during backward().
Workaround is to use torch.gather to set all values at once. Another option is to rewrite this with torch.where & torch.cat, similarly avoiding assignments to a buffer.


Best practises for “turning off” parts of the model?

Question:
Hi all,
I’m trying to do some ablation studies (or sensitivity studies in some other jargon), turning on and off different components of my model.
For now I’ve assumed that to avoid passing the input through a certain layer (say an nn.Embedding) it would be sufficient to avoid gradient contributions from that component of the model.
That will evidently make my model parameters larger than necessarily, with some of the layers not being updated past their initialisation.
Something tells me this is not exactly the best way to do these things. What would be your take? Would you rather define a specific model just for that component you want to turn on/off?

Context:
You could do something like
class Model(nn.Module):
    def init(self, args):
        self.m1 = Module1()
        if args.use_my_second_layer:
             self.m2 = Module2()
    def forward(self, x):
        h = self.m1(x)
        if args.use_my_second_layer:
             h = self.m2(x)
        return h


Backprop through iterative process with in-place CUDA ops

Question:
Hi, this is a bit of an unusual scenario:
I have a CUDA kernel implementation of an autograd Function Func with custom backward pass which I use as follows during forward pass:

x_1 = from data loader
for t = 1 : N
x_(t + 1) = Func(x_t)

Where N can very large e.g. 100 and so to decrease GPU memory usage, Func performs operations on x in-place and sets the mark_dirty flag on it. During backward pass I would like to recompute the computation graph by basically trading compute for memory. Would I need to use checkpointing to achieve this goal or does setting mark_dirty flag on a Tensor automatically will recompute the input? Or said another way how does mark_dirty recomputes the lost Tensor, does it keep a copy in memory or computes it?
Thanks!

Context:
Hi,
mark_dirty does not recompute the Tensor for you in any ways. It only allows autograd to know what was modified inplace for error checking purposes.


RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256]], which is output 0 of AsStrid edBackward0, is at version 4; expected version 3 instead

Question:
Hi, I am using PyTorch 1.10 and encountered this issue. Here is my code:
class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, embed_dim, bs):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.bs = bs
        self.embed_dim = embed_dim
        self.embedding = nn.Embedding(output_size, embed_dim)
        self.gru = nn.GRU(embed_dim, hidden_size)
        self.linear = nn.Linear(2 * hidden_size, output_size)
        self.softmax_0 = nn.LogSoftmax(dim=0)
        self.softmax_1 = nn.LogSoftmax(dim=1)
        self.hidden = None

    def forward(self, input, hidden, encoder_outputs):
        output = self.embedding(input)  # 1 * bs * ed
        output = F.relu(output)  # 1 * bs * ed
        # output: 1 * bs * hs
        # hidden: 1 * bs * hs
        output, hidden = self.gru(output, hidden)
        weights = self.dot_product(output, encoder_outputs)
        weights = self.softmax_0(weights)
        attn_output = self.weighted_sum(weights, encoder_outputs)  # bs * ed
        output = torch.cat((output[0], attn_output), dim=1)
        output = self.linear(output)  # bs * os
        output = self.softmax_1(output)  # bs * os
        return output, hidden

    def weighted_sum(self, weights, encoder_outputs):
        outputs = encoder_outputs.clone()
        res = torch.tensor([], device=device)
        for i in range(weights.size()[0]):  # n
            weight = weights[i]  # bs
            words = outputs[i]  # bs * ed
            for j in range(weight.size()[0]):  # bs
                words[j] = words[j] * weight[j]
            res = torch.cat((res, words.unsqueeze(0)), dim=0)  # n * bs * ed
        return torch.sum(res, dim=0)  # bs * ed

    def dot_product(self, a, b):
        # a: 1 * bs * hs
        # b: n * bs * hs
        res = torch.tensor([], device=device)
        decoder_word = a.squeeze(0)
        for i in range(b.size()[0]):
            temp = torch.tensor([], device=device)
            for j in range(b.size()[1]):
                dot = torch.matmul(decoder_word[j], b[i][j]).unsqueeze(0)   # HERE
                # dot = torch.randn(1, device=device)
                temp = torch.cat((temp, dot), dim=0)
            res = torch.cat((res, temp.unsqueeze(0)), dim=0)  # n * bs
        return res

    def initHidden(self):
        return torch.zeros(1, self.bs, self.hidden_size, device=device)

I found that if I replace “dot = torch.matmul(decoder_word[j], b[i][j]).unsqueeze(0)” by “dot = torch.randn(1, device=device)”, then no errors will be reported. Not sure if this information is helpful. Could anyone give me some suggestions? Thanks!

Context:
JackGao09:

words[j] = words[j] * weight[j]

Note completely sure, but possibly this operation is modifying the Tensor in-place and thus losing the ability for performing backprop correctly. Possibly you can save the result of words[j] * weight[j] in a different place?


Unable to perform Backpropagation during RNN training

Question:
Hello this is my first post ever so sorry in advance if I don’t meet the standards for posting here.
Context :
I have a problem with training a RNN I try to implement from a paper.
This RNN is called a state-space RNN with 2 equations :

A state evolution equation which corresponds to the evolution of the hidden state in the network – so we try to learn the evolution of the state. So my assumption is we need not to detach the sate inside the mini batch but we need to initialize it at each new batch. Also we need to retain the graph.
An output equation : using the current state and input predict the output at the current time step

Below the corresponding equations where x is the hidden state and y the output

My problem is : when I try to feed to the network each time step k for k from 0 to batch_size. The backpropagation works for the first batch but not for the second…
Giving the classic error : one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [15, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead.
I spent time looking for solutions to this error but : detaching the state does not seem relevant, I do not think there is an inplace operation I can correct in my code.
Please find below the trace given with torch.autograd.set_detect_anomaly(True) set, and my code. I do not understand why the ‘NL2’ layer cannot compute gradient specifically…
Epoch n° 0 started
0
1
C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\site-packages\torch\autograd\__init__.py:154: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\Alexandre\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy\__main__.py", line 45, in <module>
    cli.main()
  File "c:\Users\Alexandre\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 444, in main
    run()
  File "c:\Users\Alexandre\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\Alexandre\Desktop\Recherche\Simulation\Lip_NSSM\Tuning_grNSSM.py", line 345, in <module>
    main()
  File "c:\Users\Alexandre\Desktop\Recherche\Simulation\Lip_NSSM\Tuning_grNSSM.py", line 323, in main
    loss_course, accuracy_course = train_network(model, train_loader, test_loader, criterion, EPOCHS, LEARNING_RATE)
  File "c:\Users\Alexandre\Desktop\Recherche\Simulation\Lip_NSSM\Tuning_grNSSM.py", line 278, in train_network
    epoch_loss = model.train_model(train_loader, initial_state, optimizer, criterion)
  File "c:\Users\Alexandre\Desktop\Recherche\Simulation\Lip_NSSM\Tuning_grNSSM.py", line 213, in train_model
    output, state = self(data[k], state)    # Pass data through the network
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\Users\Alexandre\Desktop\Recherche\Simulation\Lip_NSSM\Tuning_grNSSM.py", line 124, in forward
    x_2 = self.NL2(x_1)
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\site-packages\torch\nn\modules\linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\Alexandre\anaconda3\envs\grLipNSSM\lib\site-packages\torch\nn\functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
 (Triggered internally at  ..\torch\csrc\autograd\python_anomaly_mode.cpp:104.)
  Variable._execution_engine.run_backward(
Backend QtAgg is interactive backend. Turning interactive mode on.

Init :
def __init__(self, input_dim = INPUT_DIM, hidden_dim = HIDDEN_DIM,  state_dim = STATE_DIM,  output_dim = OUTPUT_DIM, init_mode = 'random'):

        super(grLipNSSM, self).__init__()

        # Number of Input neurons

        self.input_dim = input_dim

        # Number of hidden dimensions

        self.hidden_dim = hidden_dim

        # Size of the state space

        self.state_dim = state_dim

        # Number of output dimensions

        self.output_dim = output_dim

        # Activation functions

        self.actF = torch.nn.Tanh()

        # Input layer for the state W_u

        self.input_layer = nn.Linear(self.input_dim, self.hidden_dim, bias=False)

        # First state layer

        self.state_layer = nn.Linear(self.state_dim, self.hidden_dim)

        # Nonlinear part for the state

        self.NL1 = nn.Linear(self.hidden_dim, self.state_dim)

        # Residual block

        self.residualA = nn.Linear(self.state_dim, self.state_dim, bias= False)

        self.residualB = nn.Linear(self.input_dim, self.state_dim, bias= False)

        # Output layers

        self.input_layer2 = nn.Linear(self.input_dim, self.hidden_dim, bias= False)

        self.state_layer2 = nn.Linear(self.state_dim, self.hidden_dim)

        self.NL2 = nn.Linear(self.hidden_dim, self.output_dim)

        self.residualC = nn.Linear(self.state_dim, self.output_dim, bias= False)

        self.residualD = nn.Linear(self.input_dim, self.output_dim, bias= False)

Forward pass

def forward(self, u, state = torch.zeros(1, STATE_DIM)):

        # Forward pass -- prediction of the output at time k : y_k

        x_0 = self.input_layer2(u)

        # Skip connections layers

        sc_A = self.residualA(state)

        sc_B = self.residualB(u)

        sc_C = self.residualC(state)

        sc_D = self.residualD(u)

        # Hidden layers in the non linear part

        h1 = self.state_layer(state)

        h2 = self.state_layer2(state)

        x_1 = self.actF((x_0 + h2)) #Add another activation function

        x_2 = self.NL2(x_1)

        y_k = x_2 + sc_C + sc_D

        # Updating the state

        x_3 = self.input_layer(u)

        x_4 = self.actF(x_3+ h1)

        x_5 = self.NL1(x_4)

        #Adding the linear part (skip connection)

        state = x_5 + sc_A + sc_B

        return y_k, state

Training function :
def train_model(self, train_loader, initial_state, optimizer, criterion):
        """Train neural network model with Adam optimizer for a single epoch
        params:
            * model: nn.Sequential instance                 - NN model to be tested
            * train_loader: DataLoader instance             - Training data for NN
            * initial_state :                               - Initial state for the eopch
            * optimizer: torch.optim instance               - Optimizer for NN
            * criterion: torch.nn.CrossEntropyLoss instance - Loss function
            * state : stat
        modifies:
            weights of neural network model instance
        """
        self.train()   # Set model to training mode
        sim_loss = 0
        epoch_sim_loss = 0
        for idx, (data, target) in enumerate(train_loader):
            data = data.view(BATCH_SIZE, -1)
            state = self.init_hidden()
            print(idx)
            optimizer.zero_grad()   # Zero gradient buffers
            for k in range(len(data)): # Iterates through time

                output, state = self(data[k], state)    # Pass data through the network
                sim_loss += criterion(output, target[k])    # Calculate loss
            sim_loss.backward(retain_graph=True)     # Backpropagate

            optimizer.step()    # Update weights
            epoch_sim_loss += sim_loss.item()
        return epoch_sim_loss

The error is occurring when we are doing the backprop step for the 2nd minibatch
Thank you for your help.

Context:
I’m unable to reproduce this error with the code you’ve shared.
import torch
from torch import nn

class grLipNSSM(nn.Module):

  def __init__(self, input_dim = 10, hidden_dim = 15,  state_dim = 10,  output_dim = 10, init_mode = 'random'):
          super(grLipNSSM, self).__init__()

          # Number of Input neurons
          self.input_dim = input_dim

          # Number of hidden dimensions
          self.hidden_dim = hidden_dim

          # Size of the state space
          self.state_dim = state_dim

          # Number of output dimensions
          self.output_dim = output_dim

          # Activation functions
          self.actF = torch.nn.Tanh()

          # Input layer for the state W_u
          self.input_layer = nn.Linear(self.input_dim, self.hidden_dim, bias=False)

          # First state layer
          self.state_layer = nn.Linear(self.state_dim, self.hidden_dim)

          # Nonlinear part for the state
          self.NL1 = nn.Linear(self.hidden_dim, self.state_dim)

          # Residual block
          self.residualA = nn.Linear(self.state_dim, self.state_dim, bias= False)
          self.residualB = nn.Linear(self.input_dim, self.state_dim, bias= False)

          # Output layers

          self.input_layer2 = nn.Linear(self.input_dim, self.hidden_dim, bias= False)
          self.state_layer2 = nn.Linear(self.state_dim, self.hidden_dim)
          self.NL2 = nn.Linear(self.hidden_dim, self.output_dim)
          self.residualC = nn.Linear(self.state_dim, self.output_dim, bias= False)
          self.residualD = nn.Linear(self.input_dim, self.output_dim, bias= False)

  def forward(self, u, state = torch.zeros(1, 10)):

          # Forward pass -- prediction of the output at time k : y_k

          x_0 = self.input_layer2(u)

          # Skip connections layers

          sc_A = self.residualA(state)
          sc_B = self.residualB(u)
          sc_C = self.residualC(state)
          sc_D = self.residualD(u)

          # Hidden layers in the non linear part
          h1 = self.state_layer(state)
          h2 = self.state_layer2(state)
          x_1 = self.actF((x_0 + h2)) #Add another activation function
          x_2 = self.NL2(x_1)
          y_k = x_2 + sc_C + sc_D

          # Updating the state
          x_3 = self.input_layer(u)
          x_4 = self.actF(x_3+ h1)
          x_5 = self.NL1(x_4)

          #Adding the linear part (skip connection)
          state = x_5 + sc_A + sc_B
          return y_k, state

mdl = grLipNSSM()
loss = 0
for _ in range(3):
  l,m = mdl(torch.rand(5,10))
  loss += torch.mean(torch.pow(l - torch.rand_like(l), 2))
loss.backward()

works fine for me


How to write backward function with two inputs

Question:
Now here is a distributed computing problem.
I write this function for communication between GPUs
class RemoteReceive(autograd.Function):
    @staticmethod
    def forward(ctx, input:torch.tensor = torch.tensor(1.0),from_rank:int = 0):
        dim = torch.tensor(1.0)
        dist.recv(dim,from_rank)
        size = torch.rand(int(dim))
        dist.recv(size,from_rank)
        x = torch.zeros(tuple(size.int()))

        dist.recv(x, from_rank)
        return x

    @staticmethod
    def backward(ctx, grad_output:torch.tensor = torch.tensor(1.0),to_rank:int = 0):
        dist.send(torch.tensor(grad_output.dim()*1.0),to_rank)
        #print("send to dim",to_rank,input.size.dim())
        dist.send(torch.tensor(grad_output.size())*1.0,to_rank)
        dist.send(grad_output, to_rank)
        return None

I need two input in backward. One for send to another GPU and another for choosing the rank of the GPU, but when I reference this function. like
        x = torch.rand(1)
        y = torch.rand(1)
        RemoteReceive.apply(x,1)
        print(rank,"finish forward")
        x.backward(x ,to_rank = 1)

A error occur
TypeError: backward() got an unexpected keyword argument ‘to_rank’
So how can I put these two arguments to backward,please.

Context:
Hi,
The backward that you implement is not the same as the one that gets called. The one you write will only work with gradients.
You can solve this by simply passing both the from_rank and to_rank to the apply (and so take both as input for your forward).
And in your forward save anything that you need for the backward on the ctx: ctx.to_rank = to_rank.


Convert tensor to Parameter (while keeping the graph)

Question:
I want to create a neural network generating the parameters of another NN. The NN ‘A’ generates ‘B’.
But there is a problem: the output of ‘A’ is a ‘tensor’ (with requires_grad = True) while the parameters of ‘B’ are ‘Parameter’. Since I want to backward the gradient up to the parameters of ‘A’, I am looking for a conversion tensor -> Parameter that keeps the computational graph.
A naive conversion ‘w = Parameter(y)’, where w is the generated parameter of ‘B’, and y is the output of ‘A’, does not work: the computational graph is cut between w and y (the gradient of the parameters of ‘A’ are None, while the gradient of w, which should be non-leaf, is not None).

Context: It depends on what ‘B’ does. Have you looked into torch.nn.functional APIs?


How can I call autograd.grad within an autograd.Function?

Question:
I need to implement my own function which inherits from autograd.Function, where I mannually calculate some gradient using autograd.grad. But I find this does not work (see below). I am a bit new here. Who can help me?
import torch
def simple_func():
    a = torch.tensor(3., requires_grad=True)
    c = a + 2
    grads = torch.autograd.grad(c, a, grad_outputs=torch.tensor(5.))
    print(grads)

class dummy_autograd_func(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x + 2

    @staticmethod
    def backward(ctx, dy):
        simple_func()
        return dy

# Running normally; it is fine
simple_func()

# Running within autograd.Function; it fails
x = torch.rand(3, 2, requires_grad=True)
y = dummy_autograd_func.apply(x)
s = y.sum()
s.backward()

The error message is
Traceback (most recent call last):
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 29, in <module>
    s.backward()
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply
    return user_fn(self, *args)
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 19, in backward
    simple_func()
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 7, in simple_func
    grads = torch.autograd.grad(c, a, grad_outputs=torch.tensor(5.))
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 234, in grad
    return Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Context: By default the gradient computation would be disabled in your custom autograd.Function, so you would need to enable it via torch.set_grad_enabled(True) in the backward.


Gradient of `maximum` and `minimum` functions

Question:
Hi,
This is regarding the behavior of torch.maximum and torch.minimum functions.
Here is an example:
Let a be and scalar.
Currently when computing torch.maximum(x, a), if x > a then the gradient is 1, and if x < a then the gradient is 0. BUT if x = a then the gradient is 0.5.
The same is true for torch.minimum.
Are the mathematical reasons for the 0.5 gradient when x = a? or is it for numerical stability issues?
Thanks,

Context:
I think this is expected behavior as described here 3 and implemented here 3.
CC @albanD to correct me


Making Fibonacci sphere number of points differentiable

Question:
Hey,
I am trying to generate kind of procedural displacement maps using Fibonacci sphere coordinates.
Basically the following code is generating me 2d coordinates I am using later own to create my texture.

seed_nb = torch.tensor(600,device = pyredner.get_device(), requires_grad=True)
goldenRatio = torch.tensor((1 + 5**0.5)/2,device = pyredner.get_device())
i = torch.arange(0, seed_nb.long(),device = pyredner.get_device())
theta      = 2 * torch_PI * i / goldenRatio
fibo_phi   = torch.acos(torch.ones(1,device = pyredner.get_device()) - torch.tensor(2*(i+0.5)/n,device = pyredner.get_device()))
fibo_theta = (theta%(2torch_PI))-torch_PI
# print(phi.min(),phi.max(),theta.min(),theta.max())
indices_x = (fibo_phi1800/np.pi).long()
indices_y = ( ((fibo_theta*1800/np.pi)+1800)).long()

The problem is that I want the variable “seed_nb” to be differentiable/learned to be the right value when back-propagating. So I used a long tensor with one value inside. But because or “torch.arange” it sees that the gradient of this tensor is always None, hence nothing is learned.
I am a bit stuck trying to find a way to go around this arange somehow.
Any idea ?
Justin

Context:
Hi Justin!

 lelouedec:

seed_nb = torch.tensor(600,device = pyredner.get_device(), requires_grad=True)

The problem is that I want the variable “seed_nb” to be differentiable/learned to be the right value when back-propagating.

There is no straightforward way to do what you want.  The problem is
that the number of points on your sphere is an integer and there is no
sensible way to differentiate with respect to an integer.
Helpfully, pytorch won’t let you do this:
>>> import torch
>>> torch.__version__
'1.9.0'
>>> torch.tensor (600, device = 'cpu', requires_grad = True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Only Tensors of floating point and complex dtype can require gradients

Best.
K. Frank


Zero’ing one input’s gradient for Matrix Multiply

Question:
Hi! the @ operator, or matrix multiply, is stateless and accepts 2 input tensors. During the backprop, my understanding is that it’ll calculate two gradients w.r.t. the 2 input tensors, which will each update any variables via the chain rule along the paths that produce them, respectively.
In my experiment, I want to zero out the gradient for only one input tensor and keep the other as-is. My guess is that I need to use autograd, but I’m fairly new to this and the toy example (PyTorch: Defining New autograd Functions — PyTorch Tutorials 1.7.0 documentation) doesn’t seem suffice. Please help!

Context:
Hey, I figured out how to do it with autograd.Function after reading through the pages. Let me know if it makes sense.
class TwoOpGradientController(torch.autograd.Function):
    @staticmethod
    def forward(ctx, *args):
        ctx.args_0_shape = args[0].shape
        return args[0] @ args[1]

    @staticmethod
    def backward(ctx, grad_output):
        lhs_zero_grad_input = torch.zeros_like(ctx.args_0_shape)
        rhs_grad_output = # do weird stuff to rhs gradients

        return lhs_zero_grad_input, rhs_grad_output


PyTorch optimizer.step() function doesn’t update weights

Question:
I am new to Pytorch and I tried to use SGD to perform a fitting using a statistical model .
But the problem is that the optimizer.step() part does not work. I output optimizer parameters() after each epoch, and the weights do not change.
here is my code:
shape_parameters_estimated = torch.zeros([batch, stat8model.variance.size], dtype=torch.float32, device=device, requires_grad=True)
learning_rate = 0.1
optimizer = torch.optim.Adagrad([shape_parameters_estimated], lr=learning_rate)
max_iter = 20
for iteration in range(max_iter):
theta_estimated = shape_parameters_estimated
fit_image, fit_landmarks = stat_model(theta_estimated)

loss_lm = lm_loss(target_landmark, fit_landmarks)
print('theta=', theta_estimated[0]) #I'am printing the parameter here, but doesn't change
loss_value = loss_lm
optimizer.zero_grad()
loss_value.backward()
optimizer.step()

Please, can anyone help?

Context:
Your code works fine after replacing the undefined classes:
shape_parameters_estimated = torch.zeros(1, 1, dtype=torch.float32, requires_grad=True)
stat_model = nn.Linear(1, 1)

learning_rate = 0.1
optimizer = torch.optim.Adagrad([shape_parameters_estimated], lr=learning_rate)
max_iter = 20
criterion = nn.MSELoss()

for iteration in range(max_iter):
    theta_estimated = shape_parameters_estimated
    out = stat_model(theta_estimated)

    loss = criterion(out, torch.rand_like(out))
    print('theta=', theta_estimated[0])
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Output:
theta= tensor([0.], grad_fn=<SelectBackward0>)
theta= tensor([0.1000], grad_fn=<SelectBackward0>)
theta= tensor([0.1397], grad_fn=<SelectBackward0>)
theta= tensor([0.1831], grad_fn=<SelectBackward0>)
theta= tensor([0.2331], grad_fn=<SelectBackward0>)
theta= tensor([0.2467], grad_fn=<SelectBackward0>)
theta= tensor([0.2960], grad_fn=<SelectBackward0>)
theta= tensor([0.3146], grad_fn=<SelectBackward0>)
theta= tensor([0.3339], grad_fn=<SelectBackward0>)
theta= tensor([0.3832], grad_fn=<SelectBackward0>)
theta= tensor([0.4118], grad_fn=<SelectBackward0>)
theta= tensor([0.4494], grad_fn=<SelectBackward0>)
theta= tensor([0.4717], grad_fn=<SelectBackward0>)
theta= tensor([0.4924], grad_fn=<SelectBackward0>)
theta= tensor([0.5206], grad_fn=<SelectBackward0>)
theta= tensor([0.5551], grad_fn=<SelectBackward0>)
theta= tensor([0.5686], grad_fn=<SelectBackward0>)
theta= tensor([0.5911], grad_fn=<SelectBackward0>)
theta= tensor([0.6077], grad_fn=<SelectBackward0>)
theta= tensor([0.6128], grad_fn=<SelectBackward0>)

Maybe you are detaching the computation graph in one of your modules.


Question: Topics related to Natural Language Processing

Context: Aren’t you going to edit this at least? its been 3 years now man!


Converting tensors into numpy array and vise versa

Question:
i’m working on distilbert for text classification, and want to save the tensors into a data frame so i can use it again as feature generator
to illustrate my question more:
below are token IDs for one sentence in a row in csv file i’m working on :
Token IDs: tensor([  101, 11113,  2080, 25876,  2542, 15009, 11290, 22291,  3370,  2478,
7037, 22160,  3215,  1998,  5760,  5001, 10464, 24895, 15009,  2157,
2002, 17585,  6593, 16940,  1037,  2553,  3189,  1026,  1013,  1055,
1028,  1026,  1013,  1055,  1028, 11581,  2063,  2138,  1997,  1996,
15843,  1997, 10181, 17843,  2542, 15009, 11290, 22291,  3370, 25510,
7096,  2038,  2468,  1996,  2364,  7709,  2000,  7438,  5022,  2007,
2203,  1011,  2754,   102])
1- i want to replace the text in the csv file into their tokens by converting these tensors into np.array and save this CSV file
2- then i need to know how to convert this column back from np.array to tensors (input ids and attention masks) so i can use them for fine tuning tasks

Context:
1 To convert a tensor to a numpy array use a = tensor.numpy(), replace the values, and store it via e.g. np.save.
2. To convert a numpy array to a tensor use tensor = torch.from_numpy(a).


Beginner: Should ReLU/sigmoid be called in the __init__ method?

Question:
I am trying to rebuild a Keras architecture in pytorch, which looks like this
    rnn_layer1 = GRU(25) (emb_seq_title_description)
    # [...]
    main_l = Dropout(0.1)(Dense(512,activation='relu') (main_l))
    main_l = Dropout(0.1)(Dense(64,activation='relu') (main_l))

    #output
    output = Dense(1,activation="sigmoid") (main_l)

So I tried to adjust the basic RNN example in pytorch and add ReLUs to the Linear layers.  However, I am not sure if I can call ReLU directly in the forward method or should call it in the init method.
My first try looks like this:
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = nn.ReLU(self.i2h(combined))
        output = nn.ReLU(self.i2o(combined))
        output = nn.sigmoid(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

n_hidden = 128
n_letters = 26
n_categories = 2
rnn = RNN(n_letters, n_hidden, n_categories)

However, when I look at the rnn object in python, I will not see the ReLUs, so maybe it’s not right to call nn.ReLU directly in the forward method…
RNN(
  (i2h): Linear(in_features=154, out_features=128, bias=True)
  (i2o): Linear(in_features=154, out_features=2, bias=True)
)

Context:
Since nn.ReLU is a class, you have to instantiate it first. This can be done in the __init__ method or if you would like in the forward as:
hidden = nn.ReLU()(self.i2h(combined))

However, I would create an instance in __init__ and just call it in the forward method.
Alternatively, you don’t have to create an instance, because it’s stateless, and could directly use the functional API in forward:
hidden = F.relu(...)


Question:
I don’t understand how nn.MultiheadAttention module work.

What is the meaning of k_dim and v_dim respectively in __init__? Are they related to key and value parameters in forward()?

Why must the embedding_dim be divisible by num_heads? What is the meaning of head_dim derived from such a fomulation?
I thought multi-head self-attention works this way:

embedding_dim are projected into query_dim, key_dim, value_dim respectively
after computing attention weights between query and key, the value_dim are collected according to the attention weights of each query, which forms exactly the output of current attention head.
Finally output of all heads are concatenated into one matrix of ~*head_num*value_dim and projected into embedding_dim. Thus, I don’t see where should the module divide embedding_dim by num_heads.

What is the difference between torchtext.nn.MultiheadAttention and torch.nn.MultiheadAttention?

Context:
UP! I am very confused by this as well! Can anyone clarify?
Also: how can kdim be different from embed_dim (which is hardcoded to be qdim, which also makes no sense / is not very general)? In order to take the dot product between queries and keys, they have to have the same length! The only one between q,k,v that should be able to have a different length is v.
And vdim should also determine the size of the output, but then out_proj (the Matrix to projekt that output to the final output of the layer) is set up with the shape (embed_dim, embed_dim), when it should be (vdim, embed_dim), or better (vdim, out_dim) to be more general.
It seems that this layer can only work when embed_dim=kdim=vdim, which is the case any way when you do not specify kdim and vdim, so whats the point?
I have not yet tried to play around with it to see if it really doesn’t work when kdim or vdim are different from embed_dim, but just from looking at the code and the doc, this seems all a bit messy and not very well documented at all!


If sample has more than one feature,how can I use Lstm to solve problems?

Question: After I read the Pytorch doc,I tried to solve a problem which is about price forcasting.But the doc only show creating a model with the sample that has only one feature.My sample has more than one feature,So how do I use Lstm to create model .I will appreciate if you can show me with detail codes.

Context:
You an simply represent all your features for a data sample as a single vector.
How do you features look like? Do you already have a working model with one feature?


Pack_padded_sequence with inconsistent (repeated) padding

Question:
Following the example from the docs I am trying to solve a problem where the padding is inconsistent rather than at the end of the tensor for each batch (in other words, no pun intended, I have a left-censored and right-censored problem across my batches):
 # Data structure example from docs
seq = torch.tensor([[1,2,0], [3,0,0], [4,5,6]])
 # Data structure of my problem
inconsistent_seq = torch.tensor([[1,2,0], [0,3,0], [0,5,6]])

lens = ...?
packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)

How can I solve the problem of masking these padded 0’s when running them through an LSTM using (preferably) PyTorch functionality?

Context: If I have understood your problem right, you mean that some data points are padded at the left while others are at the right. Usually the padded 0’s are to make the length of all the data points the same (usually added at the end like above). So if you have any inconsistent paddings(left or right), you might need a separate character instead of 0.


Performances : Embedding layers vs pre-trained embedding

Question:
Hello,
I read quite a lot about the importance of word embedding in the context of NLP, but i’ve never seen the following issue beeing adresed : Are pre-trained embeddings (word2vec, GloVe etc…) performing better or worse than an embedding layer trained along with the model ?
I intuitively would think that an embedding layer trained along with the model should perform better since it’s trained task-specifically by the network during back-propagation (and maybe generalize less easily ?) but didn’t really find much info on this.
I’m aware that there is probably no clear answer to this, and that this is definitely case dependant, but i’d emjoy hearing your feedback on this matter.
Cheers !

Context:
I’m not an expert on NLP, but I would assume pretrained embeddings might be treated similar to pretrained kernels in a CNN.
If you are dealing with data from the same or similar domain (e.g. English texts / “natural” images), the pretrained parameters might just work and probably even better than training from scratch, e.g. if you are working with a small dataset.
That being said, if your data comes from another domain (e.g. source code / medical images), the pretrained layers might not work pretty well and you would need to finetune them or train form scratch.
In the case of NLP and embeddings I would assume this use case to fail pretty badly, since embeddings are used as a lookup table for each word index.
If the words you are dealing with are not in the dictionary, you might get an “unknown token” for a lot of words, which might make the embedding useless.


Recommended way to create a data pipeline with torchtext>=v0.10.0

Question:
I’m pretty new to torchtext and recently I’ve digged through a lot of tutorials for NLP with Torchtext although most of them use APIs that are now moved to torchtext.legacy and current docs on PyTorch use functional methods and don’t have a standard conventional way of implementing a data pipeline like torchvision. What I currently do is extending torch.utils.data.Dataset like below:
# A custom language translation dataset using Multi30k
class Multi30kDe2En(Dataset):
    UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0, 1, 2, 3
    SPECIAL_SYMBOLS = ['<unk>', '<pad>', '<bos>', '<eos>']

    def __init__(self, base_url, urls):
        super(Multi30kDe2En, self).__init__()
        self.base_url = base_url
        self.urls = urls
        # paths to text files containing queries (e.g. train.de, train.en)
        self.paths = [extract_archive(download_from_url(self.base_url + url))[0] for url in self.urls]
        # load tokenizers
        self.de_tokenizer = get_tokenizer('spacy', language='de_core_news_sm')
        self.en_tokenizer = get_tokenizer('spacy', language='en_core_web_sm')
        self.de_vocab, self.en_vocab = self._load_vocabs()
        # list of all queries; a sentence per item
        self.de_texts = list((io.open(self.paths[0], encoding="utf8")))
        self.en_texts = list((io.open(self.paths[1], encoding="utf8")))

    def __len__(self):
        return len(self.en_texts)

    def __getitem__(self, index):
        de_text = self.de_texts[index]
        en_text = self.en_texts[index]
        de_tensor = torch.tensor([self.de_vocab[token] for token in self.de_tokenizer(de_text)], dtype=torch.long)
        en_tensor = torch.tensor([self.en_vocab[token] for token in self.en_tokenizer(en_text)], dtype=torch.long)
        return de_tensor, en_tensor

    def _load_vocabs(self):
        # REALLY LOOKING FOR AN ALTERNATIVE WAY INSTEAD OF AN UGLY ITERATOR
        def yield_tokens(filepath, tokenizer):
            with io.open(filepath, encoding='utf8') as f:
                for text in f:
                    yield tokenizer(text)

        de_vocab = build_vocab_from_iterator(yield_tokens(self.paths[0], self.de_tokenizer),
                                             specials=self.SPECIAL_SYMBOLS)
        en_vocab = build_vocab_from_iterator(yield_tokens(self.paths[1], self.en_tokenizer),
                                             specials=self.SPECIAL_SYMBOLS)
        de_vocab.set_default_index(self.UNK_IDX)
        en_vocab.set_default_index(self.UNK_IDX)

        return de_vocab, en_vocab

    @classmethod
    def collate_fn(cls, batch):
        # Just to concat bos and eos tokens to the sides of the tensor
        de_batch, en_batch = [], []
        for de, en in batch:
            de_batch.append(torch.cat([torch.tensor([cls.BOS_IDX]), de, torch.tensor([cls.EOS_IDX])], dim=0))
            en_batch.append(torch.cat([torch.tensor([cls.BOS_IDX]), en, torch.tensor([cls.EOS_IDX])], dim=0))
        de_batch = pad_sequence(de_batch, padding_value=cls.PAD_IDX).permute(1, 0)
        en_batch = pad_sequence(en_batch, padding_value=cls.PAD_IDX).permute(1, 0)
        return de_batch, en_batch

if __name__ == '__main__':
    from torch.utils.data import DataLoader

    base_url = 'https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/'
    urls = ('train.de.gz', 'train.en.gz')
    dataset = Multi30kDe2En(base_url, urls)
    dataloader = DataLoader(dataset, batch_size=1, collate_fn=Multi30kDe2En.collate_fn)
    de_batch, en_batch = next(iter(dataloader))
    print(f'German Tensor Batch : {de_batch}')
    print(f'English Tensor Batch: {en_batch}')

I’d appreciate any suggestions on other alternative ways. Thanks!

Context:
First of all, this code has a major problem which is, the vocabs are created every time a dataset is instantiated from this class (like train , val) so the validation set will have different set of vocab mappings, hence really bad results on validation set while evaluating. The better way to implement this to have consistent set of vocab is like below:
import torch
from torch.utils.data import Dataset
from torchtext.vocab import build_vocab_from_iterator
from torchtext.datasets import Multi30k
from torch.nn.utils.rnn import pad_sequence
from torchtext.data.utils import get_tokenizer
from typing import Literal

class Multi30kDe2En(Dataset):
    UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0, 1, 2, 3
    SPECIAL_SYMBOLS = ['<unk>', '<pad>', '<bos>', '<eos>']

    def __init__(self, split: Literal['train', 'valid']):
        super(Multi30kDe2En, self).__init__()
        self.split = split
        self.iter = Multi30k(split=split, language_pair=('de', 'en'))
        self.de_texts, self.en_texts = list(zip(*self.iter))
        self.de_tokenizer = get_tokenizer('spacy', language='de_core_news_sm')
        self.en_tokenizer = get_tokenizer('spacy', language='en_core_web_sm')
        self.de_vocab, self.en_vocab = self._load_vocabs()

    def __len__(self):
        return len(self.en_texts)

    def __getitem__(self, index):
        de_text = self.de_texts[index]
        en_text = self.en_texts[index]
        de_tensor = torch.tensor([self.de_vocab[token] for token in self.de_tokenizer(de_text)], dtype=torch.long)
        en_tensor = torch.tensor([self.en_vocab[token] for token in self.en_tokenizer(en_text)], dtype=torch.long)
        return de_tensor, en_tensor

    def _load_vocabs(self):
        data_iter = Multi30k(split='train', language_pair=('de', 'en'))  # vocabs must be extracted from train split
        de_texts, en_texts = list(zip(*data_iter))
        de_tokens = [self.de_tokenizer(text.rstrip('\n')) for text in de_texts]
        en_tokens = [self.en_tokenizer(text.rstrip('\n')) for text in en_texts]
        de_vocab = build_vocab_from_iterator(iter(de_tokens), specials=self.SPECIAL_SYMBOLS)
        en_vocab = build_vocab_from_iterator(iter(en_tokens), specials=self.SPECIAL_SYMBOLS)
        de_vocab.set_default_index(self.UNK_IDX)
        en_vocab.set_default_index(self.UNK_IDX)

        return de_vocab, en_vocab

    @classmethod
    def collate_fn(cls, batch):
        de_batch, en_batch = [], []
        for de, en in batch:
            de_batch.append(torch.cat([torch.tensor([cls.BOS_IDX]), de, torch.tensor([cls.EOS_IDX])], dim=0))
            en_batch.append(torch.cat([torch.tensor([cls.BOS_IDX]), en, torch.tensor([cls.EOS_IDX])], dim=0))
        de_batch = pad_sequence(de_batch, padding_value=cls.PAD_IDX).permute(1, 0)
        en_batch = pad_sequence(en_batch, padding_value=cls.PAD_IDX).permute(1, 0)
        return de_batch, en_batch

if __name__ == '__main__':
    from torch.utils.data import DataLoader

    dataset = Multi30kDe2En('train')
    dataloader = DataLoader(dataset, batch_size=16, collate_fn=Multi30kDe2En.collate_fn)
    de, en = next(iter(dataloader))
    print('done')


Multi Headed Attention: Query & Value Transformations acquire extremely low gradients

Question:
Hi,
I am trying to execute a version of multi headed attention on input batches of sequence length 10. Below is a simplified version of my code:
type or paste code here
class MultiHeadAttention(nn.Module):
    # Multi-Head Attention module

    def __init__(self, n_head, d_model, embed_dim_per_head, dropout=0.1):
        super().__init__()

        total_embed_dim = embed_dim_per_head * n_head
        self.w_qs =  nn.Linear(d_model, total_embed_dim, bias=False)
        self.w_ks =  nn.Linear(d_model, total_embed_dim, bias=False)
        self.w_vs =  nn.Linear(d_model, total_embed_dim, bias=False)
        self.fc = nn.Linear(total_embed_dim, d_model)

        self.attention = nn.MultiheadAttention(
                embed_dim = total_embed_dim,
                num_heads = n_head,
                dropout = 0.1)

        self.dropout = nn.Dropout(dropout)
        self.layer_norm = nn.LayerNorm(d_model, eps=1e-6)

    def forward(self, q, k, v):

        residual = q

        # NB, SEQ_LEN, (N_HEAD * D) -> SEQ_LEN, NB, (N_HEAD * D)
        q = self.w_qs(q).permute(1, 0, 2)
        k = self.w_ks(k).permute(1, 0 ,2)
        v = self.w_vs(v).permute(1, 0, 2)
        q, att_map = self.attention(q, k, v, need_weights=True)
        print(f'AttMap Max Values: {att_map[0].max(dim=-1)[0]}')
        # SEQ_LEN, NB, (N_HEAD * D) -> NB, SEQ_LEN, (N_HEAD * D)
        q = q.permute(1, 0, 2)
        q = self.dropout(self.fc(q))
        q += residual
        q = self.layer_norm(q)
        return q

class IntraModal_MHA(nn.Module):
    def __init__(self, num_blocks, num_heads, transform_dim, embed_dim_per_head, activation):
        super(IntraModal_MHA, self).__init__()
        self.activation = activation
        self.num_blocks = num_blocks
        self.mha_list = nn.ModuleList([
             MultiHeadAttention(num_heads, transform_dim, embed_dim_per_head)
             for i in range(num_blocks)])
        self.linear_list = nn.ModuleList([
            nn.Sequential(
                nn.Linear(transform_dim, 2 * transform_dim),
                self.activation,
                nn.Dropout(0.1),
                nn.Linear(2 * transform_dim, transform_dim),
                self.activation,
                nn.Dropout(0.1),
                )
            for i in range(num_blocks)])
        self.layer_norms = nn.ModuleList([
            nn.LayerNorm(transform_dim, eps=1e-6)
            for i in range(num_blocks)])

    def forward(self, modal_ft):
        for i in range(self.num_blocks):
            output = self.mha_list[i](modal_ft, modal_ft, modal_ft)
            residual = output
            output = self.linear_list[i](output)
            output += residual
            modal_ft = self.layer_norms[i](output)
        return modal_ft

class A2H(nn.Module):
    def __init__(self):
        super(A2H, self).__init__()
        self.activation = nn.ReLU(inplace=True)

        self.audio_transform = nn.Sequential(
                nn.Linear(128, 512),
                self.activation,
                nn.Dropout(0.1),
                )

        self.a2a_temporal_att = IntraModal_MHA(1, 1, 512, 512, self.activation)
        self.classifier_fc = nn.Linear(512, 29)

    @staticmethod
    def positional_encoding(n_position, emb_dim):
            #The sinusoid position encoding table

            position_enc = np.array([
            [pos / np.power(10000, 2 * (j // 2) / emb_dim) for j in range(emb_dim)]
            for pos in range(n_position)])
            position_enc[:, 0::2] = np.sin(position_enc[:, 0::2]) # dim 2i
            position_enc[:, 1::2] = np.cos(position_enc[:, 1::2]) # dim 2i+1
            return torch.from_numpy(position_enc).type(torch.FloatTensor)

    def forward(self, audio_ft, unused_):
        nb, seq_len, aud_ft_dim = audio_ft.shape
        audio_ft_pe = audio_ft + self.positional_encoding(seq_len, aud_ft_dim).cuda()
        audio_ft_pe = self.audio_transform(audio_ft_pe)
        a2a_temporal_out = self.a2a_temporal_att(audio_ft_pe)
        signal_out = self.classifier_fc(a2a_temporal_out)
        classifier_out = F.softmax(signal_out, dim=-1)
        return classifier_out

My optimizer is the typical SGD with momentum setting with a high LR of 0.2.
Below are the outputs for my prints which display

The gradients to the nn.Linear which transforms my input into Query and Key matrices
The maximum value of the 1st batch’s attention returned by nn.MultiheadAttention:

After Update:
a2a_temporal_att.mha_list.0.w_qs.weight : 3.750015922787675e-12
a2a_temporal_att.mha_list.0.w_ks.weight : 3.718521150719578e-12
AttMap Max Values: tensor([0.1128, 0.1126, 0.1123, 0.1128, 0.1134, 0.1130, 0.1128, 0.1122, 0.1121,
0.1125], device=‘cuda:0’, grad_fn=)
After Update:
a2a_temporal_att.mha_list.0.w_qs.weight : 4.729227426336635e-12
a2a_temporal_att.mha_list.0.w_ks.weight : 4.986864220180021e-12
AttMap Max Values: tensor([0.1131, 0.1140, 0.1128, 0.1137, 0.1142, 0.1136, 0.1132, 0.1129, 0.1123,
0.1122], device=‘cuda:0’, grad_fn=)
After Update:
a2a_temporal_att.mha_list.0.w_qs.weight : 2.1090891186292815e-11
a2a_temporal_att.mha_list.0.w_ks.weight : 1.926272509555904e-11
AttMap Max Values: tensor([0.1123, 0.1121, 0.1126, 0.1123, 0.1123, 0.1122, 0.1118, 0.1124, 0.1122,
0.1123], device=‘cuda:0’, grad_fn=)
After Update:
a2a_temporal_att.mha_list.0.w_qs.weight : 1.62918636914533e-11
a2a_temporal_att.mha_list.0.w_ks.weight : 1.50868172121843e-11
As you can see my implementation of IntraModal_MHA contains residual connections to avoid the vanishing gradient problem and resembles to that of the transformer structure.
Below is the gradient accumulation for the entire network in the 6th epoch during which my accuracy increased from 28% to 33%:

The gradients to both the w_qs and w_ks are always super low and the attention map is not converging into a sharp distribution. The input sequences contain foreground and background type samples and the objective is to classify the foreground samples. From the above objective, the attention map is expected to be sharp. Additionally, as I increase the complexity of the network by increasing the number of blocks in the IntraModal_MHA, the gradients become even more smaller.
Has anyone faced this or can someone please point out any obvious mistake I might be doing?
Thanks a lot!

Context:
For future readers: may want to check out this article: Tutorial #17: Transformers III: Training.
TL;DR: Try using the adam optimizer.

Transformers also differ from convolutional networks in that stochastic gradient descent does not work well for training (figure 2) and adaptive optimizers like Adam are required. Liu et al., 2020 observed that differentiating through the self-attention mechanism creates unbalanced gradients. In particular, the gradients for the query Φq and key Φk parameters were much smaller than those for the value parameters Φv , and so the former parameters change much more slowly. This is a direct consequence of the mathematical expression for self-attention. The Adam optimizer fixes this problem by essentially having different learning rates for each parameter. To conclude, we’ve seen that residual connections are needed to allow us to train deep networks. These cause gradient explosion, which is resolved by using layer normalization. The self-attention computation causes unbalanced gradients, which necessitates the use of Adam (figure 4). In the next section, we’ll see that layer normalization and Adam themselves cause more problems, which ultimately result in the need for learning rate warm-up.


Loss Decreasing but Accuracy Remaining the Sane

Question:
I am trying to predict opioid prescription overdoses using a dataset I found on kaggle.

      kaggle.com

U.S. Opiate Prescriptions/Overdoses

  Can you save lives through predictive modeling?

After cleaning up my data, I am left with 241 features and a total of 19807 rows in my training set. I have loaded my train_data and test_data in to dataloaders and have passed them to my RNN. My Loss is decreasing but my accuracy is remaining the exact same. What am I doing wrong?
Loading data to dataloaders
## train data
class TrainData(Dataset):

    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data

    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]

    def __len__ (self):
        return len(self.X_data)

train_data = TrainData(torch.FloatTensor(X_train),
                       torch.FloatTensor(y_train))
## test data
class TestData(Dataset):

    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data

    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]

    def __len__ (self):
        return len(self.X_data)

test_data = TestData(torch.FloatTensor(X_test),
                     torch.FloatTensor(y_test))

train_loader = DataLoader(dataset=train_data, batch_size=100, shuffle=True)

test_loader = DataLoader(dataset=test_data, batch_size=1)

RNN Architecture
class RNN(nn.Module):
  def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
    super(RNN, self).__init__()

    # Defining parameters
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim

    # Defining Layers
    # RNN Layer
    self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim,
                      batch_first=True, nonlinearity='relu')

    # Fully connected Layer
    self.fc = nn.Linear(hidden_dim, output_dim)

  def forward(self, x):
    batch_size = x.size(0)

    # initializing hidden state for first input using method defined below
    hidden = self.init_hidden(batch_size).requires_grad_()

    # Passing in input and hidden state into model and getting output
    out, hidden = self.rnn(x, hidden.detach())

    # Reshaping outputs such that it can be fit into fully connected layer
    out = out.contiguous().view(-1, self.hidden_dim)
    out = self.fc(out)

    return out

  def init_hidden(self, batch_size):
    # generate the first hidden state of 0s that will be propogated in forward pass
    # send tensor holding the hidden state to device specified earlier
    hidden = torch.zeros(self.layer_dim, batch_size, self.hidden_dim)
    return hidden

input_dim = 241

hidden_dim = 100

layer_dim = 2

output_dim = 1

model = RNN(input_dim=input_dim,
            hidden_dim=hidden_dim,
            layer_dim=layer_dim,
            output_dim=output_dim)

print(model)
# model.to(device)

n_epochs = 100
lr = 0.01

# Define Loss, Optimizer
criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

My training function:
iter = 0
for epoch in range(num_epochs):
    for i, (data, labels) in enumerate(train_loader):
        model.train()
        # Load images as tensors with gradient accumulation abilities
        # images = images.view(-1, seq_dim, input_dim).requires_grad_()
        # print(images.shape)
        # print(i)
        if i != 198:
          data = data.view(100, 1, 241).requires_grad_()
        else:
          data = data.view(7, 1, 241).requires_grad_()

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        # print(model(images))
        outputs = model(data)
        # print(outputs.shape)
        outputs = torch.sigmoid(outputs)

        labels = labels.unsqueeze(1)
        # print(labels)
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            model.eval()
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for data, labels in test_loader:
                # Resize images
                data = data.view(1, 1, 241)

                # Forward pass only to get logits/output
                outputs = model(data)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                # Total correct predictions
                correct += (predicted == labels).sum()
            print(correct)

            accuracy = 100 * correct / total
            print(accuracy)

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.item(), accuracy))

Output:
Iteration: 500. Loss: 0.20433291792869568. Accuracy: 41.11470031738281
Iteration: 1000. Loss: 0.2545611560344696. Accuracy: 41.11470031738281
Iteration: 1500. Loss: 0.2528284788131714. Accuracy: 41.11470031738281
Iteration: 2000. Loss: 0.3239484131336212. Accuracy: 41.11470031738281
Iteration: 2500. Loss: 0.3239382803440094. Accuracy: 41.11470031738281
Iteration: 3000. Loss: 0.2597212493419647. Accuracy: 41.11470031738281
Iteration: 3500. Loss: 0.4951004087924957. Accuracy: 41.11470031738281
Iteration: 4000. Loss: 0.3577985465526581. Accuracy: 41.11470031738281
Iteration: 4500. Loss: 0.3011106252670288. Accuracy: 41.11470031738281
Iteration: 5000. Loss: 0.3104074001312256. Accuracy: 41.11470031738281
Iteration: 5500. Loss: 0.2941060960292816. Accuracy: 41.11470031738281
Iteration: 6000. Loss: 0.21687862277030945. Accuracy: 41.11470031738281
Iteration: 6500. Loss: 0.36724144220352173. Accuracy: 41.11470031738281
Iteration: 7000. Loss: 0.16310080885887146. Accuracy: 41.11470031738281
Iteration: 7500. Loss: 0.2854510545730591. Accuracy: 41.11470031738281
Iteration: 8000. Loss: 0.36213111877441406. Accuracy: 41.11470031738281
Iteration: 8500. Loss: 0.44782283902168274. Accuracy: 41.11470031738281
Iteration: 9000. Loss: 0.2636875808238983. Accuracy: 41.11470031738281
Iteration: 9500. Loss: 0.2781040668487549. Accuracy: 41.11470031738281

Please let me know where I am going wrong, thank you!

Context: I am not sure why the accuracy is the same, but do not forget to do model.eval() before testing and  model.train() after testing.


How can I change self attention layer numbers and multihead attention head numbers in my model with Pytorch?

Question:
I working on sarcasm dataset and my model like below:
I first tokenize my input text:

PRETRAINED_MODEL_NAME = “roberta-base”
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)
import torch
from torch.utils.data import Dataset, DataLoader
MAX_LEN = 100

then I defined class for my dataset:
class SentimentDataset (Dataset):
    def __init__(self,dataframe):
        self.dataframe = dataframe

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, idx):
        df = self.dataframe.iloc[idx]

        text = [df["comment"]]
        label = [df["label"]]

        data_t = tokenizer(text,max_length = MAX_LEN, return_tensors="pt", padding="max_length", truncation=True)
        label_t = torch.LongTensor(label)

        return {
             "input_ids":data_t["input_ids"].squeeze().to(device),
             "label": label_t.squeeze().to(device),
        }

then I create obj from my class for training set and set other parameters:
train_dataset = SentimentDataset(train_df)
BATCH_SIZE = 32
train_dataloader = DataLoader(train_dataset, batch_size = BATCH_SIZE)
from transformers import AutoModelForSequenceClassification, AutoConfig

# For loading model stucture and pretrained weights:
model = AutoModelForSequenceClassification.from_pretrained(PRETRAINED_MODEL_NAME).to(device)

import transformers

optimizer = torch.optim.Adam(model.parameters(), lr=2e-5, weight_decay=1e-5)

Then I use dataloader for training my data:
train_dataloader = DataLoader(train_dataset, batch_size = BATCH_SIZE)
EPOCHS = 5
for epoch in range(EPOCHS):
    print("\n******************\n epoch=",epoch)
    i = 0
    logits_list = []
    labels_list = []
    for batch in train_dataloader:
        i += 1
        optimizer.zero_grad()
        output_model = model(input_ids = batch["input_ids"], labels = batch["label"])
        loss = output_model.loss
        logits = output_model.logits
        logits_list.append(logits.cpu().detach().numpy())
        labels_list.append(batch["label"].cpu().detach().numpy())
        loss.backward()
        optimizer.step()
    #scheduler.step()
        if i % 50 ==0 :
            print("training loss:",loss.item())
            #print("validation loss:",loss.item())
    logits_list = np.concatenate(logits_list, axis=0)
    labels_list = np.concatenate(labels_list, axis=0)
    logits_list = np.argmax(logits_list, axis =1)
    print(classification_report(labels_list, logits_list))

My question is how can I change self attention layers number and head of multihead attention in my model?

Context:
I do not think you can change the number of self attention layers in the network from the transformers library. Think about it. If you even manage to change the self attention layers number, the  dimension of data that goes into the subsequent layers will change, which will cause a problem with dimensions.
Moreover, the size of the  pre-trained weighs of the “roberta-base” is fixed, changing the any parameter(number of heads and self attention layers number), will make it not possible to load the pre-trained onto the  “roberta-base” architecture.
You can take a look  Transformer Multihead Attention


Question:
I’m trying to reproduce the codet5 fine-tuning results (GitHub - salesforce/CodeT5: Code for CodeT5: a new code-aware pre-trained encoder-decoder model.)
The script being used is:
python3 /home/ubuntu/CodeT5/run_gen.py \
--task summarize \
--sub_task python \
--summary_dir /home/ubuntu/CodeT5/summary \
--cache_path /home/ubuntu/CodeT5/cache \
--data_dir /home/ubuntu/CodeT5/data \
--res_dir /home/ubuntu/CodeT5/res \
--output_dir /home/ubuntu/CodeT5/output \
--save_last_checkpoints \
--always_save_model \
--do_eval_bleu \
--model_name_or_path='Salesforce/codet5-base-multi-sum' \
--tokenizer_name='Salesforce/codet5-base-multi-sum' \
--train_filename /home/ubuntu/CodeT5/data/summarize/python/train.jsonl \
--dev_filename /home/ubuntu/CodeT5/data/summarize/python/valid.jsonl \
--test_filename /home/ubuntu/CodeT5/data/summarize/python/test.jsonl \
--do_train \
--do_eval \
--do_test \
--save_steps=500 \
--log_steps=100 \
--local_rank=-1

Running it leads to the following error:
Traceback (most recent call last):
  File "/home/ubuntu/CodeT5/run_gen.py", line 387, in <module>
    main()
  File "/home/ubuntu/CodeT5/run_gen.py", line 234, in main
    outputs = model(input_ids=source_ids, attention_mask=source_mask,
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1561, in forward
    encoder_outputs = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 998, in forward
    layer_outputs = layer_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward
    self_attention_outputs = self.layer[0](
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
    attention_output = self.SelfAttention(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 483, in forward
    scores = torch.matmul(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

My guess is something fishy is going on with the source_ids, but I haven’t been able to figure it out.
UPD: a simple test shows that source_ids has shape torch.Size([8, 64]), while target_ids have torch.Size([8, 64]).

Context:
Is the sript working fine on the CPU using the same setup (i.e. input shapes etc.)?
If so, rerun the code with CUDA_LAUNCH_BLOCKING=1 python script.py args and check the stacktrace again, as it should point to the operation which is failing.


Model Predictions gets slower after each batch

Question:
I have trained a neural network model which would generate the input data’s embeddings. Training is done easily.
Now, I wanted to generate the embeddings of the training dataset from the model. Currently, how I’ve implemented is that there is a dataloader for training dataset which is passed to the model to generate the embeddings. Once, model generates the embeddings, it is stored in a numpy array.
model.eval()
FX      = np.zeros((1,300)) # embedding size is 300
print('\nGenerating Embeddings..')

    with torch.no_grad():

        for bi, data in tqdm(enumerate(dataloader), total=len(dataloader)):

            ids            = data["ids"]
            mask           = data["mask"]
            token_type_ids = data["token_type_ids"]
            target         = data["targets"]        # This is 1-hot label row
            target         = target.squeeze(1)

            ids            = ids.to(config.device, dtype=torch.long)
            mask           = mask.to(config.device, dtype=torch.long)
            token_type_ids = token_type_ids.to(config.device, dtype=torch.long)

            output         = model(ids, mask, token_type_ids)

            FX             = np.append(FX, output.detach().cpu().numpy(), 0)

Problem I’m facing is as epoch moves ahead, it gets slower solely due to storing the model output in FX. I’m also detaching the model output from the computation graph and then converting it into numpy array but still GPU memory keeps rising and epoch gets slower after batch by batch.
I have tried commenting out FX             = np.append(FX, output.detach().cpu().numpy(), 0) and I found epoch doesn’t get slowed down. It worked perfectly.
How can I store the embeddings from the model such that the epoch doesn’t get slower?

Context:
@theGuyWithBlackTie This looks more of a numpy problem rather than a pytorch problem
e.g.
import time
X = None
batch_size=1024
embedding_size=5000
for i in range(100):
    start_time = time.time()
    if X is None:
        X = np.random.rand(batch_size, embedding_size)
    else:
        X = np.append(X, np.random.rand(batch_size, embedding_size), 0)
    if i%10==0
        print("Shape is {0} at time {1}".format(X.shape, time.time() - start_time))

Shape is (82944, 5000) at time 2.2030396461486816
Shape is (83968, 5000) at time 1.750520944595337
Shape is (84992, 5000) at time 1.9004521369934082
Shape is (86016, 5000) at time 2.32991361618042
Shape is (87040, 5000) at time 2.1326563358306885
Shape is (88064, 5000) at time 2.3492512702941895
Shape is (89088, 5000) at time 2.04399037361145
Shape is (90112, 5000) at time 1.9996109008789062
Shape is (91136, 5000) at time 1.681365728378296
Shape is (92160, 5000) at time 2.7774124145507812
Shape is (93184, 5000) at time 29.736555099487305
Shape is (94208, 5000) at time 60.32349133491516
Shape is (95232, 5000) at time 93.5995888710022
Shape is (96256, 5000) at time 68.51450300216675
Shape is (97280, 5000) at time 101.32002997398376
Shape is (98304, 5000) at time 26.734136819839478
Shape is (99328, 5000) at time 27.648850679397583
Shape is (100352, 5000) at time 93.45121455192566
Shape is (101376, 5000) at time 27.221306562423706
Shape is (102400, 5000) at time 99.91760969161987

A possible solution can be using a list and then stacking it in one go
import numpy as np
batch_size=1024
embedding_size=5000
%time data = [np.random.rand(batch_size, embedding_size) for x in range(100)]
%time np.stack(data).shape

CPU times: user 3.84 s, sys: 944 ms, total: 4.79 s
Wall time: 7.07 s
CPU times: user 944 ms, sys: 2.66 s, total: 3.6 s
Wall time: 1min 11s

(100, 1024, 5000)


How can I modify the BERT embedding Module in a specific way?

Question:
Hi,
I want to modify the BERT embedding Module (from pytorch-transformers package) in the way that I can dynamically add a tensor ontop of the tokenembedding.  The tensor contains additionally information that can’t be stored in an embedding layer, because it isnt dependend on the tokenposition or the token_id/vocab. I think the picture attached in the appendix describes my interest quite well (in red writing it is clarified how I would like to modify it).

modify.png888×584 17.7 KB

Context: Hi, did u settle this? I also want to add some additional info.


How to force the weights of a Conv layer to be positive?

Question:
Hi,
I tried to use several convolution layers as learnable filters or smoothers to process time series data. However, I found the some of the weights are still negative even after convergence. And this is definitely not a good property for a smoother, so I want to force the all the weights to be positive.
What I am currently doing is to build a weight tensor x with require_grads=True manually, and envelop it with a softplus function. Just curious if there is any more efficient way to handle it? Thank you!
Best,

Context: I don’t think so, all included optimizers work on real domain, so unrestricted <-> positive mapping seems unavoidable.


Question:
I am doing an assignment on Neural Machine Translation and training an LSTM specifically . During Decoding stage I got this Error " AttributeError: ‘tuple’ object has no attribute ‘repeat’ ". I am unable to understand what the error (that I am getting) is, and how I can solve it?
Traceback (most recent call last):
File “train.py”, line 618, in
early_stopping=early_stopping)
File “train.py”, line 186, in train_epochs
decode(model, dataset, epoch)
File “/home/hisrar/NMT/kg_model.py”, line 357, in beam_decode
dec_hidden.repeat(1, batch_size * beam_size, 1)
AttributeError: ‘tuple’ object has no attribute ‘repeat’
Part of the program where I get the Error is
    batch_size, beam_size = self.config.batch_size, self.config.beam_size

    # [1, batch_size x beam_size]

    dec_input = torch.ones(1, batch_size * beam_size,
                           dtype=torch.long,
                           device=self.device) * SOS_ID

   #[num_layers, batch_size * beam_size, hidden_size]

    #dec_hidden = dec_hidden.repeat(1, beam_size,1)

    if enc_outputs is not None:
        enc_outputs = enc_outputs.repeat(1, beam_size, 1)
        enc_length = enc_length.repeat(beam_size)

screenshot955×689 48.8 KB

Context:
Hisrar:

dec_hidden

the attribute repeat is available to Tensor but the variable dec_hidden is of type Tuple. Make sure to convert your tuple to a Tensor and then it’ll work. If it’s a Tuple of 1 object do,
dec_hidden[0].repeat(1, beam_size, 1)


Unable to export character embedder to ONNX with dynamic sequence length

Question:
I have a module, whose purpose is to create word features from characters, by passing characters embeddings through an RNN and taking the last hidden state for each word. I am trying to export the module to ONNX. Here is the forward pass (notice that the RNN is initialized with batch_first=False):
def forward(self, inputs: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
    """"
    Args:
        inputs (torch.Tensor): Character feature tensor of shape [batch_size x seq_len x max_word_len]
        lengths (torch.Tensor): The lengths of each word [batch_size x seq_len]

    Returns:
        torch.Tensor: The last hidden states of all words.
            [batch_size x seq_len x (layers * directions * hidden_size)]
    """
    inputs_emb = self.embedding(inputs)  # [batch_size x seq_len x max_word_len x embedding_size]
    inputs_emb = self.dropout(inputs_emb)  # [batch_size x seq_len x max_word_len x embedding_size]

    inputs_emb_dims = inputs_emb.size()

    inputs_emb_flat = inputs_emb.view(
        -1, inputs_emb_dims[2], inputs_emb_dims[3]
    )  # [(batch_size * seq_len) x max_word_len x embedding_size]
    lengths_flat = lengths.view(-1)  # [(batch_size * seq_len)]

    seq_len_flat = inputs_emb_flat.size()[0]  # [(batch_size * seq_len)]

    # Move seq dimensions to first prior to RNN
    inputs_emb_flat = inputs_emb_flat.permute(1, 0, 2)  # [max_word_len, (batch_size * seq_len), embedding_size]

    lengths_flat_sorted, lengths_flat_idx_sorted = lengths_flat.sort(dim=0, descending=True)
    inputs_emb_flat_sorted = inputs_emb_flat[:, lengths_flat_idx_sorted]

    packed = pack_padded_sequence(
        inputs_emb_flat_sorted, lengths_flat_sorted.to("cpu"), batch_first=False, enforce_sorted=True
    )
    if self.rnn.mode == "GRU":
        _, h_n = self.rnn(packed)  # [(layers * directions) x seq_len_flat x hidden_size]
    elif self.rnn.mode == "LSTM":
        _, (h_n, _) = self.rnn(packed)  # [(layers * directions) x seq_len_flat x hidden_size]
    # Move sequence dimension to the first dimension and then
    # concatenate the forward/backward hidden states.
    h_n = (
        h_n.permute(1, 0, 2).contiguous().view(seq_len_flat, -1)
    )  # [seq_len x (layers * directions * hidden_size)]

    # Unsort the hidden states.
    _, lengths_flat_idx_unsorted_idx = lengths_flat_idx_sorted.sort(dim=0)
    h_n_unsorted = h_n[lengths_flat_idx_unsorted_idx]  # [seq_len x (layers * directions * hidden_size)]

    # Transform back to original input size
    output = h_n_unsorted.view(
        inputs_emb_dims[0], inputs_emb_dims[1], -1
    )  # [batch_size x seq_len x (layers * directions * hidden_size)]

    return output

I am using tracing and exporting my model as such:
torch.onnx.export(
    model,
    input_example,
    filepath,
    verbose=True,
    opset_version=12,
    input_names=["char_seqs", "char_seqs_lens"],
    output_names=["embeddings"],
    dynamic_axes={
        "char_seqs": {1: "seq_len"},
        "char_seqs_lens": {1: "seq_len"},
        "embeddings": {1: "seq_len"},
    },
)

When i export the model, I do not get any errors or warnings. However when i try to pass a sequence which has a different length, than the input example, i get the following error:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Expand node. Name:'Expand_39' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:483 void onnxruntime::BroadcastIterator::Append(int64_t, int64_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 500 by 1951

I do not understand why this is not working, since I believe I am taking all the necessary steps to be able to have a dynamic sequence length.
Edit 1
I believe the reason why this is not working, is because I have both a dynamic sequence length (the length of the word in characters), and the length of the word sequence (which is passed as batch_size x seq_len). Any ideas on how to circumvent this problem, would be most welcome.

Context: The solution was to make the batch size dynamic as well. This required me to input the initial hidden state to the forward pass as well, when tracing with ONNX.


Which is the labels’ order of the output Tensor?

Question:
I was trying to figure out how Pytorch sorts the labels in the output tensor. I’ve understood that with Torchvision the class_to_idx creates the order given how elements are retrieved by the ImageFolder method and this will be the idx order in the output tensor.
On the other hand, as regard NLP I am very confused.
Is it done while defining the network here below in un-explicit way?
class CharRNN(nn.Module):

    def __init__(self, tokens, n_hidden=256, n_layers=2,
                               drop_prob=0.5, lr=0.001):
        super(CharRNN, self).__init__()
        self.drop_prob = drop_prob
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.lr = lr

        # creating character dictionaries
        self.chars = tokens
        self.int2char = dict(enumerate(self.chars))
        self.char2int = {ch: ii for ii, ch in self.int2char.items()}

        ## TODO: define the LSTM
        self.lstm = nn.LSTM(len(self.chars), n_hidden, n_layers,
                            dropout=drop_prob, batch_first=True)

        self.dropout = nn.Dropout(drop_prob)

        self.fc = nn.Linear(n_hidden, len(self.chars)+1000)

    def forward(self, x, hidden):
        ''' Forward pass through the network.
            These inputs are x, and the hidden/cell state `hidden`. '''

        r_output, hidden = self.lstm(x, hidden)

        out = self.dropout(r_output)

        # Stack up LSTM outputs using view
        # you may need to use contiguous to reshape the output
        out = out.contiguous().view(-1, self.n_hidden)

        out = self.fc(out)

        # return the final output and the hidden state
        return out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

        return hidden

Context:
In your code snippet you are creating th character dictionary and mapping via the tokens input so an index will be assigned to each character by iterating it as seen here:
tokens = ['c', 'a', 'z', '0', 'b']

chars = tokens
int2char = dict(enumerate(chars))
char2int = {ch: ii for ii, ch in int2char.items()}
# > {'c': 0, 'a': 1, 'z': 2, '0': 3, 'b': 4}}


Question:
I have a setup something like this:
model = BERT()
model.to(device)
model.eval()
result = []
for data in batch:
input_ids = data.to(device)
token_ids = data.to(device)
attention_mask = data.to(device)
x = model(input_ids, token_ids, attention_mask)
y = x.cpu().detach().numpy()
result+= covert_to_string(y)
I don’t see GPU being used, I checked using nvidia-smi also. The GPU memory is about 75% being used but not the GPU cores for computation. What am I doing wrong?

Context: How are you checking if the GPU is used for computation? To validate its usage you could check the util. in nvidia-smi or check the used CUDA kernels in a profiler. In general, if the model, input and output data are all on the GPU, the computation will also be performed on the GPU as PyTorch won’t move data to the CPU without explicit to() calls.


Compute CrossEntropyLoss per sentence in MLM task

Question:
Hello everyone. I am using a HuggingFace model, to which I pass a couple of sentences. Then I am getting the logits, and using PyTorch’s CrossEntropyLoss, to get the loss. The problem is as follows:
I want the loss of each sentence. If I have 3 sentences, each with 10 tokens, the logits have size [3, 10, V], where V is my vocab size. The labels have size [3, 10], basically the correct labels for each of the 10 tokens in each sentence.
How can I get the cross entropy of each sentence then? If I do reduction='mean', I am going to get the overall mean loss (1 number). If I use reduction='none', then I get one number for each token, so basically the loss of each single token. The code I am using is

loss_fct = nn.CrossEntropyLoss(reduction=‘mean’)  # or ‘none’
masked_lm_loss = loss_fct(outputs.logits.cpu().detach().view(-1, V), target_ids.view(-1))

Perhaps I need to define the views differently in the second line?
Thanks in advance.

Context:
hi

you can use or form,
loss_fct = nn.CrossEntropyLoss(reduction=‘none’)
masked_lm_loss = loss_fct(torch.transpose(outputs.logits.cpu().detach(), 1, 2), target_ids)

and then mean over last dim
masked_lm_loss.mean(-1)

you should have 3 positive losses, one for each sentence.


Question:
(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters. Until you edit this description or create topics, this category won’t appear on the categories page.)
Use the following paragraphs for a longer description, or to establish category guidelines or rules:

Why should people use this category? What is it for?

How exactly is this different than the other categories we already have?

What should topics in this category generally contain?

Do we need this category? Can we merge with another category, or subcategory?

Context:
Hi, smth, this is my (maybe almost) first contribution. I did not intend to create a new category. I saw the word ‘distributed’ with a blue square next to it in a place that I thought would create a new post. I assumed there was an existing category ‘distributed’ to which the post was added. I’m sorry for the misunderstanding.
What do you suggest I do now?


Question:
Hi there,
I am very new to pytorch. Is Stochastic Weight Averaging supported in distributed training (more specifically, the update_bn function).
Thanks!

Context:
I see no reason why it wouldn’t work. Since SWA holds a separate copy of the model on each rank and uses the same all-reduced gradients to update the weight averages, it should theoretically work.
I would suggest giving it a try, and letting us know if you experience any issues.


Sub-path training using DDP may lead incorrect optimizing!

Question:
In sub-path training, such as NAS，there are two path A and B need to be optimized. for example, the optimizing order is B->A:
1.when using a single-gpu, everything works well if we set require_grad=False for unselected path，
2.but when using DistributedDataParallel, we need set find_unused_parameters=True in this condition，and because of  grad sync，for example, when optimizes path A in step2, the parameters‘ grad in path B will be 0 rather then None
Here comes the problem: optimizer chooses to update params which grad is not None. If we set momentum in SGD or something else in other  optimizer, when optimizing path A in step 2, because we have already optimized path B once，so parameters in B have momentum_buffer, although path B’s grad is 0,  it will also be updated by the buffer which is incorrect!
Even we set require_grad=False for path B, but once path B is optimized, its grad will be 0 rather then None.

image1712×786 65.7 KB

Context:
Thanks for posting @VisionAdam Do we know why the grad in path B becomes 0 even if we optimizing path A? I recalled DDP will just don’t touch the gradient field if it’s not being used in forward. Can you share more details why we are observing those gradients be 0?
Also, did you try setting the gradients of Path B explicitly to None and see if that fix your issue?