Autograd's gradcheck outputs RuntimeError regarding Jacobian mismatch

seankala: <p>Hello. I’m currently trying to debug a model after observing that the loss remains constant throughout the entire training process. My first intuition was to check if the gradients are flowing or not, and so after doing some research I stumbled on the <code>torch.autograd.gradcheck</code> function.</p>
<p>I tried to play around with the function to get a feel for how it works with:</p>
<pre><code class="lang-auto">torch.autograd.gradcheck(self.criterion, inputs=(input1, input2, input3))
</code></pre>
<p>which gave me the error:</p>
<pre><code class="lang-auto">*** RuntimeError: Jacobian mismatch for output 0 with respect to input 1,
numerical:tensor([[0.],
        [0.],
        [0.],
        ...,
        [0.],
        [0.],
        [0.]])
analytical:tensor([[-0.0730],
        [ 0.0224],
        [-0.0197],
        ...,
        [ 0.0574],
        [-0.0140],
        [-0.0738]])
</code></pre>
<p>I’m having a bit of trouble interpreting what this error message means. I’m aware from the documentation that the function should output <code>True</code> if the gradient is flowing. I’ve also looked at similar questions on this community as it seemed like a rather common error to run into but haven’t been able to find anything meaningful.</p>
<p>How should I proceed from here? Any tips are appreciated. Thanks in advance.</p>
<h2>Edit</h2>
<p>I’ve managed to make the function work by doing:</p>
<pre><code class="lang-auto">self.criterion = self.criterion.double()
input1 = input1.double()
input2 = input2.double()
input3 = input3.double()
</code></pre>
<p>and passing these into <code>torch.autograd.gradcheck</code> returns <code>True</code>. However, I believe it would still be helpful if anyone would be kind enough to shed some light on what the error message means.</p>
<h2>Edit 2 (Code)</h2>
<p>The code that I’m running looks like the following. The basic framework is that I’m calling a <code>train</code> function, which in turn calls a <code>train_step</code> function to conduct training.</p>
<pre><code class="lang-auto">class Trainer():
    def __init__(self, config, model, data):
        """Simple initialization of attributes for Trainer object."""
        self.config = config
        self.model = model
        self.data = data

        self.load_data() # Loads in the data into respective variables.

    def train(self):
        optimizer = optim.Adam(params=filter(lambda p: p.requires_grad, self.model.parameters()),
                               lr=self.config.learning_rate,
                               weight_decay=self.config.weight_decay)
        criterion = TripletLoss(margin=self.config.margin)

        for epoch in range(self.config.num_epochs):
            self.train_step(self.criterion, self.optimizer)

    def train_step(self, criterion, optimizer):
        self.model.train()

        step = 0
        total_loss = 0.0
        misc_data = [self.drug_info, self.profile_info, self.train_drug_list]
        batches = make_batches(self.config, self.data.train_data, misc_data)

        for batch in batches:
            self.optimizer.zero_grad()

            pos_drugs, pos_profiles, pos_labels, neg_drugs, neg_profiles, neg_labels = batch

            input_pos_drugs = [to_hgnn_data(self.config, x, 1) for x in pos_drugs]
            input_neg_drugs = [to_hgnn_data(self.config, x, 0) for x in neg_drugs]

            input_pos_profiles = torch.cuda.FloatTensor(pos_profiles)
            input_neg_profiles = torch.cuda.FloatTensor(neg_profiles)

            drug_embeddings, profile_embeddings, _ = self.model(input_pos_profiles, input_pos_drugs)
            neg_drug_embeddings, profile_embeddings, _ = self.model(input_neg_profiles, input_neg_drugs)

            loss = self.criterion(profile_embeddings, drug_embeddings, neg_drug_embeddings)

            loss.backward()
            total += loss.item()
            self.optimizer.step()
            step += 1
</code></pre>
<p>This is the basically the code that I’m running. I’ve omitted some details that aren’t directly related to this problem. The functions <code>make_batches</code> and <code>to_hgnn_data</code> are also defined in a separate <code>utils.py</code> module.</p>

ptrblck: <p>If your loss stays constant for the whole training, this might point to a detached tensor during the forward pass.<br>
Could you check the <code>.grad_fn</code> of the model output as well as look for <code>detach()</code> calls or <code>.data</code> usage?<br>
If possible, could you post the code so that we could have a look?</p>

seankala: <p>Hi, thanks for the reply. The model outputs a total of three values, and I’m only using two right now:</p>
<pre><code class="lang-auto">output1, output2, _ = self.model(input_profiles_pos, input_drugs_pos)
neg_output1, output2, _ = self.model(input_profiles_neg, input_drugs_neg)

loss = self.criterion(output2, output1, neg_output1)
</code></pre>
<p>Just for some random background information, this is a bioinformatics project and <code>profiles</code> and <code>drugs</code> refer to genes and chemical compounds. <code>self.criterion</code> is triplet loss.</p>
<p>Each of the outputs’ <code>.grad_fn</code> returns:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; output1.grad_fn
&lt;CopyBackwards object at 0x7f2caeee7278&gt;
&gt;&gt;&gt; neg_output1.grad_fn
&lt;CopyBackwards object at 0x7f2caeee7278&gt;
</code></pre>
<p>The code that I’m using is fairly long, but I’ll edit it into the original question.</p>
<p>I’m also not using any <code>.detach</code> or <code>.data</code> calls in the code.</p>

seankala: <p>I’m not sure if this is relevant or not, but I also checked the <code>.grad</code> and noticed that:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; type(output1.grad)
&lt;class 'NoneType'&gt;
&gt;&gt;&gt; type(neg_output1.grad)
&lt;class 'NoneType'&gt;
</code></pre>
<p>Perhaps this is related to the problem?</p>

ptrblck: <p>Skimming through the code I cannot find any obvious errors, so we would need your model definition to debug it.</p>
<p>No, the <code>.grad</code> attribute won’t be kept for non-leaf tensors by default.<br>
If you want to inspect this gradient, you would have to call <code>output1.retain_grad()</code> before calling <code>backward</code>.</p>