Evaluation but gradients needed

username: Zujing_Liu
cooked: <p>When i evaluated my model in which gradients should be computed in forward(), the error occured</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>The code for the error reporting section is as follows</p>
<pre><code class="lang-auto">def get_geo_outputs(self, x, feature):
        x.requires_grad_(True)
        output = self.forward_geo_layers(x, feature)
        sdf = output[:, :1]
        
        feature_vectors = output[:, 1:]
        d_output = torch.ones_like(sdf, requires_grad=False, device=sdf.device)
        gradients = torch.autograd.grad(
            outputs=sdf,
            inputs=x,
            grad_outputs=d_output,
            create_graph=True,
            retain_graph=True,
            only_inputs=True)[0]

        return sdf, feature_vectors, gradients
</code></pre>
<p>And the call stack saved by vscode is<br>
<a href="https://user-images.githubusercontent.com/68947207/258623237-0b92736a-6c28-4077-a654-8754940e1774.png" rel="noopener nofollow ugc"><img src="https://user-images.githubusercontent.com/68947207/258623237-0b92736a-6c28-4077-a654-8754940e1774.png" alt="image" width="349" height="195"></a><br>
the key functions used for evaluation are</p>
<pre><code class="lang-auto">    @torch.no_grad()
    def get_outputs_for_camera_ray_bundle(self, camera_ray_bundle: RayBundle) -&gt; Dict[str, torch.Tensor]:
        """Takes in camera parameters and computes the output of the model.

        Args:
            camera_ray_bundle: ray bundle to calculate outputs over
        """
        num_rays_per_chunk = 4096   # self.config.eval_num_rays_per_chunk
        image_height, image_width = camera_ray_bundle.origins.shape[:2]
        num_rays = len(camera_ray_bundle)
        outputs_lists = defaultdict(list)
        for i in range(0, num_rays, num_rays_per_chunk):
            start_idx = i
            end_idx = i + num_rays_per_chunk
            ray_bundle = camera_ray_bundle.get_row_major_sliced_ray_bundle(start_idx, end_idx)
            outputs = self.forward(ray_bundle=ray_bundle)
            for output_name, output in outputs.items():  # type: ignore
                if not torch.is_tensor(output):
                    # TODO: handle lists of tensors as well
                    continue
                outputs_lists[output_name].append(output)
        outputs = {}
        for output_name, outputs_list in outputs_lists.items():
            outputs[output_name] = torch.cat(outputs_list).view(image_height, image_width, -1)  # type: ignore
        return outputs
</code></pre>
<p>and</p>
<pre><code class="lang-auto">    def get_eval_image_metrics_and_images(self, step: int):
        """This function gets your evaluation loss dict. It needs to get the data
        from the DataManager and feed it to the model's forward function

        Args:
            step: current iteration step
        """
        self.model.eval()
        image_idx, camera_ray_bundle, batch = self.datamanager.next_eval_image(step)
        outputs = self.model.get_outputs_for_camera_ray_bundle(camera_ray_bundle)
        metrics_dict, images_dict = self.model.get_image_metrics_and_images(outputs, batch)
        assert "image_idx" not in metrics_dict
        metrics_dict["image_idx"] = image_idx
        assert "num_rays" not in metrics_dict
        metrics_dict["num_rays"] = len(camera_ray_bundle)
        self.model.train()
        return metrics_dict, images_dict
</code></pre>
<p>I guess that the error is due to <code>get_eval_image_metrics_and_images()</code> calling <code>self.model.eval()</code> and the <code>get_outputs_for_camera_ray_bundle()</code> with <code>@torch.no_grad()</code> context-manager that disabled gradient calculation.</p>
<p>How can i get gradients for other computation when evaluation?Thanks very much for your help</p>

username: ptrblck
cooked: <p>Remove the <code>@torch.no_grad()</code> decorator and it should work.</p>