Feature extraction has unreasonable memory consumption

username: unofficial-Jona
cooked: <p>Hi all,</p>
<p>I want to extract flow and rgb features from video frames. To do so I want to use the <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html" rel="noopener nofollow ugc">resnet50</a> model for rgb features and the <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.optical_flow.raft_large.html?highlight=raft+large#torchvision.models.optical_flow.raft_large" rel="noopener nofollow ugc">raft_large</a> for flow features. Unfortunately, even extracting features for only a single frame (2 for flow) has my memory demands spike to ~400 GB please see screenshot.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fda450fcda7a1501cd1c4318b344d3929ad00ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/5fda450fcda7a1501cd1c4318b344d3929ad00ee" title="Screenshot 2023-01-05 at 17.49.50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fda450fcda7a1501cd1c4318b344d3929ad00ee_2_690x75.png" alt="Screenshot 2023-01-05 at 17.49.50" data-base62-sha1="dFX020HZLqbdRJFBoJ0C3MPz4NE" width="690" height="75" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fda450fcda7a1501cd1c4318b344d3929ad00ee_2_690x75.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fda450fcda7a1501cd1c4318b344d3929ad00ee_2_1035x112.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fda450fcda7a1501cd1c4318b344d3929ad00ee_2_1380x150.png 2x" data-dominant-color="1E2B3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-05 at 17.49.50</span><span class="informations">2267×247 25.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Both models are loaded from the PyTorch library with default weights. To ensure same shape of rgb and flow features the last iteration of the raft_models output is used in pytorchs <a href="https://pytorch.org/vision/main/generated/torchvision.utils.flow_to_image.html?highlight=flow_to_image#torchvision.utils.flow_to_image" rel="noopener nofollow ugc">flow_to_image function</a> which is subsequently also fed into the resnet50. Features from resnet 50 are extracted at the ‘flatten’ layer of the network.</p>
<p>I’ve used the following code to disable gradient computation, hoping to decrease the overhead of the model, but the improvements are minor at best.</p>
<blockquote>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
</blockquote>
<p>What else could I do to decrease memory consumption?</p>
<p>Best, Jona</p>

username: ptrblck
cooked: <p>Depending on the input size a huge memory consumption might be expected. Could you share the input shape(s) and more information about what exactly is stored?</p>

username: unofficial-Jona
cooked: <p>the input shape of the images is (1,3,448,448) (B, C, H,W). The example above only uses one such image to extract features (respectively 2 for flow).</p>
<p>In previous trials I’ve used 32 images as input (shape=(32,3,448,448)). The memory requirements were not that much greater, which feeds my suspicion that these memory requirements are not as dependent on the model. However, besides disabling gradient computation I don’t see any way to decrease these requirements further.</p>
<p>Best, Jona</p>

username: ptrblck
cooked: <p>Thanks for the update, could you post a minimal and executable code snippet to reproduce the ~400GB usage using this input shape, please?</p>