Model Optimization using Low Rank Transformation in MMDetection

Sourabh: <p>Hello, I am new to mmdetection and model optimization. I am using mmdetection to use mask RCNN with SWIN transformer as the backbone. As the model is big I was trying to reduce its size.As far as I know , pruning as implemented in pytorch would not change the model size but would simply create a mask.I have also tried L1 unstructured quantization with success and was currently trying low rank transformations. I can see the transformation happening when I compare the <code>print(model)</code> before and after the low-rank transformation. However, the model is not able to infer anything. I see no bounding box that appear. The following is what I have tried :</p>
<pre><code class="lang-auto">config='/content/mmdetection/configs/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco.py'
# Setup a checkpoint file to load
checkpoint = 'checkpoints/mask_rcnn_swin-t-p4-w7_fpn_1x_coco_20210902_120937-9d6b7cfa.pth'
# initialize the detector
model = init_detector(config,checkpoint)
</code></pre>
<p>Once the model is initialized I use the following for low-rank transformation</p>
<pre><code class="lang-auto">def low_rank_transform(model):
    svd_classifier_layers=[]
    L=40
    for module_name, module in model.named_children():
      if module_name=='proj':
        continue
      if len(list(module.children())) &gt; 0:
        replace_layers(module)
      if isinstance(module, torch.nn.Linear):
        W=module.weight.data
        # print(W)
        # U,S,V=torch.svd(W)
        U, S, Vh = torch.linalg.svd(W)
        V = Vh.transpose(-2, -1).conj()
        W1 = U[:,:L]
        W2 = torch.diag(S[:L]) @ V[:,:L].t()
        if module.bias is not None:
          # print("W1:",W1)
          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=True)
          layer_1.weight.data = W2
          svd_classifier_layers.append(layer_1)

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=True)
          layer_2.weight.data = W1
          layer_2.bias.data = module.bias.data
          svd_classifier_layers.append(layer_2)
          # print("Layer2",layer_2.weight.data)
          # break

        else:
          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=False)
          layer_1.weight.data = W2
          svd_classifier_layers.append(layer_1)

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=False)
          layer_2.weight.data = W1
          # layer_2.bias.data = module.bias.data
          svd_classifier_layers.append(layer_2)

        try:
          # print("we are now trying to replace",module,"with",svd_classifier_layers,"in a sequnetial way")
          # n =int(module_name)
          # model[n]=nn.Sequential(*svd_classifier_layers)
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))
        except:
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))
</code></pre>
<p>Then I simply call the above function by passing only the backbone of the model.<br>
<code>low_rank_transform(model):(model.backbone)</code></p>
<p>I can see the expected change in the architecture where a single Linear layer gets replaced by two linear layers. However, nothing happens during inference.</p>
<p>Can someone help me with what is that I might be doing wrong here?</p>

Sourabh: <p>I have observed on using the inference_detector() that the confidence of the model is relatively low and which is obvious. Can someone suggest a better way to do low-rank transformations on a model? or perhaps how to change the model’s score threshold so as to get bounding boxes in the image.</p>