Adjust prediction layer of maskrcnn_resnet50_fpn for transfer learning and fine tuning

Santhoshnumberone: <h3><a name="my-approach-1" class="anchor" href="#my-approach-1"></a>My Approach</h3>
<p>I want to re-train a pre trained Object Detection Model using transfer learning and fine tune it further to improve the network for detection for the custom data that I currently have.</p>
<p><strong>For inferencing:</strong>  to check which model has already learnt features which help it at least detect(draw bounding box) around my custom data.<br>
Inferencing <a href="https://haochen23.github.io/2020/04/object-detection-faster-rcnn.html#.YmU-1S8RppQ" rel="noopener nofollow ugc">source</a></p>
<pre><code class="lang-auto"># import necessary libraries
%matplotlib inline
import matplotlib.pyplot as plt 
from PIL import Image
import torch
import torchvision.transforms as T
import torchvision
import numpy as np 
import cv2
import warnings
warnings.filterwarnings('ignore')

# load model
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

# set to evaluation mode
model.eval()


# load the COCO dataset category names
# we will use the same list for this notebook
COCO_INSTANCE_CATEGORY_NAMES = [
    '__background__', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus',
    'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'N/A', 'stop sign',
    'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
    'elephant', 'bear', 'zebra', 'giraffe', 'N/A', 'backpack', 'umbrella', 'N/A', 'N/A',
    'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',
    'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket',
    'bottle', 'N/A', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl',
    'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza',
    'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'N/A', 'dining table',
    'N/A', 'N/A', 'toilet', 'N/A', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',
    'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'N/A', 'book',
    'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'
]

def get_prediction(img_path, confidence):
  """
  get_prediction
    parameters:
      - img_path - path of the input image
      - confidence - threshold value for prediction score
    method:
      - Image is obtained from the image path
      - the image is converted to image tensor using PyTorch's Transforms
      - image is passed through the model to get the predictions
      - class, box coordinates are obtained, but only prediction score &gt; threshold
        are chosen.
    
  """
  img = Image.open(img_path)
  transform = T.Compose([T.ToTensor()])
  img = transform(img)
  pred = model([img])
  pred_class = [COCO_INSTANCE_CATEGORY_NAMES[i] for i in list(pred[0]['labels'].numpy())]
  pred_boxes = [[(i[0], i[1]), (i[2], i[3])] for i in list(pred[0]['boxes'].detach().numpy())]
  pred_score = list(pred[0]['scores'].detach().numpy())
  pred_t = [pred_score.index(x) for x in pred_score if x&gt;confidence][-1]
  pred_boxes = pred_boxes[:pred_t+1]
  pred_class = pred_class[:pred_t+1]
  return pred_boxes, pred_class

def detect_object(img_path,image_save_path, confidence=0.5, rect_th=1, text_size=1, text_th=1):
  """
  object_detection_api
    parameters:
      - img_path - path of the input image
      - confidence - threshold value for prediction score
      - rect_th - thickness of bounding box
      - text_size - size of the class label text
      - text_th - thichness of the text
    method:
      - prediction is obtained from get_prediction method
      - for each prediction, bounding box is drawn and text is written 
        with opencv
      - the final image is displayed
  """
  boxes, pred_cls = get_prediction(img_path, confidence)
  img = cv2.imread(img_path)
  img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  # print(len(boxes))
  for i in range(len(boxes)):
    cv2.rectangle(img, boxes[i][0], boxes[i][1],color=(0, 255, 0), thickness=rect_th)
    cv2.putText(img,pred_cls[i], boxes[i][0], cv2.FONT_HERSHEY_SIMPLEX, text_size, (0,255,0),thickness=text_th)

  cv2.imwrite(image_save_path, cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
  plt.figure(figsize=(20,30))
  plt.imshow(img)
  plt.xticks([])
  plt.yticks([])
  plt.show()

detect_object(source_file_path,save_file_path, confidence=0.18)
</code></pre>
<p>The features seem to align with</p>
<pre><code class="lang-auto"> "bottle","cell phone", "book"
</code></pre>
<p><strong>For training:</strong></p>
<p>I am looking into <a href="https://towardsdatascience.com/building-your-own-object-detector-pytorch-vs-tensorflow-and-how-to-even-get-started-1d314691d4ae?#8863" rel="noopener nofollow ugc">Building your own object detector — PyTorch vs TensorFlow and how to even get started?</a> where all the input features from <code>class scores</code> of <code>box_predictor</code> in <code>roi_heads</code> of <code>fasterrcnn_resnet50_fpn</code> pre trained models is set as divided by number of classes the</p>
<pre><code class="lang-auto">def get_model(num_classes):
   # load an object detection model pre-trained on COCO
   model = torchvision.models.detection.
           fasterrcnn_resnet50_fpn(pretrained=True)
# get the number of input features for the classifier
   in_features = model.roi_heads.box_predictor.cls_score.in_features
   # replace the pre-trained head with a new on
   model.roi_heads.box_predictor = FastRCNNPredictor(in_features,/
   num_classes)
   
   return model
</code></pre>
<h2><a name="maskrcnn_resnet50_fpn-2" class="anchor" href="#maskrcnn_resnet50_fpn-2"></a>maskrcnn_resnet50_fpn</h2>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca" title="MarkRCNNSsl"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca_2_528x500.png" alt="MarkRCNNSsl" data-base62-sha1="fyaD36Oh6qz1BQZVFasUBPu4Qum" width="528" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca_2_528x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca_2_792x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cfd790b5bd6707f1b9d815d6486f8d2b3bb98ca.png 2x" data-dominant-color="8ED0F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MarkRCNNSsl</span><span class="informations">964×912 65.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78c41793664b0641349dfff010587d0f13cf404b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/78c41793664b0641349dfff010587d0f13cf404b" title="MarkRCNNSl"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78c41793664b0641349dfff010587d0f13cf404b_2_690x442.jpeg" alt="MarkRCNNSl" data-base62-sha1="helq9uMqgrGDJyBXLekqFLLbc4H" width="690" height="442" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78c41793664b0641349dfff010587d0f13cf404b_2_690x442.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78c41793664b0641349dfff010587d0f13cf404b_2_1035x663.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78c41793664b0641349dfff010587d0f13cf404b_2_1380x884.jpeg 2x" data-dominant-color="F0F1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MarkRCNNSl</span><span class="informations">1920×1231 99.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Checked all the parameters those <code>requires_gradient</code></p>
<pre><code class="lang-auto"># Load model
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

# set model to evaluation mode
model.eval()

# List out all the name of the parameters whose gradient can be altered for further training
for name, param in model.named_parameters():

    # If requires gradient parameters
    if param.requires_grad:

        # Name and value of the parameter
        print("Name: ",name,"\nBool: ",param.requires_grad,"\n")
</code></pre>
<p><strong>OUTPUT</strong></p>
<pre><code class="lang-auto">Name:  backbone.body.layer2.0.conv1.weight 
Bool:  True 

Name:  backbone.body.layer2.0.conv2.weight 
Bool:  True 

Name:  backbone.body.layer2.0.conv3.weight 
Bool:  True 

Name:  backbone.body.layer2.0.downsample.0.weight 
Bool:  True 

Name:  backbone.body.layer2.1.conv1.weight 
Bool:  True 

Name:  backbone.body.layer2.1.conv2.weight 
Bool:  True 

Name:  backbone.body.layer2.1.conv3.weight 
Bool:  True 

Name:  backbone.body.layer2.2.conv1.weight 
Bool:  True 

Name:  backbone.body.layer2.2.conv2.weight 
Bool:  True 

Name:  backbone.body.layer2.2.conv3.weight 
Bool:  True 

Name:  backbone.body.layer2.3.conv1.weight 
Bool:  True 

Name:  backbone.body.layer2.3.conv2.weight 
Bool:  True 

Name:  backbone.body.layer2.3.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.0.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.0.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.0.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.0.downsample.0.weight 
Bool:  True 

Name:  backbone.body.layer3.1.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.1.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.1.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.2.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.2.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.2.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.3.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.3.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.3.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.4.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.4.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.4.conv3.weight 
Bool:  True 

Name:  backbone.body.layer3.5.conv1.weight 
Bool:  True 

Name:  backbone.body.layer3.5.conv2.weight 
Bool:  True 

Name:  backbone.body.layer3.5.conv3.weight 
Bool:  True 

Name:  backbone.body.layer4.0.conv1.weight 
Bool:  True 

Name:  backbone.body.layer4.0.conv2.weight 
Bool:  True 

Name:  backbone.body.layer4.0.conv3.weight 
Bool:  True 

Name:  backbone.body.layer4.0.downsample.0.weight 
Bool:  True 

Name:  backbone.body.layer4.1.conv1.weight 
Bool:  True 

Name:  backbone.body.layer4.1.conv2.weight 
Bool:  True 

Name:  backbone.body.layer4.1.conv3.weight 
Bool:  True 

Name:  backbone.body.layer4.2.conv1.weight 
Bool:  True 

Name:  backbone.body.layer4.2.conv2.weight 
Bool:  True 

Name:  backbone.body.layer4.2.conv3.weight 
Bool:  True 

Name:  backbone.fpn.inner_blocks.0.weight 
Bool:  True 

Name:  backbone.fpn.inner_blocks.0.bias 
Bool:  True 

Name:  backbone.fpn.inner_blocks.1.weight 
Bool:  True 

Name:  backbone.fpn.inner_blocks.1.bias 
Bool:  True 

Name:  backbone.fpn.inner_blocks.2.weight 
Bool:  True 

Name:  backbone.fpn.inner_blocks.2.bias 
Bool:  True 

Name:  backbone.fpn.inner_blocks.3.weight 
Bool:  True 

Name:  backbone.fpn.inner_blocks.3.bias 
Bool:  True 

Name:  backbone.fpn.layer_blocks.0.weight 
Bool:  True 

Name:  backbone.fpn.layer_blocks.0.bias 
Bool:  True 

Name:  backbone.fpn.layer_blocks.1.weight 
Bool:  True 

Name:  backbone.fpn.layer_blocks.1.bias 
Bool:  True 

Name:  backbone.fpn.layer_blocks.2.weight 
Bool:  True 

Name:  backbone.fpn.layer_blocks.2.bias 
Bool:  True 

Name:  backbone.fpn.layer_blocks.3.weight 
Bool:  True 

Name:  backbone.fpn.layer_blocks.3.bias 
Bool:  True 

Name:  rpn.head.conv.weight 
Bool:  True 

Name:  rpn.head.conv.bias 
Bool:  True 

Name:  rpn.head.cls_logits.weight 
Bool:  True 

Name:  rpn.head.cls_logits.bias 
Bool:  True 

Name:  rpn.head.bbox_pred.weight 
Bool:  True 

Name:  rpn.head.bbox_pred.bias 
Bool:  True 

Name:  roi_heads.box_head.fc6.weight 
Bool:  True 

Name:  roi_heads.box_head.fc6.bias 
Bool:  True 

Name:  roi_heads.box_head.fc7.weight 
Bool:  True 

Name:  roi_heads.box_head.fc7.bias 
Bool:  True 

Name:  roi_heads.box_predictor.cls_score.weight 
Bool:  True 

Name:  roi_heads.box_predictor.cls_score.bias 
Bool:  True 

Name:  roi_heads.box_predictor.bbox_pred.weight 
Bool:  True 

Name:  roi_heads.box_predictor.bbox_pred.bias 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn1.weight 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn1.bias 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn2.weight 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn2.bias 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn3.weight 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn3.bias 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn4.weight 
Bool:  True 

Name:  roi_heads.mask_head.mask_fcn4.bias 
Bool:  True 

Name:  roi_heads.mask_predictor.conv5_mask.weight 
Bool:  True 

Name:  roi_heads.mask_predictor.conv5_mask.bias 
Bool:  True 

Name:  roi_heads.mask_predictor.mask_fcn_logits.weight 
Bool:  True 

Name:  roi_heads.mask_predictor.mask_fcn_logits.bias 
Bool:  True 
</code></pre>
<p>I froze all the <code>requires_gradient</code> by setting them to <code>FALSE</code> except <code>roi_heads.box_predictor.cls_score.weight</code> and <code>roi_heads.box_predictor.cls_score.bias</code> those require tuning for new set of data to be trained</p>
<pre><code class="lang-auto"># List out all the name of the parameters whose gradient can be altered for further training
for name, param in model.named_parameters():

    # If requires gradient parameters
    if param.requires_grad:
        if name != "roi_heads.box_predictor.cls_score.weight" and name !="roi_heads.box_predictor.cls_score.bias":
            param.requires_grad = False

        # Name and value of the parameter
        print("Name: ",name,"\nBool: ",param.requires_grad,"\n")
</code></pre>
<h3><a name="my-current-case-for-confusion-3" class="anchor" href="#my-current-case-for-confusion-3"></a>My current case for confusion:</h3>
<p>I am training currently for only <strong>one class</strong></p>
<p>My current custom data gets mostly detected as <code>cell phone</code> and few other times as <code>bottle</code> and <code>book</code></p>
<p>Should I follow same approach as shown in <a href="https://towardsdatascience.com/building-your-own-object-detector-pytorch-vs-tensorflow-and-how-to-even-get-started-1d314691d4ae?#8863" rel="noopener nofollow ugc">Building your own object detector — PyTorch vs TensorFlow and how to even get started?</a> where all the input features from <code>class scores</code> of <code>box_predictor</code> in <code>roi_heads</code> of <code>fasterrcnn_resnet50_fpn</code> pre trained models is set as divided by number of classes?</p>
<pre><code class="lang-auto"># get the number of input features for the classifier
   in_features = model.roi_heads.box_predictor.cls_score.in_features
   # replace the pre-trained head with a new on
   model.roi_heads.box_predictor = FastRCNNPredictor(in_features,/
   num_classes)
</code></pre>
<p>Since <code>cell phone</code> features seem to be good for detection of custom data I currently have, is there any other way I can extract the weight and bias of the class <code>cell phone</code> for it to be used as <code>weights</code> and <code>bias</code> of this layers <code>roi_heads.box_predictor.cls_score.weights</code> and <code>roi_heads.box_predictor.cls_score.bias</code>?</p>