Detectron2 Video inferencing shows category IDs instead of class names

Kamal_Moha: <p>I’m using detectron2 to do object detection on a video after training a custom dataset. The output video does the detection, but it indicates category ids of either 1 or 2. But I want my detection to show the class names of either ‘weed’ or ‘crop’ instead.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21aa753b4b577e361e68d6f84621d1559c4d5c36.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/21aa753b4b577e361e68d6f84621d1559c4d5c36" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21aa753b4b577e361e68d6f84621d1559c4d5c36.jpeg" alt="image" data-base62-sha1="4NOXzXGaoo2ubZOo0U2goFWZqWG" width="690" height="457" data-dominant-color="717372"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">871×577 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have downloaded the config files and the model weights for my custom training and used them as shown below;</p>
<pre><code class="lang-auto">%run detectron2/demo/demo.py --config-file /content/config2.yaml --video-input /content/crop-weed.mp4 --confidence-threshold 0.6 --output output_colab.mp4 \
--opts MODEL.WEIGHTS /content/output/model_final.pth
</code></pre>
<p><strong>Expected behavior</strong></p>
<p>I expected detectron2 to do the video inference and show the class names of either ‘weed’ or ‘crop’ around the bounding box of the object detected.</p>
<p>I need help, please.</p>

ptrblck: <p>Check which part of the code adds the text to the predicted boxed and replace the ID with a mapping to the class name.</p>

Kamal_Moha: <p>That’s what I’m trying to find out. Dectectron2 starter notebook has the below code to do object detection on a video;</p>
<pre><code class="lang-auto">%run detectron2/demo/demo.py --config-file detectron2/configs/COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml --video-input video-clip.mp4 --confidence-threshold 0.6 --output video-output.mkv \
  --opts MODEL.WEIGHTS detectron2://COCO-PanopticSegmentation/panoptic_fpn_R_101_3x/139514519/model_final_cafdb1.pkl
</code></pre>
<p>And I have followed this same template as well. So I don’t know why it doesn’t work.</p>
<p>In the above example, they have used a pretrained model. But for me I want to use a custom trained model on my dataset that’s why I have different model weights &amp; yaml file.</p>
<p>I can’t see in the doc how to add texts to the predicted bounding box. Also, many people have had this problem, but detectron2 staff didn’t solve the issues.</p>
<p>That’s why I have asked the PyTorch community for help because Detectron2 is built on PyTorch</p>

Kamal_Moha: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, just to let you know.</p>
<p>Detectron2 has a <a href="https://detectron2.readthedocs.io/en/latest/_modules/detectron2/utils/video_visualizer.html" rel="noopener nofollow ugc">VideoVisualizer</a> class that takes in all the logic of doing object detection in a video. In there that’s where the metadata like labels is included.</p>
<p>There is also a <a href="https://github.com/facebookresearch/detectron2/blob/main/demo/demo.py" rel="noopener nofollow ugc">demo.py</a> that has the template for the terminal inputs.</p>
<p>I have tried using these two files to make edits to my video inferencing, but it hasn’t worked. I went into the demo.py and added a new parser-argument called <code>--vid_meta</code> that accepts the train metadata.</p>
<pre><code class="lang-auto">def get_parser():
  parser.add_argument(
          "--vid_meta",
          default=['Crop', 'Weed'],
          help='Adding Video metadata'
      )
</code></pre>
<p>And further down, I fed that argument into the <code>VideoVisualizer</code> class as metadata</p>
<pre><code class="lang-auto">if args.vid_meta:
      VideoVisualizer(metadata=args.vid_meta)
</code></pre>
<p>I have also made some changes to labels in the VideoVisualizer so that it can access the labels from the train metadata. I changed the labels in the function;<br>
<code>draw_instance_predictions()</code> in the <code>VideoVisualizer</code> class;</p>
<p>from;<br>
<code>labels = _create_text_labels(classes, scores, self.metadata."thing_classes", None)</code><br>
to;<br>
<code>labels = _create_text_labels(classes, scores, self.metadata.things_classes)</code></p>
<p>After making those edits in both <code>VideoVisualizer</code> and <code>demo.py</code>, I re-ran the video detection command as;</p>
<pre><code class="lang-auto">%run /content/demo.py --config-file /content/drive/MyDrive/Real-Time-Object-Detection/Detectron2-models/config4.yaml \
--video-input /content/crop-weed.mp4 --confidence-threshold 0.6 --output output_colab6.mp4 --vid_meta train_metadata \
--opts MODEL.WEIGHTS /content/output/model_final.pth
</code></pre>
<p>whereby train_metadata is equivalent to;</p>
<pre><code class="lang-auto">namespace(name='my_dataset_train',
          json_file='/content/Annotated-Images-Dataset-1/train/_annotations.coco.json',
          image_root='/content/Annotated-Images-Dataset-1/train',
          evaluator_type='coco',
          thing_classes=['weed-crops', 'crop', 'weed'],
          thing_dataset_id_to_contiguous_id={0: 0, 1: 1, 2: 2})
</code></pre>
<p>And then when I rerun it and downloaded the video, unfortunately it still doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
<p>But I’m very confident the solution is in configuring the <code>VideoVisualizer</code> class. I hope to get your help on this please.</p>

Kamal_Moha: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’m waiting on you to help me through this process.</p>
<p>What do you think about the <code>VideoVisualizer</code> and the <code>demo.py</code> that detectron2 provides? How can we manipulate them to be able to have label names around the bounding boxes instead of category IDs.</p>