CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Channel: Nur Muhammad Shafiullah
Date: 2022-10-12

Read the paper: https://arxiv.org/abs/2210.05663
Website: https://mahis.life/clip-fields/
Code: https://github.com/notmahi/clip-fields

Tl;dr CLIP-Field is a novel weakly supervised approach for learning a semantic robot memory that can respond to natural language queries solely from raw RGB-D and odometry data with no extra human labelling. It combines the image and language understanding capabilities of novel vision-language models (VLMs) like CLIP, large language models like sentence BERT, and open-label object detection models like Detic, and with spatial understanding capabilities of neural radiance field (NeRF) style architectures to build a spatial database that holds semantic information in it.