ObjectReact: Learning Object-Relative Control for Visual Navigation
Authors: name: Sourav Garg, name: Dustin Craggs, name: Vineeth Bhat, name: Lachlan Mares, name: Stefan Podgorski, name: Madhava Krishna, name: Feras Dayoub, name: Ian Reid

Visual navigation using only a single camera and a topological map has
recently become an appealing alternative to methods that require additional
sensors and 3D maps. This is typically achieved through an "image-relative"
approach to estimating control from a given pair of current observation and
subgoal image. However, image-level representations of the world have
limitations because images are strictly tied to the agent's pose and
embodiment. In contrast, objects, being a property of the map, offer an
embodiment- and trajectory-invariant world representation. In this work, we
present a new paradigm of learning "object-relative" control that exhibits
several desirable characteristics: a) new routes can be traversed without
strictly requiring to imitate prior experience, b) the control prediction
problem can be decoupled from solving the image matching problem, and c) high
invariance can be achieved in cross-embodiment deployment for variations across
both training-testing and mapping-execution settings. We propose a topometric
map representation in the form of a "relative" 3D scene graph, which is used to
obtain more informative object-level global path planning costs. We train a
local controller, dubbed "ObjectReact", conditioned directly on a high-level
"WayObject Costmap" representation that eliminates the need for an explicit RGB
input. We demonstrate the advantages of learning object-relative control over
its image-relative counterpart across sensor height variations and multiple
navigation tasks that challenge the underlying spatial understanding
capability, e.g., navigating a map trajectory in the reverse direction. We
further show that our sim-only policy is able to generalize well to real-world
indoor environments. Code and supplementary material are accessible via project
page: https://object-react.github.io/