Patent ID: 11966234
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 6:
7. An apparatus for controlling an ego agent, comprising:
a processor;
a memory coupled with the processor; and
instructions stored in the memory and operable, when executed by the processor, to cause the apparatus:
to capture a first two-dimensional (2D) image associated with a current view of an environment corresponding to a current location of the ego agent;
to generate, via a semantic encoder, a semantically segmented image of the first 2D image;
to generate, via a depth estimation network, a depth map of the current view of the environment based on the semantically segmented image, the depth estimation network being trained via a disposable two-stream network including a red-green-blue (RGB) encoder, a semantic encoder, and a depth decoder shared between the RGB encoder and the semantic encoder, the depth estimation network being limited to the semantic encoder and the depth decoder during inference;
to generate a three-dimensional (3D) reconstruction of the first 2D image based on the depth map and a current pose of the ego agent, the current pose being based on the first 2D image and the second 2D image, the second 2D image associated with a view of the environment that is different from the current view; and
to control an action of the ego agent based on the 3D estimate of the environment.