Patent ID: 11966234
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 12:
13. A non-transitory computer-readable medium having program code recorded thereon for controlling an ego agent, the program code executed by a processor and comprising:
program code to capture a first two-dimensional (2D) image associated with a current view of an environment corresponding to a current location of the ego agent;
program code to generate, via a semantic encoder, a semantically segmented image of the first 2D image;
program code to generate, via a depth estimation network, a depth map of the current view of the environment based on the semantically segmented image, the depth estimation network being trained via a disposable two-stream network including a red-green-blue (RGB) encoder, a semantic encoder, and a depth decoder shared between the RGB encoder and the semantic encoder, the depth estimation network being limited to the semantic encoder and the depth decoder during inference;
program code to generate a three-dimensional (3D) reconstruction of the first 2D image based on the depth map and a current pose of the ego agent, the current pose being based on the first 2D image and the second 2D image, the second 2D image associated with a view of the environment that is different from the current view; and
program code to control an action of the ego agent based on the 3D estimate of the environment.