Patent ID: 11948309
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system, comprising:
one or more processors; and
a memory communicably coupled to the one or more processors and storing instructions that when executed by the one or more processors cause the one or more processors to jointly train a machine-learning-based monocular optical flow, depth, and scene flow estimator, the instructions including:
an optical flow estimation module including instructions that when executed by the one or more processors cause the one or more processors to process a pair of temporally adjacent monocular image frames using a first neural network structure to produce an optical flow estimate and to extract, from at least one image frame in the pair of temporally adjacent monocular image frames, a set of encoded image context features;
a triangulation module including instructions that when executed by the one or more processors cause the one or more processors to triangulate the optical flow estimate to generate a depth map; and
a training module including instructions that when executed by the one or more processors cause the one or more processors to:
extract a set of encoded depth context features from the depth map using a depth context encoder; and
combine the set of encoded image context features and the set of encoded depth context features to improve performance of a second neural network structure in estimating depth and scene flow, wherein the instructions in the optical flow estimation module to process the pair of temporally adjacent monocular image frames using the first neural network structure to produce the optical flow estimate and to extract, from the at least one image frame in the pair of temporally adjacent monocular image frames, the set of encoded image context features include instructions that when executed by the one or more processors cause the one or more processors to:
extract the set of encoded image context features using a Red-Green-Blue (RGB) context encoder;
extract a set of encoded image features from the pair of temporally adjacent monocular image frames using a RGB encoder;
process the set of encoded image features using a correlation layer to generate a correlation volume; and
refine iteratively an initial estimate of optical flow based, at least in part, on the set of encoded image context features using a Gated-Recurrent-Units-based update operator and the correlation volume.