Patent ID: 11900626
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  B | IPC B  G

Claim 12:
13. A system for learning depth-aware keypoints and associated descriptors from monocular video, the system comprising:
a depth-aware keypoint model trained to learn a keypoint network and a depth network, and trained to learn the depth-aware keypoints and the associated descriptors based on a target image and a context image from successive images of the monocular video;
a keypoint lifting module to lift 2D keypoints from the target image to learn first 3D keypoints of the target image based on a learned depth map of the target image from the depth network;
a pose estimation module to estimate a pose transformation from the target image to the context image based on the first 3D keypoints of the target image to geometrically match the first 3D keypoints in the target image to first 3D keypoints in the context image and to compute second 3D keypoints of the target image based on an associated ego-motion computed between the target image and the context image according to the first 3D keypoints of the target image and the first 3D keypoints of the context image;
an ego-motion estimation module to estimate ego-motion of an ego vehicle from a target image to a context image captured by the ego vehicle based on the second 3D keypoints in the target image and the first 3D keypoints in the context image; and
a planner module to plan a trajectory of the ego vehicle according to the estimated ego-motion of the ego vehicle.