Patent ID: 11900626
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  B | IPC B  G

Claim 0:
1. A method for learning depth-aware keypoints and associated descriptors from monocular video, comprising:
training a keypoint network and a depth network to learn the depth-aware keypoints and the associated descriptors based on a target image and a context image from successive images of the monocular video;
lifting 2D keypoints from the target image to learn first 3D keypoints of the target image based on a learned depth map of the target image from the depth network;
estimating a pose transformation from the target image to the context image based on the first 3D keypoints of the target image to geometrically match the first 3D keypoints in the target image to the first 3D keypoints in the context image;
computing second 3D keypoints of the target image based on an associated ego-motion computed between the target image and the context image according to the first 3D keypoints of the target image and the first 3D keypoints of the context image;
estimating ego-motion of an autonomous ego vehicle from the target image to the context image captured by the autonomous ego vehicle based on the second 3D keypoints in the target image and the first 3D keypoints in the context image; and
planning a trajectory of the autonomous ego vehicle according to the estimated ego-motion of the autonomous ego vehicle.