Patent ID: 11900626
Assignee: TOYOTA RESEARCH INSTITUTE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  B | IPC B  G

Claim 6:
7. A non-transitory computer-readable medium having program code recorded thereon for learning depth-aware keypoints and associated descriptors from monocular video, the program code being executed by a processor and comprising:
program code to train a keypoint network and a depth network to learn the depth-aware keypoints and the associated descriptors based on a target image and a context image from successive images of the monocular video;
program code to lift 2D keypoints from the target image to learn first 3D keypoints of the target image based on a learned depth map of the target image from the depth network;
program code to estimate a pose transformation from the target image to the context image based on the first 3D keypoints of the target image to geometrically match the first 3D keypoints in the target image to first 3D keypoints in the context image;
program code to compute second 3D keypoints of the target image based on an associated ego-motion computed between the target image and the context image according to the first 3D keypoints of the target image and the first 3D keypoints of the context image;
program code to estimate ego-motion of an ego vehicle from the target image to the context image captured by the ego vehicle based on the second 3D keypoints in the target image and the first 3D keypoints in the context image; and
program code to plan a trajectory of the ego vehicle according to the estimated ego-motion of the ego vehicle.