Patent ID: 11875583
Assignee: DALIAN UNIVERSITY OF TECHNOLOGY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A dataset generation method for self-supervised learning scene point cloud completion based on panoramas, comprising the following steps:
step 1: generating initial point cloud from a panorama under a specific view by:
1.1) representing a three-dimensional world by a sphere, and representing coordinates in x, y and z directions by longitude and latitude, wherein a radius r of the sphere represents a depth value; assuming that a length of a depth panorama D1 is a same as a range of −180° to 180° in a horizontal direction of a scene, and a width of the depth panorama D1 corresponds to a range of −90° to 90° in a vertical direction; representing a coordinate of each pixel of the depth panorama D1 with the longitude and the latitude, wherein a radius of a point in the sphere corresponding to each pixel is a depth value of each pixel in the depth panorama D1; and in a spherical coordinate system, converting the latitude, longitude and depth values of each pixel into x, y and z coordinates in a camera coordinate system to generate point cloud P0;
1.2) converting the point cloud P0 in the camera coordinate system to a world coordinate system based on a camera extrinsic parameter corresponding to a view v1, and assigning a color information of a RGB panorama C1 and a normal panorama N1 to each point in the point cloud P0 in a row column order of pixel points to generate initial point cloud P1 with RGB information and initial point cloud P2 with normal information;
step 2: selecting a new occlusion prediction view based on the initial point cloud P1 and P2 by:
2.1) encoding the initial point cloud P1 by a truncated signed distance function; dividing a selected 3D space to be modeled into a plurality of blocks, and calling the block as a voxel; storing, by the voxel, a distance value between the block and a nearest object surface, and representing, by a symbol of the distance value, that the voxel is in a free space or a closed space; and conducting truncation processing if an absolute value of the distance value exceeds a set truncation distance D;
2.2) assuming that a voxel block corresponding to the view v1 is t0; updating a distance value of t0 as 0; and updating the distance value of the voxel block near t0 according to a distance from to, wherein if a distance from to is smaller, a decline of the distance value is larger;
2.3) traversing each voxel block to find a voxel block with the largest distance value; selecting the voxel block closest to a scene center if a plurality of the voxel blocks have a same distance value; randomly selecting from the voxel blocks which satisfy conditions if a distance from a scene center is still the same; and taking a center of a selected voxel block as a position of view v2 to obtain a translation matrix of the view v2, with a rotation matrix of the view v2 the same as a rotation matrix of the view v1;
step 3: generating a panorama under the view v2 from the initial point cloud P1 and P2 by:
3.1) converting the initial point cloud P1 with the RGB information and the initial point cloud P2 with normal information in the world coordinate system to the camera coordinate system based on a camera extrinsic parameter corresponding to the view v2;
3.2) in the spherical coordinate system, converting the x, y and z coordinates of each point in the point cloud P1 and the point cloud P2 respectively into latitude, longitude and radius, and corresponding to a pixel position of a 2D panorama; making a color of each point correspond to the pixel position; increasing an influence range of each point, specifically, extending a calculated each pixel (x,y) outward to pixels (x,y), (x+1,y), (x,y+1) and (x+1,y+1); and copying a information carried by each pixel to a new pixels;
3.3) firstly, initializing a depth value of each pixel of depth panorama D2 to a value 65535 that is represented by an unsigned 16-bit binary number, and initializing a color value of each pixel of a RGB panorama C2 and a normal panorama N2 as a background color; then conducting a following operation on all the pixels generated in step 3.2): acquiring a position of the pixel (x,y) and a corresponding depth value, and comparing with the depth value at the pixel (x,y) in the depth panorama D2; if a former depth value is smaller, updating the depth value at (x,y) in the depth panorama D2 and a color value at (x,y) in the RGB panorama C2 and a normal panorama N2; if a latter depth value is smaller, keeping unchanged; and after all updates are completed, obtaining the RGB panorama C2, the depth panorama D2 and the normal panorama N2 rendered under a new view v2;
step 4: generating incomplete point cloud from the panorama under a specific view by:
4.1) generating point cloud {tilde over (P)}0 from the depth panorama D2;
4.2) calculating normal direction in the world coordinate system according to the normal panorama N2, and converting the normal direction in the world coordinate system to the camera coordinate system according to the camera extrinsic parameter corresponding to the view v2, wherein the normal panorama N2 is rendered in the camera coordinate system corresponding to the view v2, but a color of the normal panorama records the normal direction in the world coordinate system;
4.3) in a process of 2D-3D rectangular projection, angle masks need to be calculated to locate a stripe area, so that a scene point cloud completion network completes a real occlusion area; calculating each point in the point cloud {tilde over (P)}0 in the camera coordinate system; denoting a vector represented by a connecting line from an origin to a point in {tilde over (P)}0 as {right arrow over (n)}1; denoting a vector of a point in a row column order calculated from the normal panorama N2 as {right arrow over (n)}2; calculating an angle α between the vector {right arrow over (n)}1 and the vector {right arrow over (n)}2; then calculating difference values between the angle α and 90° to obtain absolute values; and filtering points with the absolute value of less than 15° as angle masks;
4.4) converting the point cloud {tilde over (P)}0 in the camera coordinate system to the world coordinate system based on the camera extrinsic parameter corresponding to the view v2, and assigning the color information of the RGB panorama C2 and the normal panorama N2 to each point in the point cloud {tilde over (P)}0 in the row column order of the pixel points to generate incomplete point cloud P3 with RGB information and incomplete point cloud P4 with normal information; and
step 5: constructing a self-supervised learning dataset by:
taking the incomplete point cloud P3 with RGB information, the incomplete point cloud P4 with normal information and the angle masks as input for the training of a scene point cloud completion network, wherein the targets of the scene point cloud completion network are incomplete point cloud P1 with RGB information and incomplete point cloud P2 with normal information.