# Visual-Based Localization Using Pictorial Planar Objects in Indoor Environment
## 3 Pictorial Planner Object Based VBL  
In this section, we present an overview of our pictorial planner object based VBL framework as shown in Figure 3. The framework has 3 stages: offline object learning and libraries building, libraries and map management, and real-time device localization. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Speed**} & **Planar** & **Object** & **Object** & **Actual** & **Partial** \\  & & & **Object** & **Shape** & **Content** & **Scale** & **View** \\ \hline \multirow{2}{*}{ \begin{tabular}{c} Pixel Matching Only \\ (CNN [37; 38]) \\ \end{tabular}  } & \multirow{2}{*}{Low} & \(\alpha\) size of image & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{No} & \multirow{2}{*}{Yes} \\  & & library & & & & & \\ \hline \multirow{2}{*}{\ \begin{tabular}{c} Pixel Matching and Mapping \\ (ORB-SLAM [39; 40]) \\ \end{tabular}  } & \multirow{2}{*}{Medium} & \multirow{2}{*}{Medium} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{No} & \multirow{2}{*}{Yes} \\  & & & & & & & \\ \hline \multirow{4}{*}{ \begin{tabular}{c} ParUco ([8; 12]) \\ \end{tabular}  } & \multirow{2}{*}{High} & \multirow{2}{*}{High} & \multirow{2}{*}{Yes} & Square & Patterned & \multirow{2}{*}{Yes} & \multirow{2}{*}{No} \\  & & & & blocks & marker & & \\ \cline{1-1} \The object library building stage is the offline preparation process for the localization support. The VBL system scans all object pictures captured by cameras from different positions and angles, and generates the object feature libraries to be used for object detection. There are two main object learning components, object feature learning and object positional point extraction. The features on the planar surface of each object used for detection are learned from object picture collections. In addition, to calculate the pose of a recognized object, we also need to identify certain pixel points on the object picture to match and compare with their relative positions in the original object picture. The object positional point extraction part is used to find the positional points that not only have a unique feature but also can be used for mapping from a 2D picture surface to its 3D model. For instance, the positional points of a rectangular picture are its four corner points of the rectangle, and for other shapes, either some corner points on the picture or special ORB feature points on opposite sides of the surface. Three object libraries are used in our VBL framework: the object feature library, the positional points library, and the 3D object model. The object feature library stores the features of each object produced from the object feature learning component. Among the features, certain positional points of each object are identified and stored in the positional points library. Each positional point has a corresponding 3D point in the object's 3D model. The coordinate (x, y, z) of each point in the 3D model is represented by the actual scale of the object. The x and y are measured by the real scale from the positional point in the original object picture. The z value is 0 in the 3D model since all objects to be matched in our framework are flat surfaces. Unlike image sequence libraries used in other localization methods like SLAM, the total data size of the 3 libraries is small and can be stored in most mobile IoT devices, or small edge servers if necessary. For each target area, the 3D map of known objects in the area must be built before a device localization can be conducted. We can use some offline tools to record each object's 3D location in the real world environment. The ArUco project [10] has built an easy-to-use video analytic tool that can automatically map ArUco markers in a 3D area by shooting videos to link the relative positions of neighboring markers. We can extend such a tool and replace ArUco markers with pictorial planars to build the 3D map for all objects with planar surfaces.