Semantic-aware Next-Best-View for Multi-DoFs Mobile System in Search-and-Acquisition based Visual Perception

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:12:22

Abstract: Efficient visual perception using mobile systems is crucial, particularly in unknown environments such as search and rescue operations, where swift and comprehensive perception of objects of interest is essential. In such real-world applications, objects of interest are often situated in complex environments, making the selection of the 'Next Best' view based solely on maximizing visibility gain suboptimal. Semantics, providing a higher-level interpretation of perception, should significantly contribute to the selection of the next viewpoint for various perception tasks. In this study, we formulate a novel information gain that integrates both visibility gain and semantic gain in a unified form to select the semantic-aware Next-Best-View. Additionally, we design an adaptive strategy with termination criterion to support a two-stage search-and-acquisition manoeuvre on multiple objects of interest aided by a multi-degree-of-freedoms (Multi-DoFs) mobile system. Several semantically relevant reconstruction metrics, including perspective directivity and region of interest (ROI)-to-full reconstruction volume ratio, are introduced to evaluate the performance of the proposed approach. Simulation experiments demonstrate the advantages of the proposed approach over existing methods, achieving improvements of up to 27.13\% for the ROI-to-full reconstruction volume ratio and a 0.88234 average perspective directivity. Furthermore, the planned motion trajectory exhibits better perceiving coverage toward the target.

Main paper: 
 related work 2.1 mobile system informative path planning for visual acquisition: 
 Real-time informative path planning is typically the approach to tackle the non-model-based visual acquisition problem that has no prior information or knowledge of the environment or the target object. Thus, the non-model-based reconstruction needs to plan each view in real-time, which is different from the model-based approach that can be planned offline. There are two main approaches for evaluating new viewpoints in 3D reconstruction: surface-based methods and volumetric methods. Surface-based approaches represent the 3D shape as a mesh and evaluate new views by analyzing the mesh surface [5] . For example, Krainin et al. [17] used a surface-based approach that modelled uncertainty with a Gaussian distribution along each camera ray and measured information gain as the total entropy reduction weighted by surface area. Surface-based methods can evaluate the quality of the 3D model during reconstruction but are computationally expensive due to complex visibility calculations [33] . And more recently, dynamic objects can be accurately reconstructed by surface-based method [37] . Volumetric methods, on the other hand, represent the 3D shape with voxels, which allow for simple visibility calculations and estimating the probability that each voxel is occupied [14] . Volumetric view evaluation casts rays from the candidate next views through the voxel space to simulate how a camera would sample the scene. Volumetric approaches are computationally more efficient but may not directly provide a surface model of the 3D shape. After that, hybrid methods [21] have combined both surface and volumetric representations to gain the benefits of each. In summary, surface-based 3D reconstruction evaluates new views by analyzing an estimated 3D mesh surface [5, 19] , while volumetric methods evaluate new views by casting rays through the voxel representation [14] . The hybrid methods use both representations to improve the efficiency of 3D modelling [21] .Next-Best-View is a widely-used greedy method to find local solutions from incomplete information. It was first addressed in the 1980s [7, 24] . It determines the next viewpoint that can observe the largest information gain from the current map iteratively, finally resulting in a completed observation. The information gain metric depends on the specific application and requirements. In order to perceive the unknown volumetric information, besides the early stage approach [1] which simply counts the number of unknown voxels that can be seen, Kriegel et al. [21] use information theoretic entropy to estimate the expected observation. To achieve high completeness of reconstruction, Delmerico et al. [9] proposed proximity count and area factor volumetric information, optimizing the expected gain on a probabilistic map. In 2016, Bircher et The reconstructed mesh is visualized using the occupancy TSDF map.al. [3] presented the receding horizon NBV (RH-NBV) that adopts the core idea of model predictive control (MPC). It only executes the first edge in the best branch of RRT to avoid the dilemma of local minima. It also introduces the exponential discount term to penalize the long-distance path. In [32] , a novel utility function is formulated as the ratio of gain and cost, minimizing the number of parameters that need to be fine-tuned. Different from the NBV, which belongs to the sampling-based informative viewpoint planning method, the frontier-based method [39] consistently pursues the boundaries between the explored free and unexplored areas in the occupancy map. The frontier-based method is widely employed in high-speed flight and fast exploration tasks [2, 6] , but it is difficult to be generalized to other applications since it cannot have a flexible information gain formulation in NBV fashion. In [15] , an uncertainty-guided mapless NBV scheme is proposed, leading to more accurate scene reconstruction. In [26] , the predicted fruit shapes are explicitly used to compute information gain for fruit mapping and reconstruction. Due to the simplicity of the purpose or the environment, there is no existing research that has focused on the contribution of semantics on viewpoint selection in NBV fashion in the application of either unknown exploration or single-object reconstruction. However, under challenging environments (e.g., search and rescue), searching and perceiving the semantic informative views can help us model the object and its surroundings more efficiently. For a more closely related work [18] that proposed a semantically informed scheme for reconstruction. It presents the utility term multiplied by the entropy-formed gain, but does not formulate the semantic term explicitly. It may result in penalization on the unknown exploration capability and would be difficult to generalize to different tasks such as the search-and-acquisition mission. 
proposed method 3.1 problem description: 
 The problem considered in this work is that there exist one or more specific targets 𝐴 = {𝐴 1 , ..., 𝐴 𝑁 }, located at the unknown positions in a 3D space 𝑉 ⊂ R 3 . Unlike the other exploration approaches, the focus is not on observing all the free and occupied space (𝑉 𝑓 𝑟𝑒𝑒 and 𝑉 𝑜𝑐𝑐 ⊂ 𝑉 ) to achieve 𝑉 𝑓 𝑟𝑒𝑒 ∪𝑉 𝑜𝑐𝑐 = 𝑉 . Instead, our approach focuses on searching for and observing each target 𝐴 𝑘 ∈ 𝐴 sequentially, We begin by exploring space 𝑉 and once we identify a set of occupied voxels 𝑉 𝑜𝑏𝐴-𝑘 ⊂ 𝑉 𝑜𝑐𝑐 that have been labelled as 𝑐 𝑡𝑔𝑡 -𝑘 , it indicates that the target 𝐴 𝑘 has been found. Then the acquisition mode is initiated to retrieve not only the volume 𝑉 𝑡𝑔𝑡 -𝑘 of each target 𝐴 𝑘 , but also its surroundings (𝑉 𝑠𝑢𝑟 -𝑘 ⊂ 𝑉 𝑓 𝑟𝑒𝑒 or 𝑉 𝑠𝑢𝑟 -𝑘 ⊂ 𝑉 𝑜𝑐𝑐 ), with the objective of effectively enlarging the observed volume 𝑉 𝑜𝑏𝐴-𝑘 s.t. min |𝑉 𝑟𝑒𝑠 -𝑘 | = min |𝑉 𝑡𝑔𝑡 -𝑘 -𝑉 𝑜𝑏𝐴-𝑘 | utilizing the most extensive accessible perspective coverage within the limited time, where 𝑉 𝑟𝑒𝑠 -𝑘 represents the residue voxels of the target 𝐴 𝑘 . The searching and acquisition process will be switched to the next target 𝐴 𝑘+1 after achieving the maximum observation on 𝐴 𝑘 .Two maps are constructed to support the two-stage search-andacquisition scheme of the proposed semantic-aware NBV framework, an occupancy TSDF map and a labelled TSDF map. Figure 2 illustrates the overall system, where the occupancy TSDF map is incrementally updated from the observations. This is achieved by utilizing the point cloud input from the Intel RealSense D435i depth camera and the real-time pose of the UAV, following the approach proposed in voxblox [29] . The occupancy TSDF map provides information about the occupancy status of the environment, which is essential for the planner to generate RRT* in free space and calculate visibility gain. Additionally, we constructed a labelled TSDF map inspired by the work of Grinvald et al. [12] . This map is generated by raycasting the overlap of the segmentation results from Mask R-CNN [13] based on the RGB input and the depth segments from the depth image input. Depth segments are identified by finding the convex area of depth discontinuity in the depth image. The labelled TSDF map provides a detailed representation of the environment's geometry with semantics, which is useful for the planner to identify the semantic gain. The different types of map representations used in the system are organized into separate layers, with each layer consisting of a set of blocks that are indexed based on their position in the map. It is the same as the structure adopted in voxblox [29] . The mapping between the block positions and their locations is stored in the hash table adopting voxel hashing [28] . Finally, the acquisition result is visualized by the surface model generated from the occupancy TSDF map.From the representation of input TSDF maps, the space 𝑉 is divided into separate layers of unit-volume cubical voxel 𝑚 𝑜 ∈ M 𝑜 , 𝑚 𝑙 ∈ M 𝑙 , where M 𝑜 and M 𝑙 denote the occupancy and labelled map respectively. Each voxel 𝑚 𝑜𝑖 in the occupancy map M 𝑜 consists of an associated centre position 𝑝 𝑖 , distance 𝑑 𝑖 , weight 𝑤 𝑖 and state 𝑠 𝑖 . The centre position is represented by 𝑝 𝑖 using the coordinate of its geometric centre, and the voxel's distance from the surface boundary is represented by 𝑑 𝑖 . In order to minimize the quadratic sensing error of the 3D sensor (e.g., depth camera), we adopt the distance 𝑑 𝑖 updating approach in [32] . The weight 𝑤 𝑖 is a metric that refers to the reliability of the distance's measurement. Here we employ the weighting method formulated in voxblox [29] . The state 𝑠 𝑖 of each voxel can be marked as "FREE", "OCCUPIED", or "UNKNOWN". For the voxel 𝑚 𝑙𝑖 in the labelled map M 𝑙 , there are three additional associated properties instance label 𝑙 𝑖 , semantic category 𝑐 𝑖 and label confidence 𝑙 𝑐𝑖 . In which the instance label is the index with the highest overlap probability between the binary mask result 𝑚 𝑖 from Mask R-CNN and the result 𝑟 𝑖 from depth segmentation. The corresponding semantic category is assigned to 𝑐 𝑖 if available; otherwise, the default semantics is the background. The label confidence is the number of times the voxel has been labelled as 𝑙 𝑖 divided by the observation times.In order to perceive the unknown area and search for the target we are interested in, we define the visibility gain of a branch 𝑏 associated 𝑛 nodes {𝑏 1 , 𝑏 2 , ..., 𝑏 𝑛 } in Equation 1.EQUATIONThe visible voxels {𝑚 𝑜1 , 𝑚 𝑜2 , ..., 𝑚 𝑜𝑚 } at node 𝑏 𝑗 are obtained using the intrinsic and extrinsic parameters of the camera. Thus,EQUATIONFor simply perceiving the unknown in the 'search' stage, we employ the conventional 𝑉 _𝑔𝑎𝑖𝑛 formulation that applies a unit increase in gain if the 𝑠 𝑖 is "UNKNOWN", and there is no gain for "OCCUPIED" or "FREE" voxel.Formulation. Similar to the visibility gain, we have the semantic gain for each branch:EQUATIONAgain for each node 𝑏 𝑗 on the branch,EQUATIONThe 𝑆_𝑔𝑎𝑖𝑛 for each visible voxel 𝑚 𝑜𝑖 at the specific node 𝑏 𝑗 is formulated intuitively favours the viewpoints that can observe the new area around the labelled target voxel. As is shown in Equation 5.EQUATIONWhere 𝜂 𝑡𝑔𝑡 denotes the influence factor that refers to the significance or priority of the voxel with the target label. The exponential term represents the exponential discount on the influence regarding the distance 𝑑 𝑙𝑖 of the current voxel to the target volume 𝑉 𝑜𝑏𝐴-𝑘 of the target 𝐴 𝑘 . 𝜆 1 , 𝜆 2 are the weight term and 𝑐 1 , 𝑐 2 are the discount cost functions. In order to minimize the sensing error and refine the voxel that has already been labelled as 𝑐 𝑡𝑔𝑡 -𝑘 , we also introduced the function 𝑓 in Equation 6 as its gain.EQUATIONWhere 𝑁 𝑟𝑎𝑦𝑠 (𝑚 𝑙𝑖 ) denotes the number of rays intersecting the 𝑚 𝑙𝑖 , which is usually proportional to the inverse of depth quadratically. 𝑁 𝑒𝑥𝑝 represents the expected number of intersecting rays. The list L 𝑡𝑔𝑡 stores the voxels which have been labelled with the semantic category 𝑐 𝑡𝑔𝑡 -𝑘 , and L 𝑡𝑔𝑡 is maintained to serve the calculation of the shortest distance 𝑑 𝑙𝑖 . Inspired by [23] , we maintain the listed voxels (i.e. 𝑉 𝑜𝑏𝐴-𝑘 ) in a continuous and convex shape.Criterion. The proposed method integrates both visibility gain and semantic gain in a consistent format in Equation 7.EQUATIONWhere 𝛿𝑏 𝑗 𝑏 𝑗 -1denotes the edge distance from node 𝑏 𝑗 -1 to node 𝑏 𝑗 . 𝐾 is a bool variable controlling the mode preference switching between 'search' and 'acquisition' in our case. 𝑓 𝑜 and 𝑓 𝑙 represent the cost function penalizing on the distance of the long edge. It could be in the form of exponential penalty [4, 34] , linear penalty [8] or a reciprocal cost to reduce the complexity in tuning parameters [32] . Here, we employ the format in [32] . 𝜆 𝑜 , 𝜆 𝑙 are the constant parameters.𝑓 𝑜 (𝛿EQUATION𝑓 𝑙 (𝛿EQUATIONThe state switching of the bool variable 𝐾 ensures the smoothness of the stage changing between searching and acquisition in the manoeuvre. We also introduce a termination criterion for the acquisition stage to perform the target switching within a manoeuvre. We separate the semantic gain for each branch into three parts:EQUATIONEQUATIONEQUATIONWhere con1 refers to condition 1 𝑠 𝑖 = 𝑈 𝑛𝑘𝑛𝑜𝑤𝑛, con2 refers to 𝑠 𝑖 = 𝑂𝑐𝑐𝑢𝑝𝑖𝑒𝑑 && 𝑐 𝑖 = 𝑐 𝑡𝑔𝑡 -𝑘 and con3 refers to 𝑠 𝑖 = 𝑂𝑐𝑐𝑢𝑝𝑖𝑒𝑑 && 𝑐 𝑖 ! = 𝑐 𝑡𝑔𝑡 -𝑘 && 𝑐 𝑖 ! = 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑. The planner starts with a zero-size L 𝑡𝑔𝑡 , 𝐾 is initially assigned to 1. Once the list L 𝑡𝑔𝑡 is expanded, 𝐾 is switched to 0. The acquisition for one target is terminated if 𝑆 𝑠𝑢𝑟𝑟𝑜𝑢𝑛𝑑 is far greater than the summation of 𝑆 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 and 𝑆 𝑟𝑒 𝑓 𝑖𝑛𝑒 for 𝑐 𝑡ℎ𝑟𝑒 branches. Then 𝐾 flips to 1, the L 𝑡𝑔𝑡 is cleared, and meanwhile, the target label is switched to 𝑐 𝑡𝑔𝑡 -(𝑘+1) . Once the list L 𝑡𝑔𝑡 is further expanded, 𝐾 will be set to 0 again. This described strategy can also be represented as Algorithm 1 below.  
experimental setup: 
 Since the planner operates within a perception-planning-execution loop, realistic simulation is compulsory for the evaluation of the proposed scheme. The proposed approach is tested in the simulated world scenes in Gazebo, a 3D dynamic physical robotics simulator. The developed behaviour of the UAV is operating on the Robot Operating System (ROS) [31] . Gazebo-based simulator RotorS [11] is employed to provide an accurate model of the UAV's physics. The underlying control hierarchy of UAV is presented in [16] .The experiments are conducted in the simulation environment, three different settings with two self-build scenes (a narrow collapsed scene with an uneven lighting condition and a larger indoor house scene with ideal lighting condition) in Gazebo with the aid of the individual models by Open Robotics [30] and Google Research [10] . And all the experiment results are collected on the machine with an Intel 8C16T Core i7-11700KF at 3.6 GHz × 16 and an NVIDIA GeForce RTX 3060 graphic card. Room Scene, but the refrigerator and sink are highlighted as the specific targets.The basic system parameters are consistent throughout all the experiments described in this study, including the motion dynamics constraints of the UAV and the camera parameters for acquisition, as shown in Table 1 . For each experiment, the UAV starts at the initial pose where the target is not within the field of view (FOV).Since the proposed scheme is designed to execute the search-andacquisition manoeuvre for the specific target, once the target is found, we aim to acquire the target from multiple accessible viewpoints and achieve maximum reconstruction coverage of the target itself and potential interactions with the surroundings. For most real-world applications, the perception, planning and motion control pipeline of the UAV is expected to execute fully onboard. Thus, time usage is also a crucial aspect of the evaluations.In order to measure whether the UAV consistently perceives the target and its nearest surroundings during the acquisition stage, we calculated the perspective directivity in the target direction 𝐷 𝑡𝑔𝑡 -𝑘 for each selected view. The current position 𝑝 𝑖 , and orientation 𝑜 𝑖 of the UAV at view 𝑖 are obtained from the odometry. The ground truth position of the target 𝑝 𝑡𝑔𝑡 -𝑘 is a privileged knowledge that we defined based on the built scene and is not known to the planner. 𝑜 𝑖 𝑃 and 𝑜 𝑖 𝑌 denote the pitch and yaw angle of view 𝑖. The directivity at the target direction 𝐷 𝑖 𝑡𝑔𝑡 -𝑘 of view 𝑖 is defined as the cosine of the angle between the camera optical axis 𝑂 𝑐𝑎𝑚 and the line connecting the current position 𝑝 𝑖 with the target position 𝑝 𝑡𝑔𝑡 -𝑘 , i.e.EQUATIONEQUATION4.2.2 ROI Reconstruction Progress in Volume and ROI-to-full Reconstruction Ratio. In addition to the perspective directivity, we also periodically record the reconstructed map and analyze the global growth of the reconstruction volume as well as the growth within the region of interest. Again, the region of interest (ROI) is a privileged knowledge that is not known to the planner. It comprises the target 𝑉 𝑡𝑔𝑡 -𝑘 and the nearest surroundings 𝑉 𝑠𝑢𝑟 -𝑘 . The volume ratio of reconstructed ROI over the full reconstructed map indicates the strength of the purpose. A higher ratio implies less reconstruction redundancy in perceiving the target under the complex environment, while a lower ratio indicates more storage consumption on the non-important reconstructions. The target perceiving coverage is analyzed and compared using the motion trajectories with each pose of the UAV and the frustum of the camera. The experiments in this study are conducted in two simulation scenes with three different settings in total. Compared to the smaller scene Collapsed Room, it typically takes longer for the UAV to locate the target in the larger one. In Figure 3 (a) and (d), the target is well located within the first 30 s in each scenario. In complex scenes with more intricate structures, the proposed approach demonstrates significant advantages over the RH-NBV approach [4] , around 40% to 7% ahead at 60 s. Meanwhile, the frontier-based approach [40] also performed well in Figure 3 (d) since it has a strong pattern of exploring along the large and continuous entity, such as walls. It also can be seen in the trajectory in Figure 4 (h) that the target person in Kitchen and Dining Scene is located closer to the corner of the wall. However, the proposed approach still exhibits 12% to 3 (g).In Figure 3 (b) and (e), the proposed planner prioritizes the ROI reconstruction once the target is located, while other planners focus on perceiving other unknowns, which may belong to semantically redundant areas. The proposed approach achieves an average ROIto-full ratio of 0.3268 and 0.5046 for each scene, respectively. In comparison, the RH-NBV and frontier-based approaches achieve averages of 0.2174, 0.2333 and 0.2228, 0.3586, respectively. For the multi-object scenario in Figure 3 (h), our method shows less advantage in ROI-to-full ratio compared to the single object since more searching and target switching require more observations of the environment.The distribution of view directivity during the entire manoeuvre is shown in Figure 3 (c) and (f). The proposed planner exhibits stronger directionality and purposiveness towards the target, with a significant amount of perspective directivity (90% and 83.871%) falling in the interval [0.5, 1.0]. The proposed approach planned more views which have directivity in the range of [-1.0, 0.5) in the Kitchen and Dining scene because it takes more views to search for the target. In Figure 3 (i), The multi-object directivity distribution spends more views switching between different targets, but the ratio of occurrence in [0.5,1.0] still dominates. In Table 14 , the proposed approach shows a significant advantage over the other two planners by up to 0.88234 and 0.62581 in the average perspective directivity.Figure 4 shows the two original scenes in Gazebo and planned trajectories by each planner, where the green, red and blue trajectories denote the planned ones by our method, the RH-NBV and the Frontier-based one in order. The green trajectories exhibit the maximum coverage of the viewing angles of the target, circling around the target within the reachable region. Compared to the trajectories of the other two in Figure 4 (c), (d), (g), and (h), the proposed method also shows strong directionality and purposiveness towards the target and the target's surroundings evidently with its trajectory in Figure 4 (b) and (f), while the others seem like "wandering aimlessly and enjoying freedom". Due to space limitations, more visualization results are presented in the supplementary material. 
discussion and future work: 
 The proposed semantic-aware NBV scheme in this study demonstrates its advantages in search-and-acquisition manoeuvre under the complex environment over the existing informative path planners in the ROI reconstruction progress, ROI-to-full reconstruction volume ratio and perspective directivity. From the experimental results in Section 4, the RH-NBV planner also demonstrates good ROI reconstruction performance in the smaller scene but poor performance in the larger scene. The frontier-based planner exhibits a good ROI-to-full ratio at the beginning of the experiment, which profits from its pattern of pursuing the frontier voxels. After that, the planner intends to find the other unknowns, thus the ROI-tofull ratio drops. The significant difference here is that we are keen on perceiving the region we are interested in instead of pursuing unknown areas. More than 80% of the perspective directivity of the proposed approach falls in the interval [0.5, 1.0] in both scenes, while the other two distribute more average within four intervals. It means we are consistently looking towards the target's location once the target is well-located, while the other two are looking in all directions more evenly. The results also demonstrate the generalization potential of the proposed method by introducing the termination criterion to handle the multi-object search-and-acquisition. As the complexity of the manoeuvre increases, particularly when dealing with multiple objects, the proposed method offers a more exhaustive capture of the objects of interest.However, there are some limitations to this work. First of all, both the planning and reconstruction processes are based on the same volumetric map. The choice of voxel size is a trade-off between reconstruction precision and planning efficiency, i.e. real-time smooth planning (e.g. around 4 to 7 seconds per planning) results in a compromise in the reconstruction precision. The second one is that the proposed approach is less aggressive in exploring or searching in the larger area than the frontier-based planner. Thus, it takes longer to locate the target as the area increases. 
conclusion: 
 In this study, we presented a semantic-aware Next-Best-View aided by the multi-DoFs mobile system for autonomous visual perception under the complex and unknown environment. We formulate the novel semantic gain, combined with the conventional visibility gain in a unified form, to evaluate the "Next Best" view among the candidate views with the contribution of semantics. An adaptive strategy is introduced to control the mode switching between 'search' and 'acquisition' on the specific target under the challenging environment, and a termination criterion is designed to handle the target switching in multi-target visual acquisition. The capability of the proposed approach is demonstrated in three different settings in the simulation, achieving improvements of up to 27.13% for the ROI-to-full reconstruction volume ratio and a 0.88234 average perspective directivity. The planned motion trajectory is compared with the ones produced by existing planners, and a better target perceiving coverage is demonstrated evidently. 
: 
 Figure 1 : (a) When the refrigerator is designated as the object of interest, next view candidate 1 provides higher semantic gain while next view candidate 2 offers higher visibility gain; (b) A capture of observing a view with high semantic gain in the experiment, with the standing person is highlighted. making them well-suited for a wide range of applications. Specifically, autonomous visual acquisition by multi-DoF mobile systems (e.g. unmanned aerial vehicles, UAVs) in unknown and inaccessible environments has proven to be an effective means in search and rescue, reducing the need for professional remote control skills among emergency personnel. However, exhaustive observation is a timeconsuming and resource-intensive process. To ensure an efficient visual perception process, it is vital to select adaptive views that provide the most information. Next-Best-View (NBV) was initially presented for an unknown area exploration using the mobile robot [3, 25, 34, 38] , usually a finite iteration random tree is grown in the known free space, e.g., Rapidly-exploring Random Tree (RRT), RRT* [17, 22] then the best branch is selected by maximizing the gain (e.g., the amount of unobserved space that can be observed) while minimizing the moving cost (e.g., distance or time cost). After that, it was also adopted to the path planning for single object surface reconstruction [20, 21] , online inspection [27, 35, 36] and so on.However, the existing studies determining the next best view focus on information gain by evaluating the visibility of unknown voxels, regardless of their semantics. Unlike the previously mentioned scenarios, visual perception on the objects of interest under complex environments should be semantically selective rather than solely focused on perceiving the unknowns. In other words, the "Next Best" viewpoint in a complex environment cannot be evaluated effectively without the relevant semantic information. In Figure 1 (a), semantically informative views should be selected as a higher priority to ensure the efficiency of visual perception on the specific target using mobile systems.In this work, we propose a semantic-aware NBV scheme for efficient visual perception under complex environments and implement it in a two-stage search-and-acquisition manoeuvre aided by the multi-DoFs mobile system. We develop a novel information gain formulation which integrates both semantic gain and visibility gain. We also design an adaptive strategy to balance these two components so that the mobile robot can perform both search and acquisition operations on specified semantically important objects. We evaluate the proposed approach using different self-build scenarios in the simulation environment; an experiment capture is shown in Figure1(b). The results we obtained demonstrate that the proposed approach significantly improves the efficiency of visual perception on specified objects under complex environments through evaluating the reconstruction progress against region of interest (ROI) in volume, ROI-to-full reconstruction volume ratio and perspective directivity. Both the motion planning and reconstruction are implemented based on the voxblox [29] as the map representation, which employs Truncated Signed Distance Fields (TSDFs) to represent the object surface. Then, the RRT* is generated in the observed free space. To the best of our knowledge, this is the first work that investigates semantic-aware NBV for searchand-acquisition-based visual perception by mobile systems, which integrates the contribution from both semantic gain and visibility gain in a unified form for evaluating and selecting the next viewpoint. We demonstrate its capability in the application of different complex environments.The main contributions of this work include: (1) We present a novel information gain formulation for evaluating the candidate viewpoints that integrates both semantic gain and visibility gain. Such novel formulation can be applied to many other application scenarios in which the visual data acquired contain rich semantics of the complex environment. (2) We design an adaptive strategy with termination criterion to balance the semantic and visibility terms so that the mobile platform can perform an effective two-stage search-and-acquisition manoeuvre on the specified object or multiple objects under the complex environment. The principle behind this two-stage approach can also be applied to scenarios in which the objective of the task can be properly decomposed to facilitate effective implementation.(3) To assess this novel formulation, we also introduce several evaluation metrics to characterize the system performance and demonstrate the efficiency in perceiving the specific objects under the complex environment while the data acquisition mobile system is undergoing multi-DoFs motion.The paper content is organized as follows: an overview of the related work and how we step further is presented in Section 2. We introduce the proposed system and showcase its effectiveness in visual acquisition on the objects of interest in simulation experiments in Sections 3 and 4. Finally, we analyze the results obtained and draw conclusions in Sections 5 and 6. 
 
 Appendix:  


Decision: The paper presents a novel approach for efficient visual perception in unknown environments, integrating visibility gain and semantic gain to select the semantic-aware view.  The reviewers find the paper to be well-written; the experiments and their results are clearly presented.

On the other hand, the reviewers have risen the lack of some relevant information such as computational detail of the real-time application, algorithm complexity, ablation study. Even if most of these concerns seem to have been addressed in the rebuttal, there are still concerns among the reviewers who stated the need for improvements in the paper such as adding real-world experiments and dynamic scenarios but also explore deeper the correlation between segmentation accuracy and semantic gain and the need of more detailed and informative visualisation results.   If these concerns can be addressed in the camera-ready version, it would strengthen the paper and its contributions to the research community.  We encourage the authors to consider the comments of the reviewers and make an effort to improve the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a semantic-aware Next-Best-View (NBV) approach for visual perception in complex environments using multi-degree-of-freedom (Multi-DoFs) mobile systems. By integrating visibility gain and semantic gain into a unified information metric, the method optimizes viewpoint selection to enhance object detection and reconstruction. An adaptive strategy balances search and acquisition tasks. Simulation results show significant improvements in reconstruction accuracy and efficiency, making this approach highly effective for applications such as search and rescue operations.
strength_weakness: strength_weakness: 1 The paper introduces a novel information gain formulation that integrates both visibility gain and semantic gain, which is a significant advancement over traditional methods that focus solely on visibility.
2 The paper compares the proposed method with RH-NBV and frontier-based approaches, demonstrating its superiority in various reconstruction metrics.
3 The formulation of the information gain metric combines semantic and visibility gains, enhancing the theoretical foundation for NBV strategies.
limitations: limitations: 1 The integration of semantic gain and the adaptive strategy, while innovative, may introduce significant computational overhead. The paper does not provide detailed analysis or benchmarks on computational performance, which is crucial for real-time applications.
2 The method’s performance in larger and more dynamic environments is not thoroughly explored. The scalability to extensive real-world scenarios with more objects of interest or higher complexity remains uncertain.
paper_summary: paper_summary: This paper discusses the  efficient visual perception using mobile systems in unknown environments, particularly in operations such as search and rescue where swift and comprehensive perception of objects of interest is crucial. The authors point out that in complex environments, selecting the "Next-Best-View" (NBV) based solely on maximizing visibility gain is insufficient as it does not consider semantic information. Therefore, the paper proposes a novel information gain formulation that integrates both visibility gain and semantic gain to select the semantic-aware next best view. Additionally, an adaptive strategy with termination criteria is designed to support two-stage search-and-acquisition maneuvers for multiple objects of interest in complex environments. This framework includes two maps: an occupancy TSDF map and a labeled TSDF map, which provide information about the occupancy status of the environment and a detailed representation of the environment's geometry with semantics, respectively. The paper details the formulations for visibility gain and semantic gain and proposes an adaptive strategy and termination criteria to balance the search and acquisition phases.
strength_weakness: strength_weakness: 1. This paper proposes a novel information gain formulation that integrates visibility gain and semantic gain.
2. A two-stage search and acquisition framework with adaptive strategies and termination criteria is designed for multiple objects of interest in complex environments.
3. The paper has a logical structure, gradually unfolding from problem description to related work, method introduction, experimental results, and discussion.
4. Clear visualization results, such as motion trajectories and scene layouts, are provided. Experiments conducted in the Gazebo simulator provide a simulation of a real environment, enhancing the credibility of the research findings.
limitations: limitations: 1. The authors need to analyze the algorithmic complexity and conduct experimental comparative analysis on the time overhead. On resource-constrained mobile platforms, the resource and time overhead of the algorithm are of great importance.
2. The authors need to declare how semantic labels are obtained. The calculation of semantic gain depends on accurate semantic labeling, which may require complex image processing and machine learning models.
3. Although simulation experiments can provide well-controlled test conditions, the simulated environment may not fully replicate the complexity of the real world.
4. The reviewer know the challenges associated with rapidly establishing a real-world experimental platform. However, I would appreciate it if the authors could provide some guidance and analysis for constructing a real-world experiment.
5. The paper primarily focuses on static environments, and additional considerations may be needed for target tracking and view planning in dynamic settings. How should dynamic moving objects in the environment be handled?
paper_summary: paper_summary: In this paper, authors formulate a novel information gain that integrates both visibility gain and semantic gain in a unified form to select the semantic-aware Next-Best-View. design an adaptive strategy with termination criterion to support a two-stage search-and-acquisition manoeuvre on multiple objects of interest aided by a Multi-DoFs mobile system. Additionally, authors introduce several semantically relevant reconstruction metrics to evaluate the performance of the proposed approach. Simulation experiments demonstrate the advantages of the proposed approach over existing methods
strength_weakness: strength_weakness: The authors' idea of using semantic information to select the Next-Best-View is excellent. They have designed an initial method and conducted experimental validation based on this concept, achieving correct and relatively promising results.
limitations: limitations: （1）The specific representation of semantics in the methodology is unclear. The authors need to describe, and possibly visualize, how semantic representation is connected to semantic gain.（2）In the experimental section, both quantitative and qualitative aspects fail to adequately explain the proposed method and comprehensively demonstrate its advantages.（3）The temporal performance, its impact on reconstruction results, and the execution of tasks are not evaluated, making it difficult to assess the specific improvements brought by the method.（4）The authors have not compared their method with others nor conducted the necessary ablation studies to demonstrate the overall and modular performance superiority of their method.（5）Simulated experiments alone are insufficient to comprehensively and objectively prove the effectiveness of the method
paper_summary: paper_summary: The authors propose a novel approach for efficient visual perception in unknown environments, integrating visibility gain and semantic gain to select the semantic-aware Next-Best-View. They design an adaptive strategy for a two-stage search-and-acquisition maneuver on multiple objects of interest using a Multi-DoFs mobile system. Evaluation metrics demonstrate significant improvements over existing methods, with simulation experiments confirming better target coverage in planned motion trajectories.
strength_weakness: strength_weakness: 1. The idea of the method is clearly presented.
2. The motivation of the method is valid and the method seems to be effective and stable.
limitations: limitations: 1. The method is only compared with traditional frontier-based strategy and RH-NBV, which is a method proposed in 2018, without comparison with most advanced researches. However, as a type of efficient and universal traditional scheme, this is acceptable. But whatever, the evaluation is not sufficient.
2. As an universal algorithm, more comparison on complexity analysis with traditional frontier-based strategy and RH-NBV should be included.