High Fidelity Aggregated Planar Prior Assisted PatchMatch Multi-View Stereo

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 11:02:44

Abstract: The quality of 3D models reconstructed by PatchMatch Multi-View Stereo remains a challenging problem due to unreliable photometric consistency in object boundaries and textureless areas. Since textureless areas usually exhibit strong planarity, previous methods used planar prior and significantly improved the reconstruction performance. However, their planar prior ignores the depth discontinuity at the object boundary, making the boundary inaccurate (not sharp). In addition, due to the unreliable planar models in large-scale low-textured objects, the reconstruction results are incomplete. To address the above issues, we introduce the segmentation generated from Segment Anything Model into PM pipelines for the first time. We use segmentation to determine whether the depth is continuous based on the characteristics of segmentation and depth sharing boundaries. Then we segment planes at object boundaries and enhance the consistency of planes in objects. Specifically, we construct $\textbf{Boundary Plane}$ that fits the object boundary and $\textbf{Object Plane}$ to increase consistency of planes in large-scale textureless objects. Finally, we use a probability graph model to calculate the $\textbf{Aggregated Prior guided by Multiple Planes}$ and embed it into the matching cost. The experimental results indicate that our method achieves state-of-the-art in terms of boundary sharpness on ETH3D. And it also significantly improves the completeness weakly textured objects. We also validated the generalization of our method on Tanks\&Temples.

Main paper: 
 introduction: 
 Enhancing the multimedia application's user experience requires the usage of interactive technologies. Virtual reality (VR) and augmented reality (AR), two cutting-edge interactive multimedia that offer interactive experiences in three dimensions, have garnered a lot of attention lately. A fundamental challenge in creating 3D content for VR and AR is to obtain 3D geometry through multiview stereo. Numerous ideas originate from this line of thinking [27, 29, 41, 47, 52] and consistently raise the bar for reconstruction performances. These earlier techniques can be broadly categorized as traditional and deep learning-based methods.Deep learning-based methods [20-22, 44, 47] have demonstrated remarkable power in MVS due to their exceptional capacity to extract robust visual features. In some datasets (e.g. DTU [1] and Tanks&Temples [15] ), they achieve satisfactory results. However, most learning-based methods [20] [21] [22] 44] need to build the cost volume and increase the receptive field enormously when dealing with large-scale textureless regions, consequently leading to prohibitive memory consumption. Despite many efforts to reduce memory consumption [12, 24, 46, 48] , learning-based methods still struggle to handle datasets with large-scale textureless areas or high-resolution images using mainstream GPU devices (fig. 7d ). In addition, learning-based methods often rely on ground truth (GT), which is difficult to obtain on large-scale datasets. The above problems have led to poor performance of learning-based methods on the more challenging large-scale datasets (e.g. ETH3D [30] ).Many recently suggested traditional MVS methods [11, 29, 53 ] are essentially expanded versions of PatchMatch [4] , which computes the matching cost based on a plane hypothesis between a fixed-size reference patch and patches in source images. PM methods require less memory since they find suitable matches by using a propagation and local refinement strategy, which avoids the construction of cost volume. PM methods have the advantages of low memory consumption and no need for ground truth, making it suitable for reconstruction tasks in large-scale scenes. Nevertheless, the matching cost will become unreliable when a patch is located in a textureless region because the receptive field lacks useful feature information [19] . [16, 42, 45] introduce a coarse fitting plane hypothesis based on the assumption that textureless regions often occur on flat surfaces (such as floors). ACMMP [42] uses Delaunay Triangulation of reliable pixels to construct planar models for textureless regions (fig. 2a ), and embeds them as priors to the matching cost calculation. However, their boundaries are not sharp enough (office in fig. 1 ). That's because they don't fully consider the character of depth discontinuity at the object boundary, which is not in line with the plane assumption (green triangles in fig. 2c ). On the other hand, ACMMP can lead to many errors in large-scale lowtextured objects (pipes in fig. 1 ). This is due to the unreliability of pixel cost computations caused by photometric consistency failure.To develop a memory friendly method that ensures sharp boundaries and completeness of weak texture objects, we combine segmentation [14] with the planar model of ACMMP. We notice the character that segmentation and depth share boundaries [54] . And many methods have also proven the mutually beneficial relationship between segmentation and depth recovery [7, 37] . We use segmentation to determine the continuity of depth, and then construct new planar models to improve the quality of planar priors.Firstly, we use Delaunay triangulation to generate initial planar models (fig. 2a ) and use SAM (Segment Anything Model [14] ) to generate segmentation information. SAM will generate a mask for each object. We combine these masks as an identity map (Id Map, fig. 2b ) where pixels belonging to the same object have the same Id value. Secondly, we detect erroneous planar models that span different objects (fig. 2c ). Then we divide the error planar model into multiple small regions (collection of pixels with the same Id) and construct new planar models (Boundary Plane) for these small regions separately. In this way, planar models will fit the boundary of the object, making our depth map sharper. Thirdly, we . determine whether an object is reliable by using the texture and the size of the planar model in it. Then we attempt to fit an Object Plane using RANSAC for the object to enhance the consistency of these planar models (fig. 2d ). Finally, we integrate the initial plane, boundary plane, and object plane as an Aggregated Prior guided by Multiple Planes, which is added to the matching cost.In summary, we have the following contributions:• Introducing SAM into the PatchMatch pipelines to construct boundary planes and object planes has improved the quality of the planar model. • Embed aggregated prior guided by multiple planes into matching cost, making the depth sharper at object boundaries and more complete in textureless objects. • Our method has the advantages of low memory consumption, strong generalization ability, and no need for GT, so it can be applied to large-scale scenes reconstruction. 
related work 2.1 patchmatch-based mvs methods: 
 According to [31] , MVS methods are divided into four categories: voxel-based methods [34] , surface evolution [8] based methods, patch-based methods [10] , and depth map based methods [4, 33] .The depth map based method has attracted much attention due to its advantages of easy transmission and parallelization.Our method belongs to the last category, and we will only discuss this part of the content. PatchMatch multi-view stereo methods exploit the core idea of PatchMatch [2] , sampling and propagation, to effectively estimate depth maps for each image. In recent Secondly, we combine the initial planar models with the segmentation extracted from SAM to generate Boundary Plane (section 3.2). Thirdly, we determine whether an object is credible, and try to construct an Object Plane (section 3.3) for the incredible object. Finally, we integrate the initial plane, boundary plane, and object plane to generate an Aggregated Prior guided by Multiple planes (section 3.4) for each pixel, which will be embedded to the matching cost.years, [11, 29] have significantly improved the effectiveness of PMbased methods. ACMH designs an adaptive checkerboard sampling strategy to propagate more reliable hypotheses. ACMM [43] incorporates pyramid structure and geometric consistency into PM, allowing it to effectively analyze rich textured regions and small textureless regions but still being unable to estimate the depth for large-scale textureless regions very well. On the basis of ACMM, APD-MVS [39] adaptively deform the patch of unreliable pixel to extend the receptive field until it covers enough reliable pixels. APD-MVS greatly improves the completeness of depth maps, but its performance at boundaries is poor. Differently, [16, 28] divide images into superpixels and fit a plane for each superpixel. ACMP [45] uses the Delaunay Triangulation of reliable pixels to propose a coarse planar model for large-scale textureless regions. It adds the planar model as a prior to patch matching. ACMP significantly improves completeness in textureless regions. ACMMP [42] integrates ACMP and ACMM, resulting in improved performance. However, ACMP and ACMMP don't consider that the depth near the object boundary is often discontinuous, and the coarse plane in such a depth discontinuous area will provide an incorrect prior. Moreover, due to the unreliable matching cost calculation and difficulty in finding reliable pixels in large-scale textureless objects, the planes on these objects are often wrong.Learning-based techniques have become popular when [47] used deep learning for depth map estimation. Many works are dedicated to reducing the heavy calculation of cost volume with a coarse-to-fine strategy [12, 24, 46, 49] and RNNs [48] . Meanwhile, some researches formulate a more reliable cost volume, such as the visibility of ViS-MVSNet [51] . To achieve a more robust feature extraction, AA-RMVSNet [40] proposes an adaptive aggregation module implemented using deformable convolution. [9, 38] introduce the transformer structure into the MVS task to obtain global feature information. Transformer-based [6, 9, 18, 23, 38] introduce finely designed external structures for feature extraction but do not completely use the geometric clues embedded in the MVS scenarios [52] . Learning-based methods use networks to learn features, resulting in performance surpassing traditional methods on some datasets [1, 15] . However, in large-scale scene datasets [30] , there are problems such as the inability to obtain accurate and complete ground truth, as well as huge memory consumption, which leads to poor deep learning methods. Traditional methods still are SOTA on these datasets due to their advantages of low graphics memory consumption and strong generalization ability.There are also unsupervised methods such as [3, 13, 17, 25, 26] , which have made significant progress. However, they only use images as supervised signals, which leads to their inability to generate correct geometric information.  
method: 
 Given a set of images {𝐼 𝑖 } 𝑀 𝑖=1 and the corresponding camera parameters {𝑃 𝑖 } 𝑀 𝑖=1 , our algorithm estimates the depth map of each image. For reference image 𝐼 0 with its source images {𝐼 𝑖 } 𝑁 -1 𝑖=1 , We first use the conventional PM to obtain the initial depth map, and based on this, we use Delaunay Triangulation to obtain the initial planar models (section 3.1). Secondly, with the assistance of segmentation information from SAM (section 3.2), generate boundary planar models (section 3.2) and object planar models (section 3.3). Thidly, we generate an aggregated prior guided by multiple planaes and add it into matching cost (section 3.4). Finally, we obtained the depth map by iterating the PM process.3.1.1 PatchMatch. Firstly, PM performs random initialization, generating depth and normal vectors randomly for each pixel in 𝐼 0 as the initial planar hypothesis. The second step is propagation. For pixel p, PM use the plane hypotheses of eight pixels in its neighborhood as the hypothetical space. Thirdly, calculate the matching cost for each planar hypothesis in the hypothetical space. The reference patch is represented by a square window 𝑩 𝑝 that is centered on p. PM assumes that pixels in 𝑩 𝑝 are all on the same plane. Given a planar hypothesis 𝜃 𝑖 = [𝑛 ⊤ 𝑖 , 𝑑 𝑖 ], the projected patch 𝑩 𝑗 𝑝 on the source image 𝐼 𝑗 for 𝑩 𝑝 can be obtained by homography [32] formula. PM obtains the dissimilarity score between 𝑩 𝑝 and 𝑩 𝑗 𝑝 through one minus the NCC score [42] . After aggregating each dissimilarity score using viewpoint weights [42] , PM obtains the final matching cost 𝑐 𝑝ℎ𝑜𝑡𝑜 (𝜃 𝑖 ). For each planar hypothesis in the hypothetical space, we use the above process to obtain the corresponding cost. The fourth step is refinement, where PM selects the planar hypothesis with the minimum cost and optimizes it by introducing perturbations and randomness. PM use the optimized hypothesis as the new planar hypothesis for p. By iterating through the last three steps, PM can acquire a final depth map. Matching cost calculation is a key step in pipelines as it is crucial for the accuracy of depth maps.Prior. Here we review the planar prior in our baseline ACMMP. The planar prior is divided into two steps. Firstly, ACMMP constructs the planar model. Secondly, ACMMP calculates the planar prior and adds it to the matching cost calculation of PM.Specifically, ACMMP uses above PM process to generate an initial depth and corresponding cost map. Then, it select the pixels with the lowest local cost as the reliable points. It takes these points as vertices and uses Delaunay Triangulation to obtain triangles where adjacent triangles share the vertices (fig. 2a ). Each triangle represents a planar model in the reference camera coordinate system. The normal vector of a planar model and the distance from the origin of the camera coordinate system can be calculated based on the camera coordinates of the three triangle vertices. ACMMP assumes that the pixels inside the triangle are all located on the corresponding planar model. It calculates the depth and normal vector𝜃 𝑝 = [𝑛 ⊤𝑝 , 𝑑 𝑝 ] for each pixel based on the planar model. ACMMP calculates a planar prior, i.e. a prior probability, for each pixel by calculating the difference between 𝜃 𝑝 and its planar hypothesis where 𝜆 𝑑 is the bandwidth of depth difference, 𝜆 𝑛 is the bandwidth of normal difference. The higher the probability value, the closer the 𝜃 𝑖 is to 𝜃 𝑝 , and the more credible the 𝜃 𝑖 is. Next, ACMMP incorporates planar priors into the matching cost calculationEQUATIONEQUATIONWhen the photometric consistency cannot reflect hypothesis changes in textureless areas, the planar prior will play a major role in the hypothesis updating. 𝛼 is a empirical constant [42] .We input 𝐼 0 into SAM, which will generate masks for objects. After performing merging of these masks, we will get an identity map (Id Map) where pixels belonging to the same object will have the same Id value, as shown in fig. 2b . Then we use Canny [5] operator to extract initial contours on Id Map. For robustness, we will take a circular neighborhood with 11 centered around the initial contour pixel and label all pixels as contour pixels.We use the planar model constructed using the method in section 3.1 as the initial planar model. However, we can notice that many planes crossing the boundaries of objects (green triangles in fig. 2c ), which may construct many wrong oblique planes with part points in the foreground and part points in the background. These planes will cause errors in the cost calculation of the boundary area. To this end, we will detect these erroneous planar models and repair them using the correct planar models from their neighborhoods.First, we use contour to determine whether the triangle crosses the boundary. If the number of contour pixels in a triangle exceeds the threshold 𝛼 𝑛𝑏 , it is determined to be a boundary triangle. Otherwise it will be judged as a non-boundary triangle. For a nonboundary triangle 𝑡 (red triangles in fig. 2c ), we regard it as a correct initial planar model and will use it to fix the boundary triangle. We use the center of gravity[ 𝑥 1 +𝑥 2 +𝑥 3 3 , 𝑦 1 +𝑦 2 +𝑦 3 3] to represent its position 𝑝𝑜𝑠 𝑡 . And calculate the pixels' Id distribution within the 𝑡, taking the Id with the highest number of pixels as 𝑖𝑑 𝑡 .For a boundary triangle, we divide it into multiple regions (fig. 4 ). A region is a collection of pixels with the same Id. Here lies a natural Then we aggregate the credibility to obtain the credibility of the blue object. For an incredible object, we sample low-cost points in planar models with high credibility (Cyan triangles) and run RANSAC to fit an object plane for it.assumption: we believe that the depth changes of pixels belonging to the same object are often continuous, while the depth changes of pixels belonging to different objects are often discontinuous. So we use nearby non-boundary triangles to help construct planar models for these regions. For a region 𝑟 , we use the average coordinates of all pixels in it as the location of the region 𝑝𝑜𝑠 𝑟 . Next, we calculate the distance between the region and the non-boundary triangle belonging to the same object. We add the triangles with distance calculated from eq. ( 3) less than 𝜀 to the set 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑟 .𝑑 = ∥𝑝𝑜𝑠 𝑟 -𝑝𝑜𝑠 𝑡 ∥ 2 .(EQUATIONFor the pixel 𝑝 in 𝑟 , its possible planar models are 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑟 . We will calculate a weight for each plane in 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑟 that takes into account distance and matching costs. For 𝑝𝑙𝑎𝑛𝑒 𝑗 in 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑟 , we define its weight asEQUATIONwhere 𝜃 𝑏 𝑗 represents the planar hypothesis when 𝑝 is on the 𝑝𝑙𝑎𝑛𝑒 𝑗 , and 𝑑 represents the distance calculated by eq. ( 3). The negative exponential form ensures that the output value is within the range of 0 to 1. 𝛽 is a small constant that makes it convenient to set the weight of a planar model with cost exceeding the threshold to 0. This can mitigate the impact of outliers on weight calculation. And because the difference between distances is often much greater than the difference between costs, we use a large constant 𝛾 to control the impact of distance on weights. The closer the distance and the lower the matching cost, the more likely the pixel 𝑝 is to be on the 𝑝𝑙𝑎𝑛𝑒 𝑗 , and thus the weight of 𝑝𝑙𝑎𝑛𝑒 𝑗 will be greater. We construct multiple possible planar models 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑟 for a region and calculate weights 𝒘 𝑏 for each planar model. They will be applied in section 3.4 to construct our aggregated prior guided by multiple planes.In addition to the errors in depth discontinuous areas, we found that on large-scale textureless objects, the quality of the planar model is also poor. That's because severe low texture can lead to unreliable depth calculation, and also make the selection of reliable points difficult. Ultimately, it leads to a decrease in the quality of the planar model. Therefore, we detect the credibility of object reconstruction results and attempt to construct an object plane to improve the quality of unreliable plane models.3.3.1 Object Credibility Detection. We notice that objects are composed of multiple planar models, which in turn are composed of multiple pixels. Therefore, we define the credibility of pixels, planar models, and objects in ascending order.According to [28] , we define the credibility of a pixel 𝑝 asEQUATIONwhere 𝑉 𝑎𝑟 is the variance of the 5×5 patch around the pixel. A larger variance means that the features around the pixel are richer, and the probability of calculating the correct depth through matching cost is also higher, so it is considered that the credibility of the pixel is higher. 𝜖 𝑣𝑎𝑟 and 𝑡 𝑚𝑖𝑛 are two empirical values, and the same values as [28] are used here. For a planar model 𝑚 in object 𝑜, we define its credibility asEQUATIONwhere 𝑁 𝑚 and 𝑁 𝑜 represent the number of pixels in 𝑚 and 𝑜, respectively. The first term represents the average credibility of pixels in 𝑚. The greater the first term, the higher the credibility of 𝑚. This is because the construction of a planar model depends on the initial depth and cost generated by the conventional PM process. The stronger the texture is, the more reliable the matching cost calculation of PM will be, making the planar model more reliable. The 𝑁 𝑚 /𝑁 𝑜 in the second item represents the proportion of the plane model to the object. The larger the 𝑁 𝑚 /𝑁 𝑜 is, the lower credibility of 𝑚 will be. That's because, compared to small planar models, large planar models mean difficulty in selecting reliable points. Finally, by aggregating the credibility of planar models, we can obtain the credibility of object 𝑜 asEQUATIONwhere 𝑁 𝑢 /𝑁 𝑜 represents the proportion of the area without a planar model to the size of the object. We use 𝜆 to control the weights of the two terms. There are many areas at the boundaries of the image that do not have a planar model, so when calculating the credibility of the 𝑜, this also needs to be taken into account. We mark objects with 𝑐𝑟𝑒𝑑𝑖𝑡 𝑜 below the threshold 𝜏 as unreliable objects.For an unreliable object 𝑜, we have a pixel set 𝑅 = {𝑝 |𝑝 ∈ 𝑚, 𝑚 ∈ 𝑜, 𝑐𝑟𝑒𝑑𝑖𝑡 𝑝 > 𝜉, 𝑐𝑟𝑒𝑑𝑖𝑡 𝑚 > 𝜋, 𝑐𝑜𝑠𝑡 𝑝 < 𝜌 }. This is equivalent to selecting the low-cost credible points from the credible planar model, and we run RANSAC algorithm on 𝑅.In each iteration, we randomly select three pixels from 𝑅 and construct a plane with their world coordinates, and calculate the interior point rate of 𝑅 on this plane. We choose the plane with the highest interior point rate as the final plane. If the interior point rate of the final fitting plane is more than 𝜑, it indicates that the object has successfully constructed an object plane. Otherwise, we believe that the object is not a plane.If an object plane is successfully constructed, each pixel in the 𝑜 will have an object planar model 𝑝𝑙𝑎𝑛𝑒 𝑜 . And we calculate a weight for 𝑝𝑙𝑎𝑛𝑒 𝑜EQUATIONwhere 𝜃 𝑜 represents the planar hypothesis when 𝑝 is on the 𝑝𝑙𝑎𝑛𝑒 𝑜 , and 𝛾 𝑜 is a constant used to align with the weight of other planar models (boundary or initial planar models). 𝑤 𝑜 will be applied in section 3.4 to construct aggregated prior guided by multiple planes.So far, we have three types of planar models, namely the initial planar model, the boundary planar model that enhances boundary sharpness, and the object planar model that enhances the completeness of weakly textured objects. We will generate an aggregated prior guided by multiple planes and embed it into the calculation of matching cost in this section.Our innovative matching cost assisted with aggregated prior guided by multiple planes is derived through a probabilistic graphical model. To construct the graphical model, we define the patch on pixel 𝑝 as 𝑋 𝑟𝑒 𝑓 . Also, the patches observed on all source images via 𝜃 𝑖 are 𝑿 𝑠𝑟𝑐 𝑖 , and the visibility information of all source images is assumed to be 𝒁 𝑠𝑟𝑐 . 𝜃 𝑝 represents the initial plane, 𝜽 𝑏 represents the boundary planes (there may be more than one boundary plane) and 𝜃 𝑜 represents object plane. A pixel may have one or two types of planar models, represented by Θ, which will be reflected in eq. ( 13).The fig. 6 shows the graphical model of our approach. The joint probability isEQUATIONIn this way, the maximum a posteriori estimate of the plane hypothesis 𝜃 * is given byEQUATION)The above posterior can be factorized asEQUATION)Next, the likelihood function can be defined as follows,EQUATION)This function encodes the photometric consistency, making the low multi-view aggregated photometric consistency cost have a high probability. To solve errors in object boundaries and largescale low-textured objects, we define different planar prior based on whether three types of planar models are constructed as followEQUATION)Each row of formulas corresponds to a situation of constructing planar models. The first and second rows indicate that only the initial plane or boundary planes are provided respectively, the third row indicates that both the boundary plane and object plane are provided, and the fourth row indicates that both the initial plane and object plane are provided. 𝑃 (𝜃 𝑖 |𝜃 𝑝 ), 𝑃 (𝜃 𝑖 |𝜽 𝑏 ), and 𝑃 (𝜃 𝑖 |𝜃 𝑜 ) can be calculated by eq. ( 1). 𝑤 𝑏 𝑗 and 𝑤 𝑜 represent the weight of boundary planar model 𝜃 𝑏 𝑗 and object planar prior 𝜃 𝑜 , respectively. They can be calculated in section 3.2 and section 3.3. 𝑤 𝑝 is a constant used to represent the weight of the initial planar model. Finally, we substitute eq. ( 11), eq. ( 12), and eq. ( 13) into eq. ( 10) and take the negative logarithm algorithm to get the following aggregated prior guided by multiple planes assisted matching costEQUATIONNote that, the first term that encodes photometric consistency is the main component in the above equation. This indicates that in well-textured areas, the photometric consistency will change more obviously than the planar prior [45] . When the photometric consistency cannot reflect hypothesis changes in low-textured areas, thanks to the aggregated prior, our cost calculation can be more accurate at object boundaries and within low-textured objects. We also use the same pyramid structure as ACMMP. After iterating through the process of PM, we can obtain the depth. 
experiments 4.1 datasets and results: 
 To verify the effectiveness of our method in large-scale scenarios, we use high-resolution images in ETH3D [30] and Tanks&Temples [15] in our experiments. ETH3D is a more challenging benchmark. Firstly, it has a more diverse range of scene types (from man-made indoor and outdoor scenes to natural scenes containing a large amount of vegetation), which requires more generalized methods to avoid overfitting. Secondly, the complexity of the scene and the drastic changes in viewpoints result in a large number of weak textures, occlusion, and other difficult reconstruction areas. Finally, extremely high image resolution (6,048 × 4,032) also requires memory and computation efficient methods. Learning-based methods often fail in such challenges, however, due to the low memory consumption and excellent generalization ability of PM-based methods, the SOTA under the ETH3D benchmark are still PM-based methods.Tanks&Temples dataset also contains large-scale scenes but has a smaller resolution (about 1,920 × 1,080). We use it to demonstrate the generalization ability of our method. All our experiments are performed on a single NVIDIA Tesla V100 GPU. We use fusion technique like [42] to obtain point clouds on ETH3D dataset. fig. 1 shows a qualitative comparison, and it is evident that our method takes into account both the sharpness of object boundaries and the completeness of textureless objects. The quantitative analysis is displayed in table 1 with the first group being based on learning and the second being traditional. Furthermore, only a few learning-based approaches, such [24, 36, 51] , can successfully complete the reconstruction task on this dataset due to the high resolution of the images (fig. 7d ). However, their performance is still insufficient. We also validated the generalization of our method on Tanks&Temples. It can be seen that our method significantly improves the performance of the baseline.For all methods, we set the number of source pictures to 10 and the image size to 6, 221 × 4, 146 (ETH3D) at 100% resolution (8.04% corresponds to Tanks&Temples) in order to compare memory costs. As shown in fig. 7d , compared to the learning-based method (blue), PM methods (yellow and red) has significantly less memory consumption. Compared to the baseline, our method hardly increases From playground in fig. 7 , the baseline constructs many erroneous planar models near the boundaries of trees, while our planar models are more closely aligned with the boundaries of trees. The quality of the planar model will affect the quality of the depth map. From office in fig. 1 , it can be seen that our depth map has significantly sharper boundaries and successfully reconstructed the stack of books behind the chair, which also indicates that our method has better reconstruction ability in complex scenes. We also conducted a quantitative analysis of the boundaries. We only fuse the depth information located at the object boundary (the contour in section 3.2) on ETH3D. As shown in table 3 , our method has best 𝐹 1 scores at the object boundary compared to other methods.Our method consists of two modules, namely boundary planar prior (BP) and object planar prior (OP). From playground in fig. 7 , we can see that the boundary plane is more closely aligned with the object boundary. From meadow in fig. 7 , it can be seen that there are a large number of errors in the initial planar model on large-scale weakly textured objects such as grasslands. Our method can detect these low credibility objects and fit them to an object plane. We also conducted quantitative analysis on ETH3D. In table 4, boundary planar prior can improve both completeness and accuracy, while object planar prior mainly enhances completeness. When we combine two modules, we can achieve the best performance. To demonstrate the performance of our algorithm at the textureless object, we only fuse the depths located at these objects (The object judged to be incredible and successfully constructed Object Plane for it). We can see quantitative and qualitative experimental results in table 5 and pipes in fig. 1 , respectively. 
conclusion: 
 We present aggregated prior guided by multiple planes in this study and implement PM-based MVS technique, HFP-MVS. We introduce segmentation of SAM into the PM pipelines for the first time. We greatly enhance the accuracy of depth at object boundaries and the completeness of depth on large-scale weakly textured objects. Meanwhile, our method has the advantages of low graphics memory consumption and strong generalization, making it more suitable for reconstructing large-scale scenes (such as ETH3D). In addition, our method has strong interpretability and can serve as a prior for learning-based methods to improve their performance. However, for non-planar large-scale low-textured objects, our algorithm's planar prior quality remains unassured. Future research might address this issue by using a curved surface prior.  
high fidelity aggregated planar prior assisted patchmatch: 
 Multi-View Stereo Anonymous Author(s) Submission Id: 1917 Figure 1 : Qualitative results, boundary performance and runtime on ETH3D [30] . In office (a-d), with the help of boundary planes, we have more accurate (sharper) depth at the edge of the chair and reconstruct the stack of books behind the chair successfully. In pipes (a-d), we have more complete depth within large-scale weakly textured objects such as walls and floors due to object planes. (e) The y-axis represents 𝐹 1 score at the object boundary, and the x-axis represents the runtime. Compared to APD-MVS [39] with the best overall performance, we have a significant advantage both in boundary 𝐹 1 and runtime. 
 
 Appendix:  


Decision: This paper originally received contrasting ratings, 2 BA and WR, and after rebuttal they became all positive, 3 BA.
Reviewers criticisms regard the request of clarification of some part of the methodological pipeline and, above all, the experimental analysis, performance, and lack of significant ablations and weal comparative analysis.

After rebuttal, all the raised shortcomings seem reasonably addressed, leading to all positive ratings. For this reason, this paper can be accepted for publication to ACM MM 2024 conference.

Reviews: review: paper_summary: paper_summary: This paper focuses on improving multi-view stereo (MVS) reconstruction by addressing challenges in object boundaries and textureless areas. The paper presents a SAM-based framework, High Fidelity Aggregated Planar Prior Assisted PatchMatch Multi-View Stereo (HFP-MVS), which introduces segmentation from the Segment Anything Model (SAM) into PatchMatch pipelines. The paper proposes to use a probability graph model to calculate an Aggregated Prior guided by Multiple Planes, enhancing depth accuracy at boundaries and completeness in textureless objects.
strength_weakness: strength_weakness: - The paper proposes a method that combines the Segment Anything Model (SAM) to assist the PatchMatch process, enhancing the accuracy and completeness of the model by introducing Boundary Plane and Object Plane.
- The proposed method maintains low runtime and memory consumption and is suitable for handling large-scale scenarios.
- The paper includes many experiments and comparisons to evaluate the performance of the proposed method.
limitations: limitations: - As shown in Table 1 and Figure 7 (d), the experimental results of the proposed algorithm do not have obvious performance gain over the existing work like [1], the memory consumption of the proposed method is also higher.
- Although the authors stated in the abstract that they were the first to introduce SAM into PM pipelines, similar approaches have been adopted in previous work [2].

[1] Wang Y, Zeng Z, Guan T, et al. Adaptive patch deformation for textureless-resilient multi-view stereo[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Yuan Z, Cao J, Li Z, et al. SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024.
paper_summary: paper_summary: This paper improves ACMMP by introducing the results of the Segment Anything Model (SAM) to correct and optimize the planar prior generated in ACMMP: 1) planes spanning different objects are split, 2) planes belonging to the same object are reaggregated. The corrected planar prior then guides the MVS to achieve better reconstruction results.
strength_weakness: strength_weakness: 1. By applying the results of SAM to obtain the edges of the objects and splitting the planar prior that spans multiple objects, the incorrect planar priors that can cause negative effects to MVS will be reduced, thus improving the accuracy of the reconstruction.
2. By aggregating planar priors, this method can better handle large-scale textureless objects.
3. This method does not introduce much more additional runtime and memory consumption compared to ACMMP, but achieves better results than ACMMP.
limitations: limitations: 1. There is no special treatment for non-planar areas. If some textureless areas are non-planar and they cannot be separated using SAM, such aggregation may lead to worse results.
2. To demonstrate the improvement at the boundary areas, it is better to give a visual result of the point cloud, which is more intuitive.
3. There is no detailed explanation of how the boundary score is calculated. From my understanding, the edges are all strong textures. If only the depths on the edges are fused, the scores of these methods will not be so different unless the depths near the edges are also fused.
paper_summary: paper_summary: This proposed model HFP-MVS introduces Segment Anything Model (SAM) into the PatchMatch pipelines to construct boundary planes and object planes has improved the quality of the planar model. It constructs boundary plane that fits the object boundary and object plane to increase consistency of planes in large-scale textureless objects. Then, it uses a probability graph model to calculate the aggregated prior guided by multiple planes and embed it into the matching cost. Extensive experimental results demonstrate the effectiveness of the proposed model HFP-MVS.
strength_weakness: strength_weakness: 1. Introducing SAM into the PatchMatch pipelines to construct boundary planes and object planes has improved the quality of the planar model.
2. Extensive experimental comparisons and ablations are conducted on various widely-used benchmark datasets, which demonstrate that the proposed model HFP-MVS consistently outperforms state-of-the-art methods.
limitations: limitations: 1. This paper uses a large-scale Segment Anything Model (SAM) with powerful capacities of generalization and discrimination to generate segmentation information for reconstruction performance. Compared with other segmentation models, SAM can provide more accurate segmentation information, which naturally leads to better results for the task of multi-view stereo. But in the section of ablation study, this paper lacks a direct comparison over the impact of different segmentation models for the final results. Thus, it is difficult to identify the source of the performance improvement.
2. This paper does not provide sufficient explanation and discussion on experimental settings. For example, in the section 4.1, there is a sentence "All our experiments are performed on a single NVIDIA Tesla V100 GPU" in this paper.
3. In the Main Result, only one work from 2023 is chosen for quantitative analysis. It should be appropriate to add recent works. 
4. Some bugs. Some undefined abbreviations exist in the sections of Abstract and Introduction, such as “PM” and “MVS”. In section 1, the clear definition of the MVS task is not given. This could lead to ambiguity for readers trying to understand the incremental benefits of the proposed approach HFP-MVS over existing MVS methods.