Reconstructing, Understanding, and Analyzing Relief Type Cultural Heritage from a Single Old Photo

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 19:46:53

Abstract: Relief-type cultural heritage objects are commonly found at historical sites but often manifest with varying degrees of damage and deterioration. The traditional process of reconstructing these reliefs is laborious and requires extensive manual intervention and specialized archaeological knowledge. By utilizing a single old photo containing predamage information of a given relief, monocular depth estimation can be used to reconstruct 3D digital models. However, extracting depth variations along the edges is challenging in relief scenario due to the highly compression of the depth values, resulting in low-curvature edges. This paper proposes an innovative solution that leverages a multi-task neural network to enhance the depth estimation task by integrating the edge detection and semantic segmentation tasks. We redefine edge detection of relief data as a multi-class classification task rather than a typical binary classification task. In this paper, an edge matching module that performs this novel task is proposed to refine depth estimations specifically for edge regions. The proposed approach achieves better depth estimation results with finer details along the edge region. Additionally, the semantic and edge outputs provide a comprehensive reference for multi-modal understanding and analysis. This paper not only advances in computer vision task computer vision tasks but also provides effective technical support for the protection of relief-type cultural heritage objects.

Main paper: 
 introduction: 
 Relief-type cultural heritage objects, commonly found at historical sites worldwide, are important in humanistic areas such as art, culture, history, and architecture. Unfortunately, these valuable cultural heritage objects often suffer from varying degrees of damage and loss caused by natural or human factors [54] . While scanning and photogrammetry technology can achieve permanent three-dimensional (3D) digital preservation of their current states, they cannot restore their appearance prior to damage and loss [60] . The traditional process of restoring these reliefs is laborious, requiring extensive manual intervention and specialized archaeological knowledge [24, 56] . Moreover, reconstruction methods for typical 3D cultural relics often involve deep learning-based point cloud completion processes to reconstruct damaged and missing structures. However, these methods are generally suitable only for scenes with simple structures and minimal damage [16, 44, 47] . Unfortunately, these may not be applicable to relief-type cultural heritage objects, which are typically complex in their geometric structure and extensive in scale and quantity.Fortunately, many of these precious reliefs are often documented in old photos, as shown in Figure 2 . These 2D monocular old photos can serve as very effective references for the 3D reconstruction of relief-type cultural heritage objects. First, relief scenes are different from 3D scenes such as sculptures or 2D scenes such as paintings, and are more like special 2.5D scenes. Second, the reliefs are usually only meant to be viewed from the front or side. Therefore, these two characteristics allow a single image taken, from the front, to effectively cover most of the content, providing sufficient information for monocular algorithms. [3] ; (b) Yungang Grottoes, partially damaged [66] ; (c) Borobudur Temple, overall lost [39] . Old photos with predamage information, and photos of the present situation, are presented on the top and bottom, respectively.Pan et al. first proposed a 3D reconstruction method based on monocular depth estimation to reconstruct buried reliefs in the Borobudur temple [31] . Although they achieved 95% in reconstruction accuracy, their result lack detailed fine structures of the carved items, e.g., human figure facial features and many kinds of decorations. This limitation is due to the inadequate extraction of depth variations in the edge regions that form the fine structures. In a relief scenario, the depth values are significantly compressed compared to other 3D scenarios, leading to less pronounced changes along the edges. These edges cannot be easily detected from 2D images, as discussed in Section 2. Pan et al. identified this characteristic as "soft edge" and extracted this unique information based on the curvature changes in 3D space [20] . Subsequently, a soft edge map was straightforwardly input into their network as auxiliary information [32] . Their experimental results demonstrated that soft edge information effectively improves the depth estimation task in relief data. However, from our perspective, there are three critical limitations that still need to be addressed:(1) The edge map is extracted from 3D points and used as an additional input, which significantly reduces the accuracy of their proposed model on the test data. This decrease in accuracy occurs because, during the testing phase, 3D points from an old photo are unavailable, thereby leaving no effective method to obtain an accurate edge map. (2) The soft edge map is incorporated into the network in the later part of the decoding stage, which limits its impact on the depth estimation task. (3) They represent the soft edge map as a binary image to distinguish between edge and non-edge areas as a mask for further processing. However, the binary representation fails to convey the 3D curvature changes inherent in soft edges, which could provide effective clues for depth estimation task. In this paper, we propose solutions to the aforementioned limitations using a multi-task neural network equipped with a novel edge matching module that performs a newly defined soft edge extraction task. For the first limitation, the proposed method, based on multi-task learning (MTL), is capable of performing both soft edge detection and depth estimation tasks within the same neural network. Upon proper training, this neural network can provide accurate edge information for test data through its edge detector. To address the second limitation, we design a novel edge matching module that directs the depth estimation task to focus more intensively on the soft edge region, thereby yielding more detailed depth estimation result. For the third limitation, we redefine soft edge detection as a multi-class classification problem, aimed at determining the degree of "softness" of specific edge regions. This approach preserves the crucial curvature information, thus enhancing the effectiveness of the depth estimation task.Following the work of Pan et al., we introduce one of the UN-ESCO World Heritage Sites, Borobudur Temple in Indonesia, as our experimental subject. The temple features 2,672 bas-relief panels on its walls, constituting the largest collection of Buddhist reliefs in the world. Unfortunately, due to safety concerns, the temple's foot encasement was reinstalled, and 156 relief panels carved into this encasement were obscured by stone walls and are now hidden and invisible. For each of these hidden panels, a grayscale photograph taken in 1890 is preserved (refer to Figure 2 ) [5, 39] . We digitize the visible reliefs into 3D models and establish a training dataset including RGB inputs, depth, semantics, and soft edge labels to train our proposed model. The results of our quantitative and qualitative experiments demonstrate that our method provides a more detailed depth map than Pan et al. and other state-of-the-art (SOTA) depth estimation models, with richer detail in the edge regions, leading to superior 3D digital reconstruction models. Furthermore, our proposed multi-task neural network also delivers semantic segmentation and soft edge extraction results, achieving performance comparable to SOTA models and enhancing multimodal understanding and analysis of relief data.We summarize the contributions of this paper as follows: (1) We propose a novel multi-task network that enables the reconstruction, understanding, and analysis of relief-type cultural heritage from a monocular old photo; (2) We introduce a novel edge matching module within the network that performs a newly defined soft edge detection task, enhancing the details in the edge regions of depth estimation results and thus enabling the reconstruction of more accurate 3D digital models; (3) We propose a dynamic edgeenhanced loss function to optimize the proposed neural network; (4) We apply this method to reconstruct the Borobudur hidden reliefs their remaining old photos, thereby aiding in the preservation of this invaluable cultural heritage site. 
related work: 
 Relief Reconstruction and Generation: The traditional process of reconstructing relief-type cultural heritage objects into 3D digital models is laborious [24, 56] . Deep learning-based reconstruction of relief objects has rarely been studied, Pan et al. first proposed a monocular depth estimation-based approach to reconstruct the relief from a single monocular photo [31] , and improved the results by involving soft edges in their later work [32] . However, there are still limitations that remain to be addressed, as we describe in Section 1. However, deep learning-based reconstruction is a relatively common for other 3D relics. With scanned data of intact objects, it is possible to reconstruct cultural heritage objects by point generation approaches to fill the missing structures [16, 44, 47] . However, at present, these methods are only suitable for small areas of broken artifacts with simple geometries, thus they are not very versatile for relief scenes.Additionally, there are related studies on relief generation task (including noncultural heritage objects). Some use 3D models for depth compression to generate reliefs that do not meet the standards for the reconstruction of relief-type cultural heritages from old photos [18, 19, 63] . However, it is worth mentioning that the effect of edge information in relief has also been proven in these works. JI et al [19] demonstrated that hierarchical relationships change dramatically with a gradual decrease in depth, resulting in continuous degradations of the original details. The proposed edge optimizations in their work improved the model-to-relief reconstruction results. Other image-based methods generally rely on other prior inputs [28, 37, 43] or human-computer interactions (HCIs) [61] . Moreover, these studies are generally optimized for a few specific simple characters, such as plants, human faces or other objects with similar geometric structures [64] .Monocular Depth Estimation: Deep learning-based methods have become the mainstream solution for monocular depth estimation task because they effectively learn depth representations in an end-to-end manner. Eigen et al. proposed the first depth estimation network, which is a multi-scale fusion network, to regress the depth value [13] . Following this, considerable improvements have been made by utilizing or modifying superior network backbones [1, 15, 25] , carefully designing the regression task as a classification task [6, 27] , introducing more priors [35, 36, 52] and better objective functions [53, 62] . However, despite their promising performance, they are difficult to generalize to unseen domains, especially in relief scenes with unique data features.Recently, several works apply diffusion model-based methods to monocular depth estimation [12, 40] . By taking advantage of conditional diffusion models [21] , zero-shot depth estimation has made breakthroughs [58] . However, these models require large amounts of training data at their initial stage. Marigold is specially designed for tuning with small amounts of data, but 74k is still needed [21] . As data collection of cultural heritage objects is often limited by protection purposes or data security, no large public relief dataset is currently available. Moreover, as shown in our experimental results, the details in depth result cannot be estimated properly by these models. To address these demands, our research approach is to preparing various labels on very few sample data to perform related tasks with depth estimation based on effective MTL architecture.Edge Detection: Different from contours and boundaries, which correlate with semantically meaningful entities, edge detection aims to capture all significant intensity discontinuities in an image [67] . Traditional edge detectors [7, 23] , learning-based methods [11, 29] , convolutional neural networks [33, 46] , and recent approaches utilizing vision Transformers [34] have been explored for edge detection in 2D images. These methods focus on identifying notable brightness changes and classify pixels into two categories: edges and non-edges. However, as discussed in Section 1, soft edges, which are more indicative of 3D curvature changes, cannot be adequately represented by a binary classification. Kawakami et al. [20] extracted soft edges in reliefs using a 3D feature highlighting method that employs point opacity provided by a transparent rendering technique [49, 51] . Subsequently, Pan et al. [32] project these 3D soft edges onto a binary edge map and utilized it as an additional input for their depth estimation network. To further leverage soft edge information, we project 3D soft edges onto multi-class edge maps and use these as labels to train a deep learning-based edge detector designed for a multi-class classification task, thereby defining a novel soft edge detection task.Multi-task Learning: Multi-task learning (MTL) enhances task performance compared to independent single-task training by leveraging shared information and representations across tasks [8] . To improve MTL performance, numerous efforts have focused on designing encoder architectures [42, 50] , predicting intermediate auxiliaries [57] , and developing novel loss functions [10] . Several studies have demonstrated improved performance by jointly estimating depth and semantic segmentation [14, 48] . Ji et al. [17] further established the relationship between depth and semantics in relief scenes in their work on semantic segmentation. Inspired by these findings, we incorporate semantic segmentation into our MTL approach to jointly predict depth, semantics, and edges. Additionally, to enhance the interplay between the depth estimation and edge detection tasks, we have developed a novel edge matching module within our proposed MTL architecture. 
method 3.1 overview: 
 As shown in Figure 1 , we propose a multi-task neural network to extract various features involving the information of depth, semantics, and edges from a single old photo of a given relief-type cultural heritage object. These three kinds of feature maps provide the multi-modal data foundation for understanding and analysis of the relief scenario. Moreover, by predicting a corresponding dense depth map from a single old photo, the 3D digital model can be reconstructed. This effectively addresses the widespread issue of varying degrees of damage and loss in relief-type cultural heritage objects, because the predamage information is often documented in their historical photographs.The key to improving the depth estimation results of relief data lies in extracting more information on subtle depth variations at the edge regions. Utilizing edge images to effectively assist in the depth estimation task is a highly effective method. As we discussed in 1 Introduction, Pan et al. [32] has made preliminary attempts, but there is still significant room for improvement. Therefore, we propose a novel method based on multi-task learning to address this issue.First, the soft edge extraction task is newly defined in Section 3.2 to better extract and represent the data features of the relief-type cultural heritage objects. Second, a multi-task neural network with novel architecture is proposed in Section 3.3 to effectively utilize the soft edge information. The network can extract the depth, semantic, and edge features, and further constrain the depth estimation task to focus on the soft edge regions using a proposed edge matching module. Finally, the dynamic edge enhanced loss function is proposed in Section 3.4, so that the proposed network can be better optimized. As shown in Figure 3 , traditional edge detection tasks treat the detection of 2D edges as a binary classification problem, segregating each pixel into either edge or non-edge regions. In public datasets, the differential brightness within edge regions reflects the degree of consensus among humans during the manual labeling process [30] . This method suffices for depicting brightness transitions along edges in conventional scenes. However, the edges in relief data signify not merely changes in brightness but variations in curvature change across 3D space. Therefore, we propose to define the edge detection of relief data as a multi-classification task to better express this important feature. As shown in Figure 3 , by employing a 3D edge highlighting technique [20] , soft edge regions can be detected and subsequently projected onto a edge map. This map features continuous pixel values ranging from 0 to 255. Unlike related studies that directly binaries the map for use as a mask, we categorize the pixels into six distinct classes based on their values as detailed in Table 1 . For a given class n, the pixel value is determined by the formula 51 × n. This task is executed by the proposed soft edge detector (see Section 3.3), which is optimized by a meticulously refined loss function (see Equation 10). Overall structure: The network structure of our proposed method is shown in Figure 4 . The network follows an encoder-decoder design to perform monocular depth estimation task and semantic segmentation task, and is additionally constrained by a novel edge matching module, which performs the soft edge detection task proposed in Section 3.2. The encoder is composed of a modified SwinV2 Transformer encoder following EMSAFormer [14] and a heavyweight VIT encoder following SAM [22] . Please note that the parameters of the SAM encoder are frozen in this work because our dataset is not sufficient for overall fine-tuning. We utilize the SAM encoder to leverage its zero-shot feature extraction capability to obtain rich features. To merge the extracted features from the two independent encoders, two patch merging layers are set behind the SAM encoder to adjust the resolution and the number of channels of the intermediate results. Then, the adjusted features are fused through cross-attention to the features obtained from the SwinV2 Transformer encoder, followed by a context module. Moreover, triple skip connections are used to retain the low-level features and input to the two decoders. Additionally, the SwinV2 Transformer encoder incorporates an extra input edge image, which is output by the proposed soft edge detector of the edge matching module. The specific information about the edge matching module, soft edge detector, and two task decoders are as follows.Edge matching module: As shown in Figure 4 , the proposed module comprises two soft edge detectors designed to individually extract multi-class soft edge maps from the input monocular photo and the output depth map, respectively. Both detectors share identical structures, detailed in the subsequent paragraph. Moreover, the matching loss between the two produced soft edge maps is calculated in this module, which is part of the optimization object of the network following Equation 5. The logic behind this module is predicated on the notion that, only when the deep prediction results are sufficiently accurate can the edge information of details be extracted. This will lead to an increased similarity between the two edge result images, which in turn reduces the penalty on the optimization objective. This module effectively imposes a directed constraint on the depth prediction task, making it focus more on extracting the details outlined by the soft edges.Soft edge detector: The proposed soft edge detector is modified from LDC [45] , which is a lightweight network with just 674K parameters. As shown in (c) of Figure 4 , the detector follows a CNN architecture with four intermediate edge maps (F1 to F4); hence, the final result comes from the fusion of these maps through four skip connections. The output soft edge map follows our proposed multi-classification representation, and the pixel values reflect the curvature change of the soft edges in 3D space. This lightweight backbone is selected because we are trying to complete a more complex edge detection task with a limited dataset. The parameters pretrained on the MDBD dataset [30] are utilized in this paper. Instead of making excessive adjustments to the detector structure, we propose a meticulously refined loss function to complete the newly defined task, following Equation 10 and Equation 11.Semantic decoder and depth decoder: The decoders for the semantic segmentation task and depth estimation task are designed to suit the specific needs of each task. As shown in (a) and (b) of Figure 4 , each decoder consists of a series of convolutional layers, batch normalization layers and upconvolutional layers. Moreover, the feature maps, with multiple scales generated by skip connections as we mentioned above, are fused with the intermediate results of both decoders via the concatenation operation. For semantic segmentation, the decoder projects to the number of semantic classes. For depth estimation, the decoder regresses continuous pixel values ranging from 0 to 255. The proposed multi-task learning network optimizes a novel dynamic edge-enhanced loss function. The total loss is joined by three tasks, depth estimation, soft edge detection and semantic segmentation, following Equation 1:EQUATIONwhere 𝐿 𝑠𝑒𝑚𝑎𝑛𝑡𝑖𝑐 , 𝐿 𝑑𝑒𝑝𝑡ℎ and 𝐿 𝑒𝑑𝑔𝑒 denote the loss functions for the tasks of semantic segmentation, depth estimation and soft edge detection, respectively. 𝛼, 𝛽 and 𝛾, respectively, represent the weight coefficients for the three tasks.For the semantic segmentation task and depth estimation task, we utilize the cross-entropy loss and the Silog loss [13] following the related works, respectively:EQUATION𝐿 𝑑𝑒𝑝𝑡ℎ (𝑑EQUATIONwhere 𝑛 denotes the number of pixels with valid ground truth values, 𝑐 denotes the number of categories, which is set to 4, 𝑠 ′ denotes the label and 𝑠 denotes the predicted probability for each category after applying the softmax function. 𝑑 denotes the prediction result of depth estimation, and 𝑑 ′ denotes the ground truth of depth estimation. 𝜆 𝑑 is set to 0.15 to be invariant to global-scale changes in the predicted depth map.For the soft edge detection that we defined in this paper, a novel dynamic loss function is proposed. The total loss in this task should consider three terms: (a) the edge detection accuracy 𝐿 𝑟 predicted from the input monocular image, (b) the edge detection accuracy 𝐿 𝑑 predicted from the output depth map and (c) the matching loss 𝐿 𝑚 of the two predicted soft edge maps. The total loss 𝐿 𝑒𝑑𝑔𝑒 of the soft edge detection task can be expressed as:EQUATIONEQUATIONEQUATION)where 𝑙 is the loss function for the soft edge detector, with 𝑒 𝑟 ,𝑒 𝑑 ,𝑒 𝑔 representing the edge maps detected from the input monocular image, the edge maps detected from the input monocular image, and the ground truth. decoder is generally poor and uncertain, from which meaningless soft edge detection results are produced. If these results are used to calculate 𝐿 𝑑 and 𝐿 𝑚 , it will lead to instability 𝐿 𝑒𝑑𝑔𝑒 and total loss L, thereby affecting the learning effectiveness of each task and the entire network. Therefore, the dynamic coefficient 𝑘 is adopted to adjust the weights of 𝐿 𝑑 and 𝐿 𝑚 based on the accuracy of the depth estimation. This dynamic coefficient 𝑘 is set as follows:EQUATIONwhere d represents the accuracy of the depth estimation result using the 𝐿 1 norm, which fluctuates primarily within the range of 0.08 to 0.45. To ensure that the coefficient 𝑘 can exhibit a substantial dynamic range within the primarily active range of d, the scaling coefficient 𝜆 𝑘 is set to 𝜆 𝑘 = 2.5, and the tanh operator is adopted.To effectively obtain the soft edge map defined in Section 3.2, we propose a refined classification loss function 𝑙 as follows:EQUATIONEQUATIONwhere 𝑒 denotes the predicted result, 𝑒 ′ denotes the ground truth. 𝜆 𝑒 is a scaling factor, which is set to 0.2.In the typical classification task, different categories are independent and unordered, with equal distances between each other. However, as shown in Table 1 , the soft edge detection task that we define is based on ordered pixel values. For example, the difference in pixel values between Class 2 and Class 0 is less than the difference in pixel values between Class 5 and Class 0. However, there will be no difference when calculating loss using a typical cross-entropy loss function, which is the problem that we want to address. To make the penalty predicting Class 5 greater than the penalty predicting Class 2 for the given pixel of Class 0, 𝐿 1 function is added as a penalty term behind the typical cross-entropy loss. 
implementation details: 
 The proposed network was implemented in PyTorch and trained on a Quadro RTX 6000 GPU with 24 GB of GPU memory. The weights of the SAM ViT encoder were initialized by the ViT-B SAM model [22] (not trainable), while the weights of the SwinV2 Transformer encoder were pretrained on ImageNet. For training, we employed the Adam optimizer in combination with the OneCycleLR learning rate scheduler. This scheduler had a maximum learning rate of 0.01 and a percentage start of 0.2, spanning a total of 100 epochs. Our model was trained on the Borobudur relief dataset provided by Pan et al. [32] , consisting of 6,424 patches cropped from only 11 images with 3072 × 1024 pixels. The total training duration exceeded 34 hours, with a batch size of 4 and an input size of 640 × 512 pixels. To avoid the risk of overfitting, we augmented the images prior to feeding them into the network by applying Gaussian noise, performing random resizing and cropping, adjusting random HSV jitter, and applying horizontal flipping. For comparative experiments, we adopted the following parameter settings in the proposed loss function to achieve the best performance on depth estimation task. For Equation. 1, 𝛼 is set to 1.0, 𝛽 to 3.0, and 𝛾 to 5.0. For Equation. 5, we set 𝜆 1 to 1.0, 𝜆 2 to 0.5, and 𝜆 3 to 0.05. In this section, the effectiveness of both the proposed edge matching module and conducting the soft detection task defined as a multi-class classification problem are evaluated. We demonstrate quantitative and qualitative experimental results on monocular depth estimation and edge detection when different settings of edge information are used in the proposed network. The results of the semantic segmentation were not significantly impacted by these settings. However, our proposed method still managed to achieve competitive results in the semantic segmentation task on relief data, details of which will be discussed in Section 4.3.In our ablation studies, Exp 1 did not utilize the proposed edge matching module or any edge information. Exp 2 to Exp 5 all utilized the proposed edge matching module, each with different settings: Exp 2 involved binary edge, Exp 3 involved multi-class edge, Exp 4 involved continuous edge, and Exp 5 involved multi-class edge but without using the proposed dynamic control parameter 𝑘 in the proposed loss function as Equation 5.As shown in Figure 5 and Table 5 , the proposed edge matching module demonstrates the most significant improvement in extracting details from the edge regions. Compared to other experiments, Exp 1 displays the worst performance, with the highest error rates and the least detail in the edge regions. The newly defined soft edge detection task has also proven effective, as Exp 3 yields more accurate depth results in the edge regions than Exp 2, resulting in lower error metrics as indicated in Table 5 . While Exp 2 did manage to extract some edge details, its depth value accuracy is inferior to that of Exp 3 when compared to the Ground Truth (GT). Please pay particular attention to the changes in depth values of the leaves and human ears in Figure 5 . Additionally, the soft edge maps predicted in Exp 3, 4, and 5 present finer edges compared to the binary edge maps predicted in Exp 2, which provides more effective reference for understanding and analyzing relief data.Moreover, we also observed that Exp 3 was achieved easier convergence on our limited dataset compared to Exp 4 and 5. This underscores the effectiveness of multi-class edge classification and the introduction of the dynamic control parameter 𝑘. Exp 4 and 5 exhibited poor convergence resulting in sub-optimal error rates, with Exp 4 specifically yielding unsatisfactory extraction results for continuous edge maps as illustrated in Figure 5 . Please note that the quantitative results of the edge maps on Exp 2 are for reference only due to the use of different ground truth.In this section, we present both quantitative and qualitative comparison results of our method with other state-of-the-art approaches in depth estimation and semantic segmentation tasks as shown in Table 3 , Table 4 and Figure 6 . More results are provided in our supplementary materials.For the depth estimation task, we compare our results with soft edge-enhanced network [32] , BTS [26] , AdaBins [6] , DenseDepth [2] , Swinmim [55] , and the zero-shot model Depth Anything [59] . We observed that, except for the soft edge-enhanced network proposed by Pan et al., which only managed to capture a small portion of the fine structure, other state-of-the-art models designed for public datasets failed to capture the fine structure of the relief, particularly in the soft edge regions. The zero-shot model, Depth Anything, also failed to generalize in relief scenes, with barely any details extracted. In Figure 6 , we illustratively present detailed results from Pan et al., Swinmim, Depth Anything with our results. We achieved the best depth estimation results, capturing the clearest details necessary for forming the fine structure without requiring additional prior information on the test data. The quantitative comparisons presented in Table 3 further substantiate our claims of superior performance compared to other models. Additionally, we reconstruct relief-type cultural heritage into 3D point clouds based on the depth estimation result and calculate the cloud-to-cloud distance with the GT on our validation data. For the semantic segmentation task, we compare our results with those of JI et al. [17] , SegNet [4] , U-Net [38] , PSPNet [65] , DeepLabV3+ [9] , EMSANet [41] , and EMSAFormer [14] . As illustrated in Table 4 , our proposed method outperforms most related works, with the exception of the approach proposed by JI et al. This exception can be attributed to the use of both depth labels and soft edge labels as additional inputs in the model proposed by JI et al., which significantly enhanced the performance of their model. Furthermore, our dataset containing 11 relief panels is smaller than that used by Ji et al., which includes 26 relief panels. The proposed method has been applied to the Borobudur temple, extracting multi-modal feature maps, including depth maps, from old photo that recorded the appearance of 156 buried relief panels. This approach has enabled the 3D digital reconstruction of these panels, as illustrated in Figure 1 . While there is no ground truth available for quantitative comparisons of the buried relief objects, we provide more qualitative comparison results in our supplementary materials. 
conclusion: 
 In this study, we developed a multi-task learning-based method to predict multiple feature maps containing depth, semantics, and edges from a single old photo. The proposed method not only provided comprehensive references for multi-modal understanding and analysis but also enabled 3D digital reconstruction of relieftype cultural heritage objects. Through the optimization measures proposed in this paper, the limitations of related studies were effectively solved, and more precise 3D reconstruction models were provided, which were quantitatively and qualitatively verified on the Borobudur dataset. For future work, we may employ a point cloud completion process over the 3D reconstruction model obtained in this work to repair small portions of the side structural information that cannot be covered within a monocular old photo. Furthermore, we aim to extend our method to additional damaged or lost relief-type cultural heritage sites and develop a more comprehensive relief dataset.  
 
 Appendix:  


Decision: This study presents a unified framework designed to reconstruct single photos of relief-type cultural heritage objects, addressing challenges such as low resolution and varying degrees of damage and deterioration. The key innovation is a multi-task network that leverages three types of maps—edges, semantics, and depths—together for the reconstruction task. Experiments on public datasets showed impressive results compared to state-of-the-art methods.

The manuscript received three reviews, all recommending acceptance. The chair agrees with the reviewers that the originality of the proposed method meets the standards of ACM MM.

Reviews: review: paper_summary: paper_summary: This paper proposes a multitask model to produce three types of feature maps (depth, semantics and edges) which are then used for better predicting depth in the context of 3D digital reconstructions from 2D images.

The main application of their method is the 3D reconstruction of damaged relief-type cultural heritage objects, whose original versions are often documented in old monocular photos. They explore the prior that such objects are flat on one side, which allows their 3D model to be obtained from a single monocular picture.

The authors tackle the limitations of a previous work on 3D reconstructions from monocular pictures, namely the lack of precision in the estimation of soft edges. This is done a through multitask network, which learns both soft edge detection and depth estimation, combined with a novel edge matching module to focus specifically on soft edges.

Their model was tested on the Borobudur relief dataset, which contains more than six thousand patches of relief objects extracted from eleven high resolution monocular images.
strength_weakness: strength_weakness: The proposed method is well described, and the experimental settings are adequate. The experimental results on the Borobudur dataset show the superiority of the proposed method with respect to eight state-of-the-art depth estimation methods, including a recent approach published in CVPR 2023.
limitations: limitations: In the "Edge detection" subsection, the authors cite some methods for binary edge detection, but other contour detection models which output more than binary edges are not mentioned, such as: 

Maninis, Kevis-Kokitsi, et al. "Convolutional oriented boundaries: From image segmentation to high-level tasks." _IEEE transactions on pattern analysis and machine intelligence_ 40.4 (2017): 819-833.

Liu, Yun, et al. "Semantic edge detection with diverse deep supervision." _International Journal of Computer Vision_ 130.1 (2022): 179-198.

Since the extraction of edges and semantics is one of the main contributions of this paper, a broader discussion on other non-binary edge detectors should be included.

Figure 4 contains a lot of information, and some of the text inside it is blurry.

To better evaluate the proposed model against state-of-the-art methods, it would be relevant to include experiments on standard depth datasets, such as Cityscapes and KITTI.
paper_summary: paper_summary: This paper introduces an innovative method for three-dimensional digital reconstruction, understanding, and analysis of relief-type objects in cultural heritage, based on a single old photograph. Utilizing a multi-task neural network, it predicts multi-modal feature maps that include depth, semantics, and edges, which are crucial for enhancing the information processing capabilities of computer vision tasks. Our network is capable of extracting features ranging from simple geometric details to complex semantic layers, and it adeptly handles edge detection, a task that is often challenging to achieve simultaneously in traditional single-task models.

Moreover, their approach focuses on preserving and restoring damaged or partially lost relief artworks by reconstructing accurate 3D models from archival photographs, thus playing a vital role in the protection and study of cultural heritage. This paper also demonstrates how these reconstructed models can be used for further analyses, such as assessing the degree of material wear and exploring historical transformations, offering new perspectives and tools for comprehensive cultural heritage research.

Their method has been applied to several real-world heritage cases, including the famous reliefs at Borobudur Temple, validating its effectiveness and practicality. Experimental results show significant improvements over current state-of-the-art methods in terms of reconstruction quality, semantic understanding, and edge clarity, thereby confirming its potential contribution to the preservation and study of global cultural heritage.
strength_weakness: strength_weakness: Innovative Approach: This paper leverages an innovative multi-task neural network that predicts multi-modal feature maps including depth, semantics, and edges from a single old photograph. This approach not only enhances traditional image processing capabilities but also integrates multiple complex tasks that are typically handled separately. The ability to simultaneously manage multiple aspects of computer vision within a single framework significantly advances the field, particularly in the context of cultural heritage where detailed and accurate reconstructions are crucial.

Comprehensive Application to Cultural Heritage: The methodology proposed in this paper is particularly beneficial for the cultural heritage field, where preserving the integrity and understanding the context of artifacts is essential. By enabling high-precision 3D reconstructions from merely a single photograph, the method offers a powerful tool for historians, archaeologists, and conservators. These reconstructions can serve multiple purposes, from virtual tourism to detailed analytical studies, thus broadening the scope of how cultural heritage is interacted with and appreciated.

Extensive Experimental Validation: The paper provides extensive empirical evidence of its method's efficacy through its application on a variety of real-world objects, including intricate reliefs from the Borobudur Temple. The experiments compared favorably against current state-of-the-art techniques, showing superior performance in reconstruction accuracy, semantic interpretation, and edge definition. This extensive testing not only substantiates the robustness of the approach but also demonstrates its adaptability to different types of cultural heritage objects and conditions.

Practicality in Preserving Artifacts: The research addresses a critical need in cultural heritage preservation — the ability to accurately reconstruct and analyze ancient artifacts that may no longer be fully intact. This capability is invaluable for restoration efforts and provides a digital means to preserve artifacts' details indefinitely, which is especially important for those at risk of deterioration or further damage.
limitations: limitations: Complexity of Implementation: One significant drawback of the proposed method is its implementation complexity. The multi-task neural network, while robust, requires significant computational resources and expertise in machine learning to deploy effectively. This complexity could limit the accessibility of the technique, especially for institutions with limited technical infrastructure or expertise. Additionally, the complexity might hinder real-time applications or on-site use where computational resources are more constrained.

Dependence on High-Quality Input: The success of the method heavily relies on the quality of the input photograph. While the paper demonstrates impressive results with high-quality images, the performance in scenarios where the available photos are degraded or suboptimal—common issues in archival materials—is not thoroughly addressed. This dependence raises concerns about the method's applicability across a broader range of real-world conditions where old photographs might be faded, blurred, or otherwise impaired.

Limited Generalization Across Diverse Artifacts: Although the paper provides validation through experiments on selected datasets, there is a noticeable gap in demonstrating the method’s effectiveness across a more diverse range of cultural heritage objects. Different artifacts might have varying surface textures, materials, and degradation states, which may affect the model's ability to generalize effectively. The paper could benefit from a broader evaluation that includes a wider variety of artifact types to truly assess the robustness and versatility of the proposed method.

Potential Overfitting and Model Bias: The paper does not sufficiently address the risks associated with overfitting to the specific types of data used in training, particularly given the highly specialized nature of the data involved in cultural heritage. There is also a risk of bias in model predictions due to this overfitting, which could lead to inaccurate reconstructions of less-represented artifact types or styles, potentially skewing historical interpretations.
paper_summary: paper_summary: The paper deals with the specific object of relief-type cultural heritage. It proposes a solution for 3d reconstruction of the relief using a single old photograph. The proposed approach apprehends the relief as a 2.5D scene and exploits multiple modalities to tackle the challenge of fine depth structures reconstruction. The proposed method both helps virtually reconstruct potentially damaged elements but also offers a novel basis for analyzing such contents.
strength_weakness: strength_weakness: First of all, the paper is quite clear and pretty well written. The position of the proposed approach within the existing literature. The presentation of the existing limitations and the description of the proposed solutions to each of them are very clear and well thought out. 

The proposed idea of exploiting a multi-task approach has merits. Indeed, each output serves a global goal of reconstruction and helps in the soft-edge detection but also ensures that multiple outputs are created, thus increasing the analysis potential and helping understanding these specific contents. 
More specifically, the proposed soft-edge detector applied to both the 2D image and the output depth map, ensuring a better detection of soft edges and a more accurate 3D reconstruction overall, is very interesting.

The thought process on the loss used in the model is quite interesting and well presented. The idea of the dynamic control parameter for the loss makes a lot of sense to justify the use of the soft-edge detector module with both ends as input. 

The overall evaluation seems to prove the merits and performance of the approach. Visual results in the supplementary material are also quite convincing. Overall, both the scientific interest and the proposed approach are certain.
limitations: limitations: Although the paper is interesting and the proposed approach has merits, some aspects should be addressed in my opinion to improve overall understanding.

First of all, some minor corrections would be beneficial. First, the final sentence of the abstract is both strange and generic, maybe it should be modified or removed. Second, the light grey legend on top of figure 4 is quite unreadable, a bit darker would be better. Third, in section 4.2, mention is made of table 5 rather than table 2 in my opinion…

A second aspect that brought up questions is the classification of the soft-edge map. Although the idea is interesting and the corresponding modified loss is clear, the reason as to why there is only 6 classes and why they are created with such values is unclear. It seems arbitrary and I feel it should be justified.

A third aspect of the paper that should be addressed is the description of the network. The fact that finetuning is not an option is not problematic but I felt that the textual description of the network sometimes does not give more information than Figure 4. It would be interesting to know why such encoders have been used and not others. Textually describing the presence of convolutional layers or skip connections for instance is not informative, especially when the network is fully represented in Figure 4. Instead, more information on which classes are used for the semantic segmentation or insight on whether or not more semantic classes could lead to a better depth map could be beneficial.

Finally, mentions to the supplementary material could be made in several places to help mitigate the impression of lacking information. In my opinion, this could mostly be useful when quickly introducing the dataset (whose creation is well detailed in the Supp. Mat.) and also at the end, when presenting visual results, to guide readers towards the very interesting full-size visual comparisons of the reconstructions.

Typos : 
- l. 28 : “high compression” rather than “highly compression” ?
- l. 241, 783, 785, etc… : JI  rather than Ji ?
- l. 334 : strange reference to section 1.