OSNeRF: On-demand Semantic Neural Radiance Fields for Fast and Robust 3D Object Reconstruction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 12:18:52

Abstract: By leveraging multi-view inputs to synthesize novel-view images, Neural Radiance Fields (NeRF) have emerged as a prominent technique in the realm of 3D object reconstruction. However, existing methods primarily focus on global scene reconstruction using large datasets, which necessitate substantial computational resources and impose high-quality requirements on input images. Nevertheless, in practical applications, users prioritize the 3D reconstruction results of on-demand specific object (OSO) based on their individual demands . Furthermore, the collected images transmitted through high-interference wireless environment (HIWE) leads to negatively impact the accuracy of NeRF reconstruction, thereby limiting its scalability. In this paper, we propose a novel on-demand Semantic Neural Radiance Fields (OSNeRF) scheme, which offers fast and robust 3D object reconstruction for diverse tasks. Within OSNeRF, semantic encoder is employed to extract core semantic features of OSOs from the collected scene images, semantic decoder is utilized to facilitate robust image recovery under HIWE conditions, lightweight renderer is employed for fast and efficient object reconstruction. Moreover, a semantic control unit (SCU) is introduced to guide above components, thereby enhancing the efficiency of reconstruction. Demonstrative experiments demonstrate that the proposed OSNeRF enables fast and robust object reconstruction in HIWE, surpassing the performance of state-of-the-art (SOTA) methods in terms of reconstruction quality.

Main paper: 
 related work: 
 To bring out the motivation of OSNeRF and highlight its superiority against existing methods, we provide a comprehensive investigation about traditional 3D reconstruction method and NeRF-based reconstruction method in this section. The proposed OSNeRF pipeline. the collected images are initially processed by a semantic encoder, which compresses the semantic features for transmission in HIWE. Following this, the semantic decoder recover the images, which are then forwarded to a lightweight renderer designed to facilitate fast and robust 3D object reconstruction.Traditional 3D reconstruction methods encompass both active and passive techniques, each offering distinct approaches to capturing spatial information [19] . In the active method, a structured light source is projected into the scene to determine target locations by extracting its projected information within the scene. the authors in [20] discuss the application of multi-sensor data fusion techniques with high accuracy indoor object modeling. To realize dynamic object reconstruction, [21] proposed an efficient direct tracking on the truncated signed distance function and leverage color information to estimate the pose of the sensor. Passive method utilizes ambient environmental cues, like natural light reflections, combined with images captured by cameras and analyzed through specific algorithms to generate 3D data, offering simplicity and high feasibility compared to active methods. Among these techniques, Photometric-Stereo facilitates the determination of normal vectors by utilizing stereo vision principles [22] . It achieves this by analyzing multiple images captured under varying lighting conditions but with consistent viewpoints. Similarly, the Shape From Shading (SFS) stands out in discerning surface orientations through meticulous examination of light and shadow variations [23] . Moreover, Multi-View Stereo (MVS) is a key method for recovering 3D structure by exploiting differences in the projected positions of the same 3D points observed by multiple cameras [24] . However, traditional 3D reconstruction methods are hampered by several limitations, including subpar performance in water environments [25] and with small objects [26] , as well as substantial time that impede real-time reconstruction capabilities [27] . Traditional 3D reconstruction methods typically rely on image or point cloud data acquired from a limited number of viewpoints. This sampling restricts an accurate representation of the object, especially in occluded or detail-rich areas.Neural Radiance Fields (NeRF) represent a groundbreaking approach that learns the neural radiance field representation of a scene, enabling the synthesis of realistic novel views from limited 2D image observations. By modeling the geometry and appearance relationship of the scene, NeRF achieves high-quality reconstruction and rendering of complex 3D scenes. The remarkable performance of NeRF has inspired numerous extensions and explorations across diverse domains of 3D reconstruction. Notable extensions include human reconstruction [28] [29] [30] , dynamic object reconstruction [31] [32] [33] [34] [35] , and realization of reconstruction of large scenes [36] [37] [38] [39] , among others. These advancements showcase the versatility and potential of NeRF as a foundational framework. Nevertheless, as the complexity of the scene increases, the reconstruction process requires more hardware resources and time. To address this issue, some research efforts have focused on enhancing the rendering speed of NeRF. For instance, FastNeRF [40] introduces a novel light sampling strategy that dynamically adjusts the number of sampling points, leading to reduced repetitive calculations and improved model speed. Similarly, PlenOctrees [41] discretizes the continuous volume density and color function into a sparse octree structure, eliminating the need for redundant reasoning during real-time rendering. These methods primarily offer architectural improvements within the NeRF framework. It is important to note that the redundant information present in the input images can significantly impact reconstruction speed. Therefore, recent research has started exploring the utilization of semantic information in images to achieve more efficient reconstruction. A semantic-driven NeRF editing method is proposed in [42] , which encodes texture editing in 3D space. Sem2NeRF [43] improves rendering accuracy by encoding semantic masks into latent codes that control 3D object representation. Other methods, such as those presented in [13, 44, 45] , integrate 3D space and semantic space modeling to enhance the model's ability in scene semantic editing and realistic rendering. However, it is worth noting that all of the aforementioned approaches do not consider the distortion of input images in high-interference wireless environments (HIWE), thereby lacking robustness in practical applications. 
proposed scheme: 
 In this section, we will elaborate on the proposed OSNeRF as Figure 1 , which comprises the following constituents: 1) On-demand data collector, 2) semantic encoder and decoder, 3) Lightweight renderer, and 4) semantic control unit.The training of NeRF typically necessitates a substantial volume of input images and corresponding scene geometric information. However, for a particular demand, the user's attention may be solely directed towards reconstructing specific facets of the scene or particular vantage points. For example, firefighter's demand is to 3D reconstruct hydrants and buildings within risk scene, the ondemand data collector has the ability to selectively gather images pertaining to the hydrant and building objects present, and employ them as training data for the model. Consequently, the model's focus will be sharpened on acquiring a profound understanding of the objects' visual characteristics and structural attributes, thereby enhancing its performance on the reconstruction task. By employing a on-demand data collector, we can tailor the collection and utilization of data associated with a designated task, thereby augmenting the model's efficacy in relation to that specific demand. This methodology concurrently streamlines the scale and intricacy of the training data, whilst affording the model an opportunity to concentrate more intently on the pivotal aspects of the demand, thereby bolstering the efficiency of the training process.The semantic encoder is deployed on the data transmitter to facilitate the transmission of compressed semantic features, thereby eliminating redundant information while preserving the quality of the 3D reconstruction. The encoding process involves semantic segmentation, semantic feature extraction, and semantic feature compression.Initially, the collected raw images are normalized to enhance convergence speed and minimize computational loss. Subsequently, the normalized multi-perspective images I M and on-demand indicators O are fed into fully convolution networks 𝐹 𝜆 for further processing. i.e.,EQUATIONwhere M symbolizes the generated mask with dimensions, each element indicates whether a pixel is included or excluded as part of the OSO (On-demand Semantic Object). 𝛼 quantifies the degree of overlap between the mask and the actual annotation. 𝛽 represents the category label of the OSO. Additionally, we can calculate the predicted category label as β = 𝛾 ( M, α, O), where 𝛾 (•) is a function that generates class label predictions based on the indicator O, we accomplish semantic segmentation and obtain the desired OSO through the training process. Subsequently, the Swin Transformer [46] is utilized to extract semantic features from the input images, which are segmented into uniform patches via patch partitioning and linear embedding. These patches are sequentially processed through multiple layers of Transformer modules, each incorporating a window-based local attention mechanism. This mechanism selectively focuses on interactions among adjacent blocks, thereby decreasing both computational and memory complexities. The attention formula can be expressed as followsEQUATIONwhere Softmax(•) denotes an activation function. B, L, Z are the input embeddings obtained by linear transformation, 𝜌 is the adjustment factor. The semantic features extracted from the input images are represented as (𝑓 1 , 𝑓 2 , ..., 𝑓 𝑙 ), where 𝑙 denotes the total number of semantic features.To enhance the adaptability of semantic feature transmission across varying demands in high interference wireless environments (HIWE), we have developed a semantic-aware method that generates a feature weight vector based on both the user demand and the input data. This vector quantifies the importance of each semantic feature relative to the user demand. In our semanticaware approach, we employ Grad-CAM [47] to produce a heat map. The classifier's output probability vector is represented as g = [𝑔 1 , . . . , 𝑔 𝑙 , . . . , 𝑔 𝑐 ], the partial derivative feature vector (weight vector) 𝑖 𝑙 is obtained by𝑖 𝑙 = 𝜕𝑔 𝑙 𝜕𝑓 𝑙, where the gradient information reveals the sensitivity of the feature vector 𝒇 𝑙 to the user demand. A higher value of 𝑖 𝑙 ∈ [0, 1] indicates greater importance of the feature for fulfilling user demand. The feature weight vector of all the 𝐾 user demands is obtained as followsEQUATIONwhere 𝜆 𝑗 denotes the weight of the 𝑗-th demand, 𝑖 𝑙,𝑗 is the sensitivity vector corresponding to the 𝑗-th demand. Subsequently, the mask layer can abandon the redundancy semantic features while preserving the core semantic features (CSF) with higher 𝒘, which are most relevant to task demand and can be repressed by (𝑟 1 , 𝑟 2 , . . . 𝑟 𝑚 ) and 𝑚 is the total number of the CSF, thus achieving intelligent semantic feature compression. Specifically, the semantic compression is responsible compressing the redundant semantic features in the full semantic feature. The redundant semantic features refers to the semantic feature that are easily predicted based on image or are useless for driving the object reconstruction task. During semantic compression, we train a feature shared by the transmitter and the receiver to represent redundant information. Notably, the semantic features most relevant to the NeRF task and will not be compressed during semantic routing. Although the above method is a lossy feature compression. However, due to the introduction of basic knowledge [48] , the performance of OSNeRF remains unaffected by this process since the core semantic features for the reconstruction can be near-perfectly recovered in HIWE.For robust wireless transmission in HIWE, the power normalization layer is used to map the compressed semantic feature to We obtain the reconstructed images from the semantic decoder, which are subsequently inputted into the lightweight renderer. Each pixel in the image is associated with a ray r(𝑡) originating from the camera, determined by the camera parameters and defined as r(𝑡) = o +𝑡d. Here, o represents the origin of the light source (i.e., the camera's position), 𝑡 is the parameter along the ray, expressed as a scalar, and d is the direction of the ray corresponding to the pixel. We sample multiple points along the ray and provide them, along with their respective directions, as inputs to the neural network 𝐹 𝜃 . This allows for the prediction of both the color c 𝑟 and the depth 𝑑 𝑟 , which can be obtained as followsEQUATIONwhere 𝑁 denotes the number of samples sampled uniformly between the near and far planes, 𝛼 𝑖 𝑟 = 1exp -𝜎 𝑖 𝑟 𝛿 𝑖 𝑟 and 𝑇 𝑖 𝑟 =𝑖 -1 𝑗=1 1 -𝛼 𝑗 denote the transmittance and alpha value of each sampled point, respectively. Subsequently, we employ volume rendering [49] to generate a feature map, which serves as a neural network approximation of the radiance field. This representation captures the color and volume densities at each point and in each viewing direction within the scene. Consequently, the static object is effectively modeled as a continuous vector function, denoted asEQUATIONwhere x = (𝑥, 𝑦, 𝑧) denotes the the spatial coordinates of a point within a three-dimensional space. d denotes the observation direction. 𝐾 represents the number of channels within our feature vector. Through the process of volume rendering, we calculate the feature vector for each ray 𝑟 using the equation f 𝑟 = 𝑁 𝑖=1 𝑇 𝑖 𝑟 𝛼 𝑖 𝑟 f 𝑖 𝑟 . By selectively rendering the feature vectors for a subset of rays, we generate a feature map denoted as F. Additionally, we render a low-resolution depth value, denoted as D. Both F and D are stored in buffer for subsequent optimization. By utilizing the buffer, which contains the previous 𝐿 feature maps {F 𝑡 -𝐿 , . . . , F 𝑡 -1 } and the depth map{D 𝑡 -𝐿 , . . . , D 𝑡 -1 }, we leverage the stored semantic information to guide the selection process for determining the current sampling position. This approach significantly enhances the rendering speed. Specifically, by combining the current viewpoint's feature map with the maps in the buffer, we can generate low-resolution feature maps. Additionally, we employ a strategy similar to [9] to further reduce the number of sampling points, resulting in faster rendering of the generated low-resolution feature maps.We subsequently project the 3D point cloud directly onto a highresolution to generate feature maps that boast enhanced resolution and precision. i.e.,EQUATIONSubsequently, the reprojected high-resolution feature maps F𝑡 ′ →𝑡 are connected to the up-sampled feature maps F𝑡 and mapped onto the output multi-view images:EQUATIONAs shown in figure 3 , the core concept behind the proposed lightweight renderer is to optimize image rendering from the current viewpoint by utilizing previously restored low-resolution features and depth information stored in a buffer. The efficient renderer utilizes the reprojected feature maps at a higher resolution, along with the upsampled feature map, to generate the final image. After the low-resolution feature rendering, sample range optimization, and reprojection of previous frames, we proceed to minimize the loss function by contrasting the color of output image with the input image, which can be expressed asEQUATIONwhere 𝑀 1 represents the number of images, while 𝑀 2 corresponds to the quantity of pixels contained within each image, 𝐶 𝑚 1 r 𝑚 2 and Ĉ𝑚 1 r 𝑚 2 denote the true color and predicted color of the 𝑚 2 -th pixel on the 𝑚 1 -th image, respectively. By implementing optimization methods such as the Stochastic Gradient Descent (SGD) algorithm [50] , we can progressively update the parameters 𝜎 pertaining to the neural network 𝐹 𝜃 . In practice, we have improved the U-net [51] neural renderer by increasing the number of lowresolution feature convolution layers while decreasing the number of high-resolution feature convolution layers, thus substantially reducing the rendering time while ensuring the visual quality. These iterative updates are intended to minimize the OSNeRF loss function and ultimately achieve accurate object reconstruction.The semantic control unit (SCU) is the semantic information interaction center of our proposed OSNeRF. Its main functions can be summarised as follows.Cooperative robots behaviour control. The SCU has the ability to convert the user's demand specifications into associated semantic data, thus directing the behavior of the cooperative robots (drones, intelligent vehicle, robot dogs, etc.) towards capturing the raw images most relevant to fulfilling the task requirements. This enhances the dependability of the operational cooperative robots and heightens the efficiency of the data collection endeavor.Semantic knowledge updating. The update of semantic knowledge guarantees the preservation of consistent semantic comprehension between the encoder and decoder. Updating and promptly disseminating the most recent semantic knowledge and concepts help evade misunderstandings and disparities in the coding and decoding of semantic information. Furthermore, by updating the semantic knowledge base in situations where multiple OSNeRF systems share a common semantic repository, it ensures that these systems can comprehensively understand and interpret one another's information, thereby enriching collaborative endeavors and integration capabilities.Semantic feature routing. Compressed semantic feature routing efficiently transmits compressed semantic information to the respective decoder and renderer, employing an effective routing mechanism. Furthermore, it employs compressed semantic features to steer the model's selection of scene regions of interest, thereby precisely allocating computational resources. By concentrating computational resources on regions of elevated semantic significance, such as objects or areas of interest, the reconstruction results are enhanced in terms of both quality and efficiency. 
implementation details 4.1 dataset and baseline: 
 We train our framework on the Tandt [52] dataset (composed of 251 images of 980 × 545 px). We partition the data to 200 training scenes and 51 testing scenes. We also test our model (merely trained on Tandt) on the ABO datasets [53] , which diverse geometries with realistic materials. Moreover, in order to verify the generalizability of our proposed method, we collected a set of object images of real scenes by UAVs and robots and named them as Fyts dataset, which composed of 180 images of 1920 × 1080 px and diverse sharp and intricate textures. The above datasets have different scene and view distributions from our training dataset. Models are trained on all images of the training set for 1M iterations.For performance bench-marking, we compare the reconstruction results of the proposed OSNeRF with the SOTA schemes proposed in [13, 43, 54, 55] , in which [54] and [55] are classical NeRF methods while [43] and [13] are semantic-based NeRF methods. To conduct a comprehensive analysis of multiple NeRF-based methods, the Nerfstudio framework is used as it incorporates multiple neural implicit surface reconstruction approaches into a single framework. The NeRF training was executed using a Nvidia 4090 GPU, while the geometric comparisons of the 3D results were performed on a standard PC.OSNeRF focuses on implementing a robust and effective 3D object reconstruction in high-interference wireless environment. In particular, we will focus on the transmission of compressed semantic features. The SCU has the goal of enabling the decoder to perform inference on new data samples by exploiting a semantic paradigm. The hardware implementation will equip terminals with Jetson Nano processors for training and inference of large AI/ML models, softwaredefined radio (SDR) units provide a robust and effective implementation tested under various SNR wireless environment. The prototype system and hardware setting are illustrated in Figure . 4. Multiple cooperative robots (drones, intelligent vehicle, robot dogs) are utilized for data collection. The connectivity layer for the robotics will be provided by an advanced 3GPPcompliant core network (Rel-17/18) and an advanced 5.5G Open RAN system [56] equipped with semantic awareness platform. The high interference wireless environment is modeled as a standard Rayleigh fading channel (Signal-to-noise ratio below 10dB). SCU establish a comprehensive semantic knowledge base for managing the semantic codec and robotics, the semantic model training in above devices will be integrated into the robots tested for large-scale deployment.Pixel-level evaluation. We utilize the evaluation methodology proposed in [57] , where we initially capture multiple images of the reconstructed object from identical viewpoints. Subsequently, we analyze the pixel-level differences between the images of the original 3D object and those reconstructed from corresponding perspectives. To quantify the evaluation, we employ a comprehensive set of metrics, including PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), LPIPS (Learned Perceptual Image Patch Similarity), and FID (Fréchet Inception Distance). Semantic-level evaluation. Given the limitations of pixel-level metrics in assessing the semantic matching of reconstructed images, we propose a semantic-level evaluation to measure the performance of the generated images. To achieve this, we employ BLIP [58] , a powerful visual language model that integrates visual language understanding and construction. This model enables us to convert the multi-perspective images into textual representations. Subsequently, we utilize large language models like BERT [59] to obtain embeddings of these generated texts. Finally, we compare the differences between the embeddings using cosine similarity (CS) and BLEU [60] , the BLEU in this paper can be calculated asEQUATIONwhere 𝑛-grams means that the size of a word group. s is the transmitted sentence with length 𝑙 s and ŝ is the decoded sentence with length 𝑙 s , 𝐶 𝑘 (•) is the frequency count function for the 𝑘-th elements in 𝑛-th grams. 
experiments: 
 The experiments follow the evaluation framework presented in [61] , where we initially capture multiple images from identical viewpoints in both the processed 3D and recovered scenes. Subsequently, a meticulous analysis is conducted to assess pixel-level disparities between the images derived from the original 3D scene and those obtained from the reconstructed 3D scene, all captured from corresponding perspectives. For performance bench-marking, we compare the reconstruction results of the proposed OSNeRF with the SOTA schemes proposed in [13, 43, 54, 55] . To achieve fair and accurate comparisons, we run our method on the same experiment settings with other methods, and we try our best to directly use the reported official quantitative results in these papers or use the official code to run the experiments. The visual comparisons are shown in Figure 5 , the quantitative results are expressed in Table 1 and Table 2 . Qualitative comparison. Figure 5 showcases our reconstruction results, which demonstrate exceptional visual quality across various datasets. Utilizing input images sampled from each test scene, we apply guidance-finetuning to derive the triplane scene code and assess reconstruction quality based on previously unseen images. Despite being exclusively trained on the Tandt dataset, our model exhibits remarkable generalization to the ABO and Fyts datasets, which feature diverse scene and view distributions. Notably, OSNeRF produces more regular geometries compared to the slightly skewed and distorted shapes generated by MvsNeRF and GPNeRF. Additionally, OSNeRF excels in capturing sharp details and reflective materials. In contrast, the application of DSNeRF [52] , ABO datasets (Desk) [53] , and Fyts dataset (Hydrant and gymnasium) after 2h-processing. Note that the input images and semantic features of above NeRF methods are transmitted in HIWE (SNR=0dB) 1 . While all methods achieve reasonable PSNRs, SSIMs, LPIPs, and FID on the Tandt testing set, our method consistently surpasses other methods across these four metrics when given the same input. More notably, our results on the additional two testing datasets significantly outperform the comparison methods, effectively demonstrating the robust generalizability of our OSNeRF. Typically, comparison methods aggregate 2D image features directly across view input at ray marching points for radiance field inference. In contrast, our method prioritizes semantic information and maintains the consistency of the reconstructed image quality by leveraging the correlations among recovered semantic features. This leads to the best generalizablity and the highest rendering quality of OSNeRF across diverse testing scenes. Table 2 showcases the semantic-level evaluation results on different datasets, further validating the performance of OSNeRF. The cosine similarity, which measures the semantic similarity between the original and reconstructed images, can reach up to 0.963. This demonstrates the high degree of semantic consistency between the two, indicating that OSNeRF successfully preserves the semantic information of the objects in the reconstruction. Furthermore, the BLEU score of OSNeRF reaches a maximum of 0.942, indicating a strong alignment between the original and reconstructed images at the semantic level, which suggests that OSNeRF effectively captures and preserves the semantic features of the original objects during the reconstruction process. Despite potential fluctuations in pixel values caused by variations in brightness, contrast, or color in the reconstructed images, the semantic consistency between the original and reconstructed images remains remarkably high. This indicates that OSNeRF effectively transmits the semantic features of the OSO while preserving semantic consistency, outperforming other methods. Time comparison. We present the 3D construction results of OSNeRF for a sample object (truck) with varying optimization durations in Figure 6 . Notably, our reconstruction outcomes demonstrate substantial improvement within just 60 minutes of processing, compared to the state-of-the-art NeRF scheme depicted in Figure 5 , which required 120 minutes for rendering. The visual quality of our reconstructed images is not only comparable but also superior. In addition, in the HIWE (SNR=0dB), the processing time required for our OSNeRF to generate an image with the same PSNR has been improved by more than 35% compared to the comparison schemes, the specific details are shown in the appendix video. This advantage arises because OSNeRF exclusively transmits the semantic characteristics of the object, rather than the entire original image. This approach significantly reduces both data transfer time and the processing load on the semantic decoder. Moreover, the semantic decoder exclusively restores the multi-view image encompassing the object, enabling the lightweight renderer to efficiently utilize limited computational resources to focus on the object. Additionally, the SCU ensures efficient data transmission across various processing stages, leading to fast 3D object reconstruction. It is evident that the reconstruction quality of OSNeRF surpasses that of the compared SOTA methods in high-interference wireless environments. This superiority arises from the distortion experienced by images transmitted directly through such environments, consequently affecting the object reconstruction outcomes. In contrast to the other SOTA methods, our OSNeRF incorporates a semantic encoder that accurately extracts the core semantic features of OSO based on user demand. Even in the presence of interference, the correlation between the key semantics through the SCU and the semantic decoder ensures consistency in both pixel-level and semantic-level details of the transmitted and received images. As the SNR improves, the performance of each scheme also improves. However, our OSNeRF consistently achieves the lowest FID and the highest BLEU across all SNRs, thereby highlighting the robustness of the proposed OSNeRF.Tandt [52] ABO [53] Fyts Methods PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ PSNR ↑ SSIM ↑ LPIPS ↓ FID 
conclusion: 
 We present a novel On-demand Semantic Neural Radiance Fields (OSNeRF) scheme that can achieve fast and robust object construction in high interference wireless environment (HIWE). In contrast to traditional NeRF-based reconstruction, OSNeRF intensifies the focus on the semantic information of on-demand object (OSO). It incorporates an efficient on-demand data collector that procures multi-perspective images, employs a semantic encoder and decoder for precise feature extraction and robust image restoration in HIWE, and utilizes a lightweight renderer to expedite the reconstruction process. Additionally, we have developed a Semantic Control Unit (SCU) that orchestrates semantic-level services such as semantic routing for the above components. Experiments validates that the result of our OSNeRF performs favorably against state-of-the-art (SOTA) methods in terms of both both pixel-level and semanticlevel, which enables fast and robust 3D object reconstruction in HIWE. For the future, we will further enhance our methodology to support real-time reconstruction of dynamic objects.  
: 
 the scene [10] . By incorporating a volumetric rendering skill [11] , images of unseen camera views can be generated with convincing quality. Existing studies [12] [13] [14] mainly focus on global scene reconstruction. Nonetheless, in practical application, users tend to be more concerned with the reconstruction results of on-demand specific object (OSO) [15] . Consequently, NeRF schemes for global scene reconstruction that lack of on-demand often have significant inefficiencies [16] . Moreover, the inputs of the existing NeRF-based methods are high-quality images from the datasets. However, in real-world applications, collected images often become distorted during transmission through high interference wireless environments (HIWE) [17] , which significantly compromising the quality of object reconstruction [18] .In this paper, we propose a novel on-demand Semantic Neural Radiance Fields (OSNeRF) scheme for fast and robust 3D object reconstruction. As depicted in Fig. 1 , initially, the Semantic Control Unit (SCU) directs the cooperative robots to conduct data collection, guided by the user demand indicator, to obtain a multi-view representation of the 3D scene. Subsequently, the semantic encoder sequentially performs semantic segmentation, semantic feature extraction, and semantic feature compression on OSO images. Following that, the semantic decoder reconstructs the OSO images based on the received compressed semantic features. Finally, the restored images, containing only the core semantic features, are fed into a lightweight renderer. This scheme significantly reduces the computational complexity of NeRF while enhancing the efficiency and robustness of 3D object reconstruction in HIWE. In summary, our contributions are as follows:• A novel on-demand semantic neural radiance fields (OSNeRF) scheme is proposed, which can provide fast and robust 3D object reconstruction in HIWE. With OSNeRF, on-demand objects in the scene are selectively reconstructed according to the user's indicators. • Technically, we implement a prototype system of OSNeRF, which consists of on-demand data collector, semantic encoder, semantic decoder, lightweight renderer, and semantic control unit. By filtering redundant information and providing semantic-level reconstruction guidance, the high efficiency of 3D object reconstruction can be achieved. • Comparison experiments are conducted, the reconstruction results clearly indicate that OSNeRF outperforms existing state-of-the-art (SOTA) methods in terms of both pixel-level and semantic-level metrics. Furthermore, OSNeRF exhibits a distinct advantage on speed and robustness in HIWE. 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. The reviews are mixed, including Borderline Accept, Borderline Accept, Borderline Accept, Weak Reject. The authors have addressed majorities of the concerns from reviewers. The major concerns left by the reviewers are 1) lack of comparison on latest NeRF backbones, 2) the necessity of semantic information. While ACs agree with these two important concerns, we still think as an initial effort for an end-to-end controllable capturing and reconstruction system, this is still a valuable work. Moreover, additional experiments in rebuttal show the potential to generalize to other NeRF models, as well as the effectiveness of semantic information. Based on this, the decision is to recommend the paper for acceptance to ACM Multimedia 2024.

Still, as mentioned, we do think reviewers have raised valid concerns, and we recommend that the authors carefully read all reviewers' final feedback, and include and extend the experiments presented in rebuttal, to resolve those concerns. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach to 3D object reconstruction.
This paper leverages semantic encoding and decoding to extract and transmit core semantic features of images, allowing for high-quality 3D reconstruction even in high-interference wireless environments. The approach incorporates a lightweight renderer and a semantic control unit (SCU) to significantly enhance reconstruction efficiency and quality, particularly in resource-constrained and high-interference scenarios.
strength_weakness: strength_weakness: 1. The paper demonstrates that OSNeRF maintains high reconstruction quality even in high-interference wireless environments (HIWE).
2. OSNeRF introduces a Semantic Control Unit (SCU) to guide data collection and optimize the reconstruction process. This unit helps in enhancing the efficiency and accuracy of 3D object reconstruction by focusing computational resources on the most relevant parts of the scene as per user demand.
3. Notable performance improvement.
limitations: limitations: 1. **Writing**: Citing each baseline method in results could enhance the writing.
2. **Potential Inefficiency in Feature Transmission**: A potential weakness of OSNeRF is that transmitting intermediate semantic features might not be as noise-resistant and data-efficient as directly transmitting compressed images to the local processing unit. Compressed images typically have better noise resilience and require less data to transmit. Therefore, the authors need to clarify the advantages of performing semantic segmentation directly on the robot side, such as any specific benefits in terms of robustness, efficiency, and overall system performance.
paper_summary: paper_summary: This paper proposed OSNeRF to enable 3D reconstruction of on-demand specific objects (OSOs), particularly in high-interference wireless environments. The authors introduced semantic encoding, semantic decoder, lightweight renderer, and semantic control unit for extracting semantic features, facilitating image recovery, reconstruction, and enhancing the efficiency, respectively. Experimental results demonstrate that the proposed OSNeRF outperforms some NeRF methods in terms of reconstruction quality.
strength_weakness: strength_weakness: The paper's organization is well-structured. 

The authors have conducted a thorough evaluation using a diverse set of comparison datasets.
limitations: limitations: **Motivation**

The introduction lacks sufficient context and motivation for the proposed methodology. The paper transitions directly from the problem statement to the method description without adequately elaborating on the authors' rationale and the innovative aspects of the approach.
It would be helpful if the authors could provide a more detailed explanation for their choice of a semantic encoder-decoder architecture and how it compares to alternative techniques. 
 

**Method**

1. While the authors present an architecture that combines a transformer encoder and a convolutional decoder for image reconstruction, the overall innovation of the method appears to be limited. The core process of encoding the image with a transformer, decoding it with a convolutional network, and then reconstructing the image using NeRF does not seem to significantly advance the state-of-the-art in this area. 

2. The method description lacks the necessary clarity and specificity for a thorough understanding of the approach. For example, the distinction between the recovered RGB or semantic images in the semantic encoder-decoder module is unclear, which makes it difficult to assess the inputs and outputs of this critical component.

**Experiment**

1. While the authors have included several relevant baseline methods, the lack of details on how these approaches were adapted and configured for the current task limits the reader's ability to assess the validity of the comparative analysis. Specifically, the authors do not provide any information on how the generalizable GPNeRF method was modified to suit the OSO reconstruction task, which is necessary for a fair evaluation.

2. The choice of comparison methods appears to be somewhat outdated. It would be beneficial for the authors to consider incorporating more recent and state-of-the-art comparison methods, such as Instant-NGP [1] and Zip-NeRF [2]. 

[1] Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 2022.

[2] Zip-nerf: Anti-aliased grid-based neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
paper_summary: paper_summary: This paper proposed a novel on-demand Semantic Neural Radiance Fields (OSNeRF) scheme to resolve the transmission through the HIWE. This scheme structured with a SCU , a feature extraction encoder and lightweight render decoder. This scheme can improve the enhance the efficiency and robustness in reconstruction in HIWE.
strength_weakness: strength_weakness: 1. Writing and paper organization is great 
2. The proposed work represents a good idea for reconstruction and transmission between robots and servers. It is a novel idea to introduce this structured scheme to achieve the efficient transmission.
limitations: limitations: 1. The field of “contribution to multimedia” looks like another paper, “TSNeRF,” instead of "OSNeRF." 
2. The description of Figure 4 is not clear. 
3. The dataset chooses to use why not use NeRF synthetic datasets
4. The training takes 1M iterations and the re-rendering cost 2h, is there any optimization method to improve these high system consumption cost? 
5. The contrition claims this work can robust image recovery under HIWE conditions. Figure 6 and its explanation do not clearly explain the improvement under this condition.
paper_summary: paper_summary: This paper proposes a novel on-demand Semantic Neural Radiance Fields (OSNeRF) scheme for fast and robust 3D object reconstruction. The key points are:

1. OSNeRF enables selective reconstruction of on-demand objects in a scene based on the user's requirements, rather than reconstructing the entire scene.

2. It employs a semantic encoder to extract core semantic features from the collected images and compress them for transmission in high-interference wireless environments 

3. A semantic decoder is used to reconstruct the images containing just the core semantic features at the receiver end.

4. The reconstructed images are then fed into a lightweight renderer for fast and efficient 3D object reconstruction. 

5. A semantic control unit guides the semantic encoder, decoder and lightweight renderer to enhance reconstruction efficiency.

6. Experiments show OSNeRF outperforms state-of-the-art methods in terms of reconstruction quality, speed and robustness, especially in high-interference wireless environments.
strength_weakness: strength_weakness: Proposes a novel on-demand semantic neural radiance fields (OSNeRF) scheme that focuses on reconstructing specific objects of interest rather than the entire scene. This on-demand capability is novel and practically useful.

Integrates semantic information to enable fast and robust 3D reconstruction of objects from images transmitted over interference-prone wireless channels. The use of semantics to improve reconstruction speed and robustness, especially in challenging wireless environments, is an innovative aspect.

Employs a well-designed pipeline consisting of semantic encoder, semantic decoder, lightweight renderer and semantic control unit. The semantic encoder extracts core semantic features and compresses them for robust wireless transmission. The decoder reconstructs clean images with core semantics for input to the lightweight renderer. This is a technically sound approach.

Leverages Swin Transformer for effective semantic feature extraction and Grad-CAM for identifying features most relevant to the reconstruction task. These are appropriate technical choices.

Conducts comparison experiments that demonstrate OSNeRF outperforms state-of-the-art methods on both pixel-level and semantic-level metrics. This provides good empirical evidence for OSNeRF's reconstruction quality.

Shows that OSNeRF has advantages in terms of speed and robustness, especially in high-interference wireless environments. Evaluating performance in the practically-relevant scenario of lossy wireless transmission is a strength.

Provides a clear description of the proposed OSNeRF scheme, elaborating on the key components like semantic encoder/decoder and lightweight renderer. The writing is generally clear and easy to follow.
limitations: limitations: The paper is well-structured and clearly written. I also enjoy reading the background and its applications.

However, this article has several serious problems. 

1.The author claim that OSNeRF is compared with four SOTA schemes. 
[1] GP-NeRF: Generalized perception NeRF for context-aware 3D scene understanding.
[2] Sem2nerf: Converting single-view semantic masks to neural radiance fields.
[3] Depth-supervised nerf: Fewer views and faster training for free. 
[4] Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.

These papers are published at 2022. Considering how fast NeRF is growing in academia, they are definitely not the SOTA methods. Please compare with recent methods or methods published in at least 2023.

For instance:
[5] Simple-RF: Regularizing Sparse Input Radiance Fields with Simpler Solutions
[6] ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field
[7] Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization
[8] Sparf: Neural radiance fields from sparse and noisy poses
[9] F2-nerf: Fast neural radiance field training with free camera trajectories 

2.The proposed semantic encoder is a kind of neural feature fields, the proposed pipeline (e.g.  lightweight renderer in Fig.3) is not new in this area. The proposed advantages by the OSNeRF is a task of few-shot training, which is also not new in this area. Please give a literature review on these topics and highlight the differences between OSNeRF and existing methods.