Learning Disentangled Representations for Perceptual Point Cloud Quality Assessment via Mutual Information Minimization

Authors: ziyu  shan
yujie  zhang
yipeng  liu
yiling  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 13:49:49

Abstract: No-Reference Point Cloud Quality Assessment (NR-PCQA) aims to objectively assess the human perceptual quality of point clouds without relying on pristine-quality point clouds for reference. It is becoming increasingly significant with the rapid advancement of immersive media applications such as virtual reality (VR) and augmented reality (AR). However, current NR-PCQA models attempt to indiscriminately learn point cloud content and distortion representations within a single network, overlooking their distinct contributions to quality information. To address this issue, we propose DisPA, a novel disentangled representation learning framework for NR-PCQA. The framework trains a dual-branch disentanglement network to minimize mutual information (MI) between representations of point cloud content and distortion. Specifically, to fully disentangle representations, the two branches adopt different philosophies: the content-aware encoder is pretrained by a masked auto-encoding strategy, which can allow the encoder to capture semantic information from rendered images of distorted point clouds; the distortion-aware encoder takes a mini-patch map as input, which forces the encoder to focus on low-level distortion patterns. Furthermore, we utilize an MI estimator to estimate the tight upper bound of the actual MI and further minimize it to achieve explicit representation disentanglement. Extensive experimental results demonstrate that DisPA outperforms state-of-the-art methods on multiple PCQA datasets.

Main paper: 
 introduction: 
 With recent advances in 3D capture devices, point clouds have become a prominent media format to represent 3D visual content in various immersive applications, such as autonomous driving and virtual reality [4, 43] . These extensive applications stem from the rich information provided by point clouds (e.g., geometric coordinates, color). Nevertheless, before reaching the user-client, point clouds inevitably undergo various distortions at multiple stages, including acquisition, compression, transmission and rendering, leading to undesired perceptual quality degradation. Accordingly, it is necessary to develop an effective metric that introduces human perception into the research of point cloud quality assessment (PCQA), especially in the common no-reference (NR) situation where pristine reference point clouds are unavailable.In recent years, many deep learning-based NR-PCQA methods [21, 47, 51, 32, 3] have shown remarkable performance on multiple benchmarks, which can be applied directly to 3D point cloud data or 2D rendered images. Most of these methods [47, 22, 51, 3] tend to learn a unified representation for quality prediction, ignoring the fact that perceptual quality is determined by both point cloud Figure 1 : Statistics of SJTU-PCQA (part) [46] and predicted quality scores of NR-PCQA models (PQA-Net [21] and GPA-Net [32] ). Quality scores of different distortion types are in lines of different colors. Red circles are to highlight the score span of different contents with the same distortion. content information and distortion pattern. Although some other models [21, 33] alternately learn content and distortion representations through different training objectives, they are still based on a single-branch network and thus may lead to highly entangled features in the representation space.From the perspective of visual rules, insufficient disentanglement between representations of point cloud content and distortion disobeys the perception mechanisms of the human vision system (HVS), further limiting performance improvement. In fact, many studies [35, 16] highlight the distinct visual processing of high-level (e.g., semantics) and low-level information (e.g., distortions) in different areas of the brain. Concretely, the left and right hemispheres of the brain are specialized in processing high-level and low-level information, respectively. These findings suggest a relatively disentangled processing mechanism in our brain, challenging existing methods that seek to learn these conflicting representations using a single network indiscriminately.The difficulty of disentangled feature learning is relatively great for NR-PCQA due to data imbalance. Specifically, although a wide range of distortion types and intensities in current PCQA datasets can enable the learning of robust low-level distortion representations, it is non-trivial to learn the representations of point cloud content that lies in a considerable high dimensional space because these PCQA datasets are extremely limited in terms of content (e.g., up to 104 contents in LS-PCQA [23] ). This data limitation can lead to overfitting of NR-PCQA models regarding point cloud content, that is, when the content changes, the prediction score changes in the undesired manner, even with the same distortion pattern. As illustrated in Figure 1 (b) and (c), the NR-PCQA models PQA-Net [21] and GPA-Net [32] correctly predict the trend of quality degradation with increasing distortion intensity, but their predicted score spans deviate a lot from the ground truth in Figure 1 (a) , where the content varies but the distortion pattern remains intact. Based on these observations, we expect a new disentangled representation learning framework that can obey the separate information processing mechanism of HVS, and alleviate the difficulty of content and distortion representation learning introduced by data imbalance.In this paper, we propose a new Disentangled representation learning framework tailored for NR-PCQA, named DisPA. Motivated by the HVS perception mechanism, DisPA employs a dual-branch structure to learn representations of point cloud content and distortion (called content-aware and distortion-aware branches). DisPA has three steps to achieve disentanglement: 1) To address the problem introduced by data imbalance, we pretrain a content-aware encoder based on masked autoencoding strategy. Specifically, in this pretraining process, the distorted point cloud is rendered into multi-view images whose patches will be partially masked. The partially masked images are then fed into the content-aware encoder to reconstruct the rendered images of the corresponding reference point cloud. 2) To facilitate learning of distortion-aware representations, we decompose the distorted multi-view images into a mini-patch map through grid mini-patch sampling [40] , which can prominently present local distortions and forces the distortion-aware encoder to ignore the global content. 3) Inspired by the utilization of mutual information (MI) in disentangled representation learning [8] , we propose an MI-based regularization to explicitly disentangle the latent representations. Compared to simple linear correlation coefficients (e.g., cosine similarity), mutual information can capture the nonlinear statistical dependence between representations [15] . To achieve this, we utilize an MI estimator to estimate a tight upper bound of the MI and further minimize it to achieve straightforward disentanglement. We summarize the main contributions as follows:• We propose a novel disentangled representation learning method for NR-PCQA called DisPA, which obeys the particular HVS perception mechanism. To the best of our knowledge, DisPA is the first framework to explore representation disentanglement in PCQA.• We propose the key MI-based regularization that can explicitly disentangle the representations of point cloud content and distortion through MI minimization.• We conduct comprehensive experiments on three datasets (SJTU-PCQA [46] , WPC [20] , LS-PCQA [23] ), and achieve superior performance over the state-of-the-art methods on all of these datasets. 
related work: 
 No-Reference Point Cloud Quality Assessment. NR-PCQA aims to evaluate the perceptual quality of distorted point clouds without available references. According to the modalities, the NR-PCQA methods can be categorized into three types: projection-based, point-based and multi-modal methods.For the projection-based methods, various learning-based networks [21, 47, 52, 33, 34] adopt multiview projection for feature extraction, while Zhang et al. [53] integrates the projected images into a video to conveniently utilize video quality assessment methods to evaluate the perceptual quality. Xie et al. [45] first computes four types of projected images (i.e., texture, normal, depth and roughness) and fuses their latent features using a graph-based network. For the point-based methods, Zhang et al. [50] extracts carefully designed hand-crafted features, while Liu et al. [23] transforms point clouds into voxels and utilizes 3D sparse convolution to learn the quality representations. Some 3D native methods [37, 32, 39] divide point clouds into local patches and utilize hierarchical networks structurally like PointNet++ [29] to learn the representations. For the multi-modal methods, Zhang et al. [51] utilizes individual 2D and 3D encoders to separately extract features, and fuse them using a symmetric attention module. Other works [38, 3, 22] leverage various cross-modal interaction mechanisms to enhance the fusion between 2D and 3D modalities. Compared to previous methods that learn quality representations indiscriminately, our work solves quality representation disentanglement from a more essential perspective of mutual information, which reveals the intrinsic correlations between point cloud content and distortion pattern.Representation Learning for Image/Video Quality Assessment. As for image quality assessment (IQA), CONTRIQUE [25] learns distortion-related information on images with synthetic and realistic distortions based on contrastive learning. Re-IQA [31] trains two separate encoders to learn highlevel content and low-level image quality features through an improved contrastive paradigm. QPT [54] also learns quality-aware representations through contrastive learning, where the patches from the same image are treated as positive samples, while the negative sample are categorized into content-wise and distortion-wise samples to contribute distinctly to the contrastive loss. QPTv2 [44] is based on masked image modeling (MIM), which learns both quality-aware and aesthetics-aware representations through performing the MIM that considers degradation patterns.As for VQA, CSPT [5] learns useful feature representation by using distorted video samples not only to formulate content-aware distorted instance contrasting but also to constitute an extra self-supervision signal for the distortion prediction task. DisCoVQA [41] models both temporal distortions and contentrelated temporal quality attention via transformer-based architectures. Ada-DQA [19] considers video distribution diversity and employ diverse pretrained models to benefit quality representation. DOVER [42] divides and conquers aesthetic-related and technical-related (distortion-related) perspectives in videos, introduces inductive biases for each perspective, including specific inputs, regularization strategies, and pretraining. However, there is no current work to utilize mutual information (MI) to achieve representation disentanglement, which has not been explored in IQA/VQA.Mutual Information Estimation. Mutual information (MI) has been widely used as regularizers or objectives to constrain independence between variables [2, 7, 13, 14] . Hjelm et al. [14] performs unsupervised representation learning by maximizing MI between the input and output of a deep neural network. Kim et al. [17] learns disentangled representations by encouraging the distribution of representations to be factorial and hence independent across the dimensions. Moreover, MI minimization has been drawing increasing attention in disentangled representation learning [6, 15, 55] . Chen et al. [8] introduces a contrastive log-ratio upper bound for mutual information estimation, and extends the estimator to a variational version for general scenarios when only samples of the joint distribution are obtainable. Dunion et al. [11] minimizes the conditional mutual information between representations to improve generalization abilities under correlation shifts and enhances training performance in scenarios with correlated features. However, to our knowledge, there has been no previous work focusing on learning disentangled representations or exploring MI estimation for visual quality assessment.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please see B, which includes the limitation of our work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
mutual information estimation and minimization: 
 Given the content-aware and distortion-aware representations (x, y), our goal is to estimate the MI between x and y and further minimize it. In this section, we explain the mathematical background of how to leverage a neural network to estimate the MI between x and y.The MI between x and y can be defined as:EQUATIONwhere p(x, y) is the joint distribution, p(x) and p(y) are the marginal distributions.Unfortunately, the exact computation of MI between high-dimensional representations is actually intractable. Therefore, inspired by [6, 8, 15] , we focus on estimating the MI upper bound and further minimize it. The tight upper bound of mutual information (MI) means an upper boundary that is always higher the actual value of MI. A tight upper bound means the bound is close to the actual value of MI and equal to MI under certain conditions. An MI upper bound estimator Î(x; y) can be formulated as (proof in Appendix A):EQUATIONSince the conditional distribution p(y|x) is unavailable in our case, we approximate it using a variational distribution q ϕ (y|x) = Q ϕ (x, y) , where the conditional distribution is inferred by another light neural network Q ϕ with parameters ϕ. Then the variational form Îv (x; y) can be formulated as (in a discretized form):EQUATIONwhere {(x i , y i )} N i=1 is N samples pairs drawn from the joint distribution p(x, y). To make Îv (x; y) a tight MI upper bound, Q ϕ is trained to accurately approximate p(y|x) by minimizing the KL divergence between p(y|x) and q ϕ (y|x):min ϕ KL (p(y|x)∥q ϕ (y|x)) = min ϕ E p(x,y) [log p(y|x)]No relation with ϕ -E p(x,y) [log q ϕ (y|x)] to be minimized (4) Obviously, the first term in Equation 4 has no relation with ϕ, thus Equation 4 equals minimization of the second term. Therefore, the can be a tight MI upper bound if we minimize the following negative log-likelihood of the inferred conditional distribution:EQUATIONNow, given the representations x and y, we can train the MI estimator Îv (x; y) to predict the MI between x and y by minimizing L MI . Afterwards, we minimize Îv (x; y) for explicit disentanglement, detailed implementations will be explained in Section 4.4. 4 Proposed FrameworkQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 
overall architecture: 
 The aforementioned analysis in Section 1 reveals that HVS processes high-level and low-level features in a relatively separate manner. To obey this mechanism, as illustrated in Figure 2 (a), the architecture of DisPA is divided into two branches to learn content-aware and distortion-aware representations, respectively. Given a distorted point cloud P , we first render it into multi-view images I. The multi-view images are fed into a frozen pretrained vision transformer (ViT) [10] F with parameters Θ f to generate the content-aware representation x. Next, the multi-view images are decomposed into a mini-patch map M through grid mini-patch sampling. The mini-patch map is encoded by the distortion-aware encoder G (a Swin Transformer) [24] ) with parameters Θ g to obtain representation y. After obtaining x and y, we also use them to train the MI estimator M and obtain the estimated MI Îv (x; y) following the process in Section 3. Finally, we concatenate x and y (denoted as [•, •]) and regress it by fully-connected layers H with parameters Θ h to predict quality score q. The whole process can be described as follows:q = H([F(I; Θ f ), G(M ; Θ g )]; Θ h ) (6a) Îv (x; y) = M(F(I; Θ f ), G(M ; Θ g )) (6b)As analyzed in Section 1, the learning difficulty of content representation is more intractable than distortion due to the limited dataset scale in terms of point cloud content. To address this problem, we pretrain the content-aware encoder F via the proposed masked autoencoding strategy. As illustrated in Figure 2 (b), given a distorted point cloud P and its corresponding reference point cloud P ref , our goal is to render P andP ref into multi-view images {I (n) ∈ R H×W ×3 } Nv n=1 , {I (n) ref ∈ R H×W ×3 } Nv n=1and pretrain F by using partially maskedI (n) to reconstruct I (n) ref ,where N v is the number of views.Multi-View Rendering. Instead of directly performing masked autoencoding in 3D space, we render point clouds into 2D images to achieve pixel-to-pixel correspondence between the rendered images of P and P ref , which facilitates the computation of pixel-wise reconstruction loss between the predicted patches and the ground truth patches. To perform the rendering, we translate P (or P ref )to the origin and geometrically normalize it to the unit sphere to achieve a consistent spatial scale. Then, to fully capture the quality information of 3D point clouds, we apply random rotations before rendering P ,P ref into {I (n) } Nv n=1 and {I (n) ref } Nv n=1 .Patchifying and Masking. After obtaining the rendered image I (n) , we partition it into nonoverlapping 16 × 16 patches following [10] . Then we randomly sample a subset of patches and mask the remaining ones, where the masking ratio is relaxed to 50% instead of the high ratio in [12] (e.g., 75% and even higher) because some point cloud samples in PCQA datasets exhibit severe distortions, necessitating more patches to extract effective content-aware information. In addition, the relatively low masking ratio can mitigate the influence of the background of I (n) .Figure 3 : Illustration of mini-patch map generation.Encoding and Reconstruction. The unmasked patches of I (n) are fed into the ViT F for initial embedding and subsequent encoding to obtain the representation x. To compensate for the scarcity of point cloud content in PCQA datasets, we initialize the encoder using the parameters optimized on ImageNet-1K [9] . Next, to reconstructI (n)ref , we feed x into a decoder and reshape it into 2D pixels to generate the reconstructedÎ(n) ref .By reconstructing masked reference patches from unmasked distorted patches, the encoder F is forced to focus more on semantic information than distortion patterns. The contentaware representation can be learned by the reconstruction loss L rec :L rec = Nv n=1 Î(n) ref -I (n) ref 2 2(7)To learn an effective distortion-aware representation y, we decompose the multi-view images {I (n) } Nv n=1 to a mini-patch map through grid mini-patch sampling, following [40, 42, 52] . As illustrated in Figure 3 , the distortion pattern is well preserved and even more obvious on the minipatch map while the content information is blurred. More concretely, for each multi-view image I (n) , we first split it into uniform L × L grids, the set of grids G (n) can be described as:G (n) = {g (n) 0,0 , • • • , g (n) i,j , • • • , g (n) L,L }, g (n) i,j = I (n) [ i × H L : (i + 1) × H L , j × W L : (j + 1) × W L ] (8) where g (n) i,j ∈ R H L × WL ×3 denotes the grid in the i-th row and j-th column of I (n) . Then we sample the mini-patches from each g (n) i,j and splice all the selected mini-patches to get the mini-patch map M . Note that blank mini-patches (i.e., image background) are ignored, and the map is ensured to M ∈ R H×W ×3 by filling in the unemployed mini-patches. After the mini-patch map generation, we feed it into the distortion-aware encoder G to generate the corresponding representation y.MI-based Regularization. After obtaining content-aware and distortion-aware representations x and y, we further disentangle them by minimizing the MI upper bound in Îv (x; y) in Equation 3. As revealed in Equation 4 and 5, the key to accurately estimate a tight Îv (x; y) is to minimize the negative log-likelihood of the variational network Q ϕ (x, y). Here we implement the Q ϕ using MLPs, and model the variational distribution as an isotropic Gaussian parameterized by a mean value µ ϕ = [µ ϕ (x 1 ), ..., µ ϕ (x D )] and a diagonal covariance matrix Σ = σ 2 ϕ I, where σ ϕ = [σ ϕ (x 1 ), ..., σ ϕ (x D )], D is the feature dimension of x and y. Then the variational distribution can be inferred as:EQUATIONwhere µ ϕ and σ 2 ϕ are obtained via the last two MLP layers. ϕ is optimized by minimizing L MI in Equation 5, the negative log-likelihood of Q ϕ (x, y). It is noted that the parameters of ϕ are optimized independently with the main networks Θ f and Θ g , seeing Algorithm 1.Loss Function. After obtaining the Îv (x; y), we incorporate it into our total training objective as a regularizer to disentangle the content-aware and distortion-aware representations, the total training Algorithm 1 Disentangled Representation Learning PipelineInput: A batch of rendered images {I (n) b | Nv n=1 } B b=1 ; mini-patch map {M b } B b=1 ; networks F, G, H with parameters Θ ′ f , Θ g , Θ h ; MI estimator M with variational network Q ϕ ; optimizer; λ 1 , λ 2 Output: Updated parameters Θ ′ g , Θ ′ h and ϕ ′ // Parameters Θ ′ f is frozen 1: Encode rendered images to generate content-aware representation x ← F({I (n) b }); Θ ′ f ) 2: Encode mini-patch map to generate distortion-aware representation y ← G({M b }; Θ g ) 3: for m = 1 → N M do // Update the MI estimator by training Q ϕ 4: Compute negative log-likelihood L MI ← B b=1 Q ϕ (x, y) 5:Update ϕ by minimizingL MI ϕ ′ ←optimizer (ϕ, ∇ ϕ L MI ) 6: Compute the estimated MI Îv (x; y) ← 1 B 2 B i=1 B j=1 [log Q ϕ ′ (x i , y i ) -log Q ϕ ′ (x i , y j )] 7: Predict the quality scores qb ← H([x b , y b ]; Θ h ) 8: Compute the total loss L ← 1 B B b=1 (q b -q b ) 2 + λ 1 L rank + λ 2 Îv (x; y) 9: Update the parameters {Θ ′ g , Θ ′ h } ← optimizer({Θ g , Θ h }, {∇ Θg L, ∇ Θ h L})loss function L can be formulated as:EQUATION)where B is the batch size and L rank is a differential ranking loss following [51, 33] . The λ 1 and λ 2 are weighting factors to balance each loss term. To better recognize quality differences for the point clouds with close MOSs, the differential ranking loss [51] L rank is used to model the ranking relationship between q and q:L rank = 1 B 2 B i=1 B j=1max(0, |q i -q j |-e (q i , q j )•(q i -qj )) , e (q i , q j ) = 1, q i ≥ q j -1, q i < q j (11) Algorithm 1 summarizes the overall pipeline of the disentangled representation learning framework (one iteration), where N M is the steps of updating for variational networks per epoch, and F is initialized with the pretrained parameters Θ f after masked autoencoding. The parameters of the main network Θ g , Θ h and the variational networks ϕ are updated alternately. 
datasets and evaluation metrics: 
 Datasets. Our experiments are performed on three popular PCQA datasets, including LS-PCQA [23] , SJTU-PCQA [46] , and WPC [20] . The content-aware pretraining is based on LS-PCQA, which contains 24,024 distorted point clouds, and each reference point cloud is impaired with 33 types of distortions (e.g., V-PCC, G-PCC) under 7 levels. The disentangled representation learning is conducted on all three datasets separately using labeled data, where SJTU-PCQA includes 9 reference point clouds and 378 distorted samples impaired with 7 types of distortions (e.g., color noise, downsampling) under 6 levels, while WPC contains 20 reference point clouds and 740 distorted samples disturbed by 5 types of distortions (e.g., compression, gaussian noise).Evaluation Metrics. Three widely adopted evaluation metrics are employed to quantify the level of agreement between predicted quality scores and ground truth (i.e., Mean Opinion Score, MOS): Spearman rank order correlation coefficient (SROCC), Pearson linear correlation coefficient (PLCC), and root mean square error (RMSE). To ensure consistency between the value ranges of the predicted scores and subjective values, nonlinear Logistic-4 regression is used to align their ranges.Table 1 : Quantitative comparison of the state-of-the-art methods and proposed DisPA on LS-PCQA [23] , SJTU-PCQA [46] , WPC [20] . The best results are shown in bold, and second results are underlined. "P" / "I" stands for the method is based on the point cloud/image modality, respectively. "↑" / "↓" indicates that larger/smaller is better. Our experiments are performed using PyTorch [28] on NVIDIA 3090 GPUs. All point clouds are rendered into 6-view projected images with a spatial resolution of 512 × 512 by PyTorch3D [30] .The encoders F and G are ViT-B [10] and Swin-T [24] , respectively.Content-Aware Pretraining. The pretraining is performed for 200 epochs, the initial learning rate is 3e-4, and the batch size is 64 by default. Adam optimizer [18] is employed with weight decay of 0.0001. Each point cloud is randomly rotated 6 times before being rendered into 6-view images to fully take advantage of quality information of point clouds.Disentangle Representation Learning. The steps of updating N M of MI estimator is set to 10. We use the Adam optimizer with weight decay of 0.0001 and batch size of 16. The hidden dimension of fully-connected layers is set to 64. The learning rate is initialized with 0.003 and decayed by 0.95 exponentially per epoch. For LS-PCQA, the model is trained for 20 epochs, while 150 epochs for SJTU-PCQA and WPC. The hyper-parameter λ 1 , λ 2 is set to 1 and 0.01 according to the loss scales.Data Split. Considering the limited dataset scale, in the training stage, 5-fold cross validation is used for SJTU-PCQA and WPC to reduce content bias. Take SJTU-PCQA for example, in each fold, the dataset is split into train-test with ratio 7:2 according to the reference point clouds, where the performance on testing set with minimal training loss is recorded and then averaged across five folds to get the final result. A similar procedure is repeated for WPC where the train-test ratio is 4:1. As for the large-scale LS-PCQA, it is split into train-val-test with a ratio around 8:1:1 (no content overlap exists). The result on the testing set with the best validation performance is recorded. Note that the pretraining is only conducted on the training set of LS-PCQA.15 state-of-the-art PCQA methods are selected for comparison, including 9 FR-PCQA and 6 NR-PCQA methods. For a comprehensive comparison, we conduct the experiment in four aspects: 1) We compare a quantitative comparison of prediction accuracy, following the cross-validation configuration in Section 5.2. 2) We perform the statistical analysis in Figure 1 for DisPA. 3) We present qualitative examples to demonstrate the superiority of our model in terms of avoiding overfitting when point content varies. 4) We report the results of cross-dataset evaluation for the NR-PCQA methods to verify the generalizability of our model.Quantitative Comparison. The prediction accuracy of all selected methods are presented in Table 1 , which demonstrates the competitive performance of proposed DisPA across all three datasets, and outperforms all the FR-PCQA methods on SJTU-PCQA and WPC. Compared with the NR-PCQA methods, DisPA outperforms CoPA [33] by about 1.3% in terms of SROCC on LS-PCQA, and , CoPA [33] and DisPA) on SJTU-PCQA [46] and WPC [20] . Statistical Analysis. We perform the statistical analysis on SJTU-PCQA in Figure 4 for DisPA. Compared with the statistics of PQA-Net and GPA-Net in Figure 1 , our DisPA not only predicts quality scores more accurately, but also obviously predicts closer score spans when point cloud content varies, even when the distortion intensity is at the highest level. The statistical analysis demonstrates that the content-aware pretraining strategy can effectively address the problem of superior difficulty of learning representations for point cloud content caused by data imbalance.Qualitative Evaluation. In Figure 5 , we present examples of SJTU-PCQA and WPC with predicted scores of PQA-Net [21] and CoPA [33] , where Figure 5 (a)(b), (e)(f) share the same content, (b)-(d), (f)-(h) share the the same distortion. Note that each score is predicted on the testing set of 5-fold validation. We can see that the predicted score of our DisPA is obviously closer to the ground truth (i.e., Mean Opinion Score, MOS) and dose not deviate from the MOS when content varies, which further validates the effectiveness of content-aware pretraining and representation disentanglement.Cross-Dataset Validation. To test the generalization capability of NR-PCQA methods when encountering various data distribution, we perform cross-dataset on LS-PCQA [23] , SJTU-PCQA [46] and WPC [20] . In Table 2 , we mainly train the compared models on the complete LS-PCQA and test the trained model on the complete SJTU-PCQA and WPC, and the result with minimal training loss is recorded. This procedure is repeated for mutual cross-dataset validation between SJTU-PCQA and WPC. From Table 2 , we can see that the performance of the cross-dataset validation is relatively low due to the tremendous variation of data distribution. However, our method still present competitive performances, demonstrating the superior generalizability of DisPA. Table 2 : Cross-dataset validation on LS-PCQA [23] , SJTU-PCQA [46] and WPC [20] (complete set). The best results (PLCC) are in bold, and the second results are underlined. We conduct ablation study of DisPA on SJTU-PCQA [46] in Table 3 . From Table 3 , we have following observations: 1) Seeing 1 ⃝ and 2 ⃝, the pretraining strategy effectively improves the performance of DisPA. 2) Seeing 1 ⃝, 3 ⃝ and 4 ⃝, the philosophy of representation disentanglement brings significant improvements to our model, because using simple cosine similarity in 4⃝ for disentanglement can achieve fair performance. However, using MI for disentanglement can better constrain the dependence between representations. 3) Seeing 1 ⃝, 5 ⃝ and 6 ⃝, using single branch to infer quality scores causes poor performance, since PCQA is a combination judgement based on the interaction of distortion estimation and content recognition. 4) Seeing 1 ⃝ and 7 ⃝, the performance is close, demonstrating the robustness of our model using different training loss functions.Furthermore, as shown in Figure 6 , we conduct a t-SNE visualization to compare the representation embeddings of PQA-Net, GPA-Net, content-aware and distortion-aware branches of our DisPA on the testing set of SJTU-PCQA. PQA-Net and GPA-Net are selected for comparison because these two methods both use distortion type prediction to learn distortion-aware representations. The scattered points are color and shape marked according to distortion type and content. The distortionaware features are visualized in 3rd sub-image, where we can see that the learned distortion-aware representation shows clear and separate clustering for different distortion types, indicating a strong correlation with degradations. The content-aware features present non-clustering for distortion types but a clear boundary to split the content.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The used data is all public open. In addition, we have uploaded our code to the in a .zip file as a supplementary document.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we propose a disentangled representation learning framework (DisPA) for No-Reference Point Cloud Quality Assessment (NR-PCQA) based on mutual information (MI) minimization. As for the MI minimization, we use a variational network to infer the upper bound of the MI and further minimize it to achieve explicit representation disentanglement. In addition, to tackle the nontrivial learning difficulty of content-aware representations, we propose a novel content-aware pretraining strategy to enable the encoder to capture effective semantic information from distorted point clouds. Furthermore, to learn effective distortion-aware representations, we decompose the rendered images into mini-patch maps, which can preserve original distortion pattern and make the encoder ignore the global content. We demonstrate the high performance of DisPA on three popular PCQA benchmarks and validate the generalizability compared with multiple NR-PCQA models.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental settings and training/testing details have been included in Implemantion Details part.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The results of our paper are not accompanied by error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our work strictly conform the code the ethics, presenting no societal impact, potential harm or non-anonymity behaviors.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Honestly, our work presents no potential negative societal impacts, because the used data is all public open and the task is not potentially harmful.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work obviously poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: CC-BY 4.0 is included in our work. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have uploaded our assets (e.g., the code) in an anonymized URL link and zip file. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof of Equation 2. To show that Î(x; y) is an upper bound of I(x; y), we calculate their difference:Î(x; y) -I(x; y) = E p(x,y) [log p(y|x)] -E p(x) E p(y) [log p(y|x)]Definition of Î(x; y) in Equation 2-E p(x,y) [log p(y|x) -log p(y)]Definition of I(x; y) in Equation 1=E p(x,y) [log p(y)] -EQUATIONSince log p(y) has no relation with x, so E p(x,y) [log p(y)] = E p(y) [log p(y)]. Then the equation can be formulated as:EQUATION)Recalling that the marginal distribution can obtained by integrating the joint distribution over the values of the other random variables:EQUATIONNote that log(•) is a concave function, by Jensen's Inequality, we havelog p(y) = log E p(x) [p(y|x)]Definition of marginal distribution in Equation 14EQUATIONApplying this inequality to Equation 13, we can conclude that Î(x; y) is always greater than I(x; y). Therefore, Î(x; y) is an upper bound of I(x; y). Î(x; y) = I(x; y) occurs only when p(y|x) holds the same for any x, which means x and y are two totally independent representations.We designed a no-reference point cloud quality assessment (NR-PCQA) framework, whose experimental performances have been validated. However, our current design has two main limitations:• Limitation of pretraining datasets. Ideally, the content-aware encoder F can be pretrained on a much larger dataset, despite the already large scale of LS-PCQA [23] . In our future work, we may pretrain it on distorted natural images or create a larger dataset of point clouds with various distortions. • Estimation of mutual information (MI). Although using the current MI minimization strategy can achieve satisfactory results, we may make efforts in the future to work on a more efficient MI estimation method for high-dimensional representations.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: Following the author-reviewer discussion stage, three reviewers expressed a positive opinion of the paper. While reviewer nBew raised concerns regarding the presentation, no critical technical issues were mentioned. The AC reviewed the authors' response to reviewer nBew and believes that these concerns can be addressed in the preparation of the camera-ready version. Therefore, the AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: This article's motivation is interesting. It combines content-aware and distortion-aware characteristics to train a 3D quality assessment network. Additionally, it employs a MAE-based method to train a content-aware encoder, uses patches to focus the network on learning distortion, and applies an MI module to integrate both features.
strength_weakness: strength_weakness: 1. Regarding the masked part, there is a gap between the first and second stages because the input in the first stage is a partially masked image, while the input in the second stage is the complete image.
2. I am also a bit confused about why the constraint in the first stage is a clean/reference image. This would give the masked encoder the characteristic of restoration, while the core of quality assessment is to evaluate the quality of the image. If the image features are restored, will it affect the accuracy of the image quality assessment?
3. The motivation for the MI part should provide more details and explanations, which leaves me somewhat puzzled.
4. I am not quite sure what the distortion-aware encoder has learned. Is it truly related to degradation features? This might require some verification.
questions: questions: This section corresponds to the Weaknesses:

1. How does the network address the gap between the input images in the first and second stages?
2. Why is a clean image used as a constraint? This results in learning the restoration characteristics, which is not very helpful for quality assessment.
3. The motivation for MI needs further explanation.
4. Visualize or analyze the distortion-aware features.
limitations: limitations: No potential negative societal impact. Please address the questions/suggestions above.
paper_summary: paper_summary: This paper proposes a novel disentangled representation learning framework called DisPA to decouple the representation learning process of point cloud content and distortion. To sufficiently disentangle these two representations, the DisPA uses two branches to learn them and adopt different training philosophies separately. For the content-aware branch, DisPA pretrains one encoder using a proposed masked auto-encoding strategy, which partially masks the images projected from distorted point clouds and reconstructs the corresponding patches of the images projected from pristine point clouds; For the distortion-aware branch, the DisPA integrates the mini patches of the rendered multi-view images into a mini-patch map, which can focus on local distortions and ignore the global point cloud content. Furthermore, to disentangle the learned representations, the DisPA uses a trainable mutual information estimator to estimate the mutual information (actually the tight upper bound) between these two branches and further minimize it alternatively along with the training of the main network (i.e., the two encoders and regression layers). Finally, the experimental results demonstrate the superior performance of DisPA in terms of both prediction accuracy and generalizability.
strength_weakness: strength_weakness: 1. From my viewpoint, the DisPA can be totally used for IQA, or even more suitable, since this work does not process 3D native point cloud data but just projects point clouds into images and uses 2D networks. Have the authors tried to introduce unique attribute information that is closely related to point clouds?
2. The whole DisPA is based on projections of point clouds, so the number of viewpoints is very important to the quality score prediction. However, the authors did not discuss the impact of the number of viewpoints.
3. How much time does it take to pretrain the content-aware encoder? The authors did not discuss the pretraining details in the implementation details part.
4. Why does not the LS-PCQA follow the K-fold data splitting? The authors should explain this for the loss of a unified experiment setting.
5. What is the actual architecture of the MI estimator? And what is the relation between the MI estimator and the lightweight neural network Q_phi? The idea of alternative training of the MI estimator and the main network is easy to understand, but the components of the MI estimator need more clarification. Is it just simple MLPs?
6. The mini-patch map generation has been used in many papers [1,2,3]. However, considering this is a minor contribution and the mini-patch map is actually effective for learning disentangled representations, this limited contribution is acceptable.
7. A minor weakness: There are some small flaws (gray bounding box) with the presentation of Figure 5 when zooming in. Please ensure all figures can be presented clearly without error.

[1] Wu, Haoning, et al. "Exploring video quality assessment on user generated contents from aesthetic and technical perspectives." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Wu, Haoning, et al. "Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[3] Zhang, Zicheng, et al. "Gms-3dqa: Projection-based grid mini-patch sampling for 3d model quality assessment." ACM Transactions on Multimedia Computing, Communications and Applications 20.6 (2024): 1-19.
questions: questions: 1. Why the conditional distribution p(y|x) is unavailable in your case? This needs more explanations, because the mathematical analysis is somewhat abstract so more details are needed to make it more understandable.
2. What is the advantanges of the differential ranking loss function?
3. Can this work be trained end-to-end for IQA or video quality assessment (VQA) without modification of specific modules (just replacing the input/output)?
4. Can the masked auto-encoding strategy use natural images for pretraining?
limitations: limitations: Yes, the authors have adequately addressed the limitations and discussed the potential negative societal impact in the appendix D Limitations and Future Work.
paper_summary: paper_summary: This paper proposes a novel no-reference quality assessment model tailored for point-cloud data. A disentangled representation learning strategy is leveraged to account for both content-aware information and distortion-aware information. Comprehensive experiments are conducted and the effectiveness of this proposed model is well verified.
strength_weakness: strength_weakness: 1. The idea of framing the quality assessment into content-/semantic-aware and distortion-aware aspects is actually quite common in IQA/VQA on 2D visual data. It’d be better to review some related QA metrics for 2D data, and analyze the key difference on the implementation of this idea between point-cloud data and 2D data quality assessment.
2. Lack of comparison on computational complexity.
questions: questions: Please see the Weaknesses.
limitations: limitations: Please see the Weaknesses.
paper_summary: paper_summary: This paper explores the method for No-Reference Point Cloud Quality Assessment. The key idea is to involve the disentangled representation learning to minimize mutual information between representations of point cloud content and distortion. The authors conduct experimental performance comparisons on three public databases and compare their proposed method with 15 existing models. The proposed method achieves optimal or suboptimal results in most of the metrics. The ablation study demonstrated the necessity
of each design part.
strength_weakness: strength_weakness: The weak point of the paper is its presentation. Many terms are introduced without explaining them properly, e.g. "the tight upper bound" or "the masked autoencoding strategy”. Furthermore, the Fig 2 is not well-designed with content aware branch. The Fig2 splits the proposed architecture and the content-aware pretraining and masked autoencoding strategy, which does not help the reader to understand architecture with figure. Lastly, it would be highly beneficial to make the code publicly available to enhance collaborative efforts and facilitate the sharing of this work. Apart from this, there are quite a few typos and grammar mistakes that should be corrected, such as “the can be”, “masked auto-encoding / autoencoding
questions: questions: See Weaknesses.
limitations: limitations: See Weaknesses.

Comments: title: Would you please have a look at the author rebuttal?
Response to the authors.
comment: Thank you for the rebuttal. The authors have effectively addressed my concerns. I will raise my score to 6.
Dear Reviewer,

Thanks a lot for contributing to NeurIPS2024.

The authors have provided detailed responses to your review. Would you please have a look at them at your earliest convenience?

Thanks again. AC
Thank you for your detailed responses. I apologize for missing the fine-tuning process in the first question. In fact, applying the fine-tuned operation for MAE is general, and I thought this step was omitted in the paper, which caused my confusion. Other doubts have also been resolved, and I will improve my score to weak accept.
Thank you for the rebuttal. The authors have effectively addressed my concerns. I will raise my score to 7.