SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoxiao Sun
xiaoxiao.sun@anu.edu.au
Nidham Gazagnadou†
nidham.gazagnadou@sony.com
Vivek Sharma†
viveksharma@sony.com
Lingjuan Lyu
Hongdong Li†
hongdong.li@anu.edu.au
Liang Zheng†
liang.zheng@anu.edu.au
Hand-crafted image quality metrics, such as PSNR and SSIM, are commonly used to evaluate model privacy risk under reconstruction attacks. Under these metrics, reconstructed images that are determined to resemble the original one generally indicate more privacy leakage. Images determined as overall dissimilar, on the other hand, indicate higher robustness against attack. However, there is no guarantee that these metrics well reflect human opinions, which offers trustworthy judgement for model privacy leakage. In this paper, we comprehensively study the faithfulness of these hand-crafted metrics to human perception of privacy information from the reconstructed images. On 5 datasets ranging from natural images, faces, to fine-grained classes, we use 4 existing attack methods to reconstruct images from many different classification models and, for each reconstructed image, we ask multiple human annotators to assess whether this image is recognizable. Our studies reveal that the hand-crafted metrics only have a weak correlation with the human evaluation of privacy leakage and that even these metrics themselves often contradict each other. These observations suggest risks of current metrics in the community. To address this potential risk, we propose a learning-based measure called SemSim to evaluate the Semantic Similarity between the original and reconstructed images. SemSim is trained with a standard triplet loss, using an original image as an anchor, one of its recognizable reconstructed images as a positive sample, and an unrecognizable one as a negative. By training on human annotations, SemSim exhibits a greater reflection of privacy leakage on the semantic level. We show that SemSim has a significantly higher correlation with human judgment compared with existing metrics. Moreover, this strong correlation generalizes to unseen datasets, models and attack methods. We envision this work as a milestone for image quality evaluation closer to the human level. The project webpage can be accessed at https://sites.google.com/view/semsim.
This paper studies the evaluation of privacy risks of image classification models, with a focus on reconstruction attacks [6, 43]. During inference, a target classifier, a reconstruction attack algorithm
structed
metrics
MSE
(B) original
Figure 1: Inconsistency between existing metrics and human judgements on privacy information leakage. For each original image, we present two reconstructions produced by InvGrad [8]. Below the reconstructed images, each colored  $\checkmark$  corresponds to a different metric, indicating that the corresponding metric evaluates the reconstruction to have more information leakage. In (A), according to PSNR, MSE, SSIM and LPIPS, the first reconstructed image is evaluated to have more privacy leakage [8, 7] than the second one (i.e., the first one has a higher PSNR, SSIM values, and a lower MSE and LPIPS values). However, human annotators perceive the first image as having less privacy leakage, since they cannot recognise this recognition (in contrast to the second reconstruction, which is recognizable and suggested to have more information leakage). Such inconsistency in privacy assessment is our key observation and motivation. Moreover, we observe in (B) that even these metrics themselves often disagree with each other.
and a test set are used. For each original test image, the attack algorithm intercepts gradients of the target model to obtain a reconstructed image [7, 40]. The evaluation objective is to measure whether the reconstructed image leaks any private information of the original one.
In the literature, objective evaluation metrics [29, 25] such as peak signal-to-noise ratio (PSNR), mean squared error (MSE) and structural similarity index (SSIM) are commonly used. They measure the similarity between two images on the pixel-level. In common practice, the high similarity between the original and reconstructed image indicates a good reconstruction attack, thus a more vulnerable classification model. Conversely, the low similarity between the two images means poor reconstruction, which is believed to indicate weak privacy risk.
However, it is often subject to human perception whether privacy is leaked or preserved. In Figure 1, we show examples where hand-craft evaluation metrics, such as PSNR and SSIM, and CNN-feature-based metric learned perceptual image patch similarity (LPIPS) [39] give different judgments of privacy assessment on reconstructed images from human perception. For example, in (A), the reconstructed image that is recognizable (privacy-leaked) by annotators is evaluated as better privacy-preserved by PSNR, SSIM, and LPIPS. In (B), sometimes some of these metrics provide consistent judgments with human annotators, but their evaluation accuracy is still unstable for different images.
In light of the above discussions, this paper raises the question: is model privacy preservation ability as measured by existing metrics faithful to human perception? To answer this question, we conduct extensive experiments to study the correlation of model privacy preserving ability measured by human perception and existing evaluation metrics. Specifically, for each reconstructed image, we ask five independent annotators whether the reconstruction is recognizable. We use the average annotator responses over the test set as human perception of privacy information leakage. On a wide range of scenarios (5 datasets of different concepts, many different classification models and 4 reconstruction attack methods), we find that there is only a weak correlation between human perception and existing metrics. It suggests that a model determined as less vulnerable to reconstruction attacks by existing metrics may actually reveal more private information as judged by humans.
Recognizing such discrepancy, we propose a new learning-based metric, semantic similarity (SemSim), to measure model vulnerability to reconstruction attack. Using binary human labels that indicate whether a reconstructed image is recognizable, we train a simple neural network with a standard triplet loss function. For an unseen pair of images, we extract their features from the neural network and compute their  $\ell_2$  distance, which is referred to as the SemSim score. If a model has a low (resp. high) average SemSim score, it is considered to have a high (resp. low) risk of privacy leakage. We experimentally show models' vulnerability to reconstruction attack which is ranked by SemSim has a much stronger correlation with human perception than existing metrics. Our main contributions are summarized below.
- We find model privacy leakage against reconstruction attacks measured by existing metrics is often inconsistent with human perception.
- We propose SemSim, a learning-based and generalizable metric to assess model vulnerability to reconstruction attack. Its strong correlation with human perception under various datasets, classifiers and attack methods demonstrates its effectiveness.
- We collect human perception annotations on whether privacy is preserved for 5 datasets, 14 different architectures of each set, and 4 reconstruction methods. These annotations will become valuable benchmarks for future study.
Image quality and similarity metrics are usually used to indicate the performance of reconstruction attack approaches [42, 43, 41] and also in privacy assessment [7, 36, 32] of methods against reconstruction attacks. These metrics can be broadly categorized into pixel-level and perceptual metrics. Pixel-level metrics, such as PSNR [14, 29] and MSE [34], evaluate differences between pixel values of the original and reconstructed images [7, 36, 32], to reflect the degree of privacy leakage. Perceptual metrics, such as SSIM [35] and LPIPS [39] are designed to take into account the perceptual quality of images for privacy leakage evaluation [13]. This paper examines the effectiveness of these metrics in privacy leakage evaluation and finds they exhibit weak correlation with human annotations.
Reconstruction attacks [43, 8, 41, 42, 1] aim to recover the training samples from the shared gradients. Phong et al. [26] show provable reconstruction feasibility on a single neuron or single layer networks, which provide theoretical insights into this task. Wang et al. [33] propose an empirical approach to extract single image representations by inverting the gradients of a 4-layer network. Meanwhile, Zhu et al. [43] formulate this attack as an optimization process in which the adversarial participant searches for optimal samples in the input space that can best match the gradients. They employed the L-BFGS [19] algorithm to implement this attack. Zhao et al. [41] extend the approach with a label restoration step, hence improving speed of single image reconstruction. We focus on model privacy assessment against reconstruction attacks and evaluate different metrics using several attack methods.
Human perception annotations play an essential role in evaluating machine learning models [22, 20, 27, 38]. Most public test sets, such as the ImageNet [2] dataset from the computer vision, are annotated by humans, allowing for conventional evaluation. Moreover, human feedback has been used to improve machine learning models, such as InstructGPT [24]. In fields where human annotations were expensive to obtain, e.g., medical image analysis [37] and image generation [28], there is increasing evidence that the human judgements or evaluation is valuable and offers new insights. In our paper, we consider the information leakage of reconstructed images
3 Privacy Assessment Metrics on Reconstructed Images: A Revisit
Pipeline of privacy assessment on the reconstructed images. As shown in Figure 2, the goal of evaluation is to compare privacy risks of a series of  $K$  image classification models  $\{\mathcal{M}_k\}_{k=1}^K$ , under reconstruction attacks. The evaluation process simulates stealing data from gradients [43, 41]. Its input consists of an original image set  $\mathcal{X} = \{\mathbf{x}_i \in \mathbb{R}^{m \times n}\}_{i=1}^N$ , where  $N$  is the number of images, and a reconstruction algorithm  $\mathcal{A}$  used to attack models  $\{\mathcal{M}_k\}_{k=1}^K$ . Given a target model  $\mathcal{M}^1$ , whose parameter weights are denoted by  $\mathcal{W}$ , its gradients  $\nabla \mathcal{W}_{\mathcal{X}}$  can be calculated using the original data  $\mathcal{X}$ . The attack algorithm  $\mathcal{A}$  is applied to the target model  $\mathcal{M}$  and its gradients  $\nabla \mathcal{W}_{\mathcal{X}}$  to obtain a set of reconstructed images denoted by  $\bar{\mathcal{X}} := \mathcal{A}(\mathcal{M}, \nabla \mathcal{W}_{\mathcal{X}}) = \{\bar{\mathbf{x}}_i\}_{i=1}^N$ . Note that,  $\mathcal{A}$  can access the gradients, but has not access to  $\mathcal{X}$ . We can evaluate the privacy leakage of a target model  $\mathcal{M}$  over the original set of  $\bar{\mathcal{X}}$  as follows:
$$
\operatorname {P L} (\mathcal {M}) := \operatorname {I n f o L e a k} (\mathcal {X}, \bar {\mathcal {X}}) = \operatorname {I n f o L e a k} (\mathcal {X}, \mathcal {A} (\mathcal {M}, \nabla \mathcal {W} _ {\mathcal {X}})), \tag {1}
$$
where InfoLeak  $(\cdot ,\cdot)$  represents the amount of information leakage in reconstructed images. Therefore, it is important to have an effective metric for indicating InfoLeak  $(\cdot ,\cdot)$ .
1 Unless explicitly stated otherwise, the subscript of  $\mathcal{M}$  is omitted when this does not create ambiguity.
Figure 2: Task definition: privacy leakage assessment on reconstructed images. Given  $K$  classification models  $\mathcal{M}_1, \mathcal{M}_2, \ldots, \mathcal{M}_K$  against the image reconstruction attack  $\mathcal{A}$  (we use  $K = 3$  as an example in this figure) on a set of original images  $\mathcal{X}$ . For each model, we get a set of reconstructed images. The main goal of privacy leakage assessment on reconstructed images is to measure whether semantic information of an original image, is still accessible. We can ask human annotators to evaluate whether they can recognize the image class and then average across the set of images to obtain the overall human evaluation score of privacy leakage. In the existing literature, image quality metrics, such as PSNR, are used to measure privacy leakage. Here, the evaluation of example images shows again that PSNR deviates from human evaluation.
Information leakage formulation. As introduced in Section 2, information leakage is often assimilated to reconstruction quality and is based on a distance between an original image  $\mathbf{x}_i$  and its reconstructed counterpart  $\bar{\mathbf{x}}_i$ . Under such pointwise metric, InfoLeak  $(\cdot ,\cdot)$  of an image set  $\mathcal{X}$  and its reconstructed set  $\bar{\mathcal{X}}$  can be defined as:
$$
\operatorname {I n f o L e a k} (\mathcal {X}, \bar {\mathcal {X}}) = \frac {1}{N} \sum_ {i = 1} ^ {N} d \left(\mathbf {x} _ {i}, \bar {\mathbf {x}} _ {i}\right), \tag {2}
$$
where  $d$  can be a hand-crafted metric, such as MSE [34], PSNR [14, 29] or SSIM [35], or model based, such as LPIPS [39]. Equation (2) averages the distances or similarities over all the original - reconstructed image pairs to obtain the information leakage score of the attacked model  $\mathcal{M}$ . Apart from these, we can also use Fréchet Inception Distance (FID) [10]. It measures information leakage as the distribution difference between original and reconstructed images: InfoLeak  $(\mathcal{X},\bar{\mathcal{X}})\propto \mathrm{FID}(\mathcal{X},\bar{\mathcal{X}})$ .
4 Diagnosis of Existing Metrics and Our Proposal
4.1 Collecting human assessment of privacy leakage from reconstructed images
To evaluate whether a reconstructed image leaks privacy, human perception offers very useful judgement. In the context of image recognition and face recognition, it is to determine if the human can still recognize the reconstructed object or face.
For image classification, given an image, we provide human annotators with an incomplete list of classes. For example, for the CIFAR-100 dataset, instead of providing annotators with a list of all the 100 classes which are hard to memorize, we provide them with a list of the top-20 possible classes that includes the ground truth. We request annotators to annotate the class of a given image. If the annotate thinks the images is "incomprehensible" (i.e., severely blurry) or the right class does not appear in the candidate list, then the annotation is 'none'. We compare the human annotations between an image and its reconstructed version. If they are the same, privacy is not preserved; otherwise, privacy is preserved. The annotation pipeline and more details of the annotation process are provided in the supplementary material.
For face recognition and fine-grained image recognition, because it is by nature very difficult for a human to assign a class label from 20 candidates, we give annotators two images at a time: an original image and its reconstruction. We then ask the annotator to tell whether the two images
(A) correlation between existing metrics themselves
(B) correlation between the existing metrics and human perception
Figure 3: Correlation between existing metrics and their alignment with human perception in measuring privacy risk. 14 classification models are attacked by InvGrad [8] on the CIFAR-100 dataset. Each subfigure presents the correlation between the rankings of model privacy leakage obtained by two metrics. The correlation strength is measured by Spearman's rank correlation  $(\rho)$  [31] and Kendall's rank correlation  $(\tau)$  [16]. Between existing metrics, (A) indicates that correlation is sometimes very weak. Furthermore, (B) indicates that the correlation between existing metrics and human perception is generally weak.
contain the same person or category. If yes, then privacy is not considered as preserved; otherwise, it is. Note that in this procedure, to mitigate the potential bias of annotators, we also give reconstructed images that do not pair with the original image.
In all the above procedures, each image or image pair is labeled by 5 independent annotators. Binary labels, i.e., whether a reconstructed image is recognizable, are obtained via majority voting. In this study, we deal with five datasets: CIFAR-100, Caltech-101, Imagenette and Celeb-A and Stanford Dogs $^2$ . For each classification model being attacked, we annotate 600, 700, and 100 reconstructed images for the CIFAR-100, Caltech-101, and the other three datasets, respectively.
4.2 Correlation analysis between human perception and existing metrics
Examples from Figure 1 motivate us to conduct a more comprehensive analysis of the inconsistency between human perception and existing metrics in terms of privacy leakage. To this end, for the reconstructed image set of 14 target models, we plot their privacy risk measured by various metrics against collected human labels in Figure 3 (B). We find that the correlation strength between human evaluation and existing metrics is relatively weak. For example, Kendall's rank correlation  $\tau$  that measures rank consistency is only 0.2904 and 0.3978 for SSIM and LPIPS. Even in the best case, i.e., FID vs human, correlation is only moderate with  $\tau = 0.5604$ . It signifies that a model identified as more robust against reconstruction attacks based on existing metrics may actually be perceived as highly vulnerable according to human judgment when comparing different models.
The primary issue lies in the fact that existing metrics are computed on either a pixel-wise or patchwise basis, without considering the semantic understanding of privacy leakage. As a result, these metrics fail to accurately capture the image semantics related to privacy risks. This problem motivates us to design privacy-oriented metrics to better assess privacy leakage.
4.3 Proposed metric
To obtain a metric that is more faithful to human perception, we propose SemSim, a learning-based metric using human annotations as training data. The pipeline of SemSim is presented in Figure 4.
Training. Using binary human labels whether a reconstructed image is recognizable, we train a simple neural network  $f_{\theta}$  with a standard triplet loss function. We take the original image  $\mathbf{x}_i$  as an anchor
2The new annotated dataset is distributed under license CC BY-NC 4.0 1, which allows others to share, adapt, and build upon the dataset and restricts its use for non-commercial purposes.
Figure 4: Training and inference pipeline of SemSim. Feature extractor  $f_{\theta}$  is trained on human-annotated images with a triplet loss [30]. An original image  $\mathbf{x}$  is used as anchor, and its reconstructions are split into positive (recognizable) and negative (unrecognizable) samples based on human annotations (Section 4). The goal is to minimize the anchor distance to positive samples and maximize that to negative ones. During inference, given an original image and its reconstruction, we use  $f_{\theta}$  to extract their features and compute the  $\ell_2$  distance between the two features.
and split its reconstructions into positive  $\bar{\mathbf{x}}_i^+$  and negative  $\bar{\mathbf{x}}_i^-$  samples based on human annotations. The loss function is  $L = \sum_{i=1}^{N} \max \{d(\mathbf{x}_i, \bar{\mathbf{x}}_i^+) - d(\mathbf{x}_i, \bar{\mathbf{x}}_i^-) + \alpha, 0\}$ , where  $\mathbf{x}_i$  is an original image and  $\bar{\mathbf{x}}_i^+$  (resp.  $\bar{\mathbf{x}}_i^-$ ) stands for one of its recognizable (resp. unrecognizable) reconstruction, and  $\alpha$  is the margin. Thus, we obtain our neural network  $f_\theta$  trained on human-annotated datasets.
Inference. During the evaluation,  $f_{\theta}$  is used for extracting features for original and reconstructed images. We calculate the  $\ell_2$  distance between their feature vectors, that is  $SemSim(\mathbf{x},\bar{\mathbf{x}}) = \ell_2(f_\theta (\mathbf{x}),f_\theta (\bar{\mathbf{x}}))$ , and then average this score over test set as the overall model performance score.
Key Observations. We believe SemSim captures semantic information, which plays a crucial role in privacy preservation. There are several key factors contributing to its effectiveness. (1) Being trained on human annotations enables SemSim to capture privacy leakage semantics better than metrics based on pixel-level similarity or patch CNN features. (2) By utilizing a CNN model that extracts relevant higher-level features, SemSim captures visual information related to information leakage effectively. (3) It incorporates the relationship between the original image and recognizable/unrecognizable reconstructions, improving its accuracy in assessing privacy leakage and providing better privacy assessment. SemSim has a limitation in that it requires annotated data for training. While we show that it is very generalizable and can work better than existing metrics with limited training data (refer to Figure 7), we prioritize our future work to annotate more data for even improved generalization.
Are there other ways than human perception to assess privacy leakage? Yes. We can use a classification model trained on a dataset that contains the same categories as the reconstruction set to classify the reconstructions. If the model accurately predicts the categories, it indicates a potential privacy leakage. In our preliminary study, we used two models trained on the CIFAR-100 dataset, achieving accuracies of  $82\%$  and  $65\%$  on the test set, respectively, for recognizing reconstructed images. By using their recognition accuracies as indicators of privacy leakage, we obtained Kendall's rank correlation coefficients of 0.7023 and 0.5044 with human evaluation, respectively. These results are considered acceptable. However, there are limitations to this approach. The classification model must be trained on a dataset that matches the categories of the task, and it needs to be accurate. These limitations affect the scope of this method. Nonetheless, exploring the use of classifiers to evaluate privacy risk offers an alternative viewpoint to human perception, and it merits further investigation
Is privacy leakage on reconstructed images a binary problem? No. We simplify this problem by binarizing it. It can be continuous, where privacy information is leaked to a greater or lesser degree, depending on various factors such as the task and the type and amount of data that is leaked.
How to define privacy leakage on reconstructed images in other vision tasks? The definition depends on the task context. For example, in object counting [23], privacy information can be defined as the number of objects. Therefore, for different tasks, the definition of privacy leakage should be carefully designed and accompanied by a tailored evaluation method.
Relationship between image quality and private leakage of reconstructed images. The relationship between the image quality of a reconstructed image and its information leakage is complex. While better image quality can indicate better reconstruction performance, it does not necessarily imply higher privacy leakage. Conversely, a reconstructed image with poor image quality can still contain private information, while an image with higher quality may preserve privacy better. Therefore, the relationship between image quality and privacy leakage is not always straightforward and
Figure 5: Sample annotation results. For each original image (leftmost column), its reconstructed images are placed left to right by their PSNR values from large to small. The red cross denotes that the human annotator fails to recognize the image. We observe that human evaluation is inconsistent with PSNR ranking, e.g., some images that are top-ranked, or equivalently determined as high quality by PSNR are actually not recognizable by humans.
requires careful consideration and evaluation. These discussions also encourage us to explore new metrics that incorporate semantic-level information in order to better assess privacy leakage.