Towards Eliminating Hard Label Constraints in Gradient Inversion Attacks
Authors: Yanbo Wang
Jian Liang
Ran He
Venue: ICLR 2024 poster (2024)
Keywords: label recovery
gradient inversion attacks
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We initialize a novel method to analytically recover the augmented label as well as last layer features from single input in gradient inversion attacks, disclosing a necessary condition for all analytical-based label recovery methods.

Abstract: Gradient inversion attacks aim to reconstruct local training data from intermediate gradients exposed in the federated learning framework. Despite successful attacks, all previous methods, starting from reconstructing a single data point and then relaxing the single-image limit to batch level, are only tested under hard label constraints. Even for single-image reconstruction, we still lack an analysis-based algorithm to recover augmented soft labels. In this work, we change the focus from enlarging batchsize to investigating the hard label constraints, considering a more realistic circumstance where label smoothing and mixup techniques are used in the training process. In particular, we are the first to initiate a novel algorithm to simultaneously recover the ground-truth augmented label and the input feature of the last fully-connected layer from single-input gradients, and provide a necessary condition for any analytical-based label recovery methods. Extensive experiments testify to the label recovery accuracy, as well as the benefits to the following image reconstruction. We believe soft labels in classification tasks are worth further attention in gradient inversion attacks.

Published as a conference paper at ICLR 2024
TOWARDS ELIMINATING HARD LABEL CONSTRAINTS IN GRADIENT INVERSION ATTACKS
Yanbo Wang .2, JJan Liang": Ran Hel.2+
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
PCRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA) wangyanbo2023e ia ac cn, Liangjjan922gmaill com, rhe@nlpr. ia ac.cn
ABSTRACT
Gradient inversion attacks aim to reconstruct local training data from intermediate gradients exposed in the federated learning framework. Despite successful attacks, all previous methods, starting from reconstructing single data point and then relaxing the single-image limit to batch level, are only tested under hard label constraints. Even for single-image reconstruction, we still lack an analysis-based algorithm to recover augmented soft labels. In this work, we change the focus from enlarging batchsize to investigating the hard label constraints, considering a more realistic cireumstance where label smoothing and mixup techniques the training process. In particular, we are the first to initiate a novel algorithm to simultaneously recover the ground-truth augmented label and the input feature of the last fully-connected layer from single-input gradients, and provide a necessary condition for any analytical-based label recovery methods. Extensive experiments testify to the label recovery accuracy, as well as the benefits to the following image reconstruction. We believe soft labels in classification tasks are worth further attention in gradient inversion attacks]
used in
INTRODUCTION
Federated distributed learning (Shi et al. 2022 Luo et al. 2021) enables multiple participants to train one specific model collaboratively so as to increase efficiency and settle privacy concerns (McMa- han al. 2017 Yang al. 2019 Li et al., 2014). During the information sharing process, only intermediate gradients or parameters are transmitted to the central server under a common horizontal federated learning paradigm. which is believed effectively protect the input data privacy (Lyu et al.
2022
Wei et al. 2020; Zhang et al. 2022).
Even if federated learning is developed for its privacy protection capability, recent works have proved that with accurate gradient information, input training data could be reconstructed under ideal circumstances (Zhu & Blaschko 2020, Wang et al.] 20200) through gradient inversion attacks (GIA). Representing works achieve such goal mainly by establishing gradient matching process, using the matching loss between dummy gradients from randomly initialized input data and ground-truth
gradients optimize dummy data towards ground-trutt local data (Geiping et al. 2020 Yin et al.
2021 Zhu et al.) 2019).
When revisiting the series of works, we find that their contributions mainly focus enlarging the batchsize. They try various gradient matching functions, different regularization terms (Geiping et al. 2020 Yin et al. 2021), ,, and label recovery algorithms (Zhao al.) 2020 Ma et al. 2023) to increase the reconstruction accuracy. Therefore, works after DLG (Zhu et al. (019)) split the matching process into two steps: recover the one-hot label at first and then optimize the input image based on ground-truth labels. However, has the single-image reconstruction task been perfectly solved? The answer is negative, and we still identify crucial limitations for the basic single-image GIA when faced with soft labels. In real-world settings, label augmentation techniques, such as label smoothing (Szegeyy et al. 2016) and mixup (Zhang al. 2018a), are generally used enhance
"Corresponding author
'Ou code is publicly available at
19/1abe1 label 1aa

---

Published as a conference paper at ICLR 2024
model robustness and generalization capability. Previous works, relying the sign of gradients to decide the ground-truth one-hot label (Zhao et al. 2020; Yin et al. (021), are unable to handle augmented labels, for they will disable the sign indicator. Consequently, these label augmentations may serve as a simple defense against GIA (Huang et al. 2021), even though they are not designed to enhance model safety. Besides, previous literature also mentioned that for fully-connected networks (FCN), input data could only be reconstructed from gradients analytically when layers have non-zero bias term (Aono al. 2017p. Zhu & Blaschko (2020) propose recursive analytical data recovery algorithm regardless of the bias term, but it only works for binary classification tasks. Without strict limitations as above, current analytical methods cannot even handle the simplest FCN.
Aware of such constraints in single-image label recovery. this work initiates novel algorithm to retrieve accurate augmented labels as well as the last layer features from corresponding gradients, regardless of the existence of the bias term. We identify mathematical features of general multi-class classification tasks with cross-entropy loss and successfully break the label reconstruction task into solving one scalar from equations. To figure out the specific scalar, a simple but effective label reconstruction loss is adopted to search for it while trying to avoid local minima. Once such scalar is solved, both the input feature and the augmented label can be recovered precisely. Extensive experiments on various datasets under multiple networks demonstrate the correctness of such scalar.
Based on our algorithm, then explain why the bias term matters for previous feature recovery methods and point out a necessary condition to analytically recover augmented labels. Furthermore, convincing comparative experiments are designed to evaluate the image reconstruction profits from precisely recovered augmented labels. We conclude that the proposed method could raise image reconstruction accuracy to the level that achieved with full knowledge of input labels.
To summarize, our contributions include:
Initiating a novel analytical algorithm to accurately reconstruct the augmented label of the single input image with input features, and more importantly, disclosing a necessary condition for all analytical-based label recovery methods;
Analyzing limitations for previous bias-based recovery methods, proposing the first method to analytically reconstruct input image from FCNs under multi-class classification tasks based on recovered last-layer features regardless of the bias term;
Designing extensive experiments to demonstrate the label recovery accuracy and image reconstruction profits from our proposed algorithm, claiming that gradient inversion attacks could still be effective under real-world settings.
2 RELATED WORK
Label recovery methods. Previous methods mainly focus on retrieving one-hot labels from single images to batches. Zhao al. (2020) propose iDLG, identifying that for multi-class classification with cross- cross-entropy loss, the sign of last fully-comnected layer gradients for the entry of ground-trute class different from any other entries Yin et al. (2021) extend this theory to derive batched labels from the minimum value of each entry's gradients; Dang et al. (021) propose RI.G based on singular value decomposition to recover one-hot labels of whole batch, which achieves better results; to nullify the limit that each class could at most have one image in batch, Geng et al. (2021) improve iDLG, replacing unknown variables by the mean value to calculate labels for every class; Ma et al. (2023) then identify consistency in gradients of each class, constructing linear equations solve instance- wise one-hot labels. However, for non-one-Hee hot labels, such as label smoothing (Szegedy al. 2016) and mixup (Zhang et al., 2018a) where y; < 1 so the sign of p; - y: for ground-truth class could still be positive, previous analytical methods fail to recover accurate label distribution. Limited progress has been made in this direction.
Optimization-basel gradient inversion attacks. For optimization-baxt methods, the server initializes a data point (x, y), minimizing the gradient matching distance between the ground-truth ,e","" and the random data (I, y) to push (x, y) towards the ground-truth distribution (x",y")
arg min D (VoLo (x. VoCo (x" vv")).
(1)
Pleasee refer Section 3.1 for mathematical details.
2

---

Published as a conference paper at ICLR 2024
where D represents the matching distance. Zhu et al. (2019) first adopt Euclidean distance to simultaneously optimize input data and labels. Subsequent works try different distance functions and add various regularization terms real image priors for larger batches. Geippng et al.) (2020)
adopt cosine similarity as the distance function, and add total variance (Rudin et al. 1992]; Wang al. (2020) replace the L2 distance by Gaussian kernel function, and apply different weights for different layer gradients: Yin et al. (202I) keep the original distance function, but borrow multiple regularization terms from DeepInversion (Yin et al. 2020) to strictly penalize the image towards realistic distribution; with similar logic, image recovery from Vision Transformers (Dosovitskiy et al.
2021 is also achieved (Hatamizadeh et 2022).
Other image reconstruction methods. Aono et al. 2017) first raise the bias attack to recover features from corresponding gradients in biased connecconeected layer. Consider a fully-connected layer 2 = Wx b, aw, term, feature x could be retrieved from the division. Ma et al. (2023) also mention such methods linear equation system; Zhu & Blaschko 2020) initiate R-GAP. which solves a series of equations via Moore- Penrose pseudoinverse get features from logits layer by layer, and finally to the original image. However, These two works depend more on the network architecture and parameters: data recovery methods will fail if the layer is rank-deficient. |ann et al. (2022) propose neuron exclusivity analysis to decouple batched image reconstruction into multiple single-image reconstructions. [Balunovic al. (2022) unify several gradient matching functions with Bayesian framework and break several existing defenses in image reconstruction. Apart from these, Dang et al. (2021); Jeon et al. (2021)
8c(zyy)] 8C(z.y) x aw, ac(z.y) xT, we know the gradients of the bias 82.-
ab.
as their thcoretical intuition. After that, Fan et al. (2020) formulate image reconstruction
also include GAN (Goodfellow et 2020) for GIA, which requires extra data for GAN training.
3 LABEL INFERENCE FROM FULLY-CONNECTED NETWORKS
Threat model. Gradient inversion attack aims to reconstruct input data from gradient information in federated learning. Following the general setting where the
server wants to reconstruct private training data and labels (Li ct al., 2022 Dang et al., 202T), it knows exactly the global model architectures well as parameters 4 (white-box attack), and could get full access to gradient uploads g* VoLo (x", .y") from clients. The server cannot actively change the training protocol to facilitate a better attack.
3.1 FROM RECOVERING A LABEL. VECTOR OPTIMIZING SCALAR X
Considering the last fully-connected layer without the bias term f R' + RC which outputs logits z from input x after activation, we have f(x) Wx, where I is the dimension of the input
feature, represents the number of classes. Then the cross-entropy loss L(z,y) = ye log Ps. where y refers to the label vector and p; is the i-th entry of post-softmax probability. We pick <(-) as the softmax function, then p; o(z, i) Σ from cross-entropy loss, we have
Taking derivatives of row in weight matrix W
aC(z,y) y) dC(z,y) dz, ow,
Oz, OW, (p,- -x
Apparently, the gradient of the last fully-connected layer just scaled input vector. If do not know input x, we are unable to solve each entry of y from Eqn. (2). However, label vector y could be regarded as function of gradients g. acizy) For simplicity, pick entry of label y to finish oW.
the equation 5
Yi pj (pi vi)
Φ (f(x*)) )); gi の (f((p) - yr) -v-)"'g))) (pr
x"
gi gr
Vecctor division makes sense here because the results of item-wise division are identical for every entry.
3

---

Published as a conference paper at ICLR 2024
where x" is the ground truth input feature. If we replace the ground-truth input with Ar in which A is a non-zero scalar to be optimized, then the pseudo label could written as
D: = (f(Ag)))) Bz/().....
When =p, yr) Eqn. and Eqn. (4) are equivalent and then we get the ground-truth imput feature A, gr with label y". With different Ay. could get different distribution. Based on this, our next goal is to design a target function to optimize this scalar A,- One point worth noting is that for all equations above, any random index satisfies the formula. Therefore, each non-zero gr has its own X,. As long as we figure out one non-zero X, for any index r, the ground-truth label could be successfully recovered.
3.2 VARIANCE LOSS FUNCTION
Here we take label smoothing and mixup techniques our main focus in label augmentations. For label smoothing without access to the smoothing probability, the simplest way to regularize the label distribution is to minimize the variance of all label entries but the top one. Similarly, for mixup labels, we pick the variance loss of all but the top two entries.
Ltabel = C
Σ (u. - (E)=))(())², iES
(5)
||
where S is the exclusion set containing the indexes of top items, ||S]| is the size of S, and E means the expectation operator. During the process of picking X,, y is changing so entries in S are not fixed. Ideally, when such label perfectly recovered, the variance loss of sll-but-xxclusion-eee entries should reach the global minimum of O. It is worth highlighting that the target function is not unique, and other functions may also work. experiments we find such simple variance does perform exceptionally well, and we will show the results in Section
3.3 SEARCHING FOR THE GLOBAL MINIMUM
Based on Eqn. (5) the label recovery problem could be transformed to finding the global opti- mum in the Llabel function. Here first consider popular gradient-descent optimizers, such as Adam (Kingma & Ba 2014). L-BFGS (Liu & Nocedal 1989). etc. For most untrained networks, whose variance loss shown in Fig. [(a) the global minimum just the first local minimum (we start searching from 1), thus these gradient-descent-bane -based algorithms could find the optimum easily and quickly. However, such method suffers from multiple local minima and vanishing gradients,
especially when retrieving labels from trained networks, whose Llabel is shown in Fig. [(b) ((c)
17.5
Lobel smoothing
Label smoothingg
bruth:4444 1745
2.5
Lobel smoothing
4ruth
15.0
12.5
2.0
10.0
1.5
75
1.0
5.0
25
0.5
0.0
0.0
5.0
2236 2,277
(a) Liabel under untrained ResNet18. Liahel under trained (esNet18.(C (c) A detailed illustration of the red- Clearly the first local minimum is the The first local minimum is not the box in Fig. l(b) Liabet does global one.
global one.
not get zero.
Figure 1: Variance loss under untrained and pretrained ResNet18. 8. The gradient-based optimizer may find local minima at approximately 2.2372 while the ground-truth is 4.2745.
Under such cireumstances, Particle Swarm Optimization (Kennedy & Eberhart 1995) is better choice to find the global minimum, for it will not suffer from vanishing gradients and local minima. In experiments first use L--BGGS to try find the global optimum, then PSO if L-BFGS fails.
4

---

Published as a conference paper at ICLR 2024
3.4 THE NECESSARY CONDITION IN LABEL RECOVERY ALGORITHMS
One important conclusion derived from our label recovery problem that the ground-truth label conditioning gradient information is function of the scalar A,. Faced with such uncertainty, only have two handles for X, picking:
All entries of the valid label should sum 1.
All entries of the valid label are and only could be non-negative.
Theorem 1. For any Ar and gradient information gr. all entries of the corresponding pseudo label sum to 1.
The proof is attached in Appendix A Therefore, could only focus on the second handle. However, this non-negative principle not always usefu H either. we only get access to the gradient information, there would be range for in which any A could produce reasonable label. Here is an intuitive example of label smoothing augmentation.
fre
A, 2.00, Suse 13.5007
0.7
A.-
Cure
PoR Soretox Lavel Prossality
PoR--5 Sur-So Labbll Probabilky
0.5
Post SoftMax Label Probabilley
0.6-
0..4
0.5
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.00
1
4
5
1
5
0
4
6
Class
Class
Class
=
1,3665, Faser
0,0000
A.= 1,33, Kusae
0.0917
4,
1,00, Cuser
-1.32677
Post SuttSu Probabiity
0.7
512
Post SoftMox Labe Probability
0.7
435
Pratt SoftMox Libve Probabiltty
0.6
0.6
0.8
0.5
0.6
0.4
0.3
0.2
0.2
0.2
-02
0.59
0.0
0.0
-0.4
Figure 2: Recovered label distribution and variance loss given same gradient information. When we alter the scalar Ay, the class probability would vary. we have no more information about label distribution, all labels except the left-top and right-bottom ones could be the right labels generating exactly the same gradients.
As shown in Fig. 2 least any Xy ranging from 2..0 to -133 (where entry in probability distribution almost reaches 0) would generate labels where each entry is positive and sums to 1. That why need the variance loss supervise the label distribution. Based this, label distribution feature is necessary condition as guidance to pick one specific label from the series controlled by Ay. Label distribution feature refers to features that could be utilized to design the target function for label recovery. Such theory is also applicable to mixup labels. Actually, our algorithm could be easily adapted other augmented labels, as long as we know the special feature of that kind of label distribution and then design new loss to guide X, picking. Otherwise, these two handles without prior knowledge of label distribution are too weak to guarantee an accurate recovery.
EXPERIMENTS
4.1 LABEL RECOVERY ACCURACY
We first test the label recovery accuracy with CIFAR- 100 (Krizhevsky 2009), Flowers- (Nilsback) & Zisserman 2006) and ImageNet (Russakovsky al. 2015) on ResNet50 (He et al.) 2016) network.
UUder some circumstances this handle could help directly optimize the ground-truth scalar. More details could be found in Appendix C

---

Published as a conference paper at ICLR 2024
In label smoothing settings, randomly pick 1000 images from the testset of each dataset (for Flowers-17 17 we pick images from the whole dataset). To test the robustness our algorithm, we randomly pick the smoothing probability from (0, 0.5) for every sample. Besides, also consider the one-hot setting, which a special case for label smoothing when the smoothing probability is to testify to the algorithm performance. In the mixup setting, we also randomly pick 1000 image couples in different classes with the probability from (0, 1).
Table I: Experiments for label recovery accuracy.
Label
Accuracy (%) 11
Augmentation
CIFAR-100
ImageNet
Label smoothing (I.)
99.2
99.1
100.0
Mixup (M)
100.0
97.6
100.0
One-hot
100.0
95.6
99.9
iDL
100.0
95.6
99.9
As shown in Table firstly our proposed algorithm achieves satisfying recovery accuracy of above 95% for both label augmentations. Besides, our method also achieves identical performance on the one-hot setting the previous iDLG in multiple datasets, demonstrating its compatibility. To further test the performance of the proposed label recovery algorithm, we then focus on CIFAR- 10 and Flowers-17 17 to execute more intense experiments. For both datasets, we pick two different networks and test our algorithm both on pretrained and mandom-iniiialin networks, as shown in Table 2
Table 2: Intense experiments on the performance of label recovery algorithm. L, refers to the sum of
I loss of every entry recovered labels, if not otherwise specified.
CIFAR-10
Flowers-17
Network
Aug.
Acc. (%)
Avg. C,
Network
Aug.
Acc. (%)
Avg. C,
Untrained
I.
99.7
5.32x 10-5
Untrained
L
94.6
2.22x10-0
LeNet
M
99.7
3.62x 10-5
AlexNet
M
88.3
9.64x10-6
Trained
I.
99.9
2.39x1004
Trained
L
98.7
1.09x10-4
LeNet
M
100.0
1.51x10-4
AlexNet
M
98.9
6.50x10-8
Untrained
L.
100.0
8.78x10-5
Untrained
L
100.0
6.02x10-6
ResNet18
M
100.0
7.50x10-1
ResNet18
M 100.0
2.92x10-5
Trained
L. 99.2
3.17x10-7
Trained
L 99.1
7.36x10-5
ResNet18
M
99.0
3.94x10-6
ResNet18
M
99.9
2.30x10-4
Our algorithm could achieve above 95% accuracy on multiple datasets under various training con- ditions, which demonstrates its robustness. In addition, it does not experience obvious accuracy drop when the networks are pretrained with mixup or label smoothing techniques. Failure analysis for corner cases is in Appendix B
4.2 IMAGE RECONSTRUCTION FOR FULLY-CONNECTED NETWORKS
As mentioned in R-GAP (Zhu & Blaschko 2020), simply canceling the bias term every layer would disable previous bias attack (Aono et al. 2017 Geiping et al. (000).. With the proposed label recovery algorithm. the ground-truie feature of the last layer could directly be recovered from the cross-entropy loss of multi-class classification tasks, therefore reconstructing image analytically from fully-connected networks is not limited to a theoretical level.
Consider one fully-connected layer with input weight W and output then we have
Wx before activation,
ac 0x
OL
WT
(6)
Oz
*iDLG Zhao al. 2020) could only handle hard labels and therefore serves as comparison with proposed variance-opimizing methods one-hot setting. From experiments, we find that iDLG cannot guarantee 100% accuracy, which may contradict previous consensus. A detailed explanation attached in Appendix B.3

---

Published as a conference paper at ICLR 2024
0L OW
OC
òz
where we need pick to make (ow)) i.r non-zero. If the activation function is ReLU (Glorot et 2011), then #C is identical to (2)) of the previous layer. For other activation functions the logic is ax sumilar and we only need to multiply scalar.
Thus, after recovering this layer's input feature and gradients of the previous layer's output feature, we could recurrently reconstruct the input feature of previous layers. Such a method successfully avoids solving the output feature each layer, especially when the activation function is not strictly monotonic. As shown in Fig. [] our analytical-base algorithm could reconstruct high-quality images with label augmentations, which outperforms DLG and IG with a shorter running time.
Ground-truth
L2 loss
PSNR: 8.01
Cosine similarity
PSNR: 10.22
Proposed PSNR: 51.41
Figure 3: Image reconstruction comparisons FCN-4 network. Images with label smoothing compressed to 64 x 64 due to CUDA memory limitation in optimization-base methods. For more samples with mixup augmentation please refer Appendix D
4.3 RECONSTRUCTION FOR CNNS
In this part, further discuss the benefits accurate label recovery brings to image reconstruction. We pick widely used L2 (Zhu et al. 2019 Yin et al.) 2021) loss and cosine similarity (Geiping et al. 2020) the loss function, evaluating image reconstruction quality with the precisely recovered label (Our), ground-truth label (GT), one-hot label (iDLG), randomly initialized label (DLG) and our recovered label as initialization (DLG+Ours), respectively. Experiments are finished LeNet (LeCun al. 1998) architecture with CIFAR- 10 validation dataset. For DLG and DLG+Ours, labels are involved in the optimization process, while for the other three settings, we fix the labels and only optimize the input image. In the mixup setting, we have 45 different class tuples and for each tuple, we randomly pick one image in the corresponding class to mix with the probability sampling from (O, 1). In the label smoothing setting, for each class, we pick 3 random images and smooth every label with probability sampling from (0, 0.5). All experiments are repeated times. Evaluation metrics are widely used Peak Signal-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018b). Detailed results are shown in Table
Proposed method outperforms previous label recovery methods with large margin. For augmented labels, DLG method fails to recover labels accurately the proposed method. The optimization method unstable, so for different initializations the L, varies. Besides, For La loss,
A prior version of this paper had bug in the image reconstruction implementation, which is unrelated to label recovery algorithm. After corrections, all conclusions still hold.
7

---

Published as a conference paper at ICLR 2024
Table 3: Image reconstruction with label loss under various settings. T2 loss and Cosine similarity
refer to two ways of calculating matching distance D in Eqn.
PSNR
Match
DLG Ours
SSIM LPIPS
Trained
GT
iDLG
DLG
Ours
function
DLG
Ours
28.27
10.70
24.26
28.24
27.81
Lg loss
0.640
0.099
0.553
0.644
0.659
0.44
5x10-6
0.141
0.355
0.172
0.138
0.130
19.28
9.34
16.93
19.29
17.30
L/2 loss
0.628
0.020
0.500
0.628
0.519
0.15
8x10-6
Mixup
0.062
0.394
0.113
0.063
0.102
25.76
23.01
26.45
25.81
26.51
Cosine
0.858
0.745
0.881
0.859
0.882
0.11
5x10-6
similarity
0.033
0.053
0.027
0.033
0.027
24.56
15,51
22.71
24.67
24.84
Cosine
0.741
0.356
0.688
0.746
0.758
0.05
8x10-6
similarity
0.046
0.173
0.053
0.044
0.040
21.33
10.64
18.98
20.94
21.00
L2 loss
0,587
0.129
0.492
0.584
0.590
0.37
4x10-6
0.150
0.351
0.192
0.149
0.149
11.70
8.72
11.86
11.61
11.47
L2 loss
0.267
0.029
0.283
0.260
0.244
0.36
8x10-6
Label
0.250
0.401
0.236
0.259
0.264
smoothing
23.13
22.89
24.09
23.06
24.26
Cosine
X
0.820
0.807
0.865
0.818
0.868
0.20
4x10
similarity
0.034
0.032
0.023
0.035
0.022
18.99
17.40
18.56
18.98
20.42
Cosine
0.615
0.558
0.614
0.614
0.691
0.17
8x
similarity 0.095
0.121
0.088
0.095
0.069
one-hot labels derived from iDLG also fail provide satisfying image reconstruction quality. Proposed method successfully reaches image reconstruction quality as known labels. In all experiments, our proposed method recovers high-quality labels with L, under 8.5 x 10-6, Under all experimental settings, recovering images with recovered labels is almost identical to recovering with ground-truth labels in all metrics, demonstrating the effectiveness of our algorithm.
Proposed method could serve as good initialization for cosine similarity loss. For cosine similarity loss, under the untrained setting, optimizing an image-label tuple simultaneously could outperform the image quality that is recovered from the ground truth label. Even though it special character of cosine similarity loss itself, we do find that optimizing the image-label tuple with recovered label as initialization could increase image reconstruction quality. More importantly, such margin would be enlarged under the pretrained setting, demonstrating the value of our algorithm more convincingly.
EXTENSION STUDIES
RECONSTRUCTING IMAGE FROM GRADIENTS AND FEATURES
Given that label recovery algorithm could recover features and labels without the bias term simultancously, we could include last-layer features in the matching loss to evaluate the profits our algorithm brings to gradient inversion attacks:
N-2
Lmatch
D(g.,g;") DDD(N-119NN-1 (D((N-1,
(8)
8

---

Published as a conference paper at ICLR 2024
where and 3 refer to the weights of last-layer gradient loss and feature loss. The remaining settings are identical as in Section [4.3] As shown in T the last-layer feature does improve the image reconstruction quality and = has the best image reconstruction results under most settings. Besides, considering features and gradients of the same layer in the loss function would squeeze the final reconstruction quality. Feature information is more direct, and therefore deserves priority with higher weights.
Table 4: Image reconstruction with feature loss considered into gradient matching loss.
SSIMT LPIPS
PSNR
Match function
a == 0 (Baseline)
a B = 1
a
0 a
8 =
/ = 8 = 2
29.04
34.09
29.28
34.18
29.53
L2 loss
0.677
0.763
0.686
0.762
0.691
0.124
0.089
0.120
0.088
0.116
Mixup
25.71
28.75
25.89
28.81
25.92
Cosine
0.856
0.918
0.857
0.918
0.858
similarity
0.034
0.015
0.033
0.015
0.032
20.49
24.96
21.00
24.65
20.80
L2 loss
0.535
0.690
0.578
0.683
0.559
Label smoothing
0.176
0.114
0.162
0.117
0.167
23.02
25.79
23.27
25.85
23.34
Cosine
0.817
0.885
0.824
0.887
0.824
similarity
0.035
0.017
0.032
0.016
0.032
5.2 DIFFERENTIAL PRIVACY
The accuracy of label recovery is based accurate ground-truti gradients. Therefore, in this part, consider the impact that widely used differential privacy techniques may have on the label recovery quality. To evaluate, disturbance with variance from 10-4 to 10-1 and central to last- gradients. all experiments, pick 100 samples in the CIFAR-10 validation dataset from 10 classes with label smoothing augmentation under untrained ResNer18 8. Only the PSO algorithm utilized to find the optimum.
follow previous work (Zhu et al., 2019) add Gaussian and Laplace
Table 5: Label recovery accuracy and L, under Gaussian and Laplace disturbance. C. refer the difference between A, and Ar
Noise
Gaussian
Laplace
Variance
10-1
10-3
10-2
10-1
10-4
10-3
10-2
10-1
L.
1.02e-4
1.13e-3
1.14c-2
5,82c-1
1.61e-4
8.07c-4
7.95e-3
7.95c-1
Acc. (%)
100
100
100
45
100
100
100
36
For both Gaussian and Laplace disturbance, accuracy only drops harshly when variance reaches 10-1, 1 That means disturbance with such variance raises the variance loss of the original optimum scalar to level greater than other local minima, causing minimum exchange, while disturbance variance ranging from 10-4 4 to 1 only adds slight noise the optimal scalar.
CONCLUSION
This work proposes the first label recovery algorithm to analytically retrieve augmented labels as well as last-layer input features from gradients. We analyze the limitations of previous single-image label recovery methods, provide a necessary condition for label recovery, and design the first algorithm to recover high-quality images from unbiased fully-comnected networks under multi-class classification tasks. Extensive experiments have proved the recovery accuracy and quality, together with the benefits of following image reconstruction on both fully-connected networks and CNNs. We hope augmented labels, together with other real-world settings in GIA, could attract attention.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank Aijing Yu and Jiyang Guan in CRIPAC for their feedback on our early drafts. Besides, also would like present appreciation to the anonymous reviewers for their constructive suggestions, with which we could make our expression more fluent and informative. This work was supported by the National Natural Science Foundation of China (No. 62276256, No. U21B2045, No. U20A20223, No. 32341009), the Beijing Nova Program under Grant Z21 2211100002121000 and Young Elite Scientists Sponsorship Program by CAST.
REFERENCES
Yoshinori Aono, Takuya Hayashi, Lihua Wang, Shiho Moriai, al. Privacy- seracccpreeerving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 1365):1333-1355, 2017.
Mislav Balunovic, Dimitar Iliev Dimitrov, Robin Staab, and Martin Vechev. Bayesian framework for gradient leakage. In Proc. ICLR, 2022.
Trung Dang, Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Peter Chin, and Françoise Beaufays. Revealing and protecting labels in distributed training. In Proc. NeurIPS, pp. 1727-1738, 2021.
Peter
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition scale. In Proc. ICLR, 2021.
Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, Chang Liu, Chee Seng Chan, and Qiang Yang. Rethinking privacy preserving deep learning: How to evaluate and thwart privacy attacks. Federated Learning: Privacy and Incentive, pp. 32-50, 2020.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? In Proc. NeurlPS, pp. 16937-16944 2020.
Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chunming Rong. Towards general deep leakage in federated learning. arXiv preprint 2rXiv:2110.00044 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proc. ICAIS, pp. 315-323, 2011.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):1199:1444 144, 2020.
Ali Hatamizadeh, Hongxu Yin, Holger R Roth, Wenqi Li, Jan Kautz, Daguang Xu, and Pavlo Molchanov. Gradvit: Gradient inversion of vision transformers. In Proc. CVPR, pp. 10021-10030, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, pp. 770-778. 2016.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. In Proc. NeurIPS, pp. 7232-7241, 2021.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. In Proc. NeurIPS, pp. 29898-29908, 2021.
Kennedy and Eberhart. Particle swarm optimization. In Proc. ICNN, pp. 1942-1948, 1995.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6990 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) (6(1)):227822244 1998.
10

---

Published as a conference paper at ICLR 2024
Mu Li, David G Andersen, Jun Woo Park, Alexander Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene Shekita, and Bor- Yiing Su, Scaling distributed machine learning with the parameter server. In Proc. OSDI, pp. 583-598, 2014.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Liu. Auditing privacy defenses in federated learning via generative gradient leakage. In Proc. CVPR, pp. 10132-10142, 2022.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming. 45(1-3):503-555 528-288, 1989.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In Proc. NeurIPS, pp. 5972-5984, 2021.
Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions neural networks and learning systems, 2022.
Kailang Ma, Yu Sun, Jian Cui, Dawei Li, Zhenyu Guan, and Jianwei Liu. Instance wise batch label restoration via gradients in federated learning. Proc. ICLR, 2023.
Brendan McMMaan,, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication--ffiiienn learning of deep networks from decentralized data. In Proc. AISTATS, pp. 1273-1282, 2017.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In Proc. CVPR, pp. 1447-1454, 2006.
Xudong Pan, Mi Zhang, Yifan Yan, Jiaming Zhu, and Zhemin Yang. Exploring the security boundary of data reconstruction via neuron exclusivity analysis. In Proc. USENIX Security, pp. 3989-4006, 2022.
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-266 1992.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211 1--21112222200 2015.
Yujun Shi, Jian Liang, Wenqing Zhang, Vincent Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. In Proc. ICLR, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. CVPR, pp. 2818-2828 2016.
Yijue Wang, Jieren Deng. Dan Guo, Chenghong Wang. Xianrui Meng, Hang Liu, Caiwen Ding, and Sanguthevar Rajasekaran. Sapag: A self-adaptive privacy attack from gradients. arXiv preprint arXiv:2009.06221 06208. 2020.
Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating gradient leakage attacks in federated learning. arXiv preprint arXiv:2004.10333 2020.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 10(2):1-19, 2019.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proc. CVPR, pp. 8715-8724, 2020.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proc. CVPR, pp. 16337-16346.
2021.
11

---

Published as a conference paper at ICLR 2024
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. In Proc. ICLR, 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proc. CVPR, pp. 586-595, 2018b.
Rui Zhang. Song Guo, Junxiao Wang, Xin Xie, and Dacheng Tao. A survey on gradient inversion: Attacks, defenses and future directions. arXiv preprint 2nn:22206.07884, 2022.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020.
Junyi Zhu and Matthew Blaschko. R-gap: Recursive gradient attack on privacy. In Proc. ICLR, 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Proc. NeurlPS, pp. 14784, 2019.
12

---

Published as a conference paper at ICLR 2024
A PROOF FOR THEOREM 1
Theorem For any D and gradient information gr, all entries of the corresponding pseudo label y sum 1.
Proof. According to Eqn.
91 - (S(Ag))) - -g:/(A,B)) Therefore, we have
Σ b = Σ Φ (f(A,B))), -B:
(9)
Because & refers to the softmax function, we have
Σ à (J(A,))); =1
(10)
Therefore, only need to prove
gi/
0. From Eqn.
get g, (ps -1)xx" so
f=1
c
C
Σ
g:
Σ
pj
Vi
x
T =0
(11)
Consequently, Σ g:/ (Arg)) = 0. The theorem was proved.
B FAILURE ANALYSIS
B.I DISTURBANCE OF LOCAL MINIMA
The proposed label recovery algorithm focuses on finding the global minimum while avoiding local minima. However, in real searching processes, none of the algorithms could find the exact minimal point with Joss value equal O. Therefore, optimal points with loss below 10 as potential solutions. Under extreme conditions, some local minima also reach such low variance loss, causing recovery failure. This failure could be resolved by lowering the loss bar and increasing the population of particles in PSO, which would take more time.
increase variance loss by 1000 times and regard
B.2 OUT OF BOUND
Consider logit has its maximal entry zmaz and minimal entry 2min (mor and min subscript of entries). If sealar Xy -> +00 or Ar + -00, Soft. (x) would show one-hot distribution: Pmaz
Pmin -> both of whose variance function approach O. Therefore, if we keep searching without bounds, there exists ö so that for any Ar, if X, > 6, then Liabel < 10-9
Because ground-truth X, = (p, - yr) -2 for trained networks, p. and yr could get close so A," would be large enough to exceed bounds. This phenomenon also causes recovery failure. Extending searching range could solve such a problem, but in experiments, it consumes more time. So is also a time-accuracy trade-off.
B.3 ONE-HOT PROBABILITY
This systematic error when ground-truth labels are one-hot type, especially referring to the experiment in table T which cannot be solved by any algorithm currently, including previous iDLG (Zhao et al. 2020). Consider input x generates one-hot probability after the softmax layer, and the label is also in one-hot type. Assign the entry with value is i and j. meaning p: yj 1, then we have two situations:
i j= LCE = 00
i # LCE > gi =-gj
13

---

Published as a conference paper at ICLR 2024
For the first situation, the model predicts the input with 100% aceuracy and striet 0 loss, ground- truth A, is infinite, causing zero-division eπror. This is easy to handle because we already know that the model output is identical to the ground-trune label. In code implementations, we simply pass the optimization process, directly outputring model prediction as recovered one-hot labels.
For the second situation, it gets complicated. Originally, the ground-truth X, A-"=(p; y5)= However, the algorithm may take X, = as the ground-truth because did produce a one-hot label with zero all-but-top-one variance, even though the entry with value I not correct. In iDLG (Zhao et al. 2020), theoretically we hold that in the gradient information matrix, the entry with a negative sig.(e.g.. .......... Pj yj < 0) the ground-truth label. But in fact, the negative sign figured out by checking which entry has a gradient tensor whose sign is different from all other gradients, because gradient values themselves do not share the same sign under most circumstances, For this specific situation, only two entries of g are non-7ero, so we cannot decide which one is j. That is how the sign indicator fails. Such mathematical relation only comes out corner cases when the model is randomly initialized and predicts some inputs with extremely high probabilities. This explains why our proposed method shares identical non-perfect accuracy with iDLG, which also demonstrates the correctness and compatibility of such method.
-1.
Such one-hot prediction also influences the mixup setting, in which i and j are perceived as entries involved in mixing. When networks are properly trained, even with only a few epochs, such extreme one-hot predictions will not be generated from models anymore, implying satisfying label recovery accuracy.
C EXAMPLES OF OTHER TARGET LOSS FUNCTIONS
Ciabel = C-1|ST igs
Σ
y
C Σ
LLaahe(Ar))
| -
For the first loss function, we design intuitive experiments ResNet18 with CIFAR-10 datasets in label smoothing setting. We get 100% accuracy on the untrained network and 97.78% on the trained network. As stated in the main literature, the target loss function is not unique.
The second function is a loss to penalize negative entries in the pseudo label. Under some circum- stances, especially one-hot or mixup labels where multiple ground-truth entries are O, the change of each entry tends to be in a different direction during the optimization process. Therefore, it is possible that only the ground-truth label has all non-negative entries, implying that this loss function may only have one global minimum O. However, such property is not guaranteed. To make sure a successful recovery, under most circumstances need another necessary condition to guide target function designing.
D FCN NETWORK RECOVERY VISUALIZATION
In this part, we display all recovered 100 images from label smoothing and mixup augmentation, respectively. The reconstruction results from FCN are satisfying enough so we do not bother adding ground-truth images as comparisons. Experimental results are shown Table T
Table 6: Average image reconstruction performance overall randomly picked data.
Augmentation
SSIM
PSNR
LPIPS
Label smoothing
0.999
51.30
0.001
Mixup
1.000
66.80
0.045
14

---

Published as a conference paper at ICLR 2024
Figure 4: Recovered images with label smoothing augmentations from FCN-4. All 100 images are randomly picked with unknown smoothing probability.
Figure 5: Recovered images with mixup augmentations from FCN-4. All 100 mixed images are randomly picked with unknown mixup probability.
15

---

Published as a conference paper at ICLR 2024
E PSEUDOCODE FOR LABEL RECOVERY ALGORITHM
Here we present big picture for our label recovery algorithms with gradient-descent based optimizer and PSO optimizer, shown in Algorithm [ and Algorithm Full details can be checked in our repository.
Algorithm 1: Srradientdescent-ie label recovery algorithm
Input:
gm one entry of last-layer gradients with maximal absolute sum; Ir: learning rate; initial: initial points; coc: coefficient; bound: searching upper bound; iteration: maximum optimization iterations; func: input feature, output post-softmax probability; ear: variance loss function for label distribution.
Output: Scalar Xy
=iiiital
g=coe gm
Label=funcAA; x g)
(Label)r(Label)
Opt=optimizer(Lss, Ir = Ir)
for i in range (iteration 2): do
Label= func(A,. x g)
Loss= var(Label)
if Loss<le - 9: then
return
end
else if X, bound then break;
end Loss. Loss.backward() Opt.step()
end
-,=-1 initial // Restart from the negative side if failed.
Label=/unc(A, x g)
Loss=va((Label)
Opt=optimizer(Los,, Ir Ir)
for i in range (iteration iteration): do
Label= func(A, x g)
Loss= var(Label)
if Loss<le - 9: then
return X,
end
else if X, bound then return 11;
end
Loss. Loss. baskwarkk))d()
Opt.step()
end return -1
For L-BFGS, we set the learning rate=0.5, bound=100, iteration=200, coefficient=4, and initial=1. For PSO, we set initial=1, pop=200, max_ iter=30 by default. Enlarging the particle population as well as the iterations could enhance accuracy with the sacrifice of running time. Coefficient and initial could be fine-tuned for better performance.
16

---

Published as a conference paper at ICLR 2024
Algorithm 2: PSO-based label recovery algorithm.
Input: gm: one entry of last-layer gradients with maximal absolute sum, initial: initial points: coe: coefficient; bound: searching upper bound; ileralion: maximum optimization iterations; func: input feature, output post-softmax probability; var: variance loss
function for label distribution; interval: searching range for each step.
Output: Scalar Xy.
g=coc 8m
Label=func\A, x g)
Loss=var(Label)
lower_ lower_ bound=indiii;;; upper_ bound=lower_ wwwnd-loower_booddeeeooooooooeee bound=inderrrll
while upper_ bound<bound do
pso=PSO(Loss, lower_ bound=lower_ honlnddloww_buudd---- upper_ upund- bound, pop-pop, max iter=max___ iter) pso.run()
pssoobet
Loss=pso.besty // pso.bests and pso. besty save the minimum data
point.
Loss<le - 9: then
return X, end else
pso=PSO(Loss, lower upper bound-0.3 upper_ bound= lowwer_ bound, pop-pop, max_ iter=max_ iter)
pso.run()
end
x,=pso.bestr Loss=pso.beste
Loss<le 9: then return X, end
lower_ bound-upper, bound; upper_ bound-upper_ boundtinterval
end return -1
F MORE DATA FOR SECTION 5.2
Here we also note down the L, when faced with noisy gradients. As mentioned in Section refers to the sum of Lı loss from every entry of recovered labels. Results are shown in Fig. 6
Label Loss
2.77
Lapel Loss Vanience
4.37 10
10-4
1.32 20-3
2.76
10
10:4
9.5
10
10
54 20
8.65 2
10-1
1155 10
6.20
10-3
1.54 50
4B4 10:4
16xx1er1
30-4
10-1
10:4
10:4
6.15 x.15
10-1
6.60
10-1
10-4
10-4
15a4
10a4
10-h
10th
15-9
Geuox an disturaacce vadante at scale
Lablece dad.rre wewerrreeeeeee varlance at log scele
Figure 6: Loss comparison under different disturbance intensity
17

---

Published as a conference paper at ICLR 2024
G A FULL VERSION OF FIG.
17.5
Label smoothing
Label smoothing Ground Ground_truth:4222 truth::.2745 4444
Ground truth::118833
15.0
12.5
Laabll
10.0
Llabel
7.5
5.0
55
-3
-2
-1
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Ar
Ay
(a) Ltabel under untrained ResNet18. Clearly the (h) Llabet under trained R The first local first local minimum the global one.
minimum is the global
x10
x10
Label smoothing
2.00
Label smoothing Ground truth:4.2745
Ground truth:4.277
1.75
2.0
1.50
Claber
1.5
Caaell
1.25
1.00
0.75
0.50
0.25
0.0
0.00
2.235
2.236 2.237 Ar
2.238 2.239
4.20
2. 4.2.4......4..322224
A
(c) A detailed illustration the red-box area in (d) A detailed illustration of the global minimum area in Fig. ((b) Llabel reaches zero.
Fig. I(b) Liahet does not get zero.
Figure 7: Variance loss under untrained and pretrained ResNet18. The gradient-based optimizer may find local minima at approximately 2.2372 while the ground truth is 4.2745.
18


Reviews:
[{"review_id": "wXURXc2fkJ", "reviewer": "Reviewer_pnnF", "summary": "paper describes a simple method on label inference and also input reconstruction via analysis on one layer. this can be extended recursively to multiple layers.", "questions": "I am not very sure if this work would be of practical value if all the gradients and parameters needed to be known. for white box attack, parameters are known. however gradients should be known for this work to be valid. in practise, do we usually know gradients for each instance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "the method is simple.\n\nEq.2 is a key equation. It can be checked with some algebra to be correct. from this equation, other things follow.", "weaknesses": "Eq.(3), there is a term (g_i/x^*), since g_i is a vector and x* is a vector this makes no sense.\n\nthe same goes for the term g_i/g_r which is 'vector divide by vector'.\n\nEq.(5) the authors said \"top two entries\" but does not precisely define what it means. could it be the two items in y_i with highest values?\n\ntable 1, table 2, table 3, table 4 and other results, error bars will be needed. otherwise it makes no sense to say one number has higher values than another.  repeated experiments e.g. using differently trained networks.\n\nsection 4.3 line 5, there is a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5aLplMC8iF", "reviewer": "Reviewer_3Kd5", "summary": "This paper introduces a novel algorithm to reconstruct training data in a more realistic scenario where augmented soft labels are utilized during training. Specifically, this paper focuses on the recovery of ground-truth augmented label and last-layer features in gradient inversion attacks instead of those with hard label constraints. Through the analysis of the gradients of cross-entropy loss and introduced variance loss, the proposed algorithm can tackle the recovery of soft labels. Extensive evaluations on various datasets demonstrate the effectiveness of the proposed algorithm.", "questions": "1. Please provide more discussion of variance loss.\n2. Please provide more comparisons in label recovery evaluation.\n3. Please explain the metrics in soft label recovery.\n4. Please provide more discussion of the searching procedure in Section 3.3.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n2. The problem of soft label recovery can be challenging and interesting.\n3. The proposed variance loss seems simple and natural.\n4. The image reconstruction results of FCN regardless of the bias term seem promising.", "weaknesses": "1. The reason why the proposed variance loss can lead to the global minimum is not well-explained.\n2. The authors should include more comparison with other baselines besides iDLG in label recovery evaluations, such as Table 1.\n3. Lack details of the metrics to measure the correctness of soft label recovery.\n4. The designing of searching procedure needs more explantation, such as the starting point of $\\pm 1$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F4K4SQVyeM", "reviewer": "Reviewer_xaET", "summary": "This paper introduces a framework designed to address the label recovery problem while incorporating features such as label smoothing and mixup. Their experimental results demonstrate the adaptability of their gradient inversion attacks in practical, real-world scenarios.", "questions": "Why the label vector can be viewed as a function of the gradient in equation (3)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Seems that the performance is good compared with previous methods.", "weaknesses": "The introductory background and algorithm derivation may lack clarity, potentially causing individuals unfamiliar with this field to become confused.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVz3kKHpxc", "reviewer": "Reviewer_wnMY", "summary": "This paper proposes a method for gradient inversion attack with relaxation from hard labels to soft labels. The proposed method is based on a variance loss function and corresponding analysis is presented.", "questions": "Please refer to the Weaknesses. I would appreciate comments on the assumption of the label formats and additional results if applicable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper identifies an interesting gap for gradient inversion attack, i.e., the hard label constraints, and proposes a soft label recovery method that is closer to realistic scenarios.\n\n2. Although the proposed method is simple, it is intuitive and effective. Its simplicity also adds value to its applicability.\n\n3. Effectiveness is shown by the experimental results quantitatively and qualitatively.\n\n4. Code is provided in the supplementary.", "weaknesses": "My major question is about the assumption made regarding the label format. Does the proposed method assume the label format is known in advance or not?\n\n- First, does the method assume it is known whether a hard label or soft label is used? \n- Second, does the method assume it is known whether label smoothing or mixup is used as the format for the soft label? \n\nIf they are all known, I am curious about other versions of the results in Table 3 when the label format is unknown, corresponding to the above two settings. One setting is that we don't know whether the label is hard or soft. The other setting is that we don't know the specific format of the soft label. I think these two settings are closer to the real-world cases.\n\nMinors:\n- In Figure 2, it is better to also zoom in and visualize the global minimum similarly.\n- In Table 2 caption, experiences > experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)