Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction
Authors: Sai Qian Zhang
Ziyun Li
Chuan Guo
Saeed Mahloujifar
Deeksha Dangwal
G. Edward Suh
Barbara De Salvo
Chiao Liu
Venue: Accepted by TMLR ()

Abstract: Inverting visual representations within deep neural networks (DNNs) presents a challenging and important problem in the field of security and privacy for deep learning. The main goal is to invert the features of an unidentified target image generated by a pre-trained DNN, aiming to reconstruct the original image. Feature inversion holds particular significance in understanding the privacy leakage inherent in contemporary split DNN execution techniques, as well as in various applications based on the extracted DNN features.

In this paper, we explore the use of diffusion models, a promising technique for image synthesis, to enhance feature inversion quality. We also investigate the potential of incorporating alternative forms of prior knowledge, such as textual prompts and cross-frame temporal correlations, to further improve the quality of inverted features. Our findings reveal that diffusion models can effectively leverage hidden information from the DNN features, resulting in superior reconstruction performance compared to previous methods. 
This research offers valuable insights into how diffusion models can enhance privacy and security within applications that are reliant on DNN features.

Published in Transactions on Machine Learning Research (08/2025)
Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction
Sai Qian Zhang*
soi. soi.zhang @myn.ed edu
New York University
Ziyun Li
liziyun@meta.ccm com
Reality Labs Research, Meta
Chuan Guo
chuanguo@meta.com com
Pundamental AI Research (FAIR), Meta
Saeed Mahloujifar
sacedm@meta. com
Pundamental AI Research (FAIR), Meta
Deeksha Dangwal
sacedm@meta..c com
Reality Labs Research, Meta
Edward Suh Nvidia Nvidia/Cornell Cornell University
suksece.cornell.ed
Barbara De Salvo Reality Labs Research, Meta
barbardss@meta..cmm
Chiao Liu
chiaoliu@metta.c com
Reality Labs Research, Meta
Reviewed OpenReview: https: https://openreviee
openreview. net/ forum? id= J6MgbuBiGV
Abstract
Inverting visual representations within deep neural networks (DNNs) presents a challenging and important problem in the field of security and privacy for deep learning. The main goal invert the features of an unidentified target image generated by pro-trained DNN, aiming to reconstruct the original image. Feature inversion holds particular signifi- cance in understanding the privacy leakage inherent in contemporary split DNN execution techniques, well as in various applications based on the extracted DNN features.
In this paper, explore the use of diffusion models, promising technique for image synthe- sis, to enhance feature inversion quality. We also investigate the potential of incorporating alternative forms of prior knowledge, such as textual prompts and cross-frame temporal correlations, to further improve the quality of inverted features. Our findings reveal that diffusion models can elfectively leverage hidden information from the DNN features, result- ing in superior reconstruction performance compared to previous methods. This research offers valuable insights into how diffusion models can enhance privacy and security within applications that reliant DNN features.
1 Introduction
* Work undertaken during time Meta Work undertaken during time Meta

---

Published in Transactions on Machine Learning Research (08/2025)
Inverting visual features within DNNs presents a sig- nificant challenge in the realm of privacy for deep learning. The primary goal of feature inversion is to reverse the outputs (or intermediate results) of a pre-trained DNN and reconstruct the original im- age. This form of privacy attack, known as feature inversion atlack, can raise privacy concerns across various domains. Modern systems that perform face
Transmission
"Dorutss
(p)))
(cT))
Intermediate
User data
Edge
features
Cloud server
device
Figure 1: The DNN undergoes layer-wise partition- ing and is divided between the edge and the cloud, with the intermediate features being transferred be- tween them. In this example, we focus on the presence of a single edge device.
recognition azu (2018); ama
(202T); Schroff et al.
(2015 A Agarwaa et al.
20021; Bhat & Jain (2023);
Lezarna el. al. [ AR/ AR/VR applications Ma el al. (2021b: b Fu et al. (2023): ; Zhang et al. (2021); Chu
ct al. 2020), and image or text retricval Zhou ct al.
(2017) Lu et al. (2020 2017 : Song & Raghunathan (2020 ; Borgeaud et al. (2022) often store and process auxiliary data in the form of extracted features from the original input. For example, in face recognition system, the human face is first encoded with DNN
encoder (e.g., FaceNet [Schroff et al. (2015), CLIP Bhat & Jain (2023); Shen et al. (2023)) and the resultant feature vector is then scarched over the database for identity matching via vector comparisons. Feature
inversion attacks can be used to reconstruct the face of private users Mai et al. (0018).
Moreover, feature inversion attack also leads to a serious privacy leakage in the Split DNN computing
paradigm Hauswald et al. (2014); Park et al. (2022): ; Kang al. (2017): Toerapittayanon et al. (2016: 2017):
:
Zhang et al. (2020a : Akintoye
al.
(2023): Dong et al. (2022a); Karjee et al. [2022); Luo et al. (2023)
Mubark et al. [202
: Lee et al. [2023]
Feltin ct al. (2023) : Zeng et al. (2020); Ding et al. 2023 Kang al.
(2022] Matsubara et al. 2201 2022 pys 2020 sp (2020). Within this paradigm, a. layer-wise partitioning
of the pretrained DNN into two or more blocks, aligning with the computational capabilities of the edge
devicos, as shown in Figure 1 During the execution, the user data is first processed using one or more local DNNs that contain the initial layers. The intermediate results are then transmitted to the central server for the execution of subsequent. DNN layers. Split DNN computing has been widely adopted to accommodate the execution of increasingly large DNN on resource-constraaned devices like mobile phones, and is believed to enhance user privacy by keeping user data the local device-only the intermediate features are sent to the less secure cloud enviromment. However, this privacy enhancement turns out to be frail, recent studies have shown that the intermediate features can be inverted via feature inversion attacks to reconstruct user inputs from the intermediate outputs of parts of the DNN Mahendran & Vedaldi (2015); Dosovitskiy & Brox 2016 : He et al. (009 Dong et al. (2021) Maeng et al. 2022]; Song & Raghunathan (020) Morris et al. 2022
The broad applicability of feature inversion renders it a fundamental problem in ML security and privacy, On the other hand, feature inversion is not easy task, particularly when dealing with features extracted from later layers of a network. Intuitively, the learned feature contains more high-level semantic information about objects in an image but less information about the raw input as depth increases. As result, nearly all of the existing feature inversion methods fail when atlempting to invert features from later layers of a deep network. This explains why much of the existing research concentrates on feature inversion for shallow DNNs with lower input resolutions Mahendran & Vedaldi (2015); Dosovitskiy & Brox (2016); He et al.)
2 Maeng al. (2022).
The recent advancement of generative Al (GenAl) models opens up new possibilities to improve the quality of feature inversion attacks through their comprchensive understanding of image data distributions across real-world scenes. Among the multitude of existing GenAl techniques, Diffusion Models (DMs) Ho et al. (2020) have emerged as a remarkable breakthrough in generative modeling. Through extensive training with vast datasets comprising millions of world images, DMs obtain a high-quality, photorealistic image generation capability.
In this work, we demonstrate that recent advancements in DMs can be utilized to greatly enhance feature inversion. Instead of inverting DNN features directly to image pixels, aim to recover the input vector in the latent space of latent diffusion model (LDM) that, when converted to an image through reverse diffusion and forwarded through the DNN, matches the target DNN features. In addition, another noteworthy feature
2

---

Published in Transactions on Machine Learning Research (08/2025)
of DMs is their capacity to take textual descriptions input and produce synthetic outputs conditioned these textual prompts. We also demonstrate that this capability enables the attackers to specify the prior knowledge of a target image with natural language to utilize their existing knowledge about the victims, if available, by providing a textual description to DMs. Doing so enables the inversion of features that are much deeper into the network. Finally, in practice, as edge devices oflen process a continuous stra of input frames, we propose another variant that uses the temporal correlation in the features between consecutive input frames to enhance the reconstruction quality, Our main contributions are follows:
Peature inversion using diffusion model prior. We demonstrate that the exceptional image generalion capabilities of DMs can be effectively employed to improve DNN feature inversion. We explore two threat models that closely describe the practical scenarios. To the best of our knowledge, this marks the first research endeavor showing the use of DMs for enhancing DNN feature inversion.
Incorporating teztual prior for feature inversion. We demonstrate that incorporating textual prior information about user inputs can significantly enhance the quality of feature inversion. To inte- grale this textual prior knowledge and achieve improved feature inversion quality, we introduce new training loss terms as part of the inversion process.
Feature inversion for videos. When processing a sequence of temporally correlated inputs, we show that feature inversion can be further enhanced by considering the temporal correlation among con- secutive imput frames.
The evaluation results show that our approach exhibits significant superiority over the stato-of-the- art approaches in feature inversion quality across a variety evaluation metrics. For some backbone DNN models that are trained with self-supervised learning, we can achieve end-to-end inversion by reconstructing the input from the DNN outputs.
2 Background and Related Works
2.1 Diffusion Models
Diffusion models (DMs) Ho et al. (2020) have recently gained significant attentions for its remarkable ability to generate diverse photorcalistic images. It is a parameterized Markov chain trained through Variational inference to generate samples that match the data distribution over a finite duration. Specifically, during the forward process of DMs, given an input image 2o ~ q(x), a series of Gaussian noise generated and added to the To, resulting in a sequence noisy samples {xe). 0 < T.
q( N(re / 3, 8,1)
(1)
where 3r e (0, 1) is the variance schedule that controls the strength of the Gaussian noise in cach step.
During the reverse process, given a randomly sampled Gaussian noise N(xr; 0, I), the synthetic images are generated progressively with the following procedure:
po pa 1 N (1
(2)
where po(2e, t) and 3, are defined as follows:
1 Vor
1 Ci;
e0.t), B, 1 - Ōt-1
po( 1)
(#t
(3)
1 Cz
co.s denotes the predicted noise that is generated with trained U-Net, as Br, and In Equation 3 0g a2.
In this paper, will employ the terms "attacker" and "adversary" interchangeably.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Since their inception, there has been variety of subsequent research that builds upon DMs. Several alternative approaches to accelerate the reverse pro-
Attacker
Gradient
cess have been proposed Song et al. (2020a); Lyu et al. (2022] ; Lu el al. (2022). Latent diffusion
Lre
update
F(())
models (LDMs) were introduced in Rombach et al.
(2022) to perform the reverse process within the la- tent space of an autoencoder. The outcome of this reverse process is then fed to decoder, which gen- erates the synthetic images. LDMs offer a simple yet. efficient way of enhancing both the training and sampling efficiency of LDMs without compromising their quality. LDMs can also be integrated with text encoders for text-lo-image generation, as explored in Sahaari et al. (2022); Rombach et al. (2022). Typ- ically, these models include pre-trained text en- coder that takes user textual descriptions as imput, effectively guiding the reverse process to generate the desired synthetic output.
Transmission
IDonulis
2
User data
F,())
F F2())
Edge device
Cloud server
Figure 2: Feature inversion attack under white-box settings in a Split DNN execution scenario: The first- stage model (-) is exposed to the altacker. The at- tacker's goal is to generate an input image such that,
when passed through ()). it produces intermediato features closely matching those of a target input.
2.2 Feature Inversion Attacks against DNNs
Feature inversion has been studied by various literature. Dosovitakiy & Brox (2016) showed that DNN features can be inverted by training a network to reconstruct the corresponding input images given their features. He et al. (2019) first demonstrated that feature inversion can lead to a leakage of private input for split DNN computation, and further showed that introducing the total variation loss Rudin et al.) (992) can greatly improve feature inversion quality. Dong al. (2021) revealed that the exposure of batch normaliza- tion parameters can lead to significant enhancement in feature inversion quality. Unsplit Erdogan et al.) (2022) operated within black-box setting where the attackers lack knowledge of the model parameters, and developed techniques aimed at reconstructing both user inputs and model parameters. The study presented in Otroshi Shahreza & Marcel (2024) aims to create a facial image capable of deceiving facial recognition systems, without requiring the generated image to exactly match the user face.
Our rescarch demonstrates the use of LDMs as a prior can significantly enhance feature inversion quality, aspect not explored in prior studies. We also explore incorporating diverse prior knowledge sources, such text and cross- cross-frame correlations, to further improve reconstruction quality. These advanced techniques enable statc-of-tho-art feature inversion performance, surpassing prior methods. Then, we will discuss various application scenarios of feature inversion attacks.
2.3 Split DNN Computing
Split DNN computing has garnered significant attentions from both academia and industry, evidenced by numerous studies Hauswald et al. (2014): Teerapittayanon et al. (2016); Kang et al. (2017): Teerapittayanon
et al. [2017]; Karjee et al. (2022] ; Luo et al. (2023); Mubark et al. (2024); Lee et al. (2023); Feltin et al.
Lin et al. : Dong et al. (202bb); Zhang et al. (2024). Additionally, solutions based on split learning and inference have been actively implemented and embraced across both commercial and open- source applications pys (0200) spl (2020). Among the multiple partition strategies Kang et al. (2017):
(2023
g Zeng et al. (2020)
Ding et al. (2023) Kang et al. (2022): ; Matsubara et al. (2019 2022 ;
(2024
; Zhang et al. [2022]
Zhang et al. (2020a). layerwise partition has been widely employed Hauswald et al. (2014h; Teerapitaaynoon ct al. (2016): Kang ct al. (2017) Tocrapittayaoon et al. (2017). This approach entails splitting the DNN into two or more parts and executing on multiple devices. The study by Hauswald el al. Hauswald el al. (2014). is among the initial research efforts that moved the later stages of image classification computation to cloud servers. Neurosurgeon Kang et al. (2017) and DDNN Tecrapittayanon for automatically distributing DNN models between a mobile device and a cloud server, considering factors
al.] (2017) introduced a tochnique
like network latency and energy usage. Meanwhile, BranchyNet Teerapittayanon et al. (2016) made use of
4

---

Published in Transactions on Machine Learning Research (08/2025)
early exit points within the DNN layers to enable adaptive DNN inference based on the input complexity, further reducing the processing latency.
2.4 Applications based on Extracted Features
Modern systems often store and process auxiliary data in the form of features extracted from the DNN
encoder. For instance, in face recognition systems, the image of human face is initially encoded with a DNN,
and the resulting feature vector then searched over the database through vector comparisons azu (0188)
ama 2021 Schroff et al. (2015); Aggarwal et al. [2021]; Lezama et al. (2017).
In addition, some AR/VR lasks, such as Codec Avalar Ma el al. (2021); Richard el al. (2021 Fu et al. (2023), also rely on extracted features to operate. Codec Avatar is high fidelity animatable human face model designed for the purpose of remotcly sharing spaces with each other. To generate the Codoc Avatar, an encoding process is first performed on the transmitter headset device: cameras linked to the VR headset capture partial facial images, which are then encoded by a DNN model into feature vectors and transmitted to the receiver headset device. On the receiver side, upon the reception of the feature vectors, the decoder reconstructs the avatar's geometry and appearance, enabling the real-time rendering of the transmitter's photorealistic face.
Finally, in the field of image and text retrieval, recent studies have advocated for the adoption of vec- tor database services to facilitate scalable embedding matching and retrieval, yielding enhanced perfor- mance Zhou et al. (2017); Lu et al.] (2020; 2017); Song & Raghunathan (0020); Borgeaud et al. (002)). To operate, the data owner transmits only embeddings of the raw data from the DNN encoder, the third- party service, without revealing the actual text content. Subsequently, the database server returns search result, indicating the index of the matching document the client side.
3
Threat Models
We begin by first describing the underlying threat model for our feature inversion attack. We consider two settings, white-box and black-box, for the two variants of our attack.
3.1 Threat Model for White-Box Settings
We focus on a scenario where the target model F(.) divided into two parts: F(.) = F2oFi. Here, we use Fgt to represent the user input and 2mid to denote the intermediate feature. We assume that Fi, referred to the user model, is executed within a secure environment such as an edge device, where internal computations are protected from external access. In contrast, F2 runs in an untrusted environment (e.g., a public cloud), where the input 2mid = Fi (Z may be exposed to potential adversarics. This split computing setup is realistic and widely adopted in practice, particularly for latency-sensitive and privacy-aware applications. In such systems, is common for early layers (i.e., shallow layers) to be executed on-device to reduce bandwidth and protect raw input data, while the remaining layers are offloaded the cloud. Therefore, accessing shallow-layer outputs like 2mid is both feasible and practical under this model, especially when the communication between edge and cloud is not end-to-end encrypted or when decryption is required before cloud-side processing.
Although encryption can be applied during transmission, we assume that 2mid must be decrypted before execution in F2, which reintroduces exposure risk. In this setting, adversaries may attempt to reconstruct the original input. In the whito-box scenario, we assume the attacker has full access the structure and parameters of Fi, but not to the input Zgt or intermediate activations within F, Our goal is to reconstruct input that produces intermediate outcomes similar to the observed 2mids which can be formulated follows:
Tre arg min (F) Emsd),
(4)
where Lr)(..),, referred to as the reconstruction loss, represents the loss function employed for measuring similarity, with the la distance being used in this study. We construct x to minimize the loss function
5

---

Published in Transactions on Machine Learning Research (08/2025)
Intermadiato
1/q
results
niver
Latent Diffusion Model
DM
Cesisionn WWC
Latent
U-Net
Fi(.)
miid
Z
Inversion DNN
Normalize
U
Decoder
Intermediato results
U-Net
KV
Synthetic image
Yq
Qq
Text encoder
(a)
(b)
gradient
Figure 3: Black-bos feature inversion at- Figure 4: Feature inversion using a latent diffusion model, tack procodure: The diffusion model takes where the model takes an input vector e and generates syn- the intermediate features input and gen- thetic images that closely resemble the victim image. erates reconstructed image.
illustrated in Equation | (Figure 2). Previous studies He et al. (2019): Maeng al. (2022) have demonstrated the fcasibility of achieving high-fidelity input reconstruction when F1 is shallow and the input dimension is limited. However, as F1 gets deeper, increasing portion of information within zigt is filtered out by
DNN operations, such as pooling layers, retaining only the essential task-related information. This greatly complicates the task of feature inversion.
3.2 Threat Model for Black-Boo Settings
In the black-box variant of the feature inversion attack, the threat model resembles that of the white- box approach, with the key distinction being the relaxation of the assumption regarding the adversary's knowledge of F.). Here, the adversary can only gather information about F1 indirectly through querying it. As a result, the adversary gains access the input queries and their corresponding outputs from Fi(., as shown in Figure aa). Demote X, {1n) set of input queries sent by the adversary and Y, (vo) the corresponding outputs from (.). Next, inversion DNN F is trained to take Y, input and generate X that closely resembles Xg (Figure 3 (b)); namely:
min Σ (F
(5)
For black-box attack, the attacker only needs access to the function (i(.) by querying it with their own inputs and collecting the outputs for training. The internal architecture of (-)) does not need to be known.
3.3 Generalizability of the Threat Model
Our threat model described in Section 3.1 and Section 3.2 can also be applied to systems involving more than two participants. However, given that many real systems are typically divided into two parties Teer- apittayanon et al. [2016) 2017): Kang et al. (201)). we focus on a two-participant system for the remainder of this paper, without sacrificing generality. Additionally, by making Fj and F₂ =, our approach can also be applied to the scenario of end-to-end feature inversion, where the objective is to invert the DNN output to reconstruct the input. This presents significant privacy concern for applications that operate based extracted features, such as face recognition, Codec Avatar, etc.
4 White-box Feature Inversion
In this section, we describe the white-box attack methodologies in details. Particularly, we first present our methodology for feature inversion with LDMs. Subsequently, we investigate the impact of textual priors on feature inversion Section 4.1 and discuss multi-frame reconstruction Section 4.2
We leverage the prior knowledge embedded within the LDM to reconstruct the user input Zyt. Let D(v, e) represent the generating function of the LDM. Here, v denotes the input latent variable, which expected

---

Published in Transactions on Machine Learning Research (08/2025)
Generated image
Reconstructed
Reconstructed image
Fi(.)
Reconstruction Loss
Un
Normalize
Victim image
Cre
LDM
LDM
LDM +
TV
-Loot
"blue
"blue
"Blue sky"
20 sky"
Un
sky"
Un
(a)
(b)
(c)
(d)
Figure 6: White-box feature inversion with textual prior on the diffusion model.
Figure 5: Impact of textual prior.
Algorithm Feature Inversion with LDMs
(-). is the user DNN model. o the input latent vector of LDMs. I is total number of iterations. Emid is the intermediate result.
is the learning rate. for i do
/ of -'man(e') std(m))
Cio (D(()) zmidll (D(())
o1 v² dlas
i =i+1
1/ c'-mean(o') std(o!)
return D(v.).
follow a normal distribution Ho al. (2020), and e E(() represents the text embedding. The function E() indicates a pre-trained text encoder, and t corresponds to the text input provided by the user. In this section, we ignore the text prompt by setting the text embedding to a vector of zeroes, and will examine the influence of the text prior in Section 4.1 We then search for the input latent variable v that allows the LDM to produce synthetic output, denoted as D(v,). This output, when passed to F1(.), will result in a similar intermediate output as 2mid (Figure g
x* = arg min (D.(()((())))) Emia) + x,TVV(((..)
(6)
As the LDM necessitates input data to approximate normal distribution for photorealistic image gen- eration, implement soft restriction the variable v by normalizing it prior to forwarding it to the LDM. Specifically, we define e, g-mean(") the normalized version of U, which serves as the input for the LDM. We observe that applying normalizing operation can greatly enhance the feature inversion perfor- mance. Emid Fi (Zgge is the intermediate result generated from the user input. TV(.) represents the Total Variation Rudin al. (1922) which is used to reduce the abrupt pixel variations across the reconstructed image. TV(x)) (z) is defined follows:
std(r)
TV(x) MN Σ - |r,.i11 - -iil
where M and N represent the spatial size of the image, and X, denote the weight of the TV loss. The feature inversion process summarized in Algorithm [
4.1 White-bor Inversion with Textual Prior
Another important characteristic of LDMs is their ability to take text prompt as input and produce synthetic outputs guided by textual descriptions. We leverage this capability in our feature inversion attacks by allowing the attacker to express their prior knowledge about the user input in the form of natural language.
7

---

Published in Transactions on Machine Learning Research (08/2025)
U-Net
95 1494
164, 32, 6=2)
Corvelution
BaarhhNor (32)
(22 16, 6=2)
Corvelaate corvolution
BarchNomm 22
(16.5 8, (=2)
Deconvoiulisn
88 4 d-2)
Moomalizalinn
[4, 64, 64]
DM
Vy
Resize
ReL.U
ReLU
[64, 164,64,80 64, 64]
[32, 02,,2,,22
[16,
[8, [8,32,33 .2]
14 64, 64]
Figure 7: A sample architecture of U-Net. The dimension will adjust based on the dimension of yq. In this example, the intermediate result Bn is of dimension 64 x 56 x 56, with a batch size of 1. The LDM input has a dimension of x 64 x 64. in the (de)convolutioaal blocks denotes the stride.
Different from generic image priors such as total variation, this form of text prior can be specific to each larget image and further enhances the quality of feature inversion using diffusion models.
To incorporate the text prior into the feature inversion process, consider the private user image depicted in Figure 5 (a). Assuming the adversary possesses prior knowledge of this private image, they will convey this knowledge to the LDM through textual description. The LDM will take these textual description together with another randomly generated, normally-distributed input to produce an image visually akin to the user image (Figure (b)). Subsequently, we proceed to further train the LDM input Un to enhance the LDM's ability to refine the output, making it more closely resemble the user image Tot (Figure (c)). The resulting reconstructed image has much better quality than that reconstructed without a textual description, which is shown in Figure (d).
Considering that the relates to user input, we utilize this insight by further pushing on, the input of LDM, to approach random variable generated from a normal distribution. To achieve this, we assess the Gaussianity of en using the negentropy metric outlined in Hyvärinen & Oja (2000), resulting in an additional loss term denoted as Lizt. defined as follows:
latent coupled with the text prior can produce an output that
List
a (ogccos?(avnn.))
(8)
where 1 & 2 is hyperparameter, the expectation B(-) is taken over the elements of v.. The total loss Liot can be described as:
Liot Lye (Fi e)), Zmid) + X,TV + (eat
(9)
where e = E(t) represents the embedding of the textual description, which serves as an additional input to the LDM, Aert is the weight factor to balance the loss terms (Figure 6 The remaining inversion algorithm is similar to Algorithu [ The detailed algorithm given in the appendix.
4.2 White-box Multi-frame Reconstruction
In this section, we explore the problem on multi-frame feature inversion. This scenario closely resembles real- world situations where edge devices handle continuous stream of imput frames, such burst mode photos or video clips. In this context, the local DNN processes consecutive input frames that exhibit temporal correlation, the intermediate features are then transmitted to cloud servers for subsequent processing. The goal is to reconstruct the entire input image sequence using the intermediate results.
In particular, consider kg where & k & K, to represent a sequence of K user inputs. Additionally, let 2mid, and 0s represent the corresponding local DNN output and input latent variable for Tyt,k- Wo introduce additional loss component Le(-) aimed minimizing the disparity among the latent vectors us across these frames. To achieve this, the multi-fie frame reconstruction process can be realized by solving the following optimization problem:
min MALEEEK
Cre.k + NgVV + T))
(10)
8

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 2 Multi-frame Feature Inversion
(.). is the user DNN model.
K is the total number of input frames.
"k" is the latent vector of input at iteration m.
Emid.k is the intermediate results for input k.
total number iterations.
is the learning rale.
As d are the weights for TV loss and temporal loss, respectively. for 1 & & M do
8 Ex
Liot
for & K do
m 0k,n
"4" mean[
std
Lio Fi (D(r 2mad,a/1 + (DD((D()))) + Aello? 11
Compute the gradients based on Leoe.
for 1 & k & K do
m+ 2k
dCunt dt
m m
for k & K do M Km, (t
M -m
return D.
where 5 = (2 vx) represents the average of the input latent vectors across the K frames. The loss function Le utilized to minimize the disparity between the latent vectors for each frame. In this study, have observed that simply minimizing their le distance yields excellent reconstruction quality. The parameter de serves as a weight to balance the importance of these two loss functions. L,e,k and TV2 denote the reconstruction loss and total variation loss for reconstructing gtt. The detailed algorithm is given in Algorithm 2
5 Black-box Feature Inversion
For feature inversion attacks with black-box settings, the attacker obtains access to the input queries and their corresponding outputs from Fi).), as depicted in Figure 3 (a). Let X, = {zz) denote a set of input queries sent by the adversary and Y, {yg) represent the corresponding outputs from (-). The adversary then proceeds to train an inversion DNN Erie(.) designed take By as input and generate that closely resembles x. (Figure 3 (b)).
(ia consists of two major components: pre-trained LDM and an U-Net, which are denoted as D(.) and Fu(.), respectively. During the execution, F.(.) takes the intermediate data Bg and generates the input latent variable for the LDM, which then produces the result = D(Fa(Ha)). The training of the inversion DNN model involves minimizing the following loss function:
g arg min Pa
Σ
Cre(D(F.().))) 24) X,TV
(11)
where 02 represents the parameters of Fine()) TV loss is introduced over the reconstructed input D ((Fu(y))).
The architect ure of U-Net is illustrated in Figure 7 During the forward pass, the input No is first resized spa- tially. Following this, the intermediate output traverses through several blocks consisting of convooonvll layers, batch normalization layers, and ReLU layers. The resulting output is normalized before being sent the diffusion model for image generation. The specific dimensions of the inversion DNN will vary depending the shape of Uq-
9

---

Published in Transactions on Machine Learning Research (08/2025)
Reconstructed inputs
Inversion DNN
Input variable
FI(.)
Input
FI(.)
Pointwiie
U-Net
DM
eves
U-Nel
Loss
variable
Decoder
Loss
fa see
U-Nel
U-Net
Cot
tot
gradient
gr
gradient
Architecture of the inversion DNN for black-box multi-frame reconstruc- tion. In this example, the inversion DNN can generate four frames.
Figure 9:
(a)
(b)
Figure 10: Baseline feature inversion algorithms.
5.1 Black-box Inversion with Textual Prior
Similar to the incorporation of textual priors to enhance recon- struction quality in the white-box setting, integrating textual priors into the inversion DNN can also improve the quality of the black-box feature inversion. The training procedure out- lined in Figure S The U-Net output is directed into the DM for image generation, while additional loss function Cime()) is simultancously applied to enhance its gaussianity. The overall loss function is shown as follows:
Inversion DNN
Yg
N
U-Net
L
Intermediate results
"Whito
TV
Lrxt
Liot
0 arg min
Σ (.))( A,TV + (o (Fig04))
Figure Feature inversion with textual prior under black-box settings.
(12)
where Lix(() is the loss term that enforces the LDM inputs, (1g)), follows a gaussian distribution. Cq E(ty) is the emboddings of the textual description ty that describes g...
5.2 Black-box Multi-frame Reconstruction
In this section, we explore the problem on multi-frame feature inversion under black-box settings. par- ticular, assume group of consecutive frames with total of K images, the inversion DNN will take the intermediate results Y {) from each frame & within this group g, and produce the K outputs that will serve as the inputs of the LDM.
In order to exploit the temporal correlation within the intermediate results YJ, we introduce a pointwise convolutional layer into the inversion DNN, as depicted in Figure g This pointwise convolutional layer will incorporate a weight filter with spatial size of x 1, enabling the learning temporal correlations among the intermediate results &q,6- The output of the pointwise convolution will then be separated into K pieces each of which corresponds to one input frame, the outputs will be forwarded to the U-Net, whose architecture is shown in Figure 7 Each of the four U-Net will share its weights. The outputs from U-Net will further be delivered to the LDM, which will then reconstruct qg for each & E K. The loss function for training is shown as follows:
min (LCF)((())()))) X X.TVV
(13)
keK
where X² (e) are the groups of ground-truth consecutive frames, and TV, is the TV loss the k-th reconstructed frame within group g.
6 Evaluation Results for White-bon box Inversion
In this section, we present detailed evaluation of the white-box feature inversion technique described Section 4 We first evaluate the quality of the inverted features different. applications in Section 6.2
10

---

Published in Transactions on Machine Learning Research (08/2025)
Section and Section 6.5
Next, explore the inffuence of the textual context in Section 6.6 and the multi-
frame reconstruction in Section 6.7 Lastly, we conduct an ablation studies in Section 6.8 and Section 6.9.
6.1 Experiment settings
Datasets and models: We assess our feature inversion approach outlined in Algorithm ImaageNet Deng et al. (2009) and COCO Lin et al. (2014) datasets. We employ various DNN architectures pre-trained on ImageNet as the target models for feature inversion, including ResNet-18, ResNet-50, and Vision Transformer
(ViT). All of the pretrained models are downloaded from the official Pytorch website. Due to institutional restriction, we are unable to use the public latent diffusion model for publishing our research outcomes. Instead, we employed an LDM with an architecture highly similar to Stable Diffusion 2.1 Rombach et al.) (2022) in terms of architecture, model size and pretraining techniques. Our internal model was pretrained the dataset collected by a third party (Shutterstock) that is not public. Regarding the dataset, it consists of 385 million images: 321 million without people and 64 million with people. None of the victim images used in the feature inversion attack are included in the training dataset of the diffusion model. In addition, we have also previously conducted extensive experiments with the public Stable Diffusion 2.1, which yielded similar (and even better) results in terms of 18, PSNR, and SSIM scores than the results in this paper.
Hyperparameters: We set all X to for the reconstruction loss defined in Equation 6 Equation g and Equation 13 The reconstruction process continues for total of T 1500 iterations. We adopt the Adam optimizer with an initial learning rate of 0.1, (0.00,,).999)) To expedite the reverse procedure, the sampling steps of LDM to be 20 with a linear schedule Ho et al. (2020). We find that using 20 sampling steps can already yield high-quality feature inversion results. We investigate the impact of the sampling steps in Section 6.8 More evaluation results can be found in the appendix.
configure
Baselines: We consider two algorithms for comparison. The first approach, referred as Direct Optimiza- tion (DO), reconstructs the imput by directly optimizing Equation 4 over the image pixel space (Figure 10 (a)). This method has been utilized for input reconstruction in prior works He et al. (2019); Maeng et al. y Erdogan et al. (2022) and serves as the baseline to assess the impact of LDMs on the feature inver- sion. The second approach, known as the Decoder-based (DB) approach (Figure I0 (b)), employs only the LDM decoder for input reconstruction. Evaluating this approach helps us understand the influence of the iterative reverse process in LDMs on feature inversion attacks. It is worth noting that this Decoder-based approach has not been investigated in prior works on feature inversion either, but similar tech- niques using GAN decoders have been used in the context of gradient inversion Jeon et al. (2021); Li et al.) (2022) and model inversion Zhang et al. (2220b). Finally, we denote our method as the DM-based (DMB) approach.
(2022
6.2 Feature Inversion on Split Models for Image Classification
We first assess the reconstruction quality of our feature inversion attacks without text prior. We randomly select 100 images from the ImageNet and COCO test datasets and feed them to a pre-trained R8sNet-18. ResNet-50 and ViT-base model, respectively. For each of these target DNN models, we evenly divide them into blocks of layers and extract intermediate results at the end of each block. Subsequently, we employ the techniques outlined in Section to reconstruct the user input.
Qualitative result: Figure 11 depicts the fcature inversion results for ResNet-50 over ImageNet, respec- tively. The original image is displayed in the left column for reference. DM-based method consistently demonstrates superior reconstruction qualities across all datasets and DNN architectures. Notably, our ap- proach achieves high-quality input reconstructions, even when utilizing features from very deep layers (c.g.. layer 36 in ResNet-50), whereas other baseline methods struggle to achieve comparable performance.
Quantitative result: To quantify the quality of the reconstructed images, we utilize three metrics. The first metric Inception Score (IS) Salimans et al. (2016), which is commonly used to evaluate the quality of image generation in prior works Song et al. (2020b); Xu et al. (2019); He et al. (2019): Dong et al. 2021 For instance, generative AT models like StackGAN
al. (017)) and GAN-INT-CLLS Reed et al. 2016
11

---

Published in Transactions on Machine Learning Research (08/2025)
Original
Direct Optimization
Recoder-based
DM-based
layer 12
layer 24
layer 36
layer 12
layer 24
layer 36
layer 12
layer 24 layer 36
Figure 11: Feature inversion over ResNet-50 on ImageNet, features are extracted from the end of layer 12, layer 24 and layer 36 respectively.
typically generate images with 1S scores around 3 to 5, while diffusion models can achieve IS scores high as 10 Ho et al. (2020).
The second metric is Peak Signal-to-Noise Ratio (PSNR), which calculates the ratio between the maximum possible value of a signal and the power of distorting noise affecting the quality of its representation. The mathematical expression for PSNR as follows:
255
PSNR(1 (1') = 10log( M
(14)
MN 2
Ej=
Fori [.
where fori and Ie denote the original and reconstructed images, and they both have a size of M x N. 255 is the maximum pixel value. PSNR is a commonly used metric for assessing image quality, particularly when comparing a compressed or reconstructed image its original version. frequently employed in the image processing tasks to quantify the degree of distortion introduced. PSNR values between 30-50 dB are typically considered indicative of excellent image quality.
Lastly, the Structural Similarity Index Mcasure (SSIM) Wang et al. 2004) is metric used to assess image quality by comparing the similarity between two images. In our scenario, we evaluate SSIM between the reconstructed images and the original image. Unlike the PSNR, which focuses solely on pixel differences, SSIM evaluates changes in structural information, luminance, and contrast, making it more closely aligned with human visual perception. SSIM values range from 0 1, with higher values indicating better image quality. Table gives the mean values of average IS, PSNR and SSTM across 100 reconstructed images over ImageNet for the various model architectures and feature inversion methods. We observe three trends:
For the same model-layer pair (i.e. each column), DM-based method (DMB) achieves the high- est average IS/PSNR/SSIM compared to the Direct Optimization (DO) and Decoder-basee (DB) approaches. This shows that our diffusion-based feature inversion attack is highly effective.
For the same model, inverting features extracted from later layers results in lower 1S/ IS/PSNR/SSIM for all three methods. However, with our diffusion-based method, the reduction in reconstruction quality much less pronounced, showing that the attack is more capable of inverting later layer features.
In comparison, inverting features for ViT models more challenging, although it remains feasible to invert features using output from middle layers of ViT (e.g., layer 5).
12

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Evaluation results: 'OO 'DB', and 'DMB' refer to direct optimization, bacoder-based, and DM-
based approaches. 'L' is the feature extraction layer. PSNR is shown in db. For IS, PSNR and SSIM, higher values indicate better results.
1es-ett-1
ResNet-50
ViT-base
Metric
Method
L4
L8
L12
L12
L24
L36
L3
L4
L5
DO
5.63
3.92
1.40
5.55
3.88
1.28
5.46
3.95
1.63
IS
DB
6.84
5.97
4.20
6.90
5.86
4.38
6.76
5.76
3.93
DMB
7.23
6.86
6.48
7.36
6.90
6.55
7.14
6.77
6.58
DO
29.3
14.6
9.51
28.9
15.3
8.04
28.5
17.6
9.42
PSNR
35.2
32.6
35.4
18.3
36.8
19.9
DMB
41.0
36.3
29.1
40.2
37.0
29.9
42.6
38.9
32.5
DO
0.87
0.58
0.13
0.85
0.62
0.09
0.87
0.66
0.11
SSIM
DB
0.93
0.90
0.72
0.92
0.88
0.70
0.93
0.90
0.75
DMB
0.94
0.86
0.96
0.95
0.88
0.98
0.94
0.91
Loss changes across iterations (ResNet-50)
Layer 12
Loss
0.02
Layer 24
Layer 36
0.00
Original
Layer
Layer 24
Layer
0
250
500
750
1000
1250 1500
Iterations
Figure 13: White-box feature inversion from different layers
of YOLO.
Figure 12: Changes training loss.
In Figure 28 we illustrate the changes on the loss function throughout the reconstruction process as outlined in Algorithm 1 using the intermediate features from ResNet-50. Tt evident. that the loss values converge by the end of the 1500 iterations.
6.3 Comparison with White-box GAN-based Inversion Method
We perform another experiment. by replacing the diffusion model with StyleGAN3 Karras et al. [2021). To achieve this, replace the diffusion model in Figure | of the paper with the StyleGAN3, and train the latent vector of StyleGAN3 to reconstruct the user input. We evaluate under both white-box setting. All training settings remain consistent with those described in Sections 6.1 of the paper. The results are shown in Table 2 for white-box and black-box settings, respectively. We can see that our diffusion model-based approach outperforms the GAN-based approach.
Table 2: Evaluation results of white-box setting. "StyleGAN-baseed and 'DMB" denote StyleGAN- and DM-based approaches, respectively. 'T," denotes the layer where the features are extracted. PSNR is shown in db. For 18, PSNR and SSIM, higher values indicate better results.
ResNet- 18
ResNet-50
ViT-
Metric
Method
L44
T,8
L122
L,1
T.24
T.36
L33
IS
StyleGAN-bassel
7.02
6.22
5.06
7.04
6.12
4.88
6.89
5.92
4.00
DMB
7.23
6.86
6.48
7.36
6.90
6.55
7.14
6.77
6.58
PSNR
StyleGAN-baase -assed
37.3
34.1
22.5
36.9
34.3
21.7
35.3
34.7
21.0
DMB
41.0
36.3
29.1
40.2
37.0
29.9
42.6
38.9
32.5
SSIM
StyleGAN-basee
0.94
0.92
0.78
0.94
0.90
0.76
0.92
0.88
0.74
DMB
0.97
0.94
0.86
0.96
0.95
0.88
0.98
0.94
0.91
13

---

Published in Transactions on Machine Learning Research (08/2025)
Reconstructed Original
Figure 15: End-to-end feature inversion CLIP image encoder.
6.4 Feature Inversion Results YOLO
We present additional feature inversion results over a YOLO-v2 model based on ResNet-50 for object detec- tion (Figure 13). We select 100 samples from the COCO dataset for the inversion experiment. Specifically, for YOLO, our method can achieve higher average inception scores of 8.13, 7.22, and 6.32 using features from layers 12, 24, and 36. This is much higher than IS scores obtained by DO method (5.60, 3.75 and 1.29 for L12, L24 and L36, respectively) and DB method (6.96, 5.80 and 4.47 for L12, L24 and L36, respectively).
6.5 End-to-end Feature Inversion over CLIP
In this section, we present the results for the end- to-end feature inversion over the CLIP Radford at. (201) image encoder. CLIP is language-visual multimodal DNN capable of understanding images and text jointly in zero-shot manner, without the need for fine-tuning on specific task. Tt aligns natu- ral language prompts with images to perform wide range of tasks, including image classification, image generation, and image-text retrieval. Specifically, the CLIPP image encoder has been widely adopted for various of computer vision tasks including face recognition Bhat & Jain (2023); Shen et al. (2023), image segmentation Wang et al. (202bb) and emotion classification Bondielli et al. (2021): Deng al. (2022). We use the methodology described in Section 3.1 to invert the output features from the pretrained CLIP image encoder. Specifically,
Original image (b) Synthetic image (c) Reconstructed image
Figure 14: Limitations on utilizing text priors.
download clip-ust-base. patch32 from the huggingface official website hug 2021 and apply the settings described in the beginning of Section G for evaluation. We select 100 images from the TmageNet test dataset, some images together with their reconstructed versions are shown in Figure 15 Specifically, we obtain an IS, PSNR and SSIM of 3.54, 13.2 and 0.50, respectively. In comparison, DO method achieves a IS, PSNR and SSIM of 0.88, 5.78 and 0.11, respectively. Similarly, the DB method also attains IS, PSNR, and SSTM scores 2.21, 9.74 and 0.33, respectively. Furthermore, we note that, when compared with DNN designed for image classification, CLIP is much easier lo invert at the same layer depth. This enhancement could be attributed to CLIP's lendency to retain more original image data support wide range of downstream tasks, thereby aiding feature inversion. In contrast, DNNs for image classification typically discard redundant information, preserving only the essential data required for recognizing object classes within the image.
6.6 Impact of Text Prior on Feature Inversion
In this section, we evaluate the impact of the lext prior on the feature inversion results. Particularly, we extract intermediate features from deeper layers, such as layer 48 in Res.Net-50. When attempting to invert these features from the deep layers, we observe a substantial quality degradation over the reconstructed input with the methods in Section 3.1 This decline can be primarily attributed to the loss of critical information embedded within deep features.
14

---

Published in Transactions on Machine Learning Research (08/2023)
(a)
(b)
(c)
(d)
LBuue sky"
Sample
"G Land
Sample
"Soww
Sample
Figure 16: Impact of the text prior on feature inversion. Three samples with different scenes are selected from the ImageNet, shown in column (a). Column (b) shows the textual inputs and the corresponding synthetic images. Column (c) and (d) depict the reconstructed input with and without the text description.
To evaluate the impact of textual prior, we select multiple images from the ImageNet dalaset leaturing simple backgrounds, including scemes with sky, grassland, and snow (Figure I6 (a)). For each of these three background categories, we choose 10 images. We utilize the textual descriptions "blue sky", "a piece of grassland", and "snow" textual guidance for the LDMs during the feature inversion process. Subsequently, these selected images are forwarded through ResNet-18, ResNet-50 and ViT-based, and extract the results from the output of layer 16 for ResNet-18, layer 48 for ResNet-50, and layer 8 for ViT-base, respectively, The extracted intermediate results are then employed to reconstruct the input images using the techniques outlined in Section 4.1. Throughout the reconstruction process, X and Aes in Equation are set to and 10, respectively. The rest of the settings remain the same as those Section 6.2
The reconstruction results are shown in Figure 16 We notice significant enhancement in the fea- ture inversion quality when incorporating the tex- tual prior, as depicted in Figure 16 (c). These re- constructions, although not pixel-perfect, closely re- semble the original images in (a) in teriss of the semantic content. In comparison, the reconstruc- tions obtained without lextual prior, as shown in Figure 16 (d), are not semantically meaningful. Ta- ble shows quantitative results for feature inversion with (right) and without (left) textual prior in terms of average IS, PSNR and SSIM. Evidently, including the textual prior significantly improves the reconstruction quality quantitatively as well.
Intermediate results
Reconstructed inpul
Intermediate results
Reconstructed
U-Net
inpul
Inversion DNN
Inversion DNN
(a)
(b)
Figure 17: Basclines for black-box evaluation.
While the use of a lextual prior can greatly enhance the reconstruction quality, it should be employed judiciously, as its improper use can potentially impair the reconstruction results. To illustrate this, we utilize intermediate results from layer 16 of ResNet-18 reconstruct the user input shown Figure 14 (a), while providing the LDM with text prior "yellow pickup park along the road" for feature inversion. Surprisingly, this does not lead to improved reconstruction quality, depicted in Figure 14 (c). One possible explanation is this description fails to accurately characterize the object within the victim image, resulting in synthetic images that incorrectly represent the foreground in terms of shape, texture and position, as seen in Figure 14 (b). This misalignment further degrades inversion quality. In general, we notice that offering a simple textual description of the image background tends to enhance the reconstruction performance. These descriptions provides an overview on the background of the image, outlining key attributes like the dominant color and surroundings. Our research marks the first step investigating how textual descriptions influence feature inversion quality. Further investigation is needed to fully grasp the impact of textual priors
15

---

Published in Transactions on Machine Learning Research (08/2025)
Original
Without temporal loss
With temporal loss
Clip
Clip 2
Clip
frame
frame 2 frame 3 frame 4 frame frame 2 frame frame 4 frame frame 2 frame 3 frame
Figure 18: Feature inversion across multiple frames. Our method obtains average increase of 1.8, 7.3 and
0.39 for 1S, PSNR and SSIM.
Table 3: Feature inversion quality with and without textual priors. out/ /withh priors
left/right numbers show results with-
Metrics
ResNet-18 (L16)
ResNet-50 (L48)
ViT-basc (L8)
IS
0.33/3.60
0.35/3.84
0.47/3.00
PSNR
4.5/15.4
5.2/14.9
4.3/14.7
SSIM
0.02/0.50
0.03/0.54
0.02/0.56
6.7 Multi-frame Feature Inversion
In this section, we evaluate the multi-frame feature inversion algorithm detailed in Section 4.2 To create multi-frame inputs with high correlations, utilize the tanks and temples dataset Knapitsch et al. (2017 which contains high-resolution video clips for twelve different objects. We select ten video clips and extract four consecutive frames from each video clip, the time interval between a pair of consecutive two frames is 0.5 seconds. Subsoquently, we employ the loss function defined in Equation 10 to jointly reconstruct the four frames. We configure Ås and A Equation 10 process, we the sampling steps 10. All other settings remain consistent with the earlier sections.
1 and 5, respectively. To accelerate the reconstruction
To demonstrate the advance of the proposed loss function, conduct comparison of reconstructed image quality with and without the inclusion of the smoothing loss Le- From the results presented in Figure [8] we observe a noticeably better reconstruction quality by using the smoothing loss. We also observe that the inclusion of the smoothing loss results in improvements in IS, PSNR, and SSIM scores, with an average increase 1.8 for IS, 7.3 dB for PSNR, and 0.39 for SSIM across all the video clips.
6.8 Ablation Study Number of Diffusion Sampling Steps
In this section, we investigate how the number of sampling steps affects the reconstruction quality. We utilize the intermediate fcatures from layer of ResNet-50 across 100 inputs from the ImageNet. Subsequently, we conduct feature inversion, as outlined in Section 3.1 employing various sampling step values for the LDM, Table 4 depicts how the IS and SSIM scores evolve with different sampling steps. We observe that both scores increase the number of sampling steps increases. Nevertheless, both stabilize when sampling steps exceods 20. Therefore, in this study, we employ a sampling step value of 20 to achieve the optimal balance between feature inversion quality and training efficiency.
Table 4: IS and SSIM scores with different sampling steps.
Metrics
10 steps
15 steps
20 steps
25 steps
30 steps
IS
5.93
6.38
6.55
6.62
6.65
SSIM
0.80
0.87
0.88
0.89
0.89
16

---

Published in Transactions on Machine Learning Research (08/202))
Training changes during training IResNet-501
Inception score
8
L12
L24
L36
0.15
layer 24
layer 12
6
LO5S
0.10
layer 36
0.05
20
80
1024
2048
4096
8192
Epochs (a)
Training data size (b)
Figure 19: (a) Training loss vs. number of epochs for black-box feature inversion over ResNet-50 on ImageNet. Features are extracted from the end layer 12, layer 24 and layer 36 respectively. (b) Inception scores with different training data size on ResNet-50 with ImageNet.
Original
Direct Optimization
Decoder-based
DM-based
Original
layer 12 layer layer 36
layer 12 layer
layer 36
layer layer layer 36
Figure 20: Black-box feature inversion over ResNet-50 on ImageNet, features are extracted from the end of layer 12, layer 24 and layer 36 respectively.
6.9 Ablation Study on Loss Function Setting
We conduct an ablation study to evaluate the impact of the loss func- tion components image reconstruction quality. Specifically, var- ied the importance weights X and Atst in Equations g|| to assess their
Table 5: Ablation study on loss function setting.
influence. The evaluation was carried out by mcasuring changes in re- construction quality under white-box inversion allacks on ResNet-50, the results are shown in Table We observe that setting Atst 0, thereby removing the influence of the textual prompt, leads a de-
Setting Default
L12
L24
L36
7.36
6.90
6.55
Set X
7.11
6.47
6.20
cline in reconstructed image quality the layer depth increases. Addi- tionally, removing the TV loss consistently degrades feature inversion quality across all layer depths.
Set Atat
0 7.34
6.72
5.88
7 Results for Black-box Feature Inversion
7.1 Experiment settings
We use the same datasets, target models and LDM as described in the Section 6.1 To build the training dataset and test dataset of inversion DNN, we randomly select 4096 and 1024 images from the training and test datasets of either ImageNet or YOLO, respectively. We notice that a training data size of 4096 is enough for inversion DNN to generalize well.
Datasets and models:
17

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Evaluation results for black-bo box inversion.
NesNet--18
ResNet-50
ViT-base
Metric
Method
L4
L8
L12
L12
L24
L36
L3
L4
L5
DO
5.38
3.64
1.01
5.36
3.41
0.98
5.31
3.77
1.48
IS
DB
6.53
5.23
3.60
6.79
5.20
3.21
6.62
5.16
3.51
DMB
6.99
6.21
5.19
7.08
6.44
4.89
6.98
6.27
5.00
DO
27.6
11.1
7.74
26.3
13.1
7.11
26.6
13.2
7.80
PSNR
DB
34.3
24.3
9.6
33.4
27.0
12.3
35.3
23.9
11.5
DMB
40.4
32.5
20.6
39.2
31.4
23.9
41.6
31.5
19.5
DO
0.84
0.40
0.08
0.84
0.54
0.07
0.85
0.62
0.08
SSIM
DB
0.90
0.78
0.38
0.92
0.79
0.43
0.91
0.80
0.55
DMB
0.92
0.84
0.46
0.94
0.88
0.67
0.95
0.88
0.71
Hyperparameters: The inversion DNNs are trained over 96 epochs using batch size of 128. We assign X values of in Equations I1
12 and 13
We employ the Adam optimizer with an initial learning rate of
0.1 and 3 values of (0.9, 0.999).
Baseline: Following the baseline setups outlined 6.1 we examine the black-box versions of DO and DB. In the case of DO (Figure 17 (a)), we modify the architecture of the inversion DNN to directly reconstruct the user input without relying on the LDM. Conversely, for DB (Figure 17 (b)), we integrale the decoder from the LDM into the inversion DNN to improve the quality of reconstruction. We change the structure of the inversion DNNs for DMB, DO and DB to ensure they contain equal number of parameters. DO has been applied by the He et al. (0019) Dong et al. 2021];; Maeng et al. (0222 for feature inversion attack under the black-box setting, but. DB has not. been studied in prior works.
7.2 Feature Inversion of Split Models for Image Classification
Similar to Section 6.2 we begin by evaluating the reconstruction accuracy of our black-box feature inversion attacks withoot utilizing text priors. For every target DNN model, we partition them into blocks of layers and capture intermediate outputs at the conclusion of each block. Next, we apply the methods presented in Section 5 to reconstruct the user input. A sample training loss curve for inversion DNN is shown in Figure 19 (a).
Figure 20 illustrates the feature inversion outcomes for ResNet-50 on ImageNet, while Table 0 presents the mean IS, PSNR, and SSTM values computed across the test dataset for different model architectures and feature inversion techniques. Notably, our approach (DM-based) consistently exhibits a higher reconstruction quality across all datasets and DNN architectures. Furthermore, is worth noting that, under identical settings, the reconstruction quality tends to be lower for the black-box feature inversion attacks compared to the white-bos feature inversion attacks, as shown in Section 6.2
7.3 Comparison with Black-box GAN-based Inversion Method
We conduct an additional experiment by replacing the diffusion model with StyleGAN3 under the black- setting, while keeping all training configurations consistent with those outlined in Section 7.1 The results, presented in Table 7 demonstrate that our diffusion model-based approach achieves superior performance compared to the GAN-based alternative.
7.4 Additional Results over YOLO and CLIP
We present additional results using a YOLO-v2 model basod on ResNet-50 for object detection. The inputs are reconstructed by the inversion DNN using the intermediate results from layer 12, 24, and 36 (Figure 21 (a)). DMB achieves inception scores of 7.54, 6.99, and 6.80 using features from layers 12, 24, and 36 over the
18

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Evaluation results with black-box setting. "StyleGAN-basedd and 'DMB" denote StyleGAN-based and DM-based approaches, respectively. 'L' denotes the layer where the features are extracted. PSNR is shown in db. For 18, PSNR and SSIM, higher values indicate better results.
ResNet-18 18
ResNet-50
ViT-
Metric
Method
L4
L8
L12
L12
L224
L36
L3
L4
L5
IS
StyleGAN-
6.70
5.29
4.04
6,87
5.87
4.02
6.77
5.79
4.23
DMB
6.99
6.41
5.19
7.08
6.64
4.89
6.98
6.37
5.00
PSNR
StyloGAN-bascll
36.8
27.6
11.4
35.4
28.9
16.7
37.0
26.4
14.8
DMB
40.4
32.5
20.6
39.2
31.4
23.9
41.6
31.5
19.5
SSIM
StyleGAN-basert
0.92
0.80
0.41
0.93
0.82
0.54
0.93
0.82
0.60
DMB
0.92
0.84
0.46
0.94
0.88
0.67
0.95
0.88
0.71
Table 8: Black-box feature inversion results with/without textual priors. Left/right numbers show results
without/win priors.
Metrics
ResNet-50 (L48)
YOLO (L48)
CLIP (End-to-end)
IS
0.29/30 0.29/3.11
0.88/3.48
3.22/4.00 4.09
PSNR
4.3/13.6
5.4/14.0
12.0/17.7
SSIM
0.03/0.50
0.08/0.44
0.46/0.56
test dataset of COCO. This is much higher than IS scores obtained by DO method (5.64, 3.97 and 1.20 for L12, L24 and L36, respectively) and DB method (6.95, 5.95 and 4.46 for L12, L24 and L36, respectively).
feature inversion using the CLIP Radford et al. (021) image encoder. Specifically, we curate subset of 1024 images from the ImageNet IS, PSNR, and SSIM of 3.22, 12.0, and 0.46, respectively. Contrasting this, the DO method yields an IS, PSNR, and SSIM of 0.45, 2.62, and 0.09, respectively, while the DB method returns IS, PSNR, and SSTM scores of 2.03, 9.10, and 0.25, correspondingly. Similarly, we observe that CLIP is notably more amenable to inversion at the same layer depth compared to a DNN tailored for image classification.
Moreover, in Figure 21 (b), we showcase the outcomes of black-box
test dataset and achieve
7.5 Impact of Text Prior on Feature Inversion
In this section, we analyze the impact of textual prior on the inversion results. Especially, we extract the intermediate feature from layer 16 in ResNet-18 and layer 48 from YOLO, we extract intermediate features from deeper layers, such as layer 48 in ResNet-50, and invert the image with simple background, as described in Section 6.6 Xa and Atai in Equation are set to and 3, respectively.
Table highlights the quantitative results for feature inversion with and without textual prior in terms of average IS, PSNR and SSIM. Evidently, including the textual prior significantly improves reconstruction quality quantitatively as well. However, we also observe that a similar failure case as described in Section 6.6 under black-box setting.
7.6 Multi-frame Feature Inversion
We employ the same training and testing datasets as described in 6.7 Specifically, the training and test datascts include 1024 and 256 video clips, respectively. A, in Equation 13 are set to 1. We conduct comparison of reconstructed image quality with and without the inclusion of the pointwise convolution layer described in Figure D whose primary function is consider the temporal correlation during the reconstruction of input frames. Moreover, we adjust the structure of the DNN so that the total amount of parameters are the same for both scenarios. We note significant enhancement in IS, PSNR, and SSIM scores with the inclusion of
In this section, evaluate the multi-frame feature inversion algorithm detailed in Section 10.9
19

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Black-box Inversion on YOLO
(b) End-to-end Inversion on CLIP
Reconstructed Original
Reconsiructed Original
Figure 21: (a) Black-box feature inversion using the output from layer 36 of YOLO. (b) End-to-end black-bo box feature inversion attack over CL.IP.
the pointwise convolutional layer, resulting in an average increase of 0.2 for IS, 4.0 dB for PSNR, and 0.09 for SSIM across all video clips in the test dataset using the intermediate results from layer 36 of ResNet-50 on ImageNet datasct. More evaluation results are presented in the appendix.
7.7 Ablation Study on Training Set Size
We study the impact of training dataset size reconstruction quality. We vary the size of the ImageNet training dataset while maintaining the test dataset size at 1024, and measure feature inversion outcomes using the black-box setting. Figure 19 (b) demonstrates that IS scores stcadily incrcase as the training dataset size grows. Interestingly, even with a batch size of 1024, a notably high IS score is achieved, suggesting that a smaller training dataset can still facilitate effective generalization of the inversion DNN.
8 Discussion
In this section, we summarize some findings we observe from the evaluation results (Section 8.1 We then
discuss the potential defense strategies in Section 8.2
8.1 Insights From the Evaluation Results
A deeper DNN does not guarantco privacy: Based on the results outlined in Section 6.2 and Section 7.2 it becomnes evident that deeper DNN does not inherently ensure privacy. For instance, as observed in Table and Table 6 the quality of reconstructed input using the intermediate features at the 12th layer of ResNet- 18 is notably inferior to that of the 24th layer output of ResNet-50. This suggests that the absolute layer depth alone does not guarantee any privacy protection. Instead, what matters is the relative layer depth within the DNN. For instance, the reconstructed quality using the outputs of middle layer of ResNet-50 (e.g., 24) is approximately equal to that from the middle layer (e.g., 8) of ResNet-18.
Transformer is harder to invert than CNN: Another trend we notice is that transformers, such ViT, exhibit. better privacy protection capabilities compared to Convolutional Neural Networks (CNNs). As indicated by the results in Table | and Table P the quality of reconstructed input using features from the middle layer (c.g., 5) of ViT-base is comparable to that obtained using features from later layers in ResNet-18 or ResNet-50. This may contribute amounts to a low-pass filter Wang et al. (2022ab, this will climinate a lot of high-frequency information within the original input image, making the intermediate features harder to invert. By contrast, CNNs typically extract information across wide frequency ranges Yosinski et al. (2015), thereby retaining more essential information for fcature inversion. Nevertheless, further studies are needed.
the fact. that transformers with self-attention mechanism inherently
Self-supervised pretrained backbone models are easier to invert: The evaluation results shown in Section 6.5 and Section 7.4 illustrae that pretrained backbone models with self-supervised learning, are more amenable inversion compared to DNNs trained with supervised learning frameworks tailored for specific tasks like image classification. This due to the fact that the SSL-pretrained backbones tend preserve a rich set of information that can be beneficial for various downstream tasks. The pretraining process typically
20

---

Published in Transactions on Machine Learning Research (08/2025)
involves learning representations that capture meaningful patterns and structures in the input data, which can generalize well to different. tasks. In contrast, DNNs trained with supervised learning using labeled datasets tend to eliminate redundant information unrelated to the task during the training process, making the input data more challenging to reconstruct.
A well-constructed textual prompt can improve inversion performance: This work the first work to demonstrate that additional information in another format (i.e., textual format) can be utilized to enhance reconstruction performance. Our general finding is that a simple textual description consisting of a few words that provides overview of the image background, highlighting attributes like the dominant color and surroundings, can generally enhance reconstruction quality. Although applying an inaccurate textual prior will degrade the attack quality, if the attacker has multiple candidate textual descriptions, the best strategy is to exhaustively try all of them and select the one that achieves the optimal quality. While using textual priors in feature inversion is not our main focus, it is a promising area for future rescarch.
In real-world scenarios, descriplive information about the victim's input can often be inferred or leaked. In split computing settings, attackers may gain access to contextual prior knowledge that enables them to craft effective textual prompts, enhancing the success of feature inversion attacks. For example, in smart home or AR/VR environments, common indoor contexts such as a living room or kitchen are casily inferred. In professional applications like Codec Avatars, an attacker may know that the user is participating in a virtual meeting. Location- based AR applications can reveal contextual clues through GPS, allowing prompts such as "Eiffel Tower on a clear day." Similarly, in autonomous driving or fitness apps, route history or usage data may suggest likely scenes such as "a suburban street or "a person doing yoga indoors." These contextual cues, obtained through metadata, environmental knowledge, or user behavior, make prompt-based inversion attacks both practical and more effective.
Revealing the model weight and structure can improve the quality of the attack: Based on the evaluation results presented in Table and Table G it is evident that the feature inversion attack achieves higher quality results in the white-box setting compared to the black-box setting. This underscores the importance of weight and architecture of the user model influencing the quality of feature inversion.
8.2 Defense over Feature Inversion Attack
In this section, we explore potential defense strategies to mitigate the feature inversion attacks outlined in Section 4 and Section 5 One defensive strategy is to ensure that all DNN computation and communication happen over encrypted data, i.e. through cryptographic methods like secure multiparly computation (MPC) or homomorphic encryption (HE).
Secure MPC allows a group of n untrusting partics to collaboratively compute a public function I2,,,) over their private inputs æ1, Z2.,, without revealing any of their secret information. If the MPC scheme is expressive enough to implement large and complex neural networks like diffusion models, the implementation often has prohibitively large communication overheads and high computational complexity. For example, recent work on MPC implementation of VGG16 in the WAN setting leads to 37s latency (and training can take several weeks) Wagh et al. 22020).
Similarly, HE schemes allow certain operations, such as arithmetic or boolean functions, to be applied to the ciphertext, thereby allowing privacy-presers mation in plaintext form. But computing on ciphertexts over plaintext means both higher communication and computation costs. Additionally, HE requires additional computation steps like moiso-management via bootstrapping. Prior implementations show significant slowdown, 300s for eneryption, DNN application, and decryption (up to 30s for 5x5 convolutional layer to simple 5-layer MNIST network, and 127s for pooling) Silad-Bachhach et al. (2016). While these are promising and active research areas, at this time these methods are not widely deployed due to large overheads over an already computationally intense DNN architecture.
neural network evaluations without revealing sensitive infor-
Another feasible approach involves the use of differential privacy (DP) to introduce noise and obscure sensitive information. In particular, random noise e can be directly integrated into the intermediate results 2mid: with the magnitude of the noise being controlled to achieve the desired level of DP. Nevertheless, it's crucial to
21

---

Published in Transactions on Machine Learning Research (08/2025)
acknowledge the trade-off between usability and privacy. When higher levels of noise introduced, there will be corresponding drop in model accuracy. To mitigate this loss in accuracy, it is beneficial to take into account the influence of the injected noise during the training phase for the target DNN Fo(.).
9 Conclusion and Future Work
In this study, we demonstrate the significant performance enhancement achievable in the feature inversion process via the utilization of the diffusion model. We also highlight the polential for utilizing diverse forms prior knowledge, such as textual information and cross-frame correlations, to further improve the roconstruction quality. From the evaluation results, we show that GenAI, with its remarkable ability synthesize realistic and coherent data, can also be utilized detrimentally affect individuals' lives, particularly in the context of privacy breaches. This opens up interesting future avenues in promising direction of research.
Several directions remain for future work. First, although textual input has been shown to enhance im- age reconstruction, an open and compelling question is how to automatically generate prompts with the optimal level of detail to maximize reconstruction quality. Second, given the rapid advances in generative image models, it would be valuable to investigate the elfectiveness of newer architectures, such as diffusion transformers Peebles & Xie (2023). Finally, exploring feature inversion attacks in the context of foundation models presents important and promising area for further study.
22

---

Published in Transactions on Machine Learning Research (08/2025)
References
Azure face recognition. 2018, https://azuue. microsoft.com/ htterrooooooo//ee//erreeee//rrer
Pysyft, 2020. https://githubbe https:/ggtubbccom///peeiiie/////
Splitmn, 2020. https///blgggeemmmiiooooogggg/sp
Amazon rekognition., 2021. https///wwssammzonn.cc////eoggn/
Clip image encoder, 2021. http::///nngiiiiarccc/o////////p
Divyansh Aggarwal, Jiayu Zhou, and Anil K Jain. Fedface: Collaborative learning of face recognition model. In 2021 IEEE International Joint Conference on Biometrics (IJCB), pp. 1-8. IEEE, 2021.
Samson B Akintoye, Liangxiu Han, Huw Lloyd, Xin Zhang, Darren Dancey, Haoming Chen, and Daoqiang Zhang. Layer-wise partitioning and merging for efficient and scalable deep learning. Future Generation Computer Systems, 149:432-444, 2023.
Aaditya Bhat and Shrey Jain. Face recognition in the age of clip & billion image datasets. orXiv preprint arXiv:2301.07315, 2023.
Alessandro Bondielli, Lucia C Passaro, et al. Leveraging clip for image emotion recognition. In CEUR WORKSHOP PROCEEDINGS, volume 3015. CEUR-WS, WS, 2021.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoe, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206 2240. PMLR, 2022.
Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh. Expressive telepresence via modular codec avatars. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part XII 16, pp. 330-345. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Sinuo Deng, Lifang Wu, Ge Shi, Lehao Xing, Meng Jian, and Ye Xiang. Learning to compose diversified prompts for image emotion classification. arXie preprint 0rXiv:2201.10963 2022.
Aolin Ding, Amin Hass, Matthew Chan, Nader Sehatbakhsh, and Saman Zonouz. Resource-aware dnn partitioning for privacy-sensitive edge-cloud systems. In International Conference on Neural Information Processing, pp. 188-201. Springer, 2023.
Xin Dong, Hongxu Yin, Jose M Alvarez, Jan Kautz, Pavlo Molchanov, and HT Kung, Privacy vulnerability of split computing to data-free model inversion attacks. arXiv preprint orXiv:2107.0630, 2021.
Xin Dong, Barbara De Salvo, Meng Li, Chiao Liu, Zhongnan Qu, and Ziyun Li. Splitnets: Designing neural architectures for efficient. distributed computing on head-mounted systems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 12559 12569, 2022a.
Xin Dong, Sai Qian Zhang, Ang Li, and HT Kung. Spherefed: Hyperspherical federated learning. Tn European Conference on Computer Vision, pp. 165 184. Springer, 2022b.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4829 4837, 2016.
Ege Erdogan, Alptekin Küpçü, and A Ercüment Çiçek. Unsplit: Data-oblivious model inversion, model stcaling, and label inference attacks against split learning. In Proceedings of the 21st Workshop on Privacy in the Electronic Sociely, pp. 115-124, 2022.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Thomas Feltin, Léo Marchó, Juan-Antonio Cordero-Fuerted Frank Brockners, and Thomas H Clausen. Dnn partitioning for inference throughput acceleration at the edge. TEEE Access, 11:52236-522444 2023.
Yonggan Fu, Yuecheng Li, Chenghui Li, Jason Saragih, Peizhao Zhang, Xiaoliang Dai, and Yingvan Celine Lin. Auto-card: Efficient and robust codec avatar driving for real-time mobile telepresence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21036 21045, 2023.
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks encrypted data with high throughput and accuracy. In Interna- tional conference on machine learning, pp. 201-210. PMLR, 2016.
Johann Hauswald, Thomas Manville, Qi Zheng, Ronald Dreslinski, Chaitali Chakrabarti, and Trevor Mudge. A hybrid approach offloading mobile image classification. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8375-8379. IEEE, 2014.
Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148-162, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Newral networks, 13(4-5):411-430) 2000.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advonces in neural information processing systems, 34:29898 -99008, 2021.
Woosung Kang, Siwoo Chung, Jeremy Yuhyun Kim, Youngmoon Lee, Kilho Lee, Jinkyu Lee, Kang Shin, and Hoon Sung Chwa. Dnn-sam: polit-add-errg dnn execution for real-time object detection. In 2022 IEEE 28th Real- Time and Emaedded Technology and Applications Symposium, (RTAS), pp. 160-172. IEEE, 2022.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45(1):615-622 2017.
Jyotirmoy Karjee, Praveen Naik, Kartik Anand, and Vanamala N Bhargav. Split computing: Dun inference partition with load balancing in iot-edge platform for beyond 5g. Measurement: Sensors, 23:100409, 2022.
Tero Karras, Miika Ailtala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free free generative adversarial networks. Advances in neural information processing systems, 34:852 863, 2021.
Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large- scale scene reconstruction. ACM Transactions on Graphics 36(4), 2017.
Jaeduk Lee, Hojung Lee, and Wan Choi. Wireless channel adaptive dnn split inference for resource- constrained edge devices. IEEE Communications Letters, 2023.
José Lezama, Qiang Qiu, and Guillermo Sapiro. Not afraid of the dark: Nir-vis face recognition via cross- spectral hallucination and low-rank embedding. In Proceedings of the IEEE conference computer vision and pottern recognition, pp. 6628 6637, 2017.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Liu. Auditing privacy defenses in federated learning via generative gradient leakage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10132-10142, June 2022.
Jicyu Lin, Minghao Li, Sai Qian Zhang, and Alberto Leon-Garcia. Murmuration: On-tho-fly dnn adaptation for slo-aware distributed inference in dynamic edge environments. In Proceedings of the 53rd International Conference on Parailel Processing, pp. 792 792-801, 2024.
24

---

Published in Transactions on Machine Learning Research (08/2025)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLarrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neural Information Processing Systems, 35:5775-577 5787, 2022.
Huimin Lu, Ming Zhang, Xing Xu, Yujie Li, and Heng Tao Shen. Deep fuzzy hashing network for efficient image retrieval. IEEE transuctions on fuzzy systems, 29(1):166 176, 2020.
Xiaoqiang Lu, Yaxiong Chen, and Xuelong Li. Hierarchical recurrent neural hashing for image retrieval with hierarchical convolutional features. IEEE transactions on image processing, 27(1):106-122 2017.
Diaohan Luo, Tian Yu, Yuewen Wu, Heng Wu, Tao Wang, and Wenbo Zhang, Split: Qos-aware dnn inference on shared gpu via ovenly-sized model splitting. In Proceedings of the 52nd International Conference on Parallel Processing, pp. 605-614, 2023.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint 220 12524, 2022.
Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 64 64-73, 2021.
Kiwan Maeng, Chuan Guo, Sanjay Kariyappa, and Edward Suh. Measuring and controlling split layer privacy leakage using fisher information. arXiv preprint or arXi::2209.1011 2022.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 5188-5196, 2015.
Guangcan Mai, Kai Cao, Pong C Yuen, and Anil K Jain. On the reconstruction of face images from deep face templates. IEEE transactions on pattern analysis and machine intelligence, 41 1202, 2018.
Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, and Sameer Singh. Distilled split deep neural networks for edge-assisted real-time systems. In Proceedings of the 2019 Workshop on Hot Topics in Video Analylics and Intelligend. Edges, pp. 21 26, 2019.
Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. Split computing and early exiting for deep learning applications: Survey and rescarch challenges. ACM Computing Surveys, 55(5):1-30, 2022.
John X Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embeddings reveal (almost) as much as text. arXiv preprint arXiw:2310.066. 0.06666, 2023.
Waleed Hassan Mubark, Jagannath Guptha Kasula, and Md Yusuf Sarwar Uddin. Asap: Asynchronous split inference for accelerated cnn execution. In Proceedings of the 25th International Conference on Distributed Computing and Networking, pp. 32 44, 2024.
Hatef Otroshi Shahreza and Sébastien Marcel. Face reconstruction from facial templates by learning latent space of a generator network. Advances in Neural Information Processing Systems, 36, 2024.
Jongho Park, HyukJun Kwon, Seowoo Kim, Junyoung Lee, Minho Ha, Euicheol Lim, Mohsen Imani, and Yescong Kim. Quiltnet: efficient deep learning inference on multi-chip accelerators using model partition- ing. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 1159 1164, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IRFE/CVF international conference on computer vision, pp. 4195-4205, 2023.
25

---

Published in Transactions on Machine Learning Research (08/2025)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 10600-169. PMLR, 2016.
Alexander Richard, Colin Lea, Shugao Ma, Jurgen Gall, Fernando De la Torre, and Yaser Sheikh. Audio- and gaze-driven facial animation of codec avalars. In Proceedings of the IEEE/CVE winler conference applications of computer vision, pp. 41-50, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent dilfusion models. In Proceedinggs of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 10684-10694 2022.
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-265 268, 1992.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 36494, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 815-823, 2015.
Shuai Shen, Wanhua Li, Xiaobing Wang, Dafeng Zhang, Zhezhu Jin, Jie Zhou, and Jiwen Lu. Clip-cluster: Clip-guided attribute hallucination for face clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20786-20795, 2023.
Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp. 377-390, 2020.
Jiaming Song, Chenlin Mong, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.0280 02500, 2020a.
Mengkai Song, Zhibo Wang, Zhifei Zhang, Yang Song, Qian Wang, Ju Ren, and Hairong Qi. Analyzing user-level privacy attack against federated learning. IEEE Journal Selected Areas in Communications, 38(10):2430 2444, 2020b.
Surat Teerapitlayanon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. 2464 2469. IEEE, 2016.
Surat Teerapittayanon, Bradley McDanel, and Hsiang- lssiag-Tssung Kung, Distributed deep neural networks over the cloud, the edge and end devices. In 2017 IEEE 37th international conference on distributed computing systems (ICDCS), pp. 228-339. TEEE, 2017.
Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateck Mittal, and Tal Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning. arXiv preprint or orXiv:200,.02229 2020.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. arXiv preprint arXin:2003.05962 2022a.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVP conference computer vision and pollern recognilion, pp. 11686 11695, 2022b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncolli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(460000-0) 000, 2004.
Chugui Xu, Ju Ren, Deyu Zhang, Yaoxue Zhang, Zhan Qin, and Kui Ren. Ganobfuscator: Mitigating information leakage under gan via differential privacy. IEEE Transactions on Information Forensics and Security, 14(9):2358 -3771 2019.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arxiv preprint 4rXiv:1566.063 2015.
Lickang Zeng, Xu Chen, Zhi Zhou, Lei Yang, and Junshan Zhang. Coodge: Cooperative dnn inference with adaplive workload partitioning over heterogeneous edge devices. IBEE/ACM Trunsaclions on Networking, 29(2):595-600 2020.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907 5915, 2017.
Sai Qian Zhang, Jieyu Lin, and Qi Zhang. Adaptive distributed convolutional neural network inference at the network edge with adenn. In Proceedings of the 49th International Conference on Parallel Processing, pp. 1-11, 2020a.
Sai Qian Zhang, Jieyu Lin, and Qi Zhang. A aulti-agent reinforcement learning approach for efficient client selection in federated learning. In Proceedings of the AAAT conference on artificial intelligence, volume 36, pp. 9091-9099, 2022.
Sai Qian Zhang, Jieyu Lin, Qi Zhang, and Yu-Jia Chen. Learning client selection strategy for federated lcarning across heterogencous mobile devices. In 2024 25th International Symposium on Quality Electronic Design (ISQED), pp. 1-7. TEEE, 2024.
Xiaolan Zhang, Dawei Wang, Pierce Chuang, Shugao Ma, Deming Chen, and Yuecheng Li. F-cad: A framework to explore hardware accelerators for codec avatar decoding. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pp. 763-768. IEEE, 2021.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Gener- ative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b.
Wengang Zhou, Houqiang Li, Jian Sun, and Qi Tian. Collaborative index embedding for image retrieval. IEFE transactions on pattern analysis and machine intelligence, 40(5):1154-1166) 2017.
27

---

Published in Transactions on Machine Learning Research (08/2025)
10 Appendix
10.1
statement on Data Availability
Due to institutional restrictions, we are unable to use the public latent diffusion model for publishing our research outcomes. Instead, we employed an LDM with an architecture highly similar to Stable Diffusion 2.1 in terms of architecture, model size and pretraining techniques. Our internal model was pretrained on the dataset collected by third party (Shutterstock) that is not public. Regarding the dataset, it consists of 385 million images: 321 million without people and 64 million with people. In addition, we have also previously conducted extensive experiments with the public Stable Diffusion 2.1, which yielded similar (and even better) results than the reported results in terms of IS, PSNR, and SSIM scores. If the paper is accepled, we intend to release the code as open source. This code will enable the integration of the public LDM for conducting feature inversion attacks.
10.2 Implementation details
Table and Table 10 list the detailed settings for feature inversion described in Section 4 and Section 5 initiate the training process, the latent variable v is initialized using randomly generated vector sampled from normal distribution with standard deviation of 0.1.
To
General Configuration
Detail
Optimizer
Adam
Total iterations
1500
Base learning rate
0.1
Learning rate schedule
multiple stages
Sampling steps
20
A
1
Table 9: Detailed scttings for white-box feature inversion.
General Configuration
Detail
Optimizer
Adam
Total epochs
96
Base learning rate
0.1
Learning rate schedule
multiple stages
Sampling steps
20
Batch size
128
Training data size
4096
Test data size
1024
Table 10: Detailed settings for black box feature inversion.
In Figure 23 we illustrate the changes on the loss function throughout the reconstruction process as outlined in Algorithm 1 using the intermediate features from ResNet-50. Tt is evident. that the loss values converge by the end of the 1500 iterations.
10.3 Feature inversion training with textual prior
A Algorithm describes the algorithm for feature inversion training with text prior under white-box settings.
28

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 3 Feature Inversion with Text Prior (i()) is the user DNN model. vthh is the imput latent vector of LDMs at iteration total number iterations.
is the learning rate.
forirr is the prior knowledge described in text. E() is the pretrained text encoder.
for m & M do U =m" -mean("")
std(wm
Leot F (D E((prior))) zmiall? (DV((D + Aexellzn -11?
y m++ = dee
m m 1 e std(o
Up.
return D(v Ellprior)).
Original
Direct Optimization
Decoder-based
DM-based
layer 4
layer 8
layer 12
layer
layer
layer 12
layer 4
layer 8
layer 12
Figure 22: Feature inversion of ResNet- 18 on ImageNet with white-box seiling.
29

---

Published in Transactions on Machine Learning Research (08/2025)
Loss changes across iterations (ResNet-50)
Layer 12
Loss
0.02
Layer
Layer
0.00
250
500
750
1000
1250 1500
Iterations
Figure 23: Changes on training loss during the inversion process.
10.4
Multi-frame feature inversion training
Algorithm 2
describes the algorithm for feature inversion training with multiple frames under white-bot
settings.
10.5
Training algorithm for feature inversion under black-box settings
Algorithm
4 describes the algorithm for feature inversion training with black-box setting.
Algorithm Black-box Feature Inversion
(.). is the user DNN model.
Xg = {rg). Yg (ygy are the sets training imput samples and the corresponding intermediate results from the user DNN Fi(.).
(Ty. Mq) is a training sample. X, is the weight for the TV loss. T is total number of iterations. is the lcarning rate.
Finu() (.) is the inversion DNN. D(.) is the latent diffusion model.
Initialize 0 within ()).
for 1 1ses & E do for (Tg. Na) (X,, Y₂) do (ya)
== ((g))
List = - xza + (x)((x)
Compute the gradient and update 0.
return
10.6
More results white-box feature inversion
Figure 22 shows the feature inversion results for ResNet-18 on ImageNet dataset. Finally, Figure 24 shows the feature inversion results for VIT on ImageNet.
10.7
More results on black-box feature inversion
Figure 25
illustrate the feature inversion outcomes for ViT on TmageNet.
10.8 Impact sampling steps
In this section, we show the impact of the DM sampling steps on the feature inversion results for white- box setting. Specifically, we show the reconstructed input images (Figure 26) from layer 36 of ResNet-50 on ImageNet under whito-box settings. This serves as a supplementary addition the ablation studies discussed in Section 6.8 We observe that the feature inversion quality improves as the number of sampling steps increasing from 10 to 20.
30

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: Multi-frame feature inversion results under black-box settings. For ResNet-50 and ViT, features are inverted using the outpuls from L36 and L5, respectively. The number on the left /right represents the results obtained without/wwee /with a pointwise convolutional layer.
Metrics
IS
PSNR
SSIM
ResNet-50
4.82/4.99
23.6/27.2
0.67/0.76
ViT
4.89/5.10
19.8/22.2
0.70/0.77
Original
Direct Optimization
Decoder-based
DM-based
0.12
block block
block block 3
block block
block 3 block 4
block 5
Figure 24: Feature inversion of ViT on ImageNct with whitc-box sctting.
10.9 Multi-frame inversion with black-box settings
In this section, we show the multi-frame feature inversion results under black-box settings (Table 11 We evaluate using two target models, ResNet-50 and VIT. For ResNet-50 and VIT, their features are inverted using the intermediate results from L36 and L5, respectively. We can that involving the pointwise layer in the inversion DNN obtains a clear improvement on the rectnot quality.
10.10 More results on inversion with textual prior
In this section, we show additional results on feature inversion with textual prior (Figure 27 under white- box settings. Specifically, we use the same textual prior Figure [6 Clearly, notice significant improvement in quality when incorporating a textual prior in feature inversion.
31

---

Published in Transactions on Machine Learning Research (08/2025)
Original
Direct Optimization
Decoder-based
DM-based
Original
block 3
block 4 block 5
block 3 block 4
block 5
block 3 block 4
block 5
Figure 25: Feature inversion of VIT on ImageNet with black-box setting.
(b)
(c)
(d)
10 steps
Sample 1
"gluee
steps
Sample
20 steps
Sample
"Snow"
Figure 26: Feature inversion results with different sam-
Figure 27: Feature inversion with Lextual prior.
pling steps.
32


Reviews:
[{"review_id": "fmd8dUcx2y", "reviewer": "Reviewer_gMLz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\na. The idea of leveraging a powerful diffusion model to recover high-quality input images is interesting and reasonable. The authors use a diffusion model pretrained on a large-scale dataset, which provides strong prior knowledge to support image reconstruction.\n\nb. Both white-box and black-box settings are considered. The inclusion of the black-box scenario adds practical relevance and broadens the applicability of the proposed method.\n\nc. The authors conduct extensive experiments across different target models and various intermediate layers. The results clearly demonstrate that the proposed method outperforms the baselines.\n\nWeaknesses:\n\na. The novelty of optimizing the latent vector of a generative model is somewhat limited, as this has been explored in prior inversion works. The main contribution lies in adopting a diffusion model rather than other generative models.\n\nb. Beyond pixel-level evaluation metrics, it would be valuable to consider semantic-level metrics that better reflect perceptual quality or semantic similarity.\n\nc. More qualitative results should be provided to strengthen the empirical evidence and improve the overall persuasiveness of the experimental findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates feature inversion using deep neural networks, with the goal of reconstructing input images from intermediate representations, such as outputs from hidden layers. Rather than directly optimizing the image pixels, the authors propose optimizing the input latent vector of a latent diffusion model, which is then passed through the reverse diffusion process. The key advantage of this approach is that it leverages the generative power of diffusion models to enhance the quality of the reconstructed images.\n\nThe paper considers two scenarios: a white-box setting, where the adversary has full access to the target model, and a black-box setting, where the adversary only observes input-output pairs.\n\nExtensive experiments are conducted across various target models to evaluate the proposed method. It is compared against two baselines: direct pixel optimization and a decoder-based approach. The results show that the proposed method clearly outperforms both baselines. Additional experiments demonstrate that incorporating textual prompts can further enhance reconstruction quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "a. The experimental results, especially the quantitative ones, show that using outputs from shallow layers clearly outperforms using outputs from deeper layers. Please elaborate more on whether it is realistic to obtain outputs from shallow layers in practice, especially under the split DNN computing scenario mentioned in the paper.\n\nb. Using textual prompts can improve the quality of the reconstructed images. The authors should discuss more about how these textual prompts can be obtained in practical settings.\n\nc. Although the authors mention that their approach can also be applied to the scenario of end-to-end feature inversion, they do not compare their method with other privacy attack baselines. Meanwhile, more qualitative results should be provided to make the experiments more convincing.\n\nTypo: five lines under Section 3.3", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M2Q3EYdUli", "reviewer": "Reviewer_pjJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The ablation studies are comprehensive, where the authors include different baselines like resnet, vit, clip, yolo with different layers/depth to prove the soundsness of the proposed approach.\n2. Both qualatitive and quantative results looks good.\n3. The writing is good easy to follow.\n\n## Weakness\n1. The authors should compare with other baselins directly (instead of similar model configurations) to better prove the effectiveness of th proposed methods.\n2. DMS, thought effective, could face latency and memory problems in real world applications. The authors should have further discussions in this issue.\n3. It seems text prioirs have huge impact on the final results, then the accuracy of the description will be a bottleneck, otherwise the authors need to have experiments for further discussion.\n\n## Questions\n1. Elaborate more about the difference between black and white box setting. In 3.2, it mentions,  \"the key distinction being the relaxation of the assumption regarding the adversary’s knowledge of F1(.). Here, the adversary can only gather information about F1 indirectly through querying it.\" However, eq.4 in the black box setting can also be achieved by simply querying F1 w/o knowing its architecture?\n2. 3.3 typo, F2=?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on using diffusion models to improve image inversion quality from internediate DNN features. The authors formulate the problems into the balck box and the white box settings and propose different algorithms for the improvement. Through comprehensive ablation studies and analysis, the authors demonstrate incorporating various forms of prior knowledge, such as textual prompts and cross-frame temporal correlations, to improve the quality of inverted features with DMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Needs clarification of the weakness and questions in the section above.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M7mRsRes8C", "reviewer": "Reviewer_m7AQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Sufficient background.\n* The manuscript clearly explains the distinction between white-box and black-box types of attacks.\n* In general, the paper is very comprehensive and covers several aspects of attacks.\n* Great performance.\n* Detailed insights.\n* Promise for an open-source code and integration with public models (appendix).\n\n**Weaknesses**:\n* Missing ablations of additional losses like the TV loss, “Gaussianity” loss, and the disparity loss. Are they really necessary?\n* The authors mention that inverting ViT features is more challenging. Perhaps the combination of ViT features and U-Net-based models is incompatible, and instead Transformer-based models should be used. I appreciate the insights in the conclusions section.\n* I find the conclusion in the text-based section (“In general, we notice that offering a simple textual description of the image background tends to enhance the reconstruction performance. These descriptions provides an overview on the background of the image, outlining key attributes like the dominant color and surroundings.”) a bit counter-intuitive, as I would assume the background should not be the important part when trying to extract important information (also “provides” -> “provide”).\n* I find the comparison to baselines a bit lacking in my opinion, as there are many several works covered in the related work part of the paper. If it is impossible to compare these baselines, the authors should at least report why. I acknowledge that I’m not an expert in the privacy field and thus cannot provide an expert opinion regarding the right baselines.\n* Lacking discussion of limitations and future work.\n\n**Minor**:\n* Figures could generally be more detailed and explain what is illustrated. For example, Figures 2-4: very hard to understand what the authors want the reader to observe.\n* Page 6, Section 3.3, second paragraph: “by making $F_1 = F$ and $F_2=$,. $F_2=$?\n* Algorithm 1: how is $v$ initialized? Also, $\\epsilon$ is not a great choice for denoting learning rate in my opinion. \n* Equation 6 (page 6) appears out of nowhere. I would suggest building the loss gradually if the discussion of the TV loss comes later.\n* Page 10, “Hyperparamters”: all equation references should use a capital E, e.g. “Equation 6”. Similarly on page 16 Section 7.1.\n* Page 13, Section 6.3 - what does “We select 100 test datasets from the COCO dataset for inversion” mean?\n* Appendix: page 1 - broken reference to “Algorithm”. Similarly in page 2 Section 1.4. Also broken references to sections and figures throughout the rest of the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with a privacy attack paradigm called “feature inversion” in which privacy data is leaked via reconstructing the original input from the intermediate features. Specifically, this work focuses on the visual domain and latent-diffusion-based image generation models. The main contributions of this work include improving feature inversion using diffusion models, and extensions that add support for textual priors and inverting features for videos. The reported reconstruction performance surpasses previous methods, leading to a new state-of-the-art.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Question: it seems the pre-trained encoders were all supervised (except for CLIP, which is trained with text-image pairs). What is the authors’ opinion on using pre-trained self-supervised encoders (such as [DINOv2](https://arxiv.org/abs/2304.07193) for example)?\n* Please go over all the figures and modify the captions to be more informative.\n* See “Weaknesses” and “Minor” above.", "broader_impact_concerns": "I think that given this paper delas with privacy attacks, a \"Broader Impact\" section would be appropriate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)