Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
Authors: Amin Ghiasi
Hamid Kazemi
Steven Reich
Chen Zhu
Micah Goldblum
Tom Goldstein
Venue: ICLR 2022 Submitted (2022)

Abstract: Existing techniques for model inversion typically rely on hard-to-tune regularizers, such as total variation or feature regularization, which must be individually calibrated for each network in order to produce adequate images. In this work, we introduce Plug-In Inversion, which relies on a simple set of augmentations and does not require excessive hyper-parameter tuning.  Under our proposed augmentation-based scheme, the same set of augmentation hyper-parameters can be used for inverting a wide range of image classification models, regardless of input dimensions or the architecture. We illustrate the practicality of our approach by inverting Vision Transformers (ViTs) and Multi-Layer Perceptrons (MLPs) trained on the ImageNet dataset, tasks which to the best of our knowledge have not been successfully accomplished by any previous works. 

Under review as a conference paper at ICLR 2022
PLUG-IN INVERSION: MODEL-AGNOSTI INVERSION FOR VISION WITH DATA AUGMENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Existing techniques for model inversion typically rely hard-to-tune regular- izers, such as total variation or feature regularization, which must be individu- ally calibrated for each network in order to produce adequate images. In this work, we introduce Plug-In Inversion, which relies on a simple set of augmenta- tions and does not require excessive hyper-paraee paree-eeretterr tuning. Under our proposed augmentation-basee based scheme, the same set of augmentation hyper-parameters can used for inverting a wide range of image classification models, regardless input dimensions or the architecture. We illustrate the practicality of our approach by inverting Vision Transformers (ViTs) and Multi-Layer Perceptrons (MLPs) trained the ImageNet dataset, tasks which to the best of our knowledge have not been successfully accomplished by any previous works.
INTRODUCTION
Model inversion is an important tool for visualizing and interpreting behaviors inside neural ar- chitectures, understanding what models have learned, and explaining model behaviors. In general, model inversion seeks inputs that either activate a feature in the network (feature visualization) or yield high output response for a particular class (class inversion) (Olah al., 2017). Model in- version and visualization has been a comerstone of conceptual studies that reveal how networks decompose images into semantic information (Zeiler & Fergus, 2014, Dosovitskiy & Brox 2016).
Over time, inversion methods have shifted from solving conceptual problems solving practical ones. Saliency maps, for example, are image-specific model visualizations that reveal the inputs that most strong influence model's 's decisions (Simonyan et al. 2014).
Recent advances in network architecture have posed major challenges for existing model inversion schemes. Convolutional Neural Networks (CNN) have long been the de-facto approach for com- puter vision tasks, and are the focus of nearly all research in the model inversion field. Recently, other architectures have emerged that achieve results competitive with CNNs. These include Vision Transformers (ViTs; Dosovitskiy et al., 202Th, which are based on self-attention layers, and MLP- Mixer (Tolstikhin et al. (021) and ResMLP (Touvron et al. 2021a), which based on Multi Layer Perceptron layers. Unfortunately, most existing model inversion methods either cannot be applied to these architectures, or are known to fail. For example, the feature regularizer used DeepInversion Yin et al. 2020) cannot be applied to ViTs or MLP-based models because they do not include Batch Normalization layers (loffe & Szegedy 2015p.
In this work, we focus on class inversion, the goal of which is to find interpretable images that maximize the score a classification model assigns to a chosen label without knowledge about the model's 's training data. Class inversion has been used for variety of tasks including model interpre- tation (Mordvintsev et al. 2015), image synthesis (Santurkar et al. 2019). and data-free knowledge transfer (Yin et al.) 2020 However, current inversion methods have several key drawbacks. The quality of generated images is often highly sensitive to the weights assigned to regularization terms, need to be carefully calibrated for each individual network. In addition,
so these methods requiring batch norm parameters are not applicable to emerging architectures.
To overcome these limitations, present Plug-In In Inversion (PII), augmentation-be based approach to class inversion. PIl does not require any explicit regularization, which eliminates the need to tune regularizer-speciffe hyper-parameters for each model image instance. We show that PIl is able to

---

Under review as a conference paper at ICLR 2022
Fringilla (10)
Tibetan terrier (200)
Volcano (980)
Alp (970)
Figure I: Inverted images from a robust-ResNet50 model trained ImageNet-Ik:
invert CNNs, ViTs, and MLP networks using the same architecture aggnosst method, and with the same architectur--annotic hyper-parameters.
1.1 CONTRIBUTIONS
summarize our contributions as follows:
We provide detailed analysis of various augmentations and how they affect the quality of images produced via class inversion.
We introduce Plug-In Inversion (PIT), new class inversion technique based on these aug- mentations, and compare it to existing techniques.
We apply PI/ to dozens of different pre-trained models of varying architecture, justifying our claim that it can be 'plugged in' to most networks without modification.
In particular, we show that PI/ succeeds inverting ViTs and large MLP-based architec- tures, which our knowledge has not previously been accomplished.
Finally, we explore the potential for combining PI/ with prior methods.
In section we review existing techniques class inversion and outline the types of architectures we consider. In section 5] we explore individually the different augmentations that we use, and then describe the full PII algorithm. The remainder is devoted to results and analysis.
BACKGROUND
2.1 CLASS INVERSION
In the basic procedure for class inversion, we begin with pre-trained model and chosen target class y. We randomly initialize (and optionally pre-process) an image x in the input space of f. We then perform gradient descent to solve the following optimization problem for a chosen objective function
f arg min (x[(x,,),
x
and the result is class image f. For very shallow networks and small datasets, letting be cross- entropy even the negative confidence assigned to the true class can produce recognizable images with minimal pre-processing (Fredrikson et al. 2015). Modern deep neural networks, however, cannot be inverted as easily.
2.2 REGULARIZATION
Most prior work on class inversion for deep networks has focused on carefully designing the ob- jective function to produce quality images. This entails combining divergence term (e.g. cross- entropy) with one or more regularization terms (image priors) meant to guide the optimization to- wards image with 'natural" characteristics. DeepDream (Mordvintsev al. 2015). following
2

---

Under review as a conference paper at ICLR 2022
work feature inversion (aahendran & Vedaldi 2015p, uses two such terms: Re₂ (x) ||x|12 which penalizes the magnitude of the image vector, and total variation, defined a
RTv(x)= = Σ i
(TitI - I2s)? -
Σ (zist) - xin)
+
+
Σ
(1411541 - - 1
+
Σ (Ieti - x1
which penalizes sharp changes over small distances. DeepInversion these regularizers, along with the feature regularizer
Yin et al. 2020 uses both of
Reeat (x)
Σ (I]xx(x) -uull2 (loî(x) dfll2),
where (Ak; o. are the batch mean and variance the features output by the k-th convolutional layer, and jig, 0g are corresponding Batch Normalization statistics stored in the model (Toffe & Szegedy 2013). Naturally, this method is only applicable to models that use Batch Normalization, which leaves out ViTs, MLPs, and even some CNNs. Furthermore, the optimal weights for each regularizer in the objective function vary wildly depending architecture and training set, which presents barrier to easily applying such methods a wide array of networks.
2.3 ARCHITECTURES FOR VISION
We now present a brief overview of the three basic types of vision architectures that we will consider.
Convolutional Neural Networks (CNNs) have long been the standard in deep learning for computer vision (LeCun al. 1989, Krizhevsky et al. 2012). Convolutional layers encourage a model to learn properties desirable for vision tasks, such translation invariance. Numerous CNN models exist, mainly differing in the number, size, and arrangement of convolutional blocks and whether they include residual connections, Batch Normalization, other modifications (He al., 2016
Zagoruyko Komodakis 2016 Simonyan & Zisserman 2014).
Dosovitskiy et al. 2021] recently introduced Vision Transformers (ViTs), adapting the transformer
architectures commonly used in Natural Language Processing (Vaswani et al. (017)). These models break input images into patches, combine them with positional embeddings. and these input tokens to self-attention modules. Others have proposed variants which require less training data
Touvron al.) 2021c), have convolutional inductive biases (d' As'oli et al. 2021). make other
modifications to the attention modules (Chu et al., 2021 Liu et al. 2021b Xu et al. 1021).
Subsequently, a number of authors have proposed vision models which are based solely on Multi- Layer Perceptrons (MLPs), using insights from ViTs (Tolstikhin et al. 202 Touvron et al. 202 la Liu al. 2021a). Generally, these models use patch embeddings sumilar ViTs and alternate channel- and watch-wise linear embeddings, along with non con-eiieassties and normalization.
We emphasize that as the latter architecture types are recent developments, our work is the first to study them in the context of model inversion.
PLUG-IN INVERSION
Prior work on class inversion uses augmentations like jitter, which randomly shifts image hori- zontally and vertically, and horizontal flips to improve the quality of inverted images (Mordvintsev al. 2015 Yin et al. 2020). The hypothesis behind their use is that different views of the same image should result in similar scores for the target class. These augmentations are applied to the input before feeding to the network, and different augmentations are used for each gradient step
"Tiis is the formulation used by Yin et al. 2020 others are also common, such as the simpler version in Mahendran Vedaldi 2015).
3

---

Under review as a conference paper at ICLR 2022
used to reconstruct In this section, explore additional augmentations that benefit inversion before describing how we combine them to form the P!l algorithm.
As robust models typically easier to invert than naturally trained models (Santurkar et al. 2019 Mejia et al. 2019), we use a robust ResNet-50 (He et al. 2016) model trained on the ImageNet (Deng al. 2009) dataset throughout this section tions impact inversion. Note, we perform the demonstrations in this section under slightly different conditions and with different models than those ultimately used for PII in order highlight the ef- fects of the augmentations as clearly as possible. The reader may find thorough experimental details in the appendix, section D
toy example examine how different augmenta-
3.1 RESTRICTING SEARCH SPACE
In this section, consider augmentations improve the spatial qualities of inverted images: Centering and Zoom These are designed based on our hypothesis that restricting the input opti- mization space encourages better placement of recognizable features. Both methods start with small input patches, and each gradually increases this space in different ways to reach the intended input size. doing so, they force the inversion algorithm to place important semantic content in the center of the image.
Centering Let x be the input image being optimized. At first, we only optimize a patch at the center of x. After a fixed number of iterations, we increase the patch size outward by padding with random noise, repeating this until the patch reaches the full input size. Figure 2 shows the state of the image prior each stage of this process, as well as an image produced without centering. Without centering, the shift invariance of the networks allows most semantic content to scatter to the image edges. With centering, results remain coherent.
w/ Centering
w/o Centering Final
Init
Step 1
Step
Step
Step 4 Step 5 Step 6 Step 7
Final
Figure 2: An image at different stages of optimization with centering (left), and an image inverted without centering (right), for the Border Terrier class of robust ResNet-50.
Zoom For zoom, we begin with an image x of lower resolution than the desired result. In each step, we optimize this image for fixed number of iterations and then up-sample the result, repeating until we reach the full resolution. Figure 3 shows the state an image at each step of the zoom procedure, along with image produced without zoom. The latter image splits the object of interest at its edges. By contrast, zoom appears to find meaningful structure for the image in the early steps and refines details like texture as the resolution increases.
We note that zoom is not an entirely novel idea in inversion, Yin et al. (2020) use a similar technique 'warm-up' for better performance and speed-up. However, observe that continuing zoom throughout optimization contributes to the overall success of PII.
Zoom Centering Unsurprisingly, we have found that applying zoom and centering simultane- ously yields even better results than applying either individually, since each one provides different benefit. Centering places detailed and important features (e.g. the dog's 's eye in Figure 2 near the center and builds the rest of the image around the existing patch. Zoom helps enforce sound large-scale structure for the image and fills details later.
The combined Zoom and Centering process proceeds in 'stages', each at higher resolution than the last. Each stage begins with an image patch generated by the previous stage, which approximately minimizes the inversion loss. The patch is then up-sampled to resolution halfway between the

---

Under review as a conference paper at ICLR 2022
w/ Zoom
w/o Zoom
Init
Step
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Final
Final
Figure 3: An image during different stages of optimization with zoom (left), and an image inverted without zoom (right), for the Jay class of robust ResNet-50.
previous stage and current stage resolution, allowing to fill the center of the image, leaving border which is padded with random noise. Then next round of optimization then begins starting from this newly processed image.
3.2 COLORSHIFT AUGMENTATION
The colors of the illustrative images we have shown so far are notably different from what one might expect in natural image. This is due to ColorShift, augmentation that we present.
ColorShift is an adjustment of an image's colors by random mean and variance each channel. This can be formulated follows:
ColorShift(x) ox p.
where and a are C-dimensionaal vectors drawn from Ld(-a, a) and ( respectively, and are repeatedly redrawn after fixed number of iterations. We use a 3 1.0 in all demonstrations unless otherwise noted. At first glance, this deliberate shift away from the distribution of natural images seems counterproductive to the goal of producing a recognizable image. However, our re- sults show that using ColorShift noticeably increases the amount of visual information captured by inverted images and also obviates need for hard-to-tune regularizers to stabilize optimization.
We visualize the stabilizing effect of ColorShift in Figure In this experiment, we invert the model by minimizing the sum of cross entropy and total-variation (TV) penalty. Without ColorShift, the quality of images is highly dependent produce noisy images, while larger values produce blurry ones. Inversion with ColorShift, on the other hand, is insensitive this value and in fact succeeds when omitting the regularizer altogether.
the weight λrv of the TV regularizer; smaller values
(Ne))
8
7
6
w/ CS
w/o CS
Figure 4: Inversions of the robust ResNet-50 ATM class, with and without ColorShift and with
varying TV regularization strength. The inversion process with ColorShift is robust to changes in the Ato hyper-parameter while without it, de seems to present a trade-off between noise and blur.
²C' being the number of channels

---

Under review as a conference paper at ICLR 2022
e
e 16
32
64
Figure 5: Effect of ensemble size in the quality of inverted images for the Tugboat class of robust ResNet-50.
Other preliminary experiments show that ColorShift similarly removes the need for E or feature regularization, as our main results for PIl will show. We conjecture that by forcing unnatural col- ors into an image, ColorShift requires the optimization to find a solution which contains meaningful semantic information, rather than photo-realistic colors, order achieve high class score. Alter- natively, as seen in Figure y images optimized with an image prior may achieve high scores despite a lack of semantic information merely by finding sufficiently natural colors and textures.
3.3 ENSEMBLING
Ensembling is established tool often used in dataset security (Souri et al. 2021 enhanced infer- ence (ppiz & Maclin 1999). Similarly, we find that optimizing an ensemble composed of different ColorShifts of the same image simultaneously improves the performance of inversion methods. To this end, we minimize the average of cross-entropy losses (xf(xx,, y). where the x; are different ColorShifts of the image the current step of optimization. Figure S shows the result of applying ensembling alongside ColorShift. We observe that larger ensembles appear to give slight improve- ments, but even ensembles size 1 or two produce satisfactory results. This important for models like ViTs, where available GPU memory constrains the possible size of this ensemble; in general, we use the largest ensemble size (up particular model. More results on the effect of ensemble size can be found in Figure [4] We show the effect of ensembling using other well-known augmentations and compare them ColorShift in Appendix Section A.5 We empirically show that ColorShift is the strongest among augmentations we tried for model inversion.
maximum of = 32) that our hardware permits for
3.4 THE PLUG-IN INVERSION METHOD
We combine the jitter, ensembling. ColorShift, centering, and zoom techniques, and name the sult Plug-In Inversion, which references the ability to 'plug in' any differentiable model, including ViTs and MLPs, using single fixed of hyper-parameters. In the next section, we detail the experimental method that we used to find these hyper-parameters, after which we present our main results.
4 EXPERIMENTAL SETUP
In order to tune poper-paraaetert of PIl for use on naturally-trained models, we use the torchvision (Paszee et al., 2019) ImageNet-trainedd ResNet-50 model. We apply centering zoom simultaneously in 7 'stages.' : During each stage, we optimize the selected patch for 400 itera-
tions, applying random jitter and ColorShift at each step. We use the Adam (Kingma & Ba 2014) optimizer with momentum 8m (0.5, 0.99), initial learning rate b = 0.01, and cosine-decay. At the beginning of every stage, the learning rate and optimizer are re-initialized. We use a = B 1.0
for the ColorShift parameters, and an ensemble size of 32. We ablate these choices in Figure
to validate our selected settings. Further ablation studies can be found in the appendix in figures T0
13 and 14

---

Under review as a conference paper at ICLR 2022
Centering
Not Centering Final
Init
Step
Step 2
Step 3
Step 4
Step 5
Step
Final
e=
e=2
e= 4
16
32
64
Z
Z+C
None
Z Z+C
C
None
w/ ColorShift
w/o ColorShift
Figure 6: Ablation studies repeating the previous experiments for naturally-traine ResNet-50. (top) Inversion for target class Fly with and without Centering. (center) Effect ensembling size in the quality of inverted images for the Tench class. (bottom) The effect of various combinations of zoom, Centering, and ColorShift when inverting the Dipper class.
5 RESULTS
5.1 PII WORKS ON A RANGE OF ARCIITECCTRESS
We present the results of applying Plug-In Inversion different types of models. We again emphasize that we use identical settings for the PIl parameters in all cases.
Figure 7 depicts images produced by inverting the Volcano class for variety of architectures, in- cluding examples of CNNs, ViTs, and MLPs. While the quality of images varies somewhat between networks, all of them include distinguishable and well-placed visual information. Many more ex- amples are found in Figure 17 of the Appendix.
In Figure g we show images produced by PIl from representatives of each main type of architecture for few arbitrary classes. We note the distinct visual styles that appear in each row, which supports the perspective model inversion as tool for understanding what kind of information different networks learn during training.
5.2 COMBINING PI/ WITH DEEPINVERSION
We now consider the relative benefits of Plug-In Inversion and DeepInversion YYn et al., 2020 state-of-the-ar class inversion method for CNNs. Figure shows images from few arbitrary classes produced by P// and DeepInversion (along with reference examples from the robust model demonstration in section g We additionally show images produced by DeepInversion using the output of PII, rather than random noise, its initialization.
Using either initialization, DeepInversion clearly produces images with natural-looking colors and textures, which PII of course does not. However, DeepInversion alone results in some images that either do not clearly correspond to the target class or are semantically confusing. By comparison, PIl again produces images with strong spatial and semantic qualities. Interestingly, these qualities appear to be largely retained when applying DeepInversion after PII, but with the color and texture improvements that image priors afford (Mahendran & Vedaldi 2015

---

Under review as a conference paper at ICLR 2022
MobileNet-v2
ResNet-
ResNet-101
W- ResNet- 101-2
ShuffleNet-v2
VGG16-bn
ViT B-32
DeiT P16 224
Deit Dist P16 384
ConViT tiny
Mixer 616 224
PiT Dist 224
ResMIP 36 Dist
Swin P4 W12
Twin PCPVT
Figure 7: Images inverted from the Volcano class for various Convolutional, Transformer, and MLP- based networks using PII. For more details about networks, refer to Appendix
CONCLUSION
We studied the effect of various augmentations the quality of class-inverted images and intro- duced Plug-In Inversion, which uses these augmentations in tandem. We showed that this technique produces intelligible images from wide range of well-studied architectures, well as the recently introduced ViTs and MLPs, without a need for model specific hyper-parameter tuning. We believe that augmentation baaed based model inversion is promising direction for future research in understand- ing computer vision models.
7 ETHICAL CONSIDERATIONS
We propose Plug-In Inversion as a class inversion technique for the purpose of understanding vi-
sion models. However, we note that prior work has considered the potential of model inversion to
compromise the security of a model's training data (Fredrikson et al. 2015 Yin et al. 2020). These
areas of progress and other data privacy concerns (/hu & Han 2020 Geiping al. 2020 make clear the need for caution when sensitive data is used to train deep learning models.
REPRODUCIBILITY
All the models (including pre-trained weights) we consider in this work are publicly available from widely-used sources. Explicit details of model resources can be found section B the ap- pendix. We also make the code used for all demonstrations and experiments in this work available wttttts:///// com/ http:///w/ooooooooooooeeeeee/e/e
8

---

Under review as a conference paper at ICLR 2022
Garbage Truck
Ocean Liner
CRT
Barn
Goblet
Warplane
Screen
ResNet-110 101
V-3
DeiT Dist
ResMLP
Figure 8: Inverting different model and class combinations for different classes using PII.
Gown
Microphone
Mobile Home
Schooner
Cardoon
Volcano
PII
PIl + Deepinv
DeepInv
Figure 9; PII Inversion results for naturally-trained ResNet-50. PII be used initialization for other methods such as DeepInversion.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 04297, 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. arXiv preprint arXiv:2104 13840, 1(2):3, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint 0rXiv:2103.10697 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pailern recognition, pp. 4829- 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC confer- ence on computer and communications security, pp. 1322-1333, 2015. 2015.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how casy is it to break privacy in federated learning? arXiv preprint arXiv:2003. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint a0X.v: /6302, 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4747 4708, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprini urXiv: 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1v:1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
10

---

David Opitz and Richard Maclin. Popular ensemble methods: An empirical study. Journal of Under review as a conference paper at ICLR 2022
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard. Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint 200.:02005.08000, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021b.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp- 116-131, 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Felipe A Mejia, Paul Gamble, Zigfried Hampel-Arias, Michael Lomnitz, Nina Lopatina, Lucas Tindall, and Maria Alejandra Barrios. Robust or private? adversarial training makes models more vulnerable privacy attacks. arXiv preprint arXiv:I906.000 06449, 2019.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. 2015.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 1/ https://istill..ubb0////// 7/att///////aauurrssuuuaiaiiiii
artificial intelligence research, 11:169-198, 1999.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 6026-8037, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29: 2234-2242. 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. Advances in Neural Information Pro- cessing Systems, 32:1262-1273, 2019.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1556 2014.
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
Hossein Souri, Micah Goldblum, Liam Fowl, Rama Chellappa, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. arXiv preprint arXiv:2/066 009707 2021.
11

---

Under review as a conference paper at ICLR 2022
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-28222 2019.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training, 2021a.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021c
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ross Wightman. Pytorch image models. 2019.
https://githab..com htm. wwt..//arriiiiatar.........
gyttorch-imagemmod
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. arXiv preprint arXiv:2104.06639, 2021.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715 8715-8724, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1el::06000044440 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 318-833. Springer, 2014.
Ligeng Zhu and Song Han. Deep leakage from gradients. In Federated learning, pp. 17-31. Springer, 2020.
12


Reviews:
[{"review_id": "PYotF3PEX1Y", "reviewer": "Reviewer_SYwW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n* The authors aim for a model-agnostic and hyperparameter-robust method. This is an important direction.\n* To verify the model-agnostic features, the authors tested the proposed method on various CNNs (MobileNet, ResNet, etc.), Vision Transformers (ViT, DeiT, etc.), and MLPs.\n\n# Weaknesses\n\n* According to Section 7, the purpose of this paper is to understand vision models. I agree with one of the goals of model inversion is so, but the paper lacks a discussion of how PII helps us to understand vision models, even though the authors conducted experiments on various image recognition models.\n* The analysis is limited to qualitative ones. I would include quantitative results to compare PII with other methods. For example, DeepInversion [Yin+20] reported classification accuracy and the Inception score (IS). Santurkar+19 also shows IS.\n* The authors claimed that DeepInversion needs statistics $\\hat{\\mu}_k, \\hat{\\sigma}^2_k$ ported from batch normalization (BN) and thus is not applicable to models sans BN, such as ViT. However, I think that $\\hat{\\mu}_k, \\hat{\\sigma}^2_k$ are statistics of training data and can be obtained even for models sans BN only by passing training data once and storing the activations.\n* PII is only verified on the ImageNet dataset. I recommend the authors to use CIFAR-10 as Yin+20.\n\n# Comments\n\n* References are not well organized and written in a consistent format. For example, Santurkar+19 is accepted at NeurIPS 2019, but it is not written so. \n* An image is denoted as $x$ in p. 4, but $\\mathbf{x}$ in P5. The notation should be consistent, otherwise, it is difficult to follow.\n* Why a total-variation loss is used to visualize Figure 4, while, if I understand correctly, PII does not use the loss term?\n* Without a quantitative measure, how the hyperparameters of PII can be tuned in Section 4?\n* Section 3.3 says $e=8$ results in images of acceptable quality, while in the main experiment, $e$ is set to 32. If it is intended, it would be justified.\n* What are \"random augmentations\" in p. 6 Section 4?", "summary_of_the_paper": "This paper proposes Plug-In Inversion (PII), which can invert a trained image classifier so that it generates a class-conditional image.\nIn addition to CNNs that previous work has considered, PII can handle Vision Transformers and Vision MLPs as well.\nPII starts generating from the center of a random initialization with lower resolution. Then, the method gradually broadens the generating area (centering) and increases the resolution (zooming). During generation, colors are randomly shifted (ColorShift) and the averaged cross-entropy to $e$ color-shifted images are used.\n\n", "summary_of_the_review": "The model aims for a model-agnostic and hyperparameter-robust method to investigate vision models by model inversion. I highly appreciate this goal. Meanwhile, the current manuscript has several weaknesses. Therefore, I recommend this paper be marginally below the threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NOQ4ee3nv_L", "reviewer": "Reviewer_dsJC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Highlights: \nThe problem addressed in the paper is well written and well described in the Introduction and Background section.\n\n# Related work: \nThere should be a separate Related work section citing the work that has been done in this area previously with a thorough literature review. \n\n\n# Methods: \n1.\tThe paper mentioned several drawbacks in the Introduction section of the existing methods - the authors should put any of such examples like the generated images being highly sensitive to the weights assigned to the regularized terms, etc, and compare that with the images generated by PII to show how PII is working better.\n2.\tThe authors should have provided some theoretical justifications of their work -  why are the new sets of augmentation working better than its predecessor?\n\n# Experiments:\n\t\n1.\tOne of the main issues in this paper is the applicability of the solution of the problem is not discussed in the paper. For example, there are three major applications where Deep Inversion(Yin et al., 2020) can be used - (a) pruning, (b) knowledge transfer, and (c) continual (class incremental) learning. If the authors set Deep Inversion as the baseline, they should have formulated the experiments for these three applications and compared PII with all architectures like VIT and MLP and others against Deep Inversion. Also if there are any other applications of the solution, that should have been clearly mentioned in the paper. \n\n2.\tThe authors experimented only using the Imagenet dataset. They should have done against other datasets like CIFAR 10 - the way Deep Inversion was doing.\n\n3.\tTo prove such a simple method is working better than the previous models, the authors should have either used or introduced some metric that gives a quantitative comparison between all the inversion techniques. In this paper, the authors compared PII with different settings and against Deep Inversion qualitatively. For example, in Deep Inversion, the authors did a wide range of quantitative comparisons among various metrics like inception score, knowledge transfer results, continual learning results (table 1-7 Deep Inversion (Yin et al., 2020))\n\n# Minor:\n1.\tSection 2.3, 2nd paragraph where Vision transformers are discussed - in line #2, there is a mistake \\cite{Vaswani}\n2.\tSame for the last line in the same paragraph \\cite{CoaT and CaiT}\n", "summary_of_the_paper": "This paper attempts to solve the problem of image inversion by introducing three augmentation-based techniques. It specifically tries to solve the problem of class inversion, which generates an interpretable image from a neural network by sending the pre-initialized image to the network. Then optimization is wrt the input image instead of the weights of the network. It introduced new sets of augmentation-based techniques like centering, zooming, color shift augmentation, ensembling that have not been tried before, and finally, they combined them with VIT and MLP based vision models. They validated all the augmentations using Vision transformer and MLP based vision models and compared them against the exiting method Deep Inversion(Yin et al., 2020). The paper is well written but lack quantitative evaluation which is a very significant shortcoming. ", "summary_of_the_review": "* Since the baseline for this paper is Deep Inversion lack and there are many quantitative experiments in that paper, the lack of quantitative results in this paper is not justifiable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EgleSTnFGEE", "reviewer": "Reviewer_MNJX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a technique called Plug-In Inversion, which is to be used as-is, irrespective of the underlying model which is to be inverted. This method works by applying a particular sequence of augmentations to the image, and then performing inversion by minimizing the loss of this image with respect to a single class.\n\nStrengths:\n- The motivation behind the paper is clearly presented - the goal is to reduce the need for extensive hyperparameter optimization of an inversion system. This is an interesting motivation, given that extensive tuning of regularizers is often difficult to perform.\n- The paper is in general clear and easy to follow. The main ideas are presented in a concise manner and formulate a solid story for the paper.\n- The authors demonstrate the effects of each different augmentation used in their inversion system, both in the main paper and the appendix. This is particularly useful both for understanding how Plug-In Inversion works and how each part of the procedure affects the resulting image (namely, how the zoom + centering procedure operates, as well as the effect of the ColorShift augmentation).\n- The proposed method is evaluated on a great variety of models, and produces intelligible images for the classes presented (in the sense that they conform to the high-level notion related to the class).\n\nWeaknesses:\n- The proposed Plug-In inversion method is introduced very late in the paper (at the end of Section 3.4) even though it consists only of combining the augmentation and search space restriction techniques provided in the previous sections (3.1 - 3.3). It would have been much clearer for the reader if this fact was fully described earlier in Section 3 (for example, by moving Section 3.4 earlier in the paper), instead of delaying the presentation of the full method.\n- My greatest concern is the fact that I am not sure whether the claim of the authors that the method can be applied to multiple networks without tuning is fully supported by the experimental results. More specifically, in Figures 7 and 8 the method is applied in a variety of networks, resulting in images which, while intelligible, are of varying quality. As such, to fully support the argument of the lack of need of extensive hyperparameter tuning, one would also need to show that using the same regularizer (for example, TV, which can be applied to any model) with the same hyperparameter leads to more extensive degradation, when applied to different models. I believe that a small example to demonstrate this motivation would greatly improve the paper.\n- The above point is made even more unclear by the fact that one of the models considered by the authors does use a TV regularizer (which leads to drastic improvement in image quality).\n- The ColorShift augmentation proposed by the authors is, unless I am mistaken, a variant of color jittering (in the sense that the adjustment is made directly to the pixels of the image, rather than on hue, saturation, contrast etc.). Given that applying some form of adjustment to the color of the image is a form of data augmentation which has been used in prior work  (Krizhevsky et al., 2012, section 4.1), I am not sure if this data augmentation is as novel as the authors claim (although I do appreciate the qualitative analysis of its effects in the context of inversion).\n\nQuestions:\n-\tThe hyperparameters for ColorShift are fixed to $a=b=1$, and if I understand correctly this comes from a qualitative analysis of the results in a few simple experiments. Is this correct?\n\nMinor comments/typos:\n-\tAs a minor comment related to the above, I believe the authors should indicate in the captions of the figures which model the respective images come from.\n-\tThere is a space missing in the second-to-last line of page 6.\n-\tThe caption in Figure 10 in the appendix is incomplete.\n\nReferences:\nKrizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 25, 1097-1105.\n", "summary_of_the_paper": "This paper proposes a method to perform class inversion on image data, called Plug-In Inversion. This method consists of a sequence of augmentations on image data and is designed to be applicable to a variety of architectures. This method is evaluated on ImageNet trained models and is compared with other techniques used for the same goal.", "summary_of_the_review": "Overall, this work has an interesting motivation (alleviating the need for extensive tuning of regularizers for class inversion), but I am not certain if the benefits of Plug-In Inversion, as described by the authors, are fully supported by the experimental evidence provided. As such, given also the fact that the individual components of the method do not seem novel by themselves, I marginally lean towards rejecting this paper, pending extra discussion during the rebuttal process.\n\n**Update after rebuttal**: See response to the authors' comments below.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2zKVhvKLiqN", "reviewer": "Reviewer_vGig", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n- The proposed method works for class inversion for any architecture, including Vision Transformers and MLPs.\n- The proposed method is robust across architectures with little-to-no hyperparameter tuning\n\nWeak points of the paper:\n- The specific method proposed is not clearly presented at a mathematical level (it is described only in words)\n- The only baseline comparison I saw was relative to one method (DeepInversion).  \n\nRequested response from authors that will affect my recommendation:\n\nThe authors should justify why they believe their baseline comparisons are sufficient (or comment on what additional baselines they would present in a camera ready). \n\nThe paper does not comment on methods for adversarial examples, such as adversarial patches, either as related work or as baselines.  In the rebuttal, please justify why these comparisons are not present, or comment about them.\n\nThe authors should clarify what exactly the PII algorithm is at a mathematical level (write out the GD iteration) so that I am clear I am not misunderstanding the model.\n\nAdditional feedback with the aim to improve the paper:\n\nIf my reading is correct, the method treats the classifier as a black box, which is why it works for any architecture.  If this is true, then the paper would be much improved by clearly stating it treats the classifier as a black box.  In its current form, it comes across as being restricted to neural networks, which I do not think is the case.\n\nTypos:\nThe minimization problem on page 2 should say argmin_x L(f(x), y).  It has an incorrect parenthesis.\n\nThere are several places where the document says \"cite XYZ\".  Replace these with the citations.\n\nFigure 2 caption: \"An image a different stages\" has a grammatical issue.\n", "summary_of_the_paper": "In this paper, the authors introduce a novel method for the class inversion problem: given a trained classifier and a specific class/label, generate an example image of that class.  This method is called Plug-In Inversion (PII) and it works as follows: a gradient descent is performed to find the image x such that the output of the classifier is close to the target label under a suitable loss function.  In PII, each successive gradient update is computed using a data-augmented version of the current iterate of the image x.  The authors introduce new data augmentations and combinations thereof.  The benefits of this inversion method is that it works for any classification architecture (which is treated only as a black box), and the same hyper parameter values (e.g. amount of color shift) work across a range of significantly different classifier architectures.  The authors present the output of PII across 12 different networks (including Convolutional, Transformer, and MLP nets), and they demonstrate several examples where their method is more interpretable than DeepInversion, a state-of-the-art method for class inversion.  \n", "summary_of_the_review": "Clearly state your recommendation with 1-2 reasons\n\nI rate this paper as a weak accept for the following reasons.\n\nThe reason to accept is: They provide a class inversion method that produces interpretable visualizations for recent architectures (Vision Transformers + MLPs) that existing methods do not work for.  This could accelerate the development of better classifiers by helping researchers interpret and fix their behavior.\n\nThe reason this isn't rated higher is: The baseline comparison was inadequate.  The authors should compare their visualization method on Transformers/MLPs to the best existing approach that exists before this work.  Perhaps that approach is a plain inversion method.  Perhaps there are methods that can be borrowed from the methods of targeted adversarial examples.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for solving the problem of inverting image classifier models. The authors introduce three new augmentation-based techniques to do this. The techniques are validated using Vision Transformer and MLP models and compared against previous methods. The reviewers appreciate the problem that the paper aims to solve. However, the reviewers are not satisfied with the presentation and evaluation of the proposed approach. The main contribution of the paper is not presented clearly enough, according to the reviewers, and it remains unclear to them what aspect of model inversion the authors most want to improve on, and whether their proposed technique indeed achieves such an improvement. In their response, the authors do provide Inception scores that show that their inversion method improves the perceptual quality of generated images compared to previous approaches. The reviewers acknowledge the author response, but indicate that it does not fully resolve their concerns. I recommend that the authors update their paper to more clearly present their main contributions and conclusions, and to provide a more thorough comparison against previous methods, before submitting to another conference.
Decision: Reject