When less is more: Simplifying inputs aids neural network understanding
Authors: Robin Tibor Schirrmeister
Rosanne Liu
Sara Hooker
Tonio Ball
Venue: ICLR 2022 Submitted (2022)
Keywords: interpretability
compression
network training

Abstract: Are all bits useful? In this work, we propose SimpleBits, a method to synthesize simplified inputs by reducing information content, and carefully measure the effect of such simplification on learning. Crucially, SimpleBits does not require any domain-specific knowledge to constrain which input features should be removed. Instead, SimpleBits learns to remove the features of inputs which are least relevant for a given task. Concretely, we jointly optimize for input simplification by reducing inputs' bits per dimension as given by a pretrained generative model, as well as for the classification performance. We apply the simplification approach to a wide range of scenarios: conventional training, dataset condensation and post-hoc explanations. In this way, we analyze what simplified inputs tell us about the decisions made by classification networks. We show that our simplification approach successfully removes superfluous information for tasks with injected distractors.  When applied post-hoc, our approach provides intuition into reasons for misclassifications of conventionally trained classifiers. Finally, for dataset condensation, we find that inputs can be simplified with only minimal accuracy degradation. Overall, our learning-based simplification approach offers a valuable new tool to explore the basis of network decisions.

Under review as a conference paper at ICLR 2022
WHEN LESS IS MORE: SIMPLIFYING INPUTS AIDS NEU- RAL NETWORK UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
How do neural network classifiers behave if you make their inputs simpler? In this work, we propose SimpleBits, method to synthesize simplified inputs by reduc- ing information content, and measure the effect of such simplification on learning. To measure simplicity, we use the finding that the encoding bit size given by a pre- trained generative image model correlates well with the visual complexity of the image. Hence, we minimize the encoding bit size to simplify inputs and investi- gate the effect of such simplification narios: conventional training, dataset condensation and post-hoc explanations. all settings, optimize for simplified inputs that still contain the information learn the classification task. We show that SimpleBits successfully removes super- fluous information for tasks with injected distractors and investigate the tradeoff between input simplicity and task performance on real-world datasets. For dataset condensation, we find that inputs can be simplified with only minimal accuracy degradation. Finally, applied post-hoc to normally trained classifiers, SimpleBits provides intuition into reasons for misclassifications.
neural network behavior in several sce-
INTRODUCTION
A better understanding of the information deep networks learn can lead to new scientific discover- ies (Raghu & Schmidt, 2020). inform our understanding of differences between human and model behaviors (Makino et al. 20020) and can serve powerful auditing tools (Geirhos et al. 2020).
Removing information from the input manually one way to understand what information content relevant for learning. For example, researchers have occluded parts of the input removed specific frequency ranges from the input to see which input regions and frequency ranges are relevant for the network's prediction (Zintgraf et al. 2017 Makino al., 2020 Banerjee et al.) (021). These ablation techniques use simple heuristics such as removing at random (Hooker et al. 2019) or exploit domain knowledge about interpretable aspects of the input (input regions, frequency range) to create simpler versions of the input and analyze the network's 's prediction on these simpler inputs.
What if, instead of using heuristics, one would learn how to simplify inputs that contain prediction- relevant information? This way, one could synthesize simpler inputs and gain intuition into model behavior without relying on domain knowledge about what information may be relevant for the network. For this, one needs to define the precise meaning of "simplify an input", including useful metries for the simplification of the inputs and for the retention of task-relevant information.
In this work, we propose SimpleBits an informatioonrrdducti simplified inputs which contain less information but still remain informative for the task. To mea- sure simplicity, we use a finding initially reported as problem for density- anomaly detection generative image models tend to assign higher probability densities and hence lower bits to visu- ally simpler inputs (Kirichenko et al.] 2020; Schirmeister et al. 2020). Here, use this to our advantage and minimize the encoding bit size given by generative network trained on a general image distribution to simplify inputs. At the same time, optimize that the simplified inputs still contain the task-relevant information.
method that learns to synthesize
We then investigate what information is retained in the simplified inputs and how the simplification affects network behavior in a variety of settings. We apply SimpleBits both in a per-instance setting, where each image is processed to be simplified version of itself, and the size of training set remains

---

Under review as a conference paper at ICLR 2022
Per-Instance Data Simplification
Data Simplification with Condensation
during Training
after Training
Original datasel
Original dalasel
Original dataset
Sample batches
Condense
RRC
Trained classifier
SimpleBits
SimpleBits
SitpleBits
Train classificr
Train classifier
Figure We apply SimpleBits to a variety of tasks to aid neural network understanding. As a per-image simplifier, applied during training (left), it investigates the trade-off curve between sim- plification and accuracy. It can also be used as a post-hoc analysis tool after training (center), illuminating features of images that crucial the trained classifier. When combined with data condensation (right), the original data set can be effectively reduced both in size and in complexity.
unchanged, as well as in condensation setting, where the dataset is compressed to only a few samples per class, with the condensed samples simplified at the same time. Applied during training. SimpleBits can be used to investigate the trade-off between the information content and the task performance, After training, SimpleBits can act as an analysis tool understand what information a trained model uses for its decision making, Figure Summarizes tasks covered in this paper.
Our evaluation provides the following insights in the investigated scenarios:
1. Per-instance simplification during training. SimpleBits successfully removes super- fluous information for tasks with injected distractors. On natural image datasets, Sim- pleBits highlights plausible task-relevant information (shape, color, texture). Increasing simplification leads to accuracy decreases and we report the trade-off' between simplifying inputs and task level performance for different datasets.
2. Dataset simplification with condensation. We evaluate SimpleBits applied conden- sation setting that processes the training data into much smaller set of synthetic images. SimpleBits simplifies these images drastically reduce the encoding size without substan- tial task performance decrease. On chest radiograph dataset (Johnson et al. 2019a b), SimpleBits can uncover known radiologic features for plcural effusion and gender.
3. Post-training simplification. For a trained model, SimpleBits provides intuition into the prediction-relevant information in image. For example, by visualizing simplifications of mispredicted images, we can form hypotheses of why these images were mispredicted.
MEASURING AND REDUCING INSTANCE COMPLEXITY
How to define simplicity? We use the fact that generative image models tend to assign lower en- coding bit sizes to visually simpler inputs (Kirchennoo et al. 2020 Schimmeister et al.) (220). Con- cretely, the complexity an image & can quantified as the negative log probability mass given by pretrained generative model with tractable likelihood, G, i.e. (o)). This log probabil-
2

---

Under review as a conference paper at ICLR 2022
bpd
0.05
0.24
1.04
1.63
2.38
3.09
3.60
4.05
6.32
(given by pretrained generative model)
6
Figure 2: Visualization the bits-per-dimensoon (bpd) measure for image complexity, sorted from low to high. Image samples are taken from MNIST, Fashion-MNIST, CIFARIO and CIFAR 100, in addition completely black image sample. bpd calculated from the density produced by Glow (Kingma & Dhariwal 2018) model pretrained on 80 Million Tiny Images.
Side-by-Side MNISTs (ground truth: MNIST)
MNIST with Uniform Noise (ground truth: MNIST)
Original
7
/
0
4
7
2
SimpleBits oulput
7
4
7
2
Interpolated MNIST and CIFAR (ground truth: MNIST)
CIFARIO with Stripes (ground truth: Strips)
7
Original
SimpleBits output
7
2
Figure 3: Evaluation of distractor removal on four composite datasets. Shown are the composite original images and the corresponding simplified images produced by SimpleBits trained alongside the classifier. SimpleBits is able to almost entirely remove task- irrelevant image parts, namely Fash- ionMNIST (top left), random noise (top right), CIFAR1O (bottom left well as bottom right).
ity mass can be interpreted as the image encoding size in bits per dimension (bpd) via Shannon's theorem (Shannon 1948): bpd(x) loga pa)x) where is the dimension of the flattened æ.
The simplification loss for an input æ, given pre-trained generative model G, is as follows:
Liim (x)
log Pc (x)
Figure 2 visualizes images and their corresponding bits-per-dinne per-dimension (bpd) values given by a Glow network [Kingma & Dhariwal] 2018) trained on 80 Million Tiny Images (Torralba et al. 2008) (see supp. sec. [ST for other models). This is the generative network used across all our experiments. A visual inspection of Figure 2 suggests that lower bpd corresponds with simpler inputs, as also noted in prior work (Serrà et al., 2020). The goal of our approach, SimpleBits, is to minimize bpd of input images while preserving relevaelevant information.
We now explore how SimpleBits affects network behavior in a variety of scenarios. In each sce- nario, explain the method to optimize Leim and the retention of task-relevant information, the experimental setup, and results. All code is at httphhtpt::/ette htt///iiyyrrrr// wtw.tnyoooooo.......ooooo simp Sepllees e-bits
PER-INSTANCE SIMPLIFICATION DURING TRAINING
When plugged into the training of classifier f. SimpleBits simplifies each image such that f can still leam the original classification task from the simplified batches. We apply backpropagation through training steps: given a batch of input images Xorig before updating the classifier f. an image-to- image simplifier network generates corresponding batch of images Xsin such that: (a) images in Xsim have low bpd as measured per Lgim in Equation ( and (b) training on the simplified images leads to reduction of the classification loss the original images.
We optimize (b) by unrolling one training step the classifier. So for a single batch X V. we first compute updated classifier I' as follows:
3

---

Under review as a conference paper at ICLR 2022
Accuracy [%)
Results for Instance Simplification on CIFAR10
90
80
70
3.50
3.25
3.00
2.75
2.50
2.25
2.00
1.75
Bits per dimension
Figure 4: Selection of Simplified Training Images on CIFAR10. Simplified images from settings with varying simplification loss weight Asim- We observe that at lower bits per dimension color is retained only for some images such green color for frog blue sky for the plane. In this low bit regime, texture remains discernable for the cat and frog images.
X = simplifier Xorig)
(2)
= train (( v))
(3)
where simplifier is an image-to-image network, is the classification loss function, i.e., the cross- entropy loss, and f' is the classifier f after one optimization step using the classification loss ((((Xsim) y) on the simplified data.
To train the simplifier network, optimize for both (a) (Equation (1) and for the classifica- tion loss on the batch of original images with the updated classifier after the unrolled training step 1(f' 3) using backpropagation through training (Maclaurin et al.) 2015 Finn et al. 2017). Note that it would not be enough to instead optimize pertormance on the simplited images as the simplifier could then to collapse all simplified images of one class to one representative example.
Adding the classification losses on the simplified data both before and after the unrolled training step ensures the training is not influenced by predictions on the simplified data that are very different from the classification target and we found that to improve training stability, leading to:
Lets
y) + ((((X)im) y) ((f((X sim), y)
(4)
Further details to stabilize the training are described in S4 The total loss for the simplifier is
L= Asim L + Lets:
(5)
where Asm is a hyperparameter to control the trade-off between simplification and task completion. In subsequent experiments, we vary Asi flexibly control the extent of simplification.
Implementation Our training architecture is a normalizer-fin classifier architecture to avoid inter- pretation difficulties that may arise from normalization layers, such as image values being down- scaled by the simplifier and then renormalized again. We use Wide Residual Networks and adapt them according to the steps outlined (Brock et al. 2021); additional details are included in the Supp. Section S2 The normalizer-free free architecture reaches 94.0% on CIFAR 10 in our experiments, however we opt for a smaller variant for faster experiments that reaches 91.2%, see Supp. Section S2 For the simplifier network, use UNet architecture (Ronneberger al. 2015) that mod- iyy to be residual, more details see Supp. Section [S3] For both simplifier and classifier networks, use AdamW (Loshchilov & Hutter 2019) with Ir = 5 10-1 and weight decay 10-5
4

---

Under review as a conference paper at ICLR 2022
CIFAR10
SVHN
FASHIONMNIST
MNIST
Accuracy [%)
90
100
End of Joint Training Retr Retrained Simplified
90
80
80
75
80
70
70
60
Finetuned on Simplified
3
1
1.5
1.0
1.0
0.5
Bits per Dimension Bits per Dimension
Bits per Dimension
Bits per Dimension
Figure 5; Results for Training Image Simplifications on real Datasets, Dots show results for training with different loss weights for the simplification loss. Images with less bits per dimension lead to reduced accuracies, this already happens for smaller bpd-reductions for more complex datasets like CIFAR10 than for less complex ones like SVHN.
3.1 SimpleBits REMOVES INJECTED DISTRACTORS
We first evaluate whether our per-instance simplification during training successfully removes perfluous information for tasks with injected distractors. To that end, we construct datasets to con- tain both useful (ground truth) and redundant (distractor) information for task learning. We create four composite datasets derived from three conventional datasets: MNIST (LeCun & Cortes 2010),
FashionMNIST (Xiao al., 2017) and CIFARIO (Krizhevsky) 2009). Their construction, samples and results are described below, and shown in Figure 3
Side-by-Side MNISTS constructs each image by concatenating, left and right, one sample from Fashion-MNIST and another from MNIST. Each sample is rescaled to 16x32, so the concatenated image size remains 32x32; the order of concatenation random. The ground truth target MNIST labels, and therefore FashionMNIST acts like a distractor it is irrelevant for the classification task. As seen in Figure 3
the simplifier effectively removes the clothes side the image.
MNIST with Uniform Noise adds uniform noise to the MNIST digits, preserving the MNIST digit as the classification target. Hence the noise is the distractor and is expected to be removed. And indeed the noise is no longer visible in the simplified outputs shown in Figure 3
Interpolated MNIST and CIFARIO is constructed by interpolating between MNIST and CIFARIO images. MNIST digits the classification target. The expectation is that the simplified images should no longer contain any of the CIFARIO image information. The result shows that most of the CIFARIO background is removed, leaving only slight traces of colors.
CIFARIO with Stripes overlays either horizontal or vertical stripes onto CIFARIO 10 images, with the binary classification label 0 for horizontal and for vertical stripes. With this dataset observe the most drastic and effective information removal, where only the tip of vertical strips is retained, which by itself sufficient solve this binary classification task.
3.2 TRADE-OFF CURVES ON CONVENTIONAL DATASETS
Sec 3.1 evaluates the ability of SimpleBits to remove redundant information where ground truth is known. Now, we evaluate the trade-off between task performance and input simplification on real-world datasets. We perform instance-level simplification on MNIST, Fashion-MNIST, SVHN (Netzer et T01 and CIFARIO. 10, comparing results with varying weight Xsm for the simplification loss, including Asim O, so without using any simplification loss. At an extreme, when we have simplified to the black image seen in Figure 2 we expect to significant erosion of performance. Understanding the trade-offs inbetween the original image and this extreme can help understand how sensitive the training is to the removal of more complex features.
In Figure 5 there is a pronounced decay in performance at high levels of simplify (where final bpd is less than 2). We also observe this decay curve to be more pronounced for relatively more complex datasets such as CIFAR1O, suggesting the classifiers need to be trained with more complex features present for highest task performance. Baselines for our simplifier can be found in S8
5

---

Under review as a conference paper at ICLR 2022
The visible retention of task-relevane information in some images together with the decreased accu- racies suggests another factor: The simplified inputs may lack noise-fcatures that the classifier has to learn to be invariant to. This is analogous to how removing data augmentation can decrease accura- cies even though the augmentations themselves do not contain discriminative information. This view implies SimpleBits still be used to visualize task-relevant information and at the same motivates the adaptation of SimpleBits to analyze regularly trained classifiers after training in Section
4 DATASET SIMPLIFICATION WITH CONDENSATION
Now we investigate how SimpleBits affects training on a small synthetic condensed dataset. Multiple methods have been proposed to achieve dataset condensation (Zhao et al. 2222 Zhao & Bilen 2021
Wang et al., 2018 Maclaurin et al., 2015). via backpropagation through training (Wang al. 2018
Maclaurin et al. 2013), gradient matching (Zhao & Bilen 202I), or kernel based meta- weeaaleerning
(Nguyen et al. 2021). Due to its small size, one can visualize the full condensed dataset to under- stand what information is preserved for learning. Our aim here is to combine SimpleBits with dataset condensation to see we could obtain a both smaller and simpler training dataset than the original.
In this setting, jointly condense our training dataset inputs and simplify the synthetic inputs according to our simplification loss (Equation (T). Con- cretely, add the simplification loss Lyim to the gradient matching loss proposed by Zhao & Bilen (202)). The gradient matching loss computes the layerwise cosine distance between the gradient of the classification loss Xorig and a batch of synthetic images Xsyn
smaller number of synthetic training
the classifier parameters 0 produced by batch of original images
Lmatch (X (orig Xayn) D(Vol(f(Xotil Vel(f(Xaya)) v)).
(6)
where D is the layerwise cosine distance. The matching loss is computed separately per class. Overall, with our simplification loss, we get:
Layal (Xorig Xsyu) Lmatch (oorgg Xaya) +
Σ
llg (Zsyn)
Payn Xxyu
We perform dataset condensation on MNIST, Rashion-MNIST SVHN and CIFARIO with varying Asim for the simplification loss. We also apply dataset condensation to the chest radiograph dataset MIMIC-CXR-JPG (Johnson et al. 2019a b) for predicting pleural effusion and gender. We use the
networks from (Zhao et al. 2021). but use Adam (Kingma & Ba 2015) for optimization.
4.1 SimpleBits RETAINS CONDENSATION PERFORMANCE WHILE GREATLY SIMPLIFYING DATA
In Figure 6 we examine the accuracy for each pundensed-and-simppife. dataset. We observe that for the natural image datasets, accuracies are mostly retained when decreasing the number of bits per image. Note that the setting with highest bpd is a reimplementation of ZZao et (02I)) and therefore baseline without simplification loss. We visualize examples in Figure 6 and observe that the jointly condensed and simplified images look visually smoother, indicating that higher frequency patterns visible in the original images are not needed to reach the same accuracy. These visualiza- tions are also noticeably more smooth than the results for per-instance simplification Figure 3 which suggests that data condensation may already favor features that are less complex.
Evaluation of a medical chest radiograph dataset We also evaluate jointly condensing and sim- plifying for a dataset of chest radiograph images (Johnson et al.|2019a 2019 b). This dataset has known radiologic features for the presence pleural effusion (Jany & Welte 2019 9 Raasch et al. 1982] and difference in gender (Bellemare et al. 2003). In Figure 7| we visualize both the condensed (top row) and the jointly condensed and simplified dataset (bottom row). The overlayed shows that a visible difference between presence of feature. For pleural effusion, a larger white region the bottom of the lung occurs in the simplified pathological image, while for gender, lungs appear slightly smaller for the simplified female image.

---

Under review as a conference paper at ICLR 2022
1 Images per Class
10 Images per Class
100
FashionMNIST
Accuracy
75
CIFAR10
SVHN
50
MNIST
10
Bits per dimension
Bits per dimension
Plane
Bird
Dog
Frog
Ship
28.2%
8.94 bpd
28.5%
4.45 bpd
T-shirt
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankleboot
69.8%
5.84 bpd
68.6%
3.19 bpd
Figure 6: Dataset condensation accuracies (when retraining with the condensed dataset) vs. data simplicity. Top: Each dot represents data condensation experiment run with particular weight for the simplification loss, which results in more or less complex datasets. Accuracies can be retained with substantially reduced bits per dimension. In the --image-per-claan case (top left figure), arrows highlight the settings that are visualized in the bottom figure. Bottom: Condensed datasets with varying simplification loss weight. Each row represents the whole condensed dataset (1 image per class), with high (top row) or low (bottom row) bits per dimension. Lower bits per dimension datasets are visually simpler and smoother while retaining the accuracy.
No Finding
Pleural Effusion Overlayed
Female
Male
Overlayed
73.3%
63.1%
5.87 bpd
6.02 bpd
74.8%
64.4%
1.97 bpd
2.07 bpd
Figure 7: Condensed dataset for pleural effusion and gender prediction from chest radiographs in MIMIC-CXR. Condensed images for the classes look very similar. Color-coded mixed rightmost images reveal the differences between the classes. Green highlighted region the lower end of the lung consistent with typical radiologic features for pleural effusion (white region indicating fluid on lungs), red highlighted around lung for gender indicate smaller lung volume for the female class.
5 POST-TRAINING SIMPLIFICATION
Per-instance SimpleBits reduces accuracies when applied during training, but can it be used to inter- pret trained classifiers with high accuracies after training? A trained classifier's 's prediction may be influenced by lot of information it has seen during training. Here, SimpleBits to visualize some of the information that would help the classifier remember what has learned for that specific prediction.

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Simplification loss function after training
1: given generative network G. input at, simplified input æsim- classifier f. parameter scaling factors & < 1
2: faol <- ScaleParameters([
>. Sccale parameters of classifier down by to simulate "forgetting" Predict original and simplified with unscaled classifier Predict original and simplified with scaled classifier VaDKL (till)aam.eea)) P Compute distance of gradients on scaling factors > Compute prediction differences b Compute needed bits for simplified input
3: h f(()), hsun = (((mmmm)
4: hacd [ heimmsed = (aei
5: Lgrad = (V,DDi(hhl
6: Lped = DKLL (fll) + DICL
PICL(head (Rescd sssmm 7: Tiim log PG (Fsim))
8: return Tignd" + T.yeed
Asam
Tiim
For synthesizing the prediction rellvvant information, we simulate that the classifier forgets knowl- edge and then synthesize a simplified input that allows the classifier to relearn the relevant knowl- edge for the prediction of the original input. To simulate forgetting, we scale down all parameters Φ; 8, < 1. This scaling removes information from the model by bringing the learned parameter values closer to zero, is identical to weight decay and also makes the network simpler in terms of model encoding size (Hinton & van Camp 1993). To synthesize simplified input that helps learning to restore the pa- rameter values that are important for specific input, compare gradients between the original and simplified input.
of the trained classifier f by multiplying them with a gating value @scated,
Given input, we can compute the gradients of the KL-divergence between the rescaled network's s prediction (x) and the original network's prediction f (x)) At each iteration, the layerwise co- sine distance between these two sets of gradients (one for original datapoint and the other for the simplified) is the basis of the loss used in SimpleBits. We compute this distance only considering the gradients that are negative for the original input. Combined with the simplification loss Eq. [ this amounts to asking what input information is needed to recover parts of the original network to restore the original prediction? To ensure that the network trained towards minimizing the same prediction difference on the simplified and original data, we also add prediction difference loss Ippedd Further details about the implementation are included in Alg. [ and Section S5
In Figure 8 we visualize both the misclassified images according to the original network f and produce the corresponding simplified versions. We observe that simplified images may provide some intuition into the reason for misclassification, highlighting a variety of different features for different images. We imagine a possible practitioner workflow, where the practitioner derives a set of possible hypotheses for the misclassification from SitpleBits and tests them on the real data. We show further post-hoc simplified examples in supplementary Section S13
6 RELATED WORK
A different approach to reduce input bits while retaining classification performance is to train com- pressor that only keeps information that is invariant to predefined label-preserving augmentations. Dubois et al. (202T) implement this elegant approach in two ways. In their first variant, by train- ing a VAE to reconstruct an unaugmented input from augmented (c.g. rotated, translated, sheared) versions. In their second variant, building on the CLIP (Radford et al. 202I] model, they take image-text pairs and view all images with the same text as augmented versions of each other. This allows to use compressed CLIP encodings for classification and achieves up to 1000x compression on Imagenet without decreasing classification accuracy. Their approach focuses on achieving maxi- mum compression while our approach is focused on interpretability. Their approach requires access to predefined label-preserving augmentations and has reduced classification performance in input space compared to latent/encoding space.
Our approach simplifying individual training images builds on Raghu et al. (2021). where they learn to inject information into the classifier training. Per-instance simplification during training can be seen instance of their framework combined with the idea of input simplification. In difference to their methods, SimpleBits explicitly aims for interpretability through input simplification.
Other interpretability approaches that synthesize inputs include generating counterfactual inputs (Hvilshøj et al.) [02T] Dombrowski et al.) 202T] Goyal et al., 2019) or inputs with exaggerated teatures (Singla et al. 2020). SimpleBits differs in explicitly optimizing the inputs to be simpler.
8

---

Under review as a conference paper at ICLR 2022
class
Ship
Horse
Bird
Deer
Original Image
Predicted as
Dug
Frog
Frgg
SimpleBits output
Hypothesis for one cause misclassification
Time mip revembles antior
Black pads pan. color bottow
High froquency tenture
Circle resembles shape car
ship body
Green color
Control Image (profictiad commaily)
Expected
Gradients on Original Image
Prediction on Original Imagee
Figure 8: Post-hoc simplifications misclassified CIFAR-10 examples. For each, simplified image reveals plausible causes for the misclassification. We subsequently made alterations to compensate for the cause (from left to right: removing black dots, removing tree top, removing color, removing high frequency texture, removing circle, and removing color), and able to revert the predictions to the true class. We also show color-coded saliency maps for expected gradients (Erion et al.) 2021) and prediction difference (Zintgraf et al. 2017) for comparison (red: evidence for and blue: evidence against the predicted class). SimpleBits reveals more information than saliency methods.
Generative models have often been used various ways for interpretability such as generat- ing realistic-looking inputs (Montavon et al. 2018) and by directly training generative classifiers (Hvilshøj et al. 2021; Dombrowski al.) (222)) but we are not aware of any work except (Dubois) et al. 2021) (discussed above) explicitly generate simpler inputs.
CONCLUSION
We propose SimpleBits, information- based method to synthesize simplified inputs. Crucially, SimpleBits does not require any domain-specific knowledge to constrain which input components should be removed; instead SimpleBits itself learns to remove the components of inputs which are least relevant for given task.
As interpretability tool, we show that SimpleBits is able remove injected distractors, sug- gest plausible reasons for misclassification, and recover known radiologic features from condensed datasets. When combined with data condensation, SimpleBits retains accuracy while greatly reduc- ing the complexity of condensed images.
Our simplification approach sheds light the information required for deep network classifier to learn its task. We find that the tradcoff between task performance and input simplification varies by dataset and setting it more pronounced for more complex datasets.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We provide the following information to ensure reproducibility. Main concepts, algorithms and basic architectures are described in sections 3 and Further network architecture details are in supp. sections S2 and S3] further optimization details in supp. sections S4 and S5 Finally, the code available under https: tnnuurl com/ simple-bits
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Imon Banerjee, Ananth Reddy Bhimireddy, John L. Burns, Leo Anthony Celi, Li-Ching Chen, Ra- mon Correa, Natalic Dullerud, Marzych Ghassemi, Shih-Cheng Huang. Po-Chih Kuo, Matthew P. Lungren, Lyle J. Palmer, Brandon J. Price, Saptarshi Purkayastha, Ayis Pyrros, Luke Oakden- Hari Trivedi, Ryan Wang. Zachary Zaiman, Haoran Zhang, and Judy W. Gichoya. Reading race: AI recognise patient's 's racial identity in med- ical images. CoRR, abs/2107. 2021. URL btttpss:/aarivv.orgg//oo wttp:///aaab//..111033.0
Rayner, Chima Okechukwu, Laleh Seyyed-Kalantal
François Bellemare, Alphonse Jeanneret, and Jacques Couture. Sex differences in thoracic dimen- sions and configuration. American journal of respiratory and critical care medicine, 168(3): 305-312, 2003.
Andy Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1059-1071, PMLR, 2021. URL
Ann- Kat-Kathrin Dombrowski, Jan E Gerken, and Pan Kessel. Diffeomorphic explanations with normal- izing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL http:::: http://://ppeeeeeeeeeee net/ http://wwwwetttrrr///e/ 2-R 6G4
Yann Dubois, Benjamin Bloem-Reddy Karen Ullrich, and Chris J. Maddison. Lossy compression for lossless prediction. CoRR, abs/2106 10800, 2021. URL
2116610800
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence, pp. 1-12, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL http: /ttE:://proceediggs mlr. Dozess/v70/Eines
html
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11):665-61) 6776665-66666060 2020.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2376-2384. PMLR, 2019. URL http:ttttp/prroeeeinotm
mlr wtw.lr.press/v99/ /aaal//oyalll
html
Jakob Drachmann Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maaløe. Hierarchical vaes know what they don't know. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th In- ternational Conference Machine Learning. ICML 2021, 18-24 July 2021. Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 4117-4128. PMIR, 2021. URL ngs ml r.w.ssppssss...... prress//139//awve/ http/////ttwwo222/ html
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Lenny Pitt (ed.). Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory, COLT 1993, Santa Cruz, CA, USA, July 26-28, /C.
1993, pp. 5-13. ACM, 1993. doi: 10.1 1..11111/6380..111 168306. URL hhttps://dat 1145/1 68304 168306
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Bue, Buc, E. Fox, and R. Garnet (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL wttp:/tss.///ppcceeeeioooo mew.riiiinnnnnnuns....mmmmm

---

Under review as a conference paper at ICLR 2022
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016- 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science pp. 646-661. Springer, 2016. doi: 978 3- 31 3778-3311-44453333 46493-0 39
URL https://dcii org 1007/
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell., 16(5):550-554, 1994. doi: 10. 10.1109/34.291440. hRtphtttss///oo.ooo
1109/34 34. 291440
Frederik Hvilshøj, Alexandros losifidis, and Ira Assent. ECINN: efficient counterfactuals from
invertible neural networks. CoRR, abs/2103.13701, 2021. htt.ttsss//aaxi.orr//////
2103. 13701
Berthold Jany and Tobias Welte. Pleural effusion in adults etiology diagnosis, and treatment. Deutsches Ärzteblatt Interattonal, 11 116(21):377, 2019.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lun- gren, Chih-ying Deng. Roger Mark, and Steven Horn. Mimic-exr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019a.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng. Yifan Peng. Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 2019b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http:t 7/ 7/arxiv.com //arxiv.org/abse
6980
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeuRIPs), pp. 10215-10224 2018.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson Why normalizing flows fail to
detect out-of-distribttion data. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 6- 12, 2020, virtual, 2020. URL http:: / nww.ips..ppp........ wttpscccppppprrr/ 2000//hash wwwpaaawww.a.ehaaaaaaeaaaaeeeeae fbt.fffwwb99cccc 8eee33312556688822288288828 Abbtbeeeta...bbba-bbbaaat....... Abst. html
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: / /yann.
lecun. wwaaa//xdb//nnii.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th
International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
set//oorumidd=Skeee
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2112-2122 Lille, France, 07-09 Jul 2015. PMLR. URL https: 7/proceed ngs. ml prwss vv3///ac3 aurin15. htmll
12

---

Under review as a conference paper at ICLR 2022
Taro Makino, Stanislaw Jastrzebski, Witold Oleszkiewicz, Celin Chacko, Robin Ehrenpreis. Naziya Samreen, Chloe Chhor, Eric Kim, Jiyon Lee, Kristine Pysarenko, et al. Differences between human and machine perception in medical diagnosis. arXiv preprint MoXiv:2011.14000 2020.
Grégoire Montavon, Wojciech Samek, and Klaus-Robers Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018. ISSN 1051-2004. doi: https://doi.org/10...00 http:////....1100.......11111.10 com/ http:///ceeeerrtteee/////////110
URL https: www. sciencedi rect.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. CoRR, abs/2107.13034, 2021. URL https://arxiv..
abs/ abs/2107 13034
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V. Le. Meta pseudo labels. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, vir- sual, June 19-25. 2021, pp. 11557-11568. Computer Vision Foundation / IEEE,
2021. URL
httpas:///peccccssscco iericeess.hhee twweeeeeee..... com/ wttp:///ccoceee///////////aaaaa Pham Wheta_Pseusee Laeeudo_Labelss ________P_______ 2UMPR_2021_1 paclllppppe.... html.
BN Raasch, EW Carsky, EJ Lane, JP O' Callaghan, and ER Heitzman. Pleural effusion: explanation of some typical appearances. American Journal of Roenigenology, 139(5):899-990 1982.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 1tt1::20330000200 2021.
Aniruddh Raghu, Maithra Raghu, Simon Korbblith, David Duvenaud, and Geoffrey E. Hinton.
Teaching with commentaries. In 9th International Conference Learning Representations, ICLR 2021, Virtall Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: / //openreview. www.eee/.fooooo for:m//rrmmmmooo d= d= Rbogghhgq
Maithra Raghu and Eric Schmidt. A survey of deep learning for scientific discovery. arXiv preprint arXiv:2003. / 1755, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells III. and Alejandro Frangi (eds.), Medical Image Computing and Conputer-Asiisted Inter- vention MICCAI 2015 18th International Conference Munich, Germany, October 5 9, 2015, Proceedings, Part III, volume 9351 of Lecture Notes Computer Science, pp. 234-241. Springer, 2015. doi: 10. 000000097773999-995599 28. URL pattps://do. org 10. 10.100071777000000 978 3 319--19-11 24544-45222
Robin Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly de- tection with deep invertible networks through hierarchies of distributions and features. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httphhttss:/prrreednnnnn neurips wttppauwp.s..ppppppeeeeeepm /tc/pappe/2222//aat/// htm
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. In- put complexity and out-of-distribution detection with likelihood-based generative models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpts:://pppeeeeeeeee/// wwmperrrewww.ee/oorrmmmmmm SyXIMpVYvr
net wtw.effrffmmiii..
Claude Elwood Shannon. A mathematical theory communication. The Bell System Technical Journal, 27(3):379-44 423, July 1948.
13

---

Under review as a conference paper at ICLR 2022
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:// openreview, Seeenreveewnee/foeoee forum == x XFWgrFPS)
net/
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 30( 1):1958-11977 2008.
Tongzhou Wang. Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. CoRR. abs/1811.10059, 2018. URL http:/ttpx//ii..../ wtt.aaiwww.aa..aaaa.oooooooooeot aaaaabb//11111 abs/1811..0959
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. UtLhttp://wee bmva. org/ bmvc/ / 201 6/ httpppppspppppr//11111 087/ index. html
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine mmlr.
Learning Research, pp.
PMLR, 2021. URL
press/v139/zhao211 html
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:
net forum?
DSAKhLYLSs!
Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling, Visualizing deep neural net- work decisions:
Prediction difference analysis. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: /ppp/eeveeee../ee///////// / wwt.:eeeeooommmoooooooooooo
14

---

Under review as a conference paper at ICLR 2022
bpd
0.05
0.24
1.04
1.63
2.38
3.09
3.60
4.05
6.32
(Glow
6
trained
80TinyM)
bpd
0.02
0.23
0.75
1.25
2.02
2.48
2.93
3.71
6.03
(PixelCNN trained on CIFAR10)
6
bpd
0.09
0.31
0.86
1.42
2.22
2.52
2.89
3.77
5.87
6
(Diffusion trained on CIFAR10)
Figure S1: Visualization of the bits-per-dimension (bpd) measure for image complexity, sorted from low to high. Image samples are taken from MNIST, Reshion-MNISTT CIFARIO and CIFARIO0, in addition to completely black image sample. bpd is calculated from the density produced by Glow KKngmma & Dhariwal 2018 model pretrained on 80 Million Tiny Images, PixelCNN model trained on CIFAR10, and diffusion model trained CIFARIO.
SUPPLEMENTARY OUTLINE
This document completes the presentation of the main paper with the following:
Bits per dimensions for selected images of other generative models in SI
Details about the classifier and simplifier architectures in sections S2 and S3
Details about the optimization of per-instance simplification during and after training in sections S4 and S5
Files sizes the simplified images when compressed with PNG in section S6
Learning curves during retraining on the simplified images in section S7
Basclines for the per-instance simplification during training in section 58
Potentially spurious features revealed by SimpleBits in section S10
More images simplified during and after training in sections Mic-SimpleBiis output on the control images
S9 and
S13
including post-
More condensed and simplified datasets under varying settings in section S11
Results for the continual learning setting in section ST2
S1 BPDs OF OTHER GENERATIVE MODELS
Figure ST shows that the bits per dimension produced by other generative models than Glow also correlate well with visual complexity, validating our measure. This is consistent with prior work that found bpds of generative models trained on natural image datasets are strongly influenced by general natural image characteristics independent of any specific dataset (Kirichenko et al.) 2020 Schirrmeister et al. 2020 Havtorn et al. 202T.
S2 CLASSIFIER NETWORK DETAILS
Our classification network built the Wide ResNet architecture (Zagoruyko & Komodakis 2016 We used a version with relatively few parameters with depth = 16 and widen factor 2 to allow for fast iteration experimentation. We used ELU instead of ReL.U nonlinearities.
15

---

Under review as a conference paper at ICLR 2022
Additionally, we removed batch normalization to avoid interference of normalization layers with the simplification process. We followed the method from Brock et al. (2021) to create a normalizer free Wide ResNet. We reparameterize the convolutional layers using Scaled Weight Standardization:
Way = Wy - 44
(S1)
VNo,
m.)², and N denotes the fan-in. Further as in Brook et al. (202I). "activation functions are also scaled by non-linearity specific scalar gain 7, which ensures that the combination of the 7-scaled activation function and a Scaled Weight Standardized layer is variance preserving." Finally, the output of the residual branch is downscaled by 0.2, so the function to compute the output becomes h&+1 = h2 + 0.2 f((h;), where h, denotes the inputs to the jth residual block, and denotes the function computed by the 7th residual branch. Unlike Brock et al. (2021). we did not multiply scalars }3 with the input of the residual branch or learned zero-initializee scalars to multiply with the output of the residual branch, we did not find these two parts helpful in our setting. We also did not attempt to use Stochastic Depth (Huang et al., 2016), which may further improve upon the accuracies reported here. Due to our small batch sizes (32), we also did not use adaptive gradient clipping.
where 14 (L/N) We. of (L/N),,
S3 SIMPLIFIER NETWORK DETAILS
For the simplifier, we adapted a publicly available implementation of UNet We used num_down = downsampling steps, ELU nonlinearities ngf 64 filters in the last conv layer and a simple affine transformation layer instead of a normalization layer. Furthermore, we made the simplifier residual and ensured the output is within [0, by adding the output of the UNet to the inverse-sigmoid trannoooderrassfomeedd input and then reapplying the sigmoid function.
S4 OPTIMIZATION DETAILS PER VER-INSTANCE SIMPLIFICATION DURING TRAINING
First, we note that the single train step helps ensure a correspondence between simplified and original images, and technique others have used in meta-learning settings (Pham et al. 202T
For stabilizing the optimization of the per-instance simplification during training, we found two further steps helpful. First, we modify:
Lels [(f'( y) + ((f(Xslm) y) ((f'(X
(S2)
to
Lele 10 [[f ((oo(()))g))..))... .) y) + (Xm(( y) + (X y)
(S3)
as (a) the gradient magnitudes are much smaller from the losses after unrolling and (b) we want to prioritize the classification loss on the original data. Additionally, we dynamically turn off Laim during training, for any example a where (( (X(or) y) > 0.1, which we also found to further stabilize training.
S5 OPTIMIZATION DETAILS PER-INSTANCE SIMPLIFICATION AFTER TRAINING
We found it beneficial optimize the simplified inputs in the latent space of the pre- pre-trained Glow network and to apply our loss functions to all interpolated inputs on the path between original and simple input in latent space instead of only to the simplified input itself. The points on the path include information from the original input and prevent that the optimization some relevant information from the original input. Additionally, we also ensure that the predictions are the same for the original and interpolated inputs for both the original and the scaled model. The complete loss function can be found in Alg. [ during training we called with scaling factors sampled from (0.8, 0.95) as these values mostly led to similar but more uniformly distributed predictions than the unperturbed network.
unable to recover
https:////t com/ http://hhhhhh.ooooo////////oo/// eGAN- and and-pix2pix
16

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Simplification loss function after training full algorithm
1: given generative invertible network G, input æ and its latent code z (from G). simplified input's latent code Zaim, classification network f. parameter scaling factors <
2: Jaaoeed t- ScaleParameter (callPParmeters(SSss)
Scale parameters of classifier down
simulate "forgetting" & Sample interpolation factor uniformly between 0 and B Interpolate simple and original input in latent space
10)
4: Zmixed = a Zzim + +11 (1-a)
5: amix invent (G, 2mix)
b Invert 2mix using invertible network Predict original and mixed with classifier Predict original and mixed with scaled classifier
6: h f(x)). Amix = (((mii)
7: hscaled fscaled ( hescatdmmix facaled (Fmix.
Lguat = dDDI.. (escated) DRI. I.. (mmix
D Compute distance between gradients on
scaling factors.
9: Lpnel DxI, [miix) + DKL (Ptacated (mnixacated)
Compute prediction differences Invert using invertible network > Compute needed bits for simplified input
10: Zam invert (, Xum)
Lsimpliticutio log pa (Faim))
12: return Lignd + Lipeed + Asmm" Lrimplisicuson
Test Accuracy
90
End of Joint Training Retraining
80
70
Orig CIFAR10
6.0
5.5
5.0
4.5
4.0
PNG-compressed File Size in BPDs
Figure S2: Tradeoff between PNG storage space and accuracies. Note that the PNG bpd file sizes do not show the maximally possible savings, these can be seen from the bpd values the main manuscript.
S6 PNG-COMPRESSED FILE SIZES OF SIMPLIFIED IMAGES
Figure S2 shows the tradeoff between PNG storage space of the simplified images and the accuracies achieved when retraining.
S7 LEARNING CURVES FOR RETRAINING
Figure S3 shows learning curves during retraining on the simplified images on CIFARIO, There are no noticeable differences in training speed for more or less simplified images.
3.57 BPD
Retraining Learning Curves CIFAR10
Test Accuracy [%)
3.56 BPD
3.52 BPD
75
2.95 BPD
2.81 BPD
50
2.66 BPD
25
2.50 BPD
40
60
80
100
2.26 BPD
Training Epoch
1.86 BPD
1.76 BPD
Figure S3: Learning curves for retraining simplified images CIFAR10.
17

---

Under review as a conference paper at ICLR 2022
Per-Instance Simplification During Training, Retraining Results
CIFAR10
SVHN
FASHIONMNIST
MNIST
Accuracy (96)
Simple Bits
75
Minimize MSE and BPD
50
50
Gaussian Blurring
50
JPEG Compress
25
3
1
1.5 1.0 0.5
Bits per dimension
Bits per dimension
Bits per dimension
Bits per dimension
Figure $4: Comparison between SimpleBits and two simpler baselines: In the first one, the simplifier network is trained simultaneously reduce bpd of the simplified image and the mean squared error between the simplified and the original image. In the second one, gaussian blurring is applied to the input images, different runs vary in standard deviation used to create the gaussian blurring kernel. In the third one, we use JPEG compression with varying quality levels. Tradeoff curves are worse for the baselines than for SimpleBits.
S8 SIMPLIFIER BASELINES
We implemented three simpler baselines to check whether the losses used in SimpleBits during training help retain task-relevant information. In the first baseline, we train the simplifier to simulta- neously reduce bpd of the simplified image and the the original image. Afterwards we train the classifier on the simplified images and evaluate on the original images the same way during retraining of SimpleBits. In the second baseline, we blur the original images with a gaussian kernel, which also reduces their bpd. We vary the sigma/standard deviation for the gaussian kernel trade off smoothness and rokkinformaaiveness.. baseline, we use lossy JPEG compression with varying quality levels. As in SimpleBits and the other baselines, we estimate the bits per dimension of the cas=JJPPG-coompeeseed images through our pretrained Glow network for a fair comparison. The gaussian blurring and JPEG compres- sion each replace the simplifier, so these are fixed simplifier baselines without training a simplifier. While these three baselines also retain some task-relevant information allowing the classifier to tain above-chance accuracies (see Figure S4). the tradeoff between bpd and accuracy worse than for SitpleBits. This shows the losses used in SimpleBits help retain more lask-relevan information compared to these baselines.
squared error between the simplified and
the third
S9 MORE IMAGES SIMPLIFIED DURING TRAINING
We show larger number of images that were simplified on CIFARIO during training with the largest simplification loss weight Asim = 2.0 in Figures [S5] S66 and S7
S10 POTENTIALLY SPURIOUS FEATURES UNCOVERED BY SIMPLEBITS
BimpleBits may have the potential to reveal spurious correlations present in the dataset. We show
some simplified images that reveal potentially spurious features in Figure S8 These observations be used starting point to further investigate whether these features also affect regularly trained classifiers.
S11 MORE CONDENSED DATASETS
We also show some interesting condensed datasets that resulted when varied the architecture (Figure(S1O) or the condensation loss (from gradient matching to either negative gradient product or single-training-stee unrolling, Figure
18

---

Under review as a conference paper at ICLR 2022
Figure $5: Uncurated of simplified images with Asim 2.0, 6 per class.
S12 EVALUATION OF CONDENSED DATASETS FOR CONTINUAL LEARNING
We also evaluate the simplified condensed datasets in a continual learning setting, following (Zhao & Bilen 2021). In this task-incremental continual learning setting, the model is trained on different classification datasets sequentially. When training on new dataset, the model is additionally trained on the condensed versions of the previous datasets.
The continual learning experiment reproduces the setting from ZZhao & Bilen 2021) to first train on SVHN, then on MNIST and finally on USPS (Hull 1994), using the average accuracy across all three datasets of the classifier at the end of training as the final accuracy (see (Zhao & Bilen 202T for details).
We created simpler and faster continual training pipeline that achieves comparable results to Zhao Bilen (202T). First, we train 3 times for 50 epochs on SVHN, with cosine annealing learning rate schedule (Loshchilov & Hutter, 2017) that is restarted at each time with Ir = 0.1. Then for each MNIST and USPS, we train one cosine annealing cycle of 50 epochs for Ir = 0.1.
We first verified that we can reproduce the prior continual learning results with our simpler training pipeline and find that our training pipeline indeed even slightly outperforms the reported final re- sults (96.0% vs. 95.2% with, and 95.4% vs 93.0% without knowledge distillation) despite slightly inferior performance in the first training stage (before any continual learning, 93.6% vs 94. 1 1)), see following subsection. When using different SVHN and MNIST condensed datasets, find that can retain the original continual learning accuracies even with condensed datasets with substantially less (~99 less) bits per dimension (see Fig S12).
19

---

Under review as a conference paper at ICLR 2022
Figure $6: Uncurated set of correctly predicted simplified images with Asim 2.0, 6 per class.
S12.1 WITHOUT CONDENSED DATASET
Our training pipeline still exhibits forgetting when not using any condensed datasets of previously trained-on datasets. As Figure S13 shows, the accuracies are far lower than with just regular sequen- tial training. We performed this ablation to ensure forgetting still occurs in our training pipeline.
S13
MORE IMAGES SIMPLIFIED AFTER TRAINING
saat-hoo-impliieee images for misclassified original images in Fig- ure S14 We also show the output of SimpleBits when applied to the control images of Figure g in Figure S15 We also show an uncurated set of incorrectly predicted and correctly predicted images
We show further examples
in Figures S16 and S17
20

---

Under review as a conference paper at ICLR 2022
Figure S7: Correctly predicted simplified images with strongest color with Asim 2.0, 6 per class.
Predicted
Spurious Feature
Correctly Predicted
Incorrectly Predicted
as
Car
Red texture
Plane
Blue color
Red Brown texture
Horse
Ship
Dark bottom half, bright top half
Figure S8: Selected simplified images that highlight potentially spurious features. Two leftmost images correctly predicted, two rightmost images incorrectly predicted.
21

---

Under review as a conference paper at ICLR 2022
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
28.2%
8.94 bpd
28.5%
4.45 bpd
1
3
6
6
29.0%
18
2
3
4
5
6
7
8
9
7.55 bpd
29.4%
[0
10
2
3
4
5
6
7
8
9
4.55 bpd
T-shirt
Pullover
Sandal
Shirt
Sneaker
Bag
Ankleboot
69.8%
5.84 bpd
68.6%
3.19 bpd
1
3
4
5
7
8
92.0%
0
/
2
3
4
5
6
7
8
9
4.98 bpd
2.65 bpd
92.0%
0
1
2
3
4
5
6
7
8
9
Figure S9: Dataset condensation results with varying simplification loss weight. Top: Individual
dots represent accuracies for setting with different simplification loss weights. Accuracies can be retained even with substantially reduced bits per dimension. For image per class, arrows highlight the settings that visualized below. Below: Condensed datasets with varying simplification loss weight. Per dataset, showing condensed datasets with high (top row) and low (bottom row) bits per dimension. Lower bits per dimension datasets visually simpler and smoother while mostly retaining accuracies.
plane
car
bird
cat
deer
dog
frog
horse
ship
truck
Instance norm ReLU
28.7% Acc. HPD)
Instance norm ELU
27.3% Acc.
4.35 BPD
No norm ReLU
22.5% Acc. 4.35 BPD
No norm
22.7% Acc. BPD
ELU
Figure S10: Dataset condensation on CIFAR 10 with varying architecture
22

---

Under review as a conference paper at ICLR 2022
Grud
plane
car
bird
cat
deer
dog
frog
horse
ship
truck
Distance, Outer
28.7%
4.35 BPD
step
Grad
29.1%
Product
6.47 BPD
1-step Unrolling
5.24 BPD
Grad
27.6% Acc. 1.6
Product, 4 Outer
Steps
Grad
Product,
26.9% Acc. 0.37 BPD
ReLU,
Outer
Steps
Figure $11: Dataset condensation on CIFAR10 with varying condensation loss and varying outer loop steps, i.e. how many steps the classifier is trained at each training epoch (default in the image per class setting), after each step the condensation loss is again optimized.
(average acress datasets
training)
Continual Learning Results
96
ours KD SVHN bpd: 11.9, MNIST BPO: 4.3
Accuracy (%)
ours W. KD
ours W. KD SVHN bpd: 3.9, MNIST BPD: 2.5
94
ours W.
ours
bpd: 1.0,
92
SVHN
SVHN+ MNIST
SVHN+ MNIST +
Orig KD
USPS
Orig w.o. KD
Training Stage
Figure $12: Continual Learning Results. Results for first training on SVHN, then MNIST and then USPS for condensed datasets with varying bits per dimension. Solid lines are with and dashed lines without knowledge distillation. Note that continual learning accuracies remain similar also for substantially reduced bits per dimension. Ablations show that accuracies degrade without any condensed dataset, see supplementary.
Continual Learning Results
faverage across datass
ours KD SVHN bpd: 11.9, MNIST BPD: 4.3
seen during training)
95
ours w.o. KD
Accuracy (%)
ours w. KD SVHN bpd: 3.9, MNIST BPD: 2.5
ours w.o. KD
ours w. KD SVHN bpd: 1.0, MNIST BPD: 0.8
ours w.o. KD
80
no condensed dataset
Orig
Orig
SVHN
MNIS+ Training Stage
SVHN SWIN+
Figure S13: Continual Learning Results without Condensed Dataset (regular sequential training). Conventions as in Figure S12 Accuracies substantially worse without any condensed dataset.
23

---

Under review as a conference paper at ICLR 2022
Original Images
Predicted as
Ship
Plane
Frog
Deer
Car
Simplified
Hypothesis for one cause misclassification
Ground + plane ship
Black
Higher Freq Texture
Color and antier -like shapes
and wheelsshape
Control Image
Predicted
Plane (true class)
Ship (true class)
Bird
Deer
(true class)
(true class)
(true class)
Original Images
Predicted as
Deer
Frog
Car
Deer
Dog
Simplified
Hypathesis for one cause misclassification
Tree as antler
Color
Brighter parts form car contours
Color, earitail antler-like
Black-white dog face
Control Image
Predicted as
Horse (true class)
Cat (true class)
Cat (true class)
Dog (true class)
Bird (true class)
Figure S14: Further examples of post-hoc simplifications of originally misclassified images.
24

---

Under review as a conference paper at ICLR 2022
class
Horse
Rird
Original Image
Predicted as
Plane
Dog
Frog
Frog
SimpleBits output
Hypothesis for one cause of misclastification
Tiee top revembles antfor
Black pads of pana. color boitom
High froquency tenture
Chelle resembles shape
ship body
Green color
Control Image (prdicted correctty
SimpleBits output on control
Figure S15: Post-hoc simplifications of control images.
25

---

Under review as a conference paper at ICLR 2022
Class
Truck
Automobile
Ship
Dog
Roal Bremn
original
Predicted Class
Automobile
Dog
Airplane
Dog
Frog
Bird
Airplane
Deer
Simplified
True Class
Airplane
Truck
Bird
Cat
Bird
Dog
Truck
Airplane
Original
Predicted Class
Truck
Automobile
Dog
Cat
Automobile
Ship
Simplified
True Class
Dog
Cat
Cat
Cat
Cat
Airplane
Truck
Original
Predicted Class
Airplane
Deer
Dog
Truck
Dog
Deer
Deer
Automobile
Simplified
True Class
Truck
Cat
Bird
Ship
Dog
Bird
Cat
Truck
Original
Predicted
Frog
Dog
Airplane
Dog
Dog
Simplified
Figure S16: Uncurated post-hoc simplifications of incorrectly predicted images.
26

---

Under review as a conference paper at ICLR 2022
Class
Ship
Frog
Frog
Automobile Frog
Automobile
Original
Predicted Class
Cat
Ship
Frog
Frog
Automobile
Frog
Cat
Automobile
Simplified
True Class
Airplane
Truck
Dog
Horse
Truck
Ship
Dog
Horse
Original
Predicted Class
Airplane
Truck
Horse
Truck
Ship
Dog
Horse
Simplified
True Class
Ship
Frog
Airplane
Deer
Truck
Dog
Airplane
Truck
Original
Predicted Class
Ship
Frog
Airplane
Deer
Truck
Dog
Airplane
Truck
Simplified
True Class
Frog
Frog
Deer
Dog
Truck
Bird
Deer
Original
Predicted
Frog
Frog
Dog
Dog
Simplified
Figure $17: Uncurated post-hoc simplifications of correctly predicted images.
27


Reviews:
[{"review_id": "v8GCG5R7BA", "reviewer": "Reviewer_ZQYq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper provides a diverse set of experiments showing that SimpleBits is able to reduce the image complexity in terms of bpd (bits per dimension) while retaining model performance at the same time. The proposed method is fully data-driven and does not have any strong assumptions. \n\n**Strength**\n- The proposed loss terms $\\lambda_{sim} L_{sim} + L_{cls}$ are easy to implement\n- The paper provides ways to apply the simplification network in various settings\n- I like figure 4, where it is easy to see the progress of the simplification\n\n**Weakness**\n- Section 4 is hard to understand. What is the motivation for simulating forgetting? In Algo.1, what are $\\mathbf{h}$ and $\\mathbf{h}_{sim}$ in L4, 5? The gradients of the returned loss are taken w.r.t to which parameter, simplification network or $f$?\n- In Section 5, it is unclear if the motivation here is to condense the dataset further or interpret neural nets. \n- In the dataset condensation part, prior work (Zhao & Bilen, 2021) performs gradient matching in the inner loop and re-initializes the model parameters in the outer loop. Therefore, for each outer loop, a different neural network is used. It is unclear which model the proposed method is interpreting. In the rebuttal, the authors can explain the details of combining simplification and dataset condensation more.\n\n**Other**\n- For section 3 (simplification during training), can the proposed method identify spurious features learned by the neural nets?\n", "summary_of_the_paper": "The paper proposes to minimize the image per-instance complexity to interpret the essential ingredients for neural network learning. The proposed method, SimpleBits, works in three different settings _i)_ Simplification during training _ii)_ Simplification after training _iii)_ Simplification with dataset condensation. \n\nIn _i)_, SimpleBits is able to differentiate task-relevant information on simulated composite datasets. In _ii)_, SimpleBits is able to explain the possible reason for misclassification. In _iii)_, SimpleBits reduces the image complexity in dataset condensation even further.\n\nThe major contributions can be summarized as follows.\n\n- Simplifying images to interpret the neural models is not new, while this paper performs the novel simplification on image complexity.\n- The paper proposes a way to simplify image complexity while retaining classifier accuracy.\n- The paper shows that SimpleBits helps interpret the model learning in three different cases.", "summary_of_the_review": "The paper proposes a novel direction of simplifying images (in terms of image complexity) to interpret neural networks. The proposed method is intuitive and easy to implement, having the potential to make broader impacts. \nThe paper applies the proposed, SimpleBits, in three different settings *i)* simplification during training *ii)* simplification after training *iii)* simplification with dataset condensation. \n\nFrom the title, the paper's main goal should be understanding and interpreting neural networks, while I found the contents sway between interpreting models and simplifying images. Without further elaboration, better simplifying images does not naturally imply better model interpretation. \n\nOverall, I think the method is novel, and the experimental results look convincing, but the explanation and motivation are unclear, especially sections 4 and 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YvyakrTPvqy", "reviewer": "Reviewer_RfmX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the visualization tool proposed in the paper very interesting but I encountered two important limitations:\n\n(1) Poorly explained methodology\n\nIt's not clear what the $unrolledtrainstep$ function (Eq.3) implements. I don't think this problem formalization is common knowledge, so it should be more thoroughly explained.\nSimilarly, the $simplifier$ function (Eq.2) is not described in the main paper.\n\n\"...adding the classification losses on the simplified data both before and after the unrolled training step improves training stability\". \nThe rationale for *why* this works is not provided.\n\n(2) Lack of baselines and comparisons\n\nThe simplifier function is not investigated, so it's impossible to know if the method is sensitive to this component. The choice of the specific simplifier used is not justified.\n\nOn the condensation experiments there's no comparison with other condensation methods in the literature (Fig.7).\nThere's also no natural baseline provided, e.g. a dataset of the same size as the original (or until performance levels off) in order to understand how good this type of condensation can be in the limit.\n\nSimilarly, the results on training with simplified images (Fig.5) do not include baselines or other competing methods.\n\n(other comments)\n\nFig.6: It would be interesting to see what the SimpleBits output is for the control image. For example, does the simplified image of the deer change considerably when the background color (outside the circle) is manually corrected? \nAre there cases where after correcting the suspected cause of error, the image is still misclassified? It would be good to show some unsuccessful/hard cases.\n\nSec.4: As a visual inspection tool, this is interesting, but the manual process of altering the images is unlikely to be viable for many applications like large datasets or streaming data). Do the authors anticipate a way of automating this process? ", "summary_of_the_paper": "This paper proposes a method to simplify images by reducing their information content, and measure the resulting effects on several aspects of learning: removing redundant image features for network training, understanding classification errors via visualization, and data condensation.", "summary_of_the_review": "In my opinion the methodology section must be more thoroughly explained, and comparisons/baselines must be added. For these reasons I recommend that the paper is rejected in its current form. I think that the visualization tool proposed for diagnosing errors could be useful, so I would encourage the authors to improve the manuscript.\n\nScore: 3: reject, not good enough.\n\n============== AFTER REBUTTAL\n\nI have updated my score for this paper, following the authors' clarifications. However, I still think that the paper is incomplete with respect to baselines using alternative image compression methods or other techniques.\n\nUpdated Score: 5: marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R2-2VbtPvCX", "reviewer": "Reviewer_eYVm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Draft discusses an interesting problem. However, in the end-to-end learning framework it is likely to find task-irrelevant information in the input. While surely it is an interesting investigation to understand the amount of relevant information in the inputs, authors may better highlight practical utility of the simplified samples (or simplification).\n\n- Since the simplified images have less irrelevant information how do they affect the time taken for training the models compared to the original samples? \n\n- On the same lines, how can one leverage the trade-off between the model performance and sample simplicity? In other words, in the conventional training scenario the reader may like to know if there are any advantages/applications for sample simplification. For instance, do the simplified samples occupy lesser memory footprint than the original samples?\n\n- Proposed per-instance simplification post model training can help to explore model behavior as a posthoc interpretation. However, it looks complicated to obtain as opposed to multiple gradient based posthoc explanations. It is not clear if they are related in some way.\n\n- Shouldn't the notion of sample complexity be task specific? How is it justified to interpret the negative likelihood of the sample to be its complexity in the classification task?\n\n- Authors use bpd as the image complexity measure which treats an image with more likelihood under a generative model trained on Tiny images dataset as less complex. From Figure 2 of the draft, a complete black image has close to 0 bpd, does that mean it has very strong likelihood? However, in general such images are not common in real datasets. Authors are suggested to clarify this. \n\n- Also, how representative the computed likelihood can be, given that it is applied across different datasets (gray scale, color, etc. sometimes in different domains such as medical images) despite being learned on a specific dataset? \n\n- In the experiment with the side-by-side MNIST dataset, SimpleBits output erases major portion of the relevant information also. I wonder what is the classification accuracy on the side-by-side data and the actual MNIST test data are.", "summary_of_the_paper": "- Draft proposes a simplification method for visual recognition task where the information less relevant for the classification task is removed from the samples. In an iterative fashion the proposed approach (with no domain knowledge) synthesizes the simplified samples. Then the draft attempts to (i) measure the effect of the simplification via evaluating the models trained on the simplified samples, (ii) understand the models' behavior via posthoc interpretations, and (iii) simplify the synthetic data learned through dataset condensation.", "summary_of_the_review": "- While the draft discusses a novel and interesting problem,  I feel it lacks in some of the aspects (listed in the Main Review). The work can be considered as a good early work in this direction. If the authors can justify the motivation for chosen complexity notion, and the effectiveness of the proposed posthoc interpretation along with other minor issues, this can become an excellent paper.\n\n- Pre-rebuttal score: 5 (marginally below)\n-----------------\npost-rebuttal\n----------------\n\n- Rebuttal partially convinces me on some of the aspects (e.g. sample complexity) so I increased the score to 6. \n- However, after reading other reviews and authors' rebuttal I feel that the manuscript needs some more improvements. \n- For instance, primary aim of the manuscript is to investigate the effect of reducing information on classification, and I feel the findings are not very nontrivial or significant. In the end-to-end framework where the sample is presented to the learning algorithm as it is, one can easily see that he/she can find irrelevant information in the sample. And the framework deployed in the manuscript is very much inspired from the meta-learning approaches used for dataset distillation, condensation works. Also, the observations have not been leveraged (or harvested) in a significant way.\n- Despite my earlier points, I feel that these investigations have potential to better understand the information that the classifier networks might be relying for learning. Hence, I score 6, but I will not fight for this manuscript in a discussion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8Fh8qfCmcBB", "reviewer": "Reviewer_agcx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method has the potential to illuminate the important regions of an input image for classification while avoiding having to manually occlude parts of images to figure out the most relevant image parts. I especially like the idea of jointly optimizing the simplifier network and the classification network -- in this way, the simplifier network can explain the prediction made by the classification network, by showing the simplified version of the input image that retains the most salient features for classification, and we know that the classification network can perform reasonably well on simplified images. The integration of the simplification loss and the gradient matching loss for dataset condensation is also a great idea, because it allows us to not only condense a dataset but also illuminate the important aspects of each class in the dataset.\n\nI have some questions/concerns regarding the use of SimpleBits for posthoc explainability analysis of an already trained model (Section 4). Is the goal to train a simplifier network to explain the prediction of an already trained image classifier (by showing a simplified version of the input image that contains the most relevant parts for classification)? I am not entirely sure from the reading of Section 4. Is Algorithm 1 (or Algorithm 2 in the supplement) trying to describe how one computes the loss function for training the simplifier network in this case? Do you then perform back-propagation to update the simplifier network? Why is scaling of parameters of the trained network needed? Also, what is the purpose/meaning of the gradients of the KL-divergence between the rescaled network's prediction and the original network's prediction? Section 4 needs a lot more explanations than what is currently written -- it needs to be rewritten for better clarity. Also, how does the proposed method compares with existing posthoc techniques (qualitatively or quantitatively)?", "summary_of_the_paper": "In this paper, the authors proposed SimpleBits, which is a method to create simplified input images that retain the most relevant parts for classification while removing irrelevant details. The core of SimpleBits is the simplification loss, which is the negative log likelihood of seeing an image, according to an already trained generative model (as well as a related concept known as bits per dimension, which is simply the simplification loss divided by the dimension of a flattened image). The authors showed empirically that more complex images generally have higher bits per dimension/simplification loss.\n\nThe authors applied SimpleBits in three settings: (1) during training, (2) after training, and (3) dataset condensation/summarization. In the first setting, the authors trained both an image classifier and a simplifier network in tandem -- in particular, the simplifier network is trained by minimizing both the classification loss and the simplification loss. In the second setting, the authors tried to create simplified versions of input images, especially input images that are wrongly classified, to hypothesize (posthoc) about how an already trained network made predictions. In the third setting, the authors combined simplification loss with the gradient matching loss (Zhao and Bilen, 2021) to create a smaller dataset with simplified images, and showed that simplification does not have significant impact on accuracy.", "summary_of_the_review": "The paper is for the most part well-written. Section 4 needs serious improvement in terms of clarity -- in particular, the authors should provide intuition and clearer explanations about how they performed posthoc explainability analysis using SimpleBits/a simplifier network (e.g., explanations about L_grad and L_pred in Algorithm 1 and why scaling of parameters of the trained network is needed). The authors should also consider comparing their methods of generating simplified images with existing posthoc techniques (e.g., saliency techniques). For a given input image, is the simplified image retaining regions similar to those identified as salient by an existing saliency method?\n\n******************************\npost-rebuttal:\n\nThanks for the clarification of Algorithm 1/2. After reading the response and the revised paper, I still think that the paper will benefit from a better explanation of how to synthesize a simplified image for a given input image for an already trained neural network. I understand that Algorithm 1/2 details the process of how to compute the loss function, but there is still a lack of explanation of how to optimize the loss function in the latent space. Essentially, I want to the following question be answered clearly: how do you perform the optimization in the latent space of the pretrained invertible generative model, using the loss function you computed in Algorithm 1/2? Right now, there is only an algorithm for how to compute the loss of the optimization problem, but I cannot find the optimization algorithm itself. Therefore, I am keeping my original score of 6.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes SimpleBits, for simplifying input images to remove irrelevant details but keep relevant details for classification. This idea can be applied during/after training. Authors have significantly revised the draft to address reviewer concerns, to improve the readability and clarify concerns on complexity analysis, for which reviewers have raised scores post-rebuttal. However, even with score changes, there are commonly expressed concerns, that manuscript still needs some more improvements to be ready for publication in their post-rebuttal comments: findings are not very nontrivial or significant (reviewer eYVm), still incomplete (RfmX) or optimization algorithm is yet to be found (reviewer agcx) .
Decision: Reject