SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 14: Scatter plots of the 12 experiments from Fig. 5 (a). Each plot is model trained with a different value of weight decay on 2 classes with 50 samples in each class. Certain values of weight decay make the model more susceptible to our reconstruction scheme.
F Convolutional Neural Networks - Ablations and Observations
In this section we provide more results and visualizations to the experiments on convolutional neural network in Section 6.1.
In Fig. 15 we show ablations for the choice of the kernel-size  $(k)$  and number of output channels  $(C_{\mathrm{out}})$  for models with architecture CONV(kernel-size  $= k$  ,output-channels  $= C_{\mathrm{out}}$  )-1000-1. All models were trained on 500 images (250 images per class) from the CIFAR10 dataset, with weight-decay
term  $\lambda_{\mathrm{WD}} = 0.001$ . As can be seen, for such convolutional models we are able to reconstruct samples for a wide range of choices.
Note that the full summary of reconstruction quality versus the distance from the decision boundary for the model whose reconstructed samples are shown in Fig. 6, is shown in Fig. 15 for kernel-size 3 (first row) and number of output channels 32 (third column).
Further analysis of Fig. 15. As expected for models with less parameters, the reconstructability decreases as the number of output channels decrease. An interesting phenomenon is observed for varying the kernel size: for a fixed number of output channel, as the kernel size increases, the susceptibility of the model to our reconstruction scheme decreases. However, as the kernel size approaches 32 (the full resolution of the input image), the reconstructability increases once again. On the one hand it is expected, since for kernel-size=32 the model is essentially an MLP, albeit with smaller hidden dimension than usual (at most 64 here, whereas the typical model used in the paper had 1000). On the other hand, it is not clear why for some intermediate values of kernel size (in between 3 and 32) the reconstructability decreases dramatically (for many models there are no reconstructed samples at all). This observation is an interesting research direction for future works.
Number of Channels  $(C_{\mathrm{out}})$
Figure 15: Ablating the choice of the kernel size and output-channels for reconstruction from neural binary classifiers with architecture CONV(kernel-size=k, output-channels=Cout)-1000-1. (Please note that this figure might not be displayed well on Google Chrome. Please open in Acrobat reader.)
Visualizing Kernels. In Haim et al. [2022], it was shown that some of the training samples can be found in the first layer of the trained MLPs, by reshaping and visualizing the weights of the first fully-connected layer. As opposed to MLPs, in the case of a model whose first layer is a convolution layer, this is not possible. For completeness, in Fig. 16 we visualize all 32 kernels of the Conv layer. Obviously, full images of shape  $3 \times 32 \times 32$  cannot be found in kernels of shape  $3 \times 3 \times 3$ , which makes reconstruction from such models (with convolution first layer) even more interesting.
Figure 16: The kernels of the model whose reconstructions are shown in Fig. 6, displayed as RGB images.
G Reconstruction From a Larger Number of Samples
One of the major limitations of Haim et al. [2022] is that they reconstruct from models that trained on a relatively small number of samples. Specifically, in their largest experiment, a model is trained with only 1,000 samples. Here we take a step further, and apply our reconstruction scheme for a model trained on 5,000 data samples.
To this end, we trained a 3-layer MLP, where the number of neurons in each hidden layer is 10,000. Note that the size of the hidden layer is 10 times larger than in any other model we used. Increasing the number of neurons seems to be one of the major reasons for which we are able to reconstruct from such large datasets, although we believe it could be done with smaller models, which we leave for future research. We used the CIFAR100 dataset, with 50 samples in each class, for a total of 5000 samples.
In Fig. 17a we give the best reconstructions of the model. Note that although there is a degradation in the quality of the reconstruction w.r.t a model trained on less samples, it is still clear that our scheme can reconstruct some of the training samples to some extent. In Fig. 17b we show a scatter plot of the SSIM score w.r.t the distance from the boundary, similar to Fig. 3a. Although most of the samples are on or close to the margin, only a few dozens achieve an  $\mathrm{SSIM} > 0.4$ . This may indicate that there is a potential for much more images to reconstruct, and possibly with better quality.
(a) Full Images. Original samples from the training set (red) and reconstructed results (blue)
(b) Scatter plot (similar to Fig. 3).
Figure 17: Reconstruction from a model trained on 50 images per class from the CIFAR100 dataset (100 classes, total of 5000 datapoints). The model is a 3-layer MLP with 10000 neurons in each layer. (Please note that this figure might not be displayed well on Google Chrome. Please open in Acrobat reader.)