Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a  new side channel analysis attack framework to reconstruct images from system side channel. This new framework consists of two submodels: the VAE-LP model generates a coarse reconstruction image and the GAN model refines this reconstruction. The experiments show good results in reconstruction.

The idea of reconstructing images from system side channels is interesting and the proposed approach shows good performance. I have the following concerns about the publication of this paper:

1. The novelty is limited from the perspective of the method. The VAE with learned prior is new but little discussion is provided that why a learned prior is necessary. Can we just use a simple supervised end-to-end autoencoder to learn the mapping from side channel signals to the image? Can we use a standard VAE without the Image Encoder branch? Btw, these two are the important baseline for comparison. The authors need to explain why such a design is necessary, or better. Is this a framework specifically designed for SCA, or is it suitable for any task of image reconstruction from any signals?

2. The motivation of the GAN submodel is confusing. As shown in Figure.3, the results of VAE-LP are blurry and are difficult to identify the details, which is reasonable as the input may include very limited information. The GAN submodel adds a lot of details to these images. Some of them are far from the reference images, which is also reasonable as GAN tends to add **texture-like noises**. The refinement or the rich structures and textures are made-up by GAN for good **visual effects**. This additional information is not guaranteed to be correct. Usually, GAN is applied where we require good visual effects. Do we need this in SCA, even more important than the accuracy of the results?

3. The GAN submodel also faces the challenge of **model collapse**, which refers to the problem that the network tends to generate limited patterns to fool the discriminator. As the training dataset is highly structured and the patterns are relatively simple. Is the proposed method capable of working in the scenario of reconstructing natural images? Is the risk of model collapse a concern for the SCA task? This discussion deserves to be included in the paper.

4. Another question that may attract the attention of the machine learning field is, can we know what information in the side channel is used for image reconstruction? In the paper, the side channel information is folded into 2D matrix because of the spatial dependency. Can we visualize this dependency? Is it robust to noise or shifting?

--- Updates ---

After reading the authors' response, I decide to raise my rating to "accept".

review_title: Nice work. But the necessity of using generative models needs further investigation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proposed a representation and generation model to reconstruct the image signals from system side channel signals. The task itself is interesting and novel, demonstrating the first efforts and impressive performance on recovering noisy side channel signals. The work will potentially inspire more attempts to conduct attack and security research on computer internal signals. 

Pros:

+ A clear problem statement and description of the proposed method, and very impressive performance in both visualization and quantitative metrics.
+ A good analysis of the generalization ability of the model.

Cons:
- The method itself makes sense to me, but not novel. The authors propose to share latent codes between two modalities and train them jointly. And then use GAN to further refine the image quality for visualization. The overall pipeline is very common in image restoration and generation area. 
- The 'learn the prior' method may fail to learn a good latent space without another prior distribution regularization. It's better to compare it with first training the VAE with image encoders using fixed gaussian prior, and then train the latent space jointly while fixing the image encoder. I believe the latent space will be better regularized and smooth. However, the tasks do not require you to generate image from the latent space, the advantages may not be revealed from the reconstruction results.
- The performance of the model trained on mini-ImageNet is poor, even though the authors state that it still reconstructs the basic structures. However, it also reminds us that the good reconstruction results are highly related to the small image training database domain. It may be harder for larger databases or other unseen data. 
- Will the organization of the matrix influence the results? What if there are random zero paddings or other errors in the log?
- It's better to analyze and think about the possible solution to avoid these kinds of attacks. Otherwise, purely talking about the attacks strategies may raise some ethics concerns of the paper.

In summary, the task sounds interesting to me, but the overall method is not novel at all. The authors utilized a totally old technical model and training method in a brand new topic by formatting the input modalities. The results seem impressive in smaller dataset, but it still not convincing enough when testing the model on cross-domain data. Besides, there are possible ethics issues if no discussion related to avoiding the attacks is covered. 

Update:
Thanks for all the authors' feedback and the make-up experiments as well as revisions. I appreciate the authors' efforts on solving the privacy problem and new input forms, but I really think the paper made no contributions to 'learning representation'.  The VAE and GAN-based model is not problem-specific or novelly designed for the task. Instead, it is a general framework for any reconstruction problems. I do believe this will make a strong submission to other conferences related to information. In this case, I will not change my rating.
review_title: Interesting Applications but Not Interesting Method

final_score: rating: 7: Good paper, accept
review_content: 
#### Summary:
The authors present a framework that uses a combination of   VAE and GAN to recover private user images using Side channel analysis of memory access . A VAE-LP model first reconstructs a coarse image from side channel information which is reshaped and processed using a convolutional network. The output of the VAE-LP model is refined using a GAN to add fine details. Compelling results are demonstrated for recovery of private information and state of art metrics are reported. 

#### Strengths:
1. The paper is well written with great attention to detail. All design decisions are well motivated and have been adequately explained.
2. State of the art qualitative results are demonstrated for recovery of images using SCA .
3. The paper is an extremely interesting application of the VAE-GAN framework. 
4. The proposed solution offers a way to recover private information without requiring specialized knowledge of the system architecture, which is a great advantage as it is potentially generalizable to a wide variety of architectures.
5. To the best of the reviewer's knowledge, the treatment of related work is adequate and the positioning of the current work in the context of prior art is made very clear.
6. Analysis is performed on a wide variety of datasets indicating that the solution is not over-engineered to a particular library or architecture. 
7. The appendix presents a lot of interesting experiments and analysis. Particularly, classification performance and recovery quality on images trained using a more generic rather than domain specific dataset. 

#### Weaknesses:
1. Although the authors mention that direct comparisons are hard to provide to previous methods due to lack of public implementation. Few qualitative examples of recovered images from prior art (relegated to the appendix) would help motivate the need for the proposed approach better.
2. It is not clear why the L1 loss is necessary for the second stage in section 3.3 . Particularly, as opposed to a more well studied framework like Pix2pix[1], which imposes a L1 loss at the output with a ground truth image. Because at training time we know the GT image that we are recovering. An ablation study demonstrating the quality without this L1 loss would be insightful.
3. Reporting image level metrics is instructive in understanding how far the reconstructed image is from the ground truth image. Particularly, SSIM and LPIPS[2] score of generated image w.r.t ground truth image. 
4. Although the formulation is interesting. The novelty is somewhat limited as it is straightforward application of the VAE-GAN framework to SCA data.

#### Questions:
1. The exact nature of the trace encoding is unclear. Are memory locations fed directly into the network? If so how are the normalized, because assuming 16 bit addresses, the range of values occupied 
2. The authors mention that libraries like libjpeg generates 700k side channel "signals"  in Section 1. What is the nature of these signals? Particularly are the single bits of information/ text/ memory locations? 
3. Although beyond the scope of the presented work, this framework allows for access of private information. Such methods raise obvious ethical concerns on part of services which offer ML solution. A small statement about how this form of attack can be protected against would be instructive in strengthening the narrative. 
4. The motivation to use a CNN to process trace matrix is unclear. Particularly, are assumptions of equivariance made on the access patterns?  Apart from the architectural convenience of weight sharing is there a stronger motivation to assume a convolutional structure on the input? An MLP(or 1*1 conv) with pooling run on the same $$N \times N \times K$$ encoded trace matrix would provided some insights regarding the need for a CNN to process the trace information. 
5. It is not clear what the generated *Dec* refers to in Sec. 3.2.

#### Minor:
subsumed -> assumed ? in section 3.2 

#### Justification of rating:
Although the authors present an interesting application of the VAE-GAN framework for recovery of private information from SCA, the formulation of the problem is remarkably similar to conditional-VAE. It would be interesting to further analyze the learnt representation from the VAE-LP to see if certain "code-blocks" or functions can be isolated. Having said that, the paper is very well written and thorough in its analysis. The reviewer also believes that a systems/privacy audience would be able to appreciate the contributions of this work better.

#### Updates
In light of the responses provided by the authors, and the thorough nature of the additional experiments I am inclined to increase the ratings of this paper. The followup responses answer most concerns raised by reviewers and provide supporting evidence. However, the reviewer maintains that this work could potentially be of more interest to a privacy audience who may better be able to appreciate the use case considered in the presented work.

[1] Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[2] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
review_title: The authors present a framework to reconstruct private user images using a GAN trained on side channel information obtained from image libraries accessing the image. 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Objective of the paper:  
The objective of the paper is to show that the side channel information (based on timing, etc.) for image analysis software can be used to reconstruct reasonably accurate to highly accurate approximations of user images.  While this was previously known, in this paper, the authors use machine learning techniques to more efficiently "learn" how to use this side-channel information, developing generative models that allow image reconstruction.  

Strong Points:
1)  The use of machine learning techniques as an "attack methodology" for making use of side channel information seems to be a potentially powerful methodology.
2)  The results shown here seem quite strong.  
3)  The application (image analysis) is well chosen;  the side channels under consideration seems quite plausible.  
4)  The code is made available.  

Weak Points:
1)   There are obvious counters to this approach;  once one knows that this type of software is being targeted, one can introduce various randomization (for memory accesses, timing, etc.) to lessen the power of side channel attacks.  That is, this may only be a reasonable attack because nobody has to this point considered it important to avoid side channel information being released by image analysis software.
2)   It's a little unclear (to this reviewer) how much "information" the learning algorithm is seeded with.  That is, for these image sets, how similar/distinct are the underlying images?  Your examples seem to capture a lot of the original image, but it's not clear to me how to interpret what the algorithm is starting with.  (Essentially, would these results scale to much larger sets of images?  How much does working with the limited set of celebrity faces already help yield the strong results?  This is tackled somewhat in Appendix F, so the authors do seem aware of this point.)
3)  It seems like you're considering all the different side channels separately?  Is there any way to combine the multiple side channels (into a mega-side-channel?) and do better?  (I apologize if I'm missing something -- I'd expect to see a result set with each side information, as you've given, but then results with all combined.)
4)  This reviewer does not have good insight into why this type of side information would be useful in determining how to reconstruct images in the manner done here.  Some description or intuition as to how accessed cache lines could intuitively be used to determine an approximation to the original picture would be welcome.  (Understood you have limited space.)  
5)   The paper does seem like a first step and there remains significant room for improvement (improved side channels, improved scalability, other applications).  

Overall Rating:  
Despite the (relatively minor) weak points, the paper seems to be a strong proof of concept that machine learning can greatly increase the risk of side channel attacks, because they can essentially automate the task of determining how to use the side channel information to obtain the desired result.  This seems to be an important message for the community, in particular as I expect this framework is not specific to image analysis software, but can be applied more generally.  

Questions for Authors:
I would very much like to see some sort of experiment where noise is injected in the side channel information.  One can view this as a setting where one obtains limited side channel information for some reason, or as a setting where the program is modified in some way to lessen the amount/value of side information.   

Similarly, as discussed above, is there some intuition as to why the side information yields such reasonable results?  (Again, and how this relates to the limited data sets?)  And whether or not you could combine multiple types of side information naturally using this approach?

Other Feedback:  
Overall thee paper seems well written, and you do provide a great deal of data.  
review_title: Review of Private Image Reconstruction from System Side Channels Using Generative Models 

Meta-review: The paper shows that it is possible to reconstruct private images from CPU cache line and OS page table accesses side channels, using a generative model on top of side channel traces. The reviewers agree that the problem is interesting and the experimental evaluation makes a convincing case that such an attack is possible. The author rebuttal was useful in clarifying some aspects of the paper, and the discussion on possible mitigation strategies is a nice addition to the paper.
Decision: Accept (Poster)