Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors in this paper address a challenging, and interesting problem in the current deep neural network literature. Though it is widely believed that neural networks essentially memorize, thereby overfit to training samples, it is usually not very clear how one might be able to achieve this. This paper proposes a relatively clean and straightforward formulation to compute the training data points. The authors rely on the results of [Ji and Telgarsky, 2020] to frame the problem as discussed in Equations (2) - (5). The technique hinges on the fact that at the training data points, a linear combination of the derivatives of the network parameters can predict the parameters themselves. A very interesting observation in my opinion.

strengths_and_weaknesses: Strengths : 

1. Clean formulation of the problem in terms of  reconstruction loss to find the training points.
2. The approach produces convincing results in some of the standard computer vision datasets like MNIST and CIFAR10.

Weakness : 

1. This work cannot be extended to CNNs with skip connections  easily due to the homogeneity assumption. Which limits the test error on CIFAR10 due to the above limitation is 71%.  

questions: 
I wonder if the authors have suggestions, for a possible way to extend the current work to networks with skip connections ?

limitations: Not general enough to be applicable to Resnet networks.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new method for reconstructing parts of the training data from a given trained homogeneous binary classification network without any further prior knowledge. It relies on a theoretical result by Lyu and Li as well as Ji and Telgarsky, who showed that the gradient flow of the training loss of such a network converges in direction to parameters that solve a maximum margin problem. The optimality condition to the latter is used to phrase an optimization problem for recovering training data on the margin. The resulting approach is tested in various contexts for a 2d toy example, a binary MNIST image classification, and a binary CIFAR-10 classification, and demonstrates significant abilities to recover some parts of the training data. 
strengths_and_weaknesses: To my mind this is a really nice paper, because it exploited a theoretical result to propose a completely new method for reconstructing the data a network has been trained on, which yields significantly better results than competing model inversion schemes. I agree with the authors that a significant amount of training data can be reconstructed surprisingly well. While the investigated network architectures as well as the allowed settings (requiring all training examples to be classified correctly such that the loss converges to zero) remain limited, I still believe the results to be very significant as they lead to several further questions, including why the optimization problems works well at all, how many training data points typically are on the margin, and if training data points that are not on the margin are protected from possible recovery attacks as well as possible further characterizations for stochastic gradient descent, other training methods, multi-class classification network, or regression networks. It reveals interesting properties of the trained weights which is why I consider this to be a strong paper. 

In terms of weaknesses, the paper is of course limited to a specific setting, which is, however, understandable considering it is the first work in this direction. Besides this, I only have some minor points that I am phrasing as questions below.
questions: - The training images to be reconstructed are initialized from a zero mean Gaussian distribution, i.e. roughly half of the values being negative besides an explicit regularization term the tries to enforce a [0,1] box constraint. This seems contradictory. Is it not possible to initialize with a uniform distribution in [0,1]^d?
- Constraints (such as lambda \geq 0 or the abovementioned box constraint) are implemented as penalties instead of running projected gradient descent (i.e. enforcing hard constraints explicitly). Is there a specific reason?
- To be very picky I suggest to write (5) as $\lambda_i (y_i \phi(\tilde{\theta};x_i)-1) = 0$ because strictly speaking I do not think $\lambda_i$ has to be greater than 0 if $y_i \phi(\tilde{\theta};x_i)=1$. And in line 159 I suggest not to use the word "converge" in "we converge to some point $\theta$" because the training setting is designed such that no minimizer exists (the norm of $\theta$ should slowly go to infinity and it can only converge in direction). 
limitations: Yes, there are several limitations in terms of the investigated network architectures and a number of open research questions, but the authors have adequately addressed them.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper gives a principled and novel scheme for reconstructing training set image only use parameters of a network. The approach is elegant and principled and the results are convincing.
strengths_and_weaknesses: Strengths: Principled and elegant approach, with convincing experimental results

Weakness: One thing I am wondering about is other more complicated network architectures, especially those that have been used in practice today. For example, how about we apply this to CNN? What will happen? Note that even CNN should be regarded as a "baby version".

So to this end, I do have some concerns about the experiments regarding effectiveness of the approach on state-of-the-art neural networks
questions: 1. Being curious, have the authors considered other network architectures? (well I feel it is natural to consider them, and maybe the author have already tried the algorithm on these networks?) If not, why?

2. If a network is trained with differential privacy (there are plenty work study differentially private neural network training), how well would this approach work? Note that, it seems to me that, successful reconstruction seems to be in contradiction with DP 
limitations: Potential negative societal impact because the adversaries can reconstruct training data just from network parameters. But this is for good reasons because it means this phenomenon should receive attention and one should defense mechanisms
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper deals with an interesting problem. Reconstructing training data from weights. Mostly, it has been attempted to reconstruct data based on logit outputs only. The paper has far reaching impact on privacy. The paper proposes a method and demonstrates its effectivenss on layer MLPs.
strengths_and_weaknesses: + The problem is highly relevant.
+ Showing that training data can be accurately reconstructed from weights would be ground breaking and have far reaching practical implications with respect to privacy.
+ The optimization procedure is interesting with some nice tweeks. 
- Unfortunately, and this is a key aspect, the paper falls short in evaluation. Evaluating on a practical network and a practical training setup (in terms of amount of data) is essential for the method to have far reaching impact and also be convincing. Since for simple architectures as those used in the paper trivial reconstructions (e.g. just visualizing weights) give already good results, the additional value of the method seems limited and it is also not clear whether it works at all in a practical setup - see detailed comments.

Detailed comments:
- Evaluation should be on a reasonable architecture: not just a two layer MLP, e.g., a CNN like ResNet or VGG. The reason why I am saying this, is for linear networks, it is known that weights might resemble class averages/ prototypes - the authors state this themselves (Figure 5b) - another reference showing this in a different context is, e.g., https://arxiv.org/abs/2202.08299. So it appears that 2 layer MLPs encode samples / class prototypes. The authors somewhat confirm this in Figure 5b, though admittedly their reconstructions are clearly superior to showing weights only. However, it is not clear, whether the approach would work on deep networks, i.e., is the optimization only good for fine-tuning already good prototypes? Would it fail (completely) if they did not exist? Where failure could be that only a very small fraction (I am talking 1/10^6 or so) of reconstructed images are actual training data. The authors themselves state that not only optimization attempts are successful and given that classifiers are subject to adversarial samples, the confidence in the generalizability of the findings to more realistic networks is very low. 
- Evaluation should use a larger training dataset: The authors use only 500 samples. If they used just one sample per class, it would be quite likely that it resembles a class-average, i.e. that sample. It is not obvious whether their method works if trained on 5000 or 50000 samples.
- "not an actual examples" -> not actual samples
- we use the original test sets of MNIST/CIFAR10 with 10000/8000 images respectively, and labeled accordingl -> Cifar10 has 10000 test samples, does not it?
questions: Could you add the needed evaluation (see above) or explain why the method works well in a practical setup?
limitations: I recommend that the author have a dedicated paragraph or subsection named "Limitations", where they discuss them.
review_title: null

Meta-review: metareview: This paper proposed a new algorithm to reconstruct a subset of training examples from a trained homogeneous binary classification neural network. Although there are still some limitations such as the zero training loss and homogeneity assumption, as well as limited experiments beyond MLPs, the reviewers also acknowledge that this results is very interesting and reveals an important property of deep neural networks that could potentially have far-reaching implications for privacy and security.
confidence: Certain
Decision: Accept