Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a deep reasoning networks for de-mixing overlapping patterns with some logic constraints. There are two applications considered in the paper: de-mixing overlapping hand-written digits and inferring crystal structures of materials from X-ray diffraction data. The experiments indicate the proposed method work pretty well on these tasks.

I like the general idea of this paper, since it has the flavor of combining deep learning with logic rules, although I feel weird to view the generative decoder as thinking fast and the reasoning modules as thinking slow. The notion of thinking fast and slow in the model does not well match the intuition given in the first paragraph of the introduction. The so-called reasoning module is essentially some contraints (i.e., regualrization losses) and a training data sampler. It is far away from the concept of (symbolic or logic) reasoning. There is not too much reasoning happening here. The way the paper relaxes the discrete logic constraints to continuous and differentiable objective that can be jointly optimized by SGD is interesting, which is similar to [Harnessing deep neural networks with logic rules, ACL 2016]. The carefully designed training data sampler that samples data according to a constraint graph also resembles GraphRNN, as the authors have mentioned in the paper. I feel the combination of these techniques is definitely interesting but also somehow incremental. I am not a big fan of some big claims in the paper. The reasoning modules are not what I expect.

For the experiments, I think the authors do a good job presenting these experimental details and evaluations. These experiments are interesting and also show some advantages of the propose method. However, some baselines are also doing pretty well, indicating that the task is not difficult in general.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work proposes a framework for solving de-mixing problems. The hard constraints from human inputs about a specific problem are relaxed into continuous constraints (the "slow" reasoning part), and a reconstruction loss measures the fitness of the inferred labels with the observations (the "fast" pattern recognition part). Due to the relaxation inference becomes an optimization problem, and on a Sudoku task and a crystal-structure-phase-mapping recovery task (both de-mixing tasks), the proposed method gets very good performance (100% for all Sudoku tasks including one in the appendix).

Pros:
1. The method works well for the two demixing tasks.
2. It "led to the discovery of a new material that is important for solar fuels technology"

Cons:
1. The generative decoder seems to be pretrained on both tasks instead of learned (correct me if I misunderstood), and I'm not sure if this approach can work in cases where we don't have access to such a generative decoder, so branding the approach "deep reasoning network" might be an overclaim.
2. No reasonable baselines are used: The supervised baseline in Sudoku does not use those handcrafted constraints at all. Given pretrained decoders, a reasonable baseline would be randomized optimization methods such as simulated annealing, which might also solve the two tasks listed here.
3. This paper proposes a deep reasoning framework with relaxation and continuous optimization, but it is unclear whether this can solve general reasoning problems such as multi-hop QA or some NP-hard integer programming problems.

Questions:
1. In algorithm 1, how are the penalty weights and thresholds adjusted?
2. How to determine whether a run needs to restart?

Overall this work points an interesting direction of combining reasoning and pattern recognition in the same network and the proposal works well on two de-mixing problems. However, I am not convinced that the proposed solution can generalize to tasks other than the tasks proposed here, and the usage of pretrained generative decoders undermines the significance of this work. Therefore, I am inclined to reject this paper.


---updates after reading authors' rebuttal----
Thanks for revising the paper and addressing my concerns! However, my concern Con #2 has not been fully addressed. I think a reasonable baseline (at least for Sudoku) is simulated annealing, such as in https://www.researchgate.net/publication/220704743_Sudoku_Using_Parallel_Simulated_Annealing. I believe that with restarts those baselines would also solve the Sudoku problem.

Another concern I still have is the claim of "reasoning", and I'd suggest to narrow down the claim to be only on pattern de-mixing, since the reasoning part seems to be writing down continuous constraints from the discrete constraints (same as the concern in review #3). Although the proposed approach can solve some NP-C integer programming problems, it is unclear based on the experiments here whether it can work for general reasoning tasks (e.g., DROP https://allennlp.org/drop or listops https://arxiv.org/pdf/1804.06028.pdf) without writing new rules manually.

Besides, after reading Reviewer 3's comments, I also feel it unsuitable to train DRNet (generalization) on test set for 25 epochs even though you made it explicit in the revised paper. I'd recommend removing that experiment since it doesn't change this work that much.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new encoder-decoder framework that combines prior knowledge-based regularization and constrained reconstruction for unsupervised and weakly-supervised classification in structure rich scenarios. This framework injects prior knowledge in the form of relaxed constraints that act as regularization during the training of the encoder network. Some of the constraints concern sets of training examples. In this case, the paper proposes corresponding sampling schemes. Three experiments demonstrate the efficacy of the model. The first is a synthetically created 4x4 Sudoku made of overlaid MNIST digits. The other two are based on predicting crystal structures from x-ray diffraction measurements. Here, the first experiment is on simulated data for the Al-Li-Fe oxide system, while the other is performed on real measurements for the Bi-Cu-V oxide system.

Overall, I believe that the proposed framework could be a significant contribution to the fields of representation learning and constrained optimization. However, the paper exhibits serious shortcomings, which require revision. 

First, the positive aspects of the paper:
•	The framework is simple yet ingenious. It makes intelligent use of constraints in the form of regularization to guide the training of the encoder. Furthermore, it enables the direct design of the latent representation through the use of (pre-trained) generative models for constrained reconstruction of data points.
•	The proposed entropy-based method for relaxation of discrete constraints is intuitive and potentially adaptable for further constraints.
•	The experiments presented in this paper are well chosen. They demonstrate the contribution of the model to both general CV data as well as a specialized domain, where it can solve both simulated and real scenarios.
•	The paper provides an extensive literature survey, which makes it easy to embed the presented work in the proper context. However, I propose to remove the paragraph titled “Other less closely related work” as the connection to the current work is not clear, and the space could be used more effectively (see below).

Unfortunately, this paper has a couple of major flaws:
•	The results for DRNets (Generalization) on the MNIST Sudoku are compromised because the model trained on the test set for 25 epochs after being trained on the training set. Honestly, I was baffled to read the following sentence in the appendix: “Note that, during the test, instead of predicting the overlapping digits directly as other networks, we further optimize DRNets on the test set for 25 epochs to achieve a better result.” What is more, the main paper does not even mention this fact!
•	Although this paper relies on empirical verification of its proposition, the experimental results are almost impossible to interpret with just the information provided in the paper. Both experiments are poorly described, and even after reading the appendix several times, some serious detective work was necessary to piece together what happened in the experiments. The XRD experiments are especially hard to decipher, even with a physics background. Many vital components remain shrouded in mystery: What is a composition graph, and how are the paths sampled from it? How does the restart method, which is part of the results, work? Why are only six phases shown in the phase concentration visualizations if there were 159 possible phases? Are these the first six, a random subset, or were the other phases not realized?
•	The paper introduces the constraint-aware SGD algorithm to incorporate batching rules into the training of the encoder. On pages 2 and 6 and in Algorithm 1, I found the statement that the weights for each constrained are updated dynamically. However, that is where the information on the dynamic update method ends. Nowhere in the paper or the appendix did I find an explanation of how this is done. As this mechanism is a critical component of the proposed framework, the absence of an explanation is a significant oversight.
Other remarks:
•	In the context of global constraints, the paper talks about a constraint graph. If I understand the creation of this graph correctly, this graph has several connected components in which every element connects to every other element. As such, this seems to be a collection of sets rather than a real graph. This is especially confusing in the case of XRD, where all data points are in the same global constraint, leading to a fully connected “graph”.
•	Although I appreciate the reference to Kahneman’s model of the mind, I suggest to remove the first two paragraphs from the introduction and use the space to motivate the de-mixing problem instead. While it is a compelling (but not novel) observation, the analogy to system 1 and system 2 does not benefit the proposed work in the slightest.
•	In general, I fail to see the connection between reasoning and the proposed work. The model itself is an encoder-decoder network that cannot reason. It does not discover any new rules during training. All the reasoning has to be done manually beforehand to be then incorporated in the form of constraints. To clarify, I do believe that there is value in the presented work, but not necessarily in the way, it is advertised.

review_title: Official Blind Review #3

Meta-review: The paper received mixed reviews of WR (R1), WR (R2) and WA (R3). AC has carefully read all the reviews/rebuttal/comments and examined the paper. AC agrees with R1 and R2's concerns, specifically around overclaiming around reasoning. Also AC was unnerved, as was R2 and R3, by the notion of continuing to train on the test set (and found the rebuttal unconvincing on this point). Overall, the AC feels this paper cannot be accepted. The authors should remove the unsupported/overly bold claims in their paper and incorporate the constructive suggestions from the reviewers in a revised version of the paper.
Decision: Reject