Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: SUMMARY 

The paper proposes a new form of regularization utilizing logical constraints. The semantic loss function is built on the exploitation of symbolic knowledge extracted from data and connecting the logical constraints to the outputs of a neural network. The use of Boolean logic as a constraint provides a secondary regularization term to prevent over-fitting and improve predictions. The benefit of using the function is found primarily with semi-supervised tasks where data is partially unlabelled. The logical constraints provided by the semantic loss function allow for improved classification of unlabeled data.
Output constraints for the semantic loss function are represented with one-hot encoding, prefer- ence rankings, and paths in a grid. These three different output constraints are designed to explore different learning purposes. The semantic function was tested on both semi-supervised classifica- tion tasks as well as structure learning. The paper primarily focuses on the one-hot encoding constraint as it is viewed as a capable technique for multi-class classification.

POSITIVES 

In terms of structure, the paper was written very well. Sufficient background information was con- veyed which helped in understanding the proposed semantic loss function. A thorough breakdown is also carried out on the semantic loss function itself by explaining its axioms which help explain how the outputs of a neural network match a given constraint.
As a scientific contribution, I would say results from the experiments were able to justify the proposal of the semantic loss function. The function was able to perform better than most other implementations for semi-supervised learning tasks, and the function was tested on multiple datasets. The paper also made use of testing the function against other notable machine learning approaches, and in most cases the function performed better, but this usually was confined to semi-supervised learning tasks. During supervised learning tasks the function did not perform markedly better than older implementations. Given that, the semantic loss function did prove to be a seemingly simple approach to improving semi-supervised classification tasks.
• The background section covers the knowledge required in understanding the semantic loss function. The paper also clearly explains the meaning for some of the notation used in the definitions.
• Experiments which clearly show the benefit of using the semantic loss function. Multiple experiment types were done as well which showed evidence of the broad applicability of the function.
• In depth description of the definitions, axioms, and propositions of the semantic loss function.
• A large number of experiments exploring the usefulness of the function for multiple learning tasks, and on multiple datasets.

NEGATIVES 

I was not clear if the logical constraints are to be instantiated before learning, i.e. they are defined by hand prior to being implemented in the neural network. This is a pretty important question and drastically changes the nature of the learning process. Beyond that complaint, the paper did not suffer from any critical issues. There were some issues with spelling, and the section titled ’Algorithm’ fails to clearly define a complete algorithm using the semantic loss function. It would have helped to have two algorithms. One defining the pipeline for the semantic loss function, and another showing the implementation of the function in a machine learning framework. The semantic loss function found success only in cases were the learning task was semi-supervised, and not in cases of total supervised learning. This is not a true negative, but an observation on the effectiveness of the function.

- A few typos in the paper.
- The axioms for the semantic loss function where defined but there seemed to be a lack of a clear algorithm provided showing the pipeline implementation of the semantic loss function.
- While the semantic loss function does improve learning performance in most cases, the im- provements are confined to semi-supervised learning tasks, and with the MNIST dataset another methodology, Ladder Nets, was able to outperform the semantic loss function.

RELATED WORK

The paper proposed that logic constraints applied to the output of neural networks have the capacity to improve semi-supervised classification tasks as well as finding the shortest path. In the introduction, the paper lists Zhiting Hu et al. paper titled Harnessing Deep Neural Networks with Logic Rules as an example of a similar approach. Hu et al. paper utilized logic constraints in conjunction with neural nets as well. A key difference was that Hu et al. applied their network architecture to supervised classification tasks. Since the performance of the current papers semantic loss function with supervised tasks did not improve upon other methods, it may benefit to utilize the research by Hu et al. as a means of direct comparison for supervised learning tasks, and possibly incorporate their methods with the semantic loss function in order to improve upon supervised learning tasks.

CONCLUSION

Given the success of the semantic loss function with semi-supervised tasks, I would accept this paper. The semantic loss was able to improve learning with respect to the tested datasets, and the paper clearly described the properties of the functions. The paper would benefit by including a more concrete algorithm describing the flow of data through a given neural net to the semantic loss function, as well as the process by which the semantic loss function constrains the data based on propositional logic, but in general this complaint is more nit picking. The semantic loss function and the experiments which tested the function showed clearly that there is a benefit to this research and there are areas for it to improve.
review_title: Deep learning with symbolic information

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper suggest a method for including symbolic knowledge into the learning process. The symbolic knowledge is given as logical constraints which characterize the space of legal solutions.   This knowledge is "injected" into the learning process by augmenting the loss function with a symbolic-loss term that, in addition to the traditional loss, increases the probability of legal states (which also includes incorrect, yet legal, predictions). 

Overall the idea is interesting, but the paper does not seem ready for publication.  The  idea of semantic-loss function is appealing and is nicely motivated in the paper, however the practical aspects of how it can be applied are extremely vague and hard to understand.  Specifically, the authors define it over all assignments to the output variables that satisfy the constraints. For any non-trivial prediction problem, this would be at least computationally challenging. The author discuss it briefly mentioning a method by Darwiche-2003, but do not offer much intuition or analysis beyond that.  Their experiments focus on multiclass classification, which implicitly has a "one-vs.-all" constraint, although it's not clear why defining a formal loss function is needed (instead of just taking the argmax of the multiclass net), and even beyond that - why would it result in such significant improvements (when there are a few annotated data points)?   

The more interesting case is where the loss needs to decompose over the parts of a structural decision, where symbolic knowledge can help  constrain the output space. This has been addressed in the literature (e.g., [1], [2]) it's not clear why the authors don't compare to these models, or even attempt any meaningful evaluation.


[1] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural
networks with logic rules. ACL, 2016.

[2]Posterior Regularization for Structured Latent Variable Models.  Ganchev et-al 2010.
review_title: Overall the idea is interesting, but the paper does not seem ready for publication. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a new loss function that is directed to take into account Boolean constraints involving the variables of a classification problem. This is a nice idea, and certainly relevant. The authors clearly describe their problem, and overall the paper is well presented. The contributions are a loss function derived from a set of axioms, and experiments indicating that this loss function captures some valuable elements of the input. This is a valid contribution, and the paper certainly has some significant strengths.

Concerning the loss function, I find the whole derivation a bit distracting and unnecessary. Here we have some axioms, that are not simple when taken together, and that collectively imply a loss function that makes intuitive sense by itself. Well, why not just open the paper with Definition 1, and try to justify this definition on the basis of its properties. The discussion of axioms is just something that will create debate over questionable assumptions. Also it is frustrating to see some axioms in the main text, and some axioms in the appendix (why this division?). 

After presenting the loss function, the authors consider some applications. They are nicely presented; overall the gains are promising but not that great when compared to the state of the art --- they suggest that the proposed semantic loss makes sense. However I find that the proposal is still in search of a "killer app". Overall, I find that the whole proposal seems a bit premature and in need of more work on applications (the work on axiomatics is fine as long as it has something to add).

Concerning the text, a few questions/suggestions:
- Before Lemma 3, "this allows..." is the "this including the other axioms in the appendix?
- In Section 4, line 3: I suppose that the constraint is just creating a problem with a class containing several labels, not really a multi-label classification problem (?).
- The beginning of Section 4.1 is not very clear. By reading it, I feel that the best way to handle the unlabeled data would be to add a direct penalty term forcing the unlabeled points to receive a label. Is this fair?
- Page 6: "a mor methodological"... should it be "a more methodical"?
- There are problems with capitalization in the references. Also some references miss page numbers and some do not even indicate what they are (journal papers, conference papers, arxiv, etc).

review_title: The paper proposes a new loss function that penalizes semantic features of the data, and shows some experiments; overall the writing is good and the ideas are nice, even though the contribution is relatively small.

Meta-review: This one was really on the fence.  After some additional rounds of discussion post-rebuttal with the reviewers I think the general consensus is that it's a good paper and almost there but not quite ready for acceptance at this time.  A detailed list of issues and concerns below.

PROS:
1. good idea: an additional loss term that enforces semantic constraints on the network output (like exactly 1 output element must be 1).
2. well written generally
3. a nice variety of different experiments

CONS:
1. paper organization.  The authors start with the axioms they would like a semantic loss function to obey, then provide a general definition, then show it does obey the axioms.  The general definition is intractable in a naive implementation.  The authors use boolean circuits to tractably solve the problem but this isn't discussed enough and it's unreasonable to expect readers to just give a pass on it without some more background.  I personally would prefer an organization that presented the motivation (in english) for the loss definition; then the  definition with a description of its pieces and why they are there; then a short discussion of how to implement such a loss in practice using boolean circuits (or if this is too much put it in the appendix); and a pointer to the axiomatization in an appendix.

2. related to 1, I didn't see anything which discussed the training time of this approach.  Given that the semantic loss has to be computed in a more involved way than usual, it's not clear whether it is practical.
Decision: Reject