Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes DeepCTRL, a method for infusing domain knowledge in the form of rules into the representation learning models. The method independently encodes the input instance and the supplied rule and couples the input and rule encodings where the coupling is controlled by a parameter \alpha. Training the model involves a task-specific loss function and a “rule-satisfaction” loss function which are again coupled by the same parameter \alpha. Experiments on three tasks related to Physics, Forecasting, and Health domain demonstrate several benefits of the proposed approach which include (i) ability to specify the domain knowledge, (ii) More control on coupling the information from rule and data during the test time (iii) More interpretability. 
However, the experiments in this paper seem to be very simplistic and lack exhaustive comparisons with the prior work and thus it’s difficult to judge the practical utility of the proposed method (E.g. all the experiments involve just one rule).  
main_review: Pros:
- This paper considers an important problem of injecting domain knowledge into representation learning models. 
- The proposed method allows easy adaptation of rules to new domains and more controllability in the information coming from rules vs data, during inference.
- Experiments cover multiple domains like Physics, Health and Forecasting (however the datasets seem to be very simplistic)

Limitations:
- Lack of comparisons with the relevant baselines: In section 4.1, I believe [1] is an important baseline, which also tries to combine domain knowledge along with data for learning better models for predicting damped pendulum dynamics. The is a large body of work around learning from rules e.g. [2] and [3]. How does this work compares with these methods? I think the paper should include some more baselines since there are a lot of papers around learning from rules.
- All the experiments deal with just one rule and the datasets seem to be very simplistic. How well does this method do when we have significantly more rules?

Questions:
- In Figure 4a, why is Task-Only better than DeepCTRL for \alpha=0? Should both the methods differ significantly for \alpha=0?

References:

[1] Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting, ICLR 2021

[2] Learning from Rules Generalizing Labeled Exemplars, ICLR 2020

[3] Snorkel: Fast training set generation for information extraction, VLDB


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to integrate a combination of rule learning and data learning in the deep neural networks. The core motivation is to introduce prior knowledge and/or other learning methods and learning how to "balance" the pure data-driven part of the neural network with its rule learning component. The final neural network has two components: the rule-based encoder and the data-driven encoder that are combined by a parametrized convex combination. The loss function has also a counterpart that captures the traditional loss function with penalizations on the rule violations. The second part is similar to the Lagrangian approach to training neural networks subject to constraints that has been used successfully in prior (mostly cited) work. The novelty is the introduction of the rule encoder inside the neural network, as examplified by the third application. The paper also proposes an interesting perturbation approach to capture some of the rule behavior. 
main_review: It is an interesting paper giving an interesting direction. Some of the work described is a direct consequence of prior work on incorporating constraints in the objective function. But the paper generalizes this with two ideas
    -- the idea of integrating rule learning directly in the neural networks
    -- the idea of local perturbation, which is quite nice
These ideas are an interesting avenue for research in a variety of applications.

There are some weaknesses in the paper. The paper lists a number of limitations  of prior work
  (1)  the Lagrangian multipliers are fixed;
  (2) the loss function for the "constraint" part must be differentiable
  (3) the components cannot be balanced at inference time.

(1) is not correct and is addressed in prior work. The approach here also suffers from (2). (3) is addressed by changing how to weight the rule and data encoders during inference. It is not clear what the justification is here since the network is trained with a different weight.


limitations_and_societal_impact: The paper mentions some of its limitations at the end. The paper should mention that the rule-based encoder may codify existing biases in some possible applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes DeepCTRL, a novel training method to integrate rules into neural networks. DeepCTRL is evaluated through several applications showing how rules can be effectively incorporated and improve performance.
main_review: Overall, the idea proposed is clear and yields strong results.

Fig 4 makes a compelling case that DeepCTRL works and is useful. One question about Fig 4 (especially 4c): it seems to make the case that DeepCTRL does a better job of enforcing the soft rule than LDF. It is surprising then that in 5b and 5c DeepCTRL continues to outperform LDF even when imposing the rule would ostensibly hurt perfromance. Do the authors have an explanation for why this might be?

The results in Fig 3 do not clearly show that DeepCTRL outperforms the LDF-ratio baseline. In this case, LDF-Ratio seems to perform the best, even without following the energy damping rule. Even if its verification ratio is lower, this example does not clearly show that DeepCTRL is outperforming LDF-Ratio.

The ablation studies give reasonable confidence that some of the calls made in DeepCTRL's methodology are sound.

One relatively simple baseline that comes to mind is using a convex combination of two classifiers: the rule-based model and a neural network model. This should still preserve many of the proposed useful properties (e.g. selecting the relative strength of the rule-model at test time). Is there a reason the proposed model is better than this simple baseline and does not need to be compared to it?

**Minor points.**

- grammar in first sentece of abstract is slightly unclear
- fig 1 caption - "DEEPCTRL **provide** a controllable mechanism"
- fig 1 itself is faily confusing - what do the blue arrows represent? Why isn't the physical rule of  monotonicity obeyed by any of the curves (it seems to be even less obeyed for the bottom plot, where the rule strength is ostensibly the highest)?
limitations_and_societal_impact: The authors do not seem to explicitly address limitations and societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposed an interesting way of mixing the power of data and rules while enabling the adaptation at the inference time. This paper proposed to use perturbation of the input data points as a novel way of injection rules compared with the continuous relaxation of rules.
main_review: This learning framework as well as the perturbation-based rule injection looks very interesting and novel to me. First of all, combining rule-based prior knowledge with learned knowledge is an interesting and promising direction for improving neural networks' generalization performance. The proposed perturbation-based rule injection is very novel to me. Most existing works inject rules using their continuous relaxation. The perturbation-based rule injection provides a new way to inject a broader range of rules that are potentially hard to relax. 
Second, the idea of controlling the importance weights between data (learned knowledge) and rules is quite novel. Though it is simple from hindsight, it does enable the control of a trained model and provides some interpretability to the model itself (rather than a pure black box).  Moreover, the control of the weights between data and rules can help to solve out-of-sample data points (as shown in their experiments).
Overall I think this paper deserves a score of 8.  I recommend accepting this paper.


One suggestion: I hope the author gives a broader literature survey for methods that try to merge the power of data and rule/symbolic knowledge.

Here are some related works that I know:

Deep Reasoning Networks for Unsupervised Pattern De-mixing with Constraint Reasoning
A semantic loss function for deep learning with symbolic knowledge

Questions:
It looks quite counterintuitive to concatenate the two embeddings z_d and z_r while scaling them by alpha and (1-alpha). People often use this kind of “convex combination of two” on the summation operator rather than concatenation. Did you try if you use z = alpha * z_d + (1-alpha) *z_r? 

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper considers deep learning models for applications where rules are important (e.g. physics). The idea is to construct a "rule encoder" to incorporate user-specified rules. Specifically, this integration is done by perturbing the observations and enforce some user-specified rules to "regulate" between the model's outputs on the original & perturbed inputs. 

Reviewers conclude that the proposed approach is novel and the machine learning tasks considered in the paper are important. Initially they have concerns on missing comparisons with other approaches. This is partly addressed by further experimental results provided by the author feedback. 

Personally I think the paper can be an interesting presentation to the NeurIPS community. I find this paper also connected to existing works related to constrained optimisation/control and adversarial training: these works also consider regularising networks to satisfy some constraints and/or make sure the network returns similar results on perturbed inputs. It would be useful to discuss them in the camera ready.
Decision: Accept (Poster)