Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The authors discuss several ideas aimed at improved semi-supervised learning by adopting an appropriate "plate model" with probabilistic content, and then examining various techniques and variants. The theme is relevant but the whole effort seems a bit preliminary: there are many keywords and many discussed techniques, but the whole picture is not clear in terms of concrete contributions (and the provided testing does not clarify whether gains are realized). 

The paper contains a somewhat long introduction that in a sense includes Sections 1 and 2, then quickly goes through the proposed Expression (6), and derives consequences that are not always clearly articulated; for instance what is the point of Table 1? Also the connection with neuro-symbolic learning is interesting but it feels a bit too much; why exactly is it needed in this framework? Or is it just an optional add-on? (Besides, for the proposed approach to work I believe more testing is needed.)

A problem: as far as I can tell, Figure 1 left and center are identical. What is the difference?

A problem: is the citation (Van Engelen and Hoos 2020) indeed in the references? I could not find it.


review_title: Relevant theme but still preliminary: many ideas but still scattered results.

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Summary

This paper proposes a probabilistic model to describe semi-supervised/unsupervised learning, which is further applied to model neuro-symbolic learning. Comparing to traditional unsupervised/semi-supervised learning formulations, the proposed model imposes a prior on the label distribution instead of input features. When applying this formulation to neuro-symbolic learning, the symbolic part can be regarded as a prior on label space to constrain the learning process. Finally, the authors propose three methods to calculate the loss of violating the symbolic prior constraints on label space.
  
#### Pros

+ This paper is motivated by a very important problem. Combining statistical learning and symbolic reasoning becomes a trend recently, many algorithms and models have been proposed. However, there lacks a unified theoretical framework to understand this combination fundamentally.
+ Modelling neuro-symbolic learning as a traditional semi-supervised statistical learning problem is an interesting idea. The resulted model integrates symbolic constraints as a prior on label distribution, which is very natural.

#### Cons
- The writing of this paper could be improved. For example, the notation in this paper is sometimes confusing. The meaning of parameter $\theta^x$ is not clearly explained: In section 2, it is explained as a parameter of the conditional distribution $p(\mathrm{y}|x)$, so I took it as the parameters of a classifier $y=h_\theta^x(x)$. But in Section 4 and 5, the authors state that $\mathbf{y}$ and $\theta$ has a 1-1 corresponds in the space of $\{0,1\}^K$, so it seems that $\theta^x$ should be interpreted as pseudo-labels of examples. In section 3, the authors introduce another variable $\tilde{\theta^x}$ as estimations of $\theta^x$, which makes me more confused. Although this is a theoretical paper, I think the authors should improve the notations and provide some simple examples for helping readers understand this framework.
- The idea of using logic rules as constraints (or other non-differentiable constraints) on label space is not a novel idea in statistical learning (Zhang and Zhou, 2013). In section 3, the first category of "deterministic, distinct classes" is multi-class learning; the "deterministic non-exclusive features" is multi-label learning. In both contexts, there are many works talking about constraining the label space with prior knowledge, for example, Cesa-Bianchi et al. (2006) learns multi-label classifier with hierarchical labels; Tsochantaridis et al. (2004) proposes SVM-struct deal with structural output with support vector machines; Mei et al. (2014) introduces logic rules as label constraints within a Bayesian learning framework. The authors have surveyed a wide range of related works in the area of semi-supervised learning and neuro-symbolic learning. However, I think the works I have mentioned above is highly related to this work; the authors should discuss the relationship between this work and those early works in statistical learning.
- Last but not least, this paper lacks experiments to support the authors' claims. Meanwhile, although this paper presents a general framework, it provides little insight into this area. The authors propose three methods to compute the non-differentiable $p(\theta)$, while (B) ignores the relationship between rules (e.g., recursive theories) and treat rules independently; (C) is equivalent to multi-label learning setting or the ECOC method in multi-class learning.
  
#### Recommendation

Overall, I appreciate that this paper is trying to solve a crucial problem in neuro-symbolic learning, and the idea of viewing this problem as semi-supervised statistical learning is natural and reasonable. However, this paper should include more discussion about related works in statistical learning. Furthermore, it would be better if the authors can include some examples and experiments to demonstrate the idea of the proposed model.

#### Additional questions and comments

- The first to sub-figures in Fig. 1 are the same; the generative model should have arrows from $y$ and $\theta$ to $x$.
- In section 3, first paragraph, line 6: "... outputting $\theta^x$, an estimate of $\theta^x$". The first $\theta^x$ should be $\tilde{\theta^x}$.
- $\alpha$ first appears in Eq. 6 and Fig. 1. However, there is no explanation about what is it.
  

#### References
- Zhang, Min-Ling, and Zhi-Hua Zhou. "A review on multi-label learning algorithms." IEEE transactions on knowledge and data engineering 26.8 (2013): 1819-1837.
- Cesa-Bianchi, Nicolò, Claudio Gentile, and Luca Zaniboni. "Incremental algorithms for hierarchical classification." Journal of Machine Learning Research 7.Jan (2006): 31-54.
- Tsochantaridis, Ioannis, et al. "Support vector machine learning for interdependent and structured output spaces." Proceedings of the twenty-first international conference on Machine learning. 2004.
- Mei, Shike, Jun Zhu, and Jerry Zhu. "Robust regbayes: Selectively incorporating first-order logic domain knowledge into bayesian models." International Conference on Machine Learning. 2014.


review_title: Recommendation to Reject

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper aims at proposing a theoretical rationale for discriminative semi-supervised learning that is comparable with that of generative models. Moreover,
the paper aims at theoretically justifying a family of neuro-symbolic SSL approaches.
For the first task, the paper states that the proposal justifies entropy minimisation (Grandvalet &
Bengio, 2005), mutual exclusivity (Sajjadi et al., 2016a; Xu et al., 2018) and pseudo-labelling (Lee,
2013).
For the second task, the paper states that the proposal justifies  (Serafini & Garcez, 2016; van Krieken et al., 2019, Marra et al., 2019, Xu et al., 2018).

With respect to the first task, while I agree the the paper provides a justification for mutual exclusivity (Sajjadi et al., 2016a) 
and pseudo-labelling (Lee, 2013), I have doubts on entropy minimisation (Grandvalet & Bengio, 2005):
how do you model an entropy with terms of the form $-\sum\log p(\theta|\alpha)$?
The authors also say that previous discriminative SSL lack theoretical justification but (Xu et al., 2018) proves that their loss is a consequence of a 
number of assumptions, thus theoretically justifying it.

The paper misunderstood the scope of neuro-symbolic integration (NeSy): it is not the integration of statistics and logic but the integration of 
neural networks with (possibly probabilistic) logic. The integration of statistics and logic is well established in the field of
Statistical Relational Learning.  For example, (De Raedt et al., 2007) is not NeSy while (Manhaeve et al., 2018) is.
As such, the paper does not provide "a theoretically principled understanding of integrating ‘connectionist’ and ‘symbolic’ methods." in general, only
for some specific methods (not for example for (Manhaeve et al., 2018)). In fact, the justification is only for NeSy that are based on fuzzy logic, 
which is much simpler to manipulate than probabilistic logic as in (Manhaeve et al., 2018). Fuzzy logic is not a probabilistic logic because the
first is truth functional (the value of A and B depends only on the values of A and B) while the latter is not (A and B depends on whether A and B are
independent, mutually exclusive,...). Moreover, (Xu et al., 2018) is more than simply impositive mutual exclusivity as it  also allow a form of probabilistic reasoning by 
representing the loss formula as an arithmetic circuit obtained with automated reasoning. So overall I think that the paper claims more than it effectively provides.

The presentation of the paper makes following the exposition unnecessarily difficult. 
First, the notation is confusing, with a nonuniform representation
of vectors, which are sometimes bold and sometimes not ($\mathbb{z}$ and $\theta$), while sometimes also scalars are bold ($\mathbb{z}_k$).
Moreover, random variables and their values are also not typographically distinct.
All integrals have the integrating variables as subscripts of the integral symbol instead of as $dxdy$...
In Figure 1 the left and center subfigures are equal, I guess in the left one x and y should be exchanged.
Figure 1 left should represent the model of eq 1 but $\psi$ and $\pi$ are absent from the figure.

In eq. 6, shouldn't the integrating variable be $\tilde{\theta}^X$?
It is not clear how you derive the formulas [Gen.] from eq 1: since Figure 1 does not show parameters
$\psi$ and $\pi$, it is difficult to judge the conditional independences among variables.
The symbol $\Delta^{2^K}$ is confusing since it can also be interpreted as the simplex over a set with $2^K$ elements. Better
$(\Delta^2)^K$.
You say that Figure 3 represents a mix of two univariate Gaussians but the variance of the two Gaussians do not seem equal: if only
the mean changes, then the Gaussians should have the same height.
The paper mentions Table 4 which is absent from the paper.

Minor comments:
"proportionality constant obsolete"->"proportionality constant irrelevant"
A the bottom of page 6, last formula: the subscript of the sum should be $\mathbb{y}$
Page 7 "Note that each $\theta\in\{0,1\}^K\subseteq \Delta^{2^K}$ (from a continuous space): $\theta$ does not seem from a continuous space.
Bibliography: reference Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE
Transactions on Neural Networks, 20(3):542–542, 2009. is a book review, you want to cite the book, not the review
In Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for
deep learning with symbolic knowledge. In International Conference on Machine Learning, 2018.
the last author is Guy van den Broeck


---After reading the other reviews and the authors' comments, I sill think that the paper promises more than it delivers, even if the paper was extensively rewritten as a consequence of many problems in the original version, so I will keep my score.
review_title: promises more than it delivers

final_score: rating: 7: Good paper, accept
review_content: The authors introduce a discriminative model for semi-supervised learning for which several existing methods are special cases. In their model, for each data value, there is a distribution from which the label is sampled. Although this distribution is unknown, in their framework the sampling distribution's parameters are approximately produced by a discriminative model such as a neural network trained on the labeled data.

When the labels are vectors of features, the prior distribution of this parameter distribution can be defined so as to enforce logical constraints about which combinations of features are valid. This is a neat direction and in keeping with recent trends to integrate statistical and logical reasoning, although the paper would be strengthened if the authors gave concrete examples of a real-world dataset for which their innovation would be helpful. Likewise for their overall model, it's nice to help unify previous work, but additional discussion of impact or potential applications beyond what's been done would strengthen the paper.
review_title: Discriminative semi-supervised learning

Meta-review: Reviewers agree that this is a very promising paper, with an excellent overview of existing techniques for semi-supervised and neuro-symbolic learning. However, reviewers also agree that the paper is not ready. With one more revision for clarity, some limited empirical validation and illustration of the theory, and focus on the essential message, this could become a seminal paper for our understanding of semi-supervised learning. Luckily the reviews provided ample feedback, and the authors should be able to submit a very competitive paper next time around.
Decision: Reject