Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors consider active deep learning. They propose decomposing predictive entropy into a) vacuity (lack of evidence) and b) dissonance (contradictory evidence). They frame this in terms of "subjective logic". In practice this is achieved by having the NN output the parameters of a Dirichlet, which allows an additional degree of freedom describing variance/vacuity. Dissonance is defined in terms of the support of contradictory classes. To get improved estimates of vacuity they augment the loss with a term regularizing the Dirichlet parameters to be small (low precision) at unlabelled points with higher KDE(unlablled points) than KDE(labelled points). They propose initially weighting vacuity and later dissonance as AL proceeds. Encouraging results are presented on simulated 2D data, MNIST and CIFAR10. 

I think the basic idea of separating vacuity and dissonance is interesting, and the demonstration of the failings of existing ADL approaches is valuable. It wasn't clear to me how this "subjective logic" theory gets you to the specific definitions of vacuity/dissonance, or whether these were just proposed by the authors. Equation 6 seems to come out of nowhere (whereas the rest of the derivations using the Dirichlet are very intuitive). 

The idea of encouraging the network to be uncertain far from data is also reasonable. While some Bayesian models such as Gaussian process regression with a RBF kernel give you this for free, it is certainly true that DL methods do not have this characteristic in general. Regularizing the r to be small seems like a reasonable way to do this, but I'm not convinced by the kernel density estimate part. DNNs can operate on very high-dimensional, structured inputs. Even the simplest of these, images, requires some degree of spatial invariance (achieved using convolutions) to obtain meaningful predictions. I find it very hard to believe a KDE can do anything meaningful in such spaces, even if you could find a sensible bandwidth (which isn't discussed at all). It is possible of course that random selection of unlabelled points to regularize in this way would work just as well. Unfortunately no ablation study is performed, so we don't know what the individual contribution of the three proposals (moving from vacuity to dissonance, augmented loss and Dirichlet likelihood) is. 

How sensitive is the method to the vacuity/dissonance weighting? 

The improvement over competing methods for MNIST and CIFAR10 appears to mostly manifest after the initial 20 or so acquistions.

Do these results extend to batch AL? For many applications that's more important. 

Overall I thought this paper had some promising ideas but they need to be more thoroughly tested empirically to give some sense of how robust and generalizable the approach is. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper propose an active deep learning model. By leveraging subjective Logic, they propose to decompose the entropy of a predicted class distribution into vacuity (lack of evidence) and dissonance (conflict of strong evidence). Instead of using the predicted class distribution, they estimate the supporting evidence for each class. In the actual data sampling stage, they first sample from those high-vacuity dense region, to shape the true decision boundary, and then gradually sample from those high-dissonance region to fine-tune the decision boundary. They show better performance than the baselines on both synthetic and real datasets. 

First of all, for the readers who are not familiar with Subjective Logic or probabilistic logic in general, it is a bit hard to follow the reasoning behind the  equations in Sec 3 and  4. Specifically, what is the intuition behind the dissonance of an opinion in Eq (6)? What is the advantage of using Subjective logic framework. 

In the experiments on MNIST, it would be best to visualize the image samples selected by the active learning model in the early and later stages, so that we can have a more intuitive understanding of vacuity vs. dissonance. 
review_title: Official Blind Review #1

Meta-review: The authors propose a new perspective on active learning by borrowing concepts from subjective logic. In particular, they model uncertainty as a combination of dissonance and vacuity; two orthogonal forms of uncertainty that may invite additional labels for different reasons. The concepts introduced are not specific to deep learning but are generally applicable. Experiments on 2d data and a couple standard datasets are provided.

The derivation of the model is intuitive but it's not clear that it is "better" than any other intuitively derived model for active learning. With the field of active learning having such a long history, the field has moved towards a standard of expecting theoretical guarantees to distinguish a new method from the rest; this paper provides none. Instead anecdotal examples and small experiments are performed. Like other reviews, I am extremely skeptical about the use of KDE which is known to have essentially no inferential ability in high dimensions (such as in deep learning situations where presumably images are involved). It is hard not to feel as though deep learning is somewhat of a red herring in this paper. 

I recommend the authors lean into understanding the method from a perspective beyond anecdotes and experiments if they wish for this method to gain traction. 
Decision: Reject