Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overall comments: Novel formulation and good experimental results. The method is claimed to be a general solution to PU settings with biased positives. However, there might be potential issues related to identifiability of the posterior from the labels. Mathematical writing lacks rigor.
 
Strengths:
1) The formulation is Novel.
2) Experiments show superior performance then the state of the art.

Weaknesses:
1) The algorithms for class prior estimation in PU learning are only derived for the no bias case. Thus class prior estimated using biased positives can be wildly incorrect. And consequently, it can introduce significant bias in the learnt posterior.
2) The new loss function is a function of the posterior and the propensity function which are both learnt from the data. As implied by the authors it can’t alone be used to learn the true posterior as there is a trivial solution to the loss which assigns q_i=1 for all i and g(x_1) = l_i. The authors add terms corresponding to the prior matching cost and observation matching cost to solve the issue. However, it seems that the observation matching cost is not helpful in that regard since it is minimum when q_i*g(x_i)=l_i and consequently q_i = 1 and g(x_i) = l_i. 
3) In general, it seems that q_i and posterior cannot be uniquely identified from the labeled data. The true posterior is the minimizer of the new PU loss only if the estimated q_i indeed corresponds to the true labeling mechanism. Is it not possible that the both estimated q_i and the estimated posterior are far away from the true labeling mechanism and the true posterior, respectively? In my opinion, a consistent estimate of the true posterior is not feasible, unless assumptions are made on the nature of the bias. 
4) Lack of rigor in mathematical writing: g_\theta(X)=Py(Y|X) implies that  g_\theta is a function of the sequence, but used as the posterior at a given x in the sequence. It would be better to have separate notations for the theoretical quantities and their estimates. In equation 2 shouldn’t it be Bern(Pr(\hat{y}=1|x)). G_\theta is defined twice, as the posterior probability and also as the output of the model. In reality it is the output of the model and the probability is what it is intended to estimate. OMC is not just a function of X it is also a function of the label. Please define OMC completely as a summation over the dataset. J is used as a function of g_\theta and also as a function of Q_\phi.  q_i is used in two different ways: one where y_i is given and the other where y_i is not given. 
5) It will be useful to mention that the loss functions belong to the proper loss family so that g(x) indeed corresponds to the posterior [1].
6) Please give details on how are hyperparameters \lambda_1 and \lambda_2 are tuned.

[1] Reid MD, Williamson RC. Composite binary losses. The Journal of Machine Learning Research. 2010 Dec 1;11:2387-422.

review_title: The paper presents a novel algorithm for binary classification of partially labeled inputs that come in a form of a sequence under a biased labeling mechanism. A novel loss function is derived by incorporating the propensity score in the standard PU loss formulation which is shown to reduce to the PN loss in expectation when the estimated q_i is equal to the true q_i. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**: This paper makes the observation that sequential labelling processes introduce a bias in the obtained labels. This is relevant for weakly supervised sequential settings (eg human activity recognition), where examples are either assigned a positive label or are unlabelled. For this setting, a novel method is introduced that models this (latent) labelling process and uses the latent labels in the objective function,  this results in better performance on two datasets. While the observation of sequential bias is interesting and the performance is promising, I rate the current paper as a 'OK but not good enough': it seems to miss the insights behind the observation of sequential labelling bias and it seems to miss some experiments (ablation study, multi-class, etc).

**Sequential Labelling Bias**:
The observation about sequential bias is interesting and probably valid.
But some claims are doubtful without further backup by experiments or references, for example: "Hence, an instance is more likely to be labeled if its neighbours are also labeled, and vice versa" [P1, first paragraph]. This depends completely on the labelling procedure, eg for Human Activity Recognition: when at random times people are asked what they do now, the labels dont follow this assumption. Also in the next paragraph: it is described that HAR data might have sequential bias, but it is not shown, what the properties of this bias are. Given that this is so important in the current manuscript, I'd like to see more insights:
1. What kind of bias is observed? I'd expect that people like to label in the beginning, and then forget about it later on. So that early in the sequence more labels are provided than later. 
2. How can this biased be modelled? Please show some qualitative and quantitative results from different type of annotation/labelling procedures. Does the bias differ from domain to domain (eg medical vs human activity recognition)?
3. Does labelling bias depend on the annotator/person how labels the sequence?
4. Can the (obtained) propensity score be used for predicting labelling bias. How good is it at this? Or would a temporal smoothing over labels do equally well?

**Problem Setting**
- The problem setting assumes there are sets of sequences. Why is that? What defines such a single sequence? 
- Can the setting be generalised to multi-class  classification? In that case a positive label for one class is a (strong indicator for) a negative label for all other classes.
- x_t seems to be a scalar-value, why is that? Can the model be used with more complex input (eg images)?

**Methodology**
- Figure 1 seems to contain an error: both the SPM and CN always take the learned representation of the feature sequence, not the feature sequence itself. It is also a rather complicated figure, if the only thing to show is that the networks are trained alternately.
- For my understanding it would be nice if the relation with related work is made here more clearly. Where does this method differ from the PN network? Only in the loss? If yes, which loss is used then? 

**Experiments**
- I miss an ablation study for the effect of Rpu, PMC and OMC in the final objective (below Eq 3). What is the influence of adding PMC to the training pipeline? Or is Rpu alone sufficient?
- I miss experiments on multi-class datasets
- I miss the details on the datasets (type of features, number of examples, influence of subsampling etc).

**Conclusion**
To restate again, I think the observation of sequential bias is valid and it has to be present in many (sequential) labelling tasks. While in this paper some form of a solution is provided, I do miss the insights into this bias. I hope these can be provided during the rebuttal phase.

**Post-Rebuttal**
The authors have addressed some of my concerns during the rebuttal phase. Thanks! For me the major drawback of the current manuscript remains that 'sequential bias' is not really defined, and not really showed to exist in the datasets at hand. Only indirectly, by our new method performs better than existing methods. I think this should be improved, to increase the understanding of this topic.
review_title: TLDR; Interesting observation about sequential bias; not sufficiently substantiated (yet)

final_score: rating: 6: Marginally above acceptance threshold
review_content: --- Update after discussion ---

I thank the authors for considering my recommendations. I think the clarity of the updated paper is much improved, particularly the introduction. I further think that the authors have adequately addressed the concerns raised by the other reviewers and recommend accepting the paper.

--- Review ---

Overall, I thought this paper was good, but not great. I found the problem poorly motivated and the contributions limited in scope. The methods were reasonable and appear technically correct, if somewhat ad hoc. I recommend acceptance, though I think the paper could be substantially improved.

--- Comments ---

1. After reading the intro, it was very unclear to me what problem the paper was trying to solve and what the contributions of the paper were. In part, this is because the main problem addressed by the paper, "sequential bias", is not defined until the second page despite being reference earlier and in the abstract. Even the fact that we are dealing with sequential data wasn't made very clear until later. I recommend rewriting the intro to start by defining the problem that the paper solves, something like: Existing methods for learning instance-level classification models from sequential data with positive only labels rely on accurately estimating the conditional probability that an instance receives a label, known as the propensity score. However, these methods generally ignore the fact that positive labels are often temporally clustered together --- referred to as sequential bias --- resulting in inaccurate propensity scores and lower classification performance. We propose a method that ...

2. In section 3, I recommend giving more of a tutorial on how propensity scores are used in non-sequential settings. As it is, I wasn't sure how the propensity scores could be used to recover the true risk (and thus, why they were important) until page 5.

3. The main contribution of this paper doesn't really have anything to do with deep learning and I think tying the method so closely to NNs artificially limits the contributions. Specifically, most of section 4.5 and 4.6 could be presented without any reference to NNs. I recommend moving 4.5 and 4.6 before 4.1 - 4.3 and writing them as a general estimation procedure. 4.1 - 4.3 can then be framed as discussing the specific model specification details used in the paper. Even the name DeepSPU references deep learning when it doesn't really need to.

--- Minor comments ---

1. Page 2, par 2, line 2: "Instance" not yet defined.

2. Page 2, par 4, line 7: Prior-Matching --> Prior-Weighted

3. Page 3, par 3, line 2: I recommend $|\cdot|$ or $length(\cdot)$ rather than $||\cdot||$ to refer to the length of a sequence so as to avoid confusion with a norm.

4. Page 3, par 4, line 5: $y_i^{(j)}$ --> $y_i$

5. Page 3, par 4, line 8: I don't follow the logic that a low propensity score implies a high probability of being a true positive. Since we are conditioning on $x_i$, couldn't it also just be an instance that is obviously negative and thus unlikely to be labeled?

6. Page 3, par 5, line 3: Is there notation defined in the appendix that is not defined somewhere in the main text? If not, I'm not sure I would include this table.

7. Page 4, par 3, line 2: local feature values --> local feature values and the feature values of all preceding instances

8. The notation in the appendix differs from that used in the main paper and has several typos. 
review_title: Ok paper, but writing and organization could be improved

Meta-review: Dear Authors,

Thanks for your detailed feedback to and even communications with the reviewers. Your additional input certainly clarified some of the concerns raised by the reviewers and also improved their understanding of your work.

However, we still think that the notion of sequential bias is unclear, and the authors overclaim what they have done.
For these reasons, this paper cannot be recommended for acceptance.
I hope that the detailed feedback from the reviewers will help improve this work for future publication.

Decision: Reject