Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies semi-supervised classification when the labels are not missing at random. In this scenario, the labeled and unlabeled data cannot be considered as from the same distribution, and the existing semi-supervised learning algorithms face issues such as over-learning on popular classes and ignoring rare classes. The authors introduced a modification of the pseudo-rectifying procedure by tracking the class transitions. The rationale is that if the transition happens frequently between classes then we would like the algorithm to "memorize" it and avoid being over-confident on the imputed labels. The authors illustrate this idea with careful explanations and a number of numerical examples. The improvement on real data experiments is also impressive. 
strength_and_weaknesses: Strength:

- I enjoy reading this paper. It does a great job illustrating why each step is designed in this way and what effect it has on the training process. This paper has no theory, but the authors managed to explain the rationale using numerical examples from many different aspects. 

- The methodology is simple, intuitive, and easy to implement. The improvement on real data experiments is also quite significant. 

- I agree with the authors that it is a strong assumption that the labeled and unlabeled data are from the same distribution. The setting of missing not at random and knowing little about the missing scheme is quite common in practice. I think the authors studied a meaningful problem, and I am glad to see that they provided a simple approach that works so effectively. 


Weaknesses:

- The major weakness is that this approach lacks theoretical justification. Although the re-weighting scheme has a good explanation, we have no idea whether it will have issues in some cases. For example, it is unclear to me why the particular ${\bf H}$ should be the correct choice of weights. I was imagining that a monotone transformation of entries of ${\bf H}$ may also serve as weights, and some likelihood-ratio based weights are more natural choices. To answer this question, we need to at least have some basic theoretical understanding. For example, at least in some very simple settings (e.g., linear classification settings, squared loss in FitMatch), we wish to know what the algorithm does in each update. 

- The methodological innovation is not super large. As mentioned in the paper, the data-set level pseudo-rectifying guidance has been proposed in the literature. Using a class-level pseudo-rectifying guidance is more or less the same kind of ideas (but I do appreciate the simplicity of this approach and how it nicely fits the missing-not-at-random and imbalanced class settings). 



clarity,_quality,_novelty_and_reproducibility: Clarify: 

The paper is clearly written and easy to understand. 
Please add value bars for those heat map plots of the class transition tracking matrices. 

Quality and Novelty: 

I like this simple and intuitive idea and appreciate the numerical work. However, the methodological innovation is fair, and the paper lacks a theoretical justification of the proposed procedure. 

Reproducibility: 

The authors provide an anonymous link for downloading code. This is great!
summary_of_the_review: This is a well-motivated and well-written paper. The approach is not super novel, but it seems to have a lot of practical values. I think this is an okay paper. Some theoretical justification will greatly strengthen this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors focus on a challenging scenario called MNAR where the labeled and unlabeled data fall into different class distributions, which is different from traditional semi-supervised learning. In order to address the problem, the authors introduce a new technique called Pseudo-Rectifying Guidance (PRG). Using Markov random walk to get the class-level guidance information, PRG fully utilizes the history of each class's transition and makes the model concentrate on neglected classes more, therefore improves the performance of previous SSL models.
strength_and_weaknesses: Strength:
1.	The paper explains the necessity of solving MNAR problem clearly.
2.	The paper proposes a novel method for MNAR. The idea of focusing on label transition is quite interesting.
3.	The paper shows extensive experiments and the experimental results in the synthetic class-imbalanced settings demonstrated the effectiveness of PRG.
Weakness:
1.	The paper is a little hard to understand, especially for Section 3.1. The meaning of the symbol \xi is not clearly explained, and this symbol seems to disappear in pseudo-code although it appears in the main body.
2.	Figure 3 is the most important picture in the article ,but it is confusing. Although the style of the figure is nice, it does not help me understand PRG. There is no need to place a black box on (1,1). 

clarity,_quality,_novelty_and_reproducibility: The writing and figures in this paper should be improved. The novelty of the paper is quite well. The authors give an anonymous link of the code.
summary_of_the_review: I tend to accept this paper since the problem is meaningful and the method show plenty of novelty. The main weaknesses are writing and figures, which can be improved in order to help readers understand the idea more quickly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A practical and challenging scenario called label Missing Not At Random (MNAR) is usually ignored in previous works on semi-supervised learning (SSL). In MNAR, the labeled and unlabeled data fall into different class distributions resulting in biased label imputation, which deteriorates the performance of SSL models. In this work, class transition tracking based Pseudo-Rectifying Guidance (PRG) is devised for MNAR. PRG unifies the history information of each class transition to activate the model’s enthusiasm for neglected classes, so as the quality of pseudo-labels on both popular classes and rare classes in MNAR could be improved.
strength_and_weaknesses: Strengths
1. This paper studies an important and practical problem in semi-supervised learning, which has been not studied much.
2. This paper is written very well. The motivation of research is presented clearly along with the limitation of existing approaches.
3. It is good to represent an existing approach (Sohn et al., 2020) as the framework of PRG and explain why it fails at MNAR scenarios.

Weaknesses
1. The paper claims to solve the MNAR problem, but I think it solves only one part of the problem: when the distribution of labeled data is balanced. How well does PRG perform in the opposite setting, i.e., balanced labeled but imbalanced unlabeled data?
2. The intuitions for Equations (6) and (7) are insufficient, although they are the core techniques of the proposed approach. Why do we need to use H_{ij}’ instead of H_{ij}? Why do we need the L vector to reweight the transition matrix?
3. I have some concerns on the experiments. Please see the following: 
    1. I think the main competitor of PRG is CADR, which studies the same MNAR problem. However, the performance of CADR is presented only at Table 1 and missing from Table 2 and 3, and Figure 6. Why?
    2. In Table 1, the improvement of PRG from CADR is not always significant. In CIFAR-100 with \gamma=50, the accuracy of PRG is lower than that of CADR. When \gamma=200, the improvement is negligible considering the standard deviation. I don’t expect PRT to beat CDPR in all cases, but more explanations and analysis on the result would be helpful.

clarity,_quality,_novelty_and_reproducibility: I think is paper is clearly written with a strong motivation, novel ideas, and sufficient reproducibility.
summary_of_the_review: I liked reading this paper, and I think it proposes good ideas to an important problem. My main concerns are that (a) this paper solves only a part of the MNAR problem, and (b) the accuracy improvement is not as strong as the authors claim in the paper, and there are not sufficient analysis and explanation on the result, i.e., Table 1.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles semi-supervised learning with a focus on the most challenging setting of when there is a mismatch between the distributions of labelled and unlabelled data. The proposed method, PRG, uses the history of class transitions to sway the pseudo-labelling mechanism towards under-represented classes. This works by exploiting pairwise similarities between classes: classes that are often confused must be similar (with respect to the predictions of the model) and thus, if the model predicts one of them, we should also consider the others as possible pseudo-labels with certain probability. The authors validate PRG on CIFAR and mini-Imagenet with promising results, outperforming previous similar methods.
strength_and_weaknesses: ### Strength
- The paper proposes a simple and seemingly effective method that improves previous approaches for semi-supervised learning with imbalanced and mismatched class distributions. That is a relevant contribution that, I believe, would be useful both in practical applications and future research.
- Empirically the proposed method works well, outperforming previous competing methods by significant margins.

### Weaknesses
- The motivation of the proposed methods is somewhat weak. 
    - The definition of the transition matrix $\bf H$ is not very well motivated. It is not entire clear to me why Equation (6) is preferable to simply estimating $\bf H$ from the transitions observed in a batch of data. At least that would be a simple and telling baseline to compare against that, I believe, would add to the paper.
    - The authors say "the history of class transitions can point the way for bias removal on label imputation with an abnormal propensity on different classes caused by mismatched distributions in MNAR" but only hint as to why that is the case. We are told PRG works because it encourages the model to predict under-represented classes, but that does not necessarily require a history of transitions nor does it attenuate "over-learning the samples of the labeled popular classes". Empirically, PRG seems to work well, supporting the authors' claim to some extent, but in the current version of the paper the connection between the history of transitions and mismatched class distributions is rather loose. To be clear, I am not suggesting such a connection does not exist. I am just stating that it is not obvious (and thus potentially interesting) and the paper in its current version does not explain it very clearly.
    - The random walk motivation is interesting but feels under-explored. Why take a simple step in the random walk? If we took $k$ steps, i.e., multiply by $\bf H^k$ instead of $\bf H$ in Equation (7), we could uncover more complex patterns of misclassification than simple pairwise class relations.  For instance, if classes A and B are often misclassified as class C, but the model rarely classifies A as B and vice-versa, PRG would not capture this sort of relationship even though the transition matrix does.

- Given that the motivation of the paper is not very grounded, the experiments seem not thorough enough since they are limited to CIFAR and mini-imagenet. CIFAR is notably noisy, which my benefit PRG. It would be interesting to see if the history of transitions is a useful artefact for general types of data other than images. Even simple and easy to run tabular datasets would already contribute to the overall message of the paper.

- The paper is harder to follow than one would expect for such a simple and intuitive approach. Parts of the text are confusing and some of the notation is unnecessary or not properly introduced. I comment on some examples of this in the questions and minor issues below, and in the clarity section.

### Questions
- What exactly is $\xi$ computing? "Rectifying weight vector" is not a very precise description.
- In Equation (6), isn't $\sum_{d=1}^k L_d$ always equal to 1? So the first term on right-hand side of (6) is just the ratio of total number of predictions over the number of predictions that changed class (in a given batch), and is thus always larger than one?
- If $\bf H$ is a proper transition matrix, while do we need to renormalise $p$? Is it because of the $\xi$ term?
- In Figure 7, it is interesting that the performance drops for larger batch sizes. One would expect the estimate of the transition matrix to be improve, which in turn should translate into better or at least similar performance. Do the authors have an idea as why that is the case? Could it be that the learning rate, if kept fixed in the ablation study, was too large for a batch size of 512?

### Minor issues
- The authors use $\max(p)$ as a measure of the model's confidence. That is only one of the possible ways to quantify confidence and that choice should be mentioned more explicitly. Also, the authors imply that the model is more confident ($\max(p)$ is higher) when classifying classes that appear frequently. While this might be intuitive, I think this is an empirical observation that does not always hold for every data point and for arbitrary models, and in particular not for neural networks. If this is indeed an empirical observation, it should be made clear.
- In Figure 4, I cannot see the pattern the authors describe in "the lower right corner (corresponding to the labeled popular classes) of the heatmap is getting lighter and always lighter than the upper left corner." Maybe the Figure is not showing right or the matrices need to be renormalised to show the desired effect?
- Probably best to mention $C_{ii}=0$ already in Equation (5).
- The missing label indicator $m$ is only used once. Not sure it is worth introducing it.
clarity,_quality,_novelty_and_reproducibility: To the best of my knowledge the ideas in the paper are novel, and I have no reason to believe the results are not reproducible. The presentation of the paper leaves something to be desired. The overall motivation is not exposed clearly, as discussed previously, and the text has a number of typos and some rather unclear parts. For instance, I failed to understand the sentences below.
- "The propensity of rectifying the labels belonging to the same class to different classes is different".
- "PRG recalls those “old face” classes once assigned to the unlabeled data".
summary_of_the_review: The main idea of the paper is interesting and seems to work well in a relevant scenario. The methods are rather simple, but I see that as a potential strength, facilitating their application in future work. However, the exposition of the paper is not clear and the main ideas and motivation are under-explored or not conveyed plainly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: An idea based on rectifying guidance is presented for dealing with non MAR missing labels. The depth of the approach and the theoretical justification have not been fully understood, or they have been considered partially missing. The submission did not allow reviewers to be certain about the amount of novelty. In the short reviewing process and with the matched reviewers, the submission was not fully appreciated, and some replies made reviewers question further points which could not be resolved this quickly. Yet, it has become clear that the authors are putting a create effort into a potentially interesting topic, and it is thought that this will lead to strong outcomes. Reproducibility is appreciated.
justification_for_why_not_higher_score: This submission has failed to find interested reviewers and excitement among reviewers. For instance, it has not been clear to the reviewers how much the idea pushes research forward. Attempted clarifications during authors' responses after reviews have made some reviewers more doubtful than they were before.
Decision: Reject