Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The author proposes C2D2 that relies on the diversity of models to identify invariant patterns related to drift. C2D2's complexity is lower than author’s previous model, and it focuses on capturing confounding variables related to drift by leveraging the ensemble feature space.
strengths_and_weaknesses: ## Strengths:

1. The proposed Collaborative Concept Drift Detection (C2D2) framework provides a new approach for detecting concept drift in ensemble models by leveraging the diversity of models and their feature spaces.
2. The research provides insights into the importance of considering overlapping feature spaces and predictors for drift detection in systems with multiple models.

## Weakness:

1. It is assumed the proposed C2D2 framework is evaluated on synthetic datasets, and its effectiveness on real-world datasets is unknown.
2.  The proposed method assumes that the confounding variables related to drift are invariant across the ensemble of classifiers, which may not always be the case.
3. The study does not provide a comprehensive comparison of the time complexity of C2D2 with other state-of-the-art concept drift detection methods.
suggested_changes: As a next step, the paper could benefit from further comparison with other state-of-the-art methods in terms of time complexity. This would provide a more comprehensive evaluation of the proposed method and help researchers to better understand its strengths and limitations in comparison to other methods.

Please consider expanding FCBF to Fast Correlation Based Filtering in the abstract.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The authors propose C2D2 (Collaborative Concept Drift Detection) to detect invariant patterns that could relate to drift. First, they map features and target spaces into a correlation matrix through Fast Correlation Based Filtering. Then, SVD is used to get the invariant representation and finally a post-hoc is used to indicate the occurrence of drift.
strengths_and_weaknesses: ## Weaknesses:
1. Several phrases in the paper are difficult to follow or not entirely correct:
    1.  Section 1: *"Although the models themselves operate independently of each other, and they hold information that describes the relationship..."*
    2. Section 2: *"Similarly, our notion is to learn from multiple models in order to capture evolving concepts, but rather than triggering drift detection based on similarities between models-to-features and model-to-model relationships ..."*
    3. Section 2.1: *"To fit a model is to take a function f onto a feature vector, x such that it provides ...".* The comma is not needed here. Similarly Section 3.1: *"The correlation of every combination of x within X mapped to every y in Y is calculated by Fast Correlation Based Filtering (FCBF) ( Nguyen et al. (2012)) to form a correlation matrix, A..."*
2. The authors provide a comparison between C2D2 and the author's previous work which is anonymized and not provided in the paper. As a consequence, I cannot compare the two approaches. 
3. Reproducibility: there are a few details that are missing in the paper to reproduce the results. What model and data are the authors using?
suggested_changes: As said in the "Weaknesses" section, I cannot compare C2D2 with the author's previous work. 
More specifically, in Section 4 - Conclusion:
-  *"A limitation of XYZ is that its similarity scores are calculated via shapley values and the complexity explodes in complexity to O(2k ), where k is the entire feature size across all models. Additionally, it has an inherent trade-off between incurring high false alarms and long delays in detection."*
- *"With C2D2, however, the complexity is O(mn) and its constraint lies in its ability to identify how diverse
an ensemble needs to be in order to isolate the invariants to be related only to drift."* Why the complexity is  O(mn)?
review_title: null

Meta-review: metareview: The paper proposes a new perspective for drift detection. The idea of using diversity within an ensemble to detect drift seems intriguing, something that both I and reviewer d6SQ can appreciate.

However, I agree with Reviewer LAyt that anonymizing the main work that the paper compares its methodology to undermines our ability to review the paper — how can reviewers verify the claims of the author if we cannot read the paper that provides the contrast to this approach? A principle of science is to never take something on faith if possible. I can understand the motivation to anonymize; and can commiserate that it is sometimes unclear when submitting a blinded paper what to do.

For peer review, I'm not sure one should ever anonymize references (at all, I imagine, although perhaps there are edge cases I can’t think of), especially those that are necessary for reviewers to verify or understand one’s argument. Instead, you can write the paper such that authorship of particular papers is not claimed in a first-person way (e.g. you can say this paper builds off of X, instead of this paper builds off of our previous work X). For example, when reading this paper, that you were the authors of the paper you are contrasting to does not seem essential to the paper’s message. It is true that reviewers may sometimes guess that someone building off a framework may indeed be the author of that framework, but that is generally acceptable (I suppose it is just an aspect of scientific review culture) as long as one is attempting as best as one can to honor the spirit of blinding a paper. 

Also, it appears as if the paper provides a method but does not have any experimental results? At least I could not find any within the PDF I was able to download. Both reviewers seem to be a bit confused by this (e.g. “what model and data are the authors using” and “It is assumed the proposed C2D2 framework is evaluated on synthetic datasets”). Perhaps there is a reason to propose the method without testing it, but that should likely be explicitly acknowledged and the reason for not running the algorithm should be explained. It is hard to review a proposed method without any results to contextualize what it does in practice — you could perhaps design a simple artificial data set at least as a proof of concept?

So unfortunately, I think that this paper cannot really be CCR in its current state (due centrally to the anonymization of the previous paper, and also due to the lack of empirical results), but the ideas seem interesting and I encourage the authors to keep developing them. And I feel bad that the anonymization approach is one of the main reasons for my recommendation, I can imagine it being disappointing -- my recommendation would be to, if possible, seek a few trusted people (perhaps a few elder grad students) to ask what to do when given a tricky situation with anonymization.
confidence: 3: The area chair is somewhat confident
Decision: Revision accepted; invite to archive