Kati8/ICLR_2023_2025 huggingface dataset:
authors: Danushka Senarathna
authors: Spyros Tragoudas
authors: Kiriti Nagesh Gowda
authors: Mike Schmit
reviews: 
Comments: The article proposes novel methods to detect drift, drift type, and drift magnitude in an image processing data set. The uniqueness of the approach seems to be its ability to deal with varying frequencies of the observed classes. The technical aspects are based on the CDFs of predicted classes and a usage of previous ideas proposed by Senarathna et al. The method is demonstrated on images which are distorted by different types of plausible noise, hence ground truth for evaluation is available in this case.
The article motivates the relevance of the research with a reference to deep vision models and the non-robustness w.r.t. noise.
Strengths: The article deals with a relevant area given that models are delivered to the market and hence need to act in open environments. Further, it integrates an interesting SOTA.
Weaknesses: The article faces the following weaknesses, in my opinion:
- The article fails to give valid mathematical definitions of relevant quantities, specifically what exactly is referred to by drift magnitude? Further, it shares the challenge of many other approaches which fail to give a definition of the underlying mathematical object (mathematically referred to as a drift process in several recent publications which aim for a formal definition of the statistical object dealt with)
- The article describes the novel approach within the text. This makes it very hard to catch the steps. A separate display of the algorithm would be appreciated.
- Properties of the algorithmic steps (including potential guarantees or deeper mathematical insight) are not set apart and not formalized, leaving a mere heuristic approach.
- The types of drift are a bit unclear, especially as regards the claim that every type of drift can be detected. This is not true as it works via the supervision and according classes, see e.g. the theoretical insight and proofs provided in https://www.scitepress.org/Papers/2023/117975/117975.pdf
- The relevance of the results is not clear. On the one hand, all data are theoretical one and no real observed drift is included. On the other hand, the benefit of detecting and specifically quantifying drift is unclear, as drift can have unpredictable consequences which do not necessarily depend on its magnitude.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper addresses issues of drift type and magnitude detection in the image classification task, which is an interesting topic. A learning method that relies on handling the changeable data has been proposed with ideal experimental results. However the learning setting and novelty are not well addressed. There are some suggestions for further improvement.
Strengths: The topic of drift type and magnitude detection in the image classification task is interesting, and sufficient experiment has been given for model evaluation.
Weaknesses: 1. The problem definition is not clear, since the definition only reflects the probability change without considering the timestamp, which is an important term in data stream learning.
2. The threshold setting for drift magnitude identification still need more novelty.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: As the title specifies, the paper proposes a method for detecting drift in image data. It separates the process into the phases of detecting whether drift is present, classification of the type of drift, and estimation of the magnitude of the drift. They are certianly not the only researchers working on this.
Strengths: I find it hard to see any strengths in this work. A not very well-known algorithm for drift detection in images has been improved. That is certainly positive.
Weaknesses: Their only point of comparison is a method published by Senarathna et al. 2023 at a conference on high-performance switching and routing. The work improves upon the previous method by bringing in a method for estimating a class distribution that published in 2010 in a journal on political science. The choice of these non-standard techniques makes it hard to position this work in the context of the current state-of-the-art of drift detection, in particular as there is also no comparison to other, better known works in this area.

The paper is also not very precise in formulating its claims. The first section (2 pages!) is a very general introduction to concept drift detection that does not contribute much to a deeper understanding. This place could have been used by maybe bringing in some results from the appendix, which are frequently referenced and discussed in the main paper, but not part of the main paper. I judged the paper as a stand-alone contribution, and did not look at the appendix, which made it hard to follow the discussion.

The presented evaluation is rather straight-forward. Three standard datasets have been trained with standard neural network architectures, and drift was only simulated. The sole point of comparison is the above-mentioned work by Senarathna et al., with which I am not familiar (p.9, e.g., contains 7 references to that method, no reference to any other method). In any csae, just one comparison is not sufficiently convincing.

In addition, the paper is also not carefully written and/or proof-read. There are many very odd mistakes. For example, on p.7, after the three network architectures have been specified, there is an incomplete sentence "The ResNet50 architecture (He et al. (2016))." I am not sure what this refers to, probably it is the network that has been used to train on the CIFAR 100 dataset. Or the phrase "in occur" in the conclusions.

Minor comments:

  - "drift" does not have a plural, as far as I know.

  - "cumulative sum": Is there also a non-cumulative sum?

  - The authors use the symbol for set intersection (\cap) for denoting logical conjunction (\wedge). There is clearly a relation, but this choice is nevertheless confusing.
Recommendation: 1
Reviewer confidence: 2
Soundness/correctness: 2
Clarity: 1