Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows several methods on how to use temporal information in stream-based active learning. It shows why classical pool-based AL methods cannot be used in this domain. It also shows that some methods can even outperform pool-based methods in terms of data savings. This paper closes the gap between pool and stream-based active learning. The authors performed several experiments on the public Audi Autonomous Driving Dataset and showed marginal improvements over state-of-the-art approaches by using 1% fewer labels.
strength_and_weaknesses: + The paper is written clearly and easy to follow. 
+ The paper gives a view on how to transform pool-based methods into stream-based methods.
+ The custom dataset can be easily reproduced by the dataset description or downloaded as it is publicly available, and all hyperparameters needed to replicate the results are given.

- related work section can be improved a lot as it does not cover the current state of the art. Good literature overview of existing pool-based query methods and their (dis)advantages. 
An overview of the properties of stream-based data, as opposed to pool-based data, is missing. For example, the non-stationary nature and possible concept drift of stream-based data are very important properties that can be solved by (stream-based) Active Learning.

- Paragraph 2.2 only contains one Stream based active learning technique. There are many more that could be discussed to give a better view of previous research. For example, it would be good to add some info about “RAL - Improving stream-based AL by RL”. There is much more research on stream-based active learning than this paper suggests. 

- The obtained results can be discussed further to show the real value of the proposed approach. In the current form, it is very hard to see why the proposed approach required 1% less data than other ALs (sec. 6.2, and 6.3).
clarity,_quality,_novelty_and_reproducibility: - The paper gives a view on how to transform pool-based methods into stream-based methods. It shows that using temporal changes indeed gives a more diverse sampling strategy than standard pool-based methods. 

- The explanation of the implementation of temporal predicted loss could be improved by supplying an algorithm description.  Very clear explanation of the second method, why it is hypothesized to work well, and how it should be implemented. The TDLS method would be easier to reproduce if a step-by-step algorithm description would be supplied. 

summary_of_the_review: Overall this is a concise paper but some parts could be revised to further enhance reproducibility. 
Algorithm descriptions would be very helpful for the reader. While some related research on stream-based AL is given it does not give a correct representation of the actual research that has been done in this domain. The obtained results should be discussed in more detail to show the usefulness of the proposed approach. In its current form, it is hard to judge its value.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an Active Learning method for AL in a streaming context for an application of AL on edge (sensor) devices such as a fleet of cars or robots. Additionally, they define two contexts with different configurations of pool-based batching and streaming of data. Then, they introduce and evaluate variants of loss learning (Yoo & Kweon, 2019) on a new 4-class classification task defined on the Audi Autonomous Driving Dataset (Geyer et al., 2020) and ResNet-based architectures.
strength_and_weaknesses: **Strengths:**
- Addresses a relevant problem: early pre-processing of data close to sensors

**Weaknesses:**
- Language in-correctness and lack of clarity (see below)
- Motivation of the method (why is the classification of highways vs. construction sites important?) 
- Claims are not well supported by experiments nor by theory

clarity,_quality,_novelty_and_reproducibility: **Clarity/Open Points**
- Lack in language clarity:
   - Distinction of “uncertainty” and “diversity” into “predict one value” and “performing greedy optimization” are not appropriate, why not stick with the known categorizations?
   - References missing, for example: “While early approaches like loss learning or MC dropout are generating a single value per unseen image, state of the art approaches often include unlabeled data for unsupervised training. Other approaches taking diversity into account usually perform a greedy optimization, which requires constant access to the complete unlabeled dataset.
- Correct use of language lacking, e.g., consistency of names (“pool based” vs “stream-based”, pytorch, BatchBALD vs. BatchBald, Active learning/Learning), typos. Please have the text proof-read.
- Motivation is bad: “As classification is still the most commonly benchmarked task and shown by all authors, we focus on classification in our paper.”
   - What is the task in the autonomous driving use-case that you really want to tackle with your work? Detection of highways or construction sites? Events/Incidents?
- Related work
    - If the work cited in Section 2.1 is related work, then please differentiate your method from it. If it is not (this is what I guess) then omit it (many things already have been used to motivate AL and introduced before)
    - Where is the related work section on stream-based Active Learning? Section 2.2 misses a lot of related work on stream-based AL, e.g., Fuji2016, see below.
    - If you use AL to filter a stream of data, should you not also present related work on stream processing such as novelty detection?
- Method:
    - How do you use the time? You only describe the difference in gradients and the distance in the latent space, no temporal information
    - Section 3: Why don’t you compare to “third category methods”? Is it not possible to train VAAL on stored data in a datacenter once and then apply it on a stream on device?
    - Section 4: Construction of dataset for classification: what are the problems that arise from the inconsistent sample rate? Are the frame timestamps later used in the method? 
    - Section 5: You mention the method is sensitive to sudden occurrences. What do you mean?
- Claims that are not supported by experiments:
    - Noise of results in Figure 7 is high. Results shown are specific picks (here: orders) to support the claim, but do not show summary statistics. Also, results have high variance and difference between methods is small, drowned by noise.
    - Figure 4a: why are the data points not equidistant on the x-axis, if you select 5% of the samples for labeling?
    - Figure 6a,b: missing random baseline
- Conclusion: what other tasks do you want to extend the approach to?

**Related work (non-exhaustive list, as a starting point)**
- Fuji et al., 2016, Budgeted stream-based active learning via adaptive submodular maximization
- Narr et al., 2016, Stream-based Active Learning for efficient and adaptive classification of 3D objects
- DavideCacciarelli et al., 2022, Stream-based active learning with linear models
- Li et al., 2019, Incremental semi-supervised learning on streaming data
- Korycki et al., 2019, Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams
- Luo et al., 2017, Online decision making for stream-based robotic sampling via submodular optimization

**Minor details**
- The paper talks about “1% fewer labels” or “0.5% fewer labels needed” – this was somehow confusing to me as it did sound marginal. In retrospect, I think it would be better to relate this value to the number of samples actually being annotated
- “A collection of different scenarios [is] depicted in”
- “Classification datasets like Cifar10 (Krizhevsky et al., 2009)[,] which is often used to benchmark AL[,] are not temporally ordered data streams.“
- “The scaling factor $\delta$ is chosen to 0.5” $\rightarrow$ “we set the scaling….”

summary_of_the_review: The paper addresses the field of stream-based active learning which is definitely relevant for a lot of field applications where actual systems are being deployed. However, as the paper misses a lot of related work and does not support its central claims neither theoretically nor experimentally, I do not see ICLR as an appropriate venue for this work. Moreover, writing needs to significantly improve, making this paper not mature enough at this point.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applies stream-based active learning to datasets with time stamps. The main idea is to select data points with a large loss change $\left\lVert \frac{d}{dt} \hat{\mathcal{L}} \right\rVert$, where $\hat{\mathcal{L}}$ is an estimated loss. The authors propose three methods by following this policy, and validate their empirical quality in object detection experiments.
strength_and_weaknesses: Strengths:
As mentioned in the paper, stream-based active learning is not much studied compared with pool-based active learning. However, since data are often collected in a streaming fashion (e.g. autonomous driving), the importance of stream-based active learning is increasing. This paper proposes practical ideas for stream-based active learning, and then the results might be used in future applications.

Weaknesses:
The support for the superiority of the proposed algorithms is not so strong. There is no theoretical guarantee, and why the authors choose the proposed criteria (objectives (1) and (2)) is not explained clearly. Although the authors empirically show that the proposed algorithms perform well in some object detection tasks, but only for the specific task with specific datasets. I think more experiments are needed if the authors claim that these algorithms perform well for general stream-based active learning with time stamps.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The description of the algorithm can be improved. Section 5 describes the algorithmic details of the proposed methods, but they are not sufficient. For example, how to apply SieveStreaming++ is not clear. Since SieveStreaming++ is a method for a static set function, the objective function should be updated at an appropriate timing (minor comment: det(FF^T)+I_j should be replaced by det(FF^T+I_j)), but the authors do not address this point.

Novelty:
The novelty of this paper lies in the idea of using $\left\lVert \frac{d}{dt} \hat{\mathcal{L}} \right\rVert$ instead of the loss itself. It is a nice contribution, but not so ground-breaking.

Reproducibility:
The datasets and random seeds are addressed. However, the descriptions of the algorithms are not sufficient to reproduce the experimental results. It is helpful if the authors provide the details of the algorithms (how to estimate the loss function, when the loss estimate is updated, etc).
summary_of_the_review: Stream-based active learning with temporal information is an important problem and the proposed algorithms based on the loss change is practically useful, but it would be better to discuss in more detail the advantages of the proposed algorithms.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies active learning in a stream setup, which assumes to be more useful than the classic pool setup for applications such as autonomous driving and robotics. All three reviewers are slightly negative about the work. This submission has clear values, but overall is a bit immature to be accepted now. More experiments or theories can be added to increase its strength, and the exposition can be made more clear (especially the motivation). 
justification_for_why_not_higher_score: All three reviewers have concerns with either the experiment part a bit weak or lacking a theory. The work is also a bit weak in elucidating the motivation.
Decision: Reject