Coherence-based Label Propagation over Time Series for Accelerated Active Learning
Authors: Yooju Shin
Susik Yoon
Sundong Kim
Hwanjun Song
Jae-Gil Lee
Byung Suk Lee
Venue: ICLR 2022 Poster (2022)
Keywords: active learning
time series
pseudo labeling

Abstract: Time-series data are ubiquitous these days, but lack of the labels in time-series data is regarded as a hurdle for its broad applicability. Meanwhile, active learning has been successfully adopted to reduce the labeling efforts in various tasks. Thus, this paper addresses an important issue, time-series active learning. Inspired by the temporal coherence in time-series data, where consecutive data points tend to have the same label, our label propagation framework, called TCLP, automatically assigns a queried label to the data points within an accurately estimated time-series segment, thereby significantly boosting the impact of an individual query. Compared with traditional time-series active learning, TCLP is shown to improve the classification accuracy by up to 7.1 times when only 0.8% of data points in the entire time series are queried for their labels.

Published as a conference paper at ICLR 2022
COHERENCE-BASED LABEL PROPAGATION OVER TIME SERIES FOR ACCELERATED ACTIVE LEARNING
Yooju Shin', Susik Yoon?, Sundong Kim², Hwanjun Song", Jae-Gil Lee Byung Suk Lee" 'KAIST PIICC "Institute for Basic Science "NAVER Lab University of Vermont (yooju. shin, jaegil)ekaist ac.kk, kr, susikellling edu, sundonge ibs re.kr, hwanjun. hwanjun.songenaaeee www.:sseeeeeeerp..ccom bslee@uvm.edh:
ABSTRACT
Time-series data ubiquitous these days, but lack of the labels in time-series data is regarded as a hurdle for its broad applicability. Meanwhile, active learn- ing has been successfully adopted reduce the labeling efforts in various tasks. Thus, this paper addresses an important issue, time-series active learning. Inspired by the temporal coherence in time-series data, where consecutive data points tend to have the same label, our label propagation framework, called TCLP, automat- ically assigns queried label to the data points within an accurately estimated time-series segment, thereby significantly boosting the impact of an individual query. Compared with traditional time-series active learning. TCLP is shown to improve the classification accuracy by up to 7.1 times when only 0.8% of data points in the entire time series are queried for their labels.
INTRODUCTION
A time series is a sequence of data points at successive timestamps. Supervised learning (e.g., clas- sification) with time series requires the label of every data point, but unfortunately labels are often missing and hard to obtain due to lack of domain-specific knowledge (Shen et al. [018 Malhotra cct al. 2019: Li et al. 2020). It is worse for a time series collected for an extended length of time, as manually labeling so many data points is labor-intensive and time-consuming (Perslev et al. 2019
Tonekaboni et al., 2021). Active learning (Settles) 2009), a method that iteratively selects the most informative data point and queries a user for its label, can mitigate the high labeling cost. However, most active learning methods are not geared for time-series data, as they assume that data points are independent of one other (Sener & Savarese 2018 Yoo & Kweon 2019 Ash et al. 2020). which is obviously not true in time-series data.
Time-series data typically has the characteristic of temporal coherence; that is, temporally consecu- tive data points tend to have the same label (Wang et al., 2020) Barrow et al. 2020 Ishikawa et al., 202T]. Let us refer to sub-sequence of temporally coherent data points as a segment. For exam- ple, in motion-sensing time-series data, segment consists of data points with the same motion status (e.g., walking, running). This temporal coherence of segment can be exploited in time-series active learning. Specifically, when the label of certain data point is obtained from user, the same label can be propagated to other data points in the same segment. One challenge here is that the seg- ment length is not known but needs to be estimated. If it is too short, unnecessarily frequent queries are issued; if too long, data points on the fringe of the segment are labeled incorrectly, consequently damaging the learning performance (e.g., classification accuracy). Thus, accurate estimation of the segments is important to enable the label propagation to achieve the maximum learning performance with the minimum number of queries.
This paper addresses the label propagation segment estimation problem in time-series active learn- ing through a novel framework called Temporal Coherence-based Label Propagation (TCLP). Fig- ure [ illustrates the overall workflow in the time-series active learning centered on TCLP. TCLP receives the class probabilities (i.e.. softmax output) for the label of each data point from classifier model and estimates the extent of the segment to propagate the label. This estimation is challenging
"Coreepooniigg author.

---

Published as a conference paper at ICLR 2022
Unlabeled
A few labels added
Labels propagated
Query,
Label Propagation (TCLP)
Query
Classifier output
Classifier ((=;,,)**
8
78
C Classifier ((==0))
Figure I: Overall workflow of time-series active learning centered on TCLP. A data point that is
queried for a label determined by a query selection strategy. The label obtained is then propagated to adjacent data points guided by the TCLP framework.
in a real time series, as the classifier model output is uncertain and the time-series segments are unknown. TCLP meets this challenge by taking advantage of the temporal coherence via a quadratic plateau model (Moltisanti et al. 2019), by fitting it to the classifier model output to smooth out the fluctuations of class probabilities across consecutive data points.
To the best of our knowledge, TCLP is the first that performs label propagation for time-series active learning. The previous work closest to ours pseudo-labeling in single-timestamp supervised learning, where labels are known for at least one data point in each segment (Moltisanti et al. 2019; Ma al. 2020 Li al, 2021). The approximate location and true class of segment must be known in their work, which is often impractical in the real world. Moreover, the known labels are relatively dense in timestam supervised learning, but they are very sparse active learning -tpically, no more than 5% of segments in our experiments. Thus, finding the boundaries between segments active learning than in single-timestamp supervised learning. To cope with the sparsity of labeled data points, TCLP performs sparsity- label propagation by exploiting temperature scaling (Guo et al. (017) and plateau regularization.
more challenging
Contributions of this paper are summarized follows:
proposes a novel time-series active learning framework equipped with a sparsity-aware label propagation within accurately estimated segment.
It verifies the merit of TCLP through extensive experiments. The classification accuracy is im- proved by up to 7.1 times with TCLP compared to without label propagation. Moreover, TCLP works with any query selection strategy including core-set sampling (Sener & Savarese 2018) and BADGE (Ash et al. 2020), boosting the effect of individual labeling.
RELATED WORK
2.1 ACTIVE LEARNING
Active learning is special case of machine learning that 'actively' queries user for the labels of data points, to the effect of using fewer labels to achieve the same learning performance. Recent studies have focused on developing such query strategies for machine learning based on deep neural
networks (Settles 2009 Ren et al. 2020). These approaches exploit prediction probabilities (Beluch et al. 2018 embeddings (Sener & Savarese 2018), gradients (Ash al. 2020), and losses (Yoo & Kweon 2019) from deep neural networks to estimate the impact of each unlabeled data point if it were be labeled. However, these methods are not suitable for time-series data, because they assume that data points are independent.
Several methods have been developed for time-series or sequence data, but most of them are applica- ble to only segmented time-series data under the assumption that a time series is already divided into labeled and unlabeled segments. Treating these segments as independent and identically distributed, these methods simply apply existing active learning frameworks to the segments. For example, HHe] al.) (2015) select unlabeled segments that are far from labeled segments to maximize diversity; Peng et al. (017)) select unlabeled segments with distinctive patterns to maximize diversity; and Zhang et al. (207)) select unlabeled segments with high gradients to consider uncertainty for sen- tence classification. In addition, new neural network architectures measures have been developed for sequence-data applications such as named entity recognition (Shen et al.; 2018). video action recognition (Wang et al.) 2018). and speech recognition (Malhotra al. 2019). None of these meth- ods is applicable to our problem, which handles unsegmented time-series data.
2.2 PSEUDO-LABELING
Exeudo-labeling has been actively studied for label-deficient learning environments, such as semi- supervised learning, to exploit unlabeled data points in training classifier (Lee et al. 2013). In
2

---

Published as a conference paper at ICLR 2022
general, a pseudo-label is given to unlabeled data point based the predictions from classifier trained with labeled data points. Confidence-biie baseeedeeeeeeddddeeee methods create a pseudo-labes if it is confidently predicted by classifier (Lee et al. 2013). Consistency- based methods create pseudo-label if it is consistently predicted for the original and augmented data points (Sajjadi et al. 2016 Rizve et al.) 2021]. Graph-based methods propagate pseudo-labels from labeled data points (nodes) to unlabeled data points based on a similarity graph constructed from the features of all data points (Shi et al.) 2018 Liu et al. 2019 Wagner et al. (0188). However, these methods are not designed for time-series data, and therefore are not directly applicable to our problem.
Coherence- basedd methods are developed for tinlle-imestamp supervised learning for unsegmented time-series data; they assume that at least one data point in each segment is given true class label through beekksupprrvision Ma et al. (2020) propose probability thresholding propagation (PTP), which propagates known labels bidirectionally unless the predicted class probability for each data point decreased by more than a threshold. [Deldari et al. (202T) propose embedding similarity propagation (ESP), which propagates known labels bidirectionally unless the embedding of each data point changes rapidly. Recently, Moltisanti et al. (2019) adopt a plateau model that represents class probabilities across consecutive data points, where plateau model is constructed for each labeled data point and fitted to the classifier output: a known label is propagated as long as the value of a plateau model is higher than a threshold. While this work shares the idea of using plateau model with our work, using the plateau model as it is for active learning results in performance degradation owing to the difference in the density of known labels, will be shown in Section 4
3 TCLP: TEMPORAL COHERENCE-BASEE LABEL PROPAGATION
3.1 PRELIMINARIES AND PROBLEM SETTING
Active learning: Let D= = {( Mt), E T be a time series where T is the index set of timestamps; 2 is multi-dimensione data point at timestamp t, and yt is one of the class labels if x; is labeled or null otherwise. Let Dr S D be a labeled set, i.e., a set of labeled data points, and Du S D be an unlabeled set, i.e., set of unlabeled data points, where Dy U Dr. D. At each round of active learning, b data points are selected from Dy by query selection strategy, such as entropy sampling (Wang& & Shang 2014), core-set selection (Sener & Savarese (018), and BADGE (Ash) iet al. 2020), and their ground-truth labels are obtained from a user; these newly-labeled 6 data points are then removed from Dy and added Dz. After Dz is updated, a classifier model fo is re-trained using the updated labeled set.
Label propagation: Given a data point at timestamp ty and its label, (It, Mr,),, obtained from a user in response to a query, TCLP assigns the label Vt, to nearby data points in the timestamp range [t, : te] (t, S la I (ee) estimated according to its temporal coherence property criteria. We call the sub-sequence of data points in [ts tt] an estimated segment at fgg There are two properties: (i) accuracy, which indicates that as many data points in the segment as possible should have the same ground-truth label Vt,; and (ii) coverage, which indicates that the length of the segment (t. tx) should be as long as possible. More formally, we estimate the segment for tg by
t, = arg min 2 # t t=mt
Σ (1 (1-follee (-
(1)
where t,, < < t' holds, fo(x;) is the softmax output vector of the classifier model at timestamp t, and fo(2,)ll1,) is the estimated probability of the label yt,, In Equation (I) the accuracy is achieved by minimizing the sum of errors in the numerator, and the coverage is achieved by maximizing the candidate segment length in the denominator. Note that estimated probabilities are used to calculate the errors, since the true probabilities are not known.
Once segment estimation is done, all data points in the estimated segment (i.e., in [ts : tel) are re- moved from Dr and added to Dr, with the label yt,, thus doing coherence-baset label propagation. At each round of active learning, the segment estimation repeats for each of the b queried data points; result, the size of Dz is increased by the total length of the estimated segments. Besides, allow the data points in [t. : tzl, except tg. to be queried again in subsequent rounds so that the propagated labels can be refined subsequently.
3

---

Published as a conference paper at ICLR 2022
3.2 PLATEAU MODEL FOR SEGMENT ESTIMATION
An adequately trained classifier model fo returns higher probabilities of the (unknown) true labels for data points inside a segment and lower priorities for data points outside the segment, Besides, the output probabilities of the model are not constant within the segment because of noise in the time-series data. Thus, one natural approach to finding segment is fit plateau model to the output of the classifier model and make a plateau of probability into an estimated segment.
3.2.1 PLATEAU MODEL AND ITS FITTING
Among many functions with a plateau-shaped value, we use the function introduced by Moltisanti al. (2019):
..... Estim. prob.
Plateau
True prob.
Fit
Proh.
2w
Pno
2w
h(t; G w. 8) (e<(t-c-w) + ()(est +1)
Timestamp
Timestamp
Initial.
(b) Optimal.
where c, w, and 5 are trainable parameters for the model h. As shown in Figure 2, and respectively represent Figure 2: Plateau model and its fitting.
the center and half-width of the plateau, and s indicates the steepness of the side slopes.
The fitting of the plateau model timestamp ty is illustrated in Figure 2 At the beginning, in Figure 2h is set to by, and w and are set to initial values and updated by the following optimization
G, W. 8 arg min
Σ
1h(t;'', ,') so(ox))lve,)l
ewt 2w
where lutt; w', = for E [c- w/ of + w']. Letting e(t) be [h(tt;'', w', s') So (xx,) [V,, [vr, 1//2 in Equation (3) and E be E, e(t), the optimal values of the three parameters are obtained by repeat- ing a gradient update step.
c=d -V
w' V E, and s' 17 E. where 17 is the learning rate.
After model is fitted through enough rounds, as shown in Figure [2,, the plateau is located at the center of true segment and its width covers most of the true segment. [c (c-w c+ w]. indi- cated by the red line in Figure 2p. is determined as the estimated segment for the plateau model lg- Overall, as shown in Figures [aguree/Baand]a (3) as active learning progresses, more data points are queried, estimated probabilities becomes more accurate, and plateau models are better fitted to the estimated probabilities. Eventually, the
Fstimated prob.
Plateau
True class
Query
Prob.
(a) Plateau models the initial round.
Prob.
(b) Plateau models after the final round.
Figure 3: Updated plateau models. plateau models accurately represent the true segments in the time series.
3.2.2 SPARSITY-AWARE LABEL PROPAGATION
In active learning, known labels are typically very sparse-initially no more than 5% of the segments and growing slowly as more labels are obtained for queried data points. Our experience indicates that simply optimizing the plateau model as explained in Section 3.2.1 tends to generate plateaus much longer than true segments. See Section 4.4 for the poor performance of the typical (sparsity- unaware) plateau model. On the other hand, in single-timestamp supervised learning, where the plateau model has been successfully employed (Moltisanti et al. 2019), at least one label should exist for every segment; thus, the boundaries of segments the plateaus not overlap between segments. Thus, to overcome the lack of potential boundaries for segment estimation, we extend the procedure of the plateau model fitting in three ways.
be easily recognized while making
Calibrating the classifier output: Because modern neural network models have millions of learning parameters, the distribution of predicted probabilities in such models is often highly skewed to either or O, which means that the model is overconfident (Guo et al.) Müller et al., 2019). That is, the output of the classifier model fo could be too high even outside the true segment, thereby making the
4

---

Published as a conference paper at ICLR 2022
plateau wider than the true segment. We address this issue by employing temperature scaling (Guo let al. 2017) to calibrate the classifier output (i.e., softmax probabilities). Temperature scaling is widely used to mitigate the overconfidence issue, because it reduces the differences of the softmax probabilities while keeping their order. Specifically, temperature scaling divides the logits (inputs to the softmax layer) 2, by learned scale parameter T, i.e., as fo
softmax(z//T))
Regularizing the width of a plateau: To prevent a plateau from rapidly growing wider than a true segment, we constrain the amount of updates on the parameter w of the plateau model. Specifically, w cannot be increased more than twice its current value in a single round of active learning. Accord- ingly, the gradient update of w in Equation (4) is slightly modified to min(2w', w' -1 (er)))))
Balancing the class skewness: This issue of class label imbalance can become more severe by label propagation from sporadic queries; although the propagated labels are correct, the number of such propagated labels can vary across different classes. To reduce the effect of this potential skewness, we re-weight the loss E(je. Me) at each timestamp in training the classifier model fo. where
Ur arg max; fa(x;)] [k] and y/ is the propagated ground-truth label (Johnson & Khoshgoftaar 2019). The loss is adjusted by the inverse ratio of the number of the timestamps of given class over that of the most infrequent class. That is, if we let N₂ be the number of the timestamps assigned with the class k the parameter of the classifier model is updated as follows: O = 0 - A ming N. V Ve(jt. Vt),
No.
where Ả is another learning rate for training classifier model fa.
3.3 THEORETICAL ANALYSIS
We show that our plateau- plateau-based segment estimation is expected to produce a segment closer to the true segment than a simple threshold-based segment estimation. For ease of analysis, we consider a single segment whose true length is L, the query data point at by is located at the center of the true segment with i = yt, known. In addition, we assume that the estimated class probabilities [k]{ < t < L) are conditionally independent at different timestamps (Graves et al. 2006 Chung et al. 2015).
prreshold-basee segment estimation: A simple and straightforward way to estimate the segment is to expand its width bidirectionally as long as the estimated probability at each timestamp is higher than or equal to a threshold 8. The probability that the length of segment reaches / is
Pr(l.. 1, =1) = 21 - =)²
(5)
where = = fre [k]= 2 5). Here, the l multiplied to 2'-1(1 2²² in Equation 5 is the number of alignments possible for segment containing the center tq. As a result, the expected length is
L
Eso(m)) Ite t.l = Σ I. Pr(te -t =1)= Σ
2 2)²
1=1
1 2 (L + 1) 1. + (2L. 2L 1) 122
1 1-2
Plateau-based segment estimation: Let us fix to ty and
(90' steepness). and denote the
plateau model simply as h(t; w). Then, h(t; w) = 1 if c- w S t S c+w, and h(t; w) = otherwise. In addition, for simplicity, let us fix the denominator 2w' in Equation to L. Then, the inside of the argmin operator in Equation (3) becomes
<(l) = 1 (()-))) 10 fo [k]]] + + 11-1111-oooo fo(x))
where (= 2w) denotes the length of the segment estimated with the plateau model. e(I) is linear function of 7, where its slope is (1 11-2/1 -2f (k))))) and < < < L. Thus, Equation (7) evaluates to the minimum at either = 1 when the slope is positive or 1 = L when the slope is negative. Letting 2 be Pr(Ja (x.) [k] 2 0.5), the probabilities of = 1 and / = L are - and z, respectively. In conclusion,
the expected length of the estimated segment is
B Ite tz] = Pr(1- 1) + L Pr(1 L) + L2.
Comparison and discussion: As L increases, Equation 6 converges toward 1+: and is not affected 1- by L_, whereas Equation (8) which equals z(I. 1) increases linearly with I.. Therefore, when
5

---

Published as a conference paper at ICLR 2022
true segment is sufficiently long and is in the typical range (e.g., less than 0.9), the plateau- based segmentation (Equation (8)) is expected to produce a longer (i.e., closer to L) segment than
the threshold-basee segmentation (Equation (6
3.4 OVERALL ACTIVE LEARNING PROCEDURE WITH TCLP
Algorithm Time-series active learning with TCLP
Input:
Timestamp feature 2t, initially labeled set Dz, unlabeled set Du. query strategy Q, number of rounds R, query size b, initial classifier fag, classifier loss f, learning rate X.
Output: Final classifier J0R+1-
I: Hoo + Initialized plateau models for Dz;
2: 81 = 8u- - A min; Ve(ge, v6) for each (It, vr) E DD:
No,
3: for L...., do
4: Yt = TTEMPEATURESCALLNGG/A (x:)) H,, = Ø;
// see Section 3.2.2
5:
for h in HI,-1 do
6:
h/ +- fit on Dr H. + HI, Ut:
// see Section 3.2.1
7:
14 + queried timestamps acquired by Q from Dy; H, H, {plateau model hr, 1
8:
9:
H. f AHI
see Appendix
10:
Dr. + LABELLLOPAGATIONN
|/ see Section 3.2.2
11:
0r+1 0, min; N; Vi(je, ye) for each (æt, Ub) E Dz: Ny.
return fön+1:
Algorithm summarizes how TCLP works time-series active learning. First, plateau models are initialized with the initially labeled set D2 and stored in the set IH, (Line I and then using Dz. TCLP trains the classifier model fe, (Line Then, at each active learning round r. TCLP first per- forms calibration by inferring the data points with the classifier model fo, and scaling the softmax output, and then initializes a new set of plateau models H. (Line 4). Next, each plateau model in -I from the previous round is fitted the scaled output, and then the updated plateau model is added to IH,- (Line 6 The new plateau models are then initialized from queried timestamp la- bels (Line 7 and added IHI, (Line 8). Then, any overlapping plateaus in IiI, adjusted-either merged into one reduced to avoid the overlap
needed (Line 99. Finally, the queried labels are propagated following the plateau models in HI, (Line [Op. and the classifier model fe, is trarrdddd with the augmented labeled set D,, (Line II The complexity analysis of TCLP is presented in Appendix A
4 EVALUATION
We conduct experiments with various active learning settings to test the following hypotheses.
TCLP accelerates active learning methods faster than other label propagation methods can.
TCLP achieves both high accuracy and wide coverage in segment estimation.
TCLP overcomes the label sparsity by the extensions discussed in Section 3.2.2
4.1 EXPERIMENT SETTING
The four benchmark datasets sum- marized in Table are used. 50Salads contains videos at 30 frames per second that capture 25 people preparing a salad (Stein & McKennal 2013). and GTEA contains 15 frame videos of
Datasets:
Table 1: Summary datasets and configurations
Timestamps
Length
Wgg
Ap
50salads
288798
289
2048
200
15
0.5
GTEA
31225
34
11
2048
200
15
5
0.5
four people (Fathi et al.) 2011). For these two video datasets, we extract I3D features of 2,048 dimensional vectors each timestamp follow- ing the previous literature (Farha & Gall 2019). mHealth contains 50Hz sensor time-series record- ings human movement, measured by 3D accelerometers, 3D gyroscopes, 3D magnetometers,
mHHalth
343195
2933
200
15
0.5
and electrocardiograms (Banos et al. 2014); we extract labeled regions from the raw data and stitch

---

Published as a conference paper at ICLR 2022
NOP
PTP
ESP
TCLP
0.5
0.67
0.35
0.76
F1@25
0.45
0.23
0.55
0.17
0.23
0.12
0.33
0.05
0.01
0.05
0.1
0
0.004
0.008
0
0.002
0.004
TS accuracy
0.6
0.63
0.69
0.89
0.43
0.53
0.75
0.26
0.25
0.36
0.62
0.05
0.01
0.05
0.1
0.004
0.008
0
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 4: Classification aceuracy measured at each (1st-15th) round of active learning. The accuracy value is an average over all query selection methods. Detailed results are in Appendix
them in chronological order to make time series. HAPT represents 50Hz sensor time-series record- ings of human actions in laboratory setting, measured by 3D accelerometers and multiple 3D gyro- scopes (Anguita et al. 2013).
To evaluate the efficacy of TCLP, we combine it with six different query selection methods. CONF (Wang& & Shang] (014) selects b timestamps exhibiting the lowest con- fidence in the model's prediction, where the confidence is evaluated by using the largest predicted class probability: MARG (Settles) 2012) is similar to CONF, but it defines the confidence as the difference between the first- and lacond-largest predicted class probabilities; ENTROPY (Wang & Shang 2014) selects the top b timestamps exhibiting the largest entropy for their predicted class probabilities; CS (Sener & Savarese 2018) chooses the top b most representative timestamps in em- bedding space: BADGE (Ash et al.) 2020) computes gradients from fa each timestamp & and queries b timestamps found by k- k-MEANS++ to consider uncertainty and diversity; UTILITY is our simple selection strategy that selects b timestamps randomly from the timestamps not covered by the current set of plateau models, to increase the utility of the plateau models.
Query selection methods:
Compared label propagation methods: For a thorough comparison, we compare TCLP with three available label propagation approaches-NNe -NOP, РТР, and ESP. For this purpose, each of TCLP and the three approaches is combined with each of the aforementioned six query selection methods. NOP is the baseline without using any label propagation. As explained in Section 2.2 PTP propagates labels based on the predicted class probabilities with certain threshold (6 0.8), while ESP leverages cosine similarity between embeddings for label propagation. PTP and ESP are modified to work in an active learning setting as done by Deldari et al. (2021).
We use the multi-stage temporal convolutional network (MS- TCN) (Farha & Gall, 2019) as the classifier fo for time-series data. We use exactly the same training configuration suggested in the original work (Farha & Gall 2019). Regarding active learning hy- perparameters, the number of queried data points per round (b) and the number of active learning rounds (It) summarized in Table I For TCLP, use the initial parameters for plateau mod- els (wy and 5o) in Table I and temperature scaling with T = 2. Our experience indicates that any value of we smaller than 20% of the mean segment length adequate enough. Accuracy met- rics: Timestamp accuracy and segmental Fl score are measured at each round by five-fold cross validation; they represent the prediction accuracy at the granularity of timestamp and segment, re- spectively. The former is defined as the proportion of the timestamps with correct prediction. The latter is defined as the Fl score of segments with an overlapping threshold on the intersection over union (IoU) (Farba & Gall 2019);; that is, prediction is classified as correct if the IoU between pre- dicted and true segments is larger than the threshold. Fl @2,5, with the threshold 25%. is commonly used in the literature (Lea et al. 2017 Farha & Gall 2019); the trends with the thresholds 10% and 50% are similar.
TCLP implementation details:
7

---

Published as a conference paper at ICLR 2022
Table 2: Classification accuracy measured after the final (15th) round (the best results in bold).
Dataset
Query
Timestamp Accuracy
NOP
PTP
ESP
TCLP
NOP
PIP
ESP
TCLP
CONP
0.191=0.015
0.204=0.015
0.28010.017
0,433-00010
0.505=000017
0.451
0.462=0.031
0.559=00010
ENTROPY
0:133-0.004
0.193-0.011
0.263+0.020
0.368 +0,031
D-432-0.019
0416+00.19
0,555=0.024
0,096-0.027
MARG
0.287-0.021
0.359-0.018
0.336+0.031
0.600-0.028
0.616=00003
0.615=0.015
0.637=00.31
0.027=0.020
50salads
C5
0.022-0.021
0.426=0.018
0.480:100022
0.059
0.595
0.602=0.017
0.632:00023
0.657
BADGE
0.197-0.014 0.317-0.822
0.377+0.019
0,471 0.000
0.514-0.023
0.529=0.024
0.567=0.018
0.600-0.025
UTILITY
0.372-0.012
0.482=0.012
0.511=0.028
0.595 0.002
0.625=000228
0.642=0.022
0.669=00026
0.672=00018
AVERAGE
0.250
0.330
0.391110033
0.504
0.518.0.0288
0.51310.035
0.56910.034
0.614.10028
CONF
0.386-0.119
0.297-0.129
0.443+0.100 108
0.020
0.409=0.100
0.321=0.115
0.443=00002
0.654-0.011
ENTROPY
0.355=0.119
0.575=0.032
0.422=0.125
0.613=4.025
0.364=0.114
0.565=0.022
0.456=00100 104
0.590
MARG
0.202-0.072
0.491=0.122
0.58210.035
0.727=0.024
0.120
0.46910.099
0.515.0.022
0.659 0.015
CS
0.636-0.031
0.512-0.125
0.11010.057
0.666
0.60910.026
0.520:0.112
0,59110.033
0.630_1.007
BADGE
0.718-0.041 0.723-00033
0.710=0.029
0.703-10.029
0.705=00.00
0.682=0.010
0.692=00017
0.663-10.015
UTILITY
0.661=0.025
0.671=00000
0.700_0.004
0.656.10.025
0.008:00.018
0,608
0.646:0.019
0.644
AVERAGE
0.504=00004
0.545=00056
0.578:00.046
0.016
0.502=00.059
0,528:000047
0,562:00033
0.640
CONP
0.016=0.005
0.015=0.002
0.029:10.00 006
0.228
0.228:0.066 0.064
0.294:00.016
0.398::00063 0.445
0.685 0.051
ENTROPY
0.008_0.004
0.011=0.001
0.023100005
0.004
0.175.1.0047
0.36310.032
0.413
0.560
MARG
0.018-0.005
0.065-0.017
0.078:100077
1.363-10008
0.305::0.011
0.085
0.5
0.699
mHHcalth
CS
0.065=0.024
0.289=00008
0.187=0.049
0.594 0.094
0.590=00.034
0.680:000032
0.577=0.040
0.807=00009
BADGE
0.101 0,145
0.12910.024
0.054
0.1011000056
0.578.0.0026
0.51510,0049
0.665
UTILITY
0.076-0.006
0.462-0.103
0,43210.00
0.538 0.113
0.752::0001
0.831
0,901:0.0015
0.789:10.065
AVERAGE
0.051=0.018
0.16110.067
0.116.10.057
0.349
0:12010.000
0.56110.061
0.553.0.006
0.702.10.034
CONF
0.12
0.289_0.084
0.15910.050
0.656
0.011
0.592.0.008
0.793.0.002
0.857 0.009
ENTROPY
0.352-00.055
0.368=00062
0.381120.04
0.653
0.616:00.067
0.755=0.022
0.763-00044
0.836 0.011
MARG
0.276-0.043
0.667=00.42
0.790::0.021
0,805 0,012
0.556=00.031
0.799=0.910
0 025
0 10111
НAРT
CS
0.739_0.033
0.816-0.010
0.843:0..06
0.835
0.880-0.005
0.89810.018
0.92610.007
0.909110:911
BADGE
0.784-0.023
0.719-0.044
0.813:0.019
0.860=0.0118
0.859:0.032
0.840:0.0016
0.858:000014
0..05
UTILITY
0.846-00122
0.867-0.018 0.018
0.811=000026
0.849-00002
D. 0.936-00.006
0.937=0.000
0,916==0.011
0.934-00.004
AVERAGE
0.550
0.621 00,09
0.684200076
0.076:0.036
0.748:00.060
0.803=00.046
0.844:00.025
0.885:00013
4.2
OVERALL ACTIVE LEARNING PERFORMANCE
Figure shows the Fl @25 and timestamp accuracy at each round of varying the queried data ra- tio (= the number of queried data points/ / the total number of data points), where the accuracy values are averaged over the six query selection methods. 15 rounds are conducted for each dataset. TCLP performs the best among all label propagation approaches, with the accuracy improving much faster with a smaller number of queries than in the other approaches; this performance is attributed to the larger number of correctly propagated labels in TCLP, as will be shown in Section|4.3 Interestingly, the accuracy gain is higher in Fl @ 25 than in timestamp accuracy; this difference makes sense be- cause Fl @25 measures the accuracy at the granularity of segment and therefore reflects temporal coherence better than the granularity of timestamp. Appendix shows more details.
Table 2 shows the F1@25 and timestamp accuracy measured after the final (15th) round, i.e., at the last queried data ratio in Figure 4 for each of the six query selection methods. TCLP performs best here as well in almost all combinations of datasets, query selection methods, and accuracy metrics. Specifically, TCLP outperforms the compared label propagation approaches (NOP, РTP, and ESP) for all query selection methods except only a few cases. This result confirms that TCLP maintains its performance advantage regardless of the query selection method. Interestingly. TCLP's performance gain is most outstanding for the mHealth dataset and least outstanding for the GTEA dataset. The reason lies in the length of the segments. As shown in Table I mHealth's segments are the longest (2,933 on average) and GTEA's segments are the shortest (34 on average). Longer seg- ments certainly allow more temporal coherence be exploited in label propagation, thus resulting in higher performance. For instance, using UTILITY on the mHealth dataset, TCLP outperforms NOP by 7.1 times, PTP by 1.2 times, and ESP by 1.2 times in F1 @5 while, on the GTEA dataset, TCLP outperforms them less significantly.
4.3 LABEL. PROPAGATION PERFORMANCE
Table 3 shows the correct label propagation ratio (= the number of correctly propagated labels / the total number of data points) to verify how many labels are correctly propagated with each label prop- agation approach. Overall, fully taking advantage of the temporal coherence based the plateau model. TCLP adds far more correct labels than PTP and ESP. Specifically, using UTILITY on the mHealth dataset, the correct propagation ratio of TCLP is higher than that of PTP by 5.0 times and that of ESP by 3.7 times. It is impressive that querying only 0.8% of data points results in up to 33% of data points correctly labeled. Appendix E shows more details.
8

---

Published as a conference paper at ICLR 2022
Table Correct label propagation ratio after the final (15th) round (the best results in bold).
Correct Prepegation Ratio
Correct Propsgation Ratio
Dataser
Query
Dataset
Query
ESP
TCLP
PTP
ESP
TCLP
CONF
0.032+0.001
0.054+00000
0.129-00
CONF
0,027+00001
0.038++0002
0.109-0.017
ENTROPY
0.026:00000
0.042:0.001
0.000-00005
ENTROPY
0.02420.000
0.031::0000]
0.066=0.009
MARG
0.07110.000
0.15110.004
0.368 0.006
MARG
0,05410.001
0.06310.004
0.027
50salads
0.076:20.000
0.151=0.001
0.003
mHealth
CS
0.061+0.000
0.0811:0.001
0.210:00.018
BADGE
0.034:00.001 0.0941:0.004
0.193 0.013
BADGE
0.060110.001
0.0871:0002
0.020
UTILITY
0.081+0.000
0.120+0.002
0.352 0.003
LITILIYY
0,065+000000
0.099+0.001
0.325 0.017
AVERAGE
0.057+0.009
0.110+0.020
0.241
AVERAGE
0.049+0.007
0.088+0.010
0.186-0.034
CONF
0.27010.008
0.097 0.252100008
CONF
0.01110.001
0.019100000
0.050
ENTROPY
0.22610.009 0.220:0.010
0.403=0.018
ENTROPY
0,00910.000
0.01420.000
0.633=0.002
0.4231:0.008
0.280110.002
0.404-0.009
MARG
0.022110.001
0.04710.002
0.007
GTFA
CS
0.437
0.398+00.014
0.371-0.011
HAPT
CS
0.026++0.000
0.048+00002
0.146 0.004
BADGE
0.34820.014 0,305=00013
0.404=0.008
BADGE
0,026:20,000 0.053+0.001
0.160=0.006
UTILITY
0,516:10.002
0.45310.001
0.424=0.006
UTILITY
0.028100.000
0.058100003
0.257
AVERAGE
0.3701100041
0.33510.033
0.016
AVERAGE
0.020:10.003
0.04010.007
0.132:00033
Table 4: Classification timestamp accuracy after the final (15th) round with and without plateau width regularization and temperature scaling (the best results in bold).
Terp Scal
Na -
Yesc"" = 21
7.005
0.0.7
1.5
T-2
T 2.35
T-25 2.5
2-02.5
CONF
0:911 -:01--0055-
0.
00044
0.519
0.595-000 -115
1L559 I1B10
0.555
2.906 10117
INTROPY
0 10943
0.590-000 0.04
0.007
ILAK INRET
MARG
amm
R831 0001
0.711
1110
BL9 18820
11799
Shalab
0,912:000
06166-2027
0.00:3100
0627=0029
0.36200015
RADGE
6311:3001
0,5 0.59928022
0 0,575 021
0.594:50037
0.01:9007
6434:89144
055055025
05575ar1.05
0L56620000 0346-0012
UTILITY
6071-0017
0650=60015
0.002-0 804
0662250022
0402-0167
0651
0.00
0001-00004
0661:25.125
1166750002
AVERAGE
0.57:-0000
0.583-00311
0,50390.05
0.510-00037
0.56740888
0.591-0001
0:00
0614:0888
0.505:4.00
0.578=0.022
0.566:0630
192
INTROPY
0:511-006
0/88-000
0114-6011
1414-1889
01903-0011
0.807-11000
2.511-000 101
0199-78610
MARG:
6901 0014
0616 (0/
0005
1676 C/1-4
0132118000
0.65 0004
0.09 10015
BAR2 (U2)
DASS CC193
C3
0.95 0007
8806 00011
01011 0101
0/01
018618111
(160
061610010
0675 10H11
ILETS IU811
UTILITY
6.59
0608:
0103:0001
0.66620066
0,915
0627
055420019
0466:r1 014
0655
01667:201133
4.4 EFFECTS OF SPARSITY-AWARE LABEL PROPAGATION TECHNIQUES IN TCLP
Table shows the timestamp accuracy achieved by TCLP with and without two techniques em- ployed to handle sparsity of teseels-temprrturrr scaling for classifier output calibration and plateau width regularization in Section B.2.2 The class skewness balancing is employed by default to assure stable performance. In addition, the temperature scale factor T is varied for the temperature scal- ing technique. Compared with enabling both width regularization and temperature scaling (T = 2), removing width regularization only, temperature scaling only, and both degrades the timestamp ac- curacy by 5.0%, 4.3%, and 7.6%. respectively, in the 50salads dataset and 4.8%. 4.2%, and 8.1 1%, respectively. in the GTEA dataset. Clearly, both techniques are helpful for the label propagation, with the width regularization showing higher effect than the temperature scaling. Note that when T >>> 2, the label propagation length is suppressed, thereby causing deficiency in labels needed to train a classifier; when T < 1, the label propagation length may exceed the true segment length, thereby including wrong labels when training the classifier. The break point is affected by the av- erage length of segments in each dataset, where it occurs at higher value of T in a dataset with shorter segments: the average segment length is 34 in the GTEA dataset whereas it is 289 in the 50salads dataset.
CONCLUSION
In this paper, we present a novel label propagation framework for time-series active learning, TCLP, that fully takes advantage of the temporal coherence inherent in time-series data. The temporal co- herence is modeled by the quadratic plateau model, which plays a key role in segment estimation. Furthermore, the sparsity of known labels is relieved using temperature scaling and plateau reg- ularization. Thanks to accurate and effective label propagation, TCLP enables us to improve the performance of time-series supervised learning with much smaller labeling effort. Extensive experi- ments with various datasets show that TCLP improves the classification accuracy by up to 7.1 times when only 0.8% of data points are queried for their labels. Future work includes developing a query selection strategy that maximizes the merit of label propagation and utilizing constraint plateau model fitting based on similarity among plateau models with the same class. Overall, we expect that our work will contribute greatly various applications whose labeling cost is expensive.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was partly supported by Samsung Electronics Co., Ltd. (00201211-08051011 through the Strategic Collaboration Academic Program and Institute of Information & Communications Tech- nology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. 2020-0- 00862. DB4DL: High-Usabilityy and Performance In-Memory Distributed DBMS for Deep Learn ing).
CODE OF ETHICS
In this paper, introduce novel time-series active learning framework that propagates the queried labels to the neighboring data points. Our work reduces excessive effort and time spent on annotation for running deep learning system. We thereby contribute the society to run deep learning systems efficiently for personal, industrial, and social purposes, All datasets used in this paper are available public websites and have been already anonymized, using random numeric identifiers to indicate different subjects to preserve privacy; in addition, these datasets have been extensively cited for the studies in human activity recognition and video segmentation.
REPRODUCIBILITY
We try our best in the main text to enhance the reproducibility of our work. Section 3 explains the details the procedures for the theoretical analysis. Section clariifs the active learning set- ting, datasets, evaluation metrics, and configurations for training the classifier as well as the hyper- parameters of proposed method. The source code uploaded on /nttps://git /gtthbtt.b.uu..gg com/ kaist dm aalllab/TccI which contains the source code for active learning (main algorithm) and data preprocessing well as detailed instructions. We download all datasets used in this paper from public websites, as specified in the corresponding references.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. public domain dataset for human activity recognition using smartphones. In Proceedings of the 2013 International European Symposium on Artificial Neural Net- works, Computational Intelligence and Machine Learning, 3, 2013. URL http:
wwtcaiie..es..uc..eee//e//eeeeee o sfturaanccctttteesaaooosstttaaan
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proceedings of the 2020
International Conference Learning Representations, 2020.
Oresti Banos, Rafael Garcia, Juan A Golgado-Terrizee Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mHealthDroid: A novel framework for agile development of mobile health applications. In Proceedings of the 2014 International Workshop on Ambient Assisted Living, pp. 91-98. 2014. URL httttp::/aarcivee.. ics eeciiieddumm..
Joe Barrow, Rajiv Jain, Vlad Morariu, Varun Manjunatha, Douglas W Oard, and Philip Resnik. A joint model for document segmentation and segment labeling. In Proceedings of the 2020 Annual Meeting of the Association for Computational Linguistics, pp. 313-322, 2020.
William H Beluch, Tim Genewein, Andreas Nümberger, and Jan M Köhler. The power of ensem- bles for active learning image classification. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 9368-9377, 2018
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, 2015.
10

---

Published as a conference paper at ICLR 2022
Shohreh Deldari, Daniel V Smith, Hao Xue, and Flora D Salim. Time series change point detection with self-supervised contrastive predictive coding. In Proceedings of the 2021 Web Conference, pp. 011331243333 3135, 2021.
Yazan Abu Farha and Jurgen Gall. MS-TCN: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3575-3584 2019.
Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric tivities. In Proceedings of the 2011 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3281-3288, 2011. http::/:://bbb ic. wwt.gattec..... saeeechheduffp//
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Pro- ceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 2017 International Conference on Machine Learning, pp. 1321 1330, 2017.
Guoliang He, Yong Duan, Yifei Li, Tieyun Qian, Jinrong Hc, and Xiangyang Jia. Active learning for multivariate time series classification with positive unlabeled data. In Proceedings of the 2015 IEEE International Conference on Tools with Artificial Intelligence, pp. 178-185, 2015.
Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka. Alleviating over- segmentation errors by detecting action boundaries. In Proceedings of the 2021 IEEE/CVF Winter Conference Applications of Computer Vision, pp.
2021.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-55 1-4,,, 2019.
Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal con- volutional networks for action segmentation and detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
Dong-Hyur Lee et al. Pseudo-label: The simple and efficient sumi-super learning method for deep neural networks. In Proceedings of the 2013 International Conference on Learning Representations Workshop Challenges in Representation Learning, volume 3, pp. 896, 2013.
Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. MS-TCN++: Multi- stage temporal convolutional network for action segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Zhe Li, Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp pervision. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8365 8374, 2021.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning propagate labels: transductie propagation network for few-shot learning. In Proceed- ings of the 2019 International Conference on Learning Representations, 2019.
Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou. SF- Net: Single-frame supervision for temporal action localization. In Proceedings of the 2020 Euro- pean Conference on Computer Vision, pp. 420-437, 2020.
Karan Malhotra, Shubham Bansal, and Sriram Ganapathy. Active learning methods for low resource end-to-end speech recognition. In Proceedings of the 2019 Conference of the International Speech Communication Association, pp. 2215-2219, 2019.
Davide Moltisanti, Sanja Fidler, and Dima Damen. Action recognition from single timestamp pervision in untrimmed videos. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9915-99954 2019.

---

Published as a conference paper at ICLR 2022
Rafael Müller, Simon Kornblihh, and Geoffrey E Hinton. When does label smoothing help? In Proceedings of the 33rd Conference on Neural Information Processing Systems, pp. 4796-4705, 2019.
Fengchao Peng, Qiong Luo, and Lionel M Ni. ACTS: active learning method for time series classification. In Proceedings of the 2017 IEEE International Conference on Data Engineering, pp. 175-178, 2017.
Mathias Perslev, Michael Hejselbak Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian Igel. U-time: A fully convolutional network for time series segmentation applied to sleep staging, In Proceedings of the 2019 International Conference on Neural Information Processing Systems, pp- 4415 4426, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. arXiv preprint arXiv:2009.000 00236,, 2020.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In Proceedings of the 2021 International Conference on Learning Representations, 2021.
Mehdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. Proceedings of the 2016 International Conference on Neural Information Processing Systems, 29:1163-1171, 2016.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proceedings of the 2018 International Conference on Learning Representations, 2018.
Burr Settles. Active learning literature survey. Technical Report 1648, University of Wisconsin- Madison, Department of Computer Sciences, 2009.
Burr Settles. Active Learning. Morgan and Claypool Publishers, 2012. ISBN 1608457257.
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. In Proceedings of the 2018 International Conference on Learning Representations, 2018.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In Proceedings of the 2018 European Conference on Computer Vision, pp. 299-315, 2018
Sebastian Stein and Stephen McKenna. Combining embedded accelerometers with computer vi- sion for recognizing food preparation activities. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 729-738, 2013. URL https: lads/
7/evip. comput: ing. ng. wwwdee..deeeemm ac. wtt.:aaaw..cccceeeeaeeeaaaa wtt/aataasett/aoooppppraaaeeeeee
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In Proceedings of the 2021 International Con-
ference on Learning Representations, 2021.
Tal Wagner, Sudipto Guha, Shiva Kasiviswanathan, and Nina Mishra. Semi-supervised learning on data streams via temporal label propagation. In Proceedings of the 2018 International Conference on Machine Learning. pp. 5095-5104 2018
Dan Wang and Shang. new active labeling method for deep learning. In Proceedings of the 2014 International Joint Conference on Neural Networks, pp. 112-119, 2014.
Hanmo Wang, Xiaojun Chang, Lei Shi, Yi Yang. and Yi-Dong Shen. Uncertainty sampling for action recognition via maximizing expected average precision. In Proceedings of the 2018 International Joint Conference on Artificial Intelligence, pp. 964-970, 2018.
Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary cascade networks for temporal action segmentation. In Proceedings of the 2020 European Conference on Computer Vision pp. 34-51, 2020.
12

---

Published as a conference paper at ICLR 2022
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 93-102, 2019.
Ye Zhang, Matthew Lease, and Byron Wallace. Active discriminative text representation learning. In Proceedings of the 2017 AAAI Conference on Artificial Intelligncee 2017.
13

---

Published as a conference paper at ICLR 2022
A COMPUTATIONAL COMPLEXITY OF TCLP
In each round of active learning, let M be the number of plateau models, V be the average length of sub-sequences of predicted probability [x],)][k, and S be the number of training steps for eval- uating Equation ( Then, considering constant computational complexity for calculating the loss and gradient at each timestamp in the sub-sequences, we derive the computational complexity of plateau model fitting per round to be (MVVS). Here, M can be reduced by merging two overlap- ping plateau models with the same class. This complexity of fitting the plateau models is negligible compared with the complexity of training the classifier. For instance, according to the experiment for the 50salads dataset conducted using Intel Xcon Gold 6226R and Nvidia RTX3080, fitting the plateau models took only about 1 to 2 minutes, whereas training the classifier took about half an hour per active learning round.
B DETAILED FIGURE WITH STANDARD ERROR AND SUPERVISED ACCURACY
NOP
PIP
ESP
TCLP
MAX
0.64
0.65
0.8
0.92
F1@2
0.43
0.44
0.53
0.65
0.22
0.22
0.27
0.39
0.05
0.01
0.1
0.004
0.008
0,004
TS accuracy
0.73
0.71
0.9
0.95
0.52
0.5
0.67
0.79
0.3
0.28
0.43
0.64
0.05
0.01
0.1
0.008
0.002
0,004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 5: Classification accuracy with standard eттоr measured at each (1st-15th) round of active learning. The accuracy value an average over all query selection methods. The black line labeled MAX at the top indicates the maximum classification accuracy.
Figure 5 enriches Figure with standard error, indicated by the shadow around line, and fully- supervised classification accuracy, indicated by the horizontal line labeled MAX. At the last queried data ratio (i.e., after 15 active learning rounds) of each figure, in the 50salads dataset where only 1% of data points are queried, TCLP achieves 85% of the timestamp accuracy of fully-super classification. Similarly, in the HAPT dataset where only 0.4% of data points are queried, TCLP achieves 92% of the timestamp accuracy of fully-supervised classification. Overall, these results show that TCLP achieves the performance very close to fully-supervise classification using a very small proportion of query data points.
C ADJUSTMENT OF OVERLAPPING PLATEAUS IN TCLP
The process of adjusting overlapping updated plateaus through either merge reduction is fol- lows. Consider two plateaus hu, (q, Wi, st) (on the left) and hy, (cr, Wr, 8,)) (on he right), where (ci- Wt < G ), (ct+wr >> G. w. A (a+ ww <w, holds. If the classes assigned to these two plateaus are the same, i.e., k = ky, then they are merged to become one plateau whose width covers the segment merged from the two plateaus' segments. Hence, the half-width w' and center + wr)/2 and d' = the other hand, different classes are assigned to the two plateaus, i.e., hi * ky, their half- widtss are reduced to remove any overlap between them. As a result, separating the two plateaus at the mid- point m== (a: + (2) between their centers, after the reduction the left plateau has the half-width w/ = (m- (q /2 and the center at c Q: - wi + w/, and the right plateau has the half-width w = (c. + w, - m and the center df = G, + w, w/ Note that the labels of queried data result of this reduction. If the right plateau covers multiple queried data points whose timestamps are smaller than cy,, the timestamp of the leftmost queried data point becomes Cr. The same is applicable to the left plateau, except the change of the direction.
e' of the new plateau are w' = (c, + w,
+ w', respectively. If, on
points should not change
14

---

Published as a conference paper at ICLR 2022
D EFFICACY OF TCLP FOR EACH QUERY SELECTION METHOD
Figures show the efficacy of each label propagation (LP) approach combined with each query selection method. The performance plot shown in Figure is the average of these results over the six query selection methods. For all query selection methods, TCLP is shown to be effective in improving active learning performance compared to the other label propagation approaches.
NOP
PTP
ESP
TCLP
0.44
0.68
0.23
0.65
F1@225
0.29
0.46
0.15
0.46
0.15
0.23
0.08
0.26
TS accuracy
0.56
0.65
0.69
0.86
0.39
0.45
0.53
0.69
0.23
0.25
0.37
0.53
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 6: Efficacy of the four LP approaches with CONF query selection.
0.37
0.07
F1@:25
0.25
0.4
0.48
0.13
0.2
0.02
0.26
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
TS accuracy
0.5
0.58
0.58
0.4
0.43
0.67
0.19
0.22
0.29
0.49
0.005
0.01
0.05
0.1
0.004
0.008
0
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 7: Efficacy of the four LP approaches with ENTROPY query selection.
0.48
Fl@255
0.41
0.48
0.32
0.57
0.21
0.24
0.16
0.31
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
IS accuracy
0.74
0.91
0.57
0.29
0.39
0.62
0.005
0.01
0.05
0.1
0.004
0.008
0
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 8: Efficacy of the four LP approaches with MARG query selection.
15

---

Published as a conference paper at ICLR 2022
NOP
PTP
ESP
TCLP
0.55
0.66
0.58
0.83
F1@:25
0.37
0.44
0.39
0.58
0.19
0.22
0.19
0.32
TS accuracy
0.65
0.81
0.46
0.44
0.61
0.74
0.26
0.25
0.4
0.55
0.005
0.01
0.1
0.004
0.008
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 9: Efficacy of the four LP approaches with CS query selection.
0.47
0.71
0.31
0.85
F1@25
0.31
0.48
0.59
0.16
0.24
0.1
0.33
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
TS accuracy
0.6
0.66
0.93
0.43
0.49
0.5
0.78
0.27
0.27
0.33
0.64
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 10: Efficacy of the four LP approaches with BADGE query selection.
0.63
0.86
Fl@25
0.4
0.46
0.42
0.66
0.21
0.23
0.21
0.46
0.005
0.01
0.05
0.1
0.004
0.008
0.002
0.004
IS accuracy
0.67
0.45
0.66
0.88
0.25
0.41
0.1
0.005
0.01
0.05
0.1
0.004
0.008
0
0.002
0.004
Queried data ratio (a) 50salads.
Queried data ratio
Queried ratio
Queried ratio
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 11: Efficacy of the four LP approaches with UTILITY query selection.
16

---

Published as a conference paper at ICLR 2022
E EVALUATION OF PROPAGATED LABELS
Figures show the propagation quality of each label propagation approach, again demonstrat- ing the superiority of TCLP. These plots provide the overall trends including the final round results reported in Table 3 The correct propagation ratio (CPR), i.c. the number of correctly propagated labels the total number of data points, measured at each round using specific query selection method on each dataset. The CPR of TCLP is shown to be higher than those of the other label propagation approaches throughout the entire period (i.e., from the 1st through the 15th round).
NOP
PTP
ESP
TCLP
0.13
0.11
0.05
CPR
0.09
0.33
0.07
0.03
0.04
0.17
0.04
0.02
0.005
0.01
0.05
0.1
0.004
0.008
0.004
0.008
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 12: CPR of the four LP approaches with CONF query selection.
0.03
0.1
0.4
0.07
CPR
0.07
0.27
0.02
0.03
0.01
0.005
0.01
0.05
0.1
0.004
0.008
0.004
0.008
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 13: CPR of the four LP approaches with ENTROPY query selection.
0.37
0.42
0.2
0.15
CPR
0.24
0.28
0.13
0.10
0.12
0.15
0.07
0.05
0.005
0.01
0.05
0.1
0.004
0.008
0
0.004
0.008
Queried data ratio
Queried data ratio
Queried ratio
Queried ratio
(a) 50salads.
(h) GTEA.
(c) mHealth.
(d) HAPT.
Figure 14: CPR of the four LP approaches with MARG query selection.
0.31
0.44
0.21
0.15
CPR
0.20
0.29
0.14
0.10
0.10
0.15
0.07
0.05
0.005
0.01
0.05
0.1
0.004
0.008
0
0.004
0.008
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 15: CPR of the four LP approaches with CS query selection.
17

---

Published as a conference paper at ICLR 2022
NOP
PTP
ESP
TCLP
0.16
CPR
0.13
0.27
0.13
0.11
0.06
0.14
0.07
0.05
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 16: CPR of the four LP approaches with BADGE query selection.
0.26
CPR
0.23
0.35
0.22
0.17
0.18
0.11
0.09
0.005
0.1
0.008
0.004
Queried data ratio
Queried data ratio
Queried data ratio
Queried data ratio
(a) 50salads.
(b) GTEA.
(c) mHealth.
(d) HAPT.
Figure 17: CPR of the four LP approaches with UTILITY query selection.
18


Reviews:
[{"review_id": "oU2cxuebJ6I", "reviewer": "Reviewer_hW9X", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper are in addressing time series active learning in a way that truly respects the temporal coherence of time series and addressing issues with identifying the time series segments. These are presented in a coherent algorithm with pseudocode that is relatively easy to follow. Additionally the empirical results are quite strong. There is also some analysis of the differences in performance between the proposed algorithm and other competitors, as a function of the characteristics of the dataset---this, along with pseudocode, are unfortunately becoming quite rare in machine learning research papers.\n\nI have one issue that would be nice for the authors to address in the paper. Near the end of the experiment section, the authors mention how performance varies as a function of the parameter T in temperature scaling, and show performance increasing as T increases from 1 to 1.5 to 2. Please describe what happens as T is increased further until the point at which performance decreases. It would be good to see how that breaking point varies depending on the dataset, and why.\n\n", "summary_of_the_paper": "This paper develops a framework for time series active learning called Temporal Coherence-based Label Propagation (TCLP), that uses the temporal coherence within time series to choose as few samples as possible for domain expert labeling and assigns those labels to temporally close samples. In addition, unlike the most similar methods in the literature, the proposed method does not require estimates of the approximate locations and true classes of the segments of the time series.", "summary_of_the_review": "This paper, as far as this reviewer knows, is truly addressing time series active learning for the first time, and does so in a novel and coherent way algorithmically and experimentally.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "mttGFVp9dH3", "reviewer": "Reviewer_pVjg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: Applying plateau model (Equation 2) into time-series active learning for pseudo labeling is a good idea for promoting the performance. The paper is easy to follow, the idea is simple and the experimental results show significant improvement (considering both performance gain and label reducing) compare with other label propagation methods.\n\nRemaining questions: \n1) In Table 1, why b and w0 are inconsistent in the experiments across different datasets? Does it make any difference to have a larger b? That is, does the performance of TCLP decrease as the query size increases? Since in active learning, enlarging the batch is an effective way to reduce the number of rounds.\n2) For temperature scaling parameter T, why the larger of T, the better the performance are? It's better to include more settings of T, i.e., T < 1, T > 2 and more analysis about the influence of T, since the value of T seems to have a significant impact on performance.\n3) Besides UTILITY, it's better to add the evaluation results that trained on whole datasets as comparison.", "summary_of_the_paper": "This paper addresses the label propagation segment estimation problem in time-series active learning, and apply plateau function to model temporal coherence. The experimental results show their effectiveness compare with baseline label propagation approaches.", "summary_of_the_review": "I vote for a weak ac of the current version since the idea of using plateau model for pseudo labeling in time-series AL is interesting. Although there is no rigorous theory to tell us why TCLP improves performance (at least in this paper), that is, why plateau model could provide a better pseudo labeling, and what's the upper limit of its promotion. The experimental results are fairly good, they demonstrate TCLP's effectiveness in their experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dUL9OIxk0xa", "reviewer": "Reviewer_kJBx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The empirical result shows that this methods is a really good method.\n- The paper is a good ICLR paper, it is based on a simple idea, and it does work well in practice.\n\nWeakness:\n- Updating rules based on a learnt model is proned to overconfidence loophole. While the authors briefly discussed temperature scaling usually used to avoid overconfidence in neural networks, I would appreciate a more thorough discussion on the matter, as well as some example of failure of TCLP.\n- I was not able to open the dropbox files and to read the code, but I believe the authors have reused licensed files that are not mentionned in the paper.\n\nThere is a lot of small corrections that I will detail now:\n\npage 3: \n1/ I believe D_u \\cup D_l = D, but this is not mentioned clearly.\n\n2/ \"More formally, the estimated segment for t_q is Eq. (1)\"\nThis is a choice made by the authors, and should be more clearly stated as a choice, by using “we estimate the segment for t_q”. If this choice is standard, than it would be nice to refer to litterature that introduce this, or similar, objective.\n\n3/ What happen when the points in [t_s, t_e] are not all in D_u? For example, if segments [t_s, t_e], overlap for t_q1 and t_q2? \n\npage 4:\n1/ Eq.(4), what is t? I believe c, w, and s should be updated according to the gradient of the quantity appearing in Eq. (3) where there is a summation over t that make it disappears.\n\npage 5:\n1/ we assume that the estimated class […] is an indenpendent random variable: independent of what, where does the randomness comes from?\n\n2/ Eq.(5) is really weird, and is based on assumptions that seems highly unrealistic. I would like to see derivations for such an equality. \nIt seems to me that the authors are assuming that $f_\\theta(x_t)[y]$ is a random variable that is independent of t, for each label y.\n\n3/ \"Without loss of generality”, why don’t we loose generality?\n\nOverall I believe Section 3.3 is too poor scientifically speaking to be put inside an ICLR paper.\n\npage 6:\n1/ It would be nice to describe temperature scaling in more details.\nSimilarly, the labels propagation from H_r is not that clear, especially if there is overlaps.\n\n2/ In Algorithm 1, the loss ell and the query strategy should be added as input of the algorithms, as well as the stepsize lambda.\n\npage 7:\n1/ Is TCLP working better than other methods for active learning simply because it is working better for semi-supervised learning?\nWhat happen if we start with 1% of labeled datapoints rather then nearly 0%?\n\n2/ Can we have standard deviation on Figure 4?\n\n3/ Is the same neural network architecture used for each methods?\n\npage 10:\n1/ Error regarding the github url.", "summary_of_the_paper": "The goal of the paper is to improve active learning for time-series.\n\nThe goal is to learn a function from x to y, with data coming from a time series (x_t) and (y_t). It is assumed that (y_t) is piecewise constant.\nAt the beginning no (y_t) are known, but query can be made in a active learning framework.\n\nThis paper tries to find the pieces where (y_t) is constant based on the features (x_t). It is done using the labeled timestamps (y_t_q) at (t_q) and the current learnt model f from X to Y with a thresholding rule based on f(x_t) around the (t_q). This gives segments on which (y_t) is supposed constant egal to the (y_t_q). Those labels are added as if they were true labels. Then the model for f is updated, and some active query are made based on any model. The loop is repeated for a given number of rounds.", "summary_of_the_review": "The experiments speak for themselves.\nOverall while the idea is simple, and the math are not really satisfying, the experimental part erases most of my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "690Ahp7uFSU", "reviewer": "Reviewer_s4ST", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Good approach to use limited user feedback efficiently.\n2. Satisfactory set of experiments\n\nCons:\n1. Need some more elaboration on the theory\n2. Computational/time complexity is missing\n\n\nMain Comments:\n\n1. Equation 3: It seems like c, w, s are all estimated independently for each plateau. However, we know that very likely, these will be similar for a particular y_l (class label). Could we use such a constraint/model? E.g., we might have a [parameterized] model for (c_l, w_l, s_l) which relates to y_l.\n\n2. Equation 5: This expression needs to be elaborated and also needs to be shown that the sum of all Pr(t_e - t_s = i) where i = 1, ..., L is 1 so that we can claim Pr is a valid probability. This is not obvious.\n\n3. Algorithm 1: Do we always have r*b plateau models at the r-th round of active learning? What is the computational complexity and/or time for each round? It seems that computation time would increase with each round. Is there a way to lower the complexity by taking a sample of the most relevant plateau models?\n\n4. It is unclear why 'b' is 85 for mHealth (85 for mHealth vs 200 for others) -- no reasoning has been provided. To make sure that the settings were not selected after-the-fact just to get the best results (in table 2), the same settings should be used across all datasets unless there is a really good justification. I assume w_0 is diferent for GTEA (5 for GTEA vs 15 for others) since it has the smallest segments. It would be better if these parameters could be auto-selected or the paper explicitly stated some reasoning/rule to select appropriate values for them.", "summary_of_the_paper": "The paper presents an active learning algorithm for classifying and segmenting timeseries data. When instances are labeled by users, these labels are propagated to [temporally] neighboring instances using a plateau model (if possible) so that the information is shared more efficiently across multiple instances.", "summary_of_the_review": "This paper proposes a technique to improve accuracy in an important field of application where data volumes are high and efficient use of limited user feedback is of urgent need. The plateau model makes intuitive sense in the context employed here and looks like something we would want to add to the toolbox.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors design a framework for active learning on time-series data. The framework, called Temporal Coherence-based Label Propagation (TCLP) leverages temporal coherence to propagate expert labels to nearby points by a plateau model. In addition to describing the framework clearly with simple pseudocode, several experiments are carried out with careful analysis to validate the effectiveness of the framework.

The reviewers are mostly positive on the simple algorithm with strong empirical performance as well as the solid analysis of experimental results. They are also satisfied with most of the rebuttal feedback from the authors. Somehow there are joint concerns on the weaker theoretical results, especially in terms of their correctness. In particular, the unrealistic assumptions and over-simplification make it hard to connect the theoretical results with the actual algorithm. Several reviewers suggest the authors to move theoretical analysis section to a supplementary section as a hypothesis, and the authors are also encouraged to clearly discuss what the theory can and cannot cover.
Decision: Accept (Poster)