The Accuracy Cost of Weakness: A Theoretical Analysis of Fixed-Segment Weak Labeling for Events in Time
Authors: John Martinsson
Tuomas Virtanen
Maria Sandsten
Olof Mogren
Venue: Accepted by TMLR ()

Abstract: Accurate labels are critical for deriving robust machine learning models. Labels are used to train supervised learning models and to evaluate most machine learning paradigms. In this paper, we model the accuracy and cost of a common weak labeling process where annotators assign presence or absence labels to fixed-length data segments for a given event class. The annotator labels a segment as "present" if it sufficiently covers an event from that class, e.g., a birdsong sound event in audio data. We analyze how the segment length affects the label accuracy and the required number of annotations, and compare this fixed-length labeling approach with an oracle method that uses the true event activations to construct the segments. Furthermore, we quantify the gap between these methods and verify that in most realistic scenarios the oracle method is better than the fixed-length labeling method in both accuracy and cost. Our findings provide a theoretical justification for adaptive weak labeling strategies that mimic the oracle process, and a foundation for optimizing weak labeling processes in sequence labeling tasks.

Published in Transactions on Machine Learning Research (09/2025)
The Accuracy Cost of Weakness: A Theoretical Analysis of Fixed-Segment Weak Labeling for Events in Time
John Martinsson
john. wonn.mortinsso@@ri..e
Computer Science
RISE Research Institules of Sweden
Centre for Mathematical Sciences
Land University
Tuomas Virtanen
tuomas. virtanen @tuni.
Signal Processing Reseorch Centre
Tampere University
Maria Sandsten
mariau.sandslee@@matttaa su
Centre for Mathematical Sciences
Lund University
Olof Mogren
olof. alof.mogrendivi.co
RISE Research Institutes of Sweden Swedish Centre for Impacts of Climate Eatremes (climes)
Climate Al Nordics
Reviewed on OpenReview: https: // opwnreveww.eeee.. net/ forum? id= tTu@uXBQ18
Abstract
Accurate labels are critical for deriving robust machine learning models. Labels are used to train supervised learning models and to evaluate most machine learning paradigms. In this paper, we model the accuracy and cost of a common weak labeling process where annotators assign presence or absence labels to fixed-length data segments for given event class. The annotator labels segment. as "present" if it sufficiently covers event from that class, e.g., birdsong sound event in audio data. We analyze how the segment length affects the label accuracy and the required number of annotations, and compare this fixed-length labeling approach with an oracle method that uses the true event activations to construct the segments. Furthermore, we quantify the gap between these methods and verify that in most realistic scenarios the oracle method better than the fixed-length labeling method in both accuracy and cost. Our findings provide theoretical justification for adaptive weak labeling strategies that mimic the oracle process, and foundation for optimizing weak labeling processes in sequence labeling tasks.
1 Introduction
In supervised machine learning, labeled datasets are required for training and evaluation. During evaluation, the accuracy of the labels determine the quality of the analysis. However, in practice, labels often contain noise that varies with the input sample and label type. Noisy training labels present persistent challenge in machine learning (Liang al., (2009] Song et al. (022).. Deep learning models, in particular, are prone overfitting noisy labels, raising questions about the nature of generalization (Zhang al.) 2021). Regular- ization techniques such dropout. (Srivastava al. 2014), data augmentation (Shorten & Khoshgoftaar

---

Published in Transactions on Machine Learning Research (09/2025)
2019 and weight decay (Krogh & Hertz 1991 mitigate overfitting but fail to climinate the performance gap between training on noisy versus clean labels (Song et al., 2022
Beyond the well-documented challenges posed by noisy training labels, inaccurate evaluation labels present a significant, yet often overlooked, obstacle to reliable machine learning. When evaluation metrics are computed against noisy ground truth, the apparent "best' performing model might simply be the one that most closely reproduces the noise present in the evaluation set, rather than exhibiting superior generalization capabilities. This very issue, where noisy evaluation labels can lead to the rejection of models that have learned the true clean label distribution, is central concern addressed by Görnitz et al. (2014). This can lead to the selection of suboptimal models that perform well on the flawed evaluation data but generalize poorly to unseen, cleaner data or data from real-world applications. Consequently, performance benchmarks can be inflated and misleading, hindering meaningful comparisons between different approaches. Therefore, understanding the characteristics of label noise, not just the training data but also in the evaluation data, is crucial for developing and selecting models that are truly effective and robust.
Labels are typically obtained through human annotation, process that involves significant time and financial investment, particularly for complex data like audio time-series signals. In this work, consider form of weak labeling where the annotator assigns presence or absence labels to predefined data segments. This offers a practical and cost-effective approach for annotating large audio datasets (Martin-Morato & Mesaros 2023). To reduce cost, weak labels avoid specifying precise boundaries within the data segments, focusing instead on general presence or absence of the target class. However, this simplification introduces noise into the labels, especially for data with time- aine-varying characteristics, such as audio signals, where events can occur intermittently within the labeled segment (Turpault et al.. 2021). Understanding and mitigating this noise is critical to effectively leverage weak labels in downstream applications (Kumar & Raj 2016).
The noise in weak labels can be categorized into two types: class label noise (mislabeling event presence or absence in a segment) and segment label noise (mislabeling due to misaligned segment boundaries). While class label noise has been extensively studied (Song et al.) 2022 Zhang et al.) 2021), the effects of segment. label noise remain underexplored This type of noise significantly affects tasks such as sound event detection (Hershey et al. [022 Turpault et al. 2021 Shah et al. 2018 and medical image segmentation Yao
et al. 2023). Strategics like pscudo-labeling (Dinkel et al.) 2022). robust loss functions (Fonsoca et al. 2019) and adaptive pooling operators (McFee et al. 2018) aim to address challenges when training weak labels.
However, fully understanding the impact of weak labels requires quantifying their accuracy (Shah et al.
2018 Turpault et al. 2021].
While adaptive annotation methods have been explored in some domains, fixed-segment (FIX) annotation remains the de facto standard for large-scale sound event datasots due its simplicity and scalability. Foundational datasets such as AudioSet (Gemmeke et al. 2017aa), CHIME (Foster et al. 2015b), OpenMIC- 2018 (Humphrey et al. 2018a), YBSS-200 (Singh & Joshi 2019 and SONYC (Bello et al. 2019), as well as more recent datasets such as VGG Sound (Chen et al.
MATS (Morato & Mesaros) 2021), and MAESTRO Real (Morato et al. 2023) continue to roly on fixed segments. A practical motivation is that for many sound sources, precise boundaries are inherently ambiguous: a passing car or fading background noise lacks sharp onsets and ofisets, and audio scenes frequently contain overlapping sources. These challenges make FIX annotation a practical default even when adaptive approaches are conceptually appcaling.
2020
In this paper we use the term weak labeling exclusively to refer to temporal ambiguity introduced by segment- level human annotation. We do not use the term in the sense of pseudo-labels automatically generated annotations.
Current hods typically estimate label noise rates after collecting labels (Song et al. 2022 employing
techniques like noise transition matrices (Li et al. 2021) or cross- cross-validation (Chen et al.) 2019 J. In contrast,
predicting label noise rates before data collection remains largely unexplored. This is particularly challenging when the noise stems from human annotators, as it is difficult to formalize. In cases involving partially automated processes, however, the noise introduced by the automated component can often be modeled under specific assumptions.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Weak labeling of an audio recording
(B=7)
4o
9
42
44
95
q6
FIX (B=7)
qo
q1
42
93
q4
G5
96
FIX (B=30)
FIX (B=60)
presence event
presence label
Figure 1: Resulting presence (red) and absence (white) labels from ORC and FIX weak labeling for an audio rocording with three presence events (green). ORC weak labcling assigns labels to ground truth segments, achieving perfect label accuracy with = 7 labels. FIX weak labeling, shown for different segment lengths (B= 7, B = 30, B 60), introduces segment label noise as segments misalign with events. Longer segments reduce annotation cost but increase noise, while shorter segments align better but require more annotations. Note that too short segments (B 60) may lead to the annotator missing the presence of the event because it does not cover large enough fraction of it.
In this work, we model the automated component of commonly used weak labeling mothod for segmentation lasks: fixed-length weak labeling (FIX). We quantify the segment label noise of this process, and study the expected label accuracy. This method, commonly employed in sound event detection, involves annotators providing presence or absence labels for fixed-length segments of the data (automated component), rather than specifying precise event boundarics. By simplifying the labeling process, FIX weak labeling reduces annotation effort but introduces segment label noise when segments misalign with the actual onsets and offsets of events. To benchmark this approach, we compare it to an oracle weak labeling method, ORC weak labeling, which assigns presence or absence labels to segments dcrived using the true onsets and offsets of the events.
Figure ! illlustrates the trade-offs between annotation cost and label accuracy for the ORC and FIX weak labeling methods. ORC weak labeling achieves perfect label accuracy by aligning the segments with the ground truth presence events (green) using a minimal number of annotated segments. In contrast, FIX weak labeling shows varying accuracy depending on segment length: shorter segments improve alignment. (B = 30) but require more annotations, while longer segments (B = 7) reduce cost at the expense accuracy. addition, too short segments (B 60) can lead to the annotator missing event presence. These trade-offs are central to understanding how FTX weak labeling labeling method, provide thcoretical framework to guide data collection efforts.
be used effectively. By analyzing the FIX weak
Our analysis shows that FIX weak labeling systematically introduces segment noise, and provide closed- form expressions for expected label accuracy and the optimal segment length. These results establish a quantitative bascline for annotation design, highlight the limits of non-adaptive strategics, and motivate the development of adaptive methods that approximate oracle labeling.
In summary, our contributions include:
Closed-form expressions for label accuracy and annotation cost in FIX and ORC weak labeling, made tractable by assuming an annotator model and a simplified data distribution.
simulation study demonstrating that our theoretical framework generalizes to more complex data distributions and serves upper bound for the accuracy of FIX weak labeling.
3

---

Published in Transactions on Machine Learning Research (09/2025)
A theoretical foundation for developing adaptive weak labeling methods that better approximate ORC weak labeling, such as (Martinsson et al. 2024) for sound event. detection and Kim et al. 2023) for image segmentation.
Our analysis focuses one-dimensional data, and the assumptions are justified by common characteristics in bioacoustic sound events. These time-localized, non-stationary animal vocalizations often roquire annotators to hear significant portions of the sound to assign accurate presence labels. Note, however, that while our framework is tailored to this domain, the principles extend to annotation of events in time in other data that shares these characteristics.
2 Problem Setting
The analysis framed within a multi-pass binary labeling setting. Here, an annotator assigns binary labels (presence or absence) to data segments based on the occurrence of specific sound events. The annotator model abstracts how an annotator interacts with data by labeling segments, without requiring precise knowledge of event boundaries. While inspired by time-localized and non-stationary sound events, this framework is generalizable to any time series with similar characteristics.
It's important to emphasize that, in this weak labeling setting, the concept of overlapping events is not explicitly modcled. Overlapping events from the same class are treated as a single, longer presence event, because presence/ sersencceabbeccc aabsence labels cannot differentiate between individual event instances. For instance, in an audio recording with two birds calling simultancously, this weak labeling framework simplifies the overlap into single 'present' event. While this simplification is necessary when studying weak labeling in this setting, it fundamentally restricts our ability resolve polyphony (the identification of multiple overlapping sound events). We leave the exploration of annotator models capable of providing richer labels to future work; this is beyond the scope of the current study.
For events of different classes occurring simultaneously, annotation is typically carried out in multi-pass setup: each class is annotated independently, often by different annotators. Overlaps across classes are therefore not mutually exclusive- both events can be marked present within the same temporal window. The results from this paper are still valid when viewed for each class separately.
2.1 The Assumed Data Distribution
A sound event is defined by its start time ag E E, end time be R, and class ce E C, denoted = (de, ce). Audio recordings are assumed to have finite length T, and we assume that the events are uniformly distributed locally in time (see Section 4.2 and Appendix A.4 for more details).
2.2 The Assumed Annotator Model
For given sound event class E C, the annotator decides the presence or absence of an event of class in data segment q = bg), where d, = by ag is the fixed-length of the segment. We will refer to q query segment because it is queried for presence absence label. Let I, {0, 1} denote the weak label indicated by the annotator for query segment g, where 14 indicates presence of an event of class in and l = 0 indicates absence of that event, class in q. Detecting the presence of an event requires observing a sufficient fraction of the event within the query segment, formalized as follows:
Definition 1. The event fraction is the fraction the total event duration de = be - ae that overlaps with the query segment ?,
h(e, g) lengl de
(1)
where e q is the interscction of (ac, be) and (ag, bq)-
Definition The presence criterion & E (0, 1] the minimum event fraction required for the annotator detect the presence of in q.
h(e, g) 7-
(2)
4

---

Published in Transactions on Machine Learning Research (09/2025)
The annotator assigns a presence label (1, to q if there is sufficient overlap with any presence event of class e (h(e, g) 2 7); otherwise, it assigns an absence label (1g 0)-
The parameter reflects the annotator's sensitivity: lower 7 values indicate sensitivity to smaller event frac- tions, while higher values require larger fractions. This model of perceptual ability is particularly suited for non-stationary events (e.g., a specific birdsong) where recognizing a relative portion of the event's structure is key, as opposed to stationary sounds (e.g., an engine hum) which might be identified after a fixed absolute duration.
This framework captures variability in annotator behavior. For example, detecting "human speech' or "bird song' may only require hearing a small fraction of the event (y closer to 0), while recognizing specific phrases or bird species might demand a near-complete observation (y closer to 1). The value of y thus depends on the annotator and the complexity the event class. This model provides a flexible yet precise way simulate annotator behavior and quantify their labeling performance. However, is important to note that this model is deterministic, focusing on temporal alignment belween events and the query segment. In practice, human annotation often involves stochastic factors, such as variability in perception and judgment, which are not explicitly modeled here.
2.3 Label Accuracy
Label accuracy measures the aligmment between annotator-providee labels and ground truth labels:
Definition 3. The label accuracy is defined as
lengl dy lengl
if lg = 1,
F(e,g.7)
(3)
if lg 0.
For instance, consider a 3-sccond query segment (dg = 3) that overlaps exactly one second (lengl = 1) with a 2-second sound event (d. = 2) the class bird song (c = "bird song"). The annotalor assigns a presence label (1g = 1) with label accuracy lengl d,
if half or less of the event needs to be in the query segment
5 0.5). Contrary, the annotator assigns an absence label (l, = 0) with label accuracy - للج if more than half of the event (7 > 0.5) needs to be in the query segment. This formulation isolates the segment label noise (1 (1- F(e,q,7) introduced by the automated component (fixed-length segments) of the FIX weak labeling method.
de
3
The Label Accuracy and Cost of ORC Weak Labeling
Let start with the ORC weak labeling method. This method uses priori information about the event start and end limes and is therefore not available in practice but should be seen as upper bound on what can be achieved with weak labeling. The start and end times of the true presence and absence events are used to construct the query segments:
QORC { bu), (a1, bi)...., (a
(.))...,((BBBo--) bBoce ( 1)} {gu................. 9Bonc
where (a;, b:) is the ith ground truth presence or absence event.. The annotator indicates presence or absence for each of these segments, which by construction results in the ground truth annotations, illustrated in Figure 2 In the example, there are three target events (green), and four absence events, which means that BORc 7. In general BORC E {2M - 1, 2M + 1}, where M denotes the number of presence events. The number of absence events can be fewer than 2M + 1 if the recording starts or ends with a presence event, however, for simplicity and without losing generality, we will consider BORc = 2M + as the minimum number of query segments needed for ORC to derive the ground truth. From an annotation cost perspective, this the most cautious choice, and is also the most likely outcome. The query accuracy for each query segment since by construction the fraction of correctly labeled data in each query segment will be 1 when given the correct presence absence labels.
5

---

Published in Transactions on Machine Learning Research (09/2025)
ORC weak labeling
qo
91
42
4a
q4
95
96
presence event
presence label
Figure 2: ORC weak labeling of an audio recording with three target events (M = 3) shown green and four absence events. The BoRc 7, query segments 90, ........0 96 are derived from the ground truth segmentation of the data, and therefore the label accuracy will by definition be 1.
summary, the ORC weak labeling method produces annotations with label accuracy 1, using the minimum number of query segments needed achieve this. We use this as a reference on what can be achieved for weak labeling data.
4 The Label Accuracy and Cost of FIX Weak Labeling
The outline of this section is follows. In Section derive closed-form expression for the expected label accuracy of a query segment given that it overlaps with a single event of deterministic event length. We note that it is only in the cases of overlap between a query segment and an event that a presence label can occur under the assumed annotator model, and that the expectation label accuracy over these cases therefore can be viewed as the expected presence label accuracy. For the remainder of the paper we will simply write expected label accuracy when referring to the expectation over the overlapping cases, unless explicitly stated otherwise.
define the FIX labeling method. In Section 4.2
In the same section derive the optimal query length with respect to the expected label accuracy, the maximum expected label accuracy and the number of query segments needed (proxy for annotation cost). In Section 4.3 we explain how the expression for expected label accuracy can be used in the case of a single event of stochastic length, and in Section 4.4 we explain under which conditions this can be used when multiple events can occur. Finally, we derive a closed form expression for the expected label accuracy of an audio recording with multiple events of stochastic length Section 4.5 and provide an alternative interpretation of the theory in Section 4.6
4.1 The FIX Weak Labeling Method
The FIX weak labeling method, commonly used in practice, splits the audio recording into fixed and equal length query segments, and then an annotator is asked to provide either a presence or absence label for each of the query segments. Let BFIx denote the number of query segments used, then the query segments for an audio recording length T are defined as
@FIX = {(ao, bo), (1), (1 1)} = .g..... .. ..r...-
(5)
where the start and end timings of each query segment is qi (a;, b.) = (idg. (i 1)dg) and the fixed query segment length is dg = T Brix- We illustrate this in Figure where the presence criterion for the annotator is = 0.5. There are three presence events and four absence events, and using only BrIx = 7 query segments results annotations with an average label accuracy that is lower than 1.
We want to find an expression for the expected label accuracy for a given data distribution and query segment length. In addition, want. to understand the query length that maximize the expected label accuracy.

---

Published in Transactions on Machine Learning Research (09/2025)
FIX weak labeling
qo
91
q2
g3
aa
95
q6
presence event
presence label
Figure Illustration of the FIX weak labeling method. The audio recording contains presence events (green). The FIX method divides the recording into fixed-length query segments (e.g., qo qa). Note how the alignment between segments and presence events affects the accuracy of presence labels (red hatched).
de
t 0
q2
1.0
Fle q2. V)
0
de + dg
t
Top panel: single event (e:) of length de can occur at various end times (t) within the recording of length T. Bottom ponel: The resulting label accuracy for query segment ø2 (arbitrarily chosen for illustration) of length dg as function of the event's end time (t). Overlap between the event and the query segment leads to segment label noise and reduced label accuracy, which in this case occur when E [02, 02 + + +dy where d2 is the start time of d2. The red hatched area (A) represents the cumulative label accuracy during these overlapping scenarios. The figure illustrates the evolution of the accuracy function F(er, q.) ) as the event end time sweeps across the segment. The apparent central alignment at 0.5 is specific to this example and should not be interpreted as general property of FIX labeling.
Figure 4:
4.2 The Expected Label Accuracy of a Query Segment given Event Overlap
To derive tractable closed-form expression, analyze simplified data distribution where each recording of length T' contains a single event of deterministic length de- This idealized case is the simplest possible annotation scenario, and the resulting accuracy can therefore be interpreted as a theoretical upper bound: any added complexity, such as multiple events or variable event lengths, introduces additional opportunities for error.
The setup is illustrated in the upper panel of Figure where single event e, of length de occur any time t E [0, (indicated by the arrow). The bottom panel of Figure 4 shows the label accuracy for specific query segment (92) as the end time (t) of the event varies. The area (A) highlighted in hatched red indicates the label accuracy in the cases of overlap between the query segment and the event, and the area in hatched green indicate the label accuracy in the cases of no overlap, which is by the definition of the annotator model is always 1. Crucially, while this figure illustrates the accuracy for query segment 42. the shape of this accuracy function remains the same for other query segments; only its position along the x-axis would change.

---

Published in Transactions on Machine Learning Research (09/2025)
fix the query segment to start at time 0, ! (0, dg), and represent the event with its ending time t as tE E [0, de + dg] describes all possible overlap occurrences. That is, when t = 0 the event ends at the start of the query segment, and when t == de + dg the event starts at the end of the query segment. To formalize this, we can express the expected label accuracy in case of overlap by integrating over all possible event end times (t) where overlap occurs:
To simplify the mathematical analysis, without loss of generality,
(t- de: (). In this way,
do
Eaup (F(..
(6)
1
!
de do
F(
(7)
de + dg Jo
A
(8)
de + dg
where t ~ p denotes a random variable t distributed according to a distribution p, and p(t) denotes the probability of realization We assume that the distribution of the relative offsets t between events and overlapping qucry segments is uniformly distributed (empirically verified in Appendix (A.4). There are two sources of variation that makes this plausible: (i) the start time of the recording varies depending on when the recording session was started, and (ii) the start time of the event varies depending on when the sound source emits the event. Note that this assumption is likely to not hold if dg > de, but that leads to very weak labels which is not wanted in practice. Using this assumption we get p(t) = 1/(de + dg). and by observing that the integral Idd tde F(e,, q, d)dt describes the hatched rod area denoted in Figure we arrive at the final expression in Eq.
Remember that absence labels occur when there is overlap (always correct) and when there is overlap but the presence criterion is not fulfilled, and presence labels can only occur when there is overlap and the presence criterion is fulfilled. Therefore, inaccurate labels only occur in the case of overlap. The expected label accuracy in the case of overlap therefore describes the accuracy of the labels when segment label noise can occur, which happens around the boundaries of the true event.
In Appendix A.1 we show how to express in terms of the event length de, the query segment length dy and the presence criterion y under the assumption that the annotator presence criterion be fulfilled (d, 2 yde), and that it can not be fulfilled (d, yde). Finally, we arrive at the following four main theorems: The expected label accuracy in case of overlap between a query segment q of length d, and a single event e of deterministic length de is
Theorem 1.
de de(2yd, (+Ppd,22PP.44.)) dgld dg Jotdy'
if dg 2 Ide: if dg < yde:
f(gy) = Expp [F(e,,
(9)
when the presence criterion for the annotator is 7.
Proof. See Appendix A.I for the proof. We show how to express the area A in Eq. S in terms of de, dg and y for the two assumptions: dy 2 ydz, and dg yda.
Theorem 2. The query length that maximizes the expected label accuracy in case of overlap for a given event length de is
27 4-2 +2
d; = =
(10)
27 +
Proof. See Appendix A.2 for the proof. We compute the derivative of f(d,) with respect to dg and show
that d; is the maximum.
]
Theorem 3. The maximum expected label accuracy in case of overlap between a query segment of length dy and an event of length de when d, 2 7de is
(*(r) = ((d)) = 27 (2 + 472 + 47 42 + 1.
(11)
8

---

Published in Transactions on Machine Learning Research (09/2025)
Proof. See Appendix A.3 for the proof. We substitute dy for de in Eq. 9
Theorem 4. The number of queries Biix (cost) that needed by FIX to maximize the expected label accuracy in case of overlap for an audio recording of length T when de = is
T BFxx
(12)
dy
Proof. T/BrIx = dy which by Theorem leads to maximum label accuracy.
7
In summary, Theorem 1 gives us an expression f(dy) for the expected label accuracy when query segments of length dg are used to detect events of length de and the presence criterion for the annotator is y, We use this to find the query segment length d; that maximize the expected label accuracy, leading to Theorem 2 Theorem
2 show the query segment length d that maximizes expected label accuracy for a given event length and annotator criterion. Further, by inserting di into Theorem f = f(d)), we get Theorem 3 which is the maximum achievable expected label accuracy for given annotator criterion y. We have omitted the case dg <de when deriving f (r), since maximizing the expected label accuracy in the case when the annolator presence criterion can not be fulfilled is not very interesting, since we can not get presence labels. Note that (7) is a function of only ", meaning that the maximum expected label accuracy is independent of the target event length when considering a single deterministic event. Finally, Theorem 4 show that an annolator needs to weakly label Bixx query segments for each audio recording to achieve the maximum label accuracy expectation, which can be seen as proxy for annotation cost.
There is arguably no simpler audio data distribution to annotate than when recordings only contain single event of deterministic length (except for when event occurs at all). We can therefore treat f*() as upper bound on the maximum expected label accuracy for any audio distribution. We demonstrate this empirically in the results in Section 0 However, in practice audio recordings often contain events that vary both in length and number. Let us therefore consider how the derived theory can be useful also in these cases.
4.3 Stochastic Event Length
Events may vary in length according to some event length distribution. Let p(de) denote the probability of the outcome that an event has length de, and let de ~ p(de) denote that de is a sample from that distribution. The expected label accuracy over distribution of event lengths for a given y and query segment length d, can then be computed as
Ede-p(da) [f(d,]] 1- f(dg)o(d)ddd.
(13)
While we do not provide a closed form solution for this, we can solve the integral in Eq. 13 by numerical integration. Note that de in Theorem 2 depends on the single event length de, and to find for a distribution we would need to solve Eq. 13 for range of dy and find the one that leads to the best label accuracy. However, for some event length distributions, sctting de to the average of the distribution turns out to be a good heuristic. We perform simulation study in Section 6.1.2 lo support these clains.
4.4 Multiple Events
There may be multiple (M) events present in given audio recording. In Figure accuracy for all possible occurrences of query segment di in recording with two events (M 2). Note that we have put the subscript t on the query segment (4:) instead of the event as in the prior analysis. This formulation is entirely equivalent, but when talking about multiple events it is more intuitive to consider them as fixed in time for a given recording, and that the query segments occur relative them at random. There are now two regions where overlap occurs, one around e and one around 62. On average we get 2A/2(d + dg) = A/(de + dg) = f(d)). That is, the theory we derived for the single event case explains the multiple event case.
show the label
9

---

Published in Transactions on Machine Learning Research (09/2025)
d
de
de
-==0
e1
e2
1.0
Fle. qt. R.Y)
0.5
A
A
0.0
0
de + dq
de dq
t
Figure 5: Top panel: Two events (M 2) of length de that are fixed in time within recording of length T, and a query segment (-d, + t, Bottom panel: The resulting label accuracy of d for t E [0, 7 - dg): simulating that the can appear anywhere at random in time in relation to the events. As before, when there is overlap between the query segment and an event the label accuracy is below 1, otherwise it is always 1.
However, for this to hold we need assume that for any event the closest other event is least dy away in time. In Figure 5 this holds since the start of 62 is at least d, away from the end of e₁. If this assumption holds then the expected label accuracy for multiple events is f(dy). The assumption is plausible if events are sparse in relation to dg- Note that d; E (0, de 2+ for E (0, 1] according to Theorem 2 That is, when considering the optimal query length dī this assumption translates to that events should be no closer than approximately 1.72d, for 7 = 1, 0.81d, for y= 0.5, and for y -> O. We perform simulation study in Section 6.1.3] to see the effect of breaking this assumption, and we leave it future work to derive the expected label accuracy in case of overlap for multiple events.
4.5 The Expected Label Accuracy of an Audio Recording
We now know the expected label accuracy of query segment given event overlap, and how to use this for a stochastic event lengths and multiple events. We can use this to derive an expression for the expected label accuracy of and audio recording of finite length (I") that has multiple (M) stochastic event lengths (de p(d))).
Theorem 5. The expected label accuracy for an audio recording of length T, with M events of stochastic event length de p(d.) that are spaced at least dg apart is
2Md2₂
2Md
Mdg
Edd-p(dd)
+
(14)
Tdg
T
T
Proof. We will do this proof by picture. In Figure 5 we have two events (M 2), in general for M events the accumulated label accuracy in the cases of overlap is MA (the sum of the hatched red arcas), the total amount of overlapping cases is M + dg) and the total amount of non-overlapping cases is therefore T (de + dz) for an audio recording of length T. In the case of no overlap, the label accuracy is always 1, which means that the accumulated label accuracy in the case no overlap (sum the green hatched areas) is T (d + dg). Normalizing for the entire duration of the recording we arrive at
AM + T M(d. + +d,) 2Md22
2Md.y
Md,
+
+ 1,
(15)
T
Td,
T
T
and as before we can simply compute an expectation over the event length distribution.
Theorem tells us the expected label accuracy under FIX weak labeling with query segment length d, for audio recording of length T, with M events of stochastic event. length de p(de). If want to account
10

---

Published in Transactions on Machine Learning Research (09/2025)
for class label noise, where the annotator gives the wrong label with probability p, this can be included by simply scaling the whole expression in Eq. 14 by (1 - p). That is, the expected label accuracy for the cases of overlap allows us to express a variety of things about the expected label accuracy of an audio recording.
However, note that we have T in the denominator all terms except the term that is 1, meaning that if we let approach oo, then the expected label accuracy approaches That is, considering the accuracy of both absence and presence labels equally can lead to hiding the effect that we want to understand in this paper, which is the effect of d the accuracy of the presence labels. We could derive balanced accuracy in similar way above, but instead we choose to continue our analysis looking only at the expected label accuracy in the case of overlap.
4.6 Expected Label Accuracy given Overlap when dg åde
As result of the proof for Theorem 3 in Appendix A.3 get alternative dimensionless interpretation of the expected label accuracy when the query segment length is expressed as factor of the event d, 5de.
((dd)) (27 1-61)61-1 2,
(16)
77
and an expression for the ratio that maximizes it
d 27 + 22 + 27 +1
8
de =
(17)
2 +1
This alternative formulation illustrates that it is the ratio 8= dg/de /de that affects the expected label accuracy of a single event, and not the absolute lengths dg and de. Further, we can use this interpretation to rewrite Theorem 5
Md5 (-ō + 27) 2M E Ex-)())
(18)
T8
where 8 denotes a random variable with probability distribution p(5).
5 Simulating the Label Accuracy of FIX Weak Labeling
To validate the theory, we simulated FIX labeling of various audio recording distributions and compared the average simulated label quality with the theoretical results from Section 4.2 The code used for these simulations is released openly
We generated 1000 audio recordings of length T 100 seconds for each configuration. The mumber of events,
M, and the event length distributions varied across simulations, detailed below:
Single Event with Deterministic Length: We simulated recordings with M 1 event of deterministic length de 1 second.
Single Event with Stochastic Length from Normal Distributions: We drew event lengths from two normal distributions with the same mean but different variances (3V 0.1) and W/33, 1)), and from two normal distributions with different means but the same variance (W(0.5.0.) and V(5,0.1). For these simulations, M 1
Single Event with Stochastic Length from Gamma Distributions: We sample event lengths from two gamma distributions (offset by 0.5 seconds due to computation cost) with different shape paramoters but the same scale parameter (Gamma(0.8, 1) + 0.5 and Gamma(0.2, 1) + 0.5) with I
Single Event with Stochastic Length from Real Length Sample: We used the event length distributions for dog barks and baby cries from the NIGENS dataset (Trowitzsch et al. 2019 with
M 1.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Multiple Events with Deterministic Length: We simmlated recordings with multiple events (M = 30 and M 50) where each event had deterministic length of de = second.
For recordings with stochastic event. lengths or multiple events, the length of each of the M events was sampled from the specified distribution. Each sampled event was then placed randomly within the rocording. The slart time de of each event was drawn uniformly at random from [0, T del- If multiple events were present, overlapping events were merged into one presence event. For each generated audio recording, we simulated FIX labeling using different annotator presence criteria y E [0.01, 0.99] and range of query segment lengths dg. The query segment lengths were linearly spaced between & small fraction of the minimum event length observed in the distribution and a value several times the maximum observed event length.
We then computed the average label accuracy over the query segments that overlaps with an event in cach recording. For each query segment q we check if the annotator presence criterion (h(e, q) 2 7) fulfilled for any event F, where F. is the set of all events that overlap with q. If this is true for any of the events then is given presence label (1g- 1) otherwise it is given an absence label (4,- = 0). The label accuracy is then computed in a similar way as in Eq. 3 but since we can now have multiple events overlapping with the same query segment, we need to consider the union of all overlapping events Ueege when computing the label accuracy of assigning label by to that query segment. The total amount of overlap becomes (Weeze) 0 gl instead of le gl. However, when M this equivalent to Eq. 3 ((()eeFe) nigl = leng)) since [ = 1.
In this way, we simulated the effect of breaking the assumption that events are spaced least dy apart, and could better understand the effect this had when compared to the derived theory. Finally, for cach considered 7, empirically determined the maximum average label accuracy across all tested query lengths and the corresponding optimal query length. These empirical results were then compared to the theoretical predictions.
6 Results
In this section we present the results of the simulated annotation process, and show how these connect the derived theory. We start by looking at the expected label accuracy and the query segment length that maximize the expected label accuracy for FIX and ORC weak labeling, and then we relate this to the annotation cost.
6.1 Expected Label Accuracy given Overlap
We evaluate how different annotator presence criteria (7) influence the achievable label accuracy given overlap under FIX weak labeling. We first examine the case of single event with deterministic length, then extend our simulation study to stochastic event lengths, and finally to multiple events occurring within the same recording.
6.1.1 Single Event with Deterministic Length
The simulated results are derived using the simulation setup described in section with A 1 (a single event) and de (deterministic length). In Figure 0 we show the maximum expected label accuracy given overlap (left) and the corresponding query length that maximize the label accuracy (right) for different 7. (7) is the maximum expected label accuracy achievable with annotator presence criterion y for the considered event length. We can see that the simulated average label accuracy closely follows the expected label accuracy, and that the corresponding segment length leading to this maximum is the same in theory and simulation.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Maximum expected accuracy
1.0
(d,=1)
0.8
Optimal query length
simulated
3
2
0.2
(y)
simulated
0.0
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
Figure 6: In the left panel we show the maximum expected label accuracy, (7),, for different 7, and the average maximum label accuracy from the simulations. In the right panel we show the query length that leads to this maximum label accuracy in theory, for de = 1, and in simulation. The theory follows the simulations well.
In Figure we see that the annotator needs to hear more than 50% of the sound event to detect presence (7 0.5) then the highest achievable label accuracy is (0.5) z 0.76. This means that average there is around 34% segment label noise around the presence labels. We also see that the query length that gives the maximum label accuracy is dz 0.81. The gap to the ORC weak labeling method which always gives label accuracy of 1, is large especially for large In general, we can see how the maximum label accuracy deteriorates with a growing "/ and which query segment length to choose to maximize label accuracy in expectation.
6.1.2 Single Event with Stochastic Length
We now consider stochastic event lengths. We do this to better understand the effect the event length distribution on the maximum expected label accuracy and the optimal query length. We solve the integral in Eq. 13 by numerical integration over different event length distributions, and compare with the theory derived for single deterministic event length and simulations. In each figure present the derived thcoretical rules (*) and dē for the simplified event length distribution, the results from integration of Eq. 13 with different event length distributions p(de) (mumerical), and the simulated results using the procedure described in section (simulated) where event lengths are sampled from different distributions. Note that, since di is dcrived for a deterministic event length de, and require a choice this value, we set de to the average event length (μ) for each distributtion these experiments as heuristic. We then present the maximum expected label accuracy for different ? (left in figures) and the query segment length that maximizes the expected label accuracy (middle in figures), and the histogram for the considered event length distributions (right in figures).
In Figure and Figure we see that the mean and variance of the normal distribution have small (if any) effect on the maximum expected label accuracy, but the mean does affect which query segment length that maximizes the expected label accuracy. We also see that d follows the simulated and numerical optimal query length well for all considered normal distributions, when de is set the average event length (p) for the considered event length distribution. The average event length can be used heuristic value if we only know the average and not the true distribution to integrate over.
In Figure g we can see that gamma distribution does affect the maximum expected label accuracy, and that simply setting de to the average event length of the distribution leads to underestimating the optimal query length. Since it is not possible to optimize for both short and long events at the same time using FIX weak labeling, this type of distribution quite challenging.
In Figure 10 we validate the theory against a real sample of event lengths from either baby cries or dog barks. Numerical integration between the derived expression and the histogram predicts the simulations well.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Maximum expected label. accuracy
1.0
15.0
dg (de
normal(0.5, 0.1) normal(5, 0.1)
0.8
Optimal query length
12.5
dg (dp=
3
0.6
10.0
x x numerical
numerical
('())
pld.)
2
7.5
simulated
X
numerical
simulated
numerical
0.2
simulated
2.5
simulated
0.0
0.0
0.0
0.4
0.8
0.0
0.2
0.4
0.6
1.0
2
de
Figure We validate the theory for stochastic event lengths drawn from two normal distributions with different means, but the variance. We show the expected label accuracy (left panel), the optimal query length (middle panel), and the considered event length distributions (right pancl).
Maximum expected label accuracy
1.0
12
(d
normal(3, 0.1)
0.8
Optimal query length
normal(3, 1)
=d)=))
0.6
(y(y
numerical
numerical
2.0
0.4
numerical
simulated
1.5
simulated
X
numerical
1.0
0.2
simulated
0.5
simulated
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0
4
Y
de
Figure 8: We validate the theory for stochastic event lengths drawn from two normal distributions with different variance, but the same mean. We show the expected label accuracy (left panel), the optimal query length (middle panel), and the considered event length distributions (right panel).
Maximum expected label accuracy
1.0
(d
gamma(0.2, 1) 0.5 gamma(0.8,
0.8
Optimal query length
di (d)=p)
0.6
X
numerical
('(y)
numerical
p(de)
simulated
0.4
numerical
2-
simulated
numerical
simulated
simulated
0.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
10
2
de
Figure We validate the theory for stochastic event lengths drawn from two gamma distributions with different shape parameters, but the same scale parameter. We show the expected label accuracy (left panel), the optimal query length (middle panel), and the considered event length distributions (right panel).
14

---

Published in Transactions on Machine Learning Research (09/2025)
Maximum expected label. accuracy
1.0
baby dog
0.8
Optimal query length
d,
0.6
x
numerical
('(y)
X numerical
pldd)
simulated
X
numerical
simulated
numerical
1
0.2
simulated
simulated
0.0
0.2
0.4
0.0
100
de
Figure 10:
Barking dog and crying baby event length distributions from the NIGENS dalasel. (Trowitzsch) These annotations have been made with strong guarantee for high quality onsets and offsets.
et al. 2019).
Maximum expected label accuracy
1.0
(de =p
M=30
0.8
Optimal query length
3
x
numerical simulated
0.6
pld.)
2
0.4
(y)
0.2
numerical
simulated
0.0
0.6
de
Figure 11: We validate the theory for multiple events of length d, We show the expected label accuracy (left pancl), the optimal query length (middle panel), and the considered event length distributions (right panel). Note that presence events longer than can occur if two more events overlap. We sample 30 events with event length de occur at random for each audio recording in this simulation.
6.1.3 Multiple Events with Stochastic Length
In these simulations we allow multiple events to occur in the same recording (M > 1). In Figure we show the results of sampling 30 events of length de = for each audio recording. This does have effect the expected maximum label accuracy and the corresponding query length, though the impact is relatively modest. In Figure I2 we show the results of sampling 50 events of length d, = 1 for each audio recording, This is an extreme case, where the event density of the recording is very high. These results demonstrate that even under high event densities, the simulated maximum accuracy follows the theoretical predictions closely. This confirms that the single-event theory provides a robust upper bound even when assumptions about event sparsity are strongly violated.
6.2 Annotation Cost for Maximum Expected Label Accuracy given Overlap
Achieving maximum expected label accuracy comes at a cost, and understanding this cost trade-off essential for practical annotation efforts. The cost model we employ accounts for both the time spent listening audio and the effort required to label presence absence events.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Maximum expected label. accuracy
1.0
dg (d. =
M=50
0.8
Optimal query length
3-
x
numerical
simulated
0.6
p(d.)
2
0.4
('()
2
0.2
numerical
1-
simulated
0.0
0.0
0.2
0.6
0.2
0.8
de
Figure 12: We validate the theory for multiple events of length d, = 1. We show the expected label accuracy (left panel), the optimal query length (middle panel), and the considered event length distributions (right panel). Note that presence events longer than can occur if two more events overlap. We sample 50 events with event length de occur at random for each audio recording in this simulation.
6.2.1 Formalizing the Cost Model
The derived theory for the optimal query length allows us to analyze the cost of achieving maximum expected label accuracy under different annotator models for FIX weak labeling. We assume that the whole audio recording of length T is listened to. The key difference in cost between the FIX and ORC weak labeling method is the number of segments (B) that need to be given presence absence label. We formalize cost model as:
C(T, B) = (1 r)T +rB
(19)
where 1 - r represents the cost of listening to one second of audio (cost per second), and r represents the cost of answering a query (cost per query). The term (1 - r)T therefore represents the cost of listening T seconds of audío, and the term rB the cost of assigning B presence absence labels. Using this cost model, calculate the cost of annotating de = using either FIX or ORC weak labeling. For FIX, the nunber of queries that maximize expected label accuracy is given by Bix = T'dd (see Theorem 4 For ORC, achieving an expected label accuracy of requires at least BOCc = 2M + queries.
audio recording of length T with M sound events of length
In practice, we not know the number of events M. To explore potential overestimation of M when, for example, using a weak labeling process that tries mimic ORC weak labeling, we model BoRc as a multiple of the necessary number of queries: BoRc BBORC where {1, 8) represents the degree of overestimation. This approach captures scenarios where the number of events are either precisely estimated (s = 1) or significantly overestimated (s 8) during the annotation process. In practice, Bonc could be set based on a bound on M. For example, by estimating a maximum expected number of sound events in a recording, Mmax: based on knowledge of typical event density, or characteristics of the audio recording. We assume that overestimation by more than a factor of is unlikely. The relative cost between FIX and ORC weak labeling can then be computed as:
CFIX
C(T, BFix)
(20)
CORC
BoRc)
where ratio larger than indicates that FIX is more costly than ORC, and ratio smaller than 1 indicates that FIX is less costly than ORC.
6.2.2 Effect of annotator criteria (7) and cost ratio (r).
Figure 13 (left) shows the relative cost for varying annotator criteria y [0.1, 1). As y + 0.1, the cost of FIX increases sharply, reflecting the need for an infinitely large number of queries to achieve an expected label accuracy of 1. In practice, achieving perfect accuracy with FTX is infeasible due to the associated cost.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Default parameters: T=100, M=1
Default parameters: T=100, M=1, y=0.5
30
CulCorn
10-
2-
5-
Figure 13: The relative cost of FIX and ORC for varying annotator criteria 7 (left), and cost ratios r (right). The default parameters are: 100, 0.5, M = and 7 0.5. We simulate overestimating the number of needed queries BoRc x(2M + 1) by factor of for 21, 4, to how this alfects the relative cost. The cost of FIX is greater than the cost of ORC above the dashed red line where the cost ratio is 1.
parameters: T=100, r-0.5, y=0.5
parameters: M=1, r=0.5, y=
2.5
2.0
5
GraCoox
GolCo
1.5
1.0
20
60
80
100
20 40
60
80
100
Figure 14: The relative cost of FIX and ORC for varying number of sound events M (left) and recording lengths T (right). The default parameters overestimating the number of needed queries Boac = s(2M 1) by a factor of for E {1, 4, 8} to see how this affects the relative cost. The cost of FIX is greater than the cost of ORC above the dashed red line where the cost ratio is
T 100, 0.5, M 1 and 7 0.5. We simulate
For higher 7, the cost of FIX becomes more comparable ORC. However, combining this with Theorem 3 reveals that FIX can either match ORC in cost but with lower expected accuracy or achieve similar accuracy at much higher cost.
The right panel of Figure 13 examines the impact of the cost ratio r. Across all tested values, ORC remains less costly than FIX the default setting (T = 100, r = 0.5, = 0.5, M = 1). This confirms that the relative cost advantage ORC is robust to changes r.
6.2.3 Effect of number of events (M) and recording length (T).
Figure explores the impact of M and T on the relative cost. In the left panel, we see that for 1, ORC is less costly than FIX when the number of events below 60. However, as increases to 8, FIX becomes
17

---

Published in Transactions on Machine Learning Research (09/2025)
less costly when at most 10 events are present. These results indicate that the relative cost depends heavily on the density of sound events in the recording and the estimated annotation budget for ORC.
In the right panel, varying T shows a similar trend. For shorter recordings (high event density), ORC loses its cost advantage. However, it's important to note that the maximum achievable expected label accuracy with FTX under default settings (7 0.5) (0.5) R 0.76, whereas ORC achieves 1.0. In such cases, the additional cost of ORC may be justified by the significantly higher label quality.
While these results indicate that the relative cost depends the sound event density, we should remember that we are considering weak labeling of presence events. This implies that all M events in this analysis are treated as non-overlapping, as the annotation task does not consider temporal overlaps for this analysis. The scenario of M > 60 non-overlapping events of length 1 in a recording of length T = 100 is therefore unlikely in practice. Similarly, estimating 10 events as 80 (modeled by = 8) for an audio recording of length T = 100 represents a substantial overestimation and scems improbable given the capabilities of modern sound event detection tools.
7 Related Work
This work introduces a framework for characterizing segmentation label noise in FIX weak labeling, a largely unexplored area. Below, we review studies addressing noisy labels and approaches to mitigate their effects, with a focus on weak labeling in audio and related domains.
7.1 Understanding Noisy Labels
Noisy labels are a partial description of the target model, influencing its performance. Early work by Liang et al. (2009) introduced the concept of measurements for conditional exponential families, encompassing labels and constraints for model learning with minimal human input- goal shared by this work.
In deep learning, the ability of models to overfit noisy labels has prompted studies into the relationship between noise rate and generalization (Zhang et al., 2021 Chen et al.) 2019). Rescarch on class label noise often assumes a noise transition matrix (Li et al. 2021 but rarely considers spatially correlated errors like those arising in segmentation tasks (Yao et al. 2023). For audio, Hershey et al. (2021) demonstrated that training on strongly labeled data yields better results than weakly labeled data, highlighting the need for precise labels, particularly in evaluation. In multi-modal (asks, such as audio-visual video parsing, a key challenge is modality-specte
label noise, where video-level tag may apply to the audio stream but not the visual, vice versa (Cheng et al. 2022 Zhou et al. 2024h. Our work focuses specifically characterizing the segment label noise that arises from the temporal misalignment between fixed-length query segments and true event. boundaries.
7.2 Mitigating Noisy Labels
Several strategies address noisy labels, including regularization techniques like dropout (Srivastava et al. 2014), data augmentation (Shorten Khoshgoftaar 2019), and specialized loss functions (Fonseca et al, 2019). For weakly labeled audio, Dinkel et al. (2022) proposod a pscudo-labeling approach, iteratively refining labels to improve training performance. Despite these advances, most methods focus on training labels and offer limited insights into noisy evaluation labels, underscoring the need for frameworks that quantify label noise, such the one proposed in this work.
7.3 Strong vs. Weak Labeling
Strong labeling, where the annotator provides the event boundarics and the class label, while often precise, is resource-intensive and subject to annotator variability (Mesaros et al., 2017). In bioacoustics, experts use spectrograms for efficient annotation (Cartwright et al. 2017), but the reliance on specialists limits scalability. Weak labeling, by contrast, simplifics the annotation task, which is especially important for crowd-sourced annotations, enabling broader data collection (Martin- Morato Mesaros 2023). However, segment. label noise, especially at event. boundaries, remains a significant. challenge.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Dataset
Task
Fixed Length
CHIME Foster et al., 2015a]
Single-pass multi-label
seconds
AudioSet (Gemmeke al. 2017b
Single-pass multi-label
10 seconds
MAESTRO Real (Martin- Morato & Mesaros 2023
Single-pass multi-label
10 seconds
OPenMIC-2018 (Humphrey al.
2018b
Multi-pass binary-label
10 seconds
Table 1: Large-scale audio datasets using variations of FIX weak labeling.
Large-scale audio datasets employing FIX weak labeling are summarized in Table tasks are single-pase pass multi-label and multi- pass binary-label annotation (Cartwright et al. 2019). Single-pass multi-label annotation asks annotators to recognize the presence of multiple event classes during a single pass through the data. In contrast, multi-pass binary-label annotation asks annotators detect the presence or absence of single event class at time through multiple passes through the data.
Two common annotation
Cartwright et al. (2019) studied the trade-offs between these tasks and found that binary labeling is preferable when high recall is required. For example, AudioSet (Gemmeke et al. 2017b) employs single-pass multi- label annotation with non-overlapping 10-second segments, which limits temporal resolution. Conversely, MAESTRO Real (Martin-Morato & Mesaros 2023) uses overlapping 10-second segments with a 9-second overlap, increasing the accuracy of the derived labels.
The choice of segment length and overlap significantly impacts the utility of weak labeling. For example, while overlapping segments increase label accuracy (Martin-Morato & Mcsaros 2023), they still fail to distinguish events occurring close in time. Current work aims to better undersland the effect of different choices of the segment length for FIX weak labeling.
7.4 Contributions of This Work
Existing research focuses predominantly on class label noise or assumes noise independence. This work ex- tends these efforts by characterizing segment label noise specific to FIX weak labeling, providing a foundation for improving both training and evaluation processes in weakly labeled datasets.
8 Discussion
FIX labeling has been employed many works, with varying degrees of complexity. Theorem provides a useful rule of thumb for selecting the best segmentation length for a given event length, and Eq. 13 provides way to use this theorem to analyze stochastic event length distributions. Our results suggest that, in most cases, knowing the average event length provides good estimate, but understanding the (approxinate) distribution of event lengths improves the analysis.
Implications for Practical Annotation
Our theoretical analysis positions FTX weak labeling as baseline strategy: it is simple and scalable but inherently limited by segment label noise. By quantifying this limitation, our work provides the necessary theoretical justification for moving towards adaptive methods that aim to approximate the oracle process.
The analysis highlights the trade-offs label accuracy and annotation cost between FIX and ORC weak labeling. While FIX can be less costly under specific conditions (e.g., high event density), these conditions are unlikely to occur in real-world annotation tasks. Furthermore, even in cases where FIX is less costly, its significantly lower label accuracy (f* 0.76 vs. 1.0 for ORC) can negate its cost advantage. This gap represents the 'accuracy cost of weakness" that is inherent to any non-adaptive weak labeling strategy. This gap only increases when dg is chosen sub-optimally, which is often the case in practice due to budget considerations. Given the rarity of extreme event densities and the importance of high-quality labels, ORC is likely the better theoretical choice for most annotation tasks.
However, ORC weak labeling is not available in practice since it uses the true event boundaries. This provides clear theoretical justification for developing adaptive weak labeling methods, which aim to approximate
19

---

Published in Transactions on Machine Learning Research (09/2025)
the ORC process. For instance, methods that use active learning change-point detection to define query boundaries (Martinsson et al. 2024 Kim et al. 2023) are practical attempts to bridge the gap between FIX and ORC. Martinsson et al. (0244) empirically evaluates an adaptive change-point detection method (A-
CPD) and compares that to FIX weak labeling and ORC weak labeling, showing the benefit of an adaptive weak labeling method for annotation of sound events. Our work provides the tools to quanlify the maximum potential accuracy gain for such methods over a simple FIX baseline, offering a principled way to evaluate the trade-off' between the complexity of an adaptive strategy and its achievable accuracy. Future research should focus on mitigating the potential biases when modeling ORC weak labeling (e.g., annotation errors, overfitting to sparse events) while retaining its theoretical advantages.
Implications for Model Evaluation
Despite the extensive focus on noisy training labels, evaluation labels are often implicitly assumed to be perfect. As emphasized in the introduction, inaccurate evaluation labels present. a significant challenge. When noise is present in both training and evaluation data, we risk selecting models that merely replicate the evaluation noise, potentially overlooking those with superior generalization abilities. This echoes the central concern highlighted by [oörnttz et al. (2014). We can use Theorem 3 to understand the properties of the best performing model when the evaluation data contains FIX weak labels. For example, for 7 = 0.5 the annotations will at most have expected label accuracy of (0.5) R 0.76. The "best" performing model will therefore be a model that mimics this specific noise profile. Our theory thus provides better understanding of the target that models are optimizing for when evaluated on weakly labeled data.
This is also relevant for standard sound event detection (SED) evaluation metrics, such as the segment- basod F1 score (Mesaros et al., 2016), which divide audio into fixed-length segments. When using ground truth labels for evaluation, we effectively have an annotator with 7 + The expected label accuracy then becomes I (dgy de/(de + dg), where dg is the segment length. This formula shows that small dg minimizes segment label noise, but choosing very small segment length negates the desired effect of miligating lemporal imprecision in the ground truth and also increases computational cost. The theory presented here can help inform such trade-offs.
Theoretical Properties and Validation
The expression for expected label accuracy derived in this paper applies to the simplest scenario, where only a single event with deterministic length is present. In all of our results, we observe that f*) is greater than or equal to the expected and average label accuracy that FIX weak labeling achieve for more complex iostss This suggests that (*()) can be considered an upper bound for a given annotation process. However, formal proof showing that adding more events or introducing event length variability leads to harder distribution to annotate is beyond the scope of this paper.
To connect this theoretical framework to a real-world setting, we conducted an empirical analysis using the weakly and strongly labeled versions of the AudioSet dataset, as detailed in Appendix A..] By treating the 10-second weak labels of AudioSet as the output of a FIX process, we calculated the empirical label accuracy against the corresponding strong labels. Our theoretical model, when applied the event length distribution of the 'Animal' class, accurately predicted this empirical accuracy for a presence criterion of "y 2 0.26. This serves as empirical validation of our framework on large-scale dataset.
Generalization and Future Directions
While work is grounded in audio event detection, the core principles are broadly generalizable because the mathematics depends only on two fundamental quantities: the event duration de, and the query segment length dg. These quantities can be directly mapped to other domains, e.g,, video action spotling, electro- cardiography, seismology, or high-frequency trading. Consequently, our core results (Theorems 33) transfer directly to these domains, provided three conditions hold: (i) the events uniformly distributed locally in time, making the uniform relative offset a reasonable model (as empirically verified in Appendix A.4), (ii) the annotation process relies observing a minimum fraction ς of an event, and (iii) the events suf-
20

---

Published in Transactions on Machine Learning Research (09/2025)
ficiently sparse. This framework also be extended to higher dimensions, such analyzing the weak labeling of rectangles in images or cubes in point clouds.
Finally, if the same presence criterion 7 is applicable for all event classes, Theorem I aaplies to the joint event length distribution. However, real-world presence criteria for different event classes may vary, requiring more complex models. Future empirical studies on annotator behavior could help refine this model and improve its practical applicability.
9 Conclusions
This study introduces a novel theoretical framework for understanding the trade-offs between label accuracy and annotation cost. weak labeling methods, particularly focusing on sound event detection where weak labeling is often employed reduce annotation costs. We specifically compared fixed-length (FIX) and oracle (ORC) approaches.
We have demonstrated that FIX weak labeling, while cost-effective in specific scenarios, is inherently limited by segment label noise. The expressions we derived theoretically provide actionable insights into optimizing segment length for maximizing expected label accuracy under FIX. However, these results also underscore the fundamental trade-offs: shorter segments improve alignment with event boundaries but significantly increase annotation cost, while longer segments reduce cost at the expense of accuracy. In addition, how short these segments can be chosen depends on the ability of the annotator to detect presence of fractions of the events. In contrast, ORC labeling achieves perfect accuracy but can incur higher costs if events are very dense and the number of events are overestimated.
Our findings have several practical implications:
Annotation Strategy: FIX weak labeling remains a robust, scalable choice for many practical applications. However, when high label accuracy is essential, ORC weak labeling-on or adaptive methods approximating it should be prioritized.
Adaptive Techniques: Theoretical justification for adaptive weak labeling methods, c.g., methods based on active learning or iterative refinement, that mimic ORC weak labeling, which suggests promising avenues for improving annotation efficiency without compromising accuracy.
Evaluation Criteria: Our analysis highlights the potential biases introduced by segment-leve levell label noise in evaluating sound event detection models. Therefore, carcfully aligning evaluation criteria with the intended model properties is critical.
Future research should address several limitations and extensions identified in our study. Developing practical approaches that reliably mimic ORC weak labeling by estimating the query segments without introducing lot of unwanted bias in the labels remains open challenge. Additionally, extending this framework to multi-dimensional data and multiple presence classes could broaden its applicability to other domains, such as medical imaging and point clouds.
In conclusion, the insights presented this work offer a foundation for optimizing weak labeling processes, balancing cost and accuracy to meet the needs of diverse machine learning applications. By refining anno- fation strategies and leveraging adaptive methods, researchers enhance the quality of labeled datasets. This, in turn, will drive adyancements in supervised learning across domains, building upon the foundational understanding presented in this work.
Acknowledgments
The authors would like to express their gratitude to the following individuals and organizations for their contributions and support:
Annamaria Mesaros: Provided insightful questions and feedback early draft of this work.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Magnus Oskarsson: Contributed proof sketch for Theorem 2 and gave feedback early draft.
Edvin Listo Zec: Assisted with the proof of Theorem 2
This work was supported by the Swedish Foundation for Strategic Rescarch (SSF; FID20-0028) and Sweden's Innovation Agency (2023-01486).
References
Juan P. Bello, Claudio Silva, Oded Nov, R. Luke Dubois, Anish Arora, Justin Salamon, Charles Mydlarz, and Harish Doraiswamy. Sonye: system for monitoring, analyzing, and mitigating urban noise pollution. Commun. ACM, 62(2):68- 77, January 2019. ISSN 0001-0782. doi: 10.1145/3224204 URL https://doi.
01g/10. 1144/3224204
Mark Cartwright, Ayanna Scals, Justin Salamon, Alex Williams, Stefanic Mikloska, Duncan MacConnell, Edith Law, Juan P. Bello, and Oded Nov. Seeing sound: Investigating the effects of visualizat ions and complexity on crowdsourced audio annotations. Proceedings of the ACM on Humon-Compaln Interaction, doi: 10.1145/313466:
(CCSCW):-246, 2017. ISSN 25730142.
Mark Cartwright, Graham Dove, Ana Elisa Méndez Méndez, Juan P. Bello, and Oded Nov. Crowdsourcing Multi-label Audío Annotation Tasks with Citizen Scientists. Conference Human Factors in Compuling Systems Proceedings, pp. 1 11, 2019. doi:
Honglic Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visus dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and Utilizing Deep Neu- ral Networks Trained with Noisy Labels. In Proceedings of the 36th International Conference on Machine Learning, pp. 1070- PMLR, May 2019. URL tttp:http:///p ISSN: 2640-3498.
mlr htt...prees/////caaa hwt./////c///.../////// html
Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denoising for weakly-supervised andio-visual video parsing. In Computer Vision ECCV 2022: 17th European Conference, Tel Avin, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pp. 431-448, Berlin, Heidelberg, 2022. Verlag. ISBN 978-3-031-19829-8 doi: 10.1007/976-3-001-19993- 25.
URL https://doi... org/ 100.01107000011-1111111111110--1 3-031-
Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, and Yujun Wang. Pseudo Strong Labels for Large Scale Weakly Supervised Audio Tagging. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings, 2022- 340, 2022. ISSN 15206149. doi: 10.1109/ arXiv: 2303. Publisher: IEEE ISBN: 9781663405409.
Eduardo Fonsoca, Frederic Font, and Xavier Serra. Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 16 20, October 2019. doi: 10.1109, 10.1109/WASPPR 2.0ASPAA.0099.003220 URL httpp://ieeeepplrre.iie ieee. org/ document/8997749
ISSN: 1947-1629.
Peter Foster, Siddharth Sigtia, Sacha Krstulovic, Jon Barker, and Mark D. Plumbley. Chime-home: A datasct for sound source recognition in a domestic environment. 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, WASPAA 2015, pp. 1 1-5, 2015a. doi: 10.1109/WASII LLLLL///AAAAAA.155755888855
Peter Foster, Siddharth Sigtia, Sacha Krstulovic, Jon Barker, and Mark D. Plumbley. Chime-home: A datasct for sound source recognition in a domestic environment. In 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5, 2015b. doi: 10.1109/WASPAS 22155..3200173388
22

---

Published in Transactions on Machine Learning Research (09/2025)
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776 780, 2017a. doi: 10. /IB//ICASSP...20 CASSS..2017.770 2017.795226.
Jort F. Gemmeke, Daniel P.W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing- Proceedings, pp- 776 780, 2017b. ISSN 15206149. doi: 1A110///CASSPP-017.77990
Nico Görnitz, Anne Porbadnigk, Alexander Binder, Claudia Sannelli, Mikio Braun, and Marius Kloft. Learning and Evaluation in Presence of Non-i.i.. Label Noise. In Samuel Kaski and Jukka Corander (eds.), Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pp. 293-302, Reykjavik, Iceland, 22 22-25 Apr 2014. PMLR. URL pttps://proceedingss http:ocoddiinnnl...orrrrssss///e wttppp/eeeve/se/tttaaaea html
Mueller,
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. Can architectures for large-scale audio classification. In Proceedings of the IEEE International Conference on Acouslics, Speech and Signal Processing (ICASSP). TEEE, 2017. URL / org/abs/1609
09430
Shawn Hershey Daniel P.W. Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R. Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings, pp. 366 366-370, 2021.
ISSN 15206149. doi: 10.1109 TCA0//39SS9P92 201222..2221111 0021.9414577
Eric Humphrey, Simon Durand, and Brian McFee. Openmic-2018: An open data-set for multiple instrument recognition. In ISMIR, pp. 138-444, 2018a.
Eric J. Humphrey, Simon Durand, and Brian McFee. OpenMIC-2018: An open dataset for multiple instru- ment. recognition. Proceedings of the 19th International Sociely for Music Information Retrieval Confer- ence, ISMIR 2018, pp. 438 444, 2018b.
Hoyoung Kim, Minhycon Oh, Schyun Hwang, Suha Kwak, and Jungseul Ok. Adaptive superpixel for active learning in semantic segmentation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.
2023. doi: 10.1109 10.1109/ICCV350 /ttp:///CC51111.22...0000
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In J. Moody, S. Hanson, and R.P. Lippmann (eds.), Advances in Neural Information Processing Systems, volume 4. Mlrgan-KKuufmamn 1991. URL attps://procedngg.enrrss.. cttpsss.ccpppppppppp ftt.ss/ppppr/19//////
pdf
Amurag Kumar and Bhiksha Raj. Audio event detection using weakly labeled data. In Proceedings of the 24th ACM International Conference on Multimedia, MM '16, pp. 1038-1047, New York, NY, USA, URL
2016. Association for Computing Machinery. ISBN 9781450336031. doi:
https://doft org/ 10. wtt.or///.//..//....e 1111111144244..... 2994444128829241000
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably Fnd-to-end Label-noise Learning without Anchor Points. In Proceedings of the 38th International Conference on Machine Learning, pp. 6403 6413. PMLR, July 2021. URL https://proceeedinge. mlr. press//139/1121/1111 httmlmmmm11m111mmm11111 2640-3498.
ISSN:
Percy Liang, Michael I. Jordan, and Dan Klein. Learning from measurements in exponential families. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 641-648, Montrcal Quebec Canada, June 2009, ACM. ISBN 978- --0558--15-- doi:
URL https:
7/d1. acm. www//oi/oo.ooooooooo.oooog....tt http:///......11115155333333731. 1553457
23

---

Published in Transactions on Machine Learning Research (09/2025)
Irene Martin-Morato and Annamaria Mesaros. Strong Labeling of Sound Events Using Crowdsourced Weak Labels and Annotator Competence Estimation. IEEE/A /AM Transactions on Audio Speech and Language Processing, 31:902 914, 2023. ISSN 23299304. doi: 10.1109, 2A00///TTASLLP.22222333333
John Martinsson, Olof Mogren, Maria Sandsten, and Tuomas Virtanen. From Weak to Strong Sound Event Labels using Adaptive Change- Point Detection and Active Learning. In EUSIPCO 2024 32nd European Signal Processing Conference, 2024. URL
Brian McFee, Justin Salamon, and Juan Pablo Bello. Adaptive Pooling Operators for Weakly Labeled Sound Event Detection. JEEE/ACM, Trans. Audio, Speech and Lang. Proc., 26(11):2180-2199, November 2018. ISSN 2329-9290. doi:
URL https://doi.orrg10... 1109/AAAAT
2018. 2858559
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Metries for polyphonic sound event detection. Applied Sciences (Switzerland), 6(6), 2016. ISSN 20763417. doi: D..3300/app6001662
Annamaria Mesaros, Toni Hcittola, and Dan Ellis. Datascts and evaluation. In Tuomas Virtanen, Mark Plumbley, and Dan Ellis (eds.), Compulational Analysis of Sound Scenes and Events, chapter 6, pp. 147-179. Springer Cham, 2017.
Trene Martin Morato and Annamaria Mesaros. Mats multi-annotator tagged soundscapes, May 2021. URL attps://doi: org/oogg/// 10/1110.2220 www.5////eood..7700000 477496
Irene Martin Morato, Manu Harju, and Annamaria Mesaros. Maestro real- multi-annotator estimated strong labels, 2023.
Ankit Shah, Anurag Kumar, Alexander G. Hauplmann, and Bhiksha Raj. A Closer Look at Weak Label Learning for Audio Events. pp. 1-,0, 2018. URL http://arxiv hwtp://rrgabb///00// 09288
Connor Shorten and Taghi M. Khoshgoftaar. survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1):60, July 2019. ISSN 2196-1115, doi: 10.1180//400337-00100007770 URL https:
Janvijay Singh and Raviraj Joshi. Background sound classification in speech audio segments. In 2019 International Conference on Speech Technology and Human- Computer Dialogue (SpeD), pp. 1-6, 2019.
doi: 10.1109/SPED) S.PED.2019.00 2019.990
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac- Gil Loc. Learning from Noisy La- bels with Deep Neural Networks: A Survey, March 2022. URL http:///axxvvvo.//aa// wrgg/aa////0/../////////// 08199 arXiv:2007.08199
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Tlya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15(56): 1929-1958, 2014. URL http://mww..orrr////e//////taaas html
Ivo Trowitzsch, Jalil Taghia, Youssef Kashef, and Klaus Obermayer. The NIGENS General Sound Events Database. pp. 1-5, 2019. URL
Nicolas Turpault, Romain Serizel, Emmanuel Vincent, Nicolas Turpault, Romain Serizel, and Emmanuel Vin- cent. Analysis of weak labels for sound event tagging. 2021. http::://aa...rr....aaaaaaaaaa
Jiachen Yao, Yikai Zhang, Songshu Zheng, Mayank Goswami, Prateek Prasanna, and Chao Chen. Learn- ing to segment from noisy annotations: A spatial correction approach. In The Eleventh International Conference on Learning Representations, 2023. URL httphttsss//peeeviiieee wwt/ hett/forrumndddcc___ooM
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Rocht, and Oriol Vinyals. Undorstanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, March 2021. ISSN 0001-0782, 1557-7317. doi: 10. 10.1145/3446776.
URL https://dl......../ acm. htt.:aaa.m.momm.o.ooo//oooo /ttp///////111111/4///777777
24

---

Published in Transactions on Machine Learning Research (09/2025)
Jinyuan Zhou, Dading Guo, Yuhang Zhong, Jize Liu, Yu Yang, Yan Wang, and Kinng-suan Tan. Advancing weakly-supervised audio-visual video parsing via segment-wise pseudo labeling. International Journal of Computer Vision, 132:5308 5329, nov 2024. doi: 10.1007//1122633004--002222 URL atttps://oos org/
10. 000.107/s12228-024-01111
A
Appendix
We do not include all simplifications of expressions in the proofs, but we do provide the code for sym- bolic mathematics solver (SymPy) at GitHutf where all results can be verified. The notebook named syymboli__veiffeattion
verification _f annalysis.pyynb" can be used to verify the analysis.
A.1 Proof of Theorem
We will derive an expression for the expected query segment accuracy given overlap with a single event terms of des dgs and 7, under all possible assumptions which will prove Theorem !
Proof. We need to consider two main assumptions. The first assumption is that the presence criterion for the annotator can be fulfilled, that is, dg 2 yde, and the second assumption that the annotator presence criterion can not be fulfilled, that is, dg -de- This happens if the query segment length is short that it never cover large enough fraction of the event of interest make presence detection feasible.
Assumption 1. The annotator presence criterion can be fulfilled (d, 2 yde).
Under this assumption there are two possible cases for the relation between dg and de, either the event length is longer or equal to the query segment length, d, 2 dg (case i), or the event length is shorter than the query segment length, de < d, (case ii). In Figure 15 we plot the query segment accuracy, F(es, q, 7,),,, for E [0, de +dy for case (i) on the left, and case (ii) on the right. We describe in more detail in Appondix A.1.1 A.1.1 how the query segment accuracy behaves as a function of different amounts of overlap between the query segment and the event. Briefly, what we see in Figure 15 is that initially there is arbitrarily little overlap (4 and ((*)), an absence label given to the query segment and the accuracy is therefore Then the accuracy decrease linearly with the amount of overlap until the presence criterion is fulfilled and a presence label is given (t(e) and z(")). After that, the accuracy lincarly increase with the amount of overlap between the event and query segment until we reach a ceiling for the accuracy when either the whole query segment is inside the event (t(9) or the query segment covers the whole event ()) Finally, the overlap between the query segment and the event starts to decrease again (th") and f(ii)), and everything is symmetrical.
We contimue by dropping the case superscripts show in the figure for Ai...., As and to...., ts. and only provide the full proof for case (i), but the proof for case (ii) is similar. In both cases the area A in Eq. be divided into five distinct parts:
8 can
= 2A + 2AA + As
(21)
where A₁ and A2 counted twice due to symmetry.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Case (i): de 2 dq
Case (ii): de < da
1.00
1.00
Fle, q.W
0.75
Fleu Fle..g.v q.V)
0.75
0.50
0.50
A1
A3
A
A
A1
A2
(i]
(4)
A
A
0.25
2
0.25
0.00
0.00
65
t5
0
1
100
55
t
Figure 15: Assuming dg dey, plot the query segment accuracy, F(e,, 9),), for (e + dgl. where to = 0 and is de + dg. Case (i) where de 2 dg is shown in the left panel, and case (ii) where de < dy is shown in the right pancl.
The variables lo, , ls, represent the different states of overlap where the discontinuities of F(er, 4,7) occur, and using these we can express the areas as the following integrals:
A1 /0 7()d = 1
('(ergg77)d
(22)
and
A/a / F(er, 1( 4()
12
A2 =
(23)
due to symmetry, and
1 Fleq, q, ydt.
A3
(24)
J12
We use that the query segment accuracy F(es, 9:7) is lincar in cach interval, which mcans that the arcas be expressed as
A F(et, 4,7) + F(er, .4.7) (11 t0),
(25)
2
A2 = Flee,+ 1177) + F(erz 4,7) (12 11),
(26)
2
and
Ag F(eez 7. + F(ets, 7, (ta t2),
(27)
2
where 1- indicate that we approach the discontinuity at L from below and & from above. We now only need express to, .o....,t to and F(e q, 4..)..... F(ety 9: in terms of de, dg and y conclude the proof. For brevity, these have been provided in Table See section A.I.1 for details how to express these in terms of d,, de and 7.
We provide the steps for case (i), and leave the derivation for case (ii) to the reader. We substitute the expressions for case (i), provided in Table 2 into equations Eq. 25 27 and the resulting expressions for the
areas 1(0) 1/5) and A into Eq. which give
26

---

Published in Transactions on Machine Learning Research (09/2025)
Case (i), de 2 dg
Case (ii), de < dg
0 F F(e .97)
(6) 0
(6)
0
((6)
F F (18)
9.77)
4 yde
F(e 177)
-d
(86 yde
F F(e 7)
dy
2
( dp
F(e (.7) de
2
(i)) de
F 9.7))
2dd_
dy
( = de 3
F(ei, 9,7) = F(eE, 4.7)
(R) = da
F 9.7)
de
3
F(eis (077
Table 2: A summary of the derived expressions for to...., 13 and Feta q, ,,.., Petta q 7) for each case.
F(er (177) and :(e:::9: 7) denotes the limits when approaching t from below and above respectively.
2
dy -de
2 7de
A
=
(1 +
ryde + -1
(dg - 7do) - (de da)
2
dg
2
dg
yde
1 + (de dg)
(2dg - yde)
+ (dg + (dg - -yde)
dy
dg
1
(2ydgee -²c of -2 drd, - %)
dg
(2ndy - 2222 + dedg)
dg
de (2ndg 2y2 + dg).
dg
Finally, by substituting A for (i) in Eq. 8 we arrive at
A d2 27 de + dg)
(28)
de dy
dg dg)
which shows that Eq. 9 holds for case (i) under the assumption that dg 2 rde- Similarly, this also holds for case (ii).
Assumption 2. The annotator presence criterion not be fulfilled (dg yde).
When the presence criterion can not be fulfilled we never get any presence labels, this means that the fraction of the query segment that overlaps with an event is always incorrectly given an absence label. When the query segment completely overlaps with an event the query segment accuracy will be 0 (seen between f and to in Figure 16).
dg < yde
1.00
Fle,, q.vl
0.75
0.50
A1
A1
0.25
0.00
to
Figure 16: Assuming that dg < yde, we plot the query segment accuracy, F(et, (0e,,,7), 7), for (0, de + dg). where to 0 and t3 de + dg.
27

---

Published in Transactions on Machine Learning Research (09/2025)
The area A1 is counted twice due to symmetry. The discontimuity t occurs for the smallest t [0, + dg] for which F(ex, 9, 7) = 0, which happens for the smallest. / for which the whole query segment overlaps with the event le n g) = dg at t t= = dy. We therefore have that 1 to = 13 - t2 = dg.
When there is no overlap between the query segment and the event giving a presence label is always correct, thus F(ctg: (:7) 1. However, giving an absence label to a query segment that completely overlaps with an event gives the query segment accuracy 0, thus F(er, 9.7) 0. The total area under the curve therefore 2 dg and by normalizing with to de dgp get dg/ (d, dg), which proves the dg yd case of Eq. and concludes the proof.
A.1.1 Details on the expressions in Table 2
This section provides a detailed explanation of the values presented in Table 2 For each case (i) and (ii), we will define the specific lime points lo, 4, 42, la where the query segment accuracy function F(e,, g, 7) changes, and explain the corresponding value of the function at these points based on the overlap between the event e, and the query segment q. The states 14 and i are analogous t and to, respectively, and therefore not illustrated, The difference is that the amount of overlap between the query segment and event decreases (instead of increases) when approaching these states
28

---

Published in Transactions on Machine Learning Research (09/2025)
Case (i): de dg
[=
t=
1=
t=
8)
(i) de 2 dg
Figure 17: An illustration of how the sound event and the query segment, q overlap at the four distinct
states = to,...,tz t3 for case (i) where de dg.
0 At 100 O, the end of the event ez aligns perfectly with the beginning of the query segment q- This means there is no overlap between the event and the query segment (le,go nil 0). Therefore, assuming the annotator absence criterion applies, the query segment accuracy is F(egos 9.7) =
4, le ne
d -0 1
dg
dg
{.: The time e qd represents the point where the annotator presence criterion is first met.
Before this point (t < (1)), ). the overlap le, n ql is less than yd,, and the query segment accuracy is given by F(er, du -elannl As t approaches z from the left, (e, O gl approaches ydg, hence d,
lim F(e).9,7) = d,-dd, At t ti the presence criterion is met, and the accuracy function de
switches to ((e,,q,7) lennal As t approaches { from the right, ler n is slightly greater than dg This transition is visually represented in Figure 17 at time t t
yde, and lim F(ee, rd. d,
( At 12 = dg. the entire query segment y is fully contained within the event ex. This means the overlap is maximal: le ngg dg. Since the presence criterion is met, the query segment accuracy
lc (e
F(e,,,, (.7)
de 1. This behavior is visually represented in Figure 17 time 12. d, 4
where the green box representing the event fully covers the red box representing the query segment.
At 6 = de, the entire query segment still fully overlaps with the event 65. Similar to t2, the overlap is 10,00 n gl = dg. and therefore F(e.0.9:7) = (g Figure 17 at time 1z.
d, = 1. This is depicted
d.
Case (ii) de < dg
t
t=yy
t=t2
t=
6
(ii) de < dg
Figure 18: An illustration of how the sound event and the query segment overlap at the four distinct states 1 = lo...., la for case (ii) where d, < dg.
(i), At ((i)) 0, the end of the event e, aligns perfectly with the beginning of the query segment There is no overlap (le,go Ngg 0). Assuming the annolator absence crilerion applies, the query
segment accuracy F(e,,o) de de-le,ca nol
-0-0 -1
d
( The time (11) = ydo again marks the point where the annotator presence criterion is first met.
Before this (t ( (16) the overlap |e, O gl yde, and F(el. 9,7) dy de-ln Approaching { from
d
29

---

Published in Transactions on Machine Learning Research (09/2025)
the left, Jex nil yd,, thus l ((e), d, At t= ((8) the criterion is met, and the function becomes F(es, 7) = la,0gl Approaching from the right, Jer n qi is slightly greater than yde, so -im F( ads. This transition is shown in Figure 18 at t t= t1.
(000 At 1/2 des the beginning of the event e aligns with the beginning of the query segment q. At this point, the overlap maximal, as the entire event contained within the query segment: leg0 ngl de. Since the presence criterion is met, the query segment accuracy is F(e,ann: heggnnll
4.. This situation illustrated in Figure 18 at t = t..
dg
3 At (60) = dg. the end of the event e aligns with the end of the query segment q. Similar to 2 the entire event is contained within the query segment, so the overlap is lego n de-
(0)
(()
le le
This corresponds to the Consequently, the query segment accuracy is Flegan.9.9 9:7) = dg = de state depicted in Figure 18 at t tz-
dy
Understanding these key time points and the corresponding query segment accuracy values is crucial for calculating the area under the curve, which represents the expected query segment accuracy.
A.2 Proof of Theorem
Proof. We start by finding a unique critical point dy which makes ('(d)) = when dy 2 yde- We then show that dy is a global maximum by analyzing the boundaries of f(d,) on its' domain when dg 2 yde- We show that f(d)) 2 f(yde) and that f(d)) 2 lima,-sso f(dg). Since dz is a unique critical point we conclude that must be global maximum of the function (dly when dg 2 7de- Lastly, we show that I ((C) f(yde) 2 f(g)) when dg < yde which proves that d is global maximum of the function fdgy for d, > 0.
1. Finding the unique critical point dy-
To find the critical points, we need to compute the derivative of f(d,) with respect to dg and set it to zero. Let ((d)) de(- 7d + 2d,7 dg) and D(d,) dg(de dz)- Then f(d,) N(d.) Using the quotient rule, the derivative given by:
('d,)) D(d)) - N(d,)D'(d,)
[Ddd)]2
First, we find the derivatives of the numerator and the denominator:
d (d,)) dd,
Ide (- + 2d dg)1
d. 2n 1
d2 +1)
D(d,) dg(( + dg) de + dg
D'(d,) d ld + dg
d
del
d + 2dl
Now, we plug these into the quotient rule formula:
22 + 2dg7 + dy)llde + 2dgl f'(g)) (2) + (d + d,)1 (-2 (d + d3
30

---

Published in Transactions on Machine Learning Research (09/2025)
To find the critical points, set (dg) = which means the numerator must be zero:
[d (de + d,)] (de(-2de>* 2d,7 + dy)](de + 2d,] 0
Since de O, we can divide by de:
(2y + ()dy(de dg) - -1-2de7 2d,7 + dy) + 2dg) 0
Expanding the terms:
(27 + (y + dg) - (-2de -Adedg² + 2d.d + 4:2y + dedg + +2dg = 2yd,d + 2nd + de dg + dg (-2d67² -4d,dg² + 2d d + 4d2 + ded, 2d
Collecting and rearranging the terms to form quadratic equation in dg:
(27+1-47) 4 2)d (27 + +27 47
27 1)d + 2d6 0
(-2y 1)d ; (4) + 2d 0
(27 + 1)dg - d7 - 2d =0
Using the quadratic formula dg =
-bt /b2 toe
where 2 + 1, b= 4d == -2d67²
2a
4d 1 v( (-4de 72)2 4 + 1)( 1)(-2()
dg
2(2 + 1)
4do + /16d2n4 + 8(27 + 1)d3n²
47+2 2
Adz² = /16d2n² + 16d2 8dβn
47 +
4d.72 + / (2 + 27 +
47 +2
4d.72 + 2de bode holl////22 47 +2
2(27 +
Since y > 0, we have h/ = 7:
dg 4do - Yd 4722 + /4.2 47 -
2(27 + +1
2d dd /4,2 + 47 +2
(27 +1)
27 + / 24 + 47 +2
dey
27 +1
We note that 4 + 4, +2 = / 0.5 > 2/72 = 2y, which means that we need to choose the
posilive sign for dg 0 to be true. The value of d, that makes the derivative zero is therefore uniquely defined by:
27 + v 4 + +
dg = de 7
2 de?
27 1
where the last inequality holds because /4,2 + 47 2 = 2 2/2 + 0.5 2.
+
2. Analyze the function at the boundaries of its' domain.
To understand why this critical point corresponds to maximum, we analyze the function f(dy) as dg the boundaries of its domain.
31

---

Published in Transactions on Machine Learning Research (09/2025)
2a. f(dg) when dg gde (dg 2 yde).
f(nde)= = de (2(yde) 2d (rde)) (7de) (de + yde)
de (2d 2 - 2d + +dd)
(7de) (de + ?de)
de (rdd)
(7da) (de + ydoe)
dey
(yde) Ide (1 + +
1+7
2b. f(dg) as dg 00 (dg 2 yde). We want evaluate the limit of f(dy) d approaches infinity:
del de(-2d,72 + (27 1)dg)
lim f(dg)
lim do-000
-00000
de + d²
Divide the numerator and the denominator by the highest power of dg in the denominator, which is de.
de
2d dy
+
2
dg
lim f(d)) = lim d0
dg 400
dg
As dg + 00, the terms
2d 2 and de all approach 0. Thus, d
de
&
d.(0 + 0)
lim f(d,)) 0 0+1
dg +00
This mcans that as d, becomes very large, the function f(d,) approaches
2c. Showing that f(d)) [(yde).
We want to show that f(d)) ((dde). Or equivalently, that f(yde) 2 0. From Theorem 3 we know
that f(dz) 27 (27 +1 4 1, and from 2a we know that f(7de) = T After substitution and some algebraic manipulation, we get
47 + 67 -3
2//472 + + 20.
1+7
Since > 0, it suffices to show that
4 + 67 +3
> 2/ + 47 +
1+
Squaring both sides of the above incquality and simplifying, we obtain the equivalent incquality
(4,2 + +6y +3)
4(47 + +
1+7
After further algebraic manipulations (which we leave to the reader), we arrive at the inequality
(27 1)2 0.
32

---

Published in Transactions on Machine Learning Research (09/2025)
Since (2 + 1)² 2 0 holds for all y, and the previous steps are all equivalences, conclude that f(d)) - f((de) 20
for < S 1, and therefore,
[(T) 2 [(7do)-
2d. Showing that f(d)) 2 limd, +o f(dy).
We combine the results from 2a-2c to get
[(d)) 2 ((7de)
1
1+7
2
lim
f(dz).
du 000
2e. f(dz) dg + (rde)= (dg < yde).
Since we are approaching ydg from the left, we have that f(dg) = dg/(de + dg). This function is continuous for dg < yde, so the limit is given by the direct substitution:
dg
yde
lim (ryde-
de + dg
de 7de
ydee
de +
7
1+7
2f. Showing that f(yd2) 2 f(dg) when dy < yde- We start by noting that f(yde) 1+7 lim,, i(dd,-(7d))- Now it is sufficient to show that [(dz) = dg/(dh + de) is strictly decreasing for decreasing dy which we do by computing the derivative of f(d)) with respect to dg using the quotient rule:
1 2 1+7
(dg + +1) - d1(1)
('(,))
(dy + 7)2
dg dg
(dg 7)2
(dg + 7)2²
Since > 0 and (d, +7)² > 0 for all d, > O, we have ('(d,) > for all d, > 0. This implies that the function f(d,) is strictly increasing on the interval (0, 20). Therefore, if < c c 6, it must be the case that f(c) < f(b). Moreover, since e < b, f(c) < f(b). Thus, for any > 0, f(b) > f(c) for all 0 Now let dg =c 7d. b.
3. Combining everything (2a-2f)
We have derived a unique critical point d; 2 yd, by setting the first derivative of f(d,) to zero. We have then shown that fd))) is greater than or equal to f(d,) at the limits of its' domain when d, 2 7de. Finally, we show that f(d)) 2 f(7de) 2 (dgy when dg < yde. Therefore, the value of dg that is the global maximum of fdd) when d, is:
27 + / + 47 +2
d de 7
2 +
33

---

Published in Transactions on Machine Learning Research (09/2025)
A.3 Proof of Theorem 3
Proof. From Theorem have that
dy = de7 (27 + 447+
2
maximizes the function
f(d)) del de(-2d, 72 + (2 + 1) dg)
dg (de + dg)
We wish to show that the maximum label accuracy given overlap, f (²) - [(d)), is
2- -V
442 1 47
1.
1. Express f(dg) in terms of dimensionless variable.
Define
6 dg de
Then
dg = 8 dde de + dg = de (1 +0,,
and
d- 72 + (27 + 1)
-2 (27 + 1)5 8(1+0)
f(dg) f(8de)
(d (de + 8 de)
We can therefore write
f(0) -2 + 727+ 8 8 (1+66)
2. Identify the optimal dimensionless query length 8*.
From Theorem know that
dz d= de (27 + /47² - +
27+1
Dividing both sides by de gives
8" d'
27 / +
de
27+1 +1
We need to show that
f(5*) =
+ 1.
3. Compute f(8*) explicitly.
Let
N(6) =22² + (27+1)6 D(8) 8(1+
N(8) Then f(6)
D(5)
34

---

Published in Transactions on Machine Learning Research (09/2025)
1. Numerator at 8*
(8))) = -2 + (2 + = + (2 1 2x + v 172 + -4
2 2
Inside the brackets, (2 + 1) cancels:
N(6") = - + (2?t) 472 + +47 = 222 227 +7 V 4/2 + +
+2 = 4 +4
2. Denominator at 8*
D(0) 8(1+0).
Hence,
D(8*) - 5* 81++8' -(7 2 + 4-2 + 47+2 + 2
2g + v 4- + 47
27 +1
2
The second bracket becomes single fraction:
1 1+7 2 + v 472 + +
(27+1) + 7 (2 + v 472 +
27+1
27+1
Combining, we get
D(0*) =7 2g + v 47 + +
+2
X 12 +2 + V 472 + +
2 +1
27 1
So
D(8*) =7 (2 + / 47 + 47 + 2) (27 + 1+2 + / + +
11)
3. Form the ratio. Thus,
N(8*) D(8*) (2++ 1,
7 V 42 + +
f(8))
(27+1+2 472+11+2 (2
Cancel the common factor 7, invert the denominator and multiply:
/ 47
1(5') = (27 + 4 (27+1+22 27744 47
You can verify by direct expansion (or by symbolic algebra tool which we provide in the supple- mentary material) that
4- +
27 (2 47 +2 + 1.
47 -2
(2 + V 4- 4, 2) 4
Thus
f(8*) = -2 -2 + +1,
which proves that
-())= = 2
+ 1.
Hence, Eq. 11 holds, completing the proof.
35

---

Published in Transactions on Machine Learning Research (09/2025)
A.4 Empirical Analysis of Uniform Relative Offset Distribution Between Events and Overlapping Segments
We empirically verify that the relative offsee between events and overlapping query segments can be modeled well by a uniform distribution. An event is denoted by e = (ag. be: Ge), where a, is the start time, be is the end time, and ce is the class, where ce E ('dogs", 'baby'}. Similarly, a query segment is denoted by g == (ay: by). We define the event distribution using the labeled start times of different sound event classes from the NIGENS Trowitzsch
al. (2019) dataset, but we fix the event length de to the median event length of the respective sound class to respect the deterministic event length assumption. We then verify that the relative offset between the events and the overlapping segments, defined as by des is uniform over the range [0, de + dg). To simulate realistic scenarios that maintain a reasonable label accuracy, we let dy E { fo de, 10dz). That is, the query segment is not larger than 10 times the median event length. Note that if d, >> de then the uniform relative offset assumption is not expected to hold, but that also means that the label accuracy will be very low which is not wanted in practice. We present the results in Figure [9] For both sound event classes the distribution looks flat for all three choices of dy meaning that it can be modeled well by uniform distribution, verifying that this is plausible assumption in practical scenarios.
Offset distribution dog
Offset distribution baby
Query length d, d, = 0.1 d. (0.02s)
Query length d, d 0.1 d. (0.09s) dg = 1.0 d, (0.91s) dg 10.0d, (9.07s)
1000
1250
800
d, 1.0d, (0.245)
1000
Count
d, 10.0d. (2.40s)
Count
750
400
500
200
250
0
0.0
1.5
2.0
2.5
Offset (bg- - ax) [s]
Offset (b, ae) [s]
Figure 19: The distribution of relative offsets between events and overlapping query segments for dog (left) and baby (right) sound event classes from the NIGENS Trowitzsch et al. 2019 dataset. We use the annotated relative start times of the events, and fix the event length de to the median event length for the respective event class. The distribution looks flat and can be modeled well with a uniform distribution.
A.5 Empirical Analysis of Theory
In an attempt. to empirically validate the theory we have compared the weakly labeled version of Au- dioSet Hershey et al. (2017) with the strongly labeled version Hershey et al. (2021). The weakly labeled version of AudioSet uses 10 second segments, corresponding to d, = 10. A subset of AudioSet has been strongly labeled by indicating start and end times of the weakly labeled events. For each strongly labeled event, we compare the strong label to the weak label compute the accuracy. That is, if the weakly labeled segment of length d, indicates the presence of an event, and the corresponding strong label for that event has length de, then the accuracy is de for that event. We compute this accuracy for all sound events correspond- dy ing to the 'Animal' class, and take the average. The thcoretical accuracy for given annotator critcrion y is derived by taking the numerical average over the event lengths for the 'Animal' class to estimate the numerical integration over the event length distribution presented Eq. (13).
The results are shown in Figure 20 Note that the query segment is not chosen to maximize label accuracy as in previous analysis. Since d, 10 it is longer than most. 'Animal' events in AudioSet and restricts the maximum event length that can be annotated to de I dg as can be seen in the right panel of Figure 20 In the left panel of Figure 20 we see that the label accuracy of the weakly labeled version of AudioSet falls within the range predicted by the theory for y E (0. 1]. Assuming that the theory correct would indicate that 7 0.26 is the presence criterion that best models the weak labeling process of AudioSet. While these
36

---

Published in Transactions on Machine Learning Research (09/2025)
results do not reject the proposed theory would need to empirical estimate y based real annotators to properly validate it. This is considered as out of scope for this paper, but would be interesting future work.
Accuracy vs Animal
Event Length Distribution Animal
400
Accuracy
0.20
Frequency
300
200
0.15
Theory
100
Empirical (0.162)
0.0
0.2
0.4
0.6
0.8
1.0
2
4
6 8
10
Event Length (s)
Figure 20: The theoretical prediction of the label accuracy for different y (left) when averaging over the event length distribution (right) for the animal events in the strongly labeled subset of AudioSet. The empirical accuracy (dashed blue line) indicates the label accuracy that derived by comparing the weakly labeled version of AudioSet with the strongly labeled version. The empirical label accuracy falls within the range predicted by the theory.
37


Reviews:
[{"review_id": "ZrKTIthdrc", "reviewer": "Reviewer_Gdrx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Math derivations on the optimal segment size to use for human annotations to reduce segment noise\n- Empirical simulations to show that math even under assumptions well describes more realistic data, including some simulations with real data like AudioSet.\n- Discussion of consequences of the estimated noise and what the best practices we should work on for human annotation in the segmentation domain.\n\n**Weaknesses**\n\n(see requested changes for more details)\n\n- I have conceptual issues with the method authors are trying to analyse and point out that it is not a good way to do human annotation. I think we have adopted a long time ago segmentation process using ML model first and then fixing the segmentation with both labeling at the same time.\n- (maybe I just missed it) Empirical justification for the real data where we have long audio with many events occurring in it: showing that derivations will be well aligned with such data distribution.\n\n\n**Note** I did a pass over all proofs (some not exactly deep on the derivations exactly but all transitions I have double checked) and all seems good to me w/o obvious errors. Also the paper itself is well written - did not spot any typos so far.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of human annotation and possible introduced noise in the annotation due to the process itself. Authors consider specifically problem of segmentation annotation (annotate that particular segment of signal is classified as particular class: e.g. audio/video event segmentation) where one of the approaches to human annotation is using fixed segments and giving these fixed segments for human labeling (assign if particular event is present in that segment) only to reduce annotation costs. While this annotation process reduces the cost of human labeling, authors discuss that this process introduces segment noise: missing some events, wrongly annotating some events, etc. The main question authors consider is what will be the expected segment noise error in the annotation and how to estimate the best segment size to use in the annotation process if we know e.g. some statistic on the average event duration.\n\nThe main contribution is giving math derivations on the expected segment noise and finding the optimal segment size (minimizing this expected error) under some assumptions (the best e.g. if we know the average event duration or min / max duration we can detect the event). Furthermore authors are doing data simulation showing that their derivations under some assumptions are pretty general and well describe more complicated scenarios including some real data, e.g. AudioSet.\n\nAuthors give a nice discussion at the end of the paper on how to use, interpret their results and kind of pushing to revisit our human annotation process in future with adaptive and active learning techniques e.g.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Some conceptual questions:**\n- I found it strange that we are still doing fixed segment annotations. Already 10 years ago Google presented their tool (and as far as I know a lot of places are doing the same now) where they use a model to annotate image segmentation and then give this to human annotators to correct boundaries and correct class assignment. This speeds up annotation and of course makes it cheaper as it is now just a correction. The same e.g. done also for speech recognition. I am not familiar with the sound segmentation, but I am really surprised that this is not done yet there / or not widespread. Could you comment on this? Because otherwise I don't understand the point of finding the best segment size for the task if the process is more adaptive and based on automatic segmentation in the first place.\n- It is great that you can derive the exact estimation of the segment noise, but at the same time I feel it is kind of obvious that this scheme introduces large noise / error into annotation and yep, optimal segment will depend on the de/dq and in general on the event duration / perception for particular task. Being obvious and simple doesn't mean to me it is not valuable and we may need to do it and show due to poor practices. But is it really that community (as I am a bit different domain where we don't do such segmentation for human annotation) is actively doing it due to negligence that there will be high error?\n\n**Some questions:**\n- what happens if two different events occurred at the same time? Do you consider only one class assignment? or is it allowed to assign two classes for human annotators? How will this impact your derivations?\n- Assumption \"events are uniformly distributed locally in time\" seems too restrictive to me and this is not what happens in real data. How does this affect the derivations and generalization of the exact formulas you got?\n- \"This is arguably the simplest data distribution to annotate, and the results can therefore be viewed as an upper bound on the expected label accuracy for any more complex data distribution.\" - I quite don't get why this will be upper bound? Is this in the sense of worst case scenario? I would argue the opposite.\n- Figure 4: are you sure that the central place of segment A will be at 0.5? I think this is wrong and the picture thus is misleading (though in appendix derivations are good to me, e.g. in Table 2).\n- Could you confirm that your empirical simulations well describe math prediction for the multiclass data where several events can happen in the long audio? I think I miss this more complicated (what we have in practice) scenario a bit.\n\n**Some suggestions:**\n- While reading I was most of the time confused between weak labels as annotations and weak labels as in pseudo-labeling (and automatic annotation). All discussions with related works made me confused which piece of the weak labels you are focusing on.\n- Maybe it is good to move some discussion sections in the beginning and give the main outcomes to people first and then go into math and empirical simulations. I feel that the discussion section contains a lot of important pieces as the outcome of your work which most people need to know when they read the paper w/o going into derivations. It feels that the main message is kind of lost in the paper as it is hidden at the very end.", "broader_impact_concerns": "No any concerns based on the content and derivations done in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b04Ml80kUP", "reviewer": "Reviewer_cKD4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Strengths**\n    1. This work derives expected label accuracy for FIX and ORC under single/deterministic/multiple stochastic event scenarios, the analysis of which is very meticulous. \n    2. It proves that ORC achieves perfect accuracy with minimal segments, serving as an upper bound for FIX.  \n    3. It provides a theoretical foundation for adaptive weak labeling strategies mimicking ORC, such as change-point detection and active learning.  \n    5. The paper is well-structured, with clear logic from problem statement to experimental verification. \n\n- **Weaknesses**\n    - My main concern is that this work merely analyzes the mathematical properties of the accuracy distribution under the scenario of temporal event annotation, and the conclusions are simple and intuitive. However, it does not highlight how to guide algorithm design based on the conclusions, offering limited guidance for machine learning algorithms and experimental design. Perhaps the paper needs to propose a specific active learning algorithm to validate the proposed modeling, rather than merely conducting simulation experiments.\n    - It may be unreasonable to determine presence or absence by the proportion of the overlapping part with $d_e$. Why not model it with $|e\\cap q| / q$ ?Some discussions can be added.\n    - Figure 4 could be polished up. It is recommended to make the top panel dynamic (i.e., showing two scenarios of q and e overlapping) and use dashed lines to correspond with the bottom panel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper theoretically analyzes the accuracy and cost trade-offs of fixed-segment weak labeling (FIX) for temporal event annotation, comparing it with an oracle method (ORC) that uses true event boundaries. The authors model how segment length affects label accuracy and annotation effort, defining key concepts like event fraction overlap $h(e, q)$ and a presence criterion $\\gamma$ to quantify annotator sensitivity. Through closed-form expressions and simulations, they show that FIX introduces segment label noise, with shorter segments improving accuracy at higher costs, while longer segments reduce costs but degrade accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses above.", "broader_impact_concerns": "Not Applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aJF6bKGscf", "reviewer": "Reviewer_ZKzU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a novel theoretical framework for analyzing weak labeling methods in temporal event detection.\n- Extensive simulation studies effectively validate the theoretical results across various scenarios, including different annotator criteria, event length distributions, and multiple events.\n- The paper is well-structured and clearly written, with helpful visualizations and mathematical derivations.\n\n**Weaknesses**\n\n- While the paper extensively validates the theoretical results through simulations, it lacks real-world experimental validation. The assumptions made in the theoretical model (e.g., uniformly distributed events, deterministic annotator behavior) may not fully capture the complexities of real annotation tasks. Real annotators may exhibit variability in their perception, attention, and judgment that isn't captured by the current model. If possible, testing the framework on actual human annotation data would strengthen the paper's claims.\n- The paper compares FIX and ORC methods, but could benefit from comparisons with other weak labeling strategies, such as active learning approaches or semi-automated methods. Such comparisons would provide a more comprehensive understanding of the relative advantages of different labeling strategies in the accuracy-cost space.\n- The theoretical analysis assumes that events are spaced at least \n apart for multiple events, which may not hold in high-density scenarios. While the authors do perform simulations that relax this assumption, the theoretical implications of violating this assumption are not fully explored in the mathematical framework.\n- Some aspects of the analysis could be more explicit about their generalizability. For instance, while the framework is developed with audio events in mind, more discussion on how the results extend to other domains with temporal events (e.g., video analysis, physiological signals, financial time series) would enhance the paper's impact. I am wondering could the proposed method be applied to multimodal temporal classification tasks, like audio-visual event localization (AVEL) or video parsing (AVVP). Some works in AVVP also aim to address the label noise or noise rate, such as (Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing; Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-Wise Pseudo Labeling). If the proposed method is not applicable to these tasks, it would also be better to add these related works in the introduction of label noise in Sec. 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper presents a theoretical analysis of fixed-length (FIX) weak labeling for event detection in time series data, particularly focusing on audio recordings. The authors model the accuracy and annotation cost of this common labeling approach where annotators assign presence/absence labels to fixed-length segments. They compare this to an oracle (ORC) method that uses true event boundaries and derive closed-form expressions for label accuracy and annotation cost for both methods. The theoretical framework is validated through simulations, showing that in most realistic scenarios, the oracle method outperforms the fixed-length approach in both accuracy and cost. The paper provides a foundation for developing adaptive weak labeling strategies that mimic the oracle process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness part for details. In brief, it would be better to provide more comparisons or at least discussions for the application in real-world annotation data or other multimodal labeling noise fields.", "broader_impact_concerns": "The authors have provided discussions about several practical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)