Transcribing Against Time
Matthias Sperbera,∗, Graham Neubigb , Jan Niehuesa , Satoshi Nakamurac , Alex WaibelaaKarlsruhe Institute of Technology, GermanybCarnegie Mellon University, USAc
Nara Institute of Science and Technology, Japan

Abstract
We investigate the problem of manually correcting errors from an automatic speechtranscript in a cost-sensitive fashion. This is done by specifying a fixed time budget, and thenautomatically choosing location and size of segments for correction such that the numberof corrected errors is maximized. The core components, as suggested by previous research[1], are a utility model that estimates the number of errors in a particular segment, anda cost model that estimates annotation effort for the segment. In this work we propose adynamic updating framework that allows for the training of cost models during the ongoingtranscription process. This removes the need for transcriber enrollment prior to the actual
transcription, and improves correction efficiency by allowing highly transcriber-adaptive
cost modeling. We first confirm and analyze the improvements afforded by this method in asimulated study. We then conduct a realistic user study, observing efficiency improvementsof 15% relative on average, and 42% for the participants who deviated most strongly from
our initial, transcriber-agnostic cost model. Moreover, we find that our updating framework
can capture dynamically changing factors, such as transcriber fatigue and topic familiarity,
which we observe to have a large influence on the transcriber’s working behavior.
Keywords: Speech transcription, error correction, cost-sensitive annotation, user modeling

1. Introduction
High quality speech transcripts are required in many different tasks, including for example web-based lecture archives, training data for automatic speech recognition (ASR), andinput to downstream applications such as translation. Unfortunately, in realistic settings
automatically created transcripts often contain too many errors to be useful as-is, and human annotators must be employed to improve their quality. This manual transcription iscostly and time-consuming.
Previous works have attempted to improve the efficiency of manual supervision for speechtranscription by dividing the speech into small segments that are convenient to transcribe∗

Corresponding author
Email address: matthias.sperber@kit.edu (Matthias Sperber)

Preprint submitted to Speech Communication

August 1, 2017

[2], and choosing low-confidence segments of an ASR transcript that are more likely tocontain errors [3, 4]. Studies on cost-sensitive annotation have also shown that to maximize
supervision efficiency, it is important to consider not only the number of errors that might
be contained in a particular segment, but also the human supervision effort involved incorrecting them [5, 6, 7]. Recent work has shown that supervision efficiency can be further
increased by training models to estimate the transcriber effort and potential error reductionof each segment, and then explicitly optimizing the location of segment boundaries [1].
However, this use of models to predict transcriber effort has a downside: these models
need to be trained. Thus, before starting the main transcription task, it is necessary to
perform enrollment, where the transcriber annotates a certain amount of randomly selected
enrollment data, which is then used to train the predictive cost models. However, enrollments are time-consuming, costly, and may be impractical; for example, in a crowd sourcing
situation. In addition, these predictive models remain static and cannot account for dynamically changing factors such as the annotator’s topic familiarity, fatigue, or increasing
experience.
In this work, we introduce a framework that removes this need for enrollment by updating
cost models and corresponding choice of data to annotate on-the-fly (during the ongoingtranscription process).1 This framework allows us to work within a fixed time limit for
manual correction of a transcript or a series of transcripts. We first start with a general,
initial cost model used to compute a first selection of segments for correction. During theongoing transcription process, the cost model is gradually improved and adapted toward theparticular transcriber, and the choice of segments is updated to reflect both the updated
cost model and the actual remaining time at a particular point. Locations and lengths ofsegments to annotate are chosen to optimize annotation efficiency, as proposed in previous
work [1].
A number of challenges arise in this proposed dynamic annotation framework. (1) A
suitable time limit must be chosen and incorporated into our framework as a stopping
criterion. (2) Annotation effort is difficult to predict, because there are large differences
not only between transcribers, but also for a single transcriber between different segments.
Accurate cost modeling is important both for selecting suitable segments that are efficient
to annotate, and for the amount of selected segments to be appropriate such that the timebudget is not over- or underspent. (3) Choosing segments to annotate is a computationallydifficult problem, but needs to be done quickly, as we desire to update the segmentation
during the ongoing transcription process.
Our solutions to these challenges are as follows: (1) We propose to limit the transcription
time budget to a fixed value, in order to reflect one’s desired cost-quality tradeoff. How tochoose a suitable time budget is task specific, but we argue that it is more practical thanconfiguring a confidence threshold as was required in some previous works [3, 4, 8]. Moreover,by periodically updating segmentations (§4) we can recover from inaccurately predicted
1

We have previously presented the basic idea of this method and verified it through simulation in theproceedings of SLT2014 [9]. In this paper, we expand the description and add a full user study with 12
expert and non-expert participants.

2

correction times, making sure that the budget is not over- or underspent. (2) We propose anapproach of starting with an initial, general cost model and gradually adapting it toward theparticular transcriber (§5). This has the potential to remove the need for enrollment, while
being able to model transcriber-specific as well as dynamically changing characteristics.
(3) We propose a new, more efficient algorithm for choosing which segments of the ASR
transcript to annotate using the penalty method (§6). We demonstrate this algorithm to be
fast enough to update segmentations during the transcription process, without degrading
quality compared to the original, much slower algorithm [1].
We first conduct a partly simulated evaluation approach (§7). Results show that our
method outperforms both cost-insensitive baselines and cost-sensitive baselines without updates. An analysis shows that efficiency gains are attributed to (1) increasingly accurate
cost models, (2) adjusting to the actual remaining time budget with each update, and (3)choosing a sensible (although crude) initial cost model that includes cognitive overhead.
Finally, we conduct a realistic user study (§8) in a typical scenario where several previously
unknown transcribers conduct only a limited amount of work. We notice large differences
between different transcribers, supporting our claim that transcriber-specific modeling is
crucial. Moreover, we observe relative productivity gains of 15% on average, and 42% for
those participants who deviated most from the initial cost model. The gains of using our
updating framework were especially strong in the case where the initial ASR transcript was
already of relatively high quality.
2. Related Work
Adaptive user models have been studied in the human computer interaction community[10], with very different requirements from ours. We are not aware of previous works on
adaptive user modeling or choice of data to annotate in the context of cost-sensitive annotation or computational linguistics in general. A closely related work on quality estimation[11] adapts an automatic quality estimator online and in a multi-task fashion, as more andmore in-domain samples are annotated over time.
Efficient supervision strategies have been studied across a variety of NLP-related researchareas, and received increasing attention in recent years. Examples include post editing
for speech recognition [3], interactive machine translation [12], active learning for machinetranslation [13, 14] and many other NLP tasks [15], to name but a few studies. Most of these
do not model annotation cost explicitly. However, it has been recognized that correcting
only the instances of highest utility is often not optimal in terms of efficiency, since theseparts tend to be the most difficult to manually annotate [5, 16]. As a solution, the idea ofusing an annotator cost model to predict the supervision effort has been developed [5, 17,
18, 19, 1], which inspired our approach as well. Note that these previous works estimate
static, annotator-specific cost models via enrollment, whereas our dynamic approach does
not require enrollment.
Some studies have addressed the problem of balancing utility and cost in the context
of active learning. A greedy approach to combine both into one measure is the “bang-forthe-buck” approach [5], where utility divided by effort is used as a per-instance efficiency
3

measure. Such an approach can be effective for selecting isolated instances for annotation,
but is problematic when selecting segments that can overlap and conflict with one another,as in our task. A more theoretically founded scalar optimization objective is the net benefit(utility minus costs) as proposed by [20], but unfortunately is restricted to applications where
both can be expressed in terms of the same monetary unit. [21] and [22] use a more practical
approach that specifies a constrained optimization problem by allowing only a limited timebudget for supervision, similar to our approach. Note that works on annotation apart fromthe active learning community have usually assumed stopping criterions based on confidence
thresholds [3, 4, 8], which measure only relative improvement and may not be intuitive to
configure in practice.
3. Background: Static Segmentation
Our advocated “transcribing against time” paradigm builds upon the following threelines of work:• Roy et al. [2] argue that human effort should be spent on transcription of speech, notits segmentation. Segmentation is time-consuming if performed manually, and can bedone reliably in an automatic fashion. Segmentation is also a very important step,
because suitable segment size leads to increased transcription efficiency.
• Rodriguez et al. [23] propose the computer-assisted transcription (CAT) approach, inwhich a transcription is first created automatically, and then corrected manually where
necessary. In their approach, a human checks the complete transcript and performs
corrections whenever errors are found.2• Sperber et al. [1] argue that the decision which segments to correct and which not,should also be done automatically to reduce human effort. They show how to selectsegment locations and sizes according to a desired utility-cost tradeoff, by exploiting
confidence scores and a transcription cost model (CM). They report 25% time savings
compared to a baseline that selects segments using only confidence scores but no CM.
3.1. Constrained Optimization Problem
In particular, our work extends the optimization framework Sesla (short for Segmentation for Efficient Supervised Language Annotation), as proposed in [1]. Given an ASR
transcript to be corrected by a human annotator, Sesla determines a segmentation of thetranscript, and makes a decision whether or not to transcribe each segment, such thatannotation efficiency is optimized. The idea is that we naturally want to concentrate on
annotating parts that yield high utility, in terms of number of errors3 that are removed bysupervising them. While choosing very small segments (e.g. single words) would allow us to2

CAT also improves the automatic transcript on-the-fly using the transcriber’s corrections. This is not
the focus of our work, but could be integrated into our proposed updating framework.
3
By number of errors, we mean Levenshtein distance to a reference transcript.

4

[SKIP:0/0][SKIP:0/0][SKIP:0/0](at)[VERIFY:1/4](what’s)[VERIFY:1/4][VERIFY:2/5][SKIP:0/0]abright

cold

day …

[VERIFY:0/3]

Figure 1: Illustration of the segmentation problem. A graph of different possible segmentations for the ASR
transcript of the sentence fragment “It was a bright cold day . . . ” is shown. Edges are segments (someedges are omitted for readability), and nodes potential segment boundaries. Segments are labeled with[supervision mode:predicted utility/predicted cost]. For optimal supervision efficiency, the solid edges might
be preferable over the dashed ones.

concentrate only on parts of maximum utility, longer segments are desirable from a cognitive
point of view as they reduce cognitive overhead due to switching between different parts ofthe transcript. Sesla attempts to balance both cost and utility.
Formally, Sesla searches for a segmentation of the N words of an ASR transcript w1N+1
into M ≤ N segments with boundaries at sM= (s1 =1, s2 , . . . , sM +1 =N + 1). A segment
1boundary marker si is interpreted as placing a segment boundary before the si -th word (orthe end-of-transcript marker for sM +1 =N + 1). Each segment is further associated witha supervision mode mj ∈ K. In this work these supervision modes are either Verify(the segment should be verified and potentially corrected), or Skip (the segment will not be
verified). This segmentation is selected based on predictive models that estimate supervision
cost (here: correction time) and utility (here: number of corrected errors) for any particularsegment. For each segment wab and supervision mode k, we denote utility by utilk (wab ) andcost by costk (wab ). Note that cost and utility for the Skip mode are always 0. Figure 1
illustrates the segmentation problem.
We desire a segmentation that maximizes the total utility, while keeping the total costwithin a cost budget T :arg max
+1
M ;sM
;mM
1
1

subject to

M h

i
X
utilmj wssjj+1

(1)

j=1
M h
X

costmj

wssjj+1

i

≤T

(2)

j=1

We will introduce an efficient method for solving this optimization problem in §6. The
CM is a transcriber-dependent regression model, while the utility model (UM) is transcriberindependent and based on ASR confidences. Both will be described in §5.
4. Proposed Dynamic Updating Framework
Sesla as proposed previously produces a static segmentation once before starting thetranscription. It relies on a CM that is trained via an enrollment, which is costly and takes
up time that the transcriber might otherwise have invested in being productive. For instance,5

in [1] an enrollment of roughly 30 minutes is carried out. While investing enrollment time
will improve supervision efficiency and pay off in the long run, this approach may not be
economical at all for transcribers that supervise only a small amount of data, as is common
in crowd-sourcing situations.
In this work, we improve upon this situation by allowing transcribers to be productivefrom the start. We create an initial segmentation with a crude initial CM, and iteratively
improve the CM and consequently the segmentation as the transcription is underway. There
are several advantages to this approach: (1) The transcriber enrollment is removed, instead
the transcriber performs useful annotation from the start. (2) The segmentation will growincreasingly efficient due to the improving CM, starting out rather crude in the beginning,and eventually reaching or surpassing the efficiency that would have been reached using
a CM trained via enrollment. (3) Each updated segmentation will take into account theactual remaining time budget, which may differ from what had been predicted as remaining
when the transcriber would reach a certain position in the ASR transcript. This solves anissue pointed out in the previous work [1], in which systematic errors in time predictions
resulted in considerable over- or underspending of the given time budget. Re-segmenting
ensures that the remaining time is used optimally, for example by skipping some of the lesspromising segments if the remaining time budget had been overestimated.
Figure 2 illustrates the updating approach. An initial CM is used to create an initial
segmentation. The annotator starts transcribing the resulting segments in temporal order.
Up unto this point, the process is identical to the static method except for how the CM
is obtained. However, unlike with the static method, only the first few segments are transcribed. After that, the CM is updated based on the observed annotation times, and theremainder of the initial transcript (everything that comes after the last annotated segment)is re-segmented to reflect the updated CM and the actual remaining time budget. This cycleof transcribing and updating repeats until the last segment has been reached. Because segmentation updates reflect the actually remaining time budget at the point of each update,the end of the transcript can be expected to be reached very closely to the point at which
the time budget is exhausted. It is important to note that we follow the temporal order ofthe segments in the transcription process. This is a desirable property, as in our experience
jumping forwards and backwards through segments makes it difficult for the transcriber tograsp the content of the speech and may lead to transcription errors.
Formally, we propose to update segmentations as in Algorithm 1. We initialize the costmodel MC such that it will rely solely on a prior distribution to be defined. The transcription
position j is initialized to the first word, and an initial segmentation of the complete ASR
transcript is created. The transcriber starts transcribing the first batch of segments fromthe current position j, until B seconds have passed, at which time j is updated to thenew position at which transcription had stopped. The cost model MC is updated using
the observed supervision times.4 Finally, the remainder of the transcript that has not yet
been supervised (wjN ) is re-segmented using the updated CM and the remaining time budget
tremain , and another batch of B seconds is transcribed. Transcription is stopped when the4

Note that the UM remains unchanged.

6

initialize:

initial choice of segments
prior
costmodel transcribe 1st batch
2nd choice of segments
retrain:2nd costmodeltranscribe 2nd batch…

retrain:nth choice of segments
nth costmodeltranscribenth (last) batch
time

Figure 2: Proposed updating framework. An initial choice of segments is determined based on an initial,
transcriber-independent CM. Periodically, the CM is adapted based on the observed annotations, and the
choice of segments is updated to reflect the new CM and the actual remaining time budget.

Algorithm 1 Updating Annotation Framework. given: T (time budget), w1N (ASR transcript), B (batch size [annotation time per batchin seconds]), MU (UM, fixed)(0)
MC ← MC. initialize cost model §5.1
j←1
. set transcription position to first word
. solve optimization problem §3.1
Segment(w1N , MC , MU , T )while tremain > 0 do
. actual transcription
SuperviseBatch(wjN , min(B, tremain ))j ← resulting transcription position
Update(MC , newly observed times). §5.1N. solve optimization problem §3.1
Segment(wj , MC , MU , tremain )end while
time budget is exhausted, or the end of the transcript has been reached.
5. Predicting Cost and Utility
5.1. Cost Model
As a cost model, a regression model that predicts transcription time is learned in asupervised fashion from observed transcriptions. With the original, static method, CM
training examples were collected via transcriber enrollment, whereas we will show how to use
examples collected on-the-fly. We employ Gaussian Process (GP) regression [24], a Bayesian
technique for which state-of-the-art performance has been reported, especially in the case of
relatively low dimensionality and limited training data [19, 25]. We intrinsically confirmed its
superiority over linear regression and support vector regression in preliminary experiments.
As an additional advantage, it allows convenient specification of a prior distribution over7

transcription costs, a feature that we will exploit to bootstrap our CM. Input features
are the segment length (number of words), audio duration (seconds), and average wordlevel confidence. We expect length and duration to increase transcription cost, while low
confidence words are potentially harder and more expensive to transcribe.
To create the initial segmentation, we need a reasonable initial CM. Since transcribeddata from similar users or tasks may not be available, we rely solely on a manually definedprior inspired by our intuition about the transcription task. We argue that to create asensible initial segmentation, the CM can be crude, but should capture two key observations,
namely that longer segments take longer to supervise, and that the transcriber will need sometime to process the context switch for each new segment. We therefore specify the prior costas 2 + n seconds, where n is the segment length. Considering our transcript correction task,
this prior underestimates the actual cost, but we deliberately avoid hand-tuning the prior
to simulate a situation where little task expertise is available.5
Updates are performed by adding the new examples to the training data and retraining
the CM. As more and more examples are collected, the trained model will move farther
away from the prior distribution. For efficiency reasons, we use only the most recent 1000
examples for retraining.
5.2. Utility Model
Utility, defined as the numbers of substitution, insertion, and deletion errors removed
from the transcript, is modeled in atranscriber-agnostic
Here, we simply approxPsj+1fashion.
−1
sj+1
1
imate segment utility as util wsj= 1 − sj+1 −sj i=sj ci , where ci is a scaled word
confidence score for the i-th word, retrieved from the ASR. Note that strictly speaking, ourapproximation does not capture deletion errors. In this work, we do not update the UM
dynamically. While it may be conceivable to learn an average edit distance on-the-fly foreach transcriber, this would only scale the optimization objective (utility) by a constantfactor, without changing the resulting segmentation.
6. Computing Segmentations via the Penalty Method
To make on-the-fly updates of the segmentation practical, the optimization problem(Equations 1 and 2) must be solved rapidly so as not to incur waiting times while updating.
The previous work [1] used a general-purpose integer linear program (ILP) solver to find an
approximate solution, but this approach is not sufficiently fast for updating segmentations
during the transcription process. In this section, we introduce a new segmentation algorithm
based on the well-known penalty method [26] that we will demonstrate to be dramaticallymore time- and memory-efficient (§7.1).
Note that our optimization problem is reminiscent of other segmentation problems thatare often solvable by dynamic programming (DP) in polynomial time. Unfortunately, theglobal constraint in Equation 2 that enforces the time budget prevents us from using DP.
5

If possible, it is preferable to specify an underestimating prior rather than an overestimating prior, as
we will point out in §8.

8

Our strategy for solving the problem is to replace the constrained objective function by anunconstrained penalty function s(·):
MXsλ;mj (sj , sj+1 ) ,(3)

where sλ;k (i, j) := utilk (wij ) − λ costk (wij ).

(4)

arg max

+1
;mM
M ;sM
1
1 j=1

For a fixed value for the penalty parameter λ, the optimization problem in Equation 3
is solvable via DP as we will soon show. By varying the value of λ, we can obtain different
solutions for which we cannot increase their utility without increasing cost and vice versa,
which are also called Pareto-optimal segmentations. Note that the optimal solution to theconstrained problem corresponds to one of the Pareto-optimal solutions of the new penalized
formulation. Hence the search problem reduces to finding the optimal value for λ.
Intuitively, if a given λ results in a Pareto-optimal segmentation that overspends ourtime budget (we say this segmentation is infeasible), we should increase λ, thus penalizing
costly segments more strongly. Similarly, if the budget is underspent (the segmentation is
feasible), λ should be decreased. We follow this simple intuition by iteratively first computing
Pareto-optimal segmentations, and then adjusting λ. While finding the optimal value for λis NP-hard, we can efficiently find an arbitrarily good approximation via a binary search.
Algorithm 2 Iterative Penalized Dynamic Programming. Initialize lower-/upper-bound segmentations SL , SU and penalties λL , λU
while util(SU )/ util(SL ) > 1 +  do
λ0 ← (λU + λL )/2
S 0 ←SegmentDP(λ0 )if cost(SU ) ≥ cost(S 0 ) ≥ T then
λU , SU ← λ0 , S 0else. i.e., cost(SL ) ≤ cost(S 0 ) ≤ T
λL , SL ← λ0 , S 0
end ifend while
The segmentation algorithm is outlined in Algorithm 2. We keep track of an upperbound and a lower bound segmentation SU , SL , with corresponding penalties λU , λL . The
upper bound refers to the lowest-scoring infeasible segmentation seen so far, and the lower
bound is the highest-scoring feasible segmentation so far. For initialization, we first try anarbitrary value for the penalty and assign it to either λL or λU , depending on whether thesegmentation was feasible or not. To find the missing value for the other parameter λU (λL ),we then repeatedly multiply (divide) λ by a constant factor (here: 10), until an (in-)feasible
Pareto-optimal solution is found.
In the main loop of the algorithm, we consider a new penalty λ0 , halfway between theupper- and lower-bound values. We compute a corresponding Pareto-optimal segmentation
9

S 0 via DP, and update the lower or upper bound, depending on whether the segmentation
was feasible or not. The loop stops when the gap between upper and lower bound utilities
is below a threshold .
The DP algorithm that finds Pareto-optimal segmentations, given a penalty λ, computes
the total penalized score aj of the best segmentation of w1j , and keeps back pointers to findan optimal segmentation as follows:

a1 = max sλ;k (1, 1)k∈K

aj = max(ai + max sλ;k (i, j))
i<j

k∈K

The DP has computational complexity O(N 2 ). By limiting the segment size to be at
most R (here: 20), the complexity reduces to O(RN ). Moreover, the number of iterationsof the binary search for λ depends on the approximation threshold , but generally does notdepend on N , so the overall algorithm’s complexity is essentially linear in the length of the
ASR transcript. In practice, we observed convergence after typically less than 20 iterations.
7. Simulated Experiments
To be able to compare a large number of different settings, we first conducted partly
simulated experiments. We asked a real, non-expert transcriber to transcribe in chronologicalorder 200 segments randomly chosen from the evaluation data, balanced across different
lengths and average confidence scores. We measured the time taken to transcribe eachsegment. This enrollment process took about one hour. Based on this data, we trained anoracle time model and assumed that the human transcriber behaves according to this oracle
time model. We further assumed that the transcriber successfully transforms the ASR
transcript of every supervised segment into the corresponding reference transcript, without
making mistakes. As a specific method to train this model we trained a GP regressor,
and used this model as the gold standard that the tested CMs will try to reproduce. To
ensure that the oracle model was sufficiently different from the initial CMs to be adapted,
we applied a different prior for the oracle model. This prior was set to the least-squares
linear regression fit on the same data. We then distorted the oracle model’s predictions
with multiplicative gamma-distributed noise, to serve as our final oracle time model. The
multiplicative noise had a mean of 1 and moderate variance of 0.01. The batch size B is set
to 2.5 minutes, unless otherwise noted, i.e. the proposed updates are performed every 2.5
minutes.6 All results are averaged over 10 simulation runs, with the order of talks and noise
multipliers chosen at random.
For the simulations, we considered a correction scenario where the transcriber is given
an ASR transcription with the goal to remove as many errors as possible, given a 100
6

We expect and will confirm low batch sizes (frequent updates) to be better. 2.5 minutes is the lowest
value we tried in this set of experiments.

10minute time budget. As transcription data, we used 10 TED talks7 (short presentations
by skilled speakers, the total length was 104 minutes or 17.8k words). We concatenatedthe 10 TED talks so that the budget is allocated jointly over all talks. The erroneoustranscripts were created using the Janus speech recognition toolkit [27] with a simple TEDoptimized setup. The word error rate on our test set was 22.3%, with a total of 3978 errors.
The reference transcripts were carefully transcribed and of high accuracy. Our CMs areparametrized similar to [1]: We employ scaled lattice posterior probabilities as our UM, andGP regression with a squared exponential kernel as our CM, predicting log times to avoid
negative values.8 Noise and kernel variances for the GP regressors were set to log(5 sec).
We empirically confirmed that these parameters are sensible, but did not fine-tune thembecause in a practical situation, parameter tuning data might not be available.
We configured the proposed segmentation algorithm to find a solution within 1% of the
optimal solution, and restricted the maximum segment length to 20 words. The average
resulting segment length was 8 words or 3 seconds.
7.1. Segmentation Algorithms
We first compared the computational efficiency of segmenting according to the algorithm
proposed in §6, as opposed to the baseline method of formulating the problem as an integer
linear program and using an off-the-shelve ILP solver. As our solver, we used GUROBI,
which is highly optimized commercial software and among the fastest ILP solvers available,9
while our Java implementation of the proposed algorithm is straightforward, with little
code level optimization. We used the initial CM to segment increasingly large subsets ofour data. Both algorithms were run on a single processor, and stopped after producing asolution within 1% of their respective upper bounds. Figure 3 shows the results. It can beseen that the proposed method is considerably faster. It stays within a computation timeof around 3 seconds, while the ILP solver needs more than two minutes to segment all data.
Further, the proposed method’s memory consumption grows only linearly in the number ofwords, needed to store the DP scores. In contrast, for segmenting the complete dataset,
we observed memory consumption one or even two orders of magnitude higher for the ILP
solver. We use the proposed algorithm throughout the following experiments.
7.2. End-to-end Effectiveness of Updating Framework
In this section, we evaluate the end-to-end effectiveness of the proposed updating framework, compared to several baseline approaches, in terms of the final number of errors removed
after 100 minutes of transcription time. As Figure 4 indicates, with the dynamic-proposed
setup 1655 errors are removed when performing updates every 2.5 minutes, with the utility
decreasing for fewer updates, until only 1328 errors are removed when performing no updates
to the initial segmentation.
7

www.ted.com
GP regression was done using GPy: github.com/SheffieldML/GPy
9
See benchmark on http://plato.asu.edu/ftp/milpc.html (accessed June 6, 2016)
8

11

Comput. time [sec]150
Proposed algorithm
100

ILP solver

50

7.3s
0.7s

0

1k

112s
57s
2.8s

3k

4.7s

5k 7k 9k 11k 13k 15k 17k
Number of segmented words

Figure 3: Computation time needed to segment different amounts of data using an ILP solver or our proposed
algorithm.

dynamic-oracleCM
dynamic-proposed
dynamic-naivePrior
dynamic-fixedCM
staticranked-conflinearutility
model

cost
model

cost prior models
segment overheadsegmentation
updates

yes
yes
yes
yes
yes
yes
-

true model minus noise
incremental
incremental
enrollment
enrollment
-

yes
yes
no
no
no
-

yes
yes
yes
yes
no
no
no

Table 1: Overview of compared settings.

Figure 4 also shows several other experimental settings (see Table 7.2 for an overview ofall settings), yielding the following findings:
Dynamic updates outperform static baseline: The setting termed static is a costsensitive baseline similar to [1]. Segmentations are optimized using Sesla, but not updated
on-the-fly, and the cost-model is trained via enrollment. In this simulation, we assume thatsupervision time can be divided between enrollment and productive error correction. The
longer the enrollment, the more accurate the CM, but the shorter the time for annotation.
For a fair comparison, we use the same prior as in our proposed method. In the graph, wevary over how much time is reserved for enrollment, and find optimum utility for enrollmentsof 0 or 10 minutes, only slightly better than the confidence baseline, and 28% worse thanthe best proposed setup. Note that for large transcription projects with bigger time budgets
enrollment costs are likely to amortize and this baseline can be expected to perform morestrongly.
Even a crude, static CM is better than no CM: For a comparison to cost-insensitive
baselines, Sesla is no longer applicable, so in this experiment we consider predefined, fixed
segmentations. Previous research found that transcribing (sub-) sentences is preferable overtranscribing individual words [2, 4], so we divide our transcript into segments of 10 words. In
linear, the transcriber corrects segments in linear order from the start until the time budget
12

1800
dynamic-oracleCM
dynamic-proposed
dynamic-naivePrior
dynamic-fixedCM
staticranked-conflinear

1600

Corrected errors

1400
1200
1000
800
600
400
200
0

0

10

20

50

100

Update frequency / enrollment time [min]Figure 4: Errors corrected for dynamic-proposed and several other settings (see Table 7.2), plotted against
update frequency (first 3 methods) or enrollment time (following 2 methods) as described in the text.

is exhausted, using neither CM nor UM. In contrast, ranked-conf uses only the UM andselects segments in order of decreasing utility until the time budget is exhausted. Deviatingfrom the strict temporal order may be found confusing in practice, but does not affect thesimulation, so results for the second baseline should be interpreted as slightly optimistic. The
results show that the first baseline is much inferior, while the second baseline still slightly
underperforms the proposed approach, even without updates and only the prior CM. This
indicates that even using our crude prior CM is better than not using a CM at all.
Updating segmentations, but not CMs, is still helpful: As an ablation experiment,
the dynamic-fixedCM setting updates segmentations as proposed, but uses fixed CMs trained
via enrollment as the previous baseline. This sheds light on the benefit of updating segmentations on-the-fly to reflect the actual remaining time budget at different times. Segmentation
update frequency was fixed at 2.5 minutes, the fastest value employed in our simulations.
Compared to static, this brings considerable gains, especially when setting enrollment time
to 0, in which case 1540 errors are corrected (7% worse than with the best proposed setup).
This indicates that adjusting to the actual remaining time budget is a crucial factor in ourframework.
Naive CM prior compromises results: As an ablation experiment examining the importance of using the proposed prior, in dynamic-naivePrior we run the full proposed updating
framework, but use a naive CM prior that simply assumes transcription time to take up
1s per word, dropping the segment overhead. This leads to considerable losses, especially13when running updates infrequently, indicating that modeling segment overhead in the prior
is another significant factor.
Proposed updating framework almost matches oracle CM: As an upper bound forhow much benefit can be expected from better cost modeling, in dynamic-oracleCM we test
the full proposed updating framework, but with a CM equal to the true, oracle CM, except
for the noise. That is, the CM always predicts the correct time, except for the multiplicative
noise applied to the true model that captures the assumed unpredictable variations in working speed. Notice that this setup still somewhat benefits from segmentation updates because
prediction errors caused by the random noise can be better recovered from. Remarkably, forhigh update frequencies the dynamic-proposed setup almost reaches the dynamic-oracleCM
performance, with the latter gaining only 3.3% relative. This indicates that the combinationof CM- and segmentation updates is effective enough to render having a more accurate CM
to start with almost unnecessary.
7.3. Conclusions
Based on our simulated experiments, we draw a number of first conclusions:• Our updating framework is considerably stronger than the original Sesla (cost-sensitive
but without updates), as well as the cost-insensitive baselines.
• The higher the updating frequency, the better the results.
• The prior cost model can be crude, but modeling the segment overhead is important.
• Two factors both contribute to the effectiveness of our updating framework: segmentation is updated, making sure the time budget is not over- or underspent; the costmodel is updated to model the annotator more accurately.
8. User Study
We now turn to testing our method in a realistic setting. We choose a scenario that is
especially challenging in cost-sensitive annotation: we assume a new, previously unknown
transcriber, who is employed to annotate only one talk. We compare the strongest dynamic(proposed) setup against the strongest static (baseline) setup, according to the simulations
above. In particular, we compared the dynamic-proposed setting, which used the prior withoverhead, updates of segmentation and CM, and a high updating frequency, against the
cost-sensitive setting without dynamic updates (static). Enrollment was omitted, as it wasdeemed not worthwhile according to the simulations. Instead, we use the same prior as in the
proposed system that models the overhead for each segment, but under-predicts annotation
time in most cases. In practice, this leads to the annotator not being able to finish all
segments chosen by the algorithm.10 We do not compare to cost-insensitive baselines, as
earlier work indicated that cost-sensitive methods are superior [1].
10

In contrast, consider the case of an over-predicting CM without updates: here the annotator would
finish all segments before the time budget is exhausted, with the leftover time remaining unused. Our
under-predicting CM is preferable as it does not result in unused annotation time.

14

Talk

Reference length (# words)

Initial ASR WER

TED 1520
TED 1532
TED 1617
TED 1685

1403
1500
2175
1521

19.5
11.5
20.3
36.7

Table 2: Data overview.

For our user experiment, we use a selection of 4 TED talks that were distinct from theones used in the simulations (see Table 2). We include 12 participants in our experiment,with English skill ranging from good to native. As user interface for transcription, we extend
the Sesla Transcriber [28] by implementing the necessary updating features.11 Transcription is performed from scratch, but with the ASR hypothesis visible to the transcriber. Thiswas shown to be more effective than blindly transcribing from-scratch [29]. In addition, we
found it easier to automatically predict effort when transcribing from-scratch than for postediting. We also display the ASR hypothesis of surrounding segments to the transcriber,
and audio play-back includes a small portion of the preceding and succeeding segments toprovide enough context for successful transcription.
Transcribers were allowed a time budget of 20 minutes to correct as many errors as
possible in a particular talk. Every transcriber worked on every talk, each time pretending
it was the first piece of work for that transcriber. The 2 methods (dynamic-proposed, static)
were alternated for each transcriber, and the assignment for which talk was transcribedwith which method was shuffled between transcribers to reduce noise. Because of thisrandomized assignment, results are not directly comparable between different transcribers.
Simply averaging over transcribers, on the other hand, would not allow to draw conclusions
for individual transcribers. Instead, we use a linear mixed effects model [30] to explicitly
account for these random factors in our experiments. We used R [34] and lme4 [35] toperform our analysis, and tested statistical significance via the likelihood ratio test. A moredetailed explanation on the employed mixed effects models is presented in Appendix A.
8.1. Transcriber Characteristics
Before we turn to the evaluation of the end results, it will be helpful to analyze howtranscribers differ from one another.
Speed and accuracy varies strongly between transcribers: Figure 5 shows a plotof the edit rate (edit distance between initial and corrected transcripts) achieved within
the allotted 20-minute time budget as a proxy for how fast participants transcribed, and11

Recall that updating segmentations can take up to several seconds (§7.1), but waiting time of thismagnitude is undesirable. In our implementation, we start to compute an update in the background when
the transcriber just started annotating a new segment. By the time the annotation of that segment is finished,
the computations are usually finished, so that we can update the user interface before the annotator starts
with the next segment.

15

Success rate [%]

100
80
60
40
20
0

0

5

10
Edit rate [%]15

20

Figure 5: Transcriber characteristics: Edit rate achieved within the 20-minute time budget, plotted against
the success rate (proportion of edits that successfully improved the WER). According to this graph, thesemeasures of correction speed and quality vary considerably between the 12 transcribers, and are not strongly
correlated.

notice considerable differences between participants (edit rate between 7.2% and 15.8%).
We also measure how many of these edits actually improved the WER as an indicator for
how carefully transcribers worked, and again notice large differences (between 29% and 85%
success rate). Quantity and quality of corrections, when measured in these terms, are only
weakly correlated (Pearson correlation coefficient: 0.35).
Initial CM accuracy varies between transcribers: Figure 6 shows the per-segment
prediction accuraciesof the prior CM for eachPparticipant. We computed mean absolute error
P11(MAE) as n i=1...n |pi − yi | and bias as n i=1...n (pi − yi ), for n predicted values pi andtrue values yi . MAE is a common regression error measure, while the bias is a measure forwhether the model tends to under- or overpredict. The figure shows that the prior model
underpredicted on average for every transcriber, although there were large differences in how
much it underpredicted: the fastest transcriber was 1.3 seconds slower than predicted on
average, the slowest transcriber 10.7 seconds slower than predicted.
The differences between transcribers we observed are a strong indicator that transcriberspecific cost modeling will be worthwhile. Note also that transcribers who differ from theprior CM more strongly have a larger potential of benefitting from the proposed dynamic
CM adaptation.
8.2. Accuracy of Cost and Utility Models
To intrinsically assess CM performance, we compute the MAE of the predicted times
given the observed times. We do this for both the initial model based on only the prior, andfor trained models. For the latter, we train models for individual talks and transcribers, andperform 10-fold cross validation by training on 90% of the observed examples and testingon the remaining 10%, repeatedly across folds. The final number is averaged over all talks
and transcribers. Results are shown in Table 3 (left). It can be seen that the trained costmodels improve considerably over the initial CM, but are still not very reliable. We present
more details on convergence behavior in §8.4.
16

Prior model bias [seconds]

0
-2
-4
-6
-8
-10
-12

0

2

4
6
8
Prior model MAE [seconds]

10

12

Figure 6: Transcriber statistics: Mean absolute error (MAE) and bias of prior cost model on each transcriber’s annotations. For all 12 transcribers, the prior model underpredicted on average (the bias is negative), although the magnitude varies strongly.

Cost model

Utility model (length-normalized)

MAE prior model
MAE 10-fold cross validation

6.7 seconds MAE
0.19
4.9 seconds Pearson correlation 0.57

Table 3: Intrinsic CM and UM performance.

To evaluate the UM, we compute MAE and the Pearson linear correlation coefficient for
the initially computed segments of our 4 TED talks (both selected and skipped segments).
Utility predictions are normalized by segment length so that all segments have comparable
influence on the intrinsic results. As shown in Table 3 (right), the UM appears reasonably
strong in terms of both MAE and Pearson correlation.
8.3. Main Results: Transcription Efficiency
Proposed method improves overall efficiency: For an empirical evaluation of theend results, we fit a simple mixed effects model. The model predicts the WER improvement,
based only on a binary input that indicates whether dynamic-proposed or static was used.
The particular transcriber and talk are incorporated in the model as random effects. Figure 7 (left) shows the result: the proposed method reduced the WER by 8.2% absolute, thebaseline only by 7.1% absolute, making the baseline 15% more efficient in terms of corrected
errors per time. However, statistical significance of the used method being responsible forthe difference is quite weak (p=0.18).
Transcribers deviating from prior CM benefit most: To analyze further in whatsituations we can expect reliable gains from using the proposed method, we estimate theimpact of the used method on a per-transcriber basis. This is possible by fitting a refinedmodel with an additional random effect that models how strongly each transcriber is affectedby what method was used. The results reveal that the gains were especially strong for theslower transcribers (who deviated from the prior model the most), and much smaller for the
faster transcribers. In fact, the Spearman rank correlation between prior CM bias (Figure 6)
17

WER reduction abs. [%]8

staticdynamic-proposed

6
4
2
0

All 12 participants

Slower 6 participants

Figure 7: WER reduction for the two evaluated methods and different groups of participants.
14
dynamic-proposed
static12

WER reduction abs. [%]WER reduction abs. [%]

14
10
8
6
4
2
0

10
8
6
4
2
0

10

20
30
Initial ASR WER [%]40

dynamic-proposed
static

12

10

20
30
Initial ASR WER [%]40(b) Slower 6 participants(a) All 12 participants

Figure 8: WER reduction plotted against the initial ASR WER, for the two evaluated methods and different
groups of participants.

and observed gain is a highly positive value at 0.78. This can be explained by observing
that faster transcribers, who were more accurately modeled by the initial CM already, havelittle room for improvement from adapting the CMs. The slower transcribers, on the other
hand, deviated strongly from the initial CM, and benefitted much more from the dynamic
updating framework. Figure 7 (right) illustrates the fit of the original model when using only
the slower 50% of participants (which coincided with the 50% of participants with largest
negative prior model bias). Differences are much more pronounced: 4.1% WER reduction
for the baseline, 5.9% for the proposed method, a 42% relative productivity increase. The
difference is statistically significant (p=0.027) for this subgroup of users.
Gains are strongest when ASR WER is low: We fit another mixed effects modelthat includes the initial ASR WER as a fixed effect. The higher the initial ASR WER, themore errors can potentially be removed. We model the influence of the initial ASR WER
separately for baseline and proposed method. The results are visualized in Figure 8a. First,
note that as expected, the initial ASR WER has a strong influence, with more errors being
corrected for initially bad transcripts. Moreover, we confirm that in general our proposed
method reduces the WER more strongly than the baseline method with this more flexible
model, as well. A perhaps surprising observation is that transcripts with low initial WER
seem to benefit most from the proposed method. We hypothesize that this can be explained18

Corrections per second

0.2

dynamic-proposed
static

0.15
0.1
0.05
0

0

5

10
Clock time [minutes]

15

20

Figure 9: Efficiency over time, averaged over all transcribers and talks, and grouped into bins for eachminute on the clock. The efficiency of dynamic-proposed visibly improves over time, as cost models are
updated, while static efficiency remains constant.

as follows: When the initial WER is high, even randomly selected segments have a relatively
high chance of containing a high proportion of errors that can be corrected. For low initial
WERs, on the other hand, it is more important to pick carefully what segments to spend
time on, because many candidate segments contain only few errors. In the latter case, themore accurate CMs evidently help making an appropriate selection of segments. Figure 8b
shows the same model again for only the slower half of participants, again revealing a muchlarger gain from the proposed method. Statistical significance is again stronger: p=0.084
for the subgroup as opposed to p=0.33 for all participants.
8.4. Analysis of Dynamics
In the previous section, we confirmed that our method improves overall correction efficiency. Now, we take a closer look at the dynamic behavior of our updating framework.
Transcription efficiency improves over time: We first analyze how efficiency changes
as CMs are improved. We compute efficiency for each corrected segment as number of corrected errors in the segment divided by editing time. We expect that as the CMs improve
during transcription of a talk, so will the efficiency. Figure 9 demonstrates that this is thecase: the further the clock time progresses within the given 20 minute time budget, thehigher the per-segment efficiency for the proposed method. In this graph, results for alltranscribers are averaged. The static baseline does not change over time, as expected. Notealso for the proposed method that efficiency does not appear to have converged after the 20
minutes worth of transcription time.
CMs converge within but not across talks: To analyze long-term CM convergence,
we use the logged annotation times for all 4 talks transcribed per transcriber. We shuffle
the data points, and train CMs for each transcriber by progressively adding more data to
the training. We measure convergence by feeding the trained CMs into Sesla to selectsegments from a fixed, held-out TED talk. We measure how many words overlap between
segmentations created using different CMs: we expect that when the CM has converged,
adding more training data to it will not change the segmentation significantly, leading to a
19

0.5
0

0

20
40
60
80
Training data [minutes labeled](a) All segments shuffled.

1.0

Word overlap

Word overlap

Word overlap

1.0

0.5
0

1.0
0.5
0

0

20
40
60
80
Training data [minutes labeled](b) Adding complete talks.

0

20
40
60
80
Training data [minutes labeled](c) Adding half talks.

Figure 10: Cost model convergence, indicated by proportion of overlapping words (y-axis) between consecutive re-segmentation of a talk by using more and more CM training examples (x-axis). When shuffling
segments from all talks, cost models seem to converge (10a). When instead adding all annotations from aparticular talk at once, cost models seem to strongly diverge from the previous model (10b). When adding
annotations from half talks, adding the first half of a talk causes the model to diverge from the previous
model, while after adding the second half the model remains more similar (10c).

word overlap close to 1. Figure 10a shows the averaged convergence over all participants. The
figure suggests that CMs are almost converged after the 80 minutes of annotation. However,this analysis does not show the full picture, as becomes evident when repeating the same
experiment but without shuffling the data. That is, we first add all the annotation data fromthe first talk to the training, then the second talk, and so on. Figure 10b shows the result,and reveals that in fact CMs are diverging across talks. Apparently the transcribers’ working
speeds are relatively consistent within a transcribed talk, but vary strongly between different
talks. Potential causes could be factors such as familiarity with topic and speaking style,fatigue, etc. Figure 10c provides some more evidence to support this explanation. Here, we
add training data not for the whole talk, but progressively for half talks, keeping the sameorder. We can see that in general, when adding the first half of annotated data of a talkto the CM training, the segmentation overlap tends to be quite low, similar to the previous
experiment. However, when adding the second half, the overlap is much bigger. This again
indicates that CM training data is relatively consistent when the annotations stem from thesame talk, but much less consistent when moving to a new annotated talk. We conclude
from these observations that task-dependent differences may dominate convergence, which
would be another strong reason to prefer our proposed updating approach over a static,
enrollment-based approach that cannot adapt to these observed dynamics.
8.5. Discussion
We conclude from our experiment that our updating framework is effective in the challenging scenario investigated. Gains over the static baseline were particularly strong whena transcriber deviated strongly from the prior CM, and when initial ASR transcript was
already of relatively high quality. Importantly, we saw that transcribers have very different
working speeds, making a strong case for transcriber-dependent modeling. Moreover, theobservation that transcription effort varies depending on the particular data indicates thatCMs learned on-the-fly are a better choice than static CMS (transcriber-dependent or not),even if enrollment effort is amortized over time.
20

9. Conclusion
We proposed a new dynamic updating framework for cost-sensitive correction of speechtranscripts, along with a new, faster algorithm that determines which segments to correct.
Besides being more convenient to use than previous cost-sensitive methods that required
transcriber enrollment, we show in a simulated study that our approach yields higher supervision efficiency. Efficiency gains are attributed to updating cost models, updating segmentations, and a sensible initial cost model.
We ran a user study for the scenario where every transcriber conducts only a small
amount of work, which is especially challenging in the context of cost-sensitive annotation.
We observed productivity gains of 15% on average across all 12 participants, and 42% for the
6 participants whose working style deviated the most from the initial cost model. Besides
confirming the effectiveness of our proposed updating framework, the large gap between
different transcribers indicates that transcriber-specific cost modeling is crucial.
For future research, we suggest investigating how dynamically updating cost models can
optimally capture the per-transcriber variance in productivity caused by influences such as
fatigue and topic familiarity. Extension to other tasks, especially post-editing for machinetranslation, seems promising. A further scenario worth investigating is multi-pass transcription [36] with high quality requirements, where the first pass could be made cheaperusing our proposed method. Moreover, additional passes using our method could be allowed
if more time budget becomes available at a later point. Finally, we suggest investigating
dynamic updates of a transcriber dependent utility model.
Acknowledgments
We thank the anonymous reviewers for their valuable comments. We also thank Oliver
Adams, Philip Arthur, Tuna Murat Cicek, Angela Grimminger, Michael Heck, Matthew
Holland, Joseph Irwin, Nurul Lubis, Santiago Escobar Martinez, Patrick Lumban Tobing,and Micha Wetzel for their help with the user experiments.
Appendix A. Mixed Effects Models
We give a brief introduction to mixed effects models, and refer to the relevant literaturefor details [30]. Mixed effects models are a tool to analyze data that is influenced by “random” factors that are difficult to control for, and that potentially have a significant influenceon our observations. In our setting in particular, we are concerned about the influence ofparticular transcribers and particular talks. Many characteristics of the particular transcriber and talk can influence transcription speed and effectiveness. Because it is virtually
impossible to account for these influences, we consider them as noise that we wish to get rid
of. In fact, this is a common problem in user studies. Recently, mixed effects models havebecome a popular a way of dealing with such situations [31, 32, 33].
Mixed effects models are specified by the following components:21• Response variable: The central quantity for which we wish to determine how it is influenced
by other measured covariates. In our experiments, this is the reduction in word error rateachieved in a particular transcription run.
• Fixed effects: Numerical or categorical attributes that influence the response variable ina meaningful way. In this paper, we assume a linear relationship. Our fixed effects are(1) the user interface, and (2) the initial ASR WER which has a strong influence on how
many errors are corrected and should potentially be explicitly accounted for.
• Random effects: Categorical factors that are hard to control for or hard to understand.
Generally, the observations include only a limited sample of values out of a large set of
possible values (as is the case with the particular transcribers and talks that are part of ourexperiment). For each random effect an intercept is estimated by the model. Thus, e.g.
for each transcriber a mean WER reduction is estimated to explain some of the observed
variance. Random effects are modeled to follow a Gaussian distribution with the observed
sample mean and variance to be estimated.
• Error term: The variance in observations that is not explained by the random effects isfinally modeled by the general error term.
For the simple models in our experiments, we restrict ourselves to linear mixed effects
models. These can be seen as an extension of linear regression models, extended by the
random effects component. We tested the significance of the fixed effects in our models via
likelihood ratio test. That is, we built a null-model with the effect in question removed, andexamined whether this significantly reduced the model likelihood.
References[1] M. Sperber, M. Simantzik, G. Neubig, S. Nakamura, A. Waibel, Segmentation for Efficient Supervised Language Annotation with an Explicit Cost-Utility Tradeoff, Transactions of the Association for
Computational Linguistics (TACL) 2 (April) (2014) 169–180.
[2] B. C. Roy, D. Roy, Fast transcription of unstructured audio recordings, in: Annual Conference of theInternational Speech Communication Association (InterSpeech), Brighton, UK, 2009, pp. 1623–1626.
[3] I. Sanchez-Cortina, N. Serrano, A. Sanchis, A. Juan, A prototype for Interactive Speech Transcription
Balancing Error and Supervision Effort, in: International Conference on Intelligent User Interfaces(IUI), Lisbon, Portugal, 2012, pp. 325–326.
[4] M. Sperber, G. Neubig, C. Fügen, S. Nakamura, A. Waibel, Efficient Speech Transcription ThroughRespeaking, in: Annual Conference of the International Speech Communication Association (InterSpeech), Lyon, France, 2013, pp. 1087–1091.
[5] B. Settles, M. Craven, L. Friedland, Active Learning with Real Annotation Costs, in: Neural Information Processing Systems Conference (NIPS) - Workshop on Cost-Sensitive Learning, Lake Tahoe, USA,2008.
[6] K. Tomanek, U. Hahn, A Comparison of Models for Cost-Sensitive Active Learning, in: International
Conference on Computational Linguistics (COLING), Beijing, China, 2010, pp. 1247–1255.
[7] M. E. Ramirez-Loaiza, A. Culotta, M. Bilgic, Anytime Active Learning, in: Conference on Artificial
Intelligence (AAAI), Québec City, Canada, 2014, pp. 2048–2054.

22

[8] J. D. Valor Miró, J. A. Silvestre-Cerdà, J. Civera, C. Turró, A. Juan, Efficiency and Usability Study ofInnovative Computer-aided Transcription Strategies for Video Lecture Repositories, Speech Communication 74 (September) (2015) 65–75.
[9] M. Sperber, G. Neubig, S. Nakamura, A. Waibel, On-the-Fly User Modeling for Cost-Sensitive Correction of Speech Transcripts, in: Spoken Language Technology Workshop (SLT), Lake Tahoe, USA,2014, pp. 460–465.
[10] P. Zigoris, Y. Zhang, Bayesian Adaptive User Profiling with Explicit & Implicit Feedback, in: International Conference on Information and Knowledge Management (CIKM), Arlington, USA, 2006, pp.
397–404.
[11] J. G. C. de Souza, M. Negri, E. Ricci, M. Turchi, Online Multitask Learning for Machine Translation
Quality Estimation, in: Association for Computational Linguistics (ACL), Beijing, China, 2015, pp.
219–228.
[12] J. González-Rubio, D. Ortiz-Martı́nez, F. Casacuberta, Balancing User Effort and Translation Error inInteractive Machine Translation Via Confidence Measures, in: Association for Computational Linguistics Conference (ACL), Uppsala, Sweden, 2010, pp. 173–177.
[13] G. Haffari, M. Roy, A. Sarkar, Active Learning for Statistical Phrase-based Machine Translation, in:North American Chapter of the Association for Computational Linguistics - Human Language Technologies Conference (NAACL-HLT), Boulder, USA, 2009, pp. 415–423.
[14] J. González-Rubio, D. Ortiz-Martı́nez, F. Casacuberta, An active learning scenario for interactive
machine translation, in: International Conference on Multimodal Interfaces (ICMI), Alicante, Spain,
2011, pp. 197–200.
[15] F. Olsson, A literature survey of active machine learning in the context of natural language processing,
Tech. rep., SICS Sweden (2009).
[16] A. Miura, G. Neubig, M. Paul, S. Nakamura, Selecting Syntactic, Non-redundant Segments in Active
Learning for Machine Translation, in: North American Chapter of the Association for Computational
Linguistics (NAACL), San Diego, USA, 2016.
[17] K. Tomanek, U. Hahn, S. Lohmann, A Cognitive Cost Model of Annotations Based on Eye-Tracking
Data, in: Association for Computational Linguistics Conference (ACL), Uppsala, Sweden, 2010, pp.
1158–1167.
[18] L. Specia, Exploiting Objective Annotations for Measuring Translation Post-editing Effort, in: Conference of the European Association for Machine Translation (EAMT), Nice, France, 2011, pp. 73–80.
[19] T. Cohn, L. Specia, Modelling Annotator Bias with Multi-task Gaussian Processes: An Application
to Machine Translation Quality Estimation, in: Association for Computational Linguistics Conference(ACL), Sofia, Bulgaria, 2013.
[20] S. Vijayanarasimhan, K. Grauman, What’s It Going to Cost You?: Predicting Effort vs. Informativenessfor Multi-Label Image Annotations, in: Conference on Computer Vision and Pattern Recognition(CVPR), Miami Beach, USA, 2009, pp. 2262–2269.
[21] S. Vijayanarasimhan, P. Jain, K. Grauman, Far-sighted active learning on a budget for image and video
recognition, in: Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, USA,2010, pp. 3035–3042.
[22] P. Donmez, J. Carbonell, Proactive Learning : Cost-Sensitive Active Learning with Multiple ImperfectOracles, in: Conference on Information and Knowledge Management (CIKM), Napa Valley, USA, 2008,
pp. 619–628.
[23] L. Rodriguez, F. Casacuberta, E. Vidal, Computer Assisted Transcription of Speech, in: Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), Girona, Spain, 2007, pp. 241–248.
[24] C. E. Rasmussen, C. K. Williams, Gaussian Processes for Machine Learning, MIT Press, Cambridge,
MA, USA, 2006.
[25] L. Specia, K. Shah, J. G. C. de Souza, T. Cohn, QuEst - A Translation Quality Estimation Framework,in: Association for Computational Linguistics: System Demonstrations, Sofia, Bulgaria, 2013, pp.
79–84.
[26] A. Fiacco, G. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Tech-23

niques, John Wiley & Sons, New York, USA, 1968.
[27] H. Soltau, F. Metze, C. Fügen, A. Waibel, A One-Pass Decoder Based on Polymorphic Linguistic Context Assignment, in: Automatic Speech Recognition and Understanding Workshop (ASRU), Madonna
di Campiglio, Italy, 2001, pp. 214–217.
[28] M. Sperber, G. Neubig, S. Nakamura, A. Waibel, SESLA Transcriber: A Speech Transcription Tool
That Adapts To Your Skill And Time Budget, in: Spoken Language Technology Workshop (SLT), Lake
Tahoe, USA, 2014.
[29] M. Sperber, G. Neubig, S. Nakamura, A. Waibel, Optimizing Computer-Assisted Transcription Quality
with Iterative User Interfaces, in: Language Resources and Evaluation (LREC), Portorož, Slovenia,
2016.
[30] J. Pinheiro, D. Bates, Mixed-effects models in S and S-PLUS, Springer Science & Business Media, 2006.
[31] S. Goldwater, D. Jurafsky, C. D. Manning, Which words are hard to recognize? Prosodic, lexical,
and disfluency factors that increase speech recognition error rates, Speech Communication 52 (2010)
181–200.
[32] M. Federico, M. Negri, L. Bentivogli, M. Turchi, Assessing the Impact of Translation Errors on MachineTranslation Quality with Mixed-effects Models, in: Empirical Methods in Natural Language Processing(EMNLP), Doha, Qatar, 2014, pp. 1643–1653.
[33] S. Green, J. Heer, C. D. Manning, The efficacy of human post-editing for language translation, in:Conference on Human factors in computing systems (CHI), Paris, France, 2013, pp. 439–448.
[34] R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for StatisticalComputing, Vienna, Austria (2014).
URL http://www.r-project.org/[35] D. Bates, M. Maechler, B. Bolker, S. Walker, lme4: Linear mixed-effects models using Eigen and S4,
R package version 1 (4).
[36] S. Stüker, F. Kraft, C. Mohr, T. Herrmann, E. Cho, A. Waibel, The KIT Lecture Corpus for Speech
Translation, in: Language Resources and Evaluation (LREC), Istanbul, Turkey, 2012, pp. 3409–3414.

24