SESLA TRANSCRIBER: A SPEECH TRANSCRIPTION TOOLTHAT ADAPTS TO YOUR SKILL AND TIME BUDGET
Matthias Sperber1 , Graham Neubig2 , Satoshi Nakamura2 , Alex Waibel1
12Karlsruhe Institute of Technology, Institute of Anthropomatics and Robotics, Germany
Nara Institute of Science and Technology, Augmented Human Communications Laboratory, Japan
ABSTRACT

We present a speech transcription tool targeted at situations
in which cost is a critical or limiting factor. This tool actively
guides the transcription process by taking an automaticallycreated transcript as a starting point, and asking for correction of only the parts likely to contain errors. The transcriber
specifies a time budget, and the software uses models of transcription accuracy and cost to choose which segments shouldbe transcribed to achieve the highest error reduction. Thisapproach has been found to be 25% more efficient than costinsensitive approaches in previous work. The cost model isadapted to the transcriber on-the-fly during the transcription
process, so no user enrollment is necessary. The segmentation is updated regularly to reflect improved cost models, andto recover from potential time prediction errors. The user interface was designed to be easy to learn and efficient to use. It
allows either transcribing each segment from scratch or postediting, and has logging features that allow detailed user studies.
1. COST-SENSITIVE TRANSCRIPTIONThis paper describes a new tool for efficient manual correction of speech transcripts. Our tool uses an automaticallycreated transcript as a starting point, and guides the transcriber through the correction of a selection of segments thatare likely to contain errors. The general strategy is to focus
only on those erroneous parts, and trust the speech recognizer
for other parts, in order to cut transcription costs.
Our tool asks the user to specify a time-budget (for example, 30 minutes of annotation), and automatically chooses
an appropriate number of segments for correction such thatthe time-budget is kept. The locations and sizes of these segments are chosen such that the expected reduction of errors
is maximized given the time budget, according to the S ESLA
method (Segmentation for Efficient Supervised Language
Annotation) as described in [1]. Specifically, the tool predicts
both transcription time and error reduction for transcribing
any possible segment in a speech. Using these predictions,an optimal segmentation into segments to transcribe and segments to skip is computed. There is a trade-off to consider

when choosing between smaller and longer segments: Whilechoosing very small segments (e.g. single words) would allow the transcriber to really concentrate only on parts thathave a very high probability of error, longer segments aredesirable from a cognitive point of view as they reduce cognitive overhead due to context switches. A global constrainedoptimization strategy that is based on the time and error reduction predictions and considers all possible segmentations
allows finding an optimal trade-off in a principled way. Savings in human effort of 25% were observed, compared tothe traditional approach of choosing low-confident segmentsfrom a fixed segmentation.
To obtain the transcription time and error reduction predictions needed to find the optimal segmentation, the tool
proceeds as follows. Confidence scores provided by the automatic speech recognizer are employed to estimate chanceof error. Transcription time is predicted via Gaussian Process
regression [2], with the features segment length, audio duration, and average word confidence. The model is continually
retrained on the observed transcription times during the ongoing transcription process. The regressor is initiated with asensible prior so that rough predictions are possible even for
new users, no initial user enrollment is required. Based on theprior model, the regressor is then retrained regularly to reflect
the transcriber’s characteristics with gradually increasing accuracy. To take advantage of the improving user models, thesegmentation of the remainder of the speech being transcribedis updated regularly as well. Each segmentation update reflects the updated currently remaining time budget, and in thisway, for instance, allows removing (skipping) less promisingsegments if the transcriber’s progress has been slower thanexpected. Hence, we can rapidly recover from prediction inaccuracies that are expected to occur during practical use, andmake sure that the remaining time is used optimally. These
updating strategies have been proposed in [3], along with afast segmentation algorithm which we employ in our tool.
2. USER INTERFACE
We took care to design the user interface in a way that is botheasy to learn, and allows fast and convenient operation foradvanced users. It has evolved over several iterations based

on pilot studies with users of various backgrounds, including computer scientists, linguists, and low-cost transcribers
with low-profile educational background. To use our tool, aspeech file along with a confidence-annotated automatic transcript must be provided. It is intended for correcting speech
files of medium to long duration, such as lectures or meetings. First, a transcription time budget must be specified, anda trade-off curve with the expected number of errors that canbe removed for different time budgets is displayed to aid thedecision for a time budget (Figure 1).
As depicted in Figure 2, the transcription view displays
the speech as a vertical waveform, and the transcriber canclick anywhere in the waveform to play back the audio fromthat position. The selected segments are aligned next to it,with segments to transcribe colored white and segments toskip colored in gray. Transcription occurs in linear order from
top to bottom. The active (white) segments can be transcribedor corrected by simply clicking on that segment. The corresponding audio will be played automatically, and can be
repeated if necessary. Transcription is supported both fromscratch or post-editing style, depending on an initial setting.
Segments that should not be corrected are always displayedwith their automatic transcript. Clicking on them causes thecorresponding audio to be played, but they cannot be edited.
After finishing a segment, the transcriber can confirm the segment, or mark it as unsure. Advanced users have complete
control over the user interface via keyboard shortcuts. The
specification of the time budget is the only input needed toenable the cost-sensitive transcription; if no budget is set, thewhole talk will be transcribed. User models and the segmentation are updated fully automatically in the background without disrupting the transcription workflow. Longer periods ofinactivity are recognized, and not subtracted from the timebudget. In addition, outlier segments, such as if the transcriber got distracted, are automatically recognized and notused for the user model training. Multiple users are supported, with separate user models being trained for each user,but the transcription of the same speech cannot be shared between several transcribers. The tool logs detailed user activities such as keystrokes and audio playback to help answer
research questions.
Our transcription tool can be used on Windows and Macintosh computers, and is available for academic use fromhttp://msperber.com/sesla-transcriber.

Acknowledgments
The research leading to these results has received funding
from the European Union Seventh Framework Programme(FP7/2007-2013) under grant agreement n 287658 Bridges
Across the Language Divide (EU-BRIDGE).

Fig. 1. Time budget prompt with visualized trade-off curve.

Fig. 2. The main transcription screen.
3. REFERENCES[1] Matthias Sperber, Mirjam Simantzik, Graham Neubig,
Satoshi Nakamura, and Alex Waibel, “Segmentation for
Efficient Supervised Language Annotation with an Explicit Cost-Utility Tradeoff,” Transactions of the Association for Computational Linguistics (TACL), vol. 2, no.
April, pp. 169–180, 2014.
[2] Carl E. Rasmussen and Christopher K.I. Williams, Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, USA, 2006.
[3] Matthias Sperber, Graham Neubig, Satoshi Nakamura,and Alex Waibel, “On-the-Fly User Modeling for CostSensitive Correction of Speech Transcripts,” in Spoken
Language Technology Workshop (SLT), 2014.