DOI: 10.5281/zenodo.18734592
title: Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use

description: This repository contains transcriptions and evaluation metrics from Automatic Speech Recognition (ASR) experiments of the work "Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use" using OpenAI's Whisper models on audio files from the Gigaspeech dataset.

If you use or base your work on this project or you need more details, please cite the following article:



@ARTICLE{11186205,
  author={Arriaga, Carlos and Pozo, Alejandro and Conde, Javier and Alonso, Alvaro},
  journal={IEEE Internet Computing}, 
  title={Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use}, 
  year={2025},
  volume={29},
  number={5},
  pages={17-24},
  keywords={Delays;Real-time systems;Measurement;Hardware;Mathematical models;Transformers;Time measurement;Speech to text;Loss measurement;Artificial intelligence},
  doi={10.1109/MIC.2025.3614363}}


Overview

The dataset provides a comprehensive comparison of Whisper ASR performance across different:



Model sizes: Tiny, Base, and Large

Transcription modes: Batch (offline) and Real-time (online)

Audio splitting algorithms: VAD, Fixed-length (2s, 3s), and Feedback-based


The dataset includes 1,748 transcription files from 159 unique audio files processed under 11 different experimental configurations, along with detailed evaluation metrics.

Dataset Structure

.
├── transcriptions/          # Transcription outputs
│   ├── batchBase/          # Batch mode with Whisper Base model
│   ├── batchLarge/         # Batch mode with Whisper Large model
│   ├── batchTiny/          # Batch mode with Whisper Tiny model
│   ├── realBase2s/         # Real-time Base model with 2-second fixed chunks
│   ├── realBase3s/         # Real-time Base model with 3-second fixed chunks
│   ├── realBaseFeed/       # Real-time Base model with feedback-based splitting
│   ├── realBaseVad/        # Real-time Base model with VAD-based splitting
│   ├── realTiny2s/         # Real-time Tiny model with 2-second fixed chunks
│   ├── realTiny3s/         # Real-time Tiny model with 3-second fixed chunks
│   ├── realTinyFeed/       # Real-time Tiny model with feedback-based splitting
│   └── realTinyVad/        # Real-time Tiny model with VAD-based splitting
└── results/                # Evaluation metrics
    ├── batchBase.csv       # Metrics for batch Base model
    ├── batchLarge.csv      # Metrics for batch Large model
    ├── batchTiny.csv       # Metrics for batch Tiny model
    ├── realBase2s.csv      # Metrics for real-time Base with 2s chunks
    ├── realBase3s.csv      # Metrics for real-time Base with 3s chunks
    ├── realBaseFeed.csv    # Metrics for real-time Base with feedback
    ├── realBaseVad.csv     # Metrics for real-time Base with VAD
    ├── realTiny2s.csv      # Metrics for real-time Tiny with 2s chunks
    ├── realTiny3s.csv      # Metrics for real-time Tiny with 3s chunks
    ├── realTinyFeed.csv    # Metrics for real-time Tiny with feedback
    └── realTinyVad.csv     # Metrics for real-time Tiny with VAD


File Count by Configuration




Configuration
Number of Files




batchBase
159


batchLarge
159


batchTiny
159


realBase2s
159


realBase3s
159


realBaseFeed
159


realBaseVad
159


realTiny2s
159


realTiny3s
159


realTinyFeed
159


realTinyVad
158


Total
1,748




Audio Files

The transcribed audio files are from the Gigaspeech corpus and are identified by their unique IDs:



POD prefix: Podcast audio files (e.g., POD1000000004.txt)

YOU prefix: YouTube audio files (e.g., YOU1000000000.txt)


The original audio files are available in the Gigaspeech repository and are not included in this dataset.

Transcription Formats

Batch Transcriptions

Batch mode transcriptions (offline processing) are stored as plain text files containing the complete transcription:

Hey friends, I don't know about you, but when I was a kid I knew absolutely nothing about money.
Well there's a new show from Marketplace and Brains on that will help address just that...


Real-time Transcriptions

Real-time mode transcriptions (online processing) are stored in JSON format with sentence-level timestamps:

{"sentence": "Friends, I don't know about you, but when I was a kid.", "timestamp": 3325};
{"sentence": "I knew absolutely nothing about it.", "timestamp": 5223};
{"sentence": "money. Well, there's a new show.", "timestamp": 7280};




sentence: The transcribed text segment

timestamp: Time in milliseconds from the start of audio playback when the transcription was presented to the user


Evaluation Metrics

Each CSV file in the results/ directory contains evaluation metrics for the corresponding experimental configuration. The metrics include:




Metric
Description




FILE
Audio file identifier from Gigaspeech


WER
Word Error Rate - measures word-level accuracy


MER
Match Error Rate - alternative error metric


WIL
Word Information Lost - information-theoretic measure




Example Results

FILE,WER,MER,WIL
POD1000000039,0.2107904642409034,0.1969519343493552,0.2719032632574976
YOU1000000107,0.19279907084785133,0.18485523385300667,0.23827720179632295
POD1000000011,0.11490082244799225,0.11121517209084524,0.1535611413001976


Lower values indicate better transcription accuracy.

Experimental Configurations

Transcription Modes





Batch (Offline): The entire audio file is processed at once before transcription begins. This represents the ideal scenario where the complete audio is available.




Real-time (Online): Audio is processed in a streaming fashion, simulating real-time transcription scenarios where audio chunks are transcribed as they arrive.



Whisper Models



Tiny: Fastest, smallest model (~39M parameters)

Base: Balanced speed and accuracy (~74M parameters)

Large: Highest accuracy, slowest (~1550M parameters)


Audio Splitting Algorithms (Real-time only)



VAD (Voice Activity Detection): Segments audio based on detected speech activity

Fixed-length (2s): Splits audio into fixed 2-second chunks

Fixed-length (3s): Splits audio into fixed 3-second chunks

Feedback: Adaptive splitting based on transcription feedback


Ground Truth

Ground truth transcriptions are obtained from the Gigaspeech repository metadata. The Gigaspeech corpus provides professionally annotated transcriptions for all audio files.

Usage Examples

Loading Batch Transcriptions

# Read a batch transcription
with open('transcriptions/batchBase/POD1000000004.txt', 'r') as f:
    transcription = f.read()
print(transcription)


Loading Real-time Transcriptions

import json

# Read real-time transcription with timestamps
with open('transcriptions/realBase2s/POD1000000004.txt', 'r') as f:
    lines = f.readlines()

# Parse JSON entries
transcriptions = [json.loads(line.rstrip(';')) for line in lines]

for entry in transcriptions:
    print(f"[{entry['timestamp']}ms] {entry['sentence']}")


Loading Evaluation Metrics

import pandas as pd

# Load metrics for batch Base model
metrics = pd.read_csv('results/batchBase.csv')

# Display summary statistics
print(metrics.describe())

# Find best and worst performing files
best_file = metrics.loc[metrics['WER'].idxmin()]
worst_file = metrics.loc[metrics['WER'].idxmax()]

print(f"Best WER: {best_file['FILE']} ({best_file['WER']:.4f})")
print(f"Worst WER: {worst_file['FILE']} ({worst_file['WER']:.4f})")


Comparing Models

import pandas as pd
import matplotlib.pyplot as plt

# Load metrics for different models
batch_tiny = pd.read_csv('results/batchTiny.csv')
batch_base = pd.read_csv('results/batchBase.csv')
batch_large = pd.read_csv('results/batchLarge.csv')

# Compare average WER
models = ['Tiny', 'Base', 'Large']
wer_means = [
    batch_tiny['WER'].mean(),
    batch_base['WER'].mean(),
    batch_large['WER'].mean()
]

plt.bar(models, wer_means)
plt.ylabel('Average WER')
plt.title('Whisper Model Comparison')
plt.show()


File Naming Convention



POD[ID].txt: Transcriptions from podcast audio files

YOU[ID].txt: Transcriptions from YouTube audio files

Where [ID] is a unique 10-digit identifier from the Gigaspeech corpus


FAIR Compliance

This dataset follows the FAIR (Findable, Accessible, Interoperable, Reusable) principles for scientific data management:

Findable



Persistent Identifier: The dataset is deposited in Zenodo with a Digital Object Identifier (DOI) for permanent citation and retrieval

Rich Metadata: Comprehensive metadata including title, authors, keywords, and abstract describing the dataset

Searchable: Indexed in multiple research data repositories and discoverable through standard search engines

Unique Identifiers: Each audio file maintains its unique Gigaspeech ID for cross-referencing with the original corpus


Accessible



Open Access: Freely available for download without registration barriers

Standard Protocols: Accessible via HTTPS and standard web protocols

Long-term Preservation: Hosted on Zenodo, ensuring long-term availability and preservation

Complete Documentation: This README provides comprehensive documentation of file formats, structure, and usage


Interoperable



Standard Formats:



Transcriptions stored in plain text (.txt) and JSON formats

Metrics stored in CSV format for easy import into analysis tools

All formats are non-proprietary and widely supported



Consistent Schema: Uniform file structure across all experimental configurations

Cross-referencing: File IDs link directly to the Gigaspeech corpus for audio retrieval

Standardized Metrics: WER, MER, and WIL are widely recognized ASR evaluation metrics


Reusable



Clear Licensing: Dataset released under an open license (specify in repository)

Detailed Provenance:



Source data clearly identified (Gigaspeech corpus)

Processing methods documented (Whisper models, versions, parameters)

Experimental conditions fully described



Quality Assurance: Ground truth from professionally annotated Gigaspeech transcriptions

Comprehensive Documentation:



File formats and structures clearly explained

Usage examples provided in multiple programming languages

Citation information for proper attribution



Research Paper: Accompanied by peer-reviewed publication detailing methodology and findings


Metadata Summary




Attribute
Value




Title
Assessing Latency in ASR Systems: Whisper Transcription Dataset


Creators
Carlos Arriaga, Alejandro Pozo, Javier Conde, Alvaro Alonso


Publication Date
2025


Publisher
Zenodo


Resource Type
Dataset


Subject Area
Speech Recognition, Natural Language Processing, Machine Learning


Language
English (transcriptions)


Source Data
Gigaspeech Corpus


File Formats
.txt, .json, .csv


Size
1,748 files


Related Publication
IEEE Internet Computing, DOI: 10.1109/MIC.2025.3614363




Data Statistics



Total transcription files: 1,748

Unique audio files: 159

Experimental configurations: 11

Whisper models tested: 3 (Tiny, Base, Large)

Splitting algorithms: 4 (VAD, 2s, 3s, Feedback)

Transcription modes: 2 (Batch, Real-time)