Package: meeteval
MeetEvalA meeting transcription evaluation toolkitFeatures|Installation|Python Interface|Command Line Interface|Visualization (WIP)|CiteFeaturesMeetEval supports the following metrics for meeting transcription evaluation:Standard WERfor single utterances (Called SISO WER in MeetEval)meeteval-wer wer -r ref -h hypConcatenated minimum-Permutation Word Error Rate (cpWER)meeteval-wer cpwer -r ref.stm -h hyp.stmOptimal Reference Combination Word Error Rate (ORC WER)meeteval-wer orcwer -r ref.stm -h hyp.stmMulti-speaker-input multi-stream-output Word Error Rate (MIMO WER)meeteval-wer mimower -r ref.stm -h hyp.stmTime-Constrained minimum-Permutation Word Error Rate (tcpWER)meeteval-wer tcpwer -r ref.stm -h hyp.stm --collar 5Time-Constrained Optimal Reference Combination Word Error Rate (tcORC WER)meeteval-wer tcorcwer -r ref.stm -h hyp.stm --collar 5Diarization Error Rate (DER)by wrappingmdevalmeeteval-der md_eval_22 -r ref.stm -h hyp.stm --collar .25Additionally (WIP), MeetEval contains avisualizationtool for cpWER and tcpWER alignments that help spot errors in system outputs.InstallationFrom PyPIpipinstallmeetevalFrom sourcegitclonehttps://github.com/fgnt/meeteval
pipinstall-e./meetevalCommand-line interfaceMeetEvalsupports the following file formats as input:Segmental Time Mark(STM)Time Marked Conversation(CTM)SEGment-wise Long-form Speech Transcription annotation(SegLST), the file format used in theCHiME challengesRich Transcription Time Marked(RTTM) files (only for Diarizaiton Error Rate)[!NOTE]MeetEvaldoes not support alternate transcripts (e.g.,"i've { um / uh / @ } as far as i'm concerned").The command-line interface is available asmeeteval-werorpython -m meeteval.werwith the following signature:python-mmeeteval.wer[orcwer|mimower|cpwer|tcpwer|tcorcwer]-hexample_files/hyp.stm-rexample_files/ref.stm# ormeeteval-wer[orcwer|mimower|cpwer|tcpwer|tcorcwer]-hexample_files/hyp.stm-rexample_files/ref.stmYou can add--helpto any command to get more information about the available options.
The command nameorcwer,mimower,cpwerandtcpwerselects the metric to use.
By default, the hypothesis files is used to create the template for the average
(e.g.hypothesis.json) and per_recohypothesis_per_reco.jsonfile.
They can be changed with--average-outand--per-reco-out..jsonand.yamlare the supported suffixes.More examples can be found intests/test_cli.py.File FormatsSEGment-wise Long-form Speech Transcription annotation (SegLST)The SegLST format was used in theCHiME-7 challengeand is the default format forMeetEval.The SegLST format is stored in JSON format and contains a list of segments.
Each segment should have a minimum set of keys"session_id"and"words".
Depending on the metric, additional keys may be required ("speaker","start_time","end_time").An example is shown below:[{"session_id":"recordingA",# Required"words":"The quick brown fox jumps over the lazy dog",# Required for WER metrics"speaker":"Alice",# Required for metrics that use speaker information (cpWER, ORC WER, MIMO WER)"start_time":0,# Required for time-constrained metrics (tcpWER, tcORC-WER, DER, ...)"end_time":1,# Required for time-constrained metrics (tcpWER, tcORC-WER, DER, ...)"audio_path":"path/to/recordingA.wav"# Any additional keys can be included},...]Another example can be foundhere.Segmental Time Mark (STM)Each line in anSTMfile represents one "utterance" and is defined asSTM :== <filename> <channel> <speaker_id> <begin_time> <end_time> <transcript>wherefilename: name of the recordingchannel: ignored by MeetEvalspeaker_id: ID of the speaker or system output stream/channel (not microphone channel)begin_time: in seconds, used to find the order of the utterancesend_time: in secondstranscript: space-separated list of wordsfor example:recording1 1 Alice 0 0 Hello Bob.
recording1 1 Bob 1 0 Hello Alice.
recording1 1 Alice 2 0 How are you?
...
recording2 1 Alice 0 0 Hello Carol.
...An exampleSTMfile can be found inhere.Time Marked Conversation (CTM)The CTM format is defined asCTM :== <filename> <channel> <begin_time> <duration> <word> [<confidence>]for the hypothesis (one file per speaker).
You have to supply oneCTMfile for each system output channel using multiple-harguments sinceCTMfiles don't encode speaker or system output channel information (thechannelfield has a different meaning: left or right microphone).
For example:meeteval-werorcwer-hhyp1.ctm-hhyp2.ctm-rreference.stm[!NOTE]
Note that theLibriCSSbaseline recipe produces oneCTMfile which merges the speakers, so that it cannot be applied straight away. We recommend to useSTMorSegLSTfiles.Python interfaceFor all metrics aLow-levelandhigh-levelinterface is available.[!TIP]
You want to use thehigh-levelfor computing metrics over a full dataset.You want to use thelow-levelinterface for computing metrics for single examples or when your data is represented as Python structures, e.g., nested lists of strings.Low-level interfaceAll WERs have a low-level interface in themeeteval.wermodule that allows computing the WER for single examples.
The functions take the reference and hypothesis as input and return anErrorRateobject.
TheErrorRatebundles statistics (errors, total number of words) and potential auxiliary information (e.g., assignment for ORC WER) together with the WER.importmeeteval# SISO WERwer=meeteval.wer.wer.siso.siso_word_error_rate(reference='The quick brown fox jumps over the lazy dog',hypothesis='The kwick brown fox jump over lazy ')print(wer)# ErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2)# cpWERwer=meeteval.wer.wer.cp.cp_word_error_rate(reference=['The quick brown fox','jumps over the lazy dog'],hypothesis=['The kwick brown fox','jump over lazy '])print(wer)# CPErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2, missed_speaker=0, falarm_speaker=0, scored_speaker=2, assignment=((0, 0), (1, 1)))# ORC-WERwer=meeteval.wer.wer.orc.orc_word_error_rate(reference=['The quick brown fox','jumps over the lazy dog'],hypothesis=['The kwick brown fox','jump over lazy '])print(wer)# OrcErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2, assignment=(0, 1))The input format can be a (list of) strings or an object representing a file format frommeeteval.io:importmeetevalwer=meeteval.wer.wer.cp.cp_word_error_rate(reference=meeteval.io.STM.parse('recordingA 1 Alice 0 1 The quick brown fox jumps over the lazy dog'),hypothesis=meeteval.io.STM.parse('recordingA 1 spk-1 0 1 The kwick brown fox jump over lazy '))print(wer)# CPErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2, missed_speaker=0, falarm_speaker=0, scored_speaker=1, assignment=(('Alice', 'spk-1'),))All low-level interfaces come with a single-example function (as show above) and a batch function that computes the WER for multiple examples at once.
The batch function is postfixed with_multifileand is similar to the high-level interface without fancy input format handling.
To compute the average over multipleErrorRates, usemeeteval.wer.combine_error_rates.
Note that the combined WER isnotthe average over the error rates, but the error rate that results from combining the errors and lengths of all error rates.combine_error_ratesalso discards any information that cannot be aggregated over multiple examples (such as the ORC WER assignment).For example with the cpWER:importmeeteval.wer.wer.sisowers=meeteval.wer.wer.cp.cp_word_error_rate_multifile(reference={'recordingA':{'speakerA':'First example','speakerB':'First example second speaker'},'recordingB':{'speakerA':'Second example'},},hypothesis={'recordingA':['First example with errors','First example second speaker'],'recordingB':['Second example','Overestimated speaker'],})print(wers)# {#   'recordingA': CPErrorRate(error_rate=0.3333333333333333, errors=2, length=6, insertions=2, deletions=0, substitutions=0, missed_speaker=0, falarm_speaker=0, scored_speaker=2, assignment=(('speakerA', 0), ('speakerB', 1))),#   'recordingB': CPErrorRate(error_rate=1.0, errors=2, length=2, insertions=2, deletions=0, substitutions=0, missed_speaker=0, falarm_speaker=1, scored_speaker=1, assignment=(('speakerA', 0), (None, 1)))# }# Use combine_error_rates to compute an "overall" WER over multiple examplesavg=meeteval.wer.combine_error_rates(wers)print(avg)# CPErrorRate(error_rate=0.5, errors=4, length=8, insertions=4, deletions=0, substitutions=0, missed_speaker=0, falarm_speaker=1, scored_speaker=3)High-level interfaceAll WERs have a high-level Python interface available directly in themeeteval.wermodule that mirrors theCommand-line interfaceand accepts the formats frommeeteval.ioas input.
All of these functions require the input format to contain a session-ID and output a dict mapping from session-ID to the result of that sessionimportmeeteval# File Pathswers=meeteval.wer.tcpwer('example_files/ref.stm','example_files/hyp.stm',collar=5)# Loaded fileswers=meeteval.wer.tcpwer(meeteval.io.load('example_files/ref.stm'),meeteval.io.load('example_files/hyp.stm'),collar=5)# Objectswers=meeteval.wer.tcpwer(reference=meeteval.io.STM.parse('''recordingA 1 Alice 0 1 The quick brown fox jumps over the lazy dogrecordingB 1 Bob 0 1 The quick brown fox jumps over the lazy dog'''),hypothesis=meeteval.io.STM.parse('''recordingA 1 spk-1 0 1 The kwick brown fox jump over lazyrecordingB 1 spk-1 0 1 The kwick brown fox jump over lazy'''),collar=5,)print(wers)# {#     'recordingA': CPErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2, reference_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('1')), hypothesis_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('1')), missed_speaker=0, falarm_speaker=0, scored_speaker=1, assignment=(('Alice', 'spk-1'),)),#     'recordingB': CPErrorRate(error_rate=0.4444444444444444, errors=4, length=9, insertions=0, deletions=2, substitutions=2, reference_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('1')), hypothesis_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('1')), missed_speaker=0, falarm_speaker=0, scored_speaker=1, assignment=(('Bob', 'spk-1'),))# }avg=meeteval.wer.combine_error_rates(wers)print(avg)# CPErrorRate(error_rate=0.4444444444444444, errors=8, length=18, insertions=0, deletions=4, substitutions=4, reference_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('2')), hypothesis_self_overlap=SelfOverlap(overlap_rate=Decimal('0'), overlap_time=0, total_time=Decimal('2')), missed_speaker=0, falarm_speaker=0, scored_speaker=2)Aligning sequencesSequences can be aligned, similar tokaldialign.align, using the tcpWER matching:importmeetevalmeeteval.wer.wer.time_constrained.align([{'words':'a b','start_time':0,'end_time':1}],[{'words':'a c','start_time':0,'end_time':1},{'words':'d','start_time':2,'end_time':3}])# [('a', 'a'), ('b', 'c'), ('*', 'd')]Visualization [WIP][!IMPORTANT]
The visualization is under development!Preview:https://groups.uni-paderborn.de/nt/meeteval/viz.htmlInteractive notebook:https://fgnt.github.io/meeteval_jupyterlite/lab?path=Demo.ipynbimportmeetevalfrommeeteval.viz.visualizeimportAlignmentVisualizationfolder=r'https://raw.githubusercontent.com/fgnt/meeteval/main/'av=AlignmentVisualization(meeteval.io.load(folder+'example_files/ref.stm').groupby('filename')['recordingA'],meeteval.io.load(folder+'example_files/hyp.stm').groupby('filename')['recordingA'])# display(av)  # Jupyter# av.dump('viz.html')  # Create standalone HTML fileCiteThe toolkit and the tcpWER were presented at the CHiME-2023 workshop (Computational Hearing in Multisource Environments) with the paper"MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems".@InProceedings{MeetEval23,title={MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems},author={von Neumann, Thilo and Boeddeker, Christoph and Delcroix, Marc and Haeb-Umbach, Reinhold},booktitle={CHiME-2023 Workshop, Dublin, England},year={2023}}The MIMO WER and efficient implementation of ORC WER are presented in the paper"On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems".@InProceedings{MIMO23,author={von Neumann, Thilo and Boeddeker, Christoph and Kinoshita, Keisuke and Delcroix, Marc and Haeb-Umbach, Reinhold},title={On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems},booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},pages={1--5},year={2023},organization={IEEE}}