Package: emphases
Crowdsourced and Automatic Speech Prominence EstimationAnnotation, training, evaluation and inference of speech prominencepip install emphasesPaperWebsiteTable of contentsInferenceApplication programming interfaceemphases.from_alignment_and_audioemphases.from_text_and_audioemphases.from_fileemphases.from_file_to_fileemphases.from_files_to_filesCommand-line interfaceTrainingDownloadAnnotatePreprocessPartitionTrainMonitorEvaluationEvaluateAnalyzeCitationInferencePerform automatic emphasis annotation using our best pretrained modelimportemphases# Text and audio of speechtext_file='example.txt'audio_file='example.wav'# Detect emphasesalignment,prominence=emphases.from_file(text_file,audio_file)# Check which words were emphasizedforword,scoreinzip(alignment,prominence[0]):print(f'{word}has a prominence of{score}')Thealignmentis apypar.Alignmentobject.Application programming interfaceemphases.from_alignment_and_audiodeffrom_alignment_and_audio(alignment:pypar.Alignment,audio:torch.Tensor,sample_rate:int,checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,batch_size:Optional[int]=None,gpu:Optional[int]=None)->Tuple[Type[pypar.Alignment],torch.Tensor]:"""Produce emphasis scores for each wordArgs:alignment: The forced phoneme alignmentaudio: The speech waveformsample_rate: The audio sampling ratecheckpoint: The model checkpoint to use for inferencebatch_size: The maximum number of frames per batchgpu: The index of the gpu to run inference onReturns:scores: The float-valued emphasis scores for each word"""emphases.from_text_and_audiodeffrom_text_and_audio(text:str,audio:torch.Tensor,sample_rate:int,checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,batch_size:Optional[int]=None,gpu:Optional[int]=None)->Tuple[Type[pypar.Alignment],torch.Tensor]:"""Produce emphasis scores for each wordArgs:text: The speech transcriptaudio: The speech waveformsample_rate: The audio sampling ratecheckpoint: The model checkpoint to use for inferencebatch_size: The maximum number of frames per batchgpu: The index of the gpu to run inference onReturns:alignment: The forced phoneme alignmentscores: The float-valued emphasis scores for each word"""emphases.from_filedeffrom_file(text_file:Union[str,bytes,os.PathLike],audio_file:Union[str,bytes,os.PathLike],checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,batch_size:Optional[int]=None,gpu:Optional[int]=None)->Tuple[Type[pypar.Alignment],torch.Tensor]:"""Produce emphasis scores for each word for files on diskArgs:text_file: The speech transcript (.txt) or alignment (.TextGrid) fileaudio_file: The speech waveform audio filecheckpoint: The model checkpoint to use for inferencebatch_size: The maximum number of frames per batchgpu: The index of the gpu to run inference onReturns:alignment: The forced phoneme alignmentscores: The float-valued emphasis scores for each word"""emphases.from_file_to_filedeffrom_file_to_file(text_file:List[Union[str,bytes,os.PathLike]],audio_file:List[Union[str,bytes,os.PathLike]],output_prefix:Optional[List[Union[str,bytes,os.PathLike]]]=None,checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,batch_size:Optional[int]=None,gpu:Optional[int]=None)->None:"""Produce emphasis scores for each word for files on disk and save to diskArgs:text_file: The speech transcript (.txt) or alignment (.TextGrid) fileaudio_file: The speech waveform audio fileoutput_prefix: The output prefix. Defaults to text file stem.checkpoint: The model checkpoint to use for inferencebatch_size: The maximum number of frames per batchgpu: The index of the gpu to run inference on"""Emphases are saved as a list of five-tuples containing the word, start time,
end time, a float-valued emphasis score, and a boolean that is true if the
word is emphasized.emphases.from_files_to_filesdeffrom_files_to_files(text_files:List[Union[str,bytes,os.PathLike]],audio_files:List[Union[str,bytes,os.PathLike]],output_prefixes:Optional[List[Union[str,bytes,os.PathLike]]]=None,checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,batch_size:Optional[int]=None,gpu:Optional[int]=None)->None:"""Produce emphasis scores for each word for many files and save to diskArgs:text_file: The speech transcript (.txt) or alignment (.TextGrid) filesaudio_files: The corresponding speech audio filesoutput_prefixes: The output files. Defaults to text file stems.checkpoint: The model checkpoint to use for inferencebatch_size: The maximum number of frames per batchgpu: The index of the gpu to run inference on"""Command-line interfacepython -m emphases
    [-h]
    --text_files TEXT_FILES [TEXT_FILES ...]
    --audio_files AUDIO_FILES [AUDIO_FILES ...]
    [--output_files OUTPUT_FILES [OUTPUT_FILES ...]]
    [--checkpoint CHECKPOINT]
    [--batch_size BATCH_SIZE]
    [--gpu GPU]

Determine which words in a speech file are emphasized

options:
  -h, --help            show this help message and exit
  --text_files TEXT_FILES [TEXT_FILES ...]
                        The speech transcript text files
  --audio_files AUDIO_FILES [AUDIO_FILES ...]
                        The corresponding speech audio files
  --output_files OUTPUT_FILES [OUTPUT_FILES ...]
                        The output files. Default is text files with json suffix.
  --checkpoint CHECKPOINT
                        The model checkpoint to use for inference
  --batch_size BATCH_SIZE
                        The maximum number of frames per batch
  --gpu GPU             The index of the gpu to run inference onTrainingDownload datapython -m emphases.download --datasets <datasets>.Downloads and uncompresses datasets.N.B.We omit Buckeye for public release. This evaluation dataset can be
made bydownloading Buckeyeand matching
the files to theannotations.
The process of matching the files to the annotations was done for us and is
tricky to replicate exactly. However, due to licensing restrictions on
Buckeye, we cannot legally distribute our private, aligned annotations.Annotate dataPerforming annotation requires first installingReproducible Subjective Evaluation (ReSEval).python -m emphases.annotate --datasets <datasets>Launches a local web application to perform emphasis annotation, according to
the ReSEval configuration fileemphases/assets/configs/annotate.yaml.
Requires ReSEval to be installed.python -m emphases.annotate --datasets <datasets> --remote --productionLaunches a crowdsourced emphasis annotation task, according to the ReSEval
configuration fileemphases/assets/configs/annotate.yaml. Requires ReSEval
to be installed.Partition datapython -m emphases.partitionGeneratestrain,valid, andtestpartitions for all datasets.
Partitioning is deterministic given the same random seed. You do not need to
run this step, as the original partitions are saved inemphases/assets/partitions.Preprocesspython -m emphases.preprocessTrainpython -m emphases.train --config <config> --dataset <dataset> --gpus <gpus>Trains a model according to a given configuration. Uses a list of GPU
indices as an argument, and uses distributed data parallelism (DDP)
if more than one index is given. For example,--gpus 0 3will train
using DDP on GPUs0and3.EvaluationEvaluatepython -m emphases.evaluate --config <config> --checkpoint <checkpoint> --gpu <gpu>MonitorRuntensorboard --logdir runs/. If you are running training
remotely, you must create a SSH connection with port forwarding to view
Tensorboard. This can be done withssh -L 6006:localhost:6006 <user>@<server-ip-address>. Then, openlocalhost:6006in your browser.CitationIEEEM. Morrison, P. Pawar, N. Pruyne, J. Cole, and B. Pardo, "Crowdsourced and Automatic Speech Prominence Estimation," International Conference on Acoustics, Speech, & Signal Processing, 2024.BibTex@inproceedings{morrison2024crowdsourced,
    title={Crowdsourced and Automatic Speech Prominence Estimation},
    author={Morrison, Max and Pawar, Pranav and Pruyne, Nathan and Cole, Jennifer and Pardo, Bryan},
    booktitle={International Conference on Acoustics, Speech, & Signal Processing},
    year={2024}
}