Package: ppgs
High-Fidelity Neural Phonetic PosteriorgramsTraining, evaluation, and inference of neural phonetic posteriorgrams (PPGs) in PyTorch[Paper][Website]Table of contentsInstallationInferenceApplication programming interface (API)ppgs.from_audioppgs.from_fileppgs.from_file_to_fileppgs.from_files_to_filesppgs.from_paths_to_pathsCommand-line interface (CLI)DistanceInterpolateEditppgs.edit.grid.constantppgs.edit.grid.from_alignmentsppgs.edit.grid.of_lengthppgs.edit.grid.sampleppgs.edit.reallocateppgs.edit.regexppgs.edit.shiftppgs.edit.swapSparsifyTrainingDownloadPreprocessPartitionTrainMonitorEvaluateCitationInstallationAn inference-only installation with our best model is pip-installablepip install ppgsTo perform training, install training dependencies and FFMPEG.pipinstallppgs[train]condainstall-cconda-forgeffmpegIf you wish to use the Charsiu representation, download the code,
install both inference and training dependencies, and install
Charsiu as a Git submodule.# Clonegitclonegit@github.com/interactiveaudiolab/ppgscdppgs/# Install dependenciespipinstall-e.[train]condainstall-cconda-forgeffmpeg# Download Charsiugitsubmoduleinit
gitsubmoduleupdateInferenceimportppgs# Load speech audio at correct sample rateaudio=ppgs.load.audio(audio_file)# Choose a gpu index to use for inference. Set to None to use cpu.gpu=0# Infer PPGsppgs=ppgs.from_audio(audio,ppgs.SAMPLE_RATE,gpu=gpu)Application programming interface (API)ppgs.from_audiodeffrom_audio(audio:torch.Tensor,sample_rate:Union[int,float],checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,gpu:int=None)->torch.Tensor:"""Infer ppgs from audioArgumentsaudioBatched audio to processshape=(batch, 1, samples)sample_rateAudio sampling ratecheckpointThe checkpoint filegpuThe index of the GPU to use for inferenceReturnsppgsPhonetic posteriorgramsshape=(batch, len(ppgs.PHONEMES), frames)"""ppgs.from_filedeffrom_file(file:Union[str,bytes,os.PathLike],checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,gpu:Optional[int]=None)->torch.Tensor:"""Infer ppgs from an audio fileArgumentsfileThe audio filecheckpointThe checkpoint filegpuThe index of the GPU to use for inferenceReturnsppgsPhonetic posteriorgramshape=(len(ppgs.PHONEMES), frames)"""ppgs.from_file_to_filedeffrom_file_to_file(audio_file:Union[str,bytes,os.PathLike],output_file:Union[str,bytes,os.PathLike],checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,gpu:Optional[int]=None)->None:"""Infer ppg from an audio file and save to a torch tensor fileArgumentsaudio_fileThe audio fileoutput_fileThe .pt file to save PPGscheckpointThe checkpoint filegpuThe index of the GPU to use for inference"""ppgs.from_files_to_filesdeffrom_files_to_files(audio_files:List[Union[str,bytes,os.PathLike]],output_files:List[Union[str,bytes,os.PathLike]],checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,num_workers:int=ppgs.NUM_WORKERS,gpu:Optional[int]=None,max_frames:int=ppgs.MAX_INFERENCE_FRAMES)->None:"""Infer ppgs from audio files and save to torch tensor filesArgumentsaudio_filesThe audio filesoutput_filesThe .pt files to save PPGscheckpointThe checkpoint filenum_workersNumber of CPU threads for multiprocessinggpuThe index of the GPU to use for inferencemax_framesThe maximum number of frames on the GPU at once"""ppgs.from_paths_to_pathsdeffrom_paths_to_paths(input_paths:List[Union[str,bytes,os.PathLike]],output_paths:Optional[List[Union[str,bytes,os.PathLike]]]=None,extensions:Optional[List[str]]=None,checkpoint:Optional[Union[str,bytes,os.PathLike]]=None,num_workers:int=ppgs.NUM_WORKERS,gpu:Optional[int]=None,max_frames:int=ppgs.MAX_INFERENCE_FRAMES)->None:"""Infer ppgs from audio files and save to torch tensor filesArgumentsinput_pathsPaths to audio files and/or directoriesoutput_pathsThe one-to-one corresponding outputsextensionsExtensions to glob for in directoriescheckpointThe checkpoint filenum_workersNumber of CPU threads for multiprocessinggpuThe index of the GPU to use for inferencemax_framesThe maximum number of frames on the GPU at once"""Command-line interface (CLI)usage: python -m ppgs
    [-h]
    [--input_paths INPUT_PATHS [INPUT_PATHS ...]]
    [--output_paths OUTPUT_PATHS [OUTPUT_PATHS ...]]
    [--extensions EXTENSIONS [EXTENSIONS ...]]
    [--checkpoint CHECKPOINT]
    [--num-workers NUM_WORKERS]
    [--gpu GPU]
    [--max-frames MAX_TRAINING_FRAMES]

arguments:
    --input_paths INPUT_PATHS [INPUT_PATHS ...]
        Paths to audio files and/or directories

optional arguments:
    -h, --help
        Show this help message and exit
    --output_paths OUTPUT_PATHS [OUTPUT_PATHS ...]
        The one-to-one corresponding output paths
    --extensions EXTENSIONS [EXTENSIONS ...]
        Extensions to glob for in directories
    --checkpoint CHECKPOINT
        The checkpoint file
    --num-workers NUM_WORKERS
        Number of CPU threads for multiprocessing
    --gpu GPU
        The index of the GPU to use for inference. Defaults to CPU.DistanceTo compute the proposed normalized Jenson-Shannon divergence pronunciation
distance between two PPGs, useppgs.distance().defdistance(ppgX:torch.Tensor,ppgY:torch.Tensor,reduction:str='mean',normalize:bool=True)->torch.Tensor:"""Compute the pronunciation distance between two aligned PPGsArgumentsppgXInput PPG Xshape=(len(ppgs.PHONEMES), frames)ppgYInput PPG Y to compare with PPG Xshape=(len(ppgs.PHONEMES), frames)reductionReduction to apply to the output. One of ['mean', 'none', 'sum'].normalizeApply similarity based normalizationReturnsNormalized Jenson-shannon divergence between PPGs"""Interpolatedefinterpolate(ppgX:torch.Tensor,ppgY:torch.Tensor,interp:Union[float,torch.Tensor])->torch.Tensor:"""Spherical linear interpolationArgumentsppgXInput PPG Xshape=(len(ppgs.PHONEMES), frames)ppgYInput PPG Yshape=(len(ppgs.PHONEMES), frames)interpInterpolation valuesscalar float OR shape=(frames,)ReturnsInterpolated PPGsshape=(len(ppgs.PHONEMES), frames)"""Editimportppgs# Get PPGs to editppg=ppgs.from_file(audio_file,gpu=gpu)# Constant-ratio time-stretching (slowing down)grid=ppgs.edit.grid.constant(ppg,ratio=0.8)slow=ppgs.edit.grid.sample(ppg,grid)# Stretch to a desired length (e.g., 100 frames)grid=ppgs.edit.grid.of_length(ppg,100)fixed=ppgs.edit.grid.sample(ppg,grid)ppgs.edit.grid.constantdefconstant(ppg:torch.Tensor,ratio:float)->torch.Tensor:"""Create a grid for constant-ratio time-stretchingArgumentsppgInput PPGratioTime-stretching ratio; lower is slowerReturnsConstant-ratio grid for time-stretching ppg"""ppgs.edit.grid.from_alignmentsdeffrom_alignments(source:pypar.Alignment,target:pypar.Alignment,sample_rate:int=ppgs.SAMPLE_RATE,hopsize:int=ppgs.HOPSIZE)->torch.Tensor:"""Create time-stretch grid to convert source alignment to targetArgumentssourceForced alignment of PPG to stretchtargetForced alignment of target PPGsample_rateAudio sampling ratehopsizeHopsize in samplesReturnsGrid for time-stretching source PPG"""ppgs.edit.grid.of_lengthdefof_length(ppg:torch.Tensor,length:int)->torch.Tensor:"""Create time-stretch grid to resample PPG to a specified lengthArgumentsppgInput PPGlengthTarget lengthReturnsGrid of specified length for time-stretching ppg"""ppgs.edit.grid.sampledefgrid_sample(ppg:torch.Tensor,grid:torch.Tensor)->torch.Tensor:"""Grid-based PPG interpolationArgumentsppgInput PPGgridGrid of desired length; each item is a float-valued index into ppgReturnsInterpolated PPG"""ppgs.edit.reallocatedefreallocate(ppg:torch.Tensor,source:str,target:str,value:Optional[float]=None)->torch.Tensor:"""Reallocate probability from source phoneme to target phonemeArgumentsppgInput PPGshape=(len(ppgs.PHONEMES), frames)sourceSource phonemetargetTarget phonemevalueMax amount to reallocate. If None, reallocates all probability.ReturnsEdited PPG"""ppgs.edit.regexdefregex(ppg:torch.Tensor,source_phonemes:List[str],target_phonemes:List[str])->torch.Tensor:"""Regex match and replace (via swap) for phoneme sequencesArgumentsppgInput PPGshape=(len(ppgs.PHONEMES), frames)source_phonemesSource phoneme sequencetarget_phonemesTarget phoneme sequenceReturnsEdited PPG"""ppgs.edit.shiftdefshift(ppg:torch.Tensor,phoneme:str,value:float):"""Shift probability of a phoneme and reallocate proportionallyArgumentsppgInput PPGshape=(len(ppgs.PHONEMES), frames)phonemeInput phonemevalueMaximal shift amountReturnsEdited PPG"""ppgs.edit.swapdefswap(ppg:torch.Tensor,phonemeA:str,phonemeB:str)->torch.Tensor:"""Swap the probabilities of two phonemesArgumentsppgInput PPGshape=(len(ppg.PHONEMES), frames)phonemeAInput phoneme AphonemeBInput phoneme BReturnsEdited PPG"""Sparsifydefsparsify(ppg:torch.Tensor,method:str='percentile',threshold:Union[float,int]=0.85)->torch.Tensor:"""Make phonetic posteriorgrams sparseArgumentsppgInput PPGshape=(*, len(ppgs.PHONEMES), frames)methodSparsification method. One of ['constant', 'percentile', 'topk'].thresholdIn [0, 1] for 'contant' and 'percentile'; integer > 0 for 'topk'.ReturnsSparse phonetic posteriorgramshape=(*, len(ppgs.PHONEMES), frames)"""TrainingDownloadDownloads, unzips, and formats datasets. Stores datasets indata/datasets/.
Stores formatted datasets indata/cache/.N.B.Common voice and TIMIT cannot be automatically downloaded. You must
manually download the tarballs and place them indata/sources/commonvoiceordata/sources/timit, respectively, prior to running the following.python-mppgs.data.download--datasets<datasets>PreprocessPrepares representations for training. Representations are stored
indata/cache/.python -m ppgs.data.preprocess \
   --datasets <datasets> \
   --representatations <representations> \
   --gpu <gpu> \
   --num-workers <workers>PartitionPartitions a dataset. You should not need to run this, as the partitions
used in our work are provided for each dataset inppgs/assets/partitions/.python -m ppgs.partition --datasets <datasets>TrainTrains a model. Checkpoints and logs are stored inruns/.python -m ppgs.train --config <config> --dataset <dataset> --gpu <gpu>If the config file has been previously run, the most recent checkpoint will
automatically be loaded and training will resume from that checkpoint.MonitorYou can monitor training viatensorboard.tensorboard --logdir runs/ --port <port> --load_fast trueTo use thetorchutilnotification system to receive notifications for long
jobs (download, preprocess, train, and evaluate), set thePYTORCH_NOTIFICATION_URLenvironment variable to a supported webhook as
explained inthe Apprise documentation.EvaluatePerforms objective evaluation of phoneme accuracy. Results are stored
ineval/.python -m ppgs.evaluate --config <name> --datasets <datasets> --gpu <gpu>CitationIEEEC. Churchwell, M. Morrison, and B. Pardo, "High-Fidelity Neural Phonetic Posteriorgrams,"
ICASSP 2024 Workshop on Explainable Machine Learning for Speech and Audio, April 2024.BibTex@inproceedings{churchwell2024high,
    title={High-Fidelity Neural Phonetic Posteriorgrams},
    author={Churchwell, Cameron and Morrison, Max and Pardo, Bryan},
    booktitle={ICASSP 2024 Workshop on Explainable Machine Learning for Speech and Audio},
    month={April},
    year={2024}
}