Package: uisrnn
UIS-RNNOverviewThis is the library for theUnbounded Interleaved-State Recurrent Neural Network (UIS-RNN)algorithm.
UIS-RNN solves the problem of segmenting and clustering sequential data
by learning from examples.This algorithm was originally proposed in the paperFully Supervised Speaker Diarization.The work has been introduced byGoogle AI Blog.DisclaimerThis open source implementation is slightly different than the internal one
which we used to produce the results in thepaper, due to dependencies on
some internal libraries.We CANNOT share the data, code, or model for the speaker recognition system
(d-vector embeddings)
used in the paper, since the speaker recognition system
heavily depends on Google's internal infrastructure and proprietary data.This library is NOT an official Google product.We welcome community contributions (guidelines)
to theuisrnn/contribfolder.
But we won't be responsible for the correctness of any community contributions.DependenciesThis library depends on:python 3.5+numpy 1.15.1pytorch 1.3.0scipy 1.1.0 (for evaluation only)Getting StartedInstall the packageWithout downloading the repository, you can install thepackageby:pip3 install uisrnnorpython3 -m pip install uisrnnRun the demoTo get started, simply run this command:python3demo.py--train_iteration=1000-l=0.001This will train a UIS-RNN model usingdata/toy_training_data.npz,
then store the model on disk, perform inference ondata/toy_testing_data.npz,
print the inference results, and save the averaged accuracy in a text file.PS. The files underdata/are manually generatedtoy data,
for demonstration purpose only.
These data are very simple, so we are supposed to get 100% accuracy on the
testing data.Run the testsYou can also verify the correctness of this library by running:bashrun_tests.shIf you fork this library and make local changes, be sure to use these tests
as a sanity check.Besides, these tests are also great examples for learning
the APIs, especiallytests/integration_test.py.Core APIsGlossaryGeneral Machine LearningSpeaker DiarizationSequenceUtteranceObservation / FeatureEmbedding / d-vectorLabel / Cluster IDSpeakerArgumentsIn your main script, call this function to get the arguments:model_args,training_args,inference_args=uisrnn.parse_arguments()Model constructionAll algorithms are implemented as theUISRNNclass. First, construct aUISRNNobject by:model=uisrnn.UISRNN(args)The definitions of the args are described inuisrnn/arguments.py.
Seemodel_parser.TrainingNext, train the model by calling thefit()function:model.fit(train_sequences,train_cluster_ids,args)The definitions of the args are described inuisrnn/arguments.py.
Seetraining_parser.Thefit()function accepts two types of input, as described below.Input as list of sequences (recommanded)Here,train_sequencesis a list of observation sequences.
Each observation sequence is a 2-dim numpy array of typefloat.The first dimension is the length of this sequence. And the length
can vary from one sequence to another.The second dimension is the size of each observation. This
must be consistent among all sequences. For speaker diarization,
the observation could be thed-vector embeddings.train_cluster_idsis also a list, which has the same length astrain_sequences. Each element oftrain_cluster_idsis a 1-dim list or
numpy array of strings, containing the ground truth labels for the
corresponding sequence intrain_sequences.
For speaker diarization, these labels are the speaker identifiers for each
observation.When callingfit()in this way, please be very careful with the argument--enforce_cluster_id_uniqueness.For example, assume:train_cluster_ids=[['a','b'],['a','c']]If the label'a'from the two sequences refers to the same cluster across
the entire dataset, then we should haveenforce_cluster_id_uniqueness=False;
otherwise, if'a'is only a local indicator to distinguish from'b'in the
1st sequence, and to distinguish from'c'in the 2nd sequence, then we should
haveenforce_cluster_id_uniqueness=True.Also, please note that, when callingfit()in this way, we are going to
concatenate all sequences and all cluster IDs, and delegate to
the next section below.Input as single concatenated sequenceHere,train_sequencesshould be a single 2-dim numpy array of typefloat,
for theconcatenatedobservation sequences.For example, if you haveMtraining utterances,
and each utterance is a sequence ofLembeddings. Each embedding is
a vector ofDnumbers. Then the shape oftrain_sequencesisN * D,
whereN = M * L.train_cluster_idsis a 1-dim list or numpy array of strings, of lengthN.
It is theconcatenatedground truth labels of all training data.Since we are concatenating observation sequences, it is important to note that,
ground truth labels intrain_cluster_idacross different sequences are
supposed to beglobally unique.For example, if the set of labels in the first
sequence is{'A', 'B', 'C'}, and the set of labels in the second sequence
is{'B', 'C', 'D'}. Then before concatenation, we should rename them to
something like{'1_A', '1_B', '1_C'}and{'2_B', '2_C', '2_D'},
unless'B'and'C'in the two sequences are meaningfully identical
(in speaker diarization, this means they are the same speakers across
utterances). This part will be automatically taken care of by the argument--enforce_cluster_id_uniquenessfor the previous section.The reason we concatenate all training sequences is that, we will be resampling
andblock-wiseshuffling the training data as adata augmentationprocess, such that we result in a robust model even when there is insufficient
number of training sequences.Training on large datasetsFor large datasets, the data usually could not be loaded into memory at once.
In such cases, thefit()function needs to be called multiple times.Here we provide a few guidelines as our suggestions:Do not feed different datasets into different calls offit(). Instead,
for each call offit(), the input should cover sequences from
different datasets.For each call to thefit()function, make the size of input roughly the
same. And, don't make the input size too small.PredictionOnce we are done with training, we can run the trained model to perform
inference on new sequences by calling thepredict()function:predicted_cluster_ids=model.predict(test_sequences,args)Heretest_sequencesshould be a list of 2-dim numpy arrays of typefloat,
corresponding to the observation sequences for testing.The returnedpredicted_cluster_idsis a list of the same size astest_sequences. Each element ofpredicted_cluster_idsis a list of integers,
with the same length as the corresponding test sequence.You can also use a single test sequence fortest_sequences. Then the returnedpredicted_cluster_idswill also be a single list of integers.The definitions of the args are described inuisrnn/arguments.py.
Seeinference_parser.CitationsOur paper is cited as:@inproceedings{zhang2019fully,
  title={Fully supervised speaker diarization},
  author={Zhang, Aonan and Wang, Quan and Zhu, Zhenyao and Paisley, John and Wang, Chong},
  booktitle={International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={6301--6305},
  year={2019},
  organization={IEEE}
}ReferencesBaseline diarization systemTo learn more about our baseline diarization system based onunsupervised clusteringalgorithms, check outthis site.A Python re-implementation of thespectral clusteringalgorithm used in this
paper is availablehere.The ground truth labels for theNIST SRE 2000dataset (Disk6 and Disk8) can be foundhere.For more public resources on speaker diarization, check outawesome-diarization.Speaker recognizer/encoderTo learn more about our speaker embedding system, check outthis site.We are aware of several third-party implementations of this work:TensorFlow implementation by Janghyun1230PyTorch implementaion by HarryVolek- with UIS-RNN integrationPyTorch implementation as part of SV2TTSPlease use your own judgement to decide whether you want to use these
implementations.We are NOT responsible for the correctness of any third-party implementations.VariantsHere we list the repositories that are based on UIS-RNN, but integrated with
other technologies or added some improvements.LinkDescriptiontaylorlu/Speaker-DiarizationSpeaker diarization using UIS-RNN and GhostVLAD. An easier way to support openset speakers.DonkeyShot21/uis-rnn-smlA variant of UIS-RNN, for the paper Supervised Online Diarization with Sample Mean Loss for Multi-Domain Data.