A method comprising: receiving, at data processing hardware, a first audio sample from a first speaker and a second audio sample from a second speaker; for each audio sample of the first audio sample and the second audio sample: a) dividing, by the data processing hardware, the respective audio sample into a plurality of audio slices; b) based on the plurality of slices, generating, by the data processing hardware, a set of candidate acoustic embeddings, each candidate acoustic embedding comprising a vector representation of acoustic features; c) determining, by the data processing hardware, a similarity score for each candidate acoustic embedding in the set of candidate acoustic embeddings; d) removing, by the data processing hardware, based on the similarity score for each candidate acoustic embedding from the set of candidate acoustic embeddings, a subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings; and e) generating, by the data processing hardware, an aggregate acoustic embedding from the remaining candidate acoustic embeddings in the set of candidate acoustic embeddings after removing the subset of the candidate acoustic embeddings; and identifying, by the data processing hardware, whether the first speaker and the second speaker are the same speaker or different speakers based on the aggregate acoustic embedding generated for each audio sample of the first audio sample and the second audio sample. receiving, at data processing hardware, a first audio sample from a first speaker and a second audio sample from a second speaker; for each audio sample of the first audio sample and the second audio sample: a) dividing, by the data processing hardware, the respective audio sample into a plurality of audio slices; b) based on the plurality of slices, generating, by the data processing hardware, a set of candidate acoustic embeddings, each candidate acoustic embedding comprising a vector representation of acoustic features; c) determining, by the data processing hardware, a similarity score for each candidate acoustic embedding in the set of candidate acoustic embeddings; d) removing, by the data processing hardware, based on the similarity score for each candidate acoustic embedding from the set of candidate acoustic embeddings, a subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings; and e) generating, by the data processing hardware, an aggregate acoustic embedding from the remaining candidate acoustic embeddings in the set of candidate acoustic embeddings after removing the subset of the candidate acoustic embeddings; and a) dividing, by the data processing hardware, the respective audio sample into a plurality of audio slices; b) based on the plurality of slices, generating, by the data processing hardware, a set of candidate acoustic embeddings, each candidate acoustic embedding comprising a vector representation of acoustic features; c) determining, by the data processing hardware, a similarity score for each candidate acoustic embedding in the set of candidate acoustic embeddings; d) removing, by the data processing hardware, based on the similarity score for each candidate acoustic embedding from the set of candidate acoustic embeddings, a subset of the candidate acoustic embeddings from the set of candidate acoustic embeddings; and e) generating, by the data processing hardware, an aggregate acoustic embedding from the remaining candidate acoustic embeddings in the set of candidate acoustic embeddings after removing the subset of the candidate acoustic embeddings; and identifying, by the data processing hardware, whether the first speaker and the second speaker are the same speaker or different speakers based on the aggregate acoustic embedding generated for each audio sample of the first audio sample and the second audio sample.