Decoding with per-speaker adaptation with diarization