Short-context / padding-tail experiments with BuzzASR Danish
Hi BuzzASR team,
I've been experimenting with the Danish BuzzASR model for a local low-latency voice-control system. The experiments started with a simple question: For very short utterances, why spend so much computation processing Whisper's padded input?
Simply truncating the input does not work. What we found instead is that the padding region is not really "empty": through the encoder layers, self-attention appears to write utterance-dependent information into the padded positions. That led us to investigate whether this information could be represented much more compactly.
What we have found so far
At encoder layer 8, the speech-dependent correction in the padding tail appears to be strongly low-rank. On an initial set of 20 Danish voice commands, we found a fixed set of 6 spatial components that preserved the full-model decoder output on 20/20 utterances.
We froze those components and tested 15 newly recorded commands/sentences: same decoder output as the full model on 14/15, with EOS preserved on 15/15.
We then analyzed the temporal coefficient curves. These are also highly low-dimensional. Using a fixed rank-8 temporal basis together with the 6 spatial components reduces the utterance-specific oracle representation to 6 × 8 = 48 coefficients.
After freezing this representation, the temporal compression introduced no additional decoder errors on the 15 unseen recordings: TEMP-R8 and the 6-component representation produced identical decoder output on 15/15.
Actual short-context encoder test
We have now compared a genuine short encoder input of 1000 mel frames → 500 encoder positions with the first 500 positions of the normal full-context encoder state at layer 8, across the original 20 recordings.
Results: global cosine similarity mean 0.996874, worst utterance 0.996369, mean position-wise cosine 0.996625, and mean RMSE 0.042271. The results were remarkably consistent across all 20 utterances.
Important limitation
The 48 coefficients are currently oracle-derived from the true full encoder tail. So this is not yet a working 48-parameter replacement for the padded encoder computation, and I am not claiming a production speed-up yet.
The next experiment is to combine the genuine 500-position short-context layer-8 state with the oracle reconstructed tail, then run encoder layers 9–32 and the decoder. If that preserves the normal full-model decoder output, the remaining problem becomes predicting those 48 coefficients directly from the short-context state.
I'd be very interested to hear from the BuzzASR developers: Have you investigated the representations Whisper/BuzzASR creates in the padded region for short utterances, or experimented with shorter/dynamic encoder contexts?
If this is useful or interesting, I'm happy to share the experiment scripts, full results and methodology.
Best regards,
Lars Folmann
Denmark
Hi Lars,
Thanks for bringing this up! it's a genuinely interesting piece of analysis, and we haven't specifically studied what the encoder does with the padded region for short utterances, so this is new to us.
So from a viewpoint, Whisper's encoder is also fully bidirectional, which is part of why it's awkward for real-time/low-latency use in general. Technically, there's no structural reason the padding positions should stay "empty" imho, they're free to accumulate whatever the attention heads find useful from the real frames, and a narrow domain like short voice commands should plausibly compress that into a low-rank subspace. The layer-8 numbers you're reporting (cosine ~0.997, RMSE ~0.042, consistent across all 20 utterances) are a nice, tight result for that setting.
I also think your experiment setting right now with predicting 48 coefficients from a 500-position encoder state is a much smaller and more tractable problem than reconstructing the tail directly, therefore, the direction you're heading in next (layers 9–32 + decoder on the spliced state) seems like the right next check.
A few things we'd be curious about before this generalizes beyond the voice-control setting:
- Does the same 6-spatial/8-temporal basis hold on longer, more varied utterances (e.g. our FLEURS/Common Voice eval sets), or is the low-rank structure specific to the short, constrained-vocabulary commands you tested on?
- Is the basis language-specific, or does it transfer across the other BuzzASR language models, given they all share Whisper's architecture and only differ in fine-tuning?
- How does the predicted (non-oracle) version hold up once you have it same 14/15 or 15/15 exact-match rate, or does it degrade?
I'd be glad to take a look at your scripts and methodology if you're up for sharing them happy to help verify or extend this on our end if it holds up on a broader eval set.
Thanks again for digging into this and writing it up so clearly.
Best,
Shivam
Hi Shivam,
Thanks — this is very helpful, and particularly interesting to hear that the padded-region behaviour hasn't been specifically studied by the team.
I've moved the experiment a little further since writing the original post.
Using a genuine 1000-mel / 500-position input through layer 8, then splicing in the 6-spatial / 8-temporal oracle-derived tail reconstruction before layers 9–32, I get 19/20 exact token matches against the full encoder, with 20/20 EOS.
I also tested the already-selected spatial basis/components and temporal basis/rank on 15 new recordings/sentences that were not used when selecting them. The per-utterance amplitudes were still oracle-derived in this experiment, so this was a test of the representation rather than a deployable predictor. On those recordings, the temporal rank-8 reconstruction agreed with the sparse-6 reconstruction on 15/15, and both matched the full model on 14/15.
I then tried removing the utterance-specific amplitudes entirely and using a single fixed mean set of amplitudes derived from the original 20 examples. That produced 18/20 exact matches on the original set, but only 10/15 on the new recordings (with EOS preserved in all 15). That suggests that a surprisingly large part of the padded-tail structure can be represented by a fixed component, but some utterance-specific correction still matters.
I also tried a simple PCA/ridge approach to predict all 48 amplitudes from the genuine short-context layer-8 state. With only 20 development examples it did not generalize in leave-one-out testing — the best normalized MSE was about 1.12 relative to the target-mean baseline — so I don't think that result justifies a predictor yet.
I'm now testing whether the problem can instead be decomposed into a fixed mean tail plus a much smaller dynamic residual. The idea is to determine at decoder level how many utterance-specific degrees of freedom are actually necessary before trying to learn them from the short-context state.
Your suggestions about FLEURS/Common Voice and cross-language transfer make a lot of sense. I deliberately started with the constrained voice-command domain because it gives a clean experimental setting, but broader evaluation would be the obvious next step if the non-oracle version holds up.
I'm absolutely happy to share the scripts, intermediate results and methodology. It would be great if you or the team wanted to reproduce or extend this on a larger evaluation set.
Best,
Lars Folmann
One small addition, just to answer your three questions explicitly:
- FLEURS / Common Voice / more varied speech: Not tested yet. The additional 15 recordings I've used are independent of the original 20 used for basis selection, but they are still short, voice-control-style utterances. So I would currently only claim the low-rank result for this constrained short-command setting. Testing FLEURS/Common Voice would be the next proper generalization test.
- Cross-language transfer: Not tested yet either. Everything so far has been done with BuzzASR Danish. It would be particularly interesting to distinguish between the same basis transferring across languages and merely the same low dimensionality appearing with a different basis.
- Predicted/non-oracle version: I have started testing this. A simple PCA/ridge predictor of all 48 amplitudes from the genuine short-context layer-8 state did not generalize in leave-one-out testing on the 20-example development set (best normalized MSE ≈ 1.12 relative to the target-mean baseline), so I don't yet have a successful non-oracle predictor.
Interestingly, a single fixed mean derived from the original 20 examples still gives 18/20 exact matches there, but only 10/15 on the new recordings. That is what led me to the current experiment: treating the tail as a large fixed component plus a much smaller utterance-specific residual, and measuring at decoder level how many dynamic degrees of freedom are actually required.
So the short version is: (1) not yet, (2) not yet, (3) promising representation, but no successful learned predictor yet.
Lars