Xinnian/shared-ttt-giga-fixed-multiqa

Shared TTT — GigaSpeech Fixed Multi-QA

Shared audio/text TTT memory and reader LoRA for SALMONN-omni, trained on GigaSpeech multi-QA with causal answer-only supervision.

Model setup

Component	Configuration
Base multimodal model	llama31_er0_c30-48k, Llama 3.1 with 131,072 native context positions
Base loading	checkpoint_loader: native_context
Memory architecture	Shared audio/text TTT; fixed_write commit policy
Memory / RAW / HANDOFF budgets	2,048 / 1,024 / 3,072 slots
TTT update / commit intervals	256 / 512 write slots
TTT eta base	0.05
RMS residual alpha	0.5
Long-span reconstruction	1,024 slots; weight 1.0; modality-balanced loss
Reader memory LoRA	Rank 128, alpha 128, dropout 0.05; targets q_proj and v_proj
Trainable components	Shared TTT, prefix packer and memory reader LoRA
Frozen components	Speech encoder, speech adapter and original LLM parameters
History and query features	Encoded online; echo ratio 1.0
Memory state	Reset independently for each QA example

Training setup

Item	Configuration
Dataset version	GigaSpeech multiqa_fixed_r20261019
Training data	3,861 histories; 22,378 QA examples
Held-out data	500 histories; one QA example per history
Objective	Causal answer-only loss; TTS loss weight 0
Training duration	5 epochs; 3,530 optimizer steps
Effective batch size	32: 8 GPUs × microbatch 1 × gradient accumulation 4
Outer learning rate	5e-6
Schedule	Cosine decay; warmup ratio 0.03
Weight decay	0.01
Gradient clipping	1.0
Precision	BF16; BF16 fast weights
Seed	20260904
Export	Epoch 5 / step 3530; compact TTT, prefix packer and reader LoRA weights

The frozen base-model weights are separate dependencies. training_config.yaml contains the expanded configuration with local path placeholders.