prostreamer-E2B-pretrain

Audio-visual streaming model pretrained from google/gemma-4-E2B-it (EgoForesee StreamAV, run e2b_seg3s1_ep2). Per 1 s step it predicts wearer speaking (speak head), the wearer transcript and the current wearer action.

Training

• full fine-tune of the LLM decoder (token embedding table frozen), vision / audio projections trained, vision / audio encoders frozen
• new parameters: speak head (layers 10,20,-1), special-token deltas (lr 0.001)
• data: all datasets, av_stream v1 labels; config act_seg + seg_share 1 + tx_win 3
• 17942 steps = 2 epochs, global batch 16 (4 GPU x accum 4), lr 2e-05, seed 0
• final train losses: speak 0.2419, transcript 0.1563, action 0.6729

Files

• trainable_step17942.pt: state dict of every trained parameter (LLM decoder in bf16, projections + new parameters); load on top of the base model with the EgoForesee streamav code (stream_eval.py --ckpt)
• args.json: training arguments; train_log.jsonl: training log
• eval/: streaming val evaluation of the final checkpoint (w32k16, generated / GT-frame)

Val (all datasets, final checkpoint, generated)

wearer AP	wearer best F1	onset recall@1s	ER en	ER zh	action ROUGE-L	action verb	RTF
0.7571	0.6937	0.656	0.5202	0.5503	0.414	0.3604	0.567