Just a small update
Been currently busy training another gemma 4 31B, did about 12 run's so far, playing with swa value, rope, turn masking, so far 2048 seems to be the sweet spot for training at 8k ctx.
should be releasing a new model in about 2-3 weeks, training on local 3090s is not very fast =p.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support