will you train a smaller model, lets say a 4b qwen3.5 model, but with a wayy high train context length? because with 8k, a 30B model is pretty useless ngl, but its a fun experiment. should have ran the epxeriment on a small model though. anyways, nice work!
That’s a fair point. At the moment, I’m limited to Colab sessions because I don’t yet have a GPU of my own, so I have to make trade-offs in what I can realistically experiment with. This was my first experiment with a model larger than 8B, and part of the goal was simply to test whether I could successfully train and deploy something at this scale.
I’ll be repeating the same pipeline on a smaller model next, which should let me increase the training context length substantially and explore long-context behaviour more seriously.
it would be nicer to have more compute, but we u need to realize you just casually trained a 30B model (MoE or not), you had to load the weights (even at 4 bits, you still had to load the weights)
so try making a 4-9B model with QLoRA at around r20-a40 anyways, great model – love it!
