Yeah, we may need the initial submissions to train with.  That's not the only option -- authors could potentially pull problems from other runs of the course (maybe run peer assessment only first).  But there's a good chance we'll need to allow submissions initially and train the model after the problem has been open for a while.