Well, this is because we run multiple instances, so we don't have to cancel.  We just kill the sbt instance.

It turns out cancelling is actually flaky (cause of failure).  I'm not 100% sure why, but I'd have to dig into how play is doing reloading and such.