semantic: Replace an assert with a panic.

This needs to happen everywhere, but I thought
I would do them as particular assers annoy me.