Fix an essential assert

This is part of the verification, so it needs to be checked
always.  Reported by @dstufft.