Fix assert conditions

Normally we don't turn on the asserts but for sanity checking, we should and these fixes can help us to pass those checks.