Remove unnecessary assert

I do not remember why these exist here, but going to remove to fix an
issue