Revert "Adding some assertions to cope with weird null pointers I'm seeing"

Could these have caused a 10% slow-down?!

This reverts commit 10702732de72e3cede1f298bbcf9c8c9b31af1da.