remove overzealous assert

Do you guys mind if we remove this assert?

I can verify that everything runs fine without it... and it's causing us some issues in some cases.