Throw exception instead of assert on nested reliable packets

Fixes a denial of service vulnerability, issue #784