Remove non-essential asserts and some other optimizations