remove AggressiveInline attribute

Can't inline virtual methods, so remove where pointless
Replace BitConverter

Replicate BitConverter functionality in our own code for performance
boost since it was mainly doing checks that we had already done.