Allow LUACMSGPACK_MAX_NESTING to be set by CPP
Add tests for more edge cases

The 0xFFFFFFFF test can potentially fail on 32 bit systems if
our internal conversions are casting to wrong Lua types.
lua_Integer != lua_Number