add typecasting to RTEST macro

This patch fixes a compilation bug with native c extensions:

error: invalid operands to binary & (have ‘VALUE (*)(void)’ and ‘long unsigned int’)