Disable direct pointer access

The check for tagged pointers is broken on ARM64

Better safe than sorry!