Use a pointer-sized type to store refcounts internally