Simplify the ref counting