MAINT: Only do refcounting in take when necessary