Cleaned up access group garbage tracking logic