Remove collection from active list on delete, if it was there