Fix ConcurrentModificationException with cleaning owner lists