Fix all attributes removal on when no affiliations

When a user lost all affiliations it thought it would try (and fail)
to remove all of their attributes since the sum of all of their affilations'
attributes was nothing. Now we're ensuring we never remove the core affiliations.