As long as it's not a site-wide removal of all your links, losing one link source won't tank your rankings. Keep aggregating to offset any lost links.