It is not the number of plugins that affect your site speed, it is how heavy your plugins are.
Even though Link Whisper does a lot of linking actions, it has very clean code and doesn’t affect your site speed.