Every plugin adds to your load time, it just depends how much. Using lightweight plugins is more important than reducing the number of plugins, but both are important.