Unless you’re doing 10,000+ redirects or it’s a major link such as in the navigation, 301 redirects shouldn’t affect site speed.