No. Since GTM fire tags asynchronously, it can actually improve the website speed. However, there is one caveat here. If your container tag is very large (contains lot of tags, triggers and variables) then it can negatively affect the website speed. So if you are not using a particular tag, trigger or variable and you have no plan to use it in the future then remove it from the container.