You may lose a tiny amount of draft, but it is insignificant and not worth sacrificing the efficiency gains you will achieve by insulating.