There are many reasons why, but here are the main ones:
1) Server: your hosting is the most crucial factor that influences how fast things get to your users. We have sites on high-frequency servers that don't use a caching plugin at all, only an image compressor and a nice server-level caching and CDN rules;
2) Website structure: that has to do with how your website was built, which foundation was used to create your pages, and what current tools or plugins are possibly draining your resources (server);
3) Large images: this goes without saying, but a site with properly compressed images is a must! Also, some websites (with high-traffics) require images to be delivered by a third-party service for smaller devices;
4) Scripts: there are 2 basic types, cookies, and applications. Cookies are all the marketing tools you need in order to attract and delight your customers. Everything should be managed in Google Tag Manager, and that also needs optimization in order to not block the rendering of your pages. Applications scripts are tools inside your website that should be tackled on a case-by-case scenario, in some cases removed and/or substituted for better ones.
5) CDN: again, another must-have. There are free and paid CDN (Content Delivery Network) services. We will assess your website and advise you if you should invest further in it. All packages come with a basic FREE CDN setup via Cloudflare;
6) Fonts: if you use too many font types, you are doomed to have a slow site. Fonts require server requests for each font-weight. We need to make some decisions and go with the ones that suit better your needs. Sometimes, avoiding Google Fonts and hosting only necessary ones yourself saves you a lot of resources.