Yes, only layout shifts in the content above the fold affect your CLS score. However, optimizing all elements in a web page ensures better overall performance and website speed, which in turn will affect the traffic and ranking.