Some, such as image map will have a slight performance hit, but probably not noticeable for the majority of sites. Running other applications on the server machine can have an impact. Processes like database searches, or Java script, which dynamically generate web pages, will tend to be served more slowly than static pages. Faster CPU processor speeds can help improve performance for these processes. RAM based databases will be faster than disk based.
Many sites run their databases on a separate machine from their primary web server.
Ultimately, the limit on serving files is going to be speed of transmission of the file in the slowest part of the path from server to browser. People on ethernet have a very high speed connection to your site. You'll be able to have a lot more hits per day if everyone is connecting by ethernet, than if they are all connecting on dial-up ISP accounts using 28.8 modems. The majority of people browsing from home have 28.8 modems, and many are still using 14.4. Increasing bandwidth on the server side allows more simultaneous connections.