And you lose a user for good. Rarely do they come back to try again. You lose them and they won't come back, and if they do, they won't have a very good attitude about it. Navigation and structure is lost. If the eye is continually drawn to the wrong area and a user has to focus on your navigation, then you have lost the navigational structure of your site and it will be deemed hard to use.
Site takes too long to load.
This is a classic. If your site takes too long to download into the browser, the user will not wait and will not come back either. Humans are impatient beings for the most part, and as Web designers, we just have to live with that. Site is too advanced for the browser and therefore doesn't work or even load at all. This is not such a large issue when dealing with intranets.