Like many office workers who work in high rise buildings, I face a daily ride in the tiny little box at least twice and maybe more. Elevators are so tiny and confining. People are uncomfortable to start with because the close proximity to other people violates our personal space. Usually people face frontward and avoid eye contact.
Fear of height is said to be instinctive, so that is probably a large part of it.
I've ridden in elevators at the Sears Tower in Chicago that whiz up 100 floors to the top so fast it makes your ears pop. And I LOVE roller coasters, which are surely more dangerous than elevators. I work in a very old building. They have given it a face lift and it has been remodeled, including the elevators. Everything looks like new.