There are actually several reasons to consider hand-vectorizing an application. The performance gains possible are certainly one reason. Additionally, accuracy and reliability gains are possible. Some algorithms require careful analysis by a mathematician in order to obtain satisfactory results. A subroutine call where the name of the subroutine describes the function, with the arguments clearly specified is easier to understand than trying to decode what a loop is doing.
Modern compilers do a very good job of scalar optimization.
Some compilers even do some advanced loop optimizations, like loop unrolling. Unfortunately, these scalar compilers don't have any real understanding of the memory architecture, and what's involved in optimizing for memory bandwidth. Take a simple function like a vector multiply as an example. Unfortunately, if the vectors are in memory, the scalar compiler will actually generate code with the worst possible utilization of memory bandwidth. Knowledge of DRAM behavior is required to understand this.