SpMV is a key linear algebra algorithm and has been widely used in many important application domains. As a result, numerous attempts have been made to optimize SpMV on GPUs to leverage their massive computational throughput. Although the previous works have shown impressive progress, load imbalance and high memory bandwidth remain the critical performance bottlenecks for SpMV. In this paper, we present our novel solutions to these problems. First, we devise a new SpMV format, called blocked compressed common coordinate (BCCOO), which uses bit flags to store the row indices in a blocked common coordinate (COO) format so as to alleviate the bandwidth problem. We further improve this format by partitioning the matrix into vertical slices to enhance the cache hit rates when accessing the vector to be multiplied. Second, we revisit the segmented scan approach for SpMV to address the load imbalance problem. We propose a matrix-based segmented scan with highly efficient shared memory usage and further improve it by eliminating global synchronization. Then, we introduce an auto-tuning framework to choose optimization parameters based on the characteristics of input sparse matrices. Our experimental results on GTX480 GPUs and GTX680 GPUs show that our proposed framework achieves significant performance improvement over the vendor tuned library CUSPARSE V5.0 (up to 150% and 43% on average on GTX480, up to 228% and 66% on average on GTX680) and some most recently proposed schemes (e.g., up to 170% and 29% on average over clSpMV on GTX480, up to 190% and 56% on average over clSpMV on GTX680).