The reason Vegeta could'nt go SSJ3 is because Goku is a Honour Gene. Honour Gene's can go SSJ3. However, Vegeta wasn't a Honour, instead he is a Pride Gene. Pride Gene's can only go SSJ, ASSJ, USSJ, SSJ2, and SSJ4.