In order that you see a rainbow, it must be raining in one part of the sky, the sun shining in the opposite part, and you must turn your back to the sun. The drops bend the light to scatter it.
Light is odd in some ways.
You have experienced the same thing, roller blading over a smoother surface with one skate, while the other skate is still on a rough path. Light bends in a similar way both when it enters and leaves the raindrop. Light enters the drop at the top and bends as it enters the water. But there's something else going on. Some colors slow down more than others and so they bend more than the others.