Well actually both antennas do have about the same gain. Gain is a relative thing, in the above paragraph, I'm comparing free space gain against above ground gain, two different things entirely. If you were to mount a dipole vertically 1 foot above ground too, you'd find that that its gain is 1.90 dB lower than the free space gain of the same antenna. The second reason you may want to use the half wave vertical is to obtain a lower angle of radiation. A ground plane mounted at 1 foot above ground has a radiation peak at 25 degrees, the half wave antenna radiation peak angle drops down to 20 degrees, making it a better dx antenna.
All antenna lengths depend on various factors.
This page allows you to calculate the length for a half wave antenna. If you've experimented with half wave antennas before and know of a better formula for your QTH, feel free to change the formula to suit. This formula is for a wire antenna. Of course if you construct your antenna out of tubing, the total length of the antenna will be shorter.