Well, I have thought about that a lot and a couple years ago I started on a project to find out for myself. It has progressed to a new skydiving event. Anyone who thinks they can go pretty fast is welcome to come to Eloy at the end of the month and show their stuff. December 27 - 29 is the second annual Arizona Speed Trials. You get three jumps from 13,000 to go as fast as you can.
Fastest time wins.
I am told a show called The Extremists will be out to cover the event. The theory of speed is simple. Since gravity is your only source of energy you can't add thrust to go faster. You can only reduce drag. Drag is a function of your shape, surface friction, and the density of the medium.