[vesafb] Select an optimal mode, rather than the first acceptable mode

There is no requirement for VBE modes to be listed in increasing order
of resolution.  With the present logic, this can cause e.g. a 1024x768
mode to be selected if the user asks for 640x480, if the 1024x768 mode
is earlier in the mode list.

Define a scoring system for modes as

  score = ( width * height - bpp )

and choose the mode with the lowest score among all acceptable modes.
This should prefer to choose the mode closest to the requested
resolution, with a slight preference for higher colour depths.

Reported-by: Robin Smidsrød <robin@smidsrod.no>
Signed-off-by: Michael Brown <mcb30@ipxe.org>