vo_direct3d: fix texture-memory sub-option, extend it

This sub-option was turned into a flag when the sub-option parser was
changed to the generic one (probably accidentally). Turn it into a
proper choice-option.

Also, adjust what the options do. Though none of this probably makes
much sense; the default should work, and if it doesn't, the GPU/driver
is probably beyond help.