Magic Wand selections: variable compare modes, misc bug-fixes

I went back and forth on whether to add compare modes, and after some
testing, I think it's worth it.  This tool will always require some
fiddling, and anything that can improve accuracy is worth it.

This commit also fixes a number of miscellaneous magic wand bugs, and it
greatly improves handling of images (or layers) with alpha channels.