Magic Wand selections: fix boundary checks when feathering is active

Antialiasing still hasn't been implemented, but this at least provides
correct values when feathering radius > 0.