Fix numerical issues when grouping faces by normal

This was causing a ton of bugs down the road including:

* draw plane offset from surface
* highlight not selecting entire face