libhb: Improve nlmeans edgeboost prefilter.

Fixes a typo (1/27 instead of 1/127) that caused edgeboost to match on nearly everything, and tweaks this number slightly.
Adds a post-processing step to handle marked pixels having too few marked neighboring pixels as false positives.