libhb: Improve nlmeans edgeboost prefilter.

Fixes a typo (1/27 instead of 1/127) that caused edgeboost to match on nearly everything, and tweaks this number slightly.
Adds a post-processing step to handle marked pixels having too few marked neighboring pixels as false positives.



git-svn-id: svn://svn.handbrake.fr/HandBrake/trunk@6224 b64f7644-9d1e-0410-96f1-a4d463321fa5