Clip filtermask to signed 8bit.

Fixes another few mismatches on vp90-2-00-quantizer-55.webm.