rawdenoise: don't take sqrt of a negative number

(and expect real results)
(cherry picked from commit 8c67cb06d03edb1a67b6a2b6ab83c2e43531909c)