hqdn3d: add mutex on callback and delete callback on close

Now callbacks are thread-safe and don't block UI thread while
calculating coefs again.

Also remove callbacks on Close.