increase mse reporting precision with disabled floyd dithering