compute per-pixel error map with selectable transfer curve