May I make a suggestion, to lessen the expense of constructing an ImageSpec every time you need to_native_rectangle?

In formatspec.cpp, there's a function set_default_quantize() that's static in that file.  I would suggest that you (a) make it a public, but static, member of ImageSpec, rather than a file-local-static; (b) call it to get the quantization defaults, rather than constructing an ImageSpec; (c) doing it only conditionally if it fails the first clause:

    if (format == m_spec.format) {
        int black, white, min, max;
        ImageSpec::set_default_quantize (format, black, white, min, max);
        if (m_spec.quant_black == black && ...)
            return data;
    }