Avoid asserting in glReadPixels for unsupported formats, return GL_INVALID_ENUM instead

We should never assert in any GL code.