Avoid asserting in glReadPixels for unsupported types, return GL_INVALID_ENUM instead

We should never assert in any GL code.