Avoid asserting in glReadPixels for unsupported types, return GL_INVALID_ENUM instead

We should never assert in any GL code.
`{` should go on the same line with the `if`.