glamor: Choose max fbo size by texture + viewport instead of renderbuffer

The max size of renderbuffers and texture often match by accident, but as we always use textures, we should check for the right flag.
Also check for viewport size as this may be lower and we want to render to almost every pixmap.