Improve error handling

Tile size factor restrictions have nothing to do with colorspace or
progressive versus interlaced, and invalid Y8 modes are now handled
correctly.