SVG will be built-in. Currently, a plug-in is required to view SVG in a browser, but this is likely to change shortly. Once SVG is supported natively by the major browsers, not only will it not be necessary to download and install a plug-in, you ll never have to worry about upgrade problems when changing to a new version of a browser. The problem with this as it relates to a GIS image server is that, as you zoom in, you expect to see greater detail. In order for that detail to be displayed, it would have to have been in the image sent to the client.
More detail means longer transmission and rendering times.
Panning presents a similar problem in that the image would need to contain information you would not initially be able to see. Carrying this to its logical conclusion, providing full panning and zooming on the client would entail downloading the complete feature sets from the server. Not a practical solution, in most cases. The approach taken with the MRF SVG Server is to send only the amount of detail needed to render the image at the scale selected panning and zooming is disabled. Several techniques are used to produce images that are orders of magnitude smaller than they would be, if sent out at full resolution.