Don't add bufferviews until the config is synced

This breaks layout restoring stuff, but is required to determine the
type of view.