Repository: grafana/grafana
Pull request #103404: Dashboards: Add k8s fallback client
Author: login: toddtreece
Labels: name: area/backend
name: no-backport
description: Skip backport of PR
name: no-changelog
description: Skip including change in changelog/release notes
Created: 2025-04-03T18:20:05Z | Merged: 2025-04-06T20:38:07Z

<!--

Thank you for sending a pull request! Here are some tips:

1. If this is your first time, please read our contribution guide at https://github.com/grafana/grafana/blob/main/CONTRIBUTING.md

2. Ensure you include and run the appropriate tests as part of your Pull Request.

3. In a new feature or configuration option, an update to the documentation is necessary. Everything related to the documentation is under the docs folder in the root of the repository.

4. If the Pull Request is a work in progress, make use of GitHub's "Draft PR" feature and mark it as such.

5. If you can not merge your Pull Request due to a merge conflict, Rebase it. This gets it in sync with the main branch.

6. Name your PR as "<FeatureArea>: Describe your change", e.g. Alerting: Prevent race condition. If it's a fix or feature relevant for the changelog describe the user impact in the title. The PR title is used to auto-generate the changelog for issues marked with the "add to changelog" label.

7. If your PR content should be added to the What's New document for the next major or minor release, add the **add to what's new** label to your PR. Note that you should add this label to the main PR that introduces the feature; do not add this label to smaller PRs for the feature.

-->

**What is this feature?**

Support falling back to `storedVersion` when conversion fails.

```json
  "spec": null,
  "status": {
    "conversion": {
      "failed": true,
      "storedVersion": "v2alpha1",
      "error": "backend conversion not yet implemented"
    }
  }
```

**Why do we need this feature?**

The backend dashboard service needs to support multiple API versions.

**Who is this feature for?**

[Add information on what kind of user the feature is for.]

**Which issue(s) does this PR fix?**:

<!--

- Automatically closes linked issue when the Pull Request is merged.

Usage: "Fixes #<issue number>", or "Fixes (paste link of issue)"

-->

Fixes #

**Special notes for your reviewer:**

Please check that:
- [ ] It works as expected from a user's perspective.
- [ ] If this is a pre-GA feature, it is behind a feature toggle.
- [ ] The docs are updated, and if this is a [notable improvement](https://grafana.com/docs/writers-toolkit/contribute/release-notes/#how-to-determine-if-content-belongs-in-whats-new), it's added to our [What's New](https://grafana.com/docs/writers-toolkit/contribute/release-notes/) doc.



Diff:
diff --git a/pkg/services/dashboards/service/client/client.go b/pkg/services/dashboards/service/client/client.go
new file mode 100644
index 0000000000000..bea2619884416
--- /dev/null
+++ b/pkg/services/dashboards/service/client/client.go
@@ -0,0 +1,157 @@
+package client
+
+import (
+	"context"
+	"fmt"
+	"sync"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+type K8sClientFactory func(ctx context.Context, version string) client.K8sHandler
+
+type K8sClientWithFallback struct {
+	client.K8sHandler
+
+	newClientFunc K8sClientFactory
+	metrics       *k8sClientMetrics
+	log           log.Logger
+}
+
+func NewK8sClientWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	reg prometheus.Registerer,
+) *K8sClientWithFallback {
+	newClientFunc := newK8sClientFactory(cfg, restConfigProvider, dashboardStore, userService, resourceClient, sorter, dual)
+	return &K8sClientWithFallback{
+		K8sHandler:    newClientFunc(context.Background(), dashboardv1alpha1.VERSION),
+		newClientFunc: newClientFunc,
+		metrics:       newK8sClientMetrics(reg),
+		log:           log.New("dashboards-k8s-client"),
+	}
+}
+
+func (h *K8sClientWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	spanCtx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(spanCtx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		return result, nil
+	}
+
+	h.log.Info("falling back to stored version", "name", name, "storedVersion", storedVersion, "conversionErr", conversionErr)
+	h.metrics.fallbackCounter.WithLabelValues(storedVersion).Inc()
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	return h.newClientFunc(spanCtx, storedVersion).Get(spanCtx, name, orgID, options, subresources...)
+}
+
+func getConversionStatus(obj *unstructured.Unstructured) (failed bool, storedVersion string, conversionErr string) {
+	status, found, _ := unstructured.NestedMap(obj.Object, "status")
+	if !found {
+		return false, "", ""
+	}
+	conversionStatus, found, _ := unstructured.NestedMap(status, "conversion")
+	if !found {
+		return false, "", ""
+	}
+	failed, _, _ = unstructured.NestedBool(conversionStatus, "failed")
+	storedVersion, _, _ = unstructured.NestedString(conversionStatus, "storedVersion")
+	conversionErr, _, _ = unstructured.NestedString(conversionStatus, "error")
+	return failed, storedVersion, conversionErr
+}
+
+func newK8sClientFactory(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+) K8sClientFactory {
+	clientCache := make(map[string]client.K8sHandler)
+	cacheMutex := &sync.RWMutex{}
+	return func(ctx context.Context, version string) client.K8sHandler {
+		_, span := tracing.Start(ctx, "k8sClientFactory.GetClient",
+			attribute.String("group", dashboardv1alpha1.GROUP),
+			attribute.String("version", version),
+			attribute.String("resource", "dashboards"),
+		)
+		defer span.End()
+
+		cacheMutex.RLock()
+		cachedClient, exists := clientCache[version]
+		cacheMutex.RUnlock()
+
+		if exists {
+			span.AddEvent("Client found in cache")
+			return cachedClient
+		}
+
+		cacheMutex.Lock()
+		defer cacheMutex.Unlock()
+
+		// check again in case another goroutine created in between locks
+		cachedClient, exists = clientCache[version]
+		if exists {
+			span.AddEvent("Client found in cache after lock")
+			return cachedClient
+		}
+
+		gvr := schema.GroupVersionResource{
+			Group:    dashboardv1alpha1.GROUP,
+			Version:  version,
+			Resource: "dashboards",
+		}
+
+		span.AddEvent("Creating new client")
+		newClient := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), gvr, restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+		clientCache[version] = newClient
+
+		return newClient
+	}
+}
diff --git a/pkg/services/dashboards/service/client/client_test.go b/pkg/services/dashboards/service/client/client_test.go
new file mode 100644
index 0000000000000..561f9da150a0a
--- /dev/null
+++ b/pkg/services/dashboards/service/client/client_test.go
@@ -0,0 +1,357 @@
+package client
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/stretchr/testify/mock"
+	"github.com/stretchr/testify/require"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+)
+
+type testSetup struct {
+	handler            *K8sClientWithFallback
+	mockClientV1Alpha1 *client.MockK8sHandler
+	mockClientV2Alpha1 *client.MockK8sHandler
+	mockMetrics        *k8sClientMetrics
+	mockFactoryCalls   map[string]int
+	t                  *testing.T
+}
+
+func setupTest(t *testing.T) *testSetup {
+	mockClientV1Alpha1 := &client.MockK8sHandler{}
+	mockClientV2Alpha1 := &client.MockK8sHandler{}
+	mockMetrics := newK8sClientMetrics(prometheus.NewRegistry())
+	mockFactoryCalls := make(map[string]int)
+
+	handler := &K8sClientWithFallback{
+		K8sHandler: mockClientV1Alpha1,
+		newClientFunc: func(ctx context.Context, version string) client.K8sHandler {
+			mockFactoryCalls[version]++
+			if version == "v2alpha1" {
+				return mockClientV2Alpha1
+			}
+			if version == dashboardv1alpha1.VERSION {
+				return mockClientV1Alpha1
+			}
+			t.Fatalf("Unexpected call to newClientFunc with version %s", version)
+			return nil
+		},
+		log:     log.New("test"),
+		metrics: mockMetrics,
+	}
+
+	return &testSetup{
+		handler:            handler,
+		mockClientV1Alpha1: mockClientV1Alpha1,
+		mockClientV2Alpha1: mockClientV2Alpha1,
+		mockMetrics:        mockMetrics,
+		mockFactoryCalls:   mockFactoryCalls,
+		t:                  t,
+	}
+}
+
+func TestK8sHandlerWithFallback_Get(t *testing.T) {
+	t.Run("Get without fallback", func(t *testing.T) {
+		setup := setupTest(t)
+
+		ctx := context.Background()
+		name := "test-dashboard"
+		orgID := int64(1)
+		options := metav1.GetOptions{}
+
+		expectedResult := &unstructured.Unstructured{
+			Object: map[string]interface{}{
+				"metadata": map[string]interface{}{
+					"name": name,
+				},
+				"status": map[string]interface{}{
+					"someOtherStatus": "ok",
+				},
+			},
+		}
+
+		setup.mockClientV1Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(expectedResult, nil).Once()
+
+		result, err := setup.handler.Get(ctx, name, orgID, options)
+		require.NoError(t, err)
+		require.Equal(t, expectedResult, result)
+		require.Equal(t, 0, len(setup.mockFactoryCalls), "Factory should not be called for non-fallback case")
+
+		setup.mockClientV1Alpha1.AssertExpectations(t)
+		setup.mockClientV2Alpha1.AssertExpectations(t)
+	})
+
+	t.Run("Get with fallback due to conversion error", func(t *testing.T) {
+		setup := setupTest(t)
+
+		ctx := context.Background()
+		name := "test-dashboard-fallback"
+		orgID := int64(2)
+		options := metav1.GetOptions{ResourceVersion: "123"}
+		storedVersion := "v2alpha1"
+		conversionErr := "failed to convert"
+
+		v1alpha1Result := &unstructured.Unstructured{
+			Object: map[string]interface{}{
+				"metadata": map[string]interface{}{
+					"name": name,
+				},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed":        true,
+						"storedVersion": storedVersion,
+						"error":         conversionErr,
+					},
+				},
+			},
+		}
+
+		expectedResultFallback := &unstructured.Unstructured{
+			Object: map[string]interface{}{
+				"apiVersion": "dashboard/v2alpha1",
+				"kind":       "Dashboard",
+				"metadata": map[string]interface{}{
+					"name": name,
+				},
+			},
+		}
+
+		setup.mockClientV1Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(v1alpha1Result, nil).Once()
+		setup.mockClientV2Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(expectedResultFallback, nil).Once()
+
+		result, err := setup.handler.Get(ctx, name, orgID, options)
+		require.NoError(t, err)
+		require.Equal(t, expectedResultFallback, result)
+		require.Equal(t, 1, setup.mockFactoryCalls["v2alpha1"], "Factory should be called once with v2alpha1")
+
+		setup.mockClientV1Alpha1.AssertExpectations(t)
+		setup.mockClientV2Alpha1.AssertExpectations(t)
+	})
+
+	t.Run("Get initial error", func(t *testing.T) {
+		setup := setupTest(t)
+
+		ctx := context.Background()
+		name := "test-dashboard-error"
+		orgID := int64(3)
+		options := metav1.GetOptions{}
+		expectedErr := errors.New("initial get failed")
+
+		setup.mockClientV1Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(nil, expectedErr).Once()
+
+		_, err := setup.handler.Get(ctx, name, orgID, options)
+		require.Error(t, err)
+		require.Equal(t, expectedErr, err)
+		require.Equal(t, 0, len(setup.mockFactoryCalls), "Factory should not be called for error case")
+
+		setup.mockClientV1Alpha1.AssertExpectations(t)
+		setup.mockClientV2Alpha1.AssertExpectations(t)
+	})
+
+	t.Run("Get with fallback fails", func(t *testing.T) {
+		setup := setupTest(t)
+
+		ctx := context.Background()
+		name := "test-dashboard-fallback-error"
+		orgID := int64(4)
+		options := metav1.GetOptions{}
+		storedVersion := "v2alpha1"
+		conversionErr := "failed to convert again"
+		fallbackErr := errors.New("fallback get failed")
+
+		v1alpha1Result := &unstructured.Unstructured{
+			Object: map[string]interface{}{
+				"metadata": map[string]interface{}{
+					"name": name,
+				},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed":        true,
+						"storedVersion": storedVersion,
+						"error":         conversionErr,
+					},
+				},
+			},
+		}
+
+		setup.mockClientV1Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(v1alpha1Result, nil).Once()
+		setup.mockClientV2Alpha1.On("Get", mock.Anything, name, orgID, options, mock.Anything).Return(nil, fallbackErr).Once()
+
+		_, err := setup.handler.Get(ctx, name, orgID, options)
+		require.Error(t, err)
+		require.Equal(t, fallbackErr, err)
+		require.Equal(t, 1, setup.mockFactoryCalls["v2alpha1"], "Factory should be called once with v2alpha1")
+
+		setup.mockClientV1Alpha1.AssertExpectations(t)
+		setup.mockClientV2Alpha1.AssertExpectations(t)
+	})
+}
+
+func TestGetConversionStatus(t *testing.T) {
+	tests := []struct {
+		name                  string
+		obj                   *unstructured.Unstructured
+		expectedFailed        bool
+		expectedStoredVersion string
+		expectedError         string
+	}{
+		{
+			name: "No status field",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+		{
+			name: "Status field, but no conversion field",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status":   map[string]interface{}{"someOtherStatus": "ok"},
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+		{
+			name: "Conversion field, failed=true, with storedVersion and error",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed":        true,
+						"storedVersion": "v2alpha1",
+						"error":         "conversion failed",
+					},
+				},
+			}},
+			expectedFailed:        true,
+			expectedStoredVersion: "v2alpha1",
+			expectedError:         "conversion failed",
+		},
+		{
+			name: "Conversion field, failed=false",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed":        false,
+						"storedVersion": "v1alpha1",
+						"error":         "",
+					},
+				},
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "v1alpha1",
+			expectedError:         "",
+		},
+		{
+			name: "Conversion field, missing failed (defaults to false)",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+
+						"storedVersion": "v1alpha1",
+						"error":         "",
+					},
+				},
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "v1alpha1",
+			expectedError:         "",
+		},
+		{
+			name: "Conversion field, failed=true, missing storedVersion",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed": true,
+
+						"error": "conversion failed",
+					},
+				},
+			}},
+			expectedFailed:        true,
+			expectedStoredVersion: "",
+			expectedError:         "conversion failed",
+		},
+		{
+			name: "Conversion field, failed=true, missing error",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": map[string]interface{}{
+						"failed":        true,
+						"storedVersion": "v2alpha1",
+					},
+				},
+			}},
+			expectedFailed:        true,
+			expectedStoredVersion: "v2alpha1",
+			expectedError:         "",
+		},
+		{
+			name:                  "Empty object",
+			obj:                   &unstructured.Unstructured{Object: map[string]interface{}{}},
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+		{
+			name:                  "Nil object",
+			obj:                   nil,
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+		{
+			name: "Status not a map",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status":   "not a map",
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+		{
+			name: "Conversion not a map",
+			obj: &unstructured.Unstructured{Object: map[string]interface{}{
+				"metadata": map[string]interface{}{"name": "test"},
+				"status": map[string]interface{}{
+					"conversion": "not a map",
+				},
+			}},
+			expectedFailed:        false,
+			expectedStoredVersion: "",
+			expectedError:         "",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var input *unstructured.Unstructured
+			if tt.obj != nil {
+				input = tt.obj.DeepCopy()
+			} else {
+				input = &unstructured.Unstructured{Object: map[string]interface{}{}}
+			}
+
+			failed, storedVersion, conversionErr := getConversionStatus(input)
+			require.Equal(t, tt.expectedFailed, failed, "failed mismatch")
+			require.Equal(t, tt.expectedStoredVersion, storedVersion, "storedVersion mismatch")
+			require.Equal(t, tt.expectedError, conversionErr, "conversionErr mismatch")
+		})
+	}
+}
diff --git a/pkg/services/dashboards/service/client/metrics.go b/pkg/services/dashboards/service/client/metrics.go
new file mode 100644
index 0000000000000..b3e9336b54d77
--- /dev/null
+++ b/pkg/services/dashboards/service/client/metrics.go
@@ -0,0 +1,20 @@
+package client
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+type k8sClientMetrics struct {
+	fallbackCounter *prometheus.CounterVec
+}
+
+func newK8sClientMetrics(reg prometheus.Registerer) *k8sClientMetrics {
+	return &k8sClientMetrics{
+		fallbackCounter: promauto.With(reg).NewCounterVec(prometheus.CounterOpts{
+			Namespace: "grafana",
+			Name:      "dashboard_stored_version_fallback_total",
+			Help:      "Number of K8s dashboard client requests to storedVersion",
+		}, []string{"stored_version"}),
+	}
+}
diff --git a/pkg/services/dashboards/service/dashboard_service.go b/pkg/services/dashboards/service/dashboard_service.go
index ad6ed4cd6133b..f33674303a869 100644
--- a/pkg/services/dashboards/service/dashboard_service.go
+++ b/pkg/services/dashboards/service/dashboard_service.go
@@ -13,6 +13,7 @@ import (
 	"github.com/google/uuid"
 	"github.com/prometheus/client_golang/prometheus"
 	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
 	"golang.org/x/exp/maps"
 	"golang.org/x/exp/slices"
 	"golang.org/x/sync/errgroup"
@@ -40,9 +41,9 @@ import (
 	"github.com/grafana/grafana/pkg/services/accesscontrol"
 	"github.com/grafana/grafana/pkg/services/apiserver"
 	"github.com/grafana/grafana/pkg/services/apiserver/client"
-	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
 	"github.com/grafana/grafana/pkg/services/dashboards"
 	"github.com/grafana/grafana/pkg/services/dashboards/dashboardaccess"
+	dashboardclient "github.com/grafana/grafana/pkg/services/dashboards/service/client"
 	dashboardsearch "github.com/grafana/grafana/pkg/services/dashboards/service/search"
 	"github.com/grafana/grafana/pkg/services/featuremgmt"
 	"github.com/grafana/grafana/pkg/services/folder"
@@ -59,7 +60,6 @@ import (
 	"github.com/grafana/grafana/pkg/storage/unified/resource"
 	"github.com/grafana/grafana/pkg/util"
 	"github.com/grafana/grafana/pkg/util/retryer"
-	"go.opentelemetry.io/otel/attribute"
 )
 
 var (
@@ -374,8 +374,7 @@ func ProvideDashboardServiceImpl(
 	serverLockService *serverlock.ServerLockService,
 	kvstore kvstore.KVStore,
 ) (*DashboardServiceImpl, error) {
-	k8sHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
-
+	k8sclient := dashboardclient.NewK8sClientWithFallback(cfg, restConfigProvider, dashboardStore, userService, resourceClient, sorter, dual, r)
 	dashSvc := &DashboardServiceImpl{
 		cfg:                       cfg,
 		log:                       log.New("dashboard-service"),
@@ -386,7 +385,7 @@ func ProvideDashboardServiceImpl(
 		folderStore:               folderStore,
 		folderService:             folderSvc,
 		orgService:                orgService,
-		k8sclient:                 k8sHandler,
+		k8sclient:                 k8sclient,
 		metrics:                   newDashboardsMetrics(r),
 		dashboardPermissionsReady: make(chan struct{}),
 		publicDashboardService:    publicDashboardService,


Reviews:
user: login: marcoabreu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T18:26:52Z
user: login: toddtreece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:33:14Z
user: login: ryantxu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T20:42:01Z
user: login: marcoabreu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T20:48:43Z
user: login: toddtreece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:56:33Z
user: login: ryantxu
body: This makes sense for our usage in `/api/`

Ideally we move the frontend to use `/apis/...` and this is all a non-issue :)
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:03:23Z
user: login: marcoabreu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:56:29Z
user: login: radiohead
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:44:43Z
user: login: radiohead
body: Looks great, thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:51:23Z
user: login: toddtreece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T23:20:26Z
user: login: toddtreece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T23:20:58Z
user: login: toddtreece
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T23:21:23Z
user: login: ryantxu
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-05T13:13:04Z

Review comments:
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: marcoabreu
body: I feel like we would want to have this behind a feature flag - generally, I'd guess we want to avoid these fallbacks triggering at large as it could foster an environment where it's being relied upon more and more. So instead the root cause should be addressed that triggered the fallback. Wdyt?

Also, shall we emit events/metrics for how often this is encountered? 
created_at: 2025-04-03T18:26:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: toddtreece
body: > I feel like we would want to have this behind a feature flag - generally, I'd guess we want to avoid these fallbacks triggering at large as it could foster an environment where it's being relied upon more and more. So instead the root cause should be addressed that triggered the fallback. Wdyt?

good question. it looks like v2alpha1 is [not behind a separate feature flag currently](https://github.com/grafana/grafana/blob/04fb9f534e4e3426dd103c3d8990ac3c9a65a4a8/pkg/registry/apis/dashboard/register.go#L232-L248), so it seems like the fallback should be available if the v2alpha1 is available. does that sound ok?

> Also, shall we emit events/metrics for how often this is encountered?

i think we could add a recording rule for this log line. would that cover it? happy to add a metric if that seems better in this case.
```
h.log.Debug("falling back to stored version", "name", name, "storedVersion", storedVersion, "conversionErr", conversionErr)
```
created_at: 2025-04-03T20:33:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: ryantxu
body: The existence of the v2 api is not behind a feature flag -- and I think that is ok/good

There is a feature flag if the frontend will use it -- and there are clear error messages if the frontend tries to read v2 over the /api/ endpoint

Log line seems good to me -- this *should* be rare -- and only happens now with developers switching between different system states in the same DB
created_at: 2025-04-03T20:42:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: marcoabreu
body: I'm afraid we wouldn't log debug logs in production. So a metric sounds good. 

If we can log this in a way so that we can query more information on demand (eg source of the request), that'd be great in order to assess the spread of the issue. 

So does that mean Ryan that other services are free to create dashboards via v2 although our UI won't be able to render them?

It feels problematic that we allow any v2 calls as long as the UI is on v1
created_at: 2025-04-03T20:48:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: toddtreece
body: > I'm afraid we wouldn't log debug logs in production. So a metric sounds good.

we do as far as i know. i can also update to `Info`

> If we can log this in a way so that we can query more information on demand (eg source of the request), that'd be great in order to assess the spread of the issue.

that's one nice part about log lines. you can create metrics with recording rules and find the source of the call with the trace ID in the log line
created_at: 2025-04-03T20:56:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: marcoabreu
body: But only if we're logging them on a log level that we're actually printing, right?

I thought we would just use some telemetry instrumentation directly, to avoid log procesing. 

But yeah, let's raise to info and put up a recording rule then. 
created_at: 2025-04-03T21:56:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: radiohead
body: > It feels problematic that we allow any v2 calls as long as the UI is on v1

The issue is that the feature flag for dynamic dashboards (which makes the UI switch to v2) can be turned on and off.

Moreover, even if v2 is only ever switched ON, the existing dashboards won't work through v2, requiring the use of separate clients depending on which version a given dashboard was saved with.

Normally in k8s world multiple incompatible versions of the schema (like we have here with v0/v1 and v2) is solved by version conversion logic on the backend – we have that enabled but the logic itself is not implemented yet (it's quite tricky, since the diff between v1 and v2 is rather large).

Once we implement it properly (which we won't be able to achieve before G12) the clients will be able to fetch a given dashboard via both API versions, regardless of which one is actually stored in the storage (although the conversion might be lossy but that's an essential complexity).

You can read more about it here – https://book.kubebuilder.io/multiversion-tutorial/api-changes.

The placeholder conversions are here – https://github.com/grafana/grafana/blob/secret-service/feature-branch/apps/dashboard/pkg/migration/conversion/conversion.go.

> we do as far as i know. i can also update to Info

I think this is a good idea – I guess @hairyhenderson would be happy to turn off debug logging (if we have it enabled) so introducing new debug logs that we _might_ end up needing is not great.

created_at: 2025-04-04T08:44:43Z
author_association: CONTRIBUTOR
path: pkg/services/dashboards/service/k8s_client.go
user: login: radiohead
body: @toddtreece how do you feel about relying on discovery for determining the preferred API version? When dynamic dashboards are enabled, we make v2 the preferred version for the clients so we could check that somewhere in the initializer, to reduce the amount of round-trips (assuming that with v2 being preferred most of the dashboards will be accessible via v2)?

If that makes sense I think it'd be nice to implement that in a follow-up PR.

I've implemented this for `grafanactl` in https://github.com/grafana/grafanactl/blob/main/internal/resources/discovery.go#L67-L89, it's a bit more advanced than what we need here but the gist is the same.

(Sidebar, but I'm planning to move discovery & dynamic client code from `grafanactl` to app SDK once it's stable enough, so we'll be able to just use app SDK code here eventually)
created_at: 2025-04-04T08:49:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,125 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerFactory func(version string) client.K8sHandler
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	newClientFunc k8sHandlerFactory
+	log           log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	newClientFunc := newK8sHandlerFactory(
+		cfg,
+		restConfigProvider,
+		dashboardStore,
+		userService,
+		resourceClient,
+		sorter,
+		dual,
+	)
+	return &k8sHandlerWithFallback{
+		K8sHandler:    newClientFunc(dashboardv1alpha1.VERSION),
+		newClientFunc: newClientFunc,
path: pkg/services/dashboards/service/k8s_client.go
user: login: radiohead
body: Small nit (can be done in a follow-up PR if you agree with the suggestion) but we could cache clients per version to avoid extra initialization every time we need a different version.
created_at: 2025-04-04T08:50:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,134 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	cfg                *setting.Cfg
+	restConfigProvider apiserver.RestConfigProvider
+	dashboardStore     dashboards.Store
+	userService        user.Service
+	resourceClient     resource.ResourceClient
+	sorter             sort.Service
+	dual               dualwrite.Service
+	log                log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	primaryHandler := client.NewK8sHandler(dual, request.GetNamespaceMapper(cfg), dashboardv1alpha1.DashboardResourceInfo.GroupVersionResource(), restConfigProvider.GetRestConfig, dashboardStore, userService, resourceClient, sorter)
+
+	return &k8sHandlerWithFallback{
+		K8sHandler:         primaryHandler,
+		cfg:                cfg,
+		restConfigProvider: restConfigProvider,
+		dashboardStore:     dashboardStore,
+		userService:        userService,
+		resourceClient:     resourceClient,
+		sorter:             sorter,
+		dual:               dual,
+		log:                parentLogger.New("dashboard-k8s-handler"),
+	}
+}
+
+func (h *k8sHandlerWithFallback) Get(ctx context.Context, name string, orgID int64, options metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) {
+	ctx, span := tracing.Start(ctx, "versionFallbackK8sHandler.Get")
+	defer span.End()
+
+	span.SetAttributes(
+		attribute.String("dashboard.metadata.name", name),
+		attribute.Int64("org.id", orgID),
+		attribute.Bool("fallback", false),
+	)
+
+	span.AddEvent("v1alpha1 Get")
+	result, err := h.K8sHandler.Get(ctx, name, orgID, options, subresources...)
+	if err != nil {
+		return nil, tracing.Error(span, err)
+	}
+
+	failed, storedVersion, conversionErr := getConversionStatus(result)
+	if !failed {
+		// if the conversion did not fail, there is no need to fallback.
+		// return the result.
+		return result, nil
+	}
+
+	span.SetAttributes(
+		attribute.Bool("fallback", true),
+		attribute.String("fallback.stored_version", storedVersion),
+		attribute.String("fallback.conversion_error", conversionErr),
+	)
+
+	span.AddEvent("getFallbackHandler")
+	fallbackHandler, handlerErr := h.getFallbackHandler(storedVersion)
+	if handlerErr != nil {
+		return nil, tracing.Errorf(span, "failed to create fallback handler: %w", handlerErr)
+	}
+
+	span.AddEvent(fmt.Sprintf("%s Get", storedVersion))
+	fallbackResult, fallbackErr := fallbackHandler.Get(ctx, name, orgID, options, subresources...)
+	if fallbackErr != nil {
+		return nil, tracing.Errorf(span, "failed to get fallback version %s: %w", storedVersion, fallbackErr)
+	}
+
+	return fallbackResult, nil
+}
+
+func (h *k8sHandlerWithFallback) getFallbackHandler(storedVersion string) (client.K8sHandler, error) {
+	targetGVR := schema.GroupVersionResource{
+		Group:    "dashboard",
+		Version:  storedVersion,
+		Resource: "dashboards",
+	}
+	fallbackHandler := client.NewK8sHandler(h.dual, request.GetNamespaceMapper(h.cfg), targetGVR, h.restConfigProvider.GetRestConfig, h.dashboardStore, h.userService, h.resourceClient, h.sorter)
path: pkg/services/dashboards/service/k8s_client.go
user: login: toddtreece
body: log updated to info and counter added in c77c23c8e07 

<img width="1058" alt="image" src="https://github.com/user-attachments/assets/c9c14988-52ce-4dea-b353-89259ace5372" />

created_at: 2025-04-04T23:20:26Z
author_association: MEMBER
path: pkg/services/dashboards/service/k8s_client.go
user: login: toddtreece
body: i think discovery is a good idea, but i agree that it would be good in a follow up PR
created_at: 2025-04-04T23:20:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+package service
+
+import (
+	"context"
+	"fmt"
+
+	"go.opentelemetry.io/otel/attribute"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+
+	dashboardv1alpha1 "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v1alpha1"
+	"github.com/grafana/grafana/pkg/infra/log"
+	"github.com/grafana/grafana/pkg/infra/tracing"
+	"github.com/grafana/grafana/pkg/services/apiserver"
+	"github.com/grafana/grafana/pkg/services/apiserver/client"
+	"github.com/grafana/grafana/pkg/services/apiserver/endpoints/request"
+	"github.com/grafana/grafana/pkg/services/dashboards"
+	"github.com/grafana/grafana/pkg/services/search/sort"
+	"github.com/grafana/grafana/pkg/services/user"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
+	"github.com/grafana/grafana/pkg/storage/unified/resource"
+)
+
+type k8sHandlerFactory func(version string) client.K8sHandler
+
+type k8sHandlerWithFallback struct {
+	client.K8sHandler
+
+	newClientFunc k8sHandlerFactory
+	log           log.Logger
+}
+
+func NewK8sHandlerWithFallback(
+	cfg *setting.Cfg,
+	restConfigProvider apiserver.RestConfigProvider,
+	dashboardStore dashboards.Store,
+	userService user.Service,
+	resourceClient resource.ResourceClient,
+	sorter sort.Service,
+	dual dualwrite.Service,
+	parentLogger log.Logger,
+) client.K8sHandler {
+	newClientFunc := newK8sHandlerFactory(
+		cfg,
+		restConfigProvider,
+		dashboardStore,
+		userService,
+		resourceClient,
+		sorter,
+		dual,
+	)
+	return &k8sHandlerWithFallback{
+		K8sHandler:    newClientFunc(dashboardv1alpha1.VERSION),
+		newClientFunc: newClientFunc,
path: pkg/services/dashboards/service/k8s_client.go
user: login: toddtreece
body: cache added in https://github.com/grafana/grafana/pull/103404/commits/85fd226a82587af4794b91f2086f420ccd8778ec
created_at: 2025-04-04T23:21:23Z
author_association: MEMBER

PR comments:
user: login: toddtreece
created_at: 2025-04-03T21:25:48Z
body: /deploy-to-hg
author_association: MEMBER
user: login: ephemeral-instances-bot[bot]
created_at: 2025-04-03T21:27:16Z
body: - Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
- Your instance will be ready in ~10 minutes. [Follow the workflow progress](https://github.com/grafana/grafana/actions/runs/14253086698)
- Slack channel: #proj-ephemeral-hg-instances
- Building instance with **toddtreece/dashboard-v2-client-wrapper** oss branch and **main** enterprise branch. [How to choose a branch](https://github.com/grafana/hosted-grafana/wiki/Ephemeral-instances#deploying-a-grafana-and-grafana-enterprise-branch-together)
author_association: NONE
user: login: ephemeral-instances-bot[bot]
created_at: 2025-04-03T21:46:29Z
body: - Your instance can be accessed at: https://ephemeral15111821103404toddtr.grafana-dev.net
- The instance is not using the CDN assets.
- [How to access / How to update instance config / How to build a specific branch](https://github.com/grafana/hosted-grafana/wiki/Ephemeral-instances#choosing-which-oss-and-enterprise-branches-are-used-to-build-the-instance)
author_association: NONE
user: login: toddtreece
created_at: 2025-04-05T00:29:37Z
body: /deploy-to-hg
author_association: MEMBER
user: login: ephemeral-instances-bot[bot]
created_at: 2025-04-05T00:30:15Z
body: - Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
- Your instance will be ready in ~10 minutes. [Follow the workflow progress](https://github.com/grafana/grafana/actions/runs/14276445965)
- Slack channel: #proj-ephemeral-hg-instances
- Building instance with **toddtreece/dashboard-v2-client-wrapper** oss branch and **main** enterprise branch. [How to choose a branch](https://github.com/grafana/hosted-grafana/wiki/Ephemeral-instances#deploying-a-grafana-and-grafana-enterprise-branch-together)
author_association: NONE
user: login: ephemeral-instances-bot[bot]
created_at: 2025-04-05T00:49:00Z
body: - Your instance can be accessed at: https://ephemeral15111821103404toddtr.grafana-dev.net
- The instance is not using the CDN assets.
- [How to access / How to update instance config / How to build a specific branch](https://github.com/grafana/hosted-grafana/wiki/Ephemeral-instances#choosing-which-oss-and-enterprise-branches-are-used-to-build-the-instance)
author_association: NONE