Project: Teleport (Provisioning / Security & Compliance)
File: 0137-unified-resource-view.md

authors: Michael Myers (michael.myers@goteleport.com) state: implemented v14.0 @zmb3 && @rosstimothy note: this is a post-implementation RFD to capture implementation details A new view on the web UI (and eventually Connect) that will display all available resources such as Nodes, Databases, Apps, etc, in a single unified way. Rather than separating all resources into their own individual pages, a Unified Resource page would help consolidate every kind of resource into a single, searchable view. This will help with discoverability and make searching across resources a lot better. This will also lay the foundation for a much more helpful "Dashboard" type page where we can have things such as Pinned Resources (coming right after this). In order to get a searchable and filterable list of all resource kinds, we will create a new, in-memory store on the auth server that will initialize on auth service start and only live as long as the service is up. The watcher will update the store any time it receives an event for any of the watched kinds. We will use a very similar setup for this as our in-memory backend, but instead of storing the resource in its textual representation required by the backend, we can store resources as their concrete type(e.g. types.ServerV2). The key is in the format `name/type` to prevent name collisions across types. However, this make the default sort a little weird where resource with UUID names will come first, even though they'll be displayed with a "friendly" name. We can combat this by storing `Servers` specifically a `name+UUID/type`. This will require a separate map of `resourceKey -> UUID` because the `OpDelete` event only returns a `ResourceHeader`, rather than the resource itself (no hostname in the header). Most of the magic will happen within this new cache. It's a pretty close copy of our memory backend implementation with a few changes. The important types are below ```go // UnifiedResourceCache contains a representation of all resources that are displayable in the UI type UnifiedResourceCache struct { mu sync.Mutex log *log.Entry cfg UnifiedResourceCacheConfig // tree is a BTree with items tree btree.BTreeG[item] initializationC chan struct{} stale bool once sync.Once cache *utils.FnCache ResourceGetter } type resource interface { types.ResourceWithLabels CloneResource() types.ResourceWithLabels } ``` The cache will initialize with a `resourceWatcher` that watches every type currently available in the UI. As of this RFD, those types include server, database servers, app servers, kube servers, saml IdP service providers, and Windows desktops. This can be easily updated to include (or not include) any type by removing its type from the watcher, and the getResource call associated with that type. The cache initializes by getting every current resource of the above types and adding them to the internal bTree, and then starts a watcher to put/delete those types based on events. If the resources are requested from an uninitialized or stale UnifiedResourceCache, we will resort to getting all the current resources again, and storing the result in a `utils.FnCache` until the watcher is back online. This `fnCache` has a TTL of 15

We created a new interface `resource` that extends `ResourceWithLabels` with a new method `CloneResource()`. This is only used to have a common "copy" method when getting resources to prevent concurrent memory modification. The underlying resource watcher is fundamentally the same as usual, except we'd added a `QueueSize` of `8192`, to match other watchers such as auth and proxy. We will add a new grpc endpoint, `ListUnifiedResources`. It will function pretty close to how `ListResources` works, except without needing to differentiate between `RequestType`, as every request is a unified resource request. Originally, the thought of just extending `ListResources`, but because of the many new filters that will be added to the request, we don't want to muddy up the `ListResourcesRequest`. Some of the new filters include which types to include (can be multiple), subtypes, etc etc. ```protobuf // ListUnifiedResourcesRequest is a request to receive a paginated list of unified resources message ListUnifiedResourcesRequest { // Kinds is a list of kinds to match against a resource's kind. This can be used in a // unified resource request that can include multiple types. repeated string Kinds = 1 [(gogoproto.jsontag) = "kinds,omitempty"]; // Limit is the maximum amount of resources to retrieve. int32 Limit = 2 [(gogoproto.jsontag) = "limit,omitempty"]; // StartKey is used to start listing resources from a specific spot. It // should be set to the previous NextKey value if using pagination, or // left empty. string StartKey = 3 [(gogoproto.jsontag) = "start_key,omitempty"]; // Labels is a label-based matcher if non-empty. map<string, string> Labels = 4 [(gogoproto.jsontag) = "labels,omitempty"]; // PredicateExpression defines boolean conditions that will be matched against the resource. string PredicateExpression = 5 [(gogoproto.jsontag) = "predicate_expression,omitempty"]; // SearchKeywords is a list of search keywords to match against resource field values. repeated string SearchKeywords = 6 [(gogoproto.jsontag) = "search_keywords,omitempty"]; // SortBy describes which resource field and which direction to sort by. types.SortBy SortBy = 7 [ (gogoproto.nullable) = false, (gogoproto.jsontag) = "sort_by,omitempty" ]; // WindowsDesktopFilter specifies windows desktop specific filters. types.WindowsDesktopFilter WindowsDesktopFilter = 8 [ (gogoproto.nullable) = false, (gogoproto.jsontag) = "windowsdesktopfilter,omitempty" ]; // UseSearchAsRoles indicates that the response should include all resources // the caller is able to request access to using searchasroles bool UseSearchAsRoles = 9 [(gogoproto.jsontag) = "usesearchas_roles,omitempty"]; // UsePreviewAsRoles indicates that the response should include all resources // the caller would be able to access with their previewasroles bool UsePreviewAsRoles = 10 [(gogoproto.jsontag) = "usepreviewas_roles,omitempty"]; } // ListUnifiedResourceResponse response of ListUnifiedResources. message ListUnifiedResourcesResponse { // Resources is a list of resource. repeated PaginatedResource Resources = 1 [(gogoproto.jsontag) = "resources,omitempty"]; // NextKey is the next Key to use as StartKey in a ListResourcesRequest to // continue retrieving pages of resource. If NextKey is empty, there are no // more pages. string NextKey = 2 [(gogoproto.jsontag) = "next_key,omitempty"]; } ``` This endpoint will pull resources from our unified resource cache and run through the same filtering/rbac as `ListResources` As of the current implementation, `ListUnifiedResources` is not very performant for large

It uses `FakePaginate` under the hood, similar to the rest of the legacy Web UI resources, which will load the entire resource set into memory to then handle sorting, RBAC checks, and pagination. This becomes very taxing as every request has to load the entire set before getting the "next page". This is a limitation due to not being able to preserve a stream across RPCs (for example, how `ListNodes` works with `IterateResources`. A possible solution for the next iteration might be storing resources in a map and using the bTree per sort order to store indexes but this needs to be verified. We will leave `ListResources` alone and keep the old individual resource endpoints available until we've vetted the unified resource view. We will remove the individual navigation items from the web ui, but leave the routes/api endpoints until we've gained confidence in the new flow. Any leaf cluster that does not support Unified Resources will gracefully fallback to a "legacy" view, with the navigation and tables we currently have. The UI for the web will be changing drastically given the fact that all resources will be on the same page and we will be using a Card layout rather than the current Table. However, besides the visual changes, the actual resources listed will still be interacted with the same way. You can still click "Connect" on servers and databases, or Login for apps. The individual functionality for each "Item" will remain the same. Labels will be truncated but viewable with an expand button. This example of a `ResourceCard` is a general view of what a resource will look like. The checkboxes, while not included in the initial release of Unified Resources, will be made available shortly after to perform bulk actions such as lock, search audit logs, etc. These bulk actions will be an ever growing list of actions as time goes on. Also, filters will be added. Some filters will just be a "pseudo-filter" in the sense that it'll just help populate an advanced search string, (example, clicking a label), but a few new actual filters will be present in the requests. The biggest of them is the `Kinds` array, which will contain a list of kinds that you want to be returned. ["node","app"] will only return nodes and apps. If left empty (the default), all resource kinds will be returned This page will also use "infinite scrolling" rather than pagination. We will employ the use of a loading skeleton as we wait for the data to come in and then at a certain scroll distance from the bottom (not the exact bottom, but maybe 200 or so pixels up) we will fetch more. We will not offload previous requests and only append resources as the user scrolls. On paper, this could cause issues if someone tries to load too many resources onto the page, but we do not expect any customers to casually browse through 10,000 items. If this becomes a pain point, we can revisit. The test plan will be updated reflect all the resources are in a single view, but generally speaking the functionality that we are testing should largely remain unchanged.