Project: Teleport (Provisioning / Security & Compliance)
File: 0010-api.md

authors: Alexander Klizhentas (sasha@gravitational.com), Brian Joerger (bjoerger@gravitational.com) state: implemented Unify Golang Client experience. Automatically generate API documentation using . There are several problems with the Teleport Go client libraries: The uses the teleport Go module as a dependency. This means that the example pulls in all of the teleport module's packages and dependencies, leading to extreme import bloat. There is no separate client library with guarantees of compatibility with specific version of Teleport. See examples . Some client logic is residing in , while other logic is in `teleport/lib` packages. It's impossible to generate the library in any other language other than Go due to an unclear API surface. Code in `lib/auth` uses some concepts unfamiliar to Go users. For example, it deals with tls certificates at a low level: ```go // connectClient establishes a gRPC connection to an auth server. func connectClient() (*auth.Client, error) { tlsConfig, err := LoadTLSConfig("certs/api-admin.crt", "certs/api-admin.key", "certs/api-admin.cas") if err != nil { return nil, fmt.Errorf("Failed to setup TLS config: %v", err) } // must parse addr list panics and is not necessary authServerAddr := utils.MustParseAddrList("127.0.0.1:3025") clientConfig := auth.ClientConfig{Addrs: authServerAddr, TLS: tlsConfig} // TLS client is the only client return auth.NewTLSClient(clientConfig) } ``` A new `api` Go module will be made to expose existing and new API features. This will make it easy to import the API with a small import footprint. Users will be able to run `go get github.com/gravitational/teleport/api` to get the latest version of the API and get started. Teleport's internal gRPC Auth client will be moved into the `api/client` package and embedded in the old client for backwards compatibility. Over time, the internal client will be deprecated and replaced with the new public client. Other logic such as the web client and tls/ssh certificate logic will be moved to the `api` package as well. The new client will gain several new quality of life improvements, including credential loaders, automatic server version checking, better documentation, and more language implementations. Move the API client and its dependencies to the new `api` package to keep Teleport's API layer unified. `api` package file structure: ``` teleport/ api/ client | proto | authservice.proto | authservice.pb.go client.go | webclient | webclient.go types events | events.proto | events.pb.go wrappers | wrappers.proto | wrappers.pb.go types.proto | types.pb.go go.mod go.sum ... utils, defaults, constants, and other limited api dependencies ``` Use Go's type aliases to refactor code into the `api` package while maintaining backwards compatibility in the main `teleport` package. These aliases can be removed once the refactor is complete. ```go // Preserve full backwards compatibility type services.Role = types.Role type auth.Service = proto.Service ``` Since `api` will be a submodule and dependency of `teleport`, it will cause an import cycle if `api` has any dependencies on `teleport`. However, `teleport` packages are all dependent on each other in a convoluted dependency graph, so one of the challenges of moving the API to its own module will be to sort out these dependencies. The majority of code that needs to be moved to the `api` module is in `teleport/lib/auth/clt.go` and `teleport/lib/services`. The former will be largely extracted into `api/client` and the latter into

Any protobuf generated code relevant to the API, including `teleport/lib/events` and `teleport/lib/wrappers`, should be moved into `api`. Code in some other package, such as `lib/constants`, `lib/defaults`, `lib/utils`, `teleport/lib/events` and `teleport/lib/wrappers`, will need to be extracted into to `api` module as well. Some logic found in these package will be beyond the scope of the external API and should be refactored to minimize dependencies. This will also help to clarify the separation between api and business logic and make the client more language agnostic. For example, in `lib/services/role`, `role.CheckAndSetDefaults` has some business logic that should remain on the server side. Therefore that logic should be extracted into a new function, `ValidateRole`, and kept in `lib/services/role.go` rather than `api/types/role.go`. ```go // api/types/role.go (r *RoleV3) CheckAndSetDefaults() error { ... basic validation with business logic extracted } ``` ```go // lib/services/role.go func ValidateRole(r types.Role) error { ... business logic } ``` Some basic dependencies will be unavoidable: `github.com/gravitational/trace` `google.golang.org/grpc` `golang.org/x/net` `golang.org/x/crypto` `github.com/stretchr/testify` If the list of dependencies grows beyond 20 direct dependencies, this issue should be revisited for better tracking mechanisms. As stated before, all gRPC client related code will be moved out of `teleport/lib/auth/clt.go` and into `api/client`. Backwards compatibility will be maintained by embedding the gRPC `api/client.Client` in the internal `teleport/lib/auth.Client`. Additionally, the remaining HTTP endpoints in the internal client will be converted and added to the gRPC client. To maintain backwards compatibility with old servers, the internal client will be updated to first try the new gRPC endpoint before falling back to the existing HTTP endpoint. These fallback methods will be moved to `teleport/lib/auth/httpfallback.go` and deprecated after one major release cycle. is being used to track remaining endpoints that need to be converted to gRPC. Once all necessary endpoints have been added in gRPC and all HTTP fallback logic has been deprecated and removed, the gRPC client can completely replace the existing `lib/auth.Client`. At this point the underlying gRPC server will be the same, so the transition should be relatively straightforward. The api package will be made a sub module of teleport. It will exist in the `teleport/api` directory, and will be imported by the main `teleport` module through a go.mod using a relative path. Users will be able to get the latest api version by using `go get github.com/gravitational/teleport/api`, or a specific version with `go get github.com/gravitational/teleport/api@vX.Y.Z`. Go module versioning must follow semver and can automatically be pushed with git tags. Since the api module is a sub module, its version tags will look like `git tag api/vX.Y.Z` (this is for multi-module repositories). If the module's version is above 2, it must follow the by appending a major version suffix to the import path. The import path for each major version `X` will look like `github.com/gravitational/teleport/api/vX`. Note that if these go module requirements aren't implemented, the api module will be versioned as v0.0.0-[random-string]. Getting the latest version with `go get github.com/gravitational/teleport/api` will get the most recent version in terms of time rather than semver/teleport version. For this reason it is important to release a proper public version as soon as possible. The gRPC API and other code imported by `github.com/gravitational/teleport` must follow the rules. However, the api module itself does not need to follow the same versioning

For example, if the API module has a new feature that is not used by the main teleport module, such as the Credential Loaders, it does not need to be compatible with previous versions of Teleport. This gives us some flexibility to choose an api versioning strategy without worries of unrelated versioning promises. The API can be tagged for release as part of the existing Teleport release process. Each Teleport release will get both the `vX.Y.Z` and `api/vX.Y.Z` tags. This can be done within the current with just a few additional steps: Add git tag - `api/vX.Y.Z` If it's a major version bump... Update `api/go.mod` with the correct `vX` major version suffix. Update `teleport/go.mod` to import the new version suffix and update any `teleport/api` imports with the updated suffix. Pros: Teleport Compatibility promise - The `api` module will follow just like `tsh` and `tctl`. As noted above, this isn't strictly necessary for the api, but it may make it simpler to reason about version compatibility. Release checklist changes are minimal For minor/patch releases, there are no significant changes to the release process outside of the additional `api/vX.Y.Z` git tag. For major versions, there is an additional step to update the version suffix, but this is still quite minimal compared to managing the API version completely independently. Cons: Cluttered versions - Even when no API changes are made, there will be a new `api` tag and release. We could opt to skip this tag for such releases, though this will make the version bumps looks strange in some circumstances. e.g `api/v6.1.1` -> `api/v6.1.7`. Frequent major version bumps - The API will likely remain backwards compatible for a long time once it is marked as stable (released at v1+). However major version bumps in semver imply backwards incompatibility. It would be better to avoid major version bumps to simplify the upgrading process for users. Major version suffix rule - The API would start at v2+, so it would have to follow the major version suffix rule. `vX` must be appended to the `api` import path before each major version. This adds an additional release step that is cumbersome for frequent major releases, and can cause development issues such as merge conflicts for backports. Users who want to update to a more recent client major version will have more manual steps; `go get` the new major version and then update import paths. If we want to choose the quickest to implement API versioning strategy, this would be it. With a small additional cognitive load on the release manager, we can release the API alongside teleport within the same release process. Teleport version compatibility can easily be upheld and tracked due to the lined up versions, just like `tsh` and `tctl`, making it familiar to existing users and developers. The API can be released out of step with the main Teleport releases. When we want to release the API, a release manager would follow a slightly modified version of the specifically geared towards the api module. Once a breaking change is added to the `api` module, the major version will be incremented, and the import path will be updated with the major version

Ideally, we would keep it in `v1` or `v0` for as long as possible in order to avoid changing the import path on releases to include the major version suffix. API major version increases will be a special case unlike current Teleport releases. Pros: Following Go Module conventions - Go users will have an easier time if we base our Go Modules on semver rather than our unique versioning guidelines. Frequent major version bumps are irregular. Unstable - The API can start at `v0`, which implies instability. When we are ready to guarantee the API's backwards compatibility and stability, it can be upgraded to `v1`. If we choose to do this, I would suggest that we upgrade to `v1` once the old client is completely deprecated and the new client is used internally. Flexibility - The API will only be released when it is necessary to do so. This can be more or less frequent than Teleport, depending on what features are needed and when. Backwards Compatibility - Since the API will follow semver, it will continue to be backwards compatible with each minor release, even if we release several major versions of Teleport between those minor releases. Cons: Compatibility - It might not be immediately obvious based on the client version what server/cli versions are compatible. A could be added to the docs and `api/client` to help reason about compatibility. One solution to this problem is to version the api as `v1.[teleport-major-version].[api-minor/patch-version]`. With this strategy, any `v1.M.X` Api version will work with any `vM.Y.Z` Teleport version, while still getting the benefits of having a separately managed version. Since the client will have automatic server version checks, users should be able to quickly catch incorrect version usage regardless. Release management - The API will need to be released on its own schedule with its own versions and branches. This will burden release managers with basically a new product to worry about. In the long run, this may be less of a burden than it might seem at first. With `Release option 1`, we may have 20 or so new versions of `teleport/api` released, while with `Release option 2`, most likely there will only be a couple of new versions. This option provides the most flexibility in API releases. It also gives us the option to start with `v0`, with no stability/compatibility promises, in order to try out a strategy before committing fully to it. If treating the API as a separately versioned product is too burdensome, we can immediately transition to another release strategy. The consensus is `Release option 1`. It will be much easier to get started with its versioning since it can follow all the same release processes in place. `Release option 2` brings many more complexities into the discussion, such as new compatibility rules for interfacing with the API and new branches and tags for the api. This is a list of quality of life improvements that will be added to the new api client. The client user experience will be simplified with a new client constructor. The client constructor should be able to retrieve credentials during initialization and handle errors. It should also transparently handle dialing via a proxy if a proxy address or tunnel dialer is provided (along with valid ssh

```go import ( "github.com/gravitational/teleport/api/client" ) func main() { // TLS client is the only TLS client supported in teleport client, err := client.New(client.Config{ Addrs: []string{"localhost:3025"}, // ContextDialer is an optional context dialer ContextDialer: net.DialContext, // direct TLS credentials config Credentials: []client.Credentials{ client.TLSCreds(tls.Config), } }) ... defer client.Close() ping, err := client.Ping() ... fmt.Printf("Ping: %v\n", ping.Version) } ``` ```bash $ go mod init $ go run main.go ``` Support multiple credential providers: `client.LoadTLS(tls.Config)` loads creds from TLS config. `client.LoadKeyPair("crtpath", "keypath", "cas_path")` loads creds from tls certificates. `client.LoadProfile("profilepath", "profilename")` loads creds from the users default or specified tsh profile. `client.LoadIdentityFile("identityfilepath")` loads creds from an identity file. Support tunnel address discovery from the web proxy, as well as tunnel proxy connectivity using ssh certificates. Since tsh profiles have ssh certificates and the web proxy address, LoadProfile should be able to load the client without any user input. The user can simply `tsh login` and then the client will retrieve their credentials. This mirrors the functionality that `tctl` offers. ```go // for testing, use client spec that loads tunnel dialer and certificates from profile client, err := client.New(client.Config{ Credentials: client.Credentials{ client.LoadProfile("", ""), }, }) ``` ```bash $ tsh login $ go run main.go ``` Credential loaders should detect underlying file changes and automatically reload credentials and update the grpc transport. The client will automatically verify that its version is compatible with the server's version by calling `Ping` during client initialization, similarly to how tsh does. The client will fail to initialize if the versions are not compatible. The user can override this by providing `SkipVersionCheck: true` in `client.Config` to avoid blocking users that are in the middle of upgrading. The API's documentation will be split into 3 parts: Automatically generated . on the Teleport docs site. In depth code The majority of the API's documentation will be hosted on . This documentation is automatically generated from the package code upon each package release. Having code-centric documentation will make it easier to maintain and update. It will also improve the experience of working with the code both internally and externally. In order to improve this automatically generated documentation, code comments throughout the api package will be made thorough and user focused. There are several useful to improve the formatting of the documentation, and even add runnable . In the beginning, only essential functionality for the client package will have exceptional documentation. For example, and will have an extensive explanation of its usage with code examples. Any code added to `teleport/api` should be held to a standard of excellence in regards to documentation. Specifically, all added code should have user focused comments and examples. Older code will be updated to this standard of excellence on an ongoing basis according to demand/priority. Some documentation will not fit cleanly within the bounds of the code. Such documentation will instead go on the main Teleport docs page under the section. This section also serves as an entry point to the generated documentation. To begin, this documentation will have three sections: Introduction: short summary, leads customers to relevant documentation. Getting started: High level overview and guide to getting started. Architecture: Covers in depth topics such as managing API authorization, client connections, and

More pages can be added here over time if needed, but the in-code docs should be prioritized. The majority of code examples will be handled by in code . However, some examples require more in depth explanations or combine several client capabilities into a full working example. For now, there is only one example in the `teleport/examples` package https://github.com/gravitational/teleport/tree/branch/v6.2/examples/go-client Once another example is to be added, they will be moved to `teleport/examples/api/example-name` in order to keep all examples in one place. spoiler: The workflows example in `teleport-plugins/access/example` will be the first example to be added. The `teleport-plugins/access` package was originally created in order to utilize the Teleport Auth API without being dependent on internal Teleport code. Now that Teleport has a external API, this is no longer a concern. The access package is now basically a duplication of the api/client, with a few nonessential quality of life features over the `teleport/api/client` package: Automatically append plugin name and other determinable parameters to relevant client methods. e.g. `GetPluginData(reqID)` calls `client.GetPluginData({types.KindAccessRequest, reqID, plugin.name})`. Automatically filter and type cast access request events. Rename, alias, or convert structs to be more user friendly for those who are only using access requests. e.g `types.AccessRequestV3` becomes `access.Request` which has a flat structure as opposed to `teleport.Resource` with metadata and spec structs embedded. These benefits are minimal at best, and actually detrimental at worst. Specifically, users who plan to use the access client for access request management along with the API client for other functionality will need to learn two APIs when they only need to learn one. Additionally, the listed benefits can be largely handled with improved documentation and examples. Another downside of the separate `access` package is that there are two separate clients to maintain, version/release. This adds unnecessary strain and confusion for both maintainers and users, especially when it comes to compatibility concerns. The external plugins held in `teleport-plugins/access` will be updated to use the new client. Once this is complete, the access client can be removed, meaning its final version will be it's previous release. e.g. If we release the updated plugins in `v6.3`, The final version of the access client will be `v6.2`. `v6.2` would still be guaranteed to be compatible with teleport until `v8.0`, according to the Teleport . In order to compensate for the removal of the `access` client and its quality of life features, the API client's documentation around workflows will be improved. User focused comments and examples will be added to all of the essential workflows functions, including the stream watcher. Additionally, the will be refactored to use the API client and added to `teleport/examples/api`. We could add a `teleport/api/client/plugin` package, which wraps plugin-specific methods in the API client. There are currently about 5 methods that would be included here, but in time this may become more useful. ```go type Plugin struct { // embed client, only replace plugin-related methods. client.Client name string } // Automatically use plugin name in relevant functions func (p *Plugin) UpdatePluginData(ctx context.Context, params types.PluginDataUpdateParams) error { params.Plugin = p.name return p.Client.UpdatePluginData(ctx, params) } ``` Access hooks are too low level and require in depth knowledge of teleport and the API

High level access hooks will be added to a new `api/client/access` in the form of a simple handler similar to an http server with a router. ```go func myTeamRequest(ctx context.Context, req types.AccessRequest) (types.AccessRequestUpdate, error) { return DenyParams("No reason, just deny all."), nil } func main() { router := access.Router() ctx := context.Background() router := access.Router{} // handle access requests from my team only router.HandleUpdateFunc(myTeamRequest, MatchUserTrait("team", "myteam")) srv := access.Server{ Client: clt, Handler: router, } // Watch for incoming requests and automatically handle // any that match a HandleFunc. return srv.Run(ctx) } ``` The router will use custom `Match` functions to match a request to a route. ```go type Route struct { Handler HandlerFunc Matchers []Match } // Custom function that will attempt to match a request type Match func(req types.AccessRequest) bool func MatchUserTrait(k, v string) Match { ... } ``` Unlike an http server router, the access hooks server cannot route via a radix tree. An access request may match multiple routes based on its values. For example, a request may have `user.trait.team=myteam` and `user.label.group=security`. If both of these values have their own handler, then the request will match both handlers, despite the handlers having no detectable overlap. Since a simple tree-like solution is not an option, the router will instead loop over the routes available looking for a match with the request's associated matchers. The first one to match will be used to handle the request, so it is important for users to consider handler priority. Handler functions should be high level and easy to use for basic use cases. The most prevalent use case is a basic access request state update, so the base `HandlerFunc` will be designed to simply make a `client.SetAccessRequestState` request. ```go // Automatically send a client.SetAccessRequestState request using the update params returned. // If updateParams.ID is empty, automatically add the request ID to the returned params. // If updateParams is nil, the server will continue without making a request. type HandlerFunc func(ctx context.Context, req types.AccessRequest) (updateParams types.AccessRequestUpdate, err error) // Helper functions for putting together updateParams. func ApproveParams(reason string, roles ...string) types.AccessRequestUpdate { ... } func DenyParams(reason string) types.AccessRequestUpdate { ... } func someHandler(ctx context.Context, req types.AccessRequest) (updateParams types.AccessRequestUpdate, err error)) { if req.GetUser() == "Jill" { return ApproveParams("I know Jill"), nil } else { return DenyParams("I don't know you..."), nil } } ``` Note that if a user wants to make a client request other than `SetAccessRequestState`, such as `SubmitAccessReview` or `UpdatePluginData`, they can use a client in the handler via closure. ```go func main() { clt, err := client.New(...) ... someHandler := func(ctx context.Context, req types.AccessRequest) (updateParams types.AccessRequestUpdate, err error) { clt.SubmitAccessReview(...) // if updateParams are nil, the router won't try to run SetAccessRequestState. return nil, nil } } ``` Once the protobuf schema has been untangled from the `teleport/lib` code, it should be easy to create a bare bones `python` version of the Auth client. Python will be the first language of many new client library implementations. ```bash ls github.com/gravitational/teleport api/ python/ $ pip install teleport-client ``` ```python import teleport client = teleport.client(addr=...) client.tokens() ``` Port router code ```python @access.handle(clt, path=access.trait("team")): def handle(req): if req.user == "bob": raise access.Denied("access denied") ``` Update docs to include the `python` variant of the client as well.