Project: Pachyderm (App Definition and Development / Streaming & Messaging)
File: proto-docs.md

<a name="top"></a> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <a name="admin_admin-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="admin_v2-ClusterInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | deployment_id | | | | | warnings_ok | | | True if the server is capable of generating warnings. | | warnings | | repeated | Warnings about the client configuration. | | proxy_host | | | The configured public URL of Pachyderm. | | proxy_tls | | | True if Pachyderm is served over TLS (HTTPS). | | paused | | | True if this pachd is in &#34;paused&#34; mode. | | web_resources | | | Any HTTP links that the client might want to be aware of. | | metadata | | repeated | Cluster-level metadata. | <a name="admin_v2-ClusterInfo-MetadataEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="admin_v2-InspectClusterRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | client_version | | | The version of the client that&#39;s connecting; used by the server to warn about too-old (or too-new!) clients. | | current_project | | | If CurrentProject is set, then InspectCluster will return an error if the project does not exist. | <a name="admin_v2-WebResource"></a> WebResource contains URL prefixes of common HTTP

| Field | Type | Label | Description | | -- | - | -- | -- | | archivedownloadbase_url | | | The base URL of the archive server; append a filename to this. Empty if the archive server is not exposed. | | createpipelinerequestjsonschema_url | | | Where to find the CreatePipelineRequest JSON schema; if this server is not accessible via a URL, then a link to Github is provided based on the baked-in version of the server. | <a name="admin_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | InspectCluster | | | | <a name="auth_auth-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="auth_v2-ActivateRequest"></a> ActivateRequest enables authentication on the cluster. It issues an auth token with no expiration for the irrevocable admin user `pach:root`. | Field | Type | Label | Description | | -- | - | -- | -- | | root_token | | | If set, this token is used as the root user login token. Otherwise the root token is randomly generated and returned in the response. | <a name="auth_v2-ActivateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pachtoken | | | pachtoken authenticates the caller with Pachyderm (if you want to perform Pachyderm operations after auth has been activated as themselves, you must present this token along with your regular request) | <a name="auth_v2-AuthenticateRequest"></a> Exactly one of &#39;idtoken&#39; or &#39;onetime_password&#39; must be set: | Field | Type | Label | Description | | -- | - | -- | -- | | oidc_state | | | This is the session state that Pachyderm creates in order to keep track of information related to the current OIDC session. | | id_token | | | This is an ID Token issued by the OIDC provider. | <a name="auth_v2-AuthenticateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pachtoken | | | pachtoken authenticates the caller with Pachyderm (if you want to perform Pachyderm operations after auth has been activated as themselves, you must present this token along with your regular request) | <a name="auth_v2-AuthorizeRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | resource | | | | | permissions | | repeated | permissions are the operations the caller is attempting to perform | <a name="auth_v2-AuthorizeResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | authorized | | | authorized is true if the caller has the require permissions | | satisfied | | repeated | satisfied is the set of permission that the principal has | | missing | | repeated | missing is the set of permissions that the principal lacks | | principal | | | principal is the principal the request was evaluated for | <a name="auth_v2-DeactivateRequest"></a> <a name="auth_v2-DeactivateResponse"></a> <a name="auth_v2-DeleteExpiredAuthTokensRequest"></a> <a name="auth_v2-DeleteExpiredAuthTokensResponse"></a> <a name="auth_v2-ExtractAuthTokensRequest"></a> ExtractAuthTokens returns all the hashed robot tokens that have been issued. User tokens are not extracted as they can be recreated by logging

<a name="auth_v2-ExtractAuthTokensResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | tokens | | repeated | | <a name="auth_v2-GetConfigurationRequest"></a> <a name="auth_v2-GetConfigurationResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | configuration | | | | <a name="auth_v2-GetGroupsForPrincipalRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | principal | | | | <a name="auth_v2-GetGroupsRequest"></a> <a name="auth_v2-GetGroupsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | groups | | repeated | | <a name="auth_v2-GetOIDCLoginRequest"></a> <a name="auth_v2-GetOIDCLoginResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | login_url | | | The login URL generated for the OIDC object | | state | | | | <a name="auth_v2-GetPermissionsForPrincipalRequest"></a> GetPermissionsForPrincipal evaluates an arbitrary principal&#39;s permissions on a resource | Field | Type | Label | Description | | -- | - | -- | -- | | resource | | | | | principal | | | | <a name="auth_v2-GetPermissionsRequest"></a> GetPermissions evaluates the

user&#39;s permissions on a resource | Field | Type | Label | Description | | -- | - | -- | -- | | resource | | | | <a name="auth_v2-GetPermissionsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | permissions | | repeated | permissions is the set of permissions the principal has | | roles | | repeated | roles is the set of roles the principal has | <a name="auth_v2-GetRobotTokenRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | robot | | | The returned token will allow the caller to access resources as this robot user | | ttl | | | ttl indicates the requested (approximate) remaining lifetime of this token, in seconds | <a name="auth_v2-GetRobotTokenResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | token | | | A new auth token for the requested robot | <a name="auth_v2-GetRoleBindingRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | resource | | | | <a name="auth_v2-GetRoleBindingResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | binding | | | | <a name="auth_v2-GetRolesForPermissionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | permission | | | | <a name="auth_v2-GetRolesForPermissionResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | roles | | repeated | | <a name="auth_v2-GetUsersRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | group | | | | <a name="auth_v2-GetUsersResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | usernames | | repeated | | <a name="auth_v2-Groups"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | groups | | repeated | | <a name="auth_v2-Groups-GroupsEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="auth_v2-ModifyMembersRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | group | | | | | add | | repeated | | | remove | | repeated | | <a name="auth_v2-ModifyMembersResponse"></a> <a name="auth_v2-ModifyRoleBindingRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | resource | | | resource is the resource to modify the role bindings on | | principal | | | principal is the principal to modify the roles binding for | | roles | | repeated | roles is the set of roles for principal - an empty list removes all role bindings | <a name="auth_v2-ModifyRoleBindingResponse"></a> <a name="auth_v2-OIDCConfig"></a> Configure Pachyderm&#39;s auth system with an OIDC provider | Field | Type | Label | Description | | -- | - | -- | -- | | issuer | | | | | client_id | | | | | client_secret | | | | | redirect_uri | | | | | scopes | | repeated | | | requireemailverified | | | | | localhostissuer | | | localhostissuer ignores the contents of the issuer claim and makes all OIDC requests to the embedded OIDC provider. This is necessary to support some network configurations like Minikube. | | useraccessibleissuerhost | | | useraccessibleissuerhost can be set to override the host used in the OAuth2 authorization URL in case the OIDC issuer isn&#39;t accessible outside the cluster. This requires a fully formed URL with scheme of either http or https. This is necessary to support some configurations like Minikube. | <a name="auth_v2-Resource"></a> Resource represents any resource that has role-bindings in the system | Field | Type | Label | Description | | -- | - | -- | -- | | type | | | | | name | | | | <a name="auth_v2-RestoreAuthTokenRequest"></a> RestoreAuthToken inserts a hashed token that has previously been extracted. | Field | Type | Label | Description | | -- | - | -- | -- | | token | | | | <a name="auth_v2-RestoreAuthTokenResponse"></a> <a name="auth_v2-RevokeAuthTokenRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | token | | | | <a name="auth_v2-RevokeAuthTokenResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | number | | | | <a name="auth_v2-RevokeAuthTokensForUserRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | username | | | | <a name="auth_v2-RevokeAuthTokensForUserResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | number | | | | <a name="auth_v2-Role"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | permissions | | repeated | | | canbebound_to | | repeated | Resources this role can be bound to. For example, you can&#39;t apply clusterAdmin to a repo, so REPO would not be listed here. | | returned_for | | repeated | Resources this role is returned for. For example, a principal might have clusterAdmin permissions on the cluster, and this is what allows them to write to a repo. So, clusterAdmin is returned for the repo, even though it cannot be bound to a repo. | <a name="auth_v2-RoleBinding"></a> RoleBinding represents the set of roles principals have on a given Resource | Field | Type | Label | Description | | -- | - | -- | -- | | entries | | repeated | principal -&gt; roles. All principal names include the structured prefix indicating their

| <a name="auth_v2-RoleBinding-EntriesEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="auth_v2-Roles"></a> Roles represents the set of roles a principal has | Field | Type | Label | Description | | -- | - | -- | -- | | roles | | repeated | | <a name="auth_v2-Roles-RolesEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="auth_v2-RotateRootTokenRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | roottoken | | | roottoken is used as the new root token value. If it&#39;s unset, then a token will be auto-generated. | <a name="auth_v2-RotateRootTokenResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | root_token | | | | <a name="auth_v2-SessionInfo"></a> SessionInfo stores information associated with one OIDC authentication session (i.e. a single instance of a single user logging in). Sessions are short-lived and stored in the &#39;oidc-authns&#39; collection, keyed by the OIDC &#39;state&#39; token (30-character CSPRNG-generated string). &#39;GetOIDCLogin&#39; generates and inserts entries, then /authorization-code/callback retrieves an access token from the ID provider and uses it to retrive the caller&#39;s email and store it in &#39;email&#39;, and finally Authorize() returns a Pachyderm token identified with that email address as a subject in Pachyderm. | Field | Type | Label | Description | | -- | - | -- | -- | | nonce | | | nonce is used by /authorization-code/callback to validate session continuity with the IdP after a user has arrived there from GetOIDCLogin(). This is a 30-character CSPRNG-generated string. | | email | | | email contains the email adddress associated with a user in their OIDC ID provider. Currently users are identified with their email address rather than their OIDC subject identifier to make switching between OIDC ID providers easier for users, and to make user identities more easily comprehensible in Pachyderm. The OIDC spec doesn&#39;t require that users&#39; emails be present or unique, but we think this will be preferable in practice. | | conversionerr | | | conversionerr indicates whether an error was encountered while exchanging an auth code for an access token, or while obtaining a user&#39;s email (in /authorization-code/callback). Storing the error state here allows any sibling calls to Authenticate() (i.e. using the same OIDC state token) to notify their caller that an error has occurred. We avoid passing the caller any details of the error (which are logged by Pachyderm) to avoid giving information to a user who has network access to Pachyderm but not an account in the OIDC provider. | <a name="auth_v2-SetConfigurationRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | configuration | | | | <a name="auth_v2-SetConfigurationResponse"></a> <a name="auth_v2-SetGroupsForUserRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | username | | | | | groups | | repeated | | <a name="auth_v2-SetGroupsForUserResponse"></a> <a name="auth_v2-TokenInfo"></a> TokenInfo is the &#39;value&#39; of an auth token &#39;key&#39; in the &#39;tokens&#39; collection | Field | Type | Label | Description | | -- | - | -- | -- | | subject | | | Subject (i.e. Pachyderm account) that a given token authorizes. See the note at the top of the doc for an explanation of subject

| | expiration | | | | | hashed_token | | | | <a name="auth_v2-Users"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | usernames | | repeated | | <a name="auth_v2-Users-UsernamesEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="auth_v2-WhoAmIRequest"></a> <a name="auth_v2-WhoAmIResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | username | | | | | expiration | | | | <a name="auth_v2-Permission"></a> Permission represents the ability to perform a given operation on a Resource | Name | Number | Description | | - | | -- | | PERMISSION_UNKNOWN | 0 | | | CLUSTERMODIFYBINDINGS | 100 | | | CLUSTERGETBINDINGS | 101 | | | CLUSTERGETPACHD_LOGS | 148 | | | CLUSTERGETLOKI_LOGS | 150 | | | CLUSTERAUTHACTIVATE | 102 | | | CLUSTERAUTHDEACTIVATE | 103 | | | CLUSTERAUTHGET_CONFIG | 104 | | | CLUSTERAUTHSET_CONFIG | 105 | | | CLUSTERAUTHGETROBOTTOKEN | 139 | | | CLUSTERAUTHMODIFYGROUPMEMBERS | 109 | | | CLUSTERAUTHGET_GROUPS | 110 | | | CLUSTERAUTHGETGROUPUSERS | 111 | | | CLUSTERAUTHEXTRACT_TOKENS | 112 | | | CLUSTERAUTHRESTORE_TOKEN | 113 | | | CLUSTERAUTHGETPERMISSIONSFOR_PRINCIPAL | 141 | | | CLUSTERAUTHDELETEEXPIREDTOKENS | 140 | | | CLUSTERAUTHREVOKEUSERTOKENS | 142 | | | CLUSTERAUTHROTATEROOTTOKEN | 147 | | | CLUSTERENTERPRISEACTIVATE | 114 | | | CLUSTERENTERPRISEHEARTBEAT | 115 | | | CLUSTERENTERPRISEGET_CODE | 116 | | | CLUSTERENTERPRISEDEACTIVATE | 117 | | | CLUSTERENTERPRISEPAUSE | 149 | | | CLUSTERIDENTITYSET_CONFIG | 118 | | | CLUSTERIDENTITYGET_CONFIG | 119 | | | CLUSTERIDENTITYCREATE_IDP | 120 | | | CLUSTERIDENTITYUPDATE_IDP | 121 | | | CLUSTERIDENTITYLIST_IDPS | 122 | | | CLUSTERIDENTITYGET_IDP | 123 | | | CLUSTERIDENTITYDELETE_IDP | 124 | | | CLUSTERIDENTITYCREATEOIDCCLIENT | 125 | | | CLUSTERIDENTITYUPDATEOIDCCLIENT | 126 | | | CLUSTERIDENTITYLISTOIDCCLIENTS | 127 | | | CLUSTERIDENTITYGETOIDCCLIENT | 128 | | | CLUSTERIDENTITYDELETEOIDCCLIENT | 129 | | | CLUSTERDEBUGDUMP | 131 | | | CLUSTERLICENSEACTIVATE | 132 | | | CLUSTERLICENSEGET_CODE | 133 | | | CLUSTERLICENSEADD_CLUSTER | 134 | | | CLUSTERLICENSEUPDATE_CLUSTER | 135 | | | CLUSTERLICENSEDELETE_CLUSTER | 136 | | | CLUSTERLICENSELIST_CLUSTERS | 137 | | | CLUSTERCREATESECRET | 143 | TODO(actgardner): Make k8s secrets into nouns and add an Update RPC | | CLUSTERLISTSECRETS | 144 | | | SECRET_DELETE | 145 | | | SECRET_INSPECT | 146 | | | CLUSTEREDITCLUSTER_METADATA | 151 | | | CLUSTERDELETEALL | 138 | | | REPO_READ | 200 | | | REPO_WRITE | 201 | | | REPOMODIFYBINDINGS | 202 | | | REPO_DELETE | 203 | | | REPOINSPECTCOMMIT | 204 | | | REPOLISTCOMMIT | 205 | | | REPODELETECOMMIT | 206 | | | REPOCREATEBRANCH | 207 | | | REPOLISTBRANCH | 208 | | | REPODELETEBRANCH | 209 | | | REPOINSPECTFILE | 210 | | | REPOLISTFILE | 211 | | | REPOADDPIPELINE_READER | 212 | | | REPOREMOVEPIPELINE_READER | 213 | | | REPOADDPIPELINE_WRITER | 214 | | | PIPELINELISTJOB | 301 | | | CLUSTERSETDEFAULTS | 302 | CLUSTERSETDEFAULTS is part of PPS. | | PROJECTSETDEFAULTS | 303 | PROJECTSETDEFAULTS is part of PPS. | | PROJECT_CREATE | 400 | | | PROJECT_DELETE | 401 | | | PROJECTLISTREPO | 402 | | | PROJECTCREATEREPO | 403 | | | PROJECTMODIFYBINDINGS | 404 | | <a name="auth_v2-ResourceType"></a> ResourceType represents the type of a Resource | Name | Number | Description | | - | | -- | | RESOURCETYPEUNKNOWN | 0 | | | CLUSTER | 1 | | | REPO | 2 | | | SPEC_REPO | 3 | | | PROJECT | 4 | | <a name="auth_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Activate | | | Activate/Deactivate the auth

&#39;Activate&#39; sets an initial set of admins for the Pachyderm cluster, and &#39;Deactivate&#39; removes all ACLs, tokens, and admins from the Pachyderm cluster, making all data publicly accessable | | Deactivate | | | | | GetConfiguration | | | | | SetConfiguration | | | | | Authenticate | | | | | Authorize | | | | | GetPermissions | | | | | GetPermissionsForPrincipal | | | | | WhoAmI | | | | | GetRolesForPermission | | | | | ModifyRoleBinding | | | | | GetRoleBinding | | | | | GetOIDCLogin | | | | | GetRobotToken | | | | | RevokeAuthToken | | | | | RevokeAuthTokensForUser | | | | | SetGroupsForUser | | | | | ModifyMembers | | | | | GetGroups | | | | | GetGroupsForPrincipal | | | | | GetUsers | | | | | ExtractAuthTokens | | | | | RestoreAuthToken | | | | | DeleteExpiredAuthTokens | | | | | RotateRootToken | | | | <a name="debug_debug-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="debug_v2-App"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | pods | | repeated | | | timeout | | | | | pipeline | | | | | loki_args | | | | | profile_args | | | | <a name="debug_v2-BinaryRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filter | | | | <a name="debug_v2-DumpChunk"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | content | | | | | progress | | | | <a name="debug_v2-DumpContent"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | content | | | | <a name="debug_v2-DumpProgress"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | task | | | | | total | | | | | progress | | | | <a name="debug_v2-DumpRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filter | | | | | limit | | | Limit sets the limit for the number of commits / jobs that are returned for each repo / pipeline in the dump. | <a name="debug_v2-DumpV2Request"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | system | | | Which system-level information to include in the debug dump. | | pipelines | | repeated | Which pipelines to fetch information about and include in the debug dump. | | input_repos | | | If true, fetch information about non-output repos. | | timeout | | | How long to run the dump for. | | defaults | | | Which defaults to include in the debug dump. | | starlark_scripts | | repeated | A list of Starlark scripts to run. | <a name="debug_v2-DumpV2Request-Defaults"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | cluster_defaults | | | If true, include the cluster

| <a name="debug_v2-Filter"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pachd | | | | | pipeline | | | | | worker | | | | | database | | | | <a name="debug_v2-GetDumpV2TemplateRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filters | | repeated | | <a name="debug_v2-GetDumpV2TemplateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | request | | | | <a name="debug_v2-LokiArgs"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | max_logs | | | | <a name="debug_v2-Pipeline"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | name | | | | <a name="debug_v2-Pod"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | ip | | | | | containers | | repeated | | <a name="debug_v2-Profile"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | duration | | | only meaningful if name == &#34;cpu&#34; | <a name="debug_v2-ProfileArgs"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | profiles | | repeated | | <a name="debug_v2-ProfileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | profile | | | | | filter | | | | <a name="debug_v2-RunPFSLoadTestRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | spec | | | | | branch | | | | | seed | | | | | state_id | | | | <a name="debug_v2-RunPFSLoadTestResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | spec | | | | | branch | | | | | seed | | | | | error | | | | | duration | | | | | state_id | | | | <a name="debug_v2-SetLogLevelRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pachyderm | | | | | grpc | | | | | duration | | | | | recurse | | | | <a name="debug_v2-SetLogLevelResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | affected_pods | | repeated | | | errored_pods | | repeated | | <a name="debug_v2-Starlark"></a> Starlark controls the running of a Starlark script. | Field | Type | Label | Description | | -- | - | -- | -- | | builtin | | | One built into the pachd binary. | | literal | | | Or a script supplied in this request. | | timeout | | | How long to allow the script to run for. If unset, defaults to 1 minute. | <a name="debug_v2-StarlarkLiteral"></a> StarlarkLiteral is a custom Starlark script. | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | The name of the script; used for debug messages and to control where the output goes. | | program_text | | | The text of the &#34;debugdump&#34; personality Starlark

| <a name="debug_v2-System"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | helm | | | | | database | | | | | version | | | | | describes | | repeated | | | logs | | repeated | | | loki_logs | | repeated | | | binaries | | repeated | | | profiles | | repeated | | <a name="debug_v2-TraceChunk"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | content | | | | <a name="debug_v2-TraceRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | duration | | | | <a name="debug_v2-Worker"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pod | | | | | redirected | | | | <a name="debug_v2-SetLogLevelRequest-LogLevel"></a> | Name | Number | Description | | - | | -- | | UNKNOWN | 0 | | | DEBUG | 1 | | | INFO | 2 | | | ERROR | 3 | | | OFF | 4 | Only GRPC logs can be turned off. | <a name="debug_v2-Debug"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Profile | | stream | | | Binary | | stream | | | Dump | | stream | | | SetLogLevel | | | | | GetDumpV2Template | | | | | DumpV2 | | stream | | | Trace | | stream | | | RunPFSLoadTest | | | RunLoadTest runs a load test. | | RunPFSLoadTestDefault | | | RunLoadTestDefault runs the default load tests. | <a name="enterprise_enterprise-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="enterprise_v2-ActivateRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | license_server | | | | | id | | | | | secret | | | | <a name="enterprise_v2-ActivateResponse"></a> <a name="enterprise_v2-DeactivateRequest"></a> <a name="enterprise_v2-DeactivateResponse"></a> <a name="enterprise_v2-EnterpriseConfig"></a> EnterpriseConfig is the configuration we store for heartbeating to the license server. | Field | Type | Label | Description | | -- | - | -- | -- | | licenseserver | | | licenseserver is the address of the grpc license service | | id | | | id is the unique identifier for this pachd, which is registered with the license service | | secret | | | secret is a shared secret between this pachd and the license service | <a name="enterprise_v2-EnterpriseRecord"></a> EnterpriseRecord is a protobuf we cache in etcd to store the enterprise status. | Field | Type | Label | Description | | -- | - | -- | -- | | license | | | license is the cached LicenseRecord retrieved from the most recent heartbeat to the license server. | | lastheartbeat | | | lastheartbeat is the timestamp of the last successful heartbeat to the license server | | heartbeatfailed | | | heartbeatfailed is set if the license is still valid, but the pachd is no longer registered with an enterprise server. This is the same as the expired state, where auth is locked but not

| <a name="enterprise_v2-GetActivationCodeRequest"></a> <a name="enterprise_v2-GetActivationCodeResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | state | | | | | info | | | | | activation_code | | | | <a name="enterprise_v2-GetStateRequest"></a> <a name="enterprise_v2-GetStateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | state | | | | | info | | | | | activationcode | | | activationcode will always be an empty string, call GetEnterpriseCode to get the activation code | <a name="enterprise_v2-HeartbeatRequest"></a> Heartbeat in the enterprise service just triggers a heartbeat for testing purposes. The RPC used to communicate with the license service is defined in the license service. <a name="enterprise_v2-HeartbeatResponse"></a> <a name="enterprise_v2-LicenseRecord"></a> LicenseRecord is the record we store in etcd for a Pachyderm enterprise token that has been provided to a Pachyderm license server | Field | Type | Label | Description | | -- | - | -- | -- | | activation_code | | | | | expires | | | | <a name="enterprise_v2-PauseRequest"></a> <a name="enterprise_v2-PauseResponse"></a> <a name="enterprise_v2-PauseStatusRequest"></a> <a name="enterprise_v2-PauseStatusResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | status | | | | <a name="enterprise_v2-TokenInfo"></a> TokenInfo contains information about the currently active enterprise token | Field | Type | Label | Description | | -- | - | -- | -- | | expires | | | expires indicates when the current token expires (unset if there is no current token) | <a name="enterprise_v2-UnpauseRequest"></a> <a name="enterprise_v2-UnpauseResponse"></a> <a name="enterprise_v2-PauseStatusResponse-PauseStatus"></a> | Name | Number | Description | | - | | -- | | UNPAUSED | 0 | | | PARTIALLY_PAUSED | 1 | | | PAUSED | 2 | | <a name="enterprise_v2-State"></a> | Name | Number | Description | | - | | -- | | NONE | 0 | | | ACTIVE | 1 | | | EXPIRED | 2 | | | HEARTBEAT_FAILED | 3 | | <a name="enterprise_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Activate | | | Provide a Pachyderm enterprise token, enabling Pachyderm enterprise features, such as the Pachyderm Dashboard and Auth system | | GetState | | | | | GetActivationCode | | | | | Heartbeat | | | Heartbeat is used in testing to trigger a heartbeat on demand. Normally this happens on a timer. | | Deactivate | | | Deactivate removes a cluster&#39;s enterprise activation token and sets its enterprise state to NONE. | | Pause | | | Pause pauses the cluster. | | Unpause | | | Unpause unpauses the cluser. | | PauseStatus | | | | <a name="identity_identity-proto"></a> <p

href="#top">Top</a></p> <a name="identity_v2-CreateIDPConnectorRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | connector | | | | <a name="identity_v2-CreateIDPConnectorResponse"></a> <a name="identity_v2-CreateOIDCClientRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | client | | | | <a name="identity_v2-CreateOIDCClientResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | client | | | | <a name="identity_v2-DeleteAllRequest"></a> <a name="identity_v2-DeleteAllResponse"></a> <a name="identity_v2-DeleteIDPConnectorRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="identity_v2-DeleteIDPConnectorResponse"></a> <a name="identity_v2-DeleteOIDCClientRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="identity_v2-DeleteOIDCClientResponse"></a> <a name="identity_v2-GetIDPConnectorRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="identity_v2-GetIDPConnectorResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | connector | | | | <a name="identity_v2-GetIdentityServerConfigRequest"></a> <a name="identity_v2-GetIdentityServerConfigResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | config | | | | <a name="identity_v2-GetOIDCClientRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="identity_v2-GetOIDCClientResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | client | | | | <a name="identity_v2-IDPConnector"></a> IDPConnector represents a connection to an identity provider | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | ID is the unique identifier for this connector. | | name | | | Name is the human-readable identifier for this connector, which will be shown to end users when they&#39;re authenticating. | | type | | | Type is the type of the IDP ex. `saml`, `oidc`, `github`. | | configVersion | | | ConfigVersion must be incremented every time a connector is updated, to avoid concurrent updates conflicting. | | jsonConfig | | | This is left for backwards compatibility, but we want users to use the config defined below. | | config | | | Config is the configuration for the upstream IDP, which varies based on the type. We make the assumption that this is either yaml or JSON. | <a name="identity_v2-IdentityServerConfig"></a> IdentityServerConfig is the configuration for the identity web server. When the configuration is changed the web server is reloaded

| Field | Type | Label | Description | | -- | - | -- | -- | | issuer | | | | | idtokenexpiry | | | | | rotationtokenexpiry | | | | <a name="identity_v2-ListIDPConnectorsRequest"></a> <a name="identity_v2-ListIDPConnectorsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | connectors | | repeated | | <a name="identity_v2-ListOIDCClientsRequest"></a> <a name="identity_v2-ListOIDCClientsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | clients | | repeated | | <a name="identity_v2-OIDCClient"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | redirect_uris | | repeated | | | trusted_peers | | repeated | | | name | | | | | secret | | | | <a name="identity_v2-SetIdentityServerConfigRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | config | | | | <a name="identity_v2-SetIdentityServerConfigResponse"></a> <a name="identity_v2-UpdateIDPConnectorRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | connector | | | | <a name="identity_v2-UpdateIDPConnectorResponse"></a> <a name="identity_v2-UpdateOIDCClientRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | client | | | | <a name="identity_v2-UpdateOIDCClientResponse"></a> <a name="identity_v2-User"></a> User represents an IDP user that has authenticated via OIDC | Field | Type | Label | Description | | -- | - | -- | -- | | email | | | | | last_authenticated | | | | <a name="identity_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | SetIdentityServerConfig | | | | | GetIdentityServerConfig | | | | | CreateIDPConnector | | | | | UpdateIDPConnector | | | | | ListIDPConnectors | | | | | GetIDPConnector | | | | | DeleteIDPConnector | | | | | CreateOIDCClient | | | | | UpdateOIDCClient | | | | | GetOIDCClient | | | | | ListOIDCClients | | | | | DeleteOIDCClient | | | | | DeleteAll | | | | <a name="internalclusterstatev2-5-0commitinfo-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="v250-CommitInfo"></a> CommitInfo is the main data structure representing a commit in etcd | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | origin | | | | | description | | | description is a user-provided script describing this commit | | parent_commit | | | | | child_commits | | repeated | | | started | | | | | finishing | | | | | finished | | | | | direct_provenance | | repeated | | | error | | | | | sizebytesupper_bound | | | | | details | | | | <a name="internalcollectiontest-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="common-TestItem"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | value | | | | | data | | | | <a name="internalconfigconfig-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="config_v2-Config"></a> Config specifies the pachyderm config that is read and interpreted by the pachctl command-line tool. Right now, this is stored at $HOME/.pachyderm/config. Different versions of the pachyderm config are specified as subfields of this message (this allows us to make significant changes to the config structure without breaking existing users by defining a new config version). These structures are stored in a JSON format, so it should be safe to modify fields as long as compatibility is ensured with previous versions. | Field | Type | Label | Description | | -- | - | -- | -- | | user_id | | | | | v1 | | | Configuration options. Exactly one of these fields should be set (depending on which version of the config is being used) | | v2 | | | | <a name="config_v2-ConfigV1"></a> ConfigV1 specifies v1 of the pachyderm config (June 30 2017 - June 2019) | Field | Type | Label | Description | | -- | - | -- | -- | | pachd_address | | | A host:port pointing pachd at a pachyderm cluster. | | server_cas | | | Trusted root certificates (overrides installed certificates), formatted as base64-encoded PEM | | session_token | | | A secret token identifying the current pachctl user within their pachyderm cluster. This is included in all RPCs sent by pachctl, and used to determine if pachctl actions are authorized. | | active_transaction | | | The currently active transaction for batching together pachctl commands. This can be set or cleared via many of the `pachctl * transaction` commands. This is the ID of the transaction object stored in the pachyderm

| <a name="config_v2-ConfigV2"></a> ConfigV2 specifies v2 of the pachyderm config (June 2019 - present) | Field | Type | Label | Description | | -- | - | -- | -- | | active_context | | | | | activeenterprisecontext | | | | | contexts | | repeated | | | metrics | | | | | maxshellcompletions | | | | <a name="config_v2-ConfigV2-ContextsEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="config_v2-Context"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | source | | | Where this context came from | | pachd_address | | | The hostname or IP address pointing pachd at a pachyderm cluster. | | server_cas | | | Trusted root certificates (overrides installed certificates), formatted as base64-encoded PEM. | | session_token | | | A secret token identifying the current pachctl user within their pachyderm cluster. This is included in all RPCs sent by pachctl, and used to determine if pachctl actions are authorized. | | active_transaction | | | The currently active transaction for batching together pachctl commands. This can be set or cleared via many of the `pachctl * transaction` commands. This is the ID of the transaction object stored in the pachyderm etcd. | | cluster_name | | | The k8s cluster name - used to construct a k8s context. | | auth_info | | | The k8s auth info - used to construct a k8s context. | | namespace | | | The k8s namespace - used to construct a k8s context. | | port_forwarders | | repeated | A mapping of service -&gt; port number, when port forwarding is running for this context. | | clusterdeploymentid | | | A unique ID for the cluster deployment. At client initialization time, we ensure this is the same as what the cluster reports back, to prevent us from connecting to the wrong cluster. | | enterprise_server | | | A boolean that records whether the context points at an enterprise server. If false, the context points at a stand-alone pachd. | | project | | | The current project. | <a name="config_v2-Context-PortForwardersEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="config_v2-ContextSource"></a> | Name | Number | Description | | - | | -- | | NONE | 0 | | | CONFIG_V1 | 1 | | | HUB | 2 | | | IMPORTED | 3 | | <a name="internalmetricsmetrics-proto"></a> <p

href="#top">Top</a></p> <a name="metrics-Metrics"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | cluster_id | | | | | pod_id | | | | | nodes | | | | | version | | | | | repos | | | Number of repos | | commits | | | Number of commits -- not used | | files | | | Number of files -- not used | | bytes | | | Number of bytes in all repos | | jobs | | | Number of jobs | | pipelines | | | Number of pipelines in the cluster -- not the same as DAG | | archived_commits | | | Number of archived commit -- not used | | cancelled_commits | | | Number of cancelled commits -- not used | | activation_code | | | Activation code | | max_branches | | | Max branches in across all the repos | | pps_spout | | | Number of spout pipelines | | ppsspoutservice | | | Number of spout services | | cfg_egress | | | Number of pipelines with Egress configured | | cfg_standby | | | Number of pipelines with Standby congigured | | cfg_s3gateway | | | Number of pipelines with S3 Gateway configured | | cfg_services | | | Number of pipelines with services configured | | cfg_errcmd | | | Number of pipelines with error cmd set | | cfg_tfjob | | | Number of pipelines with TFJobs configured | | input_group | | | Number of pipelines with group inputs | | input_join | | | Number of pipelines with join inputs | | input_cross | | | Number of pipelines with cross inputs | | input_union | | | Number of pipelines with union inputs | | input_cron | | | Number of pipelines with cron inputs | | input_git | | | Number of pipelines with git inputs | | input_pfs | | | Number of pfs inputs | | input_commit | | | Number of pfs inputs with commits | | inputjoinon | | | Number of pfs inputs with join_on | | inputouterjoin | | | Number of pipelines with outer joins | | input_lazy | | | Number of pipelines with lazy set | | inputemptyfiles | | | Number of pipelines with empty files set | | input_s3 | | | Number of pipelines with S3 input | | input_trigger | | | Number of pipelines with triggers set | | resourcecpureq | | | Total CPU request across all pipelines | | resourcecpureq_max | | | Max CPU resource requests set | | resourcememreq | | | Sting of memory requests set across all pipelines | | resourcegpureq | | | Total GPU requests across all pipelines | | resourcegpureq_max | | | Max GPU request across all pipelines | | resourcediskreq | | | String of disk requests set across all pipelines | | resourcecpulimit | | | Total CPU limits set across all pipelines | | resourcecpulimit_max | | | Max CPU limit set | | resourcememlimit | | | String of memory limits set | | resourcegpulimit | | | Number of pipelines with | | resourcegpulimit_max | | | Max GPU limit set | | resourcedisklimit | | | String of disk limits set across all pipelines | | max_parallelism | | | Max parallelism set | | min_parallelism | | | Min parallelism set | | num_parallelism | | | Number of pipelines with parallelism set | | enterprise_failures | | | Number of times a command has failed due to an enterprise check | | pipelinewithalerts | | | pipelines with alerts indicator | <a name="internalpfsloadpfsload-proto"></a> <p

href="#top">Top</a></p> <a name="pfsload-CommitSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | modifications | | repeated | | | file_sources | | repeated | | | validator | | | | <a name="pfsload-FileSourceSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | random | | | | <a name="pfsload-FrequencySpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | prob | | | | <a name="pfsload-ModificationSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | put_file | | | | <a name="pfsload-PutFileSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | source | | | | <a name="pfsload-PutFileTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | file_source | | | | | seed | | | | | auth_token | | | | <a name="pfsload-PutFileTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | hash | | | | <a name="pfsload-RandomDirectorySpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | depth | | | | | run | | | | <a name="pfsload-RandomFileSourceSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | directory | | | | | sizes | | repeated | | | increment_path | | | | <a name="pfsload-SizeSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | min_size | | | | | max_size | | | | | prob | | | | <a name="pfsload-State"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commits | | repeated | | <a name="pfsload-State-Commit"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | hash | | | | <a name="pfsload-ValidatorSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | frequency | | | | <a name="internalppsdbppsdb-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="pps_v2-ClusterDefaultsWrapper"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | json | | | | <a name="pps_v2-ProjectDefaultsWrapper"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | json | | | | <a name="internalppsloadppsload-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="ppsload-State"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | branch | | | | | pfsstateid | | | | <a name="internalstoragechunk_chunk-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="chunk-DataRef"></a> DataRef is a reference to data within a chunk. | Field | Type | Label | Description | | -- | - | -- | -- | | ref | | | The chunk the referenced data is located in. | | hash | | | The hash of the data being referenced. | | offset_bytes | | | The offset and size used for accessing the data within the chunk. | | size_bytes | | | | <a name="chunk-Ref"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | size_bytes | | | | | edge | | | | | dek | | | | | encryption_algo | | | | | compression_algo | | | | <a name="chunk-CompressionAlgo"></a> | Name | Number | Description | | - | | -- | | NONE | 0 | | | GZIPBESTSPEED | 1 | | <a name="chunk-EncryptionAlgo"></a> | Name | Number | Description | | - | | -- | | ENCRYPTIONALGOUNKNOWN | 0 | | | CHACHA20 | 1 | | <a name="internalstoragefileset_fileset-proto"></a> <p

href="#top">Top</a></p> <a name="fileset-Composite"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | layers | | repeated | | <a name="fileset-Metadata"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | primitive | | | | | composite | | | | <a name="fileset-Primitive"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | deletive | | | | | additive | | | | | size_bytes | | | | <a name="fileset-TestCacheValue"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="internalstoragefilesetindexindex-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="index-File"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | datum | | | | | data_refs | | repeated | | <a name="index-Index"></a> Index stores an index to and metadata about a range of files or a file. | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | | range | | | NOTE: range and file are mutually exclusive. | | file | | | | | numfiles | | | NOTE: numfiles and size_bytes did not exist in older versions of 2.x, so they will not be set. | | size_bytes | | | | <a name="index-Range"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | offset | | | | | last_path | | | | | chunk_ref | | | | <a name="internaltasktask-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="task-Claim"></a> <a name="task-Group"></a> <a name="task-Task"></a> TODO: Consider splitting this up into separate structures for each state in a oneof. | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | state | | | | | input | | | | | output | | | | | reason | | | | | index | | | | <a name="task-TestTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="task-State"></a> | Name | Number | Description | | - | | -- | | STATE_UNKNOWN | 0 | | | RUNNING | 1 | | | SUCCESS | 2 | | | FAILURE | 3 | | <a name="internaltracingextendedextendedtrace-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="extended-TraceProto"></a> TraceProto contains information identifying a Jaeger trace. It&#39;s used to propagate traces that follow the lifetime of a long operation (e.g. creating a pipeline or running a job), and which live longer than any single RPC. | Field | Type | Label | Description | | -- | - | -- | -- | | serializedtrace | | repeated | serializedtrace contains the info identifying a trace in Jaeger (a (trace ID, span ID, sampled) tuple, basically) | | project | | | | | pipeline | | | pipeline specifies the target pipeline of this trace; this would be set for a trace created by &#39;pachctl create-pipeline&#39; or &#39;pachctl update-pipeline&#39; and would include the kubernetes RPCs involved in creating a pipeline | <a name="extended-TraceProto-SerializedTraceEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="license_license-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="license_v2-ActivateRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | activationcode | | | activationcode is a Pachyderm enterprise activation code. New users can obtain trial activation codes | | expires | | | expires is a timestamp indicating when this activation code will expire. This should not generally be set (it&#39;s primarily used for testing), and is only applied

it&#39;s earlier than the signed expiration time in &#39;activation_code&#39;. | <a name="license_v2-ActivateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | info | | | | <a name="license_v2-AddClusterRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | id is the unique, immutable identifier for this cluster | | address | | | address is the public GPRC address where the cluster can be reached | | secret | | | If set, secret specifies the shared secret this cluster will use to authenticate to the license server. Otherwise a secret will be generated and returned in the response. | | user_address | | | The pachd address for users to connect to. | | clusterdeploymentid | | | The deployment ID value generated by each Cluster | | enterprise_server | | | This field indicates whether the address points to an enterprise server | <a name="license_v2-AddClusterResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | secret | | | | <a name="license_v2-ClusterStatus"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | address | | | | | version | | | | | auth_enabled | | | | | client_id | | | | | last_heartbeat | | | | | created_at | | | | <a name="license_v2-DeactivateRequest"></a> <a name="license_v2-DeactivateResponse"></a> <a name="license_v2-DeleteAllRequest"></a> <a name="license_v2-DeleteAllResponse"></a> <a name="license_v2-DeleteClusterRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="license_v2-DeleteClusterResponse"></a> <a name="license_v2-GetActivationCodeRequest"></a> <a name="license_v2-GetActivationCodeResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | state | | | | | info | | | | | activation_code | | | | <a name="license_v2-HeartbeatRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | secret | | | | | version | | | | | auth_enabled | | | | | client_id | | | | <a name="license_v2-HeartbeatResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | license | | | | <a name="license_v2-ListClustersRequest"></a> <a name="license_v2-ListClustersResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | clusters | | repeated | | <a name="license_v2-ListUserClustersRequest"></a> <a name="license_v2-ListUserClustersResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | clusters | | repeated | | <a name="license_v2-UpdateClusterRequest"></a> Note: Updates of the enterprise-server field are not allowed. In the worst case, a user can recreate their cluster if they need the field

| Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | address | | | | | user_address | | | | | clusterdeploymentid | | | | | secret | | | | <a name="license_v2-UpdateClusterResponse"></a> <a name="license_v2-UserClusterInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | clusterdeploymentid | | | | | address | | | | | enterprise_server | | | | <a name="license_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Activate | | | Activate enables the license service by setting the enterprise activation code to serve. | | GetActivationCode | | | | | DeleteAll | | | DeleteAll deactivates the server and removes all data. | | AddCluster | | | CRUD operations for the pachds registered with this server. | | DeleteCluster | | | | | ListClusters | | | | | UpdateCluster | | | | | Heartbeat | | | Heartbeat is the RPC registered pachds make to the license server to communicate their status and fetch updates. | | ListUserClusters | | | Lists all clusters available to user | <a name="logs_logs-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="logs-AdminLogQuery"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | logql | | | Arbitrary LogQL query | | pod | | | A pod&#39;s logs (all containers) | | pod_container | | | One container | | app | | |

&#34;app&#34; (logql -&gt; {app=X}) | | master | | | All master worker lines from a pipeline | | storage | | | All storage container lines from a pipeline | | user | | | All worker lines from a pipeline/job | <a name="logs-GetLogsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | query | | | | | filter | | | | | tail | | | | | wantpaginghint | | | | <a name="logs-GetLogsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | paging_hint | | | | | log | | | | <a name="logs-JobDatumLogQuery"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | datum | | | | <a name="logs-LogFilter"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | time_range | | | | | limit | | | | | regex | | | | | level | | | Minimum log level to return; worker will always run at level debug, but setting INFO here restores original behavior | <a name="logs-LogMessage"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | verbatim | | | The verbatim line from Loki | | object | | | A raw JSON parse of the entire line | | native_timestamp | | | If a parseable timestamp was found in `fields` | | ppslogmessage | | | For code that wants to filter on pipeline/job/etc | <a name="logs-LogQuery"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | user | | | | | admin | | | | <a name="logs-PagingHint"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | older | | | | | newer | | | | <a name="logs-PipelineJobLogQuery"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | job | | | | <a name="logs-PipelineLogQuery"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | pipeline | | | | <a name="logs-PodContainer"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pod | | | | | container | | | | <a name="logs-RegexLogFilter"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pattern | | | | | negate | | | | <a name="logs-TimeRangeLogFilter"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | from | | | Can be null | | until | | | Can be null | | offset | | | Offset from which to return results | <a name="logs-UserLogQuery"></a> Only returns &#34;user&#34; logs | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | All pipelines in the project | | pipeline | | | One pipeline in a project | | datum | | | One datum. | | job | | | One job, across pipelines and projects | | pipeline_job | | | One job in one pipeline | | job_datum | | | One datum in one job | <a name="logs-VerbatimLogMessage"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | line | | | | | timestamp | | | | <a name="logs-LogLevel"></a> | Name | Number | Description | | - | | -- | | LOGLEVELDEBUG | 0 | | | LOGLEVELINFO | 1 | | | LOGLEVELERROR | 2 | | <a name="logs-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | GetLogs | | stream | | <a name="metadata_metadata-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="metadata-ClusterPicker"></a> ClusterPicker selects a cluster. Since clusters will never &#34;cascade&#34;, there is only one cluster that can be selected, the one running this API server. <a name="metadata-Edit"></a> Edit represents editing one piece of metadata. | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | project targets a named project&#39;s metadata. | | commit | | | commit targets a commit&#39;s metadata. | | branch | | | branch targets a branch&#39;s metadata. | | repo | | | repo targets a repo&#39;s metadata. | | cluster | | | cluster targets the cluster&#39;s metadata. | | replace | | | replace replaces a target&#39;s metadata with a new metadata mapping. | | addkey | | | addkey adds a new key to the target object&#39;s metadata. | | editkey | | | editkey adds or changes a key in the target object&#39;s metadata. | | deletekey | | | deletekey removes a key from the target object&#39;s metadata. | <a name="metadata-Edit-AddKey"></a> AddKey is an operation that adds a new metadata key. | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | key is the metadata key to add. It may not be the empty string. | | value | | | value is the value to assign to the metadata key. | <a name="metadata-Edit-DeleteKey"></a> DeleteKey is an operation that removes a metadata key. | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | key is the metadata key to remove. It may not be the empty

| <a name="metadata-Edit-EditKey"></a> EditKey is an operation that changes or adds a metadata key. | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | key is the metadata key to change or add. It may not be the empty string. | | value | | | value is the value to assign to the metadata key. | <a name="metadata-Edit-Replace"></a> Replace is an operation that replaces metadata. | Field | Type | Label | Description | | -- | - | -- | -- | | replacement | | repeated | replacement is the map to replace the object&#39;s metadata with. | <a name="metadata-Edit-Replace-ReplacementEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="metadata-EditMetadataRequest"></a> EditMetadataRequest is a sequence of edits to apply to metadata. | Field | Type | Label | Description | | -- | - | -- | -- | | edits | | repeated | edits is the ordered list of metadata edits to perform. | <a name="metadata-EditMetadataResponse"></a> EditMetadataResponse is the result of editing metadata. <a name="metadata-API"></a> API is the public API of the metadata service. | Method Name | Request Type | Response Type | Description | | -- | | - | | | EditMetadata | | | EditMetadata edits metadata according to the request. All edits are applied atomically at once. All edits are attempted, but any failing edit fails the entire request. | <a name="pfs_pfs-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="pfs_v2-ActivateAuthRequest"></a> <a name="pfs_v2-ActivateAuthResponse"></a> <a name="pfs_v2-AddFile"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | | datum | | | | | raw | | | | | url | | | | <a name="pfs_v2-AddFile-URLSource"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | URL | | | | | recursive | | | | | concurrency | | | | <a name="pfs_v2-AddFileSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | filesetid | | | | <a name="pfs_v2-AuthInfo"></a> AuthInfo includes the caller&#39;s access scope for a resource, and is returned by services like ListRepo, InspectRepo, and ListProject, but is not persisted in the database. It&#39;s used by the Pachyderm dashboard to render repo access appropriately. To set a user&#39;s auth scope for a resource, use the Pachyderm Auth API (in src/auth/auth.proto) | Field | Type | Label | Description | | -- | - | -- | -- | | permissions | | repeated | The callers access level to the relevant resource. These are very granular permissions - for the end user it makes sense to show them the roles they have instead. | | roles | | repeated | The caller&#39;s roles on the relevant resource. This includes inherited roles from the cluster, project, group membership,

| <a name="pfs_v2-Branch"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | name | | | | <a name="pfs_v2-BranchInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | branch | | | | | head | | | | | provenance | | repeated | | | subvenance | | repeated | | | direct_provenance | | repeated | | | trigger | | | | | metadata | | repeated | | <a name="pfs_v2-BranchInfo-MetadataEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pfs_v2-BranchPicker"></a> BranchPicker defines mutually exclusive pickers that resolve to a single branch. Currently, the only way to pick a branch is by composing a branch name with a repo. Picker messages should only be used as request parameters. | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | <a name="pfs_v2-BranchPicker-BranchName"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | name | | | | <a name="pfs_v2-CheckStorageRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | readchunkdata | | | | | chunk_begin | | | | | chunk_end | | | | <a name="pfs_v2-CheckStorageResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | chunkobjectcount | | | | <a name="pfs_v2-ClearCacheRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | tag_prefix | | | | <a name="pfs_v2-ClearCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | <a name="pfs_v2-Commit"></a> Commit is a reference to a commit (e.g. the collection of branches and the collection of currently-open commits in etcd are collections of Commit protos) | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | id | | | | | branch | | | only used by the client | <a name="pfs_v2-CommitInfo"></a> CommitInfo is the main data structure representing a commit in postgres | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | origin | | | | | description | | | description is a user-provided script describing this commit | | parent_commit | | | | | child_commits | | repeated | | | started | | | | | finishing | | | | | finished | | | | | direct_provenance | | repeated | | | direct_subvenance | | repeated | | | error | | | | | sizebytesupper_bound | | | | | details | | | | | metadata | | repeated | Metadata is user-applied annotations. | <a name="pfs_v2-CommitInfo-Details"></a> Details are only provided when explicitly requested | Field | Type | Label | Description | | -- | - | -- | -- | | size_bytes | | | | | compacting_time | | | | | validating_time | | | | <a name="pfs_v2-CommitInfo-MetadataEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pfs_v2-CommitOrigin"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | kind | | | | <a name="pfs_v2-CommitPicker"></a> CommitPicker defines mutually exclusive pickers that resolve to a single commit. Commits can be picked relatively from some other commit like a parent or start of

Alternatively, they can be picked via their global Id, which is composed of a repo picker and an id. Picker messages should only be used as request parameters. | Field | Type | Label | Description | | -- | - | -- | -- | | branch_head | | | | | id | | | | | ancestor | | | | | branch_root | | | | <a name="pfs_v2-CommitPicker-AncestorOf"></a> This models ^ syntax recursively. | Field | Type | Label | Description | | -- | - | -- | -- | | offset | | | | | start | | | | <a name="pfs_v2-CommitPicker-BranchRoot"></a> This models .N syntax. | Field | Type | Label | Description | | -- | - | -- | -- | | offset | | | | | branch | | | | <a name="pfs_v2-CommitPicker-CommitByGlobalId"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | id | | | | <a name="pfs_v2-CommitSet"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pfs_v2-CommitSetInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit_set | | | | | commits | | repeated | | <a name="pfs_v2-ComposeFileSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetids | | repeated | | | ttl_seconds | | | | | compact | | | | <a name="pfs_v2-CopyFile"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | dst | | | | | datum | | | | | src | | | | | append | | | | <a name="pfs_v2-CreateBranchRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | head | | | | | branch | | | | | provenance | | repeated | | | trigger | | | | | newcommitset | | | overrides the default behavior of using the same CommitSet

&#39;head&#39; | <a name="pfs_v2-CreateFileSetResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="pfs_v2-CreateProjectRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | description | | | | | update | | | | <a name="pfs_v2-CreateRepoRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | description | | | | | update | | | | <a name="pfs_v2-DeleteBranchRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | branch | | | | | force | | | | <a name="pfs_v2-DeleteFile"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | | datum | | | | <a name="pfs_v2-DeleteProjectRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | force | | | | <a name="pfs_v2-DeleteRepoRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | force | | | | <a name="pfs_v2-DeleteRepoResponse"></a> DeleteRepoResponse returns the repos that were deleted by a DeleteRepo call. | Field | Type | Label | Description | | -- | - | -- | -- | | deleted | | | The repos that were deleted, perhaps none. | <a name="pfs_v2-DeleteReposRequest"></a> DeleteReposRequest is used to delete more than one repo at once. | Field | Type | Label | Description | | -- | - | -- | -- | | projects | | repeated | All repos in each project will be deleted if the caller has permission. | | force | | | | | all | | | If all is set, then all repos in all projects will be deleted if the caller has permission. | <a name="pfs_v2-DeleteReposResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repos | | repeated | | <a name="pfs_v2-DiffFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | new_file | | | | | old_file | | | OldFile may be left nil in which case the same path in the parent of NewFile&#39;s commit will be used. | | shallow | | | | <a name="pfs_v2-DiffFileResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | new_file | | | | | old_file | | | | <a name="pfs_v2-DropCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | recursive | | | Setting recursive to true indicates that the drop should be applied recursively to subvenant commits. If recursive is set to false and the provided commit has subvenant commits, the drop will

| <a name="pfs_v2-DropCommitResponse"></a> <a name="pfs_v2-DropCommitSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit_set | | | | <a name="pfs_v2-EgressRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | object_storage | | | | | sql_database | | | | <a name="pfs_v2-EgressResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | object_storage | | | | | sql_database | | | | <a name="pfs_v2-EgressResponse-ObjectStorageResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | bytes_written | | | | <a name="pfs_v2-EgressResponse-SQLDatabaseResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | rows_written | | repeated | | <a name="pfs_v2-EgressResponse-SQLDatabaseResult-RowsWrittenEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pfs_v2-File"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | path | | | | | datum | | | | <a name="pfs_v2-FileInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | | | file_type | | | | | committed | | | | | size_bytes | | | | | hash | | | | <a name="pfs_v2-FindCommitsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | start | | | | | file_path | | | | | limit | | | a limit of 0 means there is no upper bound on the limit. | <a name="pfs_v2-FindCommitsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | found_commit | | | | | lastsearchedcommit | | | | | commits_searched | | | | <a name="pfs_v2-FinishCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | description | | | description is a user-provided string describing this commit. Setting this will overwrite the description set in StartCommit | | error | | | | | force | | | | <a name="pfs_v2-FsckRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fix | | | | | zombie_target | | | | | zombie_all | | | run zombie data detection against all pipelines | <a name="pfs_v2-FsckResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fix | | | | | error | | | | <a name="pfs_v2-GetCacheRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | <a name="pfs_v2-GetCacheResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | value | | | | <a name="pfs_v2-GetFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | | | URL | | | | | offset | | | | | pathrange | | | TODO: int64 sizebytes = 3; | <a name="pfs_v2-GetFileSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | type | | | | <a name="pfs_v2-GlobFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | pattern | | | | | path_range | | | | <a name="pfs_v2-InspectBranchRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | branch | | | | <a name="pfs_v2-InspectCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | wait | | | Wait causes inspect commit to wait until the commit is in the desired

| <a name="pfs_v2-InspectCommitSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit_set | | | | | wait | | | When true, wait until all commits in the set are finished | <a name="pfs_v2-InspectFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | | <a name="pfs_v2-InspectProjectRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | <a name="pfs_v2-InspectProjectV2Request"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | <a name="pfs_v2-InspectProjectV2Response"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | info | | | | | defaults_json | | | | <a name="pfs_v2-InspectRepoRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | <a name="pfs_v2-ListBranchRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | reverse | | | Returns branches oldest to newest | <a name="pfs_v2-ListCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | from | | | | | to | | | | | number | | | | | reverse | | | Return commits oldest to newest | | all | | | Return commits of all kinds (without this, aliases are excluded) | | origin_kind | | | Return only commits of this kind (mutually exclusive with all) | | started_time | | | Return commits started before this time | <a name="pfs_v2-ListCommitSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | <a name="pfs_v2-ListFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | File is the parent directory of the files we want to list. This sets the repo, the commit/branch, and path prefix of files we&#39;re interested in If the &#34;path&#34; field is omitted, a list of files at the top level of the repo is returned | | paginationMarker | | | Marker for pagination. If set, the files that come after the marker in lexicographical order will be returned. If reverse is also set, the files that come before the marker in lexicographical order will be returned. | | number | | | Number of files to return | | reverse | | | If true, return files in reverse order | <a name="pfs_v2-ListProjectRequest"></a> <a name="pfs_v2-ListRepoRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | type | | | Type is the type of (system) repo that should be returned. An empty string requests all repos. | | projects | | repeated | Filters out repos whos project isn&#39;t represented. An empty list of projects doesn&#39;t filter repos by their project. | | page | | | Specifies which page of repos should be returned. If page isn&#39;t specified, a single page containing all the relevant repos is

| <a name="pfs_v2-ModifyFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | set_commit | | | | | add_file | | | | | delete_file | | | | | copy_file | | | | <a name="pfs_v2-ObjectStorageEgress"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | url | | | | <a name="pfs_v2-PathRange"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | lower | | | | | upper | | | | <a name="pfs_v2-Project"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | <a name="pfs_v2-ProjectInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | description | | | | | auth_info | | | | | created_at | | | | | metadata | | repeated | | <a name="pfs_v2-ProjectInfo-MetadataEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pfs_v2-ProjectPicker"></a> ProjectPicker defines mutually exclusive pickers that resolve to a single project. Currently, the only way to pick a project is by using a project name. Picker messages should only be used as request parameters. | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | <a name="pfs_v2-PutCacheRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | | filesetids | | repeated | | | tag | | | | <a name="pfs_v2-RenewFileSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | ttl_seconds | | | | <a name="pfs_v2-Repo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | type | | | | | project | | | | <a name="pfs_v2-RepoInfo"></a> RepoInfo is the main data structure representing a Repo in etcd | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | created | | | | | sizebytesupper_bound | | | | | description | | | | | branches | | repeated | | | auth_info | | | Set by ListRepo and InspectRepo if Pachyderm&#39;s auth system is active, but not stored in etcd. To set a user&#39;s auth scope for a repo, use the Pachyderm Auth API (in src/client/auth/auth.proto) | | details | | | | | metadata | | repeated | Metadata are user-defined key-value pairs. | <a name="pfs_v2-RepoInfo-Details"></a> Details are only provided when explicitly requested | Field | Type | Label | Description | | -- | - | -- | -- | | size_bytes | | | | <a name="pfs_v2-RepoInfo-MetadataEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pfs_v2-RepoPage"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | order | | | | | page_size | | | | | page_index | | | | <a name="pfs_v2-RepoPicker"></a> Repo defines mutually exclusive pickers that resolve to a single repository. Currently, the only way to pick a repo is by composing a repo name and type with a project. If the type is omitted, the &#39;user&#39; type will be used as a default. Picker messages should only be used as request parameters. | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | <a name="pfs_v2-RepoPicker-RepoName"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | name | | | | | type | | | type is optional. If omitted, the default type

&#39;user&#39;. | <a name="pfs_v2-ReposSummary"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | the project the ReposSummary corresponds to | | userrepocount | | | the count of user repos in the summary | | size_bytes | | | aggregate size of all the repos returned in the summary | <a name="pfs_v2-ReposSummaryRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | projects | | repeated | a ReposSummary will be returned for every specified project | <a name="pfs_v2-ReposSummaryResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | summaries | | repeated | repo summaries for the requested projects | <a name="pfs_v2-SQLDatabaseEgress"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | url | | | | | file_format | | | | | secret | | | | <a name="pfs_v2-SQLDatabaseEgress-FileFormat"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | type | | | | | columns | | repeated | | <a name="pfs_v2-SQLDatabaseEgress-Secret"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | key | | | | <a name="pfs_v2-ShardFileSetRequest"></a> If both numfiles and sizebytes are set, shards are created based on whichever threshold is surpassed first. If a shard configuration field (numfiles, sizebytes) is unset, the storage&#39;s default value is used. | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | num_files | | | Number of files targeted in each shard | | size_bytes | | | Size (in bytes) targeted for each shard | <a name="pfs_v2-ShardFileSetResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | shards | | repeated | | <a name="pfs_v2-SquashCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | | | recursive | | | Setting recursive to true indicates that the squash should be applied recursively to subvenant commits. If recursive is set to false and the provided commit has subvenant commits, the squash will fail. | <a name="pfs_v2-SquashCommitResponse"></a> <a name="pfs_v2-SquashCommitSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit_set | | | | <a name="pfs_v2-StartCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | parent | | | parent may be empty in which case the commit that Branch points to will be used as the parent. If the branch does not exist, the commit will have no parent. | | description | | | description is a user-provided string describing this commit | | branch | | | | <a name="pfs_v2-SubscribeCommitRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | repo | | | | | branch | | | | | from | | | only commits created since this commit are returned | | state | | | Don&#39;t return commits until they&#39;re in (at least) the desired

| | all | | | Return commits of all kinds (without this, aliases are excluded) | | origin_kind | | | Return only commits of this kind (mutually exclusive with all) | <a name="pfs_v2-Trigger"></a> Trigger defines the conditions under which a head is moved, and to which branch it is moved. | Field | Type | Label | Description | | -- | - | -- | -- | | branch | | | Which branch this trigger refers to | | all | | | All indicates that all conditions must be satisfied before the trigger happens, otherwise any conditions being satisfied will trigger it. | | ratelimitspec | | | Triggers if the rate limit spec (cron expression) has been satisfied since the last trigger. | | size | | | Triggers if there&#39;s been `size` new data added since the last trigger. | | commits | | | Triggers if there&#39;s been `commits` new commits added since the last trigger. | | cron_spec | | | Creates a background process which fires the trigger on the schedule provided by the cron spec. This condition is mutually exclusive with respect to the others, so setting this will result with the trigger only firing based on the cron schedule. | <a name="pfs_v2-WalkBranchProvenanceRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | start | | repeated | if more than one picker is specified, the result stream is the concatenation of the streams of each picker. | | max_branches | | | defaults to 10,000 if unspecified | | max_depth | | | defaults to 1000 if unspecified | <a name="pfs_v2-WalkBranchSubvenanceRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | start | | repeated | if more than one picker is specified, the result stream is the concatenation of the streams of each picker. | | max_branches | | | defaults to 10,000 if unspecified | | max_depth | | | defaults to 1000 if unspecified | <a name="pfs_v2-WalkCommitProvenanceRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | start | | repeated | if more than one picker is specified, the result stream is the concatenation of the streams of each picker. | | max_commits | | | defaults to 10,000 if unspecified | | max_depth | | | defaults to 1000 if unspecified | <a name="pfs_v2-WalkCommitSubvenanceRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | start | | repeated | if more than one picker is specified, the result stream is the concatenation of the streams of each picker. | | max_commits | | | defaults to 10,000 if unspecified | | max_depth | | | defaults to 1000 if unspecified | <a name="pfs_v2-WalkFileRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | | | paginationMarker | | | Marker for pagination. If set, the files that come after the marker in lexicographical order will be returned. If reverse is also set, the files that come before the marker in lexicographical order will be returned. | | number | | | Number of files to return | | reverse | | | If true, return files in reverse order | <a name="pfs_v2-CommitState"></a> CommitState describes the states a commit can be in. The states are increasingly specific, i.e. a commit that is FINISHED also counts as

| Name | Number | Description | | - | | -- | | COMMITSTATEUNKNOWN | 0 | | | STARTED | 1 | The commit has been started, all commits satisfy this state. | | READY | 2 | The commit has been started, and all of its provenant commits have been finished. | | FINISHING | 3 | The commit is in the process of being finished. | | FINISHED | 4 | The commit has been finished. | <a name="pfs_v2-Delimiter"></a> | Name | Number | Description | | - | | -- | | NONE | 0 | | | JSON | 1 | | | LINE | 2 | | | SQL | 3 | | | CSV | 4 | | <a name="pfs_v2-FileType"></a> | Name | Number | Description | | - | | -- | | RESERVED | 0 | | | FILE | 1 | | | DIR | 2 | | <a name="pfs_v2-GetFileSetRequest-FileSetType"></a> | Name | Number | Description | | - | | -- | | TOTAL | 0 | | | DIFF | 1 | | <a name="pfs_v2-OriginKind"></a> These are the different places where a commit may be originated from | Name | Number | Description | | - | | -- | | ORIGINKINDUNKNOWN | 0 | | | USER | 1 | | | AUTO | 2 | | | FSCK | 3 | | <a name="pfs_v2-RepoPage-Ordering"></a> | Name | Number | Description | | - | | -- | | PROJECT_REPO | 0 | | <a name="pfs_v2-SQLDatabaseEgress-FileFormat-Type"></a> | Name | Number | Description | | - | | -- | | UNKNOWN | 0 | | | CSV | 1 | | | JSON | 2 | | | PARQUET | 3 | | <a name="pfs_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | CreateRepo | | | CreateRepo creates a new repo. | | InspectRepo | | | InspectRepo returns info about a repo. | | ListRepo | | stream | ListRepo returns info about all repos. | | DeleteRepo | | | DeleteRepo deletes a repo. | | DeleteRepos | | | DeleteRepos deletes more than one repo at once. It attempts to delete every repo matching the DeleteReposRequest. When deleting all repos matching a project, any repos not deletable by the caller will remain, and the project will not be empty; this is not an error. The returned DeleteReposResponse will contain a list of all actually-deleted repos. | | StartCommit | | | StartCommit creates a new write commit from a parent commit. | | FinishCommit | | | FinishCommit turns a write commit into a read commit. | | ClearCommit | | | ClearCommit removes all data from the commit. | | InspectCommit | | | InspectCommit returns the info about a commit. | | ListCommit | | stream | ListCommit returns info about all commits. | | SubscribeCommit | | stream | SubscribeCommit subscribes for new commits on a given branch. | | SquashCommit | | | SquashCommit squashes the provided commit into its children. | | DropCommit | | | DropCommit drops the provided commit. | | InspectCommitSet | | stream | InspectCommitSet returns the info about a CommitSet. | | ListCommitSet | | stream | ListCommitSet returns info about all CommitSets. | | SquashCommitSet | | | SquashCommitSet squashes the commits of a CommitSet into their children. Deprecated: Use SquashCommit instead. | | DropCommitSet | | | DropCommitSet drops the commits of a CommitSet and all data included in the commits. Deprecated: Use DropCommit

| | FindCommits | | stream | FindCommits searches for commits that reference a supplied file being modified in a branch. | | WalkCommitProvenance | | stream | WalkCommitProvenance traverses a commit&#39;s provenance graph and streams back each commit encountered. | | WalkCommitSubvenance | | stream | WalkCommitSubvenance traverses a commit&#39;s subvenance graph and streams back each commit encountered. | | CreateBranch | | | CreateBranch creates a new branch. | | InspectBranch | | | InspectBranch returns info about a branch. | | ListBranch | | stream | ListBranch returns info about the heads of branches. | | DeleteBranch | | | DeleteBranch deletes a branch; note that the commits still exist. | | WalkBranchProvenance | | stream | WalkBranchProvenance traverses a branch&#39;s provenance graph and streams back each branch encountered. | | WalkBranchSubvenance | | stream | WalkBranchSubvenance traverses a branch&#39;s subvenance graph and streams back each branch encountered. | | ModifyFile | stream | | ModifyFile performs modifications on a set of files. | | GetFile | | stream | GetFile returns the contents of a single file | | GetFileTAR | | stream | GetFileTAR returns a TAR stream of the contents matched by the request | | InspectFile | | | InspectFile returns info about a file. | | ListFile | | stream | ListFile returns info about all files. | | WalkFile | | stream | WalkFile walks over all the files under a directory, including children of children. | | GlobFile | | stream | GlobFile returns info about all files. | | DiffFile | | stream | DiffFile returns the differences between 2 paths at 2 commits. | | ActivateAuth | | | ActivateAuth creates a role binding for all existing repos | | DeleteAll | | | DeleteAll deletes everything. | | Fsck | | stream | Fsck does a file system consistency check for pfs. | | CreateFileSet | stream | | FileSet API CreateFileSet creates a new file set. | | GetFileSet | | | GetFileSet returns a file set with the data from a commit | | AddFileSet | | | AddFileSet associates a file set with a commit | | RenewFileSet | | | RenewFileSet prevents a file set from being deleted for a set amount of time. | | ComposeFileSet | | | ComposeFileSet composes a file set from a list of file sets. | | ShardFileSet | | | | | CheckStorage | | | CheckStorage runs integrity checks for the storage layer. | | PutCache | | | | | GetCache | | | | | ClearCache | | | | | ListTask | | stream | ListTask lists PFS tasks | | Egress | | | Egress writes data from a commit to an external system | | CreateProject | | | Project API CreateProject creates a new project. | | InspectProject | | | InspectProject returns info about a project. | | InspectProjectV2 | | | InspectProjectV2 returns info about and defaults for a project. | | ListProject | | stream | ListProject returns info about all projects. | | DeleteProject | | | DeleteProject deletes a project. | | ReposSummary | | | Summary API ReposSummary returns a list of summaries about the repos for each of the requested projects. | <a name="pjs_pjs-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="pjs-CancelJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running

| | job | | | | <a name="pjs-CancelJobResponse"></a> <a name="pjs-CreateJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running Job. | | spec | | | | | input | | | | | cache_read | | | | | cache_write | | | | <a name="pjs-CreateJobResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pjs-DeleteJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running Job. | | job | | | | <a name="pjs-DeleteJobResponse"></a> <a name="pjs-InspectJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running Job. | | job | | | job is the job to start walking from. If unset the context Job is assumed. | <a name="pjs-InspectJobResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | details | | | | <a name="pjs-InspectQueueRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | queue | | | | <a name="pjs-InspectQueueResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | details | | | | <a name="pjs-Job"></a> Job uniquely identifies a Job Job will be nil to indicate no Job, or an unset Job. | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pjs-JobInfo"></a> JobInfo describes a Job | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | Job is the Job&#39;s identity | | parentjob | | | parentjob is the Job&#39;s parent if it exists. | | state | | | state is the Job&#39;s state. See JobState for a description of the possible states. | | spec | | | spec is the code specification for the Job. | | input | | | input is the input data for the Job. | | output | | | output is produced by a successfully completing Job | | error | | | error is set when the Job is unable to complete successfully | <a name="pjs-JobInfoDetails"></a> JobInfoDetails is more detailed information about a Job. It contains a superset of the information in JobInfo | Field | Type | Label | Description | | -- | - | -- | -- | | job_info | | | | <a name="pjs-ListJobRequest"></a> TODO: Filter Paginate | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running Job. | | job | | | job is the job to start listing at. If nil, then the listing starts at the first job in the natural ordering. | <a name="pjs-ListJobResponse"></a> ListJobResponse lists information about Jobs ID will always be set. Info and Details may not be set depending on how much information was

| Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | info | | | | | details | | | | <a name="pjs-ListQueueRequest"></a> TODO: Filter Paginate <a name="pjs-ListQueueResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | info | | | | | details | | | | <a name="pjs-ProcessQueueRequest"></a> Queue Messages ProcessQueueRequest is the client -&gt; server message for the bi-di ProcessQueue RPC. | Field | Type | Label | Description | | -- | - | -- | -- | | queue | | | queue is set to start processing from a Queue. | | output | | | output is set by the client to complete the Job successfully. | | failed | | | failed is set by the client to fail the Job. The Job will transition to state DONE with code FAILED. | <a name="pjs-ProcessQueueResponse"></a> ProcessQueueResposne is the server -&gt; client message for the bi-di ProcessQueue RPC. | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used to act on behalf of the Job in other RPCs. The server issues this token to the client, and the client should use it when performing Job RPCs. | | input | | | input is the input data for a Job. The server sends this to ask the client to compute the output. | <a name="pjs-Queue"></a> Queue uniquely identifies a Queue Queue will be nil to identify no Queue, or to indicate unset. | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pjs-QueueElement"></a> QueueElement is a single element in a Queue. | Field | Type | Label | Description | | -- | - | -- | -- | | data | | | data is opaque data used as the input and output of Jobs | | filesets | | repeated | filesets is a list of Fileset handles, used to associate Filesets with the input and output of Jobs. Any of the filesets referenced here will be persisted for as long as this element is in a Queue. New handles, pointing to equivalent Filesets, are minted whenever they cross the API boundary. | <a name="pjs-QueueInfo"></a> QueueInfo describes a Queue | Field | Type | Label | Description | | -- | - | -- | -- | | queue | | | queue is the Queue&#39;s identity | | spec | | | spec specifies the code to be run to process the Queue. | <a name="pjs-QueueInfoDetails"></a> QueueInfoDetails contains detailed information about a Queue, which may be more expensive to get. It contains a superset of the information in QueueInfo. | Field | Type | Label | Description | | -- | - | -- | -- | | queue_info | | | | | size | | | size is the number of elements queued. | <a name="pjs-WalkJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | context | | | context is a bearer token used when calling from within a running Job. | | job | | | job is the job to start walking from. If unset, the context Job is

| <a name="pjs-JobErrorCode"></a> | Name | Number | Description | | - | | -- | | JobErrorCode_UNSPECIFIED | 0 | UNSPECIFIED means the job error code is unspecified. | | FAILED | 1 | FAILED means that the worker processing the job indicated that it failed. | | DISCONNECTED | 2 | DISCONNECTED means the worker processing the job disconnected. | | CANCELED | 3 | CANCELED means the job was canceled. | <a name="pjs-JobState"></a> | Name | Number | Description | | - | | -- | | JobState_UNSPECIFIED | 0 | UNSPECIFIED means the job state is unspecified. | | QUEUED | 1 | QUEUED means the job is currently in a queue. A QUEUED job will not have any descendants. | | PROCESSING | 2 | PROCESSING means the job is currently being processed by a worker. | | DONE | 3 | DONE means the job, and all of its descendants, are done. | <a name="pjs-API"></a> Job API | Method Name | Request Type | Response Type | Description | | -- | | - | | | CreateJob | | | CreateJob creates a new job. Child jobs can be created by setting the context field to the appropriate parent job context. | | CancelJob | | | CancelJob cancels a job. Canceling a job transitions all of the associated QUEUED and PROCESSING jobs to the DONE state and sets their error codes to CANCELED. This will terminate all ongoing processing associated with the job. Nothing will be deleted. A job can only be canceled with the parent job context. | | DeleteJob | | | DeleteJob deletes a job. DeleteJob first cancels the job, then deletes all of the metadata and filesets associated with the job. A job can only be deleted with the parent job context. | | ListJob | | stream | ListJob returns a list of jobs and information about each job. The jobs returned in the list are the child jobs of the provided job. If no job is provided, the list is the child jobs of the provided job context. The provided job must be associated with the provided job context or a descendant of the job associated with the provided job context. | | WalkJob | | stream | WalkJob returns a list of jobs in a hierarchy and information about each job. Walking a job traverses the job hierarchy rooted at the provided job. The provided job must be associated with the provided job context or a descendant of the job associated with the provided job context. | | InspectJob | | | InspectJob returns detailed information about a job. | | ProcessQueue | stream | stream | ProcessQueue should be called by workers to process jobs in a queue. The protocol is as follows: Worker sends an initial request with the queue id. For each job: Server sends a response with a job context and the associated queue element. Worker processes the job. Worker sends a request with the job output or indicates that the job failed. This RPC should generally be run indefinitely. Workers will be scaled based on demand, so the expectation is that they should be processing queues while they are up. This RPC will be canceled by the server if the current job is canceled. Workers should generally retry the RPC when disconnects occur. | | ListQueue | | stream | ListQueue returns a list of queues and information about each queue. | | InspectQueue | | | InspectQueue returns detailed information about a queue. | <a name="pps_pps-proto"></a> <p

href="#top">Top</a></p> <a name="pps_v2-ActivateAuthRequest"></a> <a name="pps_v2-ActivateAuthResponse"></a> <a name="pps_v2-Aggregate"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | count | | | | | mean | | | | | stddev | | | | | fifth_percentile | | | | | ninetyfifthpercentile | | | | <a name="pps_v2-AggregateProcessStats"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | download_time | | | | | process_time | | | | | upload_time | | | | | download_bytes | | | | | upload_bytes | | | | <a name="pps_v2-CheckStatusRequest"></a> Request to check the status of pipelines within a project. | Field | Type | Label | Description | | -- | - | -- | -- | | all | | | boolean field indicating status of all project pipelines. | | project | | | project field | <a name="pps_v2-CheckStatusResponse"></a> Response for check status request. Provides alerts if any. | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | project field | | pipeline | | | pipeline field | | alerts | | repeated | alert indicators | <a name="pps_v2-ClusterDefaults"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | createpipelinerequest | | | CreatePipelineRequest contains the default JSON CreatePipelineRequest into which pipeline specs are merged to form the effective spec used to create a pipeline. | <a name="pps_v2-ContinueCreateDatumRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | number | | | Number of datums to return in next batch. If 0, default batch size is returned. | <a name="pps_v2-CreateDatumRequest"></a> Emits a stream of datums as they are created from the given input. Client must cancel the stream when it no longer wants to receive datums. | Field | Type | Label | Description | | -- | - | -- | -- | | start | | | | | continue | | | | <a name="pps_v2-CreatePipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | tfjob | | | tfjob encodes a Kubeflow TFJob spec. Pachyderm uses this to create TFJobs when running in a kubernetes cluster on which kubeflow has been installed. Exactly one of &#39;tf_job&#39; and &#39;transform&#39; should be set | | transform | | | | | parallelism_spec | | | | | egress | | | | | update | | | | | output_branch | | | | | s3out | | | s3out, if set, requires a pipeline&#39;s user to write to its output repo via Pachyderm&#39;s s3 gateway (if set, workers will serve Pachyderm&#39;s s3 gateway API at http://&lt;pipeline&gt;-s3.&lt;namespace&gt;/&lt;job id&gt;.out/my/file). In this mode /pfs_v2/out won&#39;t be walked or uploaded, and the s3 gateway service in the workers will allow writes to the job&#39;s output commit | | resource_requests | | | | | resource_limits | | | | | sidecarresourcelimits | | | | | input | | | | | description | | | | | reprocess | | | Reprocess forces the pipeline to reprocess all

It only has meaning if Update is true | | service | | | | | spout | | | | | datumsetspec | | | | | datum_timeout | | | | | job_timeout | | | | | salt | | | | | datum_tries | | | | | scheduling_spec | | | | | podspec | | | deprecated, use podpatch below | | podpatch | | | a json patch will be applied to the pipeline&#39;s podspec before it&#39;s created; | | spec_commit | | | | | metadata | | | | | reprocess_spec | | | | | autoscaling | | | | | tolerations | | repeated | | | sidecarresourcerequests | | | | | dry_run | | | | | determined | | | | | maximumexpecteduptime | | | | <a name="pps_v2-CreatePipelineTransaction"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | createpipelinerequest | | | | | user_json | | | the JSON the user originally submitted | | effective_json | | | the effective spec: the result of merging the user JSON into the cluster defaults | <a name="pps_v2-CreatePipelineV2Request"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | createpipelinerequest_json | | | a JSON-encoded CreatePipelineRequest | | dry_run | | | | | update | | | | | reprocess | | | | <a name="pps_v2-CreatePipelineV2Response"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | effectivecreatepipelinerequestjson | | | | <a name="pps_v2-CreateSecretRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file | | | | <a name="pps_v2-CronInput"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | project | | | | | repo | | | | | commit | | | | | spec | | | | | overwrite | | | Overwrite, if true, will expose a single datum that gets overwritten each tick. If false, it will create a new datum for each tick. | | start | | | | <a name="pps_v2-Datum"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | ID is the hash computed from all the files | | id | | | | <a name="pps_v2-DatumInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | datum | | | | | state | | | | | stats | | | | | pfs_state | | | | | data | | repeated | | | image_id | | | | <a name="pps_v2-DatumSetSpec"></a> DatumSetSpec specifies how a pipeline should split its datums into datum sets. | Field | Type | Label | Description | | -- | - | -- | -- | | number | | | number, if nonzero, specifies that each datum set should contain `number` datums. Datum sets may contain fewer if the total number of datums don&#39;t divide evenly. | | sizebytes | | | sizebytes, if nonzero, specifies a target size for each datum set. Datum sets may be larger or smaller than sizebytes, but will usually be pretty close to sizebytes in size. | | perworker | | | perworker, if nonzero, specifies how many datum sets should be created for each worker. It can&#39;t be set with number or

| <a name="pps_v2-DatumStatus"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | started | | | Started is the time processing on the current datum began. | | data | | repeated | | <a name="pps_v2-DeleteJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | <a name="pps_v2-DeletePipelineRequest"></a> Delete a pipeline. If the deprecated all member is true, then delete all pipelines in the default project. | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | all | | | Deprecated. Deprecated. | | force | | | | | keep_repo | | | | | must_exist | | | If true, an error will be returned if the pipeline doesn&#39;t exist. | <a name="pps_v2-DeletePipelinesRequest"></a> Delete more than one pipeline. | Field | Type | Label | Description | | -- | - | -- | -- | | projects | | repeated | All pipelines in each project will be deleted if the caller has permission. | | force | | | | | keep_repo | | | | | all | | | If set, all pipelines in all projects will be deleted if the caller has permission. | <a name="pps_v2-DeletePipelinesResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipelines | | repeated | | <a name="pps_v2-DeleteSecretRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | secret | | | | <a name="pps_v2-Determined"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | workspaces | | repeated | | <a name="pps_v2-Egress"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | URL | | | | | object_storage | | | | | sql_database | | | | <a name="pps_v2-GPUSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | type | | | The type of GPU (nvidia.com/gpu or amd.com/gpu for example). | | number | | | The number of GPUs to request. | <a name="pps_v2-GetClusterDefaultsRequest"></a> <a name="pps_v2-GetClusterDefaultsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | clusterdefaultsjson | | | A JSON-encoded ClusterDefaults message, this is the verbatim input passed to SetClusterDefaults. | <a name="pps_v2-GetLogsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | The pipeline from which we want to get logs (required if the job in &#39;job&#39; was created as part of a pipeline. To get logs from a non-orphan job without the pipeline that created it, you need to use ElasticSearch). | | job | | | The job from which we want to get logs. | | data_filters | | repeated | Names of input files from which we want processing logs. This may contain multiple files, to query pipelines that contain multiple inputs. Each filter may be an absolute path of a file within a pps repo, or it may be a hash for that file (to search for files at specific versions) | | datum | | | | | master | | | If true get logs from the master process | | follow | | | Continue to follow new logs as they become

| | tail | | | If nonzero, the number of lines from the end of the logs to return. Note: tail applies per container, so you will get tail * &lt;number of pods&gt; total lines back. | | uselokibackend | | | UseLokiBackend causes the logs request to go through the loki backend rather than through kubernetes. This behavior can also be achieved by setting the LOKI_LOGGING feature flag. | | since | | | Since specifies how far in the past to return logs from. It defaults to 24 hours. | <a name="pps_v2-GetProjectDefaultsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | <a name="pps_v2-GetProjectDefaultsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | projectdefaultsjson | | | A JSON-encoded ProjectDefaults message, this is the verbatim input passed to SetProjectDefaults. | <a name="pps_v2-Input"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pfs | | | | | join | | repeated | | | group | | repeated | | | cross | | repeated | | | union | | repeated | | | cron | | | | <a name="pps_v2-InputFile"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | This file&#39;s absolute path within its pfs repo. | | hash | | | This file&#39;s hash | <a name="pps_v2-InspectDatumRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | datum | | | | <a name="pps_v2-InspectJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | Callers should set either Job or OutputCommit, not both. | | wait | | | wait until state is either FAILURE or SUCCESS | | details | | | | <a name="pps_v2-InspectJobSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job_set | | | | | wait | | | When true, wait until all jobs in the set are finished | | details | | | | <a name="pps_v2-InspectPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | details | | | When true, return PipelineInfos with the details field, which requires loading the pipeline spec from PFS. | <a name="pps_v2-InspectSecretRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | secret | | | | <a name="pps_v2-Job"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | id | | | | <a name="pps_v2-JobInfo"></a> JobInfo is the data stored in the database regarding a given job. The &#39;details&#39; field contains more information about the job which is expensive to fetch, requiring querying workers or loading the pipeline spec from object storage. | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | pipeline_version | | | | | output_commit | | | | | restart | | | Job restart count

due to datum failure) | | data_processed | | | Counts of how many times we processed or skipped a datum | | data_skipped | | | | | data_total | | | | | data_failed | | | | | data_recovered | | | | | stats | | | Download/process/upload time and download/upload bytes | | state | | | | | reason | | | reason explains why the job is in the current state | | created | | | | | started | | | | | finished | | | | | details | | | | | auth_token | | | | <a name="pps_v2-JobInfo-Details"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transform | | | | | parallelism_spec | | | | | egress | | | | | service | | | | | spout | | | | | worker_status | | repeated | | | resource_requests | | | | | resource_limits | | | | | sidecarresourcelimits | | | | | input | | | | | salt | | | | | datumsetspec | | | | | datum_timeout | | | | | job_timeout | | | | | datum_tries | | | | | scheduling_spec | | | | | pod_spec | | | | | pod_patch | | | | | sidecarresourcerequests | | | | <a name="pps_v2-JobInput"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | commit | | | | | glob | | | | | lazy | | | | <a name="pps_v2-JobSet"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pps_v2-JobSetInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job_set | | | | | jobs | | repeated | | <a name="pps_v2-ListDatumRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | Job and Input are two different ways to specify the datums you want. Only one can be set. Job is the job to list datums from. | | input | | | Input is the input to list datums from. The datums listed are the ones that would be run if a pipeline was created with the provided input. | | filter | | | | | paginationMarker | | | datum id to start from. we do not include this datum in the response | | number | | | Number of datums to return | | reverse | | | If true, return datums in reverse order | <a name="pps_v2-ListDatumRequest-Filter"></a> Filter restricts returned DatumInfo messages to those which match all of the filtered attributes. | Field | Type | Label | Description | | -- | - | -- | -- | | state | | repeated | Must match one of the given states. | <a name="pps_v2-ListJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | projects | | repeated | A list of projects to filter jobs on, nil means don&#39;t filter. | | pipeline | | | nil means all pipelines | | input_commit | | repeated | nil means all inputs | | history | | | History indicates return jobs from historical versions of pipelines semantics are: 0: Return jobs from the current version of the pipeline or

1: Return the above and jobs from the next most recent version 2: etc. -1: Return jobs from all historical versions. | | details | | | Details indicates whether the result should include all pipeline details in each JobInfo, or limited information including name and status, but excluding information in the pipeline spec. Leaving this &#34;false&#34; can make the call significantly faster in clusters with a large number of pipelines and jobs. Note that if &#39;input_commit&#39; is set, this field is coerced to &#34;true&#34; | | jqFilter | | | A jq program string for additional result filtering | | paginationMarker | | | timestamp that is pagination marker | | number | | | number of results to return | | reverse | | | flag to indicated if results should be returned in reverse order | <a name="pps_v2-ListJobSetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | details | | | | | projects | | repeated | A list of projects to filter jobs on, nil means don&#39;t filter. | | paginationMarker | | | we return job sets created before or after this time based on the reverse flag | | number | | | number of results to return | | reverse | | | if true, return results in reverse order | | jqFilter | | | A jq program string for additional result filtering | <a name="pps_v2-ListPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | If non-nil, only return info about a single pipeline, this is redundant with InspectPipeline unless history is non-zero. | | history | | | History indicates how many historical versions you want returned. Its semantics are: 0: Return the current version of the pipeline or pipelines. 1: Return the above and the next most recent version 2: etc. -1: Return all historical versions. | | details | | | Deprecated. Deprecated: Details are always returned. | | jqFilter | | | A jq program string for additional result filtering | | commit_set | | | If non-nil, will return all the pipeline infos at this commit set | | projects | | repeated | Projects to filter on. Empty list means no filter, so return all pipelines. | | page | | | Page indicates which page of a certain size to return. If page is left empty, all of the selected pipelines will be returned. | <a name="pps_v2-LogMessage"></a> LogMessage is a log line from a PPS worker, annotated with metadata indicating when and why the line was logged. | Field | Type | Label | Description | | -- | - | -- | -- | | project_name | | | The job and pipeline for which a PFS file is being processed (if the job is an orphan job, pipeline name and ID will be unset) | | pipeline_name | | | | | job_id | | | | | worker_id | | | | | datum_id | | | | | master | | | | | data | | repeated | The PFS files being processed (one per pipeline/job input) | | user | | | User is true if log message comes from the users

| | ts | | | The message logged, and the time at which it was logged | | message | | | | <a name="pps_v2-LokiLogMessage"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | message | | | | <a name="pps_v2-LokiRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | since | | | | | query | | | | <a name="pps_v2-Metadata"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | annotations | | repeated | | | labels | | repeated | | <a name="pps_v2-Metadata-AnnotationsEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pps_v2-Metadata-LabelsEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pps_v2-PFSInput"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | name | | | | | repo | | | | | repo_type | | | | | branch | | | | | commit | | | | | glob | | | | | join_on | | | | | outer_join | | | | | group_by | | | | | lazy | | | | | empty_files | | | EmptyFiles, if true, will cause files from this PFS input to be presented as empty files. This is useful in shuffle pipelines where you want to read the names of files and reorganize them using symlinks. | | s3 | | | S3, if true, will cause the worker to NOT download or link files from this input into the /pfs_v2 directory. Instead, an instance of our S3 gateway service will run on each of the sidecars, and data can be retrieved from this input by querying http://&lt;pipeline&gt;-s3.&lt;namespace&gt;/&lt;job id&gt;.&lt;input&gt;/my/file | | trigger | | | Trigger defines when this input is processed by the pipeline, if it&#39;s nil the input is processed anytime something is committed to the input branch. | <a name="pps_v2-ParallelismSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | constant | | | Starts the pipeline/job with a &#39;constant&#39; workers, unless &#39;constant&#39; is zero. If &#39;constant&#39; is zero (which is the zero value of ParallelismSpec), then Pachyderm will choose the number of workers that is started, (currently it chooses the number of workers in the cluster) | <a name="pps_v2-Pipeline"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | name | | | | <a name="pps_v2-PipelineInfo"></a> PipelineInfo is proto for each pipeline that Pachd stores in the database. It tracks the state of the pipeline, and points to its metadata in PFS (and, by pointing to a PFS commit, de facto tracks the pipeline&#39;s version). Any information about the pipeline not stored in the database is in the Details object, which requires fetching the spec from PFS or other potentially expensive

| Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | version | | | | | spec_commit | | | The first spec commit for this version of the pipeline | | stopped | | | | | state | | | state indicates the current state of the pipeline | | reason | | | reason includes any error messages associated with a failed pipeline | | lastjobstate | | | lastjobstate indicates the state of the most recently created job | | parallelism | | | parallelism tracks the literal number of workers that this pipeline should run. | | type | | | | | auth_token | | | | | details | | | | | userspecjson | | | The user-submitted pipeline spec in JSON format. | | effectivespecjson | | | The effective spec used to create the pipeline. Created by merging the user spec into the cluster defaults. | <a name="pps_v2-PipelineInfo-Details"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transform | | | | | tfjob | | | tfjob encodes a Kubeflow TFJob spec. Pachyderm uses this to create TFJobs when running in a kubernetes cluster on which kubeflow has been installed. Exactly one of &#39;tf_job&#39;

&#39;transform&#39; should be set | | parallelism_spec | | | | | egress | | | | | created_at | | | | | recent_error | | | | | workers_requested | | | | | workers_available | | | | | output_branch | | | | | resource_requests | | | | | resource_limits | | | | | sidecarresourcelimits | | | | | input | | | | | description | | | | | salt | | | | | reason | | | | | service | | | | | spout | | | | | datumsetspec | | | | | datum_timeout | | | | | job_timeout | | | | | datum_tries | | | | | scheduling_spec | | | | | pod_spec | | | | | pod_patch | | | | | s3_out | | | | | metadata | | | | | reprocess_spec | | | | | unclaimed_tasks | | | | | worker_rc | | | | | autoscaling | | | | | tolerations | | repeated | | | sidecarresourcerequests | | | | | determined | | | | | maximumexpecteduptime | | | | | workersstartedat | | | | <a name="pps_v2-PipelineInfos"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline_info | | repeated | | <a name="pps_v2-PipelinePage"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | order | | | | | page_size | | | | | page_index | | | | <a name="pps_v2-PipelinesSummary"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | the project the PipelinesSummary corresponds to | | active_pipelines | | | count of active pipelines | | paused_pipelines | | | count of paused pipelines | | failed_pipelines | | | count of failed pipelines | | unhealthy_pipelines | | | count of pipelines with a failed latest job | <a name="pps_v2-PipelinesSummaryRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | projects | | repeated | a PipelinesSummary will be returned for each of the requests projects | <a name="pps_v2-PipelinesSummaryResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | summaries | | repeated | the pipeline summaries for the requested projects | <a name="pps_v2-ProcessStats"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | download_time | | | | | process_time | | | | | upload_time | | | | | download_bytes | | | | | upload_bytes | | | | <a name="pps_v2-ProjectDefaults"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | createpipelinerequest | | | | <a name="pps_v2-RenderTemplateRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | template | | | | | args | | repeated | | <a name="pps_v2-RenderTemplateRequest-ArgsEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pps_v2-RenderTemplateResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | json | | | | | specs | | repeated | | <a name="pps_v2-RerunPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | reprocess | | | Reprocess forces the pipeline to reprocess all datums. | <a name="pps_v2-ResourceSpec"></a> ResourceSpec describes the amount of resources that pipeline pods should request from kubernetes, for scheduling. | Field | Type | Label | Description | | -- | - | -- | -- | | cpu | | | The number of CPUs each worker needs (partial values are allowed, and encouraged) | | memory | | | The amount of memory each worker needs (in bytes, with allowed SI suffixes (M, K, G, Mi, Ki, Gi, etc). | | gpu | | | The spec for GPU resources. | | disk | | | The amount of ephemeral storage each worker needs (in bytes, with allowed SI suffixes (M, K, G, Mi, Ki, Gi,

| <a name="pps_v2-RestartDatumRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | data_filters | | repeated | | <a name="pps_v2-RunCronRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | <a name="pps_v2-RunLoadTestRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | dag_spec | | | | | load_spec | | | | | seed | | | | | parallelism | | | | | pod_patch | | | | | state_id | | | | <a name="pps_v2-RunLoadTestResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | error | | | | | state_id | | | | <a name="pps_v2-RunPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | provenance | | repeated | | | job_id | | | | <a name="pps_v2-SchedulingSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | node_selector | | repeated | | | priorityclassname | | | | <a name="pps_v2-SchedulingSpec-NodeSelectorEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pps_v2-Secret"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | <a name="pps_v2-SecretInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | secret | | | | | type | | | | | creation_timestamp | | | | <a name="pps_v2-SecretInfos"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | secret_info | | repeated | | <a name="pps_v2-SecretMount"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | Name must be the name of the secret in kubernetes. | | key | | | Key of the secret to load into env_var, this field only has meaning if EnvVar != &#34;&#34;. | | mount_path | | | | | env_var | | | | <a name="pps_v2-Service"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | internal_port | | | | | external_port | | | | | ip | | | | | type | | | | <a name="pps_v2-SetClusterDefaultsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | regenerate | | | | | reprocess | | | must be false if regenerate is false | | dry_run | | | | | clusterdefaultsjson | | | A JSON-encoded ClusterDefaults message, this will be stored verbatim. | <a name="pps_v2-SetClusterDefaultsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | affected_pipelines | | repeated | | <a name="pps_v2-SetProjectDefaultsRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | project | | | | | regenerate | | | | | reprocess | | | must be false if regenerate is false | | dry_run | | | | | projectdefaultsjson | | | A JSON-encoded ProjectDefaults message, this will be stored verbatim. | <a name="pps_v2-SetProjectDefaultsResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | affected_pipelines | | repeated | | <a name="pps_v2-Spout"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | service | | | | <a name="pps_v2-StartCreateDatumRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | input | | | Input is the input to list datums from. The datums listed are the ones that would be run if a pipeline was created with the provided input. | | number | | | Number of datums to return in first batch. If 0, default batch size is returned. | <a name="pps_v2-StartPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | <a name="pps_v2-StopJobRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | reason | | | | <a name="pps_v2-StopPipelineRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | must_exist | | | If true, an error will be returned if the pipeline doesn&#39;t

| <a name="pps_v2-SubscribeJobRequest"></a> Streams open jobs until canceled | Field | Type | Label | Description | | -- | - | -- | -- | | pipeline | | | | | details | | | Same as ListJobRequest.Details | <a name="pps_v2-TFJob"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | tfjob | | | tfjob is a serialized Kubeflow TFJob spec. Pachyderm sends this directly to a kubernetes cluster on which kubeflow has been installed, instead of creating a pipeline ReplicationController as it normally would. | <a name="pps_v2-Toleration"></a> Toleration is a Kubernetes toleration. | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | key is the taint key that the toleration applies to. Empty means match all taint keys. | | operator | | | operator represents a key&#39;s relationship to the value. | | value | | | value is the taint value the toleration matches to. | | effect | | | effect indicates the taint effect to match. Empty means match all taint effects. | | tolerationseconds | | | tolerationseconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. If not set, tolerate the taint

| <a name="pps_v2-Transform"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | image | | | | | cmd | | repeated | | | err_cmd | | repeated | | | env | | repeated | | | secrets | | repeated | | | imagepullsecrets | | repeated | | | stdin | | repeated | | | err_stdin | | repeated | | | acceptreturncode | | repeated | | | debug | | | | | user | | | | | working_dir | | | | | dockerfile | | | | | memory_volume | | | | | datum_batching | | | | <a name="pps_v2-Transform-EnvEntry"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | key | | | | | value | | | | <a name="pps_v2-UpdateJobStateRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | state | | | | | reason | | | | | restart | | | | | data_processed | | | | | data_skipped | | | | | data_failed | | | | | data_recovered | | | | | data_total | | | | | stats | | | | <a name="pps_v2-Worker"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | name | | | | | state | | | | <a name="pps_v2-WorkerStatus"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | worker_id | | | | | job_id | | | | | datum_status | | | | <a name="pps_v2-DatumState"></a> | Name | Number | Description | | - | | -- | | UNKNOWN | 0 | or not part of a job | | FAILED | 1 | | | SUCCESS | 2 | | | SKIPPED | 3 | | | STARTING | 4 | | | RECOVERED | 5 | | <a name="pps_v2-JobState"></a> | Name | Number | Description | | - | | -- | | JOBSTATEUNKNOWN | 0 | | | JOB_CREATED | 1 | | | JOB_STARTING | 2 | | | JOB_RUNNING | 3 | | | JOB_FAILURE | 4 | | | JOB_SUCCESS | 5 | | | JOB_KILLED | 6 | | | JOB_EGRESSING | 7 | | | JOB_FINISHING | 8 | | | JOB_UNRUNNABLE | 9 | | <a name="pps_v2-PipelineInfo-PipelineType"></a> The pipeline type is stored here so that we can internally know the type of the pipeline without loading the spec from PFS. | Name | Number | Description | | - | | -- | | PIPELINTTYPEUNKNOWN | 0 | | | PIPELINETYPETRANSFORM | 1 | | | PIPELINETYPESPOUT | 2 | | | PIPELINETYPESERVICE | 3 | | <a name="pps_v2-PipelinePage-Ordering"></a> | Name | Number | Description | | - | | -- | | RECENT | 0 | | <a name="pps_v2-PipelineState"></a> | Name | Number | Description | | - | | -- | | PIPELINESTATEUNKNOWN | 0 | | | PIPELINE_STARTING | 1 | There is a PipelineInfo &#43; spec commit, but no RC This happens when a pipeline has been created but not yet picked up by a PPS server. | | PIPELINE_RUNNING | 2 | A pipeline has a spec commit and a service &#43; RC This is the normal state of a pipeline. | | PIPELINE_RESTARTING | 3 | Equivalent to STARTING (there is a PipelineInfo &#43; commit, but no RC) After some error caused runPipeline to exit, but before the pipeline is re-run. This is when the exponential backoff is in effect. | | PIPELINE_FAILURE | 4 | The pipeline has encountered unrecoverable errors and is no longer being retried. It won&#39;t leave this state until the pipeline is updated. | | PIPELINE_PAUSED | 5 | The pipeline has been explicitly paused by the user (the pipeline spec&#39;s Stopped field should be true if the pipeline is in this state) | | PIPELINE_STANDBY | 6 | The pipeline is fully functional, but there are no commits to process. | | PIPELINE_CRASHING | 7 | The pipeline&#39;s workers are crashing, or failing to come up, this may resolve itself, the pipeline may make progress while in this state if the problem is only being experienced by some workers. | <a name="pps_v2-TaintEffect"></a> TaintEffect is an effect that can be matched by a toleration. | Name | Number | Description | | - | | -- | | ALL_EFFECTS | 0 | Empty matches all effects. | | NO_SCHEDULE | 1 | &#34;NoSchedule&#34; | | PREFERNOSCHEDULE | 2 | &#34;PreferNoSchedule&#34; | | NO_EXECUTE | 3 | &#34;NoExecute&#34; | <a name="pps_v2-TolerationOperator"></a> TolerationOperator relates a Toleration&#39;s key to its value. | Name | Number | Description | | - | | -- | | EMPTY | 0 | K8s doesn&#39;t have this, but it&#39;s possible to represent something similar. | | EXISTS | 1 | &#34;Exists&#34; | | EQUAL | 2 | &#34;Equal&#34; | <a name="pps_v2-WorkerState"></a> | Name | Number | Description | | - | | -- | | WORKERSTATEUNKNOWN | 0 | | | POD_RUNNING | 1 | | | POD_SUCCESS | 2 | | | POD_FAILED | 3 | | <a name="pps_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | InspectJob | | | | | InspectJobSet | | stream | | | ListJob | | stream | ListJob returns information about current and past Pachyderm

| | ListJobSet | | stream | | | SubscribeJob | | stream | | | DeleteJob | | | | | StopJob | | | | | InspectDatum | | | | | ListDatum | | stream | ListDatum returns information about each datum fed to a Pachyderm job | | CreateDatum | stream | stream | CreateDatum prioritizes time to first datum. Each request returns a batch of datums. | | RestartDatum | | | | | RerunPipeline | | | | | CreatePipeline | | | | | CreatePipelineV2 | | | | | InspectPipeline | | | | | ListPipeline | | stream | | | DeletePipeline | | | | | DeletePipelines | | | | | StartPipeline | | | | | StopPipeline | | | | | RunPipeline | | | | | RunCron | | | | | CheckStatus | | stream | Check Status returns the status of pipelines within a project. | | CreateSecret | | | | | DeleteSecret | | | | | ListSecret | | | | | InspectSecret | | | | | DeleteAll | | | DeleteAll deletes everything | | GetLogs | | stream | | | ActivateAuth | | | An internal call that causes PPS to put itself into an auth-enabled state (all pipeline have tokens, correct permissions, etcd) | | UpdateJobState | | | An internal call used to move a job from one state to another | | RunLoadTest | | | RunLoadTest runs a load test. | | RunLoadTestDefault | | | RunLoadTestDefault runs the default load test. | | RenderTemplate | | | RenderTemplate renders the provided template and arguments into a list of Pipeline specicifications | | ListTask | | stream | ListTask lists PPS tasks | | GetKubeEvents | | stream | GetKubeEvents returns a stream of kubernetes events | | QueryLoki | | stream | QueryLoki returns a stream of loki log messages given a query string | | GetClusterDefaults | | | GetClusterDefaults returns the current cluster defaults. | | SetClusterDefaults | | | SetClusterDefaults returns the current cluster defaults. | | GetProjectDefaults | | | GetProjectDefaults returns the defaults for a particular project. | | SetProjectDefaults | | | SetProjectDefaults sets the defaults for a particular project. | | PipelinesSummary | | | PipelinesSummary summarizes the pipelines for each requested project. | <a name="protoextensions_json-schema-options-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="protoc-gen-jsonschema-EnumOptions"></a> Custom EnumOptions | Field | Type | Label | Description | | -- | - | -- | -- | | enumsasconstants | | | Enums tagged with this will have be encoded to use constants instead of simple types (supports value annotations): | | enumsasstrings_only | | | Enums tagged with this will only provide string values as options (not their numerical equivalents): | | enumstrimprefix | | | Enums tagged with this will have enum name prefix removed from values: | | ignore | | | Enums tagged with this will not be processed | <a name="protoc-gen-jsonschema-FieldOptions"></a> Custom FieldOptions | Field | Type | Label | Description | | -- | - | -- | -- | | ignore | | | Fields tagged with this will be omitted from generated schemas | | required | | | Fields tagged with this will be marked as &#34;required&#34; in generated schemas | | min_length | | | Fields tagged with this will constrain strings using the &#34;minLength&#34; keyword in generated schemas | | max_length | | | Fields tagged with this will constrain strings using the &#34;maxLength&#34; keyword in generated schemas | | pattern | | | Fields tagged with this will constrain strings using

&#34;pattern&#34; keyword in generated schemas | <a name="protoc-gen-jsonschema-FileOptions"></a> Custom FileOptions | Field | Type | Label | Description | | -- | - | -- | -- | | ignore | | | Files tagged with this will not be processed | | extension | | | Override the default file extension for schemas generated from this file | <a name="protoc-gen-jsonschema-MessageOptions"></a> Custom MessageOptions | Field | Type | Label | Description | | -- | - | -- | -- | | ignore | | | Messages tagged with this will not be processed | | allfieldsrequired | | | Messages tagged with this will have all fields marked as &#34;required&#34;: | | allownullvalues | | | Messages tagged with this will additionally accept null values for all properties: | | disallowadditionalproperties | | | Messages tagged with this will have all fields marked as not allowing additional properties: | | enumsasconstants | | | Messages tagged with this will have all nested enums encoded to use constants instead of simple types (supports value annotations): | <a name="protoextensions_json-schema-options-proto-extensions"></a> | Extension | Type | Base | Number | Description | | | - | - | | -- | | enum_options | EnumOptions | .google.protobuf.EnumOptions | 1128 | | | field_options | FieldOptions | .google.protobuf.FieldOptions | 1125 | | | file_options | FileOptions | .google.protobuf.FileOptions | 1126 | | | message_options | MessageOptions | .google.protobuf.MessageOptions | 1127 | | <a name="protoextensions_log-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="protoextensions_log-proto-extensions"></a> | Extension | Type | Base | Number | Description | | | - | - | | -- | | half | bool | .google.protobuf.FieldOptions | 50002 | | | mask | bool | .google.protobuf.FieldOptions | 50001 | | <a name="protoextensions_validate-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="validate-AnyRules"></a> AnyRules describe constraints applied exclusively to the `google.protobuf.Any` well-known type | Field | Type | Label | Description | | -- | - | -- | -- | | required | | optional | Required specifies that this field must be set | | in | | repeated | In specifies that this field&#39;s `type_url` must be equal to one of the specified values. | | notin | | repeated | NotIn specifies that this field&#39;s `typeurl` must not be equal to any of the specified values. | <a name="validate-BoolRules"></a> BoolRules describes the constraints applied to `bool` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | <a name="validate-BytesRules"></a> BytesRules describe the constraints applied to `bytes` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | len | | optional | Len specifies that this field must be the specified number of bytes | | min_len | | optional | MinLen specifies that this field must be the specified number of bytes at a minimum | | max_len | | optional | MaxLen specifies that this field must be the specified number of bytes at a maximum | | pattern | | optional | Pattern specifes that this field must match against the specified regular expression (RE2 syntax). The included expression should elide any delimiters. | | prefix | | optional | Prefix specifies that this field must have the specified bytes at the beginning of the string. | | suffix | | optional | Suffix specifies that this field must have the specified bytes at the end of the

| | contains | | optional | Contains specifies that this field must have the specified bytes anywhere in the string. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ip | | optional | Ip specifies that the field must be a valid IP (v4 or v6) address in byte format | | ipv4 | | optional | Ipv4 specifies that the field must be a valid IPv4 address in byte format | | ipv6 | | optional | Ipv6 specifies that the field must be a valid IPv6 address in byte format | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-DoubleRules"></a> DoubleRules describes the constraints applied to `double` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-DurationRules"></a> DurationRules describe the constraints applied exclusively to the

well-known type | Field | Type | Label | Description | | -- | - | -- | -- | | required | | optional | Required specifies that this field must be set | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lt specifies that this field must be less than the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive | | gte | | optional | Gte specifies that this field must be greater than the specified value, inclusive | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | <a name="validate-EnumRules"></a> EnumRules describe the constraints applied to enum values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | defined_only | | optional | DefinedOnly specifies that this field must be only one of the defined values for this enum, failing on any undefined value. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | <a name="validate-FieldRules"></a> FieldRules encapsulates the rules for each type of field. Depending on the field, the correct set should be used to ensure proper validations. | Field | Type | Label | Description | | -- | - | -- | -- | | message | | optional | | | float | | optional | Scalar Field Types | | double | | optional | | | int32 | | optional | | | int64 | | optional | | | uint32 | | optional | | | uint64 | | optional | | | sint32 | | optional | | | sint64 | | optional | | | fixed32 | | optional | | | fixed64 | | optional | | | sfixed32 | | optional | | | sfixed64 | | optional | | | bool | | optional | | | string | | optional | | | bytes | | optional | | | enum | | optional | Complex Field Types | | repeated | | optional | | | map | | optional | | | any | | optional | Well-Known Field Types | | duration | | optional | | | timestamp | | optional | | <a name="validate-Fixed32Rules"></a> Fixed32Rules describes the constraints applied to `fixed32` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is

| | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-Fixed64Rules"></a> Fixed64Rules describes the constraints applied to `fixed64` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-FloatRules"></a> FloatRules describes the constraints applied to `float` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-Int32Rules"></a> Int32Rules describes the constraints applied to `int32` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is

| | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-Int64Rules"></a> Int64Rules describes the constraints applied to `int64` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-MapRules"></a> MapRules describe the constraints applied to `map` values | Field | Type | Label | Description | | -- | - | -- | -- | | min_pairs | | optional | MinPairs specifies that this field must have the specified number of KVs at a minimum | | max_pairs | | optional | MaxPairs specifies that this field must have the specified number of KVs at a maximum | | no_sparse | | optional | NoSparse specifies values in this field cannot be unset. This only applies to map&#39;s with message value types. | | keys | | optional | Keys specifies the constraints to be applied to each key in the field. | | values | | optional | Values specifies the constraints to be applied to the value of each key in the field. Message values will still have their validations evaluated unless skip is specified here. | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-MessageRules"></a> MessageRules describe the constraints applied to embedded message values. For message-type fields, validation is performed

| Field | Type | Label | Description | | -- | - | -- | -- | | skip | | optional | Skip specifies that the validation rules of this field should not be evaluated | | required | | optional | Required specifies that this field must be set | <a name="validate-RepeatedRules"></a> RepeatedRules describe the constraints applied to `repeated` values | Field | Type | Label | Description | | -- | - | -- | -- | | min_items | | optional | MinItems specifies that this field must have the specified number of items at a minimum | | max_items | | optional | MaxItems specifies that this field must have the specified number of items at a maximum | | unique | | optional | Unique specifies that all elements in this field must be unique. This contraint is only applicable to scalar and enum types (messages are not supported). | | items | | optional | Items specifies the contraints to be applied to each item in the field. Repeated message fields will still execute validation against each item unless skip is specified here. | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-SFixed32Rules"></a> SFixed32Rules describes the constraints applied to `sfixed32` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-SFixed64Rules"></a> SFixed64Rules describes the constraints applied to `sfixed64` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is

| | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-SInt32Rules"></a> SInt32Rules describes the constraints applied to `sint32` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-SInt64Rules"></a> SInt64Rules describes the constraints applied to `sint64` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-StringRules"></a> StringRules describe the constraints applied to `string` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | len | | optional | Len specifies that this field must be the specified number of characters (Unicode code points). Note that the number of characters may differ from the number of bytes in the string. | | min_len | | optional | MinLen specifies that this field must be the specified number of characters (Unicode code points) at a minimum. Note that the number of characters may differ from the number of bytes in the string. | | max_len | | optional | MaxLen specifies that this field must be the specified number of characters (Unicode code points) at a

Note that the number of characters may differ from the number of bytes in the string. | | len_bytes | | optional | LenBytes specifies that this field must be the specified number of bytes | | min_bytes | | optional | MinBytes specifies that this field must be the specified number of bytes at a minimum | | max_bytes | | optional | MaxBytes specifies that this field must be the specified number of bytes at a maximum | | pattern | | optional | Pattern specifes that this field must match against the specified regular expression (RE2 syntax). The included expression should elide any delimiters. | | prefix | | optional | Prefix specifies that this field must have the specified substring at the beginning of the string. | | suffix | | optional | Suffix specifies that this field must have the specified substring at the end of the string. | | contains | | optional | Contains specifies that this field must have the specified substring anywhere in the string. | | not_contains | | optional | NotContains specifies that this field cannot have the specified substring anywhere in the string. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | email | | optional | Email specifies that the field must be a valid email address as defined by RFC 5322 | | hostname | | optional | Hostname specifies that the field must be a valid hostname as defined by RFC 1034. This constraint does not support internationalized domain names (IDNs). | | ip | | optional | Ip specifies that the field must be a valid IP (v4 or v6) address. Valid IPv6 addresses should not include surrounding square brackets. | | ipv4 | | optional | Ipv4 specifies that the field must be a valid IPv4 address. | | ipv6 | | optional | Ipv6 specifies that the field must be a valid IPv6 address. Valid IPv6 addresses should not include surrounding square brackets. | | uri | | optional | Uri specifies that the field must be a valid, absolute URI as defined by RFC 3986 | | uri_ref | | optional | UriRef specifies that the field must be a valid URI as defined by RFC 3986 and may be relative or absolute. | | address | | optional | Address specifies that the field must be either a valid hostname as defined by RFC 1034 (which does not support internationalized domain names or IDNs), or it can be a valid IP (v4 or v6). | | uuid | | optional | Uuid specifies that the field must be a valid UUID as defined by RFC 4122 | | wellknownregex | | optional | WellKnownRegex specifies a common well known pattern defined as a regex. | | strict | | optional | This applies to regexes HTTPHEADERNAME and HTTPHEADERVALUE to enable strict header validation. By default, this is true, and HTTP header validations are RFC-compliant. Setting to false will enable a looser validations that only disallows \r\n\0 characters, which can be used to bypass header matching rules. Default: true | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-TimestampRules"></a> TimestampRules describe the constraints applied exclusively to the

well-known type | Field | Type | Label | Description | | -- | - | -- | -- | | required | | optional | Required specifies that this field must be set | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive | | gte | | optional | Gte specifies that this field must be greater than the specified value, inclusive | | lt_now | | optional | LtNow specifies that this must be less than the current time. LtNow can only be used with the Within rule. | | gt_now | | optional | GtNow specifies that this must be greater than the current time. GtNow can only be used with the Within rule. | | within | | optional | Within specifies that this field must be within this duration of the current time. This constraint can be used alone or with the LtNow and GtNow rules. | <a name="validate-UInt32Rules"></a> UInt32Rules describes the constraints applied to `uint32` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value, inclusive. If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-UInt64Rules"></a> UInt64Rules describes the constraints applied to `uint64` values | Field | Type | Label | Description | | -- | - | -- | -- | | const | | optional | Const specifies that this field must be exactly the specified value | | lt | | optional | Lt specifies that this field must be less than the specified value, exclusive | | lte | | optional | Lte specifies that this field must be less than or equal to the specified value, inclusive | | gt | | optional | Gt specifies that this field must be greater than the specified value, exclusive. If the value of Gt is larger than a specified Lt or Lte, the range is reversed. | | gte | | optional | Gte specifies that this field must be greater than or equal to the specified value,

If the value of Gte is larger than a specified Lt or Lte, the range is reversed. | | in | | repeated | In specifies that this field must be equal to one of the specified values | | not_in | | repeated | NotIn specifies that this field cannot be equal to one of the specified values | | ignore_empty | | optional | IgnoreEmpty specifies that the validation rules of this field should be evaluated only if the field is not empty | <a name="validate-KnownRegex"></a> WellKnownRegex contain some well-known patterns. | Name | Number | Description | | - | | -- | | UNKNOWN | 0 | | | HTTPHEADERNAME | 1 | HTTP header name as defined by RFC 7230. | | HTTPHEADERVALUE | 2 | HTTP header value as defined by RFC 7230. | <a name="protoextensions_validate-proto-extensions"></a> | Extension | Type | Base | Number | Description | | | - | - | | -- | | rules | FieldRules | .google.protobuf.FieldOptions | 1071 | Rules specify the validations to be performed on this field. By default, no validation is performed against a field. | | disabled | bool | .google.protobuf.MessageOptions | 1071 | Disabled nullifies any validation rules for this message, including any message fields associated with it that do support validation. | | ignored | bool | .google.protobuf.MessageOptions | 1072 | Ignore skips generation of validation methods for this message. | | required | bool | .google.protobuf.OneofOptions | 1071 | Required ensures that exactly one the field options in a oneof is set; validation fails if no fields in the oneof are set. | <a name="proxy_proxy-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="proxy-ListenRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | channel | | | | <a name="proxy-ListenResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | extra | | | | <a name="proxy-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Listen | | stream | Listen streams database events. It signals that it is internally set up by sending an initial empty ListenResponse. | <a name="serverpfsserver_pfsserver-proto"></a> <p

href="#top">Top</a></p> <a name="pfsserver-CompactTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | inputs | | repeated | | | path_range | | | | <a name="pfsserver-CompactTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pfsserver-ConcatTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | inputs | | repeated | | <a name="pfsserver-ConcatTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pfsserver-GetFileURLTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | URL | | | | | Fileset | | | | | path_range | | | | <a name="pfsserver-GetFileURLTaskResult"></a> <a name="pfsserver-PathRange"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | lower | | | | | upper | | | | <a name="pfsserver-PutFileURLTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | dst | | | | | datum | | | | | URL | | | | | paths | | repeated | | | start_offset | | | | | end_offset | | | | <a name="pfsserver-PutFileURLTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="pfsserver-ShardTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | inputs | | repeated | | | path_range | | | | <a name="pfsserver-ShardTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | compact_tasks | | repeated | | <a name="pfsserver-ValidateTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | path_range | | | | <a name="pfsserver-ValidateTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | first | | | | | last | | | | | error | | | | | size_bytes | | | | <a name="serverworkercommon_common-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="common-Input"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | file_info | | | | | name | | | | | join_on | | | | | outer_join | | | | | group_by | | | | | lazy | | | | | branch | | | | | empty_files | | | | | s3 | | | If set, workers won&#39;t create an input directory for this input | <a name="serverworkerdatum_datum-proto"></a> <p

href="#top">Top</a></p> <a name="datum-ComposeTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetids | | repeated | | | auth_token | | | | <a name="datum-ComposeTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="datum-CrossTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetids | | repeated | | | basefileset_index | | | | | basefilesetpathrange | | | | | base_index | | | | | auth_token | | | | <a name="datum-CrossTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="datum-KeyTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | path_range | | | | | type | | | | | auth_token | | | | <a name="datum-KeyTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="datum-MergeTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetids | | repeated | | | path_range | | | | | type | | | | | auth_token | | | | <a name="datum-MergeTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="datum-Meta"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | inputs | | repeated | | | hash | | | | | state | | | | | reason | | | | | stats | | | | | index | | | | | image_id | | | | <a name="datum-PFSTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | input | | | | | path_range | | | | | base_index | | | | | auth_token | | | | <a name="datum-PFSTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | <a name="datum-SetSpec"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | number | | | | | size_bytes | | | | <a name="datum-Stats"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | process_stats | | | | | processed | | | | | skipped | | | | | total | | | | | failed | | | | | recovered | | | | | failed_id | | | | <a name="datum-KeyTask-Type"></a> | Name | Number | Description | | - | | -- | | JOIN | 0 | | | GROUP | 1 | | <a name="datum-MergeTask-Type"></a> | Name | Number | Description | | - | | -- | | JOIN | 0 | | | GROUP | 1 | | <a name="datum-State"></a> | Name | Number | Description | | - | | -- | | PROCESSED | 0 | | | FAILED | 1 | | | RECOVERED | 2 | | <a name="serverworkerpipelinetransformtransform-proto"></a> <p

href="#top">Top</a></p> <a name="pachyderm-worker-pipeline-transform-CreateDatumSetsTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | path_range | | | | | set_spec | | | | | auth_token | | | | <a name="pachyderm-worker-pipeline-transform-CreateDatumSetsTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | datum_sets | | repeated | | <a name="pachyderm-worker-pipeline-transform-CreateParallelDatumsTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | salt | | | | | filesetid | | | | | basefileset_id | | | | | path_range | | | | | auth_token | | | | <a name="pachyderm-worker-pipeline-transform-CreateParallelDatumsTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | stats | | | | <a name="pachyderm-worker-pipeline-transform-CreateSerialDatumsTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | salt | | | | | filesetid | | | | | basemetacommit | | | | | no_skip | | | | | path_range | | | | | auth_token | | | | <a name="pachyderm-worker-pipeline-transform-CreateSerialDatumsTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | filesetid | | | | | outputdeletefilesetid | | | | | metadeletefilesetid | | | | | stats | | | | <a name="pachyderm-worker-pipeline-transform-DatumSetTask"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job | | | | | filesetid | | | | | path_range | | | | | output_commit | | | | <a name="pachyderm-worker-pipeline-transform-DatumSetTaskResult"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | outputfileset_id | | | | | metafileset_id | | | | | stats | | | | <a name="storage_fileset-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="storage-AppendFile"></a> AppendFile will append the provided data to the file with the specified path. If a file with the specified path doesn&#39;t exist, it will be created. | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | | data | | | | <a name="storage-ComposeFilesetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_ids | | repeated | | | ttl_seconds | | | The TTL, in seconds, for the composite fileset that is created. | <a name="storage-ComposeFilesetResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_id | | | | <a name="storage-CreateFilesetRequest"></a> A CreateFilesetRequest corresponds to a single file modification. Supported file modifications are append and delete. A put / overwrite file modification can be performed by a delete followed by an append. TODO: Decide how to handle datums. | Field | Type | Label | Description | | -- | - | -- | -- | | append_file | | | | | delete_file | | | | <a name="storage-CreateFilesetResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_id | | | | <a name="storage-DeleteFile"></a> DeleteFile will delete the file with the specified path. If a file with the specified path doesn&#39;t exist, the delete will be a no-op. | Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | <a name="storage-FileFilter"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | path_range | | | Only emit files with paths in the provided path range. | | path_regex | | | Only emit files with paths that match the provided regular expression. | <a name="storage-PathRange"></a> PathRange is a range of paths. The range is inclusive, exclusive: [Lower, Upper). | Field | Type | Label | Description | | -- | - | -- | -- | | lower | | | | | upper | | | | <a name="storage-ReadFilesetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_id | | | | | filters | | repeated | Filters constrain which files are emitted. A file is only emitted if it makes it through all of the filters sequentially. | | empty_files | | | If true, then the file data will be omitted from the stream. | <a name="storage-ReadFilesetResponse"></a> A ReadFilesetResponse corresponds to a single chunk of data in a file. Small or empty files will be contained within a single message, while large files may be spread across multiple messages. For files spread across multiple messages, each message will have the same path and the content will be returned in append

| Field | Type | Label | Description | | -- | - | -- | -- | | path | | | | | data | | | | <a name="storage-RenewFilesetRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_id | | | | | ttl_seconds | | | The TTL, in seconds, for the fileset after renewal. | <a name="storage-ShardFilesetRequest"></a> If both numfiles and sizebytes are set, shards are created based on whichever threshold is surpassed first. If a shard configuration field (numfiles, sizebytes) is unset, the storage&#39;s default value is used. | Field | Type | Label | Description | | -- | - | -- | -- | | fileset_id | | | | | num_files | | | Number of files targeted in each shard. | | size_bytes | | | Size (in bytes) targeted for each shard. | <a name="storage-ShardFilesetResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | shards | | repeated | | <a name="storage-Fileset"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | CreateFileset | stream | | CreateFileset creates a fileset based on a stream of file modifications. A string identifier for the created fileset will be returned that can be used for subsequent fileset operations. Filesets have a fixed time-to-live (ttl), which is currently 10 minutes. Filesets needed longer than the ttl will need to be renewed. | | ReadFileset | | stream | ReadFileset reads a fileset. | | RenewFileset | | | RenewFileset renews a fileset. | | ComposeFileset | | | ComposeFileset composes a fileset. Composing a fileset involves combining one or more filesets into a single fileset. TODO: Explain how the filesets are layered and what that means for the order of file modifications. | | ShardFileset | | | ShardFileset shards a fileset. The shards of a fileset are returned as a list of path ranges that are disjoint and account for the full set of paths in the fileset. | <a name="task_task-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="taskapi-Group"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | namespace | | | | | group | | | | <a name="taskapi-ListTaskRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | group | | | | <a name="taskapi-TaskInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | | group | | | | | state | | | | | reason | | | | | input_type | | | | | input_data | | | | <a name="taskapi-State"></a> | Name | Number | Description | | - | | -- | | UNKNOWN | 0 | | | RUNNING | 1 | | | SUCCESS | 2 | | | FAILURE | 3 | | | CLAIMED | 4 | not a real state used by task logic | <a name="transaction_transaction-proto"></a> <p

href="#top">Top</a></p> <a name="transaction_v2-BatchTransactionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | requests | | repeated | | <a name="transaction_v2-DeleteAllRequest"></a> <a name="transaction_v2-DeleteTransactionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transaction | | | | <a name="transaction_v2-FinishTransactionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transaction | | | | <a name="transaction_v2-InspectTransactionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transaction | | | | <a name="transaction_v2-ListTransactionRequest"></a> <a name="transaction_v2-StartTransactionRequest"></a> <a name="transaction_v2-Transaction"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | id | | | | <a name="transaction_v2-TransactionInfo"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transaction | | | | | requests | | repeated | | | responses | | repeated | | | started | | | | | version | | | | <a name="transaction_v2-TransactionInfos"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | transaction_info | | repeated | | <a name="transaction_v2-TransactionRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | create_repo | | | Exactly one of these fields should be set | | delete_repo | | | | | start_commit | | | | | finish_commit | | | | | squashcommitset | | | | | create_branch | | | | | delete_branch | | | | | updatejobstate | | | | | stop_job | | | | | createpipelinev2 | | | | <a name="transaction_v2-TransactionResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | commit | | | At most, one of these fields should be set (most responses are empty) Only used for StartCommit - any way we can deterministically provide this before finishing the transaction? | <a name="transaction_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | BatchTransaction | | | Transaction rpcs | | StartTransaction | | | | | InspectTransaction | | | | | DeleteTransaction | | | | | ListTransaction | | | | | FinishTransaction | | | | | DeleteAll | | | | <a name="versionversionpbversion-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="versionpb_v2-Version"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | major | | | | | minor | | | | | micro | | | | | additional | | | | | git_commit | | | | | gittreemodified | | | | | build_date | | | | | go_version | | | | | platform | | | | <a name="versionpb_v2-API"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | GetVersion | | | | <a name="worker_worker-proto"></a> <p align="right"><a href="#top">Top</a></p> <a name="pachyderm-worker-CancelRequest"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | job_id | | | | | data_filters | | repeated | | <a name="pachyderm-worker-CancelResponse"></a> | Field | Type | Label | Description | | -- | - | -- | -- | | success | | | | <a name="pachyderm-worker-NextDatumRequest"></a> Error indicates that the processing of the current datum errored. Datum error semantics with datum batching enabled are similar to datum error semantics without datum batching enabled in that the datum may be retried, recovered, or result with a job failure. | Field | Type | Label | Description | | -- | - | -- | -- | | error | | | | <a name="pachyderm-worker-NextDatumResponse"></a> Env is a list of environment variables that should be set for the processing of the next

| Field | Type | Label | Description | | -- | - | -- | -- | | env | | repeated | | <a name="pachyderm-worker-Worker"></a> | Method Name | Request Type | Response Type | Description | | -- | | - | | | Status | | | | | Cancel | | | | | NextDatum | | | NextDatum should only be called by user code running in a pipeline with datum batching enabled. NextDatum will signal to the worker code that the user code is ready to proceed to the next datum. This generally means setting up the next datum&#39;s filesystem state and updating internal metadata similarly to datum processing in a normal pipeline. NextDatum is a synchronous operation, so user code should expect to block on this until the next datum is set up for processing. User code should generally be migratable to datum batching by wrapping it in a loop that calls next datum. | | .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | | -- | -- | | - | | -- | -- | | - | | <a name="double" /> double | | double | double | float | float64 | double | float | Float | | <a name="float" /> float | | float | float | float | float32 | float | float | Float | | <a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | <a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | | <a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | | <a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | | <a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | <a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | | <a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | | <a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | | <a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | <a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | | <a name="bool" /> bool | | bool | boolean | boolean | bool |