Project: Vitess (App Definition and Development / Database)
File: TabletServerParamsAsYAML.md

This issue is an expansion of #5791, and covers the details of how well use yaml to implement a hierarchy of parameters for initializing and customizing TabletServer components within a process. Using the yaml approach, we intend to solve the following problems: Multiple tablet servers need to exist within a process. Badly named command line parameters: The new yaml specifications will have simpler and easier to remember parameter names. Sections for improved readability. Unreasonable default values: The new specs will introduce a simplified approach that will save the user from tweaking too many variables without knowing their consequences. Repetition: If multiple TabletServers have to share the same parameter values, they should not be repeated. Backward compatibility: The system must be backward compatible. We will introduce two new command-line options that will be specified like this: ``` -defaults=defaults.yaml -tablets=tablets.yaml ``` The defaults option allows the user to specify the default settings for all tablets. The tablets option allows the user to specify tablet specific options. Well provide predefined defaults files that people can use as starting points. Well start with something simple like small.yaml, medium.yaml, and large.yaml. We explored the use of protobufs for converting to and from yaml. However, the protobuf JSON implementation, which was required to correctly parse enums, was unable to preserve the original values for sub-objects. This was a showstopper for making defaults work. The existing `tabletenv.TabletConfig` data structure will be converted into a struct with the new names and appropriate JSON tags. The following section lists all the properties that can be specified in a yaml file along with the existing defaults. During implementation, well need to decide between creating a universal protobuf that represents this structure vs converting the original one into something more usable. Convention used: same as the one followed by kubernetes, and well be making use of https://github.com/kubernetes-sigs/yaml for the implementation. Note that certain properties (like tabletID) are only applicable to tablet-specific files. ``` tabletID: zone-1234 init: dbName: # initdbname_override keyspace: # init_keyspace shard: # init_shard tabletType: # inittablettype timeoutSeconds: 60 # init_timeout db: socket: # db_socket host: # db_host port: 0 # db_port charSet: # db_charset flags: 0 # db_flags flavor: # db_flavor sslCa: # dbsslca sslCaPath: # dbsslca_path sslCert: # dbsslcert sslKey: # dbsslkey serverName: # dbservername connectTimeoutMilliseconds: 0 # dbconnecttimeout_ms app: user: vtapp # dbapp_user password: # dbapppassword useSsl: true # dbappuse_ssl preferSocket: true dba: user: vtdba # dbdba_user password: # dbdbapassword useSsl: true # dbdbause_ssl preferSocket: true filtered: user: vtfiltered # dbfiltered_user

# dbfilteredpassword useSsl: true # dbfiltereduse_ssl preferSocket: true repl: user: vtrepl # dbrepl_user password: # dbreplpassword useSsl: true # dbrepluse_ssl preferSocket: true appdebug: user: vtappdebug # dbappdebug_user password: # dbappdebugpassword useSsl: true # dbappdebuguse_ssl preferSocket: true allprivs: user: vtallprivs # dballprivs_user password: # dballprivspassword useSsl: true # dballprivsuse_ssl preferSocket: true oltpReadPool: size: 16 # queryserver-config-pool-size timeoutSeconds: 0 # queryserver-config-query-pool-timeout idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout prefillParallelism: 0 # queryserver-config-pool-prefill-parallelism olapReadPool: size: 200 # queryserver-config-stream-pool-size timeoutSeconds: 0 # queryserver-config-query-pool-timeout idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout prefillParallelism: 0 # queryserver-config-stream-pool-prefill-parallelism txPool: size: 20 # queryserver-config-transaction-cap timeoutSeconds: 1 # queryserver-config-txpool-timeout idleTimeoutSeconds: 1800 # queryserver-config-idle-timeout prefillParallelism: 0 # queryserver-config-transaction-prefill-parallelism oltp: queryTimeoutSeconds: 30 # queryserver-config-query-timeout txTimeoutSeconds: 30 # queryserver-config-transaction-timeout maxRows: 10000 # queryserver-config-max-result-size warnRows: 0 # queryserver-config-warn-result-size hotRowProtection: mode: disable|dryRun|enable # enablehotrowprotection, enablehotrowprotectiondryrun maxQueueSize: 20 # hotrowprotectionmaxqueue_size maxGlobalQueueSize: 1000 # hotrowprotectionmaxglobalqueuesize maxConcurrency: 5 # hotrowprotectionconcurrenttransactions consolidator: enable|disable|notOnPrimary # enable-consolidator, enable-consolidator-replicas heartbeatIntervalMilliseconds: 0 # heartbeatenable, heartbeatinterval shutdownGracePeriodSeconds: 0 # transactionshutdowngrace_period passthroughDML: false # queryserver-config-passthrough-dmls streamBufferSize: 32768 # queryserver-config-stream-buffer-size queryCacheSize: 5000 # queryserver-config-query-cache-size schemaReloadIntervalSeconds: 1800 # queryserver-config-schema-reload-time watchReplication: false # watchreplicationstream terseErrors: false # queryserver-config-terse-errors messagePostponeParallelism: 4 # queryserver-config-message-postpone-cap sanitizeLogMessages: false # sanitizelogmessages ``` There are also other global parameters. VTTablet has a total of 405 flags. We may have to later move some of them into these yamls. We'll use the unified tabletenv.TabletConfig data structure to load defaults as well as tablet-specific values. The following changes will be made: TabletConfig will be changed to match the above specifications. The existing flags will be changed to update the values in a global TabletConfig. In case of type mismatch (like time.Duration vs a "Seconds" variable), an extra step will be performed after parsing to convert the flag variables into TabletConfig members. The code will be changed to honor the new TabletConfig members. After the flag.Parse step, a copy of the global Config will be used as input to load the defaults file. This means that any command line flags that are not overridden in the yaml files will be preserved. This behavior is needed to support backward compatibility in case we decide to move more flags into the yaml. For each tablet entry, we create a copy of the result TabletConfig and read the tablet yaml into those, which will set the tablet-specific values. This will then be used to instantiate a TabletServer. The exact format of the tablets.yaml file is not fully finalized. Our options are to allow a list of files where each is for a single tablet, or, to require only one file containing a dictionary of tablet-specific overrides.