Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: context.md

DolphinScheduler provides the ability to refer to each other between parameters, including local parameters refer to global parameters and upstream and downstream parameter transfer. Due to the existence of references, it involves the priority of parameters when the parameter names are the same. see also The premise of local tasks referring global parameters is that you have already defined . The usage is similar to the usage in , but the value of the parameter needs to be configured as the key of the global parameter. DolphinScheduler allows parameter transfer between tasks. Currently, transfer direction only supports one-way transfer from upstream to downstream. The task types that support this feature are When defining an upstream node, if there is a need to transmit the result of that node to a dependency related downstream node. You need to set an `OUT` direction parameter to [Custom Parameters] of the [Current Node Settings]. If it is a SubProcess node, there is no need to set a parameter in [Current Node Settings], but an `OUT` direction parameter needs to be set in the workflow definition of the subprocess. The value of upstream parameter can be updated in downstream node in the same way as . Upstream parameter will be override when defining parameter with the same name in downstream node. Note: If there are no dependencies between nodes, local parameters cannot be passed upstream. This sample shows how to use the parameter passing function. Create local parameters and assign them to downstream through the SHELL task. The SQL task completes the query operation by obtaining the parameters of the upstream task. The user needs to pass the parameter when creating the shell script, the output statement format is `'${setValue(key=value)}'`, the key is the `prop` of the corresponding parameter, and value is the value of the parameter. Create a Node_A task, add output and value parameters to the custom parameters, and write the following script: Parameter Description: value: The direction selection is IN, and the value is 66 output: The direction is selected as OUT, assigned through the script`'${setValue(output=1)}'`, and passed to the downstream parameters When the SHELL node is defined, the log detects the format of `${setValue(output=1)}`, it will assign 1 to output, and the downstream node can directly use the value of the variable output. Similarly, you can find the corresponding node instance on the [Workflow Instance] page, and then you can view the value of this variable. Create the NodeB task, which is mainly used to test and output the parameters passed by the upstream task NodeA. When the SHELL task is completed, we can use the output passed upstream as the query object for the SQL. The id of the query is renamed to ID and is output as a parameter. Note: If the result of the SQL node has only one row, one or multiple fields, the name of the `prop` needs to be the same as the field name. The data type can choose structure except `LIST`. The parameter assigns the value according to the same column name in the SQL query

If the result of the SQL node has multiple rows, one or more fields, the name of the `prop` needs to be the same as the field name. Choose the data type structure as `LIST`, and the SQL query result will be converted to `LIST<VARCHAR>`, and forward to convert to JSON as the parameter value. Click on the Save workflow icon and set the global parameters output and value. After the workflow is created, run the workflow online and view its running results. The result of Node_A is as follows: The result of Node_B is as follows: The result of Node_mysql is as follows: Even though output is assigned a value of 1 in NodeA's script, the log still shows a value of 100. But according to the principle from : `Startup Parameter > Local Parameter > Parameter Context > Global Parameter`, the output value in NodeB is 1. It proves that the output parameter is passed in the workflow with reference to the expected value, and the query operation is completed using this value in Node_mysql. But the output value 66 only shows in the Node_A, the reason is that the direction of value is selected as IN, and only when the direction is OUT will it be defined as a variable output. Use `print('${setValue(key=%s)}' % value)`, DolphinScheduler will capture the `${setValue(key=value}` in the output to capture the parameters and pass them downstream. For example Attention: When the variable value contains the `\n` identifier, such as ` value = "hello \n world" `, value needs to be carried out in a special way. You need to use `print('${setValue(key=%s)}' % repr(value))`, otherwise the argument cannot be passed to the subsequent flow. In the workflow definition of the subprocess, define `OUT` direction parameters as output parameters, and these parameters can be passed to the downstream tasks of the subprocess node. Create an A task in the workflow definition of the subprocess, add var1 and var2 parameters to the custom parameters, and write the following script: Save the subprocess_example1 workflow and set the global parameters var1. Create a subprocess task in a new workflow, and use the subprocessexample1 workflow as the sub-node. Create a shell task as a downstream task of the sub_process task, and write the following script: Save the workflow and run it. The result of the downstream task is as follows: Although the two parameters var1 and var2 are output in the A task, only the `OUT` parameter var1 is defined in the workflow definition, and the downstream task successfully outputs var1. It proves that the var1 parameter is passed in the workflow with reference to the expected value. Different programming languages may use different logging frameworks in Kubernetes tasks. To be compatible with these frameworks, DolphinScheduler provides a universal logging data format `${(key=value)}` or `#{(key=value)}`. Users can output log data in the format in the terminal logs of their applications, where `key` is the corresponding parameter prop and `value` is the value of that parameter. DolphinScheduler will capture the `${(key=value)}` or `#{(key=value)}` in the output logs to capture the parameters and pass them downstream. For example Another special consideration, not always can DolphinScheduler collect pod logs, if the user redirects the log output stream, DolphinScheduler can not collect logs for use and can not use the output parameter, either.