Repository: argoproj/argo (go)

Issue:
argo watch fails with an "EOF" error for workflows lasting > 90 seconds
Checklist:

 * [x] I've included the version.
 * [x] I've included reproduction steps.
 * [x] I've included the workflow YAML.
 * [x] I've included the logs.
 
**What happened**:

In Argo v2.7.5, for any long / involved workflows, both `argo submit --watch` and `argo watch` fail after about 80-90 seconds, with the following error:
```FATA[0117] EOF```
(the number in brackets varies)

**How to reproduce it (as minimally and precisely as possible)**:
Just run any script that lasts at least 80-90 seconds, and watch it. Here's a simple example:
```
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: long-workflow-
spec:
  entrypoint: plan
  templates:
  - name: plan
    container:
      image: alpine:latest
      command: [sh, -c, sleep 90]
```

When I run `argo submit --watch` on this, I get this:
```
Name:                long-workflow-66s8q
Namespace:           virtual
ServiceAccount:      default
Status:              Pending
Created:             Thu Apr 23 20:35:50 +0800 (now)
Name:                long-workflow-66s8q
Namespace:           virtual
ServiceAccount:      default
Status:              Running
Created:             Thu Apr 23 20:35:50 +0800 (now)
Started:             Thu Apr 23 20:35:50 +0800 (now)
Duration:            0 seconds

STEP                    TEMPLATE  PODNAME              DURATION  MESSAGE
 ◷ long-workflow-66s8q  plan      long-workflow-66s8q  0s
Name:                long-workflow-66s8q
Namespace:           virtual
ServiceAccount:      default
Status:              Running
Created:             Thu Apr 23 20:35:50 +0800 (1 second ago)
Started:             Thu Apr 23 20:35:50 +0800 (1 second ago)
Duration:            1 second

STEP                    TEMPLATE  PODNAME              DURATION  MESSAGE
 ◷ long-workflow-66s8q  plan      long-workflow-66s8q  1s        ContainerCreating
Name:                long-workflow-66s8q
Namespace:           virtual
ServiceAccount:      default
Status:              Running
Created:             Thu Apr 23 20:35:50 +0800 (3 seconds ago)
Started:             Thu Apr 23 20:35:50 +0800 (3 seconds ago)
Duration:            3 seconds

STEP                    TEMPLATE  PODNAME              DURATION  MESSAGE
 ● long-workflow-66s8q  plan      long-workflow-66s8q  3s
FATA[0067] EOF
```

Changing the 90 in the template to 70 solves the problem; the workflow completes in time.

Note that this also applies to `argo watch`, but the timer only starts from when I start watching.

---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.


Pull request description:
fix: Re-establish workflow watch on disconnect. Fixes #2796
Checklist:

* [x] Either (a) I've created an [enhancement proposal](https://github.com/argoproj/argo/issues/new/choose) and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
* [x] The title of the PR is (a) [conventional](https://www.conventionalcommits.org/en/v1.0.0/), (b) states what changed, and (c) suffixes the related issues number. E.g. `"fix(controller): Updates such and such. Fixes #1234"`.  
* [x] I've signed the CLA.
* [x] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
* [x] My builds are green. Try syncing with master if they are not. 
* [x] My organization is added to USERS.md.

Fixes #2796

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 452a09b60..6486b729f 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -154,7 +154,7 @@ jobs:
             make test
       - run: |
           cp sonar-project.properties dist/
-          if [ $CIRCLE_BRANCH != master ]; then
+          if [ "$(echo "$CIRCLE_BRANCH" | grep 'master\|release-')" = '' ]; then
             echo "sonar.pullrequest.branch=$CIRCLE_BRANCH" >> sonar-project.properties
             echo "sonar.pullrequest.key=$CIRCLE_PR_NUMBER" >> sonar-project.properties
           fi
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index 49c4a6066..e6fb63fcf 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -21,6 +21,7 @@ Checklist:
 **Anything else we need to know?**:
 
 **Environment**:
+
 - Argo version:
 ```
 $ argo version
@@ -31,16 +32,22 @@ $ kubectl version -o yaml
 ```
 
 **Other debugging information (if applicable)**:
+
 - workflow result:
+
 ```
-argo get <workflowname>
+argo --loglevel DEBUG get <workflowname>
 ```
+
 - executor logs:
+
 ```
 kubectl logs <failedpodname> -c init
 kubectl logs <failedpodname> -c wait
 ```
+
 - workflow-controller logs:
+
 ```
 kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
 ```
diff --git a/cmd/argo/commands/root.go b/cmd/argo/commands/root.go
index 8487a940a..f9f54d70d 100644
--- a/cmd/argo/commands/root.go
+++ b/cmd/argo/commands/root.go
@@ -4,8 +4,10 @@ import (
 	"fmt"
 
 	"github.com/argoproj/pkg/cli"
+	log "github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
 
+	"github.com/argoproj/argo"
 	"github.com/argoproj/argo/cmd/argo/commands/clustertemplate"
 
 	"github.com/argoproj/argo/cmd/argo/commands/auth"
@@ -62,10 +64,16 @@ If you're using the Argo Server (e.g. because you need large workflow support or
 
 	// global log level
 	var logLevel string
+	var verbose bool
 	command.PersistentPreRun = func(cmd *cobra.Command, args []string) {
+		if verbose {
+			logLevel = "debug"
+		}
 		cli.SetLogLevel(logLevel)
+		log.WithField("version", argo.GetVersion()).Debug("CLI version")
 	}
 	command.PersistentFlags().StringVar(&logLevel, "loglevel", "info", "Set the logging level. One of: debug|info|warn|error")
+	command.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Enabled verbose logging, i.e. --loglevel debug")
 
 	return command
 }
diff --git a/docs/variables.md b/docs/variables.md
index d1bd848ab..2a0432a2f 100644
--- a/docs/variables.md
+++ b/docs/variables.md
@@ -14,8 +14,8 @@ The following variables are made available to reference various metadata of a wo
 |----------|------------|
 | `steps.<STEPNAME>.ip` | IP address of a previous daemon container step |
 | `steps.<STEPNAME>.status` | Phase status of any previous step |
+| `steps.<STEPNAME>.exitCode` | Exit code of any previous script or container step |
 | `steps.<STEPNAME>.outputs.result` | Output result of any previous container or script step |
-| `steps.<STEPNAME>.outputs.exitCode` | Exit code of any previous script step |
 | `steps.<STEPNAME>.outputs.parameters.<NAME>` | Output parameter of any previous step |
 | `steps.<STEPNAME>.outputs.artifacts.<NAME>` | Output artifact of any previous step |
 
@@ -24,8 +24,8 @@ The following variables are made available to reference various metadata of a wo
 |----------|------------|
 | `tasks.<TASKNAME>.ip` | IP address of a previous daemon container task |
 | `tasks.<TASKNAME>.status` | Phase status of any previous task |
+| `tasks.<TASKNAME>.exitCode` | Exit code of any previous script or container task |
 | `tasks.<TASKNAME>.outputs.result` | Output result of any previous container or script task |
-| `tasks.<TASKNAME>.outputs.exitCode` | Exit code of any previous script task |
 | `tasks.<TASKNAME>.outputs.parameters.<NAME>` | Output parameter of any previous task |
 | `tasks.<TASKNAME>.outputs.artifacts.<NAME>` | Output artifact of any previous task |
 
@@ -72,6 +72,7 @@ For `Template`-level metrics:
 |----------|------------|
 | `workflow.name` | Workflow name |
 | `workflow.namespace` | Workflow namespace |
+| `workflow.serviceAccountName` | Workflow service account name |
 | `workflow.uid` | Workflow UID. Useful for setting ownership reference to a resource, or a unique artifact location |
 | `workflow.parameters.<NAME>` | Input parameter to the workflow |
 | `workflow.parameters` | All input parameters to the workflow as a JSON string |
diff --git a/examples/exit-code-output-variable.yaml b/examples/exit-code-output-variable.yaml
index 42a4fbd38..c779f7241 100644
--- a/examples/exit-code-output-variable.yaml
+++ b/examples/exit-code-output-variable.yaml
@@ -16,7 +16,8 @@ spec:
             arguments:
               parameters:
                 - name: exitCode
-                  value: "{{steps.failing-container.outputs.exitCode}}"
+                  value: "{{steps.failing-container.exitCode}}"
+
     - name: failing-container
       script:
         image: alpine:3.6
diff --git a/pkg/apiclient/apiclient.go b/pkg/apiclient/apiclient.go
index 6d657ba5b..4f5033bac 100644
--- a/pkg/apiclient/apiclient.go
+++ b/pkg/apiclient/apiclient.go
@@ -3,6 +3,7 @@ package apiclient
 import (
 	"context"
 
+	log "github.com/sirupsen/logrus"
 	"k8s.io/client-go/tools/clientcmd"
 
 	clusterworkflowtmplpkg "github.com/argoproj/argo/pkg/apiclient/clusterworkflowtemplate"
@@ -38,6 +39,7 @@ func NewClient(argoServer string, authSupplier func() string, clientConfig clien
 }
 
 func NewClientFromOpts(opts Opts) (context.Context, Client, error) {
+	log.WithField("opts", opts).Debug("Client options")
 	if opts.ArgoServerOpts.URL != "" {
 		return newArgoServerClient(opts.ArgoServerOpts, opts.AuthSupplier())
 	} else {
diff --git a/server/workflow/workflow_server.go b/server/workflow/workflow_server.go
index e64103094..7045bced1 100644
--- a/server/workflow/workflow_server.go
+++ b/server/workflow/workflow_server.go
@@ -155,7 +155,9 @@ func (s *workflowServer) WatchWorkflows(req *workflowpkg.WatchWorkflowsRequest,
 	if req.ListOptions != nil {
 		opts = *req.ListOptions
 	}
-	watch, err := wfClient.ArgoprojV1alpha1().Workflows(req.Namespace).Watch(s.withInstanceID(opts))
+	opts = s.withInstanceID(opts)
+	wfIf := wfClient.ArgoprojV1alpha1().Workflows(req.Namespace)
+	watch, err := wfIf.Watch(opts)
 	if err != nil {
 		return err
 	}
@@ -163,45 +165,49 @@ func (s *workflowServer) WatchWorkflows(req *workflowpkg.WatchWorkflowsRequest,
 	ctx := ws.Context()
 
 	log.Debug("Piping events to channel")
+	defer log.Debug("Result channel done")
 
-	for next := range watch.ResultChan() {
+	for {
 		select {
 		case <-ctx.Done():
 			return ctx.Err()
-		default:
-		}
-		log.Debug("Received event")
-		wf, ok := next.Object.(*v1alpha1.Workflow)
-		if !ok {
-			return fmt.Errorf("watch object was not a workflow %v", reflect.TypeOf(next.Object))
-		}
-		logCtx := log.WithFields(log.Fields{"workflow": wf.Name, "type": next.Type, "phase": wf.Status.Phase})
-		err := packer.DecompressWorkflow(wf)
-		if err != nil {
-			return err
-		}
-		if wf.Status.IsOffloadNodeStatus() {
-			if s.offloadNodeStatusRepo.IsEnabled() {
-				offloadedNodes, err := s.offloadNodeStatusRepo.Get(string(wf.UID), wf.GetOffloadNodeStatusVersion())
+		case event, open := <-watch.ResultChan():
+			if !open {
+				log.Info("Re-establishing workflow watch")
+				watch, err = wfIf.Watch(opts)
 				if err != nil {
 					return err
 				}
-				wf.Status.Nodes = offloadedNodes
-			} else {
-				log.WithFields(log.Fields{"namespace": wf.Namespace, "name": wf.Name}).Warn(sqldb.OffloadNodeStatusDisabled)
+				continue
+			}
+			log.Debug("Received event")
+			wf, ok := event.Object.(*v1alpha1.Workflow)
+			if !ok {
+				return fmt.Errorf("watch object was not a workflow %v", reflect.TypeOf(event.Object))
+			}
+			logCtx := log.WithFields(log.Fields{"workflow": wf.Name, "type": event.Type, "phase": wf.Status.Phase})
+			err := packer.DecompressWorkflow(wf)
+			if err != nil {
+				return err
+			}
+			if wf.Status.IsOffloadNodeStatus() {
+				if s.offloadNodeStatusRepo.IsEnabled() {
+					offloadedNodes, err := s.offloadNodeStatusRepo.Get(string(wf.UID), wf.GetOffloadNodeStatusVersion())
+					if err != nil {
+						return err
+					}
+					wf.Status.Nodes = offloadedNodes
+				} else {
+					log.WithFields(log.Fields{"namespace": wf.Namespace, "name": wf.Name}).Warn(sqldb.OffloadNodeStatusDisabled)
+				}
+			}
+			logCtx.Debug("Sending event")
+			err = ws.Send(&workflowpkg.WorkflowWatchEvent{Type: string(event.Type), Object: wf})
+			if err != nil {
+				return err
 			}
 		}
-		logCtx.Debug("Sending event")
-		err = ws.Send(&workflowpkg.WorkflowWatchEvent{Type: string(next.Type), Object: wf})
-		if err != nil {
-			return err
-		}
-
 	}
-
-	log.Debug("Result channel done")
-
-	return nil
 }
 
 func (s *workflowServer) DeleteWorkflow(ctx context.Context, req *workflowpkg.WorkflowDeleteRequest) (*workflowpkg.WorkflowDeleteResponse, error) {
diff --git a/util/logs/workflow-logger.go b/util/logs/workflow-logger.go
index 85d49e86d..ca1e76c19 100644
--- a/util/logs/workflow-logger.go
+++ b/util/logs/workflow-logger.go
@@ -166,6 +166,7 @@ func WorkflowLogs(ctx context.Context, wfClient versioned.Interface, kubeClient
 							logCtx.Error(err)
 							return
 						}
+						continue
 					}
 					wf, ok := event.Object.(*wfv1.Workflow)
 					if !ok {
@@ -198,6 +199,7 @@ func WorkflowLogs(ctx context.Context, wfClient versioned.Interface, kubeClient
 							logCtx.Error(err)
 							return
 						}
+						continue
 					}
 					pod, ok := event.Object.(*corev1.Pod)
 					if !ok {
diff --git a/workflow/common/common.go b/workflow/common/common.go
index 433114731..083ddbe83 100644
--- a/workflow/common/common.go
+++ b/workflow/common/common.go
@@ -114,6 +114,8 @@ const (
 	GlobalVarWorkflowName = "workflow.name"
 	// GlobalVarWorkflowNamespace is a global workflow variable referencing the workflow's metadata.namespace field
 	GlobalVarWorkflowNamespace = "workflow.namespace"
+	// GlobalVarWorkflowServiceAccountName is a global workflow variable referencing the workflow's spec.serviceAccountName field
+	GlobalVarWorkflowServiceAccountName = "workflow.serviceAccountName"
 	// GlobalVarWorkflowUID is a global workflow variable referencing the workflow's metadata.uid field
 	GlobalVarWorkflowUID = "workflow.uid"
 	// GlobalVarWorkflowStatus is a global workflow variable referencing the workflow's status.phase field
diff --git a/workflow/controller/operator.go b/workflow/controller/operator.go
index e24334f16..f756bf770 100644
--- a/workflow/controller/operator.go
+++ b/workflow/controller/operator.go
@@ -413,6 +413,7 @@ func (woc *wfOperationCtx) getWorkflowDeadline() *time.Time {
 func (woc *wfOperationCtx) setGlobalParameters() {
 	woc.globalParams[common.GlobalVarWorkflowName] = woc.wf.ObjectMeta.Name
 	woc.globalParams[common.GlobalVarWorkflowNamespace] = woc.wf.ObjectMeta.Namespace
+	woc.globalParams[common.GlobalVarWorkflowServiceAccountName] = woc.wf.Spec.ServiceAccountName
 	woc.globalParams[common.GlobalVarWorkflowUID] = string(woc.wf.ObjectMeta.UID)
 	woc.globalParams[common.GlobalVarWorkflowCreationTimestamp] = woc.wf.ObjectMeta.CreationTimestamp.String()
 	if woc.wf.Spec.Priority != nil {
diff --git a/workflow/validate/validate.go b/workflow/validate/validate.go
index 732c6fb0f..43e6739ec 100644
--- a/workflow/validate/validate.go
+++ b/workflow/validate/validate.go
@@ -59,6 +59,7 @@ func newTemplateValidationCtx(wf *wfv1.Workflow, opts ValidateOpts) *templateVal
 	globalParams := make(map[string]string)
 	globalParams[common.GlobalVarWorkflowName] = placeholderGenerator.NextPlaceholder()
 	globalParams[common.GlobalVarWorkflowNamespace] = placeholderGenerator.NextPlaceholder()
+	globalParams[common.GlobalVarWorkflowServiceAccountName] = placeholderGenerator.NextPlaceholder()
 	globalParams[common.GlobalVarWorkflowUID] = placeholderGenerator.NextPlaceholder()
 	return &templateValidationCtx{
 		ValidateOpts: opts,


Test patch:
diff --git a/test/e2e/manifests/mixins/argo-server-deployment.yaml b/test/e2e/manifests/mixins/argo-server-deployment.yaml
index c831895f4..f4ce656a9 100644
--- a/test/e2e/manifests/mixins/argo-server-deployment.yaml
+++ b/test/e2e/manifests/mixins/argo-server-deployment.yaml
@@ -8,12 +8,11 @@ spec:
       containers:
         - name: argo-server
           args:
+            - -v
             - server
             - --namespaced
             - --auth-mode
             - client
-            - --loglevel
-            - debug
             - --secure
           imagePullPolicy:  Never
           env:
diff --git a/workflow/controller/operator_test.go b/workflow/controller/operator_test.go
index 5d731e204..df536c315 100644
--- a/workflow/controller/operator_test.go
+++ b/workflow/controller/operator_test.go
@@ -2970,3 +2970,48 @@ func TestNestedStepGroupGlobalParams(t *testing.T) {
 	assert.Equal(t, "hello world", *woc.wf.Status.Outputs.Parameters[0].Value)
 	assert.Equal(t, "global-param", woc.wf.Status.Outputs.Parameters[0].Name)
 }
+
+var globalVariablePlaceholders = `
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  name: output-value-global-variables-wf
+  namespace: testNamespace
+spec:
+  serviceAccountName: testServiceAccountName
+  entrypoint: tell-workflow-global-variables
+  templates:
+  - name: tell-workflow-global-variables
+    outputs:
+      parameters:
+      - name: namespace
+        value: "{{workflow.namespace}}"
+      - name: serviceAccountName
+        value: "{{workflow.serviceAccountName}}"
+    container:
+      image: busybox
+`
+
+func TestResolvePlaceholdersInGlobalVariables(t *testing.T) {
+	wf := unmarshalWF(globalVariablePlaceholders)
+	woc := newWoc(*wf)
+	woc.artifactRepository.S3 = new(config.S3ArtifactRepository)
+	woc.operate()
+	assert.Equal(t, wfv1.NodeRunning, woc.wf.Status.Phase)
+	pods, err := woc.controller.kubeclientset.CoreV1().Pods(wf.ObjectMeta.Namespace).List(metav1.ListOptions{})
+	assert.NoError(t, err)
+	assert.True(t, len(pods.Items) > 0, "pod was not created successfully")
+
+	templateString := pods.Items[0].ObjectMeta.Annotations["workflows.argoproj.io/template"]
+	var template wfv1.Template
+	err = json.Unmarshal([]byte(templateString), &template)
+	assert.NoError(t, err)
+	namespaceValue := template.Outputs.Parameters[0].Value
+	assert.NotNil(t, namespaceValue)
+	assert.NotEmpty(t, *namespaceValue)
+	assert.Equal(t, "testNamespace", *namespaceValue)
+	serviceAccountNameValue := template.Outputs.Parameters[1].Value
+	assert.NotNil(t, serviceAccountNameValue)
+	assert.NotEmpty(t, *serviceAccountNameValue)
+	assert.Equal(t, "testServiceAccountName", *serviceAccountNameValue)
+}
diff --git a/workflow/validate/validate_test.go b/workflow/validate/validate_test.go
index 6c9635674..939aa36a1 100644
--- a/workflow/validate/validate_test.go
+++ b/workflow/validate/validate_test.go
@@ -2339,3 +2339,49 @@ func TestInvalidMetricHelp(t *testing.T) {
 	_, err := validate(invalidMetricHelp)
 	assert.EqualError(t, err, "templates.whalesay metric 'metric_name' must contain a help string under 'help: ' field")
 }
+
+var globalVariables = `
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: global-variables-
+spec:
+  priority: 100
+  entrypoint: test-workflow
+
+  templates:
+  - name: test-workflow
+    steps:
+    - - name: step1
+        template: whalesay
+        arguments:
+          parameters:
+          - name: name
+            value: "{{workflow.name}}"
+          - name: namespace
+            value: "{{workflow.namespace}}"
+          - name: serviceAccountName
+            value: "{{workflow.serviceAccountName}}"
+          - name: uid
+            value: "{{workflow.uid}}"
+          - name: priority
+            value: "{{workflow.priority}}"    
+
+  - name: whalesay
+    inputs:
+      parameters:
+      - name: name
+      - name: namespace
+      - name: serviceAccountName
+      - name: uid
+      - name: priority
+    container:
+      image: docker/whalesay:latest
+      command: [cowsay]
+      args: ["name: {{inputs.parameters.name}} namespace: {{inputs.parameters.namespace}} serviceAccountName: {{inputs.parameters.serviceAccountName}} uid: {{inputs.parameters.uid}} priority: {{inputs.parameters.priority}}"]
+`
+
+func TestWorfklowGlobalVariables(t *testing.T) {
+	_, err := validate(globalVariables)
+	assert.NoError(t, err)
+}