Repository: argoproj/argo (go)

Issue:
Concaternated 20GB GZIP Input Archives Cause Stalling
Checklist:

 * [x] I've included the version.
 * [x] I've included reproduction steps.
 * [x] I've included the workflow YAML.
 * [x] I've included the logs.
 
**What happened**:
When a big (20-40gb) input artifact is used as an input archive the init script hangs for a long time. 

**What you expected to happen**:
The download from the object store to finish promptly.

**How to reproduce it (as minimally and precisely as possible)**:
Make a really big gzip file by concatenating an existing, big gzip file (say 20GB) making the overall file 40GB. Use this as an input to a workflow step.

**Anything else we need to know?**:
For a GZIP file that if formed from the concatenation of multiple gzipped files (e.g. bgzip output) `tar -tf` will check each 'part' to make sure it doesn't contain a tar somewhere inside. Therefore, `tar -tf` will return immediately for a gzipped tar or not a gzipped tar but will take a long time when handling concatenated gzips.

**Environment**:
- Argo version:
```
argo: v2.4.2
  BuildDate: 2019-10-21T18:39:52Z
  GitCommit: 675c66267f0c916de0f233d8101aa0646acb46d4
  GitTreeState: clean
  GitTag: v2.4.2
  GoVersion: go1.11.5
  Compiler: gc
  Platform: darwin/amd64
```
- Kubernetes version :
```
v1.14.9
```

**Other debugging information (if applicable)**:

- executor logs:
```
time="2020-02-03T16:03:05Z" level=info msg="Start loading input artifacts..."
time="2020-02-03T16:03:05Z" level=info msg="Downloading artifact: r1"
time="2020-02-03T16:03:05Z" level=info msg="Specified artifact path /ephemeral/xyz/xyz.fastq.gz overlaps with volume mount at /var/lib/edico. Extracting to volume mount"
time="2020-02-03T16:03:05Z" level=info msg="S3 Load path: /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp, key: xyz/clean_data/xyz.fastq.gz"
time="2020-02-03T16:03:05Z" level=info msg="Creating minio client s3.amazonaws.com using static credentials"
time="2020-02-03T16:03:05Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: dantenostrum, key: xyz/clean_data/xyz.fastq.gz) to /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp"
time="2020-02-03T16:07:59Z" level=info msg="[tar -tf /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp]"
time="2020-02-03T16:25:09Z" level=info msg="Successfully download file: /mainctrfs/ephemeral/xyz/xyz.fastq.gz"
time="2020-02-03T16:25:09Z" level=info msg="Downloading artifact: r2"
time="2020-02-03T16:25:09Z" level=info msg="Specified artifact path /ephemeral/xyz/xyz.fastq.gz overlaps with volume mount at /var/lib/edico. Extracting to volume mount"
time="2020-02-03T16:25:09Z" level=info msg="S3 Load path: /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp, key: xyz/clean_data/xyz.fastq.gz"
time="2020-02-03T16:25:09Z" level=info msg="Creating minio client s3.amazonaws.com using static credentials"
time="2020-02-03T16:25:09Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: dantenostrum, key: xyz/clean_data/xyz.fastq.gz) to /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp"
time="2020-02-03T16:30:11Z" level=info msg="[tar -tf /mainctrfs/ephemeral/xyz/xyz.fastq.gz.tmp]"
time="2020-02-03T16:47:38Z" level=info msg="Successfully download file: /mainctrfs/ephemeral/xyz/xyz.fastq.gz"
time="2020-02-03T16:47:38Z" level=info msg="Alloc=4675 TotalAlloc=24109 Sys=70846 NumGC=28 Goroutines=4"
```

---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.


Pull request:
fix(artifacts): Allow tar check to be ignored. Fixes #2140
Checklist:

* [x] Either (a) I've created an [enhancement proposal](https://github.com/argoproj/argo/issues/new/choose) and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
* [x] The title of the PR is (a) [conventional](https://www.conventionalcommits.org/en/v1.0.0/), (b) states what changed, and (c) suffixes the related issues number. E.g. `"fix(controller): Updates such and such. Fixes #1234"`.  
* [x] I've signed the CLA.
* [x] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
* [x] My builds are green. Try syncing with master if they are not. 
* [x] My organization is added to USERS.md.

Fixes #2140

Patch:
diff --git a/pkg/apis/workflow/v1alpha1/workflow_types.go b/pkg/apis/workflow/v1alpha1/workflow_types.go
index 2429afe0c..7026473f6 100644
--- a/pkg/apis/workflow/v1alpha1/workflow_types.go
+++ b/pkg/apis/workflow/v1alpha1/workflow_types.go
@@ -1720,6 +1720,13 @@ func (args *Arguments) GetParameterByName(name string) *Parameter {
 	return nil
 }
 
+func (a *Artifact) GetArchive() *ArchiveStrategy {
+	if a == nil || a.Archive == nil {
+		return &ArchiveStrategy{}
+	}
+	return a.Archive
+}
+
 // GetTemplateByName retrieves a defined template by its name
 func (wf *Workflow) GetTemplateByName(name string) *Template {
 	for _, t := range wf.Spec.Templates {
diff --git a/workflow/executor/executor.go b/workflow/executor/executor.go
index f8a10079e..015245b5f 100644
--- a/workflow/executor/executor.go
+++ b/workflow/executor/executor.go
@@ -133,7 +133,7 @@ func (we *WorkflowExecutor) LoadArtifacts() error {
 				log.Warnf("Ignoring optional artifact '%s' which was not supplied", art.Name)
 				continue
 			} else {
-				return errors.Errorf("", "required artifact %s not supplied", art.Name)
+				return errors.Errorf("required artifact %s not supplied", art.Name)
 			}
 		}
 		artDriver, err := we.InitDriver(&art)
@@ -171,11 +171,22 @@ func (we *WorkflowExecutor) LoadArtifacts() error {
 			return err
 		}
 
-		ok, err := isTarball(tempArtPath)
-		if err != nil {
-			return err
+		isTar := false
+		if art.GetArchive().None != nil {
+			// explicitly not a tar
+			isTar = false
+		} else if art.GetArchive().Tar != nil {
+			// explicitly a tar
+			isTar = true
+		} else {
+			// auto-detect
+			isTar, err = isTarball(tempArtPath)
+			if err != nil {
+				return err
+			}
 		}
-		if ok {
+
+		if isTar {
 			err = untar(tempArtPath, artPath)
 			_ = os.Remove(tempArtPath)
 		} else {


Test patch:
diff --git a/pkg/apis/workflow/v1alpha1/workflow_types_test.go b/pkg/apis/workflow/v1alpha1/workflow_types_test.go
index 1b7409fca..a8d61bee6 100644
--- a/pkg/apis/workflow/v1alpha1/workflow_types_test.go
+++ b/pkg/apis/workflow/v1alpha1/workflow_types_test.go
@@ -38,6 +38,11 @@ func TestArtifactLocation_HasLocation(t *testing.T) {
 	assert.True(t, (&ArtifactLocation{GCS: &GCSArtifact{Key: "my-key", GCSBucket: GCSBucket{Bucket: "my-bucket"}}}).HasLocation())
 }
 
+func TestArtifact_GetArchive(t *testing.T) {
+	assert.NotNil(t, (&Artifact{}).GetArchive())
+	assert.Equal(t, &ArchiveStrategy{None: &NoneStrategy{}}, (&Artifact{Archive: &ArchiveStrategy{None: &NoneStrategy{}}}).GetArchive())
+}
+
 func TestNodes_FindByDisplayName(t *testing.T) {
 	assert.Nil(t, Nodes{}.FindByDisplayName(""))
 	assert.NotNil(t, Nodes{"": NodeStatus{DisplayName: "foo"}}.FindByDisplayName("foo"))