Repository: go-task/task
Pull request #216 (2019-06-17 22:30:47)

Expose timestamp and checksum to status
## What changes does this PR introduce?

This introduces two special variables `{{.CHECKSUM}}` and `{{.TIMESTAMP}}` to `status` commands, to enable the checking of "up-to-dateness" of remote artifacts.

## Any background context you want to provide?

Generally, when you use `sources` - you also use `generates` - if the `generates` artifacts is newer than the `sources`, then the your task is out of date and needs to be re-run.  This works as long as you have direct access to the generated artifacts - but some tasks produce "remote artifacts" that don't live on the same file system as your sources (AMIs, Docker Images, Kubernetes Deploys) - `status` already supports `sh` based status checks - so we expanded this to evaluate remote artifacts.

Which ever method you choose to fingerprint your sources will be exposed as a template variable to `status` shell commands.

Currently, the expectation is that you would use `sources` and `generates` together and then `status` separately.   The expectation going forward would be that `sources` could be used with either `status` or `generates` to evaluate the status of the task.  `status` and `generates` are still mutually exclusive.

## Where should the reviewer start?

The test is a little artificial in that it echos the value of the checksum rather than actually _doing_ something with it - but it demonstrates how the feature could be used.

## Has this been manually tested? How?

This is part of our private fork.

andreynering: Hi @stephenprater,

Sorry for asking, but just to make sure we're on the same page... Do you plan to do the necessary fixes or do you prefer me to work on it?

If the latter, can you make sure this checkbox is checked?

<img width="239" alt="Screen Shot 2019-08-11 at 22 38 31" src="https://user-images.githubusercontent.com/7011819/62842480-f5fbba00-bc88-11e9-8963-a37d3253f397.png">

Thanks!
stephenprater: Sorry Andrey, I’ve been kind of distracted the last week or so.  I think I should be able to get to this tomorrow evening.
On Aug 11, 2019, 6:40 PM -0700, Andrey Nering <notifications@github.com>, wrote:
> Hi @stephenprater,
> Sorry for asking, but just to make sure we're on the same page... Do you plan to do the necessary fixes or do you prefer me to work on it?
> If the latter, can you make sure this checkbox is checked?
> Thanks!
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

andreynering: @stephenprater Don't hurry, take your time, we all have busy lives. I was just checking in. 😉 

Thanks!
andreynering: Can you please move the target branch from `master` to `v3` and merge `v3` into this branch? We shouldn't have any conflicts. Thanks!

Patch:
diff --git a/docs/usage.md b/docs/usage.md
index 7b9f5fd0..02a8db7e 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -266,6 +266,8 @@ The above syntax is also supported in `deps`.
 
 ## Prevent unnecessary work
 
+### By fingerprinting locally generated files and their sources
+
 If a task generates something, you can inform Task the source and generated
 files, so Task will prevent to run them if not necessary.
 
@@ -321,6 +323,9 @@ tasks:
 
 > TIP: method `none` skips any validation and always run the task.
 
+### Using programmatic checks to indicate a task is up to date.
+
+
 Alternatively, you can inform a sequence of tests as `status`. If no error
 is returned (exit status 0), the task is considered up-to-date:
 
@@ -340,15 +345,35 @@ tasks:
       - test -f directory/file2.txt
 ```
 
+
+Normally, you would use `sources` in combination with
+`generates` - but for tasks that generate remote artifacts (Docker images,
+deploys, CD releases) the checksum source and timestamps require either
+access to the artifact or for an out-of-band refresh of the `.checksum`
+fingerprint file.
+
+Two special variables `{{.CHECKSUM}}` and `{{.TIMESTAMP}}` are available
+for interpolation within `status` commands, depending on the method assigned
+to fingerprint the sources.  Only `source` globs are fingerprinted.
+
+Note that the `{{.TIMESTAMP}}` variable is a "live" Go time struct, and can be
+formatted using any of the methods that `Time` responds to.
+
+See [the Go Time documentation](https://golang.org/pkg/time/) for more information.
+
 You can use `--force` or `-f` if you want to force a task to run even when
 up-to-date.
 
 Also, `task --status [tasks]...` will exit with a non-zero exit code if any of
 the tasks are not up-to-date.
 
-If you need a certain set of conditions to be _true_ you can use the
-`preconditions` stanza.  `preconditions` are very similar to `status`
-lines except they support `sh` expansion and they SHOULD all return 0.
+### Using programmatic checks to cancel execution of an task and it's dependencies
+
+In addition to `status` checks, there are also `preconditions` checks, which are
+the logical inverse of `status` checks.  That is, if you need a certain set of
+conditions to be _true_ you can use the `preconditions` stanza.
+`preconditions` are similar to `status` lines except they support `sh`
+expansion and they SHOULD all return 0.
 
 ```yaml
 version: '2'
diff --git a/go.sum b/go.sum
index 7ff7b829..5e9ae83e 100644
--- a/go.sum
+++ b/go.sum
@@ -30,8 +30,11 @@ github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 golang.org/x/crypto v0.0.0-20180830192347-182538f80094 h1:rVTAlhYa4+lCfNxmAIEOGQRoD23UqP72M3+rSWVGDTg=
 golang.org/x/crypto v0.0.0-20180830192347-182538f80094/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d h1:g9qWBGx4puODJTMVyoPrpoxPFgVGd+z1DZwjfRu4d0I=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=
diff --git a/internal/status/checksum.go b/internal/status/checksum.go
index 572d4650..2e968938 100644
--- a/internal/status/checksum.go
+++ b/internal/status/checksum.go
@@ -84,11 +84,21 @@ func (c *Checksum) checksum(files ...string) (string, error) {
 	return fmt.Sprintf("%x", h.Sum(nil)), nil
 }
 
+// Value implements the Checker Interface
+func (c *Checksum) Value() (interface{}, error) {
+	return c.checksum()
+}
+
 // OnError implements the Checker interface
 func (c *Checksum) OnError() error {
 	return os.Remove(c.checksumFilePath())
 }
 
+// Kind implements the Checker Interface
+func (t *Checksum) Kind() string {
+	return "checksum"
+}
+
 func (c *Checksum) checksumFilePath() string {
 	return filepath.Join(c.Dir, ".task", "checksum", c.normalizeFilename(c.Task))
 }
diff --git a/internal/status/none.go b/internal/status/none.go
index 01e35060..cca3888b 100644
--- a/internal/status/none.go
+++ b/internal/status/none.go
@@ -8,6 +8,15 @@ func (None) IsUpToDate() (bool, error) {
 	return false, nil
 }
 
+// Value implements the Checker interface
+func (None) Value() (interface{}, error) {
+	return "", nil
+}
+
+func (None) Kind() string {
+	return "none"
+}
+
 // OnError implements the Checker interface
 func (None) OnError() error {
 	return nil
diff --git a/internal/status/status.go b/internal/status/status.go
index 320ca8a6..2648d27c 100644
--- a/internal/status/status.go
+++ b/internal/status/status.go
@@ -9,5 +9,7 @@ var (
 // Checker is an interface that checks if the status is up-to-date
 type Checker interface {
 	IsUpToDate() (bool, error)
+	Value() (interface{}, error)
 	OnError() error
+	Kind() string
 }
diff --git a/internal/status/timestamp.go b/internal/status/timestamp.go
index fcc52f6a..3801c1ac 100644
--- a/internal/status/timestamp.go
+++ b/internal/status/timestamp.go
@@ -41,6 +41,29 @@ func (t *Timestamp) IsUpToDate() (bool, error) {
 	return !generatesMinTime.Before(sourcesMaxTime), nil
 }
 
+func (t *Timestamp) Kind() string {
+	return "timestamp"
+}
+
+// Value implements the Checker Interface
+func (t *Timestamp) Value() (interface{}, error) {
+	sources, err := globs(t.Dir, t.Sources)
+	if err != nil {
+		return time.Now(), err
+	}
+
+	sourcesMaxTime, err := getMaxTime(sources...)
+	if err != nil {
+		return time.Now(), err
+	}
+
+	if sourcesMaxTime.IsZero() {
+		return time.Unix(0, 0), nil
+	}
+
+	return sourcesMaxTime, nil
+}
+
 func getMinTime(files ...string) (time.Time, error) {
 	var t time.Time
 	for _, f := range files {
diff --git a/internal/taskfile/var.go b/internal/taskfile/var.go
index f52e0f8b..ad0f5e90 100644
--- a/internal/taskfile/var.go
+++ b/internal/taskfile/var.go
@@ -13,17 +13,22 @@ var (
 // Vars is a string[string] variables map.
 type Vars map[string]Var
 
-// ToStringMap converts Vars to a string map containing only the static
+// ToCacheMap converts Vars to a map containing only the static
 // variables
-func (vs Vars) ToStringMap() (m map[string]string) {
-	m = make(map[string]string, len(vs))
+func (vs Vars) ToCacheMap() (m map[string](interface{})) {
+	m = make(map[string](interface{}), len(vs))
 	for k, v := range vs {
 		if v.Sh != "" {
 			// Dynamic variable is not yet resolved; trigger
 			// <no value> to be used in templates.
 			continue
 		}
-		m[k] = v.Static
+
+		if v.Live != nil {
+			m[k] = v.Live
+		} else {
+			m[k] = v.Static
+		}
 	}
 	return
 }
@@ -31,6 +36,7 @@ func (vs Vars) ToStringMap() (m map[string]string) {
 // Var represents either a static or dynamic variable.
 type Var struct {
 	Static string
+	Live   interface{}
 	Sh     string
 }
 
diff --git a/internal/templater/templater.go b/internal/templater/templater.go
index 65a2fde2..d43117e0 100644
--- a/internal/templater/templater.go
+++ b/internal/templater/templater.go
@@ -14,8 +14,12 @@ import (
 type Templater struct {
 	Vars taskfile.Vars
 
-	strMap map[string]string
-	err    error
+	cacheMap map[string](interface{})
+	err      error
+}
+
+func (r *Templater) RefreshCacheMap() {
+	r.cacheMap = r.Vars.ToCacheMap()
 }
 
 func (r *Templater) Replace(str string) string {
@@ -29,12 +33,12 @@ func (r *Templater) Replace(str string) string {
 		return ""
 	}
 
-	if r.strMap == nil {
-		r.strMap = r.Vars.ToStringMap()
+	if r.cacheMap == nil {
+		r.cacheMap = r.Vars.ToCacheMap()
 	}
 
 	var b bytes.Buffer
-	if err = templ.Execute(&b, r.strMap); err != nil {
+	if err = templ.Execute(&b, r.cacheMap); err != nil {
 		r.err = err
 		return ""
 	}
@@ -62,6 +66,7 @@ func (r *Templater) ReplaceVars(vars taskfile.Vars) taskfile.Vars {
 	for k, v := range vars {
 		new[k] = taskfile.Var{
 			Static: r.Replace(v.Static),
+			Live:   v.Live,
 			Sh:     r.Replace(v.Sh),
 		}
 	}
diff --git a/task.go b/task.go
index 4d8097e7..92cd64dc 100644
--- a/task.go
+++ b/task.go
@@ -348,7 +348,7 @@ func getEnviron(t *taskfile.Task) []string {
 	}
 
 	environ := os.Environ()
-	for k, v := range t.Env.ToStringMap() {
+	for k, v := range t.Env.ToCacheMap() {
 		environ = append(environ, fmt.Sprintf("%s=%s", k, v))
 	}
 	return environ
diff --git a/variables.go b/variables.go
index 9c227372..8307fc4d 100644
--- a/variables.go
+++ b/variables.go
@@ -2,6 +2,7 @@ package task
 
 import (
 	"path/filepath"
+	"strings"
 
 	"github.com/go-task/task/v2/internal/execext"
 	"github.com/go-task/task/v2/internal/taskfile"
@@ -20,6 +21,7 @@ func (e *Executor) CompiledTask(call taskfile.Call) (*taskfile.Task, error) {
 	if err != nil {
 		return nil, err
 	}
+
 	r := templater.Templater{Vars: vars}
 
 	new := taskfile.Task{
@@ -27,7 +29,6 @@ func (e *Executor) CompiledTask(call taskfile.Call) (*taskfile.Task, error) {
 		Desc:        r.Replace(origTask.Desc),
 		Sources:     r.ReplaceSlice(origTask.Sources),
 		Generates:   r.ReplaceSlice(origTask.Generates),
-		Status:      r.ReplaceSlice(origTask.Status),
 		Dir:         r.Replace(origTask.Dir),
 		Vars:        nil,
 		Env:         nil,
@@ -94,5 +95,24 @@ func (e *Executor) CompiledTask(call taskfile.Call) (*taskfile.Task, error) {
 		}
 	}
 
+	if len(origTask.Status) > 0 {
+		checker, err := e.getStatusChecker(&new)
+		if err != nil {
+			return nil, err
+		}
+
+		value, err := checker.Value()
+		if err != nil {
+			return nil, err
+		}
+
+		vars[strings.ToUpper(checker.Kind())] = taskfile.Var{Live: value}
+		// Adding new variables, requires us to refresh the templaters
+		// cache of the the values manually
+		r.RefreshCacheMap()
+
+		new.Status = r.ReplaceSlice(origTask.Status)
+	}
+
 	return &new, r.Err()
 }


Test patch:
diff --git a/task_test.go b/task_test.go
index ebcb51b3..1d8f9575 100644
--- a/task_test.go
+++ b/task_test.go
@@ -11,6 +11,7 @@ import (
 	"testing"
 
 	"github.com/go-task/task/v2"
+	"github.com/go-task/task/v2/internal/logger"
 	"github.com/go-task/task/v2/internal/taskfile"
 
 	"github.com/mitchellh/go-homedir"
@@ -351,12 +352,20 @@ func TestStatusChecksum(t *testing.T) {
 	}
 
 	var buff bytes.Buffer
+
+	logCapturer := logger.Logger{
+		Stdout:  &buff,
+		Stderr:  &buff,
+		Verbose: true,
+	}
+
 	e := task.Executor{
 		Dir:    dir,
 		Stdout: &buff,
 		Stderr: &buff,
 	}
 	assert.NoError(t, e.Setup())
+	e.Logger = &logCapturer
 
 	assert.NoError(t, e.Run(context.Background(), taskfile.Call{Task: "build"}))
 	for _, f := range files {
@@ -367,6 +376,20 @@ func TestStatusChecksum(t *testing.T) {
 	buff.Reset()
 	assert.NoError(t, e.Run(context.Background(), taskfile.Call{Task: "build"}))
 	assert.Equal(t, `task: Task "build" is up to date`+"\n", buff.String())
+
+	buff.Reset()
+	e.Silent = false
+	e.Verbose = true
+	assert.NoError(t, e.Run(context.Background(), taskfile.Call{Task: "build-with-checksum"}))
+	assert.Contains(t, buff.String(), "d41d8cd98f00b204e9800998ecf8427e")
+
+	buff.Reset()
+	inf, _ := os.Stat(filepath.Join(dir, "source.txt"))
+	ts := fmt.Sprintf("%d", inf.ModTime().Unix())
+	tf := fmt.Sprintf("%s", inf.ModTime())
+	assert.NoError(t, e.Run(context.Background(), taskfile.Call{Task: "build-with-timestamp"}))
+	assert.Contains(t, buff.String(), ts)
+	assert.Contains(t, buff.String(), tf)
 }
 
 func TestInit(t *testing.T) {
diff --git a/testdata/checksum/Taskfile.yml b/testdata/checksum/Taskfile.yml
index 1c620a71..2becc5f0 100644
--- a/testdata/checksum/Taskfile.yml
+++ b/testdata/checksum/Taskfile.yml
@@ -10,3 +10,17 @@ tasks:
     generates:
       - ./generated.txt
     method: checksum
+
+  build-with-checksum:
+    sources:
+      - ./source.txt
+    method: checksum
+    status:
+      - echo "{{.CHECKSUM}}"
+
+  build-with-timestamp:
+    sources:
+      - ./source.txt
+    status:
+      - echo '{{.TIMESTAMP.Unix }}'
+      - echo '{{.TIMESTAMP}}'