Repository: stern/stern
Pull request #239 (2023-02-26 17:58:03)

allow flexible log parsing and formatting
Addresses https://github.com/stern/stern/issues/227

This is an alternative solution to https://github.com/stern/stern/pull/228

Instead of hardcoding template in the code, I suggest adding a bunch of helper functions and a way to load template from disk.

This PR adds:
 * various color functions
 * TS format function `formatTsRFC3339Nano`
 * `tryParseJSON` function that does not return an error
 * `--template-file/-T` to load template from disk

Sample template that I use:
```
{{color .PodColor .PodName}} {{color .ContainerColor .ContainerName}} {{ with $msg := .Message | tryParseJSON }}[{{ colorGreen (formatTsRFC3339Nano $msg.ts) }}] {{ levelColor $msg.level }} ({{ colorCyan $msg.caller }}) {{ $msg.msg }}{{ else }} {{ .Message }} {{ end }}
```

It allows me to have an alias `alias stern="stern -T ~/.stern.tpl"` which support parsing both JSON and non-JSON logs.

niamster: @tksm @superbrothers Thank you so much for the review!
I tried to address your comments, please let me know if I missed something.

NOTE: I added `toUTC` function to simplify testing. It still might be useful in some case I think.
tksm: Thank you for the changes! I added minor comments below, but it looks good to me other than these. Could you rebase it to allow CI to check?

- https://github.com/stern/stern/pull/239#discussion_r1125155115
- https://github.com/stern/stern/pull/239#discussion_r1125160096

niamster: > Thank you for the changes! I added minor comments below, but it looks good to me other than these. Could you rebase it to allow CI to check?
> 
> * [allow flexible log parsing and formatting #239 (comment)](https://github.com/stern/stern/pull/239#discussion_r1125155115)
> * [allow flexible log parsing and formatting #239 (comment)](https://github.com/stern/stern/pull/239#discussion_r1125160096)

Thank you so for the suggestions! I merged `master` into my branch. D you want me to rebase instead (TBH I'm not a huge fan of rebase since it messes up with GH quite often, so we loose essential traces)?
tksm: > Thank you so for the suggestions! I merged master into my branch. D you want me to rebase instead (TBH I'm not a huge fan of rebase since it messes up with GH quite often, so we loose essential traces)?

Thank you. Either rebase or merge is fine with me.


Patch:
diff --git a/README.md b/README.md
index 2653df8..237ecae 100644
--- a/README.md
+++ b/README.md
@@ -95,6 +95,7 @@ Supported Kubernetes resources are `pod`, `replicationcontroller`, `service`, `d
  `--since`, `-s`             | `48h0m0s` | Return logs newer than a relative duration like 5s, 2m, or 3h.
  `--tail`                    | `-1`      | The number of lines from the end of the logs to show. Defaults to -1, showing all logs.
  `--template`                |           | Template to use for log lines, leave empty to use --output flag.
+ `--template-file`, `-T`     |           | Path to template to use for log lines, leave empty to use --output flag. It overrides --template option.
  `--timestamps`, `-t`        | `false`   | Print timestamps.
  `--timezone`                | `Local`   | Set timestamps to specific timezone.
  `--verbosity`               | `0`       | Number of the log level verbosity
@@ -133,13 +134,25 @@ will receive the following struct:
 The following functions are available within the template (besides the [builtin
 functions](https://golang.org/pkg/text/template/#hdr-Functions)):
 
-| func        | arguments             | description                                                     |
-|-------------|-----------------------|-----------------------------------------------------------------|
-| `json`      | `object`              | Marshal the object and output it as a json text                 |
-| `color`     | `color.Color, string` | Wrap the text in color (.ContainerColor and .PodColor provided) |
-| `parseJSON` | `string`              | Parse string as JSON                                            |
-| `extjson`   | `string`              | Parse the object as json and output colorized json              |
-| `ppextjson` | `string`              | Parse the object as json and output pretty-print colorized json |
+| func            | arguments             | description                                                                       |
+|-----------------|-----------------------|-----------------------------------------------------------------------------------|
+| `json`          | `object`              | Marshal the object and output it as a json text                                   |
+| `color`         | `color.Color, string` | Wrap the text in color (.ContainerColor and .PodColor provided)                   |
+| `parseJSON`     | `string`              | Parse string as JSON                                                              |
+| `tryParseJSON`  | `string`              | Attemp to parse string as JSON, return nil on failure                             |
+| `extjson`       | `string`              | Parse the object as json and output colorized json                                |
+| `ppextjson`     | `string`              | Parse the object as json and output pretty-print colorized json                   |
+| `toRFC3339Nano` | `object`              | Parse timestamp (string, int, json.Number) and output it using RFC3339Nano format |
+| `levelColor`    | `string`              | Print log level using appropriate color                                           |
+| `colorBlack`    | `string`              | Print text using black color                                                      |
+| `colorRed`      | `string`              | Print text using red color                                                        |
+| `colorGreen`    | `string`              | Print text using green color                                                      |
+| `colorYellow`   | `string`              | Print text using yellow color                                                     |
+| `colorBlue`     | `string`              | Print text using blue color                                                       |
+| `colorMagenta`  | `string`              | Print text using magenta color                                                    |
+| `colorCyan`     | `string`              | Print text using cyan color                                                       |
+| `colorWhite`    | `string`              | Print text using white color                                                      |
+
 
 ### Log level verbosity
 
@@ -247,6 +260,18 @@ Output using a custom template with `parseJSON`:
 stern --template='{{.PodName}}/{{.ContainerName}} {{with $d := .Message | parseJSON}}[{{$d.level}}] {{$d.message}}{{end}}{{"\n"}}' backend
 ```
 
+Output using a custom template that tries to parse JSON or fallbacks to raw format:
+
+```
+stern --template='{{.PodName}}/{{.ContainerName}} {{ with $msg := .Message | tryParseJSON }}[{{ colorGreen (toRFC3339Nano $msg.ts) }}] {{ levelColor $msg.level }} ({{ colorCyan $msg.caller }}) {{ $msg.msg }}{{ else }} {{ .Message }} {{ end }}{{"\n"}}' backend
+```
+
+Load custom template from file:
+
+```
+stern --template-file=~/.stern.tpl backend
+```
+
 Trigger the interactive prompt to select an 'app.kubernetes.io/instance' label value:
 
 ```
diff --git a/cmd/cmd.go b/cmd/cmd.go
index 6524a65..ab1ab00 100644
--- a/cmd/cmd.go
+++ b/cmd/cmd.go
@@ -19,6 +19,7 @@ import (
 	"encoding/json"
 	goflag "flag"
 	"fmt"
+	"os"
 	"regexp"
 	"strconv"
 	"strings"
@@ -29,6 +30,7 @@ import (
 
 	"github.com/fatih/color"
 	"github.com/pkg/errors"
+	"github.com/spf13/cast"
 	"github.com/spf13/cobra"
 	"github.com/spf13/pflag"
 	"github.com/stern/stern/stern"
@@ -62,6 +64,7 @@ type options struct {
 	version             bool
 	completion          string
 	template            string
+	templateFile        string
 	output              string
 	prompt              bool
 	podQuery            string
@@ -85,6 +88,7 @@ func NewOptions(streams genericclioptions.IOStreams) *options {
 		since:               48 * time.Hour,
 		tail:                -1,
 		template:            "",
+		templateFile:        "",
 		timestamps:          false,
 		timezone:            "Local",
 		prompt:              false,
@@ -302,6 +306,7 @@ func (o *options) AddFlags(fs *pflag.FlagSet) {
 	fs.DurationVarP(&o.since, "since", "s", o.since, "Return logs newer than a relative duration like 5s, 2m, or 3h.")
 	fs.Int64Var(&o.tail, "tail", o.tail, "The number of lines from the end of the logs to show. Defaults to -1, showing all logs.")
 	fs.StringVar(&o.template, "template", o.template, "Template to use for log lines, leave empty to use --output flag.")
+	fs.StringVarP(&o.templateFile, "template-file", "T", o.templateFile, "Path to template to use for log lines, leave empty to use --output flag. It overrides --template option.")
 	fs.BoolVarP(&o.timestamps, "timestamps", "t", o.timestamps, "Print timestamps.")
 	fs.StringVar(&o.timezone, "timezone", o.timezone, "Set timestamps to specific timezone.")
 	fs.BoolVar(&o.onlyLogLines, "only-log-lines", o.onlyLogLines, "Print only log lines")
@@ -311,6 +316,13 @@ func (o *options) AddFlags(fs *pflag.FlagSet) {
 
 func (o *options) generateTemplate() (*template.Template, error) {
 	t := o.template
+	if o.templateFile != "" {
+		data, err := os.ReadFile(o.templateFile)
+		if err != nil {
+			return nil, err
+		}
+		t = string(data)
+	}
 	if t == "" {
 		switch o.output {
 		case "default":
@@ -348,6 +360,15 @@ func (o *options) generateTemplate() (*template.Template, error) {
 			}
 			return string(b), nil
 		},
+		"tryParseJSON": func(text string) map[string]interface{} {
+			decoder := json.NewDecoder(strings.NewReader(text))
+			decoder.UseNumber()
+			obj := make(map[string]interface{})
+			if err := decoder.Decode(&obj); err != nil {
+				return nil
+			}
+			return obj
+		},
 		"parseJSON": func(text string) (map[string]interface{}, error) {
 			obj := make(map[string]interface{})
 			if err := json.Unmarshal([]byte(text), &obj); err != nil {
@@ -365,9 +386,44 @@ func (o *options) generateTemplate() (*template.Template, error) {
 			}
 			return strings.TrimSuffix(string(b), "\n"), nil
 		},
+		"toRFC3339Nano": func(ts any) string {
+			return cast.ToTime(ts).Format(time.RFC3339Nano)
+		},
+		"toUTC": func(ts any) time.Time {
+			return cast.ToTime(ts).UTC()
+		},
 		"color": func(color color.Color, text string) string {
 			return color.SprintFunc()(text)
 		},
+		"colorBlack":   color.BlackString,
+		"colorRed":     color.RedString,
+		"colorGreen":   color.GreenString,
+		"colorYellow":  color.YellowString,
+		"colorBlue":    color.BlueString,
+		"colorMagenta": color.MagentaString,
+		"colorCyan":    color.CyanString,
+		"colorWhite":   color.WhiteString,
+		"levelColor": func(level string) string {
+			var levelColor *color.Color
+			switch strings.ToLower(level) {
+			case "debug":
+				levelColor = color.New(color.FgMagenta)
+			case "info":
+				levelColor = color.New(color.FgBlue)
+			case "warn":
+				levelColor = color.New(color.FgYellow)
+			case "error":
+				levelColor = color.New(color.FgRed)
+			case "dpanic":
+				levelColor = color.New(color.FgRed)
+			case "panic":
+				levelColor = color.New(color.FgRed)
+			case "fatal":
+				levelColor = color.New(color.FgCyan)
+			default:
+			}
+			return levelColor.SprintFunc()(level)
+		},
 	}
 	template, err := template.New("log").Funcs(funs).Parse(t)
 	if err != nil {
diff --git a/go.mod b/go.mod
index cf25672..12f2ac3 100644
--- a/go.mod
+++ b/go.mod
@@ -6,6 +6,7 @@ require (
 	github.com/AlecAivazis/survey/v2 v2.3.6
 	github.com/fatih/color v1.13.0
 	github.com/pkg/errors v0.9.1
+	github.com/spf13/cast v1.5.0
 	github.com/spf13/cobra v1.6.1
 	github.com/spf13/pflag v1.0.5
 	golang.org/x/sync v0.1.0
diff --git a/go.sum b/go.sum
index 1c60613..9cee17a 100644
--- a/go.sum
+++ b/go.sum
@@ -38,6 +38,7 @@ github.com/evanphx/json-patch v5.6.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLi
 github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
 github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
 github.com/flowstack/go-jsonschema v0.1.1/go.mod h1:yL7fNggx1o8rm9RlgXv7hTBWxdBM0rVwpMwimd3F3N0=
+github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
 github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
@@ -109,8 +110,8 @@ github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:C
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
-github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
+github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -151,9 +152,12 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
+github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
 github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
+github.com/spf13/cast v1.5.0 h1:rj3WzYc11XZaIZMPKmwP96zkFEnnAmV8s6XbB2aY32w=
+github.com/spf13/cast v1.5.0/go.mod h1:SpXXQ5YoyJw6s3/6cMTQuxvgRl3PCJiyaX9p6b155UU=
 github.com/spf13/cobra v1.6.1 h1:o94oiPyS4KD1mPy2fmcYYHHfCxLqYjJOhGsCHFZtEzA=
 github.com/spf13/cobra v1.6.1/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=


Test patch:
diff --git a/cmd/cmd_test.go b/cmd/cmd_test.go
index 8b992dd..1718173 100644
--- a/cmd/cmd_test.go
+++ b/cmd/cmd_test.go
@@ -296,6 +296,42 @@ func TestOptionsGenerateTemplate(t *testing.T) {
 			"",
 			true,
 		},
+		{
+			"template-file",
+			func() *options {
+				o := NewOptions(streams)
+				o.templateFile = "test.tpl"
+
+				return o
+			}(),
+			"template message",
+			"pod1 container1 template message",
+			false,
+		},
+		{
+			"template-file-json-log-ts-float",
+			func() *options {
+				o := NewOptions(streams)
+				o.templateFile = "test.tpl"
+
+				return o
+			}(),
+			`{"ts": 123, "level": "INFO", "msg": "template message"}`,
+			"pod1 container1 [1970-01-01T00:02:03Z] INFO template message",
+			false,
+		},
+		{
+			"template-file-json-log-ts-str",
+			func() *options {
+				o := NewOptions(streams)
+				o.templateFile = "test.tpl"
+
+				return o
+			}(),
+			`{"ts": "1970-01-01T01:02:03+01:00", "level": "INFO", "msg": "template message"}`,
+			"pod1 container1 [1970-01-01T00:02:03Z] INFO template message",
+			false,
+		},
 	}
 
 	for _, tt := range tests {
diff --git a/cmd/test.tpl b/cmd/test.tpl
new file mode 100644
index 0000000..275aa9d
--- /dev/null
+++ b/cmd/test.tpl
@@ -0,0 +1,1 @@
+{{color .PodColor .PodName}} {{color .ContainerColor .ContainerName}} {{ with $msg := .Message | tryParseJSON }}[{{ colorGreen (toRFC3339Nano (toUTC $msg.ts)) }}] {{ levelColor $msg.level }} {{ $msg.msg }}{{ else }}{{ .Message }}{{ end }}
\ No newline at end of file