Repository: twpayne/chezmoi (Go)

Issue:
Accept more source file encoding when execute go templating, or allow encoding conversion after templating
## Is your feature request related to a problem? Please describe.

Just when I use powershell script with templating, I noticed that .tmpl files with utf16le encoding did not remove `{{- command -}}` pattern. Then I realized that go templating assumes utf8 input. However, windows world assumes utf16le inputs in some place, like powershell 5 script input

## Describe the solution you'd like

Either allow templating other than utf8(system-native encoding), or allow encoding conversion after templating but before execution (using a chezmoi directive?)

## Describe alternatives you've considered

Allow user to execute some command right after script template evaluation but before execution, like pre-?interpreters with argument list

## Additional context

<https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-5.1#character-encoding-in-windows-powershell> and <https://en.wikipedia.org/wiki/Windows_code_page#UTF-8,_UTF-16>, text encoding problem on windows is difficult, in some cases utf16le works but utf8 do not work, and this can be the first step to improve windows script execution experience ww


Discussion:
Thank you for opening this issue! I want chezmoi to provide a good experience on Windows, but I need help understanding it as I am not a regular Windows or PowerShell user.

As I understand it, the current situation is:

* Currently, chezmoi expects templates to be in UTF-8.
* Currently, the result of executing a chezmoi template is always in UTF-8.
* On Windows, you sometimes need to use UTF-16.
* On Windows, you might want to write templates in UTF-16.
* On Windows, you might want the result of executing a template to be in UTF-16.

If this understanding is correct, then what do think about the following?

* On all systems, templates are written in UTF-8 with no byte order mark. This ensures that templates are portable across multiple systems. 
* chezmoi adds a template directive like `chezmoi:template:encoding=utf-16-le` which tells chezmoi to convert the output of the template execution to little-endian UTF-16, which should fulfill your "allow encoding conversion after templating but before execution (using a chezmoi directive?)" suggestion.

I did some initial experiments in #4378. Would you be able to test this?
Yes, I think this work solved the problem for powershell 5.1. Here I provide a test script and my output.

<details>

```powershell run_encoding.ps1
# chezmoi:template:left-delimiter="#{{"
# chezmoi:template:encoding=utf-16-le-bom
# chezmoi:template:line-endings=crlf
#{{- if eq .chezmoi.os "windows" -}}

#!/usr/bin/env powershell -File
#also see <https://www.i18nqa.com/debug/utf8-debug.html>
# tested with 933ed88af5860182792afdaab94ac34a7232775205682cb26d5fd9b1a2336cb7  chezmoi.exe on Windows 11 24H2 x64 26100.3194
# PS C:\Users\Misaka4a92> $PSVersionTable

# Name                           Value
# ----                           -----
# PSVersion                      5.1.26100.2161
# PSEdition                      Desktop
# PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
# BuildVersion                   10.0.26100.2161
# CLRVersion                     4.0.30319.42000
# WSManStackVersion              3.0
# PSRemotingProtocolVersion      2.3
# SerializationVersion           1.1.0.1
#
#
# $ chezmoi.exe apply
# System default encoding is iso-8859-1
# — "ÉüДЦ本화が😂
# Hex representation is \xe2\x80\x94\x20\x22\xc3\x89\xc3\xbc\xd0\x94\xd0\xa6\xe6\x9c\xac\xed\x99\x94\xe3\x81\x8c\xf0\x9f\x98\x82
# — "ÉüДЦ本화が😂
# Bytes: 255 254 35 0
# Detected bom is Unicode UTF-16 Little-Endian.
# The encoding of C:\ProgramData\Temp\1730255736.encoding.ps1 is utf-16.

# https://learn.microsoft.com/en-us/dotnet/fundamentals/runtime-libraries/system-text-encoding-default
Write-Host ('System default encoding is {0}' -f [System.Text.Encoding]::Default.BodyName)

$x = '— "ÉüДЦ本화が😂'
$hex = -join [System.Text.Encoding]::UTF8.GetBytes($x).ForEach({ '\x{0}' -f $_.toString('x2') })
# $hex = '\xe2\x80\x94\x20\x22\xc3\x89\xc3\xbc\xd0\x94\xd0\xa6\xe6\x9c\xac\xed\x99\x94\xe3\x81\x8c\xf0\x9f\x98\x82'
Write-Host "$x"
Write-Host "Hex representation is $hex"
$busybox = (Get-Command -ErrorAction Ignore busybox).Path
if ($busybox) {
    Start-Process -NoNewWindow -Wait -FilePath $busybox -ArgumentList ('printf', ($hex + '\n'))
}

# from https://www.powershellgallery.com/packages/PoshFunctions/2.2.1.1/Content/Functions%5CGet-FileEncoding.ps1
[byte[]] $byte = Get-Content -Encoding byte -ReadCount 4 -TotalCount 4 -Path $MyInvocation.MyCommand.Path
Write-Host Bytes: $byte[0] $byte[1] $byte[2] $byte[3]

if ( $byte[0] -eq 0xef -and $byte[1] -eq 0xbb -and $byte[2] -eq 0xbf ) {
    # EF BB BF (UTF8)
    $bom = 'UTF8'
} elseif ($byte[0] -eq 0xfe -and $byte[1] -eq 0xff) {
    # FE FF (UTF-16 Big-Endian)
    $bom = 'Unicode UTF-16 Big-Endian'
} elseif ($byte[0] -eq 0xff -and $byte[1] -eq 0xfe) {
    # FF FE (UTF-16 Little-Endian)
    $bom = 'Unicode UTF-16 Little-Endian'
} elseif ($byte[0] -eq 0 -and $byte[1] -eq 0 -and $byte[2] -eq 0xfe -and $byte[3] -eq 0xff) {
    # 00 00 FE FF (UTF32 Big-Endian)
    $bom = 'UTF32 Big-Endian'
} elseif ($byte[0] -eq 0xfe -and $byte[1] -eq 0xff -and $byte[2] -eq 0 -and $byte[3] -eq 0) {
    # FE FF 00 00 (UTF32 Little-Endian)
    $bom = 'UTF32 Little-Endian'
} elseif ($byte[0] -eq 0x2b -and $byte[1] -eq 0x2f -and $byte[2] -eq 0x76 -and ($byte[3] -eq 0x38 -or $byte[3] -eq 0x39 -or $byte[3] -eq 0x2b -or $byte[3] -eq 0x2f) ) {
    # 2B 2F 76 (38 | 38 | 2B | 2F)
    $bom = 'UTF7'
} elseif ( $byte[0] -eq 0xf7 -and $byte[1] -eq 0x64 -and $byte[2] -eq 0x4c ) {
    # F7 64 4C (UTF-1)
    $bom = 'UTF-1'
} elseif ($byte[0] -eq 0xdd -and $byte[1] -eq 0x73 -and $byte[2] -eq 0x66 -and $byte[3] -eq 0x73) {
    # DD 73 66 73 (UTF-EBCDIC)
    $bom = 'UTF-EBCDIC'
} elseif ( $byte[0] -eq 0x0e -and $byte[1] -eq 0xfe -and $byte[2] -eq 0xff ) {
    # 0E FE FF (SCSU)
    $bom = 'SCSU'
} elseif ( $byte[0] -eq 0x0e -and $byte[1] -eq 0xfe -and $byte[2] -eq 0xff ) {
    $bom = 'SCSU'
} elseif ( $byte[0] -eq 0xfb -and $byte[1] -eq 0xee -and $byte[2] -eq 0x28 ) {
    # FB EE 28 (BOCU-1)
    $bom = 'BOCU-1'
} elseif ($byte[0] -eq 0x84 -and $byte[1] -eq 0x31 -and $byte[2] -eq 0x95 -and $byte[3] -eq 0x33) {
    # 84 31 95 33 (GB-18030)
    $bom = 'GB-18030'
} else {
    $bom = 'ASCII'
}
Write-Host ("Detected bom is $bom.")

$sr = [System.IO.FileStream]::new($MyInvocation.MyCommand.Path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite)
$sr = [System.IO.StreamReader]::new($sr, $true)

$sr.ReadLine().ToString() | Out-Null
Write-Host ('The encoding of {0} is {1}.' -f $MyInvocation.MyCommand.Path, $sr.CurrentEncoding.HeaderName)
$sr.Close()

#{{- end -}}
```

seems that utf16le without bom do not work with powershell. Most windows api expects utf16le-with-bom or current codepage. This script works with workflow files but not latest release.

</details>



Patch:
diff --git a/assets/chezmoi.io/docs/reference/templates/directives.md b/assets/chezmoi.io/docs/reference/templates/directives.md
index 6570b65ef8b..a551f853583 100644
--- a/assets/chezmoi.io/docs/reference/templates/directives.md
+++ b/assets/chezmoi.io/docs/reference/templates/directives.md
@@ -37,6 +37,33 @@ inherited by templates called from the file.
     # [[ "true" ]]
     ```
 
+## Encoding
+
+Templates are always written in UTF-8 with no byte order mark.
+
+By default, the result of executing a template is also UTF-8 with no
+byte order mark but this can be transformed into another encoding with the
+template directive:
+
+    chezmoi:template:encoding=$ENCODING
+
+where `$ENCODING` is one of:
+
+| Encoding        | Description                                  |
+| --------------- | -------------------------------------------- |
+| `utf-8`         | UTF-8 with no byte order mark                |
+| `utf-8-bom`     | UTF-8 with a byte order mark                 |
+| `utf-16-be`     | Big-endian UTF-16 with no byte order mark    |
+| `utf-16-be-bom` | Big-endian UTF-16 with a byte order mark     |
+| `utf-16-le`     | Little-endian UTF-16 with no byte order mark |
+| `utf-16-le-bom` | Little-endian UTF-16 with a byte order mark  |
+
+!!! example
+
+    ```
+    {{/* chezmoi:template:encoding=utf-16-le */}}
+    ```
+
 ## Format indent
 
 By default, chezmoi's `toJson`, `toToml`, and `toYaml` template functions use
diff --git a/internal/chezmoi/template.go b/internal/chezmoi/template.go
index f5f633d11e0..649e3de196d 100644
--- a/internal/chezmoi/template.go
+++ b/internal/chezmoi/template.go
@@ -3,6 +3,7 @@ package chezmoi
 import (
 	"bytes"
 	"encoding/json"
+	"fmt"
 	"maps"
 	"strconv"
 	"strings"
@@ -12,6 +13,8 @@ import (
 	"github.com/mattn/go-runewidth"
 	"github.com/mitchellh/copystructure"
 	"github.com/pelletier/go-toml/v2"
+	"golang.org/x/text/encoding"
+	"golang.org/x/text/encoding/unicode"
 )
 
 // A Template extends text/template.Template with support for directives.
@@ -23,6 +26,7 @@ type Template struct {
 
 // TemplateOptions are template options that can be set with directives.
 type TemplateOptions struct {
+	Encoding       encoding.Encoding
 	Funcs          template.FuncMap
 	FormatIndent   string
 	LeftDelimiter  string
@@ -107,7 +111,12 @@ func (t *Template) Execute(data any) ([]byte, error) {
 	if err := t.template.ExecuteTemplate(&builder, t.name, data); err != nil {
 		return nil, err
 	}
-	return []byte(replaceLineEndings(builder.String(), t.options.LineEnding)), nil
+
+	result := []byte(replaceLineEndings(builder.String(), t.options.LineEnding))
+	if t.options.Encoding != nil {
+		return t.options.Encoding.NewEncoder().Bytes(result)
+	}
+	return result, nil
 }
 
 // parseAndRemoveDirectives updates o by parsing all template directives in data
@@ -126,6 +135,23 @@ func (o *TemplateOptions) parseAndRemoveDirectives(data []byte) ([]byte, error)
 			key := string(keyValuePairMatch[1])
 			value := maybeUnquote(string(keyValuePairMatch[2]))
 			switch key {
+			case "encoding":
+				switch value {
+				case "utf-8":
+					o.Encoding = unicode.UTF8
+				case "utf-8-bom":
+					o.Encoding = unicode.UTF8BOM
+				case "utf-16-be":
+					o.Encoding = unicode.UTF16(unicode.BigEndian, unicode.IgnoreBOM)
+				case "utf-16-be-bom":
+					o.Encoding = unicode.UTF16(unicode.BigEndian, unicode.UseBOM)
+				case "utf-16-le":
+					o.Encoding = unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
+				case "utf-16-le-bom":
+					o.Encoding = unicode.UTF16(unicode.LittleEndian, unicode.UseBOM)
+				default:
+					return nil, fmt.Errorf("%s: unknown encoding", value)
+				}
 			case "format-indent":
 				o.FormatIndent = value
 			case "format-indent-width":


Test patch:
diff --git a/internal/cmd/main_test.go b/internal/cmd/main_test.go
index d7392012fcd..3ff60e7f30e 100644
--- a/internal/cmd/main_test.go
+++ b/internal/cmd/main_test.go
@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"bytes"
 	_ "embed"
+	"encoding/hex"
 	"errors"
 	"flag"
 	"fmt"
@@ -97,6 +98,7 @@ func TestScript(t *testing.T) {
 			"cmpmod":         cmdCmpMod,
 			"edit":           cmdEdit,
 			"expandenv":      cmdExpandEnv,
+			"hexdecode":      cmdHexDecode,
 			"httpd":          cmdHTTPD,
 			"isdir":          cmdIsDir,
 			"issymlink":      cmdIsSymlink,
@@ -260,6 +262,33 @@ func cmdHTTPD(ts *testscript.TestScript, neg bool, args []string) {
 	ts.Setenv("HTTPD_URL", server.URL)
 }
 
+// cmdHexDecode decodes each argument, which must be a file with the extension
+// .hex, writing the result to the same path but without the .hex extension.
+func cmdHexDecode(ts *testscript.TestScript, neg bool, args []string) {
+	if neg {
+		ts.Fatalf("unsupported: ! hexdecode")
+	}
+	for _, arg := range args {
+		filename := ts.MkAbs(arg)
+		if filepath.Ext(filename) != ".hex" {
+			ts.Fatalf("%s: no .hex extension", arg)
+		}
+		hexData, err := os.ReadFile(filename)
+		if err != nil {
+			ts.Fatalf("%s: %v", arg, err)
+		}
+		hexData = regexp.MustCompile(`(?m)#.*$`).ReplaceAll(hexData, nil)
+		hexData = regexp.MustCompile(`\s+`).ReplaceAll(hexData, nil)
+		data := make([]byte, hex.DecodedLen(len(hexData)))
+		if _, err := hex.Decode(data, hexData); err != nil {
+			ts.Fatalf("%s: %v", arg, err)
+		}
+		if err := os.WriteFile(strings.TrimSuffix(filename, ".hex"), data, 0o666); err != nil {
+			ts.Fatalf("%s: %v", arg, err)
+		}
+	}
+}
+
 // cmdIsDir succeeds if all of its arguments are directories.
 func cmdIsDir(ts *testscript.TestScript, neg bool, args []string) {
 	for _, arg := range args {
diff --git a/internal/cmd/testdata/scripts/templatedirectives.txtar b/internal/cmd/testdata/scripts/templatedirectives.txtar
index 63d1f558a08..b95144de399 100644
--- a/internal/cmd/testdata/scripts/templatedirectives.txtar
+++ b/internal/cmd/testdata/scripts/templatedirectives.txtar
@@ -1,17 +1,37 @@
+hexdecode golden/encoding.hex
+
 # test --left-delimiter and --right-delimiter flags to chezmoi execute-template
 exec chezmoi execute-template --left-delimiter=[[ --right-delimiter=]] '[[ "ok" ]]'
 stdout ^ok$
 
+# test that the encoding can be set in files
+exec chezmoi cat $HOME${/}encoding
+cmp stdout golden/encoding
+
+# test that missing key behavior can be set in files
+exec chezmoi cat $HOME${/}missing-key
+cmp stdout golden/missing-key
+
 # test that template delimiters can be set in files
-exec chezmoi cat $HOME${/}template
-cmp stdout golden/template
+exec chezmoi cat $HOME${/}nested-template
+cmp stdout golden/nested-template
 
--- golden/template --
-<no value>(nested)
+-- golden/encoding.hex --
+fffe                                                     # UTF-16 BOM
+480065006c006c006f002c00200077006f0072006c00640021000a00 # "Hello, world!\n"
+-- golden/missing-key --
+<no value>
+-- golden/nested-template --
+(nested)
 -- home/user/.local/share/chezmoi/.chezmoitemplates/nested --
 # chezmoi:template:left-delimiter=(( right-delimiter=))
 ((- . -))
--- home/user/.local/share/chezmoi/template.tmpl --
-# chezmoi:template:left-delimiter=[[ right-delimiter=]]
+-- home/user/.local/share/chezmoi/encoding.tmpl --
+# chezmoi:template:encoding=utf-16-le-bom
+Hello, world!
+-- home/user/.local/share/chezmoi/missing-key.tmpl --
 # chezmoi:template:missing-key=default
-[[ .MissingKey ]]([[ template "nested" "nested" ]])
+{{ .MissingKey }}
+-- home/user/.local/share/chezmoi/nested-template.tmpl --
+# chezmoi:template:left-delimiter=[[ right-delimiter=]]
+([[ template "nested" "nested" ]])