Repository: google/yamlfmt
Pull request #222 (2024-11-25 14:14:17)

Extensions
I want to format templated YAML files, which have file names like `config.yaml.gotmpl`. We also have other files, such as `README.md.gotmpl` which we do *not* want to format with yamlfmt.

Previously, yamlfmt used `strings.Split()` to extract the extension from a file. This doesn't work when the extension itself contains a dot.

With this change, yamlfmt uses `strings.HasSuffix()` to match files instead, allowing to match multi-part extensions.

That said, I'll probably end up using the "doublestar" implementation – I just didn't understand what it does before reading the code. If you end up grabbing the doc update only, I'd be happy too.

## Alternative

For my purposes, I'd actually love to have a single file to list includes *and* excludes. The "Git ignore" syntax allows you to do this, for example:

```gitignore
*.yaml
*.yml

# Ignore testdata directories
!testdata/
```

There are plenty of libraries around that make implementing this easy. I've used [github.com/go-git/go-git/v5/plumbing/format/gitignore](https://pkg.go.dev/github.com/go-git/go-git/v5/plumbing/format/gitignore) in the past, but [github.com/sabhiram/go-gitignore](https://pkg.go.dev/github.com/sabhiram/go-gitignore) appears to also support negation. What do you think about the approach? Useful addition or just more confusion for users?

braydonk: Thanks for the PR! I think the change looks good so I'll accept the code and docs changes. (I always appreciate docs improvements especially!)

Responding to some comments in the PR description:

> I want to format templated YAML files

It's worth noting that formatting does require a valid yaml document as the input, since the underlying parsing library parses the document into an AST first. So if any of the Go templating makes it no longer a valid yaml file then `yamlfmt` will fail. This has been an issue with other users trying to use `yamlfmt` on things like Helm charts for example, which are Go templated. However, depending on where you use the templating it can still be possible to interpret it as a valid yaml document.

> For my purposes, I'd actually love to have a single file to list includes and excludes.

Will the [`include` and `exclude` sections](https://github.com/google/yamlfmt/blob/main/docs/config-file.md#configuration) not work in this case? They follow the same rules as specifying via command line flags. 

Patch:
diff --git a/docs/paths.md b/docs/paths.md
index 94330eb..a98236c 100644
--- a/docs/paths.md
+++ b/docs/paths.md
@@ -6,10 +6,36 @@
 
 In standard path mode, you can specify a file or directory path directly. If specifying a file, it will simply include the file. If specifying a directory, it will include every file with the correct extension (as specified in `extensions`, default is `yml` and `yaml`).
 
+This mode does *not* support wildcards, aka. globbing. That means with `*.yaml` yamlfmt will look for a file named asterisk dot yaml. If you require globbing, use the [Doublestar mode](#doublestar) instead.
+
 ## Doublestar
 
-In Doublestar mode, paths are specified using the format explained in the [doublestar](https://github.com/bmatcuk/doublestar) package. It is almost identical to bash and git's style of glob pattern specification.
+In Doublestar mode, paths are specified using wildcard patterns explained in the [doublestar](https://github.com/bmatcuk/doublestar) package. It is almost identical to bash and git's style of glob pattern specification.
+
+To enable the doublestar mode, set `doublestar: true` in the config file or use the `-dstar` command line flag.
 
 ## Include and Exclude
 
 In both modes, `yamlfmt` will allow you to configure include and exclude paths. These can be paths to files in Standard or Doublestar modes, paths to directories in Standard mode, and valid doublestar patterns in Doublestar mode. These paths should be specified **relative to the working directory of `yamlfmt`**. They will work as absolute paths if both the includes and excludes are specified as absolute paths or if both are relative paths, however it will not work as expected if they are mixed together. It usually easier to reason about includes and excludes when always specifying both as relative paths from the directory `yamlfmt` is going to be run in.
+
+Exclude paths can be specified on the command line using the `-exclude` flag.
+Paths excluded from the command line are **added* to excluded paths from the config file.
+
+Include paths can be specified on the command line via the positional arguments, i.e. there is no flag for it.
+Paths from the command line take precedence over and **replace** any paths configured in the config file.
+
+yamlfmt will build a list of all files to format using the include list, then discard any files matching the exclude list.
+
+## Extensions
+
+*Only in standard mode*
+
+By default, yamlfmt formats all files ending in `.yaml` and `.yml`.
+You can modify this behavior using the config file and command line flags.
+
+The config file **sets** the list of extensions.
+For example, with `extensions: ["foo"]`, yamlfmt will only match files ending in `.foo` and will *not* match files ending in `.yaml` or `.yml`.
+An empty list triggers the default behavior.
+
+The `-extensions` command line flag **adds** to the list of extensions from the config file.
+For example, `-extensions yaml.gotmpl` will match files ending in `.yaml.gotmpl` *in addition to* files ending in `.yaml` and `.yml`.
diff --git a/path_collector.go b/path_collector.go
index 7b7fbcc..2b8d516 100644
--- a/path_collector.go
+++ b/path_collector.go
@@ -95,18 +95,13 @@ func (c *FilepathCollector) CollectPaths() ([]string, error) {
 }
 
 func (c *FilepathCollector) walkDirectoryForYaml(dir string) ([]string, error) {
-	paths := []string{}
+	var paths []string
 	err := filepath.Walk(dir, func(path string, info fs.FileInfo, err error) error {
 		if info.IsDir() {
 			return nil
 		}
 
-		extension := ""
-		if strings.Contains(info.Name(), ".") {
-			nameParts := strings.Split(info.Name(), ".")
-			extension = nameParts[len(nameParts)-1]
-		}
-		if collections.SliceContains(c.Extensions, extension) {
+		if c.extensionMatches(info.Name()) {
 			paths = append(paths, path)
 		}
 
@@ -115,6 +110,20 @@ func (c *FilepathCollector) walkDirectoryForYaml(dir string) ([]string, error) {
 	return paths, err
 }
 
+func (c *FilepathCollector) extensionMatches(name string) bool {
+	for _, ext := range c.Extensions {
+		// Users may specify "yaml", but we only want to match ".yaml", not "buyaml".
+		if !strings.HasPrefix(ext, ".") {
+			ext = "." + ext
+		}
+
+		if strings.HasSuffix(name, ext) {
+			return true
+		}
+	}
+	return false
+}
+
 type DoublestarCollector struct {
 	Include []string
 	Exclude []string


Test patch:
diff --git a/path_collector_test.go b/path_collector_test.go
index 9d6d90f..0ccf06e 100644
--- a/path_collector_test.go
+++ b/path_collector_test.go
@@ -193,6 +193,26 @@ func TestFilepathCollector(t *testing.T) {
 				"y.yaml": {},
 			},
 		},
+		{
+			name: "multi-part extension",
+			files: []tempfile.Path{
+				{FileName: "x.yaml"},
+				{FileName: "y.yaml.gotmpl"},
+				{FileName: "z.json"},
+			},
+			includePatterns: testPatterns{
+				{pattern: ""}, // with the test this functionally means the whole temp dir
+			},
+			extensions: []string{
+				"yaml",
+				"yml",
+				"yaml.gotmpl",
+			},
+			expectedFiles: collections.Set[string]{
+				"x.yaml":        {},
+				"y.yaml.gotmpl": {},
+			},
+		},
 	}.runAll(t, useFilepathCollector)
 }