Repository: gohugoio/hugo (language: go)

Issue:
Processed images are published without invoking .Permalink, .RelPermalink or .Publish
According to the [docs](https://gohugo.io/hugo-pipes/introduction/#asset-publishing), assets in the `assetDir` directory are only published by invoking `.Permalink`, `.RelPermalink`, or `.Publish`. This is true for the source images but once image processing methods are applied, the result is automatically published. This can have a significant impact on the size of build artifacts, especially when multiple operations are performed on images.

For my use case, I am resizing two images, merge them with the overlay filter, convert them to webp and then generate a fingerprinted permalink. This leads to a huge number of published image variants which are never used.

**Update:** The issue seems to only occur once the cache exists. When `cacheDir` is deleted before the build, the issue can no longer be reproduced.

### What version of Hugo are you using (`hugo version`)?

<pre>
$ hugo version
hugo v0.102.3-b76146b129d7caa52417f8e914fc5b9271bf56fc+extended linux/amd64 BuildDate=2022-09-01T10:16:19Z VendorInfo=gohugoio
</pre>

### Does this issue reproduce with the latest release?
~~Did not check against v0.102.x because klakegg's Docker images are not available yet but there are no related changes documented for this version.~~
Yes, tested with v0.102.3.


Discussion / hints:
@dgerhardt I am unable to reproduce this:

template 

```
{{ $i := resources.Get "a.jpg" }}
{{ $i = $i.Resize "300x webp" }}
```

commands

```bash
rm -rf resources/ public/
hugo
tree public resources
```

result

```text
public/
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        └── a_hu4cf93b69df91308fdf11e4cd6cd70889_17959_300x0_resize_q75_h2_box.webp
```

The resource is cached, but not published (it's not in the public directory).

I'm probably missing something. Can you provide a more detailed example?



> Can you provide a more detailed example?

@jmooring Sure. You can try the following example to reproduce the issue:
```
{{ $image := resources.GetRemote "https://upload.wikimedia.org/wikipedia/commons/thumb/4/41/Shtandart_%28ship%2C_1999%29%2C_S%C3%A8te_cf02.jpg/500px-Shtandart_%28ship%2C_1999%29%2C_S%C3%A8te_cf02.jpg" }}
{{ $image = $image.Resize "400x" }}
{{ $image = $image.Resize "300x" }}
{{ $image = $image.Resize "200x" }}
{{ $image = $image.Resize "100x" }}
{{ $image = $image.Crop "50x50" }}
{{ $image = $image.Filter (images.GaussianBlur 6) }}
{{ ($image | fingerprint).Permalink }}
```
This will result in 7 published image files:
```
$ ls -1p public/
categories/
_hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
_hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
index.html
index.xml
sitemap.xml
tags/
```
@dgerhardt Using your example I get this:

```text
public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
```

Here's a test repo to try:

```bash
git clone --single-branch -b hugo-github-issue-10255 https://github.com/jmooring/hugo-testing hugo-github-issue-10255
cd hugo-github-issue-10255
hugo
tree public resources
```

The template code is in `layouts/_default/home.html`.
Running hugo v0.102.3.

@jmooring Thank you for further looking into this. I've now switched from the Docker images to the the official v102.3 binary to make sure the version wasn't the cause of the issue. I was able to reproduce the issue with your repository but it only happens once the cache exists.

Fresh start without cache:
```
$ rm -rf public resources
$ hugo
$ tree -F public resources
public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
```
Only delete public this time - keep the cache:
```
$ rm -rf public
$ hugo
$ tree -F public resources
public/
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.cb37525879289f1d2049cb89754bdbbe4b5eff8b16cdb6a7ffbd90c4ce063d5d.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
├── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
└── index.html
resources/
└── _gen/
    ├── assets/
    └── images/
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_76c7cf38bf945b0ffcde1085a8d98901.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_9dc8b79c16d49a9a446321e1b68e4900.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_c2318e86b543017d2554be662dc27ffb.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_e9431206c87606187680d7f79cafe08d.jpg
        ├── _hu21ab6d3fcb85df7d479b326f59a26776_0_edd9e18643cc4ca510ad18a504ab4300.jpg
        └── _hu21ab6d3fcb85df7d479b326f59a26776_0_f56758ff620cf7b63fce42717f97af38.jpg
```
Confirmed. MRE:

```bash
git clone --single-branch -b hugo-github-issue-10255 https://github.com/jmooring/hugo-testing hugo-github-issue-10255
cd hugo-github-issue-10255
hugo && tree public resources # Results are expected
hugo && tree public resources # Results are NOT expected
```
I think I have observed this myself, and I'm also pretty sure I have this fixed in a branch that has taken me a bit longer to get over the finish line than I planned. I have been side stepped a little lately, but I plan to pick up this particular ball on Monday.
@bep Since you mentioned that you might already have a fix: Did you have the chance to have another look at it?
### Reproduced on
- `hugo v0.110.0+extended linux/amd64` (ArchLinux)
- `hugo v0.111.1-39a4a3cf676533859217805c36181c7a3dfa321c+extended windows/amd64` (Windows 10)
- `hugo v0.111.2-4164f8fef9d71f50ef3962897e319ab6219a1dad+extended windows/amd64` (Windows 10)

This also happens with image resources from a **headless bundle**, e. g. a sub-folder of a top-level section, with `publishResources = false`.

It's even worse:
- When I use a custom figure shortcode which does some image processing before generating the `<img src="{{ $image.RelPermalink}}" />`, I get all those intermediate images published and not only the final result.
- When I use the builtin figure shortcode to reference the original image, it's **missing** in the generated output (as soon as I use `publishResources = false`).
- When I use Markdown syntax to reference the original image, i. e. `![Image](./headless/image.png)`, it's **missing** in the generated output (as soon as I use `publishResources = false`)
- When I do not use `publishResources = false`, everything from the **headless bundle** folder gets published, no matter if referenced or not. Even a `Makefile` and a graphviz `.gv` file I use to generate the original image `.png` file.

This is clearly not what I was expecting. The reason why I set `publishResources = false` on the **headless bundle** was that I wanted **exactly** the referenced resources to be published. Neither any intermediate results nor sources I use to generate the resources.

https://gohugo.io/content-management/build-options/#publishresources

Dunno, if it matters, but I have a dual-language setup, i. e. the **headless bundle** has (and needs) an `index.md` and an `index.de.md` file to be usable (be referenced) from pages of both languages.

> I actually/initially thought a **headless bundle** would work like this (only referenced resources are published) without explicitly setting `publishResources = false`. I was also very surprised to see that I needed `index.LANG.md` files to access the **headless bundle** resources from other than the default language.

I'm currently trying to figure out under which conditions a page resource is (supposed to be) published:
- https://github.com/gohugoio/hugo/search?q=publishresources&type=code
- https://github.com/gohugoio/hugo/search?q=lazypublish&type=code
- https://github.com/gohugoio/hugo/search?q=publishonce&type=code
I noticed another situation in which this could cause issues. I had created a custom `image.html` partial that was processing any image I pass to it. I added that image to `./layouts/_default/baseof.html` as I needed it on every page. Since I'm also using `resources.Copy`, I ended up with 2 images (I currently have just 2 pages, didn't test with more pages). The first image was the required image (the one that I produced with `resources.Copy`) and the second one was the one copies from cache.

This was happening, even if I delete my `resources` folder, probably because I am rendering it twice. If I add it just to `index.html` by removing it from `baseof.html`, and delete the `resources` directory, I just get the image once.

I solved this by using `partialCached`, so this might not be as much of an issue, but I thought I'd let you know.
> I solved this by using partialCached

Thanks, that worked for me for a related issue.

I was using a resized image in my `_default/baseof` and the final file in `public/` was ending up empty. Extracting to a partial and using `partialCache` resolved that.

Fix patch:
diff --git a/resources/image_cache.go b/resources/image_cache.go
index 636607f9452..542a6836fc1 100644
--- a/resources/image_cache.go
+++ b/resources/image_cache.go
@@ -106,28 +106,10 @@ func (c *ImageCache) getOrCreate(
 		rp := img.getResourcePaths()
 		rp.relTargetDirFile.file = relTarget.file
 		img.setSourceFilename(info.Name)
+		img.setSourfeFilenameIsHash(true)
 		img.setMediaType(conf.TargetFormat.MediaType())
 
-		if err := img.InitConfig(r); err != nil {
-			return err
-		}
-
-		r.Seek(0, 0)
-
-		w, err := img.openDestinationsForWriting()
-		if err != nil {
-			return err
-		}
-
-		if w == nil {
-			// Nothing to write.
-			return nil
-		}
-
-		defer w.Close()
-		_, err = io.Copy(w, r)
-
-		return err
+		return img.InitConfig(r)
 	}
 
 	// create creates the image and encodes it to the cache (w).
diff --git a/resources/resource.go b/resources/resource.go
index d1acc102681..884900d5870 100644
--- a/resources/resource.go
+++ b/resources/resource.go
@@ -161,7 +161,6 @@ type baseResourceInternal interface {
 	specProvider
 	getResourcePaths() *resourcePathDescriptor
 	getTargetFilenames() []string
-	openDestinationsForWriting() (io.WriteCloser, error)
 	openPublishFileForWriting(relTargetPath string) (io.WriteCloser, error)
 
 	relTargetPathForRel(rel string, addBaseTargetPath, isAbs, isURL bool) string
@@ -216,6 +215,7 @@ func (d dirFile) path() string {
 type fileInfo interface {
 	getSourceFilename() string
 	setSourceFilename(string)
+	setSourfeFilenameIsHash(bool)
 	setSourceFs(afero.Fs)
 	getFileInfo() hugofs.FileMetaInfo
 	hash() (string, error)
@@ -304,6 +304,21 @@ func (l *genericResource) Permalink() string {
 func (l *genericResource) Publish() error {
 	var err error
 	l.publishInit.Do(func() {
+		targetFilenames := l.getTargetFilenames()
+		if l.sourceFilenameIsHash {
+			// This is a processed image. We want to avoid copying it if it hasn't changed.
+			var changedFilenames []string
+			for _, targetFilename := range targetFilenames {
+				if _, err := l.getSpec().BaseFs.PublishFs.Stat(targetFilename); err == nil {
+					continue
+				}
+				changedFilenames = append(changedFilenames, targetFilename)
+			}
+			if len(changedFilenames) == 0 {
+				return
+			}
+			targetFilenames = changedFilenames
+		}
 		var fr hugio.ReadSeekCloser
 		fr, err = l.ReadSeekCloser()
 		if err != nil {
@@ -312,7 +327,7 @@ func (l *genericResource) Publish() error {
 		defer fr.Close()
 
 		var fw io.WriteCloser
-		fw, err = helpers.OpenFilesForWriting(l.spec.BaseFs.PublishFs, l.getTargetFilenames()...)
+		fw, err = helpers.OpenFilesForWriting(l.spec.BaseFs.PublishFs, targetFilenames...)
 		if err != nil {
 			return
 		}
@@ -475,33 +490,6 @@ func (l genericResource) clone() *genericResource {
 	return &l
 }
 
-// returns an opened file or nil if nothing to write (it may already be published).
-func (l *genericResource) openDestinationsForWriting() (w io.WriteCloser, err error) {
-	l.publishInit.Do(func() {
-		targetFilenames := l.getTargetFilenames()
-		var changedFilenames []string
-
-		// Fast path:
-		// This is a processed version of the original;
-		// check if it already exists at the destination.
-		for _, targetFilename := range targetFilenames {
-			if _, err := l.getSpec().BaseFs.PublishFs.Stat(targetFilename); err == nil {
-				continue
-			}
-
-			changedFilenames = append(changedFilenames, targetFilename)
-		}
-
-		if len(changedFilenames) == 0 {
-			return
-		}
-
-		w, err = helpers.OpenFilesForWriting(l.getSpec().BaseFs.PublishFs, changedFilenames...)
-	})
-
-	return
-}
-
 func (r *genericResource) openPublishFileForWriting(relTargetPath string) (io.WriteCloser, error) {
 	return helpers.OpenFilesForWriting(r.spec.BaseFs.PublishFs, r.relTargetPathsFor(relTargetPath)...)
 }
@@ -622,6 +610,9 @@ type resourceFileInfo struct {
 	// the path to the file on the real filesystem.
 	sourceFilename string
 
+	// For performance. This means that whenever the content changes, the filename changes.
+	sourceFilenameIsHash bool
+
 	fi hugofs.FileMetaInfo
 
 	// A hash of the source content. Is only calculated in caching situations.
@@ -654,6 +645,10 @@ func (fi *resourceFileInfo) setSourceFilename(s string) {
 	fi.sourceFilename = s
 }
 
+func (fi *resourceFileInfo) setSourfeFilenameIsHash(b bool) {
+	fi.sourceFilenameIsHash = b
+}
+
 func (fi *resourceFileInfo) getSourceFs() afero.Fs {
 	return fi.sourceFs
 }


Test patch:
diff --git a/hugolib/integrationtest_builder.go b/hugolib/integrationtest_builder.go
index 6a5cd1ce332..c06c9d25f8c 100644
--- a/hugolib/integrationtest_builder.go
+++ b/hugolib/integrationtest_builder.go
@@ -137,6 +137,24 @@ func (s *IntegrationTestBuilder) AssertBuildCountTranslations(count int) {
 	s.Assert(s.H.init.translations.InitCount(), qt.Equals, count)
 }
 
+func (s *IntegrationTestBuilder) AssertFileCount(dirname string, expected int) {
+	s.Helper()
+	fs := s.fs.WorkingDirReadOnly
+	count := 0
+	afero.Walk(fs, dirname, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		if info.IsDir() {
+			return nil
+		}
+		count++
+		return nil
+	})
+	s.Assert(count, qt.Equals, expected)
+
+}
+
 func (s *IntegrationTestBuilder) AssertFileContent(filename string, matches ...string) {
 	s.Helper()
 	content := strings.TrimSpace(s.FileContent(filename))
diff --git a/resources/integration_test.go b/resources/integration_test.go
index 543c264dcf1..2075079dcc2 100644
--- a/resources/integration_test.go
+++ b/resources/integration_test.go
@@ -100,3 +100,42 @@ Width: {{ $svg.Width }}
 	b.Assert(err.Error(), qt.Contains, `error calling Width: this method is only available for raster images. To determine if an image is SVG, you can do {{ if eq .MediaType.SubType "svg" }}{{ end }}`)
 
 }
+
+// Issue 10255.
+func TestNoPublishOfUnusedProcessedImage(t *testing.T) {
+	t.Parallel()
+
+	workingDir := t.TempDir()
+
+	files := `
+-- assets/images/pixel.png --
+iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==
+-- layouts/index.html --
+{{ $image := resources.Get "images/pixel.png" }}
+{{ $image = $image.Resize "400x" }}
+{{ $image = $image.Resize "300x" }}
+{{ $image = $image.Resize "200x" }}
+{{ $image = $image.Resize "100x" }}
+{{ $image = $image.Crop "50x50" }}
+{{ $image = $image.Filter (images.GaussianBlur 6) }}
+{{ ($image | fingerprint).Permalink }}
+
+
+`
+
+	for i := 0; i < 3; i++ {
+
+		b := hugolib.NewIntegrationTestBuilder(
+			hugolib.IntegrationTestConfig{
+				T:           t,
+				TxtarString: files,
+				NeedsOsFS:   true,
+				WorkingDir:  workingDir,
+			}).Build()
+
+		b.AssertFileCount("resources/_gen/images", 6)
+		b.AssertFileCount("public/images", 1)
+		b.Build()
+	}
+
+}