Repository: gohugoio/hugo (language: go)

Issue:
Add .Data.Headers in GetRemote
# Support Link Response fields in GetRemote Result object

[Discourse 49894](https://discourse.gohugo.io/t/support-for-link-header-field-in-getremote-used-for-api-pagination/49894/3)

## Issue

Different APIs use different Response-Header fields to return additional data about the Response.

Especially paginating REST APIs (GitHub, GitLab guess there are many) used in combination with GetRemote are affected.

Hugo currently only provides access to the _Content-Type_ filed through it's Data Interface in
resource Object returned from GetRemote

I would like to have see the Link Header fields for Pagination at the resource.

-  start
-  first
-  next
-  prev(ious)
-  last
-  self

## Link Response Header Field (RFC 5988)

RFC has a complete grammar in [section 5](https://datatracker.ietf.org/doc/html/rfc5988#section-5)

Just the key points for the stuff used in pagination here:

```text
Link                 = "Link" ":" #link-value
  link-value         = "<" URI-Reference ">" *( ";" link-param )
    link-param       = ( "rel" "=" relation-types )             <-- the are more keywords than "rel" and more types
      relation-types = relation-type
                 |     <"> relation-type *( 1*SP relation-type ) <">
```

relation types are defined in
[section 622](https://datatracker.ietf.org/doc/html/rfc5988#section-6.2.2)

commonly used types are

-  self - the link to the page itself
-  help - usually links to the api documentation
-  stylesheet - for XML delivery or transformation
-  start - slighty different definition than first (non native speaker) - and there is no _end_ type
   ;-)

So we have multiple _types_ in a _rel_ field that may link to the same Url. And two different types
for the same (prev and previous) Hurray.

### Example from GitHub

> This is a one line string, newlines added for readability
>
> ```text
> Link:
> <https://api.github.com/repositories/11180687/issues?page=1&per_page=2>; rel="prev",
> <https://api.github.com/repositories/11180687/issues?page=3&per_page=2>; rel="next"
> <https://api.github.com/repositories/11180687/issues?page=233&per_page=2>; rel="last",
> <https://api.github.com/repositories/11180687/issues?page=1&per_page=2>; rel="first"
> <https://www.example.com; rel="self">
> <https://www.example.com; rel="self stylesheet">
> ```

## Implementation thoughts

Implementation effort varies on the degree of Link attribute support

1. only rel=
  1.2 all types
  1.3 list of types (minimal: next,prev(ios),first,last,start)
2 complete Link RFC
3 all or some other Headers

I would vote for 1.2 maybe 1.1 so it's full supported. Guess you have some Go libraries already in
use. The grammar for the Link attribute seems to regex capture handlebar.

### Easy access

should be dead simple $resource.Links.First $resource.Links "first" ... no puzzling around with
walking a structure or index function.

### Should not fail

If the link cannot be parsed completely it should not fail but store it in a String representation
maybe Resource.Header.LinkRaw ...

## Field that may be worth adding

-  _Content-Language_ might be interesting, so one can handle the process according to Hugo site
   languages.

## Further considerations

For simplicity all fields exposed are directly available at the .Data field.

Could be moved to a .Data.Header storing the Response headers. Just one field currently for the
Response-Header, so a deprecation now and duplicate/link it to .Data.Headers. or an interface,
getter ... i'm not in Go. The new link Field might be widely used because of paginating APIs
utilized with GetRemote and Content Adapters. imho if there should be a change, early is better ;-)

Dunno if these will be a conflict candidate later but at lest a point to consider

## Resources

-  [mdn_web_docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link)
-  [Wikipedia](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields)
-  [RFC 5988](https://datatracker.ietf.org/doc/html/rfc5988)



Discussion / hints:
a few hours of 😴  and a quit lesson 🧑‍🎓 in building Hugo  

I would say that the most practical approach would be to add needed Fields only.

the Link is returned as unicoded string and to extract only the needed rel types it looks quite simple.
In `Perl` I would use a little regex magic to create fields for every "rel=" type for each link.
[regex 101](https://regex101.com/r/1sfSDc/2)
```regex
<(.*?)>\s*;\s*.*?(rel="(first|last|next|(prev(:?ious)?)|start|self)(\s+(first|last|next|(prev(:?ious)?)|start|self))?").*?(,|$)
```

I could imagine that it would be good to have the unparsed Link stored, too.

```nocode
.Data.Links [
  rawLink = res.Header.Get("Link")
  first = "https...  
  last = "https"
  ...
]
```

the presence of each link type is determined by the query result. So only available links are in the Response. The first page fe. wouldn't provide a previous link in the response header field

## tested with 

just added a `res.Header.Get("Link")` to `responseToData` and we get a Raw string looks like Unicode.

Dumped in a Template:
```
   {{ $github_rest_api_branches := "https://api.github.com/repos/gohugoio/hugo/branches?per_page=10&page=10" }}
   {{ $github_rest_options := dict
      "headers" (dict "Accept" "application/vnd.github+json" "Content-Type" "application/json")
   }}
   {{ with resources.GetRemote $github_rest_api_branches $github_rest_options }}
      {{ warnf (debug.Dump .Data) }}
   {{ end }}

```
will result in:
```
{
  "ContentLanguage": "",
  "ContentLength": -1,
  "ContentType": "application/json; charset=utf-8",
  "Link": "\u003chttps://api.github.com/repositories/11180687/issues?per_page=10\u0026page=9\u003e; rel=\"prev\", \u003chttps://api.github.com/repositories/11180687/issues?per_page=10\u0026page=11\u003e; rel=\"next\", \u003chttps://api.github.com/repositories/11180687/issues?per_page=10\u0026page=47\u003e; rel=\"last\", \u003chttps://api.github.com/repositories/11180687/issues?per_page=10\u0026page=1\u003e; rel=\"first\"",
  "Status": "200 OK",
  "StatusCode": 200,
  "TransferEncoding": null
}```

Just a thought - What about an _Opt-In_ for complex header fields.... 

```golang
{{ $opts := dict
  "method" "post"
  "body" `{"complete": true}` 
  "headers" (dict  "Content-Type" "application/json")
  "responseHeaderFields= (slice "Links", ...)
}}
```

 Will limit the performance impact if one has many Get-Remote Calls without need for links.
An opt-In would lower this and avoid a performance regression in current usage.



Fix patch:
diff --git a/resources/resource_factories/create/remote.go b/resources/resource_factories/create/remote.go
index 3bfb5f3ccd8..e201339b01d 100644
--- a/resources/resource_factories/create/remote.go
+++ b/resources/resource_factories/create/remote.go
@@ -30,6 +30,7 @@ import (
 
 	"github.com/gohugoio/httpcache"
 	"github.com/gohugoio/hugo/common/hashing"
+	"github.com/gohugoio/hugo/common/hstrings"
 	"github.com/gohugoio/hugo/common/hugio"
 	"github.com/gohugoio/hugo/common/loggers"
 	"github.com/gohugoio/hugo/common/maps"
@@ -51,18 +52,28 @@ type HTTPError struct {
 	Body       string
 }
 
-func responseToData(res *http.Response, readBody bool) map[string]any {
+func responseToData(res *http.Response, readBody bool, includeHeaders []string) map[string]any {
 	var body []byte
 	if readBody {
 		body, _ = io.ReadAll(res.Body)
 	}
 
+	responseHeaders := make(map[string][]string)
+	if true || len(includeHeaders) > 0 {
+		for k, v := range res.Header {
+			if hstrings.InSlicEqualFold(includeHeaders, k) {
+				responseHeaders[k] = v
+			}
+		}
+	}
+
 	m := map[string]any{
 		"StatusCode":       res.StatusCode,
 		"Status":           res.Status,
 		"TransferEncoding": res.TransferEncoding,
 		"ContentLength":    res.ContentLength,
 		"ContentType":      res.Header.Get("Content-Type"),
+		"Headers":          responseHeaders,
 	}
 
 	if readBody {
@@ -72,7 +83,7 @@ func responseToData(res *http.Response, readBody bool) map[string]any {
 	return m
 }
 
-func toHTTPError(err error, res *http.Response, readBody bool) *HTTPError {
+func toHTTPError(err error, res *http.Response, readBody bool, responseHeaders []string) *HTTPError {
 	if err == nil {
 		panic("err is nil")
 	}
@@ -85,7 +96,7 @@ func toHTTPError(err error, res *http.Response, readBody bool) *HTTPError {
 
 	return &HTTPError{
 		error: err,
-		Data:  responseToData(res, readBody),
+		Data:  responseToData(res, readBody, responseHeaders),
 	}
 }
 
@@ -213,7 +224,7 @@ func (c *Client) FromRemote(uri string, optionsm map[string]any) (resource.Resou
 		}
 
 		if res.StatusCode < 200 || res.StatusCode > 299 {
-			return nil, toHTTPError(fmt.Errorf("failed to fetch remote resource from '%s': %s", uri, http.StatusText(res.StatusCode)), res, !isHeadMethod)
+			return nil, toHTTPError(fmt.Errorf("failed to fetch remote resource from '%s': %s", uri, http.StatusText(res.StatusCode)), res, !isHeadMethod, options.ResponseHeaders)
 		}
 
 		var (
@@ -280,7 +291,7 @@ func (c *Client) FromRemote(uri string, optionsm map[string]any) (resource.Resou
 		}
 
 		userKey = filename[:len(filename)-len(path.Ext(filename))] + "_" + userKey + mediaType.FirstSuffix.FullSuffix
-		data := responseToData(res, false)
+		data := responseToData(res, false, options.ResponseHeaders)
 
 		return c.rs.NewResource(
 			resources.ResourceSourceDescriptor{
@@ -345,9 +356,10 @@ func hasHeaderKey(m http.Header, key string) bool {
 }
 
 type fromRemoteOptions struct {
-	Method  string
-	Headers map[string]any
-	Body    []byte
+	Method          string
+	Headers         map[string]any
+	Body            []byte
+	ResponseHeaders []string
 }
 
 func (o fromRemoteOptions) BodyReader() io.Reader {
@@ -432,7 +444,7 @@ func (t *transport) RoundTrip(req *http.Request) (resp *http.Response, err error
 				if resp != nil {
 					msg = resp.Status
 				}
-				err := toHTTPError(fmt.Errorf("retry timeout (configured to %s) fetching remote resource: %s", t.Cfg.Timeout(), msg), resp, req.Method != "HEAD")
+				err := toHTTPError(fmt.Errorf("retry timeout (configured to %s) fetching remote resource: %s", t.Cfg.Timeout(), msg), resp, req.Method != "HEAD", nil)
 				return resp, err
 			}
 			time.Sleep(nextSleep)


Test patch:
diff --git a/resources/resource_factories/create/create_integration_test.go b/resources/resource_factories/create/create_integration_test.go
index 7b9c96e3497..faa2de56549 100644
--- a/resources/resource_factories/create/create_integration_test.go
+++ b/resources/resource_factories/create/create_integration_test.go
@@ -56,7 +56,42 @@ func TestGetRemoteHead(t *testing.T) {
 
 	b.AssertFileContent("public/index.html",
 		"Head Content: .",
-		"Head Data: map[ContentLength:18210 ContentType:image/png Status:200 OK StatusCode:200 TransferEncoding:[]]",
+		"Head Data: map[ContentLength:18210 ContentType:image/png Headers:map[] Status:200 OK StatusCode:200 TransferEncoding:[]]",
+	)
+}
+
+func TestGetRemoteResponseHeaders(t *testing.T) {
+	files := `
+-- config.toml --
+[security]
+  [security.http]
+    methods = ['(?i)GET|POST|HEAD']
+    urls = ['.*gohugo\.io.*']
+-- layouts/index.html --
+{{ $url := "https://gohugo.io/img/hugo.png" }}
+{{ $opts := dict "method" "head" "responseHeaders" (slice "X-Frame-Options" "Server") }}
+{{ with try (resources.GetRemote $url $opts) }}
+  {{ with .Err }}
+    {{ errorf "Unable to get remote resource: %s" . }}
+  {{ else with .Value }}
+    Response Headers: {{ .Data.Headers }}
+  {{ else }}
+  {{ errorf "Unable to get remote resource: %s" $url }}
+  {{ end }}
+{{ end }}
+`
+
+	b := hugolib.NewIntegrationTestBuilder(
+		hugolib.IntegrationTestConfig{
+			T:           t,
+			TxtarString: files,
+		},
+	)
+
+	b.Build()
+
+	b.AssertFileContent("public/index.html",
+		"Response Headers: map[Server:[Netlify] X-Frame-Options:[DENY]]",
 	)
 }