Repository: raviqqe/muffet
Pull request #325 (2023-06-29 13:55:30)

Allow application/xml and text/xml when parsing sitemaps
Update `(*sitemapPageParser).Parse` to support both `application/xml` and `text/xml` as valid MIME types for sitemaps, previously only `application/xml` was considered valid.  However, popular HTTP servers such as Caddy (https://caddyserver.com) return a `Content-Type: text/xml; charset=utf-8` response header when serving a `sitemap.xml` sitemap file.  Currently, it is not possible to use `/sitemap.xml` as the root page with `muffet` when serving a site with an HTTP server such as Caddy due to `muffet` rejecting the `Content-Type` header and thus refusing to parse the site's sitemap file.

The official website for the sitemap protocol
(https://www.sitemaps.org) does not explictly state any specific `Content-Type` that should be used when serving sitemap files. However, the IANA's Media Types registry
(https://www.iana.org/assignments/media-types/media-types.xhtml) lists both `application/xml` and `text/xml` as being valid media types for `xml`.  Additionally, the abstract for RFC 7303 ("XML Media Types", https://www.rfc-editor.org/rfc/rfc7303.html), the reference listed in the IANA registry for both `application/xml` and `text/xml` media types, explicitly states that `text/xml` is an alias for `application/xml`:

    This specification standardizes three media types -- application/xml,
    application/xml-external-parsed-entity, and application/xml-dtd --
    for use in exchanging network entities that are related to the
    Extensible Markup Language (XML) while defining text/xml and text/
    xml-external-parsed-entity as aliases for the respective application/
    types.  This specification also standardizes the '+xml' suffix for
    naming media types outside of these five types when those media types
    represent XML MIME entities.

RFC 7303 Section 9.2 ("text/xml Registration") also states:

    The registration information for text/xml is in all respects the same
    as that given for application/xml above (Section 9.1), except that
    the "Type name" is "text".

Hopefuly, this is enough to warrant allowing `text/xml` in addition to `application/xml`.  A new
`TestSitemapPageParserParsePageMimeTypeAlias` test has been added to ensure that `(*sitemapPageParser).Parse` doesn't return an error when its `typ` argument is `text/xml`.

nwidger: @raviqqe Sorry for not catching this earlier!  We tested the changes you made for #316 running `muffet` against a preview version of our site served with `hugo server`, which is how we check for broken links in our CI pipeline.  `hugo server` returns a `Content-Type: application/xml; charset=utf-8` header when serving a `sitemap.xml` (I believe this is due to [this check here](https://github.com/gohugoio/hugo/blob/master/media/mediaType.go#L81) in `hugo`'s source code), thus in CI the new v2.9.1 release of `muffet` works great.  However, the production site is served using Caddy, which as mentioned above returns a `Content-Type: text/xml; charset=utf-8` header when serving a `sitemap.xml` file.  That means that if we try to run `muffet` v2.9.1 against our production site, `muffet` returns with an `root page has invalid content type` error.  
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/raviqqe/muffet/pull/325?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama) Report
> Merging [#325](https://app.codecov.io/gh/raviqqe/muffet/pull/325?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama) (82e52af) into [main](https://app.codecov.io/gh/raviqqe/muffet/commit/4536090a7347a08922ba722432f342ed0c912e71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama) (4536090) will **not change** coverage.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##             main     #325   +/-   ##
=======================================
  Coverage   87.34%   87.34%           
=======================================
  Files          30       30           
  Lines         861      861           
=======================================
  Hits          752      752           
  Misses         88       88           
  Partials       21       21           
```


| [Impacted Files](https://app.codecov.io/gh/raviqqe/muffet/pull/325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama) | Coverage Δ | |
|---|---|---|
| [sitemap\_page\_parser.go](https://app.codecov.io/gh/raviqqe/muffet/pull/325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama#diff-c2l0ZW1hcF9wYWdlX3BhcnNlci5nbw==) | `100.00% <100.00%> (ø)` | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Yota+Toyama)


Patch:
diff --git a/sitemap_page_parser.go b/sitemap_page_parser.go
index c8c09ad..e3faec2 100644
--- a/sitemap_page_parser.go
+++ b/sitemap_page_parser.go
@@ -16,7 +16,7 @@ func newSitemapPageParser(f linkFilterer) *sitemapPageParser {
 }
 
 func (p *sitemapPageParser) Parse(u *url.URL, typ string, bs []byte) (page, error) {
-	if typ != "application/xml" {
+	if typ != "application/xml" && typ != "text/xml" {
 		return nil, nil
 	}
 


Test patch:
diff --git a/sitemap_page_parser_test.go b/sitemap_page_parser_test.go
index 1cc3a77..0142eac 100644
--- a/sitemap_page_parser_test.go
+++ b/sitemap_page_parser_test.go
@@ -7,7 +7,10 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
-const SITEMAP_MIME_TYPE = "application/xml"
+const (
+	SITEMAP_MIME_TYPE       = "application/xml"
+	SITEMAP_MIME_TYPE_ALIAS = "text/xml"
+)
 
 func TestSitemapPageParserParsePage(t *testing.T) {
 	p, err := newSitemapPageParser(newTestLinkFilterer()).Parse(parseURL(t, "https://foo.com/sitemap.xml"), SITEMAP_MIME_TYPE, []byte(`
@@ -31,6 +34,31 @@ func TestSitemapPageParserParsePage(t *testing.T) {
 	assert.Equal(t, map[string]struct{}(nil), p.Fragments())
 }
 
+func TestSitemapPageParserParsePageMimeTypeAlias(t *testing.T) {
+	p, err := newSitemapPageParser(newTestLinkFilterer()).Parse(parseURL(t, "https://foo.com/sitemap.xml"), SITEMAP_MIME_TYPE_ALIAS, []byte(`
+		<?xml version="1.0" encoding="UTF-8"?>
+		<urlset
+			xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
+			xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"
+			xmlns:xhtml="http://www.w3.org/1999/xhtml"
+			xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
+			xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"
+		>
+			<url>
+				<loc>https://foo.com/</loc>
+			</url>
+		</urlset>
+	`))
+
+	assert.Nil(t, err)
+	if !assert.NotNil(t, p) {
+		return
+	}
+	assert.Equal(t, "https://foo.com/sitemap.xml", p.URL().String())
+	assert.Equal(t, map[string]error{"https://foo.com/": nil}, p.Links())
+	assert.Equal(t, map[string]struct{}(nil), p.Fragments())
+}
+
 func TestSitemapPageParserParseIndexPage(t *testing.T) {
 	p, err := newSitemapPageParser(newTestLinkFilterer()).Parse(parseURL(t, "https://foo.com/sitemap.xml"), SITEMAP_MIME_TYPE, []byte(`
 		<?xml version="1.0" encoding="UTF-8"?>