Repository: xroche/httrack (2026-06-20T16:12:52Z)

Issue:
Non-HTML files given .html file extension with HTMLFirst enabled
I'm archiving a website and won't need it to work offline, so got rid of the `+*.jpg` etc. rules that were causing externally-hosted images to be included. This changed a few images referenced by a relatively small number of pages to change their file extension from `.ico`, `.jpg` or `.png` to `.html`. I'm concerned about this as the archive will eventually be hosted on a webserver that uses file extensions to automatically determine MIME types, and I anticipate problems if there's a mismatch.

The file extensions and MIME type provided by the original webserver are correct, so I don't know why it's decided to make up new ones.


Discussion:
Some relevant lines from logs etc. for one of the affected files (`vsg-small.png`):
## `hts-log.txt`
```
22:50:25	Warning: 	Unexpected 412/416 error (Requested Range Not Satisfiable) for https://www.openscenegraph.com/images/vsg-small.png, 'C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.html' could not be found on disk
```
## `new.lst`
```
[www.openscenegraph.com/images/vsg-small.png]
[www.openscenegraph.com/images/vsg-small.html]
```
## `new.txt`
```
17:38:50	314/314	------	416	error ('Requested%20Range%20Not%20Satisfiable')	text/html	date:Wed,%2022%20Nov%202023%2016:57:05%20GMT	https://www.openscenegraph.com/images/vsg-small.png	C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.png	(from https://www.openscenegraph.com/)
23:00:37	6895/6895	---M--	200	added ('OK')	image/png	etag:%221aef-57b663281bf20%22	https://www.openscenegraph.com/images/vsg-small.png	C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/www.openscenegraph.com/images/vsg-small.html	(from https://www.openscenegraph.com/images/vsg-small.png)
```

You can view everything from the time it didn't do this (and I included a bunch of stuff from domains I didn't need) at https://github.com/AnyOldName3/OpenSceneGraphDotComBackup/tree/main/OpenSceneGraph/hts-cache
I've done some digging, and haven't determined whether the 416 was caused by HTTrack submitting a dodgy request or by the server doing something wrong. As the file has the right contents and the log mentions the right MIME type, I think it's plausible that the response body was still the correct file and response header still had the right MIME type, and HTTrack automatically changed the file extension because the status code represented an error.
Actually, that's not right - the line that mentions the 416 error mentions `text/html` and it's only the later one that has a 200 status that mentions `image/png`. That would suggest to me that the server's setting the wrong MIME type when it generates an error. I still don't know whether the error is the server's fault or HTTrack's.
Definitely down to an intermittent fault as it's a different set of files affected when rerunning the mirroring process again. I've still not managed to access the server logs, so have no more information about whether it's down to malformed requests or incorrect handling of well-formed requests.
I've looked a bit more, and it's apparently also affecting loads of gzips that didn't have the problem the first time I attempted the mirroring process. As far as I can tell (I committed to a Git repo after the first attempt, so it should be accurate), the only things that changed on HTTrack's end are the following `winprofile.ini` changes:
* `HTMLFirst=1` (was `HTMLFirst=0`, which the GUI says is slower)
* `WildCardFilters=+*.css +*.js -ad.doubleclick.net/* -mime:application/foobar` (was `WildCardFilters=+*.png +*.gif +*.jpg +*.jpeg +*.css +*.js -ad.doubleclick.net/* -mime:application/foobar`, which caused images from other domains to be included)
* `CurrentUrl=https://www.openscenegraph.com/%0d%0alists.openscenegraph.org%0d%0alists.openscenegraph.org/pipermail/osg-users-openscenegraph.org%0d%0awww.openscenegraph.com%0d%0awww.openscenegraph.org` (was `CurrentUrl=https://www.openscenegraph.com/`, which required me to manually include more URLs in interactive mode).

I should probably clarify that the `old.txt` and `old.zip` mentioned below are for a previous partially successful run of the process with the newer settings, not the run with the old settings that I linked to a few posts ago.

Some of these tie in with lines in `new.txt` like
```
20:20:30	76352/76352	---M--	200	added ('OK')	application/gzip	etag:%2212a40-574b8b6a9f35d%22	http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz	C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.html	(from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)
```
In the `old.txt`, the same URL is mentioned in these two lines:
```
17:40:31	76352/76352	---M--	200	added ('OK')	application/gzip	etag:%2212a40-574b8b6a9f35d%22	http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz	C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gzip	(from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)
22:50:33	314/314	UR-MC-	416	error ('Requested%20Range%20Not%20Satisfiable')	text/html	date:Wed,%2022%20Nov%202023%2022:50:33%20GMT	http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gz	C:/Source/OpenSceneGraphDotCom/OpenSceneGraph/lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/2018-August.txt.gzip	(from http://lists.openscenegraph.org/pipermail/osg-users-openscenegraph.org/)
```
Looking in `new.zip`, there's an empty file with the same name, but in `old.zip`, it's there twice (yay, weird quirks of the zip underspecified zip format), once as an empty file, and once containing:
```
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>416 Requested Range Not Satisfiable</title>
</head><body>
<h1>Requested Range Not Satisfiable</h1>
<p>None of the range-specifier values in the Range
request-header field overlap the current extent
of the selected resource.</p>
</body></html>
```

I guess that debunks the theory that the 416 errors still led to the same file contents being served. It also makes it look like the presence of these files in the cache zip from the previous run poisons the next run, even if the file becomes available again.
I've just noticed that I've been using *Download web site(s) + questions* instead of *\* Update existing download*, which I imagine might not have been the best idea for the runs where I didn't delete everything first to start with a clean slate.
Doing a fresh run with the same settings generated no 416 errors at first, then towards the end of the process, the first number in the *Links scanned: 12345/12345 (+1234)* bit reached the same value as the second number, and started counting again from zero. During this phase, a significant percentage of the files fetched generated 416 errors.

As it was a clean run, this can't have been caused by the cache from a previous run poisoning the next run. I don't think this was the setting to deal with HTML files first as there are plenty of non-HTML files before this point in `new.txt`.
Today I tried running this again with the option to fetch HTML files first disabled, like it had been for my initial, successful run. I hit no HTTP 416 errors, and all files were given the correct extension. I still don't know whether the 416 errors were caused by malformed requests from HTTrack or by the server misbehaving, but at least this is no longer giving me grief. I also found it ran in about half the time with the option disabled, which isn't what the tooltip or documentation suggested.



Patch:
diff --git a/src/htsname.c b/src/htsname.c
index 9c4f6a0e2..3fc710302 100644
--- a/src/htsname.c
+++ b/src/htsname.c
@@ -138,6 +138,30 @@ static void cleanEndingSpaceOrDot(char *s) {
   }
 }
 
+/* Should the wire Content-Type override the URL's own extension when naming the
+   saved file? True only when the type is patchable (may_unknown2) and doing so
+   would not clobber a URL extension that already maps to a specific, non-HTML
+   type. This is the #267 mangle guard: a .png served as text/html (or with no
+   type) stays named .png. */
+static int wire_patches_ext(httrackp *opt, const char *wiremime,
+                            const char *file) {
+  char urlmime[256];
+
+  if (may_unknown2(opt, wiremime, file))
+    return 0; /* type kept verbatim (keep-list / bogus-multiple) */
+  urlmime[0] = '\0';
+  /* type implied by the URL extension, only when confidently known (flag 0) */
+  if (!get_httptype_sized(opt, urlmime, sizeof(urlmime), file, 0))
+    return 1; /* URL ext implies no known type: trust the wire type */
+  if (strfield2(wiremime, urlmime))
+    return 0; /* wire agrees with the ext: keep it (no .htm->.html churn) */
+  /* wire disagrees: keep a specific non-HTML ext against an html/empty claim */
+  if (!is_hypertext_mime(opt, urlmime, file) &&
+      (is_html_mime_type(wiremime) || !strnotempty(wiremime)))
+    return 0;
+  return 1;
+}
+
 // forme le nom du fichier à sauver (save) à partir de fil et adr
 // système intelligent, qui renomme en cas de besoin (exemple: deux INDEX.HTML et index.html)
 int url_savename(lien_adrfilsave *const afs,
@@ -325,7 +349,10 @@ int url_savename(lien_adrfilsave *const afs,
   }
 
   /* replace shtml to html.. */
-  if (opt->savename_delayed == HTS_SAVENAME_DELAYED_HARD)
+  /* HARD delays every type, except one the user pinned with --assume: honor it
+     immediately (ishtml() consults the user type), no delayed name (#56) */
+  if (opt->savename_delayed == HTS_SAVENAME_DELAYED_HARD &&
+      !is_userknowntype(opt, fil))
     is_html = -1;               /* ALWAYS delay type */
   else
     is_html = ishtml(opt, fil);
@@ -380,7 +407,7 @@ int url_savename(lien_adrfilsave *const afs,
               if (strnotempty(r.cdispo)) {        /* filename given */
                 ext_chg = 2;      /* change filename */
                 strcpybuff(ext, r.cdispo);
-              } else if (!may_unknown2(opt, r.contenttype, fil)) {        // on peut patcher à priori?
+              } else if (wire_patches_ext(opt, r.contenttype, fil)) {
                 if (give_mimext(s, sizeof(s),
                                 r.contenttype)) { // recognized extension
                   ext_chg = 1;
@@ -425,7 +452,8 @@ int url_savename(lien_adrfilsave *const afs,
               if (strnotempty(headers->r.cdispo)) {        /* filename given */
                 ext_chg = 2;      /* change filename */
                 strcpybuff(ext, headers->r.cdispo);
-              } else if (!may_unknown2(opt, headers->r.contenttype, headers->url_fil)) {    // on peut patcher à priori? (pas interdit ou pas de type)
+              } else if (wire_patches_ext(opt, headers->r.contenttype,
+                                          headers->url_fil)) {
                 char s[16];
                 if (give_mimext(
                         s, sizeof(s),
@@ -653,7 +681,8 @@ int url_savename(lien_adrfilsave *const afs,
                     if (strnotempty(back[b].r.cdispo)) {        /* filename given */
                       ext_chg = 2;      /* change filename */
                       strcpybuff(ext, back[b].r.cdispo);
-                    } else if (!may_unknown2(opt, back[b].r.contenttype, back[b].url_fil)) {    // on peut patcher à priori? (pas interdit ou pas de type)
+                    } else if (wire_patches_ext(opt, back[b].r.contenttype,
+                                                back[b].url_fil)) {
                       if (give_mimext(
                               s, sizeof(s),
                               back[b].r.contenttype)) { // recognized extension


Test patch:
diff --git a/tests/15_local-types.test b/tests/15_local-types.test
new file mode 100644
index 000000000..36ae1b70a
--- /dev/null
+++ b/tests/15_local-types.test
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Content-Type vs URL-extension naming (issue #267 family). Under the default
+# delayed type check (-%N2), a bogus/missing html-ish wire type must not clobber
+# a URL extension that maps to a specific non-HTML type. The .html "mangle" names
+# are asserted absent so a regression that re-introduces it fails here.
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" --errors 0 \
+    --found 'types/notype.png' --not-found 'types/notype.html' \
+    --found 'types/lie.png' --not-found 'types/lie.html' \
+    --found 'types/page.htm' --not-found 'types/page.html' \
+    --found 'types/photo.png' \
+    --found 'types/script.js' \
+    --found 'types/style.css' \
+    --found 'types/data.json' \
+    --found 'types/control.html' --not-found 'types/control.php' \
+    --found 'types/gend61c.png' --not-found 'types/gend61c.html' \
+    httrack 'BASEURL/types/index.html'
diff --git a/tests/16_local-assume.test b/tests/16_local-assume.test
new file mode 100644
index 000000000..8422ac6bc
--- /dev/null
+++ b/tests/16_local-assume.test
@@ -0,0 +1,11 @@
+#!/bin/bash
+#
+# --assume under the default delayed type check (-%N2), issue #56. A user type
+# pinned with --assume must be honored immediately, not lost to the delayed
+# name: photo.png served as image/png but assumed text/html is saved as .html.
+
+: "${top_srcdir:=..}"
+
+bash "$top_srcdir/tests/local-crawl.sh" --errors 0 \
+    --found 'types/photo.html' --not-found 'types/photo.png' \
+    httrack 'BASEURL/types/photo.png' --assume png=text/html
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 6e2535feb..74c70796c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -51,6 +51,8 @@ TESTS = \
 	12_crawl_https.test \
 	13_crawl_proxy_https.test \
 	13_local-cookies.test \
-	14_local-https.test
+	14_local-https.test \
+	15_local-types.test \
+	16_local-assume.test
 
 CLEANFILES = check-network_sh.cache
diff --git a/tests/local-server.py b/tests/local-server.py
index d16330578..90d7f5c46 100755
--- a/tests/local-server.py
+++ b/tests/local-server.py
@@ -118,11 +118,69 @@ def route_robots(self):
         if self.command != "HEAD":
             self.wfile.write(body)
 
+    # --- type/extension matrix (issue #267 family) -------------------------
+
+    def send_raw(self, body, content_type):
+        """Send a raw body with an explicit Content-Type, or none at all when
+        content_type is None (to observe httrack's typeless-file naming)."""
+        self.send_response(200)
+        if content_type is not None:
+            self.send_header("Content-Type", content_type)
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        if self.command != "HEAD":
+            self.wfile.write(body)
+
+    # A fake-binary PNG-ish blob for the image/typeless cases.
+    FAKE_PNG = b"\x89PNG\r\n\x1a\n" + b"\x00" * 64
+
+    # path -> (body, content_type); content_type None means no header at all.
+    TYPE_MATRIX = {
+        "/types/control.php": (b"<html><body>control</body></html>", "text/html"),
+        "/types/photo.png": (FAKE_PNG, "image/png"),
+        "/types/notype.png": (FAKE_PNG, None),
+        "/types/lie.png": (FAKE_PNG, "text/html"),
+        "/types/page.htm": (b"<html><body>htm page</body></html>", "text/html"),
+        "/types/script.js": (b"var x = 1;\n", "application/javascript"),
+        "/types/style.css": (b"body { color: red; }\n", "text/css"),
+        "/types/data.json": (b'{"k": "v"}\n', "application/json"),
+        "/types/gen.php": (FAKE_PNG, "image/png"),
+    }
+
+    def route_types_index(self):
+        body = (
+            '\t<a href="control.php">control</a>\n'
+            '\t<img src="photo.png" />\n'
+            '\t<img src="notype.png" />\n'
+            '\t<img src="lie.png" />\n'
+            '\t<a href="page.htm">htm</a>\n'
+            '\t<script src="script.js"></script>\n'
+            '\t<link rel="stylesheet" href="style.css" />\n'
+            '\t<a href="data.json">json</a>\n'
+            '\t<img src="gen.php?id=5" />\n'
+        )
+        self.send_html(body)
+
+    def route_types(self):
+        path = urlsplit(self.path).path
+        body, ctype = self.TYPE_MATRIX[path]
+        self.send_raw(body, ctype)
+
     ROUTES = {
         "/cookies/entrance.php": route_entrance,
         "/cookies/second.php": route_second,
         "/cookies/third.php": route_third,
         "/robots.txt": route_robots,
+        "/types/index.html": route_types_index,
+        "/types/control.php": route_types,
+        "/types/photo.png": route_types,
+        "/types/notype.png": route_types,
+        "/types/lie.png": route_types,
+        "/types/page.htm": route_types,
+        "/types/script.js": route_types,
+        "/types/style.css": route_types,
+        "/types/data.json": route_types,
+        "/types/gen.php": route_types,
     }
 
     # --- dispatch ----------------------------------------------------------