Repository: xroche/httrack (2026-07-27T07:07:33Z)

Issue:
--purge-old deletes a mirrored file when the connection drops before the response
A re-fetch whose connection closes before any response arrives leaves the mirrored copy untouched: nothing is written, nothing is cached. The file never reaches `hts-cache/new.lst` either, so the end-of-session purge (`opt->delete_old`, on by default) removes it as if it had vanished from the site.

Same data-loss family as #562, which was closed for the short-`Content-Length` path. A connection that dies before the status line takes a different route into `back_finalize()` and still reproduces on master (f72e7eb).

Repro, against a local server that serves `/reset.bin` in full on the first fetch and, on every later fetch, closes the connection without answering:

1. `httrack --retries=2 -O out http://127.0.0.1:PORT/index.html` mirrors index.html, ok.bin and reset.bin.
2. `httrack --retries=2 --update -O out ...`: all three attempts at reset.bin fail, and the pass-1 bytes sit on disk untouched for the whole crawl.
3. End of session: old.lst lists reset.bin, new.lst does not, and the purge deletes it. The same run with `--purge-old=0` keeps the file, which pins the mechanism.

Found while working on #714 (`--changes`), which computes the same old.lst-minus-new.lst set. The report side now refuses to call such a file gone; the deletion is untouched.



Discussion:

Patch:
diff --git a/src/htsback.c b/src/htsback.c
index 646082a8..1257427f 100644
--- a/src/htsback.c
+++ b/src/htsback.c
@@ -585,6 +585,19 @@ static int create_back_tmpfile(httrackp *opt, lien_back *const back,
   return 0;
 }
 
+/* Did the fetch fail to produce a response, as opposed to the engine
+   deliberately passing the resource over? Only the latter may be purged. */
+static hts_boolean back_transfer_failed(const int statuscode) {
+  switch (statuscode) {
+  case STATUSCODE_TOO_BIG:
+  case STATUSCODE_EXCLUDED:
+  case STATUSCODE_TEST_OK:
+    return HTS_FALSE;
+  default:
+    return statuscode <= 0 ? HTS_TRUE : HTS_FALSE;
+  }
+}
+
 /* Commit or restore a re-fetch backup (#77 follow-up): a re-fetch over an
    existing file moved the good copy to back->tmpfile before truncating url_sav.
    commit keeps the new file and drops the backup; else restore it so an aborted
@@ -1031,6 +1044,14 @@ int back_finalize(httrackp * opt, cache_back * cache, struct_back * sback,
   /* Aborted, error, or not ready: url_sav (if written) is broken; restore the
      previous copy from the backup. */
   back_finalize_backup(opt, &back[p], HTS_FALSE);
+  /* Note the surviving copy, or the end-of-update purge drops what this run
+     never managed to replace (#746). */
+  if (!back[p].testmode && back_transfer_failed(back[p].r.statuscode) &&
+      back[p].url_sav[0] != '\0' && fexist_utf8(back[p].url_sav)) {
+    filenote(&opt->state.strc, back[p].url_sav, NULL);
+    file_notify(opt, back[p].url_adr, back[p].url_fil, back[p].url_sav, 0, 0,
+                back[p].r.notmodified);
+  }
   return -1;
 }
 


Test patch:
diff --git a/tests/96_local-refetch-keep.test b/tests/96_local-refetch-keep.test
index 02bba14d..a493069f 100644
--- a/tests/96_local-refetch-keep.test
+++ b/tests/96_local-refetch-keep.test
@@ -1,19 +1,16 @@
 #!/bin/bash
 #
-# A re-fetch cut mid-header must leave the mirrored file alone (#748): unfixed,
-# the aborted read's leftover buffer lands on it. err.bin is the control, an
-# HTTP 500 on the same resource already keeping its copy; stay.bin answers
-# normally with a new body, so a fix that stopped overwriting anything fails.
-#
-# Purging is off: an update purge deletes both files for an unrelated reason
-# (#746), which would hide what this asserts.
+# A re-fetch cut mid-header must leave the mirrored file alone (#748) and keep
+# it out of the update purge (#746). err.bin is the control, an HTTP 500 on the
+# same resource already surviving both; stay.bin answers normally with a new
+# body, so a fix that stopped overwriting anything fails.
 
 set -euo pipefail
 
 : "${top_srcdir:=..}"
 
 bash "$top_srcdir/tests/local-crawl.sh" \
-    --rerun-args '--update --purge-old=0' \
+    --rerun-args '--update' \
     --log-found 'after 2 retries at link .*keep/page\.html' \
     --log-found 'after 2 retries at link .*keep/data\.bin' \
     --file-matches 'keep/page.html' 'KEEP-PAGE-V1' \