Repository: CloudCannon/pagefind (language: python)

Issue:
Unable to index compressed pages
How do you index compressed html files?

I have a large static site with around ~100k html files. My static site generator outputs gzipped compressed html files, which get uploaded to S3. The compression saves a ton of money a month on hosting costs, and every modern browser than auto-detect and depress gzipped html files, so it's a necessary publication format.

However, I'm walking through you [getting started docs](https://pagefind.app/docs/config-sources/) and `npx pagefind` is failing when it tries to walk my build directory:

    Running Pagefind v0.9.1 (Extended)
    Running from: "/var/project"
    Source:       "build"
    Bundle Directory:  "_pagefind"

    [Walking source directory]
    Found 90544 files matching **/*.{html}

    [Parsing files]
    Did not find a data-pagefind-body element on the site.
    ↳ Indexing all <body> elements on the site.
    90544 pages found without an <html> element. 
    Pages without an outer <html> element will not be processed by default. 
    If adding this element is not possible, use the root selector config to target a different root element.

    [Reading languages]
    Discovered 0 languages: 

    [Building search indexes]
    Total: 
      Indexed 0 languages
      Indexed 0 pages
      Indexed 0 words
      Indexed 0 filters
      Indexed 0 sorts
    Error: Pagefind wasn't able to build an index. 
    Most likely, the directory passed to Pagefind was empty or did not contain any html files.

So it looks like it's seeing my files but not finding any <html> tags, even though each page definitely contains an <html> tag. Why is this?

My build directory isn't flat and is structured like:

    build
        category1
             subcategory1
                 index.html
             subcategory2
                 index.html
             subcategory3
                 index.html
              ...
         ...

Is it failing to find any files because of my structure, or because of the gzip compression?

Either way, how would I resolve this?

For practicality reasons, I'm unable to change the folder structure nor remove gzip compression.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index d61eaea0..40a9ee76 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -9,7 +9,7 @@ on:
 
 env:
   CARGO_TERM_COLOR: always
-  HUMANE_VERSION: "0.3.7"
+  HUMANE_VERSION: "0.5.0"
   WASM_PACK_VERSION: "v0.10.3"
 
 jobs:
@@ -28,7 +28,7 @@ jobs:
             ~/.cargo/git
             ~/.rustup
             target
-          key: ${{ runner.os }}-stable
+          key: ${{ runner.os }}-stable-min165
       - name: Setup
         run: |
           rustup install stable
diff --git a/pagefind/Cargo.toml b/pagefind/Cargo.toml
index 4ba05aa3..90cc8bd5 100644
--- a/pagefind/Cargo.toml
+++ b/pagefind/Cargo.toml
@@ -56,6 +56,7 @@ lol_html = "0.3"
 html-escape = "0.2.11"
 bit-set = "0.5"
 flate2 = "1.0"
+async-compression = { version = "0.3", features = ["tokio", "gzip"] }
 minifier = "0.2.2"
 sha-1 = "0.10"
 serde_json = "1"
@@ -73,6 +74,7 @@ portpicker = "0.1"
 actix-web = "4"
 actix-files = "0.6"
 lexical-core = "0.8.5"
+path-slash = "0.2"
 
 [features]
 
diff --git a/pagefind/features/errors.feature b/pagefind/features/errors.feature
index 68557dd4..acff3f3f 100644
--- a/pagefind/features/errors.feature
+++ b/pagefind/features/errors.feature
@@ -19,7 +19,7 @@ Feature: Graceful Pagefind Errors
             """
         When I run my program
         Then I should see "Running Pagefind" in stdout
-        Then I should see "Failed to parse file public/dog/index.html" in stdout
+        Then I should see "Failed to parse file" in stdout
         Then I should see the file "public/_pagefind/pagefind.js"
         When I serve the "public" directory
         When I load "/"
diff --git a/pagefind/features/input_quirks.feature b/pagefind/features/input_quirks.feature
new file mode 100644
index 00000000..d3d240e0
--- /dev/null
+++ b/pagefind/features/input_quirks.feature
@@ -0,0 +1,33 @@
+Feature: Input Quirk Tests
+    Background:
+        Given I have the environment variables:
+            | PAGEFIND_SOURCE  | public |
+            | PAGEFIND_verbose | true   |
+        Given I have a "public/index.html" file with the body:
+            """
+            <p data-title>Nothing</p>
+            """
+
+    Scenario: Index gzipped input files
+        Given I have a gzipped "public/cat/index.html" file with the body:
+            """
+            <h1>Hello World</h1>
+            """
+        When I run my program
+        Then I should see "Running Pagefind" in stdout
+        Then I should see the file "public/_pagefind/pagefind.js"
+        When I serve the "public" directory
+        When I load "/"
+        When I evaluate:
+            """
+            async function() {
+                let pagefind = await import("/_pagefind/pagefind.js");
+
+                let search = await pagefind.search("world");
+
+                let data = await search.results[0].data();
+                document.querySelector('[data-title]').innerText = data.meta.title;
+            }
+            """
+        Then There should be no logs
+        Then The selector "[data-title]" should contain "Hello World"
diff --git a/pagefind/features/metadata.feature b/pagefind/features/metadata.feature
index b216317e..73535895 100644
--- a/pagefind/features/metadata.feature
+++ b/pagefind/features/metadata.feature
@@ -47,6 +47,30 @@ Feature: Metadata
             </body>
             </html>
             """
+        # Covering fallback titles
+        Given I have a "public/kea/index.html" file with the content:
+            """
+            <html>
+            <head>
+                <title>Website | Kea</title>
+            </head>
+            <body>
+                <p>NZ Kea Post.</p>
+            </body>
+            </html>
+            """
+        Given I have a "public/kaka/index.html" file with the content:
+            """
+            <html>
+            <head>
+                <title>Website | Kaka</title>
+            </head>
+            <body>
+                <h1></h1>
+                <p>NZ Kaka Post.</p>
+            </body>
+            </html>
+            """
         When I run my program
         Then I should see "Running Pagefind" in stdout
         Then I should see the file "public/_pagefind/pagefind.js"
@@ -166,3 +190,18 @@ Feature: Metadata
             """
         Then There should be no logs
         Then The selector "[data-result]" should contain "Dog Post. | /puppy.jpg | dog | generic | generic"
+
+    Scenario: Title metadata falls back to the title element
+        When I evaluate:
+            """
+            async function() {
+                let pagefind = await import("/_pagefind/pagefind.js");
+
+                let search = await pagefind.search("NZ");
+
+                let data = await Promise.all(search.results.map(result => result.data()));
+                document.querySelector('[data-result]').innerText = data.map(d => d.meta.title).sort().join(', ');
+            }
+            """
+        Then There should be no logs
+        Then The selector "[data-result]" should contain "Website | Kaka, Website | Kea"
diff --git a/pagefind/features/multisite/multisite_base.feature b/pagefind/features/multisite/multisite_base.feature
index bd510d16..2eb97364 100644
--- a/pagefind/features/multisite/multisite_base.feature
+++ b/pagefind/features/multisite/multisite_base.feature
@@ -70,7 +70,7 @@ Feature: Multisite Search
                     }]
                 });
                 pui.triggerSearch("web");
-                await new Promise(r => setTimeout(r, 500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
             }
             """
         Then There should be no logs
diff --git a/pagefind/features/ui/ui_base.feature b/pagefind/features/ui/ui_base.feature
index 0d274c3f..42eb446a 100644
--- a/pagefind/features/ui/ui_base.feature
+++ b/pagefind/features/ui/ui_base.feature
@@ -39,7 +39,7 @@ Feature: Base UI Tests
             """
             async function() {
                 window.pui.triggerSearch("world");
-                await new Promise(r => setTimeout(r, 500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
             }
             """
         Then There should be no logs
diff --git a/pagefind/features/ui/ui_hooks.feature b/pagefind/features/ui/ui_hooks.feature
index a827632b..2870fd02 100644
--- a/pagefind/features/ui/ui_hooks.feature
+++ b/pagefind/features/ui/ui_hooks.feature
@@ -30,7 +30,7 @@ Feature: UI Hooks
             """
             async function() {
                 window.pui.triggerSearch("search");
-                await new Promise(r => setTimeout(r, 500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
 
                 // TODO: Add more web test steps to humane instead of throwing js
                 let el = document.querySelector(".pagefind-ui__result-image");
diff --git a/pagefind/features/ui/ui_strings.feature b/pagefind/features/ui/ui_strings.feature
index 8f9b79b5..92b947c7 100644
--- a/pagefind/features/ui/ui_strings.feature
+++ b/pagefind/features/ui/ui_strings.feature
@@ -29,7 +29,7 @@ Feature: UI Test Strings
             """
             async function() {
                 window.pui.triggerSearch("garbage");
-                await new Promise(r => setTimeout(r, 500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
             }
             """
         Then There should be no logs
@@ -66,7 +66,7 @@ Feature: UI Test Strings
             """
             async function() {
                 window.pui.triggerSearch("garbage");
-                await new Promise(r => setTimeout(r, 500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
             }
             """
         Then There should be no logs
diff --git a/pagefind/src/fossick/mod.rs b/pagefind/src/fossick/mod.rs
index a3caa9d2..5a1af998 100644
--- a/pagefind/src/fossick/mod.rs
+++ b/pagefind/src/fossick/mod.rs
@@ -1,12 +1,15 @@
+use async_compression::tokio::bufread::GzipDecoder;
 #[cfg(feature = "extended")]
 use charabia::Segment;
 use hashbrown::HashMap;
 use lazy_static::lazy_static;
 use pagefind_stem::{Algorithm, Stemmer};
+use path_slash::PathExt as _;
 use regex::Regex;
 use std::io::Error;
 use std::path::{Path, PathBuf};
 use tokio::fs::File;
+use tokio::io::AsyncBufReadExt;
 use tokio::io::{AsyncReadExt, BufReader};
 use tokio::time::{sleep, Duration};
 
@@ -55,16 +58,39 @@ impl Fossicker {
 
         let mut br = BufReader::new(file);
         let mut buf = [0; 20000];
-        while let Ok(read) = br.read(&mut buf).await {
-            if read == 0 {
-                break;
+
+        let is_gzip = if let Ok(read) = br.fill_buf().await {
+            read.len() >= 3 && read[0] == 0x1F && read[1] == 0x8B && read[2] == 0x08
+        } else {
+            false
+        };
+
+        if is_gzip {
+            let mut br = GzipDecoder::new(br);
+            while let Ok(read) = br.read(&mut buf).await {
+                if read == 0 {
+                    break;
+                }
+                if let Err(error) = rewriter.write(&buf[..read]) {
+                    println!(
+                        "Failed to parse file {} — skipping this file. Error:\n{error}",
+                        self.file_path.to_str().unwrap_or("[unknown file]")
+                    );
+                    return Ok(());
+                }
             }
-            if let Err(error) = rewriter.write(&buf[..read]) {
-                println!(
-                    "Failed to parse file {} — skipping this file. Error:\n{error}",
-                    self.file_path.to_str().unwrap_or("[unknown file]")
-                );
-                return Ok(());
+        } else {
+            while let Ok(read) = br.read(&mut buf).await {
+                if read == 0 {
+                    break;
+                }
+                if let Err(error) = rewriter.write(&buf[..read]) {
+                    println!(
+                        "Failed to parse file {} — skipping this file. Error:\n{error}",
+                        self.file_path.to_str().unwrap_or("[unknown file]")
+                    );
+                    return Ok(());
+                }
             }
         }
 
@@ -182,13 +208,20 @@ impl Fossicker {
 }
 
 fn build_url(page_url: &Path, options: &SearchOptions) -> String {
-    let url = page_url
-        .strip_prefix(&options.source)
-        .expect("File was found that does not start with the source directory");
+    let trimmed = page_url.strip_prefix(&options.source);
+    let Ok(url) = trimmed else {
+        options.logger.error(format!(
+            "File was found that does not start with the source directory: {}\nSource: {:?}\nFile: {:?}",
+            trimmed.err().unwrap(),
+            options.source,
+            page_url
+        ));
+        return "/unknown/".to_string();
+    };
 
     format!(
         "/{}",
-        url.to_str().unwrap().to_owned().replace("index.html", "")
+        url.to_slash_lossy().to_owned().replace("index.html", "")
     )
 }
 
@@ -249,5 +282,20 @@ mod tests {
 
         let p: PathBuf = "hello/world/about.html".into();
         assert_eq!(&build_url(&p, &opts), "/about.html");
+
+        let p: PathBuf = "hello/world/about/index.htm".into();
+        assert_eq!(&build_url(&p, &opts), "/about/index.htm");
+    }
+
+    #[cfg(target_os = "windows")]
+    #[test]
+    fn windows_url() {
+        std::env::set_var("PAGEFIND_SOURCE", "C:\\hello\\world");
+        let config =
+            PagefindInboundConfig::with_layers(&[Layer::Env(Some("PAGEFIND_".into()))]).unwrap();
+        let opts = SearchOptions::load(config).unwrap();
+
+        let p: PathBuf = "C:\\hello\\world\\about\\index.htm".into();
+        assert_eq!(&build_url(&p, &opts), "/about/index.htm");
     }
 }
diff --git a/pagefind/src/fossick/parser.rs b/pagefind/src/fossick/parser.rs
index 0d612edb..666e8601 100644
--- a/pagefind/src/fossick/parser.rs
+++ b/pagefind/src/fossick/parser.rs
@@ -263,9 +263,13 @@ impl<'a> DomParser<'a> {
                             }
 
                             // Try to capture the first title on the page (if unset)
-                            if tag_name == "h1" && !data.meta.contains_key("auto_title") {
+                            if tag_name == "h1" && !data.meta.contains_key("auto_title") && !node.current_value.trim().is_empty() {
                                 data.meta.insert("auto_title".into(), normalize_content(&node.current_value));
                             }
+                            // Try to capture the actual title of the page as a fallback for later
+                            if tag_name == "title" && !data.meta.contains_key("auto_page_title") {
+                                data.meta.insert("auto_page_title".into(), normalize_content(&node.current_value));
+                            }
 
                             // If we bail out now, the content won't be persisted anywhere
                             // and the node + children will be dropped.
@@ -463,6 +467,11 @@ impl<'a> DomParser<'a> {
                 data.meta.insert("title".into(), title);
             }
         }
+        if let Some(title) = data.meta.remove("auto_page_title") {
+            if !data.meta.contains_key("title") {
+                data.meta.insert("title".into(), title);
+            }
+        }
 
         // Merge the collected metadata over top of the defaults, if any
         data.default_meta.extend(data.meta);
diff --git a/pagefind/src/output/mod.rs b/pagefind/src/output/mod.rs
index 76684138..5785959b 100644
--- a/pagefind/src/output/mod.rs
+++ b/pagefind/src/output/mod.rs
@@ -4,8 +4,8 @@ use std::time::Duration;
 
 use crate::index::PagefindIndexes;
 use crate::SearchOptions;
-use flate2::write::GzEncoder;
-use flate2::Compression;
+use flate2::write::GzEncoder; // TODO: Replace flate2 with async-compression since we
+use flate2::Compression; //   // require that crate for the input compression anyway.
 use futures::future::join_all;
 use hashbrown::HashMap;
 use include_dir::{include_dir, Dir};


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 2ef472c7..c5dc3439 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -8,7 +8,7 @@ on:
 
 env:
   CARGO_TERM_COLOR: always
-  HUMANE_VERSION: "0.3.7"
+  HUMANE_VERSION: "0.5.0"
   WASM_PACK_VERSION: "v0.10.3"
 
 jobs:
@@ -50,7 +50,7 @@ jobs:
             ~/.cargo/git
             ~/.rustup
             target
-          key: ${{ runner.os }}-${{ matrix.rust }}
+          key: ${{ runner.os }}-${{ matrix.rust }}-min165
 
       - uses: actions/setup-node@v3
         with:
@@ -59,6 +59,12 @@ jobs:
       - name: Install Rust
         run: |
           rustup install ${{ matrix.rust }}
+          rustup default ${{ matrix.rust }}
+
+      - name: Check versions
+        run: |
+          cargo --version
+          rustc --version
 
       - name: Install wasm-pack
         uses: jetli/wasm-pack-action@v0.3.0