Repository: CloudCannon/pagefind (language: python)

Issue:
Normalize user's search text
I'm interested in normalizing the user's input text when doing a search.

For example, in the language I'm working with, the user might type in `ײ` (a single Unicode character) and I want to normalize it to `יי` (two Unicode characters) before doing the search, because my indexed pages use the two-character version. This is somewhat analogous in English to making text lowercase, or in other languages to stripping away _àccênt_ symbols, if the nuance is not important for search.

I've been able to achieve this by using this code on my search page:
```
<div id="search"></div>
<script>
    let pagefind = new PagefindUI({ element: "#search" });
    $("#search").on("keyup", function() {
      normalizedInput = normalizeUnicode(document.getElementsByTagName("input")[0].value);
      pagefind.triggerSearch(normalizedInput);
    });
</script>
```
where I've defined the function `normalizeUnicode(text)` elsewhere to make the necessary string replacements. However, this actually overwrites what's in the input box itself. I want to leave the user's input text as is (visually), but search using the replacements I've defined. (For example, if a user searched `áccent`, I want to search for `accent` but still leave the form `áccent` in the input box.) Is this possible? Thanks!


Discussion / hints:
This is a good call! 

Ideally, normalizing between `áccent` and `accent` will be done automatically by Pagefind — currently just a few things around punctuation and casing are normalized. 

I haven't yet determined the best place to perform this logic, since I would like it to be bidirectional like the current normalization. A search for `accent` that returns `áccent` will need to have logic implemented in the Rust portion of the codebase, so it's a little harder to hook custom logic in for that direction. (i.e. in this case having `יי` return results for `ײ` — the search _result_ needs to be normalized, not the search term)

As a step in the right direction I could add an option to PagefindUI similar to [processResult](https://pagefind.app/docs/ui/#process-result), something like:

```js
new PagefindUI({
    element: "#search",
    processTerm: function (term) {
        return normalizeUnicode(term);
    }
});
```

That wouldn't handle the bidirectional aspect, but it sounds like it might still fit your use-case. Thoughts?
Your approach with `processTerm` could work very well for me! I agree that there is value in performing the same kind of text normalization on the search terms and the documents, but in my case the documents are already pretty much standardized, and so I just want to make some tweaks to the user's search term.

If `processTerm` would allow me to normalize the search term and perform a search, but **not** show the user what manipulations I've made to their term, that is great for me. Thank you!

(P.S. your video on pagefind is excellent and inspired me to use the tool on my jekyll site.)
Awesome to hear! I'll look at implementing this very soon 🙂

Fix patch:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 4cf3ff0f..cf931624 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -29,9 +29,14 @@ jobs:
             ~/.rustup
             target
           key: ${{ runner.os }}-stable-min165
-      - name: Setup
-        run: |
-          rustup install stable
+          
+      - name: Install Rust
+        uses: actions-rs/toolchain@v1
+        with:
+            toolchain: stable
+            override: true
+            default: true
+            components: rustfmt, clippy
 
       - name: Install wasm-pack
         uses: jetli/wasm-pack-action@v0.3.0
@@ -257,11 +262,16 @@ jobs:
           echo "TARGET_CC=clang" >> $GITHUB_ENV
           echo "CFLAGS_aarch64_unknown_linux_musl=--sysroot=/usr/aarch64-linux-gnu" >> $GITHUB_ENV
           echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=/usr/aarch64-linux-gnu/bin/ld" >> $GITHUB_ENV
+      
       - name: Install Rust
-        run: |
-          rustup install ${{ matrix.rust }}
-          rustup target add ${{ matrix.target }}
-          rustup show
+        uses: actions-rs/toolchain@v1
+        with:
+            toolchain: ${{ matrix.rust }}
+            target: ${{ matrix.target }}
+            override: true
+            default: true
+            components: rustfmt, clippy
+
       - name: Install wasm-pack
         uses: jetli/wasm-pack-action@v0.3.0
         with:
diff --git a/pagefind/features/multisite/multisite_base.feature b/pagefind/features/multisite/multisite_base.feature
index 38f914a6..76566f68 100644
--- a/pagefind/features/multisite/multisite_base.feature
+++ b/pagefind/features/multisite/multisite_base.feature
@@ -67,7 +67,7 @@ Feature: Multisite Search
                     }]
                 });
                 pui.triggerSearch("web");
-                await new Promise(r => setTimeout(r, 2500)); // TODO: await el in humane
+                await new Promise(r => setTimeout(r, 3500)); // TODO: await el in humane
             }
             """
         Then There should be no logs
@@ -147,5 +147,5 @@ Feature: Multisite Search
                 }
             }
             """
-        # There will be a log since the versions do not match
-        # Then There should be no logs
+# There will be a log since the versions do not match
+# Then There should be no logs
diff --git a/pagefind/features/ui/ui_hooks.feature b/pagefind/features/ui/ui_hooks.feature
index 2870fd02..083d331c 100644
--- a/pagefind/features/ui/ui_hooks.feature
+++ b/pagefind/features/ui/ui_hooks.feature
@@ -3,6 +3,40 @@ Feature: UI Hooks
         Given I have the environment variables:
             | PAGEFIND_SOURCE | public |
 
+    Scenario: Pagefind UI can provide a hook to process search terms
+        Given I have a "public/index.html" file with the body:
+            """
+            <h1>Search</h1>
+            <div id="search"></div>
+            <script src="/_pagefind/pagefind-ui.js" type="text/javascript"></script>
+
+            <script>
+                window.pui = new PagefindUI({
+                    element: "#search",
+                    processTerm: (t) => t.replace("word", "search"),
+                });
+            </script>
+            """
+        When I run my program
+        Then I should see "Running Pagefind" in stdout
+        Then I should see the file "public/_pagefind/pagefind.js"
+        When I serve the "public" directory
+        When I load "/"
+        When I evaluate:
+            """
+            async function() {
+                window.pui.triggerSearch("word");
+                await new Promise(r => setTimeout(r, 1500)); // TODO: await el in humane
+
+                // TODO: Add more web test steps to humane instead of throwing js
+                let el = document.querySelector(".pagefind-ui__result-link");
+                if (el.getAttribute("href") !== "/") {
+                    throw new Error("Search term should have been normalized by processTerm");
+                }
+            }
+            """
+        Then There should be no logs
+
     Scenario: Pagefind UI can provide a hook to process results
         Given I have a "public/index.html" file with the body:
             """
diff --git a/pagefind/src/output/stubs/search.js b/pagefind/src/output/stubs/search.js
index 00fa2d30..a8437e1b 100644
--- a/pagefind/src/output/stubs/search.js
+++ b/pagefind/src/output/stubs/search.js
@@ -153,9 +153,13 @@ class PagefindInstance {
             let compressed_wasm = await fetch(wasm_url);
             compressed_wasm = await compressed_wasm.arrayBuffer();
             const final_wasm = this.decompress(new Uint8Array(compressed_wasm), "Pagefind WebAssembly");
+            if (!final_wasm) {
+                throw new Error("No WASM after decompression");
+            }
             this.wasm = await this.backend(final_wasm);
         } catch (e) {
             console.error(`Failed to load the Pagefind WASM:\n${e.toString()}`);
+            throw new Error(`Failed to load the Pagefind WASM:\n${e.toString()}`);
         }
     }
 
diff --git a/pagefind_ui/svelte/ui.svelte b/pagefind_ui/svelte/ui.svelte
index 951f1b64..dbc34a18 100644
--- a/pagefind_ui/svelte/ui.svelte
+++ b/pagefind_ui/svelte/ui.svelte
@@ -23,6 +23,7 @@
     export let reset_styles = true;
     export let show_images = true;
     export let process_result = null;
+    export let process_term = null;
     export let show_empty_filters = true;
     export let debounce_timeout_ms = 300;
     export let pagefind_options = {};
@@ -143,6 +144,9 @@
 
     const search = async (term, filters) => {
         search_term = term || "";
+        if (typeof process_term === "function") {
+            term = process_term(term);
+        }
         loading = true;
         searched = true;
         await waitForApiInit();
diff --git a/pagefind_ui/ui.js b/pagefind_ui/ui.js
index 6ff9c806..bbed16f5 100644
--- a/pagefind_ui/ui.js
+++ b/pagefind_ui/ui.js
@@ -19,6 +19,7 @@ class PagefindUI {
         let resetStyles = opts.resetStyles ?? true;
         let showImages = opts.showImages ?? true;
         let processResult = opts.processResult ?? null;
+        let processTerm = opts.processTerm ?? null;
         let showEmptyFilters = opts.showEmptyFilters ?? true;
         let debounceTimeoutMs = opts.debounceTimeoutMs ?? 300;
         let mergeIndex = opts.mergeIndex ?? [];
@@ -30,6 +31,7 @@ class PagefindUI {
         delete opts["resetStyles"];
         delete opts["showImages"];
         delete opts["processResult"];
+        delete opts["processTerm"];
         delete opts["showEmptyFilters"];
         delete opts["debounceTimeoutMs"];
         delete opts["mergeIndex"];
@@ -44,6 +46,7 @@ class PagefindUI {
                     reset_styles: resetStyles,
                     show_images: showImages,
                     process_result: processResult,
+                    process_term: processTerm,
                     show_empty_filters: showEmptyFilters,
                     debounce_timeout_ms: debounceTimeoutMs,
                     merge_index: mergeIndex,
diff --git a/pagefind_web/build.js b/pagefind_web/build.js
new file mode 100644
index 00000000..dd5cb820
--- /dev/null
+++ b/pagefind_web/build.js
@@ -0,0 +1,12 @@
+const fs = require("fs");
+const path = require("path");
+
+const web_js_file = path.join(__dirname, "pkg/pagefind_web.js");
+let web_js = fs.readFileSync(web_js_file, { encoding: "utf-8" });
+
+// document.currentScript.src breaks in the module context
+// and is never a code path we need, so we nix that from the
+// generated javascript.
+web_js = web_js.replace(/document\..?currentScript\..?src/, `"UNHANDLED"`);
+
+fs.writeFileSync(web_js_file, web_js);
diff --git a/pagefind_web/local_build.sh b/pagefind_web/local_build.sh
index 904bdc57..9269512f 100755
--- a/pagefind_web/local_build.sh
+++ b/pagefind_web/local_build.sh
@@ -23,6 +23,7 @@ if [ "$1" = "debug" ]; then
 else
     wasm-pack build --release -t no-modules 
 fi
+node build.js
 mv pkg/pagefind_web.js ../pagefind/vendor/pagefind_web.$WASM_VERSION.js
 # Append pagefind_dcd to the decompressed wasm as a magic word read by the frontend
 printf 'pagefind_dcd' > ../pagefind/vendor/wasm/pagefind_web_bg.unknown.$WASM_VERSION.wasm
diff --git a/pagefind_web/local_fast_build.sh b/pagefind_web/local_fast_build.sh
index 853dda30..c951f952 100755
--- a/pagefind_web/local_fast_build.sh
+++ b/pagefind_web/local_fast_build.sh
@@ -23,6 +23,7 @@ if [ "$1" = "debug" ]; then
 else
     wasm-pack build --release -t no-modules 
 fi
+node build.js
 mv pkg/pagefind_web.js ../pagefind/vendor/pagefind_web.$WASM_VERSION.js
 # Append pagefind_dcd to the decompressed wasm as a magic word read by the frontend
 printf 'pagefind_dcd' > ../pagefind/vendor/wasm/pagefind_web_bg.unknown.$WASM_VERSION.wasm


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 9ed0ab2b..7b745b50 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -57,9 +57,13 @@ jobs:
           node-version: 16
 
       - name: Install Rust
-        run: |
-          rustup install ${{ matrix.rust }}
-          rustup default ${{ matrix.rust }}
+        uses: actions-rs/toolchain@v1
+        with:
+            toolchain: ${{ matrix.rust }}
+            target: ${{ matrix.target }}
+            override: true
+            default: true
+            components: rustfmt, clippy
 
       - name: Check versions
         run: |
@@ -71,9 +75,6 @@ jobs:
         with:
           version: ${{env.WASM_PACK_VERSION}}
 
-      - name: Install Chrome
-        uses: browser-actions/setup-chrome@latest
-
       - name: Install humane
         uses: supplypike/setup-bin@v1
         with: