mozilla/rust — pull request #13431 by lifthrasiir (2014-04-09 18:24:53+00:00)
Title: rustdoc: Smaller Search Index

This is a series of inter-related commits which depend on #13402 (Prune the paths that do not appear in the index). Please consider this as an early review request; I'll rebase this when the parent PR get merged.

----

This PR aims at reducing the search index without removing the actual information. In my measurement with both library and compiler docs, the search index is 52% smaller before gzipped, and 16% smaller after gzipped:

```
1719473 search-index-old.js
1503299 search-index.js (after #13402, 13% gain)
724955 search-index-new.js (after this PR, 52% gain w.r.t. #13402)

262711 search-index-old.js.gz
214205 search-index.js.gz (after #13402, 18.5% gain)
179396 search-index-new.js.gz (after this PR, 16% gain w.r.t #13402)
```

Both the uncompressed and compressed size of the search index have been accounted. While the former would be less relevant when #12597 (Web site should be transferring data compressed) is resolved, the uncompressed index will be around for a while anyway and directly affects the UX of docs. Moreover, LZ77 (and gzip) can only remove *some* repeated strings (since its search window is limited in size), so optimizing for the uncompressed size often has a positive effect on the compressed size as well.

Each commit represents the following incremental improvements, in the order:

1. Parent paths were referred by its AST `NodeId`, which tends to be large. We don't need the actual node ID, so we remap them to the smaller sequential numbers. This also means that the list of paths can be a flat array instead of an object.
2. We remap each item type to small predefined numbers. This is strictly intended to reduce the uncompressed size of the search index.
3. We use arrays instead of objects and reconstruct the original objects in the JavaScript code. Since this removes a lot of boilerplates, this affects both the uncompressed and compressed size.
4. (I've found that a centralized `searchIndex` is easier to handle in JS, so I shot one global variable down.)
5. Finally, the repeated paths in the consecutive items are omitted (replaced by an empty string). This also greatly affects both the uncompressed and compressed size.

There had been several unsuccessful attempts to reduce the search index. Especially, I explicitly avoided complex optimizations like encoding paths in a compressed form, and only applied the optimizations when it had a substantial gain compared to the changes. Also, while I've tried to be careful, the lack of proper (non-smoke) tests makes me a bit worry; any advice on this would be appreciated.