January 2024 data update

README.md

@@ -8,16 +8,14 @@ library_name: txtai
 tags:
 - sentence-similarity
 datasets:
-- olm/olm-wikipedia-20221220
+- neuml/wikipedia-20240101
 ---
 
 # Wikipedia txtai embeddings index
 
 This is a [txtai](https://github.com/neuml/txtai) embeddings index for the [English edition of Wikipedia](https://en.wikipedia.org/).
 
-This index is built from the [OLM Wikipedia December 2022 dataset](https://huggingface.co/datasets/olm/olm-wikipedia-20221220).
-Only the first paragraph of the [lead section](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section) from each article is included in the index.
-This is similar to an abstract of the article.
+This index is built from the [Wikipedia January 2024 dataset](https://huggingface.co/datasets/neuml/wikipedia-20240101). Only the first paragraph of the [lead section](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Lead_section) from each article is included in the index. This is similar to an abstract of the article.
 
 It also uses [Wikipedia Page Views](https://dumps.wikimedia.org/other/pageviews/readme.html) data to add a `percentile` field. The `percentile` field can be used
 to only match commonly visited pages.
@@ -49,7 +47,43 @@ embeddings.search("""
 
 An embeddings index generated by txtai is a fully encapsulated index format. It doesn't require a database server or dependencies outside of the Python install.
 
-The Wikipedia index works well as a fact-based context source for conversational search. In other words, search results from this model can be passed to LLM prompts as the
-context in which to answer questions.
+The Wikipedia index works well as a fact-based context source for retrieval augmented generation (RAG). In other words, search results from this model can be passed to LLM prompts as the context in which to answer questions.
 
 See this [article](https://neuml.hashnode.dev/embeddings-in-the-cloud) for additional examples on how to use this model.
+
+## Build the index
+
+The following steps show how to build this index. These scripts are using the latest data available as of 2024-01-01, update as appropriate.
+
+- Install required build dependencies
+```bash
+pip install txtchat mwparserfromhell datasets
+```
+
+- Download and build pageviews database
+```bash
+mkdir -p pageviews/data
+wget -P pageviews/data https://dumps.wikimedia.org/other/pageview_complete/monthly/2023/2023-12/pageviews-202312-user.bz2
+python -m txtchat.data.wikipedia.views -p en.wikipedia -v pageviews
+```
+
+- Build Wikipedia dataset
+
+```python
+from datasets import load_dataset
+
+# Data dump date from https://dumps.wikimedia.org/enwiki/
+date = "20240101"
+
+# Build and save dataset
+ds = load_dataset("neuml/wikipedia", language="en", date=date)
+ds.save_to_disk(f"wikipedia-{date}")
+```
+
+- Build txtai-wikipedia index
+```bash
+python -m txtchat.data.wikipedia.index \
+       -d wikipedia-20240101 \
+       -o txtai-wikipedia \
+       -v pageviews/pageviews.sqlite
+```

config.json

@@ -14,15 +14,15 @@
   "content": true,
   "dimensions": 768,
   "backend": "faiss",
-  "offset": 6013092,
+  "offset": 6172387,
   "build": {
-    "create": "2023-02-20T21:57:46Z",
-    "python": "3.7.16",
+    "create": "2024-01-10T20:34:13Z",
+    "python": "3.8.18",
     "settings": {
-      "components": "IVF2193,SQ8"
+      "components": "IVF2222,SQ8"
     },
     "system": "Linux (x86_64)",
-    "txtai": "5.4.0"
+    "txtai": "6.4.0"
   },
-  "update": "2023-02-20T21:57:46Z"
+  "update": "2024-01-10T20:34:13Z"
 }

documents

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fecb90d975ac6d48caabd1b7a5a4b94350b1af6c052e28dbc6fab4afa6051708
-size 3138019328
+oid sha256:3074eee183918963965a7cdb7c6371150131bf9d99e04d141b20095cd8183b2c
+size 3237478400

embeddings

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb7bd5798472eb259459edf5037231198814b639649239b05799253db2df8529
-size 4672920160
+oid sha256:6f4dea2a142773ae579125431036c7adb632e597a7cd9ed04fbe5473e5f83201
+size 4796622400