Repository: binh-vu/kgdata (language: python)

Issue:
Cannot initiate wikidata.db.WDProxyDB
I cannot create an instance of WDProxyDB because it inherits RocksDBDict, which is a class created by Rust Pyo3. The reason is that RocksDBDict's subclass must have the same `__init__`'s signature as in RocksDBDict.


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 9a704c2..5c3ba5b 100644
--- a/README.md
+++ b/README.md
@@ -1,93 +1,20 @@
-<h1 align="center">kgdata</h1>
+# kgdata ![PyPI](https://img.shields.io/pypi/v/kgdata) ![Documentation](https://readthedocs.org/projects/kgdata/badge/?version=latest&style=flat)
 
-<div align="center">
+KGData is a library to process dumps of Wikipedia, Wikidata. What it can do:
 
-![PyPI](https://img.shields.io/pypi/v/kgdata)
-![Documentation](https://readthedocs.org/projects/kgdata/badge/?version=latest&style=flat)
+- Clean up the dumps to ensure the data is consistent (resolve redirect, remove dangling references)
+- Create embedded key-value databases to access entities from the dumps.
+- Extract Wikidata ontology.
+- Extract Wikipedia tables and convert the hyperlinks to Wikidata entities.
+- Create Pyserini indices to search Wikidata’s entities.
+- and more
 
-</div>
-
-KGData is a library to process dumps of Wikipedia, Wikidata and DBPedia. For a complete documentation, please see [the website](https://kgdata.readthedocs.io/).
-
-## Contents
-
-<!--ts-->
-
-- [Usage](#usage)
-  - [Wikidata](#wikidata)
-  - [Wikipedia](#wikipedia)
-- [Installation](#installation)
-<!--te-->
-
-## Usage
-
-### Wikidata
-
-```
-Usage: python -m kgdata.wikidata [OPTIONS] COMMAND [ARGS]...
-
-Options:
-  --help  Show this message and exit.
-
-Commands:
-  classes              Wikidata classes
-  entities             Wikidata entities
-  entity_labels        Wikidata entity labels
-  entity_redirections  Wikidata entity redirections
-  properties           Wikidata properties
-  wp2wd                Mapping from Wikipedia articles to Wikidata entities
-```
-
-You need the following dumps:
-
-1. entity dump ([`latest-all.json.bz2`](https://dumps.wikimedia.org/wikidatawiki/entities/20200518/wikidata-20200518-all.json.bz2)): needed to extract entities, classes and properties.
-2. `wikidatawiki-page.sql.gz` and `wikidatawiki-redirect.sql.gz` ([link](https://dumps.wikimedia.org/wikidatawiki)): needed to extract redirections of old entities.
-
-Then, execute the following steps:
-
-1.  Download the wikidata dumps (e.g., [`latest-all.json.bz2`](https://dumps.wikimedia.org/wikidatawiki/entities/20200518/wikidata-20200518-all.json.bz2)) and put it to `<wikidata_dir>/step_0` folder.
-1.  Extract entities, entity Labels, and entity redirections:
-    - `kgdata wikidata entities -d <wikidata_dir> -o <database_directory> -c`
-    - `kgdata wikidata entity_labels -d <wikidata_dir> -o <database_directory> -c`
-    - `kgdata wikidata entity_redirections -d <wikidata_dir> -o <database_directory> -c`
-1.  Extract ontology:
-    - `kgdata wikidata classes -d <wikidata_dir> -o <database_directory> -c`
-    - `kgdata wikidata properties -d <wikidata_dir> -o <database_directory> -c`
-
-For more commands, see `scripts/build.sh`.
-If compaction step (compact rocksdb) takes lots of time, you can run without `-c` flag.
-If you run directly from source, replacing the `kgdata` command with `python -m kgdata`.
-
-We provide functions to read the databases built from the previous step and return a dictionary-like objects in the module: [`kgdata.wikidata.db`](/kgdata/wikidata/db.py). In the same folder, you can find models of Wikidata [entities](/kgdata/wikidata/models/qnode.py), [classes](/kgdata/wikidata/models/wdclass.py), and [properties](/kgdata/wikidata/models/wdproperty.py).
-
-### Wikipedia
-
-Here is a list of dumps that you need to download depending on the database/files you want to build:
-
-1. [Static HTML Dumps](https://dumps.wikimedia.org/other/enterprise_html/): they only dumps some namespaces. The namespace that you likely to use is 0 (main articles). For example, enwiki-NS0-20220420-ENTERPRISE-HTML.json.tar.gz.
-
-Then, execute the following steps:
-
-1. Extract HTML Dumps:
-   - `kgdata wikipedia -d <wikipedia_dir> enterprise_html_dumps`
+For a full documentation, please see[the website](https://kgdata.readthedocs.io/).
 
 ## Installation
 
-### From pip
-
-You need to have gcc in order to install `cityhash`
+From PyPI (using pre-built binaries):
 
 ```bash
 pip install kgdata
 ```
-
-### From Source
-
-This library uses Apache Spark 3.0.3 (`pyspark` version is `3.0.3`). If you use different Spark version, make sure that version of `pyspark` package is matched (in `pyproject.toml`).
-
-```bash
-poetry install
-mkdir dist; zip -r kgdata.zip kgdata; mv kgdata.zip dist/ # package the application to submit to Spark cluster
-```
-
-You can also consult the [Dockerfile](./Dockerfile) for guidance to install from scratch.
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 120bb72..c91ed6e 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,5 +1,5 @@
 furo>=2022.6.21,<2023.0.0
-sphinx>=5.0.2,<6.0.0
+sphinx>=5.0.2,<5.1.0
 sphinxcontrib.katex
 matplotlib
 sphinx-copybutton
diff --git a/docs/source/api.rst b/docs/source/api.rst
index 4f6abed..ad34635 100644
--- a/docs/source/api.rst
+++ b/docs/source/api.rst
@@ -7,5 +7,12 @@ Python APIs
     
     kgdata.spark
     kgdata.splitter
+    kgdata.wikidata.config
     kgdata.wikidata.db
-    kgdata.wikidata.models
\ No newline at end of file
+    kgdata.wikidata.search
+    kgdata.wikidata.datasets
+    kgdata.wikidata.models
+    kgdata.wikipedia.config
+    kgdata.wikipedia.datasets
+    kgdata.wikipedia.models
+    
\ No newline at end of file
diff --git a/docs/source/changelog.md b/docs/source/changelog.md
new file mode 100644
index 0000000..b938966
--- /dev/null
+++ b/docs/source/changelog.md
@@ -0,0 +1,9 @@
+# CHANGE LOG
+
+## [2.3.3](https://github.com/binh-vu/kgdata/tree/2.3.3) (2022-07-06)
+
+[Full Changelog](https://github.com/binh-vu/kgdata/compare/1.7.1...2.3.3)
+
+**Changed:**
+
+- Upgrade hugedict from version 1 to version 2 using Rust. Existing RocksDB located in: `<db_folder>/primary` should change to just `<db_folder>`
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 4f2424b..262c87e 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -54,16 +54,20 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
-python_version = "3"
+# -- Options for intersphinx -------------------------------------------------
 
+python_version = "3"
 intersphinx_mapping = {
     "python": (
         f"https://docs.python.org/{python_version}",
         None,
     ),
     "numpy": ("https://numpy.org/doc/stable", None),
+    "hugedict": ("https://hugedict.readthedocs.io/en/latest/", None),
 }
 
+# -- Options for autosummary and autodoc -------------------------------------
+
 autosummary_generate = True
 autodoc_typehints = "both"
 autodoc_typehints_format = "short"
@@ -74,6 +78,9 @@
     "undoc-members": True,
     "member-order": "bysource",
 }
+add_module_names = False
+
+# -- Options for external links ----------------------------------------------
 
 extlinks = {"source": ("https://github.com/binh-vu/kgdata/blob/master/%s", "%s")}
 
@@ -96,7 +103,6 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 html_css_files = [
-    # "https://cdn.jsdelivr.net/npm/katex@0.10.0-beta/dist/katex.min.css",
     "css/custom.css",
     "css/ibm-plex.min.css",
 ]
diff --git a/docs/source/development.rst b/docs/source/development.rst
index 12bc385..eacc8c5 100644
--- a/docs/source/development.rst
+++ b/docs/source/development.rst
@@ -12,7 +12,7 @@ Install from Source
 You can also consult the :source:`Dockerfile` for guidance to install from scratch.
 
 Setup Documentation
---------------------
+-------------------
 
 1. Installing dependencies & copying required files
 
diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst
index 25ea594..b31bda9 100644
--- a/docs/source/getting-started.rst
+++ b/docs/source/getting-started.rst
@@ -1,10 +1,12 @@
+.. include:: glossary.rst
+
 Getting Started
 ===============
 
 Installation
 ------------
 
-kgdata is distributed on `PyPI <https://pypi.org/project/kgdata/>`_ so you can install with pip as below.
+|kgdata| is distributed on `PyPI <https://pypi.org/project/kgdata/>`_ so you can install with pip as below.
 
 .. code:: bash
 
@@ -14,7 +16,7 @@ kgdata is distributed on `PyPI <https://pypi.org/project/kgdata/>`_ so you can i
 .. note::
 
     You need to have gcc in order for pip to compile `cityhash`. 
-    Also, this library uses Apache Spark 3.0.3 (``pyspark`` version is ``3.0.3``).
+    Also, this library uses Apache Spark 3.3.0 (``pyspark`` version is ``3.3.0``).
     If you use different Spark version, try to install from source (see :ref:`Install from Source`).
 
 
diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst
new file mode 100644
index 0000000..3be6350
--- /dev/null
+++ b/docs/source/glossary.rst
@@ -0,0 +1,1 @@
+.. |kgdata| replace:: `kgdata <https://kgdata.readthedocs.io/>`__
\ No newline at end of file
diff --git a/docs/source/index.rst b/docs/source/index.rst
index e03e07e..faf0db5 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -1,12 +1,13 @@
+.. include:: glossary.rst
 .. kgdata documentation master file, created by
    sphinx-quickstart on Fri Jul 22 19:47:50 2022.
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-kgdata documentation
+|kgdata| documentation
 ==================================
 
-kgdata is a Python library to process dumps of Wikipedia, Wikidata. What it can do:
+|kgdata| is a Python library to process dumps of Wikipedia, Wikidata. What it can do:
 
 - Clean up the dumps to ensure the data is consistent (resolve redirect, remove dangling references)
 - Create embedded key-value databases to access entities from the dumps.
diff --git a/docs/source/wikidata.rst b/docs/source/wikidata.rst
index 0114b7a..11c55a0 100644
--- a/docs/source/wikidata.rst
+++ b/docs/source/wikidata.rst
@@ -1,7 +1,31 @@
+.. include:: glossary.rst
+
 Wikidata
 ========
 
-::
+.. admonition:: Prerequisite
+
+   Please see :ref:`Wikidata dumps` for the list of dumps that are required to generate all datasets.
+
+.. admonition:: Data organization
+
+   |kgdata| organizes Wikidata's data in a folder ``<wikidata_dir>``, e.g., ``/data/wikidata/20200518``. The dumps are stored in a subfolder called dumps ``<wikidata_dir>/dumps`` (e.g., ``/data/wikidata/20200518/dumps``). Other datasets after processed are stored in sibling folders. The list of folders can be found in :py:mod:`kgdata.wikidata.config`.
+
+Wikidata datasets
+-----------------
+
+List of available datasets can be found in :py:mod:`kgdata.wikidata.datasets`.
+
+.. automodule:: kgdata.wikidata.datasets.__main__
+
+Wikidata databases
+------------------
+
+List of available databases can be found by running ``python -m kgdata.wikidata``.
+
+.. code:: bash
+
+   $ python -m kgdata.wikidata --help
 
    Usage: python -m kgdata.wikidata [OPTIONS] COMMAND [ARGS]...
 
@@ -16,27 +40,18 @@ Wikidata
      properties           Wikidata properties
      wp2wd                Mapping from Wikipedia articles to Wikidata entities
 
-You need the following dumps:
-
-1. entity dump
-   (`latest-all.json.bz2 <https://dumps.wikimedia.org/wikidatawiki/entities/20200518/wikidata-20200518-all.json.bz2>`__):
-   needed to extract entities, classes and properties.
-2. ``wikidatawiki-page.sql.gz`` and ``wikidatawiki-redirect.sql.gz``
-   (`link <https://dumps.wikimedia.org/wikidatawiki>`__): needed to
-   extract redirections of old entities.
-
-Then, execute the following steps:
+We provide functions to read the databases built from the previous step
+and return a dictionary-like objects in the module:
+:py:mod:`kgdata.wikidata.db`. You can find main models of Wikidata in here:
+:py:mod:`kgdata.wikidata.models.wdentity`, :py:mod:`kgdata.wikidata.models.wdclass`, :py:mod:`kgdata.wikidata.models.wdproperty`.
 
-1. Download the wikidata dumps (e.g.,
-   `latest-all.json.bz2 <https://dumps.wikimedia.org/wikidatawiki/entities/20200518/wikidata-20200518-all.json.bz2>`__)
-   and put it to ``<wikidata_dir>/step_0`` folder.
-2. Extract entities, entity Labels, and entity redirections:
+1. Extract entities, entity Labels, and entity redirections:
 
    -  ``kgdata wikidata entities -d <wikidata_dir> -o <database_directory> -c``
    -  ``kgdata wikidata entity_labels -d <wikidata_dir> -o <database_directory> -c``
    -  ``kgdata wikidata entity_redirections -d <wikidata_dir> -o <database_directory> -c``
 
-3. Extract ontology:
+2. Extract ontology:
 
    -  ``kgdata wikidata classes -d <wikidata_dir> -o <database_directory> -c``
    -  ``kgdata wikidata properties -d <wikidata_dir> -o <database_directory> -c``
@@ -49,4 +64,32 @@ directly from source, replacing the ``kgdata`` command with
 We provide functions to read the databases built from the previous step
 and return a dictionary-like objects in the module:
 :py:mod:`kgdata.wikidata.db`. You can find main models of Wikidata in here:
-:py:mod:`kgdata.wikidata.models.wdentity`, :py:mod:`kgdata.wikidata.models.wdclass`, :py:mod:`kgdata.wikidata.models.wdproperty`.
\ No newline at end of file
+:py:mod:`kgdata.wikidata.models.wdentity`, :py:mod:`kgdata.wikidata.models.wdclass`, :py:mod:`kgdata.wikidata.models.wdproperty`.
+
+Wikidata dumps
+--------------
+
+The dumps are available at `dumps.wikimedia.org <https://dumps.wikimedia.org/wikidatawiki/>`__.
+
+We need the following dumps:
+
+1. entity dump
+   (e.g., `latest-all.json.bz2 <https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2>`__):
+   needed to extract entities, classes and properties.
+2. `wikidatawiki-page.sql.gz <https://dumps.wikimedia.org/wikidatawiki/latest/wikidatawiki-latest-page.sql.gz>`__ and `wikidatawiki-redirect.sql.gz <https://dumps.wikimedia.org/wikidatawiki/latest/wikidatawiki-latest-redirect.sql.gz>`__: needed to
+   resolve redirections of old entities.
+
+Below are some useful scripts to download the dumps. First, set the correct parameters:
+
+.. code:: bash
+
+   export VERSION=20200518
+   export DIR=<wikidata_dir>
+
+.. code:: bash
+
+   mkdir -p $DIR/dumps
+   cd $DIR/dumps
+   wget https://dumps.wikimedia.org/wikidatawiki/entities/$VERSION/wikidata-$VERSION-all.json.bz2
+   wget https://dumps.wikimedia.org/wikidatawiki/$VERSION/wikidatawiki-$VERSION-page.sql.gz
+   wget https://dumps.wikimedia.org/wikidatawiki/$VERSION/wikidatawiki-$VERSION-redirect.sql.gz
\ No newline at end of file
diff --git a/docs/source/wikipedia.rst b/docs/source/wikipedia.rst
index 6a2cb7e..c4403a4 100644
--- a/docs/source/wikipedia.rst
+++ b/docs/source/wikipedia.rst
@@ -1,17 +1,41 @@
+.. include:: glossary.rst
+
 Wikipedia
 =========
 
-Here is a list of dumps that you need to download depending on the
-database/files you want to build:
+.. admonition:: Prerequisite
+
+   Please see :ref:`Wikipedia dumps` for the list of dumps that are required to generate all datasets.
+
+.. admonition:: Data organization
+
+   |kgdata| organizes Wikipedia's data in a folder ``<wikipedia_dir>``, e.g., ``/data/wikipedia/20200518``. The dumps are stored in a subfolder called dumps ``<wikipedia_dir>/dumps`` (e.g., ``/data/wikipedia/20200518/dumps``). Other datasets after processed are stored in sibling folders. The list of folders can be found in :py:mod:`kgdata.wikipedia.config`.
+
+
+Wikipedia datasets
+-----------------
+
+List of available datasets can be found in :py:mod:`kgdata.wikipedia.datasets`.
+
+.. automodule:: kgdata.wikipedia.datasets.__main__
+
+
+Wikipedia dumps
+---------------
+
+We need the following dumps:
+
+1. `Static HTML Dumps <https://dumps.wikimedia.org/other/enterprise_html/>`__: they only dumps some namespaces. The namespace that you likely to use is 0 (main articles). For example, ``enwiki-NS0-20220420-ENTERPRISE-HTML.json.tar.gz``.
+
+Below are some useful scripts to download the dumps. First, set the correct parameters:
 
-1. `Static HTML
-   Dumps <https://dumps.wikimedia.org/other/enterprise_html/>`__: they
-   only dumps some namespaces. The namespace that you likely to use is 0
-   (main articles). For example,
-   enwiki-NS0-20220420-ENTERPRISE-HTML.json.tar.gz.
+.. code:: bash
 
-Then, execute the following steps:
+   export VERSION=20200420
+   export DIR=<wikipedia_dir>
 
-1. Extract HTML Dumps:
+.. code:: bash
 
-   -  ``kgdata wikipedia -d <wikipedia_dir> enterprise_html_dumps``
\ No newline at end of file
+   mkdir -p $DIR/dumps
+   cd $DIR/dumps
+   wget https://dumps.wikimedia.org/other/enterprise_html/runs/$VERSION/enwiki-NS0-$VERSION-ENTERPRISE-HTML.json.tar.gz
\ No newline at end of file
diff --git a/kgdata/__main__.py b/kgdata/__main__.py
deleted file mode 100644
index 26f1d83..0000000
--- a/kgdata/__main__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-import click
-from kgdata.wikidata.__main__ import wikidata
-from kgdata.wikipedia.cli import wikipedia
-
-
-@click.group()
-def cli():
-    pass
-
-
-cli.add_command(wikidata)
-cli.add_command(wikipedia)
-
-
-if __name__ == "__main__":
-    cli()
diff --git a/kgdata/dataset.py b/kgdata/dataset.py
index 2bebc25..f1fadb0 100644
--- a/kgdata/dataset.py
+++ b/kgdata/dataset.py
@@ -2,13 +2,16 @@
 from dataclasses import dataclass
 import glob
 from pathlib import Path
-from typing import Callable, Generic, List, Tuple, TypeVar, Union, Optional
+from typing import Callable, Generic, List, Tuple, TypeVar, Union, Optional, cast
 
 from kgdata.spark import get_spark_context
-from sm.misc import deserialize_byte_lines, deserialize_lines, identity_func
+from pyspark import RDD
+from sm.misc import deserialize_lines
+from hugedict.misc import identity, Chain2
 from tqdm import tqdm
 
 V = TypeVar("V")
+V2 = TypeVar("V2")
 
 
 @dataclass
@@ -16,33 +19,47 @@ class Dataset(Generic[V]):
     # pattern to files (e.g., /*.gz)
     file_pattern: Union[str, Path]
     deserialize: Callable[[str], V]
-    filter: Optional[Callable[[str], bool]] = None
+    # this filter function is applied before deserialization
+    prefilter: Optional[Callable[[str], bool]] = None
+    # this filter function is applied after deserialization
+    postfilter: Optional[Callable[[V], bool]] = None
 
     # whether the deserialize function is an identity function
     # only happens when is this a list of string
+    # just to avoid unnecessary function calls
     is_deser_identity: bool = False
 
     @staticmethod
     def string(file_pattern: Union[str, Path]) -> Dataset[str]:
         return Dataset(
-            file_pattern, deserialize=identity_func, filter=None, is_deser_identity=True
+            file_pattern,
+            deserialize=identity,
+            prefilter=None,
+            is_deser_identity=True,
         )
 
     def get_files(self) -> List[str]:
         return glob.glob(str(self.file_pattern))
 
-    def get_rdd(self):
+    def get_rdd(self) -> RDD[V]:
         rdd = get_spark_context().textFile(str(self.file_pattern))
-        if self.filter is not None:
-            rdd = rdd.filter(self.filter)
+        if self.prefilter is not None:
+            rdd = rdd.filter(self.prefilter)
 
         if not self.is_deser_identity:
-            return rdd.map(self.deserialize)
+            rdd = rdd.map(self.deserialize)
+        else:
+            rdd = cast(RDD[V], rdd)
+
+        if self.postfilter is not None:
+            rdd = rdd.filter(self.postfilter)
 
         return rdd
 
     def get_dict(self: Dataset[Tuple[str, str]], rstrip: bool = True):
-        assert self.filter is None, "Does not support filtering for non-rdd usage yet."
+        assert (
+            self.prefilter is None and self.postfilter is None
+        ), "Does not support filtering for non-rdd usage yet."
         output = {}
         if rstrip:
             for file in tqdm(self.get_files(), desc="read dataset"):
@@ -55,3 +72,19 @@ def get_dict(self: Dataset[Tuple[str, str]], rstrip: bool = True):
                     k, v = self.deserialize(line)
                     output[k] = v
         return output
+
+    def does_exist(self) -> bool:
+        return len(self.get_files()) > 0
+
+    def map(self, func: Callable[[V], V2]) -> Dataset[V2]:
+        """Transform record from its origin type to another type.
+
+        Args:
+            func: transformation function
+        """
+        return Dataset(
+            file_pattern=self.file_pattern,
+            deserialize=Chain2(func, self.deserialize),
+            prefilter=self.prefilter,
+            is_deser_identity=False,
+        )
diff --git a/kgdata/dbpedia/instances_extraction.py b/kgdata/dbpedia/instances_extraction.py
index 2e89284..ae4bc71 100644
--- a/kgdata/dbpedia/instances_extraction.py
+++ b/kgdata/dbpedia/instances_extraction.py
@@ -1,3 +1,4 @@
+from kgdata.wikipedia.datasets.grouped_articles import grouped_articles
 import orjson
 import os
 from operator import itemgetter
@@ -8,7 +9,6 @@
 from kgdata.config import DBPEDIA_DIR
 from kgdata.misc.ntriples_parser import ntriple_loads, ignore_comment
 from kgdata.spark import get_spark_context, ensure_unique_records
-from kgdata.wikipedia.deprecated.rdd_datasets import id2groups
 
 Triple = Tuple[str, str, str]
 
@@ -370,7 +370,11 @@ def merged_instances_fixed_wiki_id_en(
     sc = get_spark_context()
     if not os.path.exists(outfile):
         merge_instances_rdd = merge_instances_rdd or merge_instances_en()
-        id2groups_rdd = id2groups_rdd or id2groups()
+        id2groups_rdd = id2groups_rdd or (
+            grouped_articles()
+            .get_rdd()
+            .flatMap(lambda x: [(id, x) for title, id in x["group"]])
+        )
 
         def get_wiki_page_ids(x):
             if isinstance(x["http://dbpedia.org/ontology/wikiPageID"], list):
diff --git a/kgdata/dbpedia/table_extraction.py b/kgdata/dbpedia/table_extraction.py
index ec32c7e..c443388 100644
--- a/kgdata/dbpedia/table_extraction.py
+++ b/kgdata/dbpedia/table_extraction.py
@@ -6,6 +6,7 @@
 from operator import attrgetter
 from pathlib import Path
 from typing import List
+from kgdata.wikipedia.datasets.grouped_articles import grouped_articles
 
 import numpy as np
 from bs4 import Tag, NavigableString
@@ -29,8 +30,7 @@
     left_outer_join,
     does_result_dir_exist,
 )
-from kgdata.wikidata.deprecated.rdd_datasets import wikidata_wikipedia_links
-from kgdata.wikipedia.prelude import get_title_from_url, title2groups
+from kgdata.wikipedia.prelude import get_title_from_url
 from sm.misc.deser import get_open_fn
 
 """This module provides functions to extract tables from DBPedia raw table dumps.
@@ -552,11 +552,17 @@ def p_2_resolve_article_id(x):
 
         tables_rdd = tables_rdd or relational_tables_en()
 
+        title2groups = (
+            grouped_articles()
+            .get_rdd()
+            .flatMap(lambda x: [(title, x) for title, id in x["group"]])
+        )
         # (title, url)
         links_rdd = (
             tables_rdd.flatMap(p_0_get_article_urls).distinct().map(p_1_add_title)
         )
-        links_rdd.join(title2groups()).map(p_2_resolve_article_id).map(
+
+        links_rdd.join(title2groups).map(p_2_resolve_article_id).map(
             orjson.dumps
         ).saveAsTextFile(
             outfile, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
diff --git a/kgdata/pyserini.py b/kgdata/pyserini.py
new file mode 100644
index 0000000..a3f1662
--- /dev/null
+++ b/kgdata/pyserini.py
@@ -0,0 +1,180 @@
+"""Utilities for Pyserini."""
+
+from __future__ import annotations
+from dataclasses import asdict, dataclass
+from enum import Enum
+from functools import partial
+import os
+from pathlib import Path
+from typing import Callable, TypedDict, Union
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist
+from kgdata.wikidata.config import WDDataDirCfg
+from kgdata.wikidata.db import get_wdprop_db
+from pyserini.search.lucene import LuceneSearcher
+import orjson
+
+from pyserini.analysis import Analyzer as PyseriniAnalyzer, get_lucene_analyzer
+from kgdata.wikidata.datasets.properties import properties
+from pyserini.index.lucene import IndexReader
+from sm.misc.deser import deserialize_json, serialize_json
+from sm.misc.funcs import identity_func
+
+
+class AnalyzerType(str, Enum):
+    DefaultEnglishAnalyzer = "default_analyzer"
+    TrigramAnalyzer = "trigram_analyzer"
+
+
+@dataclass(eq=True)
+class IndexSettings:
+    analyzer: AnalyzerType = AnalyzerType.TrigramAnalyzer
+
+    def get_analyzer(self):
+        if self.analyzer == AnalyzerType.DefaultEnglishAnalyzer:
+            return identity_func
+        if self.analyzer == AnalyzerType.TrigramAnalyzer:
+            return TrigramAnalyzer()
+        raise NotImplementedError(f"Analyzer type {self.analyzer} is not implemented")
+
+    def to_dict(self) -> dict:
+        return {
+            "version": 1,
+            "analyzer": self.analyzer.value,
+        }
+
+    @staticmethod
+    def from_dict(o: dict) -> IndexSettings:
+        assert o["version"] == 1
+        return IndexSettings(analyzer=AnalyzerType(o["analyzer"]))
+
+
+class TrigramAnalyzer:
+    def __init__(self):
+        self.pyserini_analyzer = PyseriniAnalyzer(
+            get_lucene_analyzer(
+                language="en", stemming=True, stemmer="krovetz", stopwords=True
+            )
+        )
+
+    def __call__(self, text: str):
+        return " ".join(
+            [
+                ngram
+                for token in self.pyserini_analyzer.analyze(text)
+                for ngram in self.ngram_filter(token, min=3, max=4)
+            ]
+        )
+
+    @staticmethod
+    def ngram_filter(text, min, max):
+        if len(text) <= min:
+            return [text]
+
+        size = len(text)
+        output = []
+        for start in range(0, size - min + 1):
+            for nchar in range(min, max + 1):
+                end = start + nchar
+                if end > size:
+                    break
+                output.append(text[start:end])
+        return output
+
+    def __getstate__(self):
+        return {}
+
+    def __setstate__(self, state):
+        self.__init__()
+
+
+class PyseriniDoc(TypedDict):
+    id: str
+    contents: str
+
+
+def retokenize(doc: PyseriniDoc, analyzer: Callable[[str], str]):
+    doc["contents"] = analyzer(doc["contents"])
+    return doc
+
+
+def build_pyserini_index(
+    dataset: Dataset[PyseriniDoc],
+    data_dir: Union[str, Path],
+    index_dir: Union[str, Path],
+    settings: IndexSettings,
+    n_files: int = 256,
+    optimize: bool = True,
+):
+    """Build a pyserini index from the given dataset.
+
+    Args:
+        dataset: The dataset to build the index from.
+        data_dir: The directory to store the serialized docs.
+        index_dir: The directory to store the index.
+        settings: The settings to use for the index.
+        n_files: The number of files to store the docs (to avoid huge number of files).
+        optimize: Whether to optimize the index.
+    """
+    data_dir = Path(data_dir)
+    index_dir = Path(index_dir)
+
+    if not does_result_dir_exist(data_dir):
+        rdd = (
+            dataset.get_rdd()
+            .map(partial(retokenize, analyzer=settings.get_analyzer()))
+            .map(orjson.dumps)
+        )
+
+        if n_files > 0:
+            rdd = rdd.coalesce(n_files)
+
+        rdd.saveAsTextFile(str(data_dir))
+
+        for file in Path(data_dir).iterdir():
+            if file.name.startswith("part-"):
+                file.rename(file.parent / f"{file.name}.jsonl")
+
+        serialize_json(settings.to_dict(), data_dir / "_SUCCESS", indent=4)
+    else:
+        assert (
+            IndexSettings.from_dict(deserialize_json(data_dir / "_SUCCESS")) == settings
+        )
+
+    if not does_result_dir_exist(index_dir):
+        from jnius import autoclass
+
+        index_dir.mkdir(parents=True, exist_ok=True)
+        extra_args = []
+        if settings.analyzer != AnalyzerType.DefaultEnglishAnalyzer:
+            extra_args.append(f"-pretokenized")
+
+        if optimize:
+            extra_args.append("-optimize")
+
+        JIndexCollection = autoclass("io.anserini.index.IndexCollection")
+        JIndexCollection.main(
+            [
+                "-collection",
+                "JsonCollection",
+                "-input",
+                str(data_dir),
+                "-index",
+                str(index_dir),
+                "-generator",
+                "DefaultLuceneDocumentGenerator",
+                "-storePositions",
+                "-threads",
+                str(os.cpu_count()),
+                "-memorybuffer",
+                "8192",
+            ]
+            + extra_args
+        )
+        (index_dir / "_SUCCESS").touch()
+        serialize_json(settings.to_dict(), index_dir / "_SUCCESS", indent=4)
+    else:
+        assert (
+            IndexSettings.from_dict(deserialize_json(index_dir / "_SUCCESS"))
+            == settings
+        )
diff --git a/kgdata/spark.py b/kgdata/spark.py
index 8ff10b2..15009d1 100644
--- a/kgdata/spark.py
+++ b/kgdata/spark.py
@@ -1,20 +1,15 @@
-from functools import partial
+"""Utility functions for Apache Spark."""
+
 import glob
-import gzip
-from multiprocessing.shared_memory import SharedMemory
-from math import ceil
-from multiprocessing import Pool, Queue, Process
-import zstandard as zstd
-from uuid import uuid4
-from hugedict.misc import identity
-from hugedict.parallel import Parallel
 import orjson
 import os
 import shutil
 from operator import add, itemgetter
 from pathlib import Path
 from typing import Any, TypeVar, Callable, List, Union, Tuple, Optional
-from pyspark import SparkContext, SparkConf
+from pyspark import RDD, SparkContext, SparkConf
+from loguru import logger
+
 
 # SparkContext singleton
 _sc = None
@@ -29,40 +24,61 @@ def get_spark_context():
     """
     global _sc
     if _sc is None:
-        conf = SparkConf().setAll(
-            [
-                (
+
+        def has_key(key):
+            return any(
+                k in os.environ
+                for k in [
+                    key,
+                    key.upper(),
+                    key.replace(".", "_"),
+                    key.upper().replace(".", "_"),
+                ]
+            )
+
+        def get_key(key):
+            lst = [
+                os.environ[k]
+                for k in [
                     key,
-                    os.environ[key]
-                    if key in os.environ
-                    else os.environ[key.replace(".", "_")],
-                )
-                for key in [
-                    "spark.master",
-                    "spark.ui.port",
-                    "spark.executor.memory",
-                    "spark.executor.cores",
-                    "spark.executor.instances",
-                    "spark.driver.memory",
-                    "spark.driver.maxResultSize",
+                    key.upper(),
+                    key.replace(".", "_"),
+                    key.upper().replace(".", "_"),
                 ]
-                if key.replace(".", "_") in os.environ or key in os.environ
+                if k in os.environ
             ]
-        )
+            assert len(lst) > 0, f"{key}: {has_key(key)}"
+            return lst[0]
+
+        opts = [
+            (key, get_key(key))
+            for key in [
+                "spark.master",
+                "spark.ui.port",
+                "spark.executor.memory",
+                "spark.executor.cores",
+                "spark.executor.instances",
+                "spark.driver.memory",
+                "spark.driver.maxResultSize",
+            ]
+            if has_key(key)
+        ]
+        logger.debug("Spark Options: {}", opts)
+        conf = SparkConf().setAll(opts)
         _sc = SparkContext(conf=conf)
 
         # add the current package, run `python setup.py bdist_egg` if does not exist
-        egg_file = [
-            str(file)
-            for file in (
-                Path(os.path.abspath(__file__)).parent.parent / "dist"
-            ).iterdir()
-            if file.name.endswith(".egg") or file.name.endswith(".zip")
-        ]
-        if len(egg_file) > 0:
-            assert len(egg_file) == 1, f"{len(egg_file)} != 1"
-            egg_file = egg_file[0]
-            _sc.addPyFile(egg_file)
+        dist_dir = Path(os.path.abspath(__file__)).parent.parent / "dist"
+        if dist_dir.exists():
+            egg_file = [
+                str(file)
+                for file in (dist_dir).iterdir()
+                if file.name.endswith(".egg") or file.name.endswith(".zip")
+            ]
+            if len(egg_file) > 0:
+                assert len(egg_file) == 1, f"{len(egg_file)} != 1"
+                egg_file = egg_file[0]
+                _sc.addPyFile(egg_file)
 
     return _sc
 
@@ -136,8 +152,8 @@ def ensure_unique_records(rdd, keyfn, print_error: bool = True):
 
 
 def left_outer_join(
-    rdd1,
-    rdd2,
+    rdd1: RDD[R1],
+    rdd2: RDD[R2],
     rdd1_keyfn: Callable[[R1], K1],
     rdd1_fk_fn: Callable[[R1], List[K2]],
     rdd2_keyfn: Callable[[R2], K2],
diff --git a/kgdata/splitter.py b/kgdata/splitter.py
index 810d7bc..6ac51ac 100644
--- a/kgdata/splitter.py
+++ b/kgdata/splitter.py
@@ -28,7 +28,7 @@ def default_currentbyte_constructor(
 ) -> Callable[[], int]:
     """Get a function that returns the current byte position that the file reader is currently at."""
     if isinstance(file_object, BZ2File):
-        return file_object.buffer._buffer.raw._fp.tell  # type: ignore
+        return file_object._buffer.raw._fp.tell  # type: ignore
 
     if isinstance(file_object, GzipFile):
         return file_object.fileobj.tell  # type: ignore
@@ -106,6 +106,7 @@ def split_a_file(
         file_size = 1
 
     data_size_file_size = datasize(file_size)
+    success = True
     try:
         with file_object as f, tqdm(
             total=file_size,
@@ -114,7 +115,11 @@ def split_a_file(
             unit_scale=True,
         ) as pbar:
             last_bytes = 0
-            tell = currentbyte_constructor(f)
+            try:
+                tell = currentbyte_constructor(f)
+            except:
+                success = False
+                raise
             for i, line in enumerate(record_iter(f)):
                 queues[i % n_writers].put(line)
                 current_bytes = tell()
@@ -125,7 +130,6 @@ def split_a_file(
         for q in queues:
             q.put(None)
 
-        success = True
         for p in writers:
             p.join()
             success = success and (p.exitcode == 0)
diff --git a/kgdata/wikidata/__main__.py b/kgdata/wikidata/__main__.py
index 0466382..e852d52 100644
--- a/kgdata/wikidata/__main__.py
+++ b/kgdata/wikidata/__main__.py
@@ -1,10 +1,18 @@
 import glob
+import gc
 from pathlib import Path
-from typing import List
+import shutil
+from typing import List, cast
 import click, os
 from click.types import Choice
-from hugedict.loader import FileFormat, load
-from hugedict.misc import Chain2, zstd6_compress, Chain3
+from hugedict.prelude import (
+    RocksDBDict,
+    RocksDBOptions,
+    rocksdb_load,
+    init_env_logger,
+    rocksdb_build_sst_file,
+    rocksdb_ingest_sst_files,
+)
 from kgdata.wikidata.config import WDDataDirCfg
 from kgdata.wikidata.datasets.entity_redirections import entity_redirections
 from kgdata.wikidata.datasets.properties import properties
@@ -19,6 +27,7 @@
     get_entity_label_db,
     get_entity_redirection_db,
     get_wdprop_domain_db,
+    get_wdprop_range_db,
     get_wp2wd_db,
     get_wdclass_db,
     get_wdprop_db,
@@ -28,6 +37,9 @@
 from operator import itemgetter
 
 from sm.misc.funcs import identity_func
+from sm.misc.deser import deserialize_jl
+from sm.misc.timer import Timer
+import ray
 
 
 @click.command(name="entities")
@@ -41,25 +53,26 @@
 )
 @click.option("-l", "--lang", default="en", help="Default language of the Wikidata")
 def db_entities(directory: str, output: str, compact: bool, lang: str):
-    """Wikidata entities"""
+    """Build a key-value database of Wikidata entities"""
     WDDataDirCfg.init(directory)
 
     dbpath = Path(output) / "wdentities.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_entity_db(dbpath).db,
-        infiles=entities(lang=lang).get_files(),
-        format=FileFormat.jsonline,
-        key_fn=Chain2(str.encode, itemgetter("id")).exec,
-        value_fn=Chain2(zstd6_compress, orjson.dumps).exec,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
+    options = cast(RocksDBDict, get_entity_db(dbpath)).options
+    gc.collect()
+
+    rocksdb_load(
+        dbpath=str(dbpath),
+        dbopts=options,
+        files=entities(lang=lang).get_files(),
+        format={
+            "record_type": {"type": "ndjson", "key": "id", "value": None},
+            "is_sorted": False,
+        },
+        verbose=True,
+        compact=True,
     )
-    if compact:
-        logger.info("Run compaction...")
-        db.compact_range()
 
 
 @click.command(name="entity_labels")
@@ -79,21 +92,53 @@ def db_entity_labels(directory: str, output: str, compact: bool, lang: str):
     dbpath = Path(output) / "wdentity_labels.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_entity_label_db(dbpath, create_if_missing=True, read_only=False).db,
-        infiles=entities(lang=lang).get_files(),
-        format=FileFormat.jsonline,
-        key_fn=Chain2(str.encode, itemgetter("id")).exec,
-        value_fn=Chain3(
-            orjson.dumps, WDEntityLabel.to_dict, WDEntityLabel.from_wdentity_raw
-        ).exec,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
-    )
-    if compact:
-        logger.info("Run compaction...")
-        db.compact_range()
+    temp_dir = dbpath / "_temporary"
+    if temp_dir.exists():
+        shutil.rmtree(temp_dir)
+    temp_dir.mkdir()
+
+    options = cast(
+        RocksDBDict,
+        get_entity_label_db(dbpath, create_if_missing=True, read_only=False),
+    ).options
+    gc.collect()
+
+    ray.init()
+
+    @ray.remote
+    def _build_sst_file(infile: str, temp_dir: str, options: RocksDBOptions):
+        kvs = sorted(
+            [
+                (
+                    obj["id"].encode(),
+                    orjson.dumps(WDEntityLabel.from_wdentity_raw(obj).to_dict()),
+                )
+                for obj in deserialize_jl(infile)
+            ],
+            key=itemgetter(0),
+        )
+        tmp = {"counter": 0}
+
+        def input_gen():
+            if tmp["counter"] == len(kvs):
+                return None
+            obj = kvs[tmp["counter"]]
+            tmp["counter"] += 1
+            return obj
+
+        outfile = os.path.join(temp_dir, Path(infile).stem + ".sst")
+        rocksdb_build_sst_file(options, outfile, input_gen)
+        return outfile
+
+    ray_opts = ray.put(options)
+    with Timer().watch_and_report("Creating SST files"):
+        refs = []
+        for file in entities(lang=lang).get_files():
+            refs.append(_build_sst_file.remote(file, str(temp_dir), ray_opts))
+        sst_files = ray.get(refs)
+
+    with Timer().watch_and_report("Ingesting SST files"):
+        rocksdb_ingest_sst_files(str(dbpath), options, sst_files, True)
 
 
 @click.command(name="entity_redirections")
@@ -112,21 +157,23 @@ def db_entity_redirections(directory: str, output: str, compact: bool):
     dbpath = Path(output) / "wdentity_redirections.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_entity_redirection_db(
-            dbpath, create_if_missing=True, read_only=False
-        ).db,
-        infiles=entity_redirections().get_files(),
-        format=FileFormat.tabsep,
-        key_fn=identity_func,
-        value_fn=identity_func,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
+    options = cast(
+        RocksDBDict,
+        get_entity_redirection_db(dbpath, create_if_missing=True, read_only=False),
+    ).options
+    gc.collect()
+
+    rocksdb_load(
+        dbpath=str(dbpath),
+        dbopts=options,
+        files=entity_redirections().get_files(),
+        format={
+            "record_type": {"type": "tabsep", "key": None, "value": None},
+            "is_sorted": False,
+        },
+        verbose=True,
+        compact=True,
     )
-    if compact:
-        logger.info("Run compaction...")
-        db.compact_range()
 
 
 @click.command(name="classes")
@@ -146,19 +193,23 @@ def db_classes(directory: str, output: str, compact: bool, lang: str):
     dbpath = Path(output) / "wdclasses.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_wdclass_db(dbpath).db,
-        infiles=classes(lang=lang).get_files(),
-        format=FileFormat.jsonline,
-        key_fn=Chain2(str.encode, itemgetter("id")).exec,
-        value_fn=orjson.dumps,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
+    options = cast(
+        RocksDBDict,
+        get_wdclass_db(dbpath),
+    ).options
+    gc.collect()
+
+    rocksdb_load(
+        dbpath=str(dbpath),
+        dbopts=options,
+        files=classes(lang=lang).get_files(),
+        format={
+            "record_type": {"type": "ndjson", "key": "id", "value": None},
+            "is_sorted": False,
+        },
+        verbose=True,
+        compact=True,
     )
-    if compact:
-        logger.info("Run compaction...")
-        db.compact_range()
 
 
 @click.command(name="properties")
@@ -188,51 +239,57 @@ def db_properties(
     dbpath = Path(output) / "wdprops.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_wdprop_db(dbpath).db,
-        infiles=properties(lang=lang).get_files(),
-        format=FileFormat.jsonline,
-        key_fn=Chain2(str.encode, itemgetter("id")).exec,
-        value_fn=orjson.dumps,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
+    options = cast(
+        RocksDBDict,
+        get_wdprop_db(dbpath),
+    ).options
+    gc.collect()
+
+    rocksdb_load(
+        dbpath=str(dbpath),
+        dbopts=options,
+        files=properties(lang=lang).get_files(),
+        format={
+            "record_type": {"type": "ndjson", "key": "id", "value": None},
+            "is_sorted": False,
+        },
+        verbose=True,
+        compact=True,
     )
-    if compact:
-        logger.info("Compacting wdprops.db...")
-        db.compact_range()
 
     for name in extra:
         if name == "domains":
             dbpath = Path(output) / "wdprop_domains.db"
             dbpath.mkdir(exist_ok=True, parents=True)
-            infiles = property_domains(lang=lang).get_files()
-            db = get_wdprop_domain_db(
-                dbpath, create_if_missing=True, read_only=False
-            ).db
+            files = property_domains(lang=lang).get_files()
+            options = cast(
+                RocksDBDict,
+                get_wdprop_domain_db(dbpath, create_if_missing=True, read_only=False),
+            ).options
+            gc.collect()
         elif name == "ranges":
             dbpath = Path(output) / "wdprop_ranges.db"
             dbpath.mkdir(exist_ok=True, parents=True)
-            infiles = property_ranges(lang=lang).get_files()
-            db = get_wdprop_domain_db(
-                dbpath, create_if_missing=True, read_only=False
-            ).db
+            files = property_ranges(lang=lang).get_files()
+            options = cast(
+                RocksDBDict,
+                get_wdprop_range_db(dbpath, create_if_missing=True, read_only=False),
+            ).options
+            gc.collect()
         else:
             raise NotImplementedError(name)
 
-        db = load(
-            db=db,
-            infiles=infiles,
-            format=FileFormat.tuple2,
-            key_fn=str.encode,
-            value_fn=orjson.dumps,
-            n_processes=8,
-            shm_mem_ratio=12,
-            shm_mem_limit_mb=128,
+        rocksdb_load(
+            dbpath=str(dbpath),
+            dbopts=options,
+            files=files,
+            format={
+                "record_type": {"type": "tuple2", "key": None, "value": None},
+                "is_sorted": False,
+            },
+            verbose=True,
+            compact=True,
         )
-        if compact:
-            logger.info("Compacting {} db...", name)
-            db.compact_range()
 
 
 @click.command(name="wp2wd")
@@ -252,19 +309,56 @@ def db_wp2wd(directory: str, output: str, compact: bool, lang: str):
     dbpath = Path(output) / "wp2wd.db"
     dbpath.mkdir(exist_ok=True, parents=True)
 
-    db = load(
-        db=get_wp2wd_db(dbpath, create_if_missing=True, read_only=False).db,
-        infiles=wp2wd(lang=lang).get_files(),
-        format=FileFormat.tuple2,
-        key_fn=str.encode,
-        value_fn=str.encode,
-        n_processes=8,
-        shm_mem_ratio=12,
-        shm_mem_limit_mb=128,
+    options = cast(
+        RocksDBDict,
+        get_wp2wd_db(dbpath, create_if_missing=True, read_only=False),
+    ).options
+    gc.collect()
+
+    rocksdb_load(
+        dbpath=str(dbpath),
+        dbopts=options,
+        files=wp2wd(lang=lang).get_files(),
+        format={
+            "record_type": {"type": "tuple2", "key": None, "value": None},
+            "is_sorted": False,
+        },
+        verbose=True,
+        compact=True,
     )
-    if compact:
-        logger.info("Run compaction...")
-        db.compact_range()
+
+
+@click.command(name="search.entities")
+@click.option("-d", "--directory", default="", help="Wikidata directory")
+@click.option("-o", "--output", help="Output directory")
+@click.option("-l", "--lang", default="en", help="Default language of the Wikidata")
+def search_entities(directory: str, output: str, lang: str):
+    from kgdata.wikidata.search import build_index
+
+    WDDataDirCfg.init(directory)
+    build_index(name="entities", index_parent_dir=Path(output), lang=lang)
+
+
+@click.command(name="search.properties")
+@click.option("-d", "--directory", default="", help="Wikidata directory")
+@click.option("-o", "--output", help="Output directory")
+@click.option("-l", "--lang", default="en", help="Default language of the Wikidata")
+def search_props(directory: str, output: str, lang: str):
+    from kgdata.wikidata.search import build_index
+
+    WDDataDirCfg.init(directory)
+    build_index(name="props", index_parent_dir=Path(output), lang=lang)
+
+
+@click.command(name="search.classes")
+@click.option("-d", "--directory", default="", help="Wikidata directory")
+@click.option("-o", "--output", help="Output directory")
+@click.option("-l", "--lang", default="en", help="Default language of the Wikidata")
+def search_classes(directory: str, output: str, lang: str):
+    from kgdata.wikidata.search import build_index
+
+    WDDataDirCfg.init(directory)
+    build_index(name="classes", index_parent_dir=Path(output), lang=lang)
 
 
 @click.group()
@@ -278,7 +372,11 @@ def wikidata():
 wikidata.add_command(db_classes)
 wikidata.add_command(db_properties)
 wikidata.add_command(db_wp2wd)
+wikidata.add_command(search_entities)
+wikidata.add_command(search_props)
+wikidata.add_command(search_classes)
 
 
 if __name__ == "__main__":
+    init_env_logger()
     wikidata()
diff --git a/kgdata/wikidata/config.py b/kgdata/wikidata/config.py
index 4f791fa..d260380 100644
--- a/kgdata/wikidata/config.py
+++ b/kgdata/wikidata/config.py
@@ -1,3 +1,5 @@
+"""Locations of Wikidata dumps and datasets on disk."""
+
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Union
@@ -21,15 +23,19 @@ def __init__(self, datadir: Path) -> None:
 
         self.entity_ids = datadir / "entity_ids"
         self.entity_types = datadir / "entity_types"
+        self.entity_metadata = datadir / "entity_metadata"
         self.page_ids = datadir / "page_ids"
         self.entities = datadir / "entities"
 
         self.classes = datadir / "classes"
+        self.class_count = datadir / "class_count"
         self.properties = datadir / "properties"
         self.property_domains = datadir / "property_domains"
         self.property_ranges = datadir / "property_ranges"
         self.wp2wd = datadir / "wp2wd"
 
+        self.search = datadir / "search"
+
         self.entity_redirections = datadir / "entity_redirections"
 
     def get_entity_dump_file(self):
diff --git a/kgdata/wikidata/datasets/__main__.py b/kgdata/wikidata/datasets/__main__.py
index 4b24ff5..c97392c 100644
--- a/kgdata/wikidata/datasets/__main__.py
+++ b/kgdata/wikidata/datasets/__main__.py
@@ -1,3 +1,29 @@
+"""
+Run the command ``python -m kgdata.wikidata.datasets`` to generate the datasets.
+
+.. code:: bash
+
+   $ python -m kgdata.wikidata.datasets --help
+   
+   Usage: python -m kgdata.wikidata.datasets [OPTIONS]
+
+   Options:
+   -s, --source TEXT   Wikidata directory  [required]
+   -d, --dataset TEXT  Dataset name  [required]
+   --help              Show this message and exit. 
+
+Examples::
+
+    python -m kgdata.wikidata.datasets -s $WD_DIR -d entities
+    python -m kgdata.wikidata.datasets -s $WD_DIR -d classes
+    python -m kgdata.wikidata.datasets -s $WD_DIR -d properties
+
+.. note::
+
+    For the commands in the above examples to run correctly, replaced ``$WD_DIR`` with the path to the Wikidata directory, e.g., ``export WD_DIR=/data/wikidata/20211213``
+
+"""
+
 from importlib import import_module
 from typing_extensions import Required
 import click
diff --git a/kgdata/wikidata/datasets/class_count.py b/kgdata/wikidata/datasets/class_count.py
new file mode 100644
index 0000000..1c4f85b
--- /dev/null
+++ b/kgdata/wikidata/datasets/class_count.py
@@ -0,0 +1,54 @@
+from typing import Tuple
+from operator import add
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist, saveAsSingleTextFile
+from kgdata.wikidata.config import WDDataDirCfg
+from kgdata.wikidata.datasets.classes import classes
+from kgdata.wikidata.datasets.entities import entities
+from kgdata.wikidata.datasets.entity_types import entity_types, get_instanceof
+import orjson
+
+
+def class_count(lang="en") -> Dataset[Tuple[str, int]]:
+    cfg = WDDataDirCfg.get_instance()
+
+    if not does_result_dir_exist(cfg.class_count):
+        ds = entity_types(lang)
+        if ds.does_exist():
+            rdd = ds.get_rdd()
+        else:
+            rdd = entities(lang).get_rdd().map(get_instanceof)
+
+        class_count = rdd.flatMap(lambda x: [(c, 1) for c in x[1]]).reduceByKey(add)
+        (
+            classes(lang)
+            .get_rdd()
+            .map(lambda x: (x.id, 0))
+            .leftOuterJoin(class_count)
+            .map(lambda x: (x[0], x[1][1] if x[1][1] is not None else 0))
+            .map(orjson.dumps)
+            .saveAsTextFile(
+                str(cfg.class_count),
+                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
+            )
+        )
+
+    # return Dataset(cfg.class_count / "*.gz", deserialize=orjson.loads)
+    ds = Dataset(cfg.class_count / "*.gz", deserialize=orjson.loads)
+    rdd = (
+        classes(lang)
+        .get_rdd()
+        .map(lambda x: (x.id, x.label))
+        .join(ds.get_rdd())
+        .map(lambda x: [x[0], x[1][0], x[1][1]])
+        .sortBy(lambda x: x[2], ascending=True)
+        .map(lambda x: "\t".join([str(y) for y in x]))
+    )
+
+    saveAsSingleTextFile(
+        rdd, str(cfg.class_count / "../class_count_sorted.tsv"), shuffle=False
+    )
+    # print(ds.get_rdd().sortBy(lambda x: x[1], ascending=False).take(100))
+    # print(ds.get_rdd().sortBy(lambda x: x[1], ascending=True).take(1000)[900:])
+
+    return ds
diff --git a/kgdata/wikidata/datasets/entity_metadata.py b/kgdata/wikidata/datasets/entity_metadata.py
new file mode 100644
index 0000000..89a7d59
--- /dev/null
+++ b/kgdata/wikidata/datasets/entity_metadata.py
@@ -0,0 +1,45 @@
+import os
+from functools import partial
+from pathlib import Path
+from typing import Union, cast, Set, Dict
+
+from kgdata.config import WIKIDATA_DIR
+from kgdata.spark import does_result_dir_exist, get_spark_context, saveAsSingleTextFile
+from kgdata.wikidata.config import WDDataDirCfg
+from kgdata.wikidata.datasets.entities import deser_entity, entities, ser_entity
+from kgdata.wikidata.datasets.entity_dump import entity_dump
+from kgdata.wikidata.datasets.entity_ids import entity_ids
+from kgdata.wikidata.datasets.entity_redirections import entity_redirections
+from kgdata.wikidata.models.wdentity import WDEntity, WDValue
+from loguru import logger
+import orjson
+import sm.misc as M
+from pyspark.rdd import RDD
+from pyspark import Broadcast
+from kgdata.dataset import Dataset
+
+
+def entity_metadata(lang: str = "en") -> Dataset[WDEntity]:
+    """Keep all data of the entities but its properties (set it to empty dictionary)."""
+
+    cfg = WDDataDirCfg.get_instance()
+    outdir = cfg.entity_metadata.parent / (cfg.entity_metadata.name + "_" + lang)
+
+    if not does_result_dir_exist(outdir):
+        (
+            entities()
+            .get_rdd()
+            .map(empty_ent_props)
+            .map(ser_entity)
+            .saveAsTextFile(
+                str(outdir),
+                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
+            )
+        )
+
+    return Dataset(file_pattern=outdir / "*.gz", deserialize=deser_entity)
+
+
+def empty_ent_props(ent: WDEntity) -> WDEntity:
+    ent.props = {}
+    return ent
diff --git a/kgdata/wikidata/datasets/properties.py b/kgdata/wikidata/datasets/properties.py
index aeb0e80..99135c0 100644
--- a/kgdata/wikidata/datasets/properties.py
+++ b/kgdata/wikidata/datasets/properties.py
@@ -22,6 +22,7 @@ def properties(lang="en") -> Dataset[WDProperty]:
             .get_rdd()
             .flatMap(get_property_ids)
             .distinct()
+            .coalesce(128, shuffle=True)
             .saveAsTextFile(
                 str(cfg.properties / "ids"),
                 compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
@@ -39,6 +40,7 @@ def properties(lang="en") -> Dataset[WDProperty]:
             .filter(lambda ent: ent.id in prop_ids.value)
             .map(lambda x: WDProperty.from_entity(x).to_dict())
             .map(orjson.dumps)
+            .coalesce(128, shuffle=True)
             .saveAsTextFile(
                 str(cfg.properties / "properties"),
                 compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
@@ -101,6 +103,7 @@ def merge_ancestors(o):
             .map(merge_ancestors)
             .map(WDProperty.to_dict)
             .map(orjson.dumps)
+            .coalesce(128, shuffle=True)
             .saveAsTextFile(
                 str(cfg.properties / "full_properties"),
                 compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
@@ -130,4 +133,3 @@ def get_property_ids(ent: WDEntity) -> List[str]:
             prop_ids.update(stmt.qualifiers.keys())
 
     return list(prop_ids)
-
diff --git a/kgdata/wikidata/datasets/wp2wd.py b/kgdata/wikidata/datasets/wp2wd.py
index a439ec6..852572e 100644
--- a/kgdata/wikidata/datasets/wp2wd.py
+++ b/kgdata/wikidata/datasets/wp2wd.py
@@ -12,7 +12,7 @@
 
 
 def wp2wd(lang="en") -> Dataset[Tuple[str, str]]:
-    """Get alignments between wikidata qnode and wiki articles."""
+    """Get alignments between wiki article titles and wikidata qnode."""
     cfg = WDDataDirCfg.get_instance()
     site = lang + "wiki"
 
diff --git a/kgdata/wikidata/db.py b/kgdata/wikidata/db.py
index ac2bfef..9661ed0 100644
--- a/kgdata/wikidata/db.py
+++ b/kgdata/wikidata/db.py
@@ -1,6 +1,8 @@
+"""Wikidata embedded key-value databases."""
+
 from functools import partial
 from pathlib import Path
-from typing import *
+from typing import Dict, Union, TypeVar, Optional, Callable, Set, List, Type, cast
 
 from hugedict.types import HugeMutableMapping
 from kgdata.wikidata.models.wdentity import WDEntity
@@ -9,7 +11,7 @@
 from hugedict.misc import zstd6_compress_custom, zstd_decompress_custom, identity
 import requests
 
-from hugedict.prelude import RocksDBDict, RocksDBOptions
+from hugedict.prelude import RocksDBDict, RocksDBOptions, RocksDBCompressionOptions
 from kgdata.wikidata.models import (
     WDClass,
     WDProperty,
@@ -21,40 +23,14 @@
 
 
 class WDProxyDB(RocksDBDict, HugeMutableMapping[str, V]):
-    def __init__(
-        self,
-        EntClass,
-        dbpath: Union[Path, str],
-        dboptions: Optional[RocksDBOptions] = None,
-        compression: bool = False,
-        create_if_missing=True,
-        readonly=False,
-    ):
-        super().__init__(
-            path=str(dbpath),
-            options=RocksDBOptions(create_if_missing=create_if_missing)
-            if dboptions is None
-            else dboptions,
-            readonly=readonly,
-            deser_key=partial(str, encoding="utf-8"),
-            deser_value=zstd_decompress_custom(partial(deserialize, EntClass))
-            if compression
-            else partial(deserialize, EntClass),
-            ser_value=zstd6_compress_custom(serialize) if compression else serialize,
-        )
-
-        if not hasattr(EntClass, "from_entity"):
-            self.extract_ent_from_entity: Callable[[WDEntity], V] = identity
-        else:
-            self.extract_ent_from_entity: Callable[[WDEntity], V] = getattr(
-                EntClass, "from_entity"
-            )
+    def set_extract_ent_from_entity(self, func: Callable[[WDEntity], V]):
+        self.extract_ent_from_entity = func
+        return self
 
     def __getitem__(self, key: str):
-        item = self.get(key)
-        if item == b"\x00":
-            raise KeyError(key)
-        elif item is None:
+        try:
+            item = self._get(key.encode())
+        except KeyError:
             qnodes = query_wikidata_entities([key])
             if len(qnodes) == 0:
                 # no entity
@@ -63,9 +39,11 @@ def __getitem__(self, key: str):
             else:
                 ent = self.extract_ent_from_entity(qnodes[key])
                 self._put(key.encode(), self.ser_value(ent))
-
             return ent
-        return self.deser_value(item)
+
+        if item == b"\x00":
+            raise KeyError(key)
+        return self.deser_value(item)  # type: ignore
 
     def get(self, key: str, default: Optional[V] = None):
         try:
@@ -73,11 +51,10 @@ def get(self, key: str, default: Optional[V] = None):
         except KeyError:
             return default
 
-    def __contains__(self, key):
-        item = self.get(key)
-        if item == b"\x00":
-            return False
-        if item is None:
+    def __contains__(self, key: str):
+        try:
+            item = self._get(key.encode())
+        except KeyError:
             qnodes = query_wikidata_entities([key])
             if len(qnodes) == 0:
                 # no entity
@@ -86,10 +63,18 @@ def __contains__(self, key):
 
             ent = self.extract_ent_from_entity(qnodes[key])
             self._put(key.encode(), self.ser_value(ent))
+            return True
+
+        if item == b"\x00":
+            return False
+
         return True
 
-    def does_not_exist_locally(self, key):
-        item = self.get(key)
+    def does_not_exist_locally(self, key: str):
+        try:
+            item = self._get(key.encode())
+        except KeyError:
+            return True
         return item == b"\x00"
 
 
@@ -148,28 +133,35 @@ def get_wdclass_db(
     read_only=False,
     proxy: bool = False,
 ) -> HugeMutableMapping[str, WDClass]:
-    dboptions = RocksDBOptions(
-        create_if_missing=create_if_missing,
-        compression_type="lz4",
-        bottommost_compression_type="zstd",
-    )
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     if proxy:
-        db = WDProxyDB(
-            WDClass,
-            dbpath=dbfile,
-            dboptions=dboptions,
-            compression=False,
-            readonly=read_only,
-        )
+        constructor = WDProxyDB
+        create_if_missing = True
     else:
-        db = RocksDBDict(
-            path=str(dbfile),
-            options=dboptions,
-            readonly=read_only,
-            deser_key=partial(str, encoding="utf-8"),
-            deser_value=partial(deserialize, WDClass),
-            ser_value=serialize,
-        )
+        constructor = RocksDBDict
+
+    db = constructor(
+        path=str(dbfile),
+        options=RocksDBOptions(
+            create_if_missing=create_if_missing,
+            compression_type="lz4",
+            bottommost_compression_type="zstd",
+        ),
+        readonly=read_only,
+        deser_key=partial(str, encoding="utf-8"),
+        deser_value=partial(deserialize, WDClass),
+        ser_value=serialize,
+    )
+
+    if proxy:
+        return cast(WDProxyDB, db).set_extract_ent_from_entity(WDClass.from_entity)
     return db
 
 
@@ -179,27 +171,34 @@ def get_wdprop_db(
     read_only=False,
     proxy: bool = False,
 ) -> HugeMutableMapping[str, WDProperty]:
-    dboptions = RocksDBOptions(
-        create_if_missing=create_if_missing,
-        compression_type="lz4",
-    )
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     if proxy:
-        db = WDProxyDB(
-            WDProperty,
-            dbfile,
-            dboptions=dboptions,
-            compression=False,
-            readonly=read_only,
-        )
+        constructor = WDProxyDB
+        create_if_missing = True
     else:
-        db = RocksDBDict(
-            path=str(dbfile),
-            options=dboptions,
-            readonly=read_only,
-            deser_key=partial(str, encoding="utf-8"),
-            deser_value=partial(deserialize, WDProperty),
-            ser_value=serialize,
-        )
+        constructor = RocksDBDict
+
+    db = constructor(
+        path=str(dbfile),
+        options=RocksDBOptions(
+            create_if_missing=create_if_missing,
+            compression_type="lz4",
+        ),
+        readonly=read_only,
+        deser_key=partial(str, encoding="utf-8"),
+        deser_value=partial(deserialize, WDProperty),
+        ser_value=serialize,
+    )
+
+    if proxy:
+        return cast(WDProxyDB, db).set_extract_ent_from_entity(WDProperty.from_entity)
     return db
 
 
@@ -209,28 +208,39 @@ def get_entity_db(
     read_only=False,
     proxy: bool = False,
 ) -> HugeMutableMapping[str, WDEntity]:
-    # no compression as we pre-compress the qnodes -- get much better compression
-    dboptions = RocksDBOptions(
-        create_if_missing=create_if_missing,
-        compression_type="none",
-    )
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "2", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("2")
+
     if proxy:
-        db: RocksDBDict[str, WDEntity] = WDProxyDB(
-            WDEntity,
-            dbfile,
-            compression=True,
-            readonly=read_only,
-            dboptions=dboptions,
-        )
+        constructor = WDProxyDB
+        create_if_missing = True
     else:
-        db = RocksDBDict(
-            path=str(dbfile),
-            options=dboptions,
-            readonly=read_only,
-            deser_key=partial(str, encoding="utf-8"),
-            deser_value=zstd_decompress_custom(partial(deserialize, WDEntity)),
-            ser_value=zstd6_compress_custom(serialize),
-        )
+        constructor = RocksDBDict
+
+    db = constructor(
+        path=str(dbfile),
+        options=RocksDBOptions(
+            create_if_missing=create_if_missing,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=-14, level=6, strategy=0, max_dict_bytes=16 * 1024
+            ),
+            zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+        readonly=read_only,
+        deser_key=partial(str, encoding="utf-8"),
+        deser_value=partial(deserialize, WDEntity),
+        ser_value=serialize,
+    )
+
+    if proxy:
+        return cast(WDProxyDB, db).set_extract_ent_from_entity(identity)
+
     return db
 
 
@@ -239,6 +249,14 @@ def get_entity_redirection_db(
     create_if_missing=False,
     read_only=True,
 ) -> HugeMutableMapping[str, str]:
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     dboptions = RocksDBOptions(
         create_if_missing=create_if_missing,
         compression_type="lz4",
@@ -258,6 +276,14 @@ def get_entity_label_db(
     create_if_missing=False,
     read_only=True,
 ) -> HugeMutableMapping[str, WDEntityLabel]:
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     dboptions = RocksDBOptions(
         create_if_missing=create_if_missing,
         compression_type="lz4",
@@ -278,6 +304,14 @@ def get_wp2wd_db(
     read_only=True,
 ) -> HugeMutableMapping[str, str]:
     """Mapping from wikipedia article to wikidata id"""
+    version = Path(dbpath) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     dboptions = RocksDBOptions(
         create_if_missing=create_if_missing,
         compression_type="lz4",
@@ -297,6 +331,14 @@ def get_wdprop_range_db(
     create_if_missing=False,
     read_only=True,
 ) -> HugeMutableMapping[str, WDPropertyRanges]:
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     dboptions = RocksDBOptions(
         create_if_missing=create_if_missing,
         compression_type="lz4",
@@ -316,6 +358,14 @@ def get_wdprop_domain_db(
     create_if_missing=False,
     read_only=True,
 ) -> HugeMutableMapping[str, WDPropertyDomains]:
+    version = Path(dbfile) / "_VERSION"
+    if version.exists():
+        ver = version.read_text()
+        assert ver.strip() == "1", ver
+    else:
+        version.parent.mkdir(parents=True, exist_ok=True)
+        version.write_text("1")
+
     dboptions = RocksDBOptions(
         create_if_missing=create_if_missing,
         compression_type="lz4",
diff --git a/kgdata/wikidata/deprecated/cli.py b/kgdata/wikidata/deprecated/cli.py
deleted file mode 100644
index 8627be6..0000000
--- a/kgdata/wikidata/deprecated/cli.py
+++ /dev/null
@@ -1,202 +0,0 @@
-import os, glob
-from hugedict.misc import Chain2, identity, zstd6_compress
-import orjson
-import ujson
-from enum import Enum
-from pathlib import Path
-from typing import Literal
-
-import click
-from loguru import logger
-from operator import itemgetter
-from hugedict.loader import load, load_single_file, FileFormat
-
-
-@click.command()
-@click.option("-b", "--build", help="Build database")
-@click.option("-d", "--directory", default="", help="Wikidata directory")
-@click.option("-o", "--output", help="Output directory")
-@click.option(
-    "-c",
-    "--compact",
-    is_flag=True,
-    help="Whether to compact the results. May take a very very long time",
-)
-def wikidata(
-    build: Literal[
-        "qnodes",
-        "wdclasses",
-        "wdprops",
-        "enwiki_links",
-        "qnode_labels",
-        "qnode_identifiers",
-    ],
-    directory: str,
-    output: str,
-    compact: bool,
-):
-    try:
-        assert build in [
-            "qnodes",
-            "wdclasses",
-            "wdprops",
-            "enwiki_links",
-            "qnode_redirections",
-            "qnode_labels",
-            "qnode_identifiers",
-        ]
-    except ValueError:
-        logger.error("Invalid build value: {}. Exiting!", build)
-        return
-
-    directory = directory.strip()
-    output_dir = Path(output.strip())
-    output_dir.mkdir(exist_ok=True, parents=True)
-
-    if directory != "":
-        os.environ["WIKIDATA_DIR"] = directory
-
-    from kgdata.config import WIKIDATA_DIR
-    from kgdata.wikidata.s00_prep_data import prep01
-    from kgdata.wikidata.deprecated.rdd_datasets import (
-        qnodes_en,
-        wiki_article_to_qnode,
-        qnodes_identifier,
-    )
-    from kgdata.wikidata.deprecated.models.ontology import (
-        save_wdprops,
-        save_wdclasses,
-        make_ontology,
-        make_superclass_closure,
-        examine_ontology_property,
-    )
-    from kgdata.spark import does_result_dir_exist
-    from kgdata.wikidata.deprecated.db import (
-        get_qnode_db,
-        get_wdprop_db,
-        get_wdclass_db,
-        get_qnode_redirection_db,
-        get_qnode_label_db,
-    )
-    from kgdata.wikidata.redirections import extract_redirections
-
-    logger.info("Wikidata directory: {}", WIKIDATA_DIR)
-    logger.info("Build: {}. Compaction: {}", build, compact)
-
-    # build qnode redirections do not need entity dumps
-    if build == "qnode_redirections":
-        redirection_file = extract_redirections(
-            page_dump=os.path.join(WIKIDATA_DIR, "step_0/wikidatawiki-page.sql.gz"),
-            redirect_dump=os.path.join(
-                WIKIDATA_DIR, "step_0/wikidatawiki-redirect.sql.gz"
-            ),
-            outdir=os.path.join(WIKIDATA_DIR, "redirections"),
-        )
-        dbpath = os.path.join(output_dir, "qnode_redirections.db")
-        db = load_single_file(
-            db=get_qnode_redirection_db(
-                dbpath, create_if_missing=True, read_only=False
-            ).db,
-            infile=redirection_file,
-            format=FileFormat.tabsep,
-            key_fn=identity,
-            value_fn=identity,
-        )
-        if compact:
-            logger.info("Run compaction...")
-            db.compact_range()
-        return
-
-    # process the raw wikidata dump
-    prep01(
-        indir=os.path.join(WIKIDATA_DIR, "step_0"),
-        outdir=os.path.join(WIKIDATA_DIR, "step_1"),
-        overwrite=False,
-    )
-    # extract qnodes from wikidata english
-    qnode_files = os.path.join(WIKIDATA_DIR, "step_2/qnodes_en")
-    if not does_result_dir_exist(qnode_files):
-        qnodes_en(outfile=qnode_files)
-
-    if build == "qnodes":
-        dbpath = os.path.join(output_dir, "qnodes.db")
-        db = load(
-            db=get_qnode_db(dbpath).db,
-            infiles=glob.glob(os.path.join(qnode_files, "*.gz")),
-            format=FileFormat.jsonline,
-            key_fn=Chain2(str.encode, itemgetter("id")).exec,
-            value_fn=Chain2(zstd6_compress, orjson.dumps).exec,
-            n_processes=8,
-            shm_mem_ratio=12,
-            shm_mem_limit_mb=128,
-        )
-
-        if compact:
-            logger.info("Run compaction...")
-            db.compact_range()
-        return
-
-    if build == "qnode_labels":
-        dbpath = os.path.join(output_dir, "qnode_labels.db")
-        db = load(
-            db=get_qnode_label_db(dbpath, create_if_missing=True, read_only=False).db,
-            infiles=glob.glob(os.path.join(qnode_files, "*.gz")),
-            format=FileFormat.jsonline,
-            key_fn=Chain2(str.encode, itemgetter("id")).exec,
-            value_fn=extract_id_label,
-            n_processes=8,
-            shm_mem_limit_mb=8,
-            shm_mem_ratio=20,
-        )
-        if compact:
-            logger.info("Run compaction...")
-            db.compact_range()
-        return
-
-    if build == "qnode_identifiers":
-        qnodes_identifier(outfile=os.path.join(output_dir, "qnode_identifiers.txt"))
-        return
-
-    if build == "enwiki_links":
-        wiki_article_to_qnode()
-        dbpath = os.path.join(output_dir, "enwiki_links.db")
-        db = load(
-            db=get_qnode_db(dbpath).db,
-            infiles=glob.glob(os.path.join(WIKIDATA_DIR, "step_2/enwiki_links/*.gz")),
-            format=FileFormat.tuple2,
-            key_fn=str.encode,
-            value_fn=str.encode,
-            n_processes=8,
-            shm_mem_limit_mb=8,
-            shm_mem_ratio=20,
-        )
-        if compact:
-            logger.info("Run compaction...")
-            db.compact_range()
-        return
-
-    if build in ["wdclasses", "wdprops"]:
-        make_ontology()
-        make_superclass_closure()
-        # TODO: uncomment to verify if the data is correct
-        # examine_ontology_property()
-        dbpath = os.path.join(output_dir, f"{build}.db")
-        if build == "wdprops":
-            db = get_wdprop_db(dbpath, create_if_missing=True, read_only=False).db
-            save_wdprops(indir=os.path.join(WIKIDATA_DIR, "ontology"), db=db)
-        elif build == "wdclasses":
-            db = get_wdclass_db(dbpath, create_if_missing=True, read_only=False).db
-            save_wdclasses(indir=os.path.join(WIKIDATA_DIR, "ontology"), db=db)
-            logger.info("Finish saving wdclasses to DB")
-        else:
-            raise Exception("Unreachable!")
-        if compact:
-            logger.info("Run compaction...")
-            db.compact_range()
-        return
-
-
-def extract_id_label(odict):
-    label = odict["label"]
-    label = label["lang2value"][label["lang"]]
-    return orjson.dumps({"id": odict["id"], "label": label})
diff --git a/kgdata/wikidata/deprecated/db.py b/kgdata/wikidata/deprecated/db.py
deleted file mode 100644
index bf566a5..0000000
--- a/kgdata/wikidata/deprecated/db.py
+++ /dev/null
@@ -1,281 +0,0 @@
-from pathlib import Path
-from typing import *
-import rocksdb
-import gzip
-from hugedict.misc import zstd6_compress_custom, zstd_decompress_custom, identity
-import requests
-
-from hugedict.prelude import RocksDBDict
-from kgdata.wikidata.deprecated.models.qnode import QNode, QNodeLabel
-from kgdata.wikidata.deprecated.wdclass import WDClass
-from kgdata.wikidata.deprecated.wdproperty import WDProperty
-
-V = TypeVar("V", bound=Union[QNode, WDClass, WDProperty])
-CompressionType = rocksdb.CompressionType  # type: ignore
-
-
-class WDProxyDB(RocksDBDict[str, V]):
-    def __init__(
-        self,
-        EntClass,
-        dbfile: Union[Path, str],
-        compression: bool,
-        create_if_missing=True,
-        read_only=False,
-        db_options: Optional[Dict[str, Any]] = None,
-    ):
-        super().__init__(
-            dbpath=dbfile,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            deser_key=bytes.decode,
-            ser_key=str.encode,
-            deser_value=zstd_decompress_custom(EntClass.deserialize)
-            if compression
-            else EntClass.deserialize,
-            ser_value=zstd6_compress_custom(EntClass.serialize)
-            if compression
-            else EntClass.serialize,
-            db_options=db_options,
-        )
-
-        if not hasattr(EntClass, "from_qnode"):
-            self.extract_ent_from_qnode: Callable[[QNode], V] = identity
-        else:
-            self.extract_ent_from_qnode: Callable[[QNode], V] = getattr(
-                EntClass, "from_qnode"
-            )
-
-    def __getitem__(self, key: str):
-        item = self.db.get(key.encode())
-        if item == b"\x00":
-            raise KeyError(key)
-        elif item is None:
-            qnodes = query_wikidata_entities([key])
-            if len(qnodes) == 0:
-                # no entity
-                self.db.put(key.encode(), b"\x00")
-                raise KeyError(key)
-            else:
-                ent = self.extract_ent_from_qnode(qnodes[key])
-                self.db.put(key.encode(), self.ser_value(ent))
-            return ent
-        return self.deser_value(item)
-
-    def get(self, key: str, default: Optional[V] = None):
-        try:
-            return self[key]
-        except KeyError:
-            return default
-
-    def __contains__(self, key):
-        item = self.db.get(key.encode())
-        if item == b"\x00":
-            return False
-        if item is None:
-            qnodes = query_wikidata_entities([key])
-            if len(qnodes) == 0:
-                # no entity
-                self.db.put(key.encode(), b"\x00")
-                return False
-
-            ent = self.extract_ent_from_qnode(qnodes[key])
-            self.db.put(key.encode(), self.ser_value(ent))
-        return True
-
-    def does_not_exist_locally(self, key):
-        item = self.db.get(key.encode())
-        return item == b"\x00"
-
-
-def get_wikipedia_to_wikidata_db(
-    dbpath: Union[Path, str],
-    create_if_missing=False,
-    read_only=True,
-) -> RocksDBDict[str, str]:
-    db_options = {"compression": CompressionType.lz4_compression}
-    return RocksDBDict(
-        dbpath,
-        create_if_missing=create_if_missing,
-        read_only=read_only,
-        deser_key=bytes.decode,
-        ser_key=str.encode,
-        deser_value=bytes.decode,
-        ser_value=str.encode,
-        db_options=db_options,
-    )
-
-
-def get_qnode_label_db(
-    dbfile: Union[Path, str],
-    create_if_missing=False,
-    read_only=True,
-) -> RocksDBDict[str, QNodeLabel]:
-    db_options = {
-        "compression": CompressionType.lz4_compression,
-    }
-    return RocksDBDict(
-        dbfile,
-        create_if_missing=create_if_missing,
-        read_only=read_only,
-        deser_key=bytes.decode,
-        ser_key=str.encode,
-        deser_value=QNodeLabel.deserialize,
-        ser_value=QNodeLabel.serialize,
-        db_options=db_options,
-    )
-
-
-def get_qnode_redirection_db(
-    dbfile: Union[Path, str],
-    create_if_missing=False,
-    read_only=True,
-) -> RocksDBDict[str, str]:
-    db_options = {
-        "compression": CompressionType.lz4_compression,
-    }
-    return RocksDBDict(
-        dbfile,
-        create_if_missing=create_if_missing,
-        read_only=read_only,
-        deser_key=bytes.decode,
-        ser_key=str.encode,
-        deser_value=bytes.decode,
-        ser_value=str.encode,
-        db_options=db_options,
-    )
-
-
-def get_qnode_db(
-    dbfile: Union[Path, str],
-    create_if_missing=True,
-    read_only=False,
-    proxy: bool = False,
-) -> RocksDBDict[str, QNode]:
-    # no compression as we pre-compress the qnodes -- get much better compression
-    db_options = {"compression": CompressionType.no_compression}
-    if proxy:
-        db: RocksDBDict[str, QNode] = WDProxyDB(
-            QNode,
-            dbfile,
-            compression=True,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            db_options=db_options,
-        )
-    else:
-        db = RocksDBDict(
-            dbpath=dbfile,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            deser_key=bytes.decode,
-            ser_key=str.encode,
-            deser_value=zstd_decompress_custom(QNode.deserialize),
-            ser_value=zstd6_compress_custom(QNode.serialize),
-            db_options=db_options,
-        )
-    return db
-
-
-def get_wdclass_db(
-    dbfile: Union[Path, str],
-    create_if_missing=True,
-    read_only=False,
-    proxy: bool = False,
-) -> RocksDBDict[str, WDClass]:
-    db_options = {
-        "compression": CompressionType.lz4_compression,
-        "bottommost_compression": CompressionType.zstd_compression,
-    }
-    if proxy:
-        db = WDProxyDB(
-            WDClass,
-            dbfile,
-            compression=False,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            db_options=db_options,
-        )
-    else:
-        db = RocksDBDict(
-            dbpath=dbfile,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            deser_key=bytes.decode,
-            ser_key=str.encode,
-            deser_value=WDClass.deserialize,
-            ser_value=WDClass.serialize,
-            db_options=db_options,
-        )
-    return db
-
-
-def get_wdprop_db(
-    dbfile: Union[Path, str],
-    create_if_missing=True,
-    read_only=False,
-    proxy: bool = False,
-) -> RocksDBDict[str, WDProperty]:
-    db_options = {"compression": CompressionType.lz4_compression}
-    if proxy:
-        db = WDProxyDB(
-            WDProperty,
-            dbfile,
-            compression=False,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            db_options=db_options,
-        )
-    else:
-        db = RocksDBDict(
-            dbpath=dbfile,
-            create_if_missing=create_if_missing,
-            read_only=read_only,
-            deser_key=bytes.decode,
-            ser_key=str.encode,
-            deser_value=WDProperty.deserialize,
-            ser_value=WDProperty.serialize,
-            db_options=db_options,
-        )
-    return db
-
-
-def query_wikidata_entities(
-    qnode_ids: Union[Set[str], List[str]],
-    endpoint: str = "https://www.wikidata.org/w/api.php",
-) -> Dict[str, QNode]:
-    assert len(qnode_ids) > 0, qnode_ids
-    resp = requests.get(
-        endpoint,
-        params={
-            "action": "wbgetentities",
-            "ids": "|".join(qnode_ids),
-            "format": "json",
-        },
-    )
-    assert resp.status_code, resp
-    data = resp.json()
-    if "success" not in data:
-        assert "error" in data, data
-        # we have invalid entity id format, if we only query for one entity
-        # we can tell it doesn't exist, otherwise, we don't know which entity are wrong
-        if len(qnode_ids) == 1:
-            return {}
-        raise Exception(
-            f"Invalid entity ID format. Don't know which entities are wrong. {qnode_ids}"
-        )
-    else:
-        assert data.get("success", None) == 1, data
-    qnodes = {}
-
-    for qnode_id in qnode_ids:
-        if "missing" in data["entities"][qnode_id]:
-            continue
-        qnode = QNode.from_wikidump(data["entities"][qnode_id])
-        qnodes[qnode.id] = qnode
-        if qnode.id != qnode_id:
-            # redirection -- the best way is to update the redirection map
-            origin_qnode = qnode.shallow_clone()
-            origin_qnode.id = qnode_id
-            qnodes[qnode_id] = origin_qnode
-    return qnodes
diff --git a/kgdata/wikidata/deprecated/models/ontology.py b/kgdata/wikidata/deprecated/models/ontology.py
deleted file mode 100644
index 9ed5842..0000000
--- a/kgdata/wikidata/deprecated/models/ontology.py
+++ /dev/null
@@ -1,298 +0,0 @@
-from functools import partial
-import os, gzip
-from operator import add, itemgetter
-from pathlib import Path
-from typing import Dict, Optional, Set, Union
-from graph.interface import BaseEdge, BaseNode
-from hugedict.misc import identity
-
-import rocksdb
-from loguru import logger
-from tqdm.auto import tqdm
-
-from kgdata.config import WIKIDATA_DIR
-from kgdata.spark import does_result_dir_exist, get_spark_context, saveAsSingleTextFile
-from kgdata.wikidata.models import QNode, WDClass, WDProperty
-from kgdata.wikidata.deprecated.rdd_datasets import qnodes_en
-from sm.misc import serialize_jl, serialize_json, deserialize_jl
-from graph.retworkx import digraph_find_cycle, RetworkXStrDiGraph
-
-"""
-This module provides function to extract Wikidata ontology from its dump. 
-"""
-
-
-def make_ontology(outfile: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    """Get the list of classes and predicates from Wikidata
-    See the data structure: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Site_Links
-    """
-    # list of class and prop ids
-    step0_file = os.path.join(outfile, "step_0")
-    # list of class and prop full qnode
-    step1_file = os.path.join(outfile, "step_1")
-    # count the frequency of props per class
-    step2_file = os.path.join(outfile, "freq_stats.jl")
-
-    class_qnode_file = os.path.join(outfile, "classes.jl")
-    prop_qnode_file = os.path.join(outfile, "properties.jl")
-
-    if not does_result_dir_exist(step0_file):
-
-        def p_0_get_property_and_class(qnode: QNode):
-            if qnode.type == "property":
-                qnode_ids = [
-                    stmt.value.as_entity_id() for stmt in qnode.props.get("P1647", [])
-                ]
-                qnode_ids.append(qnode.id)
-                return qnode_ids
-
-            if "P279" in qnode.props:
-                # this qnode is a class, but we want to get the parent classes as well.
-                qnode_ids = [
-                    stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])
-                ]
-                qnode_ids.append(qnode.id)
-                return qnode_ids
-
-            return [stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])]
-
-        qnodes_en().flatMap(p_0_get_property_and_class).distinct().coalesce(
-            256
-        ).saveAsTextFile(
-            step0_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    if not does_result_dir_exist(step1_file):
-        sc = get_spark_context()
-
-        qnode_ids = sc.broadcast(set(sc.textFile(step0_file).collect()))
-        qnodes_en().filter(lambda qnode: qnode.id in qnode_ids.value).map(
-            QNode.serialize
-        ).saveAsTextFile(
-            step1_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    item_rdd = lambda: get_spark_context().textFile(step1_file).map(QNode.deserialize)
-
-    if not os.path.exists(class_qnode_file):
-
-        def p_1_extract_class(qnode: QNode):
-            if qnode.type != "item":
-                return None
-
-            return WDClass(
-                id=qnode.id,
-                label=qnode.label,
-                description=qnode.description,
-                datatype=qnode.datatype,
-                aliases=qnode.aliases,
-                parents=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])}
-                ),
-                properties=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1963", [])}
-                ),
-                different_froms=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1889", [])}
-                ),
-                equivalent_classes=sorted(
-                    {stmt.value.as_string() for stmt in qnode.props.get("P1709", [])}
-                ),
-                parents_closure=set(),
-            )
-
-        class_rdd = (
-            item_rdd()
-            .map(p_1_extract_class)
-            .filter(lambda x: x is not None)
-            .map(WDClass.serialize)
-        )
-        saveAsSingleTextFile(class_rdd, class_qnode_file)
-
-    if not os.path.exists(prop_qnode_file):
-
-        def p_2_extract_property(qnode: QNode):
-            if qnode.type != "property":
-                return None
-            return WDProperty(
-                id=qnode.id,
-                label=qnode.label,
-                description=qnode.description,
-                datatype=qnode.datatype,
-                aliases=qnode.aliases,
-                parents=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1647", [])}
-                ),
-                see_also=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1659", [])}
-                ),
-                equivalent_properties=sorted(
-                    {stmt.value.as_string() for stmt in qnode.props.get("P1628", [])}
-                ),
-                subjects=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1629", [])}
-                ),
-                inverse_properties=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1696", [])}
-                ),
-                instanceof=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])}
-                ),
-            )
-
-        prop_rdd = (
-            item_rdd()
-            .map(p_2_extract_property)
-            .filter(lambda x: x is not None)
-            .map(WDProperty.serialize)
-        )
-        saveAsSingleTextFile(prop_rdd, prop_qnode_file)
-
-    if not os.path.exists(step2_file):
-
-        def p_3_count_props(qnode: QNode):
-            if qnode.type == "property":
-                return []
-            class_ids = []
-            for stmt in qnode.props.get("P31", []):
-                class_ids.append(stmt.value.as_entity_id())
-
-            triples = []
-            for p in qnode.props:
-                if p == "P31":
-                    continue
-                for cid in class_ids:
-                    triples.append(((cid, p), 1))
-            for cid in class_ids:
-                triples.append(((cid, "_ccount"), 1))
-            return triples
-
-        sp_count = qnodes_en().flatMap(p_3_count_props).reduceByKey(add).collect()
-        class_freq = {}
-        for (s, p), c in sp_count:
-            if s not in class_freq:
-                class_freq[s] = {"props": {}, "count": 0, "class": s}
-            if p == "_ccount":
-                class_freq[s]["count"] = c
-            else:
-                class_freq[s]["props"][p] = c
-
-        serialize_jl(sorted(class_freq.values(), key=itemgetter("class")), step2_file)
-
-
-def make_superclass_closure(outdir: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    def get_all_parents(index: Dict[str, dict], id: str, parents: Set[str]):
-        for pid in index[id]["parents"]:
-            if pid not in parents:
-                if pid not in index:
-                    continue
-                parents.add(pid)
-                get_all_parents(index, pid, parents)
-
-    superclasses_closure_file = os.path.join(outdir, "superclasses_closure.jl")
-    superproperties_closure_file = os.path.join(outdir, "superproperties_closure.jl")
-
-    if not os.path.exists(superclasses_closure_file):
-        logger.info("Make superclass closure...")
-        # need to make sure that they are in the same order
-        classes = deserialize_jl(os.path.join(outdir, "classes.jl"))
-        classes_order = [c["id"] for c in classes]
-        classes = {c["id"]: c for c in classes}
-
-        class_closure = {}
-        for c in tqdm(classes.values()):
-            parents = set()
-            get_all_parents(classes, c["id"], parents)
-            class_closure[c["id"]] = list(parents)
-
-        class_closure = [(cid, class_closure[cid]) for cid in classes_order]
-        serialize_jl(class_closure, superclasses_closure_file)
-
-    if not os.path.exists(superproperties_closure_file):
-        logger.info("Make superproperties closure...")
-        predicates = deserialize_jl(os.path.join(outdir, "properties.jl"))
-        predicates_order = [p["id"] for p in predicates]
-        predicates = {p["id"]: p for p in predicates}
-
-        predicate_closure = {}
-        for r in tqdm(predicates.values()):
-            parents = set()
-            get_all_parents(predicates, r["id"], parents)
-            predicate_closure[r["id"]] = list(parents)
-
-        predicate_closure = [(pid, predicate_closure[pid]) for pid in predicates_order]
-        serialize_jl(predicate_closure, superproperties_closure_file)
-
-
-def examine_ontology_property(indir: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    def print_cycle(edges: list[BaseEdge[str, str]]):
-        print("\t>", " -> ".join([e.source for e in edges]) + " -> " + edges[-1].target)
-
-    def find_cycles(g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]]):
-        nodes = g.nodes()
-        for node in nodes:
-            cycle = digraph_find_cycle(g, node.id)
-            if len(cycle) == 0:
-                continue
-
-            while True:
-                print_cycle(cycle)
-                e = cycle[0]
-                g.remove_edges_between_nodes(e.source, e.target)
-                cycle = digraph_find_cycle(g, cycle[-1].target)
-                if len(cycle) == 0:
-                    break
-
-    # classes = WDClass.from_file(indir)
-    # g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]] = RetworkXStrDiGraph(
-    #     multigraph=False
-    # )
-    # for c in tqdm(classes.values()):
-    #     for pcid in c.parents:
-    #         g.add_edge(BaseEdge(-1, c.id, pcid, ""))
-
-    # print("Detecting cycles in the ontology classes...")
-    # find_cycles(g)
-
-    predicates = WDProperty.from_file(indir)
-
-    g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]] = RetworkXStrDiGraph(
-        multigraph=False
-    )
-    for p in tqdm(predicates.values()):
-        for ppid in p.parents:
-            g.add_edge(BaseEdge(-1, p.id, ppid, ""))
-
-    print("Detecting cycles in the ontology properties...")
-    find_cycles(g)
-
-
-def save_wdprops(indir: Union[str, Path], db):
-    if indir == "":
-        indir = os.path.join(WIKIDATA_DIR, "ontology")
-
-    wdprops = WDProperty.from_file(indir, load_parent_closure=True)
-    wb = rocksdb.WriteBatch()
-    for id, item in tqdm(wdprops.items(), total=len(wdprops)):
-        wb.put(id.encode(), item.serialize())
-    db.write(wb)
-
-
-def save_wdclasses(indir: Union[str, Path], db):
-    if indir == "":
-        indir = os.path.join(WIKIDATA_DIR, "ontology")
-
-    wb = rocksdb.WriteBatch()
-    for idx, item in tqdm(
-        enumerate(WDClass.iter_file(indir, load_parent_closure=True)),
-        desc="Write WDClass",
-    ):
-        wb.put(item.id.encode(), item.serialize())
-        if idx % 1000 == 0:
-            db.write(wb)
-            wb = rocksdb.WriteBatch()
-    db.write(wb)
-
-
-if __name__ == "__main__":
-    examine_ontology_property()
diff --git a/kgdata/wikidata/deprecated/models/qnode.py b/kgdata/wikidata/deprecated/models/qnode.py
deleted file mode 100644
index 356e291..0000000
--- a/kgdata/wikidata/deprecated/models/qnode.py
+++ /dev/null
@@ -1,398 +0,0 @@
-from dataclasses import dataclass
-from typing import List, Dict, Literal, Tuple, TypedDict, Union, Optional
-
-import orjson
-
-
-DataValueString = str
-DataValueWikibaseEntityId = TypedDict(
-    "DataValueWikibaseEntityId",
-    {
-        "entity-type": Literal["item", "property"],
-        "id": str,
-        # WARNING: not all entity IDs have a numeric ID – using the full ID is highly recommended.
-        "numeric-id": str,
-    },
-)
-DataValueGlobeCoordinate = TypedDict(
-    "DataValueGlobeCoordinate",
-    {
-        "latitude": float,
-        "longitude": float,
-        "precision": float,
-        # deprecated, No longer used. Will be dropped in the future
-        "altitude": None,
-        # The URI of a reference globe. This would typically refer to a data item on wikidata.org. This is usually just an indication of the celestial body (e.g. Q2 = earth), but could be more specific, like WGS 84 or ED50.
-        "globe": str,
-    },
-)
-DataValueQuantity = TypedDict(
-    "DataValueQuantity",
-    {
-        # The nominal value of the quantity, as an arbitrary precision decimal string. The string always starts with a character indicating the sign of the value, either “+” or “-”.
-        "amount": str,
-        # Optionally, the upper bound of the quantity's uncertainty interval, using the same notation as the amount field. If not given or null, the uncertainty (or precision) of the quantity is not known. If the upperBound field is given, the lowerBound field must also be given.
-        "upperBound": str,
-        # Optionally, the lower bound of the quantity's uncertainty interval, using the same notation as the amount field. If not given or null, the uncertainty (or precision) of the quantity is not known. If the lowerBound field is given, the upperBound field must also be given.
-        "lowerBound": str,
-        # The URI of a unit (or “1” to indicate a unit-less quantity). This would typically refer to a data item on wikidata.org, e.g. http://www.wikidata.org/entity/Q712226 for “square kilometer”.
-        "unit": str,
-    },
-)
-DataValueTime = TypedDict(
-    "DataValueTime",
-    {
-        # See more: https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_json.html
-        "time": str,
-        "timezone": int,
-        "before": int,
-        "after": int,
-        "precision": int,
-        "calendarmodel": str,
-    },
-)
-DataValueMonolingualText = TypedDict(
-    "DataValueMonolingualText",
-    {
-        "text": str,
-        "language": str,
-    },
-)
-DataValueType = Literal[
-    "string",
-    "wikibase-entityid",
-    "globecoordinate",
-    "quantity",
-    "time",
-    "monolingualtext",
-]
-
-
-@dataclass
-class DataValue:
-    __slots__ = ("type", "value")
-
-    # https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Data_Values is moved to https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_json.html
-    # the new document does not have all types specified in https://www.wikidata.org/wiki/Help:Data_type such as monolingualtext
-    # so keep in mind the type may not be exhausted
-    type: DataValueType
-    value: Union[
-        DataValueString,
-        DataValueWikibaseEntityId,
-        DataValueGlobeCoordinate,
-        DataValueQuantity,
-        DataValueTime,
-        DataValueMonolingualText,
-    ]
-
-    def is_string(self):
-        return self.type == "string"
-
-    def is_time(self):
-        return self.type == "time"
-
-    def is_quantity(self):
-        return self.type == "quantity"
-
-    def is_mono_lingual_text(self):
-        return self.type == "monolingualtext"
-
-    def is_globe_coordinate(self):
-        return self.type == "globecoordinate"
-
-    def is_entity_id(self):
-        return self.type == "wikibase-entityid"
-
-    def is_qnode(self):
-        return self.type == "wikibase-entityid" and self.value["id"][0] == "Q"  # type: ignore
-
-    def is_pnode(self):
-        return self.type == "wikibase-entityid" and self.value["id"][0] == "P"  # type: ignore
-
-    def as_entity_id(self) -> str:
-        return self.value["id"]  # type: ignore
-
-    def as_qnode_id_safe(self) -> str:
-        assert self.is_qnode()
-        return self.value["id"]  # type: ignore
-
-    def as_pnode_id_safe(self) -> str:
-        assert self.is_pnode()
-        return self.value["id"]  # type: ignore
-
-    def as_string(self) -> str:
-        return self.value  # type: ignore
-
-    def as_qnode_label_safe(self) -> str:
-        assert self.is_qnode()
-        """Label of the qnode doesn't come with the data. It is set using `set_qnode_label` function."""
-        return self.value["label"]  # type: ignore
-
-    def set_qnode_label(self, label: Optional[str]):
-        self.value["label"] = label  # type: ignore
-
-    def set_qnode_label_safe(self, label: str):
-        assert self.is_qnode()
-        self.value["label"] = label  # type: ignore
-
-    def to_string_repr(self) -> str:
-        if isinstance(self.value, str):
-            return self.value
-        return orjson.dumps(self.value).decode()
-
-
-@dataclass
-class SiteLink:
-    __slots__ = ("site", "title", "badges", "url")
-
-    site: str
-    title: str
-    badges: List[str]
-    url: Optional[str]
-
-
-@dataclass
-class Statement:
-    __slots__ = ("value", "qualifiers", "qualifiers_order")
-
-    value: DataValue
-    # mapping from qualifier id into data value
-    qualifiers: Dict[str, List[DataValue]]
-    # list of qualifiers id that records the order (as dict lacks of order)
-    qualifiers_order: List[str]
-
-    @staticmethod
-    def from_dict(o):
-        o["qualifiers"] = {
-            k: [DataValue(**v) for v in vals] for k, vals in o["qualifiers"].items()
-        }
-        o["value"] = DataValue(**o["value"])
-        return Statement(**o)
-
-
-class MultiLingualString(str):
-    # two characters language code: en, th, de, fr, etc.
-    lang: str
-    lang2value: Dict[str, str]
-
-    def __new__(cls, lang2value: Dict[str, str], lang):
-        object = str.__new__(cls, lang2value[lang])
-        object.lang = lang
-        object.lang2value = lang2value
-        return object
-
-    def as_lang(self, lang: str) -> str:
-        return self.lang2value[lang]
-
-    @staticmethod
-    def en(label: str):
-        return MultiLingualString(lang2value={"en": label}, lang="en")
-
-    def serialize(self):
-        return {"lang2value": self.lang2value, "lang": self.lang}
-
-    def __getnewargs__(self) -> Tuple[Dict[str, str], str]:
-        return (self.lang2value, self.lang)
-
-
-class MultiLingualStringList(List[str]):
-    def __init__(self, lang2values: Dict[str, List[str]], lang):
-        super().__init__(lang2values[lang])
-        self.lang2values = lang2values
-        self.lang = lang
-
-    def serialize(self):
-        return {"lang2values": self.lang2values, "lang": self.lang}
-
-
-@dataclass
-class QNodeLabel:
-    # contains only label & id of qnode
-    __slots__ = ("id", "label")
-    id: str
-    label: str
-
-    @staticmethod
-    def deserialize(s):
-        return QNodeLabel(**orjson.loads(s))
-
-    def serialize(self):
-        return orjson.dumps({"id": self.id, "label": self.label})
-
-
-@dataclass
-class QNode:
-    __slots__ = (
-        "id",
-        "type",
-        "datatype",
-        "label",
-        "description",
-        "aliases",
-        "props",
-        "sitelinks",
-    )
-
-    id: str
-    # possible values ["item", "property"]
-    type: str
-    label: MultiLingualString
-    # the datatype is not described in the documentation: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON
-    # however, seems to appear for the property, and will tell us whether the property is an external identifier or not
-    # possible values:
-    # ['quantity', 'wikibase-form', 'geo-shape', 'math', 'wikibase-item', 'musical-notation', 'commonsMedia', 'wikibase-property', 'wikibase-lexeme', 'tabular-data', 'time', 'wikibase-sense', 'external-id', 'monolingualtext', 'string', 'globe-coordinate', 'url']
-    datatype: Optional[str]
-    description: MultiLingualString
-    aliases: MultiLingualStringList
-    props: Dict[str, List[Statement]]
-    sitelinks: Dict[str, SiteLink]
-
-    def serialize(self):
-        # make it self, so we can serialize
-        odict = {
-            k: getattr(self, k)
-            for k in [
-                "id",
-                "type",
-                "label",
-                "datatype",
-                "description",
-                "aliases",
-                "props",
-                "sitelinks",
-            ]
-        }
-        for k in ["label", "description", "aliases"]:
-            odict[k] = odict[k].serialize()
-        return orjson.dumps(odict, option=orjson.OPT_SERIALIZE_DATACLASS, default=list)
-
-    def shallow_clone(self):
-        return QNode(
-            id=self.id,
-            type=self.type,
-            label=self.label,
-            datatype=self.datatype,
-            description=self.description,
-            aliases=self.aliases,
-            props=self.props,
-            sitelinks=self.sitelinks,
-        )
-
-    @staticmethod
-    def deserialize(s):
-        o = orjson.loads(s)
-        return QNode.from_dict(o)
-
-    @staticmethod
-    def from_dict(o):
-        o["props"] = {
-            k: [Statement.from_dict(v) for v in vals] for k, vals in o["props"].items()
-        }
-        o["sitelinks"] = {k: SiteLink(**v) for k, v in o["sitelinks"].items()}
-        o["label"] = MultiLingualString(**o["label"])
-        o["description"] = MultiLingualString(**o["description"])
-        o["aliases"] = MultiLingualStringList(**o["aliases"])
-        return QNode(**o)
-
-    @staticmethod
-    def from_wikidump(qnode, lang: str = "en") -> "QNode":
-        """Extract essential information from qnode in the form that are easier to work with
-
-        Read more about ranks and truthy statements:
-            - https://www.wikidata.org/wiki/Help:Ranking
-            - https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Truthy_statements
-
-        Parameters
-        ----------
-        qnode : dict
-            qnode
-        lang : str, optional
-            language, default is 'en'
-
-        Returns
-        -------
-        QNode
-        """
-        props = {}
-        sitelinks = {}
-
-        for prop, stmts in qnode["claims"].items():
-            prop_value = []
-            for stmt in stmts:
-                if stmt["rank"] == "deprecated":
-                    continue
-                mainsnak = stmt["mainsnak"]
-                if mainsnak["snaktype"] != "value":
-                    assert "datavalue" not in mainsnak
-                    continue
-                datavalue = mainsnak["datavalue"]
-
-                try:
-                    value = datavalue["value"]
-                    type = datavalue["type"]
-                except:
-                    print(datavalue)
-                    raise
-
-                qualifiers = {}
-                if "qualifiers" in stmt:
-                    assert "qualifiers-order" in stmt
-                    for qualifier_id, qualifier_snaks in stmt["qualifiers"].items():
-                        qualifier_values = []
-                        for qualifier_snak in qualifier_snaks:
-                            if qualifier_snak["snaktype"] != "value":
-                                assert "datavalue" not in qualifier_snak["snaktype"]
-                                continue
-
-                            qualifier_dvalue = qualifier_snak["datavalue"]["value"]
-                            qualifier_dtype = qualifier_snak["datavalue"]["type"]
-
-                            qualifier_values.append(
-                                DataValue(value=qualifier_dvalue, type=qualifier_dtype)
-                            )
-                        qualifiers[qualifier_id] = qualifier_values
-                    qualifiers_order = stmt["qualifiers-order"]
-                else:
-                    qualifiers_order = []
-
-                prop_value.append(
-                    Statement(
-                        value=DataValue(value=value, type=type),
-                        qualifiers=qualifiers,
-                        qualifiers_order=qualifiers_order,
-                    )
-                )
-            props[prop] = prop_value
-
-        for key, sitelink in qnode.get("sitelinks", {}).items():
-            sitelinks[key] = SiteLink(
-                sitelink["site"],
-                sitelink["title"],
-                badges=sitelink["badges"],
-                url=sitelink.get("url", None),
-            )
-
-        label = {k: v["value"] for k, v in qnode["labels"].items()}
-        if lang not in label:
-            label[lang] = ""
-        label = MultiLingualString(label, lang)
-
-        description = {k: v["value"] for k, v in qnode["descriptions"].items()}
-        if lang not in description:
-            description[lang] = ""
-        description = MultiLingualString(description, lang)
-
-        aliases = {k: [x["value"] for x in lst] for k, lst in qnode["aliases"].items()}
-        if lang not in aliases:
-            aliases[lang] = []
-        aliases = MultiLingualStringList(aliases, lang)
-        return QNode(
-            id=qnode["id"].upper(),
-            type=qnode["type"],
-            datatype=qnode.get("datatype", None),
-            label=label,
-            description=description,
-            aliases=aliases,
-            props=props,
-            sitelinks=sitelinks,
-        )
diff --git a/kgdata/wikidata/deprecated/online_api.py b/kgdata/wikidata/deprecated/online_api.py
deleted file mode 100644
index 6591ada..0000000
--- a/kgdata/wikidata/deprecated/online_api.py
+++ /dev/null
@@ -1,269 +0,0 @@
-import ujson, time, rdflib, os, logging
-from tqdm.auto import tqdm
-from SPARQLWrapper import SPARQLWrapper, JSON
-from typing import *
-from datetime import datetime
-from pathlib import Path
-from ruamel.yaml import YAML
-
-"""
-Functions for querying Wikidata through their SPARQLEndpoint. 
-
-Based on the scripts wikidata_download and wikidata_read in `sm-unknown/scripts`
-"""
-
-logger = logging.getLogger("kg_data.wikidata")
-
-# the endpoint that we are going to query
-# SPARQL_ENDPOINT = os.environ.get('WIKIDATA_SPARQL_ENDPOINT', "https://query.wikidata.org/sparql")
-SPARQL_ENDPOINT = "https://dsbox02.isi.edu:8888/bigdata/namespace/wdq/sparql"
-logger.info(f"Going to use {SPARQL_ENDPOINT} to query wikidata")
-# a predicate telling that we've got all direct predicates of an entity at a specific time
-ACQUIRED_FULL_ONEHOP = "https://isi.edu/ontology/sm/acquired_full_onehop"
-
-
-with open(str(Path(os.path.abspath(__file__)).parent / "wikidata_prefixes.yml"), "r") as f:
-    WIKIDATA_PREFIXES = dict(YAML().load(f))
-    WIKIDATA_INVERT_PREFIXES = {v: k for k, v in WIKIDATA_PREFIXES.items()}
-assert len(WIKIDATA_PREFIXES) == len(WIKIDATA_INVERT_PREFIXES)
-
-
-def get_abs_url(url: str) -> str:
-    global WIKIDATA_PREFIXES
-    if url.startswith("http://") or url.startswith("https://"):
-        return url
-    prefix, url = url.split(":", 1)
-    return WIKIDATA_PREFIXES[prefix] + url
-
-
-def get_rel_url(url: str) -> str:
-    global WIKIDATA_INVERT_PREFIXES
-    if url.startswith("http://") or url.startswith("https://"):
-        x, y = url.rsplit("/", 1)
-        return f"{WIKIDATA_INVERT_PREFIXES[x + '/']}:{y}"
-    return url
-
-
-def query_wikidata(query: str, return_format: Optional[str]="json", endpoint=SPARQL_ENDPOINT):
-    global WIKIDATA_PREFIXES
-
-    query = "\n".join([f"PREFIX {ns}: <{url}>" for ns, url in WIKIDATA_PREFIXES.items()]) + "\n" + query
-    start = time.time()
-    sparql = SPARQLWrapper(endpoint)
-    sparql.setMethod("POST")
-    sparql.setQuery(query)
-    if return_format is not None:
-        sparql.setReturnFormat(return_format)
-
-    result = sparql.query().convert()
-    print(f">>> Finish the query in {time.time() - start:.5f}")
-    return result
-
-
-def download_entity_urls(conditions: List[Tuple[str, str]], outfile: str, limit: Optional[int]=None, offset: int=0):
-    """
-    Download entity's URLs
-
-    @param conditions: list of <p>, <o> condition
-    @param outfile: the file that we are going to append triples to
-    @param limit
-    @param offset
-    """
-    global SPARQL_ENDPOINT, WIKIDATA_PREFIXES
-    conditions = [
-        f"<{get_abs_url(p)}> <{get_abs_url(o)}>"
-        for p, o in conditions
-    ]
-    query = f"""
-        select distinct ?s
-        where {{ ?s {"; ".join(conditions)}. }}
-        {f"LIMIT {limit}" if limit is not None else ""}
-        OFFSET {offset}
-    """
-
-    start = time.time()
-    # setup and send query
-    sparql = SPARQLWrapper(SPARQL_ENDPOINT)
-    sparql.setQuery(query)
-    sparql.setReturnFormat("json")
-    result = sparql.query().convert()
-    print(f">>> Finish the query in {time.time() - start:.5f}")
-
-    triples = []   # triples that is going to write to outfile
-    ent_uris = []  # list of entity URLs
-
-    # parse result
-    for subj in (o['s'] for o in result['results']['bindings']):
-        assert subj['type'] == 'uri'
-        ent_uris.append(subj['value'])
-
-        for po in conditions:
-            triples.append(f"<{subj['value']}> {po} .\n")
-
-    # write triples to file
-    with open(outfile, "a") as f:
-        for triple in triples:
-            f.write(triple)
-    return ent_uris
-
-
-def download_entity_properties(ent_urls: List[str], outfile: str, batch_size: int=5):
-    """
-    Query entity's properties
-
-    @param outfile: the file that we are going to append triples to
-    """
-    global SPARQL_ENDPOINT, WIKIDATA_PREFIXES, ACQUIRED_FULL_ONEHOP
-    batch_ent_urls = [
-        [f"<{get_abs_url(x)}>" for x in ent_urls[i:i+batch_size]]
-        for i in range(0, len(ent_urls), batch_size)
-    ]
-
-    for burls in tqdm(batch_ent_urls):
-        curr_time = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
-        sparql = SPARQLWrapper(SPARQL_ENDPOINT)
-        sparql.setQuery(f"""
-            DESCRIBE ?item
-            WHERE {{ VALUES ?item {{ {" ".join(burls)} }} }}
-        """)
-        sparql.setReturnFormat("rdf+xml")
-        g = sparql.query().convert()
-        triples = g.serialize(format="ntriples").decode()
-        with open(outfile, "a") as f:
-            f.write(triples)
-            for url in burls:
-                f.write(f"{url} <{ACQUIRED_FULL_ONEHOP}> \"{curr_time}^^<http://www.w3.org/2001/XMLSchema#dateTime>\" .\n")
-
-
-def download_predicates(pred_urls: List[str], outfile: str, batch_size: int=5, filter_duplication: bool=True):
-    """
-    Download wikidata property ONLY.
-    """
-    # in wikidata, to get info about a predicate, we have to query its entity
-    ent_preds = set()
-    for pred in pred_urls:
-        if pred.startswith("http://www.wikidata.org/prop/P") or pred.startswith("https://www.wikidata.org/prop/P"):
-            ent_preds.add(pred.replace("/prop/", "/entity/"))
-    
-    outfile_url = Path(outfile).parent / f"{Path(outfile).stem}.urls.txt"
-
-    if filter_duplication and os.path.exists(outfile_url):
-        # we always write these ent_preds to a separated file, so we can ensure uniqueness
-        with open(outfile_url, "r") as f:
-            existing_ent_preds = {line.strip() for line in f}
-            n_ent_preds = len(ent_preds)
-            ent_preds = ent_preds - existing_ent_preds
-            print(">>> filter duplication from %s down to %s", n_ent_preds, len(ent_preds))
-            
-    if len(ent_preds) > 0:
-        # write the predicate we are going to query
-        ent_preds = list(ent_preds)
-        with open(outfile_url, "a") as f:
-            for e in ent_preds:
-                f.write(e + "\n")
-        # query them
-        download_entity_properties(ent_preds, outfile, batch_size)
-
-
-def read_entity(subj: str, g: rdflib.Graph, keep_props: Set[str]=set(), keep_langs: Set[str]=set(), ignore_props: Set[str]=set(), ignore_prop_patterns: list=[]):
-    """
-    Read all properties of an entity from a graph
-
-    @param subj: the entity that we want to obtain full information
-    @param g: the graph that contains triples of the entity, if it is Fuseki instance, we will query the triple store first
-    @param keep_props: set of properties of predicates that we want to keep (e.g., skos:altLabel, rdfs:label). We keep all when it's empty
-    @param keep_langs: set of languages of property's values  that we want to keep (e.g., en). Keep all when it's empty. If you want to also keep literals that don't have language tag, add None to the set
-    @param ignore_props: set of properties that we will discard
-    @param ignore_prop_patterns: set of regex patterns of properties that we will discard
-    """
-    odict = {
-        '@id': subj
-    }
-    if not isinstance(g, rdflib.Graph):
-        g = g.describe("DESCRIBE ?item WHERE { VALUES ?item { <%s> } }" % subj)
-
-    for p, o in g.predicate_objects(rdflib.term.URIRef(subj)):
-        str_p = str(p)
-        if len(keep_props) > 0 and str_p not in keep_props:
-            continue
-        # remove the o.language null check, so that we have option to keep literals that don't have language tag
-        if len(keep_langs) > 0 and isinstance(o, rdflib.term.Literal) and o.language not in keep_langs:
-            continue
-
-        if str_p in ignore_props:
-            continue
-
-        if any(ptn.match(str_p) for ptn in ignore_prop_patterns):
-            continue
-
-        if str_p not in odict:
-            odict[str_p] = []
-        odict[str_p].append(o)
-    return odict
-
-
-def read_batch_entities(subjs: List[str], g: rdflib.Graph, keep_props: Set[str]=set(), keep_langs: Set[str]=set(), ignore_props: Set[str]=set(), ignore_prop_patterns: list=[], return_type: str='list'):
-    """
-    A batch version of the `read_entity` function
-    """
-    if not isinstance(g, rdflib.Graph):
-        g = g.describe("describe ?item where { values ?item { %s }}" % (
-            " ".join((
-                "<%s>" % subj for subj in subjs
-            ))
-        ))
-    
-    if return_type == 'list':
-        return [read_entity(subj, g, keep_props, keep_langs, ignore_props, ignore_prop_patterns) for subj in subjs]
-    else:
-        return {
-            subj: read_entity(subj, g, keep_props, keep_langs, ignore_props, ignore_prop_patterns)
-            for subj in subjs
-        }
-
-
-def read_predicates(infile_or_g: str, keep_props: Set[str]=set(), keep_langs: Set[str]=set()) -> Dict[str, Dict[str, Any]]:
-    """
-    Read information about wikidata predicates that has been stored using `wikidata_download.download_predicates` function
-
-    @param infile_or_g: the file that contains downloaded triples in `ntriples` format or an fuseki instance
-    @param keep_props: set of properties of predicates that we want to keep (e.g., skos:altLabel, rdfs:label). We keep all when it's empty
-    @param keep_langs: set of languages of property's values  that we want to keep (e.g., en). Keep all when it's empty.
-    """
-    predicates = {}
-
-    if isinstance(infile_or_g, str):
-        # parse graph and read predicates
-        start = time.time()
-        g = rdflib.Graph()
-        g.parse(infile_or_g, format='ntriples')
-        print(">>> read graph take %.5f seconds" % (time.time() - start))
-
-        # set of predicates
-        subjs = set()
-        for s in g.subjects():
-            if s.startswith("http://www.wikidata.org/entity/P") or s.startswith("https://www.wikidata.org/entity/P"):
-                subjs.add(s)
-        
-        for subj in tqdm(subjs):
-            predicates[subj.replace("/entity/", "/prop/")] = read_entity(subj, g, keep_props, keep_langs)
-    else:
-        g = infile_or_g
-        res = g.query("""select distinct ?s where {
-    ?s ?p ?o.
-    FILTER (strStarts(STR(?s) , 'http://www.wikidata.org/entity/P'))
-}""")['results']['bindings']
-        subjs = []
-        for r in res:
-            subjs.append(r['s']['value'])
-
-        batch_size = 100
-        with tqdm(total=len(subjs)) as pbar:
-            for i in range(0, len(subjs), batch_size):
-                b_subjs = subjs[i:i+batch_size]
-                ents = read_batch_entities(b_subjs, g, keep_props, keep_langs)
-                for subj, ent in zip(b_subjs, ents):
-                    predicates[subj.replace("/entity/", "/prop/")] = ent
-                pbar.update(len(b_subjs))
-    
-    return predicates
diff --git a/kgdata/wikidata/deprecated/ontology.py b/kgdata/wikidata/deprecated/ontology.py
deleted file mode 100644
index 9ed5842..0000000
--- a/kgdata/wikidata/deprecated/ontology.py
+++ /dev/null
@@ -1,298 +0,0 @@
-from functools import partial
-import os, gzip
-from operator import add, itemgetter
-from pathlib import Path
-from typing import Dict, Optional, Set, Union
-from graph.interface import BaseEdge, BaseNode
-from hugedict.misc import identity
-
-import rocksdb
-from loguru import logger
-from tqdm.auto import tqdm
-
-from kgdata.config import WIKIDATA_DIR
-from kgdata.spark import does_result_dir_exist, get_spark_context, saveAsSingleTextFile
-from kgdata.wikidata.models import QNode, WDClass, WDProperty
-from kgdata.wikidata.deprecated.rdd_datasets import qnodes_en
-from sm.misc import serialize_jl, serialize_json, deserialize_jl
-from graph.retworkx import digraph_find_cycle, RetworkXStrDiGraph
-
-"""
-This module provides function to extract Wikidata ontology from its dump. 
-"""
-
-
-def make_ontology(outfile: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    """Get the list of classes and predicates from Wikidata
-    See the data structure: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Site_Links
-    """
-    # list of class and prop ids
-    step0_file = os.path.join(outfile, "step_0")
-    # list of class and prop full qnode
-    step1_file = os.path.join(outfile, "step_1")
-    # count the frequency of props per class
-    step2_file = os.path.join(outfile, "freq_stats.jl")
-
-    class_qnode_file = os.path.join(outfile, "classes.jl")
-    prop_qnode_file = os.path.join(outfile, "properties.jl")
-
-    if not does_result_dir_exist(step0_file):
-
-        def p_0_get_property_and_class(qnode: QNode):
-            if qnode.type == "property":
-                qnode_ids = [
-                    stmt.value.as_entity_id() for stmt in qnode.props.get("P1647", [])
-                ]
-                qnode_ids.append(qnode.id)
-                return qnode_ids
-
-            if "P279" in qnode.props:
-                # this qnode is a class, but we want to get the parent classes as well.
-                qnode_ids = [
-                    stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])
-                ]
-                qnode_ids.append(qnode.id)
-                return qnode_ids
-
-            return [stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])]
-
-        qnodes_en().flatMap(p_0_get_property_and_class).distinct().coalesce(
-            256
-        ).saveAsTextFile(
-            step0_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    if not does_result_dir_exist(step1_file):
-        sc = get_spark_context()
-
-        qnode_ids = sc.broadcast(set(sc.textFile(step0_file).collect()))
-        qnodes_en().filter(lambda qnode: qnode.id in qnode_ids.value).map(
-            QNode.serialize
-        ).saveAsTextFile(
-            step1_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    item_rdd = lambda: get_spark_context().textFile(step1_file).map(QNode.deserialize)
-
-    if not os.path.exists(class_qnode_file):
-
-        def p_1_extract_class(qnode: QNode):
-            if qnode.type != "item":
-                return None
-
-            return WDClass(
-                id=qnode.id,
-                label=qnode.label,
-                description=qnode.description,
-                datatype=qnode.datatype,
-                aliases=qnode.aliases,
-                parents=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])}
-                ),
-                properties=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1963", [])}
-                ),
-                different_froms=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1889", [])}
-                ),
-                equivalent_classes=sorted(
-                    {stmt.value.as_string() for stmt in qnode.props.get("P1709", [])}
-                ),
-                parents_closure=set(),
-            )
-
-        class_rdd = (
-            item_rdd()
-            .map(p_1_extract_class)
-            .filter(lambda x: x is not None)
-            .map(WDClass.serialize)
-        )
-        saveAsSingleTextFile(class_rdd, class_qnode_file)
-
-    if not os.path.exists(prop_qnode_file):
-
-        def p_2_extract_property(qnode: QNode):
-            if qnode.type != "property":
-                return None
-            return WDProperty(
-                id=qnode.id,
-                label=qnode.label,
-                description=qnode.description,
-                datatype=qnode.datatype,
-                aliases=qnode.aliases,
-                parents=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1647", [])}
-                ),
-                see_also=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1659", [])}
-                ),
-                equivalent_properties=sorted(
-                    {stmt.value.as_string() for stmt in qnode.props.get("P1628", [])}
-                ),
-                subjects=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1629", [])}
-                ),
-                inverse_properties=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1696", [])}
-                ),
-                instanceof=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])}
-                ),
-            )
-
-        prop_rdd = (
-            item_rdd()
-            .map(p_2_extract_property)
-            .filter(lambda x: x is not None)
-            .map(WDProperty.serialize)
-        )
-        saveAsSingleTextFile(prop_rdd, prop_qnode_file)
-
-    if not os.path.exists(step2_file):
-
-        def p_3_count_props(qnode: QNode):
-            if qnode.type == "property":
-                return []
-            class_ids = []
-            for stmt in qnode.props.get("P31", []):
-                class_ids.append(stmt.value.as_entity_id())
-
-            triples = []
-            for p in qnode.props:
-                if p == "P31":
-                    continue
-                for cid in class_ids:
-                    triples.append(((cid, p), 1))
-            for cid in class_ids:
-                triples.append(((cid, "_ccount"), 1))
-            return triples
-
-        sp_count = qnodes_en().flatMap(p_3_count_props).reduceByKey(add).collect()
-        class_freq = {}
-        for (s, p), c in sp_count:
-            if s not in class_freq:
-                class_freq[s] = {"props": {}, "count": 0, "class": s}
-            if p == "_ccount":
-                class_freq[s]["count"] = c
-            else:
-                class_freq[s]["props"][p] = c
-
-        serialize_jl(sorted(class_freq.values(), key=itemgetter("class")), step2_file)
-
-
-def make_superclass_closure(outdir: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    def get_all_parents(index: Dict[str, dict], id: str, parents: Set[str]):
-        for pid in index[id]["parents"]:
-            if pid not in parents:
-                if pid not in index:
-                    continue
-                parents.add(pid)
-                get_all_parents(index, pid, parents)
-
-    superclasses_closure_file = os.path.join(outdir, "superclasses_closure.jl")
-    superproperties_closure_file = os.path.join(outdir, "superproperties_closure.jl")
-
-    if not os.path.exists(superclasses_closure_file):
-        logger.info("Make superclass closure...")
-        # need to make sure that they are in the same order
-        classes = deserialize_jl(os.path.join(outdir, "classes.jl"))
-        classes_order = [c["id"] for c in classes]
-        classes = {c["id"]: c for c in classes}
-
-        class_closure = {}
-        for c in tqdm(classes.values()):
-            parents = set()
-            get_all_parents(classes, c["id"], parents)
-            class_closure[c["id"]] = list(parents)
-
-        class_closure = [(cid, class_closure[cid]) for cid in classes_order]
-        serialize_jl(class_closure, superclasses_closure_file)
-
-    if not os.path.exists(superproperties_closure_file):
-        logger.info("Make superproperties closure...")
-        predicates = deserialize_jl(os.path.join(outdir, "properties.jl"))
-        predicates_order = [p["id"] for p in predicates]
-        predicates = {p["id"]: p for p in predicates}
-
-        predicate_closure = {}
-        for r in tqdm(predicates.values()):
-            parents = set()
-            get_all_parents(predicates, r["id"], parents)
-            predicate_closure[r["id"]] = list(parents)
-
-        predicate_closure = [(pid, predicate_closure[pid]) for pid in predicates_order]
-        serialize_jl(predicate_closure, superproperties_closure_file)
-
-
-def examine_ontology_property(indir: str = os.path.join(WIKIDATA_DIR, "ontology")):
-    def print_cycle(edges: list[BaseEdge[str, str]]):
-        print("\t>", " -> ".join([e.source for e in edges]) + " -> " + edges[-1].target)
-
-    def find_cycles(g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]]):
-        nodes = g.nodes()
-        for node in nodes:
-            cycle = digraph_find_cycle(g, node.id)
-            if len(cycle) == 0:
-                continue
-
-            while True:
-                print_cycle(cycle)
-                e = cycle[0]
-                g.remove_edges_between_nodes(e.source, e.target)
-                cycle = digraph_find_cycle(g, cycle[-1].target)
-                if len(cycle) == 0:
-                    break
-
-    # classes = WDClass.from_file(indir)
-    # g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]] = RetworkXStrDiGraph(
-    #     multigraph=False
-    # )
-    # for c in tqdm(classes.values()):
-    #     for pcid in c.parents:
-    #         g.add_edge(BaseEdge(-1, c.id, pcid, ""))
-
-    # print("Detecting cycles in the ontology classes...")
-    # find_cycles(g)
-
-    predicates = WDProperty.from_file(indir)
-
-    g: RetworkXStrDiGraph[str, BaseNode, BaseEdge[str, str]] = RetworkXStrDiGraph(
-        multigraph=False
-    )
-    for p in tqdm(predicates.values()):
-        for ppid in p.parents:
-            g.add_edge(BaseEdge(-1, p.id, ppid, ""))
-
-    print("Detecting cycles in the ontology properties...")
-    find_cycles(g)
-
-
-def save_wdprops(indir: Union[str, Path], db):
-    if indir == "":
-        indir = os.path.join(WIKIDATA_DIR, "ontology")
-
-    wdprops = WDProperty.from_file(indir, load_parent_closure=True)
-    wb = rocksdb.WriteBatch()
-    for id, item in tqdm(wdprops.items(), total=len(wdprops)):
-        wb.put(id.encode(), item.serialize())
-    db.write(wb)
-
-
-def save_wdclasses(indir: Union[str, Path], db):
-    if indir == "":
-        indir = os.path.join(WIKIDATA_DIR, "ontology")
-
-    wb = rocksdb.WriteBatch()
-    for idx, item in tqdm(
-        enumerate(WDClass.iter_file(indir, load_parent_closure=True)),
-        desc="Write WDClass",
-    ):
-        wb.put(item.id.encode(), item.serialize())
-        if idx % 1000 == 0:
-            db.write(wb)
-            wb = rocksdb.WriteBatch()
-    db.write(wb)
-
-
-if __name__ == "__main__":
-    examine_ontology_property()
diff --git a/kgdata/wikidata/deprecated/rdd_datasets.py b/kgdata/wikidata/deprecated/rdd_datasets.py
deleted file mode 100644
index e5b392d..0000000
--- a/kgdata/wikidata/deprecated/rdd_datasets.py
+++ /dev/null
@@ -1,1018 +0,0 @@
-import copy, itertools, numpy as np, fastnumbers, math, shutil
-from functools import reduce
-import os, orjson, functools, warnings
-import pickle, codecs
-from operator import add, itemgetter, attrgetter
-from collections import Counter, defaultdict
-from loguru import logger
-from kgdata.config import WIKIDATA_DIR
-from kgdata.spark import (
-    get_spark_context,
-    ensure_unique_records,
-    left_outer_join,
-    does_result_dir_exist,
-    cache_rdd,
-)
-from kgdata.wikipedia.prelude import get_title_from_url, title2groups
-from kgdata.wikidata.deprecated.models.qnode import QNode
-from kgdata.wikidata.models import WDProperty, WDClass
-
-
-def qnodes(indir: str = os.path.join(WIKIDATA_DIR, "step_1")):
-    # get qnodes from the dumps prepared by `s00_prep_data.py`. Return the data in the raw format as the qnode class
-    # stores only data for a specific language!
-    # see the data structure: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Site_Links
-    sc = get_spark_context()
-    return sc.textFile(os.path.join(indir, "*.gz")).map(orjson.loads)
-
-
-def qnodes_en(
-    qnodes_rdd=None,
-    outfile: str = os.path.join(WIKIDATA_DIR, "step_2/qnodes_en"),
-    raw: bool = False,
-):
-    """Get all QNodes in Wikidata (only english)"""
-    sc = get_spark_context()
-    if not does_result_dir_exist(outfile):
-        qnodes_rdd = qnodes_rdd or qnodes()
-        qnodes_rdd.map(functools.partial(QNode.from_wikidump, lang="en")).map(
-            QNode.serialize
-        ).saveAsTextFile(
-            outfile, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-        ensure_unique_records(
-            sc.textFile(outfile).map(QNode.deserialize), keyfn=attrgetter("id")
-        )
-
-    if raw:
-        return sc.textFile(outfile).map(orjson.loads)
-    return sc.textFile(outfile).map(QNode.deserialize)
-
-
-def qnodes_identifier(
-    qnodes_rdd=None, outfile: str = os.path.join(WIKIDATA_DIR, "step_2/identifiers.txt")
-):
-    """Make a list of identifiers so that we can quickly identify if a node is missing or not"""
-    if not os.path.exists(outfile):
-        sc = get_spark_context()
-        qnodes_rdd = qnodes_rdd or qnodes_en()
-        ids_rdd = qnodes_rdd.map(lambda x: x.id).distinct()
-        ids_rdd.coalesce(1).saveAsTextFile(outfile + ".tmp")
-        assert os.path.join(outfile + ".tmp", "_SUCCESS")
-        os.rename(os.path.join(outfile + ".tmp", "part-00000"), outfile)
-        shutil.rmtree(outfile + ".tmp")
-
-
-def wikidata_wikipedia_links(
-    qnodes_rdd=None, outfile=os.path.join(WIKIDATA_DIR, "step_2/wiki_links")
-):
-    """Get alignments between wikidata qnode and wiki articles.
-
-    Parameters
-    ----------
-    qnodes_rdd : RDD, optional
-        RDD of all qnodes in wikidata, by default None
-    outfile : str, optional
-        output file, by default "/workspace/sm-dev/data/wikidata/step_2_links"
-
-    Returns
-    -------
-    RDD
-        RDD of { "id": <qnode>, "sitelinks": <sitelinks of qnode> }
-    """
-    sc = get_spark_context()
-
-    if not does_result_dir_exist(outfile):
-        # see the data structure: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Site_Links
-        def p_0_discard_property(x):
-            # only two types: property and item
-            return x["type"] == "item"
-
-        def p_1_keep_useful_props(x):
-            return {"id": x["id"], "sitelinks": x["sitelinks"]}
-
-        rdd = qnodes_rdd or qnodes()
-        rdd.filter(p_0_discard_property).map(p_1_keep_useful_props).map(
-            orjson.dumps
-        ).saveAsTextFile(
-            outfile, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    return sc.textFile(outfile).map(orjson.loads)
-
-
-def wiki_article_to_qnode(
-    qnodes_rdd=None, outdir=os.path.join(WIKIDATA_DIR, "step_2/"), lang="en"
-):
-    site = f"{lang}wiki"
-    outfile = os.path.join(outdir, f"{site}_links")
-
-    if not does_result_dir_exist(outfile):
-
-        def extract_links(qnode: QNode):
-            if site not in qnode.sitelinks:
-                return None
-            title = qnode.sitelinks[site].title
-            assert title is not None and isinstance(title, str) and len(title) > 0
-            return title, qnode.id
-
-        qnodes_rdd = qnodes_rdd or qnodes_en()
-        # records = qnodes_rdd.filter(lambda x: 'enwiki' in x.sitelinks).take(5)
-        # size = qnodes_rdd.map(extract_links).filter(lambda x: x is not None).count()
-        # print(size)
-        # return
-        qnodes_rdd.map(extract_links).filter(lambda x: x is not None).map(
-            orjson.dumps
-        ).saveAsTextFile(
-            outfile, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-
-def wiki_reltables_instances(
-    qnodes_rdd=None,
-    outfile: str = os.path.join(WIKIDATA_DIR, "step_2/wiki_reltables_instances"),
-):
-    """Get a subset of qnodes that have links with the wikipedia relational tables"""
-    warnings.warn("wiki_reltables_instances is deprecated", DeprecationWarning)
-
-    step0_file = os.path.join(outfile, "step_0")
-    step1_file = os.path.join(outfile, "step_1")
-    step2_file = os.path.join(outfile, "step_2")
-
-    sc = get_spark_context()
-
-    if not does_result_dir_exist(step0_file):
-        # get qnode ids that connects with links in rel tables
-        from kgdata.dbpedia.table_extraction import (
-            wikipedia_links_in_relational_tables_en,
-        )
-
-        links_rdd = wikipedia_links_in_relational_tables_en()
-
-        titles = set(links_rdd.map(itemgetter("title")).collect())
-        titles = sc.broadcast(titles)
-
-        def p_2_filter_no_wiki_link(item):
-            # see the data structure: https://www.mediawiki.org/wiki/Wikibase/DataModel/JSON#Site_Links
-            for x in item["sitelinks"].values():
-                if x["title"] in titles.value:
-                    return True
-            return False
-
-        rdd = wikidata_wikipedia_links()
-        rdd.filter(p_2_filter_no_wiki_link).map(itemgetter("id")).coalesce(
-            64
-        ).saveAsTextFile(
-            step0_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    if not does_result_dir_exist(step1_file):
-        # create a list of target qnodes that qnodes in rel tables links to
-        qnode_ids = sc.broadcast(set(sc.textFile(step0_file).collect()))
-
-        def p_3_filter_by_qid(item):
-            return item["id"] in qnode_ids.value
-
-        def p_4_extract_link_qid(qnode):
-            qnode = QNode.from_wikidump(qnode, lang="en")
-            targets = [qnode.id]
-            for stmts in qnode.props.values():
-                for stmt in stmts:
-                    if stmt.value.is_qnode():
-                        targets.append(stmt.value.as_entity_id())
-
-            return targets
-
-        qnodes_rdd = qnodes_rdd or qnodes()
-        qnodes_rdd.filter(p_3_filter_by_qid).flatMap(
-            p_4_extract_link_qid
-        ).distinct().saveAsTextFile(
-            step1_file, compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
-        )
-
-    step20_file = os.path.join(step2_file, "step_0")
-    step21_file = os.path.join(step2_file, "step_1")
-    step22_file = os.path.join(step2_file, "step_2")
-
-    if not os.path.exists(os.path.join(step2_file, "_SUCCESS")):
-        # create a final list that join those qnodes together
-        if not does_result_dir_exist(step20_file):
-            qnode_ids = sc.broadcast(set(sc.textFile(step1_file).collect()))
-
-            def p_5_filter_by_qid(item):
-                return item["id"] in qnode_ids.value
-
-            qnodes_rdd = qnodes_rdd or qnodes()
-            qnodes_rdd.filter(p_5_filter_by_qid).map(
-                functools.partial(QNode.from_wikidump, lang="en")
-            ).map(QNode.serialize).saveAsTextFile(
-                step20_file,
-                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-            )
-
-        if not does_result_dir_exist(step21_file):
-            # create rdd1 of records in rel tables
-            original_qnode_ids = sc.broadcast(set(sc.textFile(step0_file).collect()))
-
-            def p_6_filter_by_qid(item):
-                return item.id in original_qnode_ids.value
-
-            sc.textFile(step20_file).map(QNode.deserialize).filter(
-                p_6_filter_by_qid
-            ).map(QNode.serialize).saveAsTextFile(
-                step21_file,
-                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-            )
-
-        if not does_result_dir_exist(step22_file):
-            # create rdd2 of records that we want to join
-            def p_7_extract_essential_info(qnode):
-                return {"id": qnode.id, "label": qnode.label}
-
-            def rdd1_fk_fn(qnode: QNode):
-                targets = set()
-                for stmts in qnode.props.values():
-                    for stmt in stmts:
-                        if stmt.value.is_qnode():
-                            targets.add(stmt.value.as_entity_id())
-                return list(targets)
-
-            def rdd1_join_fn(qnode: QNode, targets):
-                targets = dict(targets)
-                for stmts in qnode.props.values():
-                    for stmt in stmts:
-                        if stmt.value.is_qnode():
-                            qnode_id = stmt.value.as_entity_id()
-                            if targets[qnode_id] is not None:
-                                stmt.value.set_qnode_label(targets[qnode_id]["label"])
-                            else:
-                                stmt.value.set_qnode_label(None)
-                return targets
-
-            rdd1 = sc.textFile(step21_file).map(QNode.deserialize)
-            rdd2 = (
-                sc.textFile(step20_file)
-                .map(QNode.deserialize)
-                .map(p_7_extract_essential_info)
-            )
-            rdd1_keyfn = attrgetter("id")
-            rdd2_keyfn = itemgetter("id")
-
-            left_outer_join(
-                rdd1,
-                rdd2,
-                rdd1_keyfn,
-                rdd1_fk_fn,
-                rdd2_keyfn,
-                rdd1_join_fn,
-                QNode.serialize,
-                step22_file,
-                compression=True,
-            )
-
-        with open(os.path.join(step2_file, "_SUCCESS"), "w") as f:
-            f.write("DONE")
-
-    return sc.textFile(step22_file).map(QNode.deserialize)
-
-
-def wikidata_graph_structure(
-    qnodes_rdd=None, outfile: str = os.path.join(WIKIDATA_DIR, "step_2/graph_structure")
-):
-    """Get the RDD data that contains incoming links between nodes and outgoing links between nodes.
-    The graph ignores the instance of class to prevent one node has so many links
-    """
-    qnodes_rdd = qnodes_rdd or qnodes_en()
-    step0_file = os.path.join(outfile, "outedges")
-    step1_file = os.path.join(outfile, "inedges")
-    step2_file = os.path.join(outfile, "data")
-
-    if not does_result_dir_exist(step0_file):
-        logger.info("Extract outgoing links...")
-
-        def extract_outedges(qnode: QNode):
-            outedges = {}
-            for p, stmts in qnode.props.items():
-                if p == "P31":
-                    continue
-
-                for stmt in stmts:
-                    if stmt.value.is_qnode():
-                        if p not in outedges:
-                            outedges[p] = set()
-                        outedges[p].add(stmt.value.as_entity_id())
-
-                    for q, qvals in stmt.qualifiers.items():
-                        edge = f"{p}-{q}"
-                        for qval in qvals:
-                            if qval.is_qnode():
-                                if edge not in outedges:
-                                    outedges[edge] = set()
-                                outedges[edge].add(qval.as_entity_id())
-
-            for edge in outedges:
-                outedges[edge] = list(outedges[edge])
-
-            return {"id": qnode.id, "outedges": outedges}
-
-        qnodes_rdd.map(extract_outedges).map(orjson.dumps).saveAsTextFile(
-            step0_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    if not does_result_dir_exist(step1_file):
-        logger.info("Extract incoming links...")
-        sc = get_spark_context()
-        rdd = sc.textFile(step0_file).map(orjson.loads)
-
-        def inverse(odict):
-            inverse_qnodes = {}
-            for p, qnodes in odict["outedges"].items():
-                for qnode in qnodes:
-                    if qnode not in inverse_qnodes:
-                        inverse_qnodes[qnode] = {"id": qnode, "inedges": {}}
-                    if p not in inverse_qnodes[qnode]["inedges"]:
-                        inverse_qnodes[qnode]["inedges"][p] = set()
-                    inverse_qnodes[qnode]["inedges"][p].add(odict["id"])
-            return [(n["id"], n) for n in inverse_qnodes.values()]
-
-        def combine(a, b):
-            for p, pvals in b["inedges"].items():
-                if p in a["inedges"]:
-                    a["inedges"][p].update(pvals)
-                else:
-                    a["inedges"][p] = pvals
-            return a
-
-        rdd.flatMap(inverse).reduceByKey(combine).map(itemgetter(1)).map(
-            lambda x: orjson.dumps(
-                x, option=orjson.OPT_SERIALIZE_DATACLASS, default=list
-            )
-        ).saveAsTextFile(
-            step1_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    if not does_result_dir_exist(step2_file):
-        sc = get_spark_context()
-        outedges = sc.textFile(step0_file).map(orjson.loads).map(lambda x: (x["id"], x))
-        inedges = sc.textFile(step1_file).map(orjson.loads).map(lambda x: (x["id"], x))
-
-        def merge_nodes(args):
-            key, (inedge, outedge) = args
-            return {
-                "id": key,
-                "inedges": inedge["inedges"] if inedge is not None else {},
-                "outedges": outedge["outedges"] if outedge is not None else {},
-            }
-
-        inedges.fullOuterJoin(outedges).map(merge_nodes).map(
-            orjson.dumps
-        ).saveAsTextFile(
-            step2_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    return get_spark_context().textFile(step2_file).map(orjson.loads)
-
-
-def wikidata_schema(
-    qnodes_rdd=None, outfile: str = os.path.join(WIKIDATA_DIR, "step_2/schema")
-):
-    """Create a schema of Wikidata.
-
-    1. `node_classes` contains a list of qnodes, each with its classes, its outgoing qnodes (not props) and their classes.
-    2. `node_degree` contains a list of qnodes and their degree
-    3. `node_schema` contains schema of each qnode
-    4. `class_schema` schema of class computed from `node_schema`
-    """
-    qnodes_rdd = qnodes_rdd or qnodes_en()
-
-    node_class_file = os.path.join(outfile, "node_classes")
-    node_degree_file = os.path.join(outfile, "node_degree")
-    node_schema_file = os.path.join(outfile, "node_schema")
-    class_schema_file = os.path.join(outfile, "class_schema")
-
-    def extract_classes(qnode: QNode):
-        out_qnodes = set()
-        for p, stmts in qnode.props.items():
-            for stmt in stmts:
-                if stmt.value.is_qnode():
-                    out_qnodes.add(stmt.value.as_entity_id())
-
-                for q, qvals in stmt.qualifiers.items():
-                    for qval in qvals:
-                        if qval.is_qnode():
-                            out_qnodes.add(qval.as_entity_id())
-
-        return {
-            "id": qnode.id,
-            "classes": list(
-                {stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])}
-            ),
-            "out_qnodes": list(out_qnodes),
-        }
-
-    if not does_result_dir_exist(node_class_file):
-
-        def join(x, ys):
-            ys = dict(ys)
-            return {
-                "id": x["id"],
-                "classes": list(x["classes"]),
-                "out_qnodes": {
-                    k2: list(ys[k2]["classes"])
-                    for k2 in x["out_qnodes"]
-                    if ys[k2] is not None
-                },
-            }
-
-        node_class_rdd = qnodes_rdd.map(extract_classes).persist()
-        left_outer_join(
-            rdd1=node_class_rdd,
-            rdd2=node_class_rdd.map(lambda x: {"id": x["id"], "classes": x["classes"]}),
-            rdd1_keyfn=itemgetter("id"),
-            rdd1_fk_fn=itemgetter("out_qnodes"),
-            rdd2_keyfn=itemgetter("id"),
-            join_fn=join,
-            ser_fn=orjson.dumps,
-            outfile=node_class_file,
-        )
-
-    if not does_result_dir_exist(node_degree_file):
-
-        def extract_freq(x):
-            count = [(x["id"], len(x["out_qnodes"]))]
-            for xi in x["out_qnodes"]:
-                count.append((xi, 1))
-            return count
-
-        qnodes_rdd.map(extract_classes).flatMap(extract_freq).foldByKey(0, add).map(
-            orjson.dumps
-        ).saveAsTextFile(
-            node_degree_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    if not does_result_dir_exist(node_schema_file):
-
-        def extract_qnode_schema(qnode: QNode):
-            schema = {}
-            for prop, stmts in qnode.props.items():
-                if prop == "P31":
-                    continue
-                # storing the values of objects & literals for calculating the probability of class & datatype
-                prop_qnodes = defaultdict(int)
-                prop_literal_dtypes = defaultdict(int)
-
-                qualifier_qnodes = defaultdict(lambda: defaultdict(int))
-                qualifier_dtypes = defaultdict(lambda: defaultdict(int))
-                n_qualifiers = defaultdict(int)
-                for stmt in stmts:
-                    if stmt.value.is_qnode():
-                        prop_qnodes[stmt.value.as_entity_id()] += 1
-                    else:
-                        prop_literal_dtypes[stmt.value.type] += 1
-
-                    for qual, qual_values in stmt.qualifiers.items():
-                        for qual_val in qual_values:
-                            if qual_val.is_qnode():
-                                qualifier_qnodes[qual][qual_val.as_entity_id()] += 1
-                            else:
-                                qualifier_dtypes[qual][qual_val.type] += 1
-                        n_qualifiers[qual] += 1
-
-                schema[prop] = {
-                    "n_stmts": len(stmts),
-                    "qnodes": dict(prop_qnodes),
-                    "literal_types": dict(prop_literal_dtypes),
-                    "qualifiers": {
-                        qual: {
-                            "qnodes": dict(qualifier_qnodes[qual]),
-                            "literal_types": dict(qualifier_dtypes[qual]),
-                            "n_qualifiers": n_qualifiers[qual],
-                        }
-                        for qual in qualifier_qnodes
-                    },
-                }
-
-            return qnode.id, schema
-
-        sc = get_spark_context()
-
-        # building an rdd that contain class information of outgoing qnodes and a weight that try to represent the centrality of a node
-        node_class_rdd = (
-            sc.textFile(node_class_file).map(orjson.loads).map(lambda x: (x["id"], x))
-        )
-        node_degree_rdd = sc.textFile(node_degree_file).map(orjson.loads)
-
-        def merge(args):
-            k, (gclass, gcen) = args
-            gclass["degree"] = gcen
-            return gclass["id"], gclass
-
-        vertice_rdd = node_class_rdd.leftOuterJoin(node_degree_rdd).map(merge)
-
-        # merge the qnode schema with the rdd above to complete the schema with qnode classes
-        def merge_class_info(args):
-            qnode_id, (qnode_schema, qnode_linked_info) = args
-            for prop, prop_schema in qnode_schema.items():
-                class_freqs = defaultdict(int)
-                for qid, qnode_count in prop_schema.pop("qnodes").items():
-                    for clsid in qnode_linked_info["out_qnodes"].get(qid, []):
-                        class_freqs[clsid] += qnode_count
-                prop_schema["classes"] = dict(class_freqs)
-
-                for qual, qual_schema in prop_schema["qualifiers"].items():
-                    class_freqs = defaultdict(int)
-                    for qid, qnode_count in qual_schema.pop("qnodes").items():
-                        for clsid in qnode_linked_info["out_qnodes"].get(qid, []):
-                            class_freqs[clsid] += qnode_count
-                    qual_schema["classes"] = dict(class_freqs)
-
-            return {
-                "id": qnode_id,
-                "classes": qnode_linked_info["classes"],
-                "schema": qnode_schema,
-                "degree": qnode_linked_info["degree"],
-            }
-
-        qnodes_rdd.map(extract_qnode_schema).leftOuterJoin(vertice_rdd).map(
-            merge_class_info
-        ).map(orjson.dumps).coalesce(2048).saveAsTextFile(
-            node_schema_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    if not does_result_dir_exist(class_schema_file):
-
-        def cls2instance(x):
-            instances = []
-            for clsid in x["classes"]:
-                cschema = {"id": clsid, "n_instances": 1, "props": {}}
-                for prop, prop_schema in x["schema"].items():
-                    prop_schema = copy.deepcopy(prop_schema)
-                    prop_schema["n_instances"] = 1
-                    example = {"objects": {}, "literals": {}, "qualifiers": {}}
-                    for c in prop_schema["classes"]:
-                        example["objects"][c] = [(x["id"], x["degree"])]
-                    for c in prop_schema["literal_types"]:
-                        example["literals"][c] = [(x["id"], x["degree"])]
-                    for qual, qual_schema in prop_schema["qualifiers"].items():
-                        example["qualifiers"][qual] = {"objects": {}, "literals": {}}
-                        for c in qual_schema["classes"]:
-                            example["qualifiers"][qual]["objects"][c] = [
-                                (x["id"], x["degree"])
-                            ]
-                        for c in qual_schema["literal_types"]:
-                            example["qualifiers"][qual]["literals"][c] = [
-                                (x["id"], x["degree"])
-                            ]
-                    prop_schema["examples"] = example
-                    assert len(example["objects"]) == len(prop_schema["classes"])
-                    cschema["props"][prop] = prop_schema
-
-                instances.append((clsid, cschema))
-            return instances
-
-        def merge_schema(schema1, schema2):
-            if isinstance(schema1, tuple):
-                return schema1
-            if isinstance(schema2, tuple):
-                return schema2
-
-            assert schema1["id"] == schema2["id"]
-            top_k_examples = 5
-
-            schema1["n_instances"] += schema2["n_instances"]
-            for prop, prop_schema2 in schema2["props"].items():
-                if prop not in schema1["props"]:
-                    schema1["props"][prop] = prop_schema2
-                    continue
-
-                # merge exist prop schema
-                prop_schema1 = schema1["props"][prop]
-                prop_schema1["n_instances"] += prop_schema2["n_instances"]
-                prop_schema1["n_stmts"] += prop_schema2["n_stmts"]
-                for k, v in prop_schema2["classes"].items():
-                    if k not in prop_schema1["classes"]:
-                        prop_schema1["classes"][k] = v
-                        prop_schema1["examples"]["objects"][k] = prop_schema2[
-                            "examples"
-                        ]["objects"][k]
-                    else:
-                        prop_schema1["classes"][k] += v
-                        prop_schema1["examples"]["objects"][k] = sorted(
-                            prop_schema1["examples"]["objects"][k]
-                            + prop_schema2["examples"]["objects"][k],
-                            key=itemgetter(1),
-                            reverse=True,
-                        )[:top_k_examples]
-                for k, v in prop_schema2["literal_types"].items():
-                    if k not in prop_schema1["literal_types"]:
-                        prop_schema1["literal_types"][k] = v
-                        prop_schema1["examples"]["literals"][k] = prop_schema2[
-                            "examples"
-                        ]["literals"][k]
-                    else:
-                        prop_schema1["literal_types"][k] += v
-                        prop_schema1["examples"]["literals"][k] = sorted(
-                            prop_schema1["examples"]["literals"][k]
-                            + prop_schema2["examples"]["literals"][k],
-                            key=itemgetter(1),
-                            reverse=True,
-                        )[:top_k_examples]
-
-                for qual, qual_schema2 in prop_schema2["qualifiers"].items():
-                    if qual not in prop_schema1["qualifiers"]:
-                        prop_schema1["qualifiers"][qual] = prop_schema2["qualifiers"][
-                            qual
-                        ]
-                        prop_schema1["examples"]["qualifiers"][qual] = prop_schema2[
-                            "examples"
-                        ]["qualifiers"][qual]
-                    else:
-                        qual_schema1 = prop_schema1["qualifiers"][qual]
-                        qual_schema1["n_qualifiers"] += qual_schema2["n_qualifiers"]
-                        for k, v in qual_schema2["classes"].items():
-                            if k not in qual_schema1["classes"]:
-                                qual_schema1["classes"][k] = v
-                                prop_schema1["examples"]["qualifiers"][qual]["objects"][
-                                    k
-                                ] = prop_schema2["examples"]["qualifiers"][qual][
-                                    "objects"
-                                ][
-                                    k
-                                ]
-                            else:
-                                qual_schema1["classes"][k] += v
-                                prop_schema1["examples"]["qualifiers"][qual]["objects"][
-                                    k
-                                ] = sorted(
-                                    prop_schema1["examples"]["qualifiers"][qual][
-                                        "objects"
-                                    ][k]
-                                    + prop_schema2["examples"]["qualifiers"][qual][
-                                        "objects"
-                                    ][k],
-                                    key=itemgetter(1),
-                                    reverse=True,
-                                )[
-                                    :top_k_examples
-                                ]
-                        for k, v in qual_schema2["literal_types"].items():
-                            if k not in qual_schema1["literal_types"]:
-                                qual_schema1["literal_types"][k] = v
-                                prop_schema1["examples"]["qualifiers"][qual][
-                                    "literals"
-                                ][k] = prop_schema2["examples"]["qualifiers"][qual][
-                                    "literals"
-                                ][
-                                    k
-                                ]
-                            else:
-                                qual_schema1["literal_types"][k] += v
-                                prop_schema1["examples"]["qualifiers"][qual][
-                                    "literals"
-                                ][k] = sorted(
-                                    prop_schema1["examples"]["qualifiers"][qual][
-                                        "literals"
-                                    ][k]
-                                    + prop_schema2["examples"]["qualifiers"][qual][
-                                        "literals"
-                                    ][k],
-                                    key=itemgetter(1),
-                                    reverse=True,
-                                )[
-                                    :top_k_examples
-                                ]
-
-            return schema1
-
-        # get_spark_context().textFile(node_schema_file) \
-        #     .map(orjson.loads).flatMap(cls2instance) \
-        #     .filter(lambda x: x[0] == 'Q2367225') \
-        #     .map(orjson.dumps) \
-        #     .saveAsTextFile(node_schema_file + "_tmp", compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec")
-        # lst = get_spark_context().textFile(node_schema_file).map(orjson.loads).filter(lambda x: x['id'] == 'Q30').count()
-        # print(lst)
-        # return
-
-        # def test_irregular(schema):
-        #     for p in schema['props'].values():
-        #         for lst in p['examples']['objects'].values():
-        #             if lst[0][0] == 'Q30':
-        #                 return True
-        #     return False
-        # result = get_spark_context().textFile(node_schema_file + "_tmp").map(orjson.loads).map(itemgetter(1)).filter(test_irregular).collect()
-
-        # print(len(result))
-        # x = get_spark_context().textFile(class_schema_file).map(lambda x: x.split("\t", 1)[1]).map(orjson.loads).filter(lambda x: x['id'] == 'Q2367225').collect()
-        # print(x[0]['props']['P118'])
-        # result2 = get_spark_context().textFile(class_schema_file).map(lambda x: x.split("\t", 1)[1]).map(orjson.loads).filter(lambda x: x['id'] == 'Q2367225').filter(test_irregular).collect()
-        # print(len(result2))
-
-        # print("here")
-        # return
-        #     .reduceByKey(merge_schema) \
-        #     .count()
-        # print(result)
-        # # prop_schema1, prop_schema2, schema1, schema2 = result[0][1]
-        # return
-        # import IPython
-        # IPython.embed()
-
-        rdd = (
-            get_spark_context()
-            .textFile(node_schema_file)
-            .map(orjson.loads)
-            .flatMap(cls2instance)
-            .reduceByKey(merge_schema)
-            .map(lambda x: x[0] + "\t" + orjson.dumps(x[1]).decode())
-            .coalesce(512)
-            .saveAsTextFile(
-                class_schema_file,
-                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-            )
-        )
-
-    def extract(line: str):
-        id, data = line.split("\t")
-        return id, orjson.loads(data)
-
-    return get_spark_context().textFile(class_schema_file).map(extract)
-
-
-def wikidata_quantity_prop_stats(
-    qnodes_rdd=None,
-    outfile: str = os.path.join(WIKIDATA_DIR, "step_2/quantity_prop_stats"),
-):
-    qnodes_rdd = qnodes_rdd or qnodes_en()
-
-    raw_quantity_values_file = os.path.join(outfile, "raw_quantity_values")
-    quantity_stats_file = os.path.join(outfile, "quantity_stats")
-
-    if not does_result_dir_exist(raw_quantity_values_file):
-        wd_props = WDProperty.from_file()
-        identifiers = {p.id for p in wd_props.values() if p.datatype == "external-id"}
-        identifiers = get_spark_context().broadcast(identifiers)
-
-        def extract_quantity(val: dict):
-            assert len(val) <= 4, f"So that they only have 4 props: {val.keys()}"
-            return (
-                val["amount"],
-                val.get("unit", None),
-                val.get("upperBound", None),
-                val.get("lowerBound", None),
-            )
-
-        def extract_data_prop(qnode: QNode):
-            raw_dprop_values = []
-            for p, stmts in qnode.props.items():
-                # ignore identifiers
-                if p in identifiers.value:
-                    continue
-                p_vals = []
-                p_qual_values = {}
-                for stmt in stmts:
-                    if stmt.value.is_quantity():
-                        p_vals.append(extract_quantity(stmt.value.value))
-
-                    for q, qvals in stmt.qualifiers.items():
-                        new_qvals = []
-                        for qval in qvals:
-                            if qval.is_quantity():
-                                new_qvals.append(extract_quantity(qval.value))
-                        if len(new_qvals) > 0:
-                            p_qual_values[q] = new_qvals
-                if len(p_vals) > 0 or len(p_qual_values) > 0:
-                    raw_dprop_values.append(
-                        (p, {"id": p, "value": p_vals, "qualifiers": p_qual_values})
-                    )
-            return raw_dprop_values
-
-        qnodes_rdd.flatMap(extract_data_prop).map(orjson.dumps).saveAsTextFile(
-            raw_quantity_values_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-    if not does_result_dir_exist(quantity_stats_file):
-        # max_f32 = np.finfo(np.float32).max
-        max_i36 = np.iinfo(np.int32).max * 16
-
-        def create_stats(pinfo):
-            defaultstats = {
-                "units": set(),
-                "size": 0,
-                "int_size": 0,
-                "min": float("inf"),
-                "max": -float("inf"),
-                "sum": 0,
-                "sum_square": 0,
-                "n_overi36": 0,
-            }
-            value = add_stats(copy.copy(defaultstats), pinfo["value"])
-            qualifiers = {
-                q: add_stats(copy.copy(defaultstats), qvals)
-                for q, qvals in pinfo["qualifiers"].items()
-            }
-            return {"id": pinfo["id"], "value": value, "qualifiers": qualifiers}
-
-        def add_stats(statdict, quan_lst):
-            if len(quan_lst) == 0:
-                return statdict
-            units = list({x[1] for x in quan_lst})
-            olst = [fastnumbers.fast_real(x[0]) for x in quan_lst]
-            assert all(isinstance(x, (int, float)) and not math.isinf(x) for x in olst)
-
-            lst = [x for x in olst if x <= max_i36]
-
-            statdict = copy.copy(statdict)
-            statdict["units"] = statdict["units"].union(units)
-            statdict["size"] += len(olst)
-            statdict["int_size"] += sum([isinstance(x, int) for x in olst])
-            if len(lst) > 0:
-                statdict["min"] = min(statdict["min"], min(lst))
-                statdict["max"] = max(statdict["max"], max(lst))
-            statdict["sum"] += sum(lst)
-            statdict["sum_square"] += sum([x**2 for x in lst])
-            statdict["n_overi36"] += len(olst) - len(lst)
-            return statdict
-
-        def merge_stats(odict1, odict2):
-            statdict = copy.copy(odict1)
-            statdict["units"] = statdict["units"].union(odict2["units"])
-            statdict["size"] += odict2["size"]
-            statdict["int_size"] += odict2["int_size"]
-            statdict["min"] = min(statdict["min"], odict2["min"])
-            statdict["max"] = max(statdict["max"], odict2["max"])
-            statdict["sum"] += odict2["sum"]
-            statdict["sum_square"] += odict2["sum_square"]
-            statdict["n_overi36"] += odict2["n_overi36"]
-            return statdict
-
-        def compute_statistic(p1info, p2info):
-            assert p1info["id"] == p2info["id"]
-            pinfo = {"id": p1info["id"]}
-            pinfo["value"] = merge_stats(p1info["value"], p2info["value"])
-            pinfo["qualifiers"] = copy.copy(p1info["qualifiers"])
-            for q, qinfo in p2info["qualifiers"].items():
-                if q not in pinfo["qualifiers"]:
-                    pinfo["qualifiers"][q] = copy.copy(qinfo)
-                else:
-                    pinfo["qualifiers"][q] = merge_stats(pinfo["qualifiers"][q], qinfo)
-            return pinfo
-
-        def compute_statistic_postprocess(pinfo):
-            lst = [pinfo["value"]] + list(pinfo["qualifiers"].values())
-            for item in lst:
-                if item["size"] == 0:
-                    item["units"] = set()
-                    item["min"] = None
-                    item["max"] = None
-                    item["mean"] = 0
-                    item["std"] = 0
-                    item.pop("sum")
-                    item.pop("sum_square")
-                else:
-                    item["mean"] = (item["n_overi36"] * max_i36 + item["sum"]) / item[
-                        "size"
-                    ]
-                    nsum_square = item["n_overi36"] * (max_i36**2) + item.pop(
-                        "sum_square"
-                    )
-                    nsum = item["n_overi36"] * max_i36 + item.pop("sum")
-                    item["std"] = math.sqrt(
-                        nsum_square / item["size"] - (nsum / item["size"]) ** 2
-                    )
-            return pinfo
-
-        get_spark_context().textFile(raw_quantity_values_file).map(orjson.loads).map(
-            lambda x: (x[0], create_stats(x[1]))
-        ).reduceByKey(compute_statistic).map(itemgetter(1)).map(
-            compute_statistic_postprocess
-        ).map(
-            lambda x: orjson.dumps(
-                x, option=orjson.OPT_SERIALIZE_DATACLASS, default=list
-            )
-        ).coalesce(
-            8
-        ).saveAsTextFile(
-            quantity_stats_file,
-            compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
-        )
-
-
-if __name__ == "__main__":
-    # wikidata_quantity_prop_stats()
-    qnodes_identifier()
-    exit()
-
-    # rdd = qnodes()
-    # values = rdd.filter(lambda x: x['id'] == "L252247-F2").take(1)
-    # print(orjson.dumps(values[0]).decode())
-    wiki_article_to_qnode(lang="en")
-    # rdd = qnodes_en()
-    # # print(rdd.count())
-    # # values = rdd.take(1)
-    # values = rdd.filter(lambda x: x.id == "Q3034572").take(1)
-    # print(values)
-    exit(0)
-
-    # rdd = qnodes()
-    # values = rdd.filter(lambda x: x['id'] == "Q2188143").take(1)
-    # print(values)
-
-    import json
-
-    # wdclasses = WDClass.from_file()
-    # wdprops = WDProperty.from_file()
-    #
-    # # wikidata_graph_structure(rdd)
-    # rdd = wikidata_schema()
-    #     schema = rdd.filter(lambda x: x[1]['id'] == 'Q12140').collect()[0][1]
-
-    # clbl = lambda x: f"{wdclasses[x].label} ({x})"
-
-    # info = {
-    #     "id": schema['id'],
-    #     "label": wdclasses[schema['id']].label,
-    #     "n_instances": schema['n_instances'],
-    #     "props": {}
-    # }
-    # for prop, prop_schema in sorted(schema['props'].items(), key=lambda x: x[1]['n_stmts'], reverse=True):
-    #     lbl = f"{wdprops[prop].label} ({prop})"
-    #     info['props'][lbl] = prop_schema
-    #     for c in prop_schema['classes']
-
-    # with open("/nas/home/binhvu/workspace/sm-dev/sm-unknown/test.json", "w") as f:
-    #     f.write(json.dumps(info, indent=4))
-    #     exit(0)
-    # from sm_unk.prelude import M
-    # M.serialize_json(rdd.filter(lambda x: x['id'] == 'Q5').take(1)[0], WIKIDATA_DIR + "/test.json", indent=4)
-
-    # def norm(args):
-    #     qnode_id, schema = args
-    #     for prop, prop_schema in schema['props'].items():
-    #         norm = {
-    #             'prob': prop_schema['n_instances'] / schema['n_instances'],
-    #             'avg_n_stmts': prop_schema['n_stmts'] / prop_schema['n_instances'],
-    #             # somehow we have statements that don't have any values
-    #             'prob_value_object': prop_schema['f_so_ip'] / max(prop_schema['f_so_ip'] + prop_schema['f_sd_ip'], 1),
-    #             'prob_value_literal': prop_schema['f_sd_ip'] / max(prop_schema['f_so_ip'] + prop_schema['f_sd_ip'], 1),
-    #             'prob_class_value': {},
-    #             'prob_datatype_value': {},
-    #             'qualifiers': {},
-    #             'examples': prop_schema['examples']
-    #         }
-
-    #         for k, v in prop_schema['f_s_ipc'].items():
-    #             norm['prob_class_value'][k] = v / prop_schema['f_so_ip']
-    #         for k, v in prop_schema['f_s_ipd'].items():
-    #             norm['prob_datatype_value'][k] = v / prop_schema['f_sd_ip']
-    #         for qual, qual_schema in prop_schema['qualifiers'].items():
-    #             norm['qualifiers'][qual] = {
-    #                 'avg_n_qualifiers': qual_schema['g_q_ipq'] / qual_schema['g_s_ipq'],
-    #                 'prob': qual_schema['g_s_ipq'] / (prop_schema['f_so_ip'] + prop_schema['f_sd_ip']),
-    #                 'prob_value_object': qual_schema['g_qo_ipq'] / (qual_schema['g_qo_ipq'] + qual_schema['g_qd_ipq']),
-    #                 'prob_value_literal': qual_schema['g_qd_ipq'] / (qual_schema['g_qo_ipq'] + qual_schema['g_qd_ipq']),
-    #                 'prob_class_value': {},
-    #                 'prob_datatype_value': {},
-    #             }
-
-    #             for k, v in qual_schema['g_q_ipqc'].items():
-    #                 norm['qualifiers'][qual]['prob_class_value'][k] = v / qual_schema['g_qo_ipq']
-    #             for k, v in qual_schema['g_q_ipqd'].items():
-    #                 norm['qualifiers'][qual]['prob_datatype_value'][k] = v / qual_schema['g_qd_ipq']
-
-    #         schema['props'][prop] = norm
-    #     return schema
-
-    # resp1 = rdd2.map(itemgetter(1)).filter(lambda x: x['id'] == 'Q15991303').take(1)[0]
-    # resp2 = rdd.filter(lambda x: x['id'] == 'Q15991303').take(1)[0]
-
-    # M.serialize_json(resp1, WIKIDATA_DIR + "/test.json", indent=4)
-    # M.serialize_json(norm((1, resp1)), WIKIDATA_DIR + "/test1.json", indent=4)
-    # M.serialize_json(resp2, WIKIDATA_DIR + "/test2.json", indent=4)
-
-    # print(rdd.take(1))
-
-    # def test(qnode):
-    #     for p, vals in qnode.props.items():
-    #         if p in {'P3036'}:
-    #             return True
-    #     return False
-
-    # result = rdd.filter(test).map(lambda x: x.id).distinct().take(1000)
-    # print(result)
-
-    # rdd = wikidata_wikipedia_links()
-    # print(rdd.filter(lambda x: x['id'] == 'Q6323219').collect())
-    # rdd = wiki_reltables_instances()
-    # print(rdd.filter(lambda x: x['id'] == 'Q885507').take(1))
-
-    # rdd = wikidata_en_properties()
-    # rdd = wikidata_en_ontology()
-    # print(rdd.count())
-    # input(">>> Press any key to finish")
diff --git a/kgdata/wikidata/deprecated/wdclass.py b/kgdata/wikidata/deprecated/wdclass.py
deleted file mode 100644
index c448ded..0000000
--- a/kgdata/wikidata/deprecated/wdclass.py
+++ /dev/null
@@ -1,151 +0,0 @@
-import os
-from dataclasses import dataclass, field, asdict
-from pathlib import Path
-from pickle import load
-from typing import List, Dict, Set, Union, Generator
-from loguru import logger
-
-import orjson
-
-from kgdata.config import WIKIDATA_DIR
-from sm.misc import deserialize_json, deserialize_jl
-from kgdata.wikidata.deprecated.models.qnode import (
-    QNode,
-    MultiLingualString,
-    MultiLingualStringList,
-)
-
-
-@dataclass
-class WDClass:
-    __slots__ = (
-        "id",
-        "label",
-        "description",
-        "datatype",
-        "aliases",
-        "parents",
-        "properties",
-        "different_froms",
-        "equivalent_classes",
-        "parents_closure",
-    )
-    id: str
-    label: MultiLingualString
-    description: MultiLingualString
-    datatype: str
-    aliases: MultiLingualStringList
-    parents: List[str]
-    properties: List[str]
-    different_froms: List[str]
-    equivalent_classes: List[str]
-    # not include itself
-    parents_closure: Set[str]
-
-    @staticmethod
-    def from_file(
-        indir: Union[str, Path] = os.path.join(WIKIDATA_DIR, "ontology"),
-        load_parent_closure: bool = False,
-    ) -> Dict[str, "WDClass"]:
-        records = deserialize_jl(os.path.join(indir, "classes.jl"))
-        records = [WDClass.from_dict(c) for c in records]
-        records = {r.id: r for r in records}
-
-        if load_parent_closure:
-            logger.info("Load ancestors...")
-            parents_closure = deserialize_jl(
-                os.path.join(indir, "superclasses_closure.jl")
-            )
-            for rid, parents in parents_closure:
-                records[rid].parents_closure = set(parents)
-
-        return records
-
-    @staticmethod
-    def iter_file(
-        indir: Union[str, Path] = os.path.join(WIKIDATA_DIR, "ontology"),
-        load_parent_closure: bool = False,
-    ) -> Generator["WDClass", None, None]:
-        if load_parent_closure:
-            with open(os.path.join(indir, "classes.jl"), "r") as f, open(
-                os.path.join(indir, "superclasses_closure.jl"), "r"
-            ) as g:
-                while True:
-                    try:
-                        record = orjson.loads(next(f))
-                    except StopIteration:
-                        try:
-                            next(g)
-                        except StopIteration:
-                            break
-                        raise Exception(
-                            "Invalid superclasses_closure file. Number of rows doesn't match with the classes file"
-                        )
-
-                    cid, parents = orjson.loads(next(g))
-                    assert record["id"] == cid
-                    record["parents_closure"] = parents
-                    yield WDClass.from_dict(record)
-        else:
-            with open(os.path.join(indir, "classes.jl"), "r") as f:
-                for line in f:
-                    yield WDClass.from_dict(orjson.loads(line))
-
-    @staticmethod
-    def deserialize(s):
-        o = orjson.loads(s)
-        return WDClass.from_dict(o)
-
-    @staticmethod
-    def from_dict(o):
-        o["label"] = MultiLingualString(**o["label"])
-        o["description"] = MultiLingualString(**o["description"])
-        o["aliases"] = MultiLingualStringList(**o["aliases"])
-        o["parents_closure"] = set(o["parents_closure"])
-        return WDClass(**o)
-
-    @staticmethod
-    def from_qnode(qnode: QNode):
-        return WDClass(
-            id=qnode.id,
-            label=qnode.label,
-            description=qnode.description,
-            datatype=qnode.datatype,  # type: ignore
-            aliases=qnode.aliases,
-            parents=sorted(
-                {stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])}
-            ),
-            properties=sorted(
-                {stmt.value.as_entity_id() for stmt in qnode.props.get("P1963", [])}
-            ),
-            different_froms=sorted(
-                {stmt.value.as_entity_id() for stmt in qnode.props.get("P1889", [])}
-            ),
-            equivalent_classes=sorted(
-                {stmt.value.as_string() for stmt in qnode.props.get("P1709", [])}
-            ),
-            parents_closure=set(),
-        )
-
-    def serialize(self):
-        odict = {
-            k: getattr(self, k)
-            for k in [
-                "id",
-                "label",
-                "description",
-                "datatype",
-                "aliases",
-                "parents",
-                "properties",
-                "different_froms",
-                "equivalent_classes",
-                "parents_closure",
-            ]
-        }
-        for k in ["label", "description", "aliases"]:
-            odict[k] = odict[k].serialize()
-        return orjson.dumps(odict, option=orjson.OPT_SERIALIZE_DATACLASS, default=list)
-
-    def get_uri(self):
-        return f"http://www.wikidata.org/entity/{self.id}"
diff --git a/kgdata/wikidata/deprecated/wdproperty.py b/kgdata/wikidata/deprecated/wdproperty.py
deleted file mode 100644
index b59ee99..0000000
--- a/kgdata/wikidata/deprecated/wdproperty.py
+++ /dev/null
@@ -1,173 +0,0 @@
-import glob
-import gzip
-import os
-from dataclasses import dataclass, field
-from pathlib import Path
-from typing import List, Dict, Set, Union
-
-import orjson
-
-from kgdata.config import WIKIDATA_DIR
-from kgdata.wikidata.deprecated.models.qnode import (
-    QNode,
-    MultiLingualString,
-    MultiLingualStringList,
-)
-from sm.misc.deser import deserialize_jl, deserialize_json
-
-
-@dataclass
-class WDProperty:
-    id: str
-    label: MultiLingualString
-    description: MultiLingualString
-    # wikibase-lexeme, monolingualtext, wikibase-sense, url, wikibase-property,
-    # wikibase-form, external-id, time, commonsMedia, quantity, wikibase-item, musical-notation,
-    # tabular-data, string, math, geo-shape, globe-coordinate
-    datatype: str
-    aliases: MultiLingualStringList
-    parents: List[str]
-    see_also: List[str]
-    equivalent_properties: List[str]
-    subjects: List[str]
-    inverse_properties: List[str]
-    instanceof: List[str]
-    parents_closure: Set[str] = field(default_factory=set)
-
-    @staticmethod
-    def from_file(
-        indir: Union[str, Path] = os.path.join(WIKIDATA_DIR, "ontology"),
-        load_parent_closure: bool = False,
-    ) -> Dict[str, "WDProperty"]:
-        records = deserialize_jl(os.path.join(indir, "properties.jl"))
-        records = [WDProperty.from_dict(c) for c in records]
-        records = {r.id: r for r in records}
-
-        if load_parent_closure:
-            parents_closure = deserialize_jl(
-                os.path.join(indir, "superproperties_closure.jl")
-            )
-            for rid, parents in parents_closure:
-                records[rid].parents_closure = set(parents)
-
-        return records
-
-    @staticmethod
-    def deserialize(s):
-        o = orjson.loads(s)
-        return WDProperty.from_dict(o)
-
-    @staticmethod
-    def from_dict(o):
-        o["label"] = MultiLingualString(**o["label"])
-        o["description"] = MultiLingualString(**o["description"])
-        o["aliases"] = MultiLingualStringList(**o["aliases"])
-        o["parents_closure"] = set(o["parents_closure"])
-        return WDProperty(**o)
-
-    @staticmethod
-    def from_qnode(qnode: QNode):
-        try:
-            return WDProperty(
-                id=qnode.id,
-                label=qnode.label,
-                description=qnode.description,
-                datatype=qnode.datatype,  # type: ignore
-                aliases=qnode.aliases,
-                parents=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P279", [])}
-                ),
-                see_also=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1659", [])}
-                ),
-                equivalent_properties=sorted(
-                    {stmt.value.as_string() for stmt in qnode.props.get("P1628", [])}
-                ),
-                subjects=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1629", [])}
-                ),
-                inverse_properties=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P1696", [])}
-                ),
-                instanceof=sorted(
-                    {stmt.value.as_entity_id() for stmt in qnode.props.get("P31", [])}
-                ),
-            )
-        except:
-            print(qnode)
-            raise
-
-    def serialize(self):
-        odict = {
-            k: getattr(self, k)
-            for k in [
-                "id",
-                "label",
-                "description",
-                "datatype",
-                "aliases",
-                "parents",
-                "see_also",
-                "equivalent_properties",
-                "subjects",
-                "inverse_properties",
-                "instanceof",
-                "parents_closure",
-            ]
-        }
-        for k in ["label", "description", "aliases"]:
-            odict[k] = odict[k].serialize()
-        return orjson.dumps(odict, option=orjson.OPT_SERIALIZE_DATACLASS, default=list)
-
-    def get_uri(self):
-        return f"http://www.wikidata.org/prop/{self.id}"
-
-    def is_object_property(self):
-        return self.datatype == "wikibase-item"
-
-    def is_data_property(self):
-        return not self.is_object_property()
-
-    def is_transitive(self):
-        return "Q18647515" in self.instanceof
-
-
-@dataclass
-class WDQuantityPropertyStats:
-    id: str
-    value: "QuantityStats"
-    qualifiers: Dict[str, "QuantityStats"]
-
-    @staticmethod
-    def from_dir(
-        indir: str = os.path.join(
-            WIKIDATA_DIR, "step_2/quantity_prop_stats/quantity_stats"
-        )
-    ) -> Dict[str, "WDQuantityPropertyStats"]:
-        odict = {}
-        for infile in glob.glob(os.path.join(indir, "*.gz")):
-            with gzip.open(infile, "rb") as f:
-                for line in f:
-                    data = orjson.loads(line)
-                    item = WDQuantityPropertyStats(
-                        data["id"],
-                        QuantityStats(**data["value"]),
-                        {
-                            q: QuantityStats(**qstat)
-                            for q, qstat in data["qualifiers"].items()
-                        },
-                    )
-                    odict[item.id] = item
-        return odict
-
-
-@dataclass
-class QuantityStats:
-    units: List[str]
-    min: float
-    max: float
-    mean: float
-    std: float
-    size: float
-    int_size: int
-    n_overi36: int
diff --git a/kgdata/wikidata/models/propertystats.py b/kgdata/wikidata/models/propertystats.py
index 45398a2..781787d 100644
--- a/kgdata/wikidata/models/propertystats.py
+++ b/kgdata/wikidata/models/propertystats.py
@@ -1,19 +1,12 @@
 import glob
 import gzip
 import os
-from dataclasses import dataclass, field
-from pathlib import Path
-from typing import List, Dict, Set, Union
+from dataclasses import dataclass
+from typing import List, Dict
 
 import orjson
 
 from kgdata.config import WIKIDATA_DIR
-from kgdata.wikidata.deprecated.models.qnode import (
-    QNode,
-    MultiLingualString,
-    MultiLingualStringList,
-)
-from sm.misc.deser import deserialize_jl, deserialize_json
 
 
 @dataclass
diff --git a/kgdata/wikidata/redirections.py b/kgdata/wikidata/redirections.py
deleted file mode 100644
index 8f80320..0000000
--- a/kgdata/wikidata/redirections.py
+++ /dev/null
@@ -1,259 +0,0 @@
-import glob
-import os
-import re
-import gzip
-import csv
-from pathlib import Path
-from typing import List, Union
-from collections import defaultdict
-from kgdata.config import WIKIDATA_DIR
-from tqdm import tqdm
-from multiprocessing import Process, Queue
-
-
-def save_record_id_to_wikidata_id(outfile, q):
-    file_counter = 0
-    n_records_per_file = 64000
-
-    writer = gzip.open(outfile.format(auto=file_counter), "wb")
-    n_records = 0
-
-    while True:
-        record = q.get()
-        if record is None:
-            break
-
-        if not record.startswith("INSERT INTO"):
-            continue
-
-        for v in parse_sql_values(record):
-            if is_wd_item(v[2]):
-                pair = v[0], v[2]
-                writer.write("\t".join(pair).encode("utf-8"))
-                writer.write(b"\n")
-
-        n_records += 1
-
-        if n_records % n_records_per_file == 0:
-            writer.close()
-            file_counter += 1
-            writer = gzip.open(outfile.format(auto=file_counter), "wb")
-
-    writer.close()
-
-
-def extract_redirections(
-    page_dump: Union[str, Path],
-    redirect_dump: Union[str, Path],
-    outdir: Union[str, Path],
-) -> Path:
-    """Extract redirections from Wikidata redirection dumps and store the mapping in a key value file
-
-    Args:
-        page_dump: wikidata page dump (e.g., wikidatawiki-page.sql.gz)
-        redirect_dump: wikidata redirect dump (e.g., wikidatawiki-redirect.sql.gz)
-        outdir: output directory
-
-    Returns: a list of tsv files containing the redirection from old item to new item
-    """
-    # step 0 is to gather the internal id of each wikidata item
-    out_step0 = Path(outdir) / "step_0"
-    out_step0.mkdir(exist_ok=True, parents=True)
-
-    redirection_file = Path(outdir) / "redirections.tsv"
-    if redirection_file.exists():
-        return redirection_file
-
-    if not (out_step0 / "_SUCCESS").exists():
-        n_writers = 8
-        queues = []
-        writers = []
-
-        for i in range(n_writers):
-            outfile = os.path.join(out_step0, "{auto:05d}.%s.gz" % i)
-            queues.append(Queue())
-            writers.append(
-                Process(target=save_record_id_to_wikidata_id, args=(outfile, queues[i]))
-            )
-            writers[i].start()
-
-        with gzip.open(str(page_dump), "rt", encoding="utf-8", newline="\n") as f:
-            for i, line in tqdm(enumerate(f), total=11514):
-                queues[i % n_writers].put(line)
-
-            print(">>> Finish! Waiting to exit...")
-            for q in queues:
-                q.put(None)
-
-            for p in writers:
-                p.join()
-
-            with open(out_step0 / "_SUCCESS", "w") as g:
-                g.write("done")
-
-    i_wd = {}
-    with tqdm(desc="load internal wikidata id") as pbar:
-        for file in glob.glob(str(out_step0 / "*.gz")):
-            with gzip.open(file, "r") as f:
-                for line in f:
-                    i, wd = line.decode().strip().split("\t")
-                    assert i not in i_wd
-                    i_wd[i] = wd
-                    pbar.update()
-
-    buff_obj = []
-    with gzip.open(str(redirect_dump), "rt", encoding="utf-8", newline="\n") as f:
-        with tqdm(desc="Wikidata redirects") as p_bar:
-            for line in f:
-                if not line.startswith("INSERT INTO"):
-                    continue
-                for v in parse_sql_values(line):
-                    if is_wd_item(v[2]) and i_wd.get(v[0]):
-                        p_bar.update()
-                        buff_obj.append((i_wd[v[0]], v[2]))
-
-    # if you encounter any error, that means the following logic is no longer correct,
-    # uncomment the following line, and run the function verify_redirection to verify them one by one
-    # with open(redirection_file, "w") as f:
-    #     for k, v in buff_obj:
-    #         f.write("\t".join([k, v]) + "\n")
-
-    redirections = defaultdict(set)
-    for before_item, next_item in tqdm(buff_obj):
-        redirections[before_item].add(next_item)
-
-    for before_item in tqdm(list(redirections.keys())):
-        next_items = redirections[before_item]
-        if len(next_items) == 1:
-            redirections[before_item] = list(next_items)[0]
-            continue
-
-        # there must be only one final item on this list, otherwise, it's not consistent
-        final_items = [item for item in next_items if item not in redirections]
-        assert len(final_items) == 1, next_items
-        final_item = final_items[0]
-        redirect_items = [item for item in next_items if item != final_item]
-
-        # now verify for each item, it is being redirected to the same final item
-        for item in redirect_items:
-            ptr = item
-            for i in range(1000):
-                if redirections[ptr] == final_item:
-                    break
-                ptr = redirections[ptr]
-                if ptr not in redirections:
-                    raise Exception(
-                        f"Item {before_item} is mapped to both {final_item} and {item} -> {ptr}"
-                    )
-            else:
-                raise Exception("Likely to encounter a redirection loop.")
-        redirections[before_item] = final_item
-
-    with open(redirection_file, "w") as f:
-        for k, v in redirections.items():
-            f.write("\t".join([k, v]) + "\n")
-
-    return redirection_file
-
-
-def verify_redirection(infile: Union[str, Path]):
-    """Verify the extracted redirection"""
-    print("Verify redirection:")
-    with open(infile, "r") as f:
-        redirections = [line.strip().split("\t") for line in f]
-
-    counter = defaultdict(set)
-    for prev, now in redirections:
-        counter[prev].add(now)
-
-    for prev, now in counter.items():
-        if len(now) > 1:
-            print("Redirect to more than one entity:", prev, list(now))
-
-
-"""
-Some functions below are borrowed from Phuc Nguyen <phucnt@nii.ac.jp> in his mtab_dev repo on Github (MIT License).
-"""
-
-
-def convert_num(text):
-    if not text:
-        return None
-    try:
-        text = removeCommasBetweenDigits(text)
-        # tmp = representsFloat(text)
-        # if not tmp:
-        #     return None
-        #
-        # return parseNumber(text)
-        return float(text)
-    except ValueError:
-        return None
-
-
-def get_wd_int(wd_id):
-    result = None
-    if wd_id and len(wd_id) and wd_id[0].lower() in ["p", "q"] and " " not in wd_id:
-        result = convert_num(wd_id[1:])
-    return result
-
-
-def is_wd_item(wd_id):
-    if get_wd_int(wd_id) is None:
-        return False
-    else:
-        return True
-
-
-def removeCommasBetweenDigits(text):
-    """
-    :example:
-    >>> removeCommasBetweenDigits("sfeyv dsf,54dsf ef 6, 6 zdgy 6,919 Photos and 3,3 videos6,")
-    'sfeyv dsf,54dsf ef 6, 6 zdgy 6919 Photos and 33 videos6,'
-    """
-    if text is None:
-        return None
-    else:
-        return re.sub(r"([0-9]),([0-9])", "\g<1>\g<2>", text)
-
-
-def parse_sql_values(line):
-    values = line[line.find("` VALUES ") + 9 :]
-    latest_row = []
-    reader = csv.reader(
-        [values],
-        delimiter=",",
-        doublequote=False,
-        escapechar="\\",
-        quotechar="'",
-        strict=True,
-    )
-    for reader_row in reader:
-        for column in reader_row:
-            if len(column) == 0 or column == "NULL":
-                latest_row.append(chr(0))
-                continue
-            if column[0] == "(":
-                new_row = False
-                if len(latest_row) > 0:
-                    if latest_row[-1][-1] == ")":
-                        latest_row[-1] = latest_row[-1][:-1]
-                        new_row = True
-                if new_row:
-                    yield latest_row
-                    latest_row = []
-                if len(latest_row) == 0:
-                    column = column[1:]
-            latest_row.append(column)
-        if latest_row[-1][-2:] == ");":
-            latest_row[-1] = latest_row[-1][:-2]
-            yield latest_row
-
-
-if __name__ == "__main__":
-    extract_redirections(
-        Path(WIKIDATA_DIR) / "20211220/step_0/wikidatawiki-page.sql.gz",
-        Path(WIKIDATA_DIR) / "20211220/step_0/wikidatawiki-redirect.sql.gz",
-        Path(WIKIDATA_DIR) / "20211220/redirections",
-    )
-    verify_redirection(Path(WIKIDATA_DIR) / "20211220/redirections/redirections.tsv")
diff --git a/kgdata/wikidata/s00_prep_data.py b/kgdata/wikidata/s00_prep_data.py
deleted file mode 100644
index 7a98b6f..0000000
--- a/kgdata/wikidata/s00_prep_data.py
+++ /dev/null
@@ -1,95 +0,0 @@
-import bz2
-import gzip
-import os
-import shutil
-import glob
-from multiprocessing import Process, Queue
-
-from tqdm.auto import tqdm
-
-from kgdata.config import WIKIDATA_DIR
-
-"""This is the module that split and prepare the raw data downloaded from Wikidata to make it easier to process with Spark
-"""
-
-
-def save2file(outfile, q):
-    file_counter = 0
-    n_records_per_file = 64000
-
-    writer = gzip.open(outfile.format(auto=file_counter), "wb")
-    n_records = 0
-
-    while True:
-        record = q.get()
-        if record is None:
-            break
-
-        n_records += 1
-        if n_records % n_records_per_file == 0:
-            writer.close()
-            file_counter += 1
-            writer = gzip.open(outfile.format(auto=file_counter), "wb")
-
-        # fix the json record (replace the `inplace_fix_file_in_prep01` function)
-        if record[-3:] == b"},\n":
-            record = record[:-3]
-            writer.write(record)
-            writer.write(b"}\n")
-        elif record[-2:] == b"}\n":
-            writer.write(record)
-        elif record == b"]\n":
-            continue
-        else:
-            print(record)
-            raise Exception("Unreachable!")
-
-    writer.close()
-
-
-def prep01(
-    indir: str = os.path.join(WIKIDATA_DIR, "step_0"),
-    outdir: str = os.path.join(WIKIDATA_DIR, "step_1"),
-    overwrite: bool = False,
-):
-    if os.path.exists(outdir):
-        if not overwrite:
-            return
-        shutil.rmtree(outdir)
-    os.mkdir(outdir)
-
-    match_files = glob.glob(os.path.join(indir, "*.bz2"))
-    if len(match_files) != 1:
-        raise Exception(
-            f"Invalid input directory. Expect to have only one .bz2 file but got {len(match_files)} files"
-        )
-    infile = match_files[0]
-
-    n_writers = 8
-
-    queues = []
-    writers = []
-
-    for i in range(n_writers):
-        outfile = os.path.join(outdir, "{auto:05d}.%s.gz" % i)
-        queues.append(Queue())
-        writers.append(Process(target=save2file, args=(outfile, queues[i])))
-        writers[i].start()
-
-    with bz2.open(infile, "rb") as f:
-        line = f.readline()[:-1]
-        assert line == b"["
-
-        for i, line in tqdm(enumerate(f), total=85883865):
-            queues[i % n_writers].put(line)
-
-    print(">>> Finish! Waiting to exit...")
-    for q in queues:
-        q.put(None)
-
-    for p in writers:
-        p.join()
-
-
-if __name__ == "__main__":
-    prep01()
diff --git a/kgdata/wikidata/search.py b/kgdata/wikidata/search.py
new file mode 100644
index 0000000..1f8ffab
--- /dev/null
+++ b/kgdata/wikidata/search.py
@@ -0,0 +1,165 @@
+"""Searching Wikidata using Pyserini Index."""
+
+from dataclasses import dataclass
+import os
+from pathlib import Path
+from typing import List, Literal, Optional, Union
+from kgdata.spark import does_result_dir_exist
+from kgdata.wikidata.config import WDDataDirCfg
+from kgdata.wikidata.datasets.classes import classes
+from kgdata.wikidata.datasets.entities import entities
+from kgdata.wikidata.db import get_wdprop_db
+from kgdata.wikidata.models import WDClass, WDEntity, WDProperty
+from pyserini.search.lucene import LuceneSearcher
+import orjson
+from tqdm import tqdm
+
+from kgdata.wikidata.datasets.properties import properties
+from pyserini.index.lucene import IndexReader
+from sm.misc.deser import deserialize_json
+from sm.misc.funcs import identity_func
+from kgdata.pyserini import (
+    AnalyzerType,
+    PyseriniDoc,
+    build_pyserini_index,
+    IndexSettings,
+)
+
+
+@dataclass
+class SearchReturnType:
+    """A placeholder for the return type of the search function."""
+
+    docid: str
+    score: float
+
+
+class WDSearch:
+    def __init__(self, index: Union[str, Path]):
+        self.index_dir = Path(index)
+        self.searcher = LuceneSearcher(str(index))
+        self.settings = IndexSettings.from_dict(
+            deserialize_json(self.index_dir / "_SUCCESS")
+        )
+
+        self.analyzer = self.settings.get_analyzer()
+
+    def search(self, query: str, limit: int = 10) -> List[SearchReturnType]:
+        res = self.searcher.search(self.analyzer(query), k=limit)
+        return [SearchReturnType(docid=x.docid, score=x.score) for x in res]
+
+    def batch_search(
+        self, queries: List[str], limit: int = 10
+    ) -> List[List[SearchReturnType]]:
+        queries = [self.analyzer(query) for query in queries]
+        query_ids = [str(x) for x in range(len(queries))]
+
+        n_processes = os.cpu_count()
+        assert isinstance(n_processes, int)
+
+        result = self.searcher.batch_search(
+            queries, query_ids, k=limit, threads=n_processes
+        )
+        return [
+            [SearchReturnType(docid=x.docid, score=x.score) for x in result[qid]]
+            for qid in query_ids
+        ]
+
+    def analyze(self, text: str):
+        return self.analyzer(text)
+
+    def __getstate__(self):
+        return {"index_dir": self.index_dir}
+
+    def __setstate__(self, state):
+        self.__init__(state["index_dir"])
+
+
+def build_index(
+    name: Literal["entities", "props", "classes"],
+    index_parent_dir: Optional[Union[str, Path]] = None,
+    lang: str = "en",
+):
+    cfg = WDDataDirCfg.get_instance()
+    settings = IndexSettings(analyzer=AnalyzerType.TrigramAnalyzer)
+
+    data_dir = cfg.search / name / settings.analyzer.value
+    if index_parent_dir is None:
+        index_parent_dir = cfg.search
+    else:
+        index_parent_dir = Path(index_parent_dir)
+    index_dir = index_parent_dir / f"{name}_{settings.analyzer.value}"
+
+    if name == "entities":
+        dataset = entities(lang=lang)
+    elif name == "props":
+        dataset = properties(lang=lang)
+    else:
+        assert name == "classes"
+        dataset = classes(lang=lang)
+
+    dataset = dataset.map(to_doc)
+    dataset.postfilter = ignore_empty
+
+    build_pyserini_index(
+        dataset,
+        data_dir=data_dir,
+        index_dir=index_dir,
+        settings=settings,
+        n_files=-1 if name == "entities" else 256,
+    )
+    return index_dir
+
+
+def to_doc(record: Union[WDEntity, WDProperty, WDClass]) -> PyseriniDoc:
+    return {"id": record.id, "contents": str(record.label)}
+
+
+def ignore_empty(record: PyseriniDoc) -> bool:
+    return record["contents"].strip() != ""
+
+
+if __name__ == "__main__":
+    cfg = WDDataDirCfg.init("/nas/ckgfs/users/binhvu/wikidata/20211213")
+
+    index = build_index("entities")
+    index = build_index("classes")
+
+    index = build_index("props")
+
+    # db = get_wdprop_db(cfg.datadir / "databases/wdprops.db")
+    # search = WDSearch(index)
+    # hits = search.search("histric coutry", limit=50)
+    # for i, hit in enumerate(hits):
+    #     print(f"{i+1:2} {hit.score:.5f} {hit.docid:6} {db[hit.docid].label}")
+
+    # from pyserini.index.lucene import IndexReader
+
+    # # Initialize from a pre-built index:
+    # index_reader = IndexReader.from_prebuilt_index("robust04")
+
+    # # Initialize from an index path:
+    # index_reader = IndexReader("indexes/index-robust04-20191213/")
+    # print(ngram_filter("administrative", min=3, max=4))
+    # build_wdprops()
+
+    # db = get_wdprop_db("/data/binhvu/sm-dev/data/home/databases/wdprops.db")
+
+    # cfg = WDDataDirCfg.get_instance()
+
+    # index_dir = cfg.search / "properties" / f"index_{IndexSettings.analyzer}"
+    # searcher = LuceneSearcher(str(index_dir))
+
+    # def search(q):
+    #     hits = searcher.search(q, k=20)
+
+    # search(trigram_analyzer("ocated"))
+    # term = "cities"
+    # index_reader = IndexReader(str(cfg.search / "properties" / "index"))
+    # # Analyze the term.
+    # analyzed = index_reader.analyze(term)
+    # print(f'The analyzed form of "{term}" is "{analyzed[0]}"')
+
+    # # Skip term analysis:
+    # df, cf = index_reader.get_term_counts(analyzed[0], analyzer=None)
+    # print(f'term "{term}": df={df}, cf={cf}')
diff --git a/kgdata/wikipedia/config.py b/kgdata/wikipedia/config.py
index 8cb385d..a05a9fc 100644
--- a/kgdata/wikipedia/config.py
+++ b/kgdata/wikipedia/config.py
@@ -18,8 +18,16 @@ def __init__(self, datadir: Path) -> None:
         # directorys contain dumps and their splitted files
         # for the name of the dumps, see the corresponding function `self.get_X_file` in this class
         self.dumps = datadir / "dumps"
+        self.articles = datadir / "articles"
+        self.grouped_articles = datadir / "grouped_articles"
         self.html_articles = datadir / "html_articles"
         self.html_tables = datadir / "html_tables"
+        self.relational_tables = datadir / "relational_tables"
+        self.linked_relational_tables = datadir / "linked_relational_tables"
+        self.easy_tables = datadir / "easy_tables"
+
+    def get_article_file(self):
+        return self._get_file(self.dumps / "*pages-articles*.xml.bz2")
 
     def get_html_article_file(self):
         return self._get_file(self.dumps / "*NS0-*ENTERPRISE-HTML.json.tar.gz")
diff --git a/kgdata/wikipedia/datasets/__main__.py b/kgdata/wikipedia/datasets/__main__.py
index 6cffc35..f10cfee 100644
--- a/kgdata/wikipedia/datasets/__main__.py
+++ b/kgdata/wikipedia/datasets/__main__.py
@@ -1,18 +1,49 @@
+"""
+Run the command ``python -m kgdata.wikipedia.datasets`` to generate the datasets.
+
+.. code:: bash
+
+    $ python -m kgdata.wikipedia.datasets --help
+
+    Usage: python -m kgdata.wikipedia.datasets [OPTIONS]
+
+    Options:
+    --wp-dir TEXT       Wikipedia directory  [required]
+    --wd-dir TEXT       Wikidata directory
+    -d, --dataset TEXT  Dataset name  [required]
+    --help              Show this message and exit. 
+
+Examples::
+
+    python -m kgdata.wikipedia.datasets --wp-dir $WP_DIR -d relational_tables
+    python -m kgdata.wikipedia.datasets --wp-dir $WP_DIR --wd-dir $WD_DIR -d linked_relational_tables
+
+.. note::
+
+    For the commands in the above examples to run correctly, replaced ``$WP_DIR`` with the path to the Wikipedia directory and ``$WD_DIR`` with the path to the Wikidata directory, e.g., ``export WD_DIR=/data/wikipedia/20220420``
+
+"""
+
 from importlib import import_module
 from typing_extensions import Required
 import click
 from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikidata.config import WDDataDirCfg
 import kgdata.wikipedia.datasets
 from loguru import logger
 
 
 @click.command("Generate a specific dataset")
-@click.option("-s", "--source", required=True, help="Wikipedia directory")
+@click.option("--wp-dir", required=True, help="Wikipedia directory")
+@click.option("--wd-dir", default="", help="Wikidata directory")
 @click.option("-d", "--dataset", required=True, help="Dataset name")
-def main(source: str, dataset: str):
-    logger.info("Wikidata directory: {}", source)
+def main(wp_dir: str, wd_dir: str, dataset: str):
+    logger.info("Wikipedia directory: {}", wp_dir)
 
-    WPDataDirConfig.init(source)
+    WPDataDirConfig.init(wp_dir)
+    if wd_dir.strip() != "":
+        logger.info("Wikidata directory: {}", wd_dir)
+        WDDataDirCfg.init(wd_dir)
 
     module = import_module(f"kgdata.wikipedia.datasets.{dataset}")
     getattr(module, dataset)()
diff --git a/kgdata/wikipedia/datasets/articles.py b/kgdata/wikipedia/datasets/articles.py
new file mode 100644
index 0000000..55a4e87
--- /dev/null
+++ b/kgdata/wikipedia/datasets/articles.py
@@ -0,0 +1,84 @@
+from bz2 import BZ2File
+from gzip import GzipFile
+from typing import BinaryIO, Union
+import orjson
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist
+from kgdata.splitter import split_a_file
+from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikipedia.models.page_article import WikiPageArticle
+import xml.etree.ElementTree as ET
+
+
+def articles() -> Dataset[WikiPageArticle]:
+    """Extract articles from XML dumps"""
+    cfg = WPDataDirConfig.get_instance()
+
+    if not does_result_dir_exist(cfg.articles):
+        split_a_file(
+            infile=cfg.get_article_file(),
+            outfile=cfg.articles / "part.ndjson.gz",
+            n_writers=8,
+            override=True,
+            n_records_per_file=64000,
+            record_iter=iter_from_dump,
+        )
+        (cfg.articles / "_SUCCESS").touch()
+    return Dataset(cfg.articles / "*.gz", deserialize=deser_page_article)
+
+
+def deser_page_article(line: Union[str, bytes]) -> WikiPageArticle:
+    return WikiPageArticle.from_dict(orjson.loads(line))
+
+
+def iter_from_dump(infile: Union[BZ2File, GzipFile, BinaryIO]):
+    # this seems to be the only way to specify default namespace in python 3.7
+    ET.register_namespace("", "http://www.mediawiki.org/xml/export-0.10/")
+
+    tree = iter(
+        ET.iterparse(
+            infile,
+            events=(
+                "start",
+                "end",
+            ),
+        )
+    )
+    event, root = next(tree)
+
+    for event, elem in tree:
+        if (
+            event != "end"
+            or elem.tag != "{http://www.mediawiki.org/xml/export-0.10/}page"
+        ):
+            continue
+
+        id = elem.find("{http://www.mediawiki.org/xml/export-0.10/}id").text
+        ns = elem.find("{http://www.mediawiki.org/xml/export-0.10/}ns").text
+        title = elem.find("{http://www.mediawiki.org/xml/export-0.10/}title").text
+        redirect_title = elem.find(
+            "{http://www.mediawiki.org/xml/export-0.10/}redirect"
+        )
+        if redirect_title is not None:
+            redirect_title = redirect_title.get("title")
+        model = elem.find(
+            "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}model"
+        ).text
+        format = elem.find(
+            "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}format"
+        ).text
+        text = (
+            elem.find(
+                "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}text"
+            ).text
+            or ""
+        )
+
+        yield orjson.dumps(
+            WikiPageArticle(
+                id, ns, title, redirect_title, model, format, text
+            ).to_dict()
+        )
+
+        # avoid using too much memory, remove the elem. leverage the fact that pages_articles if mediawiki > page
+        root.remove(elem)
diff --git a/kgdata/wikipedia/datasets/easy_tables.py b/kgdata/wikipedia/datasets/easy_tables.py
new file mode 100644
index 0000000..3a50bb8
--- /dev/null
+++ b/kgdata/wikipedia/datasets/easy_tables.py
@@ -0,0 +1,184 @@
+from typing import Callable, List
+from functools import partial
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist
+from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikipedia.datasets.linked_relational_tables import (
+    linked_relational_tables,
+    ser_linked_tables,
+    deser_linked_tables,
+)
+from kgdata.wikipedia.models.linked_html_table import LinkedHTMLTable
+
+
+def easy_tables() -> Dataset[LinkedHTMLTable]:
+    """Tables that can be labeled automatically easily.
+    The table is easy or not is determined by :py:func:`kgdata.wikipedia.easy_table.is_easy_table`
+
+    """
+    cfg = WPDataDirConfig.get_instance()
+
+    # step 1: generate stats of which tables passed which tests
+
+    # step 2: filter the tables
+    if not does_result_dir_exist(cfg.easy_tables):
+        tests = [
+            EasyTests.min_rows,
+            EasyTests.min_links_all_columns,
+            EasyTests.min_link_coverage_all_columns,
+            EasyTests.min_existing_links_all_columns,
+        ]
+
+        # print(
+        #     (
+        #         linked_relational_tables()
+        #         .get_rdd()
+        #         .filter(partial(is_easy_table, tests=tests))
+        #         .count()
+        #     )
+        # )
+        (
+            linked_relational_tables()
+            .get_rdd()
+            .filter(partial(is_easy_table, tests=tests))
+            .map(ser_linked_tables)
+            .coalesce(192, shuffle=True)
+            .saveAsTextFile(
+                str(cfg.easy_tables),
+                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
+            )
+        )
+
+    return Dataset(
+        file_pattern=cfg.easy_tables / "*.gz", deserialize=deser_linked_tables
+    )
+
+
+def is_easy_table(
+    tbl: LinkedHTMLTable, tests: List[Callable[[LinkedHTMLTable], bool]]
+) -> bool:
+    """Determine if a table is easy or not.
+
+    Args:
+        tbl: Input table.
+        tests: List of tests. Each test is a function that takes a table and returns a boolean.
+    """
+    return all(test(tbl) for test in tests)
+
+
+def get_n_headers(tbl: LinkedHTMLTable) -> int:
+    n_headers = 0
+    try:
+        for row in tbl.table.iter_rows():
+            if not row.get_cell(0).is_header:
+                break
+            n_headers += 1
+    except KeyError as e:
+        raise KeyError(str(e) + " " + str((">>>", tbl.table.id, tbl.table.url)))
+    return n_headers
+
+
+class EasyTests:
+    MIN_ROWS = 10
+    MIN_FREQ_LINKS = 0.7
+    MIN_LINK_SURFACE = 0.9
+    MIN_EXISTING_LINKS = 0.8
+
+    @staticmethod
+    def min_rows(tbl: LinkedHTMLTable) -> bool:
+        """Determine if a table has at least min_rows rows.
+
+        Args:
+            tbl: Input table.
+            min_rows: Minimum number of rows.
+        """
+        # minus one for the header
+        n_headers = get_n_headers(tbl)
+        return tbl.table.n_rows() - n_headers >= EasyTests.MIN_ROWS
+
+    @staticmethod
+    def min_link_coverage_all_columns(tbl: LinkedHTMLTable) -> bool:
+        n_headers = get_n_headers(tbl)
+        nrows, ncols = tbl.table.shape()
+        nrows -= n_headers
+
+        if nrows == 0:
+            return False
+
+        for ci in range(ncols):
+            percent_surface = (
+                sum(
+                    sum((link.end - link.start) for link in tbl.links.get((ri, ci), []))
+                    / max(0.1, tbl.table.get_cell(ri, ci).value.len())
+                    for ri in range(1, nrows + 1)
+                )
+                / nrows
+            )
+            if percent_surface > 0 and percent_surface < EasyTests.MIN_LINK_SURFACE:
+                return False
+        return True
+
+    @staticmethod
+    def min_links_all_columns(tbl: LinkedHTMLTable) -> bool:
+        n_headers = get_n_headers(tbl)
+        nrows, ncols = tbl.table.shape()
+        nrows -= n_headers
+
+        if nrows == 0:
+            return False
+
+        for ci in range(ncols):
+            nlinks = sum(
+                int(len(tbl.links.get((ri, ci), [])) > 0) for ri in range(1, nrows + 1)
+            )
+            if nlinks > 0 and nlinks / nrows < EasyTests.MIN_FREQ_LINKS:
+                return False
+        return True
+
+    @staticmethod
+    def single_links_all_columns(tbl: LinkedHTMLTable) -> bool:
+        n_headers = get_n_headers(tbl)
+        nrows, ncols = tbl.table.shape()
+        nrows -= n_headers
+
+        for ci in range(ncols):
+            count = 0
+            total = 0
+            for ri in range(1, nrows + 1):
+                if (ri, ci) not in tbl.links:
+                    continue
+                nonempty_links = [
+                    link for link in tbl.links[ri, ci] if link.end > link.start
+                ]
+                if len(nonempty_links) > 0:
+                    count += int(len(nonempty_links) == 1)
+                    total += 1
+            if count != total:
+                return False
+        return True
+
+    @staticmethod
+    def min_existing_links_all_columns(tbl: LinkedHTMLTable) -> bool:
+        n_headers = get_n_headers(tbl)
+        nrows, ncols = tbl.table.shape()
+        nrows -= n_headers
+
+        for ci in range(ncols):
+            n_existing_links = 0
+            total = 0
+            for ri in range(1, nrows + 1):
+                if (ri, ci) not in tbl.links:
+                    continue
+                nonempty_links = [
+                    link for link in tbl.links[ri, ci] if link.end > link.start
+                ]
+                if len(nonempty_links) > 0:
+                    n_existing_links += sum(
+                        int(link.wikidata_id is not None) for link in nonempty_links
+                    ) / len(nonempty_links)
+                    total += 1
+
+            if total > 0 and n_existing_links / total < EasyTests.MIN_EXISTING_LINKS:
+                return False
+
+        return True
diff --git a/kgdata/wikipedia/datasets/grouped_articles.py b/kgdata/wikipedia/datasets/grouped_articles.py
new file mode 100644
index 0000000..f718c9b
--- /dev/null
+++ b/kgdata/wikipedia/datasets/grouped_articles.py
@@ -0,0 +1,115 @@
+from dataclasses import dataclass
+import gzip
+import os
+from typing import TypedDict, Tuple, List
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist
+from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikipedia.datasets.articles import articles
+import orjson
+from sm.misc.deser import get_open_fn
+from tqdm import tqdm
+
+
+class GroupedArticles(TypedDict):
+    # (title, id)
+    final: Tuple[str, str]
+    group: List[Tuple[str, str]]
+
+
+def grouped_articles() -> Dataset[GroupedArticles]:
+    """Group wikipedia pages/articles that are belong to the same entity"""
+    cfg = WPDataDirConfig.get_instance()
+    batch_size = 64000
+
+    if not does_result_dir_exist(cfg.grouped_articles):
+        wiki_links = []
+        for infile in tqdm(articles().get_files(), desc="read file"):
+            with get_open_fn(infile)(infile, "rb") as f:
+                for line in f:
+                    r = orjson.loads(line)
+                    wiki_links.append((r["id"], r["title"], r["redirect_title"]))
+
+        # verify if we have the case of one source node is link to two target nodes, then we build dict that manually curate those nodes
+        tmp = {}
+        manually_curated_source2target = {}
+        title2id = {}
+        for source_id, source, target in tqdm(wiki_links):
+            if source not in tmp:
+                tmp[source] = target
+                title2id[source] = source_id
+            else:
+                assert source not in manually_curated_source2target
+                if target is None:
+                    manually_curated_source2target[source] = tmp[source]
+                    # don't have to update the id since this we discard this article
+                else:
+                    manually_curated_source2target[source] = target
+                    title2id[source] = source_id
+
+                print("`%s` | `%s` | `%s`" % (source, target, tmp[source]))
+
+        # build reverse map
+        reverse_map = {}
+        leaves = set()
+        for source_id, source, target in tqdm(wiki_links, desc="build reverse map"):
+            if source in manually_curated_source2target:
+                continue
+
+            if target is None:
+                assert source not in leaves
+                leaves.add(source)
+                continue
+
+            if target not in reverse_map:
+                reverse_map[target] = [source]
+            else:
+                reverse_map[target].append(source)
+
+        for source, target in manually_curated_source2target.items():
+            if target is None:
+                leaves.add(source)
+                continue
+            if target not in reverse_map:
+                reverse_map[target] = [source]
+            else:
+                reverse_map[target].append(source)
+
+        # now travel upward to group
+        visited = set()
+
+        def trace_upward(reverse_map, group, ptr):
+            assert ptr not in visited
+            visited.add(ptr)
+
+            for parent in reverse_map.get(ptr, []):
+                group.append((parent, title2id[parent]))
+                trace_upward(reverse_map, group, parent)
+
+        groups: List[GroupedArticles] = []
+        for leaf in tqdm(leaves, desc="grouping"):
+            if leaf not in reverse_map:
+                groups.append(
+                    {"final": (leaf, title2id[leaf]), "group": [(leaf, title2id[leaf])]}
+                )
+            else:
+                group = [(leaf, title2id[leaf])]
+                trace_upward(reverse_map, group, leaf)
+                groups.append({"group": group, "final": (leaf, title2id[leaf])})
+
+        # write result
+        count = 0
+        cfg.grouped_articles.mkdir(parents=True, exist_ok=True)
+
+        for i in tqdm(range(0, len(groups), batch_size), desc="writing result"):
+            with gzip.open(
+                cfg.grouped_articles / ("part.%05d.ndjson.gz" % count), "wb"
+            ) as f:
+                for g in groups[i : i + batch_size]:
+                    f.write(orjson.dumps(g))
+                    f.write(b"\n")
+                count += 1
+
+        (cfg.grouped_articles / "_SUCCESS").touch()
+
+    return Dataset(cfg.grouped_articles / "*.gz", deserialize=orjson.loads)
diff --git a/kgdata/wikipedia/datasets/html_articles.py b/kgdata/wikipedia/datasets/html_articles.py
index 1bf88b3..72a3959 100644
--- a/kgdata/wikipedia/datasets/html_articles.py
+++ b/kgdata/wikipedia/datasets/html_articles.py
@@ -46,3 +46,7 @@ def html_articles() -> Dataset[HTMLArticle]:
 
 def deser_html_articles(line: Union[str, bytes]) -> HTMLArticle:
     return HTMLArticle.from_dump_dict(orjson.loads(line))
+
+
+def ser_html_articles(article: HTMLArticle) -> bytes:
+    return orjson.dumps(article.to_dict())
diff --git a/kgdata/wikipedia/datasets/html_tables.py b/kgdata/wikipedia/datasets/html_tables.py
index c3b2a30..4f06f64 100644
--- a/kgdata/wikipedia/datasets/html_tables.py
+++ b/kgdata/wikipedia/datasets/html_tables.py
@@ -1,30 +1,23 @@
-import orjson
-from typing import TypedDict, Union
-from loguru import logger
 from kgdata.dataset import Dataset
 from kgdata.spark import does_result_dir_exist
-from kgdata.wikipedia.datasets.html_articles import html_articles
 from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikipedia.datasets.html_articles import html_articles
 from kgdata.wikipedia.models.html_article import HTMLArticle
-from table_extractor.table_extractor import HTMLTableExtractor
-from table_extractor.models.html_table import HTMLTable
-import sm.misc as M
+from loguru import logger
+from rsoup.rsoup import ContextExtractor, Table, TableExtractor
 
 
-def html_tables() -> Dataset[HTMLTable]:
+def html_tables() -> Dataset[Table]:
     """Extracting all tables (at the lowest level) and their surrounding context from Wikipedia articles."""
 
     cfg = WPDataDirConfig.get_instance()
 
-    # resp = html_articles().get_rdd().filter(lambda x: x.page_id == 673723).take(1)
-    # M.serialize_pkl(resp, "/tmp/debug.pkl")
-    # return
-
     if not does_result_dir_exist(cfg.html_tables):
         (
             html_articles()
             .get_rdd()
             .flatMap(extract_tables)
+            .coalesce(1024, shuffle=True)
             .saveAsTextFile(
                 str(cfg.html_tables),
                 compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
@@ -34,21 +27,29 @@ def html_tables() -> Dataset[HTMLTable]:
     return Dataset(
         file_pattern=cfg.html_tables / "*.gz",
         deserialize=deser_table,
-        filter=lambda x: x[0] == "{",
+        # can be json object, or string. it is string when we fail to extract tables from the articles
+        prefilter=lambda x: x[0] == "{",
     )
 
 
-def deser_table(x: Union[str, bytes]) -> HTMLTable:
-    return HTMLTable.from_dict(orjson.loads(x))
+def deser_table(x: str) -> Table:
+    return Table.from_json(x)
+
+
+def ser_table(x: Table) -> str:
+    return x.to_json()
 
 
 def extract_tables(article: HTMLArticle):
+    extractor = TableExtractor(context_extractor=ContextExtractor())
     try:
-        tables = HTMLTableExtractor(article.url, article.html, "lxml").extract_tables(
-            auto_span=False, auto_pad=False, extract_context=False
+        tables = extractor.extract(
+            article.url,
+            article.html,
+            auto_span=True,
+            auto_pad=True,
+            extract_context=True,
         )
-
-        return [orjson.dumps(tbl.to_dict()) for tbl in tables]
     except Exception as e:
         logger.exception(
             "Error while extracting tables from article {}: {}",
@@ -57,13 +58,17 @@ def extract_tables(article: HTMLArticle):
         )
         return [article.url]
 
+    # postprocess wikipedia tables
+    for table in tables:
+        for cell in table.iter_cells():
+            value = cell.value
+            for uid in value.iter_element_id():
+                if (
+                    value.get_element_tag_by_id(uid) == "a"
+                    and value.get_element_attr_by_id(uid, "href") is None
+                ):
+                    cls = value.get_element_attr_by_id(uid, "class")
+                    assert "selflink" in cls, cls
+                    value.set_element_attr_by_id(uid, "href", article.url)
 
-if __name__ == "__main__":
-    resp = M.deserialize_pkl("/tmp/debug.pkl")
-    for article in resp:
-        tables = HTMLTableExtractor(article.url, article.html).extract_tables()
-        for tbl in tables:
-            x = tbl.to_dict()
-            orjson.dumps(tbl.to_dict())
-
-    # print(resp[0].page_id)
+    return [tbl.to_json() for tbl in tables]
diff --git a/kgdata/wikipedia/datasets/linked_relational_tables.py b/kgdata/wikipedia/datasets/linked_relational_tables.py
new file mode 100644
index 0000000..1afc92e
--- /dev/null
+++ b/kgdata/wikipedia/datasets/linked_relational_tables.py
@@ -0,0 +1,134 @@
+from kgdata.wikidata.datasets.wp2wd import wp2wd
+from kgdata.wikipedia.datasets.html_tables import deser_table, html_tables, ser_table
+from kgdata.wikipedia.misc import get_title_from_url, is_wikipedia_url
+from kgdata.wikipedia.models.linked_html_table import LinkedHTMLTable, WikiLink
+import orjson, ujson
+from typing import Dict, Iterable, List, Literal, Tuple, TypedDict, Union
+from loguru import logger
+from kgdata.dataset import Dataset
+from kgdata.spark import does_result_dir_exist, left_outer_join
+from kgdata.wikipedia.datasets.html_articles import html_articles
+from kgdata.wikipedia.config import WPDataDirConfig
+from kgdata.wikipedia.models.html_article import HTMLArticle
+
+from rsoup.rsoup import Table, Cell
+import sm.misc as M
+from importlib import import_module
+
+
+def linked_tables(
+    table_dataset_name: Literal["html_tables", "relational_tables"], lang: str = "en"
+) -> Dataset[LinkedHTMLTable]:
+    """Convert Wikipedia links in HTML tables to links in Wikidata using sitelinks
+
+    Args:
+        table_dataset_name: the table dataset to convert
+        lang: the language of Wikidata
+    """
+    cfg = WPDataDirConfig.get_instance()
+
+    if table_dataset_name == "relational_tables":
+        outdir = cfg.linked_relational_tables
+    else:
+        raise NotImplementedError(table_dataset_name)
+
+    if not does_result_dir_exist(outdir):
+        module = import_module(f"kgdata.wikipedia.datasets.{table_dataset_name}")
+        table_dataset: Dataset[Table] = getattr(module, table_dataset_name)()
+
+        tbl2titles = (
+            table_dataset.get_rdd()
+            .flatMap(extract_title_to_tables)
+            .groupByKey()
+            .leftOuterJoin(wp2wd(lang).get_rdd())
+            .flatMap(lambda x: [(tbl_id, (x[0], x[1][1])) for tbl_id in x[1][0]])
+            .groupByKey()
+        )
+
+        (
+            table_dataset.get_rdd()
+            .map(lambda tbl: (tbl.id, tbl))
+            .leftOuterJoin(tbl2titles)
+            .map(merge_link_to_table)
+            .map(ser_linked_tables)
+            .coalesce(1024, shuffle=True)
+            .saveAsTextFile(
+                str(outdir),
+                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
+            )
+        )
+
+    ds = Dataset(file_pattern=str(outdir / "*.gz"), deserialize=deser_linked_tables)
+    return ds
+
+
+def linked_relational_tables(lang: str = "en") -> Dataset[LinkedHTMLTable]:
+    """Convert Wikipedia links in HTML tables to links in Wikidata using sitelinks
+
+    Args:
+        lang: the language of Wikidata
+    """
+    return linked_tables("relational_tables", lang)
+
+
+def deser_linked_tables(x: str) -> LinkedHTMLTable:
+    return LinkedHTMLTable.from_json(x)
+
+
+def ser_linked_tables(tbl: LinkedHTMLTable) -> bytes:
+    return tbl.to_json()
+
+
+def extract_title_to_tables(tbl: Table) -> List[Tuple[str, str]]:
+    """Extract (link, table id) in a table"""
+    urls = set()
+    for ri, row in enumerate(tbl.rows):
+        for ci, cell in enumerate(row.cells):
+            urls = urls.union((x.wikipedia_url for x in extract_cell_links(cell)))
+    return [(get_title_from_url(url), tbl.id) for url in urls]
+
+
+def merge_link_to_table(
+    x: Tuple[str, Tuple[Table, Union[None, Iterable[Tuple[str, Union[str, None]]]]]]
+) -> LinkedHTMLTable:
+    tbl_id, (tbl, title_and_wd) = x
+    if title_and_wd is None:
+        # happen when the table has no links
+        title2wd = {}
+    else:
+        title2wd = dict(title_and_wd)
+
+    links: Dict[Tuple[int, int], List[WikiLink]] = {}
+
+    if title_and_wd is not None:
+        for ri, ci, cell in tbl.enumerate_cells():
+            cell_links = extract_cell_links(cell)
+            if len(cell_links) > 0:
+                links[ri, ci] = []
+
+            for link in cell_links:
+                title = get_title_from_url(link.wikipedia_url)
+                link.wikidata_id = title2wd[title]
+                links[ri, ci].append(link)
+
+    return LinkedHTMLTable(
+        table=tbl,
+        links=links,
+    )
+
+
+def extract_cell_links(cell: Cell) -> List[WikiLink]:
+    links = []
+
+    value = cell.value
+    for eid in value.iter_element_id():
+        if value.get_element_tag_by_id(eid) == "a":
+            href = value.get_element_attr_by_id(eid, "href")
+            if href is not None and is_wikipedia_url(href):
+                el = value.get_element_by_id(eid)
+                link = WikiLink(
+                    start=el.start, end=el.end, wikipedia_url=href, wikidata_id=None
+                )
+                links.append(link)
+
+    return links
diff --git a/kgdata/wikipedia/datasets/relational_tables.py b/kgdata/wikipedia/datasets/relational_tables.py
index b21adf7..cc3e49a 100644
--- a/kgdata/wikipedia/datasets/relational_tables.py
+++ b/kgdata/wikipedia/datasets/relational_tables.py
@@ -1,35 +1,51 @@
-from kgdata.wikipedia.datasets.html_tables import html_tables
-import orjson
-from typing import TypedDict, Union
-from loguru import logger
+from kgdata.wikipedia.datasets.html_tables import deser_table, html_tables, ser_table
 from kgdata.dataset import Dataset
 from kgdata.spark import does_result_dir_exist
-from kgdata.wikipedia.datasets.html_articles import html_articles
 from kgdata.wikipedia.config import WPDataDirConfig
-from kgdata.wikipedia.models.html_article import HTMLArticle
-from table_extractor.table_extractor import HTMLTableExtractor
-from table_extractor.models.html_table import HTMLTable
+from rsoup.rsoup import Table
 import sm.misc as M
 
 
-def relational_tables():
+def relational_tables() -> Dataset[Table]:
     cfg = WPDataDirConfig.get_instance()
 
-    tables = html_tables().get_rdd().take(1)
-    table = tables[0]
-    return tables
-
-
-def is_relational_table(tbl: HTMLTable) -> bool:
+    if not does_result_dir_exist(cfg.relational_tables):
+        (
+            html_tables()
+            .get_rdd()
+            .filter(is_relational_table)
+            .map(ser_table)
+            .coalesce(1024, shuffle=True)
+            .saveAsTextFile(
+                str(cfg.relational_tables),
+                compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec",
+            )
+        )
+
+    return Dataset(
+        file_pattern=cfg.relational_tables / "*.gz",
+        deserialize=deser_table,
+    )
+
+
+def is_relational_table(tbl: Table) -> bool:
     if len(tbl.rows) == 0:
         return False
 
-    if not all("th" in c.elements for c in tbl.rows[0].cells):
-        return False
+    rows = tbl.rows
+    n_headers = 0
+    for i in range(len(rows) - 1):
+        if not all(c.is_header for c in rows[i].cells):
+            break
+        n_headers += 1
 
-    if not all(not c.is_header for r in tbl.rows[1:] for c in r.cells):
+    if n_headers == 0:
         return False
 
+    for i in range(n_headers, len(rows)):
+        if not all(not c.is_header for c in rows[i].cells):
+            return False
+
     return True
 
 
diff --git a/kgdata/wikipedia/deprecated/__init__.py b/kgdata/wikipedia/deprecated/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/kgdata/wikipedia/deprecated/rdd_datasets.py b/kgdata/wikipedia/deprecated/rdd_datasets.py
deleted file mode 100644
index 2d138d4..0000000
--- a/kgdata/wikipedia/deprecated/rdd_datasets.py
+++ /dev/null
@@ -1,60 +0,0 @@
-import orjson, os
-
-from kgdata.config import WIKIPEDIA_DIR
-from kgdata.spark import get_spark_context, ensure_unique_records, itemgetter
-
-
-"""Containing Spark RDD datasets that is built from extracted files in `wikidump_io` module
-"""
-
-
-def title2groups(
-    infile: str = os.path.join(WIKIPEDIA_DIR, "pages_articles_en/article_groups/*.gz")
-):
-    """Get RDD that maps from article title into a cluster of articles
-
-    Parameters
-    ----------
-    infile : str, optional
-        input file obtained from group_pages function, by default "${DATA_DIR}/wikipedia/pages_articles_en/article_groups/*.gz"
-
-    Returns
-    -------
-    RDD:
-        (title, { "final": (<title>, <wiki_id>), "group": [(<title>, <wiki_id>)] })
-    """
-    sc = get_spark_context()
-    return (
-        sc.textFile(infile)
-        .map(orjson.loads)
-        .flatMap(lambda x: [(title, x) for title, id in x["group"]])
-    )
-
-
-def id2groups(
-    infile: str = os.path.join(WIKIPEDIA_DIR, "pages_articles_en/article_groups/*.gz")
-):
-    """Get RDD that maps from article id into a cluster of articles
-
-    Parameters
-    ----------
-    infile : str, optional
-        input file obtained from group_pages function, by default "${DATA_DIR}/wikipedia/pages_articles_en/article_groups/*.gz"
-
-    Returns
-    -------
-    RDD:
-        (id, { "final": (<title>, <wiki_id>), "group": [(<title>, <wiki_id>)] })
-    """
-    sc = get_spark_context()
-    return (
-        sc.textFile(infile)
-        .map(orjson.loads)
-        .flatMap(lambda x: [(id, x) for title, id in x["group"]])
-    )
-
-
-if __name__ == "__main__":
-    rdd = title2groups()
-    # rdd = id2groups()
-    ensure_unique_records(rdd, itemgetter(0))
diff --git a/kgdata/wikipedia/deprecated/table_parser.py b/kgdata/wikipedia/deprecated/table_parser.py
deleted file mode 100644
index 3e012f6..0000000
--- a/kgdata/wikipedia/deprecated/table_parser.py
+++ /dev/null
@@ -1,137 +0,0 @@
-from parsimonious import Grammar, NodeVisitor
-from parsimonious.expressions import (
-    ZeroOrMore,
-    OneOf,
-    OneOrMore,
-    Optional as PEGOptional,
-    Sequence as PEGSequence,
-)
-
-from kgdata.wikipedia.deprecated.wikimodels import Table, Row, Cell
-
-"""
-(DEPRECATED) Module for parsing tables from wikipedia markup.
-
-You should use existing wikiparser to get the result as users see in the page as wikiparser are very lenient to user errors.
-"""
-
-
-class TableVisitor(NodeVisitor):
-    grammar = Grammar(
-        r"""
-        # https://en.wikipedia.org/wiki/Help:Table
-        # define the table, attributes of each mark must be on the same line as the mark, except the "|}" does not have any attrs
-        tbl = _n "{|" attrs? _e rows "|}" _n
-        
-        # rows
-        rows = first_row? row*
-        first_row = (row_cells)+
-        row = "|-" _ attrs? _e (row_cells)*
-        row_cells = header_cells / data_cells
-        # the header row is determined by the first ! character, even if the rest is || it still recognized as header! that's why we should use the cell delimiter instead of just !!
-        header_cells = !"|-" "!" cell (same_row_cell)* _n
-        data_cells = !"|-" !"|}" "|" cell (same_row_cell)* _n
-        
-        # cells. Note that there is one exception in the wiki table parser that || after the value of previous multiline cell (e.g., `cell 2 \n continue || cell 3`) is still belong
-        # to the previous cell, which against its documentation that || is equal to `\n|` so we decide to stick to the doc as it's easier for us
-        cell = (attrs "|")? cell_value
-        same_row_cell = _cell_delimiter cell
-        cell_value = tbl / ~r"((?!(!!|\|\||\|-|\|}|(\n *\|)|(\n *!))).)*"s
-        _cell_delimiter = "!!" / "||"
-        
-        # html attributes
-        attrs = (_ ident _ '=' _ attr_value _)+
-
-        # basic expr
-        ident = ~r"[a-z0-9]*"i
-        # sometimes they don't use quoted for attribute values as they should be
-        attr_value = ~'"[^\"]+"' / ~"'[^\']+'" / ~"[a-z0-9%]+"i
-        
-        # define space only, new line only, and space with new line
-        # space
-        _ = ~r"[ \t]*"
-        # space and new line
-        _n = ~r"[ \n]*"
-        # end row
-        _e = _ "\n" _n
-    """
-    )
-
-    def visit_tbl(self, node, children):
-        (_n, otag, attrs, _e, rows, ctag, _n) = children
-        return Table(rows=rows, attrs=attrs or {}, caption=None)
-
-    def visit_attrs(self, node, children):
-        # extract html attributes
-        attr_seq = children
-        attrs = {}
-        for attr in attr_seq:
-            _, ident, _, _eq, _, attr_value, _ = attr
-            attrs[ident.text] = attr_value
-        return attrs
-
-    def visit_attr_value(self, node, children):
-        if node.text.startswith("'") or node.text.startswith('"'):
-            return node.text[1:-1]
-        return node.text
-
-    def visit_rows(self, node, children):
-        op_first_row, rep_rows = children
-        rows = []
-        if op_first_row is not None:
-            rows.append(op_first_row)
-        rows += rep_rows
-        return rows
-
-    def visit_first_row(self, node, children):
-        # an array of cells (2D) so have to flatten
-        row = Row(cells=[c for cs in children for c in cs])
-        return row
-
-    def visit_row(self, node, children):
-        _l, _, attrs, _e, cells = children
-        # an array of cells (2D) so have to flatten
-        return Row(cells=[c for cs in cells for c in cs], attrs=attrs or {})
-
-    def visit_header_cells(self, _node, children):
-        _l, _l, cell_with_attrs, rep_cells, _n = children
-        data = [cell_with_attrs]
-        data[0].is_header = True
-        for cell in rep_cells:
-            if cell.is_literal():
-                cell.is_header = True
-            data.append(cell)
-        return data
-
-    def visit_data_cells(self, _node, children):
-        _l, _l, _l, cell_with_attrs, rep_cells, _n = children
-        data = [cell_with_attrs] + rep_cells
-        return data
-
-    def visit_cell(self, _node, children):
-        opt_attr, cell_value = children
-        if opt_attr is not None:
-            attrs, _bar = opt_attr
-        else:
-            attrs = {}
-        return Cell(value=cell_value, attrs=attrs)
-
-    def visit_same_row_cell(self, node, children):
-        return children[1]
-
-    def visit_cell_value(self, _node, children):
-        tbl_or_data = children[0]
-        if isinstance(tbl_or_data, Table):
-            return tbl_or_data
-        return tbl_or_data.text.strip()
-
-    def generic_visit(self, node, children):
-        if isinstance(node.expr, (ZeroOrMore, OneOrMore, PEGSequence)):
-            return children
-        if isinstance(node.expr, OneOf):
-            return children[0]
-        if isinstance(node.expr, PEGOptional):
-            if len(children) == 0:
-                return None
-            return children[0]
-        return node
diff --git a/kgdata/wikipedia/deprecated/wikidump_io.py b/kgdata/wikipedia/deprecated/wikidump_io.py
deleted file mode 100644
index e6f093b..0000000
--- a/kgdata/wikipedia/deprecated/wikidump_io.py
+++ /dev/null
@@ -1,451 +0,0 @@
-from dataclasses import asdict
-import glob
-import gzip
-import os
-from hugedict.parallel import Parallel
-from kgdata.config import DEFAULT_DATA_DIR
-import orjson
-import ujson
-
-from kgdata.wikipedia.deprecated.wikimodels import *
-from sm.misc.deser import get_open_fn
-from pathlib import Path
-from tqdm import tqdm
-import xml.etree.ElementTree as ET
-
-"""Module containing code for reading and extracting wikipedia dump from DBPedia at 2016 so it is easier for us to work with. 
-
-This module is deprecated.
-"""
-
-logger = logging.getLogger("wikipedia")
-
-
-def iter_page_articles(compressed_file: str):
-    """Iterate through each page and yield the content.
-
-    Parameters
-    ----------
-    compressed_file : str
-        path to the `page_articles_en.xml.bz2`. Must contains only one latest revision per page. The compression format is going to be determined from the extension (.gz or .bz2)
-    """
-    ET.register_namespace("", "http://www.mediawiki.org/xml/export-0.10/")
-
-    with get_open_fn(compressed_file)(compressed_file, "r") as f:
-        tree = iter(
-            ET.iterparse(
-                f,
-                events=(
-                    "start",
-                    "end",
-                ),
-            )
-        )
-        event, root = next(tree)
-
-        for event, elem in tree:
-            if (
-                event != "end"
-                or elem.tag != "{http://www.mediawiki.org/xml/export-0.10/}page"
-            ):
-                continue
-
-            id = elem.find("{http://www.mediawiki.org/xml/export-0.10/}id").text
-            ns = elem.find("{http://www.mediawiki.org/xml/export-0.10/}ns").text
-            title = elem.find("{http://www.mediawiki.org/xml/export-0.10/}title").text
-            redirect_title = elem.find(
-                "{http://www.mediawiki.org/xml/export-0.10/}redirect"
-            )
-            if redirect_title is not None:
-                redirect_title = redirect_title.get("title")
-            model = elem.find(
-                "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}model"
-            ).text
-            format = elem.find(
-                "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}format"
-            ).text
-            text = (
-                elem.find(
-                    "{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}text"
-                ).text
-                or ""
-            )
-
-            yield WikiPageArticle(id, ns, title, redirect_title, model, format, text)
-
-            # Article to avoid using too much memory: https://www.ibm.com/developerworks/xml/library/x-hiperfparse/
-            # elem.clear()
-            # leverage the fact that pages_articles if mediawiki > page
-            root.remove(elem)
-
-        del tree
-
-
-def split_page_articles(
-    bz2_infile: Union[Path, str], outdir: Union[Path, str], pages_per_file: int = 64000
-):
-    """Split the big `page_articles_en.xml.bz2` to multiple files (compressed in gzip) for faster processing
-
-    Parameters
-    ----------
-    bz2_infile : str
-        path to the `page_articles_en.xml.bz2`
-    outdir : str
-        path to the out files. the splitted files will automatically be `data_<pages_per_files>.<chunk_id>` before the `gz` extension
-    pages_per_file : int, optional
-        numbers of pages per file, by default 50000
-    """
-    # get human readable size
-    ppf = pages_per_file
-    units = ["", "k", "m", "b"]
-    i = 0
-    while ppf >= 1000:
-        ppf = ppf / 1000
-        i += 1
-
-    assert ppf == int(ppf), "Please choose a better number"
-    assert Path(outdir).exists(), "Invalid output directory"
-
-    ppf = f"{int(ppf)}{units[i]}"
-    filename_format = os.path.join(outdir, f"data_{ppf}.%05d.gz")
-
-    # this seems to be the only way to specify default namespace in python 3.7
-    ET.register_namespace("", "http://www.mediawiki.org/xml/export-0.10/")
-
-    with tqdm(desc="split_page_articles") as pbar:
-        with get_open_fn(bz2_infile)(bz2_infile, "r") as f:
-            batch = []
-            file_counter = 0
-
-            tree = iter(
-                ET.iterparse(
-                    f,
-                    events=(
-                        "start",
-                        "end",
-                    ),
-                )
-            )
-            event, root = next(tree)
-
-            for event, elem in tree:
-                if (
-                    event != "end"
-                    or elem.tag != "{http://www.mediawiki.org/xml/export-0.10/}page"
-                ):
-                    continue
-
-                batch.append(elem)
-                pbar.update(1)
-
-                # avoid using too much memory, remove the elem. leverage the fact that pages_articles if mediawiki > page
-                root.remove(elem)
-
-                if len(batch) > pages_per_file:
-                    subtree = ET.Element("mediawiki")
-                    for el in batch:
-                        subtree.append(el)
-                    outfile = filename_format % file_counter
-                    with gzip.open(outfile, "wb") as f:
-                        ET.ElementTree(subtree).write(f, encoding="utf-8")
-                    batch = []
-                    del subtree
-                    file_counter += 1
-
-            if len(batch) > 0:
-                subtree = ET.Element("mediawiki")
-                for el in batch:
-                    subtree.append(el)
-                outfile = filename_format % file_counter
-                with gzip.open(outfile, "wb") as f:
-                    ET.ElementTree(subtree).write(f, encoding="utf-8")
-                file_counter += 1
-                batch = []
-                del subtree
-
-
-def extract_raw_tables(
-    infile: str, outfile: str, max_pages: Optional[int] = None, report: bool = False
-):
-    """Extract tables from infile, the infile has to be compatible with the one using in the `iter_page_articles` function.
-
-    Parameters
-    ----------
-    infile : str
-        path to the `page_articles_en.xml`. Must contains only one latest revision per page. The compression format is going to be determined from the extension (.gz or .bz2)
-    outfile : str
-        path to the output file, compressed using gzip and in JSON format
-    max_pages : Optional[int], optional
-        [description], by default None
-    """
-    if max_pages is None:
-        max_pages = float("inf")
-    if report:
-        pbar = tqdm(desc=f"Extract tables in file {Path(infile).stem}")
-
-    with gzip.open(outfile, "wb") as f:
-        counter = 0
-        for page in iter_page_articles(infile):
-            tables = _extract_raw_tables(page)
-            if len(tables) == 0:
-                continue
-
-            record = WikiPageExtractedTables(
-                page.id, page.ns, page.title, page.text, tables
-            )
-            f.write(ujson.dumps(asdict(record)).encode())
-            f.write(b"\n")
-            counter += 1
-
-            if report:
-                pbar.update(1)
-
-            if counter >= max_pages:
-                break
-
-    if report:
-        pbar.close()
-
-
-def _extract_raw_tables(
-    page: WikiPageArticle, silent: bool = True, log_error: bool = False
-):
-    """Extract tables from an article. This function does not return nested tables as it is included in the outer table
-
-    Parameters
-    ----------
-    page : WikiPageArticle
-        markup content of a wikipedia article
-    silent: bool
-        whether throwing error when encounter inconsistent open/close tag
-    log_error: bool
-        whether we will log the error
-
-    Returns
-    -------
-    List[str]
-        list of the raw tables in wikipedia
-    """
-    otag, ctag = "{|", "|}"
-    tables = []
-    tag_indices = []
-    text = page.text
-
-    for m in re.finditer(r"{\|", text):
-        tag_indices.append((m.start(0), otag))
-
-    # no specified in any wikipedi
-    for m in re.finditer(r"\|}(?!})", text):
-        tag_indices.append((m.start(0), ctag))
-
-    tag_indices.sort(key=itemgetter(0))
-    if len(tag_indices) == 0:
-        return tables
-
-    # in case there is an error in the markup
-    i = 0
-    while i < len(tag_indices) and tag_indices[i][1] == ctag:
-        i += 1
-
-    if i >= len(tag_indices):
-        if log_error:
-            logger.debug(
-                "tag-close inconsistent: url=%s\ntags=%s\n-------content=%s\n-------",
-                page.url,
-                tag_indices,
-                text,
-            )
-        if not silent:
-            raise Exception("Inconsistent between open tag and close tag")
-
-        return tables
-
-    stack = [tag_indices[i][0]]
-    for idx, tag in islice(tag_indices, i + 1, None):
-        if tag == otag:
-            stack.append(idx)
-        else:
-            # close tag
-            if len(stack) == 0:
-                # error in the markup, encounter it on: https://en.wikipedia.org/wiki/Template:Manx_Electric_Railway?curid=14227720
-                if log_error:
-                    logger.debug(
-                        "tag-close inconsistent: url=%s\ntags=%s\n-------content=%s\n-------",
-                        page.url,
-                        tag_indices,
-                        text,
-                    )
-                if not silent:
-                    raise Exception("Inconsistent between open tag and close tag")
-            elif len(stack) == 1:
-                # yield the most outer table
-                tables.append(text[stack.pop() : idx + 2])
-            else:
-                stack.pop()
-
-    return tables
-
-
-def iter_raw_tables(infile: str):
-    """Iterate through each page that contains extracted raw tables. Normally, we use this function after run `extract_raw_tables`
-
-    Parameters
-    ----------
-    infile : str
-        input file
-    """
-    with get_open_fn(infile)(infile, "rb") as f:
-        for line in f:
-            yield WikiPageExtractedTables(**ujson.loads(line))
-
-
-def extract_page_identifications(infile: str, outfile: str):
-    with get_open_fn(outfile)(outfile, "wb") as f:
-        for page in iter_page_articles(infile):
-            f.write(
-                orjson.dumps(
-                    {
-                        "id": int(page.id),
-                        "ns": page.ns,
-                        "redirect_title": page.redirect_title,
-                        "title": page.title,
-                    }
-                )
-            )
-            f.write(b"\n")
-
-
-def group_pages(
-    indir: Union[Path, str], outdir: Union[Path, str], batch_size: int = 64000
-):
-    """Group wikipedia pages/articles that are belong to the same entity
-
-    Parameters
-    ----------
-    infile : str
-        patterns to find extracted pages
-    outdir : str
-        output directory
-    batch_size: int
-        number of records per file
-    """
-    infiles = sorted(glob.glob(os.path.join(indir, "*.gz")))
-    Path(outdir).mkdir(exist_ok=True)
-
-    wiki_links = []
-    for infile in tqdm(infiles, desc="read file"):
-        with get_open_fn(infile)(infile, "rb") as f:
-            for line in f:
-                r = orjson.loads(line)
-                wiki_links.append((r["id"], r["title"], r["redirect_title"]))
-
-    # verify if we have the case of one source node is link to two target nodes, then we build dict that manually curate those nodes
-    tmp = {}
-    manually_curated_source2target = {}
-    title2id = {}
-    for source_id, source, target in tqdm(wiki_links):
-        if source not in tmp:
-            tmp[source] = target
-            title2id[source] = source_id
-        else:
-            assert source not in manually_curated_source2target
-            if target is None:
-                manually_curated_source2target[source] = tmp[source]
-                # don't have to update the id since this we discard this article
-            else:
-                manually_curated_source2target[source] = target
-                title2id[source] = source_id
-
-            print("`%s` | `%s` | `%s`" % (source, target, tmp[source]))
-
-    # build reverse map
-    reverse_map = {}
-    leaves = set()
-    for source_id, source, target in tqdm(wiki_links, desc="build reverse map"):
-        if source in manually_curated_source2target:
-            continue
-
-        if target is None:
-            assert source not in leaves
-            leaves.add(source)
-            continue
-
-        if target not in reverse_map:
-            reverse_map[target] = [source]
-        else:
-            reverse_map[target].append(source)
-
-    for source, target in manually_curated_source2target.items():
-        if target is None:
-            leaves.add(source)
-            continue
-        if target not in reverse_map:
-            reverse_map[target] = [source]
-        else:
-            reverse_map[target].append(source)
-
-    # now travel upward to group
-    visited = set()
-
-    def trace_upward(reverse_map, group, ptr):
-        assert ptr not in visited
-        visited.add(ptr)
-
-        for parent in reverse_map.get(ptr, []):
-            group.append((parent, title2id[parent]))
-            trace_upward(reverse_map, group, parent)
-
-    groups = []
-    for leaf in tqdm(leaves, desc="grouping"):
-        if leaf not in reverse_map:
-            groups.append(
-                {"final": (leaf, title2id[leaf]), "group": [(leaf, title2id[leaf])]}
-            )
-        else:
-            group = [(leaf, title2id[leaf])]
-            trace_upward(reverse_map, group, leaf)
-            groups.append({"group": group, "final": (leaf, title2id[leaf])})
-
-    # write result
-    count = 0
-    for i in tqdm(range(0, len(groups), batch_size), desc="writing result"):
-        with gzip.open(os.path.join(outdir, "data.%05d.gz" % count), "wb") as f:
-            for g in groups[i : i + batch_size]:
-                f.write(orjson.dumps(g))
-                f.write(b"\n")
-            count += 1
-
-
-if __name__ == "__main__":
-    count = 0
-    data_dir = DEFAULT_DATA_DIR / "wikipedia-20211201"
-    (data_dir / "step_2").mkdir(exist_ok=True)
-
-    # split_page_articles(
-    #     bz2_infile=list((DEFAULT_DATA_DIR / "wikipedia-20211201/step_0").glob("*.bz2"))[
-    #         0
-    #     ],
-    #     outdir=data_dir / "step_1",
-    #     pages_per_file=64000,
-    # )
-
-    # pp = Parallel()
-    # pp.map(
-    #     extract_page_identifications,
-    #     inputs=[
-    #         (infile, data_dir / "step_2" / infile.name)
-    #         for infile in tqdm((data_dir / "step_1").glob("*.gz"))
-    #     ],
-    #     show_progress=True,
-    # )
-
-    # group_pages(
-    #     indir=data_dir / "step_2",
-    #     outdir=data_dir / "step_3",
-    # )
-
-    testfile = data_dir / "step_3/data.00000.gz"
-    with get_open_fn(testfile)(testfile, "r") as f:
-        for line in f:
-            print(orjson.loads(line))
-            break
diff --git a/kgdata/wikipedia/deprecated/wikimodels.py b/kgdata/wikipedia/deprecated/wikimodels.py
deleted file mode 100644
index 67d6c96..0000000
--- a/kgdata/wikipedia/deprecated/wikimodels.py
+++ /dev/null
@@ -1,78 +0,0 @@
-import logging
-from dataclasses import dataclass, field
-from typing import *
-from urllib.parse import urlparse, unquote_plus
-
-"""Module containing code for parsing wikipedia dump from DBPedia at 2016. This module is deprecated as we do not parse the data directly any more
-"""
-
-logger = logging.getLogger("wikipedia")
-
-
-@dataclass
-class WikiPageArticle:
-    # Original schema: https://www.mediawiki.org/xml/export-0.10.xsd
-    # properties of PageType
-    # page.id
-    id: int
-    # page.ns
-    ns: str
-    # page.title
-    title: str
-    # page.redirect.title (since RedirectType only has title property), flag this current revision is a redirect
-    redirect_title: Optional[str]
-
-    # properties of revision
-    # page.revision.model
-    model: str
-    # page.revision.format
-    format: str
-    # page.revision.text
-    text: str
-
-    @property
-    def url(self):
-        return f"https://en.wikipedia.org/wiki/{self.title.replace(' ', '_')}?curid={self.id}"
-
-
-@dataclass
-class WikiPageExtractedTables:
-    id: int
-    ns: str
-    title: str
-    text: str
-    raw_tables: List[str]
-
-    @property
-    def url(self):
-        return f"https://en.wikipedia.org/wiki/{self.title.replace(' ', '_')}?curid={self.id}"
-
-
-@dataclass
-class Cell:
-    value: Union[str, "Table"]
-    attrs: Dict[str, str] = field(default_factory=dict)
-    is_header: bool = False
-
-    def is_literal(self):
-        return isinstance(self.value, (str,))
-
-
-@dataclass
-class Row:
-    cells: List[Cell]
-    attrs: Dict[str, str] = field(default_factory=dict)
-
-
-@dataclass
-class Table:
-    rows: List[Row]
-    caption: Optional[str] = None
-    attrs: Dict[str, str] = field(default_factory=dict)
-
-    @staticmethod
-    def parse(text: str):
-        # lazy import here to avoid circular import
-        from kgdata.wikipedia.deprecated.table_parser import TableVisitor
-
-        return TableVisitor().parse(text)
diff --git a/kgdata/wikipedia/misc.py b/kgdata/wikipedia/misc.py
index d7a8b89..4076f3c 100644
--- a/kgdata/wikipedia/misc.py
+++ b/kgdata/wikipedia/misc.py
@@ -1,3 +1,4 @@
+from typing import Optional
 from urllib.parse import unquote_plus, urlparse
 
 
@@ -18,3 +19,14 @@ def get_title_from_url(url: str) -> str:
     path = path[6:]
     title = unquote_plus(path).replace("_", " ")
     return title.strip()
+
+
+def is_wikipedia_url(url: str, lang: Optional[str] = None) -> bool:
+    """This function checks if a URL is a wikipedia URL.
+
+    Args:
+        url: A URL.
+        lang: A 2-char language code (e.g., en). If None, all languages are acceptable.
+    """
+    host = "wikipedia.org" if lang is None else f"{lang}.wikipedia.org"
+    return urlparse(url).netloc.endswith(host)
diff --git a/kgdata/wikipedia/models/html_article.py b/kgdata/wikipedia/models/html_article.py
index 53b7242..5fd6720 100644
--- a/kgdata/wikipedia/models/html_article.py
+++ b/kgdata/wikipedia/models/html_article.py
@@ -78,6 +78,23 @@ def from_dump_dict(o: dict) -> HTMLArticle:
             wikitext=o["article_body"]["wikitext"],
         )
 
+    def to_dict(self) -> dict:
+        return {
+            "name": self.name,
+            "page_id": self.page_id,
+            "date_modified": self.date_modified,
+            "url": self.url,
+            "lang": self.lang,
+            "wdentity": self.wdentity,
+            "additional_entities": [ent.to_dict() for ent in self.additional_entities],
+            "is_part_of": self.is_part_of,
+            "categories": [o.to_dict() for o in self.categories],
+            "templates": [o.to_dict() for o in self.templates],
+            "redirects": [o.to_dict() for o in self.redirects],
+            "html": self.html,
+            "wikitext": self.wikitext,
+        }
+
 
 @dataclass
 class AdditionalEntity:
@@ -85,8 +102,18 @@ class AdditionalEntity:
     url: str
     aspects: List[str]
 
+    def to_dict(self) -> dict:
+        return {
+            "identifier": self.identifier,
+            "url": self.url,
+            "aspects": self.aspects,
+        }
+
 
 @dataclass
 class NameAndURL:
     name: str
     url: str
+
+    def to_dict(self) -> dict:
+        return {"name": self.name, "url": self.url}
diff --git a/kgdata/wikipedia/models/linked_html_table.py b/kgdata/wikipedia/models/linked_html_table.py
new file mode 100644
index 0000000..7c25b8a
--- /dev/null
+++ b/kgdata/wikipedia/models/linked_html_table.py
@@ -0,0 +1,50 @@
+from __future__ import annotations
+import copy, orjson
+from dataclasses import asdict, dataclass
+from typing import Dict, List, Optional, Tuple, Union
+from rsoup.rsoup import Table
+
+
+@dataclass
+class LinkedHTMLTable:
+    table: Table
+    # mapping from (row, col) => links)
+    links: Dict[Tuple[int, int], List[WikiLink]]
+
+    def to_json(self) -> bytes:
+        links = [
+            [ri, ci, [l.to_dict() for l in links]]
+            for (ri, ci), links in self.links.items()
+        ]
+        return orjson.dumps([self.table.to_base64(), links])
+
+    @staticmethod
+    def from_json(s: Union[str, bytes]) -> LinkedHTMLTable:
+        tbl, links = orjson.loads(s)
+        links = {
+            (ri, ci): [WikiLink.from_dict(l) for l in links] for ri, ci, links in links
+        }
+        return LinkedHTMLTable(
+            table=Table.from_base64(tbl),
+            links=links,
+        )
+
+
+@dataclass
+class WikiLink:
+    start: int
+    end: int  # exclusive
+    wikipedia_url: str
+    wikidata_id: Optional[str] = None
+
+    def to_dict(self) -> dict:
+        return {
+            "start": self.start,
+            "end": self.end,
+            "wikipedia_url": self.wikipedia_url,
+            "wikidata_id": self.wikidata_id,
+        }
+
+    @staticmethod
+    def from_dict(o: dict):
+        return WikiLink(**o)
diff --git a/kgdata/wikipedia/models/page_article.py b/kgdata/wikipedia/models/page_article.py
index c77e55b..4f52b24 100644
--- a/kgdata/wikipedia/models/page_article.py
+++ b/kgdata/wikipedia/models/page_article.py
@@ -28,3 +28,26 @@ class WikiPageArticle:
     @property
     def url(self):
         return f"{WikipediaConfig.WikiURL}/wiki/{self.title.replace(' ', '_')}?curid={self.id}"
+
+    @staticmethod
+    def from_dict(o: dict):
+        return WikiPageArticle(
+            id=o["id"],
+            ns=o["ns"],
+            title=o["title"],
+            redirect_title=o["redirect_title"],
+            model=o["model"],
+            format=o["format"],
+            text=o["text"],
+        )
+
+    def to_dict(self):
+        return {
+            "id": self.id,
+            "ns": self.ns,
+            "title": self.title,
+            "redirect_title": self.redirect_title,
+            "model": self.model,
+            "format": self.format,
+            "text": self.text,
+        }
diff --git a/kgdata/wikipedia/prelude.py b/kgdata/wikipedia/prelude.py
index 8b583bc..2e334e1 100644
--- a/kgdata/wikipedia/prelude.py
+++ b/kgdata/wikipedia/prelude.py
@@ -1,5 +1,2 @@
 from .models import *
-from .deprecated.wikidump_io import *
-from .deprecated.table_parser import *
-from .deprecated.rdd_datasets import *
 from .misc import *
diff --git a/pyproject.toml b/pyproject.toml
index b7bf86c..94481d4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "kgdata"
-version = "2.3.3"
+version = "3.0.3"
 description = "Library to process dumps of knowledge graphs (Wikipedia, DBpedia, Wikidata)"
 authors = ["Binh Vu <binh@toan2.com>"]
 license = "MIT"
@@ -8,16 +8,13 @@ homepage = "https://github.com/binh-vu/kgdata"
 repository = "https://github.com/binh-vu/kgdata"
 readme = "README.md"
 
-[tool.poetry.scripts]
-kgdata = 'kgdata.__main__:cli'
-
 [tool.poetry.dependencies]
 python = "^3.8"
 orjson = "^3.6.8"
 tqdm = "^4.64.0"
 beautifulsoup4 = "^4.9.3"
 cityhash = "^0.2.3"
-pyspark = "3.0.3"
+pyspark = "3.3.0"
 loguru = ">=0.6.0"
 rdflib = "^6.1.1"
 six = "^1.16.0"
@@ -31,9 +28,10 @@ requests = "^2.28.0"
 sem-desc = "^3.5.2"
 click = "^8.1.3"
 parsimonious = "^0.8.1"
-hugedict = "^2.3.0"
-web-table-extractor = "^1.3.0"
+hugedict = "^2.5.1"
+rsoup = "^2.5.1"
 lxml = "^4.9.0"
+pyserini = "^0.17.0"
 
 [tool.poetry.dev-dependencies]
 python-dotenv = ">=0.19.0"
diff --git a/scripts/build.sh b/scripts/build.sh
index bfd0ede..2071e97 100644
--- a/scripts/build.sh
+++ b/scripts/build.sh
@@ -1,8 +1,10 @@
 set -e
 
-kgdata wikidata -b entities -d data/wikidata/20211213 -o data/wikidata/20211213/databases -c -l en
-kgdata wikidata -b classes -d data/wikidata/20211213 -o data/wikidata/20211213/databases -c -l en
-kgdata wikidata -b properties -d data/wikidata/20211213 -o data/wikidata/20211213/databases -c -l en
-kgdata wikidata -b wp2wd -d data/wikidata/20211213 -o data/wikidata/20211213/databases -c -l en
-kgdata wikidata -b entity_labels -d data/wikidata/20211213 -o data/wikidata/20211213/databases -c -l en
-kgdata wikidata -b entity_redirections -d data/wikidata/20211220 -o data/wikidata/20211213/databases -c
+export RUST_LOG=info
+
+python -m kgdata.wikidata entities -d data/wikidata/20211213 -o data/databases/20211213_v2 -c -l en
+python -m kgdata.wikidata entity_labels -d data/wikidata/20211213 -o data/databases/20211213_v2 -c -l en
+python -m kgdata.wikidata classes -d data/wikidata/20211213 -o data/databases/20211213_v2 -c -l en
+python -m kgdata.wikidata properties -d data/wikidata/20211213 -o data/databases/20211213_v2 -c -l en
+python -m kgdata.wikidata wp2wd -d data/wikidata/20211213 -o data/databases/20211213_v2 -c -l en
+python -m kgdata.wikidata entity_redirections -d data/wikidata/20211213 -o data/databases/20211213_v2 -c
diff --git a/kgdata/wikidata/deprecated/__init__.py b/scripts/experiment_wdentity_db_settings/wdentity/__init__.py
similarity index 100%
rename from kgdata/wikidata/deprecated/__init__.py
rename to scripts/experiment_wdentity_db_settings/wdentity/__init__.py
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/gen_data.py b/scripts/experiment_wdentity_db_settings/wdentity/gen_data.py
new file mode 100644
index 0000000..06e8894
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/gen_data.py
@@ -0,0 +1,47 @@
+from genericpath import exists
+from multiprocessing import parent_process
+import ray, glob, orjson
+from pathlib import Path
+from sm.prelude import M
+
+from hugedict.misc import zstd6_compress_custom
+
+ray.init()
+
+
+@ray.remote
+def generate_data(infile: Path, outdir: Path, name: str):
+    (outdir / name).mkdir(exist_ok=True, parents=True)
+
+    if name == "default":
+        value_fn = orjson.dumps
+    elif name == "zstd-6":
+        value_fn = zstd6_compress_custom(orjson.dumps)
+    else:
+        assert False, name
+
+    with M.get_open_fn(outdir / name / infile.name)(
+        outdir / name / infile.name, "wb"
+    ) as f:
+        with M.get_open_fn(infile)(infile, "rb") as g:
+            for line in g:
+                object = orjson.loads(line)
+                key = object["id"].encode()
+                value = value_fn(object)
+
+                f.write(len(key).to_bytes(8, byteorder="little"))
+                f.write(key)
+                f.write(len(value).to_bytes(8, byteorder="little"))
+                f.write(value)
+
+
+def run(name: str):
+    data_dir = Path(__file__).parent.parent / "data"
+    refs = []
+    for file in (data_dir / "entities").glob("*.gz"):
+        refs.append(generate_data.remote(file, data_dir, name))
+    ray.get(refs)
+
+
+# run("default")
+# run("zstd-6")
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/load_db.py b/scripts/experiment_wdentity_db_settings/wdentity/load_db.py
new file mode 100644
index 0000000..1d4513f
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/load_db.py
@@ -0,0 +1,152 @@
+import glob
+from pathlib import Path
+import shutil
+from hugedict.prelude import rocksdb_load, RocksDBOptions, RocksDBCompressionOptions
+from tap import Tap
+
+
+class LoadDBArgs(Tap):
+    name: str
+    option: str
+
+
+db_options = {
+    "default": {"name": "", "opts": RocksDBOptions(create_if_missing=True)},
+    "compress-1": {
+        "name": "--compress-type=zstd",
+        "opts": RocksDBOptions(create_if_missing=True, compression_type="zstd"),
+    },
+    "compress-2": {
+        "name": "--compress-type=lz4",
+        "opts": RocksDBOptions(create_if_missing=True, compression_type="lz4"),
+    },
+    "compress-3": {
+        "name": "--compress-type=zstd-6",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=15,
+                level=6,
+                strategy=0,
+                max_dict_bytes=0,
+                # max_dict_bytes=16 * 1024,
+            ),
+            # zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+    },
+    "compress-4": {
+        "name": "--compress-type=zstd-6-rem",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=15,
+                level=6,
+                strategy=0,
+                max_dict_bytes=16 * 1024,
+            ),
+            zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+    },
+    "compress-5": {
+        "name": "--compress-type=zstd-6-w14",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=14,
+                level=6,
+                strategy=0,
+                max_dict_bytes=0,
+            ),
+        ),
+    },
+    "compress-6": {
+        "name": "--compress-type=zstd-6-w14-rem",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=14,
+                level=6,
+                strategy=0,
+                max_dict_bytes=16 * 1024,
+            ),
+            zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+    },
+    "compress-7": {
+        "name": "--compress-type=bottom-zstd-6-w14-rem",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="lz4",
+            bottommost_compression_type="zstd",
+            bottommost_compression_opts=RocksDBCompressionOptions(
+                window_bits=14,
+                level=6,
+                strategy=0,
+                max_dict_bytes=16 * 1024,
+            ),
+            bottommost_zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+    },
+    "compress-8": {
+        "name": "--compress-type=zstd-6-w14-d10-rem",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=14,
+                level=6,
+                strategy=0,
+                max_dict_bytes=16 * 1024,
+            ),
+            zstd_max_train_bytes=10 * 16 * 1024,
+        ),
+    },
+    "compress-9": {
+        "name": "--compress-type=zstd-3-w14-rem",
+        "opts": RocksDBOptions(
+            create_if_missing=True,
+            compression_type="zstd",
+            compression_opts=RocksDBCompressionOptions(
+                window_bits=14,
+                level=3,
+                strategy=0,
+                max_dict_bytes=16 * 1024,
+            ),
+            zstd_max_train_bytes=100 * 16 * 1024,
+        ),
+    },
+}
+
+
+def load(data_dir: Path, name: str, opt_name: str, opts):
+    dbpath = data_dir / "databases" / (name + opt_name)
+    if dbpath.exists():
+        shutil.rmtree(dbpath)
+
+    rocksdb_load(
+        str(dbpath),
+        opts,
+        glob.glob(str(data_dir / name / "part-*.gz")),
+        {
+            "record_type": {"type": "bin_kv", "key": None, "value": None},
+            "is_sorted": False,
+        },
+        True,
+        True,
+    )
+
+
+if __name__ == "__main__":
+    data_dir = Path(__file__).parent.parent / "data"
+
+    args = LoadDBArgs().parse_args()
+    load(
+        data_dir,
+        args.name,
+        db_options[args.option]["name"],
+        db_options[args.option]["opts"],
+    )
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/load_db.sh b/scripts/experiment_wdentity_db_settings/wdentity/load_db.sh
new file mode 100644
index 0000000..700f724
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/load_db.sh
@@ -0,0 +1,11 @@
+# python benchmarks/wdentity/load_db.py --name default --option default
+# python benchmarks/wdentity/load_db.py --name default --option compress-1
+# python benchmarks/wdentity/load_db.py --name default --option compress-2
+# python benchmarks/wdentity/load_db.py --name default --option compress-3
+# python benchmarks/wdentity/load_db.py --name default --option compress-4
+# python benchmarks/wdentity/load_db.py --name default --option compress-5
+# python benchmarks/wdentity/load_db.py --name default --option compress-6
+# python benchmarks/wdentity/load_db.py --name default --option compress-7
+python benchmarks/wdentity/load_db.py --name default --option compress-8
+python benchmarks/wdentity/load_db.py --name default --option compress-9
+# python benchmarks/wdentity/load_db.py --name zstd-6 --option default
\ No newline at end of file
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/read_speed.py b/scripts/experiment_wdentity_db_settings/wdentity/read_speed.py
new file mode 100644
index 0000000..15dbc54
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/read_speed.py
@@ -0,0 +1,83 @@
+from functools import partial
+from pathlib import Path
+import random
+import orjson
+import ray
+from sm.prelude import M
+from tap import Tap
+from hugedict.misc import zstd6_compress_custom, zstd_decompress_custom
+from hugedict.prelude import RocksDBDict
+from wdentity.load_db import db_options
+
+
+class ReadSpeedArgs(Tap):
+    name: str
+    option: str
+
+
+data_dir = Path(__file__).parent.parent / "data"
+
+
+if not (data_dir / "keys.txt").exists():
+    ray.init()
+
+    @ray.remote
+    def read_keys(infile, rate: float, seed: int):
+        random.seed(seed)
+
+        keys = []
+        with M.get_open_fn(infile)(infile, "rb") as f:
+            for line in f:
+                key = orjson.loads(line)["id"]
+                if random.random() <= rate:
+                    keys.append(key)
+        return keys
+
+    rate = 0.01
+    refs = []
+    for file in (data_dir / "entities").glob("*.gz"):
+        refs.append(read_keys.remote(file, rate, 155))
+
+    lst: list[str] = [k for ref in list(ray.get(refs)) for k in ref]
+    M.serialize_lines(lst, data_dir / "keys.txt")
+
+
+def test_runtime(name: str, option: str):
+    cfg = db_options[option]
+    dbpath = data_dir / "databases" / (name + cfg["name"])
+
+    if name == "zstd-6":
+        deser_value = zstd_decompress_custom(orjson.loads)
+        ser_value = zstd6_compress_custom(orjson.dumps)
+    else:
+        deser_value = orjson.loads
+        ser_value = orjson.dumps
+
+    db = RocksDBDict(
+        str(dbpath),
+        options=cfg["opts"],
+        deser_key=partial(str, encoding="utf-8"),
+        deser_value=deser_value,
+        ser_value=ser_value,
+    )
+    timer = M.Timer()
+    keys = M.deserialize_lines(data_dir / "keys.txt", trim=True, n_lines=10000)
+    with timer.watch():
+        count = 0
+        for key in keys:
+            count += len(db[key])
+
+    rows = []
+    if not (data_dir / "runtime.txt").exists():
+        rows = [["name", "type", "runtime"]]
+    rows.append([name + cfg["name"], "read", timer.get_time()])
+
+    with open(data_dir / "runtime.txt", "a") as f:
+        for row in rows:
+            f.write(",".join(map(str, row)) + "\n")
+
+    return count
+
+
+args = ReadSpeedArgs().parse_args()
+test_runtime(args.name, args.option)
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/read_speed.sh b/scripts/experiment_wdentity_db_settings/wdentity/read_speed.sh
new file mode 100644
index 0000000..d934fdd
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/read_speed.sh
@@ -0,0 +1,16 @@
+export PYTHONPATH=$(pwd)/benchmarks
+
+for i in {1..20}; do
+    echo $i
+    # python benchmarks/wdentity/read_speed.py --name default --option default
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-1
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-2
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-3
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-4
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-5
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-6
+    # python benchmarks/wdentity/read_speed.py --name default --option compress-7
+    python benchmarks/wdentity/read_speed.py --name default --option compress-8
+    python benchmarks/wdentity/read_speed.py --name default --option compress-9
+    # python benchmarks/wdentity/read_speed.py --name zstd-6 --option default
+done
\ No newline at end of file
diff --git a/scripts/experiment_wdentity_db_settings/wdentity/report.ipynb b/scripts/experiment_wdentity_db_settings/wdentity/report.ipynb
new file mode 100644
index 0000000..8206964
--- /dev/null
+++ b/scripts/experiment_wdentity_db_settings/wdentity/report.ipynb
@@ -0,0 +1,307 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 51,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/workspace/hugedict/.venv/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np, seaborn as sns, matplotlib.font_manager as fm\n",
+    "import pandas as pd, os, subprocess\n",
+    "from pathlib import Path\n",
+    "from sm.prelude import M"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fm.fontManager.addfont(\"/home/rook/Dropbox/Fonts/CMU/ttf/cmunss.ttf\")  # set font\n",
+    "sns.set_theme(font='CMU Sans Serif', font_scale=1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 73,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "default--compress-type=zstd-6-w14-rem: 586MB - 34.94% (586/1677)\n",
+      "default--compress-type=zstd-6-rem: 586MB - 34.94% (586/1677)\n",
+      "default--compress-type=bottom-zstd-6-w14-rem: 586MB - 34.94% (586/1677)\n",
+      "default--compress-type=zstd-6-w14-d10-rem: 604MB - 36.02% (604/1677)\n",
+      "default--compress-type=zstd-3-w14-rem: 656MB - 39.12% (656/1677)\n",
+      "default--compress-type=zstd-6-w14: 1228MB - 73.23% (1228/1677)\n",
+      "default--compress-type=zstd-6: 1228MB - 73.23% (1228/1677)\n",
+      "default--compress-type=zstd: 1288MB - 76.80% (1288/1677)\n",
+      "zstd-6: 1396MB - 83.24% (1396/1677)\n",
+      "default--compress-type=lz4: 1667MB - 99.40% (1667/1677)\n",
+      "default: 1677MB - 100.00% (1677/1677)\n"
+     ]
+    }
+   ],
+   "source": [
+    "db2size = {\n",
+    "    dir.name: int(subprocess.check_output([\"du\", \"-hsm\", str(dir.absolute())]).decode().split(\"\\t\")[0])\n",
+    "    for dir in Path(\"benchmarks/data/databases\").iterdir()\n",
+    "}\n",
+    "target = \"default\"\n",
+    "for db, size in sorted(db2size.items(), key=lambda x: x[1]):\n",
+    "    print(f\"{db}: {size}MB - {M.percentage(size, db2size[target])}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 74,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead tr th {\n",
+       "        text-align: left;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead tr:last-of-type th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr>\n",
+       "      <th></th>\n",
+       "      <th colspan=\"3\" halign=\"left\">runtime</th>\n",
+       "      <th>percentage</th>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th></th>\n",
+       "      <th>mean</th>\n",
+       "      <th>sem</th>\n",
+       "      <th>count</th>\n",
+       "      <th></th>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>name</th>\n",
+       "      <th></th>\n",
+       "      <th></th>\n",
+       "      <th></th>\n",
+       "      <th></th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=lz4</th>\n",
+       "      <td>0.428515</td>\n",
+       "      <td>0.007680</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.769434</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default</th>\n",
+       "      <td>0.475079</td>\n",
+       "      <td>0.008573</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.853044</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-6-w14</th>\n",
+       "      <td>0.531722</td>\n",
+       "      <td>0.008675</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.954750</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-6</th>\n",
+       "      <td>0.532959</td>\n",
+       "      <td>0.007505</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.956971</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd</th>\n",
+       "      <td>0.540787</td>\n",
+       "      <td>0.007639</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.971028</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-3-w14-rem</th>\n",
+       "      <td>0.546975</td>\n",
+       "      <td>0.008001</td>\n",
+       "      <td>20</td>\n",
+       "      <td>0.982139</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>zstd-6</th>\n",
+       "      <td>0.556922</td>\n",
+       "      <td>0.008446</td>\n",
+       "      <td>20</td>\n",
+       "      <td>1.000000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-6-w14-d10-rem</th>\n",
+       "      <td>0.572827</td>\n",
+       "      <td>0.012252</td>\n",
+       "      <td>20</td>\n",
+       "      <td>1.028557</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-6-w14-rem</th>\n",
+       "      <td>0.582554</td>\n",
+       "      <td>0.010322</td>\n",
+       "      <td>20</td>\n",
+       "      <td>1.046023</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=zstd-6-rem</th>\n",
+       "      <td>0.595775</td>\n",
+       "      <td>0.011158</td>\n",
+       "      <td>20</td>\n",
+       "      <td>1.069763</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>default--compress-type=bottom-zstd-6-w14-rem</th>\n",
+       "      <td>0.596663</td>\n",
+       "      <td>0.009663</td>\n",
+       "      <td>20</td>\n",
+       "      <td>1.071358</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                               runtime                  \\\n",
+       "                                                  mean       sem count   \n",
+       "name                                                                     \n",
+       "default--compress-type=lz4                    0.428515  0.007680    20   \n",
+       "default                                       0.475079  0.008573    20   \n",
+       "default--compress-type=zstd-6-w14             0.531722  0.008675    20   \n",
+       "default--compress-type=zstd-6                 0.532959  0.007505    20   \n",
+       "default--compress-type=zstd                   0.540787  0.007639    20   \n",
+       "default--compress-type=zstd-3-w14-rem         0.546975  0.008001    20   \n",
+       "zstd-6                                        0.556922  0.008446    20   \n",
+       "default--compress-type=zstd-6-w14-d10-rem     0.572827  0.012252    20   \n",
+       "default--compress-type=zstd-6-w14-rem         0.582554  0.010322    20   \n",
+       "default--compress-type=zstd-6-rem             0.595775  0.011158    20   \n",
+       "default--compress-type=bottom-zstd-6-w14-rem  0.596663  0.009663    20   \n",
+       "\n",
+       "                                             percentage  \n",
+       "                                                         \n",
+       "name                                                     \n",
+       "default--compress-type=lz4                     0.769434  \n",
+       "default                                        0.853044  \n",
+       "default--compress-type=zstd-6-w14              0.954750  \n",
+       "default--compress-type=zstd-6                  0.956971  \n",
+       "default--compress-type=zstd                    0.971028  \n",
+       "default--compress-type=zstd-3-w14-rem          0.982139  \n",
+       "zstd-6                                         1.000000  \n",
+       "default--compress-type=zstd-6-w14-d10-rem      1.028557  \n",
+       "default--compress-type=zstd-6-w14-rem          1.046023  \n",
+       "default--compress-type=zstd-6-rem              1.069763  \n",
+       "default--compress-type=bottom-zstd-6-w14-rem   1.071358  "
+      ]
+     },
+     "execution_count": 74,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pd.read_csv(\"benchmarks/data/runtime.txt\")\n",
+    "tmp = df[['name', 'runtime']].groupby('name').agg([\"mean\", \"sem\", \"count\"]).sort_values((\"runtime\", \"mean\"))\n",
+    "tmp['percentage'] = tmp[('runtime', 'mean')] / tmp[('runtime', 'mean')]['zstd-6']\n",
+    "tmp"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 61,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<seaborn.axisgrid.FacetGrid at 0x7f05472cebb0>"
+      ]
+     },
+     "execution_count": 61,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAABc0AAAHkCAYAAADsPwdhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKcklEQVR4nO3deXhU5dk/8DuLQNgMi6CiqFDBgkWDgChUQX2rFVRcwFYtitYq6g9p3SuKFsUNrVXf1mqtS1WqVq11qSi1dkGtaFlaFRcEyyZIAkQiAknO74+OeY1hTUImk3w+18XlzJnnnLnP5Nw543dOnslKkiQJAAAAAAAgstNdAAAAAAAA1BdCcwAAAAAASBGaAwAAAABAitAcAAAAAABShOYAAAAAAJAiNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApuekuoL4oKyuPoqKSdJcBAAAAAMA2ssMOrTY7xpXmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACk1KvQvLi4OMaMGRO9e/eOIUOGxNSpUzc5/o033ogTTzwxCgoK4pBDDokPPvigjioFAAAAAKAhyk13AV921VVXRW5ubkybNi0+/PDDOOOMM6Jz587RrVu3KmPfeOONOPfcc+OGG26Igw8+OJYsWRJ5eXlpqBoAAAAAgIYiK0mSJN1FREQsXbo0DjnkkHjppZeiY8eOERFx/fXXx9q1a2P8+PFVxg8dOjSOPfbYOOOMM2rl+cvKyqOoqKRWtgUAAAAAQP2zww6tNjum3kzPMmfOnGjbtm1FYB4R0atXr5g5c2aVsbNnz4758+fHcccdV4cVAgAAAADQ0NWb6VmKiooiPz+/0rL8/PxYvnx5lbH/+te/Yo899og//vGP8cADD8SSJUvipJNOiksuuaRGNeTm1pvPEAAAAAAASIN6E5pHRGRnVw6tc3JyYu3atVXGrVq1KubNmxeffPJJPPPMM/Gf//wnRo4cGd27d49hw4ZV87mzok2bFtVaFwAAAACAhqHehOZNmzaN9evXV1q2bt26aNas2QbH7rzzznH++edHRESXLl1i+PDh8cILL1Q7NC8vT6K4+LNqrQsAAAAANAyLFi2MX//67oiIOP30M6NTp13SXBG1aUsunK43oXm7du2isLCw0rLCwsLYaaedqozt2LFj7LDDDpWW7bzzzvHqq6/WqIbS0vIarQ8AAAAAZLZf//pXMWvWjIiIuPfeX8Wll16Z5oqoa/VmEu8ePXpESUlJzJ07t2LZjBkzYq+99qoytqCgIObOnVvpyvTly5fHLrv41AcAAAAAqL5FixZU3F64cMEmRtJQ1ZvQvFWrVjF06NCYNGlSlJSUxKxZs+Lpp5+OESNGVBnbqVOn2G+//eKOO+6I8vLymDNnTjz88MMbHAsAAAAAAFuq3oTmERHjxo2L5s2bx6BBg+L888+PCRMmRM+ePSMiYsiQIfGHP/yhYuzEiRNj/vz5ccABB8SFF14YEyZMiH79+qWrdAAAAIBNWrx4UUyceFVMnHhVLF68KN3lALARWUmSJOkuoj4oKyuPoqKSdJcBAAAANFDXXfeTmDXrnxERse++vc2TDPXUeeedGcuXfxIREe3b7xB33HF3miuiNu2wQ6vNjqlXV5oDAAAANFTmSQbIDEJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABAitAcAAAAAABSctNdAAAA9dfixYvivvvujoiI0047M3beuVOaKwIAaHyys7MiOzsr3WU0GllZWZVu5+a67rgulZcnUV6epLUGoTkAABt1//33xOzZMyMi4oEH7olLL70yvQUBADQy2dlZ0SY/L7JzctJdSqPx5Q8osrOzok2bFmmspvEpLyuLFSvXpDU4F5oDALBRixYtqLi9cOGCTYwEAGBbyM7OiuycnHjukXuiaNmSdJfTKJQUr6p0+8Hbr0ljNY1L2w47xZEnnhHZ2VlCcwAAAABg44qWLYlli13EUBfKykor3fa6Nz4m5AEAAAAAgBShOQAAAAAApJieBYC0WLx4Udx3390REXHaaWfGzjt3SnNFAAAAAK40ByBN7r//npg9e2bMnj0zHnjgnnSXAwAAABARQnMA0mTRov/7IpWFC32pCgAAAFA/CM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABASm66CwAAAADSIzs7K7Kzs9JdRqORlZVV6XZurmsZ61J5eRLl5Um6ywAygNAcAAAAGqHs7KzIz28eOTmC27ry5Q8osrOzok2bFmmspvEpKyuPlSs/E5wDmyU0BwAAgEYoOzsrcnKy46lH/xHLP/k03eU0Cqs//bzS7Xv+d2oaq2lc2u/QKo4ZsX9kZ2cJzYHNEpoDAABAI7b8k09j6eKV6S6jUSgrLa902+sOUD8JzQGAjGLu1bpl7tX0MvcqAADUPaE5AJAxsrOzIr9NXuRk56S7lEbD3KvpVVZeFitXrBGcAwBAHRKaAwAZIzs7K3Kyc+KXf3kgFq9amu5yGoWVn62qdHv8H25KYzWNy87bd4yzDh5p7lUAgDrWMq9JrFlXWnGbxkdoDgBknMWrlsZHhQvTXUajUFpeVum21x0AgIauR+f28dZHn1TcpvERmgMAAAAApLTMaxL779Up3WWQRr7JCQAAAAAAUoTmAAAAAACQIjQHAAAAAIAUc5oDpGRnZ0V2dla6y2g0srKyKt3OzfU5bl0qL0+ivDxJdxkAAABQ7wjNAeK/gXmbNnmRnZ2T7lIajS9/QPHf179FGqtpfMrLy2LFijWCcwAAAPgKoTlAfHGVeU7Me+buWFO4JN3lNArrV6+sdPvt+3+SvmIambx2O8UeQ8+M7OwsoTkAAAB8hdAc4EvWFC6JNUv/k+4yGoWkrKzSba87AAAAUB+YQBYAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AwEZtt33TDd4GAABoqHLTXQAAAPVX/n47xcpkScVtAACAhk5oDgDARm23fdPY4dDd010GAABAnTE9CwAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAACAOtAiL///bjdvk75CANgkXwQKAAAAUAf23H1AJPOn/ff2bgemuRoANqZeXWleXFwcY8aMid69e8eQIUNi6tSp6S4JAAAAoFa0yGsTBV8fGgVfHxot8lxpDlBf1asrza+66qrIzc2NadOmxYcffhhnnHFGdO7cObp161Zl7OGHHx6ffPJJZGf/X+5/5513Rp8+feqyZAAAAAAAGpB6E5ovXbo0pkyZEi+99FLk5eVFz549Y9iwYTF58uQYP358lfHr16+PG2+8MQ477LA0VAtATXVouV2s+Ly04jYAAABAfVBvpmeZM2dOtG3bNjp27FixrFevXjFz5sz0FQXANnPUXm2jW7u86NYuL47aq226ywEAAACIiHoUmhcVFUV+fn6lZfn5+bF8+fKNrjN58uTYf//9Y/DgwfHcc89t4woBqE0dWjSJ7/fZMb7fZ8fo0KJJussBAAAAiIh6ND1LRFSanzwiIicnJ9auXbvR8VlZWfHnP/85PvjggzjllFOic+fOsffee1f7+XNz681nCEAdy8nR/zQ+mXjcZ2LNUFOOe2Bb8fuFxigTj/tMrBlqKt3Hfb0JzZs2bRrr16+vtGzdunXRrFmzDY6/6667Yscdd4zmzZtHr169ok+fPvHII49UOzTPzs6KNm1aVGtdAMhErVvnpbsEYAvoVQCoPc6rkBnS3av1JjRv165dFBYWVlpWWFgYO+200wbHf+1rX6t0v02bNrF48eJqP395eRLFxZ9Ve30gs+XkZKf9FzLUteLiNVFWVp7uMraKXqUxysReBTKD8yqNUSaeV/UqjdG27NUtuXC63oTmPXr0iJKSkpg7d2507do1IiJmzJgRe+21V5Wxa9asidLS0mjVqlXFskWLFkWPHj1qVENpaWb90gSAmigrK3fugwygV9kSixcvivvuuzsiIk477czYeedOaa4IoH5yXoXMkO5erTeTIrVq1SqGDh0akyZNipKSkpg1a1Y8/fTTMWLEiCpjp02bFueee27Mmzcv1q9fH4899li8++67ccopp6ShcgAAgPS6//57YvbsmTF79sx44IF70l0OAEBGqzdXmkdEjBs3LsaPHx+DBg2KFi1axIQJE6Jnz54RETFkyJA466yz4uijj47DDjssli9fHmeffXYsW7YsunfvHvfdd1906dIlzXsAAABQ9xYtWlBxe+HCBZsYCQDA5tSr0Lxly5Zx8803b/CxZ599ttL973znO/Gd73ynLsoCAAAAAKCRqDfTswAAAAAAQLrVqyvNoTb4EiQAAAAAoLpcaU6D40uQAAAAAIDqEprT4PgSJAAAAACguoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAACm56S6gMcjOzors7Kx0l9FoZGVlVbqdm+uzobpUXp5EeXmS7jIAAAAAoFqE5ttYdnZW5Oc3j5wcwW1d+fIHFNnZWdGmTYs0VtP4lJWVx8qVnwnOAaCRc+FI3XLhSHq5cAQAGhah+TaWnZ0VOTnZ8b+Tp8WiZavSXU6jsKJ4TaXbP/7Zc2mspnHp1GH7OPe7AyI7O8v/NABAI5adnRVt8vMiOycn3aU0Gi4cSa/ysrJYsXKN98AA0EAIzevIomWrYv6iFekuo1EoLSuvdNvrDgBQt7KzsyI7Jydm/uKXsXrxknSX0yisXbmy0u2/X3FV2mppbFruvFPsO/osF44AQAMiNAcAALaJ1YuXRPFHH6W7jEahvLS00m2vOwBA9ZnoDgAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5jQ4uc22/9Lt/PQVAgAAAABkHKE5DU6rXftHk9adoknrTtFq1/3TXQ4AAAAAkEFy010A1LbcZttHmz0PT3cZAAAAAEAGcqU5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAZLi2udtV3G73pdsAAGw9oTkAAECGG7x9m9itabPYrWmzGLR9m3SXAwCQ0XLTXQAAAAA103a77eKE9h3SXQYAQIPgSnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAlHoVmhcXF8eYMWOid+/eMWTIkJg6dWq6SwIAAAAAoBHJ3doV3nrrrfjrX/8ab731Vnz00UexevXqKC0tjebNm0f79u2ja9eu0a9fvxg0aFC0bNlyq7Z91VVXRW5ubkybNi0+/PDDOOOMM6Jz587RrVu3ja4zd+7cOPHEE+Owww6L66+/fmt3BwAAAAAAKmxxaP7nP/85br311pg3b17ss88+0a1bt+jdu3c0b948cnNz4/PPP4+VK1fG/Pnz49Zbb40rrrgijjvuuDjvvPOiTZs2m93+0qVLY8qUKfHSSy9FXl5e9OzZM4YNGxaTJ0+O8ePHb3CdJEniqquuij59+mz5HgMAAAAAwEZsNjRfv359XHnllfHOO+/EmWeeGf/zP/8TTZs23eyG586dG5MnT47jjz8+Jk2aFL17997k+Dlz5kTbtm2jY8eOFct69eoVd99990bXefTRR6NDhw7Rtm3b+PTTTzdbEwAAAAAAbMpm5zR/8MEHY4899ognn3wyhg4dukWBeURE165dY9y4cXH33XfHjTfeuNnxRUVFkZ+fX2lZfn5+LF++fIPjP/nkk7jjjjviggsu2KJ6AAAAAABgczZ7pfl3vvOdyMvLq/YTdO3aNe69994tGpudXTnDz8nJibVr125w7IQJE+Lkk0+OnXfeudq1fVVubu1/L2pOTr36rlWoE5l43GdizVBTmXjcZ2LNUFOZeNxnYs1QU5l43GdizVBTmXjcZ2LNUFPpPu43G5pvLDBft25dNGnSpOJ+cXFxvPbaa7H77rtX+eLOLQndmzZtGuvXr6/yHM2aNasy9qWXXor3338/br755s1ud0tlZ2dFmzYtam170Ji1bl39D9qAuqNXITPoVcgMehUyg16FzJDuXt3iLwL9qssuuyx22223GDNmTCxdujRGjBgRLVq0iGXLlsU555wTp59++lZtr127dlFYWFhpWWFhYey0005Vxj7//PNRWFgYgwYNioiI1atXR5Ik8be//S0eeeSR2GWXXbZ6f8rLkygu/myr19ucnJzstP+Qoa4VF6+JsrLydJexVfQqjZFehcygVyEz6FXIDHoVMsO27NUtuXC62qH5q6++Gt///vcjIuI3v/lN7LDDDvG73/0uXnnllbj44ou3OjTv0aNHlJSUxNy5c6Nr164RETFjxozYa6+9qoz96hzpl156aUREXH/99dXZlQqlpZn1SxPqq7Kycv0EGUCvQmbQq5AZ9CpkBr0KmSHdvVrtyWGKi4ujbdu2ERHx+uuvx+DBgyMiYo899oiVK1du9fZatWoVQ4cOjUmTJkVJSUnMmjUrnn766RgxYkR1SwQAAAAAgK1S7dC8W7du8fvf/z7+/ve/x7///e849NBDIyJi3rx50alTp2ptc9y4cdG8efMYNGhQnH/++TFhwoTo2bNnREQMGTIk/vCHP1S3XAAAAAAA2KxqT88yYcKEuOSSS+Kuu+6KCy64oGIald/+9rdxxBFHVGubLVu23OiXez777LMbXa+m07IAAAAAAEBEDULznj17xjPPPFNl+a233hplZWU1KgoAAAAAANJhi0LzxYsXb/WGd955561eBwAAAAAA0mmLQvNDDjkksrKytmrD77zzTrUKAgAAAACAdNmi0Pzpp5+usmz69OnxyiuvxPe///2KQL2kpCQmTZoUI0eOrN0qAQAAAACgDmxRaL7nnntWWXb55ZfHjTfeGLvvvnul5VdccUXccsstMWzYsNqoDwAAAAAA6kx2dVd89913o7CwsMryJk2axL/+9a8aFQUAAAAAAOlQ7dC8V69ece2118bSpUsrlhUVFcXEiROje/futVIcAAAAAADUpS2anmVDrr/++hg9enQceuih0blz58jOzo758+fHbrvtFj//+c9rs0YAAAAAAKgT1Q7NO3XqFL///e/jtddeizlz5kSSJPH1r389+vfvH9nZ1b6AHQAAAAAA0qbaoXlERHZ2dhx44IFx4IEH1lY9AAAAAACQNtUOzUtLS+OFF16I999/P9auXVvl8YsvvrhGhQEAAAAAQF2rdmh+0UUXxQsvvBDdu3ePFi1aVHosKyurxoUBAAAAAEBdq3Zo/tJLL8VNN90URx55ZG3WAwAAAAAAaVPtb+xs165d9OrVqzZrAQAAAACAtKp2aH7xxRfHjTfeGAsXLqzNegAAAAAAIG2qPT3Lz372s1iyZEm8+OKL0axZs2jWrFmlx1999dUaFwcAAAAAAHWp2qH5D37wg9qsAwAAAAAA0q7aofmxxx5bm3UAAAAAAEDaVTs0j4hIkiSmTJkSr7zySpSWlkZBQUEcc8wx0aRJk9qqDwAAAAAA6ky1vwh09erVcdJJJ8XEiRNj1apV8emnn8Ytt9wSw4cPj1WrVtVmjQAAAAAAUCeqfaX5DTfcEG3atIn777+/4sryzz77LM4///y48cYb49prr621IgEAAAAAoC5U+0rzF198McaOHVtpKpbmzZvHBRdcEFOnTq2V4gAAAAAAoC5VOzRPkiTKy8urLN/QMgAAAAAAyATVDs0PP/zwuOmmm2LdunUVy1avXh033nhjHH744bVSHAAAAAAA1KVqz2l+ySWXxA9+8IM4+OCDo1evXlFWVhazZs2KPfbYIy6++OLarBEAAAAAAOpEtUPzFi1axIMPPhh/+tOfYvr06VFWVhbDhg2LI444InJzq71ZAAAAAABIm2qn2x9//HGsWLEiDjvssDjssMMqlj/77LPRr1+/2GGHHWqlQAAAAAAAqCvVntP8uuuui5dffrnK8g8++CBuv/32mtQEAAAAAABpUe3Q/NVXX41vf/vbVZZ/+9vfjqlTp9aoKAAAAAAASIdqh+br1q2LnJycKsvz8vJizZo1NSoKAAAAAADSodqhee/eveO3v/1tleWPPPJI7LvvvjWpCQAAAAAA0qLaXwR62WWXxSmnnBKzZ8+Ob37zm5GdnR3Tpk2Lt956Kx588MHarBEAAAAAAOpEta8033PPPeO5556LHj16xIsvvhhPPfVU7LrrrvHEE09Et27darNGAAAAAACoE9W+0jwiol27dnHZZZfVVi0AAAAAAJBW1b7SPCLivvvui2OPPTYGDBgQixYtioiIsWPHxksvvVQrxQEAAAAAQF2qdmj+v//7v/Hss8/GrbfeGp999lmUl5dHRMS3v/3t+MUvflFrBQIAAAAAQF2p9vQskydPjl/84hex2267VVr+jW98I959990aFwYAAAAAAHWt2leal5eXR5MmTaosX7RoUbRr165GRQEAAAAAQDpUOzQfPnx4jB8/PoqKiiqWffzxxzFhwoT47ne/WyvFAQAAAABAXap2aD5mzJjo3r17HHroobF27do455xz4ogjjogBAwbED37wg9qsEQAAAAAA6kS15zTPycmJq6++OkaPHh3/+te/IisrK/bdd99o3759bdYHAAAAAAB1ptqh+dy5cyMrKyu6dOkSO+64Y7z55ptx5513xh577BHDhw/f4HznAAAAAABQn1V7epYJEybEq6++GhERs2fPjlGjRsW6devikUceiSuvvLLWCgQAAAAAgLpS7dD83//+dwwYMCAiIh577LEYMGBA/OQnP4lJkybFiy++WGsFAgAAAABAXal2aF5aWloxBcvrr78eBx54YERENG/ePMrKymqnOgAAAAAAqEPVntN8wIABMXHixOjSpUssXbo0Dj/88IiImDlzZnTp0qXWCgQAAAAAgLpSoznNO3ToEO+991786le/ig4dOkTEf+c3P+WUU2qtQAAAAAAAqCvVvtK8bdu2G/zCzx//+Mc1KggAAAAAANJls1eav/zyy1FUVFSjJ3n88cdrtD4AAAAAANSFzYbmbdq0iZNPPjmeeOKJKC0t3aqNL1iwIMaOHRt///vfq10gAAAAAADUlc1Oz7LPPvvEz3/+87jsssvi1ltvjW9961tx4IEHxp577hmdOnWK7Oz/y91XrVoVCxYsiNmzZ8fLL78cM2bMiLPPPjtOP/30bboTAAAAAABQG7ZoTvM99tgjfvvb38Zf/vKXePzxx+Oiiy6Kzz77LCIimjVrFtttt12sXr06kiSJJEmiW7ducfjhh8fEiROjffv223QHAAAAAACgtmzVF4EefPDBcfDBB0dZWVl89NFHsWDBgigpKYnS0tLIy8uL9u3bR9euXaN169bbql4AAAAAANhmtio0/0JOTk506dIlunTpUtv1AAAAAABA2mz2i0ABAAAAAKCxqFeheXFxcYwZMyZ69+4dQ4YMialTp2507IwZM2LEiBFRUFAQRx11VEyfPr0OKwUAAAAAoCGq1vQs28pVV10Vubm5MW3atPjwww/jjDPOiM6dO0e3bt0qjVu4cGGMHTs2JkyYEAMHDoznn38+zj777PjTn/4U+fn56SkeAAAAAICMV2+uNF+6dGlMmTIlLrnkksjLy4uePXvGsGHDYvLkyVXGtmnTJsaPHx8HHXRQZGdnx5FHHhnbbbdd/Pvf/05D5QAAAAAANBT1JjSfM2dOtG3bNjp27FixrFevXjFz5swqY1u0aBGHHHJIxf0VK1bEypUro1OnTnVRKgAAAAAADVSNp2f59NNPY9myZbHbbrtFbm5uFBYWRn5+fuTk5GzVdoqKiqpMrZKfnx/Lly/f5Hpr1qyJq6++Og466KDYY489trb8SnJza/8zhJycevO5BNSZTDzuM7FmqKlMPO4zsWaoqUw87jOxZqipTDzuM7FmqKlMPO4zsWaoqXQf99UOzVetWhWXXXZZzJkzJ5YtWxZ//OMfY9ddd40rrrgi+vXrF6eddtpWbzM7u/KLkZOTE2vXrt3o+AkTJsRjjz0WTZo0iUcffXSrn6/yc2dFmzYtarQN4L9at85LdwnAFtCrkBn0KmQGvQqZQa9CZkh3r1Y7NJ84cWJsv/328eKLL0afPn0qlo8aNSouv/zyrQ7NmzZtGuvXr6+0bN26ddGsWbONrnPFFVfEuHHj4oUXXoiTTz45Hnjggdhzzz236nm/UF6eRHHxZ9Vad1NycrLT/kOGulZcvCbKysrTXcZW0as0RnoVMoNehcygVyEz6FXIDNuyV7fkwulqh+Z/+tOf4ne/+12VaVh22WWXWLp06VZvr127dlFYWFhpWWFhYey0006bXC8rKysOP/zweOSRR+Kuu+6Km266aauf+wulpZn1SxPqq7Kycv0EGUCvQmbQq5AZ9CpkBr0KmSHdvVrtyWHatWsXS5YsqbL8n//8Z+y+++5bvb0ePXpESUlJzJ07t2LZjBkzYq+99qoydtmyZVWW5eTkxGef1f6V4gAAAAAANB7VDs3PPffcuPDCC+Oll16KrKysWLNmTTz77LPxk5/8JMaMGbPV22vVqlUMHTo0Jk2aFCUlJTFr1qx4+umnY8SIEVXGvvzyyzFu3LiYP39+JEkSL7/8crzyyitxwgknVHd3AAAAAACg+tOzHH300ZGbmxs33HBDfPbZZ3H00UfHnnvuGRMnToxDDz20WtscN25cjB8/PgYNGhQtWrSICRMmRM+ePSMiYsiQIXHWWWfF0UcfHSNGjIisrKw4//zz46OPPoqddtoprrvuuhg8eHB1dwcAAAAAAKofmkdEHHnkkXHkkUfGqlWrIisrK1q3bl2jYlq2bBk333zzBh979tlnK90fPnx4DB8+vEbPBwAAAAAAX1aj0PwL22+/fW1sBgAAAAAA0qraofnbb78dP/3pT+Pdd9+N9evXV3n81VdfrVFhAAAAAABQ16odml9wwQXx6aefxsiRI2OHHXaozZoAAAAAACAtqh2aL1q0KB5++OHYe++9a7MeAAAAAABIm+zqrti3b98oLS2tzVoAAAAAACCtqn2l+cSJE2P06NExePDg6NatWzRr1qzS4wcffHCNiwMAAAAAgLpU7dB8ypQpMWfOnHj77berPJaVlRXvvPNOjQoDAAAAAIC6Vu3Q/I477ohRo0bF6NGjo2XLlrVZEwAAAAAApEW15zTPysqKk08+WWAOAAAAAECDUe3Q/KSTToqnn366NmsBAAAAAIC0qvb0LIWFhfHMM8/Eo48+GnvuuWc0bdq00uM/+9nPalwcAAAAAADUpWqH5uvXr4/DDz+8NmsBAAAAAIC0qnZoft1119VmHQAAAAAAkHbVntMcAAAAAAAami2+0nzkyJFxxx13ROvWrSMi4rLLLtvkeFeiAwAAAACQabY4NE+SpNL9kpKSWi8GAAAAAADSaYtD89/85jeV7t922221XgwAAAAAAKRTtec0nz59epSWllZZPnv27Jg1a1aNigIAAAAAgHSodmg+cuTIWLVqVZXly5cvj1tuuaVGRQEAAAAAQDpUOzRPkiSysrKqLG/dunW8/fbbNSoKAAAAAADSYYvnNP/CIYccEllZWZGVlRXHH398ZGf/X+5eXl4eS5cujT333LNWiwQAAAAAgLqw1aH59ddfH0mSxKmnnhqjRo2KVq1aVTyWlZUVLVu2jP79+9dqkQAAAAAAUBe2OjTv169fRET07ds3hg0bFq1bt671ogAAAAAAIB22OjT/wm9+85varAMAAAAAANKu2qF5cXFxPPLII/H+++/H2rVrqzz+s5/9rEaFAQAAAABAXat2aH7++efHrFmzYsCAAdGyZcvarAkAAAAAANKi2qH5m2++Gb/85S/jgAMOqM16AAAAAAAgbbKru+Iuu+wSO++8c23WAgAAAAAAaVXt0Pyqq66Kq666Kv7xj3/EihUrYs2aNZX+AQAAAABApqn29Czf//73o7S0NF599dXIysqq8vg777xTo8IAAAAAAKCuVTs0/9WvflWbdQAAAAAAQNpVOzTv169fbdYBAAAAAABpV+3QfHPzlufl5VV30wAAAAAAkBbVDs0LCgo2OJf5F8xpDgAAAABApql2aP7AAw9UWbZmzZqYPHlydO/evUZFAQAAAABAOtT6nOYHHHBAXHjhhdUuCAAAAAAA0iW7tjfYpEmTKC4uru3NAgAAAADANlftK80feuihDS4vLCyMt99+u9oFAQAAAABAulQ7NL/nnnuqLMvKyorWrVvHpZdeWqOiAAAAAAAgHaodmr/00ku1WQcAAAAAAKRdrc9pHhGxevXqbbFZAAAAAADYprYqNJ89e3aMHTs2Ro8eHdOnT6/y+Lp16+Kee+6JQw89tNYKBAAAAACAurLFofmcOXPilFNOifLy8mjTpk18//vfj9dff73i8SeffDKOOOKIuPvuu+Pss8/eJsUCAAAAAMC2tMVzmt95550xYsSIGDduXERE9OjRI2699db4wQ9+EDfffHMsXrw4Ro0aFaNGjYoWLVpss4IBAAAAAGBb2eIrzd9444045phjKu4fc8wx8c9//jPGjh0bAwcOjKlTp8Z5550nMAcAAAAAIGNt8ZXmK1asiI4dO1bcb9WqVTRt2jQefPDB2HvvvbdJcQAAAAAAUJe2+ErzsrKyyMnJqbxydnZsv/32tV4UAAAAAACkwxZfaR4R8fHHH8fatWsrLVu6dGmVMD0iYuedd65ZZQAAAAAAUMe2KjQ/4YQTKt1PkiS+973vbXDsO++8U/2qAAAAAAAgDbY4NL/uuuu2ZR0AAAAAAJB2WxyaH3vssduyDgAAAAAASLst/iJQAAAAAABo6ITmAAAAAACQIjQHAAAAAICUehWaFxcXx5gxY6J3794xZMiQmDp16kbHvv7663HUUUfFvvvuG8ccc0y88sordVgpAAAAAAAN0RZ/EWhduOqqqyI3NzemTZsWH374YZxxxhnRuXPn6NatW6Vxq1atiiuuuCJuuumm6NmzZzz00ENx1llnxdNPPx277757eooHAAAAACDj1ZsrzZcuXRpTpkyJSy65JPLy8qJnz54xbNiwmDx5cpWxM2fOjJNOOil69eoVOTk5MXLkyBgwYED87ne/S0PlAAAAAAA0FPUmNJ8zZ060bds2OnbsWLGsV69eMXPmzCpjDz744BgxYkSlZXvuuWcsWrRoW5cJAAAAAEADVm9C86KiosjPz6+0LD8/P5YvX77B8Xl5eZXuz5s3L3bddddtVR4AAAAAAI1AvZrTPDu7coafk5MTa9eu3ex6c+bMiX/+859xxRVX1Oj5c3Nr/zOEnJx687kE1JlMPO4zsWaoqUw87jOxZqipTDzuM7FmqKlMPO4zsWaoqUw87jOxZqipdB/39SY0b9q0aaxfv77SsnXr1kWzZs02ud6yZcviwgsvjJ/97GeVpnbZWtnZWdGmTYtqrw/8n9at8zY/CEg7vQqZQa9CZtCrkBn0KmSGdPdqvQnN27VrF4WFhZWWFRYWxk477bTRdZYvXx5nn312jBs3Lvr27Vuj5y8vT6K4+LMabWNDcnKy0/5DhrpWXLwmysrK013GVtGrNEZ6FTKDXoXMoFchM+hVyAzbsle35MLpehOa9+jRI0pKSmLu3LnRtWvXiIiYMWNG7LXXXhscX1xcHOecc05cdNFF0b9//1qpobQ0s35pQn1VVlaunyAD6FXIDHoVMoNehcygVyEzpLtX682kSK1atYqhQ4fGpEmToqSkJGbNmhVPP/10jBgxosrYNWvWxHnnnRfnnntuHHDAAWmoFgAAAACAhqjeXGkeETFu3LgYP358DBo0KFq0aBETJkyInj17RkTEkCFD4qyzzoqjjz46pkyZEtOnT49//etfldYfPHhw3HLLLekoHQAAAACABqBeheYtW7aMm2++eYOPPfvssxW3hw0bFsOGDaujqgAAAAAAaCzqzfQsAAAAAACQbkJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABAitAcAAAAAABShOYAAAAAAJAiNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABAitAcAAAAAABShOYAAAAAAJAiNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAlHoVmhcXF8eYMWOid+/eMWTIkJg6depm13n77bfjoYceqoPqAAAAAABo6OpVaH7VVVdFbm5uTJs2LW688cYYN25cvPfeexscW15eHq+99lqceeaZsX79+jquFAAAAACAhig33QV8YenSpTFlypR46aWXIi8vL3r27BnDhg2LyZMnx/jx46uMP+ecc2LRokVpqBQAAAAAgIaq3lxpPmfOnGjbtm107NixYlmvXr1i5syZGxx/5513xtNPPx1dunSpowoBAAAAAGjo6k1oXlRUFPn5+ZWW5efnx/Lly9NTEAAAAAAAjU69mZ4lIiI7u3KGn5OTE2vXrq2z58/Nrf3PEHJy6s3nElBnMvG4z8SaoaYy8bjPxJqhpjLxuM/EmqGmMvG4z8SaoaYy8bjPxJqhptJ93Neb0Lxp06ZVvtBz3bp10axZszp5/uzsrGjTpkWdPBc0dK1b56W7BGAL6FXIDHoVMoNehcygVyEzpLtX601o3q5duygsLKy0rLCwMHbaaac6ef7y8iSKiz+r9e3m5GSn/YcMda24eE2UlZWnu4ytoldpjPQqZAa9CplBr0Jm0KuQGbZlr27JhdP1JjTv0aNHlJSUxNy5c6Nr164RETFjxozYa6+96qyG0tLM+qUJ9VVZWbl+ggygVyEz6FXIDHoVMoNehcyQ7l6tN5MitWrVKoYOHRqTJk2KkpKSmDVrVjz99NMxYsSIdJcGAAAAAEAjUW9C84iIcePGRfPmzWPQoEFx/vnnx4QJE6Jnz54RETFkyJD4wx/+UDH20ksvjYKCgnjjjTdi0qRJUVBQEFOmTElX6QAAAAAANAD1ZnqWiIiWLVvGzTffvMHHnn322Ur3r7/++rj++uvroiwAAAAAABqJenWlOQAAAAAApJPQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABAitAcAAAAAABShOYAAAAAAJAiNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaA4AAAAAAClCcwAAAAAASBGaAwAAAABAitAcAAAAAABShOYAAAAAAJAiNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKfUqNC8uLo4xY8ZE7969Y8iQITF16tSNjl28eHGcfvrpUVBQEMcdd1y8+eabdVgpAAAAAAANUW66C/iyq666KnJzc2PatGnx4YcfxhlnnBGdO3eObt26VRn7wx/+MHr37h133nlnvP7663HuuefGM888E+3bt09D5QAAAAAANAT15krzpUuXxpQpU+KSSy6JvLy86NmzZwwbNiwmT55cZezMmTPjvffei7Fjx0aTJk1i4MCB0bdv3/j9739f94UDAAAAANBg1JvQfM6cOdG2bdvo2LFjxbJevXrFzJkzq4x9++23o3v37tG0adOKZd/4xjc2OBYAAAAAALZUvZmepaioKPLz8ysty8/Pj+XLl2/R2DZt2mxw7JbKzs6Ktm1bVHv9jcnK+u9/LznjkCgrK6/17UN9kpPz38/htt8+L5IkzcVspS96dc8TxkZSXpbeYmAby8rOiYjM7tUL/ufsKNWrNHC5DaBX+170o0hK9SoNW1Zu5vfqd04d6P9XafAawv+vHjdqTJSVOa/SsOXk1I/zar0JzSMisrMrX/iek5MTa9eurfHYLZGVlRU5OVnVXn9ztm/ZbJttG+qbr/ZnJtmuRet0lwB1JpN7tXVeq3SXAHUmk3u1aWvnVRqPTO7VFv5/lUYkk3u1eUvnVRqPdPdqvflN0bRp01i/fn2lZevWrYtmzaqevJs0abLFYwEAAAAAYEvVm9C8Xbt2UVhYWGlZYWFh7LTTTlXGtm/ffovHAgAAAADAlqo3oXmPHj2ipKQk5s6dW7FsxowZsddee1UZ26tXr3jvvfdi9erVmx0LAAAAAABbqt6E5q1atYqhQ4fGpEmToqSkJGbNmhVPP/10jBgxosrYr33ta1FQUBA33XRTrFu3Lv70pz/FP//5zzjmmGPSUDkAAAAAAA1FVpLUn+8MXr16dYwfPz7++te/RosWLeKiiy6KIUOGRETEkCFD4qyzzoqjjz46IiKWLVsWl19+ebz55pvRoUOHGD9+fBxwwAHpLB8AAAAAgAxXr0JzAAAAAABIp3ozPQsAAAAAAKSb0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAFKE5AAAAAACkCM0BAAAAACBFaE69de2118all1662XErV66M8847LwoKCuK4446r0XM+8cQTccwxx9RoGzRsjksyzX333Rff+9730l0GDcyW/i5877334rTTTovevXvHwIED49VXX62D6shkzrNsK5deemlce+216S6jXtN/ZJotfZ+7ZMmSOO+886JPnz7Rv3//eOqpp+qgus3Tc2SaTO+5rZWb7gKgpq677rro0KFDTJ8+PT7//PN0lwMR4bgEmDdvXnzve9+Liy66KH7961/HihUrorS0NN1l0UA4z1IX7rjjjrjzzjvjnnvuif333z/d5dQb+o9MUlRUFCeddFKMGDEibrnllli7dm2sXLky3WVtFT1HJmkIPfcFV5qT8WbOnBkHHnhg5ObmRsuWLWt120888YQrNKkWx2X91NBeu0MOOST+8Y9/pLsMGpju3bvHwoULa7yd6667LoYMGRInnHBCZGdnR7t27aJjx461UCE4z9ZXDem1++CDD2LGjBnRrFmzdJdS7+i/+qmhvXa19T739ttvj27dusXo0aOjSZMm0apVq9h1111rocK6o+fqp4b22um5qoTmZLz169enuwSownEJNGbLly+Pv/3tb3HiiSemuxQaKOdZtqUkSeKKK66I008/Pd2l1Ev6j0xRVlYWf/jDHzL+/YieI1M0lJ77gtCceuPNN9+MIUOGREFBQZx77rmxaNGiisfeeeedOPHEE6NPnz4xevTo+OSTTyIiok+fPrF48eL40Y9+VPEnk//4xz/ipJNOiv333z8OOeSQePnllyu2849//CP69OlT6Xm7d+8e77zzTpV6Lr300hg/fny88cYbUVBQEIcddtg22Gvqu4Z4XC5YsCBGjRoVBQUF0b9//7jkkksq3oj97W9/i2OPPTZ69+4dxx9/fLz++usV691+++1x4YUXxpVXXhkDBgyIgw46KF599dV46KGH4pvf/GYUFBTE7bffXjH+i/nyrr322hg8eHAUFBTEnXfeWWl7l19+eTzwwANx8MEHx7Rp0yIi4oEHHoiDDjooBg4cGHfddVfF+LvuuisOOOCA6N27dxx11FHx6quvRpIkMXHixOjXr1/st99+cfzxx8ecOXM2uN8beu3Wrl0b/fv3rzTPcmlpaQwcODBmzpwZCxcujO7du8e9994bRx11VPTq1SvGjBkT69atqxi/sXq3xiuvvBIFBQUV//bee+/Ya6+94qOPPoqPPvooTj755Nh3332jf//+cdFFF8W8efOioKAgFi9eHGeeeWYUFBTEY489FhERc+fOjeHDh0evXr3i1FNPjXfffbdaNdEwPPbYY1WOrW984xvx+uuvx7Bhw6KgoCAGDhwYN910U8VxGBExdOjQKCgoiFdeeSUiNv27cEPeeuutaNasWcybNy+OOeaY6NWrV1x88cWVegcinGedZ+vmPPtVt912W6XfjT169IiDDz644vHJkydH69atY8CAAbXyfPWV/tN/Df197rx582L16tWxfv36OPHEE2OfffaJ73//+/Hpp59Wa19qSs/pOT23YYcccki8+OKLccEFF8S3vvWtiIhYsWJFnH/++dGnT5847rjjYtasWZXGP/zwwxX7MmrUqFi0aFGcf/75UVBQEIceemjMnj27Wq9fJQnUA6tWrUr69euXTJo0KSkuLk6WLVuWDBkyJLnkkkuSTz75JOnXr1/y6KOPJuXl5ckDDzyQnHXWWRXrDh48OHnxxRcr7k+bNi2ZM2dOkiRJMn369GTvvfdOFi1alCRJkrz22mvJfvvtV+m5u3Xrlrz99ttJkiTJ448/nhx99NEVjz3++OPJKaecss32m/qtIR6Xn376aTJgwIDkvvvuS8rKypLPPvss+d3vfldR14EHHphMnz49SZIk+fvf/57sv//+ybvvvpskSZLcdtttSY8ePZK//e1vSWlpaXL55Zcn++67b3LLLbck5eXlya9+9avk61//erJq1aqKOnv37p28/vrrSVlZWfLwww8n3bp1S15//fWK7e23337JmWeemSxcuDBJkiR59NFHk29+85vJggULklWrViUjR45Mnn/++eSDDz5I9t1332Tx4sVJkiTJnDlzkg8++CD5y1/+kgwePDhZuXJlUl5envzzn/9MPv74443u/4Zeu5/+9KeVfnbPP/98xeu9YMGCpFu3bsmDDz6YrF27Npk5c2ZSUFCQ3H777ZusN0mS5PTTT0/222+/Tf7bf//9k//85z+V6lm7dm0ybNiw5IYbbkiSJEnGjh2bXHfddUlpaWlSUlKSvPzyyxVjBw8enLz22msV99evX59861vfSi6++OKksLAw+fTTT5Pvfe97fo+RJEmSFBUVJQcffHDy4IMPJsOHD0/uu+++pLy8PFmxYkUybdq0inHdunVLFixYUHF/U78LN+b3v/990rNnz+THP/5xsmbNmuTjjz9Ohg4dmtxxxx3bdB/JLM6zzrN1eZ695JJLkmuuuaZKzfPnz0/222+/5M9//nOSJEny8ccfJwceeGDy0UcfJUmSJPvtt1+lc21Dof/0X2N4nzt9+vSkW7duyejRo5Pi4uJk5cqVyamnnppcdtllG11nW9Fzek7PbdzgwYOTvn37Jvfff3/y+eefJ0mSJN/97neTSy+9NFm3bl3y5ptvJoceemiyZs2aivFDhw5NVq1alSxdujQZOHBgsv/++ydz5sxJSkpKkhNOOCE577zzNvmcW8KV5tQLL730UuTn58cFF1wQrVq1ih122CH22WefiPjvJ4rdunWL4cOHR1ZWVnzve9+LJUuWbPQLMA488MDo3r17RPz3U9k2bdps9NNB2JSGeFw+/vjj0alTpzj11FMjOzs78vLy4vjjj4+IiF/96ldx2mmnVVyZMGDAgDjhhBPi3nvvrVi/e/fuMXDgwMjJyYkjjzwyPvvsszjzzDMjKysrTjrppCgrK4slS5ZUjN9ll12ib9++kZ2dHd/97ndj8ODBMWXKlIrHmzZtGrfeemt06tQpIiJ+/etfxw9/+MPYZZddonXr1nHhhRfGCy+8EE2aNInS0tL485//HIsXL47u3btH165dIy8vL4qLi+Mvf/lLFBYWRkFBwVbPmXzSSSfFK6+8UnGlx6OPPlrlz8mOOuqoaNKkSeyzzz5x3nnnVezDxuqNiLjnnnvijTfe2OS/1157rcr8bhMnToymTZvGj370o4iIaNasWbz11lsxY8aMyM3NrXQl3FfNnDkzPvnkk/jJT34Sbdu2jZYtW0b//v236vWgYUqSJC699NLYd9994+STT45mzZrFm2++GW+99Va0bNkyDjzwwI2uu6nfhRvTtGnT2G677WL8+PHRrFmz6NixY5x66qkV/QERzrPOs/+nLs+zX7Zu3boYO3ZsjBgxIgYNGhQREddcc02cdNJJ0blz563az0yj//TfFxry+9ymTZtGRMSVV14ZrVq1iu233z7OOuustLwf0XN67gt6bsOGDh0aI0eOjKZNm8asWbPinXfeiSuuuCK222676N27dxx00EHx5ptvVow/8cQTo3Xr1tGhQ4fYf//9o1+/ftG9e/do3rx5DBs2bLN/GbslhObUCx9//PFG35guXLgwZs6cGX369Kn499FHH23023eXL18eN954Yxx33HHRv3//WLZsWZSXl2/D6mmoMuW4nD17dqU/vzrhhBMqTvJf/Pviz4vnzZtX8Qbrq+bPn1/xBuMLu+66a3z44YcV97/8piEnJydatmxZ8WU0eXl5ERGb3K8999wzli9fXnG/R48e0bx584r7CxYsiAkTJlS8pqNGjYpVq1bFrrvuGrfddlv87ne/i0MPPTSOPfbYWLhwYfTt2zd+/OMfxy9/+csYOHBgnHrqqbFy5co47rjjKu3/22+/vdGaOnToEEcccUQ8/vjjsXTp0pg5c2YcffTRm9yHwsLCTdZbXc8991z88Y9/jJ/+9KeRm5sbERGXX3557LbbbnHuuedG3759N/lnekuWLIkdd9yx4s0KfOHuu++OefPmxTXXXBMREZMmTYrs7OwYOXJkHHDAAfHkk09udN1N/S6MiNh///0r9duKFSuiY8eOsf3220eTJk0qxnXq1KmidyDCeTbCeXZj+7CtzrNfde2111YKEz7//PP4+9//Hg8++GAMGDAgBgwYEJ9++mn8v//3/2L06NG19rz1gf7Tfxvbh0x9n7uh1+WLn2mHDh0qxnXq1Ck+/fTTOp8yTs/puY3tQ2PpuY0dR1/48rRCCxYsiLVr18ZBBx1U8Xo8+eSTUVRUVDHmy8dPbm5upefMy8uLsrKyLXmZNim3xluAWrDjjjtW+RQoSZKI+O8v1mOPPTZ+8pOfbHY7a9eujWOOOSaOOuqomDRpUnTu3LliPqQN+eI5YEMy5bjs1atXzJgxo8ryDS3beeedK+Z3+6pOnTpVehMTEfHhhx/GTjvtVHE/Kyur0uNfvb8577//fuyxxx4V97fbbrtKj3fu3DmuvfbainmVv2zw4MExePDgWL16dVx55ZXx05/+NG6++eY47rjj4rjjjouioqI4//zz49e//nU88cQTW1XXaaedFmPHjo02bdrEt7/97U1+K/37779f8Zpsqt5TTz11s/Oo5ebmxhNPPBG77rprzJ8/P6688sq4+eabK73mLVu2jGuuuSauueaaeOONN+L000+Pww47LLp06VJlezvuuGMsW7Ys1q1bVxFW+j3Hm2++Gb/4xS/i4Ycfrji2O3ToELfeemuUl5fH1KlT40c/+lH8z//8zwaP/U39Loz479yZX5WXlxeffvppFBUVRdu2bSMi4pNPPtnkFZ80Ps6zzrMb24dtcZ79queeey6ef/75eOqppyrChGbNmlX5uXbv3j1uv/32irmEGwr9p/82tg+Z+j53Y6/LLrvsEu+880707NkzIv77fqRjx46VPtivC3pOz21sHxpLz7Vr126Dx9EXvtyTnTt3jl133bXSXxJ8VU2Pny3hSnPqhUGDBsWyZcvirrvuis8//zzmzJkTf/nLXyIi4thjj42XX345/vCHP0SSJLFy5cp47LHHNvgN0qWlpVFUVBQHHHBAdOnSJd56661YtmxZxeN5eXmxbt26KC0tjYj//onUpjRr1iz+85//xJo1a+KNN96oxT0mEzTE4/L444+P9957L+67774oLy+P1atXx4MPPhiFhYVxxhlnxG9+85uKL2j561//Go8//niMGjVqq57jyz788MN4/fXXo7S0NB566KH4+9//XvEnexsyatSoGD9+fHz00UdRXl4eH374Ybz44osxf/78eO6552Lt2rWRl5cXeXl5kZubG7NmzYq//vWvUVpaGi1btowmTZpU/I/vhmzstevRo0d06NBhg38+F/HfL1MsLS2N119/Pe68886KMRurNyLi/vvvjxkzZmzy3/Tp02PXXXeNtWvXxtixY+PEE0+s8idyTzzxRPznP/+JiIi2bdtGVlZWpf+5/+CDD6KoqCjee++92HfffSM/Pz+uv/76WL16dSxYsCCeeeaZLfxp0RAVFRXFj370o7jooovi61//esXyhx9+OJYuXRrZ2dmRn58f2dnZkZOTExH/d1x9/PHH8dFHH23yd+HGNGvWLI4//vi44YYbYt26dbF48eL4+c9/Ht/5zne26f6SWZxnnWe/sK3Ps181f/78uOKKK+LGG2+MHXfccYtf74ZE/+m/LzT097kjR46MSZMmRUlJSaxYsSImTZqUlvcjek7PfUHPbV6vXr2iffv2ccMNN8Tnn38ea9asib/+9a8xd+7crdpOTbnSnHohPz8/brvttrjyyivjzjvvjMMOOyx69+4dERHt27ePu+66K6699tq4+uqro02bNnHkkUducDstWrSIMWPGxHnnnRetWrWKfv36RX5+fsXje+21V+y9995x8sknR7du3SI/P7/Kp5Fftv/++8d2220XgwYNin79+sXXv/71aNGiRa3uO/VXQzwu27VrFw888EBMmDAhbr/99mjdunUccsghkZeXFwceeGBMnDgxJkyYEAsXLowuXbrE7bffvtm5izelU6dO8fzzz8c555wTrVu3jptvvjm+9rWvbXT88OHDY/369XH66afHqlWromvXrnHOOedETk5OPPvss3H11VfHunXrYu+9947rr78+Vq1aFffdd19ccMEFUVpaGgcccECceuqpG93+pl677373u3HPPffEN77xjSrrlZaWxiGHHBJr1qyJU045JYYPH77JerfWww8/HO+8807Mnz8/Hn744Yrlv/71r6OkpCRGjx4dixcvjmbNmsUPf/jDij/tPProo+Omm26Khx9+OM4999zo1q1b3H777XHxxRfHwIEDo3///tG3b9+YN2/eVtdEw/Dzn/88Pv7447jpppvipptuqlh+xBFHxL333hvLly+P1q1bx3XXXVfxZ7DDhg2LsWPHRpcuXeLiiy+O3XbbbaO/CzflwgsvjGuuuSYGDx4czZs3j7PPPjuOOeaYbbavZB7nWefZL2zr8+xXXX/99VFSUhJjx46tWJaTk9OoLpLRf/rvCw39fe7IkSOjqKgojjjiiMjJyYmTTjopzjzzzK3ej5rSc3ruC3puy/z85z+PCRMmxEEHHRRNmjSJ/v37x0UXXbTV26mJrMTfbQOwDTzxxBNx//33x1NPPZXuUrbIbbfdFjvuuGOMGDGiYtnChQvj0EMPjenTp0fr1q3TWB0AVOY8C+mj/6Bu6TnSwZXmADRqK1asiCRJ4o9//GM8/vjj6S4HABoU51lIH/0HdUvPNSxCcwAatQsvvDBmzpwZV199daVvWwcAas55FtJH/0Hd0nMNi+lZAAAAAAAgJTvdBQAAAAAAQH0hNAcAAAAAgBShOQAAAAAApAjNAQAAAAAgRWgOAAAAAAApQnMAAAAAAEgRmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAGSYSy+9NK644oq49dZbo1+/ftGnT5/45S9/GRERZWVlcdddd8XQoUNjn332iWHDhsWsWbMqrd+9e/d47rnn4rTTTotvfOMbMXTo0Hj//ffjxRdfjEMPPTR69eoVY8aMiXXr1lWs89lnn8WVV14Zffr0icGDB8fkyZPrdJ8BAKCuCM0BACADPfPMM9G0adN44oknYvjw4fHTn/40li1bFhERf/vb32LcuHHx7LPPRs+ePeOss86KkpKSSuvfdNNNMWbMmHj88ccjKysrzjrrrHjqqafi3nvvjR/96EcxZcqUePHFFyvGn3/++bFs2bJ45JFH4vrrr4977rknXnjhhTrdZwAAqAtCcwAAyEBt2rSJs846K3bZZZc499xzI0mS+OCDDyInJyd+85vfRP/+/WOXXXaJsWPHxooVK2L27NmV1j/mmGOid+/e0a1btxg+fHgsWrQoLr744ujcuXOcdtppsfvuu8e7774bERFvvvlmvPPOO/Gzn/0sunbtGvvvv39cffXV8dRTT6Vj1wEAYJvKTXcBAADA1vva174W2dn/vQamZcuWERGxdu3aKuPat28fERGff/55peXdu3evuN2yZcvIy8uLzp07Vyzr0KFDrF+/PiIiZs+eHUVFRdG/f/+Kx5Mkia5du9bS3gAAQP0hNAcAgAzUrFmzjT72j3/8I5588sn497//HcuXL9+i9Zs2bbrR7WVlZcUBBxwQV155ZaXlTZo02YqKAQAgMwjNAQCgAXnooYfi9ttvj9NPPz2GDRsW7du3jyFDhtRom717944777wz2rVrV3FVOwAANFTmNAcAgAZkyZIl0b59++jbt2/ssssu8dprr9V4m7169Yq+ffvG6NGjY/bs2bFgwYJ44oknYvz48bVQMQAA1C9CcwAAaEBGjRoVOTk5ccopp8Spp54aH3/8cWy33XY13u7NN98cPXv2jLPPPjuGDBkSTz75ZBx11FG1UDEAANQvWUmSJOkuAgAAAAAA6gNXmgMAAAAAQIrQHAAAAAAAUoTmAAAAAACQIjQHAAAAAIAUoTkAAAAAAKQIzQEAAAAAIEVoDgAAAAAAKUJzAAAAAABIEZoDAAAAAECK0BwAAAAAAFKE5gAAAAAAkCI0BwAAAACAlP8PzZPymPdnhYMAAAAASUVORK5CYII=",
+      "text/plain": [
+       "<Figure size 1500x500 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "ax = sns.catplot(x=\"name\", y=\"runtime\", kind=\"bar\", data=df, aspect=3)\n",
+    "ax.set(xlabel=\"name\", ylabel=\"Runtime (seconds)\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.9.7 ('.venv': venv)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "3a4380b13e5c3181737ad0a440ee2e5072f6f89a6f577e15b530ebafc9227677"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}


Test patch:
diff --git a/scripts/experimenting_rustbackend/rust_deser_test.rs b/scripts/experimenting_rustbackend/rust_deser_test.rs
new file mode 100644
index 0000000..0eae815
--- /dev/null
+++ b/scripts/experimenting_rustbackend/rust_deser_test.rs
@@ -0,0 +1,240 @@
+use std::marker::PhantomData;
+use std::path::Path;
+use std::str;
+
+use crate::{container::Container, macros::def_pyfunction};
+use std::fmt;
+
+use postcard::{from_bytes, to_allocvec, to_vec};
+use pyo3::conversion::IntoPy;
+use pyo3::type_object::PyTypeObject;
+use pyo3::types::{PyBytes, PyDict, PyString};
+use pyo3::{prelude::*, types::PyTuple};
+use serde::de::{MapAccess, Visitor};
+use serde::ser::SerializeMap;
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
+
+
+#[derive(Serialize, Deserialize)]
+struct Object<'s> {
+    id: &'s str,
+    type_: &'s str,
+    datatype: Option<&'s str>,
+    label: MultiLingualString<'s>,
+    description: MultiLingualString<'s>,
+    props: WrappedPyDict<PyString, Vec<WDStatement<'s>>>,
+}
+
+#[derive(Serialize, Deserialize)]
+// struct WDStatement<'s> { dict }
+#[derive(Serialize, Deserialize)]
+struct MultiLingualString<'s> {
+    lang2value: WrappedPyDict<PyString, PyString>,
+    lang: &'s str,
+}
+
+trait PySer {
+    fn pyserialize(&self) -> PyResult<Vec<u8>>;
+    fn pydeserialize(data: &[u8]) -> PyResult<&Self>
+    where
+        Self: Sized;
+}
+
+impl PySer for PyString {
+    fn pyserialize(&self) -> PyResult<Vec<u8>> {
+        return Ok(self.to_str().unwrap().as_bytes().to_owned());
+    }
+    fn pydeserialize(data: &[u8]) -> PyResult<&Self>
+    where
+        Self: Sized,
+    {
+        let py = unsafe { Python::assume_gil_acquired() };
+        Ok(PyString::new(py, str::from_utf8(data).unwrap()))
+    }
+}
+
+#[repr(transparent)]
+struct WrappedPyDict<K, V> {
+    dict: Py<PyDict>,
+    kmarker: PhantomData<K>,
+    vmarker: PhantomData<V>,
+}
+
+impl<K, V> WrappedPyDict<K, V> {
+    fn new(dict: Py<PyDict>) -> Self {
+        Self {
+            dict,
+            kmarker: PhantomData,
+            vmarker: PhantomData,
+        }
+    }
+}
+
+impl<K, V> Serialize for WrappedPyDict<K, V>
+where
+    for<'py> K: PySer + PyTryFrom<'py>,
+    for<'py> V: PySer + PyTryFrom<'py>,
+{
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let py = unsafe { Python::assume_gil_acquired() };
+        let obj = self.dict.as_ref(py);
+
+        let mut map = serializer.serialize_map(Some(obj.len())).unwrap();
+
+        for (key, value) in obj.iter() {
+            map.serialize_entry(
+                &key.downcast::<K>().unwrap().pyserialize().unwrap(),
+                &value.downcast::<V>().unwrap().pyserialize().unwrap(),
+            )
+            .unwrap()
+        }
+        map.end()
+    }
+}
+
+impl<'de, K, V> Deserialize<'de> for WrappedPyDict<K, V>
+where
+    for<'py> K: PySer + PyTryFrom<'py> + ToPyObject,
+    for<'py> V: PySer + PyTryFrom<'py> + ToPyObject,
+{
+    fn deserialize<D>(deserializer: D) -> Result<WrappedPyDict<K, V>, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        deserializer.deserialize_map(WrappedPyDictVisitor {
+            kmarker: PhantomData,
+            vmarker: PhantomData,
+        })
+    }
+}
+
+struct WrappedPyDictVisitor<K, V> {
+    kmarker: PhantomData<K>,
+    vmarker: PhantomData<V>,
+}
+
+impl<'de, K, V> Visitor<'de> for WrappedPyDictVisitor<K, V>
+where
+    for<'py> K: PySer + PyTryFrom<'py> + ToPyObject,
+    for<'py> V: PySer + PyTryFrom<'py> + ToPyObject,
+{
+    type Value = WrappedPyDict<K, V>;
+
+    fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+        formatter.write_str("a dict")
+    }
+
+    fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
+    where
+        A: MapAccess<'de>,
+    {
+        let py = unsafe { Python::assume_gil_acquired() };
+        let mut dict = PyDict::new(py);
+        while let Some((key, value)) = map.next_entry::<Vec<u8>, Vec<u8>>().unwrap() {
+            dict.set_item(
+                K::pydeserialize(&key).unwrap(),
+                V::pydeserialize(&value).unwrap(),
+            )
+            .unwrap();
+        }
+
+        Ok(WrappedPyDict {
+            dict: dict.into_py(py),
+            kmarker: PhantomData,
+            vmarker: PhantomData,
+        })
+    }
+}
+
+impl<'s> MultiLingualString<'s> {
+    pub fn extract<'t>(py: Python<'_>, object: &'t PyAny) -> PyResult<MultiLingualString<'t>> {
+        let lang_1 = object.getattr("lang")?;
+        let lang = lang_1.cast_as::<PyString>()?.to_str()?;
+
+        let lang2value_1 = object.getattr("lang2value")?;
+        let aaa = lang2value_1.cast_as::<PyDict>()?;
+        let lang2value = aaa.into_py(py);
+
+        Ok(MultiLingualString {
+            lang2value: WrappedPyDict::new(lang2value),
+            lang,
+        })
+    }
+}
+
+#[pyfunction]
+pub fn serent(py: Python<'_>, object: &PyAny) -> PyResult<Py<PyBytes>> {
+    let id = object.getattr("id")?.downcast::<PyString>()?.to_str()?;
+    let type_ = object.getattr("type")?.cast_as::<PyString>()?.to_str()?;
+
+    let datatype_1 = object.getattr("datatype")?;
+    let datatype = if datatype_1.is_none() {
+        None
+    } else {
+        Some(datatype_1.cast_as::<PyString>()?.to_str()?)
+    };
+
+    let label = MultiLingualString::extract(py, object.getattr("label")?)?;
+    let description = MultiLingualString::extract(py, object.getattr("description")?)?;
+
+    let obj = Object {
+        id,
+        type_,
+        datatype,
+        label,
+        description,
+    };
+
+    // let serdat = bincode::serialize(&obj).unwrap();
+    let serdat = to_allocvec(&obj).unwrap();
+    Ok(PyBytes::new(py, &serdat).into_py(py))
+}
+
+#[pyfunction]
+pub fn deserent(py: Python<'_>, dat: &PyBytes) -> PyResult<Py<PyTuple>> {
+    // let raw = bincode::deserialize::<Object>(dat.as_bytes()).unwrap();
+    let raw = from_bytes::<Object>(dat.as_bytes()).unwrap();
+    Ok(PyTuple::new(
+        py,
+        &[
+            raw.id.to_object(py),
+            raw.type_.to_object(py),
+            raw.datatype.to_object(py),
+            raw.label.lang2value.dict.to_object(py),
+            raw.label.lang.to_object(py),
+            raw.description.lang2value.dict.to_object(py),
+            raw.description.lang.to_object(py),
+        ],
+    )
+    .into_py(py))
+}
+
+
+pub(crate) fn register(py: Python<'_>, m: &PyModule) -> PyResult<()> {
+    let submodule = PyModule::new(py, "rocksdb")?;
+
+    def_pyfunction!(submodule, "hugedict.hugedict.rocksdb", load);
+    def_pyfunction!(submodule, "hugedict.hugedict.rocksdb", primary_db);
+    def_pyfunction!(submodule, "hugedict.hugedict.rocksdb", stop_primary_db);
+    def_pyfunction!(submodule, "hugedict.hugedict.rocksdb", serent);
+    def_pyfunction!(submodule, "hugedict.hugedict.rocksdb", deserent);
+
+    submodule.add_class::<Options>()?;
+    submodule.add_class::<CompressionOptions>()?;
+    submodule.add_class::<RocksDBDict>()?;
+    submodule.add_class::<SecondaryDB>()?;
+    submodule.add("PrefixExtractor", Container::type_object(py))?;
+    submodule.add("fixed_prefix", Container::type_object(py))?;
+    submodule.add("fixed_prefix_alike", Container::type_object(py))?;
+
+    m.add_submodule(submodule)?;
+
+    py.import("sys")?
+        .getattr("modules")?
+        .set_item("hugedict.hugedict.rocksdb", submodule)?;
+
+    Ok(())
+}
diff --git a/scripts/experimenting_rustbackend/test.py b/scripts/experimenting_rustbackend/test.py
new file mode 100644
index 0000000..1db0107
--- /dev/null
+++ b/scripts/experimenting_rustbackend/test.py
@@ -0,0 +1,66 @@
+import copy
+from pathlib import Path
+import orjson
+from typing import *
+from sm.prelude import M
+from kgdata.wikidata.models import WDEntity
+from kgdata.wikidata.models.multilingual import MultiLingualString
+from hugedict.hugedict.rocksdb import deserent, serent
+from dataclasses import dataclass
+
+
+@dataclass
+class TestEnt:
+    __slots__ = (
+        "id",
+        "type",
+        "datatype",
+        "label",
+        "description",
+    )
+
+    id: str
+    type: Literal["item", "property"]
+    label: MultiLingualString
+    datatype: Optional[str]
+    description: MultiLingualString
+
+    def to_dict(self):
+        return {
+            "id": self.id,
+            "type": self.type,
+            "label": self.label.to_dict(),
+            "datatype": self.datatype,
+            "description": self.description.to_dict(),
+        }
+
+    @staticmethod
+    def from_dict(o):
+        o["label"] = MultiLingualString(**o["label"])
+        o["description"] = MultiLingualString(**o["description"])
+        return TestEnt(**o)
+
+
+data_dir = Path(__file__).parent.parent / "data"
+data_dir / "entities"
+file = list((data_dir / "entities").glob("*.gz"))[0]
+
+obj = M.deserialize_jl(file, n_lines=1)[0]
+ent = WDEntity.from_dict(copy.deepcopy(obj))
+test_ent = TestEnt.from_dict(
+    {k: obj[k] for k in ["id", "type", "datatype", "label", "description"]}
+)
+
+with M.Timer().watch_and_report("rust dumps", precision=7):
+    bin = serent(ent)
+
+with M.Timer().watch_and_report("rust loads", precision=7):
+    ent2 = deserent(bin)
+
+with M.Timer().watch_and_report("orjson dumps", precision=7):
+    bjson = orjson.dumps(test_ent.to_dict())
+
+with M.Timer().watch_and_report("orjson loads", precision=7):
+    test_ent2 = orjson.loads(bjson)
+
+print(">>>", M.percentage(len(bin), len(bjson)))
diff --git a/kgdata/wikidata/deprecated/models/__init__.py b/tests/kg_data/wikidata/__init__.py
similarity index 100%
rename from kgdata/wikidata/deprecated/models/__init__.py
rename to tests/kg_data/wikidata/__init__.py
diff --git a/tests/kg_data/wikidata/test_db.py b/tests/kg_data/wikidata/test_db.py
new file mode 100644
index 0000000..3b3963b
--- /dev/null
+++ b/tests/kg_data/wikidata/test_db.py
@@ -0,0 +1,47 @@
+from pathlib import Path
+from kgdata.wikidata.db import get_entity_db, get_wdclass_db, get_wdprop_db
+import gc
+
+from sm.misc.funcs import assert_not_null
+
+
+def test_get_entity_db(tmp_path: Path):
+    db = get_entity_db(tmp_path / "entities.db", proxy=True)
+
+    assert db["Q513"].label == "Mount Everest"
+    assert assert_not_null(db.get("Q513")).label == "Mount Everest"
+
+    del db
+    gc.collect()
+
+    db = get_entity_db(tmp_path / "entities.db", proxy=False)
+    assert db["Q513"].label == "Mount Everest"
+
+
+def test_get_wdclass_db(tmp_path: Path):
+    db = get_wdclass_db(tmp_path / "classes.db", proxy=True)
+
+    assert db["Q8502"].label == "mountain"
+    assert assert_not_null(db.get("Q8502")).label == "mountain"
+
+    del db
+    gc.collect()
+
+    db = get_wdclass_db(tmp_path / "classes.db", proxy=False)
+    assert db["Q8502"].label == "mountain"
+
+
+def test_get_wdprop_db(tmp_path: Path):
+    db = get_wdprop_db(tmp_path / "props.db", proxy=True)
+
+    assert db["P131"].label == "located in the administrative territorial entity"
+    assert (
+        assert_not_null(db.get("P131")).label
+        == "located in the administrative territorial entity"
+    )
+
+    del db
+    gc.collect()
+
+    db = get_wdprop_db(tmp_path / "props.db", proxy=False)
+    assert db["P131"].label == "located in the administrative territorial entity"
diff --git a/tests/kg_data/wikipedia/test_table_parser.py b/tests/kg_data/wikipedia/test_table_parser.py
deleted file mode 100644
index bad6b28..0000000
--- a/tests/kg_data/wikipedia/test_table_parser.py
+++ /dev/null
@@ -1,286 +0,0 @@
-from kgdata.wikipedia.prelude import Table as T, Row as R, Cell as C, TableVisitor
-import pytest
-
-
-def test_parse_canonical_table():
-    tbl = TableVisitor().parse("""
-{| class="wikitable"
-|-
-! Name and special position!! Appointment date
-|-
-| '''[[Arifin Zakaria]]'''<br />[[Chief Justice of Malaysia|Chief Justice]] || 12 September 2011
-|-
-| Hassan Lah || 10 August 2011
-|}
-    """)
-    assert tbl == T(
-        attrs={"class": "wikitable"},
-        rows=[
-            R(cells=[
-                C(value='Name and special position', is_header=True),
-                C(value='Appointment date', is_header=True)
-            ]),
-            R(cells=[
-                C(value=
-                  r"'''[[Arifin Zakaria]]'''<br />[[Chief Justice of Malaysia|Chief Justice]]"
-                  ),
-                C(value='12 September 2011')
-            ]),
-            R(cells=[C(value="Hassan Lah"),
-                     C(value="10 August 2011")])
-        ])
-
-
-def test_parse_with_cell_properties():
-    tbl = TableVisitor().parse("""
-    {| class="wikitable"
-    |-
-    ! Column 1 !! Column 2 !! Column 3
-    |-
-    | rowspan="2" | A || colspan="2" style="text-align: center;" | B
-    |-
-    | C <!-- column 1 occupied by cell A -->
-    | D
-    |-
-    | E
-    | rowspan="2" colspan="2" style="text-align: center;" |F
-    |-
-    | G <!-- column 2+3 occupied by cell F -->
-    |-
-    | colspan="3" style="text-align: center;" | H
-    |}
-    """)
-    assert tbl == T(
-        attrs={"class": "wikitable"},
-        rows=[
-            R(cells=[
-                C(value='Column 1', is_header=True),
-                C(value='Column 2', is_header=True),
-                C(value='Column 3', is_header=True)
-            ]),
-            R(cells=[
-                C(value='A', attrs={"rowspan": "2"}),
-                C(value='B',
-                  attrs={
-                      "colspan": "2",
-                      "style": "text-align: center;"
-                  })
-            ]),
-            R(cells=[
-                C(value="C <!-- column 1 occupied by cell A -->"),
-                C(value="D")
-            ]),
-            R(cells=[
-                C(value="E"),
-                C(value="F",
-                  attrs={
-                      "rowspan": "2",
-                      "colspan": "2",
-                      "style": "text-align: center;"
-                  })
-            ]),
-            R(cells=[C(value="G <!-- column 2+3 occupied by cell F -->")]),
-            R(cells=[
-                C(value="H",
-                  attrs={
-                      "colspan": "3",
-                      "style": "text-align: center;"
-                  })
-            ])
-        ])
-
-
-def test_parse_multiline_cells():
-    result = TableVisitor().parse("""
-    {|class="wikitable" style="float: right; margin:1em;"
-    |-
-    | Col 1 content
-    continue
-    | Col 2
-    continue | still col 2
-    | Col 3
-    continue || should not be col 3
-    |}
-    """)
-    assert result == T(attrs={
-        "class": "wikitable",
-        "style": "float: right; margin:1em;"
-    },
-                       rows=[
-                           R(cells=[
-                               C(value="Col 1 content\n    continue"),
-                               C(value="Col 2\n    continue | still col 2"),
-                               C(value="Col 3\n    continue"),
-                               C(value="should not be col 3")
-                           ])
-                       ])
-
-
-
-def test_parse_optional_first_row_delimiter():
-    result = TableVisitor().parse("""
-    {| class=wikitable style="text-align:center;"
-    ! H1
-    ! H2
-    |-
-    | A || B
-    |}""")
-    assert result == T(attrs={"class": "wikitable", "style": "text-align:center;"},
-                       rows=[
-                           R(cells=[
-                               C(value="H1", is_header=True),
-                               C(value="H2", is_header=True)
-                           ]),
-                           R(cells=[
-                               C(value="A", is_header=False),
-                               C(value="B", is_header=False)
-                           ]),
-                       ])
-
-
-def test_parse_empty_row():
-    result = TableVisitor().parse("""
-    {| class="wikitable sortable"
-    !align="left" width="220"|Charts (2007)
-    !align="center"|Peak<br>position
-    |-
-    |align="left"|Australian [[ARIA Charts|ARIA]] Top 20 Compilations Chart
-    |align="center"|1
-    |-
-    |}""")
-    assert result == T(
-        attrs={"class": "wikitable sortable"},
-        rows=[
-            R(cells=[
-                C(value="Charts (2007)",
-                  attrs={
-                      "align": "left",
-                      "width": "220"
-                  },
-                  is_header=True),
-                C(value="Peak<br>position",
-                  attrs={"align": "center"},
-                  is_header=True)
-            ]),
-            R(cells=[
-                C(value=
-                  "Australian [[ARIA Charts|ARIA]] Top 20 Compilations Chart",
-                  attrs={"align": "left"},
-                  is_header=False),
-                C(value="1", attrs={"align": "center"}, is_header=False)
-            ]),
-            R(cells=[])
-        ])
-
-
-def test_parse_multi_empty_cells():
-    result = TableVisitor().parse("""
-    {| class="wikitable"
-    |-
-    | Rank ||||
-    |}
-    """)
-    assert result == T(
-        attrs={'class': 'wikitable'},
-        rows=[
-            R(cells=[
-                C(value="Rank"),
-                C(value=""),
-                C(value=""),
-            ])
-        ]
-    )
-    result = TableVisitor().parse("""
-    {| class="wikitable"
-    |-
-    ! Rank !! Name !! CD1 !! CD2 !! OD !! FD
-    |-
-    ! WD
-    | [[Renée Roca]] / [[James Yorke (figure skater)|James Yorke]] ||||||||
-    |}
-    """)
-    assert result == T(
-        attrs={'class': 'wikitable'},
-        rows=[
-            R(cells=[
-                C(value='Rank', is_header=True),
-                C(value='Name', is_header=True),
-                C(value='CD1', is_header=True),
-                C(value='CD2', is_header=True),
-                C(value='OD', is_header=True),
-                C(value='FD', is_header=True)
-            ]),
-            R(cells=[
-                C(value='WD', is_header=True),
-                C(value=
-                  '[[Renée Roca]] / [[James Yorke (figure skater)|James Yorke]]',
-                  is_header=False),
-                C(value='', is_header=False),
-                C(value='', is_header=False),
-                C(value='', is_header=False),
-                C(value='', is_header=False)
-            ])
-        ])
-
-
-@pytest.mark.skip
-def test_unknown_space():
-    result = TableVisitor().parse("""
-{| class="wikitable"
-|-
-! Name !! Position !! Year
-|-
-| Choc Sanders || [[Guard (American football)|Guard]] || 1928
-|-
-| Marion Hammon || [[Tackle (American football)|Tackle]] || 1929
-|-
-| Speedy Mason || [[halfback (American football)|Halfback]] || 1931
-|-
-| Clyde Carter || [[Tackle (American football)|Tackle]] || 1934
-|-
-| Harry Shuford<br />[[Bob Wilson (American football)|Bobby Wilson]] || [[Fullback (American football)|Fullback]]<br />[[halfback (American football)|Halfback]]|| 1934
-|-
-| Harry Shuford<br />[[Bob Wilson (American football)|Bobby Wilson]]<br />[[Truman Spain|Truman "Big Dog" Spain]]<br />[[J. C. Wetsel|J.C. "Iron Man" Wetsel]]|| [[Fullback (American football)|Fullback]]<br />[[halfback (American football)|Halfback]]<br />[[Tackle (American football)|Tackle]]<br />[[Guard (American football)|Guard]] || 1935
-|-
-| Kelly Simpson || [[End (American football)|End]] || 1941
-|-
-| Tom Dean || [[Tackle (American football)|Tackle]] || 1945
-|-
-| [[Doak Walker|Doak "The Doaker" Walker]] || [[halfback (American football)|Halfback]] || 1947
-|-
-| Doak Walker || [[halfback (American football)|Halfback]] || 1948
-|-
-| Doak Walker || [[halfback (American football)|Halfback]] || 1949
-|-
-| [[Kyle Rote|Kyle "The Mighty Mustang" Rote]] || [[halfback (American football)|Halfback]] || 1950
-|-
-| Dick Hightower || [[Center (American football)|Center]] || 1951
-|- [[Forrest Gregg]] || [[Tackle (American football)|Tackle]] || 1955
-|-
-| [[Don Meredith|Don "Dandy Don" Meredith]] || [[Quarterback]] || 1958
-|-
-| Don Meredith || [[Quarterback]] || 1959
-|-
-| [[John LaGrone]] || [[Guard (American football)|Guard]] || 1966
-|-
-| [[Jerry LeVias]] || [[Wide Receiver]] || 1968
-|-
-| Robert Popelka || [[Defensive End]] || 1972
-|-
-| [[Louie Kelcher]]<br />[[Oscar Roan]] || [[Guard (American football)|Guard]]<br />[[Tight End]] || 1974
-|-
-| [[Emanuel Tolbert]] || [[Wide Receiver]] || 1978
-|-
-| [[John Simmons (American football)|John Simmons]] || [[Defensive back]] || 1980
-|-
-| [[Harvey Armstrong]] || [[Defensive Tackle]] || 1981
-|-
-| [[Eric Dickerson]] || [[Running Back]] || 1982
-|-
-| [[Russell Carter]] || [[Defensive back]] || 1983
-|-
-| [[Reggie Dupard]] || [[Running Back]] || 1985
-|-
-| John Stewert || [[Placekicker]] || 1993
-|}
-    """)
\ No newline at end of file