Repository: allenai/ir_datasets (language: python)

Issue:
HC4
**Dataset Information:**

The HC4 collection that is accepted at ECIR 2022

<brief description>

**Links to Resources:**

https://github.com/hltcoe/HC4/tree/main/resouces/hc4

**Dataset ID(s) & supported entities:**

 - Dataset ID: hc4/{language id: zh, fa, ru}/{train, dev, test}
 - Will have {Chinese, Farsi, Russian} documents, English queries(title/description/narrative), an English report associated with each topic and qrels. 

**Checklist**

Mark each task once completed. All should be checked prior to merging a new dataset.
  
 - [x] Dataset definition (in `ir_datasets/datasets/[topid].py`)
 - [x] Tests (in `tests/integration/[topid].py`)
 - [x] Metadata generated (using `ir_datasets generate_metadata` command, should appear in `ir_datasets/etc/metadata.json`)
 - [x] Documentation (in `ir_datasets/etc/[topid].yaml`)
   - [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
 - [x] Downloadable content (in `ir_datasets/etc/downloads.json`)
   - [x] Download verification action (in `.github/workflows/verify_downloads.yml`). Only one needed per `topid`.
   - [x] ~~Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in `downloads.json`.~~
  
**Additional comments/concerns/ideas/etc.**

The document id, qrels, and topics will be distributed through a public github repository. Users need to download the actual documents through Common Crawl. Script for downloading and validating will be provided along with the doc ids. 

The structure will be very similar to the future NeuCLIR collection. Whether these two collections will be distributed through the same repository is TBD. 



Discussion / hints:
For the IDs, I suggest:

```
hc4 # empty top-level placeholder --- or a concatenation of all three corpora? (Would that be useful?)
hc4/{zh,fa,ru} # docs only (for the specified language)
hc4/{zh,fa,ru}/{train,dev,test} # docs (inherited from above), queries&qrels (for particular subset)
```

What's the nature of the report that's associated with each query? (e.g., how long is it?)
Thanks, Sean. This makes sense. 
The top-level should be just placeholder. At this point, I don't think combining all three languages makes sense. 
Even though there are some topics that span across languages (i.e. same title and description), but the narratives are different. So they should be considered different queries. 

Reports range from 1 to 5 paragraphs. Conceptually, they are written by the analysts prior to the search to reflect some background of the information need.  
Forgot to mention, we also have the human and machine translation of the title and descriptions. 
All titles and descriptions have MT translations in all 3 languages. 
Human translation are only available for the language we that we have qrels for. For example, a topic 1 is available for Chinese and Farsi but not Russian, its title and description would come with English,  human translated Chinese, human translated Farsi, MT translated Chinese, Farsi and Russian. 

I am thinking about something like this and leave fields that are not available as None. But this looks tedious and not really robust. 
```python
class HC4Query(NamedTuple):
    query_id: str
    title_en: str
    ht_title_zh: str
    ht_title_fa: str
    ht_title_ru: str
    mt_title_zh: str
    mt_title_fa: str
    mt_title_ru: str
    # and other stuff
```

Alternatively, we could make it another level in the dataset id. So something like this --
```
hc4 # empty top-level placeholder --- or a concatenation of all three corpora? (Would that be useful?)
hc4/{zh,fa,ru} # docs only (for the specified language)
hc4/{zh,fa,ru}/{train,dev,test} # docs (inherited from above), qrels (for particular subset)
hc4/{zh,fa,ru}/{train,dev,test}/{org,ht,mt} # queries with different languages/sources
```

But this structure will lose the ability to provide MT queries that don't come with qrels (i.e. the MT translated title/descriptions in Russian for topic 1 in the example above). 

What do you think, @seanmacavaney? 


There's always something new!

What would you think about having it a dict with the keys as the language code? E.g.,

```python
class HC4Query(NamedTuple):
    query_id: str
    title: str # can we omit en from the title? Maybe it can be assumed since it's a CLIR task
    ht_titles: Dict[str, str] # e.g., {"zh": "XXX", "fa": "YYY"}
    mt_titles: Dict[str, str]
    # and other stuff
```

Pros:
 - All data are available
 - Doesn't expand # of sub-datasets

Cons:
 - Tools that use irds may not expand / know how to handle a dict -- most just support pulling from single fields.
   -  Then again, this is a pretty darn specialized setting. Not only would the system need to know to pull from a dict in a field, but also to match up the translation to the corresponding document language. So maybe this isn't such a big deal.
 - Dict is mutable -- I've tried to keep the structures immutable when possible. Python doesn't have a frozendict either.


But wait, isn't this already under a particular language? E.g., `hc4/zh/train`. What would be the reason to have a english-to-farsi query translation under a chinese corpus? Am I misunderstanding HC4 -- is it a combined corpus?
I personally is leaning toward keeping the the flat structure to allow more/easier adaptions.
How about a middle ground -- 
```python
class HC4Query(NamedTuple):
    query_id: str
    title: str # can we omit en from the title? Maybe it can be assumed since it's a CLIR task
    ht_titles: Dict[str, str] # e.g., {"zh": "XXX", "fa": "YYY"}
    mt_titles: Dict[str, str]
    # and other stuff

   def __getattr__(self, key):
       key = key.split("_") # e.g. "ht_zh_titles"
       if key[0] == 'ht':
           return self.ht_titles[key[1]]
       elif key[0] == 'mt':
           return self_mt_titles[key[1]]
       else:
           raise AttributeError
```



> What would be the reason to have a english-to-farsi query translation under a chinese corpus? Am I misunderstanding HC4 -- is it a combined corpus?

Probably not really some strong use cases. Our original intention is to provide them so if people really want to test Farsi-to-Chinese CLIR, there is at least a translated query that they can use. But this is really an edge case. 


> Probably not really some strong use cases. Our original intention is to provide them so if people really want to test Farsi-to-Chinese CLIR, there is at least a translated query that they can use. But this is really an edge case.

If that's the case, I'd say:

```
hc4 # empty top-level placeholder --- or a concatenation of all three corpora? (Would that be useful?)
hc4/{zh,fa,ru} # docs only (for the specified language)
hc4/{zh,fa,ru}/{train,dev,test} # docs (inherited from above), qrels (for particular subset)
```

With:

```
class HC4Query(NamedTuple):
    query_id: str
    title: str # en
    ht_title: str # whatever language we're currently under (e.g., if this is `hc4/zh/train`, query is in zh)
    mt_title: str
    # and other stuff
```

I think this would be the easiest to use in most cases. If you want total CLIR, use `title`. If you want total monolingual, use `ht_title`. If you want CLIR that assumes the query has already been translated, use `mt_title`. And it will work with whatever language you're currently using for docs.

fa-to-zh (or similar) CLIR would still be achievable if you, say, use `[ht|mt]_title` from `hc4/fa/train` with docs and qrels from `hc4/zh/train`. Which isn't so bad, especially for an edge case that you'd really have to know what you were doing anyway for.
> fa-to-zh (or similar) CLIR would still be achievable if you, say, use [ht|mt]_title from hc4/fa/train with docs and qrels from hc4/zh/train. Which isn't so bad, especially for an edge case that you'd really have to know what you were doing anyway for.

So the issue with this is that the set of topics in `hc4/fa/train` would be different than `hc4/zh/train`. So let's say topic 2 is exclusively judged in Chinese subset. It will only appear in `hc4/zh/train` and not in `hc4/fa/train`. But this in fa-to-zh CLIR setting, topic 2 with MT translated queries in Farsi is exactly what we want here. 
So that means they'd also have to filter down to the intersection of the topics in `hc4/fa/train` and `hc4/zh/train`?

I think I could live with that, considering that we expect this use case to be rare and the simplicity it gets us (and the users!) in the topic class.

The documentation could mention that some of the topics are aligned across languages to enable this.
If that's can be achieved by a filter, I think that is good enough for us. 
But consider this case -- 
```
Topic 1 -- judged in Chinese
Topic 2 -- judged in Farsi
Topic 3 -- judged in both Chinese and Farsi
```

And in the above scheme, we will have 
```
hc4/zh/train -- containing topic 1 and 3 with [ht|mt]_titles both Chinese
hc4/fa/train -- containing topic 2 and 3 with [ht|mt]_titles both Farsi
```
I don't think we want to have all topic 1, 2, and 3 appear in both the `hc4/fa/train` and `hc4/zh/train` query iterators. 

In this example, if we want to do fa-to-zh (query in fa and documents in zh), we want topic 1+3(queries in `hc4/zh/train` with Farsi MT results but the `mt_title` fields will be containing Chinese MTs) because of the qrels instead of 2+3. 

Yeah, that's why I was saying in this edge case, the user would have to take the intersection of qids in `hc4/fa/train` and `hc4/zh/train` -- leaving just Topic 3. They'd also have to filter the qrels, true.
Agreed. And for users who really need the full capability of fa-to-zh CLIR, they can also go back to the original source to extract the queries. 

Fix patch:
diff --git a/.github/workflows/verify_downloads.yml b/.github/workflows/verify_downloads.yml
index d3dd1e2e..7bd91472 100644
--- a/.github/workflows/verify_downloads.yml
+++ b/.github/workflows/verify_downloads.yml
@@ -1037,3 +1037,29 @@ jobs:
       with:
         name: "wikir.json"
         path: "wikir.json"
+
+  hc4:
+    if: "!github.event.inputs.dataset || github.event.inputs.dataset == 'hc4'"
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+    - name: Test
+      env:
+        IR_DATASETS_DL_DISABLE_PBAR: 'true'
+      run: |
+        python -m test.downloads --filter "^hc4/" --output "hc4.json" --randdelay 60
+    - name: Upload artifact
+      if: always() # don't skip if the Test step fails (defeats the whole purpose)
+      uses: actions/upload-artifact@v2
+      with:
+        name: "hc4.json"
+        path: "hc4.json"
+  
\ No newline at end of file
diff --git a/README.md b/README.md
index 9548a75f..fd38c2d5 100644
--- a/README.md
+++ b/README.md
@@ -168,6 +168,7 @@ Available datasets include:
  - [DPR Wiki100](https://ir-datasets.com/dpr-w100.html)
  - [GOV](https://ir-datasets.com/gov.html)
  - [GOV2](https://ir-datasets.com/gov2.html)
+ - [HC4](https://ir-datasets.com/hc4.html)
  - [Highwire (TREC Genomics 2006-07)](https://ir-datasets.com/highwire.html)
  - [Medline](https://ir-datasets.com/medline.html)
  - [MSMARCO (document)](https://ir-datasets.com/msmarco-document.html)
@@ -240,3 +241,4 @@ Contributors to this repository:
  - Thomas Jänich (University of Glasgow)
  - Jan Heinrich Reimer (Martin Luther University Halle-Wittenberg)
  - Maik Fröbe (Martin Luther University Halle-Wittenberg)
+ - Eugene Yang (Johns Hopkins University)
diff --git a/ir_datasets/datasets/__init__.py b/ir_datasets/datasets/__init__.py
index dc5b9182..12cf367a 100644
--- a/ir_datasets/datasets/__init__.py
+++ b/ir_datasets/datasets/__init__.py
@@ -41,3 +41,4 @@
 from . import wikir
 from . import trec_fair_2021
 from . import trec_cast # must be after wapo,car,msmarco_passage
+from . import hc4
diff --git a/ir_datasets/datasets/hc4.py b/ir_datasets/datasets/hc4.py
new file mode 100644
index 00000000..64a580e4
--- /dev/null
+++ b/ir_datasets/datasets/hc4.py
@@ -0,0 +1,172 @@
+from typing import Dict, NamedTuple
+import json
+import ir_datasets
+from ir_datasets.formats.base import BaseDocs, BaseQueries
+from ir_datasets.indices.lz4_pickle import PickleLz4FullStore
+from ir_datasets.util import DownloadConfig
+from ir_datasets.formats import TrecQrels
+from ir_datasets.datasets.base import Dataset, YamlDocumentation
+
+NAME = 'hc4'
+
+LANG_CODE_CONVERT = {
+    'zh': 'zho',
+    'fa': 'fas',
+    'ru': 'rus'
+}
+
+DOC_COUNTS = {
+    'zh': 646305,
+    'fa': 486486,
+    'ru': 4721064
+}
+
+class HC4Doc(NamedTuple):
+    doc_id: str
+    title: str
+    text: str
+    url: str
+    time: str
+    cc_file: str
+
+
+class HC4Docs(BaseDocs):
+    
+    def __init__(self, docs_dlc, subset_lang):
+        self._docs_dlc = docs_dlc
+        self._subset_lang = subset_lang
+        self._count = DOC_COUNTS[subset_lang]
+
+    def docs_path(self, force=True):
+        return self._docs_dlc.path(force)
+
+    @ir_datasets.util.use_docstore
+    def docs_iter(self):
+        with self._docs_dlc.stream() as f:
+            for line in f:
+                line = json.loads(line)
+                line['doc_id'] = line['id']
+                del line['id']
+                yield HC4Doc(**line)
+
+    def docs_store(self):
+        return PickleLz4FullStore(
+            path=f'{self.docs_path(force=False)}.pklz4',
+            init_iter_fn=self.docs_iter,
+            data_cls=self.docs_cls(),
+            lookup_field='doc_id',
+            index_fields=['doc_id'],
+            count_hint=self._count,
+        )
+
+    def docs_cls(self):
+        return HC4Doc
+
+    def docs_namespace(self):
+        return NAME
+
+    def docs_count(self):
+        return self._count
+
+    def docs_lang(self):
+        return self._subset_lang 
+
+
+class HC4Query(NamedTuple):
+    query_id: str
+    title: str
+    description: str
+    ht_title: str
+    ht_description: str
+    mt_title: str
+    mt_description: str
+    narrative_by_relevance: Dict[str, str]
+    report: str
+    report_url: str
+    report_date: str
+    translation_lang: str
+
+class HC4Queries(BaseQueries):
+    def __init__(self, queries_dlc, subset_lang):
+        self._queries_dlc = queries_dlc
+        self._subset_lang = subset_lang
+
+        self._subset_lang_hc4 = LANG_CODE_CONVERT[self._subset_lang]
+    
+    def queries_path(self):
+        return self._queries_dlc.path()
+    
+    def queries_cls(self):
+        return HC4Query
+
+    def queries_namespace(self):
+        return NAME
+    
+    def queries_iter(self):
+        with self._queries_dlc.stream() as f:
+            for line in f:
+                line = json.loads(line)
+                if self._subset_lang_hc4 in line['languages_with_qrels']:
+                    yield self._produce_query(line)
+    
+    def _produce_query(self, line):
+        resources = {}
+        for tp in line['topics']:
+            if tp['lang'] == 'eng':
+                resources['org'] = tp
+            elif tp['lang'] == self._subset_lang_hc4:
+                if tp['source'] == 'human translation':
+                    resources['ht'] = tp
+                else: # machine translation
+                    resources['mt'] = tp
+        return HC4Query(
+            query_id=line['topic_id'],
+            title=resources['org']['topic_title'],
+            description=resources['org']['topic_description'],
+            ht_title=resources['ht']['topic_title'],
+            ht_description=resources['ht']['topic_description'],
+            mt_title=resources['mt']['topic_title'],
+            mt_description=resources['mt']['topic_description'],
+            narrative_by_relevance=line['narratives'][self._subset_lang_hc4],
+            report=line['report']['text'],
+            report_url=line['report']['url'],
+            report_date=line['report']['date'],
+            translation_lang=self._subset_lang
+        )
+
+
+QREL_DEFS = {
+    3: 'Very-valuable. Information in the document would be found in the lead paragraph of a report that is later written on the topic.',
+    1: 'Somewhat-valuable. The most valuable information in the document would be found in the remainder of such a report.',
+    0: 'Not-valuable. Information in the document might be included in a report footnote, or omitted entirely.',
+}
+
+def _init():
+    subsets = {}
+    base_path = ir_datasets.util.home_path()/NAME
+    dlc = DownloadConfig.context(NAME, base_path)
+    documentation = YamlDocumentation(f'docs/{NAME}.yaml')
+
+    base = Dataset(documentation('_')) # dummy top level ds
+
+    for lang in ['zh', 'fa', 'ru']:
+        lang_docs = HC4Docs(dlc[f'{lang}/docs'], subset_lang=lang)
+        subsets[lang] = Dataset(
+            lang_docs,
+            documentation(lang)
+        )
+        for sep in ['train', 'dev', 'test']:
+            subsets[f'{lang}/{sep}'] = Dataset(
+                lang_docs,
+                HC4Queries(dlc[f'{sep}/topics'], subset_lang=lang),
+                TrecQrels(dlc[f'{lang}/{sep}/qrels'], QREL_DEFS),
+                documentation(f'{lang}/{sep}'),
+            )
+    
+    ir_datasets.registry.register(NAME, base)
+    for s in sorted(subsets):
+        ir_datasets.registry.register(f'{NAME}/{s}', subsets[s])
+
+    return base, subsets
+
+base, subsets = _init()
diff --git a/ir_datasets/docs/bibliography.bib b/ir_datasets/docs/bibliography.bib
index 929584e2..ee19c0a2 100644
--- a/ir_datasets/docs/bibliography.bib
+++ b/ir_datasets/docs/bibliography.bib
@@ -838,3 +838,15 @@ @inproceedings{Froebe2022Anchors
   title =                 {{The Power of Anchor Text in the Neural Retrieval Era}},
   year =                  2022
 }
+
+@article{Lawrie2022HC4,
+  author = {Dawn Lawrie and James Mayfield and Douglas W. Oard and Eugene Yang},
+  title = {HC4: A New Suite of Test Collections for Ad Hoc CLIR},
+  booktitle = {{Advances in Information Retrieval. 44th European Conference on IR Research (ECIR 2022)},
+  year = {2022},
+  month = apr,
+  publisher = {Springer},
+  series = {Lecture Notes in Computer Science},
+  site = {Stavanger, Norway},
+  url = {https://arxiv.org/abs/2201.09992}
+}
\ No newline at end of file
diff --git a/ir_datasets/docs/hc4.yaml b/ir_datasets/docs/hc4.yaml
new file mode 100644
index 00000000..1523371f
--- /dev/null
+++ b/ir_datasets/docs/hc4.yaml
@@ -0,0 +1,154 @@
+_:
+  pretty_name: 'HC4 (HLTCOE CLIR Common-Crawl Collection)'
+  desc: '
+<p>
+HC4 is a new suite of test collections for ad hoc Cross-Language Information Retrieval (CLIR), 
+with Common Crawl News documents in Chinese, Persian, and Russian, topics in English and in 
+the document languages, and graded relevance judgments.
+</p>
+<ul>
+  <li>Documents: Web pages from Common Crawl in Chinese, Persian, and Russian.</li>
+  <li>
+  Queries: English TREC-style title/description queries. Narrative field contains an example passage for each relevance level.
+  Human and machine translation of the titles and descriptions in the target language (i.e., document language) are provided in the query object. 
+  (Titles and descriptions are machine-translated into all three target languages even in the laguages that they are not assessed to facillate CLIR 
+  other than English-to-X pairs, e.g., Persian-to-Chinese. Please refer to the original dataset repository for these additional resources.)
+  </li>
+  <li>Report: Each query comes with an English report that is designed to be written by professional searchers prior to the search.</li>
+  <li>Qrels: Documents are judged in three levels of relevance. Please refer to the dataset paper for the full definition of the levels. </li>
+  <li><a href="https://github.com/hltcoe/HC4">Repository</a></li>
+  <li><a href="https://arxiv.org/abs/2201.09992">Dataset Paper</a></li>
+</ul>'
+  bibtex_ids: ['Lawrie2022HC4']
+  data_access: '
+<p>
+To access the docuemnts of this dataset, you will need to download the documents from Common Crawl. 
+The script for downloading and validating the documents are in <a href="https://github.com/hltcoe/HC4">HLTCOE/HC4</a>. 
+Please use the following command to download the documents:
+</p>
+<code>
+git clone https://github.com/hltcoe/HC4<br/>
+cd HC4<br/>
+pip install -r requirements.txt<br/>
+python download_documents.py --storage ~/.ir_datasets/hc4/ \ <br/>
+                             --zho ./resources/hc4/zho/ids.jsonl.gz \ <br/>
+                             --fas ./resources/hc4/fas/ids.jsonl.gz \ <br/>
+                             --rus ./resources/hc4/rus/ids.*.jsonl.gz \ <br/>
+                             --jobs {number of process} \ <br/>
+                             --check_hash <br/>
+</code>
+<p>
+After download, please also post-process the downloaded file to verify all and only specified documents are downloaded, and 
+modify the ordering of the collection to match the original specified ordering in the id files. 
+</p>
+<code>
+python fix_document_order.py --hc4_file ./data/{lang}/hc4_docs.jsonl \ <br/>
+                             --id_file ./resources/hc4/{lang}/ids*.jsonl.gz \ <br/>
+                             --check_hash
+</code>
+<p>
+You can also store the documents in other directory and create a soft link for <kbd>~/.ir_datasets/hc4/</kbd>.
+</p>
+'
+
+zh:
+  desc: '
+<p>
+The Chinese collection contains English queries and Chinese documents for retrieval. 
+Human and machine translated queries are provided in the query object for running monolingual retrieval or cross-language retrival 
+assuming the machine query tranlstion into Chinese is available. 
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+zh/train:
+  desc: '
+<p>
+Train split of <a class="ds-ref">hc4/zh</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+zh/dev:
+  desc: '
+<p>
+Development split of <a class="ds-ref">hc4/zh</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+zh/test:
+  desc: '
+<p>
+Test split of <a class="ds-ref">hc4/zh</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+fa:
+  desc: '
+<p>
+The Persian collection contains English queries and Persian documents for retrieval. 
+Human and machine translated queries are provided in the query object for running monolingual retrieval or cross-language retrival 
+assuming the machine query tranlstion into Persian is available. 
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+fa/train:
+  desc: '
+<p>
+Train split of <a class="ds-ref">hc4/fa</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+fa/dev:
+  desc: '
+<p>
+Development split of <a class="ds-ref">hc4/fa</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+fa/test:
+  desc: '
+<p>
+Test split of <a class="ds-ref">hc4/fa</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+ru:
+  desc: '
+<p>
+The Russian collection contains English queries and Russian documents for retrieval. 
+Human and machine translated queries are provided in the query object for running monolingual retrieval or cross-language retrival 
+assuming the machine query tranlstion into Russian is available. 
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+ru/train:
+  desc: '
+<p>
+Train split of <a class="ds-ref">hc4/ru</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+ru/dev:
+  desc: '
+<p>
+Development split of <a class="ds-ref">hc4/ru</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
+
+ru/test:
+  desc: '
+<p>
+Test split of <a class="ds-ref">hc4/ru</a>.
+</p>
+'
+  bibtex_ids: ['Lawrie2022HC4']
diff --git a/ir_datasets/etc/downloads.json b/ir_datasets/etc/downloads.json
index 009a4d02..9fb0ed4a 100644
--- a/ir_datasets/etc/downloads.json
+++ b/ir_datasets/etc/downloads.json
@@ -3582,5 +3582,80 @@
     "size_hint": 248875419,
     "expected_md5": "e9c5b81c9df6fdc0e2986fa8ffb8ff12"
   }
+},
+
+"hc4": {
+  "train/topics": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/train.topics.v1-0.jsonl",
+    "size_hint": 104898,
+    "expected_md5": "cf3a43c4085e28ce03f37704771e0e36"
+  },
+  "dev/topics": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/dev.topics.v1-0.jsonl",
+    "size_hint": 81444,
+    "expected_md5": "4c34c546c3e90de4733c3c9411ce9c6d"
+  },
+  "test/topics": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/test.topics.v1-0.jsonl",
+    "size_hint": 964434,
+    "expected_md5": "a311237913a7335d45fe261e9ff7f11e"
+  },
+  "fa/docs": {
+    "instructions": "To use this dataset, you need to download the document files using the HC4 document download the post-processing script here: <https://github.com/hltcoe/HC4>.\nTo proceed, symlink the source file here: {path}",
+    "cache_path": "fas/hc4_docs.jsonl"
+  },
+  "fa/train/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/fas/train.qrels.v1-0.txt",
+    "size_hint": 5152,
+    "expected_md5": "4a343957837ce996a7275a71a30ed806"
+  },
+  "fa/dev/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/fas/dev.qrels.v1-0.txt",
+    "size_hint": 24574,
+    "expected_md5": "d66b62e3733a66b151f895e2f57ee144"
+  },
+  "fa/test/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/fas/test.qrels.v1-0.txt",
+    "size_hint": 113490,
+    "expected_md5": "842c307e3e01a688897fadefc7a9672f"
+  },
+  "zh/train/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/zho/train.qrels.v1-0.txt",
+    "size_hint": 15686,
+    "expected_md5": "67d956dcc5b373ae8b6ce5f360b03987"
+  },
+  "zh/docs": {
+    "instructions": "To use this dataset, you need to download the document files using the HC4 document download the post-processing script here: <https://github.com/hltcoe/HC4>.\nTo proceed, symlink the source file here: {path}",
+    "cache_path": "zho/hc4_docs.jsonl"
+  },
+  "zh/dev/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/zho/dev.qrels.v1-0.txt",
+    "size_hint": 20126,
+    "expected_md5": "f252fb5edeee1fa38ccc8ee1c2a6e6f0"
+  },
+  "zh/test/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/zho/test.qrels.v1-0.txt",
+    "size_hint": 123795,
+    "expected_md5": "1bc5cfcefc49805884142b8b32f1a6ea"
+  },
+  "ru/docs": {
+    "instructions": "To use this dataset, you need to download the document files using the HC4 document download the post-processing script here: <https://github.com/hltcoe/HC4>.\nTo proceed, symlink the source file here: {path}",
+    "cache_path": "rus/hc4_docs.jsonl"
+  },
+  "ru/train/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/rus/train.qrels.v1-0.txt",
+    "size_hint": 4232,
+    "expected_md5": "2e2c52e404a6ee0fe676cb88d63c26bb"
+  },
+  "ru/dev/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/rus/dev.qrels.v1-0.txt",
+    "size_hint": 11532,
+    "expected_md5": "24305749fd39fb0392be171e76ef2510"
+  },
+  "ru/test/qrels": {
+    "url": "https://raw.githubusercontent.com/hltcoe/HC4/main/resources/hc4/rus/test.qrels.v1-0.txt",
+    "size_hint": 133650,
+    "expected_md5": "1d1112350a11289496426ea558c321a2"
+  }
 }
 }
diff --git a/ir_datasets/etc/metadata.json b/ir_datasets/etc/metadata.json
index 44b3bff1..6f8a09da 100644
--- a/ir_datasets/etc/metadata.json
+++ b/ir_datasets/etc/metadata.json
@@ -173,6 +173,19 @@
   "gov2/trec-tb-2006/efficiency/stream3": {"docs": {"_ref": "gov2"}, "queries": {"count": 25000}, "qrels": {"count": 31984, "fields": {"relevance": {"counts_by_value": {"0": 26091, "1": 5467, "2": 426}}}}},
   "gov2/trec-tb-2006/efficiency/stream4": {"docs": {"_ref": "gov2"}, "queries": {"count": 25000}},
   "gov2/trec-tb-2006/named-page": {"docs": {"_ref": "gov2"}, "queries": {"count": 181}, "qrels": {"count": 2361, "fields": {"relevance": {"counts_by_value": {"1": 807, "0": 1554}}}}},
+  "hc4": {},
+  "hc4/fa": {"docs": {"count": 486486, "fields": {"doc_id": {"max_len": 36, "common_prefix": ""}}}},
+  "hc4/fa/dev": {"docs": {"_ref": "hc4/fa"}, "queries": {"count": 10}, "qrels": {"count": 565, "fields": {"relevance": {"counts_by_value": {"0": 456, "3": 63, "1": 46}}}}},
+  "hc4/fa/test": {"docs": {"_ref": "hc4/fa"}, "queries": {"count": 50}, "qrels": {"count": 2522, "fields": {"relevance": {"counts_by_value": {"0": 2101, "1": 215, "3": 206}}}}},
+  "hc4/fa/train": {"docs": {"_ref": "hc4/fa"}, "queries": {"count": 8}, "qrels": {"count": 112, "fields": {"relevance": {"counts_by_value": {"1": 23, "3": 22, "0": 67}}}}},
+  "hc4/ru": {"docs": {"count": 4721064, "fields": {"doc_id": {"max_len": 36, "common_prefix": ""}}}},
+  "hc4/ru/dev": {"docs": {"_ref": "hc4/ru"}, "queries": {"count": 4}, "qrels": {"count": 265, "fields": {"relevance": {"counts_by_value": {"0": 186, "1": 67, "3": 12}}}}},
+  "hc4/ru/test": {"docs": {"_ref": "hc4/ru"}, "queries": {"count": 50}, "qrels": {"count": 2970, "fields": {"relevance": {"counts_by_value": {"0": 2297, "1": 411, "3": 262}}}}},
+  "hc4/ru/train": {"docs": {"_ref": "hc4/ru"}, "queries": {"count": 7}, "qrels": {"count": 92, "fields": {"relevance": {"counts_by_value": {"1": 31, "3": 23, "0": 38}}}}},
+  "hc4/zh": {"docs": {"count": 646305, "fields": {"doc_id": {"max_len": 36, "common_prefix": ""}}}},
+  "hc4/zh/dev": {"docs": {"_ref": "hc4/zh"}, "queries": {"count": 10}, "qrels": {"count": 466, "fields": {"relevance": {"counts_by_value": {"0": 374, "3": 62, "1": 30}}}}},
+  "hc4/zh/test": {"docs": {"_ref": "hc4/zh"}, "queries": {"count": 50}, "qrels": {"count": 2751, "fields": {"relevance": {"counts_by_value": {"0": 2277, "3": 282, "1": 192}}}}},
+  "hc4/zh/train": {"docs": {"_ref": "hc4/zh"}, "queries": {"count": 23}, "qrels": {"count": 341, "fields": {"relevance": {"counts_by_value": {"0": 173, "1": 140, "3": 28}}}}},
   "highwire": {"docs": {"count": 162259, "fields": {"doc_id": {"max_len": 8, "common_prefix": ""}}}},
   "highwire/trec-genomics-2006": {"docs": {"_ref": "highwire"}, "queries": {"count": 28}, "qrels": {"count": 27999, "fields": {"relevance": {"counts_by_value": {"0": 24934, "1": 1237, "2": 1828}}}}},
   "highwire/trec-genomics-2007": {"docs": {"_ref": "highwire"}, "queries": {"count": 36}, "qrels": {"count": 35996, "fields": {"relevance": {"counts_by_value": {"0": 31501, "1": 4495}}}}},


Test patch:
diff --git a/test/integration/hc4.py b/test/integration/hc4.py
new file mode 100644
index 00000000..50bd6203
--- /dev/null
+++ b/test/integration/hc4.py
@@ -0,0 +1,109 @@
+import unittest
+from ir_datasets.datasets.hc4 import HC4Doc, HC4Query
+from ir_datasets.formats import TrecQrel
+from .base import DatasetIntegrationTest
+
+
+class TestHC4(DatasetIntegrationTest):
+    def test_hc4_zh_docs(self):
+        self._test_docs('hc4/zh', count=646305, items={
+            0: HC4Doc(doc_id='a2b32b07-a008-438a-a8c3-b8331a1e306d', title='云南网，云南省重点新闻网站，世界看云南的窗口，云南看世界的视角', text='今年夏季，一片浪漫花海在抚仙湖边横空出世，既美化了抚仙湖，又为澄江的旅游带来了旺盛的人气。', url='http://www.yunnan.cn', time=None, cc_file='crawl-data/CC-NEWS/2016/08/CC-NEWS-20160826124520-00000.warc.gz'),
+            20000: HC4Doc(doc_id='407c5f73-0a81-4da7-8294-496c8dc7c493', title='學測指考一例一休？勞動部：不影響', text='我要評比\n\n一例一休上路，大考中心認為學測、指考均利用周末舉行，恐無法負擔加班費衍生成本，考慮調漲報名費。勞動部對此回應「不受影響」，監考人員受僱於大考中心，監考天數若未達連續6天，就不會產生休息日加班費的問題。\n\n至於入闈出題者，勞動部指出，依勞動部7休1函釋，入闈者經勞資雙方合意，可挪移例假，最多可以連上12天班，不算違法，也可以安排休息日。\n\n(中時)', url='http://www.chinatimes.com/realtimenews/20170109005061-260405', time='2017-01-09T17:25:51+08:00', cc_file='crawl-data/CC-NEWS/2017/01/CC-NEWS-20170109103449-00080.warc.gz'),
+            646304: HC4Doc(doc_id='5eaf1d5b-1591-4b42-9e6f-2c878b721d5e', title='六福村徵鬼！當鬼王陪玩笑傲飛鷹 讓你選擇笑死還是嚇死', text='▲應徵上鬼王還能陪遊客一起搭乘笑傲飛鷹。（圖／六福村提供，下同）\n\n\n\n記者陳凱力／新竹報導\n\n中國人怕鬼，西洋人也怕鬼，全世界的人都怕鬼！鬼門要關了，但六福村主題遊樂園不關門，迎接即將到來的萬聖節，六福村下達急急如律令，號召全台敢秀、愛熱鬧、腦中還有1000種大膽、充滿創意、千奇百怪嚇朋友鬼點子的「鬼王」，一起到村子裡來群魔亂舞開趴吧！\n\n請繼續往下閱讀...\n\n▲六福村開出100名鬼缺。\n\n\n\n徵的是鬼！徵的是鬼！六福村徵的就是鬼！猛鬼團隊限額招募100個名額，教你畫上血淋淋、恐怖指數破表的的鬼妝穿梭在園區，工作內容就是要把遊客嚇得「吱吱叫」，讓園區充滿高分貝驚聲尖叫，一邊當鬼王還能暢玩遊樂設施，像是鬼王陪遊客搭上宛如屍速列車的笑傲飛鷹，神出鬼沒鬼吼軌叫，從白天惡靈狂歡派對到夜晚最恐怖的萬聖遊行惡靈夜驚魂，整個園區都是鬼王舞台，絕對是北台灣下半年度最恐怖又最有趣的打工。\n\n六福村行銷經理黃淑芳說，六福村連續三年推出的萬聖節慶典活動「墓碑鎮」令遊客十分驚艷，面對今年即將到來的萬聖節，延續好評投入更多心力物力準備，猛鬼們將會出沒在10月1日起跑的六福萬聖節慶，滿滿一個月的主題活動，而這項「鬼」工作適合鬼靈精怪的朋友來挑戰，經審查錄取後，將有專業特殊化妝老師教你如何變成一副鬼模樣，並安排豐富的肢體訓練、驚嚇訓練等課程，工作期間更提供免費餐點。園方指出，通過履歷第一階段的朋友將接獲面試邀請，接受「驚嚇測試」項目包含情境演技、角色投入和劇本演繹等測試。\n\n▲100名猛鬼們將會出沒在10月1日起跑的六福萬聖節慶。\n\n這是一份擁有充分舞台的工作，戲劇系、表演科系學生將優先錄取，通過履歷第一階段的朋友將接獲面試邀請，接受「驚嚇測試」項目包含情境演技、角色投入和劇本演繹等測試，擁有其他特殊專長如體操、特技更是加分條件。你，準備好要挑戰當鬼王了嗎？我們不見不散！\n\n更多活動詳情及優惠，歡迎至六福村官網 、六福村官方線上購票 、FB粉絲專頁或猛鬼人力銀行網站查詢。', url='https://www.ettoday.net/news/20190828/1523501.htm', time='2019-08-28T15:00:00', cc_file='crawl-data/CC-NEWS/2019/08/CC-NEWS-20190828113707-01025.warc.gz')
+        })
+    
+    def test_hc4_fa_docs(self):
+        self._test_docs('hc4/fa', count=486486, items={
+            0: HC4Doc(doc_id='9064520f-bc4d-4118-a30e-7d99f5adc612', title='گشن\u200c شاخ پارسی در گردباد تعصب تباری', text='پذيرش > دیدگاه > گشن\u200c شاخ پارسی در گردباد تعصب تباری\n\nزبان و ادبیات پارسی یک هویت فرهنگی است. مرز هویت فرهنگی را نمی\u200cتوان با مرزهای سیال سیاسی و تباری اشتباه گرفت. فرهنگ تعریفش را نه از سیاست می\u200cگیرد و نه از تبار و قبیله. فرهنگ مجموعه داشته\u200cهایی است خیلی فراتر از این مقوله\u200cهای تقلیل\u200cگرا. این سیاست و تبار است که در فرایند تاریخ و تحول، در درون فرهنگ شکل می\u200cگیرند. بنابراین، سیاست و تبار برساخته\u200cهای فرهنگ اند و نه عکس آن. از این\u200cرو، نمی\u200cتوان هیچ سیاست و تبار خاصی را از میان چندین خرده\u200cفرهنگ و کتله\u200cهای تباری برساخته\u200cشده، وارث اصلی و یگانه\u200cی فرهنگ مادر خواند. سهمیه\u200cبندی این میراث بر مبنای تفکیک "خلف" از "ناخلف"، چیزی نیست جز یک انتحار فرهنگی و سلاخی پیکر واحد و تنومند فرهنگ. انتحار فرهنگی صرفن می\u200cتواند از دست یک سیاست فاشیستی و عصبیت کور تباری و قبیلوی برآمده باشد! وارث به\u200cحق و شایسته\u200cی فرهنگ مادر در هر زمانی، صرف نظر از نسبت تباری و قبیلوی و سیاسی، کسانی یا جمعی می\u200cتوانند بود که در جهت استمرار و تعالی آن فرهنگ کاری کرده و خون دلی خورده باشند. لذا، این\u200cکه شخصی با یک روی\u200cکرد کماکان ملاعمری از یک آدرس فرهنگی آمده می\u200cگوید من در انتخابات گذشته به فلانی رأی ندادم اما اگر بسمدانی کاندید می\u200cکرد، به او رأی می\u200cدادم، مگر جز پیروی از یک سیاست فاشیستی و حیوانیت تباری، چیز دیگری را هم می\u200cتواند به نمایش گذارد؟ آقا، تویی که سنگ میراث\u200cداری از کلیت زبان و ادبیات پارسی در یک حوزه\u200cی تمدنی را به سینه می\u200cزنی، اگر خوره\u200cی حیوانیت تباری در مغز جانت خانه نکرده، دیگر این جیره\u200cبندی\u200cها به چی معنا؟ اصلن این یعنی چی که داد از زبان و ادبیات پارسی می\u200cزنیم، اما حرکات و رفتارمان نشان از یک عصبیت افراطی ناسیونالیستی دارند و در حقیقت، در سمت تکه-پاره شدن این پارسی گام بر می\u200cداریم؟ این نعل وارونه دیگر به چی منظور که بر گرده\u200cی یک حوزه تمدنی بزرگ، تعصب تباری\u200cمان را سوار می\u200cکنیم که باری اگر حرف بر سر حقانیت مسأله باشد، این گل سرسبد تبار شاید کم\u200cترین سهم و نقش را در دامن قند پارسی داشته؟ (نابغه\u200cی دیگری از این\u200cسو در یک شبه\u200cتوطئه\u200cی ناسیونالیستی و ضد فرهنگی رفته بابت تمام بی\u200cفرهنگی\u200cهای مهاجرین افغانستانی در ایران(!!) از آن کشور معذرت تاریخی خواسته!!)\n\n\n\nفرهنگی که امروزه یگانه مشخصه و معرف آن زبان و ادبیات دیرینه\u200cسال پارسی است، حوزه\u200cی تمدنی\u200cیی را شامل می\u200cشود که اگر این مرزکشی\u200cهای سیاسی چند سال پسین را در نظر بگیریم افغانستان، تاجیکستان و ایران به یک اندازه در آن سهم دارند و در درون هریک از این کشورها نیز خرده\u200cفرهنگ\u200cها و تبارهایی زیست دارند که هیج\u200cیک بر دیگری ارجحیت و برتری ندارند تا به تنهایی "وارث خلف" آن حوزه\u200cی تمدنی و تاریخش قلمداد شود، جز آنی که در راستای بقا و بالندگی این حوزه\u200cی تمدنی و افتخاراتش، کار و پیکار کرده اند. حوزه\u200cی تمدنی زبان و ادبیات پارسی، ملک طلق هیچ سیاست و تبار یگانه و خاصی نیست! یک خاوری ساکن مشهد همانقدر می\u200cتواند وارث این حوزه\u200cی تمدنی و افتخاراتش باشد که یک ایرانی ساکن تهران و... اساس اشتراک این خرده\u200cفرهنگ\u200cها و تبارها در درون این حوزه\u200cی تمدنی واحد، زبان و ادبیات تاریخی\u200cشان است. فقط زبان و ادبیات در این معنا می\u200cتواند برای هریک از این خرده\u200cهویت\u200cهای شناور و سرگردان، یک هویت واحد باشد و به تعبیر هایدگر: «خانه\u200cی وجودشان». بلی، برای تصاحب شدن و مصادره کردن پارسی که هرگز تصاحب پذیر نمی\u200cتواند بود، نه سکه\u200cی رنگ\u200cرفته\u200cی تبار ارزشی دارد و نه سیاست\u200cهای فاشیستی هتلروار! فقط یک هویت واحد (زبان) است که می\u200cتواند به قول ویتگنشتاین: «مرزهای جهان\u200cمان را تعیین کند» و بس!\n\nبیایید با عصبیت\u200cهای زشت و عق\u200cآور تباری و سیاسی، دامن فرهنگ گران\u200cسنگ پارسی را لکه\u200cدار نکنیم. وگرنه آن بزرگانی که از جان و زندگی\u200cشان مایه گذاشته این کاخ بلند را بنا گذاشته اند و ما امروزه به طرز بی\u200cشرمانه\u200cیی آن\u200cها را نیز به نفع جنون مبتذل تباری خود مصادره می\u200cکنیم، بر روی\u200cمان تف خواهند انداخت!\n\nبه امید بقا و درخشش مدام پارسی و شرمنده باد آنانی که با انتحار فرهنگی\u200cشان می\u200cخواهند این کاخ بلندمان را ویران و کلبه\u200cی حقیرانه\u200cی تباری و سیاسی\u200cشان را بر آوار آن بنا کنند!', url='http://kabulpress.org/article239985.html', time=None, cc_file='crawl-data/CC-NEWS/2016/08/CC-NEWS-20160826124520-00000.warc.gz'),
+            20000: HC4Doc(doc_id='67d086f7-0d09-44e5-b3b8-92caf7930827', title='ورزشکاران حرفه\u200cای در مردم شوق ورزش ایجاد کنند', text='یزد ـ استاندار یزد با اشاره به اینکه مردم به ورزشکاران علاقمند هستند و به آنها توجه نشان می دهند، گفت: شایسته است ورزشکاران حرفه\u200cای در مردم شوق ورزش ایجاد کنند.\n\nبه گزارش خبرنگار مهر، سیدمحمد میرمحمدی در مراسم افتتاحیه شانزدهمین صعود بین المللی کوهنوردان سراسر کشور به شیرکوه در سالن همایش های هلال احمر استان یزد اظهار داشت: مردم از ورزشکاران حرفه ای و محبوب الگوگیری می کنند و خوب است ورزشکاران از این امر به عنوان یک فرصت برای جذب مردم به ورزش استفاده کنند.\n\nوی با اشاره به اینکه حضور کوهنوردان نخبه کشور در یزد در مردم شور و شعف ایجاد می کند، افزود: این حضور و همدلی ها در مردم برای انجام ورزش اشتیاق ایجاد می کند.\n\nاستاندار یزد با شاره به اینکه همواره از گذشته بر این موضوع که عقل سالم در بدن سالم است، تاکید شده است، عنوان کرد: اگر مردم را به ورزش کردن و تحرک تشویق کنیم به طور قطع از بار مشکلات و بیماری های موجود در جامعه کاسته می شود ضمن اینکه مردم انگیزه و نشاط بیشتری برای ادامه زندگی خواهند داشت.\n\nمیرمحمدی با بیان اینکه از ورزش کوهنوردی لذت می برم، افزود: یکی از زیبایی\u200cها و جذابیت\u200cهای کوهنوردی صعود است و در صعود روح تعالی وجود دارد.\n\nوی خاطرنشان کرد: اخلاق ورزشی در ورزش کوهنوردی شکل و صورت دیگری دارد زیرا اخلاق ورزشی از همین همدلی ها و دور هم بودن ها ناشی می شود.\n\nاستاندار یزد از حضور قهرمانان بین المللی نظیر عظیم قیچی\u200cساز، همچنین هشت نفر دیگر از کوهنوردان بین المللی و آقایان عطری و گلشن نیا قهرمانان کوهنوردی استان یزد در این همایش ابراز خرسندی کرد و افزود: حضور ۴۰۰ کوهنورد حرفه ای از سراسر کشور افتخاری برای استان یزد است.\n\nوی اظهار امیدواری کرد: سفر به استان یزد و صعود به شیرکوه خاطرات خوشایندی برای کوهنوردان کشور از یزد بر جای بگذارد.\n\nبه گزارش مهر، در این مراسم از کوهنوردان برتر استان یزد تجلیل شد و از سوی رئیس فدراسیون کوهنوردی نیز پرچمی به استاندار یزد اهدا شد.', url='http://www.mehrnews.com/news/3895020/%D9%88%D8%B1%D8%B2%D8%B4%DA%A9%D8%A7%D8%B1%D8%A7%D9%86-%D8%AD%D8%B1%D9%81%D9%87-%D8%A7%DB%8C-%D8%AF%D8%B1-%D9%85%D8%B1%D8%AF%D9%85-%D8%B4%D9%88%D9%82-%D9%88%D8%B1%D8%B2%D8%B4-%D8%A7%DB%8C%D8%AC%D8%A7%D8%AF-%DA%A9%D9%86%D9%86%D8%AF', time='2017-02-02T08:04:10+00:00', cc_file='crawl-data/CC-NEWS/2017/02/CC-NEWS-20170202041952-00044.warc.gz'),
+            486485: HC4Doc(doc_id='40ab1e8e-a937-4c9e-89a9-9495a021e47e', title='بهره\u200cبرداری از طرح آبرسانی تهران به سد تنظیمی کرج تا خرداد ۹۹', text='به گزارش خبرگزاری مهر به نقل از وزارت نیرو، سید حسن رضوی در حاشیه بازدید از تصفیه\u200cخانه ششم تهران از افزایش سرعت عملیات اجرایی کانال سرریز تصفیه\u200cخانه ششم پایتخت خبر داد و اظهار داشت: به\u200cدنبال رفع موانع اجرایی کانال سرریز تصفیه\u200cخانه ششم تهران واقع در اراضی لتمال کن، از جمله پرداخت هزینه باقی\u200cمانده و تغییر پیمانکار مربوطه، عملیات اجرایی این بخش که حدود ۱۵۰۰ متر کانال بتنی با مقطع پنج متر در پنج متر است، سرعت بیشتری به خود گرفت.\n\nوی افزود: بعد از شروع عملیات اجرایی از تیر ماه امسال در محدوده یاد شده، ۶۰ هزار متر مکعب خاکبرداری و ۱۱۰۰ متر مکعب بتن\u200cریزی انجام شد و انتظار می\u200cرود با افزایش سرعت کار، تا پایان سال جاری بخش عمده کانال مذکور اجرا و وارد مرحله آب\u200cاندازی شود.\n\nمدیرعامل شرکت آب منطقه\u200cای تهران افزود: طرح آبرسانی به تهران از سد تنظیمی کرج در مجموع حدود ۹۶ درصد پیشرفت دارد که با تدابیر اتخاذ شده و با تزریق به\u200cموقع منابع مالی تا خرداد ماه سال آینده به مرحله بهره\u200cبرداری می\u200cرسد.\n\nگفتنی است، کانال سرریز تصفیه\u200cخانه ششم تهران، بخش انتهایی طرح آبرسانی به تهران از سد تنظیمی امیرکبیر است که سرریز تصفیه\u200cخانه ششم تهران را به رودخانه کن هدایت می\u200cکند. بهره\u200cبرداری از تونل انتقال آب کرج – تهران و تصفیه\u200cخانه ششم، منوط به تکمیل این کانال است.', url='https://www.mehrnews.com/news/4704991/%D8%A8%D9%87%D8%B1%D9%87-%D8%A8%D8%B1%D8%AF%D8%A7%D8%B1%DB%8C-%D8%A7%D8%B2-%D8%B7%D8%B1%D8%AD-%D8%A2%D8%A8%D8%B1%D8%B3%D8%A7%D9%86%DB%8C-%D8%AA%D9%87%D8%B1%D8%A7%D9%86-%D8%A8%D9%87-%D8%B3%D8%AF-%D8%AA%D9%86%D8%B8%DB%8C%D9%85%DB%8C-%DA%A9%D8%B1%D8%AC-%D8%AA%D8%A7-%D8%AE%D8%B1%D8%AF%D8%A7%D8%AF-%DB%B9%DB%B9', time='2019-08-28T10:56:28+00:00', cc_file='crawl-data/CC-NEWS/2019/08/CC-NEWS-20190828113707-01025.warc.gz')
+        })
+
+    def test_hc4_ru_docs(self):
+        self._test_docs('hc4/ru', count=4721064, items={
+            0: HC4Doc(doc_id='b899e0e9-61e6-40e9-b9e1-1f19cd2559c6', title='IE 6 style', text='Сегодня мы отмечаем один из главных праздников Республики, знамен', url='http://ugo-osetia.ru/index.php/joomla-pages-iii/category-list/23-ie-6-style', time=None, cc_file='crawl-data/CC-NEWS/2016/08/CC-NEWS-20160826124520-00000.warc.gz'),
+            20000: HC4Doc(doc_id='c5481f83-c357-4ce5-9570-5d74a8efc442', title='Киев кошмарит итальянских политиков из-за намеченной ими поездки в Крым', text='Члены региональных советов нескольких областей Италии, которые запланировали поездку в Крым, получили письма с угрозами от консульства Украины. Об этом журналистам сообщили итальянские политики.\n\n«Письмо было направлено губернаторам. В тексте указывалось, что наши советники нарушают статью уголовного кодекса Украины со всеми вытекающими последствиями. Настоящие угрозы. Это неприемлемо, чтобы третья страна вмешивалась в политическую деятельность представителей другого государства», — заявил советник регионального совета Венето Лучано Сандона (в этот регион входят такие крупные города, как Венеция, Падуя, Верона и Виченца). По словам Сандоны, кроме политиков из региональных советов Венето, Ломбардии, Лигурии, Эмилии-Романьи и Тосканы в делегацию вошли также крупные предприниматели Италии.\n\nНапомним: многочисленная делегация итальянских депутатов и бизнесменов намерена посетить Крым 13—16 октября. Делегаты планируют ознакомиться с бизнес-условиями и инвестиционными возможностями полуострова.', url='http://anna-news.info/node/65960', time=None, cc_file='crawl-data/CC-NEWS/2016/10/CC-NEWS-20161011083804-00021.warc.gz'),
+            4721063: HC4Doc(doc_id='39000c4d-17ea-4b71-9859-09fd60c866c9', title='Три экстремала прощались с летом, прыгая с моста в центре Петербурга', text='Школьники прыгали в воду, проделывая в воздухе различные кульбиты.\n\nУтром 28 августа на Красном мосту в Петербурге заметили трех молодых людей. На записи, которую сделал один из пользователей интернета, видно, как школьники прыгают с высоты в воду, проделывая в воздухе различные кульбиты.\n\nОдин из юношей, прыгая с моста в реку Мойку, кричал: "Света, прощай!". Его примеру последовали два товарища, которые разделись до трусов и бесстрашно кинулись в воду.\n\nАвтор ролика предположил, что таким образом молодые люди прощались с летом. Другой в ответ написал, что "с мозгами они попрощались чуть раньше".\n\nРанее мы писали, что после падения на рельсы на станции метро "Гражданский проспект" женщина осталась без рук.\n\nВидео: ВКонтакте / Признавашки ДТП и ЧП Санкт-Петербург', url='https://piter.tv/event/Tri_molodih_cheloveka_prigali_s_Krasnogo_mosta_v_Mojku/', time=None, cc_file='crawl-data/CC-NEWS/2019/08/CC-NEWS-20190828113707-01025.warc.gz')
+        })
+
+    def test_hc4_zh_query(self):
+        self._test_queries('hc4/zh/train', count=23, items={
+            0: HC4Query(query_id='1001', title='Extremist covert communication advancements', description='How are extremist groups advancing covert communication capabilities with technology?', ht_title='极端分子隐匿通讯的进步', ht_description='极端分子组织如何利用科技来发展隐匿通讯能力？', mt_title='Extremist covert communication advancements', mt_description='EXtremist groups advert communication capabwith technology?', narrative_by_relevance={'very_valuable': '', 'somewhat_valuable': '', 'not_that_valuable': '', 'non_relevant': ''}, report='Even though older communication methods like radio are still used, the revolution in communication technology over the past 10–15 years has dramatically changed how terrorist organizations communicate. E-mails, fax transmissions, websites, cell phones, and satellite telephones have made it possible for organizations to contemplate a global strategy. However, too great a reliance on this new technology leaves organizations vulnerable to sophisticated monitoring of communication and triangulation of its source. When Osama bin Laden found out that his satellite phone conversations were being intercepted, he ceased using this method to communicate.', report_url='https://en.wikipedia.org/w/index.php?title=Tactics_of_terrorism&oldid=834204722#Communications', report_date='2018-04-04', translation_lang='zh'),
+            3: HC4Query(query_id='1004', title='Urban fire with multiple fatalities', description='Details on large urban fires leading to multiple lives lost', ht_title='多人死亡的城市火灾', ht_description='大型城市火灾导致多人死亡的细节', mt_title='城市火灾，有多种脂肪', mt_description='关于大量城市火灾的细节，以减少损失', narrative_by_relevance={'very_valuable': '', 'somewhat_valuable': '', 'not_that_valuable': '', 'non_relevant': ''}, report="A conflagration is a large and destructive fire that threatens human life, animal life, health, and/or property. It may also be described as a blaze or simply a (large) fire. A conflagration can begin accidentally, be naturally caused (wildfire), or intentionally created (arson). Arson can be for fraud, murder, sabotage or diversion, or due to a person's pyromania. A very large fire can produce a firestorm, in which the central column of rising heated air induces strong inward winds, which supply oxygen to the fire. Conflagrations can cause casualties including deaths or injuries from burns, trauma due to collapse of structures and attempts to escape, and smoke inhalation.", report_url='https://en.wikipedia.org/w/index.php?title=Conflagration&oldid=872678379', report_date='2018-12-08', translation_lang='zh')
+        })
+        self._test_queries('hc4/zh/dev', count=10, items={
+            4: HC4Query(query_id='6', title="Gibraltar's Sovereignty After Brexit", description="How will Gibraltar's sovereignty be impacted by Brexit negotiations between Spain and the UK?", ht_title='直布罗陀在脱欧以后主权的情况', ht_description='西班牙与英国之间的脱欧谈判对直布罗陀主权有什么影响？', mt_title='Gustar的累累累累累累累累累累累累累累累累累累累累累', mt_description='在英国和英国的Brexit negotiations中，G é altar的soverignty将如何受到影响？', narrative_by_relevance={'very_valuable': 'Very valuable documents discuss the negotiations between the UK and Spain specifically regarding the status of Gibraltar, and the different possible scenarios for Gibraltar. Very valuable information also includes what residents of Gibraltar prefer, proposed plans between the UK and Spain for Gibraltar, and a brief history of the disagreement between the UK and Spain over Gibraltar and the main areas of disagreement specifically regarding Brexit.', 'somewhat_valuable': "Somewhat valuable information references Gibraltar in the UK's Brexit negotiations and that the UK has a disagreement with Spain, but doesn't go into detail about it and doesn't mention what the implications are for Gibraltar if the UK exits the EU.", 'not_that_valuable': 'Information discussed Brexit negotiations between the UK and EU, but did not mention the issue of Gibraltar or disagreements with Spain.', 'non_relevant': 'Gibraltar was mentioned in regards to Brexit, but no details were provided and it was not the main part of the article.'}, report='Sovereignty\nSee also: Disputed status of Gibraltar\n\nThe day after the result, Spain\'s acting Foreign Minister, José Manuel García-Margallo, renewed calls for joint Spanish–British control of the peninsula. These calls were strongly rebuffed by Gibraltar\'s Chief Minister. After the result Spain reiterated its position that it wanted to jointly govern Gibraltar with the United Kingdom and said it would seek to block Gibraltar from participating in talks over future deals between the UK and EU.\n\nIn April 2017, British Prime Minister Theresa May reiterated that "the UK would seek the best possible deal for Gibraltar as the UK exits the EU, and there would be no negotiation on the sovereignty of Gibraltar without the consent of its people.” \n\nIn April 2018, Spanish Foreign Minister Alfonso Dastis announced that Spain hoped to sign off a bilateral agreement with Britain over Gibraltar before October so as not to hinder a Brexit transition deal. Talks between London and Madrid had progressed well. While reiterating the Spanish long-term aim of "recovering" Gibraltar, he said that Spain would not hold Gibraltar as a "hostage" to the EU negotiations. ', report_url='https://en.wikipedia.org/w/index.php?title=Effect_of_Brexit_on_Gibraltar&oldid=870688779', report_date='2018-11-26', translation_lang='zh')            
+        })
+        self._test_queries('hc4/zh/test', count=50, items={
+            0: HC4Query(query_id='102', title='Shipwrecks and Historical Chinese Trade', description='What information about trade and shipping has been discovered by investigating historical shipwrecks of Chinese vessels?', ht_title='沉船与中国古代贸易', ht_description='通过调查中国历史中的沉船事故，发现了哪些古代贸易和航运信息？', mt_title='Shewrecks and Historical Chinese Trade', mt_description='在中国船舰的高压造船厂中，有哪些关于贸易和航运的信息被破坏了？', narrative_by_relevance={'very_valuable': 'There were two reports found with very valuable information detailed the specific types of cultural relics found on a Song Dynasty shipwreck found in the South China Sea, which indicated porcelain and iron trades with foreign nations during that time period.   ', 'somewhat_valuable': 'These was one report briefly mentioned some cosmetic boxes found in a Chinese historical shipwreck that suggest possible foreign trades.    ', 'not_that_valuable': 'N/A', 'non_relevant': 'There were multiple articles found with reports on cultural relics found on various Chinese historical shipwrecks.  But with no mention of trades.   '}, report='The Chinese had wide connections through trade in Asia and had been sailing to Arabia, East Africa, and Egypt since the Tang Dynasty (AD 618–907). Between 1405 and 1421 the third Ming emperor Yongle sponsored a series of long range tributary missions in the Indian Ocean under the command of admiral Zheng He (Cheng Ho).\n\nA large fleet of new junk ships was prepared for these international diplomatic expeditions. The largest of these junks—that the Chinese termed bao chuan (treasure ships)—may have measured 121 metres (400 feet) stem to stern, and thousands of sailors were involved. The first expedition departed in 1405. At least seven well-documented expeditions were launched, each bigger and more expensive than the last. The fleets visited Arabia, East Africa, India, Malay Archipelago and Thailand (at the time called Siam), exchanging goods along the way. They presented gifts of gold, silver, porcelain and silk; in return, received such novelties as ostriches, zebras, camels, ivory and giraffes. After the emperor\'s death, Zheng He led a final expedition departing from Nanking in 1431 and returning to Beijing in 1433. It is very likely that this last expedition reached as far as Madagascar. The travels were reported by Ma Huan, a Muslim voyager and translator who accompanied Zheng He on three of the seven expeditions, his account published as "Ying-Yai Sheng-Lam" (Overall Survey of the Ocean\'s Shores) (1433).\n\nThese long distance journeys were not followed up, as the Chinese Ming dynasty retreated in the haijin, a policy of isolationism, having limited maritime trade. Travels were halted abruptly after the emperor\'s death, as the Chinese lost interest in what they termed barbarian lands turning inward, and successor emperors felt the expeditions were harmful to the Chinese state; Hongxi Emperor ended further expeditions and Xuande Emperor suppressed much of the information about Zheng He\'s voyages.', report_url='https://en.wikipedia.org/w/index.php?title=Age_of_Discovery&oldid=858657320', report_date='2018-09-08', translation_lang='zh'),
+        })
+
+    def test_hc4_fa_query(self):
+        self._test_queries('hc4/fa/train', count=8, items={
+            0: HC4Query(query_id='1008', title='French protest fuel taxes', description='Information on protests in France against rising fuel taxes and possible outcomes of the protests.', ht_title='اعتراض فرانسه به مالیات سوخت', ht_description='اطلاعات مربوط به اعتراضات در فرانسه علیه افزایش مالیات سوخت و نتایج احتمالی اعتراضات', mt_title='اعتراض فرانسه به مالیات سوخت', mt_description='اطلاعات در مورد اعتراضات در فرانسه در برابر افزایش مالیات سوخت و نتایج احتمالی اعتراضات.', narrative_by_relevance={'very_valuable': '', 'somewhat_valuable': '', 'not_that_valuable': '', 'non_relevant': ''}, report='After Macron was inaugurated on 15 May 2017, there were numerous risks from Labour unions about the prospect of a large organized protest. The CGT Union has attempted numerous times to organise a large-scale demonstration against Macron with one currently taking place on September 12, 2017. Macron has actively tried to prevent this by opening Labor code reform negotiations with trade unions. The reception among the unions has been mixed with the head of the FO union supporting the negotiations, the CFDT deciding to stay neutral, not participating in the September 12th protests and the CGT denouncing the negotiations alongside its ally SUD. Jean-Luc Melenchon from La France Insoumise has spoken in support of the September 12th protest encouraging members to attend. Melenchon himself organized a protest on July 12, 2017.\n\nPresident Donald Trump\'s state visit to France during Bastile Day was met with protests, protesters gathered around Place de la République to create a "No Trump Zone". Protesters were reportedly protesting about the Trump visit and Macron\'s policies with the ranks of the protesters being made up of socialists, pro-Palestinian groups, migrants’ rights activists, environmentalists and anti-fascists. Despite mass protests, 59% of French people approve of Trump\'s visit.\n\nFollowing Prime Minister Edouard Philippe\'s announcement of the plans for immigration reform, a small protest was led by a group of LGBT activists in Paris holding up a sign reading "Macron starves migrants, queers without borders"\n\nA series of protests by wine producers in the South of France have been ongoing since Francois Hollande\'s presidency. These demonstrations generally involve arson, sabotage and assault. These protests are caused by the importation of wine rather than buying it from French producers and the loss of culture. These protests have led to a 25% decrease in sales for Spanish wine producers. Spanish tankers transporting wine are usually the target of these attacks.\n\nPro-Palestinan protesters began to demonstrate against Macron offering Israel Prime Minister Netanyahu a place at the Paris Holocaust Ceremony. The French Communist Party also opposed Netanyahu\'s visit. The organizers of the protest were unknown but Le Muslim Post, a religious radioshow promoted the demonstration, encouraging listeners to attend. ', report_url='https://en.wikipedia.org/w/index.php?title=Protests_against_Emmanuel_Macron&oldid=801910503', report_date='2018-09-22', translation_lang='fa')
+        })
+        self._test_queries('hc4/fa/dev', count=10, items={
+            0: HC4Query(query_id='1', title='Asteroids Endangering Earth', description='Articles related to asteroids that pose danger of impact to Earth.', ht_title='سیارک ها\xa0کره زمین را به خطر می اندازند', ht_description='مقالات مربوط به سیارک ها که خطر ضربه به کره زمین را دارند', mt_title='سیارات در معرض خطر زمین', mt_description='مقالات مربوط به سیارک \u200c هایی که خطر برخورد با زمین را تهدید می \u200c کنند.', narrative_by_relevance={'very_valuable': 'Mention of asteroids striking Earth with description', 'somewhat_valuable': 'Mention of asteroids striking Earth but with little information', 'not_that_valuable': 'May talk about asteroids without real mention of their danger', 'non_relevant': 'Describes meteorites, planets, stars with no relation to striking Earth'}, report='2011 UL21 briefly had about a 1 in a million chance of impacting in 2029. Its cumulative impact probability dropped to 1 in 71 million by 2 November 2011 when the observation arc reached 15 days. It was removed from the Sentry Risk Table on 4 November 2011 when all impact scenarios for the next 100 years or more were ruled out. During 2029, the closest approach to Earth is 1.6 AU. Palomar Observatory precovery images from 1989 and 1990 have extended the observation arc to 22 years. Its next notable close approach to the Earth will be on June 27, 2024 at a distance of 0.044 AU (6,600,000 km; 4,100,000 mi).\n\nWith an absolute magnitude of 15.8, it is one the brightest and therefor largest potentially hazardous asteroids (PHA) detected since (242450) 2004 QY2. The next largest PHA (based on absolute magnitude) discovered in 2011 is 2011 WO41 with an absolute magnitude of 16.8. ', report_url='https://en.wikipedia.org/w/index.php?title=(415029)_2011_UL21&oldid=877055001', report_date='2019-01-06', translation_lang='fa')
+        })
+        self._test_queries('hc4/fa/test', count=50, items={
+            0: HC4Query(query_id='103', title='African Extremist School Girls Kidnapping', description='What is known about instances of extremist or terrorist groups kidnapping girls from schools in Africa?', ht_title='ربودن دختران دانش\u200cآموز توسط گروه های افراطی آفریقایی', ht_description='راجع به وقوع ربودن دختران دانش\u200cآموز توسط گروه های افراطی یا تروریستی از مدرسه ها در آفریقا چی درک  می شود؟', mt_title='دختران مدرسه افراطی آفریقایی Kidnapping', mt_description='در مورد نمونه \u200c هایی از گروه \u200c های افراطی یا گروه \u200c های تروریستی که دختران را از مدارس آفریقا می \u200c ربایند چه می \u200c توان گفت ؟', 
+                        narrative_by_relevance={'very_valuable': "Details about kidnapping operations by Boko Haram in Nigerian schools (victims often drugged, taken from schools overnight), numbers of victims, affected locations; Details about international responses including that of UNICEF; reports on Nigerian domestic response; information on the terrorist group's motives (pressure Nigerian government, use girls in suicide attacks, forced conversion to Islam) and connections to ISIL and Al Qaeda; ", 'somewhat_valuable': 'Reports of girls fleeing prior to a planned attack, reports about girls that were kidnapped but whose fate is still unknown, reports about girls being freed after government negotiations with terrorist group ', 'not_that_valuable': 'Information about celebrations after girls are rescued and returned home; general background information on Boko Haram when referenced in connection with kidnappings in Africa; side references African school girl kidnappings in reports focused on other attacks', 'non_relevant': "Boko Haram attacks in Nigeria not related to girls' kidnappings, suicide attacks by terrorist groups in countries outside of Africa, girls education in Muslim countries, American students accused of a planned school attack in Florida"},
+                        report='On the night of 14–15 April 2014, 276 female students were kidnapped from the Government Secondary School in the town of Chibok in Borno State, Nigeria. Responsibility for the kidnappings was claimed by Boko Haram, an extremist, Islamic, terrorist organization based in northeastern Nigeria. 57 of the schoolgirls managed to escape over the next few months and some have described their capture in appearances at international human rights conferences. A child born to one of the girls and believed by medical personnel to be about 20 months old also was released, according to the Nigerian president\'s office.\n\nSince then hopes were raised on various occasions that the 219 remaining girls might be released. Newspaper reports suggested that Boko Haram was hoping to use the girls as a negotiating pawns in exchange for some of their commanders in jail.\n\nIn May 2016, one of the missing girls, Amina Ali, was found. She claimed that the remaining girls were still there, but that six had died. A further 21 girls were freed in October 2016, while another was rescued the next month. Another was found in January 2017. 82 more girls were freed in May 2017. One of the girls was rescued in January 2018.\n\n\nContents\n1\tBackground\n2\tKidnapping\n3\tAftermath\n3.1\tEvents in 2014\n3.2\tEvents in 2015\n3.3\tEvents in 2016\n3.4\tEvents in 2017\n3.5\tEvents in 2018\n4\tReaction\n5\tSocial media and celebrity involvement\n6\tInternational governmental response\n7\tSee also\n8\tReferences\n9\tExternal links\nBackground\nMain article: Boko Haram insurgency\nThe terrorist group Boko Haram wants to institute an Islamic caliphate in Nigeria and is in particular opposed to western-style modern education, which they say lures people away from following Islamic teaching as a way of life. By 2014, tens of thousands of people had been killed in attacks perpetrated by the group, and the Nigerian federal government declared a state of emergency in May 2013 in Borno, Yobe and Adamawa states in its fight against the insurgency. The resulting crackdown led to the capture or killing of hundreds of Boko Haram members, with the remainder retreating to mountainous areas from which they began increasingly to target civilians. However, the campaign failed to stabilise the country. A French military operation in Mali also pushed Boko Haram and AQIM terrorists into Nigeria.\n\n\nBorno State is in northeast Nigeria\nBoko Haram began to target schools in 2010, killing hundreds of students by 2014. A spokesperson for the group said such attacks would continue as long as the Nigerian government continued to interfere with traditional Islamic education. 10,000 children have been unable to attend school as a result of activities by Boko Haram. Boko Haram has also been known to kidnap girls, whom it believes should not be educated, and use them as cooks or sex slaves.\n\nBoko Haram\'s attacks intensified in 2014. In February, the group killed more than 100 Christian men in the villages of Doron Baga and Izghe. That same month, 59 boys were killed in the Federal Government College attack in northeastern Nigeria. In March, the group attacked the Giwa military barracks, freeing captured militants. The Chibok abduction occurred on the same day as a bombing attack in Abuja in which at least 88 people died. Boko Haram was blamed for nearly 4,000 deaths in 2014. Training received from al-Qaeda in the Islamic Maghreb and al Qaeda in the Arabian Peninsula has helped Boko Haram intensify its attacks.\n\nKidnapping\n\nDamage to the school\nOn the night of 14–15 April 2014, a group of militants attacked the Government Girls Secondary School in Chibok, Nigeria. They broke into the school, pretending to be guards. According to a diary written by two of the girls (Naomi Adamu and Sarah Samuel) the militants had intended to steal an "engine block" and were initially unsure what to do with the girls. They told the girls to get out and come with them. Some girls were loaded into trucks and the rest had to walk several miles until other trucks came to take them away possibly into the Konduga area of the Sambisa Forest where Boko Haram were known to have fortified camps. Houses in Chibok were also burned down in the incident. The school had been closed for four weeks prior to the attack due to the deteriorating security situation, but students from multiple schools had been called in to take final exams in physics.\n\nThere were 530 students from multiple villages registered for the Senior Secondary Certificate Examination, although it is unclear how many were in attendance at the time of the attack. The children were aged 16 to 18 and were in their final year of school. There was initial confusion over the number of girls kidnapped but on 21 April 2014, parents said 234 girls were missing. A number of the students escaped the kidnappers by jumping off the trucks. According to the police, approximately 276 children were taken in the attack, of whom 53 had escaped as of 2 May. Other reports said that 329 girls were kidnapped, 53 had escaped and 276 were still missing.\n\nAmnesty International said it believes the Nigerian military had four hours\' advance warning of the kidnapping, but failed to send reinforcements to protect the school. Nigeria\'s armed forces have confirmed that the Nigerian military had four-hour advance notice of the attack but said that their over-extended forces were unable to mobilize reinforcements.\n\nJonathan N.C. Hill of King\'s College London, has pointed out that Boko Haram kidnapped these girls after coming increasingly under the influence of al-Qaeda in the Islamic Maghreb, and asserts that the group\'s goal is to use girls and young women as sexual objects and as a means of intimidating the civilian population into non-resistance. Hill describes the attacks as similar to kidnapping of girls in Algeria in the 1990s and early 2000s.\n\nAftermath\nEvents in 2014\n\nCEE-HOPE Nigerian organized an event to commemorate one year anniversary of bring back our girls\nNon-Muslim students were forced to convert to Islam. The girls were forced into marriage with members of Boko Haram, with a reputed "bride price" of ₦2,000 each ($6/£4). Many of the students were taken to the neighbouring countries of Chad and Cameroon, with sightings reported of the students crossing borders with the militants, and sightings of the students by villagers living in the Sambisa Forest. The forest was considered a refuge for Boko Haram. Local residents were able to track the movements of the students with the help of contacts across north eastern Nigeria. A diary described how some girls escaped but were returned to Boko Haram by local villagers and whipped.\n\nThe Guardian reported that the British Royal Air Force conducted Operation Turus in response to the Chibok schoolgirls\' kidnapping by Boko Haram in Nigeria in April 2014. A source involved with the Operation told the Observer that “The girls were located in the first few weeks of the RAF mission,” and that “We [RAF] offered to rescue them, but the Nigerian government declined,” this was because it viewed the matter as a “national issue” to be resolved by Nigerian intelligence and security services.\n\nOn 2 May 2014, police said they were still unclear as to the exact number of students kidnapped. They asked parents to provide documents so an official count could be made, as school records had been damaged in the attack. On 4 May, the Nigerian President, Goodluck Jonathan, spoke publicly about the kidnapping for the first time, saying the government was doing everything it could to find the missing girls. At the same time, he blamed parents for not supplying enough information about their missing children to the police.\n\nOn 5 May 2014, a video in which Boko Haram leader Abubakar Shekau claimed responsibility for the kidnappings emerged. Shekau claimed that "Allah instructed me to sell them...I will carry out his instructions." and "Slavery is allowed in my religion, and I shall capture people and make them slaves." He said the girls should not have been in school and instead should have been married since girls as young as nine are suitable for marriage.\n\nFollowing the kidnapping incident, Boko Haram again abducted another eight girls, aged between 12–15, from northeastern Nigeria, a number later raised to eleven.\n\nChibok is primarily a Christian village and Shekau acknowledged that many of the girls seized were not Muslims: "The girls that have not accepted Islam, they are now gathered in numbers...and we treat them well the way the Prophet Muhammad treated the infidels he seized."\n\nOn 5 May 2014, at least 300 residents of the nearby town of Gamboru Ngala were killed in an attack by Boko Haram militants after Nigerian security forces had left the town to search for the kidnapped students. On 9 May, former Boko Haram negotiator, Shehu Sani, stated that the group wanted to swap the abducted girls for its jailed members. On 11 May, Kashim Shettima, Governor of Borno State in Nigeria, said that he had sighted the abducted girls and that the girls were not taken across the borders of Cameroon or Chad. On 12 May, Boko Haram released a video showing about 130 kidnapped girls, each clad in a hijab and a long Islamic chador, and demanded a prisoner exchange.\n\nA journalist-brokered deal to secure the release of the girls in exchange for 100 Boko Haram prisoners held in Nigerian jails was scrapped at a late stage on 24 May 2014 after President Goodluck Jonathan consulted with U.S., Israeli, French and British foreign ministers in Paris, where the consensus was that no deals should be struck with terrorists, and that a solution involving force was required.\n\nOn 26 May, the Nigerian Chief of Defence Staff announced that the Nigerian security forces had located the kidnapped girls, but ruled out a forceful rescue attempt for fears of collateral damage.\n\nOn 30 May, it was reported that a civilian militia in the Baale region of Northeastern Nigeria found two of the kidnapped girls raped, "half-dead," and tied to a tree. Villagers said the Boko Haram group had left the two girls, and killed four other disobedient girls and buried them. 223 were still missing.\n\nSir Andrew Pocock, British High Commissioner to Nigeria said that a couple of months after the kidnapping a group of up to 80 of the Chibok girls were seen by American \'eye in the sky\' technology but nothing was done. The girls, a camp and evidence of ground transport vehicles were spotted next to a local landmark called the \'Tree of Life\' in the Sambisa forest.\n\nOn 24 June, it was reported that 91 more women and children were abducted in other areas of Borno State. One source estimated in June that there could be as many as 600 girls held by Boko Haram in three camps outside Nigeria.\n\nOn 26 June, it was announced that Levick, a Washington, D.C. public relations firm, had received "a contract worth more than $1.2 million" from the government of Nigeria to work on "the international and local media narrative" surrounding the Chibok schoolgirl kidnapping.\n\nOn 1 July, a businessman suspected of carrying out the kidnappings of the school girls, as well as the bombing of a busy market in northeastern Nigeria, was arrested. Military sources said that he was also accused of helping the Islamist militant group kill the traditional leader Idrissa Timta, the Emir of Gwoza.\n\nOn 15 July, Zakaria Mohammed (\'the Butcher\'), a high-ranking member of Boko Haram, was arrested at Darazo-Basrika Road while fleeing from the counter insurgency operations going on around the Balmo Forest.\n\nOn 12 October 2014, it was reported that four girls from the original kidnapped group had escaped and walked three weeks to freedom in Nigeria. They said they had been held in a camp in Cameroon and raped every day.\n\nEvents in 2015\nStephen Davis, a former Anglican clergyman, contacted three Boko Haram commanders who said they might be prepared to release Chibok schoolgirls and went to Nigeria in April 2015. He was given proof of life (a video of them being raped) and was told 18 were seriously ill, some with HIV. Davis got initial agreement that Boko Haram would release these ill girls. However, after three attempts the deal fell through when another group abducted the girls believing they could make money out of them and Davis left Nigeria. Davis commented that it was not difficult to locate the five or six main Boko Haram camps. He could find them on Google Earth.\n\nIn May 2015, it was reported that the Nigerian military had reclaimed most of the areas previously controlled by Boko Haram in Nigeria including many of the camps in the Sambisa forest where it was suspected the Chibok girls had been kept. Although many women had been freed, none of the Chibok girls had been found. It was reported that some of the girls had been sold into slavery for N2,000 (about $10) each, others had been forcibly married to Boko Haram fighters and they may have been killed. Kashim Shettima, the Borno state governor said he suspected the Chibok girls were being kept in underground bunkers.\n\nEvents in 2016\nIn January 2016 the Nigerian military were reported to have freed 1,000 women held captive by Boko Haram but none of them were Chibok girls.\n\nIn April 2016 Boko Haram released a video showing 15 girls who appeared to be some of the kidnapped Chibok girls. The video was apparently taken in December 2015 and the girls seemed to be well fed and not distressed.\n\nOn 17 May 2016, Amina Ali Nkeki, one of the girls was found along with her baby and Mohammad Hayyatu, a suspected Boko Haram militant who claimed to be her husband, by the vigilante Civilian Joint Task Force group in the Sambisa Forest. All three were suffering from severe malnutrition. She was then taken to house of the group\'s leader Aboku Gaji who recognised her. The group then reunited the girl with her parents. She met Nigerian President Muhammadu Buhari on 19 May. Government officials announced the same day that the Nigerian army and vigilante groups had killed 35 Boko Haram militants, freed 97 women and children and claimed one of the women was a Chibok schoolgirl. However, there were doubts that this girl, Serah Luka, was really one of the kidnapped Chibok schoolgirls. On 21 May 2016, Amir Muhammad Abdullahi, who claimed to be the Boko Haram second in command and speaker for several senior militants, offered to surrender so long as they would not be harmed and in return they would release hostages including the Chibok girls. However he said of Chibok girls; "...frankly, just about a third of them remain, as the rest have been martyred”.\n\nIn August 2016 Boko Haram released a video of what appeared to be about 50 Chibok girls, some of them holding babies, with an armed masked spokesman who demanded the release of jailed fighters in exchange for the girls\' freedom, The masked gunman said some of the Chibok girls had been killed by Nigerian air strikes and 40 had been married. The film was apparently released on the orders of Abubakar Shekau, the leader of one of the factions of Boko Haram.\n\nIn October 2016, 21 of the Chibok schoolgirls had been freed by Boko Haram after negotiations between the group and the Nigerian government brokered by International Committee of the Red Cross and the Swiss government. On 16 October, President Buhari\'s spokesperson stated that the ISIL-allied faction of Boko Haram was willing to negotiate the release of 83 more of the girls. According to him, the splinter group had stated that the rest of the girls were under the control of Shekau-led faction. 2 days later, Pogu Bitrus, the chairman of the Chibok Development Association, claimed that more than 100 of the missing girls apparently did not want to return home because they had either been brainwashed or were fearful of the stigma they will receive.\n\nAnother girl named Maryam Ali Maiyanga was found and rescued by the Nigerian Army on 5 November along with a baby by the Nigerian Army. The spokesman for the Army, Sani Usman, said that they discovered her in Pulka of Borno state while screening escapees from Boko Haram\'s Sambisa forest base. She was confirmed to be one of the kidnapped girls by Bring Back Our Girls.\n\nEvents in 2017\n\nU.S. President Donald J. Trump, Ivanka Trump, and Chibok schoolgirls Joy Bishara and Lydia Pogu at the White House on June 27, 2017.\nOne of the kidnapped girls, Rakiya Abubakar, was reported on January 5, 2017 to have been found by the Nigerian Army along with a 6-month-old baby while they were interrogating suspects detained in army raids on the Sambisa forest. Her identity was later confirmed by Bring Back Our Girls group.\n\nOn May 6, 82 of the schoolgirls were released following successful negotiations between the Nigerian government involving the exchange of five Boko Haram leaders. The negotiations were carried by Mustapha Zanna, barrister and owner of an orphanage in Maiduguri. The deal also involved the intervention of the Swiss government and the Red Cross. 3 million Euros (about 3.7 million US$) were paid as ransom money in two duffel bags for the total of 103 girls released in October 2016 and May 2017. A Nigerian government spokesman stated that though originally 83 girls were to be released in May 2017, one of them chose to stay with her husband instead of being freed.\n\nEvents in 2018\nThe Nigerian military stated on 4 January 2018 that it had rescued Salomi Pogu, one of the kidnapped girls. Col. Onyema Nwachukwu stated that she was rescued near Pulka village in Borno. Her name was in the list of the kidnapped Chibok schoolgirls. She was found in the company of another young woman and her child. In February 2018 most of the released girls were studying at the American University of Nigeria not far from the original scene of the kidnapping at Chibok. It was estimated that 13 girls were presumed dead and 112 were still missing.', report_url='https://en.wikipedia.org/w/index.php?title=Chibok_schoolgirls_kidnapping&diff=825075173&oldid=823837916', report_date='2018-02-11', translation_lang='fa')
+        })
+
+    def test_hc4_ru_query(self):
+        self._test_queries('hc4/ru/train', count=7, items={
+            0: HC4Query(query_id='1005', title='political implications of the Kemerovo fire', description='What are the political implications of the public reaction to the Kemerovo fire?', ht_title='', ht_description='', mt_title='политические последствия Кемеровского пожара', mt_description='Каковы политические последствия реакции общественности на кемеровский пожар?', narrative_by_relevance={'very_valuable': '', 'somewhat_valuable': '', 'not_that_valuable': '', 'non_relevant': ''}, report='Politicians across the globe sent their condolences. 28 March 2018 was declared the national day of mourning in Russia.\n\nA rally was held in Kemerovo demanding the resignation of Kemerovo Mayor Ilya Seredyuk and the longtime regional governor, Aman Tuleyev. ', report_url='https://en.wikipedia.org/w/index.php?title=2018_Kemerovo_fire&oldid=833479706#Aftermath', report_date='2018-03-31', translation_lang='ru')
+        })
+        self._test_queries('hc4/ru/dev', count=4, items={
+            0: HC4Query(query_id='3', title='British royal news impacts', description='What political and economic impacts does news about the British royal family have domestically and abroad?', ht_title='Влияние британских королевских новостей', ht_description='Какое политическое и экономическое влияние новости о британской королевской семье имеют внутри страны и за рубежом?', mt_title='Британские королевские новости влияют', mt_description='Какие политические и экономические последствия имеют новости о британской королевской семье как внутри страны, так и за рубежом?', narrative_by_relevance={'very_valuable': 'Information regarding economic and political impacts of the British royal family in the UK and worldwide', 'somewhat_valuable': 'Information related to the British royal family and their interactions with politics or the economy', 'not_that_valuable': 'information about the royal family', 'non_relevant': 'information not about the royals'}, report="Announcement of engagement\nPrincess Eugenie of York in 2013\n\nPrincess Eugenie of York is the second daughter of Prince Andrew, Duke of York and Sarah, Duchess of York. Jack Brooksbank is a British nightclub manager, a distant relation of the Brooksbank baronets,[notes 1] and a third cousin twice removed of Princess Eugenie through Thomas Coke, 2nd Earl of Leicester.[notes 2] The couple have been dating for seven years; they were first introduced by friends in a ski break in Verbier, Switzerland, where Brooksbank was working at the time.\n\nOn 22 January 2018, Buckingham Palace announced that Princess Eugenie of York would marry Jack Brooksbank in the autumn. They were engaged earlier the same month while in Nicaragua with Brooksbank giving the Princess an oval-cut Padparadscha sapphire surrounded by a halo of diamonds set on a gold band with two further diamonds on the shoulders. The ring bears a striking similarity to the engagement ring of Princess Eugenie's mother.\n\nEugenie was eighth in the line of succession to the British throne at the time the engagement was announced. As of May 2018, she is ninth in the line of succession. Although Eugenie is a member of the British royal family, she does not require the Queen's permission to marry.[notes 3] The Duke and Duchess of York expressed their delight at the news and the British Prime Minister Theresa May congratulated the couple on her Twitter account. After the announcement the couple gave an interview to Matt Baker of BBC One.\n\nThe official engagement photographs were taken in the Picture gallery at Buckingham Palace. ", report_url='https://en.wikipedia.org/w/index.php?title=Wedding_of_Princess_Eugenie_and_Jack_Brooksbank&oldid=841040200', report_date='2018-05-13', translation_lang='ru')
+        })
+        self._test_queries('hc4/ru/test', count=50, items={
+            10: HC4Query(query_id='127', title='Brumadinho dam collapse cause', description='What caused the collapse of the dam at the Córrego do Feijão iron ore mine owned by Vale in Brumadinho, Brazil?', ht_title='Причина обрушения плотины Брумадиньо', ht_description='Что стало причиной обрушения плотины на железорудном руднике Коррего-ду-Фейхао, принадлежащем Vale в Брумадиньо, Бразилия?', mt_title='Причина обрушения плотины Брумадиньо', mt_description='Что вызвало обрушение плотины на руднике железной руды Коррего-ду-Фейяу, принадлежащей Вале в Брумадинью, Бразилия?', narrative_by_relevance={'very_valuable': 'states reason and provides responsible party', 'somewhat_valuable': 'company may have known about failures', 'not_that_valuable': 'people were arrested', 'non_relevant': 'mentions victims or unrelated topics (gas trade, cars)'}, report="Background\nBento Rodrigues Village right after the similar Mariana dam disaster of 2015\n\nThe Brumadinho dam failure happened three years and two months after the Mariana dam disaster, which killed 19 people and destroyed the village of Bento Rodrigues. The Mariana disaster is considered the worst environmental disaster in Brazil's history and is still under investigation.\n\nExperts say that Brazil's weak regulatory structures and regulatory gaps allowed the dam's failure. Three years after the Mariana dam collapse, the companies involved in that environmental disaster have paid only 3.4% of R$785 million in fines.\n\nAt the time of the Mariana dam disaster in November 2015, the department in charge of inspecting mining operations in the state of Minas Gerais, the National Department of Mineral Production (DNPM), was worried about the retirement of another 40% of public employees over the course of the next two years.\n\nAccording to the national registry of the National Mining Agency, the Córrego do Feijão dam, built in 1976 by the Ferteco Mineração (acquired by Vale in 2001), was classified as a small structure with low risk of high potential damage. In a statement, the State Department of Environment and Sustainable Development reported that the venture was duly licensed. In December 2018, Vale obtained a license to reuse waste from the dam (about 11.7 million cubic meters) and to close down activities. The dam had not received tailings since 2014 and, according to the company, underwent bi-weekly field inspections. ", report_url='https://en.wikipedia.org/w/index.php?title=Brumadinho_dam_disaster&oldid=tml', report_date='2019-02-10', translation_lang='ru')
+        })
+
+    def test_hc4_zh_qrels(self):
+        self._test_qrels('hc4/zh/train', count=341, items={
+            0: TrecQrel(query_id='1001', doc_id='062bf68a-5971-40f9-9cb1-6d7bf55592e2', relevance=0, iteration='0'),
+            123: TrecQrel(query_id='1009', doc_id='4393e311-34f8-4b4a-810b-97ab75e45916', relevance=0, iteration='0')
+        })
+        self._test_qrels('hc4/zh/dev', count=466, items={
+            0: TrecQrel(query_id='1', doc_id='07fdce5f-e00b-4a40-b9eb-f0f78cfaefc4', relevance=0, iteration='0'),
+            123: TrecQrel(query_id='3', doc_id='ed7d7890-fffb-4620-a952-f5cc20a2368f', relevance=0, iteration='0')
+        })
+        self._test_qrels('hc4/zh/test', count=2751, items={
+            0: TrecQrel(query_id='102', doc_id='1b7ba71e-3294-4817-be08-300e22ead35c', relevance=0, iteration='0'),
+            123: TrecQrel(query_id='104', doc_id='f98d9c3c-089c-4124-946e-afbdc53e4fc9', relevance=0, iteration='0')
+        })
+    
+    def test_hc4_fa_qrels(self):
+        self._test_qrels('hc4/fa/train', count=112, items={
+            0: TrecQrel(query_id='1008', doc_id='0223ebb7-4d7e-47bc-a282-d8eeb8c9c49e', relevance=1, iteration='0'),
+            67: TrecQrel(query_id='1022', doc_id='a8e5a616-2a0d-43f2-af2b-1b8ec622ce6f', relevance=0, iteration='0')
+        })
+        self._test_qrels('hc4/fa/dev', count=565, items={
+            0: TrecQrel(query_id='1', doc_id='079eba9b-fda2-4c26-ac18-2c35a7a08e88', relevance=0, iteration='0'),
+            67: TrecQrel(query_id='4', doc_id='b9703bc0-af7e-4332-973a-9617dbd69b60', relevance=3, iteration='0')
+        })
+        self._test_qrels('hc4/fa/test', count=2522, items={
+            0: TrecQrel(query_id='103', doc_id='0d4e7624-69fb-4d55-9103-5ab04abb28c3', relevance=0, iteration='0'),
+            67: TrecQrel(query_id='107', doc_id='25872c00-4ca5-4584-b9c4-cb88b3f47e2c', relevance=0, iteration='0')
+        })
+
+    def test_hc4_ru_qrels(self):
+        self._test_qrels('hc4/ru/train', count=92, items={
+            0: TrecQrel(query_id='1005', doc_id='047cae7c-c674-41ea-a4c7-1d0d68c73b47', relevance=1, iteration='0'),
+            77: TrecQrel(query_id='1026', doc_id='1f22c2cf-303d-4d09-9d54-26cd6a53c9d3', relevance=0, iteration='0')
+        })
+        self._test_qrels('hc4/ru/dev', count=265, items={
+            0: TrecQrel(query_id='3', doc_id='07d5796e-e55f-469f-b158-88a8aa53131d', relevance=0, iteration='0'),
+            77: TrecQrel(query_id='6', doc_id='62d31f31-1efa-4da2-8944-ac35a3cd77fa', relevance=1, iteration='0')
+        })
+        self._test_qrels('hc4/ru/test', count=2970, items={
+            0: TrecQrel(query_id='101', doc_id='00d525d6-ae24-43c0-b5f5-125d796552d1', relevance=0, iteration='0'),
+            77: TrecQrel(query_id='103', doc_id='3bd61245-80a3-4125-a095-f1314b1509f4', relevance=3, iteration='0')
+        })
+
+
+if __name__ == '__main__':
+    unittest.main()