Repository: castorini/pyserini

Issue:
Issues with latest MIRACL 2CR
On my iMac Pro (Intel), I'm getting the following failures:

```
condition bm25-mdpr-tied-pft-msmarco-hybrid.bn:
  - split: dev
      nDCG@10: 0.6540 [OK]
      R@100  : 0.9321 [FAIL] expected 0.9100

condition bm25-mdpr-tied-pft-msmarco-hybrid.zh:
  - split: dev
      nDCG@10: 0.5255 [FAIL] expected 0.5254
      R@100  : 0.9587 [OK]
```



Discussion:
On `orca`, I'm getting the following failures:

```
condition mdpr-tied-pft-msmarco-ft-all.ru:
  - split: train
      nDCG@10: 0.6011 [OK]
      R@100  : 0.8188 [OK]
  - split: dev
      nDCG@10: 0.3932 [FAIL] expected 0.3933
      R@100  : 0.6707 [OK]

condition bm25-mdpr-tied-pft-msmarco-hybrid.bn:
  - split: dev
      nDCG@10: 0.6540 [OK]
      R@100  : 0.9321 [FAIL] expected 0.9100

condition bm25-mdpr-tied-pft-msmarco-hybrid.zh:
  - split: dev
      nDCG@10: 0.5255 [FAIL] expected 0.5254
      R@100  : 0.9587 [OK]
```
After #1431 - On my iMac Pro (Intel), I'm getting the following (small) differences:

```
condition bm25-mdpr-tied-pft-msmarco-hybrid.te:
  - split: train
      nDCG@10: 0.5999 [FAIL] expected 0.6000
      R@100  : 0.8717 [OK]
  - split: dev
      nDCG@10: 0.6021 [OK]
      R@100  : 0.8569 [OK]

condition bm25-mdpr-tied-pft-msmarco-hybrid.zh:
  - split: train
      nDCG@10: 0.5209 [OK]
      R@100  : 0.9576 [OK]
  - split: dev
      nDCG@10: 0.5255 [FAIL] expected 0.5254
      R@100  : 0.9587 [OK]
```

On `orca`:

```
condition mdpr-tied-pft-msmarco-ft-all.ru:
  - split: train
      nDCG@10: 0.6011 [OK]
      R@100  : 0.8188 [OK]
  - split: dev
      nDCG@10: 0.3932 [FAIL] expected 0.3933
      R@100  : 0.6707 [OK]

condition bm25-mdpr-tied-pft-msmarco-hybrid.zh:
  - split: train
      nDCG@10: 0.5209 [OK]
      R@100  : 0.9576 [OK]
  - split: dev
      nDCG@10: 0.5255 [FAIL] expected 0.5254
      R@100  : 0.9587 [OK]
```

Resolving patch:
diff --git a/scripts/repro_matrix/run_all_miracl.py b/scripts/repro_matrix/run_all_miracl.py
--- a/scripts/repro_matrix/run_all_miracl.py
+++ b/scripts/repro_matrix/run_all_miracl.py
@@ -30,12 +30,12 @@
 
 def print_results(metric, split):
     print(f'Metric = {metric}, Split = {split}')
-    print(' ' * 32, end='')
+    print(' ' * 35, end='')
     for lang in languages:
         print(f'{lang[0]:3}    ', end='')
     print('')
     for model in models:
-        print(f'{model:30}', end='')
+        print(f'{model:33}', end='')
         for lang in languages:
             key = f'{model}.{lang[0]}'
             print(f'{table[key][split][metric]:7.3f}', end='')
@@ -125,9 +125,18 @@ def extract_topic_fn_from_cmd(cmd):
                                                                      trec_eval_metric_definitions[metric], runfile))
                             if math.isclose(score, float(expected[metric])):
                                 result_str = ok_str
-                            # Flaky test: small difference on Mac Studio (M1 chip)
-                            elif name == 'mdpr-tied-pft-msmarco.hi' and split == 'train' \
-                                    and math.isclose(score, float(expected[metric]), abs_tol=2e-4):
+                            # Flaky tests
+                            elif (name == 'mdpr-tied-pft-msmarco.hi' and split == 'train'
+                                  and math.isclose(score, float(expected[metric]), abs_tol=2e-4)) or \
+                                 (name == 'mdpr-tied-pft-msmarco-ft-all.ru'
+                                  and split == 'dev' and metric == 'nDCG@10'
+                                  and math.isclose(score, float(expected[metric]), abs_tol=2e-4)) or \
+                                 (name == 'bm25-mdpr-tied-pft-msmarco-hybrid.te'
+                                  and split == 'train' and metric == 'nDCG@10'
+                                  and math.isclose(score, float(expected[metric]), abs_tol=2e-4)) or \
+                                 (name == 'bm25-mdpr-tied-pft-msmarco-hybrid.zh'
+                                  and split == 'dev' and metric == 'nDCG@10'
+                                  and math.isclose(score, float(expected[metric]), abs_tol=2e-4)):
                                 result_str = okish_str
                             else:
                                 result_str = fail_str + f' expected {expected[metric]:.4f}'


Test patch: