Repository: ScandEval/ScandEval (language: python)

Issue:
[DATASET REQUEST] Icelandic Winogrande
**Describe the dataset**
Icelandic Winogrande is hosted [at this Hugging Face repo](https://huggingface.co/datasets/mideind/icelandic-winogrande) and consists of ~1.09k samples. It is a multiple choice coreference resolution dataset with two options, and has been recommended as challenging by native speakers.

**Rationale**
This is a high-quality dataset for a low-resource language. It would be put in the reasoning category, probably replacing HellaSwag-is, since the latter is machine translated. Since there are only 1k samples we can only use it for few-shot evaluation, however.



Discussion / hints:
Is this something that would make sense for us at Miðeind to implement or will you who are more familiar with the codebase add it?
> Is this something that would make sense for us at Miðeind to implement or will you who are more familiar with the codebase add it?

I think adding it in is very quick, but the problem is really its size. Of course it would be fine as a test-only split, but we're currently aiming to also enable finetuning evaluation on generative tasks (for instance, QLoRA finetuning of decoder models or T5-finetuning), and it's unclear to me what we would do with test-only datasets in that case.

I mean, one tentative fix one could do is to keep this as a test split, and generate a synthetic training and validation split for the dataset, but that would probably be of a much lower standard, so not quite sure how to deal with it.

@gardarjuto I'm not sure exactly how much work it is for you guys, but do you think it would be possible to effectively double the dataset, to have 1024+256+1024 for train/val/test? Alternatively, we would have to split the dataset into very small splits, like 256+256+512, which would result in a large variance in test scores.
Thanks for the answer @saattrupdan! 
Currently, unfortunately I would say that we don't have the resources to double the dataset size.

WinoGrande-style examples take a surprising amount of effort to construct since you both have to design a WinoGrad ambiguous sentence as well as carefully ensuring that it doesn't include biased language which would hint at the answer. For Icelandic, this is a painstakingly difficult process since gendered nouns, inflections and verb forms can very easily give away the answer for free if you're not careful. I'd estimate not less than 5 minutes per example, making for over 14 full 7-hour work days (as a lower bound) which is just not feasible for us at this time.

This difficulty of crafting Icelandic WinoGrande examples also means that any synthetic or machine translated data will almost invariably be so unusable that training on them is likely counterproductive, worse than doing nothing at all. I therefore suggest that we use this only for few-shot evaluation.

Fix patch:
diff --git a/.gitignore b/.gitignore
index ceb6df14b..72164bf80 100644
--- a/.gitignore
+++ b/.gitignore
@@ -101,3 +101,6 @@ scandeval_benchmark_results.jsonl
 # Experiments
 gai-experiments.xlsx
 ~$gai-experiments.xlsx
+
+# Upload of leaderboard
+src/upload/
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7d29d4f42..1b555a7b4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,24 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 
+## [Unreleased]
+### Added
+- Added the Icelandic common sense reasoning dataset Winogrande-is, being a manually
+  translated version of the English Winogrande dataset. This also means that the
+  HellaSwag-is dataset has been marked as unofficial, and will thus not automatically
+  be included when benchmarking models on the Icelandic common sense reasoning task.
+
+### Changed
+- Updated `vllm` dependency to `>=0.3.3,<0.4.0`, which allows the benchmarking of the
+  new Gemma and OLMO models, without the bug from vLLM v0.3.2.
+
+### Fixed
+- Do not show message regarding missing flash attention if CUDA is not available.
+- Only use bfloat16 as quantisation compute type if it is available and that
+  `torch_dtype` is set to "bfloat16" in the Hugging Face configuration - otherwise we
+  use float16.
+
+
 ## [v12.1.0] - 2024-02-29
 ### Changed
 - Flash attention will now default to being used if `flash_attn` has been installed. If
diff --git a/README.md b/README.md
index e60c7ec69..379702563 100644
--- a/README.md
+++ b/README.md
@@ -9,7 +9,7 @@ ______________________________________________________________________
 [![Paper](https://img.shields.io/badge/arXiv-2304.00906-b31b1b.svg)](https://arxiv.org/abs/2304.00906)
 [![License](https://img.shields.io/github/license/ScandEval/ScandEval)](https://github.com/ScandEval/ScandEval/blob/main/LICENSE)
 [![LastCommit](https://img.shields.io/github/last-commit/ScandEval/ScandEval)](https://github.com/ScandEval/ScandEval/commits/main)
-[![Code Coverage](https://img.shields.io/badge/Coverage-76%25-yellowgreen.svg)](https://github.com/ScandEval/ScandEval/tree/main/tests)
+[![Code Coverage](https://img.shields.io/badge/Coverage-73%25-yellow.svg)](https://github.com/ScandEval/ScandEval/tree/main/tests)
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](https://github.com/ScandEval/ScandEval/blob/main/CODE_OF_CONDUCT.md)
 
 
diff --git a/makefile b/makefile
index 908111d77..16787db4f 100644
--- a/makefile
+++ b/makefile
@@ -1,6 +1,6 @@
 # This ensures that we can call `make <target>` even if `<target>` exists as a file or
 # directory.
-.PHONY: docs help
+.PHONY: help
 
 # Exports all variables defined in the makefile available to scripts
 .EXPORT_ALL_VARIABLES:
@@ -86,21 +86,6 @@ setup-git:
 	@git config --local user.email ${GIT_EMAIL}
 	@poetry run pre-commit install
 
-docs:  ## Generate documentation
-	@poetry run pdoc --docformat google src/scandeval -o docs
-	@echo "Saved documentation."
-
-view-docs:  ## View documentation
-	@echo "Viewing API documentation..."
-	@uname=$$(uname); \
-		case $${uname} in \
-			(*Linux*) openCmd='xdg-open'; ;; \
-			(*Darwin*) openCmd='open'; ;; \
-			(*CYGWIN*) openCmd='cygstart'; ;; \
-			(*) echo 'Error: Unsupported platform: $${uname}'; exit 2; ;; \
-		esac; \
-		"$${openCmd}" docs/{{ cookiecutter.project_name }}.html
-
 test:  ## Run tests
 	@$(MAKE) --quiet test-cuda-vllm
 	@$(MAKE) --quiet test-cuda-no-vllm
diff --git a/poetry.lock b/poetry.lock
index 372c711e9..40caf6b5f 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -165,26 +165,6 @@ yarl = ">=1.0,<2.0"
 [package.extras]
 speedups = ["Brotli", "aiodns", "brotlicffi"]
 
-[[package]]
-name = "aioprometheus"
-version = "23.12.0"
-description = "A Prometheus Python client library for asyncio-based applications"
-optional = true
-python-versions = ">=3.8.0"
-files = [
-    {file = "aioprometheus-23.12.0-py3-none-any.whl", hash = "sha256:b1a77259131153ef820b494e76439b278434eaf2a5e25dc0b8bf3d835f455960"},
-]
-
-[package.dependencies]
-orjson = "*"
-quantile-python = ">=1.1"
-starlette = {version = ">=0.14.2", optional = true, markers = "extra == \"starlette\""}
-
-[package.extras]
-aiohttp = ["aiohttp (>=3.3.2)"]
-quart = ["quart (>=0.15.1)"]
-starlette = ["starlette (>=0.14.2)"]
-
 [[package]]
 name = "aiosignal"
 version = "1.3.1"
@@ -3025,75 +3005,14 @@ typing_extensions = "*"
 [package.extras]
 testing = ["flax", "pytest", "pytest-xdist"]
 
-[[package]]
-name = "orjson"
-version = "3.9.14"
-description = "Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy"
-optional = true
-python-versions = ">=3.8"
-files = [
-    {file = "orjson-3.9.14-cp310-cp310-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:793f6c9448ab6eb7d4974b4dde3f230345c08ca6c7995330fbceeb43a5c8aa5e"},
-    {file = "orjson-3.9.14-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a6bc7928d161840096adc956703494b5c0193ede887346f028216cac0af87500"},
-    {file = "orjson-3.9.14-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:58b36f54da759602d8e2f7dad958752d453dfe2c7122767bc7f765e17dc59959"},
-    {file = "orjson-3.9.14-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:abcda41ecdc950399c05eff761c3de91485d9a70d8227cb599ad3a66afe93bcc"},
-    {file = "orjson-3.9.14-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df76ecd17b1b3627bddfd689faaf206380a1a38cc9f6c4075bd884eaedcf46c2"},
-    {file = "orjson-3.9.14-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d450a8e0656efb5d0fcb062157b918ab02dcca73278975b4ee9ea49e2fcf5bd5"},
-    {file = "orjson-3.9.14-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:95c03137b0cf66517c8baa65770507a756d3a89489d8ecf864ea92348e1beabe"},
-    {file = "orjson-3.9.14-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:20837e10835c98973673406d6798e10f821e7744520633811a5a3d809762d8cc"},
-    {file = "orjson-3.9.14-cp310-none-win32.whl", hash = "sha256:1f7b6f3ef10ae8e3558abb729873d033dbb5843507c66b1c0767e32502ba96bb"},
-    {file = "orjson-3.9.14-cp310-none-win_amd64.whl", hash = "sha256:ea890e6dc1711aeec0a33b8520e395c2f3d59ead5b4351a788e06bf95fc7ba81"},
-    {file = "orjson-3.9.14-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:c19009ff37f033c70acd04b636380379499dac2cba27ae7dfc24f304deabbc81"},
-    {file = "orjson-3.9.14-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:19cdea0664aec0b7f385be84986d4defd3334e9c3c799407686ee1c26f7b8251"},
-    {file = "orjson-3.9.14-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:135d518f73787ce323b1a5e21fb854fe22258d7a8ae562b81a49d6c7f826f2a3"},
-    {file = "orjson-3.9.14-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d2cf1d0557c61c75e18cf7d69fb689b77896e95553e212c0cc64cf2087944b84"},
-    {file = "orjson-3.9.14-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b7c11667421df2d8b18b021223505dcc3ee51be518d54e4dc49161ac88ac2b87"},
-    {file = "orjson-3.9.14-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2eefc41ba42e75ed88bc396d8fe997beb20477f3e7efa000cd7a47eda452fbb2"},
-    {file = "orjson-3.9.14-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:917311d6a64d1c327c0dfda1e41f3966a7fb72b11ca7aa2e7a68fcccc7db35d9"},
-    {file = "orjson-3.9.14-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4dc1c132259b38d12c6587d190cd09cd76e3b5273ce71fe1372437b4cbc65f6f"},
-    {file = "orjson-3.9.14-cp311-none-win32.whl", hash = "sha256:6f39a10408478f4c05736a74da63727a1ae0e83e3533d07b19443400fe8591ca"},
-    {file = "orjson-3.9.14-cp311-none-win_amd64.whl", hash = "sha256:26280a7fcb62d8257f634c16acebc3bec626454f9ab13558bbf7883b9140760e"},
-    {file = "orjson-3.9.14-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:08e722a8d06b13b67a51f247a24938d1a94b4b3862e40e0eef3b2e98c99cd04c"},
-    {file = "orjson-3.9.14-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a2591faa0c031cf3f57e5bce1461cfbd6160f3f66b5a72609a130924917cb07d"},
-    {file = "orjson-3.9.14-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e2450d87dd7b4f277f4c5598faa8b49a0c197b91186c47a2c0b88e15531e4e3e"},
-    {file = "orjson-3.9.14-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:90903d2908158a2c9077a06f11e27545de610af690fb178fd3ba6b32492d4d1c"},
-    {file = "orjson-3.9.14-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ce6f095eef0026eae76fc212f20f786011ecf482fc7df2f4c272a8ae6dd7b1ef"},
-    {file = "orjson-3.9.14-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:751250a31fef2bac05a2da2449aae7142075ea26139271f169af60456d8ad27a"},
-    {file = "orjson-3.9.14-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9a1af21160a38ee8be3f4fcf24ee4b99e6184cadc7f915d599f073f478a94d2c"},
-    {file = "orjson-3.9.14-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:449bf090b2aa4e019371d7511a6ea8a5a248139205c27d1834bb4b1e3c44d936"},
-    {file = "orjson-3.9.14-cp312-none-win_amd64.whl", hash = "sha256:a603161318ff699784943e71f53899983b7dee571b4dd07c336437c9c5a272b0"},
-    {file = "orjson-3.9.14-cp38-cp38-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:814f288c011efdf8f115c5ebcc1ab94b11da64b207722917e0ceb42f52ef30a3"},
-    {file = "orjson-3.9.14-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a88cafb100af68af3b9b29b5ccd09fdf7a48c63327916c8c923a94c336d38dd3"},
-    {file = "orjson-3.9.14-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ba3518b999f88882ade6686f1b71e207b52e23546e180499be5bbb63a2f9c6e6"},
-    {file = "orjson-3.9.14-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:978f416bbff9da8d2091e3cf011c92da68b13f2c453dcc2e8109099b2a19d234"},
-    {file = "orjson-3.9.14-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:75fc593cf836f631153d0e21beaeb8d26e144445c73645889335c2247fcd71a0"},
-    {file = "orjson-3.9.14-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:23d1528db3c7554f9d6eeb09df23cb80dd5177ec56eeb55cc5318826928de506"},
-    {file = "orjson-3.9.14-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:7183cc68ee2113b19b0b8714221e5e3b07b3ba10ca2bb108d78fd49cefaae101"},
-    {file = "orjson-3.9.14-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:df3266d54246cb56b8bb17fa908660d2a0f2e3f63fbc32451ffc1b1505051d07"},
-    {file = "orjson-3.9.14-cp38-none-win32.whl", hash = "sha256:7913079b029e1b3501854c9a78ad938ed40d61fe09bebab3c93e60ff1301b189"},
-    {file = "orjson-3.9.14-cp38-none-win_amd64.whl", hash = "sha256:29512eb925b620e5da2fd7585814485c67cc6ba4fe739a0a700c50467a8a8065"},
-    {file = "orjson-3.9.14-cp39-cp39-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:5bf597530544db27a8d76aced49cfc817ee9503e0a4ebf0109cd70331e7bbe0c"},
-    {file = "orjson-3.9.14-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ac650d49366fa41fe702e054cb560171a8634e2865537e91f09a8d05ea5b1d37"},
-    {file = "orjson-3.9.14-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:236230433a9a4968ab895140514c308fdf9f607cb8bee178a04372b771123860"},
-    {file = "orjson-3.9.14-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3014ccbda9be0b1b5f8ea895121df7e6524496b3908f4397ff02e923bcd8f6dd"},
-    {file = "orjson-3.9.14-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ac0c7eae7ad3a223bde690565442f8a3d620056bd01196f191af8be58a5248e1"},
-    {file = "orjson-3.9.14-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fca33fdd0b38839b01912c57546d4f412ba7bfa0faf9bf7453432219aec2df07"},
-    {file = "orjson-3.9.14-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:f75823cc1674a840a151e999a7dfa0d86c911150dd6f951d0736ee9d383bf415"},
-    {file = "orjson-3.9.14-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:6f52ac2eb49e99e7373f62e2a68428c6946cda52ce89aa8fe9f890c7278e2d3a"},
-    {file = "orjson-3.9.14-cp39-none-win32.whl", hash = "sha256:0572f174f50b673b7df78680fb52cd0087a8585a6d06d295a5f790568e1064c6"},
-    {file = "orjson-3.9.14-cp39-none-win_amd64.whl", hash = "sha256:ab90c02cb264250b8a58cedcc72ed78a4a257d956c8d3c8bebe9751b818dfad8"},
-    {file = "orjson-3.9.14.tar.gz", hash = "sha256:06fb40f8e49088ecaa02f1162581d39e2cf3fd9dbbfe411eb2284147c99bad79"},
-]
-
 [[package]]
 name = "outlines"
-version = "0.0.34"
+version = "0.1.dev536+g88af3e6"
 description = "Probabilistic Generative Model Programming"
 optional = true
 python-versions = ">=3.8"
-files = [
-    {file = "outlines-0.0.34-py3-none-any.whl", hash = "sha256:911588a7e64a4f193b97fb4c501d98ccfd4e95a98f6a3ada67a280bf0c373c50"},
-    {file = "outlines-0.0.34.tar.gz", hash = "sha256:594e7204c770b47a62eb5c2ba7d25ea0ab2e16882b5f04556712a0228d3d3309"},
-]
+files = []
+develop = false
 
 [package.dependencies]
 cloudpickle = "*"
@@ -3117,6 +3036,12 @@ transformers = "*"
 serve = ["fastapi", "pydantic (>=2.0)", "ray (==2.9.0)", "uvicorn", "vllm (>=0.3.0)"]
 test = ["accelerate", "beartype (<0.16.0)", "coverage[toml] (>=5.1)", "datasets", "diff-cover", "huggingface-hub", "llama-cpp-python (>=0.2.42)", "pre-commit", "pytest", "pytest-benchmark", "pytest-cov", "pytest-mock", "responses", "transformers"]
 
+[package.source]
+type = "git"
+url = "https://github.com/saattrupdan/outlines"
+reference = "feat/add-transformers-integration"
+resolved_reference = "88af3e6aca7dc790f121035eb8d986733dd1121a"
+
 [[package]]
 name = "packaging"
 version = "23.2"
@@ -3332,6 +3257,20 @@ nodeenv = ">=0.11.1"
 pyyaml = ">=5.1"
 virtualenv = ">=20.10.0"
 
+[[package]]
+name = "prometheus-client"
+version = "0.20.0"
+description = "Python client for the Prometheus monitoring system."
+optional = true
+python-versions = ">=3.8"
+files = [
+    {file = "prometheus_client-0.20.0-py3-none-any.whl", hash = "sha256:cde524a85bce83ca359cc837f28b8c0db5cac7aa653a588fd7e84ba061c329e7"},
+    {file = "prometheus_client-0.20.0.tar.gz", hash = "sha256:287629d00b147a32dcb2be0b9df905da599b2d82f80377083ec8463309a4bb89"},
+]
+
+[package.extras]
+twisted = ["twisted"]
+
 [[package]]
 name = "protobuf"
 version = "3.20.3"
@@ -3734,6 +3673,7 @@ files = [
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
+    {file = "PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290"},
     {file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
     {file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
     {file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
@@ -3741,8 +3681,16 @@ files = [
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
+    {file = "PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b"},
     {file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
     {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a08c6f0fe150303c1c6b71ebcd7213c2858041a7e01975da3a99aed1e7a378ef"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
+    {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df"},
     {file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
@@ -3759,6 +3707,7 @@ files = [
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
+    {file = "PyYAML-6.0.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6"},
     {file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
     {file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
     {file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
@@ -3766,21 +3715,12 @@ files = [
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
+    {file = "PyYAML-6.0.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5"},
     {file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
     {file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
     {file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
 ]
 
-[[package]]
-name = "quantile-python"
-version = "1.1"
-description = "Python Implementation of Graham Cormode and S. Muthukrishnan's Effective Computation of Biased Quantiles over Data Streams in ICDE'05"
-optional = true
-python-versions = "*"
-files = [
-    {file = "quantile-python-1.1.tar.gz", hash = "sha256:558629e88c497ef3b9b1081349c1ae6a61b53590e317724298ff54c674db7969"},
-]
-
 [[package]]
 name = "rapidfuzz"
 version = "3.6.1"
@@ -5256,31 +5196,32 @@ test = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "coverage-enable-subprocess
 
 [[package]]
 name = "vllm"
-version = "0.3.1"
+version = "0.3.3"
 description = "A high-throughput and memory-efficient inference and serving engine for LLMs"
 optional = true
 python-versions = ">=3.8"
 files = [
-    {file = "vllm-0.3.1-cp310-cp310-manylinux1_x86_64.whl", hash = "sha256:d86fa826c4c62ac7a654fb98f47e5693eadb1708ea3e8fa5333c8361b9bed663"},
-    {file = "vllm-0.3.1-cp311-cp311-manylinux1_x86_64.whl", hash = "sha256:88776a0360d2b66039bf6a88ed4885826fda2fe1829c25f4291f898f909adf0c"},
-    {file = "vllm-0.3.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:5c9391a01be4aa2bc53dc95a5dd7b71b31725e8ae7493a15caa38b5ba80e9cb9"},
-    {file = "vllm-0.3.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:45d3d46a6d577c517a9760e17f18a0c4a902d2207a2187b841d20e5b51699291"},
-    {file = "vllm-0.3.1.tar.gz", hash = "sha256:c61cf77c40ab9a3d0658f9cbe5a63d72f60ccbfed16624b224b78bd5c8cba60a"},
+    {file = "vllm-0.3.3-cp310-cp310-manylinux1_x86_64.whl", hash = "sha256:a49f2de8088b36d74f0a29cfaf0d23c09ac01b707aff1a6b606c0ba991074285"},
+    {file = "vllm-0.3.3-cp311-cp311-manylinux1_x86_64.whl", hash = "sha256:12676b8fae922fff52bdb029b5295692a1393be800e5eec8fe3ee8eef7c080fd"},
+    {file = "vllm-0.3.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:fdbe8d3727312472159921ace712007749600ca557860d03e16a52ac1127c163"},
+    {file = "vllm-0.3.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:128d5906cf10c66c0fc8553344766cbfede6d924ae4c02589d6b66ab556b954a"},
+    {file = "vllm-0.3.3.tar.gz", hash = "sha256:24b70159bbcfd441bfa9d3e226ba8f5db74837c5325fea4a2104cf46c5d8246e"},
 ]
 
 [package.dependencies]
-aioprometheus = {version = "*", extras = ["starlette"]}
 cupy-cuda12x = "12.1.0"
 fastapi = "*"
 ninja = "*"
 numpy = "*"
+outlines = ">=0.0.27"
+prometheus-client = ">=0.18.0"
 psutil = "*"
 pydantic = ">=2.0"
 pynvml = "11.5.0"
 ray = ">=2.9"
 sentencepiece = "*"
 torch = "2.1.2"
-transformers = ">=4.37.0"
+transformers = ">=4.38.0"
 triton = ">=2.1.0"
 uvicorn = {version = "*", extras = ["standard"]}
 xformers = "0.0.23.post1"
@@ -5720,4 +5661,4 @@ openai = ["levenshtein", "openai", "tiktoken"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.10,<3.12"
-content-hash = "1fb765e3476d257e9c0b50f613545a1d08fb41f8e4a8074a62ae4eb03d27685f"
+content-hash = "ee5c5498ebdf4bf2dcc52f6a2596f5f69fdd680f4815cf6ad097eab95f1814ce"
diff --git a/pyproject.toml b/pyproject.toml
index bf9516e3a..68b56cd79 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -40,8 +40,8 @@ rouge-score = { version = "^0.1.2", optional = true }
 bert-score = { version = "^0.3.13", optional = true }
 demjson3 = { version = "^3.0.6", optional = true }
 bitsandbytes = { markers = "sys_platform != 'darwin' or platform_machine != 'arm64'", version = "^0.42.0", optional = true }
-vllm = { markers = "sys_platform != 'darwin'", version = ">=0.3.0,!=0.3.2,<0.4.0", optional = true }
-outlines = {version = "^0.0.34", optional = true}
+vllm = { markers = "sys_platform != 'darwin'", version = ">=0.3.3,<0.4.0", optional = true }
+outlines = { git = "https://github.com/saattrupdan/outlines", rev = "feat/add-transformers-integration", optional = true }
 
 # Needed for loading OLMO based models
 ai2-olmo = { version = "^0.2.4", optional = true }
@@ -130,6 +130,7 @@ filterwarnings = [
     "error",
     "ignore::UserWarning",
     "ignore::DeprecationWarning",
+    "ignore::PendingDeprecationWarning",
     "ignore::ImportWarning",
 ]
 log_cli_level = "info"
diff --git a/src/scandeval/benchmark_config_factory.py b/src/scandeval/benchmark_config_factory.py
index 3b39aef55..24240c97c 100644
--- a/src/scandeval/benchmark_config_factory.py
+++ b/src/scandeval/benchmark_config_factory.py
@@ -3,6 +3,7 @@
 import importlib.util
 import logging
 import os
+import sys
 
 import torch
 
@@ -138,7 +139,7 @@ def build_benchmark_config(
 
     if use_flash_attention is None:
         use_flash_attention = importlib.util.find_spec("flash_attn") is not None
-        if not use_flash_attention and first_time:
+        if not use_flash_attention and first_time and torch_device.type == "cuda":
             message = (
                 "Flash attention has not been installed, so this will not be used. "
                 "To install it, run `pip install -U wheel && "
@@ -154,6 +155,10 @@ def build_benchmark_config(
                 )
             logger.info(message)
 
+    # Set variable with number of iterations
+    if hasattr(sys, "_called_from_test"):
+        num_iterations = 2
+
     return BenchmarkConfig(
         model_languages=model_languages,
         dataset_languages=dataset_languages,
diff --git a/src/scandeval/benchmark_dataset.py b/src/scandeval/benchmark_dataset.py
index 2607a475f..ed2261d5d 100644
--- a/src/scandeval/benchmark_dataset.py
+++ b/src/scandeval/benchmark_dataset.py
@@ -148,14 +148,8 @@ def benchmark(self, model_id: str) -> tuple[ScoreDict, dict[str, bool | int]]:
             benchmarking_generative_model=benchmarking_generative_model,
         )
 
-        # Set variable with number of iterations
-        if hasattr(sys, "_called_from_test"):
-            num_iter = 2
-        else:
-            num_iter = self.benchmark_config.num_iterations
-
         if self.dataset_config.task != SPEED:
-            train, val, tests = self._load_data(num_iter=num_iter, rng=rng)
+            train, val, tests = self._load_data(rng=rng)
             prepared_train, prepared_val, prepared_tests = self._load_prepared_data(
                 train=train,
                 val=val,
@@ -168,7 +162,7 @@ def benchmark(self, model_id: str) -> tuple[ScoreDict, dict[str, bool | int]]:
 
         # Set up progress bar
         itr = tqdm(
-            iterable=range(num_iter),
+            iterable=range(self.benchmark_config.num_iterations),
             desc="Benchmarking",
             disable=not self.benchmark_config.progress_bar,
         )
@@ -333,13 +327,11 @@ def _get_metadata(
         return metadata_dict
 
     def _load_data(
-        self, num_iter: int, rng: np.random.Generator
+        self, rng: np.random.Generator
     ) -> tuple[Dataset, Dataset, list[Dataset]]:
         """Load the raw bootstrapped datasets.
 
         Args:
-            num_iter:
-                The number of iterations to run.
             rng:
                 The random number generator to use.
 
@@ -390,7 +382,9 @@ def _load_data(
             test = test.select(range(2))
 
         # Bootstrap the test set
-        test_bidxs = rng.integers(0, len(test), size=(num_iter, len(test)))
+        test_bidxs = rng.integers(
+            0, len(test), size=(self.benchmark_config.num_iterations, len(test))
+        )
         tests = [test.select(test_bidxs[idx]) for idx in range(test_bidxs.shape[0])]
 
         return train, val, tests
@@ -436,7 +430,7 @@ def _load_prepared_data(
 
         # Prepare the train and validation datasets
         with tqdm(
-            total=4 if hasattr(sys, "_called_from_test") else 12,
+            total=2 + self.benchmark_config.num_iterations,
             desc="Preprocessing data splits",
             disable=hasattr(sys, "_called_from_test"),
         ) as pbar:
diff --git a/src/scandeval/dataset_configs.py b/src/scandeval/dataset_configs.py
index b8bff3a7d..1a6824b76 100644
--- a/src/scandeval/dataset_configs.py
+++ b/src/scandeval/dataset_configs.py
@@ -748,7 +748,7 @@ def get_dataset_config(dataset_name: str) -> DatasetConfig:
 WIKI_LINGUA_NL_CONFIG = DatasetConfig(
     name="wiki-lingua-nl",
     pretty_name="the Dutch part of the truncated version of the summarisation dataset "
-    " WikiLingua",
+    "WikiLingua",
     huggingface_id="ScandEval/wiki-lingua-nl-mini",
     task=SUMM,
     languages=[NL],
@@ -1076,6 +1076,21 @@ def get_dataset_config(dataset_name: str) -> DatasetConfig:
     prompt_label_mapping=dict(a="a", b="b", c="c", d="d"),
     num_few_shot_examples=5,
     max_generated_tokens=3,
+    unofficial=True,
+)
+
+WINOGRANDE_IS = DatasetConfig(
+    name="winogrande-is",
+    pretty_name="the Icelandic common-sense reasoning dataset "
+    "Winogrande-is, manually translated from the English Winogrande dataset",
+    huggingface_id="ScandEval/winogrande-is",
+    task=COMMON_SENSE,
+    languages=[IS],
+    prompt_prefix="Eftirfarandi eru fjölvalsspurningar (með svörum).",
+    prompt_template="Spurningar: {text}\nSvara: {label}",
+    prompt_label_mapping=dict(a="a", b="b", c="c", d="d"),
+    num_few_shot_examples=5,
+    max_generated_tokens=3,
 )
 
 HELLASWAG_DE_CONFIG = DatasetConfig(
diff --git a/src/scandeval/model_setups/hf.py b/src/scandeval/model_setups/hf.py
index 5848846eb..e8935c308 100644
--- a/src/scandeval/model_setups/hf.py
+++ b/src/scandeval/model_setups/hf.py
@@ -246,18 +246,6 @@ def load_model(
         if load_in_4bit and importlib.util.find_spec("bitsandbytes") is None:
             raise NeedsExtraInstalled(extra="generative")
 
-        bnb_config = (
-            BitsAndBytesConfig(
-                load_in_4bit=load_in_4bit,
-                bnb_4bit_compute_dtype=(
-                    torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
-                ),
-                bnb_4bit_use_double_quant=True,
-            )
-            if load_in_4bit
-            else None
-        )
-
         config = self._load_hf_model_config(
             model_id=model_id,
             num_labels=dataset_config.num_labels,
@@ -267,6 +255,21 @@ def load_model(
             model_cache_dir=model_config.model_cache_dir,
         )
 
+        use_bf16 = (
+            self.benchmark_config.device == torch.device("cuda")
+            and torch.cuda.is_bf16_supported()
+            and config.to_dict().get("torch_dtype") == "bfloat16"
+        )
+        bnb_config = (
+            BitsAndBytesConfig(
+                load_in_4bit=load_in_4bit,
+                bnb_4bit_compute_dtype=torch.bfloat16 if use_bf16 else torch.float16,
+                bnb_4bit_use_double_quant=True,
+            )
+            if load_in_4bit
+            else None
+        )
+
         use_vllm = (
             model_config.task in GENERATIVE_MODEL_TASKS
             and self.benchmark_config.device == torch.device("cuda")
diff --git a/src/scandeval/structured_generation_utils.py b/src/scandeval/structured_generation_utils.py
index 7642c7909..2c95e6d1c 100644
--- a/src/scandeval/structured_generation_utils.py
+++ b/src/scandeval/structured_generation_utils.py
@@ -1,24 +1,20 @@
 """Utility functions related to structured generation."""
 
 import importlib.util
-import json
 import math
-from collections import defaultdict
-from typing import TYPE_CHECKING, Any, Callable, DefaultDict
+from typing import TYPE_CHECKING, Any, Callable
 
 import torch
 from pydantic import BaseModel, conlist, create_model
-from transformers import SPIECE_UNDERLINE, PreTrainedTokenizerBase
-
-from .exceptions import NeedsExtraInstalled
+from transformers import PreTrainedTokenizerBase
 
 if importlib.util.find_spec("outlines") is not None:
-    from outlines.fsm.fsm import RegexFSM
-    from outlines.fsm.json_schema import build_regex_from_schema
-    from outlines.serve.vllm import JSONLogitsProcessor
+    from outlines.integrations.transformers import JSONPrefixAllowedTokens
+    from outlines.integrations.vllm import JSONLogitsProcessor
 
 if TYPE_CHECKING:
-    from outlines.serve.vllm import LLM
+    from outlines.integrations.transformers import JSONPrefixAllowedTokens
+    from vllm import LLM
 
 
 def get_ner_schema(ner_tag_names: list[str]) -> type[BaseModel]:
@@ -39,112 +35,112 @@ def get_ner_schema(ner_tag_names: list[str]) -> type[BaseModel]:
     return schema
 
 
-class JSONPrefixAllowedTokens:
-    """Class to be used in `PreTrainedModel.generate` to allow tokens based on JSON."""
-
-    def __init__(
-        self,
-        ner_tag_names: list[str],
-        tokenizer: PreTrainedTokenizerBase,
-        forbidden_token_ids: list[int] = list(),
-    ):
-        """Compile the FSM that drives the JSON-guided generation.
-
-        Args:
-            ner_tag_names:
-                The NER tag names.
-            tokenizer:
-                The tokenizer to use for tokenizing the JSON Schema.
-            forbidden_token_ids:
-                The token IDs that are forbidden to be generated.
-        """
-        if importlib.util.find_spec("outlines") is None:
-            raise NeedsExtraInstalled(extra="generative")
-
-        schema = get_ner_schema(ner_tag_names=ner_tag_names)
-        schema_str = json.dumps(schema.model_json_schema())
-        regex_string = build_regex_from_schema(
-            schema=schema_str, whitespace_pattern=r" ?"
-        )
-        tokenizer = self.adapt_tokenizer(tokenizer=tokenizer)
-        fsm = RegexFSM(regex_string, tokenizer)
-        self.forbidden_token_ids = forbidden_token_ids
-        self.fsm = fsm
-
-    def __call__(self, batch_id: int, sent: torch.Tensor) -> list[int]:
-        """Use the FSM to get the allowed tokens for the next token.
-
-        Args:
-            batch_id:
-                The batch ID.
-            sent:
-                The input tensor.
-
-        Returns:
-            The allowed tokens.
-        """
-        sent = sent.tolist()
-        seq_id = hash(tuple(sent))
-
-        if len(sent) == 0:
-            self.fsm_state: DefaultDict[int, int] = defaultdict(int)
-        else:
-            last_token = sent[-1]
-            last_seq_id = hash(tuple(sent[:-1]))
-            self.fsm_state[seq_id] = self.fsm.next_state(
-                self.fsm_state[last_seq_id], last_token
-            )
-
-        allowed_tokens = [
-            token_id
-            for token_id in self.fsm.allowed_token_ids(self.fsm_state[seq_id])
-            if token_id not in self.forbidden_token_ids
-        ]
-        return allowed_tokens
-
-    @staticmethod
-    def adapt_tokenizer(tokenizer: PreTrainedTokenizerBase) -> PreTrainedTokenizerBase:
-        """Adapt vLLM's tokenizer to use to compile the FSM.
-
-        The API of Outlines tokenizers is slightly different to that of
-        `transformers`. In addition we need to handle the missing spaces to
-        Llama's tokenizer to be able to compile FSMs for this model.
-
-        Args:
-            tokenizer:
-                The tokenizer to adapt.
-
-        Returns:
-            The adapted tokenizer.
-        """
-        tokenizer.vocabulary = tokenizer.get_vocab()
-        tokenizer.special_tokens = set(tokenizer.all_special_tokens)
-
-        def convert_token_to_string(token: str) -> str:
-            """Patched way to convert a token to a string.
-
-            Args:
-                token:
-                    The token to convert.
-
-            Returns:
-                The string representation of the token.
-            """
-            string = tokenizer.convert_tokens_to_string([token])
-
-            # A hack to handle missing spaces to HF's Llama tokenizers
-            if token.startswith(SPIECE_UNDERLINE) or token == "<0x20>":
-                return " " + string
-
-            return string
-
-        tokenizer.convert_token_to_string = convert_token_to_string
-        return tokenizer
+# class JSONPrefixAllowedTokens:
+#     """Class to be used in `PreTrainedModel.generate` to allow tokens based on JSON."""
+#
+#     def __init__(
+#         self,
+#         ner_tag_names: list[str],
+#         tokenizer: PreTrainedTokenizerBase,
+#         forbidden_token_ids: list[int] = list(),
+#     ):
+#         """Compile the FSM that drives the JSON-guided generation.
+#
+#         Args:
+#             ner_tag_names:
+#                 The NER tag names.
+#             tokenizer:
+#                 The tokenizer to use for tokenizing the JSON Schema.
+#             forbidden_token_ids:
+#                 The token IDs that are forbidden to be generated.
+#         """
+#         if importlib.util.find_spec("outlines") is None:
+#             raise NeedsExtraInstalled(extra="generative")
+#
+#         schema = get_ner_schema(ner_tag_names=ner_tag_names)
+#         schema_str = json.dumps(schema.model_json_schema())
+#         regex_string = build_regex_from_schema(
+#             schema=schema_str, whitespace_pattern=r" ?"
+#         )
+#         tokenizer = self.adapt_tokenizer(tokenizer=tokenizer)
+#         fsm = RegexFSM(regex_string, tokenizer)
+#         self.forbidden_token_ids = forbidden_token_ids
+#         self.fsm = fsm
+#
+#     def __call__(self, batch_id: int, sent: torch.Tensor) -> list[int]:
+#         """Use the FSM to get the allowed tokens for the next token.
+#
+#         Args:
+#             batch_id:
+#                 The batch ID.
+#             sent:
+#                 The input tensor.
+#
+#         Returns:
+#             The allowed tokens.
+#         """
+#         sent = sent.tolist()
+#         seq_id = hash(tuple(sent))
+#
+#         if len(sent) == 0:
+#             self.fsm_state: DefaultDict[int, int] = defaultdict(int)
+#         else:
+#             last_token = sent[-1]
+#             last_seq_id = hash(tuple(sent[:-1]))
+#             self.fsm_state[seq_id] = self.fsm.next_state(
+#                 self.fsm_state[last_seq_id], last_token
+#             )
+#
+#         allowed_tokens = [
+#             token_id
+#             for token_id in self.fsm.allowed_token_ids(self.fsm_state[seq_id])
+#             if token_id not in self.forbidden_token_ids
+#         ]
+#         return allowed_tokens
+#
+#     @staticmethod
+#     def adapt_tokenizer(tokenizer: PreTrainedTokenizerBase) -> PreTrainedTokenizerBase:
+#         """Adapt vLLM's tokenizer to use to compile the FSM.
+#
+#         The API of Outlines tokenizers is slightly different to that of
+#         `transformers`. In addition we need to handle the missing spaces to
+#         Llama's tokenizer to be able to compile FSMs for this model.
+#
+#         Args:
+#             tokenizer:
+#                 The tokenizer to adapt.
+#
+#         Returns:
+#             The adapted tokenizer.
+#         """
+#         tokenizer.vocabulary = tokenizer.get_vocab()
+#         tokenizer.special_tokens = set(tokenizer.all_special_tokens)
+#
+#         def convert_token_to_string(token: str) -> str:
+#             """Patched way to convert a token to a string.
+#
+#             Args:
+#                 token:
+#                     The token to convert.
+#
+#             Returns:
+#                 The string representation of the token.
+#             """
+#             string = tokenizer.convert_tokens_to_string([token])
+#
+#             # A hack to handle missing spaces to HF's Llama tokenizers
+#             if token.startswith(SPIECE_UNDERLINE) or token == "<0x20>":
+#                 return " " + string
+#
+#             return string
+#
+#         tokenizer.convert_token_to_string = convert_token_to_string
+#         return tokenizer
 
 
 def get_ner_prefix_allowed_tokens_fn(
     ner_tag_names: list[str], tokenizer: PreTrainedTokenizerBase
-) -> JSONPrefixAllowedTokens:
+) -> Callable[[int, torch.Tensor], list[int]]:
     """Get the prefix allowed tokens function for the NER task, used in `transformers`.
 
     Args:
@@ -156,6 +152,11 @@ def get_ner_prefix_allowed_tokens_fn(
     Returns:
         The prefix allowed tokens function for the NER task.
     """
+    schema = get_ner_schema(ner_tag_names=ner_tag_names)
+    json_generation_prefix_allowed_tokens_fn = JSONPrefixAllowedTokens(
+        schema=schema, tokenizer_or_pipe=tokenizer, whitespace_pattern=r" ?"
+    )
+
     forbidden_token_ids = list()
     forbidden_tokens = ["\n", "\n\n", "\n\n\n", "\t", "\t\t", "\t\t\t"]
     for forbidden_token in forbidden_tokens:
@@ -163,11 +164,25 @@ def get_ner_prefix_allowed_tokens_fn(
             list(tokenizer(forbidden_token, add_special_tokens=False).input_ids)
         )
     forbidden_token_ids = list(set(forbidden_token_ids))
-    return JSONPrefixAllowedTokens(
-        ner_tag_names=ner_tag_names,
-        tokenizer=tokenizer,
-        forbidden_token_ids=forbidden_token_ids,
-    )
+
+    def prefix_allowed_tokens_fn(batch_id: int, sent: torch.Tensor) -> list[int]:
+        """Functions used to bias the generation of a `transformers` model.
+
+        Args:
+            batch_id:
+                The batch ID.
+            sent:
+                The input tensor.
+
+        Returns:
+            The allowed tokens.
+        """
+        allowed_tokens = json_generation_prefix_allowed_tokens_fn(
+            batch_id=batch_id, sent=sent
+        )
+        return [token for token in allowed_tokens if token not in forbidden_token_ids]
+
+    return prefix_allowed_tokens_fn
 
 
 def get_ner_logits_processors(
diff --git a/src/scripts/create_winogrande_is.py b/src/scripts/create_winogrande_is.py
new file mode 100644
index 000000000..dd586d407
--- /dev/null
+++ b/src/scripts/create_winogrande_is.py
@@ -0,0 +1,126 @@
+"""Create the Winogrande-is dataset and upload it to the HF Hub."""
+
+from collections import Counter
+
+import pandas as pd
+from datasets import Dataset, DatasetDict, Split, load_dataset
+from huggingface_hub import HfApi
+from requests import HTTPError
+from scripts.constants import (
+    MAX_NUM_CHARS_IN_INSTRUCTION,
+    MAX_NUM_CHARS_IN_OPTION,
+    MAX_REPETITIONS,
+    MIN_NUM_CHARS_IN_OPTION,
+)
+from sklearn.model_selection import train_test_split
+
+
+def main() -> None:
+    """Create the Winogrande-is dataset and upload it to the HF Hub."""
+    # Define the base download URL
+    repo_id = "mideind/icelandic-winogrande"
+
+    # Download the dataset
+    dataset = load_dataset(path=repo_id, split="train")
+    assert isinstance(dataset, Dataset)
+
+    # Convert the dataset to a dataframe
+    df = dataset.to_pandas()
+    assert isinstance(df, pd.DataFrame)
+
+    # Rename the columns
+    df.rename(
+        columns=dict(
+            sentence="instruction",
+            option1="option_a",
+            option2="option_b",
+            answer="label",
+        ),
+        inplace=True,
+    )
+
+    # Change the label from 1/2 to a/b
+    df.label = df.label.map({"1": "a", "2": "b"})
+
+    # Remove the samples with overly short or long texts
+    df = df[
+        (df.instruction.str.len() <= MAX_NUM_CHARS_IN_INSTRUCTION)
+        & (df.option_a.str.len() >= MIN_NUM_CHARS_IN_OPTION)
+        & (df.option_a.str.len() <= MAX_NUM_CHARS_IN_OPTION)
+        & (df.option_b.str.len() >= MIN_NUM_CHARS_IN_OPTION)
+        & (df.option_b.str.len() <= MAX_NUM_CHARS_IN_OPTION)
+    ]
+
+    def is_repetitive(text: str) -> bool:
+        """Return True if the text is repetitive."""
+        max_repetitions = max(Counter(text.split()).values())
+        return max_repetitions > MAX_REPETITIONS
+
+    # Remove overly repetitive samples
+    df = df[
+        ~df.instruction.apply(is_repetitive)
+        & ~df.option_a.apply(is_repetitive)
+        & ~df.option_b.apply(is_repetitive)
+    ]
+
+    # Make a `text` column with all the options in it
+    df["text"] = [
+        row.instruction.replace("\n", " ").strip() + "\n"
+        "Svarmöguleikar:\n"
+        "a. " + row.option_a.replace("\n", " ").strip() + "\n"
+        "b. " + row.option_b.replace("\n", " ").strip()
+        for _, row in df.iterrows()
+    ]
+
+    # Make the `label` column case-consistent with the `text` column
+    df.label = df.label.str.lower()
+
+    # Only keep the `text` and `label` columns
+    df = df[["text", "label"]]
+
+    # Remove duplicates
+    df.drop_duplicates(inplace=True)
+    df.reset_index(drop=True, inplace=True)
+
+    # Create validation split
+    val_size = 128
+    traintest_arr, val_arr = train_test_split(df, test_size=val_size, random_state=4242)
+    traintest_df = pd.DataFrame(traintest_arr, columns=df.columns)
+    val_df = pd.DataFrame(val_arr, columns=df.columns)
+
+    # Create test split
+    train_size = 64
+    train_arr, test_arr = train_test_split(
+        traintest_df, train_size=train_size, random_state=4242
+    )
+    train_df = pd.DataFrame(train_arr, columns=df.columns)
+    test_df = pd.DataFrame(test_arr, columns=df.columns)
+
+    # Reset the index
+    train_df = train_df.reset_index(drop=True)
+    val_df = val_df.reset_index(drop=True)
+    test_df = test_df.reset_index(drop=True)
+
+    # Collect datasets in a dataset dictionary
+    dataset = DatasetDict(
+        train=Dataset.from_pandas(train_df, split=Split.TRAIN),
+        val=Dataset.from_pandas(val_df, split=Split.VALIDATION),
+        test=Dataset.from_pandas(test_df, split=Split.TEST),
+    )
+
+    # Create dataset ID
+    dataset_id = "ScandEval/winogrande-is"
+
+    # Remove the dataset from Hugging Face Hub if it already exists
+    try:
+        api = HfApi()
+        api.delete_repo(dataset_id, repo_type="dataset")
+    except HTTPError:
+        pass
+
+    # Push the dataset to the Hugging Face Hub
+    dataset.push_to_hub(dataset_id)
+
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/tests/test_benchmark_config_factory.py b/tests/test_benchmark_config_factory.py
index d91862b97..fd4d4bfa7 100644
--- a/tests/test_benchmark_config_factory.py
+++ b/tests/test_benchmark_config_factory.py
@@ -17,15 +17,19 @@
 
 
 @pytest.fixture(scope="module")
-def all_dataset_names(all_dataset_configs) -> Generator[list[str], None, None]:
+def all_official_dataset_names(all_dataset_configs) -> Generator[list[str], None, None]:
     """Fixture for all linguistic acceptability dataset configurations."""
-    yield [cfg.name for cfg in all_dataset_configs]
+    yield [cfg.name for cfg in all_dataset_configs if not cfg.unofficial]
 
 
 @pytest.fixture(scope="module")
-def all_la_dataset_names(all_dataset_configs) -> Generator[list[str], None, None]:
+def all_official_la_dataset_names(
+    all_dataset_configs,
+) -> Generator[list[str], None, None]:
     """Fixture for all linguistic acceptability dataset configurations."""
-    yield [cfg.name for cfg in all_dataset_configs if LA == cfg.task]
+    yield [
+        cfg.name for cfg in all_dataset_configs if LA == cfg.task and not cfg.unofficial
+    ]
 
 
 @pytest.mark.parametrize(
@@ -101,14 +105,14 @@ def test_prepare_languages(input_language_codes, input_language, expected_langua
             None,
             list(get_all_languages().values()),
             list(get_all_tasks().values()),
-            "all_dataset_names",
+            "all_official_dataset_names",
         ),
         (
             "linguistic-acceptability",
             None,
             list(get_all_languages().values()),
             [LA],
-            "all_la_dataset_names",
+            "all_official_la_dataset_names",
         ),
         (
             None,