Repository: huggingface/yourbench
Pull request #27: Housekeeping
Author: login: alozowski
Created: 2025-03-21T23:13:32Z | Merged: 2025-03-22T20:00:31Z

Several "housekeeping" changes:
- Improved file structure to store the examples clearly 
- Updated `pyproject.toml` to include project authors
- Add missing `__init__.py` files
- Updated `.gitignore` and Makefile


Diff:
diff --git a/.gitignore b/.gitignore
index 2281e6a3..26f582a7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -165,3 +165,9 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Linter cache
+.ruff_cache
+
+# OS specific cache
+.DS_Store
\ No newline at end of file
diff --git a/Makefile b/Makefile
index f388eda6..c86bf1dd 100644
--- a/Makefile
+++ b/Makefile
@@ -4,13 +4,13 @@
 style:
 	@echo "Applying style fixes to $(file)"
 	ruff format $(file)
-	ruff check --fix $(file) --line-length 119
+	ruff check --fix $(file)
 
 # Checks code quality for the specified file or directory without applying fixes
 check:
 	@echo "Checking code quality for $(file) without fixes"
 	ruff format --diff $(file)
-	ruff check $(file) --line-length 119
+	ruff check $(file)
 
 # Applies PEP8 formatting and checks the entire codebase
 all: style
@@ -20,4 +20,4 @@ all: style
 # Checks the entire codebase without applying fixes (for CI)
 quality: check
 	@echo "Checking quality of the entire codebase without fixes"
-	$(MAKE) check file=.
\ No newline at end of file
+	$(MAKE) check file=.
diff --git a/configs/minimal.yaml b/configs/minimal.yaml
deleted file mode 100644
index a875420d..00000000
--- a/configs/minimal.yaml
+++ /dev/null
@@ -1,37 +0,0 @@
-# minimal example
-
-hf_configuration:
-  token: $HF_TOKEN 
-  hf_organization: $HF_ORGANIZATION 
-  private: false 
-  global_dataset_name: sumuks/yourbench-example
-
-# === MODEL CONFIGURATION ===
-model_list: 
-  # to use openai inference
-  - model_name: gpt-4o-mini
-    
-model_roles:
-  ingestion:
-    - gpt-4o-mini
-
-pipeline:
-  ingestion:
-    run: true
-    source_documents_dir: data/example/raw
-    output_dir: data/example/processed
-  upload_ingest_to_hub:
-    run: true
-    source_documents_dir: data/example/processed
-  summarization:
-    run: true
-  chunking:
-    run: true
-  single_shot_question_generation:
-    run: true
-  multi_hop_question_generation:
-    run: true
-  lighteval:
-    run: true
-  
-  
diff --git a/configs/example.yaml b/example/configs/example.yaml
similarity index 97%
rename from configs/example.yaml
rename to example/configs/example.yaml
index 2d602e44..0a1ac57f 100644
--- a/configs/example.yaml
+++ b/example/configs/example.yaml
@@ -35,9 +35,9 @@ pipeline:
   ingestion:
     run: true
     # set this to where your raw documents are located
-    source_documents_dir: data/example/raw
+    source_documents_dir: example/data/raw
     # .... and this to where you want them to be processed to
-    output_dir: data/example/processed
+    output_dir: example/data/processed
 
   # to convert your documents to a huggingface dataset
   upload_ingest_to_hub:
diff --git a/data/example/raw/hurricane_faqs.pdf b/example/data/raw/hurricane_faqs.pdf
similarity index 100%
rename from data/example/raw/hurricane_faqs.pdf
rename to example/data/raw/hurricane_faqs.pdf
diff --git a/data/example/raw/the_bitter_lesson.html b/example/data/raw/the_bitter_lesson.html
similarity index 100%
rename from data/example/raw/the_bitter_lesson.html
rename to example/data/raw/the_bitter_lesson.html
diff --git a/pyproject.toml b/pyproject.toml
index a2b162f3..9e3ce87c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,12 @@
 [project]
 name = "yourbench"
 version = "0.2.0"
+authors = [
+    { name = "Sumuk Shashidhar", email = "sumuks2@illinois.edu" },
+    { name = "Alina Lozovskaia", email = "alina.lozovskaia@huggingface.co" },
+    { name = "Clémentine Fourrier", email = "clementine@huggingface.co" },
+    { name = "Nathan Habib", email = "nathan.habib@huggingface.co" },
+]
 description = "Dynamic Evaluation Set Generation with Large Language Models"
 readme = "README.md"
 requires-python = ">=3.12, <3.13"
@@ -36,3 +42,26 @@ build-backend = "setuptools.build_meta"
 
 [tool.setuptools.packages.find]
 include = ["yourbench*"]
+
+[tool.ruff]
+line-length = 119
+exclude = ["**/*.ipynb"]
+
+lint.ignore = ["E501", "C901", "F841"]
+lint.select = ["C", "E", "F", "I", "W"]
+lint.fixable = ["A", "B", "C", "D", "E", "F", "I", "W"]
+preview = true
+
+[tool.ruff.lint.isort]
+length-sort = true
+lines-after-imports = 2
+no-lines-before = ["standard-library", "local-folder"]
+known-local-folder = ["yourbench"]
+known-first-party = ["datasets", "huggingface_hub"]
+split-on-trailing-comma = true
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+skip-magic-trailing-comma = false
+line-ending = "auto"
diff --git a/yourbench/__init__.py b/yourbench/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/yourbench/pipeline/__init__.py b/yourbench/pipeline/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/yourbench/pipeline/chunking.py b/yourbench/pipeline/chunking.py
index a54e22f8..0a4636b8 100644
--- a/yourbench/pipeline/chunking.py
+++ b/yourbench/pipeline/chunking.py
@@ -71,7 +71,7 @@
     )
     logger.info("Loaded 'perplexity' metric with model_id='gpt2'.")
 except Exception as perplexity_load_error:
-    logger.warning(
+    logger.info(
         f"Could not load perplexity metric from 'evaluate'. Skipping perplexity. Error: {perplexity_load_error}"
     )
     _perplexity_metric = None
@@ -82,7 +82,7 @@
 
     _use_textstat = True
 except ImportError:
-    logger.warning(
+    logger.info(
         "Package 'textstat' not installed. Readability metrics will be skipped."
     )
     _use_textstat = False
diff --git a/yourbench/utils/__init__.py b/yourbench/utils/__init__.py
new file mode 100644
index 00000000..e69de29b


Reviews:
user: login: clefourrier
body: lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-22T11:02:27Z
user: login: alozowski
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-22T19:51:17Z
user: login: clefourrier
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T08:02:51Z

Review comments:
diff_hunk: @@ -165,3 +165,9 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Linter cache
+.ruff_cache
+
+# OS specific cache
path: .gitignore
user: login: clefourrier
body: This one you should add to a gitignore on your machine as it's mac specific, but let's leave it here for now
created_at: 2025-03-22T11:02:21Z
author_association: MEMBER
diff_hunk: @@ -165,3 +165,9 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Linter cache
+.ruff_cache
+
+# OS specific cache
path: .gitignore
user: login: alozowski
body: It’s possible to ignore this locally, but other Mac users would need to do the same. We can add more OS-specific files later if needed
Ref: https://stackoverflow.com/questions/49305201/gitignore-only-on-local
created_at: 2025-03-22T19:51:16Z
author_association: COLLABORATOR
diff_hunk: @@ -165,3 +165,9 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Linter cache
+.ruff_cache
+
+# OS specific cache
path: .gitignore
user: login: clefourrier
body: Yes, that's precisely my point - this is not project specific, it's OS specific. You should be managing it at your machine's level :)
created_at: 2025-03-24T08:02:50Z
author_association: MEMBER

PR comments:
user: login: alozowski
created_at: 2025-03-21T23:31:40Z
body: Let's skip the quality check in this PR but I'll make a separate PR only for the quality check 
author_association: COLLABORATOR
user: login: clefourrier
created_at: 2025-03-22T11:05:49Z
body: Ofc, first fix the style :)
author_association: MEMBER
user: login: alozowski
created_at: 2025-03-22T11:17:08Z
body: @clefourrier this PR changes the style check as I corrected the style check a bit -- let's merge it without the style check and then I create another PR and apply the style for all the files? Separately, ofc, not for the whole code base :D 
author_association: COLLABORATOR
user: login: clefourrier
created_at: 2025-03-22T11:22:16Z
body: Ok :)
author_association: MEMBER