DOI: 10.5281/zenodo.17053215
title: C-PASS: An Organization-centric Framework for Compliance and Penalty Assessment Using Large Language Models

description: Supplementary Materials – Annotated Corpora and Outputs
This package contains the annotated corpora and intermediate outputs associated with the paper submission "C-PASS: An Organization-centric Framework for Compliance and Penalty Assessment Using Large Language Models".



Contents
Results from each stage of the processing pipeline and the baseline approaches are provided in separate folders:





 deontic_modality_classification (Stage 1.1)/  



Deontic_modality_ground_truth_CDCA: Ground truth deontic modality classification (Obligation, Permission, Prohibition Other) for CDCA. 

Deontic_modality_ground_truth_EPMA: Ground truth deontic modality classification (Obligation, Permission, Prohibition, Other) for EPMA. 

Setfit_predictions_CDCA_and_EPMA: Deontic modality classification outputs using Setfit for CDCA and EPMA.  







definition_retrieval (Stage 1.2)/  



definitions_CDCA: Definitions retrieved from 'Other' clauses in CDCA. 

definitions_EPMA: Definitions retrieved from 'Other' clauses in EPMA.  






binding_clause_identification (Stage 1.3)/  



CDCA_profile_1_output_with_ground_truth: Binding clause identification output and ground truth for CDCA profile 1. 

CDCA_profile_2_output_with_ground_truth: Binding clause identification output and ground truth for CDCA profile 2. 

EPMA_profile_1_output_with_ground_truth: Binding clause identification output and ground truth for EPMA profile 1. 

EPMA_profile_2_output_with_ground_truth: Binding clause identification output and ground truth for EPMA profile 2.  






compliance_task_generation (Stage 2)/



CDCA_compliance_tasks: Compliance tasks and associated elements from CDCA clauses.

CDCA_compliance_tasks_evaluation: "LLM-as-a-Judge" evaluation scores and justifications for CDCA compliance tasks for GPT-4.1. 

EPMA_compliance_tasks: Compliance tasks and associated elements from EPMA clauses.

EPMA_compliance_tasks_evaluation: "LLM-as-a-Judge" evaluation scores and justifications for EPMA compliance tasks for GPT-4.1.  

per_LLM_evaluation_output_profile_1/: "LLM-as-a-Judge" evaluation scores for profile 1 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct).  

per_LLM_evaluation_output_profile_2/: "LLM-as-a-Judge" evaluation scores for profile 2 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct).






baseline_approaches/



baseline_deontic_modality_classification (Stage 1.1)/



deontic_modality_classification_baseline: Zero-shot GPT-4.1 results for deontic modality classification for CDCA and EPMA.



baseline_binding_clause_identification (Stage 1.3)/



binding_clause_identification_baseline_CDCA: Baseline results for CDCA profile 1 and profile 2.

binding_clause_identification_baseline_EPMA: Baseline results for EPMA profile 1 and profile 2.



baseline_deontic_modality_classification (Stage 2)/



Profile 1/: Baseline results and "LLM-as-a-Judge" evaluation scores for profile 1 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct).

Profile 2/: Baseline results and "LLM-as-a-Judge" evaluation scores for profile 2 on CDCA and EPMA for all LLMs (GPT-4.1, GPT-4o, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct).






Each folder contains the corresponding outputs and/or ground truth for that stage.