SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Task	GPT-4	GPT-3.5	Claude-1
corporate_lobbying	81.7	59.1	75.8
learnedHands_benefits	87.9	62.1	66.7
learnedHands.business	81.6	58.6	47.7
learnedHands_consumer	76.2	59.3	58.1
learnedHands_courts	52.6	54.2	46.9
learnedHands_crime	81.0	62.4	59.0
learnedHands_divorce	84.0	59.3	53.3
learnedHands/domestic_violence	83.9	60.9	62.6
learnedHands_education	91.1	57.1	55.4
learnedHands_employment	69.9	67.7	49.3
learnedHands_estrates	96.6	59.0	74.7
learnedHands_family	86.2	57.1	50.9
learnedHands_health	87.2	65.0	49.6
learnedHands_housing	85.0	63.9	58.7
learnedHands-immigration	98.5	79.9	73.1
learnedHands_torts	70.6	60.0	53.2
learnedHandsTraffic	95.3	49.8	52.3
Table 66: Commercial models on issue-spotting tasks.
Task	Flan	Llama-2	OPT	Vicuna	WizardLM
corporate_lobbying	55.9	55.9	50.9	50.3	50.2
learnedHands_benefits	68.2	50.0	50.0	4.5	0.0
learnedHands.business	61.5	50.6	48.9	1.1	46.6
learnedHands_consumer	72.8	50.0	45.0	0.0	44.6
learnedHands_courts	58.9	49.5	50.0	50.0	0.0
learnedHands_crime	83.0	50.1	51.5	48.4	26.7
learnedHands_divorce	57.3	49.3	50.0	49.3	0.0
learnedHands/domestic_violence	68.4	48.9	50.0	0.0	0.0
learnedHands_education	89.3	50.0	46.4	50.0	26.8
learnedHands_employment	74.2	49.2	49.9	23.1	0.0
learnedHands_estrates	67.4	50.0	68.5	49.4	38.8
learnedHands_family	5.4	50.1	63.6	49.3	0.0
learnedHands_health	66.8	49.6	63.7	50.0	38.1
learnedHands_housing	68.7	49.8	49.5	47.8	0.0
learnedHands-immigration	79.9	51.5	53.0	50.0	53.0
learnedHands_torts	60.6	50.0	50.2	49.8	46.1
learnedHandsTraffic	84.2	49.8	58.6	10.8	38.7
Table 67: 13B models on issue-spotting tasks.
Task	Bloom	Falcon	Incite-Base	Incite-Inst.	Llama-2	MPT	OPT	Vicuna
corporate_lobbying	40.3	44.8	50.1	49.7	50.0	49.6	50.9	45.5
learned_hands_benefits	48.5	51.5	50.0	51.5	51.5	56.1	50.0	0.0
learned_hands.business	55.7	60.3	50.0	55.7	50.0	50.0	59.8	0.0
learned_hands consumer	45.9	45.0	49.7	48.0	50.0	46.6	48.9	7.5
learned_hands_courts	53.6	50.5	50.5	48.4	49.5	57.8	49.5	0.0
learned_hands_crime	50.3	51.6	50.0	54.1	51.0	53.9	51.7	0.0
learned_hands_divorce	48.0	50.0	50.0	62.0	46.7	58.0	49.3	0.0
learned_hands_domestic_violence	48.9	48.9	51.7	59.8	51.1	49.4	50.6	0.0
learned_hands_education	53.6	62.5	50.0	60.7	50.0	53.6	48.2	0.0
learned_hands_employment	50.6	49.7	49.9	54.8	49.4	51.1	49.7	0.0
learned_hands_estrates	51.7	51.1	50.0	46.1	50.6	62.4	55.6	0.0
learned_hands_family	55.0	57.3	49.6	60.8	50.2	56.0	53.3	0.0
learned_hands_health	49.1	52.7	50.0	59.3	52.2	57.1	53.5	0.0
learned_hands_housing	49.2	49.6	49.8	47.1	50.5	49.3	50.1	0.0
learned_hands-immigration	53.7	47.0	50.0	64.9	50.0	67.9	62.7	0.0
learned_hands_torts	51.2	48.8	50.0	56.0	50.0	48.4	50.0	0.0
learned_hands_traffic	54.7	50.0	49.8	54.3	50.2	56.5	57.2	13.3
Table 68: 7B models13d issue-spotting tasks.
Task	Bloom	Flan	Incite	Opt
corporate_lobbying	26.2	51.9	47.6	51.0
learnedHands_benefits	43.9	43.9	47.0	53.0
learnedHands.business	51.1	49.4	47.7	44.8
learnedHands_consumer	39.7	70.8	50.2	38.1
learnedHands_courts	56.2	46.9	47.4	60.4
learnedHands_crime	49.7	60.6	51.9	51.7
learnedHands_divorce	58.0	50.0	61.3	56.0
learnedHands/domestic_violence	36.2	49.4	52.9	51.7
learnedHands_education	50.0	67.9	51.8	46.4
learnedHands_employment	50.7	46.6	48.0	50.1
learnedHands_estrates	42.7	66.3	52.8	62.4
learnedHands_family	52.1	47.6	59.0	64.6
learnedHands_health	46.9	58.4	53.1	52.2
learnedHands_housing	51.2	51.7	47.1	51.8
learnedHands-immigration	59.0	73.1	53.7	56.7
learnedHands_torts	44.4	66.2	51.2	56.7
learnedHandsTraffic	46.8	65.3	46.4	65.3
Table 69: 3B models on issue-spotting tasks.
Task	GPT-4	GPT-3.5	Claude-1
abercrombie	85.3	63.2	66.3
diversity_1	100.0	88.4	83.4
diversity_2	99.8	87.3	92.9
diversity_3	97.0	89.4	84.3
diversity_4	100.0	90.1	97.9
diversity_5	93.2	92.6	81.0
diversity_6	90.5	77.3	56.4
hearsay	83.8	69.2	76.4
personal_jurisdiction	91.4	63.3	81.9
successor_liability	57.1	52.3	72.7
telemarketing_sales_rule	82.4	63.1	71.9
ucc_v_common_law	98.8	100.0	88.8
Table 70: Commercial models on rule-conclusion tasks.
Task	Flan	Llama-2	OPT	Vicuna	WizardLM
abercrombie	42.1	40.0	0.0	22.1	44.2
diversity_1	76.0	50.0	55.0	25.5	60.5
diversity_2	59.4	62.1	53.9	48.4	57.1
diversity_3	78.6	65.8	50.0	52.6	77.8
diversity_4	53.2	68.3	49.6	55.4	82.5
diversity_5	53.5	57.7	52.4	52.7	53.2
diversity_6	50.0	50.0	48.1	50.0	54.3
hearsay	64.0	56.5	52.3	48.7	66.3
personal_jurisdiction	62.6	57.9	51.3	0.0	64.0
successor_liability	52.7	51.2	26.4	0.0	39.1
telemarketing_sales_rule	69.8	74.3	50.0	63.1	72.7
ucc_v_common_law	98.1	77.9	52.5	0.0	79.8
Table 71: 13B models on rule-conclusion tasks.
Task	Bloom	Falcon	Incite-Base	Incite-Inst.	Llama-2	MPT	OPT	Vicuna
abercrombid	7.9	24.2	27.4	34.7	32.6	34.7	17.9	2.1
diversity_1	47.6	51.7	56.8	61.5	73.4	54.9	58.0	55.3
diversity_2	50.0	56.3	50.5	65.7	50.0	50.0	49.8	49.1
diversity_3	51.9	53.6	50.0	50.8	62.8	50.0	49.9	49.3
diversity_4	58.3	68.7	50.0	67.9	72.9	50.0	49.7	49.7
diversity_5	57.9	50.0	50.4	49.4	51.3	50.0	50.0	46.8
diversity_6	50.0	50.0	50.0	44.7	50.0	50.0	49.7	50.0
hearsay	51.7	53.7	50.0	73.2	64.1	61.7	36.4	30.7
personal_jurisdiction	51.8	54.6	40.0	46.1	52.6	43.8	50.6	50.0
successor_ liability	21.7	39.1	37.2	32.6	43.4	35.7	38.8	0.0
telemarketing_4 sales_rule	55.3	51.8	58.6	61.3	55.8	54.5	43.6
ucc_vcommon_law	50.0	50.0	50.0	50.0	56.6	50.0	50.0	0.0
Table 72: 7B models on rule-conclusion tasks.
Task	Bloom	Flan	Incite	Opt
abercrombie	20.0	31.6	25.3	26.3
diversity_1	50.0	50.0	50.3	51.5
diversity_2	50.0	50.0	49.3	52.6
diversity_3	50.0	50.0	46.2	50.0
diversity_4	50.0	50.0	62.5	51.4
diversity_5	51.8	50.0	52.8	50.0
diversity_6	50.5	50.0	51.5	50.0
hearsay	51.2	57.5	57.0	49.7
personal_jurisdiction	50.0	51.1	46.1	50.0
successor_liability	21.7	44.6	24.8	14.4
telemarketing_sales_rule	44.5	51.4	53.5	55.9
ucc_v_common_law	50.0	88.8	50.0	50.0
Table 73: 3B models on rule-conclusion tasks.
Task	GPT-4	GPT-3.5	Claude-1
canada_tax-court_outcomes	98.9	80.0	76.9
definition_classification	96.6	80.2	87.9
definition Extraction	81.8	85.0	82.7
function_of_decision_section	43.3	35.2	37.6
legal_reasoning_causality	84.5	72.1	66.9
oralArgument questionpurpose	37.4	28.4	35.1
overruling	95.2	88.9	95.4
scalr	77.9	58.8	64.7
textualism_tool_dictionaries	93.9	65.1	71.2
textualism_tool_plain	84.7	73.2	70.2
Table 74: Commercial models on rhetorical-understanding tasks.
Task	Flan	Llama-2	OPT	Vicuna	WizardLM
canada_tax-court_outcomes	71.7	34.4	1.8	2.9	75.1
definition_classification	80.8	51.2	57.4	50.0	81.5
definition Extraction	80.5	85.0	80.1	62.4	82.0
function_of_decision_section	34.6	18.0	20.8	14.2	13.2
legal_reasoning_causality	78.6	52.8	52.3	48.4	46.2
oralArgument_questionpurpose	24.5	20.0	15.0	16.1	29.3
overruling	94.2	92.3	78.3	52.7	89.5
scalr	66.5	56.7	20.2	5.0	45.6
textualism_tool_dictionaries	93.9	66.5	54.9	5.6	61.1
textualism_tool_plain	81.5	72.6	51.7	43.6	74.6
Table 75: 13B models on rhetorical-understanding tasks.
Task	Bloom	Falcon	Incite-Base	Incite-Inst.	Llama-2	MPT	OPT	Vicuna
canada_tax_court_outcomes	0.0	0.0	0.0	5.2	35.4	0.0	28.5	0.0
definition_54.4 classification	80.2	50.1	65.4	50.0	57.5	58.3	42.5
definition Extraction	77.4	77.7	80.6	73.4	84.1	80.9	73.5	4.2
function_of_decision_section	14.2	2.3	10.2	10.6	16.7	13.9	22.3	0.0
legal Reasoning_causality	47.4	59.4	57.6	56.0	53.2	55.5	50.0	38.7
oral Argument question_purpose	13.7	15.5	20.2	27.0	14.5	17.2	14.4	1.2
overruling	82.3	67.9	50.3	75.2	92.2	72.4	53.3	28.7
scalr	18.6	20.7	22.0	23.0	33.7	25.7	19.3	0.0
textualism_48.5 tool_dictionaries	60.9	55.7	59.6	47.4	59.5	51.0	19.8
textualism_50.0 toolplain	57.2	61.6	55.9	50.0	60.5	51.0	4.6
Table 76: 7B models on rhetorical-understanding tasks.
Task	Bloom	Flan	Incite	Opt
canada_tax court outcomes	16.0	62.3	11.7	0.3
definition_classification	51.0	78.5	51.3	57.2
definition Extraction	59.1	77.6	59.4	70.6
function_of_decision_section	9.9	34.8	26.7	14.1
legal_reasoning_causality	50.1	67.5	49.5	55.7
oralArgument questionpurpose	5.3	19.9	21.5	14.3
overruling	63.4	93.6	54.4	54.8
scalr	17.7	64.5	21.5	20.0
textualism_tool_dictionaries	39.7	92.9	55.6	54.8
textualism_tool_plain	51.5	82.2	50.7	55.9
Table 77: 3B models on rhetorical-understanding tasks.
Table 78: Commercial models on interpretation tasks.
Task	GPT-4	GPT-3.5	Claude-1
consumer Contracts_qa	93.6	85.9	90.3
contract_nli_confidentiality_of_agreement	96.3	96.3	92.7
contract_nli_explicit_identification	82.4	81.1	65.0
contract_nli_inclusion_of verbally_conveyed_information	90.7	83.0	83.4
contract_nli_limited_use	86.6	85.4	80.8
contract_nli_no_licensing	92.5	76.7	79.2
contract_nli Notice_on_compelled_disclosure	97.2	97.2	96.5
contract_nliisable_acquisition_of_similar_information	96.1	96.6	93.8
contract_nliisable_copy	80.4	77.7	72.8
contract_nliisable_development_of_similar_information	98.5	99.3	96.3
contract_nliisable_post-agreement Possession	94.6	89.3	92.2
contract_nli_return_of_confidential_information	95.6	92.5	89.4
contract_nli_sharing_with_employees	94.6	94.8	95.9
contract_nli_sharing_with THIRD-parties	93.3	75.0	86.6
contract_nli_survival_of_obligations	94.0	74.5	78.3
contract_qa	96.2	93.6	98.7
cuad_affiliate牌照-licensuree	90.9	90.9	85.9
cuad_affiliate牌照-licensor	92.0	95.5	89.8
cuad_anti-assignment	91.4	89.1	92.4
cuad AUDIT RIGHTS	97.9	89.5	92.7
cuad_cap_on_liability	95.6	94.1	92.9
cuad_change_of_control	88.9	89.7	89.2
cuad_competitive_restriction_exception	84.1	80.0	71.4
cuad_covenant_not_to_sue	95.8	88.0	89.3
cuad-effective_date	92.8	75.0	74.2
cuad_exclusivity	92.9	89.0	87.3
cuadexpiration_date	82.0	87.0	78.8
cuad_governing_law	99.3	98.3	98.7
cuad_insurance	99.2	95.3	94.9
cuad_ip Ownership Assignment	91.7	91.0	89.2
cuad_irrevocable_or_perpetuallicense	97.5	95.4	88.6
cuad_joint_ip Ownership	94.3	91.1	85.4
cuad Licence Granted	94.0	90.3	91.0
cuad_liquidated_damages	96.4	86.4	90.9
cuad_minimum_commitment	89.1	86.1	88.6
cuad_favored_nation	96.9	95.3	96.9
cuad_no-solicit_ofCustomers	100.0	98.8	92.9
cuad_no-solicit_of_employees	100.0	97.9	96.5
cuad_non-compete	93.0	91.0	90.5
cuad_non-disparagement	97.0	95.0	87.0
Table 78 - continued from previous page
Task	GPT-4	GPT-3.5	Claude-1
cuad_non-transferable牌照	90.2	82.1	87.6
cuad Notice period to terminate renewal	95.9	97.7	96.4
cuad post-termination services	94.6	89.0	77.8
cuad_pricerestrictions	95.7	87.0	89.1
cuad_renewal_term	96.1	95.9	95.6
cuad_revenue-profit-sharing	95.3	91.2	88.9
cuad_rofr-rofo-rofn	88.6	81.9	86.2
cuad_source_code_escrow	96.6	91.5	94.1
cuadtermination_for_convenience	96.7	94.2	95.8
cuad THIRDParty_beneficiary	89.7	83.8	82.4
cuad_uncaptured_liability	85.4	70.4	74.1
cuad_unlimited-all-you-can-eat-license	93.8	93.8	87.5
cuad_volume restriction	80.7	68.6	80.1
cuad WARRANTY_duration	77.8	81.2	78.1
insurance_policy Interpretation	69.6	55.0	64.0
jcrew_blocker	100.0	88.9	55.6
maud Ability_to_consummate Concept_issubject_to_mae_carveouts	50.0	50.0	31.5
maud_fiscal_point_of_view_is_the_sole consideration	50.0	38.8	50.0
maud.accuracy_of_fundamental_target_rws Bringdown_STANDARD	29.3	33.3	10.4
maud Accuracy_of_target_general_rwBringdown Timing_answer	63.6	51.0	46.9
maud Accuracy_of_target_capitalization_rw(outstanding_shares)_bringdown_STANDARD_answer	20.7	16.2	13.4
maud additional_MATCHing_right_period_for_modifications_(cor)	57.4	43.3	24.5
maud application_of_buyer-consent Requirement(-negative_inter_covenant)	63.7	68.8	35.3
maud_buyer_consent Requirement_(ordinary_course)	50.0	60.8	28.9
maud_change_in_lawsubject_to_disproportionate_impact_modifier	53.0	48.3	65.2
maud Changes_in_gaap_or_other_accounting_principlesSubject_to_disproportionate_impact_modifier	51.7	47.4	53.8
maud cor permitted_in_response_to_intervening_event	50.0	52.5	50.1
maud cor permitted_with-board_fiduciary_determination_only	21.4	50.0	50.6
maud cor standard(intervening_event)	0.5	36.5	0.0
maud cor standard_(superior Offer)	40.5	45.5	0.0
maud_definition.contains knowledge Requirement___answer	25.0	33.7	20.7
maud_definition Includes_asset Deals	33.3	30.5	0.5
maud_definition Includes_stock Deals	33.3	37.5	27.4
maud fiduciary_exceptionBoard_determination_STANDARD	40.1	27.5	0.0
maud fiduciary_exceptionBoard_determination_trigger(no.shop)	50.0	48.8	50.0
maud_fls(mae)_standard	25.0	44.6	22.5
maud_general_economic_and_financial_conditionssubject_to_disproportionate_impact_modifier	54.2	56.0	53.6
maud includes_consistent_withpast Practice	54.2	55.3	84.2
Table 78 - continued from previous page
Task	GPT-4	GPT-3.5	Claude-1
maud_initializer_MATCHing_rights_period_(cor)	15.4	31.9	20.7
maud_initializer_MATCHing RIGHTS period (ftr)	49.4	32.8	14.0
maud_intervening_event_-required_to_occurs_after_sign\_answer	51.9	51.4	12.5
maud_knowledge-definition	51.1	49.1	38.5
maud_liability_STANDARD_for_no-shop_breach_by_target_non-do_ representatives	44.2	51.9	49.4
maud_ordinary_course_effortsstandard	91.1	70.3	53.6
maud_pandemic_or_other_public_health_event_subject_to_ disproportionate_impact_modifier	48.7	50.0	51.3
maud_pandemic_or_other_public_health_eventSpecific_reference_to_ pandemic-related_governmentalresponses_or_MEasures	79.5	70.9	60.8
maud_Relational_language(mae)_applies_to	57.9	47.2	26.0
maudSpecific_performance	51.5	90.6	73.7
maud_tail_period_length	68.1	39.5	60.4
maud_type_of consideration	99.5	82.7	75.1
opp115_data_retention	67.0	70.5	55.7
opp115_data_security	87.5	84.2	55.6
opp115_do_not_track	99.1	93.6	90.0
opp115_firstParty.collectionition_use	76.7	80.6	63.0
opp115_international_andspecific_audiences	92.3	82.6	79.4
opp115_policy_change	91.9	89.3	83.8
opp115 THIRDParty_sharing.collectionition	80.1	77.0	71.0
opp115_user_access,_edit_and_deletion	90.2	87.7	79.3
opp115_user_choice_control	82.9	79.3	71.3
privacy_policy_entailment	85.5	78.8	89.6
privacy_policy_qa	71.3	65.5	63.0
proa	99.0	90.6	88.5
ssl_company_defendants	65.0	65.3	16.5
sslindividual_defendants	29.6	25.8	11.1
ssl PLAINTIFF	92.0	86.5	86.7
sara_entailment	86.8	68.4	67.6
sara_numeric	8.3	4.2	6.2
supply_chain_disclosure_best practice_accountability	71.5	69.5	74.6
supply_chain_disclosure_best practice Audits	74.4	76.6	75.5
supply Chain disclosure best practice certification	76.6	77.7	77.4
supply Chain disclosure best practice training	83.3	87.1	85.3
supply Chain disclosure best practice verification	68.3	59.4	64.3
supply Chain disclosure disclosed Accountability	77.0	80.4	75.5
supply Chain disclosure disclosed audits	81.6	83.7	80.0
supply Chain disclosure disclosed certification	71.2	67.3	75.8
supply Chain disclosure disclosed training	89.1	83.0	75.6
Table 78 - continued from previous page
Task	GPT-4	GPT-3.5	Claude-1
supply_chain_disclosure_disclosed Verification	56.6	62.0	67.6
unfair_tos	9.1	13.7	5.5
Table 79: 13B models on interpretation tasks.
Task	Flan	LLaMA-2	OPT	Vicuna	WizardLM
consumer Contracts_qa	92.6	68.1	24.8	37.9	67.8
contract_nli_confidentiality_of_agreement	85.4	50.0	54.9	48.8	76.8
contract_nli_explicit_identification	81.4	49.4	51.6	50.0	67.0
contract_nli_inclusion_of verbally_conveyed_information	56.2	50.0	53.0	50.0	58.2
contract_nli_LIMIT_use	71.1	44.5	60.6	49.5	57.1
contract_nli_no_licensing	54.2	53.3	47.0	51.2	58.4
contract_nli Notice_on_compelled_disclosure	90.8	52.1	61.3	55.6	73.2
contract_nliisable_acquirement_of_similar_information	89.3	50.0	45.5	52.2	61.2
contract_nliisable_copy	84.3	47.8	47.8	50.0	56.2
contract_nliisable_development_of_similar_information	98.5	50.0	55.1	54.4	86.0
contract_nliisable_post-agreement Possession	93.4	48.8	44.0	50.0	52.5
contract_nli_return_of_confidential_information	89.4	50.0	44.3	51.3	66.2
contract_nli_sharing_with_employees	92.2	48.2	70.5	50.0	65.7
contract_nli_sharing_with THIRD-parties	86.7	49.5	50.9	50.5	64.4
contract_nli_survival_of_obligations	74.6	50.0	44.8	50.6	59.6
contract_qa	96.3	82.7	56.8	73.1	35.9
cuad_affiliate Licence-licensuree	83.8	58.1	49.0	50.0	65.7
cuad_affiliate licence-licensor	90.9	72.7	53.4	50.0	54.5
cuad_anti-assignment	85.0	52.2	48.1	50.2	76.8
cuad AUDIT RIGHTS	87.1	51.6	71.7	50.7	58.6
cuad_cap_on_liability	81.5	74.0	0.6	50.0	57.5
cuad_change_of_control	75.5	56.2	52.9	49.0	74.8
cuad_competitive_restriction_exception	81.8	51.8	38.2	50.0	50.0
cuad_covenant_not_to_sue	86.0	66.6	49.4	50.0	56.5
cuad_EFFECTive_date	92.8	50.0	50.4	48.7	58.1
cuad_exclusivity	84.0	86.1	61.7	49.6	58.7
cuadexpiration_date	60.5	51.6	53.1	51.0	73.2
cuad_governing_law	99.5	90.6	71.7	59.5	54.3
cuad_insurance	90.2	55.5	66.3	55.0	55.7
cuad_ip Ownership Assignment	89.8	74.0	49.8	49.7	59.9
cuad_irrevocable_or_perpetuallicense	95.7	85.4	67.9	50.0	73.9
cuad_joint_ip Ownership	76.0	53.6	50.5	50.0	60.4
Table 79 - continued from previous page
Task	Flan	LLaMA-2	OPT	Vicuna	WizardLM
cuadlicense Granted	92.8	62.2	50.8	49.9	68.7
cuad_liquidated_damages	73.6	50.9	48.6	49.5	71.8
cuad_minimum_commitment	62.3	54.5	51.6	49.9	47.0
cuad_most_favored_nation	75.0	56.2	43.8	50.0	57.8
cuad_no-solicit_ofCustomers	97.6	57.1	56.0	50.0	85.7
cuad_no-solicit_of_employees	95.8	97.9	69.0	50.0	75.4
cuad_non-compete	91.6	52.7	46.8	49.3	55.2
cuad_non-disparagement	83.0	73.0	67.0	49.0	74.0
cuad_non-transferable LICENSE	69.6	55.0	54.2	49.4	75.6
cuad Notice_period_toTerminate_renewal	92.8	50.0	54.5	50.0	83.3
cuad_post-termination_services	88.7	50.7	51.4	49.6	62.5
cuad_price restricions	76.1	71.7	50.0	47.8	50.0
cuad_renewal_term	89.1	50.5	50.3	57.5	83.9
cuad_revenue-profit_sharing	72.1	58.4	54.5	49.7	52.7
cuad_rofr-rofo-rofn	65.1	50.4	54.1	50.0	59.6
cuad_source_code_escrow	66.1	58.5	67.8	50.0	52.5
cuadtermination_for_convenience	91.4	50.9	49.8	55.8	84.2
cuad THIRDParty_beneficiary	85.3	54.4	63.2	47.1	69.1
cuad_uncaptured_liability	50.3	55.4	46.6	51.0	74.5
cuad_unlimited-all-you-can-eat-license	83.3	52.1	64.6	47.9	54.2
cuad_volume_restriction	55.6	50.0	55.3	50.0	54.0
cuad_warranty_duration	74.7	57.5	55.9	50.6	63.1
insurance_policy_interpretation	44.8	46.6	38.0	13.2	51.9
jcrew_blocker	86.7	66.7	51.1	50.0	8.9
maud Ability_to_consummate_concept_isSubject_to_mae_carveouts	50.0	47.3	51.8	50.0	8.2
maud_financial_point_of_view_is_the_sole considereation	53.6	51.0	50.0	53.1	49.5
maud.accuracy_of_fundamental_target_rws Bringing_down_STANDARD	12.5	33.3	33.3	31.7	33.3
maud Accuracy_of_target_general_rw Bringing_down_timing_answer	46.6	50.0	49.1	49.8	50.0
maud Accuracy_of_target_capitalization_rw(outstanding_shares)_bringdown_STANDARD_answer	6.1	25.8	26.9	20.9	24.8
maud additional_MATCHing_rights_period_for_modifications_(cor)	0.0	25.0	19.8	18.4	12.9
maud application_of_buyer_consentrequirement_(negative_interim_covenant)	63.4	47.8	45.0	3.1	49.4
maud_buyer_consent Requirement_(ordinary_course)	34.4	56.5	44.4	45.1	50.0
maud_change_in_lawsubject_to_disproportionate_impact Modifier	50.0	55.9	46.6	21.3	0.0
Table 79 - continued from previous page
Task	Flan	LLaMA-2	OPT	Vicuna	WizardLM
maud_changes_in_gaap_or_other_accounting_principlessubject_to_disproportionate_impact Modifier	50.0	57.5	47.1	20.6	0.0
maud_cor_permitted_in_response_to_intervening_event	50.0	57.6	58.8	26.8	47.0
maud_cor_permitted_with_board_fiduciary_determination_only	50.0	50.0	51.2	46.7	42.0
maud_cor_STANDARD_(intervening_event)	0.0	24.0	16.7	10.0	23.3
maud_cor_STANDARD_(superior Offer)	11.9	16.8	3.0	0.0	26.9
maud_definition.contains knowledge Requirement__answer	0.0	28.9	25.0	24.5	24.1
maud_definition INCLUDES_asset Deals	2.8	35.4	32.7	0.9	34.2
maud_definition includes stock Deals	4.6	31.8	24.7	30.1	17.3
maud_fiduciary_exception board_determination_STANDARD	1.2	6.5	12.5	0.5	14.1
maud_fiduciary_exception board_determination_trigger_(no.shop)	50.0	59.2	44.7	8.8	48.8
maud_fls_(mae)_standard	17.1	5.0	24.5	25.0	4.6
maud_general_economic_and_financial_conditionssubject_to_disproportionate_impact Modifier	50.0	53.6	52.4	6.0	0.0
maud Includes_consistent_withpast practice	62.3	50.0	52.6	53.2	61.0
maud_init_MATCHing_rights_period_(cor)	0.0	8.3	20.8	0.9	14.2
maud_init_MATCHing_rights_period_(ftr)	0.0	13.3	23.2	19.1	18.8
maud_intervening_event__required_to_occurs_after_signing__answer	39.2	46.2	48.9	47.1	47.1
maud_knowledge-definition	50.6	46.0	48.4	0.0	51.2
maud_liability_STANDARD_for_no-shop_breach_by_target_non-do_representatives	48.7	48.7	49.4	0.6	50.0
maud_ordinary_course_effortsstandard	81.1	57.8	33.7	0.0	67.3
maud_pandemic_or_other_public_health_eventSubject_to_disproportionate_impact Modifier	48.1	52.2	46.2	7.6	31.0
maud_pandemic_or_other_public_health_eventSpecific_reference_to_pandemic-related_governmentalresponses_or_MEasures	50.0	50.0	51.9	50.7	37.4
maud_Relational_language_(mae)_applies_to	51.0	44.2	1.4	51.7	5.1
maudspecific_performance	94.9	52.1	50.0	0.0	56.7
maud_tail_period_length	25.8	51.5	52.5	13.5	34.6
maud_type_of consideration	73.6	39.1	30.2	27.8	27.2
opp115_data_retention	55.7	51.1	45.5	50.0	63.6
opp115_data_security	75.1	49.8	51.2	55.5	59.4
opp115_do_not_track	79.1	50.0	42.7	51.8	88.2
opp115_firstPartyCollection_use	75.0	67.0	68.5	52.3	55.2
opp115_international_andSpecific_audiences	80.1	59.5	18.2	50.4	66.2
opp115_policy_change	70.5	64.9	60.9	52.8	56.5
Table 79 - continued from previous page
Task	Flan	LLaMA-2	OPT	Vicuna	WizardLM
opp115ThirdPartySharingCollection	71.4	54.9	58.8	52.4	60.9
opp115_user_access,_edit_and_deletion	75.4	54.3	60.9	49.1	59.4
opp115_user_choice_control	80.9	53.7	50.0	47.5	58.0
privacy_policy_entailment	58.9	56.2	50.1	0.6	65.9
privacy_policy_qa	52.5	50.5	50.9	0.0	55.6
proa	94.8	76.0	52.1	50.0	80.1
sslacompany_defendants	34.4	63.4	56.6	3.1	2.8
sslindividual_defendants	21.9	21.6	16.2	0.0	0.0
ssl Plaintiff	88.8	26.4	31.6	0.0	0.0
sara_entailment	35.3	58.1	48.9	15.4	50.0
sara_numeric	1.0	0.0	0.0	0.0	0.0
supply_chain_disclosure_bestPracticeaccountability	73.6	49.4	48.8	67.2	52.1
supply Chain disclosure best practice audits	69.2	49.2	66.6	46.5	65.7
supply Chain disclosure best practice certification	69.9	51.6	56.1	67.2	64.8
supply Chain disclosure best practice training	81.0	49.7	49.4	71.5	64.9
supply Chain disclosure best practice verification	70.5	49.1	50.8	51.6	52.5
supply Chain disclosure disclosedaccountability	80.7	49.4	45.8	59.1	58.8
supply Chain disclosure disclosed audits	83.3	49.3	48.9	64.1	64.7
supply Chain disclosure disclosed certification	68.7	49.4	53.8	61.6	59.3
supply Chain disclosure disclosed training	87.0	49.2	48.0	58.1	62.8
supply Chain disclosure disclosed verification	64.3	49.3	48.6	58.0	50.4
unfair_tos	10.0	12.8	10.0	8.3	13.8
Table 81: 3B models on interpretation tasks.
Task	BLOOM	Flan	Incite	Opt
consumer Contracts_qa	0.8	93.2	46.1	31.7
contract_nli_confidentiality_of_agreement	57.3	86.6	56.1	46.3
contract_nli_explicit_identification	57.2	81.2	63.6	53.3
contract_nli_inclusion_of verbally_conveyed_information	50.1	82.2	67.8	60.9
contract_nli_limited_use	63.1	72.8	60.1	63.0
contract_nli_no_licensing	50.0	65.1	43.5	38.5
contract_nli Notice_on_compelled_disclosure	57.0	66.2	65.5	71.8
contract_nli_permissible_acquisition_of_similar_information	59.6	69.1	46.6	54.5
contract_nli_permissible_copy	51.3	83.7	55.3	56.2
contract_nli_permissible_development_of_similar_information	60.3	75.0	49.3	59.6
contract_nli_permissible_post-agreement Possession	43.8	66.3	43.4	42.4
contract_nli_return_of_confidential_information	51.6	82.3	61.3	51.9
contract_nli_sharing_with_employees	53.3	69.7	47.1	52.8
contract_nli_sharing_with THIRD-parties	50.0	80.5	51.4	49.6
contract_nli_survival_of_obligations	50.0	72.9	49.4	44.1
contract_qa	35.9	0.0	82.9	44.6
cuad_affiliate LICENSE-licensed	59.6	66.7	60.1	68.7
cuad_affiliateLICENSE-licensed	48.9	77.3	71.6	50.0
cuad_anti-assignment	50.0	74.7	51.5	33.7
cuad AUDIT RIGHTS	50.0	56.3	56.2	64.6
cuad_cap_on_liability	49.2	50.9	49.4	26.8
cuad_change_of_control	51.4	59.6	60.8	47.4
cuad_competitive_restriction_exception	46.4	73.2	45.5	40.5
cuad_covenant_not_to_sue	50.0	59.1	49.4	44.8
cuad-effective_date	51.7	92.4	47.0	65.7
cuad_exclusivity	64.6	55.2	53.9	60.9
cuadExpiration_date	50.2	67.7	52.1	69.3
cuad_governing_law	56.8	84.4	50.5	74.8
cuadinsurance	53.5	53.9	57.5	70.3
cuad_ip Ownership Assignment	49.0	57.6	57.3	65.1
cuad_irrevocable_or_perpetuallicense	51.8	76.1	53.6	71.1
cuad_joint_ip Ownership	48.4	65.1	64.6	68.8
cuadlicense Granted	50.1	69.1	51.4	69.2
cuad_liquidated_damages	50.0	52.3	57.3	34.5
cuad_minimum_commitment	52.7	54.1	50.6	57.0
cuad_favored_nation	50.0	57.8	56.2	43.8
cuad_no-solicit_ofCustomers	52.4	47.6	59.5	32.1
cuad_no-solicit_of_employees	48.6	63.4	66.9	42.3
cuad_non-compete	48.0	63.1	56.1	30.3
cuad_non-disparagement	49.0	57.0	60.0	44.0