Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?
Authors: Xiaoxi Kang
Lizhen Qu
Lay-Ki Soon
Adnan Trakic
Terry Yue Zhuo
Patrick Charles Emerton
Genevieve Grant
Venue: EMNLP 2023 Findings (2023)
Keywords: Legal Reasoning
IRAC method
Natural Language Processing
Generative Language Models
In-context Learning
Question Decomposition
TL;DR: Evaluation on ChatGPT for legal question answering with IRAC

Abstract: Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?
Xiaoxi Kang', Lizhen Qu Lay-Ki Soon', Adnan Trakic 2
Terry Yue Zhuo?, Patrick Charles Emerton", Genevieve Grant
School of Information Technology. Monash University Malaysia', Monash University", Deakin University"
[xiaoxi banooxi.knnn,.. kang, www..ooooll..oo } @monash. edu
[lizhen.quu qu, adnan. trakic, terry z genevieve. seraeiiieeeeeeeen @monash. p. www.ton@deekin..c...mmm edu. au
Abstract
understanding to what degree CHATGPT is able to perform reasoning for legal scenario analysis in the same way as legal professionals. Herein, we chose IRAC (Alsagoff, 1996), standing for Issue, Rule, Application, and Conclusion, as the framework for legal analysis because it is the most popular anal- ysis methodology used by legal professionals and law schools.
Large Language Models (LLMs), such as CHATGPT, have drawn a lot of attentions re- cently in the legal domain due its emergent ability to tackle a variety of legal tasks. How- ever, it is still unknown if LLMS are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed novel corpus consisting of scenar- ios pertain to Contract Acts Malaysia and Aus- tralian Social Act for Dependent Child. CHAT- GPT is applied to perform analysis on the cor- pus using the IRAC method, which is a frame- work widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC anal- ysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical as- sessment of CHATGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our exper- imental results shed lights on possible future research directions to improve alignments be- tween LLMs and legal experts in terms of legal reasoning.
Aens sovers sumaandin sibes indludinn RM Aenor mihee ortis to say liee the cum of R0Up SOC en the undentaanding that (oby innt mutio any the beleece lon
and c001--000 roving
churs aga aea
womess bet are changes her mind
monieebale
Milbourt incettion
think kint surti lent
rrppymment of hott RuF aor mett not fee this arstnt
Whothor and
change the cerand
tho whole colrt
/Settoo so n RutSection (raettin
andersanningg
coins apainstvooo salance ofarr
Siitiioon [pmenioutg (AMI)) 17 aginee recagi conepedain Bilwe the deetnoeme
top in actacmert det jaroiice) supported coreadorst on Lirter Luail
2018 wwwpooooeeeeeeeeeellllooooooooee ANNutt ansopplahh lncle, as toe teulloo inddeenndeemttin 1I IEE (the uncae stee thei Biok will inl make uny caatee opaensl Feme for the betinnaa the 100100000 cordiaiooieciiii pon SiS tne agroes to secept hours
by terver
curfe considerttion
Eproniso))
butiida (Sociion
roccraann
peeee soo joons coramont (ONL)) cvvosssavo deecbbe shiile
the
Cup dnneddeeeeen
24 soo www.eaadeoeeon.....oooooooo. want unaat servesatee lobut the Nouboume (Aon
Sopprsseenniiii ill saurei aesana can saan seattt
Figure 1: An example IRAC analysis conducted by a legal professional.
1 Introduction
A typical example of IRAC analysis is depicted in Fig. 1. Given scenario regarding the Contract Act Malaysia (CAM). the task is to answer the legal question recognized as an issue in I. In this exam- ple, the rules (R) are the relevant statutes in CAM. In a common law system, rules may also include precedents. The analysis or application (A) con- sists of the reasoming paths leading to the answer in the conclusion (C). Herein, a reasoning path is a sequence of reasoning steps, where each step in- volves taking a statement from the available facts in a given scenario and then applying relevant rule to it. Section 2 describes IRAC in details and its importance in legal reasoning.
Since CHATGPT was released by OpenAl in November 2022, there are fast-growing interests in applying LLMs to analyzing legal documents (Katz et al., 2023) and reasoning tasks (Huang and Chang, 2022). Although the recently released LLMs demonstrate strong abilities to solve chal- lenging tasks requiring reasoning, people find that they often follow different, or even wrong reason- ing paths to obtain correct answers (Paul et al., 2023; Tang et al., 2023). This issue is also referred to as a misalignment problem between LLMs and humans. As this problem has not been investi- gated in the legal domain, this work focuses on
Interpretability of model outputs is crucial for legal professionals in real-world legal applications
"Corresponding author.

---

(Wu al., 2022). For scenario analysis using IRAC, it helpful for models produce individ- ual reasoning paths and their associated rules or precedents so that legal professionals are able to understand why models draw certain conclusions. Moreover, it is noteworthy that legal reasoning is deseasible such that conclusions can be overturned by considering new evidence making different assumptions due to missing information (Sartor, 1995). However, prior legal datasets do not in- clude intermediate reasoning paths understandable by legal professionals and neglect the aspect of defeasible reasoning.
CHATGPT benefits from adding similar ex- ample scenarios with IRAC analysis during in-context learning, only if similar scenarios can be found.
Decomposing legal questions into simpler ones consistently improve the accuracy of identifying legal concepts, such as "invitation treat". However, this approach does not always improve the correctness of produced reasoning paths.
Background
To address the above issues, we constructed the first semi-structured IRAC corpus, coined SIRAC! which includes a set legal scenarios pertinent to CAM and Australian Social ACT (ASA) respec- tively. To make IRAC analysis understandable by both LLMs and legal professionals, we proposed a semi-structured language and used it to codify IRAC analysis for each scenario. As all sce- narios are analyzed by senior law students with IRAC, we conducted detailed comparative stud- ies between their results and the ones produced by CHATGPT, by applying different prompting and in-context learning techniques (Dong et al., 2022). As our scenarios are complex and involve 7.05 reasoning paths on average, we also decom- posed the legal question of scenario into multiple simpler questions and instructed CHATGPT ad- dress them separately. As result, obtained the following novel findings via extensive analysis conducted by the law students:
A legal problem could be simple as ascertaining the amount of money due from tenant to the land- lord under a contract affected by a certain unseen event like COVID-19, 19, or a more complex one in- volving contracts between large corporate entities negotiating cross-border sales and related matters. Legal problems require unique reasoning skills to be solved. These unique skills are applied for solv- ing legal problems in rather systematic manner using the IRAC methodology (Alsagoff, 1996).
Before one begins solve a legal problem using the IRAC's legal reasoning process, essential to acquire factual details from the client. These facts will lead to the correct identification of the legal issue. Given the identified issue, the next step is to determine the law that governs that aspect of the legal problem. The application of the law the analysis in the third stage is perhaps the most detailed stage of IRAC. In this stage, the law is practically applied to the facts and the issues question. As no two legal problems would be iden- tical, one needs to be aware of possible variations. This stage is particularly important because it is here that legal reasoning skills are truly tested. The final stage of IRAC is the conclusion, which is a natural outcome of applying the law to the facts and issues. However, applying the same law the same facts and issues could result in different conclusions, and the correct answer depends on the application of the appropriate law to the facts, which could be interpreted differently.
Without IRAC analysis from legal profession- als, CHATGPT achieves an Fl of 0.49 on aver- age for answering legal questions of scenarios. However, CHATGPT fails to produce com- plete and correct reasoning paths toward an- swers for any evaluated scenario albeit some of the answers are correct.
We demonstrated the importance of providing correct intermediate reasoning paths to CHAT- GPT. The average Fl score of the final an- swers estimated by CHATGPT more than 0.86, when the complete human- written reasoning paths except final answers are fed to the model.
improved
Dataset Construction
We constructed SIRAC to evaluate to what degree LLMs and the other AI models are able conduct IRAC analysis in the same manner as legal pro- fessionals. Our goals are to promote research on devising novel legal AI models that are i) equipped
'Github link: https///githuu..cccccrrrrstaaaae http:/uwbbooocccc/aaaeeeeatteaat

---

with strong defeasible reasoning capability, and ii) interpretable by both legal and computer science professionals. We start with selecting scenarios per- taining to CAM and ASA, followed by conducting full-fledged IRAC analysis on all scenarios using a proposed semi-structured language. The data quality is ensured by our strict procedures.
Rule. We follow the format that is widely used by legal professionals to reference relevant statutes and prior cases. An example depicted in Fig.
Application. Law students were instructed to annotate complete reasoning paths toward conclu- sions. A reasoning path is a sequence of reasoning steps and the last step is the answer to the issue of scenario. In addition, we borrowed the key ideas from default logic (Reiter, 1980) when per- forming defeasible reasoning. It differs from the well-known monotonic logic, such as first-order logic, largely by introducing defaults (assumptions) and default rules. Both defaults and default rules are applied when coping with missing information. For example, if the marriage status is unknown for a twenty-year-old defendant, it is reasonable to assume that he she is single during reasoning. Default reasoning allows changes of intermediate conclusions if the new facts or evidences are in- cluded.
3.1 Selection of Scenarios
We have selected CAM and ASA scenarios in the common-law systems, which are under-explored by legal AI research communities so that LLMs are less likely memorize similar scenarios, relevant statutes and precedents during pre-training. The IRAC analysis of a single legal scenario may take several hours or days, even for law professors and the best law students. At this stage, we do not con- sider scenarios pertaining to more than one areas of law, involving multiple topics and complex legal issues, in order to keep the analysis time within our budget.
As a reasoning path may involve arguments and assumptions articulated in natural language, logical operators, and references to statutes or precedents, we introduce semi-structures language to pre- cisely describe the reasoning paths, so that they are easy-to-interpret by legal professionals and easy-to- process by computer programs. Inspired by neuro- symbolic approaches (Hitzler, 2022), we design the language by using mix of the language used in statutes and symbols for logical operators. We take into account the language used to formulate statutes because it is unambiguous for legal profes- sionals, and the same words were consistently used for the same concepts. The introduction of logical operators, such as AND and OR, is expected to facilitate future research incorporating symbolic or neuro-symbolic reasoning into legal reasoning. In addition, we annotate mentions of legal concepts with brackets because they often play an important role in legal analysis.
We started with selecting a set of statutes in ASA before creating scenarios. In particular, Section 2, Section 3 and Section 4 in ASA are selected for the purpose of this study. Based on the statutes, law students were instructed to write 30 scenarios inspired by real-world cases so that i) each sce- nario is designed to answer legal question "Is the child dependent child"; ii) all relevant legal rules are covered by the selected sections. As result, LLMs will be given complete information to perform IRAC analysis.
For the scenarios pertinent CAM, we in- creased the complexity of IRAC analysis by con- sidering scenarios with more diverse legal issues, requiring the application of both statutes and prece- dents decisions made in previous similar cases. Herein, 20 scenarios were selected from tutorials, text books and examination questions related to CAM. Each scenario is associated with one topic and there are 10 topics in total. Each scenario per- tains to a different yes-no question as the issue. For example, "Whether Bob and Jane can change their mind and demand repayment of the whole debt?", as shown in Fig. 1.
Conclusion. We apply the same semi-structured language to formulate conclusions in the corre- sponding section of the IRAC structure. For each scenario, cach conclusion is essentially the answer to the corresponding issue.
3.2 Annotation of IRAC Analysis
To facilitate the manual annotation work, we build an online annotation tool A.I for law stu- dents, which supports annotations of legal concepts, rules, logical operators, and analysis using the semi- structured language in the IRAC framework.
Issue. As issues have already been determined for creating the scenarios, law students focused on analyzing relevant statutes and precedents, in order to obtain correct conclusions to the issues.

---

3.3 Quality Assurance
average reasoning path is 9.3 because the scenario more complex compare to ASA.
To ensure the quality of the scenarios and annota- tions, we applied a strict procedure in selecting the annotators, as well as checking their progress. Annotators Selection and Training. We recruited five senior law students from different universities after interviewing all the candidates. The recruited students were expected to be familiar with ASA and CAM. Before they joined the annotation project, we provided test cases to them, in order to get them familiar with the semi-structured language and the annotation tool. After they submitted the test cases, we asked the law professors to verify their answers before they started the actual annotation work. Scenario Quality. To ensure the quality, the sce- narios of CAM were selected from reliable sources, such as law textbooks and examination questions. Each scenario related to ASA was checked by an- other student besides the one who created it. All scenarios were double-checked by one expert in the area to ensure that they are reasonable and meet all criteria mentioned above. In case of disagree- ments on the criteria of the scenarios, the expert discussed with all the law students and revised the corresponding guidelines, if necessary, ensure consistency.
Scenarios
Issues
Rules
Ave Length
STRAC_ _SA
30
3
4.8
SIRAC CAM
20
55
9.3
SIRAC
50
21
58
7.05
Table 1: Basic statisties of SIRAC
Comparison with Related Datasets. We compare SIRAC with the current datasets for legal QA tasks along six dimensions: i) if the tasks require legal reasoning; ii) if the data is
by ma- chines; iii) if the reasoning paths are intepretable by legal professionals; iv) if IRAC analysis is ap- plied; v) if there are complete and detailed reason- ing paths annotated; vi) if the reasoning requires both statutes and prior cases. As summarized in Ta- ble 2, LEGALBENCH (Guha et al., 2022) is the only one applying the IRAC methodology. However, they do not perform full IRAC analysis scenar- ios. Hence, the corresponding reasoning paths are incomplete. The length of the annotated paths are also fairly short, which contain less than three steps. The other similar dataset is SARA (Holzenberger and Van Durme, 2021), which contains both ques- tion answering tasks and reasoning paths. However, they employed Prolog to codify reasoning paths, which is rather challenging for legal professionals to understand.
IRAC Analysis Quality. We created detailed guidelines for performing and annotating IRAC analysis, as well as the usage of the annotation tool. We had regular meetings with the law students to check on their progress and discuss any potential problems. The solved problems were incremen- tally reflected in the guidelines. In addition, we regularly assigned the same scenarios to different law students to let them work independently, such that we can calculate inter-annotator agreement (IAA) and use the TAA scores to check the consis- tency of their annotations. Moreover, the answers to the issues of the CAM scenarios were checked against the sample answers provided by the text- books, examination questions and tutorials, so that the correctness is guaranteed.
Assessment of ChatGPT
Our preliminary results show that CHATGPT is one of the best performing LLMs in legal reason- ing. Therefore, it was chosen to assess its ability in performing IRAC analysis. Instead of evaluat- ing only the final results of statutory reasoning by GPT3 (Blair-Stance et al., 2023a), we employed CHATGPT to perform full IRAC analysis, and the quality of its outputs was assessed by the law stu- dents in the respective sections: Rule, Application, and Conclusion. Issues were not evaluated because they are part of the inputs. Moreover, we also carried out in-context learning and question decom- position, which have been found useful in other reasoning tasks (Mialon et al., 2023), to improve the performance of CHATGPT.
3.4 Data Statistics
Basic Statistics. We have 50 annotated legal scenar- ios in total. Table 1 summarizes the statistics of the dataset. For CAM and ASA, the rules include ref- erences to precedents. In the reasoning paths, each reasoning steps is the statement from the facts of the scenario and applied with the rule. The average length of the reasoning paths is 7.05. For CAM the
4.1 Evaluation Measures
In this work, we focus on understanding the pros and cons of CHATGPT on this task, as well as

---

Daturet
Reasoning
Friendly for systems
Pricndly for legul profesionals?
IRAC applied
reasoning paths
Detailed
Analy requiring
precedents
SARA
Yes
Yes
No
No
No.
No
(Holzesherger Nan Durne, 2021)
COLIBE
Yes
Yes
No
No
No
No
(Rabelo al., 2022)
CUAD
(Hendrycks et 2023)
No
Yes
No
No
No
No
No
Mayhe
No
(Wang et ol,, 3023)
TEGAL
Yes
Maybe
No
BENCH (Guha all, 2022)
Semi-straciuned
Yes
Yis
IRAC.
Table 2: Comparison of the datasets for legal QA and legal scenario analysis.
its alignment to the reasoning behavior of hu- mans. Therefore, the human evaluation measures are based the marking rubries used by law schools, and the ones for evaluating language mod- els. Throughout all experiments of human evalua- tion, we formulate the majority the measures as a statement and score each them with either -1, or 1, indicating disagree, neutral, and agree respec- tively. The statements are formulated in the way that the higher scores the better. All measures are assessed first for each selected scenario and their results summarized for later analysis. Appendix A.3.2 shows the questions we asked the annotators.
General Comment. We also ask annotators to comment the reasoning performance of CHAT- GPT, in particular its strengths and weaknesses.
Conclusion.
Correciness is commonly evalu- ated feature (Hämäläinen and Alnajjar, 2021). Hu- mans will evaluate the answer to identify whether it addresses the question and whether it consistent with the source material.
For all IRAC analysis, we also evaluate the flu- ency of the texts generated by CHATGPT, in order to assess if they are understandable by humans.
To evaluate the quality of the assessment results, we reassigned 18 finished scenarios three differ- ent annotators and asked them to evaluate the sce- narios in terms the above measures. We choose Cohen's 's Kappa score (Cohen, 1960; Zhann et al., 2023) to compute the agreement. The Kappa score for all the evaluation measures is 0.55, while the Kappa score for the analysis without the assump- tion evaluation is 0.75. Our investigation shows that the questions for the assumptions are subjec- tive for law students, which however is common problem in law education, as confirmed by a lec- turer teaching law for several years.
The rules and precedents are evaluated with regard information relevance. The cor- responding statement is formulated "The ref- erenced statutes and precedents are correct and complete.".
Rule.
Application. We apply multiple measures to as- sess the legal analysis in this section.
Correctness of Concept Identification. This mea- sures whether all important legal concepts correctly identified, and is formulated as "All key concepts are correctly identified."
Articulation of Reasoning. This is concerns with articulation of reasoning steps and if the logical structures between steps are correct. The corre- sponding statement "CHATGPT performs and articulates reasoning correctly and consistently." Evaluation of Assumptions. As legal reasoning is defeasible, we evaluate the generated assumptions by using the following questions:
4.2 Results and Discussions
We started with evaluating the final answers gen- erated by CHATGPT, followed by analyzing its reasoning steps and references.
Evaluation of Conclusion. We fed a scenario and its issue as the input to CHATGPT. As the questions in all scenarios are all yes-no questions, we appended "Answer in one word." to the last line of the inputs so that could automatically extract the answers from CHATGPT. An example input is depicted in Fig. 2. In addition to the scenarios pertaining to ASA and CAM, we included the 276 scenarios pertinent to the Internal Revenue Code (IRC) from the SARA dataset, where IRC is the
How many assumptions made?
Among the assumptions made by CHATGPT, how many assumptions are correct?
Compare the assumptions from CHATGPT and humans, how many of them are matched?

---

domestic portion of federal tax law in the United States.
As sonnelletiiniing coito, sclad Rid 10,030 ovirg Buds and Rir R 500D uoveg bo.kend
Astdt hatine has to pay tuts muss of (IM 240 the in dhcorandd mg Rudt thee in fir maie any Sibvegres lako deeges
daaro upe soeel for the ludoree the fborl
und
indinaan Cap anaiiooon dolooeent deai Lake Jene fanko tie the cinege
anfiicnnt and wornts
sepugriset innt of hual RiS (40 vkck vubilto be Jic lo
Ansa severa outstanding dcos, including, RM 16,000 oving Bob Rv 200 June. 1IM
offers
Bob the
the understandng that Bob
Shhehhe deo and Anne can change their mindd and deriand te ceverenn of the wit delec?
Cucption:
clants saparial de fox iodanss of Ra 10,000 dusl But ugran iut kaies cl ungate minal and roims the ha oice ohe nee nateeding denn Jane- agrens in accept the sinneemmin ricket to tie bouane Cup in outher of dee oibe Lake Juew for lhai the lickl how rotl vollic and and oxmn inpoy ment hatt the HAt son codt veltton testes
Chaltall Aroove
An twayee, woooll wisir ths thit cast comes veey for wille to wuger fan's ri and Siovee enpagtment of the whol dete, segwniding on the speciti ircurnete tan the in tieets with Arres and the cad lo live Mu syyia
in cileecane of Tidd, sitte appeeenn lartwers is and Aeoon rooeeri iie ous legelly strdding than BoD but
Weither Bobvend ane can duange thet mind anddenend reporment of the ofhele cietat Aenoow Die
boune by apreoment i ondimer not be ablect domand repoymeett the
sktt owell by fer Honeesoy the veroement was ment unethvr or if lilare
com monkenttens yow thene andd wnee lo Bott auvoaners, in theur moreent meyyreet live ahiic change mind and domand repayment of the full awedd
Oncemned uneee)
arroresstic and Bvb Arna
care
acceper numuartiis tolest hie hourn cup cortonment onehe
Figure 2: An example prompt for answer evaluation.
not be conaldored tapalty in dinnng
fimay considened lapelty
soca foom paymenen ihto cone, tone ab doer and repoymant of the
owed Kono, Hlowoees cane zareed to accapn the excepemioo tislctn t. corirtattion
alto cionaee anki koso their riie dedood
iik dee
of the
di
of whickl
Precision
Recall
F1
IRC
0.29
0.43
0.35
con dss
infoom
Ganoomecs Corazgt
Paticaao of
3000
1001000
correct 96A010000
ASA
0.75
0.60
0.67
CAM
0,50
0.40
0.44
Average
0.51
0.48
0.49
Commont It sttntiflss and of the boy concoptz which cor sidovation, wil 1.55 not loordity thrsstars is for the socons part of the amoooo Docplt timillying or totttaaai as adansee lapel canropp, doss dieases mushiaed shace tul supty i sconaroo
Table 3: Result of answers produced by CHATGPT.
Figure 3: An example reasoning path.
As shown in Table 3, CHATGPT's performance varied across these domains, with the highest pre- cision in the ASA (0.75), and the lowest in IRC (0.29). We also compared F1 score is lowest on the IRC which is 0.35 and highest 0.67 one ASA. The average Fl score of CHATGPT 0.49. Which we can see there are still a lot of room for improvement in the following steps.
or precedents in the involved reasoning paths. Out of 40 examples, there is only one example from AC-ACCA having the analysis with correct ref- erences statutes and precedents. The perfor- mance is also poor on the articulation of reasoning. We notice that the formulation of the analysis from CHATGPT is sometimes very confusing and logi- cally inconsistent.
Evaluation of Application and Rule. Apart from conclusions, we evaluated the ability of CHAT- GPT by letting it produce intermediate reasoning paths. It can be achieved by removing "Answer in one word." " from the last line. As SARA does not contain human annotated reasoning paths, we evaluated the sections of Application only on the scenarios of CAM and ASA.
The evaluation results of assumptions are listed in Table 4. We evaluated the result into two dif- ferent perspectives: assumption and analysis. The assumption measures if the answer from CHAT- GPT makes reasonable assumptions. For example: if my son stay at my home whole day and doing nothing. Then we can assume that my son is wholly dependent on me. However, we also noticed that, although some of the assumptions were identified by CHATGPT, they were not analysed correctly. For example: CHATGPT answered "my son is not wholly dependent on me", which is a wrong as- sumption. We compared the result from different law and different methods. The result indicate that, after using the in-context learning and decomposed questions, more assumptions were identified and discussed correctly. Nonetheless, the improvement of analysis still lesser than the assumptions.
We selected 20 scenarios per area for human evaluation. Among the 40 scenarios we evaluated, only two of them produced high quality reasoning paths, which are considered as agreed in the ques- tion about articulation of reasoning. Further details can be found in the example depicted in Fig 3.
Table 5 summarizes CHATGPT's performance in CAM and ASA in terms of all evaluation mea- sures, including fluency, information relevance, ar- ticulation of reasoning. From the results we can see that CHATGPT receives much less agree (1) than that evaluated on the answers. In contrast, annotators agree that the fluency of CHATGPT out- puts is high, in line with the observations from re- lated work. However, CHATGPT does not provide enough information such as references to statutes
Overall, although CHATGPT could produce cor- rect conclusions in IRAC analysis, its analysis part in Application mostly are not aligned with those from legal professionals. The references to law and

---

precedents are often missing or incorrect.
PLAA
F1,0UM
1.23
Assamptions
Anelysi
Precsina
Becall
F Scure
Preciioo
Recall
FT Seorc
Overall
0.45
0.68
11.54
0.24
0.43
0.34
CAM
0.56
0.56
0:61
0.35
0.41
0.36
0.73
0.3
In-conlest Overnil
0.90
In-cuteest
0.95
0.32
Incowrlett ASA
0.94
0.97
096
0.85
0.88
0,7
0.5
Decompestion Uveeaal
0.75
0.33
0.81
0..3
0.57
0.52
Decomposition CAM
0.58
0.50
0.89
0.49
0.50
049
Decompottiion ASA
0.57
0.58
033
15% 30% 755 100%
Table 4: Performance of CHATGPT in terms of Preci- sion, Recall, and F1 Score.
Figure 4: Result of the add reasoning paths.
Exompee Queddoo
(Dutoe going overneau months Michadi, who polics ofticer luspeemed
Impact of Adding Reasoning Paths. LLMs are
Debble promioed Michoel RRS 000
leter resuming ovessass Detgee
known to rely on "shortcuts" for predictions occa- sionally (Du et al., 2022), hence their robustness is questionable. In this experiment, we verify CHATGPT's performance can be improved by pro- gressively providing it with human-authored parts of reason paths, in order to shed light on future research directions of legal reasoning. Herein, add 20%, 40%, and 80% of the reasoning paths notated by law students to the inputs after the legal question. The final outcome has been progressively improved. Figure shows that the more analysis we provided in a prompt, the higher Fl scores we gained from the final answers. The F1-score is able to reach 0.89 /1.0 starting from the lowest 0.10/0.0 for CAM/ASA respectively. This observation sug- gests that LLMs, e.g. CHATGPT, should focus on generating correct intermediate steps. We also found out that this observation consistent both areas of law that we evaluated.
realiees that she doesin need to soy anything to Mocoa beciuse Michaells just doing duty pobco cll)
Debdenn premnen te supporked by comashe comaaheaiions?))
Eoarppe annee:
[(1) Oadrsefpcnni ceeg now done on desine (Deteidgraraaar Boar THEN theme conditectioon pon---olpppppiiiio mardaaa (CL ASLAROOIKK BRO THE UNIEED AYPELLANITC,
IS RAAORRGGN COUNTY COUNCE AND a OTEERS RESPONDDNTS (1925) A.C. 37pp171)
22) (HOWEXERJF It keep an eye on her bouse whin sheil ceneprooniee recended the hound
(Mll hes (poorrooooonn loodga spolco zffoorppoble subg 1HEN performarzs kees zyo
UNITED FOOTBAL CL LTD [1968] C2 77310)
(4) (Jichoel)) comisee) s keep an eye fer 1o3se while she proniee amount
acccul services HEN e'w aamount vulld coraccraaoon. dhiediis a BHEERiILL UNITEL rOCTBALL
jrassji..s 17544)
Deco
woyld supporied by consideations
ecoepplee
folloiing cuaaton
Figure 5: An example of adding Examples to Prompts.
statistics of our data, we can tell that all scenarios are related to one topic and two sections pertinent to ASA. The followings are the main findings ac- cording to our human evaluation:
The analysis parts are improved in 50% of the given scenarios after adding examples.
Effects of Adding Examples to Prompts. To understand the performance of CHATGPT using in-context learning, we added the most similar ex- ample terms of topics or related statutes to the prompt for each scenario. Fig gives an example of in-context learning.
The analysis parts are improved for the scenar- ios that are similar to the provided examples.
Table 4 displays the Fl score, changing from 0.34 to 0.66 on the analysis part. The ASA is able cover more legal points and F1 scores are improved up 0.96. Referring the example provided, CHATGPT able to answer similar the one shown in the example question.
We evaluated the same 40 scenarios as before. The quality of the reasoning paths is improved by 27.5% especially for ASA, because the scenarios in this area are similar to each other. From the
Effects of Decomposed Questions. If CHAT- GPT is unable to capture the main legal concepts
Infommstion relevance
Couzpii Identificttion
Asticnlatin Reusoning
Correctness
Fluency
20
9
13
Comestace
Eagrosmani
Conaapt
ASA
12
11
7
Reasoring
11
0
-1
CAM
0
10
0
11
10
CLM
0
-1
16
17
21
17
Total
22
17
Tual
15
Table 5: Result for reasoning paths.
Table 6: Result for adding examples to prompts.

---

Carneinas
Improemen Flunny
Inforrekkee relcoree
Caregp lnviiivaio
Amirahiiin Reaverno
5 Related Work
10
20
16
ASA
10
20
In this section, we present related work, highlight- ing the similarities as well the differences be- tween theirs and ours. Previous work related to our paper can be categorized into the following subsections.
10
18
CAM
2
30
14
1401
14
Table 7: Result for decomposed questions.
5.1 LegalQA with Reasoning Task
There are several research related to legal reason- ing. Nils (Holzenberger and Van Durme, 2021) presented the reasoning step with Prolog (De Raedt et al., 2007). Their research focused on identifying the rules applicable in the scenarios. They also attempted to do the reasoning by rewriting the rule with Prolog (De Raedt et al., 2007). However, it is not applicable to other law and Prolog is not easily comprehensible by legal professionals. (Yu et al., 2022) proved that the best accuracy can be obtained by applying the IRAC methodology. Their work did not evaluate the reasoning path, which is the analysis part of IRAC. The accuracy was only based on the conclusion, which is not compre- hensive evaluation of IRAC methodology. Guha (Guha et al., 2022) separated IRAC to four individ- ual tasks. They pplied different data corpus and fit into IRAC analysis. However, they did not apply the complete IRAC on a single case. In compari- son to all these related works, our work presented in this paper covers the complete IRAC method- ology. More importantly, the reasoning traces are presented in a comprchensible method, for both legal and IT professionals.
for the reasoning analysis, we investigated how it can be improved by decomposing a question into smaller questions? Inspired by Chain-Of- Thouuhtss (Wei ct al., 2022), we attempted to dissect the is- sue questions into smaller questions and therefore guided CHATGPT towards more accurate answers. Fig 6 shows examples of decomposed questions. The list of all decomposed questions can be referred in appendix 1. Our annotators decomposed the issue questions based on the legal concepts that need to be mentioned in the reasoning paths.
From the sample results, we can observe that with decomposed questions. CHAIGPT is able to apply the analysis to the facts identified from the scenario, followed by matching and aligning them with the legal concepts.
Decaampracd Qucationa
the promicc spportted by cono d
in supprteed fed tu conprtenttoon then othelter the consicienn are wil??
volil tham witede frey wribh hearndl the parmiise
54 thyy aas loandd by thes pron lase will Rob and change thik minal deeand lopaymmnt of the whole deat?
CharGet Answer:
promiia Rub and derul
the rilrose Amni's dedii suppertcal
consateralion. (Donact Answe))
lill wappe by
odadiee the Iecanmoiiens wiks?
prodiiin of
the consilication
payterl
5.2 Reviewing CHATGPT Performance
they
Given the proliferation of CHATGPT since Novem- ber 2022, a lot of research works were carried out to review the performance of the CHATGPT in spe- cific domains (Zhuo et al., 2023). Tiffany (Kung et al., 2022) found that CHAIGPT was able to pass all the three examinations without any spe- cialized training or reinforcement. In addition. CHATGPT provided a high level of concordance and insight in its explanations. Their work paves the way for domain-speettn GPT. Andrew (Blair-Stanek et al., 2023b) and Savel (Savelka, 2023) reviewed CHATGPT perfor- mance based different input prompts. From their results, GPT-3 had 78% accuracy, raising doubts about GPT-3's ability to handle basic legal work. Similar to (Guha et al., 2022), there is no evaluation on the reasoning traces. Although it was mentioned
would bound by promise made by feri, agreed eccest paymentin
rimaining ring as he accepted the sweepstate tokeet as settement the dett Anno (Conect Answery thiyy and found by loriii, will Rob and change their mind and demend resemmentt ofthic whole dcar?
seohange reloasing
would bound promice,
ourn herge his minclard and dinnoral rnpegmmen of fflerternnii iingdei deitl, his sineedy aguuza to aoceep the paymont
Annu coec ange macssing gg Jonc connor oomond repoymenn of the oc21 acatton sattppstaae beace, as ne dewety aucrepted tw bud berbert foe Jennam hos by the promose made by Armn (Conrec deswer))
remaninng
toyeelly Aurm Thenwtrre, both loot
Figure 6: Example of Decomposed questions.
Table 7 shows the result of the decomposed ques- tions. We can see improvement from previous an- swers. From Table 4, the overall legal concept identification maintains high performance with a precision of 0.75, recall of 0.88, and an F1-score of 0.81. The results prove that decomposed ques- tions help in identifying legal concepts related to the scenario more effectively.
evaluations on CHAT-

---

that CHATGPT performs poorly, no specific mea- sures were used to support the claim objectively.
7 Limitations
While SIRAC offers a comprehensive resource for IRAC analysis, both by legal professionals and machine learning models, we need to acknowledge the existing constraints.
5.3 Alignment Problems between Large Language Models (LLMs) and Humans
Large language models (LLMs) have achieved suc- cess at a range of tasks such as question answering, summarisation, and dialogue. Hannah (Kirk et al.. 2023) proposed three-tiered policy framework that allows users to experience the benefits of per- sonalised alignment. Similarly, Jakob (Mökander al., 2023) also used a three-layered approach for the LLMs which include governance audits, model audits, and application audits. Atoosa (Kasirzadeh and Gabriel, 2023) developed philosophical anal- ysis of building blocks of linguistic communica- tion between conversational agents and human in- terlocutors. Bakker (Bakker et al., 2022) studied LLMs alignment from different perspectives, their work highlights that there is still gap between the alignment of LLMs and humans. However, there is a need strike the balance of providing more information to LLMs and not burdening the human experts involved unnecessarily. This observation inspires us to investigate the right amount of in- formation LLMs require from humans in order to achieve satisfactory performance.
Lack of legal domain It is challenging to en- gage legal professionals who can understand the law of different countries. Hence, there is limit to the extent of analysis that could be performed on some published legal dataset. At this stage, STR AC is limited to only two types of law ASA and CAM. However, the methodology proposed in this paper is applicable to different laws. ACRAC covers two different laws from different countries. In the future, we plan to engage more legal profes- sional who can contribute to expanding the dataset to other types of law.
Lack of data resources for CHATGPT revision due to the limited resources, we were able to en- gage a small group of annotators to assist us in eval- uating the outcome produced by CHATGPT. While the analysis is sufficient for us to obtain concrete insights, we hope we to engage more annotators for further strengthen our research contribution.
8 Ethics
All the tasks carried out in this paper aim to assess the reasoning traces for the legal scenario analysis by CHATGPT. The entities mentioned the legal scenarios are anonymized by the annotators. The court cases include in our dataset does not reveal the real case. We only include the court case name and responding paragraph numbers. In fact, court cases are accessible by the public and often used for further analysis by the commentators for law studies. As such, the court cases used do not require additional ethics approval.
6 Conclusion
We constructed a novel dataset, coined SIRAC, to evaluate the ability of LLMs for IRAC analy- sis. SIRAC contains 50 legal scenarios pertinent to ASA and CAM. Every scenario was annotated with reasoning paths with an average length of 7.05, described by our proposed semi-structured language. SIR AC is not only useful for the studies and analysis by legal professionals, but offers a fertile pool of resources for legal NLP.
Acknowledgements
Our evaluation results on CHATGPT show that the powerful LLMS can produce reasonable an- swers but mostly fail to yield correct reasoning paths aligned with legal experts. Its performance can be further improved by providing parts of the annotated reasoning paths, including similar anno- tated scenarios for in-context learning and decom- posing complex issues into simpler questions. As those techniques are not specific to the investigated areas of law, it desirable to understand the extend to which such empirical findings still hold in the other legal areas in the future work.
This research made possible through the gener- ous support of Monash Inter-Faculty Seeding Grant. We wish to extend our sincere gratitude to all the dedicated annotators who have made invaluable contributions to this project. We would like to ex- tend special thanks to Dr. Sia Chin Chin from Taylor's University, whose provision of the exami- nation questions as a reliable source for the legal scenarios, along with assistance in recruiting quali- fied annotators, has been instrumental in advancing our research endeavors.

---

References
Atoosa Kasirzadeh and lason Gabriel. 2023. In conver- sation with artificial intelligence: aligning language models with human values. Philosophy & Technol- ogy, 36(2):1-24.
Ahmad SA Alsagoff. 1996. Principles of the Law of Contract in Malaysia. LexisNexis.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell-Gillnghham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, al. 2022. Finc-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Infor- mation Processing Systems, 01238876-38888
Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael J Bommarito II. 2023. Nat- ural language processing in the legal domain. arXiv preprint a302. 12039.
Hannah Rose Kirk. Bertie Vidgen, Paul Röttger, and Scott A Hale. 2023. Persomalisation within bounds: A risk taxonomy and policy framework for the align- ment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453
Andrew Blair-Stanck, Nils Holzenberger, and Benjamin Van Durme. 2023a. Can gpt-3 perform statutory reasoning? arXiv preprint urXiv:2302.0610 060000000000.
Tiffany Kung. Morgan Cheatham, ChatGPT. Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga. Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. 2022. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. medRxiv, page 2022. 1022.12.19.221 12..2.2.9..22433.
Andrew Blair-Stanck, Nils Holzenberger, and Benjamin Van Durme. 2023b. Can gpt-3 perform statutory reasoning? arXiv preprint wrXii:2302.06110
Jacob Cohen. 1960. coefficient agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-44
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, models: a survey. arXiv preprint arXiv:2302.07844
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. 2007. Problog: A probabilistic prolog and its appli- cation in link discovery. IJCAI, volume pages Hyderabad.
2023. Augmented language
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.000
Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. 2023, Auditing large language models: three-layered approach. Al and Ethics, pages 1-31.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2022. Shortcut learning of large language models in natural language understanding; A survey. arXiv preprint 2208 1....7188 17857.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.0900.
Neel Guha, Daniel E Ho. Julian Nyarko, and Christo- pher Ré. 2022. Legalbench: Prototyping collabo- rative benchmark for legal reasoning. arXiv preprint arXiv:2209.06 720.
Juliano Rabelo, Randy Goebel, Yo-ng Kim, Yoshi- nobu Kano, Masaharu Yoshioka, and Ken Satoh. 2022. Overview and discussion of the competition on legal information extraction/entailment (coliee) 2021. The Review of Socionetwork Strategies, 16(1):111- 133.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268.
Raymond Reiter. 1980. A logic for default reasoning. Artificial intelligence, 13(1-2):81-132.
Pascal Hitzler. 2022. Neuro-symbolic artificial intelli- gence: The state of the art.
Giovanni Sartor. 1995. Defeasibility in legal reasoning. Springer.
Nils Holzenberger and Benjamin Van Durme. 2021. Factoring statutory reasoning as language understand- ing challenges. arXiv preprint arXiv:2105.07900
Jaromir Savelka. 2023. Unlocking practical applica- tions legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. arXiv preprint arXiv:2305.04417.
Jie Huang and Kevin Chen-Chuan Chang. 2022. To- wards reasoning in large language models: A survey, arXiv preprint arXiv:2212.10403:
Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large language models can be lazy leam- ers: Analyze shorteuts in in-context learning. arXiv preprimi arXiv:2305.1725:
Mika Hämäläinen and Khalid Alnajjar. 2021. Human evaluation of creative nig systems: interdisci- plinary survey on recent papers.

---

Steven H Wang, Antoine Scardigli, Leonard Tang. Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. 2023. Maud: An expert-annotated le- gal nlp dataset for merger agreement understanding. arXiv preprint arXiv:2301.005:
A
Appendix
A.1
Data Annotation Guidelines
We conduct the data annotation workshops with the annotators a weekly basis meeting, The annota- tors trained with the expert. We also devleoped data annotation platform for the annotators. We provided training for the
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, 2022. Chain thought prompting elicits reasoning in large language models. arXiv preprint arXtv:2201.11903
Yiquan Wu. Yifei Liu, Weiming Lu. Yating Zhang, Feng. Changlong Sun, Fei Wu, and Kun Kuang. 2022. Towards interactivity and interpretability: A rationale-based legal judgment prediction framework. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 4787-4799.
Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Le- gal prompting: Teaching language model think like lawyer. arXiv preprint arXiv:2212.01326.
Haolan Zhan, Zhuang Li, Yufci Wang, Linhao Luo, Tao Feng. Xiaoxi Kang. Yuncheng Hua, Lizhen Qu, Lay-
Ki Soon, Suraj Sharma, Ingrid Zukerman, Zhaleh Semnani-Azaa and Gholamreza Haffari. 2023. So- cialdial: A benchmark for socially- aware dialogue systems.
Terry Yue Zhuo Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Red teaming chatgpt via jail- breaking: Bias, robustness, reliability and toxicity.

---

1. Introduction
This data annotation is working the contract law-related scenario. The general task will include legal concept identification the given scenario use IRAC(Issues, Rules, Analysis, Conclusion) to process the soenarios.
The second part the annotation the evaluation the chatGPT's answer the same input legal scenarios.
2. Scenario
are expecting to legal scenarios related to contract and Australia social act.
The of scenario is from questions, tutorial questions and books.
The annotator is expecting to highlight the scenario with the legal concept. You can choose the legal concept in given list add legal concept you may think is useful in this scenario.
3. IRAC analysis
IRAC analysis, there an IRAC form you out. The following are details you need to in.
Issue: Questions and problems for the given scenario. Suggest starting the question with whether.
Section related: select sections related to the given issue Related court case: Enter the related court case for the given issue
Analysis: It free text You can drag the predefined relations such IF... THEN...... and the legal concept you highlighted the scenario write the analysis in point form. To
avoid ambiguity and misunderstanding of the syntax. It suggested to use standard legal words all the time.
Conclusion: Answer the given issues.

---

4. Annotation Platform Example
Salhoaal
Example annotation
CHANGE MODE ANNOTATION
Vanensa a vinyl records store ounner. She placed an advrtisement on social media platform selingg a limited viny! for
$500, On Monday, receives
Niko, that
Houwever, told Vanessa
unable make payment moment facing financial diffcuuties then requested Vanessa keep the vinyl for him
more days. Vanessa repiled saying that "Fine. will reserve for
you
Tuesdda, another customer, Ken, called purchase that whyl. offers $700 to purchase which Vanessa accapts. Vaneesaa then immediate)) call hiko inform him that the viny sold someone eise but phone unreacheble
message saying
out Niko went out of town and his phone was out of service. When he came back to collect the vinyl on Wednesday at find that
threatins
of contract Advise
Veneess her liabiity, if aryy Niko Kees
IRAC Analysis
Issues
Was there a contract between Vanessa Niko?
Decompostion Questions
advertieemet put by Vinesse en invitation to treet offer?
Was there acceptance on the part of Vanessa? vinyf Wednesday
Bpm?
Select Related sections
selected
Update Section
Court cases
couricase
Related court
Page number
#1
Eckhardt Marine GMBH v Sheritt
3/4
Preston Corp Sdn Bhdd v Edward
2/4
Affin Bonk Bhd Mohd Kasim lbc
125/17
Highlight Texts and Tags
Related Sections and Courtcase

---

Andlsis
The advertieement placed by Vanessa sell limited edition viry! for $500 Invitation treat (Eckhardt Marine VMBH v Sheriff, High Court Malaya, Seremban Ors (2001) MU (CAJIBAIILGGALIZZIIOON.
2. IF Vanessa's is advertisement invitation to treat, then jshe receives a call from customer, Niko, to reserve that vinyA. Zej/Preston Corp Son
(Niko's reply invitation treat])
Leong [1962]
IF (Fine, will reserve the vinyl for you until Wednesday 6pm. If don't hear from you by then, I will seil the vinyl for
Concuusion
breached selling vinyl party
Export File
EXPORT
5. ChatGPT answer analysis
Annotation Guidelines
evaluation conducted the experiment. following are details when go through the result of chatGPT's outcome. The annotators used the following scoring for the factors below: -1 means disagree, means neutral and means agree for the annotators rating factors below.
1. Correctness is commonly evaluated feature \cite(Mika2021), Humans will evaluate identify whether addresses question and whether consistent with source material.
First example of Correctness:
Human's answer:
Since $2500 [consideration to creditor 1's promise] and a sweepstake ticket [consideration to creditor's promise] is sufficient consideration. then [creditor and [creditor 2) cannot demand repayment the whole debt.
Chat GPT's answer:
Since $2500 jconsideration to creditor 1's promise] and sweepstake ticket (consideration to creditor's promise] not sufficient consideration. then (creditor 1] and [creditor 2) can demand repayment of the whole debt.
Scoring: -1, the answer incorrect and not matched.
Second example of Correctness:
Human's answer:
Since $2500 (consideration creditor 1's promise] and sweepstake ticket (consideration to creditor's promise] is sufficient consideration, then [creditor 1] and [creditor 2) cannot demand repayment of the whole debt.

---

Chat GPT's answer:
[creditor 1] and [creditor 2] cannot demand repayment of the whole debt.
Scoring: O, even though the answer was correct, there is no explanation.
Third example of Correctness;
Human's s answer:
Since A is still in full time education, A dependent child.
Chat GPT's answer:
could be considered a dependent child as she was still in full time education.
Scoring: correct with explanation.
2. (Answerability) Annotators are typically asked to rate the output on fixed scale with specific level of quality. The following are the factors that the annotators need rate for the given
a) (Fluency) refers to the generated text with respect to the English language such grammar word choice for each question. is used determine whether the answer is understandable for normal human beings.
First example of Fluency:
Yes, that is correct. Alice not married and filing tax retum for 2015, she can claim a personal exemption of $4000 under section 151(a) of the Internal Revenue Code.
However, actual outcome depend on the specific terms the contract between Jennifer and Lawrence. is possible that the contract may have included provisions for circumstances in which Jennifer may choose disobey Lawrence's instructions.
Scoring: 1, as the choice of words and grammar was accurate and easy to understand.
Second example of Fluency;
James start (starts) work at 8am.
Than (then), they proceed to buy the goods.
Their (they) have accepted the offer by way replying the email.
Scoring: -1, grammar mistakes.
(Informative Relevance) refers to generated text related to material,
whether the sentence includes accurate sections related statutes or court cases.
First example informative relevance; Chat GPT's answer:
She can claim personal exemption of $4000 under Section (51 of the Internal
Revenue Code.
Scoring: 1, the sections are detailed and mentioned in the answer.
Second example of informative relevance;
Chat GPT's answer:
She can claim personal exemption of $4000 under the Internal Revenue Code.

---

Scoring: the is general referring the specific section, couldn't be considered as informative and relevant enough.
Third example of informative relevance;
Chat GPT's answer:
She can claim a personal exemption of $4000.
Scoring: -1, the answer is too general without referring the statute.
c) (Articulation argument) refers the generated
demonstrating imagination ar flair for the given scenario. This is to determine whether the generated text identified important legal concepts when doing the analysis.
First example of Articulation of argument:
The personal exemption amount for 2015 was $4000 for individuals who are not married and not the dependent of another taxpayer.
Scoring: 1, "Taxpayer" is the key legal concept that needs to be identified.
Second example of Articulation argument;
Since the person does provide financial support, this amounts to being in substantial care of person X. As such. the kid is a dependent child of person X. Scoring: "substantial care" is the key legal concept that needs to be identified.
Third example of Articulation of argument;
Since the agreement agreement of aporenticeship and that it necessary for Jennifer's in order her earn an income, then she would
disobeying the instruction despite the lack of capacity.
Scoring: "capacity" and 'necessary for life" 's the key legal concept that needs addressed.
(Default Reasoning)
This is the part where the annotators evaluate the reasoning performance for each generated text. We will use the precision and recall matrix to evaluate the reasoning step. Following example the assumption that is highlighted by annotators. Usually, there is limited information from the scenarios given. In order have a good analysis for reasoning, there always should be assumptions based on the application of law.
Examples Default Reasoning by annotators::
a) Is ChatGPT making good assumption? (-1,0,1)
This is evaluate the overall performance of the reasoning done by ChatGPT.
Scoring:
-1: Most assumptions made were inaccurate and not matched.
O: Most assumptions made were accurate but not matched/ assumption made were inaccurate but matched
1: Most assumptions made were accurate and matched.
b) How many assumptions made by ChatGPT? (Answer numbers)
Examples of an assumption:

---

In of B, if agreement between and made writing and was legally binding, then B may be bound by the agreement and may not be able to demand repayment of the balance of the debt owed by X.
Since they unemployed, they may be considered dependent child under the Australian social act.
It is likely that she is fully dependent on you, hence, she is dependent child. It could be arqued that the agreement between A and falls under this
exception
Scoring: These are each considered as assumption. However, it is subjective when comes to determining the numbers of correct assumptions. (Tips: Focus key words such then B", "since A, then may/ may not B", 'likely that", could be argued that").
Among the assumptions made by ChalGPT, how many assumptions are correct? (Answer in numbers)
First example correct assumptions:
Chat GPT's answer: It is important to note that the definition of a dependent child and the criteria for determining dependency can vary between different government programs. Scoring: correct assumption, even though was not mentioned Human's answer, these are still relevant factors to consider. Unmatched, but correct, still correct assumption.
Second example of correct assumptions:
Human's 's answer:
Since the kid dependent then the kid is dependent child of X.
Chat GPT's answer:
can confirm that the kid a dependent child.
Scoring: correct assumption, Regardless of whether there is reasoning/ justification! reference the law/ matched not matched, long correct, then considered correct assumption.
d) Compare the assumptions from ChatGPT and humans, how many of them are matched? (Answer in number)
Example of matched assumptions:
Human's assumption
Since A was in an agreement of apprenticeship. then A would be liable for breaching the agreement for disobeying B's instructions.
Chat GPT's assumption
A failed to follow B's instruction, then she is in breach of the agreement..
Scoring: matched assumption. Even though reasoning/ justification/ reference to the law, so long as matched, then considered as matched assumption.
e) Give general comment on the performance of ChalGPT's reasoning, along with the strengths and weaknesses.

---

Examples for comments:
Lack of discussion on the main issue.
Too descriptive and merely restating the facts.
Explanations were provided, legal issues were being raised and applied accurately.
Examples of strength:
Good application the law.
Clear explanations were provided.
Examples weakness:
Lack of reference to the law.
Irrelevant information included.
Main issues mentioned.
6. Requirements to join the program
The commit time for this project will be around 5-10 hours per week
Students need to pass the pre-test before they start on the work formal
session
Students need be familiar with Contract Law and Australia Social Act and get
least this subject
Data annotation will be done through face-to-face or Zoom meetings for the formal
session; the venue will be Monash University.
There a progress update meeting every week. The expert will join the meeting
check the weekly progress.

---

16. whether the contract is voidable by fraud A.2 Number of the issues
20. whether Southern Cross Hospital has to return the X-ray
1. Is the child dependent child?
21. whether BB have to pay compensation for breach of contract.
2. Whether is there a valid contract between Alan and Cate?
A.3 Evaluation Details
3. whether there is contract between Ellie and Frank, Frank and Galvin.
Evaluation of Assumptions. As legal reasoning is defeasible, we evaluate the generated assump- tions by using the following questions:
4. whether Peter can compel Trevor to pay him again
Is ChatGPT making good assumption? (- 1,0,1) This is to evaluate the overall perfor- mance of the reasoning done by ChatGPT. Scoring:
5. Whether XYZ is bound by Sam's actions.
6. whether Jennifer could be liable for breach of agreement
-I: Most assumptions made were inaccu- rate and matched.
7. whether Jennifer could be liable for breach of agreement
0: Most assumptions made were accurate but not matched/ assumption made were inaccurate but matched
8. whether B and C can change their mind and demand repayment of whole debt.
1: Most assumptions made were accurate and matched.
9. whether Debbie's promise is supported by valid considerations
How many assumptions are made by Chat- GPT? (Answer in numbers)
10. whether the exemption clause is effective.
Among the assumptions made by ChatGPT, how many assumptions are correct? (Answer in numbers)
11. whether Jim could rely on exemption clause
12. whether Freedom Flight Sdn Bhd could keep the sum paid by Monash Soccer Team and whether they could claim the balance 50
Compare the assumptions from ChatGPT and humans, how many of them matched? (An- swer number)
13. whether Ron Realty Sdn Bhd can rely on doc- trine of frustration
Give general comment the performance of ChatGPT's reasoning, along with the strengths and weaknesses.
14. whether Cathy is legally obliged to pay Dave RM500 and RM1000 and whether Cathy is legally obliged to buy a car for Dave.
A.3.1 Decomposed questions
The following is the decomposed question for each scenario.
15. whether there is any enforceable contract be- tween Lilian and Mike, whether whether there is contract between Lilian and Neil for gold and silver dye. Whether there an enforce- able term on the quality red dye
17. whether Dr Pritam can rescind the contract
18. whether Bakar have any contractual right against Ali and Chong.
19. what is the damages Alpha Pty Ltd entitled to among the options of a,b,c,d and

---

Decomposed Questions
If child above and below
If the child wholly substantially dependent adult?
ASA
If child's income does not exceed 6403 the financial year?
If the child is above 16 and below 22?
child wholly substantially dependent adult?
If the child's income does not exceed 6403 in the financial year?
ASA.2
If adult is legally responsible for day to day care, welfare and development of the child and the child the adult's If the child not dependent child of someone else and wholly and substantially in the of the adult If the child is in full-time education If the child in receipl of income which exceeds 107.70 week
AS
If child above and below If the child wholly substantially dependent adult?
AS
does exceed
year?
If adult is legally responsible for day to day care, welfare and development of the child and the child is the adult's If child If the child is in full-time education If child
dependent child someone else wholly substantially in the the
ASA
receipt income which
107.70 per
If the child above 16 and below If the child wholly substantielly dependent adult?
ASA
If child's income does not exceed 6403 the financial year?
If the child above 16 and below 22?
child wholly subslanlially dependent adult?
07
if the child's income does not exceed 6403 in the financial year?
If adult is legally responsible for day to day care, welfare and development of the child and the child is the adult's if the child If the child is in full-time education If child
dependent child of someone else wholly and substantially in the of the adult
A8
receipt income which exceeds 107.70 per
If child above and below If the child wholly substantially dependent adult?
A9
If child's income does not exceed 6403 the financial year?
the child is above 16 and below 22?
child wholly substantially dependent adult?
ASA 10
If the child's income does not exceed 6403 in the financial year?
If adult legally responsible for day to day care, welfare and development child child adult's If the child nol a dependent child of someone else and wholly and subslanlially in the care of the adult If the child full-time education
ASA 11
If the child in receipt of income which exceeds 107. 70 per week
If the child above 16 and below 22?
If child wholly substantially dependent adult? If the child's income does not exceed 6403 the financial year? If
ASA 12
legally responsible for day day care, welfare and development child child adult's If the child is not a dependent child of someone else and wholly and substantially in the care of the adult child full-time education
ASA 13
If the child is in receipt of income which exceeds 107.70 70 per week
If adult is legally responsible for day to day care, welfare and develapment of the child and the child the adull's If the child nol a dependent child of someone else and whally and subslanlially in the care of the adult if the child in full-lime education
ASA
If child in recelpt income which exceeds 107.70 per week
If the child is above 16 and below 22?
If child wholly substantially dependent adult?
ASA 15
if the child's income does not exceed 6403 in the financial year?

---

If the child above 16 and below 22?
If child wholly substantially dependent adult?
ASA
If the child's income does not exceed 6403 the financial year?
If
legally responsible day to day care,
and development child
If the child is not a dependent child of someone else and is whally and substantially in the care of the adult child full-lime education
ASA 17
If the child is in receipl of income which exceeds 107.70 per week
If the child above 16 and below 22?
If the child wholly or substantially dependent on adult?
ASA
If the child's income does exceed 6403 financial year?
If child above and below If the child wholly substantially dependent adult?
ASA
If child's income does not exceed 6403 financial year?
If adult is legally responsible for day to day care, welfare and development of the child and the child is in the adult's If child If the child is in full-time education If child
dependent child someone else wholly and substantially in the the
2SA
receipt income which exceeds 107.70 per
1. Was the adverlisement put oui by Alan an invitation to treat or an offer
2. Whether Alan has accepled Cale's offer to buy the book at 1000
3. Whether Alan is bound to keep his offer open for 7 days
CAM 01
4. Whether there is valid acceptance by Cate for Alan's offer to sell 2000
advertisement is invitation treat or an 2. whether Frank has accepted Ellie's offer
1. whether
3. Whether there valid revocation offer by Ellie
4. whether Galvin has accepted Frank's offer?
communicate his revocation offer through third parly?
CAM 02
6. Could Galvin revoke his acceplance after sending the text?
3.1. Does agent have any expressed implied authority?
3.2. If the agent has authority, then whether the agent's action exceeds his authority?
CAM 03
3.3. the agent's actions have exceeded authority, then would the principal bound by the agent's actions?
1. Does Sam have any authority?
2. Whether Sam's authority implied express
3. whether Sam's action of purchasing land on behalf XYZ Bhd exceeds his authority?
4. Whether Sam has apparent authority
CAM 04
5. What can Bhd do Sam has exceeded his authority?
1. Is Jennifer a competent party to to contract
2.
she bound by agreement? 3. Is the agreement between Jennifer and Lawrence a contract for necessary?
CAM
1. Jennifer a competent party to to contract
2. she
would she bound by the agreement?
CAM 08
3. the agreement between Jennifer and Lawrence contract for necessary?
1. B's promise forgo the rest of the debt supported considerations?
2. Does the part payment of debt by father amount valid considerations?
3. C's promise to forgot the debt supported by considerations?
CAM 07
4. Does the sweepstake ticket amount to valid considerations?
1. Does Michaelss action amount to valid considerations?
CAM 08
2. Does Michael's action amount valid considerations despite his publi duty as police officer?
1. whether the exemplion clause on the licker is incorporated into the contract
2. Does the small print the
side of the licket amount sufficient notice?
3. whether the exemplion clause on the sign is incorporated into the contract
4. if the exemption clause incorporated, would be effective?
5. how would the court interpret the exemplion clause
6. Would the exemption clause effective the stain marks caused negligence?
7. What can Annie do if the exemption clause is incorporated
CAM 09

---

1. the exemption clause on the docket incorporated into the contract?
2. the exemption clause on docker incorporated by past dealings?
3. how would the court interpret exemption clause if incorporated into the contract?
4. Would Jim's actions deviation from direct route affect the effediieness the exemption clause?
CAM 10
5. Does Jim's actions of deviation from direct route amount to fundamental breach?
whether
frustrated by Iraffic
2. Does the air traffic controllers' strike render the contracl impossible
frustrated?
4. What can Freedom Flight Sdn Bhd do if the contract is frustrated??
Freedom Flight Sdn
money they received Monash Soccer the
1. Does the economic downtum amount to a frustrating event?
2. Does the economic downtum renders performance of contract impossible radically different?
3. Would Ron Really Sdn Bhd still able to rely on doctrine frustration if economic downlum made the perf
CAM 12
4. Would Ron Really Sdn Bhd be able rely doctrine frustration the contract between them contain
1. Cathy's promise to pay Dave RM500 for not smoking supported by valld considerations
supported by valid considerations 3. Cathy's promise to pay Dave 1000for
2. Cathy's promise to buy Dave
work supported by valid considerations 4. Does the relationship between Cathy and Dave affect the enforceabiliy of the promise to RM500, RM 1000 Cathy's promise to Dave supported by intention
legal relations?
1. Is the agreement between Lilian and Mike supported by Intention to create legal relations? each?
2. Did Lilian
Nell's offer
gold dye
and silver dye
3. Does Neifs sllence amount to valid acceptance?
4. Neil's
regarding the quality of dye enforosable term?
5. If Neil's statement regarding quality of dye is not
an enforceable term, what can Lilian do?
CAM 14
6. Lilian rely
goods 1957?
If there suggesiion fact which is untrue
If zakariah does not belleve suggestion of fact true
If Zakariah made statement with intention to induce party to enter into the contract
CAM 15
If consent is caused by fraud
non-disclosure If non-disclosure of fact induced Dr Pritam to enter into contract
duly
consent is caused by misrepresentation
1. and bakar are in a contract
2. If ali's action deprived him from performing his contractual obligations with Bakar
CAM 17
3. Bakar third party to Ali and Chong
1. alpha can recover RM 1000 day
2. Alpha can recover RM1000 the entirety weeks 3. Alpph can recover RM5000 spent replacement
4. aipha can recover RM 50000
CAM 18
5.1f Beta have actual knowledge RM50000 contract
If party intended for oral statement to have contractual effect as a term of the contract If
importance of oral statement If there is long interval If oral stalement is included in the contract If oral statement forms collateral contract If oral statement induce contract If oral statement is supported by intention If oral statement is inconsistent with written term If wrillen term is incorporated by signature If exceplions to signalure rule is applicable
CAM 19

---

If party intended for oral statement to have contractual effect as a term of the contract party express importance oral statement If there is long interval
included in the contract
If oral statement forms collateral contract contract
If oral statement is supported by intenlion is inconsistent with written If written term is incorporated by signature exceptions signalure applicable
CAM

---

A.3.2 Questions for evaluation
Ql: Is the answer make correct conclusion?
Q2: Is there any improvement of the answer? (Only applied for in in-context learning and de- composition questions)
Q3: Is the answer fluency?
Q4: Is the answer provided information rele- vance?
Q5: Is the answer have correct Concept Iden- tification?
Q6: How many assumptions are made?
Q7: Among the assumptions made by CHAT- GPT, how many assumptions are correct?
Q8: Compare the assumptions from CHAT- GPT and humans, how many of them are matched?
Q9:
Please provide your general comment for the given answer.


Reviews:
[{"review_id": "meuXB2Oz01", "reviewer": "Reviewer_uuNg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your helpful comments on our work. The following are our responses to each reason to reject: \n\nR1:  Lack of comprehensive review of IRAC Methodology.\nResponse: We introduce IRAC in Sec. 2, which should provide enough information for understanding our work. In addition, we discuss the relevant works we are aware of, which use LLMs for IRAC analysis in Sec. 5.1. Herein, (Yu et al., 2022) covers only the conclusion parts of IRAC and (Guha et al., 2022) divide IRAC into independent simplified tasks and each task is applied to different simplified scenarios. Thus they do not provide comprehensive evaluation from issues to conclusions on the same scenarios. In contrast, our work is the most comprehensive and complete evaluation of an LLM for IRAC analysis on legal scenarios, and provides the first dataset for such an evaluation. Moreover, IRAC is the most widely used legal reasoning framework for legal professionals. A comprehensive review of this technology is conducted often in textbooks and long surveys for law students and junior legal professionals. Therefore, we believe a comprehensive review of IRAC goes beyond the scope of this work.\n\nR2:  Limited Contribution to Specific Countries.\nResponse: Legal reasoning is challenging even for legal professionals. Therefore, only top law students, lawyers and law professors are able to help with annotations and evaluation of the IRAC methodology applied to scenarios. The analysis of legal scenarios is expensive and time consuming, which usually takes more than two hours per case. The evaluation of legal analysis even takes significantly more time. The Jurisdictions and the legal systems vary significantly so that legal professionals in one legal system cannot work on the other legal systems. As a result, our annotators cannot work on the legal scenarios from the other countries. We believe, finding top law scholars and top law students in foreign countries and having the capability to pay them remotely is beyond the capabilities of most research labs in this field. Our strategy is to start with the common law system, and choose the countries that it is unlikely that LLMs already memorize a large number of similar cases and relevant statutes so that the legal reasoning is not simplified to a retrieval problem. We identify the challenges and open problems by using the dataset we constructed and claim the findings in the common law system. We hope, our work will promote similar research in other jurisdictions and countries. In our point of view, although it is desirable to build a dataset and conduct evaluation across all jurisdictions and legal systems, it needs the efforts from the community, not just a single research lab. Similar to the first work on machine translation in our community, which just translated 60 Russian sentences into English, our work kick-starts a new line of research and shows that current LLMs fail to achieve reasonable performance on such tasks. \n\nR3: Small Dataset.\nResponse: As stated in Line 039-042, this work focuses on understanding to what degree ChatGPT is able to perform reasoning for legal scenario analysis in the same way as legal professionals. The key finding is that on all scenarios analyzed by ChatGPT, most of the intermediate reasoning steps in the analysis parts are wrong, though some of the conclusions are correct. None of the scenarios has a completely correct analysis part. When we feed correct human-written partial reasoning paths to ChatGPT, the reasoning performance of ChatGPT is improved significantly and consistently. Hence, this key finding is not based on selection bias or randomness, and the finding is adjusted by the corresponding interventions (providing partial reasoning paths). The same is applied to the other findings. In other words, the dataset is large enough for the findings identified in our work. We agree that it is always better to have larger datasets for the community. However, this task is special due to the high requirements on financial resources and annotators. If we want the dataset to have similar size as the other NLP tasks, probably only the richest labs in the world are able to conduct this line of research. The outcomes of this line of research are for social good. Hence, we believe it is better off encouraging more research teams to participate in this line of research, as long as their findings can be well adjusted by their empirical evidence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Novel Corpus Construction: The paper presents the creation of a novel dataset, SIRAC, containing annotated legal scenarios, which not only aids legal professionals but also serves as a valuable resource for legal Natural Language Processing (NLP) research.\n\n(2) Evaluation of ChatGPT for IRAC Analysis: The paper provides the first empirical assessment of ChatGPT's performance in legal reasoning using the IRAC method. This evaluation sheds light on the strengths and limitations of LLMs in legal tasks.\n", "reasons_to_reject": "(1) Lack of Related Work on IRAC Methodology: The paper introduces the IRAC methodology for legal reasoning (Alsagoff, 1996), but it lacks a comprehensive review of related work on this specific method. Including a broader discussion of previous research and approaches using IRAC would provide a more thorough understanding of its effectiveness and limitations in legal analysis.\n\n(2) Limited Contribution to Specific Countries: The paper focuses on legal scenarios from Malaysia and Australia, which may limit the generalizability of the findings to other legal systems and jurisdictions. Expanding the dataset to include scenarios from a more diverse set of countries would enhance the applicability and relevance of the study.\n\n(3) Small Dataset: The SIRAC dataset contains only 50 legal scenarios related to ASA and CAM. This limited dataset size may impact the robustness and generalizability of the results. A larger and more diverse dataset would strengthen the validity of the findings and improve the model's performance on various legal scenarios.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the application of Large Language Models (LLMs), specifically ChatGPT, in the legal domain for performing legal analysis using the IRAC (Issue, Rule, Application, Conclusion) method. To assess ChatGPT's ability to reason like legal professionals, the authors construct a novel dataset called SIRAC, containing legal scenarios related to the Contract Acts of Malaysia and the Australian Social Act for Dependent Children. Each scenario is annotated with a complete IRAC analysis in a semi-structured format. The experimental results reveal that while ChatGPT can produce reasonable answers, it struggles to generate correct reasoning paths aligned with legal experts. The paper highlights potential research directions to enhance the alignment between LLMs and legal professionals in legal reasoning tasks.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aFEyZ1xOxp", "reviewer": "Reviewer_623p", "summary": "", "questions": "- The evaluation is conducted on 2 very small areas of law, how generalizable could those results be? It seems that those areas are also quite \"easy\" areas compared to some more potentially complex legal domains that rely on more statutes/ more precedents? ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful feedback and comments. Below are our responses to the reasons to reject.\n\nR1: Impact of adding reasoning paths.\nResponse: The purpose of our experiments on feeding progressively partial reasoning paths to ChatGPT are two-folds. First, the previous subsection shows that the reasoning paths produced by ChatGPT did not align with those of legal professionals in general. We show that providing correct reasoning paths, even part of them, indeed help ChatGPT to improve reasoning performance. Particularly, wrong reasoning paths are the key to poor performance of ChatGPT. Hence, the future research direction is to generate higher quality reasoning paths for both performance improvement and explanabilty. Second, the experiments do not aim to suggest that people should provide correct reasoning paths when using ChatGPT for legal analysis. Those experiments were carefully designed and carried out to demonstrate to what extent correct reasoning paths are useful. E.g. what if 20%, 40% or 80% of them are correct. How would correct reasoning paths influence final conclusions? We will revise this paragraph to make our purpose clearer.\n\nR2: usefulness of a neuro-symbolic approach for legal reasoning.\nResponse: The neuro-symbolic approach aims to make the reasoning of LLMs more robust and generalizable across different scenarios. The recent evaluations of LLMs show that LLMs cannot work well in several tasks, e.g. numerical reasoning, complex reasoning, and compositional generalization. Unfortunately, legal reasoning indeed associates with some of the key weaknesses of LLMs. To address those weaknesses, we introduce symbols only at the places where LLMs often fail. When building a stronger LLM, such symbols will be associated with the corresponding functions and use of symbols will support advanced (symbolic) reasoning techniques, which will produce more reliable results than using language alone. Moreover, the language used in statutes already use the same words for the same logical relations. Hence, we only need to associate such keywords with the corresponding operators without rewriting the reasoning processes. In the legal domain, some scholars are trying similar ideas by making legislations more structured, even though they do not have a computer science background. \n\nAnswers to the questions for the authors: \n\nQ1: Small areas of law\nAnswer: The reasons that our findings are generalizable to many areas of law are three-folds. First, our dataset and evaluation covers all four aspects of IRAC analysis for each scenario. Second, the scenarios in Contract Law are from tutorials and exams for law students, they cover all main topics in that area, so that they are not simple. Law students often need several hours to analyze them. The running examples in the main body of the paper are chosen due to the limited space. Third, the evaluation we conducted does not involve any assumptions or heuristic rules specific to the two selected areas. The more complex scenarios can be considered as having more statutes and precedents involved. Compared to the existing datasets on legal reasoning, such as SARA (Holzenberger, N., & Van Durme, B. (2021), our datasets are far more comprehensive and closer to real-world scenarios than those datasets. \nFor the typos and grammar style and presentation improvements: \n\n1: Introducing semi-structured language earlier.\nAnswer: We will revise this manuscript to introduce semi-structured language in Introduction.\n\n2: What is the “average length”?\nAnswer: The length of a reasoning path is defined as the number of involved reasoning steps. A reasoning step involves the application of a single rule or a single argument from a precedent to draw conclusions. The average length is thus the average number of reasoning steps across the reasoning paths of interest.\n\n3. Missing spacing for the reference \nAnswer: We will revise the spacing in line 551.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic of the study is very relevant and should be of interest not only to the NLP community interested in domain-specific applications, but also to legal professionals or scholars. \n- The methodology,  implementation and evaluation are very clearly and precisely explained (both in the main paper and in annex), from the used legal cases, the annotators recruitment and level of expertise, the questions asked etc. This is highly appreciable, helps the reader to follow and demonstrate the quality of the survey. \n- Legal NLP generally lacks resources and datasets, and for this reason, introducing this new dataset in English is important. Moreover, this dataset is a valuable contribution on 2 aspects: 1/ it includes scenarios from Australia and Malaysia which are two under-represent jurisdictions in the existing legal NLP English resources. 2/ It provides human annotations on legal scenarios in a very precise manner including the legal reasoning that is thought as a demonstration and justification, which is very valuable for a variety of NLP tasks including legal QA. \n- The assumption made line 327 that the desired output is aligned with the human reasoning behaviour is important. \n- The results provide a good foundation for further research in legal QA/generative NLP. ", "reasons_to_reject": "- line 449 \"Impact of adding reasoning paths\": this experiment yields important results, however, one could wonder if providing too much information to ChatGPT is realistic? It seems natural that the more information you provide, the better it performs. However the goal of using a QA system in the legal domain is rather to ask a simple question and get an answer, rather than already providing a reasoning path. \n- The idea of introducing a neuro-symbolic approach and logical operators is interesting. Similarly to the above comment, I wonder if it wouldn't be more meaningful and realistic to consider only natural language, for purposes of simplicity, wider understanding including by the users and because having to re-write a reasoning process takes time. This may be counterproductive. ", "questions_to_authors": "- The evaluation is conducted on 2 very small areas of law, how generalizable could those results be? It seems that those areas are also quite \"easy\" areas compared to some more potentially complex legal domains that rely on more statutes/ more precedents? ", "paper_topic_and_main_contributions": "The paper conducts an analysis of the legal reasoning capabilities of ChatGPT on a range of legal problems, creating at the same time an annotated dataset for legal reasoning on a sample from Contracts acts from Malaysia and the Australian social act for dependent children. The authors demonstrate 1/ the misalignment of the \"reasoning\" steps of ChatGPT compare to a legal professional and 2/ the lack of autonomy of ChatGPT when reasoning about a legal case, that improves as the prompts get more detailed. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be beneficial to explain earlier in the paper what is meant by \"semi-structured language\", for example in introduction. The reader only understands that key element of the study around line 216 when it has been mentioned several times before. \n- Section 3.4: the compute of the \"average length\" is not clear. The reader understands that it is a good proxy for complexity, one would like to understand what is considered a \"reasoning step\". \n- Section 5.1: there are a few spaces missing in the citations: line 551 \"Andew(Bl...\", line 552, line 575\n"}}, {"review_id": "rGz2vy6gUK", "reviewer": "Reviewer_Vapu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your insightful comments and constructive feedback on our work. The following are our responses to the reasons to reject: \n\nR1: Experiments were conducted only using ChatGPT.\nResponse: Our work constructs a dataset for IRAC analysis and focuses on understanding to what degree ChatGPT is able to perform reasoning for legal scenario analysis similar to legal professionals. To address this research question, we need to evaluate the best performing LLM for IRAC analysis. If such a model fails to achieve that goal, the other LLMs fail as well. ChatGPT was the best candidate when we started experimenting on the constructed dataset. It performed far better than the other open source LLMs back then. It was the only model that generated meaningful results. The other LLMs failed almost completely, including drawing meaningful conclusions. Given the evaluation cost is high for each model due to the use of legal professionals as annotators, the high cost for the other models would not bring in new information but show that the other models fail almost completely.  \n\nR2: Missing a series of hypotheses that are intended to be evaluated in the work.\nResponse: As stated in Line 039-042, this work focuses on understanding to what degree ChatGPT is able to perform reasoning for legal scenario analysis in the same way as legal professionals. We have a series of hypotheses to support this research goal, but the hypotheses were not all formulated in the typical formats of hypotheses.\n\nHypothesis I: The key hypothesis in Line 411 and 447 is that ChatGPT cannot produce the same or correct reasoning steps as those of legal professionals, despite the conclusions may be correct. Our empirical study supports this hypothesis and demonstrates the key challenges for current LLMs.\n\nHypothesis II: The key hypothesis in Line 449 and 465 is that reasoning paths, especially intermediate reasoning steps, are important for drawing correct conclusions. We support this hypothesis by evaluating ChatGPT with progressively partial reasoning paths as additional inputs. The results show the effectiveness of reasoning paths for final conclusions. This also sheds light on how future models should improve. Not just predicting the final conclusions but also focusing on generating correct intermediate reasoning steps.\nHypothesis III: Line 466 - 489 are concerned with the hypothesis that similar to humans, demonstration with reasoning examples is useful for improving reasoning performance. In the corresponding experiments, we show that in-context learning with the reasoning examples similar to the input scenario is effective in improving reasoning performance. This motivates future research on how to build and retrieve similar examples for improving reasoning.\nHypothesis IV: Line 490 - 515 focus on the hypothesis that a key factor of poor performance is that the initial question is too complex for ChatGPT to answer. Then we show that when decomposing a legal question into simpler questions and let ChatGPT answer them respectively, the performance is greatly improved. \n\nWe will revise the corresponding sections to make hypotheses, findings, and support of hypotheses more clear.\n\nR3: No analysis or insights are presented as to why the results are unsatisfactory and how they could be improved.\nResponse: Line 434 - 447, we show that ChatGPT produces unsatisfactory analysis parts because i) most of the intermediate reasoning steps fail to include correct references to statutes and precedents; ii) the articulation of reasoning in the analysis parts is sometimes confusing and logically inconsistent; iii) most of the analysis parts from ChatGPT do not align with those from legal professionals.\nLine 448 - 465 shows that the above limitations can be mitigated by feeding partial reasoning paths written by professionals to ChatGPT. It sheds light on the research direction about how to overcome the problems. Moreover, Line 466 - 489 shows the research direction how to further improve the reasoning performance by providing similar in-context examples.\nLine 490 - 515 shows that one of the reasons for unsatisfactory performance is the complexity of legal questions. If we decompose the question into simpler ones, the performance will be greatly improved. The future research direction is to automatically generate decomposing questions for complex legal questions.\n\nR4: The proposed dataset is not available. \nThe proposed dataset is provided in the submitted supplementary materials. However, we got to know after the submission that the supplementary materials may not visible to reviewers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It focuses on the generation of resources in a complex research line such as reasoning by language models and legal reasoning.\n\n- A series of experiments are presented to evaluate the feasibility of the proposal.\n\n- A set of guidelines used to create the dataset are provided.", "reasons_to_reject": "- The proposal is presented as a general study on the legal reasoning capacity of LLMs. However, it is only experienced using ChatGPT.\n\n- Missing is the proposal of a series of hypotheses that are intended to be evaluated in the work, beyond the general study on whether ChatGPT is capable of performing legal reasoning.\n\n- No analysis or insights are presented as to why the results are unsatisfactory and how they could be improved.\n\n- The proposed dataset is not available.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new corpus focused on the study of the capacity of LLMs to carry out legal analysis using the IRAC method. The dataset is composed of 50 scenarios belonging to the Australian Social Act and to the Contract Acts of Malaysia.\n\nIn addition, an empirical evaluation is proposed in the aforementioned corpus, using ChatGPT to perform legal analysis using IRAC. This result shows the difficulties that this LLM has to carry out correct reasoning following the legal methodology used.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims at assessing the legal reasoning capabilities of ChatGPT and introduces a novel dataset annotated on the bases of the IRAC methods. The experimental results reported offer useful insights into the limitations of current technology. While one reviewer was concerned about the limited coverage of the data set to only two countries and a few cases, it still offers an interesting example for future research. However, the paper needs improvements.  

**Pros** 

- The dataset is valuable and has the potential to help research on legal NLU;  

- the topic is timely and highly relevant not only for NLP 

- the paper is clear and easy to follow; 

- evaluation experiments are adequate and methodologically sound, although not sufficient to generalize the claims to other LLMs; 

**Cons** 

- the discussion of related works could better motivate the soundness of the IRAC methods as guiding principles for analysis;  

- the coverage of legal cases is limited, albeit perhaps sufficient to support the claims of this paper. A critical comparison with the other larger datasets mentioned by reviewers would be beneficial; 

- the experimental set up should be better motivated and explained, to avoid misunderstandings;
Decision: Accept-Findings