Domain$o1$s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains
Authors: Xu Chu, Zhijie Tan, Hanlin Xue, Guanyu Wang, Tong Mo, Weiping Li
Accepted at: acl25

Domain0ls: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains

Xu Chu†, Zhijie Tan†, Hanlin Xue, Guanyu Wang, Tong Mo, Weiping Li*

1School of Software and Microelectronics, Peking University, Beijing, China
chuxu@stu.pku.edu.cn, besttangent@stu.pku.edu.cn, colinneverland@stu.pku.edu.cn
2301210418@stu.pku.edu.cn, motong@ss.pku.edu.cn, wpli@ss.pku.edu.cn

†Equal contribution   *Corresponding author

Abstract

Large Language Models (LLMs) are widely applied to downstream domains. However, current LLMs for high-stakes domain tasks, such as financial investment and legal QA, typically generate brief answers without reasoning processes and explanations. This limits users’ confidence in making decisions based on their responses. While original CoT shows promise, it lacks self-correction mechanisms during reasoning. This work introduces Domain0ls, which enhances LLMs’ reasoning capabilities on domain tasks through supervised fine-tuning and tree search. We construct CoT-stock-2k and CoT-legal-2k datasets for fine-tuning models that activate domain-specific reasoning steps based on their judgment. Additionally, we propose Selective Tree Exploration to spontaneously explore solution spaces and sample optimal reasoning paths to improve performance. We also introduce PROOF-Score, a new metric for evaluating domain models’ explainability, complementing traditional accuracy metrics with richer assessment dimensions. Extensive experiments on stock investment recommendation and legal reasoning QA tasks demonstrate Domain0ls’s leading performance and explainability. Our code is available at https://github.com/Hyalinesky/Domain0ls.

1 Introduction

In specific domains such as finance (Xing, 2024; Jeong, 2024; Cheng et al., 2024c), law (Cheong et al., 2024; Colombo et al., 2024), and biomedicine (Labrak et al., 2024; Wang et al., 2023a), Large Language Models (LLMs) are widely used for tasks like recommendation (e.g., stock investment recommendation (Koa et al., 2024; Qin et al., 2024; Takayanagi et al., 2023)) and question answering (e.g., legal reasoning QA (Guha et al., 2024; Wang et al., 2023b; Ujwal et al., 2024)). However, popular approaches mainly adopt direct prediction paradigms that immediately generate brief answers to questions (Cheng et al., 2024a,c; Yue et al., 2023), leading to answers lacking explainability. In practical applications within high-stakes domains like finance and law, users may not trust results lacking explainability (Biran and McKeown, 2017) to guide decision-making. While Chain-of-Thought (CoT) reasoning demonstrates the ability to enhance models’ step-by-step thinking and domain problem solving (Li et al., 2024b; Jiang and Yang, 2023; Miao et al., 2024) and provides explainable reasoning processes, its single-pass generated reasoning chains lack error correction mechanisms. If errors occur in early reasoning steps, the model continues reasoning along the flawed path, affecting the subsequent reasoning process, as shown in Figure 1. This poses challenges for solving domain tasks, as flawed reasoning processes may introduce legal and ethical risks.

Recently introduced o1-type models (OpenAI, 2024; OpenO1 Team, 2024; Zhao et al., 2024), with their exceptional reasoning capabilities, demonstrate powerful performance surpassing reasoning methods like CoT in mathematics, physics, and coding. Compared to LLMs using CoT, o1-type models feature longer reasoning chains and reasoning time. They are considered to perform multi-stage reasoning rather than generating complete reasoning chains in single-pass, which enhances the accuracy of LLM reasoning. However, despite high-stakes domains requiring high-quality reason-

![Comparison of Domain0ls and other paradigms on a demonstrative example. Domain0ls expands reasoning paths and obtains optimal ones through tree search.](page_1016_563_393_180.png)

Figure 1: Comparison of Domain0ls and other paradigms on a demonstrative example. Domain0ls expands reasoning paths and obtains optimal ones through tree search.
ing, extending o1-type models’ capabilities to these domains remains an unexplored research gap.

In this paper, we design Domaino1s to provide explainable answers for high-stakes domain problems. Domaino1s includes two model variants, Domaino1s-finance and Domaino1s-legal. As shown in Figure 1, Domaino1s can (1) perform autonomous step-by-step reasoning, and (2) expand reasoning paths through tree search to obtain optimal ones. To achieve (1), we use GPT-4o (OpenAI, 2024) to generate CoT data and construct CoT-stock-2k and CoT-legal-2k datasets for supervised fine-tuning. During dataset construction, we employ 26 special tokens (e.g., <SUMMARY>) to prompt GPT-4o to distinguish different steps in the reasoning process explicitly. In the supervised fine-tuning process, we remove these special tokens from the answers, enabling the model to autonomously select and organize intermediate steps in the reasoning chain. To achieve (2) during answer generation, we introduce a novel Selective Tree Exploration method to find the optimal reasoning paths. This method uses the average perplexity of tokens in each reasoning step to decide whether to explore new paths and select the best path. Compared to traditional search methods (Weng et al., 2022; Jiang et al., 2023; Chen and Liu, 2024), Selective Tree Exploration balances search performance and time cost. We evaluate Domaino1s on stock investment recommendation (Koa et al., 2024) and legal reasoning QA (Guha et al., 2024) datasets. Unlike most domain benchmarks (Koa et al., 2024; Yang et al., 2022; Guha et al., 2024), we point out that focusing solely on answer accuracy makes it difficult to determine whether models properly reason through given contexts rather than relying on shortcuts or overfitting. We emphasize the necessity of evaluating domain models’ explainability and introduce a new evaluation metric PROOF-Score (Principled rating for reasoning completeness, domain safety, and factual accuracy) to fill this gap. Results show that Domaino1s improves reasoning accuracy while providing high-quality, explainable reasoning processes. Our contributions are:
• Domaino1s is proposed for explainable answers, with two model variants.
• CoT-stock-2k and CoT-legal-2k datasets are constructed for fine-tuning. Selective Tree Exploration is proposed as a reasoning path search method that balances performance and time cost.
• PROOF-Score is proposed to evaluate the explainability of domain model answers, introducing a new perspective for domain model evaluation.
• Domaino1s achieves leading performance, demonstrating the effectiveness of its reasoning capabilities in solving high-stakes domain tasks.

2 Related Works

2.1 LLMs for Specific Domains
LLM applications in specific domains typically follow three approaches: training from scratch, fine-tuning, and prompt learning. While training from scratch (e.g., BloombergGPT (Wu et al., 2024)) shows promising results, it requires significant computational resources and data (Yang et al., 2023; Ling et al., 2023; Xie et al., 2023b). Fine-tuning emerges as a cost-effective alternative, with researchers using GPT-4 (Li et al., 2024a) or low-cost automated methods (Cheng et al., 2024c; Koa et al., 2024) to generate fine-tuning data. Prompt learning methods enhance model capabilities without parameter modification through template engineering or knowledge retrieval (Li et al., 2023; Cui et al., 2023; Huang et al., 2023), such as CoT (Wei et al., 2022) reasoning. o1-type models are typically constructed to equip LLMs with CoT reasoning capabilities through fine-tuning, followed by multi-pass search to obtain better reasoning paths.

2.2 Single-Pass vs. Multi-Pass
Prompt-based methods like CoT (Wei et al., 2022; Zhang et al., 2022; Lyu et al., 2023) improve single-pass reasoning through better prompt templates. However, errors in intermediate reasoning steps can propagate through the chain. In contrast, search-based methods explore multiple reasoning paths in the solution space, treating each reasoning step as a node in the tree, and selecting the optimal path to improve reasoning quality (Qi et al., 2024).

2.3 Sampling Reasoning Paths
Research on mathematical reasoning (Brown et al., 2024; Wang et al., 2024) indicates that sampling different reasoning paths can improve performance compared to greedy one-time decoding. Best-of-N search (Weng et al., 2022; Jiang et al., 2023) generates N complete answers, allowing LLM to select the best response based on final results, but may miss high-quality intermediate reasoning steps (Xie et al., 2024; Chen et al., 2024b). Sentence-level Beam Search (Chen and Liu, 2024) generates multiple candidate sentences, selects the best one, and
Example 1: Stock Investment Recommendation
Question: For a given set of facts, what was the overall impact on Apple’s stock price movement. Answer with Positive/Negative. Here are the facts:
2022-09-29
- Bank of America downgrades Apple from “Buy” to “Neutral.”
- Apple’s stock plummets, down by over 5% on the day.
…
Ground Truth: Positive.

Answer from Qwen-2.5
The overall impact of the given facts on Apple’s stock price movement is positive but negative. The facts indicate a series of downgrades, warnings about demand, negative market reactions, and other unfavorable factors, such as stock price having fallen low, which performance drags. These have had a negative impact on Apple’s stock price.

Negative

Answer from Domain01s-finance
<SUMMARY> … We will examine several factors to drive the analysis, including historical performance valuation metrics, market position, strategic initiatives, investor sentiment, risks, and recent performance. <SUMMARY>
<HISTORICAL CONTEXT> Historically, Apple’s stock has shown resilience but faced challenges this year, down 22% by mid-September, impacting overall market sentiment. … </HISTORICAL CONTEXT>
<INVESTOR SENTIMENT> Investor sentiment appears mixed, with notable trading activity in options reflecting apprehension amid stage-four challenges. … </INVESTOR SENTIMENT>
<OVERALL IMPACT> Despite the confluence of negative operational, strategic, and market pressures, the overall impact on Apple appears tilted towards a positive outlook. … </OVERALL IMPACT>
Price Movement: Positive. <OVERALL IMPACT> ✓

Example 2: Legal Reasoning QA
Question: Here is a service agreement: If you are a consumer, the laws of the country in which you reside will apply to any claim … and you may resolve your claim in any competent court in that country that has jurisdiction over the claim.…
As a Facebook user who lives in Canada, can I sue Facebook in Canada?
Ground Truth: Yes.

Answer from Qwen-2.5
If you are a consumer and reside in a country other than the United States, any claim against Facebook must be resolved in the competent court in the country where you reside. However, for non-consumer cases, including those from Canadian residents, the claim must be resolved exclusively in the U.S. District Court for the Northern District of California or a state court located in San Mateo County, California. No

Answer from Domain01s-legal
<SUMMARY> … For non-consumers, disputes must be resolved in U.S. courts. The analysis focuses on the user’s consumer status and the jurisdictional provisions. <SUMMARY>
<REASONING> 1. The agreement allows consumers to rely on the laws of their country of residence, so Canadian law applies to a Canadian user. … </REASONING>
<EVIDENCE> - The agreement states: “If you are a consumer, the laws of the country in which you reside will apply” and “you may resolve your claim in any competent court in that country.” … </EVIDENCE>
<CONCLUSION> A Canadian Facebook user classified as a consumer can sue Facebook in Canada. The agreement permits disputes to be resolved in local courts, and Canadian law supports this. The answer is Yes. <CONCLUSION> ✓

Figure 2: Comparison between the base model Qwen-2.5-Instruct (Qwen-Team, 2024) and Domain01s. The base model shows notable reasoning errors. In contrast, Domain01s breaks problems into multiple reasoning steps and reaches well-supported conclusions through systematic analysis. Details in Appendix C.

iteratively continues this process, but may get stuck in local optima. Stage-level Beam Search (Xu et al., 2024) offers a compromise by generating and selecting optimal candidates for each reasoning step rather than sentences.

3 Methodology

In this section, we first present the formal definition of LLM-based multi-step reasoning. Then, we introduce Domain01s from two aspects: enhancing reasoning capabilities and solution expansion & sampling. For aspect 1, Domain01s facilitates a progressive reasoning process. For aspect 2, Domain01s improves reasoning performance through tree search to obtain optimal reasoning paths. A comparison of reasoning examples with the base model is shown in Figure 2.

3.1 Preliminaries

For a given question \( q \), the solution process can be decomposed into multiple reasoning steps. Consider a complete solution consisting of up to \( T \) reasoning steps. The state \( S_t \) comprising all reasoning steps from step 0 to \( t \) can be represented as:

\[
S_t = \{ s_0, s_1, \ldots, s_t \}, 0 \leq t < T, t \in \mathbb{Z},
\]

where \( s_t \) represents the \( t \)-th reasoning step, state \( S_t \) represents the collection of reasoning processes from step 0 to \( t \). An action \( a_t (0 \leq t < T-1) \) is defined as choosing the next reasoning step \( s_{t+1} \). The LLM constitutes a policy model, where the transition \( f(S_{t+1}|a_t, S_t) \) from one state to the next is implemented by auto-regressively generating \( s_{t+1} \) through the input sequence. To guide the LLM in selecting more reasonable subsequent reasoning step \( s_{t+1} \), a value function \( V(s_{t+1}) \) is defined to evaluate the expected return of LLM’s strategy.

3.2 Enhancing Reasoning Capabilities

To enhance Domain01s’s reasoning capabilities in high-stakes domains (finance and legal), we employ supervised fine-tuning to let the model generate CoT-style responses. Since existing domain datasets or databases lack the detailed reasoning processes required for training Domain01s models, we constructed two new datasets, CoT-stock-2k and CoT-legal-2k, using the training sets from stock investment recommendation (Koa et al., 2024) and legal reasoning QA (Guha et al., 2024; Li et al., 2022) datasets respectively. The construction details are as follows:

Stock Investment Recommendation. Contains price data and tweet information from the top 5 stocks across 11 industries during 2020-2022. The task is to predict stock price movement (positive or negative) for the next trading day based on facts extracted from tweets over the past 5 days. Due to the high volume of daily tweets, we fine-tuned Qwen-2.5-Instruct (Qwen-Team, 2024) to generate daily tweet summaries. We utilized GPT-4o (OpenAI, 2024) to generate CoT data, explicitly prompting it to decompose the answer generation process into 10 structured reasoning steps, including market factors (Fama and French, 1993), company
Figure 3: Solution expansion & sampling illustration. Best-of-N search generates N complete responses and selects the best one; Sentence-level Beam Search generates multiple candidates for each sentence and selects the best one; Similarly, Stage-level Beam Search generates multiple candidates for each reasoning step and selects the best one. In contrast, our Selective Tree Exploration dynamically expands each reasoning step node, explores multiple reasoning steps as candidates only when necessary, and selects the best option at each step. Our method balances search performance and computational time overhead.

strategies (Porter and Kramer, 1985), and investor sentiment (Baker and Wurgler, 2006):

• Summary: Extract key facts from tweets about question q, identify main analysis focus. • Historical context: Review historical performance and market context. • Valuation: Assess current valuation metrics (e.g., P/E, price targets, market views). • Market size and dominance: Evaluate company’s industry standing and influence. • Strategic initiatives: Review recent strategic moves (partnerships, innovation) and growth potential. • Investor sentiment: Gauge investor mood through trading patterns and market discussion. • Risks and concerns: Identify key investor concerns and risk factors. • Recent performance: Analyze recent price movements and drivers. • Consolidation: Review financial/stock structure changes (buybacks, profitability). • Overall impact: Synthesize all analysis points, clearly indicate overall impact, and provide a final prediction (positive or negative) for the next trading day’s stock price.

Legal Reasoning QA. Includes legal reasoning questions across multiple categories such as legal rule application, reasoning, and legal question classification, presented as multiple choice or true/false questions. We utilized GPT-4o (OpenAI, 2024) to generate CoT data, explicitly prompting it to decompose the answer generation process into 4 structured reasoning steps:

• Summary: Extract key points from question q and identify analysis focus. • Reasoning: Apply step-by-step logic to reach answers. • Evidence: Systematically present supporting text and verify reasoning. • Conclusion: Synthesize the analysis and state the final answer.

When explicitly prompting GPT-4o to generate multiple structured reasoning steps, we require the model to use special tokens (e.g., <SUMMARY>) for segmentation. However, we want Domain0ls to organize and initiate necessary steps independently during reasoning to maintain general capabilities. Therefore, we remove all special tokens from the answers during supervised fine-tuning. After training, the model activates each reasoning step based on its own judgment.

3.3 Solution Expansion & Sampling

After supervised fine-tuning, the model can output responses in CoT format. To further enhance the model’s reasoning abilities, we enable the model to explore the solution space, and autonomously expand and sample reasoning paths. During sampling, we introduce \( V(s_{t+1}) \) to evaluate the expected return of reasoning step \( s_{t+1} \). Although \( V(s_{t+1}) \) can be constructed through direct introduction or training of step-level reward models (Chen et al., 2024a; Xie et al., 2024; Xu et al., 2024), this creates additional model training and inference overhead. In our implementation, we use a more direct but effective approach. We introduce the perplexity \( p \) of LLM when generating \( s_{t+1} \) to serve as \( V(s_{t+1}) \):

\[
p = \exp(-\frac{1}{N} \sum_{i=1}^{N} \log(\frac{e^{z_{i,k}}}{\sum_{j=1}^{M} e^{z_{i,j}}}))
\]

where \( N \) represents the number of tokens in \( s_{t+1} \), \( z_{i,k} \) is the logit value of the actually generated token \( k \) at position \( i \), \( z_{i,j} \) is the logit value of candidate token \( j \) at position \( i \), \( M \) is the vocabulary size representing the number of all candidate tokens, and \( \frac{e^{z_{i,k}}}{\sum_{j=1}^{M} e^{z_{i,j}}} \) is the softmax probability of the actually generated token. Overall, we propose Selective Tree Exploration for solution expansion & sampling, following these phases:

(1) Calculate the perplexity value \( p \) of tokens at the generation step.

(2) If \( p \geq \theta \) (\( \theta \) is the sampling threshold), regenerate the step until \( p < \theta \) or reach the maximum regeneration count \( K \) (i.e., maximum beam size). If \( p \) of all \( K \) generations are no less than threshold \( \theta \), greedily sample the candidate with minimum \( p \) from the \( K \) candidates.

(3) Continue to generate the next step based on the selected step, repeat phases (1)-(3) until the complete answer is generated.

As shown in Figure 3, compared to Best-of-N Search (Weng et al., 2022; Jiang et al., 2023),
Question: Here is a service agreement: If you are a consumer, the laws of the country in which you reside will apply to any claim, ... and you may resolve your claim in any competent court in that country that has jurisdiction over the claim. ... Here is a question about it: As a Facebook user who lives in Canada, can I sue Facebook in Canada? Answer with Yes/No.
Ground Truth: Yes.

Figure 4: The role of solution expansion & sampling. Intermediate steps in single inference (without sample) may contain errors, while solution expansion & sampling can find better reasoning paths.

Sentence-level Beam Search (Chen and Liu, 2024) and Stage-level Beam Search (Xu et al., 2024), Selective Tree Exploration balances search performance and time cost. When \( \theta \) is set to 0, Selective Tree Exploration becomes Stage-level Beam Search as it explores \( K \) paths at each step. When \( \theta \) is set to an extremely large value, Selective Tree Exploration degenerates into CoT with a single inference chain. In other cases, Selective Tree Exploration only expands reasoning paths when necessary, which reduces unnecessary overhead.

To illustrate the role of solution expansion & sampling, as shown in Figure 4, when inference without sampling, although the model generates the reasoning process, errors in intermediate steps (starting from <REASONING>) lead to error accumulation, ultimately resulting in incorrect results. Through exploration and expansion of solution paths, better reasoning paths can be found, leading to more accurate results.

4 Experiments

In this section, we evaluate the performance of Domain01s on stock investment recommendation and legal reasoning QA tasks. Our work aims to address the following questions: RQ1: How does Domain01s perform in answer accuracy compared to other LLM methods? RQ2: What are the limitations of accuracy-based evaluation metrics in domain tasks, and how can we better evaluate model performance? RQ3: How do fine-tuning and solution expansion & sampling help improve the performance of Domain01s?

4.1 Experimental Settings

Baselines. To validate Domain01s’s performance on high-stakes domain tasks, we compare it with general purpose LLMs and domain LLMs trained or fine-tuned with domain data.

General Purpose LLMs: We choose Qwen-2.5-Instruct (Qwen-Team, 2024) and Llama-3-Instruct (AI@Meta, 2024) as general purpose LLM baselines due to their remarkable performance on many downstream tasks. We also select OpenO1-Llama and OpenO1-Qwen (OpenO1 Team, 2024) as representatives of o1-type model baselines.

Financial Domain LLMs: Finance-LLM (Cheng et al., 2024c), Finance-Chat (Cheng et al., 2024c), Finance-Llama-3 (Cheng et al., 2024b), FinGPT-Forecaster (Yang et al., 2023), Llama-2-taiwan-btc (Lanz, 2024), and SEP (Koa et al., 2024).

Legal Domain LLMs: Open-Australian-Legal-LLM (Butler, 2023), DISC-LawLLM (Yue et al., 2023), Law-LLM (Cheng et al., 2024c), Law-Chat (Cheng et al., 2024c), and Lawma (Dominguez-Olmedo et al., 2024).

Datasets. For the stock investment recommendation task, we select the stock prediction dataset provided by Koa et al. (Koa et al., 2024). This dataset contains price data and tweet information for the top 5 stocks from 11 industries during 2020-2022, comprising 7,866 test question entries. The task is constructed to predict whether a stock will rise or fall on the next trading day based on facts contained in tweets from the previous 5 days. Any neutral answers are considered incorrect. Due to the large volume of daily tweets, we fine-tune Qwen-2.5-Instruct (Qwen-Team, 2024) to generate daily tweet summaries and apply these summaries as input for all models.

For the legal reasoning QA task, we select LegalBench (Guha et al., 2024), a dataset composed of numerous legal QA datasets and benchmarks. LegalBench includes 5 categories of legal tasks. We select three reasoning-related categories: Rule-application/Rule-conclusion, Interpretation, and Rhetorical-understanding, encompassing 9 datasets with a total of 35,053 test questions. Question types include true/false and multiple-choice questions.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Model size</th>
    <th colspan="5">Interpretation</th>
    <th colspan="3">Rule-application/Rule-conclusion</th>
    <th colspan="3">Rhetorical-analysis</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>CC</th>
    <th>CAUD</th>
    <th>MAUD</th>
    <th>PP</th>
    <th>IP</th>
    <th>PJ</th>
    <th>Scalr</th>
    <th>TTP</th>
    <th>TTD</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>7B</td>
    <td>86.36</td>
    <td>80.08</td>
    <td>78.75</td>
    <td>52.38</td>
    <td>48.12</td>
    <td>64.00</td>
    <td>78.98</td>
    <td>99.07</td>
    <td>76.96</td>
    <td>73.86</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>8B</td>
    <td>85.86</td>
    <td>81.20</td>
    <td>67.43</td>
    <td>61.63</td>
    <td>50.37</td>
    <td>54.00</td>
    <td>75.83</td>
    <td>100.00</td>
    <td>78.18</td>
    <td>72.72</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>8B</td>
    <td>85.10</td>
    <td>81.31</td>
    <td>74.54</td>
    <td>62.36</td>
    <td>50.37</td>
    <td>60.00</td>
    <td>80.03</td>
    <td>91.52</td>
    <td>77.58</td>
    <td>73.65</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>7B</td>
    <td>84.85</td>
    <td>80.13</td>
    <td>79.11</td>
    <td>59.27</td>
    <td>48.87</td>
    <td>66.00</td>
    <td>80.38</td>
    <td>88.78</td>
    <td>76.64</td>
    <td>73.78</td>
  </tr>
  <tr>
    <td>Open-Australian-Legal</td>
    <td>1.5B</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>1.20</td>
    <td>17.64</td>
    <td>1.50</td>
    <td>22.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.70</td>
  </tr>
  <tr>
    <td>DISC-LawLLM</td>
    <td>13B</td>
    <td>50.00</td>
    <td>32.98</td>
    <td>64.77</td>
    <td>48.09</td>
    <td>19.55</td>
    <td>56.00</td>
    <td>70.05</td>
    <td>5.60</td>
    <td>20.60</td>
    <td>40.85</td>
  </tr>
  <tr>
    <td>Law-LLM</td>
    <td>7B</td>
    <td>10.86</td>
    <td>1.59</td>
    <td>30.87</td>
    <td>3.05</td>
    <td>2.26</td>
    <td>0.00</td>
    <td>58.49</td>
    <td>8.41</td>
    <td>13.33</td>
    <td>14.32</td>
  </tr>
  <tr>
    <td>Law-Chat</td>
    <td>7B</td>
    <td>80.30</td>
    <td>82.31</td>
    <td>39.75</td>
    <td>51.69</td>
    <td>33.83</td>
    <td>48.00</td>
    <td>76.36</td>
    <td>54.21</td>
    <td>52.73</td>
    <td>57.69</td>
  </tr>
  <tr>
    <td>Lawma</td>
    <td>8B</td>
    <td>47.73</td>
    <td>34.14</td>
    <td>69.93</td>
    <td>53.31</td>
    <td>47.37</td>
    <td>36.00</td>
    <td>78.46</td>
    <td>6.54</td>
    <td>26.67</td>
    <td>44.46</td>
  </tr>
  <tr>
    <td>Domain-CoT-legal</td>
    <td>7B</td>
    <td>87.88</td>
    <td>80.59</td>
    <td>80.47</td>
    <td>65.81</td>
    <td>50.37</td>
    <td>70.00</td>
    <td>86.69</td>
    <td>94.40</td>
    <td>77.58</td>
    <td>77.09</td>
  </tr>
  <tr>
    <td>DomainO1s-legal</td>
    <td>7B</td>
    <td><b>88.64</b></td>
    <td>81.76</td>
    <td>80.33</td>
    <td><b>66.54</b></td>
    <td><b>52.63</b></td>
    <td><b>72.00</b></td>
    <td><b>88.97</b></td>
    <td>95.33</td>
    <td><b>78.78</b></td>
    <td><b>78.33</b></td>
  </tr>
</table>

Table 1: Model accuracy (%) on legal reasoning QA tasks. Avg. represents the mean accuracy across all tasks.

<table>
  <tr>
    <th>Model</th>
    <th>Model Size</th>
    <th>Accuracy</th>
    <th>MCC</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>7B</td>
    <td>51.18</td>
    <td>-0.017</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>8B</td>
    <td>51.41</td>
    <td>0.017</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>8B</td>
    <td>50.87</td>
    <td>0.014</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>7B</td>
    <td>51.02</td>
    <td>0.010</td>
  </tr>
  <tr>
    <td>Finance-LLM</td>
    <td>7B</td>
    <td>48.05</td>
    <td>-0.075</td>
  </tr>
  <tr>
    <td>Finance-Chat</td>
    <td>8B</td>
    <td>47.16</td>
    <td>-0.004</td>
  </tr>
  <tr>
    <td>Finance-Llama-3</td>
    <td>8B</td>
    <td>49.03</td>
    <td>-0.047</td>
  </tr>
  <tr>
    <td>FinGPT</td>
    <td>7B</td>
    <td>46.13</td>
    <td>0.016</td>
  </tr>
  <tr>
    <td>Llama-2-taiwan-btc</td>
    <td>7B</td>
    <td>50.66</td>
    <td>-0.002</td>
  </tr>
  <tr>
    <td>SEP</td>
    <td>7B</td>
    <td>48.35</td>
    <td>0.018</td>
  </tr>
  <tr>
    <td>Domain-CoT-finance</td>
    <td>7B</td>
    <td>51.52</td>
    <td>0.020</td>
  </tr>
  <tr>
    <td>DomainO1s-finance</td>
    <td>7B</td>
    <td><b>51.98</b></td>
    <td><b>0.021</b></td>
  </tr>
</table>

Table 2: Model accuracy (%) and MCC on stock investment recommendation tasks.

Implementation Details. In this work, our DomainO1s is developed based on Qwen-2.5-Instruct (Qwen-Team, 2024). During the fine-tuning phase for enhancing reasoning capabilities, we set the learning rate, epoch, batch size, gradient accumulation, and maximum tokens length to 5e-5, 120, 2, 2, and 2048 respectively. The \( \theta \) and \( K \) in the sampling process are set to 1.1 and 2 respectively. The experimental hardware, software, and other configuration details can be found in Appendix A.

4.2 Prediction Performance (RQ1)

In this section, we compare DomainO1s with relevant baselines to evaluate the answer accuracy.

Table 1 and Table 2 report the quantitative results for legal reasoning QA and stock investment recommendations tasks respectively. For all models where answers cannot be directly parsed from responses, we use GPT-3.5-turbo-16k (Ouyang et al., 2022) to extract the chosen options from responses for fair comparison. Additionally, given that not all stock price movements are necessarily caused by the provided text, accuracy results may not fully indicate a model’s reasoning capabilities, as they include some random guesses for non-informative text (Koa et al., 2024). Following stock prediction research (Ding et al., 2015; Feng et al., 2018), we also calculate the Matthews Correlation Coefficient (MCC) as an evaluation metric, which considers the ratios of true and false positives and negatives (Chicco and Jurman, 2020; Chicco et al., 2021). We observe that DomainO1s outperforms its base model Qwen-2.5-Instruct on almost all tasks, despite being fine-tuned on only a small amount of data. Moreover, DomainO1s and Domain-CoT (model with reasoning-enhanced fine-tuning, without solution expansion & sampling) achieve the best accuracy or MCC on nearly all tasks, even surpassing LLMs that are carefully designed and trained on domain datasets, especially on legal reasoning tasks as shown in Table 1. Although these legal LLMs learn domain knowledge through pre-training or fine-tuning, they lack the reasoning capability to derive correct answers, in contrast to our models. We also analyze the reasoning chain length and inference time of DomainO1s and baselines, see Appendix D.

4.3 Explainability Evaluation Pipeline (RQ2)

In previous research, most domain tasks use accuracy as the primary evaluation metric (Koa et al., 2024; Yang et al., 2022; Guha et al., 2024). This evaluation metric makes it difficult to distinguish between models that truly understand and reasonably utilize context and those that simply rely on partial text or overfit on pre-trained domain knowledge (Zhang et al., 2024; Bordt et al., 2024). We sample two subsets from the test sets of stock investment recommendation and legal reasoning QA, with details available in Appendix F.

In the stock investment recommendation task, stock tweets are manually classified into Positive
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Finance-Llama-3</th>
    <th colspan="3">Domaino1s-finance</th>
  </tr>
  <tr>
    <th>Tweets</th>
    <th>Response</th>
    <th>Pos.</th>
    <th>Neg.</th>
    <th>Acc.</th>
    <th>Pos.</th>
    <th>Neg.</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td>0.1</td>
    <td>0.9</td>
    <td>0.018</td>
    <td>0.982</td>
    <td>50.45</td>
    <td>0.173</td>
    <td>0.827</td>
    <td>50.91</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>0.7</td>
    <td>0.182</td>
    <td>0.818</td>
    <td>48.64</td>
    <td>0.391</td>
    <td>0.609</td>
    <td>49.55</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.5</td>
    <td>0.545</td>
    <td>0.455</td>
    <td>48.18</td>
    <td>0.527</td>
    <td>0.473</td>
    <td>49.09</td>
  </tr>
  <tr>
    <td>0.7</td>
    <td>0.3</td>
    <td>0.882</td>
    <td>0.118</td>
    <td>51.36</td>
    <td>0.777</td>
    <td>0.223</td>
    <td>50.91</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td>0.1</td>
    <td>1.000</td>
    <td>0.000</td>
    <td>50.00</td>
    <td>0.882</td>
    <td>0.118</td>
    <td>50.45</td>
  </tr>
</table>

Table 3: Model’s accuracy and prediction ratios for Positive (Pos.) and Negative (Neg.) of the stock trend under varying proportions of Pos. and Neg. tweets in stock investment prediction tasks.

and Negative tweets and combined in different ratios as model inputs. We compare Domaino1s-finance with Finance-Llama-3. As shown in Table 3, when the Positive:Negative ratio of tweets is 0.5:0.5, models’ responses maintain a similar 1:1 ratio between Positive and Negative predictions. However, when either Positive or Negative tweets dominate the input, Finance-Llama-3 typically ignores tweets with the opposite sentiment and bases its answer solely on the majority sentiment. In contrast, Domaino1s-finance still considers the minority sentiment tweets and generates answers by comprehensively evaluating all tweets. However, both models achieve similar accuracy, making it challenging to determine through accuracy metrics alone whether the models truly understand and reasonably utilize the context in the inputs, rather than overfitting or hallucinating. For the legal reasoning task subset, key conditions are removed from the question text, making it impossible to answer the tasks correctly. As shown in Figure 5, although Law-Chat achieves higher accuracy than Domaino1s-legal, its answers are mostly random responses generated from overfitted legal knowledge, while Domaino1s-legal refuses to answer due to the absence of necessary reasoning conditions, resulting in an accuracy close to 0. This indicates that accuracy alone is insufficient to determine whether models blindly overfit using domain knowledge to generate irrelevant answers.

For high-stakes domain tasks such as stock investment recommendations and legal reasoning QA, non-transparent text comprehension or inadequate reasoning processes may lead to wrong conclusions or generate advice that violates ethical or legal principles. To evaluate the explainability of domain model responses, we propose a new evaluation metric called PROOF-Score (Principled rating for reasoning completeness, domain safety, and factual accuracy). PROOF-Score uses GPT-4o (OpenAI, 2024) to generate a score from 1 to 7 for response, considering three aspects:
• Reasoning Completeness (RC): Evaluates the completeness and logical coherence.
• Domain Safety (DS): Measures the safety and appropriateness in specific domains.
• Factual Accuracy (FA): Evaluates the factual accuracy of statements.
Detailed prompts can be seen in Appendix E. Here, we define:

\[
\text{PROOF-Score} = \frac{RC + DS + FA}{3}.
\]

Table 4 shows PROOF-Scores of models on two tasks. Domaino1s achieves the highest scores on both tasks, even though we do not train specifically for these three metrics. This indicates Domaino1s can inherently consider these factors to generate better responses. We also observe that even when a model’s response is incorrect in terms of results, GPT-4o may still give a high PROOF-Score because these responses contain clear and reasonable logic. This may be inappropriate for tasks requiring strict accuracy, where prediction accuracy should be considered the primary metric. However, for tasks lacking standard answers or without unique correct answers (e.g., long-term investment advice,
<table>
  <tr>
    <th>Method</th>
    <th>Acc</th>
    <th>time(s)</th>
  </tr>
  <tr>
    <td>w/o Sample</td>
    <td>86.69</td>
    <td>8.35</td>
  </tr>
  <tr>
    <td>Best-of-N Search</td>
    <td>87.56</td>
    <td>40.26</td>
  </tr>
  <tr>
    <td>Sentence-level Beam Search</td>
    <td>84.93</td>
    <td>334.20</td>
  </tr>
  <tr>
    <td>Stage-level Beam Search</td>
    <td>88.44</td>
    <td>133.68</td>
  </tr>
  <tr>
    <td><b>Selective Tree Exploration</b></td>
    <td><b>89.14</b></td>
    <td><b>15.18</b></td>
  </tr>
</table>

Table 5: Accuracy (%) and average inference time comparison between our Selective Tree Exploration and other search methods on the Scalar dataset. Our method (with \( \theta = 1.1 \)) outperforms other approaches under the same beam size settings.

<table>
  <tr>
    <th>Method</th>
    <th>K</th>
    <th>Acc</th>
    <th>time(s)</th>
  </tr>
  <tr>
    <td>w/o Sample</td>
    <td>1</td>
    <td>86.69</td>
    <td>8.35</td>
  </tr>
  <tr>
    <td rowspan="4">Selective Tree Exploration</td>
    <td>2</td>
    <td>88.97</td>
    <td>24.88</td>
  </tr>
  <tr>
    <td>3</td>
    <td>89.14</td>
    <td>45.77</td>
  </tr>
  <tr>
    <td>4</td>
    <td>89.84</td>
    <td>72.55</td>
  </tr>
  <tr>
    <td>5</td>
    <td>90.01</td>
    <td>93.95</td>
  </tr>
</table>

Table 6: Accuracy (%) and average inference time of Domain01s-legal on the Scalar dataset under different beam size K settings. \( \theta \) is set to 1.05.

asset allocation recommendations), using PROOF-Score becomes effective in evaluating the explainability of model responses.

4.4 Ablation Study (RQ3)

In this section, we evaluate the impact of fine-tuning and solution expansion & sampling on Domain01s’s performance. We primarily focus on accuracy metrics in this section, while presenting explainability analysis in Appendix G.

Enhancing Reasoning Fine-tuning. As shown in Table 1 and Table 2, Domain-CoT represents the model configuration using only reasoning-enhanced fine-tuning without solution expansion & sampling. Compared to the base model Qwen-2.5-Instruct, Domain-CoT achieves performance improvements on almost all datasets, which demonstrates that reasoning-enhanced fine-tuning improves the model’s reasoning capabilities on domain tasks.

Solution Expansion & Sampling. Table 5 shows the performance comparison on Scalar (a dataset in LegalBench) between best-of-N search (Weng et al., 2022; Jiang et al., 2023), Sentence-level Beam Search (Chen and Liu, 2024), Stage-level Beam Search (Xu et al., 2024), and our Selective Tree Exploration. The baseline search methods use the setup from Xu et al. (Xu et al., 2024), which uses the policy model to evaluate the relative quality of reasoning chains or steps, in contrast to our perplexity-based approach. Results demonstrate that under the same beam setting of \( K = 3 \), Selective Tree Exploration achieves comparable or better performance compared to all baseline approaches (with and without search) while requiring less computational time for inference than other search methods.

To better illustrate the effectiveness of our Selective Tree Exploration as exploration paths increase, we evaluate model performance under different settings of \( K \) and \( \theta \) on the Scalar dataset. As shown in Table 6, using Selective Tree Exploration brings performance improvements compared to methods without sampling (\( K = 1 \)). Model accuracy improves as \( K \) increases, indicating that our Selective Tree Exploration is scalable. As shown in Table 7, model accuracy improves as \( \theta \) decreases, as this similarly expands the paths explored by Selective Tree Exploration. However, both increasing \( K \) and decreasing \( \theta \) lead to longer inference time. Due to computational resource constraints, we only set \( K = 2, \theta = 1.1 \). However, we demonstrate that increasing beam size \( K \) and decreasing sampling threshold \( \theta \) will lead to performance improvements.

<table>
  <tr>
    <th>Method</th>
    <th>\( \theta \)</th>
    <th>Acc</th>
    <th>time(s)</th>
  </tr>
  <tr>
    <td>w/o Sample</td>
    <td>10000</td>
    <td>86.69</td>
    <td>8.35</td>
  </tr>
  <tr>
    <td rowspan="4">Selective Tree Exploration</td>
    <td>1.4</td>
    <td>87.21</td>
    <td>8.40</td>
  </tr>
  <tr>
    <td>1.3</td>
    <td>87.91</td>
    <td>8.44</td>
  </tr>
  <tr>
    <td>1.2</td>
    <td>88.97</td>
    <td>10.16</td>
  </tr>
  <tr>
    <td>1.1</td>
    <td>89.14</td>
    <td>15.18</td>
  </tr>
  <tr>
    <td></td>
    <td>1.0</td>
    <td>89.49</td>
    <td>51.03</td>
  </tr>
</table>

Table 7: Accuracy (%) and average inference time of Domain01s-legal on the Scalar dataset under different sampling threshold \( \theta \) settings. \( K \) is set to 3.

5 Acknowledgment

This work is supported by the National Key R&D Program of China [2022YFF0902703].

6 Conclusion & Future Works

In this work, we introduce Domain01s and its two model variants for finance and legal domains, guiding LLMs towards explainable high-stakes domain answers. We construct two datasets to fine-tune Qwen-2.5-Instruct and propose Selective Tree Exploration for enabling LLMs to perform multi-stage reasoning. The superior performance on datasets demonstrates Domain01s’s exceptional potential in high-stakes domains.

In future work, we plan to build larger training datasets to enhance domain models’ reasoning abilities. We also plan to create Domain01s variants
using domain-specific pre-trained base models to better solve tasks requiring domain expertise.

7 Limitations

Despite the promising results achieved by Domain01s, there are some limitations. First, while our Selective Tree Exploration method effectively balances search performance and computational costs, the additional inference time required for tree exploration may impact the model’s real-time application scenarios, such as in situations requiring high response speed. Second, although we construct high-quality CoT datasets using GPT-4o, the relatively small size of training data (2,000 examples each for finance and legal domains) may limit the model’s ability to handle extremely rare or complex domain-specific cases. Additionally, while PROOF-Score provides a comprehensive evaluation framework, research on using LLMs as judges suggests that further refinement and elaboration of evaluation metrics may be beneficial (Gu et al., 2024). Finally, our current implementation focuses on stock recommendation and legal reasoning tasks, and the generalizability of our approach to other domain applications requires further investigation. These limitations point to promising directions for future research, such as optimizing inference efficiency, expanding training datasets, and extending the framework to broader domain applications.

8 Ethical Considerations

8.1 Fairness and Accessibility

We recognize that the computational resources required for training and inference of large language models (LLMs) and tree search exploration may limit accessibility for researchers and practitioners with fewer resources. To address this, we will open-source our implementation and provide efficient variants that can run on consumer-grade hardware. Additionally, we will release the training datasets (CoT-stock-2k and CoT-legal-2k) to enhance reproducibility and facilitate broader participation in this research direction.

8.2 Potential Risks in Financial and Legal Applications

For financial applications, we acknowledge that Domain01s-finance’s advice, while explainable, should be viewed as restricted investment references. To mitigate potential risks:

• We explicitly state that Domain01s-finance’s outputs should serve as one of many considerations when users make actual investment decisions.
• We implement safety checks in the Domain Safety (DS) metric of PROOF-Score to detect potentially harmful or high-stake advice.
• We emphasize the importance of human oversight and professional judgment in interpreting model reasoning.

For legal applications, Domain01s-legal is intended to assist rather than replace legal professionals. To mitigate potential risks:
• We explicitly state that Domain01s-legal is proposed as a support tool rather than a substitute for professional legal advice.
• We detect responses that contradict legal facts by evaluating the Factual Accuracy (FA) metric of PROOF-Score.
• We emphasize the importance of human oversight and professional judgment in interpreting model reasoning.

8.3 Privacy and Data Security

We have taken multiple measures to protect privacy and ensure data security:
• Our datasets have been carefully screened and curated to exclude sensitive personal information.
• The model’s inference process is designed to focus on public information.
• Implement rate limiting and access controls after model and dataset open-sourcing to prevent potential misuse.

8.4 Environmental Impact

We acknowledge the environmental impact of training and running large language models. To minimize this:
• Our proposed Selective Tree Exploration method is designed to improve computational efficiency and reduce inference overhead.
• We provide guidance on optimal hyperparameter settings and encourage the selection of hyperparameter configurations that balance computational costs with model performance to reduce unnecessary computation.

Through these considerations and safeguards, we aim to ensure Domain01s makes positive contributions to the field while minimizing potential risks and negative impacts. We encourage ongoing dialogue with stakeholders and welcome community
feedback to further improve the ethical implementation of our technology.

References

AI@Meta. 2024. The llama 3 herd of models. ArXiv, abs/2407.21783.

Malcolm Baker and Jeffrey Wurgler. 2006. Investor sentiment and the cross-section of stock returns. The journal of Finance, 61(4):1645–1680.

Or Biran and Kathleen R. McKeown. 2017. Human-centric justification of machine learning predictions. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pages 1461–1467. ijcai.org.

Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Bemirna Nushi, and Rich Caruana. 2024. Elephants never forget: Memorization and learning of tabular data in large language models, 2024. URL https://arxiv.org/abs/2404.06209.

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. In Submitted to The Thirteenth International Conference on Learning Representations. Under review.

Umar Butler. 2023. Open australian legal llm.

Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024a. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553.

Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024b. Step-level value preference optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7889–7903, Miami, Florida, USA. Association for Computational Linguistics.

Zheng Chen and Zhejun Liu. 2024. Sentence-level heuristic tree search for long text generation. Complex & Intelligent Systems, 10(2):3153–3167.

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024a. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550, Miami, Florida, USA. Association for Computational Linguistics.

Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024b. Instruction pre-training: Language models are supervised multitask learners. arXiv preprint arXiv:2406.14491.

Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024c. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations.

Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. 2024. (a) i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2454–2469.

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937.

Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21:1–13.

Davide Chicco, Niklas Tötsch, and Giuseppe Jurman. 2021. The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData mining, 14:1–22.

Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, et al. 2024. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883.

Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR, abs/2306.16092.

Michael Desmond, Zahra Ashktorab, Qian Pan, Casey Dugan, and James M Johnson. 2024. Evalllum: Llm assisted evaluation of generative outputs. In Companion Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 30–32.

Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock prediction. In Twenty-fourth international joint conference on artificial intelligence.

Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna Gummadri, Moritz Hardt, and Michael Livermore. 2024. Lawma: The power of specialization for legal tasks. arXiv preprint arXiv:2407.16615.

Eugene F Fama and Kenneth R French. 1993. Common risk factors in the returns on stocks and bonds. Journal of financial economics, 33(1):3–56.

Fuli Feng, Huimin Chen, Xianguan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. 2018. Enhancing stock movement prediction with adversarial training. arXiv preprint arXiv:1810.09936.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambreno, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36.

Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report. arXiv preprint arXiv:2305.15062.

Cheonsu Jeong. 2024. Fine-tuning and utilization methods of domain-specific llms. arXiv preprint arXiv:2401.02981.

Cong Jiang and Xiaolei Yang. 2023. Legal syllogism prompting: Teaching large language models for legal judgment prediction. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 417–421.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhui Chen. 2023. Tigerscore: Towards building explainable metric for all text generation tasks. Transactions on Machine Learning Research.

Kelvin JL Koa, Yunshan Ma, Ritchie Ng, and Tat-Seng Chua. 2024. Learning to generate explainable stock predictions using self-reflective large language models. In Proceedings of the ACM on Web Conference 2024, pages 4304–4315.

Yannis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, Bangkok, Thailand. Association for Computational Linguistics.

David Lanz. 2024. Llama2-7b-taiwan-btc-qlora. https://huggingface.co/DavidLanz/llama2_7b_taiwan_btc_qlora.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36.

Jonathan Li, Rohan Bhamphoria, and Xiaodan Zhu. 2022. Parameter-efficient legal domain adaptation. arXiv preprint arXiv:2210.13712.

Xiang Li, Zhenyu Li, Chen Shi, Yong Xu, Qing Du, Mingkui Tan, and Jun Huang. 2024b. AlphaFin: Benchmarking financial analysis with retrieval-augmented stock-chain framework. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 773–783, Torino, Italia. ELRA and ICCL.

Yuxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. ArXiv, abs/2303.14070.

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tamnoy Chowdhury, Yun-Qing Li, Hejie Cui, Tian Yu Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Carl Yang, and Liang Zhao. 2023. Beyond one-model-fits-all: A survey of domain specialization for large language models. ArXiv, abs/2305.18703.

Qing Lyu, Shreya Havaladar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.

Jing Miao, Charat Thongprayoon, Supawadee Supapadungsuk, Pajaree Krisananap, Yeshwant Radhakrishnan, and Wisit Cheungpasitporn. 2024. Chain of thought utilization in large language models and application in nephrology. Medicina, 60(1):148.

OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

OpenAI. 2024. Learning to reason with llms. [Accessed 19-09-2024].

OpenO1 Team. 2024. Open-o1: A model matching proprietary power with open-source innovation. https://github.com/Open-Source-O1/Open-O1. [Accessed 19-11-2024].

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Michael E Porter and Mark R Kramer. 1985. Advantage creating and sustaining superior performance. Competitive Advantage; Harvard Business School Publishing: Brighton, MA, USA.

Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195.

Chuan Qin, Jun Chang, Wenting Tu, and Changrui Yu. 2024. Followakoinvestor: Stock recommendation by hearing voices from all kinds of investors with machine learning. Expert Systems with Applications, 249:123522.

Qwen-Team. 2024. Qwen2.5: A party of foundation models.

Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah. 2020. Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8415–8426.
Takehiro Takayanagi, Kiyoshi Izumi, Atsuo Kato, Naoyuki Tsunedomi, and Yukina Abe. 2023. Personalized stock recommendation with investors’ attention and contextual information. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3339–3343.

Utkarsh Ujwal, Sai Sri Harsha Surampudi, Sayantan Mitra, and Tulika Saha. 2024. "reasoning before responding": Towards legal long-form question answering with interpretability. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4922–4930.

Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023a. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975.

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, Bangkok, Thailand. Association for Computational Linguistics.

Steven Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dmitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. 2023b. MAUD: An expert-annotated legal NLP dataset for merger agreement understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16369–16382, Singapore. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561.

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabrovolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2024. Bloomberggpt: A large language model for finance, 2023. ArXiv preprint: https://arxiv.org/pdf/2303.17564.pdf.

Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023a. The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges. arXiv preprint arXiv:2304.05351.

Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023b. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443.

Yuxi Xie, Anirudh Goyal, Wenye Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451.

Frank Xing. 2024. Designing heterogeneous llm agents for financial sentiment analysis. ACM Transactions on Management Information Systems.

Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. 2024. Llava-cot: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440.

Yumo Xu and Shay B Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970–1979.

Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031.

Linyi Yang, Jiazheng Li, Ruihai Dong, Yue Zhang, and Barry Smyth. 2022. Numhtml: Numeric-oriented hierarchical transformer model for multi-task financial forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11604–11612.

Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chencheng Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, et al. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:2309.11325.

Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. 2024. Uncovering overfitting in large language model editing. arXiv preprint arXiv:2410.07819.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.

Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
A  Experimental Setup

All experiments are conducted using an AMD EPYC 7H12 64-Core processor as CPU and four 48GB NVIDIA RTX 6000 Ada GPUs. For each variant of Domaino1s, fine-tuning takes approximately 48 GPU hours per run. The system environment uses CUDA version 12.4, Python version 3.10.15, PyTorch version 2.5.1, and transformers version 4.45.2. The random seed is set to 42.

We employ LoRA (Low-Rank Adaptation) for fine-tuning. The base model is Qwen2.5-7B-Instruct. We use the qwen template with Flash Attention enabled. The training dataset is preprocessed using 16 workers with a maximum sequence length of 2,048 tokens.

The LoRA hyperparameters are set as follows: rank = 8, alpha = 16, and dropout = 0, targeting all model layers. For optimization, we use the AdamW optimizer with a learning rate of 5e-5 and cosine learning rate scheduling. The training runs for 120 epochs. We employ mixed-precision training using bfloat16 format.

The batch size is set to 2 per device with a gradient accumulation of 2 steps, effectively creating a batch size of 16 (2 × 2 × 4 GPUs). Gradient clipping is applied with a maximum norm of 1.0. The model checkpoints are saved every 100 steps, with loss logging occurring every 5 steps.

B  CoT Data Generation

Figure 6 and Figure 7 are the prompt templates for instructing GPT-4o to generate responses in CoT format.

Figure 8 is the prompt template for instructing GPT-4o and Qwen-2.5-Instruct to generate tweet summaries.

C  Answer Demonstration

Figures 9 and 10 demonstrate complete question-answering examples for stock investment recommendation and legal reasoning QA tasks using Domaino1s and the base model Qwen-2.5-Instruct. Domaino1s does not explicitly output special tokens (e.g., <SUMMARY>), but reason according to the structured reasoning process constructed in the CoT-stock-2k and CoT-legal-2k datasets.

As shown in Figure 9, Qwen-2.5-Instruct reaches an incorrect answer by focusing only on partial information (the Negative parts) while ignoring the overall context. In contrast, Domaino1s-finance

<table>
  <tr>
    <th>Prompt for stock data generation</th>
  </tr>
  <tr>
    <td>For a given set of facts, estimate their overall impact on [stock_name]'s stock price movement. You need to generate intermediate steps following this process to arrive at the correct answer:<br><br>&lt;SUMMARY&gt; Summarize all facts, clearly identify the core focus and research subject of the analysis, outline the overall background and target scenario. Also indicate what the following analysis should focus on.&lt;/SUMMARY&gt;<br><br>&lt;HISTORICAL CONTEXT&gt; Analyze the impact of historical background and long-term performance, especially regarding past stock performance and market environment. &lt;HISTORICAL CONTEXT&gt;<br><br>&lt;EVALUATIONS&gt; Analyze current stock valuation levels and their impact on investor decisions. Include P/E ratios, analyst price targets, and market views on valuation. &lt;EVALUATIONS&gt;<br><br>&lt;MARKET SIZE AND DOMINANCE&gt; Emphasize the company's position and scale in the industry or market, analyze its market influence. &lt;MARKET SIZE AND DOMINANCE&gt;<br><br>&lt;STRATEGIC INITIATIVES&gt; Analyze recent strategic actions, including partnerships, innovations, patents, and their potential impact on future growth. Discuss major technological, business, or collaborative actions. Do recent strategic actions support long-term growth? Do these actions enhance company competitiveness? &lt;STRATEGIC INITIATIVES&gt;<br><br>&lt;INVESTOR SENTIMENT&gt; Analyze investors sentiment towards the stock, including trading activity, options performance, and market discussions. Discuss investor behavior (such as trading volume, attention) and market sentiment. &lt;INVESTOR SENTIMENT&gt;<br><br>&lt;RISKS AND CONCERNS&gt; Analyze risk factors currently concerning investors, including high valuations, market volatility, or external risks. &lt;RISKS AND CONCERNS&gt;<br><br>&lt;RECENT PERFORMANCE&gt; Analyze recent stock price performance and its driving factors, including price milestones, short-term trends, and market performance. &lt;RECENT PERFORMANCE&gt;<br><br>&lt;CONSOLIDATIONS&gt; Analyze adjustments in company finances or stock structure, such as stock buybacks, profitability, or long-term return trajectory. &lt;CONSOLIDATIONS&gt;<br><br>&lt;OVERALL IMPACTS&gt; Synthesize all analysis points, clearly indicate the direction of overall impact (positive or negative, cannot be neutral), and provide final conclusions. Give your response in this format: Price Movement: Positive/Negative. &lt;OVERALL IMPACTS&gt;<br><br>Here are the facts.<br>[facts]<br><br>In your generation process, do not use '&lt;', '&gt;', or '—'. Only respond using the above special tags and their generated content in this format:</td>
  </tr>
</table>

Figure 6: Prompt template for stock investment recommendation.

<table>
  <tr>
    <th>Prompt for legal data generation</th>
  </tr>
  <tr>
    <td>Given a legal question, you need to follow this process to generate intermediate steps and arrive at the correct answer:<br><br>&lt;SUMMARY&gt; Summarize the materials, text, or paragraphs provided in the question. Also identify what the subsequent analysis needs to focus on in order to answer the question&lt;/SUMMARY&gt;<br><br>&lt;REASONING&gt; Generate step-by-step analysis to answer the question based on the provided text/&lt;REASONING&gt;<br><br>&lt;EVIDENCE&gt; List evidence from the presented text that supports the reasoning, and reflect on whether the answers in the reasoning section are correct&lt;/EVIDENCE&gt;<br><br>&lt;CONCLUSION&gt; Provide a brief summary and draw the final conclusion&lt;/CONCLUSION&gt;<br><br>Here are the question:<br>[question]<br><br>In your generation process, do not use '&lt;', '&gt;', or '—'. Only respond using the above special tags and their generated content in this format:</td>
  </tr>
</table>

Figure 7: Prompt template for legal reasoning QA.

<table>
  <tr>
    <th>Prompt for tweets summary generation</th>
  </tr>
  <tr>
    <td>Below are multiple tweets about [stock] from [date], separated by '---':<br>[--- ... ]<br><br>Please provide a brief, factual summary of the key events, news, and developments mentioned in these tweets. Focus only on objective facts and events, without any analysis or sentiment. Format the summary as bullet points. Keep each point concise and avoid repetition.<br><br>Requirements:<br>1. Only include events and facts that were explicitly mentioned<br>2. Remove any duplicated information<br>3. Keep each bullet point under 15 words<br>4. Do not include opinions, analysis, or market sentiment<br>5. Focus on company/stock related events only<br>You only need to generate your summary.</td>
  </tr>
</table>

Figure 8: Prompt template for tweet summarization.
<table>
  <tr>
    <th>Model</th>
    <th>time(s)</th>
    <th>Length</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>6.91</td>
    <td>166.2</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td><b>5.08</b></td>
    <td>95.2</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>20.27</td>
    <td>454.8</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>21.09</td>
    <td>465.1</td>
  </tr>
  <tr>
    <td>Finance-LLM</td>
    <td>14.70</td>
    <td>131.3</td>
  </tr>
  <tr>
    <td>Finance-Chat</td>
    <td>14.52</td>
    <td>130.8</td>
  </tr>
  <tr>
    <td>Finance-Llama-3</td>
    <td>5.34</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>FinGPT</td>
    <td>6.29</td>
    <td>14.2</td>
  </tr>
  <tr>
    <td>Llama-2-taiwan-btc</td>
    <td>13.63</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td>SEP</td>
    <td>13.182</td>
    <td>119.6</td>
  </tr>
  <tr>
    <th>Domain-CoT-finance</th>
    <td>18.37</td>
    <td><b>512.1</b></td>
  </tr>
  <tr>
    <th>Domaino1s-finance</th>
    <td>27.38</td>
    <td>509.8</td>
  </tr>
</table>

Table 8: Inference time and reasoning chain length on stock investment recommendation tasks.

<table>
  <tr>
    <th>Model</th>
    <th>time(s)</th>
    <th>Length</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>0.65</td>
    <td>1.3</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>0.71</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>8.82</td>
    <td>261.2</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>9.53</td>
    <td>265.9</td>
  </tr>
  <tr>
    <td>Open-Australian-Legal</td>
    <td>8.33</td>
    <td>263.5</td>
  </tr>
  <tr>
    <td>DISC-LawLLM</td>
    <td>2.38</td>
    <td>9.8</td>
  </tr>
  <tr>
    <td>Law-LLM</td>
    <td>4.18</td>
    <td>97.4</td>
  </tr>
  <tr>
    <td>Law-Chat</td>
    <td>0.64</td>
    <td>1.2</td>
  </tr>
  <tr>
    <td>Lawma</td>
    <td><b>0.63</b></td>
    <td>1.0</td>
  </tr>
  <tr>
    <th>Domain-CoT-finance</th>
    <td>8.17</td>
    <td>268.5</td>
  </tr>
  <tr>
    <th>Domaino1s-legal</th>
    <td>13.54</td>
    <td><b>269.8</b></td>
  </tr>
</table>

Table 9: Inference time and reasoning chain length on legal reasoning QA tasks.

comprehensively considers both Positive and Negative facts to draw conclusions. As shown in Figure 10, Qwen-2.5-Instruct starts making reasoning errors after generating "However," incorrectly classifying the user as a non-consumer, leading to an incorrect result. In comparison, Domaino1s-legal avoids errors through structured reasoning paths and tree search.

D Answer Length and Inference Time

In this section, we present the reasoning chain length and inference time of Domaino1s and baselines in generating answers for stock investment recommendations and legal reasoning QA tasks. The reasoning chain length is measured by the average number of words rather than tokens in the responses to ensure fair comparison across different models. As shown in Table 8 and Table 9, o1-type models (OpenO1-Llama, OpenO1-Qwen, and our Domaino1s) have longer reasoning chains than other baselines, among which our Domaino1s and Domain-CoT have the longest reasoning chains. Although Domaino1s exhibits longer inference time compared to the baselines, this is attributed to its generation of longer and higher-quality reasoning chains and the search for optimal reasoning paths, ultimately leading to superior accuracy metrics.

E PROOF-Score Generation

Figure 11 is the prompt template for instructing GPT-4o to generate PROOF-Scores.

F Construction of Sub-datasets

We select 8 data points from each of the 55 stocks in the stock investment recommendation dataset. Each data point contains more than 10 positive tweets and 10 negative tweets. We construct five groups of tweets, with each group containing 10 tweets but different ratios of positive to negative tweets: 1:9 (1 positive, 9 negative), 3:7 (3 positive, 7 negative), 5:5 (5 positive, 5 negative), 7:3 (7 positive, 3 negative), and 9:1 (9 positive, 1 negative). Figure 12 shows a construction example and model responses. The example has a Positive:Negative ratio of 7:3. Tweets with yellow background are positive, while those with blue background are negative. Finance-Llama-3's response only considers the positive tweets, completely ignoring the negative ones. In contrast, Domaino1s-finance considers both positive and negative tweets to arrive at the correct answer.

We extract 500 questions from the legal reasoning QA dataset. Each question contains key conditions necessary for answering the question. We remove these key conditions from the questions, making them impossible to answer. Figure 13 shows a construction example.

G Explainability Analysis

Table 10 demonstrates the ablation experiments of PROOF-Score without fine-tuning (w/o Fine-tune) and without solution expansion & sampling (w/o Sample). In the w/o Fine-tune experiment, we use the Qwen-2.5-Instruct model without fine-tuning on our data and prompt it to separate each step with "\n" to facilitate our solution expansion & sampling. The results indicate that the PROOF-Score of the model without fine-tuning is lower than Domaino1s, demonstrating that Domaino1s learns to generate superior-quality reasoning processes from our constructed high-quality fine-tuning datasets. Meanwhile, the PROOF-Score of the model without solution expansion & sampling is similar to Domaino1s, which suggests that the
Question

For a given set of facts, estimate their overall impact on AAPL’s stock price movement. Answer with Positive/ Negative. Here are the facts:
- Bank of America downgrades Apple (AAPL) from “Buy” to “Neutral.”
- Rosenblatt raises $AAPL price target to $189 from $160 and upgrades to Buy.
- Apple’s market cap falls over 20% in one day.
- Apple’s market cap falls, contributing to the S&P 500 staying positive while Apple trades negative.
- Apple’s CEO warned about potential demand concerns.
- Apple’s stock price hit a new low after the viral TikTok incident.
- Apple’s stock hits a new low of the day.
- Apple’s share of the market reaches a record low.
- Apple’s stock price falls due to lack iPhone users without GPS.
- Apple downgraded to “sell” by Jefferies.
- Apple given a “strong buy” rating by JPMorgan.
- Apple’s stock price falls due to supply chain issues.
- Apple’s stock options trading activity, with significant put volume.
- Apple’s stock included in a 10% decline pattern of FAANG stocks.
- Apple’s stock price falls due to increased short selling activity.
- Apple targeted in discussions about potential short squeeze.

2022-09-30
- Apple stock entered bear market territory on September 30, 2022.
- Apple is down 22% year-to-date; meta down 60%, amazon down 34%.
- Apple’s stock price fell to a new low of the day.
- Apple fired its long-serving procurement head due to a TikTok incident.
- Apple’s stock declined 4.91% on September 30, 2022.
- Apple’s stock price fell to a new low after the viral TikTok incident.
- Apple’s Korea offices were raided by antitrust officials.
- Large trades occurred in Apple: $1 billion plus.
- Apple’s stock price fell to a new low of the day.
- Apple’s 14 Pro and 14 Mix demand reports were mentioned.
- Apple’s stock price broke a ten-dollar threshold.
- Apple’s stock price fell due to supply chain issues.
- Apple’s stock failed to protect key support levels.
- Apple’s stock is down 3% from previous closing price.
- Apple’s stock price fell due to increased negative factors.
- Apple’s stock showed algo-driven patterns.
- Apple’s price target for next week was highlighted as a top analyst estimate.

2022-10-01
- Rosenblatt re-identifies AAPL as strong stock for trading.
- AAPL reported among top stocks by mention on trading day of October 1, 2022.
- Jim Cramer associated with AAPL’s stock performance issues.
- AAPL ranked sixth in year-to-date stock performance list.
- Apple’s stock price fell to a new low of the day.
- AAPL targeted in live day-trading room offers.
- A wave 5 throw-over pattern identified for AAPL.
- Apple’s stock price fell to a new low of the day the following week.
- AAPL stock showed a significant price drop, entering bear market territory.
- AAPL’s head of procurement fired due to TikTok incident.
- Apple’s stock price fell due to increased negative factors.
- Apple mentioned in discussions around most overvalued stocks.
- Rosenblatt raised AAPL’s stock price target on strong demand for devices.
- AAPL included in option trade highlights with significant percentage increase.

2022-10-02
- Apple’s Test Drive registration is open for trading tools.
- Last week, SAAPL had an 8.13% weekly decline since March 2020.
- SAAPL was downgraded by Bank of America.
- SAAPL’s stock price fell to a new low of the day.
- SAAPL production cuts were reported alongside SNKE inventory builds.
- SAAPL’s stock and options saw significant positive gains in options trades.
- SAAPL’s stock price fell to a new low of the day during Stage 4.
- SAAPL hit extreme oversold conditions for the first time in months.
- SAAPL’s last large trade was on September 23, 2022.
- Apple’s stock price fell to a new low of the day.
- Free trials and stock options alerts available on Discord.
- SAAPL included in discussions of potential stock market bottom scenarios.

2022-10-03
- Bank of America downgrades all Big Tech stocks, with Apple (AAPL) experiencing a significant decline.
- $AAPL saw a 8% drop into quarter end, unique among US Big Tech names.
- Apple’s September App Store net revenue fell 5% year-over-year.
- TSMC is raising prices for all clients, including Apple.
- Apple (AAPL) option activity included multiple calls and puts with high open interest.
- Apple (AAPL) option activity included multiple calls and puts with high open interest.
- Apple (AAPL) hit new 12-week lows as of October 3, 2022.
- Apple (AAPL) option activity included multiple calls and puts with high open interest.
- Apple (AAPL) calls were active going into the week of October 3, 2022.
- Apple (AAPL) included in various trading watches and options lists.
- Apple (AAPL) hit price targets and saw analyst price adjustments.

Qwen-2.5-Instruct Answer

The overall impact of the given facts on AAPL’s stock price movement is predominantly negative. The facts indicate a series of downgrades, warnings about demand, negative market reactions, and other unfavorable factors, such as stock price hitting new lows and dragging market performance. These all had a negative impact on AAPL’s stock price.

Negative

Domain1s-finance Answer

The analysis focuses on Apple’s stock performance amid a series of operational, strategic, and market responses impacting its valuation and investor sentiment as of late September to early October 2022. The broader tech sector has faced challenges, particularly in relation to surging inflation, supply chain issues, and location tracking technology, and investor reactions around big tech stock declines. We will examine several factors to drive the analysis, including historical performance, valuation metrics, market position, strategic initiatives, investor sentiment, risks, and recent performance.

Historically, Apple’s stock has shown resilience but faced challenges this year, down 22% by mid-September, amid broader tech sector weakness. The inclusion of Apple as a key player in the tech sector has persisted despite setbacks. The inclusion of Big Tech segments typically volatile, weighs on its performance. Historically, Apple has shown capability for strong comebacks, driven by product cycles and strategic positioning. However, this period aligns during a broader tech market decline.

Apple’s stock has been targeted for its high valuation issues, with fluctuating analyst price targets reflecting mixed sentiment. Analysts gave mixed ratings with Rosenblatt an upgrade, Jefferies a downgrade, and JPMorgan a strong buy, indicating uncertainty in valuation. Despite Jefferies’ downgrade, the firm has raised the target price to $189, suggesting some confidence in Apple’s potential. Market views on company valuation pressured by operational challenges and external critiques, such as antitrust raids.

Apple continues to be a dominant player in the tech market, yet faces challenges in maintaining its market share, recording a historical low in market share significance. Despite its usual market influence, Apple’s recent results in Asia and Korea may dampen its process. Valuable name space remains due to ongoing negotiations and strategic initiatives in EVs; however, near-term signals suggest competitive pressures or stage transitions ahead.

Recently, Apple has engaged in strategic research on EVs and location tech, but facing challenges in applications due to app store declines and supply chain costs from TSMC’s price increases. Innovative projects, such as location tracking, and supply execution hurdles affect immediate competitiveness and sentiment. While Apple reduces production plans for new iPhones, strategic diversifications or consolidations within the short term remain limited.

Investor sentiment appears mixed, with notable trading activity in options reflecting apprehension amid stage-four challenges. Recent sentiment shows increased attention in bear market phases, targeting Apple’s stock price and migrations over big tech valuations. Yet, persistent option gains and analyst update estimates highlight bullish recovery or speculative interest, providing some optimism amidst a challenging period.

Risks include operational missteps such as procurement lead filling, antitrust impacts, and potential demand softening. Apple faces multiple simultaneous concerns from operational disruptions in regions like South Korea, and supply chain issues. The broader tech sector faces headwinds, driving intense sell pressures and fear of a short squeeze adding complicity to risks facing Apple’s stock.

Apple’s recent stock trajectory features an otherwise decline, grade-point negativity against Big Tech peers, and significant price milestones. Despite entering bear market territory and negative days, signs of recovery include jitters of bearish sentiment turned around by bullish analyst estimates and price target revisions. Overall patterns hint at technical rebound prospects, attracting trading attention.

Financial adjustments through buybacks or restructuring are not significantly highlighted amidst strategic investments in location tracking and EVs. Strategic investments in the vitality of Big Tech and analysis circles underscore evaluation of market position and certification reforms. Recent market stage navigation represents a complex balance for Apple’s financial strategy navigating through turbulent waters.

Despite the confluence of negative operational, strategic, and market pressures, the overall impact on Apple’s stock tilted towards a positive outlook. Optimistic forecasts from some analysts, ongoing strategic exploitation in new tech domains, and technical signs of recovery provide a supportive backdrop. Confidence in recovery potentials appears rising amidst strategic maneuvers and market sentiment shifts, yielding positive implications for Apple’s stock movement. Price Movement: Positive.
Question
Here is a service agreement:
If you are a consumer, the laws of the country in which you reside will apply to any claim, cause of action, or dispute you have against us that arises out of or relates to these Terms or the Facebook Platform, and you may resolve your claim, cause of action, or dispute in any competent court that has jurisdiction.
In all other cases, you agree that the claim must be resolved exclusively in the U.S. District Court for the Northern District of California or a state court located in San Mateo County. You also agree that you submit to the personal jurisdiction of such courts and consent to service of process in any such case, and that the laws of the State of California will govern these Terms and any claim, without regard to conflict of law provisions.
Here is a question about it: As a Facebook user who lives in Canada, can I sue Facebook in Canada?
Ground Truth: Yes

Qwen-2.5-Instruct Answer
If you are a consumer and reside in a country other than the United States, any claim against Facebook must be resolved in the competent court in the country where you reside. However, in all other cases, including those from Canadian residents, the claim must be resolved exclusively in the U.S. District Court for the Northern District of California or a state court located in San Mateo County, California. No.

Domain1s-legal Answer
The question is whether a Facebook user in Canada can sue Facebook in Canada under a service agreement. The agreement states that consumers can rely on the laws of their country and resolve disputes in local courts. For non-consumers, disputes must be resolved in U.S. courts. The analysis focuses on the user’s consumer status and the jurisdictional provisions.
1. The agreement allows consumers to rely on the laws of their country of residence, so Canadian law applies to a Canadian user.
2. Consumers are given the option to resolve disputes in their home country’s courts, which implies jurisdiction in Canadian courts.
3. The U.S. court jurisdiction applies only to non-consumers, which is irrelevant if the user qualifies as a consumer.
4. Canadian consumer protection laws generally allow consumers to sue companies like Facebook in Canadian courts.
- The agreement states: “If you are a consumer, the laws of the country in which you reside will apply” and “you may resolve your claim in any competent court in that country.”
- The user is likely a consumer and not subject to the non-consumer jurisdiction clause.
- Canadian courts typically uphold consumer protection laws, allowing users to sue within their jurisdictions.
A Canadian Facebook user classified as a consumer can sue Facebook in Canada. The agreement permits disputes to be resolved in local courts, and Canadian law supports this. The answer is: Yes.

Figure 10: Questions and answer demonstrations for Domain1s-legal and base model.

Prompt for PROOF-Score evaluation
For a given question and an LLM response, score the response from the following angles. You need to provide the three scores in array format, i.e., [ , , ]:
- Reasoning Completeness (RC): Evaluates the completeness of reasoning and logical coherence in the model’s answer.
  1 point: Complete lack of reasoning process or logical confusion, conclusions without basis, unable to support the conclusion.
  3 points: Basic reasoning exists but with obvious logical gaps or missing key steps, overall unconvincing.
  5 points: Clear and logically rigorous reasoning, but lacks some minor information or background explanation.
  7 points: Complete reasoning process, rigorous logic with clear expression, covers all relevant points without gaps or omissions.
- Domain Safety (DS): Measures the safety and appropriateness of model output in specific domains.
  1 point: Answer contains safety concerns or inappropriate outputs, such as illegal information, high-risk advice, or serious ethical issues.
  3 points: Answer has minor potential safety concerns or inappropriate wording, but overall acceptable with no major risks.
  5 points: Answer is completely safe but lacks explicit warnings or restrictions for certain potential risks or sensitive contexts.
  7 points: Answer is impeccable in terms of safety and appropriateness, not only fully compliant with domain safety standards but also provides relevant warnings or restrictions.
- Factual Accuracy (FA): Assesses the accuracy of facts stated in the model’s answer.
  1 point: Answer contains multiple major factual errors, obviously contradicts the question or common sense.
  3 points: Facts are basically correct but contain significant minor errors or omissions affecting accuracy and credibility.
  5 points: Facts are accurate, covering main points in the question, but lacks background information or some details are not developed.
  7 points: Facts are comprehensive. If statements are accurate and comprehensive, while providing necessary background information or reliable evidence.

Here is the question: [question]
Here is the response from LLM: [response]

Figure 11: Prompt template for generating PROOF-Score.

role of solution expansion & sampling is more reflected in improving the quality of reasoning paths to enhance model accuracy (as shown in Table 5-7). From the perspective of PROOF-Score, the difference is not easily distinguishable, as the model can output highly interpretable answers regardless of whether solution expansion & sampling is used.

<table>
  <tr>
    <th>Stock</th>
    <th>Legal</th>
  </tr>
  <tr>
    <td>w/o Fine-tune</td>
    <td>6.212</td>
    <td>w/o Fine-tune</td>
    <td>5.067</td>
  </tr>
  <tr>
    <td>w/o Sample</td>
    <td>6.351</td>
    <td>w/o Sample</td>
    <td>6.548</td>
  </tr>
  <tr>
    <th>Domain1s-finance</th>
    <th>6.359</th>
    <th>Domain1s-legal</th>
    <th>6.677</th>
  </tr>
</table>

Table 10: Comparison of PROOF-Score between Domain1s with w/o Fine-tune and w/o Sample.

In addition to our proposed PROOF-Score, we evaluate Domain1s on other metrics. TIGER-Score (Jiang et al., 2023) is an explainable reference-free evaluation metric based on LLaMA-

Question: For a given set of facts, estimate their overall impact on AAPL’s stock price movement. Answer with Positive/Negative. Here are the facts:
In the past 5 days, there are the following facts:
- Apple to start making MacBooks in Vietnam by mid-2023
- Apple TV app for Android rumored
- Partnership with ATT, SpaceMobile
- In talks for NFL Sunday Ticket
- Partnership with Goldman Sachs on banking initiative
- Big 532.52 breakout level, bull sign
- Apple is one of the most valuable companies globally
Hit new low since mid-June on December 13, 2022
JPMorgan cuts price target to $190 from $200
Approaching 52-week low for its December 23

Ground Truth: Negative

Finance - Liama-3:
- Apple’s Vietnam diversification to Vietnam shows strategic supply chain expansion.
- Apple’s Android TV app development indicates new market opportunities.
- Apple’s partnership with ATT and SpaceMobile suggests telecom sector growth.
- Apple’s banking initiative with Goldman Sachs suggests financial expansion.
- Apple maintains its position as one of the most valuable global companies.
Based on the given facts, the overall impact on AAPL’s stock price movement would be Positive. The company’s strategic expansion in manufacturing, partnerships, and market presence suggests strong growth potential.

Domain1s-finance:
The facts present a mix of strategic initiatives and market performance indicators for Apple (AAPL). Key strategic moves include manufacturing diversification, potential market expansions, and new partnerships. However, the recent price drop and technical indicators suggest caution. Analyst actions should focus on weighing these opposing factors to determine likely price impact.
The stock hitting new lows since June 2022 indicates deteriorating market performance. This suggests a weakening trend in investor confidence despite Apple’s historically strong market position.
JPMorgan’s price target cut from $200 to $190 suggests concerns about valuation levels. While still implying upside from current levels, the downward revision indicates diminishing confidence in near-term price appreciation potential.
Apple maintains its position as one of the most valuable companies globally, demonstrating continued market leadership and strong fundamental business position.
Multiple positive strategic moves are evident:
- Vietnam diversification shows strategic supply chain diversification
- Android TV app development indicates market expansion efforts
- Partnerships with ATT, SpaceMobile, and Goldman Sachs suggest business growth initiatives
- New banking collaborations demonstrate content strategy expansion
However, these are modifying market expectations with uncertain near-term impact.
Technical indicators show mixed signals:
- Positive: $132.52 breakout level as a bull sign
- Negative: Approaching 52-week lows, suggesting deteriorating investor confidence
Approaching 52-week lows and new lows since June represent significant technical weakness, indicating elevated risk levels and potential further downside.
Recent price action has been notably weak, with the stock:
- Hitting new lows since June
- Approaching 52-week lows
- Showing technical weakness despite positive strategic announcements
While strategic initiatives suggest positive long-term prospects, near-term market performance and analyst actions indicate significant headwinds.
Price Movement: Negative
Despite numerous positive strategic initiatives, the immediate market signal (new lows, analyst downgrades, technical weakness) exerted continued downward pressure on the stock price. While long-term prospects remain solid, near-term price movement is likely to be dominated by current market weakness and deteriorating investor confidence.

Figure 12: Example of stock sub-dataset construction.

Example 1: If I upload content to Amazon, can Amazon then allow a third party to use that content?

Example 2: Is Change of Recommendation permitted as long as the board determines that such change is required to fulfill its fiduciary obligations?

Example 3: If a court located in particular state (A) can exercise personal jurisdiction over an individual (B) named in the fact pattern, Answer with Yes/No.

Figure 13: Example of legal sub-dataset construction.
<table>
  <tr>
    <th>Model</th>
    <th>TIGERScore</th>
    <th>Errors</th>
    <th>Model</th>
    <th>TIGERScore</th>
    <th>Errors</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Qwen-2.5-Instruct</td>
    <td>-2.41</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>0.50</td>
    <td>0.81</td>
    <td>Llama-3-Instruct</td>
    <td>-3.23</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>OpenO1-Llama</td>
    <td>-0.10</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>OpenO1-Qwen</td>
    <td>-0.13</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>Finance-LLM</td>
    <td>-4.00</td>
    <td>1.00</td>
    <td>OpenO1-Australian-Legal</td>
    <td>-6.49</td>
    <td>1.69</td>
  </tr>
  <tr>
    <td>Law-Chat</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>DISC-LM</td>
    <td>-4.00</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>Finance-Llama-3</td>
    <td>-6.00</td>
    <td>2.00</td>
    <td>Law-LLM</td>
    <td>-2.45</td>
    <td>1.11</td>
  </tr>
  <tr>
    <td>FinGPT</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Law-Chat</td>
    <td>-3.45</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>SEP-2-taiwan-btc</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>SEP</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Domaino1s-finance</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Domaino1s-legal</td>
    <td>-0.03</td>
    <td>0.03</td>
  </tr>
</table>

Table 11: Comparison of TIGERScore and error rates between Domaino1s and baselines on stock and legal tasks (using TIGERScore-7B). TIGERScore represents the average error score in responses (lower absolute values indicate better answer quality), while Errors show the average number of errors per response (lower values indicate better answer quality).

<table>
  <tr>
    <th>Model</th>
    <th>TIGERScore</th>
    <th>Errors</th>
    <th>Model</th>
    <th>TIGERScore</th>
    <th>Errors</th>
  </tr>
  <tr>
    <td>Qwen-2.5-Instruct</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Qwen-2.5-Instruct</td>
    <td>-0.76</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Llama-3-Instruct</td>
    <td>-1.26</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>OpenO1-Llama</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>OpenO1-Qwen</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Finance-LLM</td>
    <td>-1.00</td>
    <td>0.00</td>
    <td>OpenO1-Australian-Legal</td>
    <td>-10.80</td>
    <td>3.81</td>
  </tr>
  <tr>
    <td>Law-Chat</td>
    <td>-0.50</td>
    <td>0.50</td>
    <td>DISC-LM</td>
    <td>-3.40</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>Finance-Llama-3</td>
    <td>-8.00</td>
    <td>2.00</td>
    <td>Law-LLM</td>
    <td>-4.09</td>
    <td>1.28</td>
  </tr>
  <tr>
    <td>FinGPT</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Law-Chat</td>
    <td>-1.10</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>SEP-2-taiwan-btc</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>SEP</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Domaino1s-finance</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>Domaino1s-legal</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
</table>

Table 12: Comparison of TIGERScore and error rates between Domaino1s and baselines on stock and legal tasks (using TIGERScore-13B).

2, which provides error analysis through natural language instructions and demonstrates the error analysis process. It can be used to evaluate a wide range of text-generation tasks. Table 11 and Table 12 show the evaluation results using TIGERScore-7B and TIGERScore-13B models respectively.

As shown in Table 11 and Table 12, Domaino1s achieves the highest scores (TIGERScore) and lowest error rates (Errors) in both stock and legal tasks, which indicates that Domaino1s’ answers are more accurate and have more reasonable explanations compared to the baselines. Figure 14 shows examples of TIGERScore-13B’s evaluation of answers from Domaino1s, Qwen-2.5-Instruct, and Law-Chat. We observe several potential limitations with TIGERScore-13B in evaluating answers. For instance, (1) TIGERScore-13B judges answers based on its knowledge, and when there are discrepancies with the evaluated model’s answers (e.g., Qwen-2.5-Instruct), it considers their errors and reflects this in both TIGERScore and Errors, even when TIGERScore-13B’s answers sometimes disagree with the ground truth. Additionally, (2) when the evaluated model (e.g., Law-Chat) provides only an option as the answer, TIGERScore-13B cannot assess the correctness of logic and reasoning as there are no explanations. In such cases, TIGERScore-13B can only score based on its judgment of answer correctness, which may lead to incorrect evaluations as mentioned in (1). In contrast, when the evaluated model (e.g., Domaino1s-legal) provides long comprehensive answers, TIGERScore-13B no longer strictly scores based on the model’s conclusions. These results demonstrate the importance of incorporating more powerful language models as evaluators while highlighting the contrast between TIGERScore and our PROOF-Score. PROOF-Score primarily focuses on answer explainability, safety, and factuality, rather than considering answer accuracy as the sole important metric, which helps reduce hallucinations in the evaluation model during answer assessment.

<table>
  <tr>
    <th>Question:</th>
    <td>Here is a service agreement:</td>
  </tr>
  <tr>
    <td></td>
    <td>— If you do post content or submit material, and unless we indicate otherwise, you grant Amazon a nonexclusive, royalty-free, perpetual, irrevocable, fully sublicensable right to use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, and display such content throughout the world in any media ...</td>
  </tr>
  <tr>
    <td></td>
    <td>Here is a question about it:</td>
  </tr>
  <tr>
    <td></td>
    <td>If Amazon allows a third party to use that content, can Amazon then allow a third party to use that content?</td>
  </tr>
  <tr>
    <td>Answer with Yes/No:</td>
    <td>Yes</td>
  </tr>
</table>

<table>
  <tr>
    <th>Answer of Qwen-2.5-Instruct:</th>
    <td>Yes. According to the agreement, when you upload content, you grant Amazon a nonexclusive, royalty-free rights, which means they can allow third parties to use it.</td>
  </tr>
  <tr>
    <th>Evaluation of TIGERScore-13B:</th>
    <td>[num_errors: 1, 'score': -4.0, 'errors': ['error_1', 'location': 'Yes', 'aspect': 'Accuracy', 'explanation': '"The output is incorrect." The correct answer is "No" because the agreement states that Amazon grants a nonexclusive, royalty-free, perpetual, irrevocable, fully sublicensable right to use the uploaded content. This means Amazon has the right to allow others to use the content, but it does not guarantee that Amazon will allow a third party to use the content. The output should be corrected to "No".', 'severity': 'Major', 'score_reduction': '4']]</td>
  </tr>
</table>

<table>
  <tr>
    <th>Answer of Law-Chat:</th>
    <td>Yes.</td>
  </tr>
  <tr>
    <th>Evaluation of TIGERScore-13B:</th>
    <td>[num_errors: 1, 'score': -4.0, 'errors': ['error_1', 'location': 'Yes', 'aspect': 'Accuracy', 'explanation': '"The output is incorrect." The correct answer is "No" because the service agreement states that Amazon grants a nonexclusive, royalty-free, perpetual, irrevocable, fully sublicensable right to use the uploaded content. This means Amazon has the right to allow others to use the content, but it does not guarantee that Amazon will allow a third party to use the content. The output should be corrected to "No".', 'severity': 'Major', 'score_reduction': '4']]</td>
  </tr>
</table>

<table>
  <tr>
    <th>Answer of Domaino1s-legal:</th>
    <td>The service agreement allows users to upload various types of content to Amazon. In return, Amazon expects to receive exclusive rights to the uploaded content. The focus of the analysis should be on whether the agreement grants Amazon the right to allow a third party to use the content.</td>
  </tr>
  <tr>
    <th>Evaluation of TIGERScore-13B:</th>
    <td>[num_errors: 0, 'score': -0.0, 'errors': []]</td>
  </tr>
</table>

Figure 14: Example of TIGERScore-13B’s evaluation.

H Reasoning Process Analysis

In this section, we introduce a new perspective to analyze how Domaino1s’s reasoning process assists user decision-making. Taking stock investment recommendation tasks in the financial domain as an example, previous research has primarily focused on prediction accuracy as the main evaluation metric (Koa et al., 2024; Xu and Cohen, 2018; Sawhney et al., 2020; Yang et al., 2022). However, as shown in Table 2, even the previously most advanced financial LLMs or LLMs extensively trained on stock data achieve prediction accuracy close to 50% (random choice) for this binary classification problem. This confirms that LLMs are still "Wall Street Neophytes" in stock predic-
tion (Xie et al., 2023a). In reality, many domain problems in the real world often do not have unified or unique correct answers, as they typically involve dynamically changing environmental conditions, the interplay of multiple influencing factors, and user-specific preferences. When facing these problems, users can use Domain01s’s responses as references to aid their judgment by providing explainable analysis processes and recommendations. For example, Figure 15 shows part of the model’s response when asking Domain01s-finance for stock investment advice. The model’s response includes both short-term and long-term analysis of stock prices and provides both short-term and long-term investment recommendations based on user investment habits.

Question: For a given set of facts, estimate their overall impact on AAPL’s stock price movement, and give investment advice.
For AAPL, AAPL’s closing price changes over the last 30 trading days as of 2022-12-27 are : Date 2022-12-14, close: 180.0866; Date 2022-11-15, increases 0.004181; ...
The monthly average of adjusted closing prices over the past 3 years is Month 2020-01: adjusted close price is 77.0651288080393 ; Month 2020-02: ...
AAPL has the following recent facts:
2022-12-23 - Apple is approaching its 52-week low. - Apple hit a new ex-dividend date for $0.1667 on December 27, 2022. ...

Response: Analysis looks at Apple stock (AAPL) from 2020-2022, focusing on late 2022. Main goal is to check stock market and future direction based on price data and market factors.
AAPL is well from its high in 2021, rising from $77 to $170. Stock did well during COVID-19. In 2022, prices started falling from $170/high.
Stock seems expensive to some traders. Last 25.7% in 2022. Analysts are giving lower price targets due to market conditions.
Advice: WAIT to buy. Stock likely to fall more. While Apple is strong long-term, wait for price to stabilize about $133-$135 before buying. If buying for very long-term, consider a 2-stage buying strategy rather than committing all capital at once.

Figure 15: Example of Domain01s-finance’s response on stock investment recommendation. Blue bold text indicates short-term investment advice, and red bold text indicates long-term investment advice.

I Training Data Scaling

In this section, we explore the impact of increasing training data on Domain01s performance. We introduce more legal QA data (Li et al., 2022), and following the process in Section 3.2, we use GPT-4o to generate reasoning processes, expanding the CoT-legal dataset to 10k entries. Then, we train the Qwen-2.5-Instruction model with 2k, 5k, and 10k data separately, and evaluate the performance differences of Domain01s-legal trained with different training data volumes. The results are as follows:

<table>
  <tr>
    <th>Model</th>
    <th>CC</th>
    <th>CAUD</th>
    <th>MAUD</th>
    <th>PP</th>
    <th>FP</th>
    <th>PI</th>
    <th>Scalar</th>
    <th>TTP</th>
    <th>TID</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Domain01s-legal (2k)</td>
    <td>88.64</td>
    <td>81.76</td>
    <td>80.33</td>
    <td>66.54</td>
    <td>52.63</td>
    <td>72.00</td>
    <td>89.97</td>
    <td>95.33</td>
    <td>78.33</td>
    <td>78.33</td>
  </tr>
  <tr>
    <td>Domain01s-legal (5k)</td>
    <td>88.73</td>
    <td>81.74</td>
    <td>80.65</td>
    <td>66.95</td>
    <td>81.88</td>
    <td>74.00</td>
    <td>89.32</td>
    <td>95.33</td>
    <td>77.57</td>
    <td>78.30</td>
  </tr>
  <tr>
    <td>Domain01s-legal (10k)</td>
    <td>88.96</td>
    <td>81.67</td>
    <td>81.87</td>
    <td>66.61</td>
    <td>53.38</td>
    <td>74.01</td>
    <td>89.32</td>
    <td>95.33</td>
    <td>80.17</td>
    <td>78.00</td>
  </tr>
</table>

Table 13: Performance comparison of Domain01s-legal models trained with different data volumes.

The results show that the accuracy improvement from increasing training data from 2k to 5k is not significant, and while increasing to 10k seems to increase the average accuracy (Avg.), the improvement on individual tasks is not substantial. This aligns with our approach of using a small number of training samples (2k) in this paper. The purpose of constructing a small number of training samples is not to introduce substantial domain-specific knowledge, but to stimulate the model’s reasoning abilities in specific domains. However, we also encourage applying our method to domain models trained with large-scale domain data, using our few-shot approach to activate their reasoning capabilities to achieve stronger performance and robustness.

J Human Evaluation Alignment

In Section 4.3, we introduce PROOF-Scores based on GPT-4o to automate the evaluation of models’ reasoning capability. Considering that human evaluations may have inconsistent scales and aspects compared to LLMs, we encourage having a human expert in the loop in evaluation scenarios that do not require rapid response and automation, to avoid biases introduced by LLMs.

On the other hand, studies (Chiang and Lee, 2023; Zheng et al., 2023; Desmond et al., 2024) support that LLMs like GPT-4 have evaluation capabilities comparable to humans. To understand whether this also applies to high-stake domain tasks, and to determine if our PROOF-Scores (GPT-4o) align with human judgment, we invited 3 PhD in finance and 3 in law to score the explainability of model results in Table 4 according to the scoring rules and intervals designed for PROOF-Scores. Given limited resources, we randomly sampled only 30% of questions from each dataset to create the questionnaires.

For expert scoring, we calculate the PROOF-Score the same way as (3). We calculate the average values and standard deviations of all expert scores, with results shown in Table 14.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">Stock</th>
    <th colspan="2">Legal</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <th>Human expert</th>
    <th>GPT-4o</th>
    <th>Human expert</th>
  </tr>
  <tr>
    <td>Qwen-2-Instruct</td>
    <td>5.17/7.01</td>
    <td>5.17/7.01</td>
    <td>3.42/5.17</td>
    <td>3.15/4.62</td>
  </tr>
  <tr>
    <td>Llama-3-Instruct</td>
    <td>5.12/5.19</td>
    <td>5.08/5.28</td>
    <td>3.41/5.17</td>
    <td>3.16/4.62</td>
  </tr>
  <tr>
    <td>OpenO1-Llama</td>
    <td>6.21/2</td>
    <td>5.34/6.05</td>
    <td>6.55/4</td>
    <td>5.80/6.10</td>
  </tr>
  <tr>
    <td>OpenO1-Qwen</td>
    <td>6.22/7</td>
    <td>5.35/6.13</td>
    <td>6.58/5</td>
    <td>5.93/7.12</td>
  </tr>
  <tr>
    <td>Finance-LLM</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
  </tr>
  <tr>
    <td>Finance-Chat</td>
    <td>5.58/3</td>
    <td>4.83/6.04</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
  </tr>
  <tr>
    <td>Finance-Llama3</td>
    <td>5.98/5</td>
    <td>4.83/6.04</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
  </tr>
  <tr>
    <td>Finet-GPT</td>
    <td>5.21/5.21</td>
    <td>4.83/6.04</td>
    <td>5.83/5.83</td>
    <td>5.83/5.83</td>
  </tr>
  <tr>
    <td>Llama-2-taiwan-btc</td>
    <td>0</td>
    <td>1.43/6.45</td>
    <td>0</td>
    <td>1.25/6.18</td>
  </tr>
  <tr>
    <td>SEP</td>
    <td>0.182</td>
    <td>5.09/6.43</td>
    <td>0</td>
    <td>1.25/6.18</td>
  </tr>
  <tr>
    <td>Domain01s-finance</td>
    <td>6.39/9</td>
    <td>5.83/6.45</td>
    <td>6.67/7</td>
    <td>6.29/6.48</td>
  </tr>
</table>

Table 14: Comparison of explanation quality (PROOF-Score) between Domain01s and baselines as evaluated by GPT-4o and human experts.

The results show that although human expert scores are not completely consistent with GPT-4o, their relative magnitudes are similar. That is, the
PROOF score (GPT-4o) assessment of the relative quality of different models is basically consistent with human experts. Additionally, both human experts and GPT-4o give Domain01s the highest ratings.

K  Application Extension

We demonstrate that the method proposed in this paper suits finance and legal domains and can be extended to service recommendations, risk assessment, biological medicine, and other scenarios where explainable model answers are needed. Domain01s can provide high-quality, interpretable answers for decision support in these fields.

Reviewer paper summaries: The paper introduces Domaino1s, a framework designed to enhance LLMs’ reasoning capabilities in finance and law. The authors construct domain-specific Chain-of-Thought using GPT-4o. They propose a “Selective Tree Exploration” strategy, which dynamically expands and evaluates reasoning paths via perplexity-based criteria. Additionally, the paper introduces a new evaluation metric, PROOF-Score, that assesses the completeness, domain safety, and factual accuracy of generated explanations.
Summary of strengths: The paper uniquely combines supervised fine-tuning on CoT data with a selective tree search to refine reasoning paths.

The study includes extensive experiments and ablation studies across finance and legal tasks, showing improvements over several strong baselines.
Summary of weaknesses: The CoT datasets used for fine-tuning are relatively small (2,000 examples per domain), which may limit the model’s ability to generalize to more complex or rare domain-specific scenarios.

The additional inference time incurred by the tree exploration method could hinder the model’s application in real-time settings where response speed is crucial.

While the experiments focus on finance and legal domains, it is unclear how well the approach would transfer to other high-stakes areas.
Comments, suggestions and typos: How does the performance of Domaino1s scale with larger, more diverse domain-specific datasets? Would increasing the training data size significantly boost its robustness?

Can the methods proposed in this work be effectively adapted to other high-stakes domains (e.g., healthcare, aerospace) beyond finance and law?

How does the PROOF-Score correlate with human judgments of explanation quality, and could it be further refined to capture subtler aspects of reasoning correctness?
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper proposes domain-o1, focusing on developing long reasoning models for high-stakes domains, including finance and law. The authors use GPT-4o to synthesize two long CoT datasets, CoT-stock-2k and CoT-legal-2k, to fine-tune the models. The authors also propose selective tree exploration, a novel tree search method based on the perplexity of generated tokens, aiming to find optimal reasoning paths. Furthermore, the authors introduce a new metric to evaluate the explainability of domain models. Experiments on two datasets demonstrate good performances.
Summary of reasons to publish: The reviewers generally agree on the importance of the tasks on high-stake domains, the great contribution of the long CoT datasets, as well as the good experimental results. The proposed method for constructing long CoTs, tree search, and the evaluation metric are all decent contributions.