# Holistic Spatial Reasoning for Chinese Spatial Language Understanding
## 4 Experiment  
### Experimental Setting  
**Dataset. ** We evaluated our proposed method on the SpaCE [14] dataset, a Chinese spatial language understanding corpus that comprises more than 10,000 examples. SpaCE encompasses a wide range of domains, such as news, literature, textbooks, sports corpuses, and encyclopedias. The dataset comprises annotations of abnormal spatial semantics recognition, spatial role labeling, and spatial scene matching. The detailed dataset statistics are listed in Table 2.  
In this work, we adopted the official dev split for the test set and processed zero-shot/1-shot CoT. We also reproduced some supervised models for comparison, and we split the official training set into 7:1 for training and validation. For the SpSM task, there was no training set, but 10 examples were used for evaluation ([https://2030 nlp.github.io/SpaCE2023/index.html](https://2030 nlp.github.io/SpaCE2023/index.html), accessed on 1 September 2023), and we only reported its results with the LLM-based methods. **Baselines. **We provide a comparison of our proposed method with extensive baseline methods. We first report results on conventional supervised methods, such as **BERT-like** language models. We adopted a state-of-the-art framework, MRC-SRL [42] for the SpSRL task. The MRC-SRL treat the SRL task as an multiple-choice machine reading comprehension (MRC) problem [43]. For ASpSR, we used a conventional BERT-based text classification framework [44], where the pooled outputs of the BERT backbone are used to predict whether the given sentence has abnormal expressions through a binary classification layer. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Split**} & **ASpSR** & \multicolumn{2}{c}{**SpRL**} & \multicolumn{2}{c}{**SpSM**} \\ \cline{2-6}  & **\#SENT** & **\#SENT** & **\#SPAN** & **\#ROLE** & **\#SENT** \\ \hline Train & 10,993 & 1529 & 25,108 & 15 & - \\ Test & 1602 & 700 & 3815 & 15 & 10 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The statistics of the SpaCE dataset. “#SENT” means the number of sentences. “#SPAN” means the number of SRL spans. “#ROLE” means the number of role types. We also include the results of a prompt-learning paradigm for the LLM. We report three types of prompting baselines: (1) **Vanilla Prompt (zero-shot). ** We include vanilla prompt learning [45], where only the description of the task's target is fed to the LLM as an input. (2) **In-Context Learning (ICL). ** In-context learning [27] adds demonstrations to the prompt template, thus strongly guiding the generation of output answers for desired tasks. (3) **Vanilla CoT (zero-shot). ** Following reference [37], vanilla CoT appends \"Let us think step by step\" to the prompt with or without any demonstration examples, guiding the LLM to decompose a problem into steps. (4) **Vanilla CoT-ICL. ** Based on zero-shot CoT, CoT-ICL add demonstrations to further guide the generation in an ICL paradigm. Technically, we list the input template of the three methods in Table 3.  
**Implementation Details. ** For supervised ASpSR, we use a BERT-base backbone and a binary linear classifier. For supervised SpRL, we follow reference [42] to use a BERT-base model as the base encoder and use two special symbols <p> and </p> to mark the predicate of the input sentence. We adopt Adam as optimizer, and the warmup rate is 0.05, the initial learning rate is \(1\times 10^{-5}\). For LLM-based methods, we select ChatGPT3.5 ([[https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt)]([https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt)), accessed on 1 September 2023), which is the top-performing open-source large language model. The temperature value of ChatGPT is set to 0.2. The detailed parameters are presented in Appendix B. **Evaluation. ** For the ASpSR and SpSM tasks, we use the accuracy as the evaluation metric. For the SpRL task, we use the F1 score to evaluate the sequence labeling performance; this is denoted as RL-F1, and the extra label classification is denoted as Ext-F1.