# Holistic Spatial Reasoning for Chinese Spatial Language Understanding
## 4 Experiment
### Main Results  
Table 4 details the performance of the baselines and our CoT method. We observed that the vanilla prompts for LLMs were inferior to existing supervised methods, but the ICL and CoT knowledge significantly boosted it, especially when at least one demonstration was used. Our Spatial-CoT method further outperformed the others on all three spatial language understanding tasks. Overall, SpRL was more difficult than ASpSR and SpSM. Under the one-shot setting, the vanilla prompt and vanilla CoT methods were transformed into ICL-based methods, and these are denoted as ICL and vanilla CoT-ICL. The demonstration is randomly selected from the training set. We can see that the ICL could enhance the LLM significantly when one demonstration was provided. Specifically, with the zero-shot setting, the LLM-based methods were comparable to the full-setting supervised method and even outperformed it on the SpRL task, demonstrating the generation capabilities of LLMs. Moreover, our Spatial-CoT showed the following scores: 8.84 ASpSR ACC, 2.96 RL-F1, 3.01 Ext-F1, and 8.02 SpSM F1. These were absolute improvements on the SpaCE dataset in comparison with the vanilla prompt methods. The ICL method was able to enhance the prompt-based and CoT methods by providing a demonstration to guide the LLM, earning 9.11 ASpSR Acc, 0.06 RL-F1, and 2.19 SpSM F1 \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**ASpSR**} & \multicolumn{2}{c}{**SpRL**} & \multicolumn{2}{c}{**SpSM**} \\ \cline{2-7}  & \multicolumn{1}{c}{**Acc**} & \multicolumn{1}{c}{**RL-F1**} & \multicolumn{1}{c}{**Ext-F1**} & \multicolumn{1}{c}{**F1**} & \multicolumn{1}{c}{**Rationale Score**} \\  & \multicolumn{1}{c}{} & & & & & **(Human)** \\ \hline \multicolumn{7}{l}{\(\bullet\)_Fine-tuning baselines_} & & & & & \\ BERT/MRC-SRL (1-shot) [42] & 50.33 \(\pm\) 5.25 (46.85) & 9.72 \(\pm\) 2.17 (54.27) & 6.19 \(\pm\) 2.03 (49.82) & - & - \\ BERT/MRC-SRL (Full) [42] & 78.26 \(\pm\) 1.89 (58.99) & 46.49 \(\pm\) 0.13 (39.94) & 49.10 \(\pm\) 0.81 (36.04) & - & - \\ \hline \multicolumn{7}{l}{\(\bullet\)_Prompt-based methods_} & & & & \\ Vanilla Prompt (zero-shot) & 71.19 \(\pm\) 0.07 (37.05) & 43.01 \(\pm\) 0.36 (44.54) & 46.20 \(\pm\) 0.25 (46.85) & 35.10 \(\pm\) 0.21 (39.32) & 1.79 \\ Vanilla CoT (zero-shot) & 73.29 \(\pm\) 0.02 (38.10) & 44.13 \(\pm\) 0.48 (46.31) & 47.12 \(\pm\) 0.25 (39.94) & 35.66 \(\pm\) 0.32 (48.03) & 2.64 \\ ICL (1-shot) & 77.13 \(\pm\) 0.48 (49.38) & 45.29 \(\pm\) 1.02 (58.33) & 49.15 \(\pm\) 2.21 (52.66) & 39.29 \(\pm\) 1.86 (52.45) & 2.77 \\ Vanilla CoT-ICL (1-shot) & 80.30 \(\pm\) 0.70 (46.08) & 46.04 \(\pm\) 1.52 (59.95) & 51.28 \(\pm\) 1.89 (58.63) & 43.40 \(\pm\) 0.18 (57.62) & 2.80 \\ \hline \multicolumn{7}{l}{\(\bullet\)_Ours_} & & & & \\ Spatial-CoT (zero-shot) & 80.03 \(\pm\) 0.02 (35.40) & 46.19 \(\pm\) 0.03 (39.72) & 52.91 \(\pm\) 0.07 (28.79) & 45.12 \(\pm\) 0.02 (40.02) & 3.01 \\ Spatial-CoT-ICL (1-shot) & **81.41 \(\pm\) 0.12** (28.71) & **48.61 \(\pm\) 0.08** (48.10) & **55.33 \(\pm\) 0.15** (43.22) & **47.76 \(\pm\) 0.03** (49.36) & **3.72** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results of three tasks on seven SpaCE datasets. BERT/MRC-SRL lines mean that we use BERT classifier for ASpSR task and use MRC-SRL for SpRL task. We report the standard deviation in the brackets. **Bold:** best result.