# Conformal Test Martingale-Based Change-Point Detection for Geospatial Object Detectors
## 4 Experiments
### Comparison Experiments  
The comparison experiments were conducted for all of the four domain shift cases proposed in Section 3.1. For each domain shift case, the test data from the source domain came sequentially, followed by the test data from the target domain, and ten experiments were conducted by shuffling the order of the samples within each domain. The experimental results are recorded in Tables 2-5, where Mean Delay refers to the mean value of the delay steps to raise the first true alarm of domain shift [23]. POD and POFD are widely used indicators for detection in statistics [34], where POD represents probability of detectionand POFD represents probability of false detection. All of the input features are \(v(z_{i})\) introduced in Section 3.2. Alg.1 is the conformal test martingale calculated by Algorithm 1, whose nonconformity measure is Formula (3) and parameter \(J\) is set to 0.01 according to [17]. CTM-C is a recently proposed transformation process of CTM based on a cautious betting strategy, which takes the evidence of violating IID assumption into account [24]. CTM-C is also a martingale theoretically whose performance is close to or even better than existing well-performed change-point detection techniques. Therefore, CTM-C is the state-of-the-art approach of CTM-based methods. The parameter \(\epsilon\) of CTM-C is the threshold related to the evidence of violating IID assumption, and we set it as 5, 10, or 15 in the comparison experiments and denote the algorithms as CTM-C(\(\epsilon=5\)), CTM-C(\(\epsilon=10\)) and CTM-C(\(\epsilon=15\)), respectively. Alg.2 is the transformation of the outputs of Alg.1 based on Algorithm 2, where Alg.2(\(r_{n}=a\)) means \(r_{n}=a\) for any \(n\). Alg.2(Adaptive) is Alg.2 with \(r_{n}\) calculated by Formula (4), where \(R=-0.1\), \(M=10\). Since we focus on the detection speed of each algorithm, the lowest Mean Delay is shown in bold and underlined in the Tables. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Algorithm** & **Mean Delay** & **POFD** & **POD** \\ \hline Alg.1 & 41.3 & 0.0003 & 0.9827 \\ CTM-C(\(\epsilon=5\)) & 37.3 & 0.0000 & 0.9843 \\ CTM-C(\(\epsilon=10\)) & 33.5 & 0.0000 & 0.9855 \\ CTM-C(\(\epsilon=15\)) & 34.7 & 0.0000 & 0.9854 \\ Alg.2(\(r_{n}=0\)) & 41.2 & 0.0003 & 0.9827 \\ Alg.2(\(r_{n}=0.1\)) & 40.8 & 0.0007 & 0.9829 \\ Alg.2(\(r_{n}=0.5\)) & **30.5** & 0.1806 & 0.9872 \\ Alg.2(Adaptive) & 34.5 & 0.0003 & 0.9855 \\ \hline \hline \end{tabular}  
* The lowest Mean Delay is shown in bold and underlined.  
\end{table} 
Table 2: Change-Point Detection From Clear to Blur Domain. In Table 2, we can see that the performance of Alg.1 and Alg.2(\(r_{n}=0\)) is nearly the same, which indicates that only limiting the lower bound of \(S_{n}^{*}\) cannot improve much. By increasing \(r_{n}\), the change point can be detected quickly, whereas larger \(r_{n}\) leads to more false alarms. By setting \(r_{n}=0.5\), the violation of validity with Algorithm 2 is observed, since the POFD is higher than the 5% significance level. This implies that the momentum parameter of Algorithm 2 should be chosen properly for empirical validity. Finally, our proposed Alg.2(Adaptive) has low POFD and Mean Delay simultaneously, as it only accelerates the increase of \(S_{n}^{*}\) when the distribution of \(p\)-values is far away from uniformity on \([0,1]\), i.e., the \(r_{n}\) is far away from 0.5. In comparison, the Mean Delay of CTM-C is really low and CTM-C(\(\epsilon=10\)) performs best, since it achieves the lowest Mean Delay with the POFD lower than 5%. In this domain shift case, our proposed Alg.2(Adaptive) is close to the performance of CTM-C(\(\epsilon=10\)). With similar analysis, the experimental results shown in Tables 3-5 demonstrate that Alg.2(Adaptive) performs best for the other three domain shift cases, which confirms the effectiveness and efficiency of our proposed transformation process with adaptive momentum. Since Mean Delay is the most important indicator under the premise of low POFD, we also show Mean Delay in Figure 4 for visualization.