# Weibull-Open-World (WOW) Multi-Type Novelty Detection in CartPole3D
## 5 Discussion  
We report the detection, characterization, and control performance of the 1&2 step lookahead with EVT-based detection in Table 1 for four different novelty subtypes, each with three levels of difficulty. We also report the DQN performance in that table. The DQN agent was pure control and did not report novelty. We performed a Wilcoxon signed-rank test for significance (used for all comparisons in this paper), and the difference in performance was statistically significant with \(p<0.01\). For relation and interaction subtypes, both baseline and control agents were robust, making detection more difficult. When the control agent was designed, the designers expected the semantics of \"relations\" to include things such as spatial positioning, and variations in world geometry (e.g., floor tilt, friction) would be detected in cart/pole dynamics. We failed to detect them until they were in the \"hard\" category; the impact of these may be below the noise inherent in the system from random collisions and the inability to exactly model the cart/pole. When the control agent was designed, the designers thought the semantics of \"interaction\" would be interactions between the agents and our cart. Thus, we expected our collision/failure measures would capture dissimilarity in that subtype. The experimental results show that interactions were all nuisance novelties. Collision/failure are more likely appropriate measures of action (e.g., attacking agents). We now hypothesize that interactions would be agent-to-agent measur \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & & & & & & & & & & \\ \cline{2-10}  & \multirow{2}{*}{easy} & 1.0 & 3.77 & 0.56 \(\pm\) 0.03 & 1.0 & 1.0 & 0.98 \(\pm\) 0.13 & 0.83 \(\pm\) 0.02 & 0.67 \(\pm\) 0.08 \\ \cline{2-10}  & \multirow{2}{*}{medium} & 1.0 & 3.73 & 0.56 \(\pm\) 0.03 & 1.0 & 1.0 & 0.92 \(\pm\) 0.28 & 0.75 \(\pm\) 0.03 & 0.60 \(\pm\) 0.07 \\ \cline{2-10}  & \multirow{2}{*}{hard} & 1.0 & 3.77 & 0.54 \(\pm\) 0.06 & 1.0 & 1.0 & 0.88 \(\pm\) 0.32 & 0.64 \(\pm\) 0.06 & 0.53 \(\pm\) 0.05 \\ \cline{2-10}  & \multirow{2}{*}{Subtotal} & 1.0 & 3.76 & 0.55 \(\pm\) 0.05 & 1.0 & 1.0 & 0.93 \(\pm\) 0.26 & 0.74 \(\pm\) 0.09 & 0.60 \(\pm\) 0.09 \\ \hline  & easy & 1.0 & 15.27 & 0.51 \(\pm\) 0.03 & 1.0 & 1.0 & 0.80 \(\pm\) 0.40 & 0.63 \(\pm\) 0.03 & 0.60 \(\pm\) 0.07 \\ \cline{2-10}  & \multirow{2}{*}{medium} & 1.0 & 6.40 & 0.48 \(\pm\) 0.05 & 1.0 & 1.0 & 0.42 \(\pm\) 0.49 & 0.44 \(\pm\) 0.05 & 0.47 \(\pm\) 0.05 \\ \cline{2-10}  & \multirow{2}{*}{hard} & 1.0 & 4.03 & 0.47 \(\pm\) 0.04 & 1.0 & 1.0 & 0.17 \(\pm\) 0.37 & 0.32 \(\pm\) 0.03 & 0.37 \(\pm\) 0.04 \\ \cline{2-10}  & \multirow{2}{*}{Subtotal} & 1.0 & 8.57 & 0.49 \(\pm\) 0.05 & 1.0 & 1.0 & 0.46 \(\pm\) 0.50 & 0.47 \(\pm\) 0.13 & 0.48 \(\pm\) 0.11 \\ \hline  & easy & 0.0 & 0.0 & 0.54 \(\pm\) 0.04 & 0.0 & 0.0 & 0.98 \(\pm\) 0.13 & 0.89 \(\pm\) 0.02 & 0.75 \(\pm\) 0.09 \\ \cline{2-10}  & \multirow{2}{*}{medium} & 0.0 & 0.0 & 0.55 \(\pm\) 0.03 & 0.0 & 0.0 & 1.0 \(\pm\) 0.0 & 0.88 \(\pm\) 0.02 & 0.72 \(\pm\) 0.09 \\ \cline{2-10}  & \multirow{2}{*}{hard} & 0.10 & 7.43 & 0.55 \(\pm\) 0.03 & 0.0 & 0.0 & 1.0 \(\pm\) 0.0 & 0.85 \(\pm\) 0.02 & 0.70 \(\pm\) 0.08 \\ \cline{2-10}  & \multirow{2}{*}{Subtotal} & 0.03 & 2.48 & 0.55 \(\pm\) 0.03 & 0.0 & 0.0 & 0.99 \(\pm\) 0.07 & 0.87 \(\pm\) 0.03 & 0.72 \(\pm\) 0.09 \\ \hline  & easy & 0.0 & 0.0 & 0.53 \(\pm\) 0.03 & 0.0 & 0.0 & 0.98 \(\pm\) 0.13 & 0.89 \(\pm\) 0.02 & 0.79 \(\pm\) 0.09 \\ \cline{2-10}  & \multirow{2}{*}{medium} & 0.03 & 5.10 & 0.54 \(\pm\) 0.04 & 0.0 & 0.0 & 0.98 \(\pm\) 0.13 & 0.90 \(\pm\) 0.02 & 0.78 \(\pm\) 0.10 \\ \cline{2-10}  & \multirow{2}{*}{hard} & 0.0 & 0.0 & 0.54 \(\pm\) 0.03 & 0.0 & 0.0 & 0.98 \(\pm\) 0.13 & 0.90 \(\pm\) 0.02 & 0.78 \(\pm\) 0.10 \\ \cline{2-10}  & \multirow{2}{*}{Subtotal} & 0.01 & 1.70 & 0.54 \(\pm\) 0.03 & 0.0 & 0.0 & 0.98 \(\pm\) 0.13 & 0.90 \(\pm\) 0.02 & 0.78 \(\pm\) 0.09 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance of EVT-based detection with 1&2 step lookahead WOW-agent. FP rate was 0 for all EVT-based tests.