# Chinese Medical Question Answer Matching Using End-to-End Character-Level Multi-Scale CNNs
## 4 Experiments and Results
### Results  
In this section, a detailed examination of the experimental results are presented. Table 3 summarizes the results for different approaches on cMedQA. \begin{table}
\begin{tabular}{l l l l l} \hline \hline  & **Embeddings** & **Model** & **Dev (\%)** & **Test (\%)** \\ \hline
1 & & Random Selection & 01.00 & 01.00 \\
2 & & Word Matching (Jieba) & 37.05 & 36.60 \\
3 & & Word Matching (ICTCLAS) & 36.15 & 37.30 \\
4 & None & Character Matching & 33.65 & 34.90 \\
5 & & BM25 (Jieba) & 37.60 & 40.00 \\
6 & & BM25 (ICTCLAS) & 41.35 & 41.35 \\
7 & & BM25 (Character) & 44.80 & 45.40 \\ \hline
8 & Word (Jieba) & & 15.60 & 16.80 \\
9 & Word (ICTCLAS) & Average of Embeddings & 18.05 & 18.75 \\
10 & Character Embeddings & & 24.90 & 24.00 \\ \hline
11 & Word (Jieba) & & 24.55 & 23.65 \\
12 & Word (ICTCLAS) & Embedding Matching & 27.85 & 29.10 \\
13 & Character Embeddings & & 30.80 & 32.30 \\ \hline
14 & & SingleCNN [1] & 46.30 & 47.65 \\
15 & Word Embeddings (Jieba) & BiLSTM [2] & 51.70 & 52.70 \\
16 & & MultiCNNs & 48.40 & 51.15 \\ \hline
17 & & SingleCNN [1] & 53.25 & 53.75 \\
18 & Word Embeddings (ICTCLAS) & BiLSTM [2] & 56.15 & 57.65 \\
19 & & MultiCNNs & 54.05 & 55.40 \\ \hline
20 & & SingleCNN & 64.50 & 64.05 \\
21 & Character Embeddings & BiLSTM & 61.65 & 63.20 \\
22 & & MultiCNNs & **65.35** & **64.75** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The top-1 accuracy results of the models. biLSTM: bi-directional long short-term memory network; Multi-CNN: multi-scale CNN. Rows 1 to 13 provide a survey of the results of the baseline methods without using neural network architectures. More specifically, as can be seen from Rows 2 to 4, which show the results of matching based methods, the results of the two methods differ by less than 1%. Due to the abundant information contained in words, word-based methods (Rows 2 and Row 3) surpass the character-based method (Row 4). In comparison to word (character) matching methods, BM25 methods (Rows 5 to 7) utilize more statistical information and show up to 11% improvement. Row 8 to Row 13 show the results of embedding-based shallow methods. Character embedding methods (Row 10,13) take the lead, which indicates that the semantic information of embeddings is better extracted from characters. Word embeddings preprocessed by the ICTCLAS toolkit rank second. However, the average of the embeddings methods is inferior to word matching methods, which may be due to their incomplete semantic information extraction capability. The following three rows (Rows 14 to 16) illustrate the results of three deep neural models when they are fed with word embeddings preprocessed by the Jieba toolkit. It is observed that the biLSTMs architecture stands out amongst these three models. The potential reason may be that recurrent neural networks include the variant LSTM which is capable of extracting sequence information. They are able to extract the semantic information of the whole sentence, which can effectively reduce the semantic gap between the questions and answers. Our results are consistent with experimental results reported by [PERSON] et al. [19]. Moreover, the multiCNN model shows more than 2 to 3% improvement on the development and test data-sets, respectively. The next three rows (Rows 17 to 19) show the similarity trends: biLSTM is ranked first, followed by multiCNNs. It is shown that the ICTCLAS method is much more accurate for words segmentation as compared to Jieba in most cases, although it runs slower than Jieba ([[http://thulac.thunlp.org/](http://thulac.thunlp.org/)]([http://thulac.thunlp.org/](http://thulac.thunlp.org/))). It is also shown in the table that the word embedding methods based on ICTCLAS (Rows 17 to 19) outperform the methods based on Jieba (Rows 14 to 16). The following three rows (Rows 20 to 22) summarize the results of these models on character embeddings. The multiCNNs architecture achieves the highest performance due to the benefits of multi-scale local information extraction. The singleCNN model ranks second.