Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings
Year: 2023

Abstract:
Ambiguity is a major obstacle to providing services based on sentence classification. However, because of the structural limitations of the service, there may not be sufficient contextual information to resolve the ambiguity. In this situation, we focus on ambiguity detection so that service design considering ambiguity is possible. We utilize similarity in a semantic space to detect ambiguity in service scenarios1 and training data. In addition, we apply taskspecific embedding to improve performance. Our results demonstrate that ambiguities and resulting labeling errors in training data or scenarios can be detected. Additionally, we confirm that it can be used to debug services.

heading: 1 Introduction
text: The ability to accurately access the meanings of sentences is a key component of voice recognitionbased agent services. This task is made difficult by the inherent ambiguity of some sentences, which can refer to different meanings in different contexts. Adot( ) 2 is a voice recognition-based service agent, akin to Amazon’s Alexa, for Korean users. This paper describes how we dealt with ambiguity, from a perspective of Adot’s developer.
For example, as illustrated in Figure 1, the Korean word "앞" has two different meanings depending on the context. As a result, when the media content is being played, the command "앞으로 가봐" may have two completely opposite meanings.("move forwards" vs. "move to previous"). In a test conducted within our company, approximately 61.7 percent of respondents interpreted it as “move forwards” and 38.3 percent as “move to previous”.
1A scenario refers to one expected input sentence and the analysis result of it predefined for service design.
2Adot is a virtual assistant service developed by SK Telecom, a telecom company in South Korea. Though officially denoted as A., we will refer to it as Adot in this study to avoid misunderstanding.
Although there is no universally valid interpretation in this scenario, each respondent was certain that their own interpretation was the correct one. (Likert-scale score = 5.2/7). This semantic ambiguity leads to a labeling error of training data and scenarios in specification3. From the perspective of service providers who must provide specific services in response to user input, we contend that this is a challenging problem. In addition, ambiguity is very important in terms of cost. Manual inspection of the training data or specification is required for ambiguity handling, which is expensive in terms of cost and resources.
Recently, many researchers have attempted to handle ambiguity with various approaches such as semantic space (Rodd, 2020), entity linking (Yin et al., 2019), and attribute attention (Liu et al., 2019). In addition, the primary purpose of these studies is to resolve ambiguity using context. In spite of this, we do not have sufficient contextual information to apply these disambiguation methods because of the characteristics of a very short input sentence (which has 6.97 Korean letters and 2.05 words on average) and a one-turn based service.
Although contextual information required for disambiguation is not available within the scope
3We call the set of scenarios as the specification.
425
of the sentence classifier, we can still exploit nonlinguistic traits, such as user habits and device information, which are accessible from the outside classifier to determine service. Therefore, we chose to approach this problem through detection rather than disambiguation. Detecting the presence of ambiguity in scenarios would allow for the consideration of external contextual information when providing the service.
This paper proposes a process of detecting ambiguity in scenarios or training data through similarity. To improve ambiguity detection performance, we apply task-specific embedding. We conduct an experiment detecting ambiguity and labeling error from ambiguity in Adot’s scenarios and training data. Additionally, we investigate Pearson correlation coefficients between similarity and the degree to which users expect two sentences to belong to the same class (this is referred to as user-aware class relevance). This study’s findings affirm that similarity provides a means to identify scenarios and training data with potential ambiguity. In this process, we confirm that user-aware class relevance correlates with the similarity. Furthermore, it was confirmed that the training data that resulted in the misclassification could be specified.

heading: 2 Related Works
text: Ambiguity is a long-standing problem in natural language processing (NLP) tasks such as word sense disambiguation (Navigli, 2009), entity disambiguation (Barba et al., 2022), and database search result disambiguation (Qian et al., 2021) in the taskoriented dialogue systems. Ezzini et al. (2021) utilized domain-specific data to address the structural ambiguity of sentences. This ambiguity is largely divided into four categories (Berry et al., 2003).
• Lexical ambiguity occurs when a word has several meanings
• Syntactic ambiguity occurs when a given sequence of words can be given more than one grammatical structure
• Semantic ambiguity occurs when a sentence has more than one way of reading it
• Pragmatic ambiguity occurs when a sentence has several meanings in the context in which it is uttered.
These studies are focused on resolving the ambiguity. However, we did not have contextual information to apply to these disambiguation methods.
In NLP, the sentence similarity task, which evaluates the similarity between pairs of sentences using several techniques, has been investigated extensively. Traditionally, edit distance has been used to quantify superficial similarities (Levenshtein et al., 1966). Despite being straightforward, it is noteworthy because of its similarity to manual identification of training data that induce misclassification. Recent studies have measured the similarity between pairs of sentences by projecting them into a meaningful semantic space using various neural networks, such as Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018), or using Siamese architectures, such as SentenceBERT (Reimers and Gurevych, 2019).
In this paper, inspired by (Ezzini et al., 2021), We utilize a task-specific embedding, extracted from a trained classification model, to identify semantic ambiguities.

heading: 3 Task Design

heading: 3.1 Ambiguity Detection in Scenarios
text: When ambiguity exists in scenarios, the accurate classification result is not aligned with the user’s expectations. If the user expects a class (class-B) other than the class (class-A) defined in the specification to be correct for an input a, it implies that classification of the input under class-B is sufficiently valid or that the user has experienced receiving a service corresponding to class-B for at least one input a′ similar to a. Let’s suppose that the command “앞으로가봐” we looked at in section1 is defined as “move forward” in the specification. A significant proportion, 38.3%, interpret this command as “move to previous”, and the similar phrase “앞으로 다시 가봐” is commonly used in Korea with the meaning “move to previous”. (The interjected word ‘다시’ usually means ‘again’.) From the perspective of the user, we attempted to identify the ambiguity of the scenarios.
First, pairs of sentences are collected, and the correlation between similarity and user-aware class relevance is evaluated. If this correlation is sufficiently high, user-aware class relevance could be estimated based on similarity. In a given scenario, if specific training data y that has the highest similarity with a particular scenario sentence x is placed
in a different class than the class-X defined for scenario sentence x, it implies that the user may not anticipate class-X for sentence x and ambiguity may exists in the scenario. Subsequently, ambiguity or the mislabeled scenario is manually inspected in the specified scenarios to validate this process.

heading: 3.2 Ambiguity Detection in Training Data
text: The method of 3.1 could not be applied as it is to detect the ambiguity of the training data. In the process of generating training data, several similar data are generated all at once. Ambiguous training data is rarely observed in isolation; furthermore, the data demonstrate a marked degree of similarity to one another. Finding similar data with different classes for single data did not work well.
We noted that the ambiguity in training data manifests as mislabeled data that do not conform to the specification. Models trained using this mislabeled data yield outputs that deviate significantly from the designer’s expectation. In this paper, we begin with misclassified inputs and attempt to find the training data that caused the corresponding misclassification and mistraining.
Hence, input sentences misclassified by Adot are collected. Among all training data with misclassified labels, data exhibiting high similarity with misclassified inputs are assumed to be training data that induce misclassification. These include corresponding data with ambiguity or those mislabeled from ambiguity. We inspect the specified data manually to confirm any ambiguity or labeling errors. Additionally, the labels of the specified data are modified to the labels of the expected classes, and the model is retrained. It is verified whether the classification results of the model are corrected as expected. Via this, we confirm whether the specified data are the cause of the misclassification.

heading: 4 Methods - Task-Specific Embeddings
text: We note that each node of the trained classification model stores weights adjusted for the task, and the values output by each node in the inference process contain fragmentary information useful for inference (Wang et al., 2020). We create task-specific embedding vectors using the output values of each node in the classification model. Via this process, we expect to be able to create an embedding with better expressive power for ambiguity detection, albeit biased, compared to off-the-shelf embeddings, such as pre-trained LM or Word2Vec. Assuming a
classification model fθ with n layers on top of the embedding layer, we create a vector vi representing the i-th layer as follows: (v1 = embedding vector, vn = model output vector.)
Whole Layer Vector (WLV) WLV consists of output of all nodes from the model’s embedding layer to the layer below the final output layer. WLV contains the most amount of data and reflects all information flows in the model.
WLV = concat(v1, v2, ..., vn−2, vn−1)

heading: After Representation Layer Vector (ARLV)
text: WLV contains considerable information, but the embedding layer accounts for most of it, which is problematic (for instance, 25,640 nodes out of a total of 40,424 nodes are included in the embedding layer of Adot’s classification model). ARLV is defined using nodes from the layer following the embedding layer to the layer immediately preceding the output layer to avoid undue influence of the embedding layer.
ARLV = concat(v2, v3, ..., vn−2, vn−1)
Conclusion Layer Vector (CLV) Although WLV and ARLV contain a significant amount of data, they ignore discrete functions and consequently struggle with the corresponding differences. For example, when a layer uses an activation function such as RELU, the difference in the value input to the function might be negligible. However, the resultant value after processing the function might be completely different. CLV is only defined using the layer immediately preceding the output layer of the entire model to account for these characteristics. We anticipate that the overall model’s judgment process is structured in this layer.

heading: CLV = vn−1

heading: 5 Experiments

heading: 5.1 Datasets

heading: 5.1.1 Scenario Ambiguity Test Set
text: The following experiment utilizes the task described in section 3.1 to detect scenario ambiguities from the user’s perspective.
Correlation Test Dataset 913 pairs of sentences are collected from domain classification validation data for the Adot service (Domain information is used as a label in this experiment). Domains, in
the context of Adot, are defined service areas, and include categories consisting of 30 categories such as general, music, video, weather, and schedule. Sentences are randomly selected from the data of eight domains that frequently result in classification failure. The classes of each pair may be identical or different. We measure Pearson correlation between similarity and user-aware class relevance of sentence pair.
Ambiguity Test Dataset To evaluate ambiguity detection, 29,136 sentences with designer-defined labels are collected. These sentences are scenarios defined for the Adot service. For these scenarios of specification, the scenarios with possible ambiguity are detected using the process mentioned in section 3.1. Lastly, any actual ambiguity or labeling error corresponding to the specified scenarios is manually inspected.

heading: 5.1.2 Training Data Ambiguity Test Set
text: The following experiment based on the task designed in section 3.2 is performed to detect ambiguity in training data.
Ambiguity Test Set This test dataset consists of 2,300 sentences incorrectly classified by Adot. Based on the misclassified sentences, training data exhibiting high similarity with these sentences are assumed to be training data that caused the misclassification. We manually check for ambiguity or labeling errors in those specific sentences.
Side-Effect Test Set Identification of data that induces misclassification also requires that inputs analyzed successfully before correction must must remain so after correction. To confirm this, 20,000 sentences classified correctly by Adot are collected, and their classification result is verified after the specified data are modified and retrained.

heading: 5.2 Baseline - Similarity Methods
text: Edit Distance A method for measuring the superficial similarity. It is very similar to the manual debugging process.
Embedding Vector Similarity A method for measuring the semantic similarity. Cosine similarity of embedding vectors for sentences is measured. Two types of representative off-the-shelf embeddings are prepared:Sent2Vec and KoGPT2. Sent2vec, was trained on both Korean Wikipedia and Adot’s training data. KoGPT2 is a pre-trained language model developed by SKT-AI, based on
GPT-2. These embeddings are also used to comprise an embedding layer for the classification models discussed below. We describe the embedding methods in detail in the Appendix A.

heading: 5.3 Classification models used in the Experiments
text: We employed classification models to generate taskspecific embeddings and assess the detection performance of training data that cause misclassification.
Adot Classifier This is the model used for domain classification in Adot’s service. It is formed by concatenating a convolutional neural network (CNN) that uses a part of speech-tagged morpheme as a token with another CNN that uses character and dictionary information as a token followed by three fully connected layer on top.
KoGPT2 Classifier It is a classifier based on pretrained-LM. As in the basic classifiers of preLM packages, three fully connected layers are added on top of KoGPT2.
These models are trained using training data for domain classification of Adot’ service. The data consists of 2.2 M sentences with 30 corresponding classes. We set the batch size to be 256 and finetune the model for 15 epochs using a learning rate of (1e-5). On the development dataset, the Adot and KoGPT classification models achieved accuracy rates of 98.4% and 94.2%, respectively. Because presenting a high-performance model is not within the scope of this study, a detailed description of the model is included in the Appendix A.2.

heading: 5.4 Human Evaluation
text: The task designed in section 3.1 includes the user test. The evaluator rates the degree to which two sentences belong to the same class on a scale of 1–7. An example of the test set given to the evaluator is included in the appendix B.2. For this experiment, two groups of evaluators are recruited.
Ordinary Service User (User) This group consists of six ordinary Adot users. Each user exhibits above-average linguistic proficiency and possesses a bachelor’s degree or higher academic qualifications. Through correlation with this group, we checked whether user-aware class relevant could be estimated with similarity.
Expert Evaluator (Expert) This group consists of seven people with experience in generating training data or evaluating quality of service for Adot.
Each participant exhibits five years of annotation experience on average and demonstrates a thorough understanding of class boundaries.

heading: 5.5 Evaluation Metrics
text: Experimental performance is evaluated in terms of specific metrics. If the data requires correction but is not ambiguous, it is considered to be a simple mislabeling error. In our experiment, the terms commonly used to define the formulation for evaluation metrics are Nx, s, and d, where Nx, s, and d denote the number of x ∈ {s, d}, scenario or training data identified as having an ambiguity, and input data, respectively.
Ambiguous Data Ratio (ADR) measures the ratio of scenario or training data containing ambiguity in the identified scenarios or training data, Nsa/Ns, where Nsa and Ns denote the number of sa and the number of s, respectively. sa represents the scenario or training data with actual ambiguity among s. The existence of ambiguity was confirmed by a person directly checking if multiple interpretations were possible.
Mislabeled Data from the ambiguity Ratio (MDR) measures the ratio of scenario or data containing ambiguity and labeling error in the identified scenario or data, Nsm/Ns. sm denotes scenario or training data with actual ambiguity and labeling error among s. Mislabeled Data (sm) was verified by a person directly checking the need for label correction.
Correction Ratio (CR) Measure the classification success rate for retrained model on previously misclassified sentences after modifying data and retraining, Ndf /Ndfs . df represents input data misclassified by Adot, while dfs denotes data correctly classified after correction and retraining from within the df .
Accuracy preservation Ratio (APR) measures the classification success rate for previously successfully analyzed sentences after modifying the data and retraining, Ndss/Nds . dss denotes input data from the ds that was correctly classified after correction and retraining, while ds represents data successfully classified by Adot.

heading: 5.6 Results
text: Pearson Correlation Analysis As shown in Table 1, we show the result of pearson correlation analysis between user-aware class relevance and
the measured similarities. In general, various similarity methods achieve significant positive correlations 4, which suggests that similarity methods can be used as a proxy estimator for user-aware class relevance. This result is regarded as an evidence of our hypothesis described in Section 3. This result reinforces our hypothesis that presenting data with high similarities but differing labels can aid in identifying ambiguous scenarios. Moreover, similarity methods more correlates with users than experts who know the service criteria accurately. Following (Pang et al., 2020), we also measure the mean inter-rater agreement. The study results indicate high agreement and yield two conclusions. Firstly, the methods utilized, particularly Adot-CLV, demonstrate a correlation almost equivalent to the average inter-rater correlation (0.589 vs. 0.633, respectively). Secondly, the manual evaluation explanation can be considered sufficient.
Similarity with ordinary users was significantly higher than with experts. To ascertain the underlying cause of this phenomenon, we computed the concordance of each sentence pair at three distinct points, namely the service area, entity properties, and predicate. We investigated the correlation between manual evaluation and concordances. Notably, the expert group demonstrated a markedly high correlation in the service area, whereas the ordinary user group exhibited comparable correlations across all three areas. Further elaboration on this matter is presented in the appendix C, as it is not directly pertinent to the primary focus of our paper, namely ambiguity detection.
Specification Ambiguity Detection As shown in Table 1, we presents the ambiguity and labeling errors resulting from ambiguity detected in the scenarios. Since the specification is the gold standard for services, it is mostly comprised of scenarios that have a clear meaning. Nonetheless, we can identify scenarios that contain ambiguity or require label modification. Interestingly, the number of data between containing actual ambiguity ((sa) = (Ns × ADR), KoGPT2+ARLV(sa) = 210, Adot+ARLV-(sa) = 152) or requiring modification ((sm) = (Ns × MDR), Sent2Vec(sm) = 40, Adot+ARLV-(sm) = 36) are not different significantly considering the number of scenar-
4In the case of Edit Distance, which takes the value 0 when completely identical and 1 when completely different, the sign is the opposite of that of the other similarity measurement methods.
ios (29,136).During the scenario ambiguity detection experiment, we found a noticeable contrast in the number of identified scenarios, denoted by Ns. Task-specific embedding tended to have a lower Nsthan off-the-shelf embedding. Furthermore, the upper layer-generated embedding displayed a lower Ns. Ns determined the performance difference in ADR and MDR.
Training Data Ambiguity Detection Table 1 shows the performance in training data ambiguity detection. We verify that the ambiguity and labeling error is viewed as the cause of misclassification. In practice, approximately 70% of the specified data contain ambiguity, and approximately 45% require label correction. CR and APR represent the performance of identifying training data that cause misclassification. The results also demonstrate that similarity can be used to identify training data that cause misclassification. Moreover, it is confirmed that ARLV and CLV of task-specific embeddings are more suitable for the detection task.
Effect of Task-specific Embedding As reported in Table 1, task-specific embeddings outperform other methods in both ARLV and CLV, while WLV does not achieve significant improvement. Possibly, this is because off-the-shelf embeddings occupy the majority of WLV, as discussed in the definition of ARLV (See in Section 4). To understand why this phenomenon happens, we visualize the
representation of training data samples obtained from each embedding method using T-SNE. As illustrated in Figure 2, the task-specific embeddings (i.e., ARLV and CLV) clearly represent the training data samples compared to general embeddings.
The task-specific embeddings based on the Adot model outperformed those based on KoGPT2. However, we do not consider the transformer-based embedding to be unsuitable for the task. The observed difference in performance appears to stem from Adot’s specialization in classifying this data, given that Adot has undergone extensive optimization as part of its use in the corresponding service. The differences in classification accuracy between the Adot and KoGPT2 models, as mentioned in the section 5.3, support this interpretation(98% vs. 94%). Observing Figure 2, one can discern a subtle yet distinct data segregation demonstrated by the Adot-based embeddings as compared to those based on KoGPT2. This differential expressive capacity seems to manifest as a performance discrepancy. In the case of general embeddings, KoGPT2 exhibits greater separation than Sent2Vec, which, in practice, is reflected by the overall superior performance of Embedding-KoGPT2 compared to Embedding-Sent2Vec.

heading: 6 Discussions
text: Powerful Task-Specific Representation This study demonstrates the feasibility of detecting am-
biguity through similarity and highlights the potential benefits of utilizing task-specific embedding to enhance performance. The experimental results show a positive correlation between model size and detection performance according to the scaling law. Future research efforts for ambiguity detection will focus on generating task-specific embeddings using larger models for superior results.
Multilingual Ambiguity Detection This study was conducted for Korean-based services. Most users of Adot are Korean, and the inputs contain many Korean characteristics. And since the commands input to the smart speaker were the data of the study, fairly short and imperative sentences are targeted. Experimentation is required to determine if similar results can be obtained in other languages and other types of sentences.

heading: 7 Conclusion
text: We introduce two processes to detect training data and scenarios that induce ambiguity. Moreover, task-specific embedding is adopted to improve detection performance. Comprehensive analysis reveals that compared to off-the-shelf embeddings, Sent2Vec and KoGPT2, task-specific embedding is much more suitable for ambiguity detection. Moreover, the results demonstrate the viability of identifying training data that cause misclassification. In the future work, we aim to compare the performance of task-specific embedding using a wider range of models and node selection methods.

heading: Acknowledgements
text: This research was conducted based on the experience of developing and operating the A.( ) service by SK Telecom in South Korea. This research was supported and funded by the Korean National Police Agency. [Project Name: XR CounterTerrorism Education and Training Test Bed Establishment/Project Number: PR08-04-000-21]. This work was supported by Chungnam National University.

heading: A.1 Development Environment
text: We experimented on NVIDIA A100 (40GB). The Adot model was implemented and trained within the Keras and Tensorflow environments, while finetuning of KoGPT was conducted in the PyTorch framework.

heading: A.2 Classification Models
text: The average Adot user only uses six Korean characters in an utterance, and there are typically 5 million calls per day. Owing to these characteristics, a lightweight and simple model capable of stable CPU prediction was required. Convolutional Neural network (CNN) based models are currently used in Adot. This domain classification model was created using Y. Kim’s CNN for sentence classification (Kim, 2014). It has a model concatenated with character-based CNN and CNN models based on part-of-speech (POS) tagged morphemes reflecting the Korean features. The character-based CNN model uses the dictionary-based embedding layer as a single channel to use the content domain’s dictionary information. The configuration of the fully connected layer on top of the convolutional layer is also slightly different from the model in our study. Figure 3 shows an outline of Adot’s domain classification model.

heading: A.3 Embeddings
text: We represented a sentence data in the semantic space through two methods.

heading: A.3.1 Sentence Representation Using Word Embedding
text: Word embedding is a representative method of expressing meaning in a semantic space. We developed two kinds of embeddings, character-level embedding and POS tagged morpheme-level embedding, to express sentences. A sentence is repre-
sented by concatenating it through two embeddings. Our embedding developed with skip-gram and negative sampling based on McCormick (2016) tutorial. Table 4 shows the detailed parameters of the embeddings we used. These parameters were determined empirically to maximize the performance of the Adot’s domain classifier.
These embeddings were trained from Korean Wikipedia sentences. Korean Wikipedia is a set of documents written by many people and reflects the popular usage of the Korean language by Koreans. However, sentences on the Wikipedia are long, while the utterances coming into the Adot service are relatively short, colloquial, and that there are many content object names. To reflect this part, the embedding trained from the wiki were tuned with the training data of Adot.

heading: A.3.2 Sentence Representation Using KoGPT2
text: GPT-2 is a natural language processing model that uses machine learning algorithms to generate input sample text into text with syntactic, grammatical, and informational consistency. KoGPT-2, an open source-based GPT-2 model trained in Korean. Character Byte Pair Encoding (KBPE) was used as tokenizer. Korean Wikipedia, Modu Corpus, and the Blue House National Petition and private data like news were used as training data. When a sentence passes through the KoGPT2, the output values from the last layer were used as sentence representation.

heading: A.4 Training Data
text: The training data of domain classifier of Adot’s was used as the training data of this experiment. Adot encompasses thirty service domains, including ‘general’, ‘video’, ‘music’, ‘schedule’, and ‘weather’. A person directly constructed about 2.2 million training data according to the design of the service. The data is primarily Korean sentences. Some foreign language sentences are also included. The table5 includes an extensive description of the model and training data. The data used in this experiment is from the December 2021 version, and it’s important to note that its structure and content differ from the current Adot service data.

heading: B Testset Details

heading: B.1 Designer-Annotator testset
text: To improve performance, we have collected utterance logs from Adot users and used them for accu-
racy evaluation. A test set was constructed using the utterances input in April 2022.
Consensus Error& Ambiguity testset Among those utterances, utterances that were classified incorrectly were targeted. To prevent data modification that occurred in both directions of the classes, only utterances with the correct class ‘‘general’’ were collected. The “general” class is for device manipulation such as volume control in Adot. We constructed 2,300 test cases. Because these data are misclassification data collected from the Adot classification model, only the Adot classification model was applied to this experiment.
Side-effect test set This set is a test set to evaluate the side effects caused by label modification. Through modification, even if it is possible to correctly classify misclassified utterances, the classification results of utterances that have been accurately analyzed should not be changed. We collected 20,000 utterances that the current classifier accurately analyzes and expect the data to be analyzed accurately even after modification.

heading: B.2 Scenario ambiguity testset
text: We built two test sets to detect consensus errors between designers and users.
Correlation testset First, sentence pairs were collected to confirm the correlation between similarity and the degree to which the user thinks the classes are consistent. 913 pairs of sentences were collected from validation data for the Adot service. Among them, test sets were collected from eight domains in which domain classification frequently fails: ‘commerce,’ ‘food,’ ‘general,’ ‘news,’ ‘schedule,’ ‘video,’ ‘music,’ and ‘radio.’ Like a domain classifier, domain information was used as a label for classification. Each sentence pair is randomly collected within the verification data of 8 domains and may have the same class or a different class. Figure 4 and 5 shows example of testset.
Ambiguity detection testset To test consensus error detection between designers and users, 29,136 sentences in which designers defined labels were collected. These sentences are specifications defined for Adot service scenarios and defined by the designer of Adot service. For this Spec, the sentences in which Consensus error may occur were indicated through the method mentioned in A. It was confirmed whether there was an actual ambiguity or consensus error for the specified sentences.

heading: C Expert-ordinary user correlation comparison
text: With ordinary users, similarity was significantly higher than with experts. To determine the reason behind this phenomenon, concordance is estimated at three points for each sentence pair—service area, entity properties, and predicate. In addition, the correlation between manual evaluation and concordances is estimated. The results are presented in Table 6. Although the expert group exhibits a very high correlation in the service area, the ordinary user group exhibits similar correlations in all three areas. Let us consider the pair, “Play Netflix” and “Play Spotify” Ordinary users may believe that the classes are similar because of the similarity of the predicates. In contrast, experts believe that Netflix and Spotify provide different services (video vs. music)—so there is no agreement irrespective of the predicate. The results confirm that the similarity is more similar to the judgment of ordinary users than that of experts. Consequently, similarity is deemed to be useful in detecting in designer-user consensus errors.

heading: Future Work
text: In the future work, we aim to compare the performance of task-specific embedding using a wider range of models and node selection methods