Knowing-how & Knowing-that: A New Task for Machine Reading Comprehension of User Manuals
Year: 2023

Abstract:
The machine reading comprehension (MRC) of user manuals has huge potential in customer service. However, current methods have trouble answering complex questions. Therefore, we introduce the Knowing-how & Knowingthat task that requires the model to answer factoid-style, procedure-style, and inconsistent questions about user manuals. We resolve this task by jointly representing the sTeps and fActs in a gRAph (TARA), which supports a unified inference of various questions. Towards a systematical benchmarking study, we design a heuristic method to automatically parse user manuals into TARAs and build an annotated dataset to test the model’s ability in answering real-world questions. Empirical results demonstrate that representing user manuals as TARAs is a desired solution for the MRC of user manuals. An in-depth investigation of TARA further sheds light on the issues and broader impacts of future representations of user manuals. We hope our work can move the MRC of user manuals to a more complex and realistic stage.

heading: 1 Introduction
text: User manuals are supposed to be helpful in using products, getting involved in promotions, or other goals of interest (Ryle, 2009; Chu et al., 2017; Bombieri et al., 2021). Though well-designed and instructive, they are seldom read by users because “Life is too short to read manuals” (Blackler et al., 2016). Towards high user satisfaction, professional customer service representatives (CSRs) are hired to do the reading and answer user questions about the manuals. The machine reading comprehension (MRC) of user manuals thus has huge potential, as it would not only reduce high labor costs but enable the service ready for customers 24/7.
Let’s see how a CSR approaches various questions given the user manual1 in Figure 1. Specifi-
∗ Corresponding author 1We make a few modifications (e.g., removing the applica-
tion name) to this real-world user manual for anonymity.
cally, she can answer Q1 by looking up the arguments of facts, i.e., the value ( 100% ) of the hit rate. For Q2, she needs to reason about the steps in the procedure as the answer is not explicitly exhibited in the user manual — the user manual directs the user to “Sign in” and “Scan”, and eventually to the “payment page”. After that, the subsequent description does not direct the user anywhere else. A user can infer from the procedure that one can scratch the card after immediately getting it on the payment page. Therefore, the location to scratch the card is on the payment page . The most difficult one is Q3, which raises a case inconsistent with the descriptions in the user manual — going along with the user manual (S-4), the user didn’t get the scratch card. We call this type of question an inconsistent question. A possible response to
10550
the inconsistent question may be an answer (R1) or a high-utility question leading to an answer (R2R4). Towards such responses, the CSR needs to reason about not only the steps but the facts binding the steps — the inconsistency may be caused by incorrect operations in the previous step (e.g., the user didn’t pay by scanning the QR code ) or unsatisfiable constraints in the facts (e.g., the user already had 10 scratch cards ). Therefore, aligned with the CSR, a practical MRC model must also be able to draw a unified inference across steps and facts to properly answer various user questions.
With a splendid library of studies (Rajpurkar et al., 2016; Zhu et al., 2020), current MRC models have shown strong power to answer factoid-style questions (e.g., Q1). More recent studies (Tandon et al., 2020; Goyal et al., 2021; Zhang et al., 2021) have mitigated the weakness of factoid-style MRC on procedure-style questions. However, these studies mainly focus on questions about the states of entities that can be answered by single spans. They have trouble answering more complex procedurestyle questions with multiple-span answers or about the dynamics of actions (e.g., Q2), let alone inconsistent questions like Q3. We also notice that existing models are only designed for single types of questions. This doesn’t meet the real-world scenarios where factoid-style, procedure-style, and inconsistent questions are all involved (cf., Table 4).
To address these limitations, we propose the Knowing-how & Knowing-that task2 — given a user manual, the model is required to answer factoid-style, procedure-style or inconsistent questions about it. We resolve this task by jointly representing the sTePs and fActs in a heterogeneous gRAph (TARA), cf., Figure 2. This representation allows us to make the unified inference of various questions. We further benchmark the proposed task with a Heuristic method (HUM) to automatically represent User Manuals as TARAs and a denselyannotated dataset (OHO) to test the model’s ability in answering real-world questions. Specifically, HUM is designed to be independent of labelled data with the ultimate goal of building industryscale applications and particular attention to low costs and good generalization capability. The annotated dataset (OHO) is derived from Online Help dOcuments and real-world user questions from an
2We name it after Gilbert Ryle’s thought that the possession of Knowing-how (i.e., procedural knowledge) and Knowing-that (i.e., factoid knowledge) is “a mark of intelligence” (Ryle, 2009).
e-commerce company. We provide the gold answers to user questions and gold TARAs of user manuals. Experiments demonstrate the superiority of TARA, the efficiency of HUM, and the significant challenges of OHO. With an in-depth investigation of TARA, we discuss the issues and broader impact. We expect our work can advance the MRC of user manuals and inspire future research on smart customer service. We highlight our work as follows: • We introduce a new task and take the primary step
to the MRC of user manuals in a more realistic setting, where various questions are involved. • We resolve the task by jointly representing steps and facts in a graph. We also benchmark the task with an efficient method and densely annotate a testing set derived from real-world scenarios. • The experiment reveals the superiority of the proposed representation and the significant challenges of the new task. • We will release the dataset and codes to assist further studies once this paper is accepted.

heading: 2 Related Work
text: MRC for user manuals Earlier studies on the MRC of user manuals focus on factoid-style comprehension (Zhang et al., 2012; Mysore et al., 2019; Jiang et al., 2020; Nabizadeh et al., 2020). Recent research pays attention to the procedure-style comprehension of user manuals and aims to track the state values of pre-defined entities (Bosselut et al., 2017; Amini et al., 2020; Tandon et al., 2020; Goyal et al., 2021; Zhang et al., 2021). Although such models may answer some procedurestyle questions, they are in trouble answering more complex questions like inconsistent ones. Several studies design structured representations for user manuals (Kiddon et al., 2015; Maeta et al., 2015; Vaucher et al., 2020; Kuniyoshi et al., 2020). However, they only represent the steps in user manuals. This largely limits the model’s ability to answer questions that need inference about both steps and facts. Contrary to the above research, we study the MRC of user manuals that are suitable for factoidstyle, procedure-style, and inconsistent questions and design a heterogeneous graph to represent both steps and facts. The representation supports the unified inference of various questions and can be constructed automatically by a heuristic method.
Datasets for user manuals We have done a thorough survey (cf., Table 1) about existing datasets
for user manuals. The types of user manuals include recipes (Zhang et al., 2012; Mori et al., 2014; Kiddon et al., 2015; Yamakata et al., 2020; Jiang et al., 2020), technical manuals (e.g., device maintenance, surgical practices) (Zhang et al., 2012; Nabizadeh et al., 2020; Goyal et al., 2021; Zhong et al., 2020), and scientific experiment (Mysore et al., 2019; Vaucher et al., 2020; Kuniyoshi et al., 2020). There is a lack of a dataset about ecommerce scenarios where the MRC of user manuals is urgently needed due to the high labor costs for customer service. Besides, we have noticed that existing annotations are all about user manuals. None of them has offered real-world QA pairs to test the practical performance of MRC models. To address these limitations, we collect the OHO dataset with online helping documents from an e-commerce company. We then contribute annotations for datadriven evaluations of the model’s ability to answer real-world questions and represent user manuals.

heading: 3 Representation of user manuals
text: Definitions We wish to design a representation for user manuals that supports the unified inference of various questions. It can be reduced to two fun-
damental tasks: how to extract actions (entities) and their arguments from the user manual, and what is the relation between an action and an entity (two actions or two entities). This motivates us to propose the TARA representation, which jointly describes steps and facts in a heterogeneous graph with two sets of nodes, three sets of arguments, and five sets of relations, as defined in Table 2. In this way, the user manual in Figure 1 is represented as a graph3, part of which is displayed in Figure 2.
Unified inference We use TARA to represent Q1, Q2, and Q3, as shown in Figure 2. The inference of answers can be divided into two stages. First, we extract a sub-graph from the representation of the user manual that is most similar to the structure of the question. Second, we identify the answers by directly inferring the conflict arguments between the sub-graph and the question4. Specifically, if the
3Although the STATE of Entity shares the same arguments with an Action node, it cannot be upgraded into a node because it doesn’t refer to an action and cannot be linked to other Action nodes. Similarly, the ATT of an Entity node cannot be a node because it can’t be linked to other Entity nodes except for the parent one. Meanwhile, as the user dominates the whole user manual, it cannot be the target of any STATE.
4This operation reduces the cognition loads to identify Wh-word (e.g., when, what) and its variants in user questions.
conflict is caused by the values of arguments, the answer is the value from the sub-graph. For example, the answer for Q1 is the FN value (“100%”) of the hit rate. If the conflict is caused by the existence of Action-ARG, the answer is inferred from the nearest Action node, the same argument of which is not null, that can walk to the current Action node via NEXT relations. For example, the answer to Q2 is located in the LOC argument of the “get” node, which is linked to “scratch” via NEXT. The third type of conflict is the inconsistency of MOD arguments, namely, the Action node of the question has a MOD argument valued “negation”. The answer is inferred from the nearest Action node and Entity nodes pointed to the sub-graph. For example, all possible responses to Q3 are obtained from the “pay” node, the “user” node, and the “scratch card” node. In summary, TARA allows the unified inference of factoid-style, procedure-style, and inconsistent questions by jointly representing steps and actions in the same graph.

heading: 4 The HUM Method
text: Towards practical industry-scale applications, it requires extra attention to labor costs and generalization. Thus, we choose to design an efficient method based on pre-trained semantic dependency parsing tools and domain-independent heuristics.
First, we segment the user manual sentence by sentence and feed the sentences into a pre-trained semantic dependence parsing tool. If a sentence is imperative, we leverage “user” as the subject of the sentence. For instance, “Sign in the APP” is modified as “User sign in the APP”. Then, we
eliminate the vertices to compose more meaningful phrases — the offsprings of a predicate’s child are eliminated and combined with the child. An example5 is shown in Figure 3. In addition, to maintain the leading role of the user, if the user is the patient of a predicate, we change its role with the agent and mark the predicate as “reverse”.
Second, we use the active verbs, whose agents are the user, as the Action nodes. These nodes are linked with the “NEXT” relations based on their order in the user manual and semantic dependencies. Besides the user node created by default, other Entity nodes are initialized as the unique patients with the user as agents6. The relation from the user Node to any Action node is “AGT”. We also add an edge with the “PAT” relation from an Action node to the Entity node, whose value serves as the patient of the Action node. Particularly, if an entity has attributives, we create a new entity with the “SUB” relation to it by combining the attributives. The “same user” node is created in this
5To save space, some examples are only in one language. 6As the “scratch card” and “card” share the same semantics, we treat both of them as the “scratch card” entity. In addition, although “limit” is not an active verb, its patient (“promotion”) is treated as an Entity node as its agent is the user.
way. Most of Action-ARG, Entity-ARG, and ARGARG are generated by the alignment with the tags from off-the-shelf tools like LTP (Che et al., 2021), HanLP (He and Choi, 2020), etc. The alignment rules are defined in Table 3. Besides, if an entity acts as an attribute to an agent, we use the agent as an ATT argument to the entity. For example, “total number” is an ATT argument to the “scratch card” entity. We also use a predicate as the STATE argument of its agent when the agent is the user, but the predicate is a state verb ( e.g., “have”) or when the agent is not the user.
Automated Inference for QA To infer the answers to user questions, we leverage the HUM method to represent the user manual and user question, simultaneously. Then, we extract the subgraph that is most similar to the representation of the user question via the sub-graph matching algorithm (Zou et al., 2011). The extraction only considers the nodes in the graph. The similarity score between two nodes (x and y) is computed by
S(x, y) = 1−D(x, y)/max(|x|, |y|), (1) where D computes the Levenshtein distance, and | ∗ | is the length of ∗. After getting the sub-graph matched to the user question, we do the inference described in Section 3 to get the answer.
5 The OHO dataset
We gather the frequently asked questions (FAQs) summarized by the customer service department of an e-commerce company. We remove FAQs that aren’t attached to any documents and that are attached to documents with only images. Finally,
we get a dataset of 2000 user manuals, each of which has an FAQ.
Answer annotation Each annotator is shown an FAQ and its attached user manual, which is displayed sentence by sentence. A toy example for the annotation of Q3 is shown in Figure 4. The annotator is required to select which sentences in the user manual can be possible responses to the question and then select the type of the question. Each sentence is treated as a text span in the answer. The statistics of the annotations are described in Table 4. It reveals that the procedure-style and inconsistent questions take up more than half part (52.45%) of the real-world user questions about user manuals.
Representation annotation This annotation is done on 200 user manuals of OHO. As some user manuals describe more than one task, there are 346 graphs that need to be annotated. It is not practical for the annotator to draw a heterogeneous graph for each user manual. Following Dalvi et al. (2018), we reduce the task as the annotation of answers to nine basic questions derived from Table 2. The basic questions are defined in Table 5. Specifically, the annotations of B1 and B2 are completed first. This is because the annotations of B3-B4 and B6B8 depend on the results of B1-B2. Similarly, the annotations of B5 and B9 are completed last as they depend on B4. The statistics are shown in Table 6. Although the basic questions are factoid-style questions, we can use them and gold answers to roughly estimate the maximum potential of models — if a model can correctly answer these basic questions,
it can construct TARA for user manuals and thus have a chance to answer higher-level (procedurestyle, inconsistent or more) questions via unified inferences.
Quality control We employ three CSRs with at least one-year working experience from the same department that summarizes the FAQs. We give a training session to the workers to help them fully understand our annotation requirements. The annotation tasks are released to the workers via an online annotation platform managed by the e-commerce company. On average, it takes about 3 minutes for an annotator to annotate the answer to an FAQ and about 2.8 hours to annotate all answers to basic questions of a user manual. An annotated result is accepted only if all three workers agree; otherwise, we invite two experts (one is the manager of the customer service department and the other is a postdoc working on computational linguistics) to make the final decision. The experts closely discuss with each other to ensure the consistency of results.

heading: 6 Experiments
text: To systematically benchmark the Knowing-how &Knowing-that task evaluation, we perform two experiments based on the OHO dataset. First, with the help of answer annotations, we compare the performances of HUM with seven baselines in answering real-world user questions, i.e., the FAQ-answering task. Second, with the help of representation annotations, we investigate the maximum potential of HUM, i.e., the B-answering task. Specifically, we use the LTP tool (Che et al., 2021) for word segmentation and semantic dependency parsing.

heading: 6.1 Experiment I: FAQ-answering Task
text: Baselines We compare HUM with seven baselines: 1) QA pattern matching (Peng et al., 2010; Jain and Dodiya, 2014), which utilizes a number of
QA patterns written by experts; 2) lexical matching (Alfonseca et al., 2001; Yang et al., 2018), which computes the lexical similarity scores between the FAQ and candidate sentence via EQ (1); 3) semantic matching, which computes similarity scores of user questions and candidate sentences based on BERT embeddings (Devlin et al., 2018); 4) keyword matching (Moldovan et al., 2000), which measures the similarity between the keywords of the user question and candidate sentence; 5) pre-trained LM (i.e., PERT) (Cui et al., 2022), which has been trained with MRC tasks on other corpora; 6) self-supervised LM (Nie et al., 2022), which leverages the self-supervised strategy to train the LM with more than 360000 user manuals7 with interrogative masks (Lewis et al., 2019); 7) finetuned LM, which fine-tunes the PERT models using a subset of OHO, the results are obtained from two-fold cross-validation. We also create a variant of HUM with oracle TARAs, named HUM-oracle. This method is tested on the 200 FAQs with user manuals that have gold TARAs.
Evaluation metrics We use the precision (P), recall (R), and F1 scores to measure the performances. To make fair comparisons with methods designed for one-span answers (i.e., QA pattern matching, pre-trained LM, and self-supervised LM), we also report P@1, R@1, and F1@1 values.
Results Table 7 shows the performance of HUM and baselines on the FAQ-answering task. We can see that HUM-oracle significantly outperforms others on all metrics. This demonstrates that representing user manuals as TARAs is a desired resolution to answer various user questions. We also make the following observations. 1) The keyword matching method gets a higher recall score than HUM, but its
7They are online helping documents from the e-commerce company without FAQs and annotations.
precision score is about half HUM. The matching of keywords between the user question and the user manual is similar to the extraction of the sub-graph from the heterogeneous graph of the user manual. So, this method can find all related spans to the user question. However, without the unified inference on TARA, it cannot reject the noise spans and thus get low precision scores. Similar results can also be found in the other matching methods. 2) The QA pattern matching has a high precision score because of sophisticated QA patterns (covering more than 80% of FAQs) but the worst recall score due to the high flexibility of answer expressions. 3) We are not surprised that the pre-trained LM and selfsupervised LM rank last, as they are only exposed to factoid-style questions during the training phase thus they are insufficient to answer procedure-style and inconsistent questions. 4) The results of finetuned LM are slightly better than that of pre-trained LM and self-supervised LM but are still far behind HUM. We conjecture this is because of the lack of massive annotated data and the lack of unified inference. 5) After carefully checking HUM, we notice that about 5% of FAQs are not answered because HUM can’t extract sub-graphs from user manuals. This suggests a possible future work is to eliminate such errors via more sub-graph matching algorithms, etc. 6) Except for HUM-oracle, the
best F1 score is only 0.76, indicating the intrinsic challenge of the Knowing-how & Knowing-that task and there is substantial room for a model to represent user manuals as TARAs like an expert.
Auxiliary results We conduct a further study about methods that can produce multiple-span answers. As shown in Figure 5, in most cases, the performances decrease with the increase of answer spans. This suggests that it is more difficult to predict answers with more spans than answers with fewer spans. Compared with other methods, the fine-tuned LM rank last on all metrics and has poorer robustness when answering different types of user questions — the performance for inconsistent questions is significantly worse than that for procedure-style and factoid-style questions. It indicates that, despite the achievement of state-ofthe-art results on many NLP tasks, the popular finetuned LM is not suitable for industry-scale MRC of user manuals without spending huge costs to annotate massive data. Notably, for all types of user questions, the F1 scores of HUM surpass other methods by a large margin on one-span and twospan answers and are slightly lower than the best value on three-span answers. This reveals the sufficiency and robustness of HUM to cope with the challenges raised by various types of user questions and multiple-span answers.

heading: 6.2 Experiment II: B-answering Task
text: We take a close look at HUM to investigate its ability to represent user manuals as TARAs and its maximum potential to answer higher-level questions. We create variants of HUM by replacing the backbone semantic parsing tool (i.e., LTP) with Biaffine (Dozat and Manning, 2018), MFVI (Wang et al., 2019), and HanLP (He and Choi, 2020). In addition, we employ five postgraduate students who major in computer science as annotators. They are asked to annotate the answers to basic questions. Before starting the annotation work, they are shown 10 samples annotated by experts without attending the training session. In this way, we obtain human (non-expert) upper bounds for this task.
Evaluation metrics Inspired by the evaluation of information extraction, we employ the precision, recall, and F1 scores based on BLEU scores (denoted as P†, R†, and F1†, respectively) (Tandon et al., 2020) to measure the performances of HUM on B1-B5. For B6-B9, we report the standard precision, recall, and F1 scores.
Results The results are reported in Table 8. The last column reports the human upper bounds. Although we have tried four state-of-the-art semantic parsing tools, the best-performing method only reaches ∼0.24 F1 scores, indicating the significant challenges to automatically constructing TARAs for user manuals. Specifically, the results of B6 are lower than that of B1, including the human results. This is because a worker’s annotations to B6 are based on his annotations to B1, namely, the accumulative errors issue. The values of the human upper bound (∼0.92 precision score, ∼0.88 recall score and ∼0.90 F1 score) demonstrate that the task is feasible, well-defined and leaves plenty of room for more advanced semantic dependency parsing tools, more efficient heuristics, etc.
Issues and broader impact of TARA After investigating the bad cases, we find that, in addition to the accumulative errors, there are issues with co-reference problems, complex discourse parsing beyond sentences, etc. We leave the resolutions to these issues as future work. We observe that the performances of HUM on the B-answering task are much worse than that on the FAQ-answering task. Although the basic questions are factoid-style questions, they can compose all possible higher-level questions besides procedure-style and inconsistent ones. For example, Q2 can be decomposed into a
combination of B1, B3, and B6. The FAQs in the B-answering task only take a small part of the questions composed from the basic questions. Thus, the B-answering task is much more challenging than the FAQ-answering task. This observation also strongly demonstrates that better performance on the B-answering task will largely improve HUM’s ability to answer real-world questions. Besides, we’d like to talk about other potential benefits that can be gained from the TARAs. In addition to the unified inference of various questions, the joint representation of dynamic actions and entities also sheds light on the reasoning and planning of new tasks. Possible applications involve the arrangement of unordered actions, error detection of the draft user manuals, the automated composition of user questions, task-oriented information seeking, etc. The model, with the ability to correctly answer basic questions, can further be applied in downstream scenarios other than QA. For example, it is beneficial to build an intelligent training bot for new staff from customer service departments.

heading: 7 Conclusion and Future work
text: We propose the Knowing-how & Knowing-that task that requires the model to answer various questions about a user manual. We resolve it by representing the steps and facts of a user manual in a unified graph (TARA). To benchmark it, we design an efficient method and annotate a testing set derived from real-world customer service scenarios. Experiments reveal the superiority of TARA, the efficiency of HUM, and the significant challenges of OHO and the new task.
We take the primary step to study the MRC of user manuals in a more challenging setting, where various questions are involved. We hope our work can benefit further research on smart customer service. There are several directions for further work. First, to improve the performance of HUM, we will research resolutions to accumulative errors, co-reference problems, complex discourse parsing beyond sentences, etc. Second, to explore the potential of the proposed representation, we will introduce it to other tasks that need better interpretability, like task-oriented information seeking. Third, we will investigate the potential of unified inference for more complex user manuals, e.g., user manuals with multiple agents. Lastly, we plan to deploy HUM to online customer service to gain more insights for further improvements.

heading: 8 Limitations
text: We now explain the limitations and potential risks of our work. First, it seems the Knowing-how & Knowing-that task is a bit unfriendly to supervised learning methods as we only annotate the testing set. However, towards practical industry-scale applications, we encourage future work to utilize the current annotations and contribute to more efficient heuristic, unsupervised, self-supervised, or weakly supervised methods, etc. Second, each user manual in OHO only contains one user (agent). However, there are a number of user manuals involving more than one agent, e.g., “invite your friend as a new user and get cash back”. This motivates us to explore multi-agent user manuals in our future work. Third, in addition to the textual content, many user manuals contain visual information like images and GIFs. Hence, it will be more desirable to add such user manuals and study the Knowing-how & Knowing-that task in multi-modal settings.

heading: 9 Ethics Statement
text: This paper presents a new task for machine comprehension of user manuals. Although the user manuals and FAQs involved in the task are collected from an e-commerce company, they are designed for normal users and have been widely used by the public for some time. We also have carefully checked these data to make sure they don’t contain any personally identifiable information or sensitive personally identifiable information. Thus, we believe there are no privacy concerns.
All user manuals and FAQs are reviewed at least three times by the company’s staff before being released to the public. Besides, we have been authorized by the company to make OHO publicly available for academic research. Thus, we believe the dataset doesn’t contain any harmful information and is qualified for distribution.
The annotators of OHO consist of CSRs, a postdoc, and undergraduate students. As the dataset is about user manuals and the job is to answer questions about the user manuals, we believe there are no physical or mental risks to the annotators.

heading: Acknowledgement
text: This work was supported in part by the National Natural Science Foundation of China (No. 62206191 and No. 62272330); in part by the China Postdoctoral Science Foundation (No.2021TQ0222 and No. 2021M700094); in part by the Natural
Science Foundation of Sichuan (No. 2023NSFSC0473), and in part by the Fundamental Research Funds for the Central Universities (No. 2023SCU12089 and No. YJ202219).

heading: A Settings of the baselines for the FAQ-answering task
text: The detailed settings of the baselines models of the FAQ-answering task are as follows.
• QA pattern matching (Peng et al., 2010; Jain and Dodiya, 2014) uses handwritten rules to match questions and answers with corresponding syntactic formats (Ravichandran and Hovy, 2002; Soubbotin and Soubbotin, 2001; Greenwood and Gaizauskas, 2003). For example, given Q1 following the pattern of what be <entity>?, the answer is the sentence in the user manual following the pattern of <entity> be <value>. • Lexical matching (Alfonseca et al., 2001; Yang et al., 2018). We use EQ (1) to calculate the lexical similarity between question and candidate sentence (Coates and Breitinger, 2022). Then we choose the top two sentences with the highest score as the final answer. • Semantic matching (Devlin et al., 2018). We represent a sentence by averaging the token embeddings in this sentence. Then we compute the cosine similarity between question representation and candidate representation. The top two sentences with the highest score are used as the final answers. • Keyword matching (Moldovan et al., 2000). We use the TF-IDF algorithm to extract ten keywords for each user manual. Meanwhile, we use the keywords extraction API of the iFLYTEK open platform to obtain the keywords of the question text. We calculate the matching score between each candidate answer and question according to the following formula:
S =16× |KQ|+ 16× |KA|+ 16× |KQ ∩KA|+ 16× |ΓKQ ∩ ΓKA | − 4× √ Dmax(K{Q}) (2)
where |KQ| represents the number of candidate answer keywords existing in the question text, |KA| represents the number of question keywords existing in the candidate answers, |KQ∩KA| represents the number of words that are both question keywords and answer keywords, ΓKQ ∩ ΓKA represents how many question keywords are on the same sub_tree of the candidate answer parsing tree, where the parsing tree of the candidate answer is obtained from the semantic dependency parsing of LTP (Che et al., 2021), Dmax(K{Q}) represents the distance between the two question keywords that are farthest apart in the candidate answer text.
• Pre-trained LM. We use the Chinese machine reading comprehension model based on PERTlarge (Cui et al., 2022), which has been fine-tuned on a mixture of Chinese MRC datasets. It is highly competitive in many tasks of machine reading comprehension and sequence labeling. It obtained 90.8, 95.7, and 79.3 F1-score on CMRC 2018 Dev, DRCD Dev, and SQUAD-Zen Dev (Answerable) data set respectively. We concatenate the question text and the user manual and then feed it to the model to predict the start and end positions of answer spans. • Self-supervised LM (Nie et al., 2022). Selfsupervised learning is the approach that trains the model using the constructed data sets of potential QA data mined from a large amount of corpus (Baral). It can obtain a quantity of data without manual annotation to fine-tune the pretraining model. Inspired by previous work (Lewis et al., 2019), we mask actions, entities, and corresponding arguments and relations in the source user manuals and replace them with interrogative words (when, where, why, how, etc.) to construct QA data. We finally constructed 800,000 QA pairs from 360,000 user manuals as the training dataset. For model training, we still choose the PERT-large model, whose pre-training paradigm is also self-supervised learning. Following the pre-training paradigm, we concatenate the question text and the original user manual as the input of the model and then predict the start and end positions of the answer span. We choose the AdamW (Loshchilov and Hutter, 2017) as the optimizer, the learning rate is set to 1e-5, and a total of three epochs are trained. • Fine-tuned LM. The labelled dataset OHO is divided into two parts. We fine-tune the pre-training model on half of the data and then evaluate the performance of the model on the other half of the data. To ensure that the model can output multiple possible answer spans for each question, we splice the question text and candidate sentences in the source user manual into the model in turn and then predict whether each candidate sentence is the answer to the question, and we still choose PERT-large as the pre-training model. The GPU we used for model training is Tesla P100, the max length of the model input is set to 512, the batch size is set to 4, the learning rate is set to 1e-5, and a total of six epochs are trained. The results are obtained from two-fold cross-validation.

heading: B Case Study of the B-answering task
text: We here discuss the issues of HUM in generating TARAs for user manuals. Figure 6 presents a bad case caused by the co-reference problem — the model fails to identify that “they” refers to “realname authentication users”, leading to a missing STATE argument of the “real-name authentication users” node. Figure 7 presents a bad case caused by accumulative errors — the model generates noise relations in TARA after it wrongly treats “pass” as an Action node.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 6
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 6
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 5
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 5
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 5
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 5
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, towards practical industry-scale applications, we encourage future work to utilize the current annotations and contribute to more efficient heuristic, unsupervised, self-supervised, or weakly supervised methods, etc. Second, each user manual in OHO only contains one user (agent). However, there are a number of user manuals involving more than one agent, e. g. , “invite your friend as a new user and get cash back”. This motivates us to explore multi-agent user manuals in our future work. Third, in addition to the textual content, many user manuals contain visual information like images and GIFs. Hence, it will be more desirable to add such user manuals and study the Knowing-how & Knowing-that task in multi-modal settings