SlowGuess/NAACL_2022_OCR huggingface dataset:
MultiWOZ, SGD dataset requires more contextual information (SGD has less average tokens per turn and more turns per dialogue). Thus the mapping from relatively noisy clusters to ground truth creates errors for downstream tasks, especially that the evaluation metric require exact match of slot types.
However, the results are still comparable. Although predicting a magnitude smaller number of clusters to be less favoured in evaluation, our method still achieves similar or better performance.
A.8 Comparison to DSI on DST
We note that the DST results on MultiWOZ for DSI is lower than that reported in Min et al. (2020). As shown in Section 4.1, the original number was reported by mapping predicted slot types to target ontology at the turn level (before accumulating for the final prediction), where a small subset is used. This process makes mapping more evident (for ex
ample, instead of mapping a predicted slot type to the target 30 slot types, it only compares a slot type to one of two slot types that appear in the reference). Hence, it overestimates the performance and is very different from how a human expert would assign labels when inducing the schema for a new corpus without (turn-level) annotation. Since DST is evaluated to make sure that the slot type matches, an incorrect slot type matching would result in a 0 true positive score. The actual performance in our experiments is thus lower.
In addition, we follow the exactly same settings including training and evaluation scripts (on DST) with their provided pre-processed span-level data and suggested hyperparameters. We use the same metrics and scripts to evaluate all methods. Accordingly, all the numbers reported in Table 2 are fair and comparable.
Lastly, since we use fuzzy matching scores (Rastogi et al., 2020; Min et al., 2020), turn-level performance is accumulated to the joint level. For that reason, different from joint goal accuracy commonly used where all slot types and values are required to be exactly match, partial true positives are counted again in future turns. For example, if the current turn predicts "train leave-at: 10" with the target dialog state "train leave-at: 10:00", even if the next turn predicts nothing correctly, this partial score is counted in the joint level score in the next turn. This procedure follows the setting of Min et al. (2020). In fact, in their reported performance of DSI-GM on MultiWOZ 2.1 with precision and recall of 52.5, 39.3, and 49.2, 43.2 at turn and joint level respectively, the actual F1 scores are actually 45.0 for the turn level and 46.0 at the joint level. Similar to ours, they also received higher score on the joint level due to accumulative partial scores (by calculating F1 using their reported precision and recall scores directly). Since we follow the same evaluation script and metrics, the results and conclusion we have in our experiments comparing different methods are comparable.
A.9 Further analysis of schema induction among datasets (including realistic data)
When we apply our method on internal customer data for slot schema induction, we follow the same pipeline introduced in Section 3. For intent schema induction, we consider both the system turn and utterance turn as the context to our multi-step clustering to find schema with the hierarchy. Because
our method is data-driven and does not require heuristics, it can induce expected slots explained before (e.g. "until the 30th"). We observed empirically satisfying performance but the results cannot be reported publicly because of restrictions. Therefore, we only report results on the public datasets to compare to previous research, as well as to inspire follow-up works for comparison.
We also applied our approach to the Ubuntu dialog corpus (Lowe et al., 2015). Compared to general TOD systems where a user and an knowledgeable agent communicate with each other, this data is collected from online forms to discuss technical issues. The utterances are less conversational, and include coding scripts, making it very noisy. We experiment on this more realistic dataset only for reference, since it is significantly different from building a TOD systems to interact with real users where schema is critical. We sample 8k utterances from the training data, and apply our method on both the intent level and the slot level. On the intent level, our method generates 70 clusters from the first step, and 154 clusters after three steps. Apart from greetings (which appear very frequently), we can induce intents such as suggesting one question is off topic (e.g. "this is a support channel; please leave and go to xxx channel"). There are also some more evident intent clusters such as suggested command line, suggested url, and questions for installations in a specific setup (e.g. "how to installfirefox on 64 bit"). When the input sentence is long with mixed intents, our method may group these into one large cluster (such as providing suggestions to a specific problem, which is more similar to dialog act). We may choose to mix slot- and utterance-level clustering to solve such an issue by treating each complete segment in an utterance as a long span. On the slot level induction, our method generates 36 clusters from the first step, and 287 clusters after three steps. Our method can induce slot types such as "ubuntu version" and "software name". However, compared to MultiWOZ and SGD, the induced slots are much nosier with lower precision where meaningless verbs (e.g. "set up" are grouped). Meanwhile, there a many other slot types that are not meaning such as a cluster regarding part of a path (e.g., "/var/"), which may be due to that we use the same LM trained on TOD dat which does not handle code scripts. Further in-domain pre-training within the Ubuntu dialog corpus may solve this issue. To conclude,
even though this dataset is noisy and different from TOD, our method is still applicable to discover useful schema on both the intent and the slot level without any supervision.
A.10 Applying induced schema on testing data
After inducing the schema on the training data, we may apply the induced schema directly to a different set of data (such as testing data) for downstream applications (such as DST). Since we already induced slot clusters and mapped them to ground truth, we do not need to follow the same span extraction before clustering again. Alternatively, we adopt the following procedure. We extract all candidate phrases in the same way, but instead of clustering, we map the extracted phrases to clustered groups. Specifically, similar to mapping induced latent clusters to ground truth groups in schema induction, we find the most similar latent cluster to the candidate in the contextualized embedding space, and assign the cluster name to the phrase as its slot type. We observed that even though the schema is not induced on the testing data, the performance on both turn and joint level maintains (36.58 and 48.98).
A.11 Related work in schema induction of other natural language processing tasks
Similar to grammar induction and unsupervised parsing, schema induction can help to eliminate the time-consuming manual process and serves as the first step to build a large corpus (Klein and Manning, 2002; Klasinas et al., 2014). Related tasks include event type induction (Huang et al., 2016, 2018), semantic frame induction (Yamada et al., 2021), and semantic role induction (Lang and Lapata, 2010; Michael and Zettlemoyer, 2021). Relationship in these tasks such as predicate and head or patient and agent are relatively evident compared to that in conversational dialog. In addition, most of previous research requires either strong statistical assumptions based on pre-defined parsers, or existing ontologies and annotations for some seen types, and formulate the problem similar to word sense disambiguation on predicate-object pairs (Shen et al., 2021). In contrast, our method does not require any formal syntactic or semantic supervision.
A.12 Incorporating task-specific annotation requirements for schema induction
Our method is data-driven, indicating that if two tokens appear frequently (thus form a span), it might be a good idea to consider them as a slot together. Our motivation here is to induce the most probabilistic schema based on distributed representations. Incorporating annotation requirement is not specific to schema induction from corpus, and is a broader concept of neuro-symbolic integration by merging symbolic rules with connectionist models like neural networks.
However, if there is a specific requirement, we can either inject inductive bias similar to Shi et al. (2019); Kim et al. (2020) to change the attention distribution (so that the requirement-specific bias can result in smaller or larger divergence explicitly). We can also add such requirements as rules directly on certain spans. In this way, we can incorporate the requirements. In comparison, previous methods relying on supervised parser are not applicable.
A.13 Detailed schema induction results
Table 10 shows detailed results comparison on different proposed methods on schema induction. All methods result in a similar number of clusters, while span-based further pre-training methods constrained on unsupervised PCFG structures achieve the best performance overall.
slot type	slot value
method	# clusters	precision	recall	f1	precision	recall	f1
Baseline
DSI	522	96.15	80.65	87.72	41.53	57.40	37.18
Parser only
NP	88	94.44	54.84	69.39	42.26	67.80	47.46
DSI cand.	113	100.00	74.19	85.19	56.46	60.80	49.71
PCFG	339	96.43	87.10	91.53	62.14	58.01	53.62
CoreNLP	292	96.15	80.65	87.72	57.80	63.18	54.43
Language model only
BERT	340	96.00	77.42	85.71	62.11	58.60	55.80
SpanBERT	343	96.30	83.87	89.66	56.34	51.95	45.21
TOD-BERT	219	96.30	83.87	89.66	63.58	57.64	50.89
TOD-Span	374	96.00	77.42	85.71	54.88	69.13	55.29
freq	100	93.33	45.16	60.87	47.31	63.32	45.97
freq w/o stop	337	95.65	70.97	81.48	48.63	63.66	48.27
PMI	369	100.00	80.65	89.29	53.97	73.60	56.38
PMI_seg	551	96.55	90.32	93.33	60.37	66.68	58.33
SCP	374	96.00	77.42	85.71	55.06	61.23	51.78
Language model contrained on unsupervised PCFG
BERT	350	96.15	80.66	87.72	58.85	57.49	52.32
SpanBERT	203	96.30	83.87	89.66	60.54	48.23	44.51
TOD-BERT	245	96.43	87.10	91.53	55.40	57.26	48.13
TOD-Span	290	100.00	93.55	96.67	61.34	67.26	58.71
freq	379	100.00	83.87	91.23	56.67	68.19	57.19
freq w/o stop	315	96.55	90.32	93.33	56.40	66.43	53.74
PMI	335	96.55	90.32	93.33	57.90	67.50	56.91
PMI_seg	275	96.55	90.32	93.33	55.19	65.04	54.54
SCP	290	100.00	90.32	94.92	53.62	65.31	53.00
Table 10: Schema induction results for different proposed methods. High precision scores indicate that the clusters are relatively clean without noises that result in fuzzy representations for mapping.