Log-FGAER: Logic-Guided Fine-Grained Address Entity Recognition from Multi-Turn Spoken Dialogue
Authors: Xue Han
Yitong Wang
Qian Hu
Pengwei Hu
Chao Deng
Junlan Feng
Venue: EMNLP 2023 Main (2023)
Keywords: Fine-grained address entity recognition
probabilistic soft logic
address extraction
data augmentation

Abstract: Fine-grained address entity recognition (FGAER) from multi-turn spoken dialogues is particularly challenging. The major reason lies in that a full address is often formed through a conversation process. Different parts of an address are distributed through multiple turns of a dialogue with spoken noises. It is nontrivial to extract by turn and combine them. This challenge has not been well emphasized by main-stream entity extraction algorithms. To address this issue, we propose in this paper a logic-guided fine-grained address recognition method (Log-FGAER), where we formulate the address hierarchy relationship as the logic rule and softly apply it in a probabilistic manner to improve the accuracy of FGAER. In addition, we provide an ontology-based data augmentation methodology that employs ChatGPT to augment a spoken dialogue dataset with labeled address entities. Experiments are conducted using datasets generated by the proposed data augmentation technique and derived from real-world scenarios. The results of the experiment demonstrate the efficacy of our proposal.

Log-FGAER: Logic-Guided Fine-Grained Address Entity Recognition from Multi-Ture Spoken Dialogue
Xue Han'*, Yitong Wang"*, Qian Hu', Pengwei Hu', Chao Deng', Junlan Feng'
UJUTTAN Team, China Mobile Research Institute, Beijing, China
The Xinjiang Technical Institute of Physics and Chemistry, Urumqi, China (hanxueai, wangyitongyil, huqianai, dengchao, fengjuunlanJechinambblleee. thail com
Abstract
Existing name entity recognition (NER) meth- ods usually combine the pre-trained language mod- cls (c.g. BERT) with supervised models such as MILSTM/LSTM--A
Fine-grained address entity recognition (FGAER) from multi-turn spoken dialogues is particularly challenging. The major reason lies that full address often formed through a conversation process. Different parts of an address are distributed through multiple turns of dialogue with spoken noises. It is nontrivial to extract by turn and combine them. This challenge has not been well emphasized by main-stream entity extraction algorithms. To address this issue, we propose in this paper logic-guided fine-grained address recognition method LLog-FGABR where we formulate the address hicrarchy relationship as the logic rule and softly apply it in probabilistic manner to improve the accuracy of FGAER. addition, we provide an ontology-ban data augmentation methodology that employs ChatGPT to augment a spoken dialogue dataset with labeled address entities. Experiments are conducted using datasets generated by the proposed data augmentation technique and derived from real-world scenarios. The results of the experiment demonstrate the efficacy of our proposal.
(Xu et al., 2019; Zhang et al., 2020; Mai et al., 2018) or with some expansive ex- ternal knowledge base (Dogan et al., 2019). These methods, which are trained and evaluated on text from well-written sources, do not perform well in spoken dialogue context, because the enti- ties in the spoken scenario are distributed across a multi-turn context with a lot of noise. Other studies explore the NER problem in a spoken or dialogue context (Muralidharan et al., 2021; Chen et al., 2022; Luoma and Pyysalo, 2020; Hanh et al., 2021), but their labels are not applicable to the address domain. Because of the diversity and com- plexity of multi-turn spoken dialogue scenarios, annotating for the FGAER task is laborious and time-consuming (Shan et al., 2020).
To address these issues, we propose logic- guided fine-grained address entity recognition method (Log-FGAER), The Log-FGAER based on the fact that address entities follow spatially con- strained relationships (Zandbergen, 2008). As re- sult, the multi-turn spoken dialogue must maintain discourse structure to get the full standard address for the duration of address acquiring in accordance with the spatial constraint relationship between the geographical address entities. Figure (a)) shows an example of a 4-level spatial dependency tree structure, in which each node is a fine-grained ad- dress entity, each level represents an entity type (e.g. "'ity" and "district"), and each edge is a dependency relation between the entities. A full standard address is made up of nodes in the path from the root the leaf node. We can improve entity recognition inference by presenting such an interdependent hierarchy relationship as the logic rule and then using probabilistic soft logic (PSL) to regulate the loss objective.
1 Introduction
Fine-grained address entity recognition from calls is an important task in many applications (Wu and Juang, 2022; Yang et al., 2022), such as obtain- ing delivery addresses from E-commerce assistant post-sale service (Eligüzel et al., 2020). To be specific, the scenario considered in this paper is the extraction of fine-grained address entities dis- tributed through the multi-turn spoken dialogue contexts. Figure depicts an example of such a dialogue between a customer and an after-service call center in which the fine-grained address enti- ties are recognized and then typically combined to form a full standard address for further delivery.
These authors contributed to the work equally.
The corresponding author.
We further introduce an ontology- entoloyybbased data aug-

---

mui-turn dialogue contexts
Where are you Surhou city Hangsu Prov2
Jiangau Province 1E39IE
level 0:Province
SaNo BONS WuR
I'm in Wujjang District, Shengze Town Wujjang District. Rage antieesa
Yanchong City
Buzhou City
level 1:Clty
Okay, we will send someone professional to contact you. aHA, MyReendd AMBS
Xiangshui District Gusu District
Wujiang District BSTR
+ level 2:Dtstrict
Address Extraction
Yunho Town 0E
Bujin Town Shalic
Shengpe
level 3:Town
Jlangau Province AL 9
Sushau City 25-11
Wujlang Risinic MiI
Shengse Town! 1E
Full Standard Address
(a)
(b)
Figure 1: An example of location extraction task from multi-ture turn dialogue. hierarchy relationship address entities extracted from (b).
The path in red in (a) reflects the
mentation paradigm to build labeled multi-turn dialogue dataset for the FGAER task, which could inspire future research. The paradigm first extracts utterance and response (UR) templates from pub- licly available dialogue datasets and then leverages the largely available full standard address to get labeled fine-grane rreeggained address entities. The address entities in the UR templates could be altered using the technique we propose to expand current dia- logues. To imitate real-world conversation circum- stances, noise injection approach and ChatGPT augmentation method are also proposed.
competitive or even superior performance.
Related Works
2.1 Fine-grained Named Entity Recognition
Fine-grained named entity recognition aims solve the restriction of coarse-grained NER that produces labels from only small set of entity classes. To solve the lack of datasets where entity boundaries are properly annotated while covering a large spectrum of entity types, (Dogan et al., 2019) proposes to combine learning models (ELMo) with an expansive knowledge base (Wikidata). (Mai et al., 2018) improves the neural network-based Japanese FG-NER performance by removing the CNN layer and utilizing dictionary and type embed- dings. The most similar work with is DLocRL (Xu et al., 2019), which trains a BiLSTM-CRF based POI recognizer for fine-grained location recognition and linking in tweets. This work only considers the POI locations, which is one of the entity types in our scenario.
We conduct experiments on both the created dataset and a small set of human-labeled datasets derived from real-world customer care conversa- tions. Experimental results show that Log-FGAER improves baselines in both cases.
The contributions of this paper are summarized follows:
We make use of probabilistic soft logic to im- prove the FGAER tasks after utilizing logic rules to indicate the interdependent hierarchy relationship between fine- fineggrined address en- tities, which reflects the discourse structure of address-acquiring dialogue.
2.2 Named Entity Recognition for dialogue
In dialogue systems, NER is a vital task for under- standing user intent but suffers from the problems that NE labels are too coarse-grained or the entity types are not linked to a useful ontology. (Muralid- haran et al., 2021) proposes a method of encod- ing the external knowledge of entities not directly using the toponym dictionary but by obtaining in- formation from the knowledge graph. (Bowden et al., 2018) leverages external knowledge such as the Google Knowledge Graph API to solve the problems in the dialogue contexts. However, these methods have quite different entity types, which are not applicable for our scenario.
We present an ontology-based and ChatGPT assisted data augmentation paradigm for gen- erating multi-turn spoken dialogue with la- beled fine-grained address entities requiring little human effort, thereby addressing the is- sue scarce labeled datasets and high label- ing costs.
Logic-guided method is plug-and-play mod- ule compatible with all existing FGAER tech- niques. Experiments indicate that our Log- FGAER with PSL enhancement can obtain

---

2.3 Probabilistic Soft Logic
reasoning is incorporated as additional supervision signals regulate the loss objective of FGAER task. We first introduce several concepts and no- tations in the PSL language following (Cai et al., 2022), and illustrate how the PSL rule is used to define templates for the hierarchical constraint de- pendency inference in FGAER task.
recent years, PSL rules have been applicable to various machine learning topics. In the knowledge graph construction area, (Chen et al., 2019) infers the confidence score of unseen relations by intro- ducing PSL to represent the uncertain knowledge graph. In the temporal relation extraction topic, (Zhou et al., 2021) leverages PSL regularization to jointly train relation classifier with sentence em- beddings from deep language model and perform the global temporal inference with a time graph. As for causal inference, (Du et al.. 2021) acquires additional evidence information from a large-scale causal event graph logical rules for causal rea- soning. (Cai et al., 2022) proposes neural-logit based soft logic enhanced event temporal reasoning model for acquiring unbiased temporal common sense knowledge. (Mohler et al., 2020) models the procedural state changes over time with PSL. Most of the works inject the logic knowledge to neural networks, by introducing logic-driven loss functions. These works encourage us to apply PSL rules to the FGAER task.
Definition The atom formula / is made up of a predicate p together with its arguments. takes on continuous values in the unit interval [o, 1].
We define three kinds of predicates for our see- nario: HIGH, LOW, and BETWEEN. HIGH and LOW both take two arguments to work: an in- put sequence and the type of address entity. BE- TWEEN takes three arguments: an input sequence and two address entity types. For example, the atom HIGH(S, city) represents that the highest level of address entity containing in the sequence S is city. The atom BETWEEN(S, cily, lown) means that the types of entities in S are between city and town in the hierarchy structure.
Definition 2. Complex formulae are created by combining atom formulae with logical operations (e.g., conjunction (A) and disjunction (V)).
3 Method
Definition 3. A logic rule : + f2 is an implication formed by combining fi and f2 with logical connectives. fi and f2 can be atomic or complex formulae, respectively.
3.1 Preliminary
Problem Formulation. The FGAER task is for- mulated as a sequence labeling problem (Xu et al.. 2019). A multi-turn spoken dialogue is denoted as several turns of sequences D (S,, Si), where S; represents one turn of question and answer in the dialogue. The FGAER's input S, is defined as {si, sy} where sj denotes the juh token. The fine-grained address entities are spread across in D. The output is a tag sequence Y; = (.,,). for each input S;, where Yj 7. y is the set of pre- defined tags following the BIOE tagging scheme. Each tag (e.g., B-city and I-district) in y is indi- cated by B-(begin), I-(inside), and E-(end) of fine-gratd fine-grained address entity with its type in a type set. For a character that not inside of any entity, its tag is O-(outside). The FGAER task aims at discovering a list of address entities (E",..,,"") E").
Definition 4. The soft truth values of the formula and the logic rule are denoted by the interpreta- tions (f) and I (r), which refer the probability that f and hold.
Definition 5. Eukasiewicz t-norms (Klir and Yuan, 1995) adopted to relax logic in PSL are de- fined as below. With the t-norms to relax logic, we can turn the logie rule (r) and simple or complex formulas (f) into differentiable functions, which can then be used as learning goals to control the loss objective: ((fiff) = maz ( 1,0) and [rr : fi + f2) = min(1, I(fi) + I(f2)
With the above definitions, can now define the logic rule that explain how the address entities in the current turn of dialogue sequence relate to those in the previous and following turns.
E - [Sstaat;.... Send] is substring of S; satisfy- ing start < end.
Probabilistic Soft Logic. As mentioned before, the discourse structure of multi-turn spoken di- alogue usually reflects the spatial constraint rela- tionship between the geographical address entities, which can be modeled with Probabilistic Soft Logic (PSL) (Kimmig et al., 2012). The PSL probabilistic
3.2 Log-FGAER
Definition 6. Since multi-turn spoken dialogues maintain a discourse structure to get the full stan- dard address during address acquisition, we can assume that the address entity types mentioned in

---

PSL Ruis and Regulartzation:
prior Requence:
The loveet leval
BETWEEN
SUCOII InsuL ESDUROLE
(6722-67-0.011
DMAT sequence:
The Righeet lewel
DETWEEN (S, mint1, max 742.:/0-..62222........-
Hararchical Atom:
LOWis .nly
100 Aeaan BETWEENS, twrmd
I TRNIN +0
Input
Calculation Example
Culpus
CRF
ERNIE encoder
ERNIE encoder
b)
Finetune encoder
FGAER model
The main framework of Log-FGAER. a) The inputs of the encoder are: the current turn of the sequence and sequences before and after it. byThe ERNIE encoder fine-tuned using PSL regularization to learn the spatial hierarchy constraints from the contexts for the current imput sequence. An example of how to calculate the logic regularization. d) The embedding from the trained (PSL)ERNIE is then sent to the CRF layer for the FGAER task.
Figure 2:
the current sequence S, are higher in the adminis- trative division hierarchy than the highest level of address entity types in the next sequence S but lower than the lowest level of types in the previous sequence S,-1. Such a kind of relationship could be formulated as logic rule Raddr A 11 l2 + f:
More specifically, the PLM encoder accepts sequence input that may contain address entities and then outputs a token-level embedding of the sequence. The ERNIE (Sun et al., 2019) pro- posed as the PLM backbone because has been en- hanced for named entity recognition by incorporat- ing entity semantic information. Given S, as input, we could get the embedding H; = (,))) where Hę; = { .... h2} and hj denotes the out- put of the pre-trained ERNIE that corresponds to the input token 8j E S,. The predicted probability distribution Úy for 8j then obtained by adding on top of the ERNIE linear layer Freed-Forward Network) with softmax, as shown below:
4 (5.(5 A : (SGH()
(1)
. BETWEEN(S,, a,
Eq 1, the atom formulae 1 LOW(S, in- dicates that the lowest-level address type of S;-1 is a. Similarly, l2 HIGH(S b) means that the highest-level address type of S is b. The address entity types contained in S; are generally between and b in the hierarchy structure. Because the hierar- chical logic rule Raddr have the form A 12 we can obtain a soft truth value by upplying the Lukasiewicz t-norm the interpretation of Radddr Figure 2 illustrates the framework of Log- FGAER, which consists of two components: 1) a pre-trained language model (PLM) encoder that is fine-tuned with PSL regularization; and 2) an FGAER module that is composed of the fine-tuned PLM encoder and CRF layer.
1 (()) softmaa
(2)
In the above equation, W and b are trainable parameters. Yj is the ground-truth label for 8j- We compute the prediction loss Lee using the cross-entropy objective as below:
Lice
[. Σ CrossEEntoopu(us.. 0.)
1
(3)
Following that, we will show how to compute the logic regulation loss Lingie and then combine it with the Lee to ensure that the predicted distribution Új is compatible with the hierarchical logic rule Raddr, shown in Definition 6. To calculate Liogic should get I(Radar) first, which is the truth value of logic rule Raddr- Fig- ure 2(c) illustrates an I(Raddr) calculation exam- ple. I(LOW (S,, city)), also known as T(I1), in- tended to be the predicted probabilities of the char- acters "Hangzhou city" to be the entity type of
1) Fine-tune the PLM encoder with PSL reg- ularization. In this component, the PSL rule from Definition 6 is used to calculate the logic regulation loss, which is then combined with the original PLM model's cross-entropy loss to fine-tune the PLM encoder. In this manner, the embeddings generated by the fine-tuned PLM encoder would incorporate the spatial constraint dependency information in the dialogue context.

---

"city". Similarly, I(BETWEEN(S; city, town)), also known as I(f), represents the predicted proba- bilities of "Jianggan District" and "Jiubao Town" in St. whose entity types are between "city" and "town". After getting the values of 1(4), 1(12) and I(f), I(Raddr) which denotes the degree that the rule Raddr are satisfied by the predicted dis- tribution, can be derived by incorporating the Lukasiewicz t-norm.
simply crawled from web pages such as Google Maps'. Then leveraging the recently well-studied address entity segmentation methods(Chen et al., 2015), which refers to the process of splitting un- structured standard addresses into address enti- ties, we were able to label the full standard address data with fine-grained entity types with minimal human effort.
The principle of PSL is: the larger the truth values are, the better the logic rules are satisfied, Based on this principle, we formulate the distance of I(Radrr) to be true as a regularization term to penalize the distorted distribution that violates the logic rule Raddr (Cai et al., 2022). The logic regu- lation loss Loogic is calculated as below:
Are currently Hangzhou City? Cofrolie,MMnn Yeo, I'm in Baiyang Town. Jianggan District asdendss
Ontelogy ?
City Mergeroa Town Rekyling
Are you currently [Slot] Yes. I'm in [Slot]. [Slot]
slot entity category
City, Cistfir, Town
Are you currenily in Longyan City? oweeeaaeze
Ontology
Yos, Xichang 400 10c wow
Xiniuo District.
Giesier Xankn
Figure 3: Ontology- based dialogue enrichment.
Llogic = 1 ((aadr))
By using the labeled full standard address, the ontology- basee dialogue data enrichment method can expand the limited sampled UR template pairs that include address entities, extracting from cur- rent small-size labeled dialogue datasets. Take Figure 3 as an example to explain our enrichment method: "Hangzhou city", "Jianggan District" and "Baiyang Town" in the origin UR pair are replaced by new values in ontology @. About 578 groups of UR pair templates we used to create dialogue datasets are extracted from the CUCC call center dataset² and the Taobao E-commercial Conversa- tion Corpus 3. The candidate values for each of the ontologies are derived from Tianchi Competitions' full standard address dataset, which has 8.8K full standard addresses with 16 types. More full stan- dard address data can be crawled from websites and labeled using address segmentation methods with limited efforts following (Chen al., 2015).
We finalize the loss function by applying weighted sum of Eg 3 and Eq 4:
LERNIE = -oco + Aooogi
(5)
where À is the weight for PSL regularization term. The objective is to minimize the LERNIE and to finetune the ERNIE encoder parameters.
With the regularization of PSL rule on the out- puts of the ERNIE encoder, our model can make embeddings capture more contextual information, which could further benefit the sequence labeling task described in the FGAER module.
2) FGAER module is made up of the ERNIE en- coder that has been fine-tuned with PSL regulation and CRF layer (Lafferty et al., 2001), denoting (PSL)ERNIE-CRE The loss function Lenr of FGAER is calculated by feeding H from Eq into the CRF layer. The objective to minimize LCRF The trained FGAER model could be used in the inference stage for the FGAER task.
4.2 Noise injection
To bridge the gap between actual customer support scenarios and ontology-based meethod-generatee dialogues, we present the noise injection method, which captures the typical speaking habits of peo- ple in real- situations. To be specific, we im- plement five strategies following the prior works (Tian et al., 2021; Feng et al., 2021; Liu et al., 2020): 1) Pause, 2) Repeat, 3) Restart, Inserr,, and 5)Repair. The details could be referenced to the above mentioned work. To generate the in-
4 Ontology Based Dialogue Data Augmentation Leveraging ChatGPT
4.1
Ontology-based dialogue enrichment
The goal of saooloyy-based dialogue enrichment is to address the issue of shortage of dialogue datasets with labeled fine-grained address entities. As previously stated, labeling the fine-grained en- tities is costly and time-consuming. Fortunately, large-scale, unlabeled, full standard addresses (for example, Ecommerce Park, Hanghai Road, Jiubao Town, Jianggan District, Hangzhou City) could be
2 hhtps:/wwww.googl..co wttp:///ontttttt.ooooooiioonaaaa https://github hhaps:/giituu..oo//oo////tt////p http:///wh....ooooooe/////ee////

---

Original texts
Altered texts
U: Please provide the detailed address, where are now?
U: Could you kindly furnish specini address of your location?
Can you plews share the precise wheresbouts of your preseni position?
Sentence level
U: May I request the claborate address your current whereabouts?
R.My current location Hangzhou, located Zhejinn Province.
R: I am in Hangzhou, Zhejiang Province.
R.1 lind myself in Hangahou, which is situiied in Zhejann Province.
R.1 am locared Hangzhou, specifically Zhejiang Province.
U:Repeal plewse I didn't catch what you suid clearty.
R.I currently
dental bospital.
Dialogue level
U: Repeat please, dida't hear you clearly.
U:Could you pleuse repeat that? Your wards weren't I clear to me.
I am in the dental hospital.
R.I currenlly located the denial hospital.
U:l apologize for the incorvenience Could you repeat your statement?
R.I am currently in the dental hospical.
Table 1: Generated examples from ChatGIT augmentation method.
correct pronunciation, we employ xpinyin and Pinyin2Hanzi to mutually convert pinyin and Chi- nese characters.
Lofaa cook staede
4.3 ChatGPT improves diversity
We employ ChatGPT powerful data augmenta- tion strategy (Ubani et al., 2023), to increase data diversity by improving current UR templates and transforming created dialogues (after Noise injec- tion) into new forms that differ from the original but retain the same meaning.
Sertanceen
Figure The average Cosine Similarity, Levenshtein Ratio, Jaro Distance, Baro-Winkler Distance, and Jac- card Similarity for 100 pairs of generated samples and original samples are listed individually at the sentence and dialogue levels.
We have designed prompts for rephrasing at the sentence level (UR pair templates) and dia- logue level (after Noise injection). The specific prompt for sentence-level rephrasing "Please rewrite the following sentence give: (texi), and give me rephrased results". The number UR pair templates is increased from 578 to 778 for Ontology- -ased dialogue enrichment. The prompt for dialogue-level rephrasing is "Please rewrite the following dialogue give: (text), and give 2 rephrased answers while remaining the number of sentences". The amount of generated dataset ex- ceeded Ik after ChatGPT enhancement. Table shows the samples generated using ChatGPT. We also consider the degree of similarity between the new samples and the ones we already have, as il- lustrated in Figure 4. We are able to better capture data invariance and increase the sample size by leveraging ChatGPT's outstanding generating ca- pabilities.
FGAER-dia is labeled multi-turn dialogue dataset of 9.8k groups of dialogue contexts with 16 types of fine-grained address entities. The generated di- alogue's average length of characters is 167 and the maximum length is 366 in FGAER-dia. The maximum number of turns is 10. Our proposed dataset and the code for our Log-RGAER method are openly available?.
CUCC-labeled The CUCC call-center dialogue dataset provided by DataFountain Competition? is transcribed by real-world downstream speech recognition system, including plenty of transcrip- tion errors. We filter out 200 dialogues containing address entities and manually label the filtered data with 16 different types of address entities, to eval- uate the Log-FGAER's performance on the multi- turn dialogues from real-life service environment. Table 5 in Appendix contains more detailed ex- amples of fine-grained address entity types.
5 Experiments
5.2 Implementation details
5.1 Datasets
FGAER-dia We create a labeled multi-turn dia- logue dataset called FGAER-dia, which is con- structed according to the paradigm in Section 4.
In our experiments, ERNIE is performed with BaiDu PaddlePaddlc 8, which is a framework of deep learning. For hyperparameter settings, the max sequence length of wex-eext,
httgss///ggttibboom////r/ll/ttot hhtps://www.paddlqpddd.ccc

---

FGAER-dia
CUCC-labeled
Methods
P
R
F1
P
R
F1
Baseline-I
92.32
91.09
91.55
65.72
61.72
61.75
BERT-based
Baseline-2 2
92.26
91.62
91.83
72.43
67.74
68.53
Baseline-3
90.85
90.88
90.86
64.66
64.64
63.35
RoBERTa-based
Baseline-4
92.79
91.68
92.21
69.59
65.77
66.86
Baseline-5
93.60
93.23
93.41
69.77
73.37
70.54
Baseline-6
92.10
92.46
92.28
83.39
83.99
83.69
Baseline-7
93.28
92.78
93.03
87.47
82.91
85.13
ERNIE-based
Baseline-8
93.15
91.68
92.41
82.10
84.17
83.12
Log-FGAER
94.15
93.63
93.88
86.20
87.12
86.65
Table Experiment results on FGAER-dia and CUCC-laheled datasets, and best performance highlighted boldface.
BERT, and ERNIE is 256. The hidden state size BiLSTM 128 and the dropout rate set as 0.1 for BiLSTM output. We use the A opti- mizer in mini-batch size of 64 with learning rate 7 5 x 10 5 for each experiment. The weight decay of ERNIE is 0.01. The best À is 0.15. The split of 80% for train, 10% for validation and 10% for test is used. The metrics used for evaluation are precision(P), recall(R) and Micro-FI metrics. The models with the best validation loss that are selected from all epochs, are saved and utilized for testing, For hardware, we use a four-core CPU and an NVIDIA Tesla V100 GPU.
Baseline-7 based on the ERNIE-CRF struc- ture.
Baseline-8 applies ERNIE-BILSTM-CKF to deal with FGAER task.
5,4 Results and Analysis
The following insights can be drawn from the re- sults as listed in Table 2:
1) Compared baselines using the same model structures but different PLM encoders(e.g.. Baseline-2, Baseline-4 and Baseline-8 all add BiLSTM-CRF layers encoders), ERNIE significantly outperforms FGAER task.
5.3 Baseline models.
We compare the performance of Log-FGAER with several strong baseline methods listed below.
2) We compare the state-of-the-ar (SOTA) struc- tures (Baseline 1-8) for the FGAER task. On the FGAER-diz dataset, Bascline-7 achieves better FI- score performance. As result, we combine the proposed Logic-guided method with Baseline-7 to obtain the Log-FGAER.
Baseline-1 uses the structure of Berth-CRF
Baseline-2 employs BERT-BILSTMM-RRF as the structure.
3) With the Logic-guided method enhanced, Log- -Gg-FGAER outperforms other basclines on both FGAER-di and CUCC-labeled datasets.
Baseline-3 adopts Chinese RoBERTa-wwm- 1-CF as the architecture.
To further display the progress of Log-RGAER, we exclusively all entity types with an Fl-score below 80%. As shown Table 3 and Figure 5, Log- FGAER can efficiently enhance the F1-scores of almost all entity types that are prone to prediction errors. In particular, the Log-FGAER consistently outperforms Bascline-7 (same structure without PSL) by 3.28 percent F1-score on entity type "subpoi". These findings demonstrate that the logic inference introduced our method can capture and utilize geographical hierarchy information con- tained in the context.
Baseline-4 adds BiLSTM layer between RoBERTa-wwm-ext and CRF layers.
Baseline-5 has the same structure as Baseline- 4, using a distillation approach to transfer knowledge from labeled full standard ad- dresses dialogues (Wang et al., 2022).
Baseline-6 only finetunes the ERNIE encoder.
"htnps//hugginggfaee.c htt.::/innnnnaecocceeerbebbaeeee hhtp:////iiiaa.ooooyommuuccscits

---

Methods
DISTRICT
COMMUNITY
POI
SUBPOI
INTERSECTION
overall
Baseline-6
70.45
62.36
65.86
65.79
70.10
92.28
Baseline-7
70.88
62.91
66.43
67.18
70.48
93.03
Baseline-8
70.13
62.30
66.13
58.07
70.50
92.41
Log--GAAE
70.98
63.12
66.28
70.46
71.38
93.88
Table 3: Fl-scores of Log-FGAER and ERNIE-based baselines on FGAER-dia.
better than comparable approaches in multi-turn dialogues. We compare to CMV(Luoma and Pyysalo, 2020) and Multi-head Attention methods. CMV(Luoma and Pyysalo, 2020) explores cross- sentence contexts for NER with BERT. Multi-head Attention employs the ERNIE- CRN structure to produce novel representations with context infor- mation. This is accomplished by adding address tities from the preceding and subsequent sequences to the current sequence. Take Figure 2(a) as an example: "Hangzhou City" is added the front, while "Jiubao Town" is inserted at the end of S,, forming a new representation as the input. In this way, the contextual information could be learned by Transformer multi- attention mechanism. The comparisons are also evaluated on the FGAER-dia. Log- FOg-RGAER outperforms the other two ap- proaches, as shown in Table demonstrating the efficiency our logic-guided method in exploiting contextual address hierarchy information.
Figure 5: Fl-scores of Log-FGAER and ERNIE--basdd baselines FGAER-dia.
5.5 Ablation Study
Additional experiments were conducted to demon- strate the efficacy representing contextual in- formation as logic rules that better illustrate the contextual information than comparable methods.
1) To begin, we demonstrate how our proposed method can be utilized an add-on module to improve existing sequence labeling models. We obtain (PSL)ERNIE and (PSL)ERNIE-BiLSTT)) CRF after merging the proposed Probabilistic Soft Logic-guided methodology with Baseline- 6 (ERNIE) and Baseline-8 (ERNIE-BiLSTME CRF). As shown in Figure 6, both ERNIE and URNIE-BiLSTM-CRF improve over various entity types with PSL enhancement. The PSL strategy has proven to be an independent and simple-to- implement module that can be applied to traditional sequence labeling models to make significant im- provements.
Methods
R
F1
Multi-head attention
91.87
92.02
91.94
CMV
93.09
93.17
93.12
Log-FGAER
94.15
93.63
93.88
Table 4: Results of models using different methods obtain contextual information FGAER-dia.
Conclusion
In this paper, we propose an effective method named LoA-R to extract address entities from spoken dialogue contexts using the PSL rule. Lever- aging the spatially constrained relationship of ad- dress entities in the dialogues, we improve entity recognition inference by presenting the interdepen- dent hierarchy relationship as the logic rule to reg- ulate the loss objective. We also propose data augmentation paradigm construct the labeled multi-turn dialogue dataset for training and eval- uation our proposed method. Experiments on the multi-turn spoken dialogue dataset show the effectiveness of our method.
Figure 6: scores (PSL)ERNIE and (PSI)ERNIE- BiLSTM-CRF models against the same structures with- out PSL regularization on FGAER-dia.
2)Second, we attempt to demonstrate that the PSL strategy expresses contextual information

---

Limitations
Annual Meeting the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1744-1753.
While our model achieves satisfactory results, it still has several limitations. This study has some potential limitations based on the datasets. Due to the lack of large-scale bench datasets, we cre- ated the FGAER dia and CUCC-labeled datasets, which were both originally constructed in Chinese. addition, both of these datasets contain small amount of data. This can lead to an increasing risk of biased or incomplete conclusions and gen- cralizations when applied to real-world scenarios. These challenges and constraints will be addressed in future work.
Xuelu Chen, Muhao Chen, Weijia Shi, Yizhou Sun, and Carlo Zaniolo. 2019. Embedding uncertain knowl- edge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3363- 3370.
Cihan Dogan, Aimore Dutra, Adam Gara, Alfredo Gemma, Lei Shi, Michael Sigamani, and Ella Wal- ters. 2019, Fine-grained named entity recognition using elmo and wikidata.
Li Du Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2021. Excar: Event graph knowledge enhanced ex- plainable causal reasoning. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 2354-2366
Ethics Statement
This work adheres to ethical guidelines and respon- sible practices in ACL policies. The authors have external confficts of interest declare and have not been required to seek any ethics clearances in order to undertake this work. The datasets used in this work are from previously published works or competitions and, in our view, do not have any at- tached privacy or ethical issues. We provided data augmentation approaches to improve the evalua- tion of our proposal. The augmentation-gennet conversations aim to provide an optional way to in- spire research rather than violate ethical guidelines and should be used in strictly controlled manner. Transparency and responsible reporting are priori- ties, and we welcome community engagement to enhance ethical considerations in our work.
Nazmiye Eliglizel, Cihan Çetinkaya, and Türkay Dereli. 2020. Comparison of different machine learning techniques on location extraction by utilizing geo- tagged tweets: A case study. Advanced Engineering Informatics, 46: 16:10115: 151.
Steven Y Feng. Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation ap- proaches for nlp. arXiv preprint arXiv: 105.03075.
Tran Thi Hong Hanh, Antoine Douce, Nicolas Sidere, Jose G. Moreno, and Senja Pollak. 2021. Named entity recognition architecture combining contextual and global features. International Conference Asian Digital Libraries, page 264 226.
Angelika Kimmig, Stephen Bach, Matthias Broecheler. Bert Huang, and Lise Getoor. 2012. A short intro- duction to probabilistic soft logic. In Proceedings of the NIPS workshop on probabilistic progranming: foundations and applications, pages 1-4.
References
Kevin K. Bowden, Jiaqi Wu, Shereen Oraby, Amita Misra, and Marilyn Walker. 2018 Slugnerds: A named entity recognition tool for open domain dia- logue systems.
George Klir and Bo Yuan. 1995. Fuzzy sets and fuzzy logic, volume 4. Prentice hall New Jersey.
Bibo Cai, Xiao Ding, Bowen Chen, Li Du, and Ting Liu. 2022. Mitigating reporting bias for semi-supervised temporal commonsense inference with probabilistic soft logic. The Siitt AAAI Conference on Artificiul Intelligence.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li. Weiran Nie, Cheng Li, Wei Peng, and Minlie Huang. 2020. Robustness testing of language understanding in task- dialog. arXiv preprint a222:2201..15662.
Boli Chen, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Meishan Zhang, and Fei Huang. 2022. Aishell-ner: Named entity recognition from chinese speech. In ICASSP 2022 2022 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 8352-8356.
Jouni Luoma and Sampo Pyysalo. 2020. Exploring cross-sentence contexts for named entity recogni- tion with bert. In Proceedings of the 28th Inter- national Conference on Computational Linguistics. page 904-914.
Xinchi Chen, Xipeng Qiu, Chenxi Zhu, and Xuan-Jing Huang. 2015. Gated recursive neural network for chi- nese word segmentation. In Proceedings of the 53rd

---

Khai Mai, Thai Hoang Pham. Minh Trung Nguyen. Tuan Due Nguyen, Danushka Bollegala, Ryohei Sasano, and Satoshi Sekine. 2018. An empirical study on fine-grained named entity recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 711-22
Speech and Signal Processing (ICASSP), pages 7747- 7751.
P.A. 7andhergen. 2008. A comparison of address point, parcel and street geocoding techniques. In Comput. Environ. Urban Syst. volume 32, page 214-232.
Michael Mohler, Sean Monahan, and Mare Tomlinson. 2020. Modeling procedural state changes over time with probabilistic soft logic. In The Thirty-Thit International Flairs Conference.
Haisong Zhang. Lemao Liu, Haiyun Jiang, Yangming Li, Enbo Zhao, Kun Xu, Linfeng Song, Suncong Zheng. Botong Zhou, Jianchen Zhu, al. 2020. Texs- mart: text understanding system for fine ane-graieed ner and enhanced semantic analysis. arXiv preprint 20..2001..1633
Deepak Muralidharan, Joel Ruben Antony Moniz, We- icheng Zhang, Stephen Pulman, Lin Li, Megan Barnes, Jingjing Pan, Jason Williams, and Alex Acero. 2021. Dexter: Deep encoding of external knowledge for named entity recognition in virtual assistants.
Yichao Zhou, Yu Yan. Rujun Han, J Harry Caufield, Kai-Wei Chang, Yizhou Sun, Peipei Ping, and Wei Wang. 2021. Clinical temporal relation extraction with probabilistic soft logic regularization and global inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14647- 14655.
Shuangli Shan, Zhixu Li, Qiang Yang, An Liu, Lei Zhao, Guanfeng Liu, and Zhigang Chen. 2020. Geo- graphical address representation learning for address matching. World Wide Web, 23(3):2005-2222
A Appendix
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng. Xuyi Chen, Han Zhang, Xin Tian, Danxiang 7hu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced represen- tation through knowledge integration. arXiv preprint arXiv: 0900:9904.022..
Label
Interpretation
Exammle
PROVINCE
name of province
Zhejiang Province
BL411T:
CITY
name of city
Tangzhou City
Xin Tian, Xinxian Huang, Dongfeng He, Yingzhan Lin, Siqi Bao, Huang He, Liankai Huang, Qiang Ju, Xiyuan Zhang, Jian Xie, et 2021. Tod-da: Towards boosting the robustness of task-oriented di- alogue modeling on spoken conversations. arXiv preprint arxiv:2112.12441
DISTRICT
name of districr in city
Xibu District
TOWN
name town or boulevard
ROAD
read
Hanghai Road
XX
INTERSECTION
road junction
Intersection
ROADNO
rad mumher
#5
Solomon Ubani, Suleyman Olcay Polat, and Rodney Nielsen. 2023. Zeroshotdataaug: Generating and aug- menting training data with chatgpt. arXiv preprint arXiv:2304.1 14334.
DEVZONE
ceonomic development
Development Zone
TE
COMMUNITY
community village
Ningun Community
R
name the point of
Qushoa Hospital
Yitong Wang. Xue Han, Feng Zhou, Yiting Wang, Chao Deng, and Junlan Feng. 2022. Distill-aer: Fine- grained address entity recognition from spoken dia- logue via knowledge distillation. In NLPCC 2022: Natural Language Processing and Chinese Comput- ing, page 643-655.
SURPOI
name second point interest
Radiology Department
[b
phrase indicating relative position
Opposite
100%
DISTANCE
arnunii of space between two points
100m
HOUSENO
house number
Block 83
24
Ting- Wei Wu and Biing-Hwang Juang. 2022. Knowl- edge augmented bert mutual network in multi- turtt spoken dialogues. ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7487-7491.
CELLNO
cell number
Unit A2
FLOORNO
florr numher
Level 60
Table 5: Interpretations and examples for each entity type used in our datasets, according to the address hier- archical order from top bottom.
Canwen Xu, Jing Li, Xiangyang Luo, Jiaxin Pei, Chen- liang Li, and Donghong Ji. 2019. Dloerl: deep learning pipeline for finc-grained location recogni- tion and linking in tweets. In The World Wide Web Conference, pages 3391-3397.
Jiudong Yang, Peiying Wang. Yi Zhu, Mingchao Feng. Meng Chen, and Xiaodong He. 2022. Gated multi- modal fusion with contrastive learning for turn-taking prediction in human-robot dialogue. ICASSP 2022 -022 IEEE International Conference on Acoustics,


Reviews:
[{"review_id": "EHiv0yjcgv", "reviewer": "Reviewer_NiYR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful suggestions, which will significantly improve the paper.\n\n__Question__: As stated in Definition 6, the proposed approach assumes that addresses become more fine-grained along the dialog, that is, an address entity type that is higher in the administrative division hierarchy does not appear in a dialog turn subsequent to one that includes an address entity type that is deeper in the hierarchy (e.g the province is not provided after the city). I'm not fluent in Chinese, however, I believe that this may not be always the case, independently of the language. For instance, a broader entity type can be used for disambiguation purposes. Thus, this may be a flaw in the approach that impacts its performance in real conversations.\n\n__Answer__: Thanks for giving us the chance to clarify this concern. There are different kinds of conversations, even goal-driven dialogue. Yes, it may not always be the case that the fine-grained address entity mentioned follows the hierarchy during the dialog turn. For example, an airport booking conversation may only mention the city. As a result, as explained in the Introduction section, the scenarios we considered primarily involved an address-acquiring conversation, such as e-commence shipping, an emergency call for help, registering after-sales services, and so on, in which a full address is frequently formed through a conversation process. These scenarios are quite diverse, and it is extremely beneficial to assist in automatically collecting fine-grained address entities, such as in an emergency.  According to the statistics we have from real-world dialogues in our system, approximately 82.14% of the dialogues follow the top-down geographical hierarchy guideline.\n\nIn addition, our method is not limited to a particular address naming system but could also be applied to other addressing systems that follow certain geographic location rules, according to references [1–3].\n\n_[1]Manoruang D, Asavasuthirakul D. Quality analysis of online geocoding services for Thai text addresses[J]. Engineering and Applied Science Research, 2019, 46(2): 86-97._\n\n_[2]Wang X, Zhang Y, Chen M, et al. An evidence-based approach for toponym disambiguation[C]//2010 18th International Conference on Geoinformatics. IEEE, 2010: 1-7._\n\n_[3]Naaman M, Song Y J, Paepcke A, et al. Automatic organization for digital photographs with geographic coordinates[C]//Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries. 2004: 53-62._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Automatically extracting addresses from a dialog is an important step for automating processes such as online ordering and emergency dispatching. Thus, this paper addresses a relevant problem that is not widely regarded in the literature. The proposed approach is compared to several strong baselines and the discussion of the results is thorough, including analysis per entity type.", "reasons_to_reject": "As stated in Definition 6, the proposed approach assumes that addresses become more fine-grained along the dialog, that is, an address entity type that is higher in the administrative division hierarchy does not appear in a dialog turn subsequent to one that includes an address entity type that is deeper in the hierarchy (e.g the province is not provided after the city). I'm not fluent in Chinese, however, I believe that this may not be always the case, independently of the language. For instance, a broader entity type can be used for disambiguation purposes. Thus, this may be a flaw in the approach that impacts its performance in real conversations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of fine-grained address entity extraction from multi-turn dialogs in Chinese. The proposed approach improves the loss function of an ERNIE-based model by relying on probabilistic soft logic to consider information regarding the administrative division hierarchy and the way addresses are typically provided in a dialog. Additionally, several augmentation methods are used to obtain more data for training the model: the address entities in existing dialogs are replaced by other entities of the same type, noise is injected in the dialogs to simulate speaking habits in real-world conversations, and ChatGPT is used generate variations of the dialogs. The proposed approach achieves 93.88 F1 score on a dataset generated using this augmentation approach and 86.65 on a real-world dataset, surpassing several competitive baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The citation format when the reference is used as the subject is not correct. For instance, in line 127, instead of \"(Mai et al., 2018) improves\", it should read Mai et al. (2018) improves\n\n(039) \"further delivery\" makes no sense here\n\n(Figure 1) \"mui\" -> multi\n\n(131) \"with us\" -> to ours\n\n(396) \"could be referenced to\" -> are described in\n\n(411-419) The \"I give\" and \"remaining\" parts of the prompts sound weird\n\n(419) dataset -> data"}}, {"review_id": "dRGtP1ctuw", "reviewer": "Reviewer_ke6G", "summary": "", "questions": "A. Is separating fine-tuning PLM encoder with PSL regularization from training CRF a deliberate choice and your recommendation vs training PLM-CRF with PSL regularization together? I.e., do you recommend adding PSL regularization only to fine-tuning PLM instead of training the full model end-to-end?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time, which will substantially improve this paper.\n\n__Question 1__: Is separating fine-tuning PLM encoder with PSL regularization from training CRF a deliberate choice and your recommendation vs training PLM-CRF with PSL regularization together? I.e., do you recommend adding PSL regularization only to fine-tuning PLM instead of training the full model end-to-end?\n\n__Answer__: We appreciate your professional inquiry. As a matter of fact, we have conducted the experiment of training PLM-CRF with PSL regularization together, and the result is not as good as first fine-tuning the PLM encoder with PSL regularization before training the CRF layer. This result will be included in the revision. Our findings show that fine-tuning the PLM encoder with PSL regularization can capture address information, which might aid the subsequent CRF layer in geographical tag decoding.\n\n__Question 2__：It would be good to see some analyses justifying the hierarchy relationship among addresses in a dialog. The main relationship encoded with PSL assumes that the scope of location decreases as dialog progresses, but users can easily break such hierarchy, and the conventional order of providing addresses may change depending on the language (e.g., Chinese vs English). It’s not a huge concern, however, as the proposed method is general enough to encode other relationships among entities.\n\n__Answer__: Thank you for providing us the opportunity to clarify this concern. The scenarios we considered primarily involved an address-acquiring conversation, such as e-commence shipping, an emergency call for help, registering after-sales services, and so on, in which a full address is frequently formed through a conversation process. According to the statistics we have from real-world dialogues in our system, approximately ___82.14%___ of the dialogues follow the top-down geographical hierarchy guideline.\nIn addition, this method is not limited to a particular address naming system (such as Chinese) but could also be applied to other addressing systems that follow certain geographic location rules, according to references [1–3].\n\n_[1]Manoruang D, Asavasuthirakul D. Quality analysis of online geocoding services for Thai text addresses[J]. Engineering and Applied Science Research, 2019, 46(2): 86-97._\n\n_[2]Wang X, Zhang Y, Chen M, et al. An evidence-based approach for toponym disambiguation[C]//2010 18th International Conference on Geoinformatics. IEEE, 2010: 1-7._\n\n_[3]Naaman M, Song Y J, Paepcke A, et al. Automatic organization for digital photographs with geographic coordinates[C]//Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries. 2004: 53-62._\n\n__Quesion 3__: In the experiments, the proposed Log-FGAER adds the proposed logic-guided enhancement to Baseline-7. Since Baseline-7 is already mostly outperforming the rest of the baselines, comparing Log-FGAER with the rest of the baselines is not very informative. A more informative experiment would be to compare each baseline’s performance with and without the proposed enhancement (basically expand the first ablation study to other baselines in the main experiments). Such experiment can better support the claim that the proposed enhancement can be combined with any existing FGAER method.\n\n__Answer__: Thank you for your useful suggestions. We will incorporate the ablation study into the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed logic-guided enhancement for FGAER is well-motivated, novel, and general. Log-FGAER can be combined with any existing FGAER model without modifying model architecture, making it easy to adopt in practice. The proposed method is also not limited to encoding spatial dependence of addresses; other domain knowledge could be encoded with PSL in a similar way to improve NER performance, making the method an effective way to combine domain knowledge with data-driven approaches. \n- Even though data augmentation with LLMs, especially for paraphrasing, is not new, the proposed end-to-end data augmentation approach leveraging crawled addresses, entity segmentation method, and ChatGPT is a practically useful addition. ", "reasons_to_reject": "- It would be good to see some analyses justifying the hierarchy relationship among addresses in a dialog. The main relationship encoded with PSL assumes that the scope of location decreases as dialog progresses, but users can easily break such hierarchy, and the conventional order of providing addresses may change depending on the language (e.g., Chinese vs English). It’s not a huge concern, however, as the proposed method is general enough to encode other relationships among entities. \n- In the experiments, the proposed Log-FGAER adds the proposed logic-guided enhancement to Baseline-7. Since Baseline-7 is already mostly outperforming the rest of the baselines, comparing Log-FGAER with the rest of the baselines is not very informative. A more informative experiment would be to compare each baseline’s performance with and without the proposed enhancement (basically expand the first ablation study to other baselines in the main experiments). Such experiment can better support the claim that the proposed enhancement can be combined with any existing FGAER method. ", "questions_to_authors": "A. Is separating fine-tuning PLM encoder with PSL regularization from training CRF a deliberate choice and your recommendation vs training PLM-CRF with PSL regularization together? I.e., do you recommend adding PSL regularization only to fine-tuning PLM instead of training the full model end-to-end?", "paper_topic_and_main_contributions": "The paper tackles the problem of fine-grained address entity recognition (FGAER) from multi-turn dialogs, which is challenging due to the presence of noise in spoken dialogs and addresses often spanning multiple turns. The paper proposes using probabilistic soft logic (PSL) to modify the loss function to encode the spatial dependence of addresses in a dialog, which is shown to outperform competitive baselines in the experiments. The proposed enhancement can be combined with any existing FGAER models. To address the lack of labeled dialogs with address entities, the paper proposes an ontology-based data augmentation with ChatGPT to generate additional data. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 54 - Not clear why the referenced methods are not considered due to their labels, since I assume you can still use the same model architectures and train on your own labeled data.\n- Line 148-150 – Similarly, not clear why those methods are not applicable just because of different entity types. \n- Line 181-182 – a nitpick on the notations: typically upper case letters are random variables and lower case letters are realized instances of these random variables.\n- Line 396 – Referring the implementation details to survey papers is not very helpful; it’d be better to just have the implementation details in the appendix. \n- Line 418 – nitpick: “retaining” would be a better choice of word than “remaining”\n- Figure 5 – nitpick: not sure if it makes sense to draw lines in this figure, since there’s no particular order with the entities on x-axis; better to just keep the dots or draw barplots. "}}, {"review_id": "TBa0R1EZBb", "reviewer": "Reviewer_QkBv", "summary": "", "questions": "Question A: Based on the logic rules in Section 3, it seems that a big assumption in this work is that earlier turns in the dialogue have lower-level locations such as a city, while later turns have higher-level locations such as a district. Is this assumption always true? What other sequence-labeling tasks have a similarly hierarchical nature besides addresses?\nQuestion B: What is the computational cost of these experiments (i.e., training and inference)?\nQuestion C: Might you compare BERT-style models to LLM (fine-tuned or few-shot) baselines?\nQuestion D: Is the improvement (3.28% F1) from the probabilistic logic loss significant?\nQuestion E: What is your ethics statement?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the perceptive suggestions provided by reviewers, which will substantially enhance the quality of the paper.\n\n__Question A__: Based on the logic rules in Section 3, it seems that a big assumption in this work is that earlier turns in the dialogue have lower-level locations such as a city, while later turns have higher-level locations such as a district. Is this assumption always true? What other sequence-labeling tasks have a similarly hierarchical nature besides addresses? \n\n__Answer__：\nThank you for providing us the opportunity to clarify this concern. For the question “is this assumption always true”? The scenarios we considered primarily involved an address-acquiring conversation, such as e-commence shipping, an emergency call for help, registering after-sales services, and so on, in which a full address is frequently formed through a conversation process. According to the statistics we have from real-world dialogues in our system, approximately __82.14%__ of the dialogues follow the top-down geographical hierarchy guideline.\n\nOther tasks, in addition to address entity recognition, use the hierarchical structure rules available in the data to solve their problems. For example, [1] explores the subsumption hierarchy among the relational phrases and constructs a hypernymy graph of phrases using PSL rules. [2] illustrates the hierarchical structure of mobility-related entities, attributes, attribute values, and relations in clinical notes. \n\n_[1]Grycner A, Weikum G, Pujara J, et al. Relly: Inferring hypernym relationships between relational phrases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015: 971-981._\n\n_[2]Thieu T, Maldonado J C, Ho P S, et al. A comprehensive study of mobility functioning information in clinical notes: entity hierarchy, corpus annotation, and sequence labeling[J]. International journal of medical informatics, 2021, 147: 104351._\n\n__Question B__: What is the computational cost of these experiments (i.e., training and inference)? \n\n__Answer__:\nDuring the ERNIE fine-tuning stage:\nFor an input dialogue sequence of length $N$, the computational complexity is: $O(DLN^2+4SJ)=O(DLN^2+4N)$,\nwhere D represents the dimension size of the hidden layer and L is the number of layers. The turn of this dialogue is $S$, and the average length of each turn is $J$. We iterate through each turn in the dialogue, searching its two preceding turns and two subsequent turns seperately for address entities ($O(4SJ)$)\n\nDuring the CRF training stage:\nThe complexity of training CRF is $O(NK^2)$,\nwhere K represents the number of label types.\n\nDuring the inference stage:\nThe complexity is $O(DLN^2+NK^2)$\n\n__Question C__: Might you compare BERT-style models to LLM (fine-tuned or few-shot) baselines? \n\n__Answer__: BERT-style models, if we understand the question right, refer to BERT or Roberta. As illustrated in baselines 1-4, we fine-tuned RoBERTa and BERT using neural network layers (BiLSTM and CRF).\n\n__Question D__: Is the improvement (3.28% F1) from the probabilistic logic loss significant? \n\n__Answer__: We also evaluate the Log-FGAER performance over a small number of the the real-life call-center dialogue data. Our method’s accuracy increased by roughly ___5%___ when compared to an approach with the same structure but no logical rules. We conduct case studies to see how Log-FGAER works: When checking the labeling results, Log-FGAER corrects the entity type of \"clinical pharmacy department\" in a sentence to \"SUBPOI\" instead of the previously incorrect type of \"COMMUNITY,\" a mistake that is frequently made. This alteration is consistent with the logical rule we established, which states that the entity type of \"clinical pharmacy department\" should be located between 'POI' and 'HOUSENO.' This real-world example demonstrates the efficacy of our method.\n\n__Question E__: What is your ethics statement?\n\n__Answer__: We will include the following ethics statement at the end of the paper:\nThis work adheres to ethical guidelines and responsible practices in ACL policies. The authors have no external conflicts of interest to declare and have not been required to seek any ethics clearances in order to undertake this work. The datasets used in this work are from previously published works or competitions and, in our view, do not have any attached privacy or ethical issues. We provided data augmentation approaches to improve the evaluation of our proposal. The augmentation-generated conversations aim to provide an optional way to inspire research rather than violate ethical guidelines and should be used in a strictly controlled manner. Transparency and responsible reporting are priorities, and we welcome community engagement to enhance ethical considerations in our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions include a novel approach for combining a standard cross-entropy loss for a LM with a probabilistic logic model’s loss, as well as several techniques for augmenting small dialogue datasets (i.e., replacing slots in existing templates with different slot values, injecting noise, and generating new templates with ChatGPT). The paper is well-written, and the experiments comparing to other contextual embedding baselines are thorough and convincing. The code and dataset are open-source.", "reasons_to_reject": "This is a narrow use case for the technique, specific to detecting addresses with the 16 labels shown in Table 5, for e-commerce assistants delivering purchased items.", "questions_to_authors": "Question A: Based on the logic rules in Section 3, it seems that a big assumption in this work is that earlier turns in the dialogue have lower-level locations such as a city, while later turns have higher-level locations such as a district. Is this assumption always true? What other sequence-labeling tasks have a similarly hierarchical nature besides addresses?\nQuestion B: What is the computational cost of these experiments (i.e., training and inference)?\nQuestion C: Might you compare BERT-style models to LLM (fine-tuned or few-shot) baselines?\nQuestion D: Is the improvement (3.28% F1) from the probabilistic logic loss significant?\nQuestion E: What is your ethics statement?\n", "paper_topic_and_main_contributions": "This paper discusses a probabilistic soft logic approach for modeling the hierarchy of parts of an address (e.g., district -> city) during a multi-turn dialogue, which is combined with a pre-trained language model (i.e., ERNIE-CRF) for BIOE labeling of addresses. The contributions include a novel approach for combining a standard cross-entropy loss for a LM with a probabilistic logic model’s loss, as well as several techniques for augmenting small dialogue datasets (i.e., replacing slots in existing templates with different slot values, injecting noise, and generating new templates with ChatGPT).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "•\tSpell out POI the first time it’s used\n•\t“spread across to” -> “spread across”\n•\t“entity containing in” -> “entity contained in”\n•\t“logic rule that explain” -> “logic rule that explains”\n•\tSome spaces are missed between text and opening/closing parentheses (section 4)\n•\tSummarize the noise injection prior work in the appendix\n•\tAdd examples of where probabilistic logic fixes mistakes, with analysis, to the appendix\n"}}]

Author rebuttal:

Meta-review: The paper proposes a method for logic-guided fine-grained address recognition method, where address hierarchy has been utilized as the logic rule and apply this in a probabilistic manner to improve the accuracy. Automatic extraction of addresses from conversation is a nice step. The paper needs revision to address the following issues: ablation and qualitative analysis to demonstrate the justification of hierarchy relationships of addresses; ablation by including logic rules to each of the baselines.
Decision: Accept-Main