Knowledge-Selective Pretraining for Attribute Value Extraction
Authors: Hui Liu
Qingyu Yin
Zhengyang Wang
Chenwei Zhang
Haoming Jiang
Yifan Gao
Zheng Li
Xian Li
Chao Zhang
Bing Yin
William Yang Wang
Xiaodan Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: attribute value extraction
pretraining
TL;DR: We propose a pretraining method to improve the generalization ability of the QA-based AVE model.

Abstract: Attribute Value Extraction (AVE) aims to retrieve the values of attributes from the product profiles. The state-of-the-art methods tackle the AVE task through a question-answering (QA) paradigm, where the value is predicted from the context (i.e. product profile) given a query (i.e. attributes). Despite of the substantial advancements that have been made, the performance of existing methods on rare attributes is still far from satisfaction, and they cannot be easily extended to unseen attributes due to the poor generalization ability. In this work, we propose to leverage pretraining and transfer learning to address the aforementioned weaknesses. We first collect the product information from various E-commerce stores and retrieve a large number of (profile, attribute, value) triples, which will be used as the pretraining corpus. To more effectively utilize the retrieved corpus, we further design a Knowledge-Selective Framework (KSelF) based on query expansion that can be closely combined with the pretraining corpus to boost the performance. Meanwhile, considering the public AE-pub dataset contains considerable noise, we construct and contribute a larger benchmark EC-AVE collected from E-commerce websites. We conduct evaluation on both of these datasets.  The experimental results demonstrate that our proposed KSelF achieves new state-of-the-art performance without pretraining. When incorporated with the pretraining corpus, the performance of KSelF can be further improved, particularly on the attributes with limited training resources.

Knowledge-Selective Pretraining for Attribute Value Extraction
Hui Liu', Qingyu Yin?; Zhengyang Wang", Chenwei Zhang', Haoming Jiang', Yifan Gao?, Zheng Li2, Xian Li2, Chao Zhang?,
Bing Yin', William Yang Wang', Xiaodan Zhu'
Ingenuity Labs Research Institute & ECE, Queen's 's University, Canada 2 "Amazon.com Inc, Palo Alto, CA, USA
[hui 1iu, xiaodan. zhu)equeensu.ca ca com
{qingyy, zhengywa, cwzhang, jaa.minaaan.i. yi
Abstract
Profile
Zebco Slingshot Spincasting Reel and Fishing Rod Combo, 5-Foot 6-Inch 2-Piece Fishing Pole
Attribute Value Extraction (AVE) aims to re- trieve the values of attributes from the product profiles. The state-of-the fi--the-rtt methods tackle the AVE task through paradigm, where the value is predicted from the context (i.e. product profile) given a query (i.e. attributes). Despite of the substantial advance- ments that have been made, the performance of existing methods on rare attributes still far from satisfaction, and they cannot be easily extended to unseen attributes due to the poor generalization ability. In this work, we pro- pose leverage pretraining and transfer learn- ing to address the aforementioned weaknesses. We design a Knowledge- Showledge-Selective Selective Framework (KSelF) based on query expansion that can be closely combined with pretraining corpus to boost the performance. Meanwhile, consider- ing the public AB-pub dataset contains consid- erable noise, we construct larger benchmark EC-AVE collected from E-commerce websites. We conduct evaluation on both of these datasets. The experimental results demonstrate that our proposed KSeIF achieves state-of-the- performance without pretraining. When incor- porated with the pretraining corpus, the per- formance of KSelF can be further improved, particularly on the attributes with limited train- ing resources.
(Attribute, Value)
(Brand, Zebco)
[Length, 5-foot 6-inch) [Weight, None)
(QA)
(Color, None)
Figure 1: Example of (altribute, value) pairs of prod- uct. For the attributes that are not presented in the profile, the values are "None".
"Zebco" from the profile for the attribute Brand. Automating this process can considerably enrich the product information with fewer human efforts, which has become a hot research topic for many E-commerce stores (Xu et al., 2019).
Existing research on AVE mainly falls into two paradigms: (1) sequence-tagging, and (2) question answering (QA). The sequence-tagginn paradigm formulates AVE named entity recognition prob- lem, where the model is built to identify the at- tribute types of the tokens in product profiles (Put- thividhya and Hu, 2011; More, 2016; Zheng et al., 2018; Xu et al., 2019; Liang et al., 2020; Zhu et al., 2020; Jiang et al., 2021; Wang et al., 2022). How- ever, the sequence-tagginn methods suffer from the limitations that they do not scale to cope with a large number of attributes and cannot be extended to unseen attributes (Wang et al., 2020). Hence, recent works have proposed to formulate AVE as a QA task, where the product profile and attribute are taken as the context and query, respectively (Wang et al., 2020; Shinzato et al., 2022; Ding et al., 2022; Yang et al., 2022), and the model is trained to ex- tract the attribute value span from the context. The QA paradigm has shown to be more flexible and scalable than requence-tagging methods, achieving state-of-the-art performance on the AVE task.
1 Introduction
With the fast expansion of E-commerce systems, Attribute Value Extraction (AVE), which aims to retrieve the values of attributes from the product profiles, has attracted significantly more research attention since can be used to supplement prod- uct information and hence serve better product search (Xiao et al., 2021: Luo al., 2022), recom- mendations (Hwangbo et al., 2018), and other E- commerce tasks. As shown in Figure 1, the goal of AVE is to extract values of different attributes, e.g..
While the QA paradigm has yielded improved re- sults, they still face significant limitations that need to be addressed. Previous research demonstrates that the based methods do not perform well on
"corresponding author

---

the attributes with limited training resources (Wang al.. 2020; Shinzato et al., 2022), even though Shinzato et al. (2022) propose to build better repre- sentations by expanding the query with values of the same attribute from the training set and accord- ingly achieves promising overall performance. The sub-optimal performance on attributes with limited training resources hinders the practical application of QA-based AVE methods to E-commerce, be- cause new and unseen attributes continue to arise with the introduction of new products.
usefulness of knowledge and employ informative knowledge for query expansion. Therefore, on the basis of query expansion-based AVE, we pro- pose a Shooledge-Seleectie When performing query expansion, the proposed method first categorizes the retrieved values into the more-related and less-related, based on their similarity to the current product. Then we con- struct the expanded queries by properly exploiting the two types of attributes during training and in- ference. KSelF can be closely combined with the constructed pretraining corpus, allowing for more efficient utilization of training data, which further improves the generalization ability of the model. To benchmark the KSelF, we conduct experi- ments the AF-pub dataset (Xu al., 2019). However, we notice that AE-pub (Xu et al., 2019) contains significant noise. To encourage future research on this task, we further propose a new benchmark EC-AVE of high quality collected from well-structured E-commerce website. The re- sults demonstrate that our proposed KSelF achieves new state-of-the performance on both datasets without pretraining. When pretrained the large- scale corpus and then finetuned on the downstream datasets, KSelF can further attain better perfor- mance, particularly on the attributes with limited training resources. We summarize our contribu- tions as follows:
Framework (KSelF).
To overcome the aforementioned problems, we develop a task-specific pretraining method for the QA-based AVE, which provides the model strong generalization capabilities, allowing the model to quickly adapt to rare and unseen attributes. Inspired by the recent popularity of pretrained language models (Devlin et al., 2019; Liu et al., 2019; Chen al., 2020; He et al., 2021) that leverage large- scale unlabeled corpora from the Web perform pretraining, we collect product profiles as well as the corresponding product attributes from various public E-commerce websites. To automatically and efficiently construct data for pretraining. we con- duct string match in the collected product informa- tion, and obtain the large-scale corpus consisting of (profile, attribute, value) triples. Using these data, we further pretrain public language model for an AVE task employing the QA paradigm.
To further improve the performance, we adopt the query expansion technique. Given a product and an attribute to be filled with value, Shinzato et al. (2022) employs values the same attributes retrieved from training corpus as knowledge to ex- pand the query, improving upon previous meth- ods that only regard the attributes as queries. The assumption behind their method is that such ex- panded knowledge provides informative hints helping identify the correct value span of the target attribute. However, these retrieved values consti- tute vast set which may include those values that are not closely related to the current product, thus providing little hints and even introducing unex- pected noise. For example, the value of attribute "material" for "shirt" has nothing to do with that for "desk". This is particularly severe in the pre- training scenario because the pretraining data is typically in large-scale and there will be a huge number of helpless values from a variety of prod- uets for the same attribute. Idcally, a desirable solution should be able to effectively identify the
We propose a novel pretraining method for QA- QA-based AVE model that improves model performance on rare and unseen attributes.
We propose a knowledge-selective query ex- pansion framework that is capable of more efficiently exploiting the training data.
We conduct extensive experiments on the two AVE datasets, demonstrating that our method achieves ofate-of-the-att art performance.
2 Related Work
2.1 Attribute Value Extraction
The goal of Attribute Value Extractive (AVE) is to extract the values of attributes from the product profilc. Early works adopt rulc-based techniques including domain-specifi dictionary or vocabu- lary identify the important phrases and attributes (Gopalakrishnan et al., 2012; Vandic et al., 2012). With the wide application of deep learning, recent works mainly adopt two paradigms for the AVE

---

task: (1) sequence tagging and (2) question an- swering. For the sequence tagging paradigm, the main idea to is train model to predict the attribute types of tokens the product profile sequentially (Putthividhya and Hu, 2011; Shinzato and Sekine, 2013; More, 2016). To achieve this goal, differ- approaches have been proposed. Zheng et al. (2018) build an end-to- end BiLSTM-CRE model with separate tag sets for different attributes, while Xu et al. (2019) adopt BERT-CRE and propose to adopt global set of tags for all the attributes, which enables the model to fit larger attribute sets. Kumar and Saladi (2022) further use reinforcement learning to enhance the performance of sgquence-tagging models.
query, hence they cannot be directly applied the AVE task, In this work, we build the pretrain- ing corpus collected from the Web and pretrain the model with the constructed corpus, which, to the best of our knowledge, is the first work that applies the idea of pretraining to AVE.
ent sequence-tagging-
3 Background
3.1 Task Definition
Given a product profile a = {21, 22, ...... ,T) and an attribute a = {a1, 02, ... am) where n and m are the number of tokens in the profile and the attribute respectively, the goal of AVE is to predict the value of the current attribute, which is single text span in with beginning position P. and ending position as Pe S P. S P. S n). If x does not contain a value span for the current at- tribute, which we call as a negative instance in this paper, a special value None should be predicted.
However, the sequence-tagging paradigm can only be used for a fixed attribute set, and thus suf- fers from the lack of scalability and generalizability. To extend the model to large-scale attribute sets, Wang et al. (2020) utilize the question-answent paradigm where the product profile and the at- context and question. The model AVEQA is proposed which concatenates the profile and the attribute into sin- gle query string, and then predicts the value span from the profile. To encourage the models to fully utilize the advantage of the QA paradigm, Shinzato et al. (2022) expand the query string with values of the same attribute from the training set, since these values can serve the knowledge to learn better query representations. Despite of the promising performance achieved by the QA-based models, they are shown to perform poorly on the attributes that are not seen in the training set, demonstrating sub-optimal generalization on the new attributes. We hence build our method upon the QA paradigm and propose knowledge-sellctive query expan- sion framework to encourage the model to more efficiently exploit the training data.
3.2 Query Expansion for QA-Based AVE
tribute are respectively regarded
Consider the product profile æ and the attribute a. A vanilla query is constructed as:
q CCS;; æ; SEP; a)
(1)
q will be fed into a pretrained BERT (Devlin et al., 2019), and the contextualized vector representation of [CLS] and æ from the last layer will be attained respectively as hcus E Rd and H, E R where is the hidden size. Then the two vector represen- tations will be concatenated as H = [ACLS: Hz). and further be used to predict the starting position and the ending position as:
Pg = arg max(SoftmaxxW5HHH)
Pa arg max i2
(B)))
where HI is the contextual embedding of the 7th token in H. W,, and W. are parameters that map the embedding to output logits. For the negative instances where there no value presented in the product profile, the ground truth of P3 and Pa is set to be 0, which is the position of the [CLS] token.
2.2 Pretrained Language Models
Pretrained language models have recently em- braced great success in a variety of NLP tasks (De- vlin et al., 2019; Radford et al., 2019; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019; Chen et al., 2020; He et al., 2021). They pretrained with large-scale corpus and then adapted different downstream tasks or datasets from various domains, showing good generalization ability. However, the existing pretrained models lack the ability to pre- dict a value span from the context conditioned on
To provide better representation of the attribute, Shinzato al. (2022) propose several techniques to expand Eq. 1. Admittedly, values of attribute can be used for illustrating the attribute. Shinzato et al. (2022) propose to utilize the values of a that appear in the training data as the run-time knowledge as va = [Ma,1; SEP; Pa.2; SEP:...

---

Profile
HIGH-QUALITY MATERIAL: This dino pullover is made of high-quality
polyester fiber, which will make you feel soft, comfortable and breathable. The tops very soft and comfortable you to wear, and be matched with any clothing, such as jeans, leggings, casual pants and so on. NICE GIFT: Fashionable and casual style, it good choice whether you wear it by yourself or give as a gift your family and friends.
VERSATILE:
String
matchy
Specifications
Extracted Tuples
Material: Polyester
Size: 5/M/L
(... is made of high high-quality polyester fiber, which ...
material,
polyester)
Style: Casual Style
(... fashionable and casual style, it is good ...
style
casual style)
Figure 2: Process of harvesting (profile, attribute, value) triples from product profile and their specifications.
where Va,i is pears in the training data. Hence, Eq. can be expanded with the run-time knowledge as qe [CLS; æ; SEP; a; va.; Another technique they adopt is domain token mixing (Britz et al., 2017) by using two special tokens SEEN and UNSEEN to denote if run-time knowledge is pro- vided in the query. Hence, for an example with profile and attribute a, two queries can be built for model training:
value of attribute that ap-
#Triple 4M
Length 47.75
#Attributes 5270
#Pos. 2.75M
#Neg. 1.25M
Table 1: Statistics of pretraining corpus. Pos. and Neg. denote positive instances and negative instances respectively. Length denotes the average length of the profiles the triples.
Triple Extraction This step aims retrieve (pro- file, attribute, value) triples from the product de- scriptions and specifications collected in the last step. We denote D = {d/í E [1,2. ......... _DII} as the descriptions of a product that contain [D] sen- tences, and S = {az vylj E as the specifications that contain |S] attribute- value pairs. We conduct string match between the descriptions and the specifications. If a value 0j appears in a description sentence d, a triple (di, aj, U2) will be retrieved where d,, aj. Vjy are the profile, attribute, and value, respectively, and will be used for pretraining our AVE model.
9. = [CLS; x; SEP; SEEN; a: SEP; va)
qu - [CLS; e; SEP: UNSEEN; a)
During the inference stage, SEEN and UNSEEN tokens are adopted for seen attributes and unseen attributes, respectively.
... [1.2.... [S]]]
4 Method
4.1
Pretraining Data Construction
Here, we introduce how to build the pretraining corpus, of which the overall construction process is shown Figure 2. In general, we collect the pre- training corpus from various E-commerce websites. The construction process contains two-stages: (1) product profile collection and (2) triples extraction.
The triple set collected through the foremen- tioned two steps contains a total of 5,270 attributes, while the attributes exhibit an apparent long-tail distribution which is common in many datasets (Zhang et al., 2021). In order mitigate the nega- tive effects brought by the extreme long-tail distri- bution, we keep the maximum instance number of one attribute to 10K and perform down- bomn-sampling accordingly. Meanwhile, the triples collected with the two steps are all positive instances, while an AVE dataset may also include negative instances as discussed in Section 3.1. Hence, we randomly combine some product descriptions and attributes from the product profiles, and use the special value None to form a negative instance. In this way, we finally obtain a pretraining corpus of 4M triples with 5270 attributes. Table presents a detailed statistics of the collected pretrain corpus.
Product Profile Collection When presenting product, an E-commerce website often includes both the product descriptions and the correspond- ing product specifications. The product descrip- tions are some sentences that describe the details or features of a product, while the specifications con- sist of some product attributes and the correspond- ing values, as shown in the examples in Figure 4 of Appendix D. We visit multiple E-commerce web- sites and for each product, crawl its descriptions and specifications, if any. As such, obtain 22M products with their corresponding descriptions and specifications from various product categories in- cluding clothing, books, among others.

---

Product Information
Current product-
Expanded Queries for Training
Profile:
Men's Running
[CLS] Nike Men's Sunning shoes [S8P] [SEEN) Brand (SEP) Adidas (SEP) Nike
(Nike)
[115] Nik Men's Running shoss [S81] [SSEN] Brannd (SEP) Samnung (SEP) Zeioco
Qarery Training dola
buld Query
[CLS Nike Men's Running shoes [SEP] (SEEN) Brand (SEP) Adidde (SEP) Samsung (SEP) Mike (SEP) Zebro
(C55) Nale Men's Running show [SE [UNSSFiN) Brand
Expanded Queries
Training Data
Values "Brand"
Dxg Al, P.) (Ag. P2) (H..
Retriewe Varoas
[shnes]]
[(Nletronid)
[Eshing
[C1S] Kile Men's lunning shors [SEV] [SEEN) Brand [SPP) Adides (SEP) (SKP) Samnung (SFP)) Zebra
acoribute:
Adids Nike
Samsung
Zebco
[P] for UNIIINN arteibtte:
(CLS) Nike Men's Running shoes (SEP) (UNSEE) Brand
More-retated
Lesp-related
Anowledge
knowledge
Figure 3: Overall illustration of our proposed knowledge-selective query expansion framework. Given the current product information, the goal is to predict the value of "Brand". As a first step, the values of "Brand" are retrieved from the training data run-time knowledge, which is further separated into subsets, more-related knowledge and less-related knowledge. Then the two subsets, together with the product information, will form different queries which are used during the training stage and the inference stage.
4.2 Knowledge-Selective Framework
which used to categorize the knowledge (i.e. at- tribute values) into more-related and less-related ones. KSelF then constructs the expanded queries by properly exploiting these two types of attributes during training and inference. Figure 3 shows the overview of the proposed framework.
Despite of the performance achieved by Shinzato et al. (2022) briefed in Section 3.2, there are short- comings in their proposed method.
First, the proposed run-time knowledge con- tains all the seen values of an attribute, while some of the values may be less related to the current profile æ, and hence provide limited information in explaining the current attribute a. As shown in the example in Figure 3, given the current product profile "Nike Men's Run- retrieve various values
Knowledge Categorization Given a product pro- file æ and an attribute a, we first retrieve all the triples from training data which the attribute is a as T = {(x,, a, v. l2 E [I, |I[]] For a triple (æ; a, v;) E T, we calculate the simi- larity between two profiles æ and I,, where the calculation method is differently designed for the pretraining stage and fine-tuning stuge.
ning Shoes", of attribute "Brand" from the training data, including values of the Shoes category like "Adidas" and "Nike", and values from other categories including Electronic and Fishing. However, only the values from the Shoes cate- gory are highly related to the current profile and can help to explain the current attribute, while the values of Electronic and Fishing may contribute little even become noise in the run-time knowledge.
For the fine-tuning stage, we conduct entity retrieval (De Cao al., 2021) on both æ and æ,, The similarity defined as the number of common entitics the two profiles.
For the pretraining stage, the similarity is de- fined as the bag-of-word similarity of the two profiles for the consideration of both precision and computation efficiency.
Second, pretrained models limit the length of the input query. The values in the run-time knowledge are ranked in descending order of frequency, which may result in the truncation of the more-related values the query long.
More details of similarity measurements can be found in Appendix A. After obtaining the simi- larities between æ and each profile T, we can separate T into two sub-sets Tn, and Tj according to a predefined threshold on the profile similarity socres. The values in Im are from the triples where the profiles are more similar to æ, which we call more-related knowledge, while the values in Fj are less-related knowledge.
The above problems will become even more severe when the method is combined with the pretrain- ing corpus since one attribute may contain large number of values from different product categories.
To overcome the aforementioned weaknesses, we propose a Knowledge-Selent Selective Framework (KSelF) based on query expansion. KSeIF first measures the similarity score between profiles,
Query Expansion with Selected Knowledge We expand queries with the run-time knowledge

---

according to the knowledge categorization. Specif- ically, we have two types of expansions:
AE-pub
BC-AVE
split
Train
Dev
Test
Train
Dev
Test
Total #
76823
10975
21950
105625
15817
31876
g." = [CLS; æ; SEP; SEEN; a; SEP; " 9. = CCLS; æ; SEP; SEEN; a; SEP; od
Pos. #
59954
8518
17154
72719
10772
21897
Neg. #
16869
2457
4796
32906
5045
9979
Attr. #
1800
635
872
347
394
422
Unseen. #
154
246
58
75
where e0 and vy are the values from Im and Ti, respectively. Following Shinzato et al. (2022), the values in eaa or va are sorted in descending order according to their frequency.
Table Statistics of the datasets adopted in this work. "Pos.", "Neg.", "Attr.", and "Unseen." denote pos- itive instance, negative instance, attribute, and unseen attribute, respectively.
Our proposed KSeIF can efficiently reduce the size of the run-time knowledge exposed during the instance training, and can be applied with or without the pretraining corpus. During the training stage, KSelF inherits the advantage of Shinzato et al. (2022) by constructing all the four queries Is. Iu 1."h and d' which can exploit the run-time knowledge more efficiently to explain the meaning of the attribute, while reduce the dependence on the values in the run-time knowledge. During the inference stage, for knowledge is a combination of the more-related knowledge and the less-related knowledge, and the query built as:
E-commerce website. During the collection pro- cess, we guarantee that the EC-AVE dataset has no overlaps with the pretraining corpus to avoid data leakage. Table shows statistics of two datasets.
5.1.2 Implementation Details
Following Shinzato et al. (2022), we build our model upon pretrained BERT in the uncased base version for fair comparisons. We first pretrain the model with the pretraining corpus for 5 epochs, with the maximum profile length as 64 and the maximum query length 192. During the finetun- ing stage, continuely train from the pretrained model for 20 epochs on downstream datasets with the maximum profile length as 32 and maximum query length as 192. We follow the settings in Shin- zato et al. (2022) for the rest hyper-parameters.
attribute, the run-time
ls,test [CLS; a; SEP; a; SEP; o o
while for an unseen attribute, the query is con- structed as:
5.1.3 Evaluation Metrics
Quilest = [CLS; 2; SEP; UNSEEN; a).
We adopt precision, recall and F1 score as met- rics. We follow Exact Match criteria (Rajpurkar et al., 2016) to compute the scores, where the full sequence of extracted value needs to be correct.
4.3 Pretraining with KSelF
In the pretraining stage, we first initialize KSelF with the a public pretrained model, e.g., BERT (De- vlin et al., 2019), and then continue pretrain for the QA-based AVE task using the generated pretrain- ing corpus introduced in Section 4.1. During the finetuning stage, the model further tuned with any available downtream datasets.
5.1.4 Baseline Methods
GPT-3.5 (Brown al., 2020): We design a prompt to have GPT-3.5 to predict the values a generative scheme. More details can found in Appendix B. Dictionary: We follow Shinzato et al. (2022) to perform simple dictionary matching by returning the most frequent title for a given attribute.
5 Experiments
value included in the given
5.1
Experimental Setup
5.1.1 Datasets
SUOpen (Xu et al., 2019): This is the method that achieves best performance under the sequence- tagging paradigm.
A widely adopted benchmark for the task of AVE is AE-pub (Xu et al., 2019). However, this dataset contains many noisy annotations and some dull at- tributes (e.g. "feature 1", "type"). These problems bring extra challenges for fairly evaluating an AVE system. To mitigate this problem, we propose a new dataset EC-AVE collected from a stru-sttre
BERT-QA (Shinzato al., 2022): This is the method that achieves the state-of-the-ae perfor- mance on the AVE task!
'The code of BERT-QA is not publicly available, hence we report the performance of our reproduction.

---

AB-pub
EC-AVE
Macro R
Micro R
Macro R
Micro
P
F1
F1
P
F1
P
F1
GPT-3.5
18.89
13.14
14.37
78.85
27.68
40.97
50.28
26.47
31.51
81.4
36,3
50.21
Dictionary
30.24
29.50
28.16
82.19
70.86
76.10
48.09
62.30
50.20
73.02
58,50
64.96
SUOpenTag
39.91
35.53
36.76
95.54
83.04
88.85
76.08
63.82
67.91
96.60
82.38
88.93
BERT-QA
46.24
42.47
43.23
93.66
85.54
89.42
76.41
69.14
71.05
94.89
83,99
89.11
BERT-QA + pretrain
47.02
42.81
43.69
93.69
85.85
89.60
78.84
70.33
72.99
95.67
85.82
90.48
KSeIF
47.61
44.44
45.02
93.64
86.13
89.73
76.91
69.15
71.42
95.75
84,91
90.00 90.95
KSeIF + pretrain
48.30
45.49
45.93*
94.06
86.28
90.00"
80.90
71.20
74.30"
96.32
86.15
Table 3: Performance of different methods on AB-pub dataset and EC-AVE dataset. significant with p < 0.05 compared with KSelF.
means the gains are statistically
BERT-QAApreetain:: We first pretrain the BERT- QA model with our constructed pretraining corpus, and finetune on downstream datascts. The pre- training scheme and finetuning scheme follow the settings in Shinzato et al. (2022).
QA on the corpus. According to the performance reported in the Table 3. the pretraining boosts the performance of BERT-QA on both datasets. However, the improvement of BERT-QA is more marginal than that of KSelF, which further sup- ports our motivation that the proposed knowledge- selective query expansion framework is capable of more efficiently exploiting the training data.
5.2
Experimental Results
5.2.1
Overall Performance
The results presented in Table 3, and we re- port both macro and micro performance. There are mainly four observations we can conclude from the table. (1) Surprisingly, GPT-3.5 can achieve Macro/Micro F1 of 14.37/40.97 and 31.51/50.21 on AE-pub and EC-AVE respectively. is capable of predicting the values in an unsupervised gen- erative way to some extent, which demonstrates the potentials of large pretrained language models in solving the AVE task. (2) The QA-based meth- ods significantly outperform the sequence-taggie based SUOpen'agg showing the advantages of QA paradigm. (3) When the pretraining corpus is not exploited, our proposed KSell' outperforms BERT- QA and achieves new state-of-the performance on both two datasets. Such results demonstrate that the knowledge selection method can encourage the model to utilize the training data more efficiently, and can enhance the performance of the QA-based models. (4) The pretraining method can further boost the performance of KSelF, As shown in Ta- ble 3, when KSeIF firstly pretrained and then finetuned on downstream datasets, both Macro Fl and Micro Fl are improved while the improvement on Macro Fl is much more significant because of the long-tail distribution of the attributes. Such re- sults consolidate the conclusion that the proposed pretraining method brings benefits to the AVE.
5.2.2 Results Rare Attributes
We summarize the performance of BERT-QA and KSelF on attributes with different frequencies in Table 4. As reported in the table, both AE-pub and EC-AVE, KSelF outperforms BERT-QA on fre- quent attributes (frequency 2 10), while the perfor- mance is slightly worse than BERT-QA on unseen attributes. The reason that KSelF can more ef- ficiently utilize the values in the training data as run-time knowledge, while unseen attributes do not appear in the training data and thus the proposed KSelF has little effect them since run-time knowledge can be retrieved for unseen attributes.
The goal of pretraining is to enhance the model's generalizability and improve its performance on rare attributes. The results in Table 4 demonstrate that the performance of the two QA-based meth- ods is significantly enhanced on unseen attributes. For the AE-pub dataset, Macro Fl and Micro Fl are lower for both methods on attributes with a frequency between and 10 when pretraining is incorporated. The main reason is that AE-pub con- tains many grammatically wrong attributes and dull attributes, shown in the examples in Appendix C. The pretraining corpus is not able improve the generalizability
these attributes since the cor- pus are collected from E-commerce websites and the instances are well organized both literally and semantically. Meanwhile, we can see from the ta- ble that the improvement introduced by pretraining
To more comprehensively evaluate the effective- ness of our constructed pretraining corpus and our proposed KSelF, we also explore to pretrain BERT-

---

AE-pub
EC-AVE
Frequency Metrics
High
Low
Unseen
High
Low
Unseen
Ma-F1
Mi-FI
Ma-F1
Mi-F1
Ma-FI
Mi-F1
Ma-F1
Mi-F1
Ma-FI
Mi-Fi
Ma-F1
Mi-FI
BERT-QA
64.16
90,44
40.22
64.62
23.60
38.64
86,45
90.14
15.00
56.00
28.39
53.62
BERT-QA + pretrain
62.99
90.61
39.15
63.07
28.51
44.95
87.33
91.41
18.33
61.54
34.24
57.60
KSelF
64.89
90.73
44.48
67.28
22.57
38.10
87.25
91.05
13.33
52.17
27.75
53.25
KSelF pretrain
65.15
90.99
42.52
65.95
28.95
45.81
87.89
91.82
21.67
64.29
37.90
60.47
Table 4: Performance attributes with different frequencies. Ma-F1 and Mi-FI denote Macro F1 and Micro-F1 respectively. High frequency means the number of occurrence of the attribute in the training data is greater than or equal 10, while low frequency means the number is less than 10 but greater than 0. Unseen means the attribute never appear in the training data.
AE-pub
EC-AVE
Profile: Original official Nike Air Jordan 11 retro win like 96 men's basketball shoes sneakers sports AJ11 classic outdoor 378037-623
Ma-F1
Mi-Fi
Ma-F1
Mi-F1
w/o KS
35.92
66.17
48.62
64.33
Attribute: model number
Value: 378037-623
w/ KS
38.36
69.01
49.47
66.44
KSelF: 378037
KPelF+Pre. 378037-623
Table 5: Zero-shot results the two datasets. KS and Ma-FI/Mi-F1 denote knowledge selection and Macro/Micro Fl respectively.
Profile: pes crankbait fishing wobbler 14g 10cm artificial
crank bait bass trout fishing lure pike trolling pesca minnow fishing tackle
Attribute: lure length
Value; 10em
on unseen attributes of KSelF' is more substantial than that of BERT-QA, which further proves that when large-scale dataset is available, KSelF can uti- lize the pretraining data more efficiently and hence strengthen the model performance.
KSeIF: 14g
KSelF+Pre. 10cm
Profile: running street creative funny belly pockets outdoor sports zipper mobile phone pockets simulation butt anti harassment waist bag Attribute: opening method
Value: zipper
KSeIF: belly pockets
KSelF+Pre. zipper
5.2.3 Zero-shot Results
Table 6: Example predictions of KSelF and KSelF with pretraining (Pre.) from AE-pub.
We further evaluate the performance of our pre- trained model under extreme zero-shot setting. Specifically, we initialize the model with the pre- trained checkpoints either with or without knowl- edge selection, and evaluate the model on both AE-pub and EC-AVE without any further finetun- ing. The results are reported in Table As we can see, even if the model is not finetuned on the corre- sponding downstream datasets, it still exhibits the ability of generalization to some extent. Moreover, when pretraining step is performed without our pro- posed knowledge selection scheme, the zero-shot performance is much worse than pretraining with the proposed scheme. Such results also demon- strate the advantages of our proposed knowledge- selective query expansion framework.
can improve its accuracy from 53.1% % to 53.7% on the attribute "model number", of which the val- ues are mostly
tokens. Another advantage of pretraining is the enhancement of performance on unseen attributes. Here, "lure length" and "opening method" in the second and third examples are all unseen attributes as they do not appear in the train- ing data, As we shown in the table, KSelF fails to predict their values while the pretraining method enables the model to correctly extractvalues from the product profiles.
Conclusions
Our study aims to improve the performance of At- tribute Value Extraction for rare attributes, which is of significant importance for real-life applica- tions. We propose a pretraining method to im- prove the generalization ability of the QA-based AVE model. In addition, we propose knowledge- selective query expansion framework that can effec- tively exploit the training data, Our experimental results on both the AE-pub and EC-AVE datasets show that the proposed pretraining brased KSelF
5.3
Case Study
The primary benefit of pretraining is the improved generalizability on attributes with limited training resources. To have a better understanding about the proposed method, we present some examples in Table 6. The first example demonstrates that the pretraining can enhance the model's ability on pre- dicting the rare values, e.g. "378037-623". Further investigations show that with pretraining, KSelF

---

achieves new SOTA performance.
of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 8635- 8648, Online. Association for Computational Lin- guistics.
Limitations
Though our proposed KSeIF has achieved the state- of-the-art performance and the performance is fur- ther improved with the pretraining corpus, the method can only tackle attributes/values appear- ing in the textual part of the product profile. It is often the case that some attributes and values ap- pear in the non-text part, e.g. product image, while our proposed method does not cover such pairs. Although multimodal AVE also plays an important role in E-commerce, we do not discuss it this work and leave it for future work.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-XI: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2983 Florence, Italy, Asso- ciation for Computational Linguistics.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Ethics Statement
Jacob Devlin, Ming-War Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Our proposed methods do not introduce any so- cial/cthical bias to the model, while there are po- tentially concerns in the AE-Pub dataset. We notice that some products are unisex according to the pro- files while the annotation for the attribute "gender" is either "man" or "woman". Such problems in an- notations may introduce gender bias in the training data, which will further influence the behaviors of the model. To avoid such problems, cleaner and fetter-formatted datasets are desired, and hence we build the EC-AVE dataset, which can provide fairer and more comprehensive evaluations to the AVE task.
Yifan Ding, Yan Liang, Nasser Zalmout, Xian Li, Chris- tan Grant, and Tim Weninger. 2022. Ask-and-verify: Span candidate generation and verification for at- tribute value extraction. In EMNLP 2022.
Vishrawas Gopalakrishnan, Suresh Parthasarathy Iyen- gar, Amit Madaan, Rajeev Rastogi, and Srinivasan Sengamedu. 2012. Matching product titles using web-based enrichment. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM '12, page 605-614, New York, NY, USA. Association for Computing Machinery.
References
Denny Britz, Quoe Le, and Reid Pryzant. 2017. Effec- tive domain mixing for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 118-126, Copenhagen, Denmark. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates, Inc.
Hyunwoo Hwangbo, Yang Sok Kim, and Kyung Jin Cha. 2018. Recommendation system development for fashion retail e-commerce. Electron. Commen. Rec. Appl., 28(C):44-10)
Haoming Jiang, Danqing Zhang, Tianyu Cao, Bing Yin, and Tuo Zhao. 2021. Named entity recognition with small strongly labeled and large weakly labeled data. arXiv prepriat arXiv:2106.08::
Kushal Kumar and Anoop Saladi. 2022. Pave: Lazy- mdp hased ensemble to improve recall of product attribute extraction models. In Proceedings of the 31 ACM International Conference on Information & Knowledge Management, pages 3233-3242.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-groundde pre- training for data-to-text generation. In Proceedings

---

Chen Liang, Yue Yu, Haoming Jiang. Siawpeng Er, Rui- jia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 1054-1064.
Qifan Wang. Li Yang, Bhargav Kanagal, Sumit Sanghai, D Sivakumar, Bin Shu, Zac Yu, and Jon Elsas. 2020. Learning to extract attribute value from product via question answering: multi-task approach. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pages 47-55.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint /90. 1692..
Qifan Wang, Li Yang, Jingang Wang. Jitin Krishnan, Bo Dai, Sinong Wang, Zenglin Xu, Madian Khabsa, and Hao Ma. 2022. SMARTAVE: Structured mul- timodal transformer for product attribute value traction. Findings of the Association for Compu- tational Linguistics: EMNLP 2022, pages 263-276, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chen Luo, William Headden, Neela Avudaiappan. Haoming Jiang, Tianyu Cao, Qingyu Yin, Yifan Gao, Zheng Li, Rahul Goutam, Haiyang Zhang, et al. 2022. Query attribute recommendation at amazon search. In Proceedings of the 16th ACM Conference on Rec- ommender Systems, pages 506-508.
Liqiang Xiao, Jun Ma. Xin Luna Dong. Pascual Martinez-Gómez, Nasser Zalmout, Wei Chen, Tong Zhao, Hao He, and Yaohui Jin. 2021. End-to-end conversational search for online shopping with utter- ance transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 3477-3488 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Ajinkya More. 2016. Attribute extraction from product titles in ecommerce. arXiv preprint 160800600804670.
Duangmance Putthividhya and Junling Hu, 2011. Boot- strapped named entity recognition for product at- tribute extraction. In Proceedings of the 2011 Con- ference on Empirical Methods in Natural Language Processing, pages 1557-1567, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Huimin Xu, Wenting Wang. Xin Mao, Xinyu Jiang, and Man Lan. 2019. Scaling up open tagging from tens to thousands: Comprehension empowered attribute value extraction from product title. In Proceedings of the 57th Annual Meeting of the Association for Com- putational I Linguistics, pages 521 4-5223,, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18))9
Li Yang. Qifan Wang, Zac Yu, Anand Kulkarni, Sumit Sanghai, Bin Shu, Jon Elsas, and Bhargav Kanagal. 2022. Mave: A product dataset for multi-source attribute value extraction. In Proceedings of the Fif- teenth ACM International Conference on Web Search and Duta Mining, WSDM '22, page 1256-1265, New York, NY. USA. Association for Computing Machin- ery.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392 Austin, Texas. Association for Computational Linguistics.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xinet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in Neural Infor- mation Processing Systems, volume 32, Curran Asso- ciates, Inc.
Keiji Shinzato and Satoshi Sekine. 2013, Unsupervised extraction of attributes and their values from product description. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1339-1347, Nagoya, Japan. Asian Federation of Natural Language Processing.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. 2021. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition (CVPR), pages 2361-2370.
Keiji Shinzato, Naoki Yoshinaga, Yandi Xia, and Wei- Te Chen. 2022. Simple and effective knowledge- driven query expansion for QA-based product at- tribute extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 227-234, Dublin, Ireland. Association for Computational Lin- guistics.
Guineng Zheng, Subhabrata Mukherjee, Xin Luna Dong, and Feifei Li. 2018. Opentag: Open attribute value extraction from product profiles. KDD 188 page 1049-1058, New York, NY, USA. Association for Computing Machinery.
Damir Vandic, Jan-Willem van Dam, and Flavius Fras- incar. 2012. Faceted product search powered by the semantic web. Decision Support Systems, 53(3):425_. 437.
Tiangang Zhu, Yue Wang, Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. Multimodal

---

joint attribute prediction and value extraction for E- commerce product. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2129-2139, Online. As- sociation for Computational Linguistics.

---

A Details of Similarity Calculation
Example 1
Profile: adidas men shoes originals forum lo refined low-top men's skateboarding shoes cotton fabric adidas sports sneakers for men Attribute: brand name Value: adidas
We adopt different measurements of similarity for pretraining stage and fine-tuning stage for two rea- sons:
1. Entity retrieval for the E-Commerce domain is very time-consuming and model computation cost is very high. Considering the size of the pretraining corpus, only apply entity retrieval during the fine-tuning stage.
Example 2:
Profile: handing overlength 8m-12m 13m 14m 15m high carbon super hard fishing rod telescopic rod sea fishing rod taiwan fishing rod Attribute: maerial (material)
2. We conduct comprehensive analyses over the bag-of-word (BoW) similarity. It turns out that achieve acceptable performance on the similarity calculation between different product profiles by applying BoW similarity. Considering the large size of the pretraining corpus, BoW similarity is a fairly good al- ternative to reduce the time and computation costs during the pretraining stage.
Value: carbon
Example 3:
Profile: outdoor mens hiking ski jacket camouflage thick warm assault jacket wind- proof camping jacket men plus size Attribute: patern (pattern)
Value: camouflage
Specifically, for the entity similarity, we regard two profiles I; and œ and as similar profile if (1) æ; and æj contain at least one common entity for the entity measurement, or (2) more than 1/3 of the to- kens in æ; appear in ; for the BoW measurement.
Example 4:
Profile: catch u 5-22g m/mh spinning casting rod carbon pole 1.8m 2 tips travel sea
spinning fishing rod Attribute: feature Value: casting rod
GPT-3.5 Implementation Details
We adopt GPT-3.5, specifically the model "gpt-3.5- turbo", to generate the values based on the given product profile and attribute We design different prompts and choose the that achieves the best performance on the Dev set. The prompt we adopt is:
In contrast, our proposed EC-AVE dataset is much better formatted and of larger size, and we present two examples below:
Example 1
Profile: huffy frozen 2 olaf preschool scooter
handlebar bin three wheels & wide deck
bullet point brake style foot bullet point suspension type rigid Attribute: brake style Value: foot
Q: Given the product profile does the profile contain the value of the attribute a? If so, return the answer in the format of (attribute, value); if not, return "No".
Example 2:
We have GPT-3.5 to generate the content in the blank, and the generated content will be parsed and the retrieved results are regarded as the correspond- ing predicted value.
Profile: uxpro psd010bf heating only with fan digital thermostat bullet point digital accuracy bullet point battery powered only bullet point
large led illuminated digital display
Attribute: model number
Examples of Datasets
Value: psd010bf
In this section, will present some examples of in this work. The current public benchmarkAL--UU contains many instances of which the attributes are either grammatically wrong, like Example and 3, or dull, like Example 4:
D Examples of E-commerce
the datasets
In Figure 4, we present three website examples that contain product descriptions and the corresponding specifications.

---

Women Dinosaur Print Sweater Aesthetic
Polyaite
Appliable doader: Uhine
Harajuku Oversized Pullover E Girl Streetwear
Applicable Aoe Ranye: Adul
Preppy Sweater 20hmiing
Style: Cesoal Style
Festure: Soharamaaraabi hesps/aai///mm//////n/aooo Haskinneb
522 99 $35499
Switeble Gocasions: occasions
Fil Orcerunatiioo.
Size: S/N/L
Celter
Color, Whttee Whtts/Baaci/haeee Beaaaa///aaee//eeee//eee
AA
Salact
Fox
Product Specifications
Catton
1821- QUALIT MATERBALL This pullore
polyaser fiee.
veicch will mulo you has lunk, canta rabll anartec
DESSON
antiraalllll impiccooll nocciangg seeee
VENSSITLE The 1as
raich zd cotking g auth eo loppligg canall our and DCCASIiNn
accauness ordiargg 8. gift yor home's rilens.
tepo
MIDE CIFT nachomiitt and stytc
theike wour
Product Descriptions
(a) An example from Amazon
About diss
catsi
seover proket Ner wowter wedereeedeeee citer
Cluck
Product Descriptions
Conder
Mara
Product Specifications
(b) An example from Walmart
1
Product Descriptions
About item
alrec
Product Specifications
(c) An example from Target
Figure 4: Examples from various E-commerce websites that contain product descriptions and the corresponding specifications.


Reviews:
[{"review_id": "aVhmcluEWq", "reviewer": "Reviewer_YCV3", "summary": "", "questions": "* This work basically applies distant supervision to construct pretraining dataset. Is the precision of distant supervision100%? Does tge wrong product problem occur, i.e. the profile extracted is not related to the product of interest?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments.\n\n## Comment 1\n\n> Comment 1.1: contributions w.r.t, previous work (Shinzato et al., 2022)\n\nWe consider that innovation and novelty connote different respects. Our work is based on intuitive motivation and we are the first to propose the pretraining method for attribute value extraction.  While the proposed method is not convoluted, it outperforms all prior methods and achieves SOTA performance on the benchmark datasets. Simple and effective models are often preferred in real-life applications. In addition, our method is shown to be effective also on the rare and unseen attributes, which has not been studied in previous works but is important for many applications.\n\nSecond, the Knowledge Selective Framework can be closely combined with pretraining. As shown in Table 3 and Table 4, the baseline method proposed by Shinzato et al. (2022) cannot be effectively applied in pretraining due to the lack of fine-grained run-time knowledge categorization, while KSelF is shown to be effective with or without pretraining according to Table 3.\n\nThirdly, we collect and release a new benchmark dataset which is larger and better-formatted compared with the previous benchmark. This benchmark can help with the future research in this field. We take it as another major contribution of our work.\n\n> Comment 1.2: Improvements over the baselines\n\nWe note that the proposed KSelF achieves improvement over all the metrics on both of the datasets. The improvement is statistically significant with p < 0.05. We will add the significance test results in revision. Thank you for your suggestions.\n\nAccording to Table 3, KSelf+pretrain improves the previous SOTA performance from 43.23 to 45.93 on Macro-F1 and from 89.42 to 90.00 on Micro-F1 on the AE-pub dataset, and from 71.05 to 74.30 on Macro-F1 and from 89.11 to 90.95 on Micro-F1 on the EC-AVE dataset. Such improvement is substantial. Even if we take BERTQA+pretrain as a comparison, our method outperforms it by 43.9 to 45.93 on Macro-F1 and from 89.60 to 90.00 on Micro-F1 on the AE-pub, and 72.99 to 74.30 on Macro-F1 and from 90.48 to 90.95 on Micro-F1 on the EC-AVE dataset.\n\nNote that among different metrics, the reason why the improvement is less significant on the Micro-F1 is that Micro-F1 takes all the true-positives into calculation. Both datasets exhibit a long-tail distribution on attributes, so there are more examples on the frequently seen attributes, which makes the absolute improvement on Micro-F1 less significant. The similar phenomenon can be observed in previous works such as Shinzato et al. (2022) as well.\n\n> Comment 1.3: Fairness to report GPT 3.5's performance according to one version of prompt.\n\nAs stated in Appendix B, we design different prompts and choose the one that achieves the best performance on the development set for GPT-3.5. Hence the performance of GPT-3.5 is well validated. We assume that there might be better methods that can achieve good performance with GPT-3.5, but it is out of the scope of this paper. We leave such a problem as future work on how to build AVE frameworks based on LLMs.\n\n## Comment 2\n\n> Comment 2.1: How entity retrieval works for calculating the similarity in fine-tuning.\n\nThe entity retrieval aims to categorize the run-time knowledge into more-related or less-related. For this step, we utilize two different ways, as shown in Appendix A. In brief, we adopt the entity similarity during finetuning, while adopting bag-of-word similarity during pretraining.\n\n> Comment 2.2: \"what are entities\" and \"can we retrieve attributes instead\"\n\nOur setting is to extract attribute values from product profiles (specifically product titles). We adopt the entity retrieval tool (De Cao et al., 2021). So the entities retrieved from the profile mostly include brands (Nike, Samsung, etc.), item names (shoe, cloth, etc.), human beings (boy, woman, etc.). These entities well present the main topic of a product profile, and hence can be used for calculating the similarities between product profiles.\n\nAs you can see, we cannot retrieve attributes instead. The attributes cannot represent the topic of a product profile because different product profiles may contain the same attributes. For example, a product related to \"cloth\" and a product related to \"electronic\" may both contain the attribute \"brand\", but they cannot serve as effective run-time knowledge for each other.\n\n> Comment 2.3: \"how time-consuming is entity retrieval\"\n\nThe goal of entity retrieval is to detect the similarity between two product profiles, so we want the entity retrieval tool to be universal and open-domain. We adopt the tool from De Cao et al., 2021 in our paper. For this tool, it took around 3 seconds to finish a single example, which is pretty time-consuming. \n\n> Comment 2.4: \"what's the performance without this entity retrieval?\"\n\nOur method is built upon the categorization of more-related knowledge and less-related knowledge, while entity retrieval is adopted to satisfy this goal. If no entity retrieval is adopted, then our method no longer stands. Or if we remove the categorization step, KSelF will degrade to BERT-QA. Thank you so much for the comments, we will make this clearer.\n\n> Comment 2.5: \"which may not be accurate enough to measure the similarity between two profiles.\"\n\nThe product titles are very different from normal language. The product titles are often informative yet concise, and the tokens included are mostly related to certain aspects of the product. As we are conducting attribute value extraction on product titles, the bag-of-word similarity works pretty well if the threshold is reasonable and carefully selected. We do not have golden labels to evaluate the performance of this step, but human checks over 30 random samples during our experiments on each dataset show that averagely less than 2 out of 14 values are \"less-related\" to the current product profile, which indicates that the performance of bag-of-word similarity is high enough to satisfy our goal. \n\nMeanwhile, we want to point out that the goal of categorization for more-related knowledge and less-related knowledge is to encourage the model to exploit the run-time knowledge more effectively, and hence a 100% accuracy is not required.\n\n## Comment 3\n\n> Comment 3.1: \"the proposed method is designed to solve text-only AVE task.\"\n\nThe text-based AVE is a well-established task and has been widely studied for years. In this paper, we strictly follow the setting employed by previous research to solve the text-based AVE task, i.e., retrieving attributes/values from text.\n\n> Comment 3.2: only deal with cases where attribute values can be explicitly found in the text\n\nWe follow previous works by solving the AVE task from the retrieval-based scheme. We want to point out that existing works on the AVE task mostly focus on the retrieval-based scheme, by either adopting a sequence-tagging paradigm or a QA paradigm. Both of the two paradigms mainly deal with values that are explicitly presented in the text. For the attributes that are implicitly presented, a different paradigm is required which is out of the scope of this paper. \n\n> Comment 3.3: Overall the impact and coverage of this work.\n\nOur work strictly follows the problem setting that is well-established in previous works, which is also the setting widely adopted in current research, where AVE is conducted by retrieving attributes/values from text and the attributes/values are explicitly presented in the text. Considering the large number of published works on this task previously and the impact of solutions, we do not agree with the claim that the impact and coverage are limited.\n\n## Question\n\n> Question 1: \"This work basically applies distant supervision to construct pretraining dataset. Is the precision of distant supervision100%? Does the wrong product problem occur, i.e. the profile extracted is not related to the product of interest?\"\n\nFirstly, we point out that the precision of distant supervision can never reach 100%. The goal is to categorize the run-time knowledge into the more-related and the less-related by our designed similarity measurements. So there will be errors occurring since the similarity measurements are unsupervised. But such errors are not frequent since we carefully design the entity-based similarity measurement and the bag-of-word-based similarity measurement. Experiments show that our method works and is effective. \n\nMeanwhile, even if all the intermediate categorization is wrong, our method will degrade to the baseline method of BERT-QA, which is the previous SOTA method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* KSelF tries to address the limitations of existing methods. KSelF first categorizes the knowledge (i.e., attribute values) into more-related and less-related ones, which further uses the seen values of an attribute to expand the query. \n* KSelF achieves state-of-the-art performance on two benchmark datasets.\n* KSelF outperforms existing methods on rare and unseen attributes.", "reasons_to_reject": "* The work is purely augmented from Shinzato et al. (2022) without significant innovation. The improvements from the baselines are very limited, e.g. from 89.60 micro-F1 to 90.00. GPT-3.5's performance highly relies on how the prompt was designed. It may not be fair to report its performance according to one version of prompt. \n\n* Calculating similarity scores between profiles seems to be the key to KSelF. It's not clear that how entity retrieval works for calculating the similarity in the fine-tuning stage, e.g. what are the entities in the profile? can we retrieve attributes instead? how time-consuming is it? what's the performance without this entity retrieval? And the similarity measurement in pretraining stage is based on bag-of-word similarity, which may not be accurate enough to measure the similarity between two profiles. This could lead to the selection of irrelevant knowledge to expand the query. \n\n* As mentioned by the author, the proposed method is designed to solve text-only AVE task. Also, the method can only deal with cases where attribute values can be explicitly found in the text. For example, the attribute is water_resistant and value is True, which should be able to be inferred from the profile: \"water resistant up to 50 meters\". Overall the impact and coverage of this work is limited.", "questions_to_authors": "* This work basically applies distant supervision to construct pretraining dataset. Is the precision of distant supervision100%? Does tge wrong product problem occur, i.e. the profile extracted is not related to the product of interest?", "paper_topic_and_main_contributions": "The paper proposes a new method for attribute value extraction (AVE) called Knowledge-Selective Pretraining for Attribute Value Extraction (KSelF). KSelF addresses the limitations of existing AVE methods by leveraging pretraining and query expansion. The pretraining method is based on a large-scale corpus of product profiles and attributes collected from E-commerce websites. The query expansion method uses a knowledge-selective framework to select informative knowledge from the pretraining corpus to expand the query. The experiments on two benchmark datasets show that KSelF achieves state-of-the-art performance on both datasets. KSelF also outperforms existing methods on rare and unseen attributes.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "MAVE: A Product Dataset for Multi-source Attribute Value Extraction", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 321: 5.270 attributes?"}, "extra_text": {}}, {"review_id": "u2OZ6yDHkX", "reviewer": "Reviewer_a4W9", "summary": "", "questions": "Question A: This paper pointed out that attribute values for the same attribute on products from different categories are in relatively disjoint space as a motivation for selecting knowledge when performing query expansion (i.e., categorizing values into more and less related). However, since product profiles are often associated with categories (or can be classified if not), the same can be done by retrieving values for query expansion within products of the same category. Are there any other properties of product profiles that KSelF could disambiguate to improve performance?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n## Comment 1\n\n> Comment 1.1: \"The pre-training method relies on psudo-labeling based on a naive approach (extract string match) \"\n\nOur main contributions in this work include: (1) We are the first to propose pretraining on the AVE task. (2) We design a Knowledge-Selective Framework that can be closely combined with pretraining to effectively improve the overall performance, as well as the performance on the rare and unseen attributes. (3) We construct and release a new benchmark dataset that can encourage future research in this field.\n\nConstructing a task-specific pretraining corpus is expensive and time-consuming, especially when the pretraining corpus is particularly large. Existing works mostly adopt pseudo-labeling methods to save the cost on the construction. Although our adopted approach is simple and not convoluted, it is shown to be very effective according to the experiments, and we consider that simple and effective models are often preferred in real-life applications.\n\n## Comment 2\n\n> Comment 2.1: The benchmark contributions as a newer, larger, and cleaner open dataset is available.\n\nFirstly, we construct and release a new benchmark dataset EC-AVE which is larger and cleaner than the AE-pub dataset. Experiments on both two dataset show that our method is effective. \n\nSecondly, we are not sure which newer dataset is referred to. If the MAVE dataset is referred to, the goal of this dataset is different from ours. MAVE aims to retrieve attributes from multiple sources like product descriptions, which is very different from our setting where the product profile is the product title. AVE on product titles is not easy as the product titles are more concise and informative, and sometimes the titles are not coherent natural language. Harvesting attributes/values from product titles is a more challenging task compared to the setting in the MAVE dataset. Meanwhile, as reported in the MAVE paper, the performance of the BERT-QA baseline on the MAVE dataset is 98.14, which is very saturated (some researchers have also reported that the MAVE dataset is noisy and too simple), while the Micro-F1 scores achieved by BERT-QA on the two datasets adopted in this paper are 89.42 and 89.11 respectively. The lower numbers further show that AVE on product titles is more challenging than the setting in the MAVE dataset.\n\n## Question\n\n> Question A: \"Are there any other properties of product profiles that KSelF could disambiguate to improve performance?\"\n\nYes. Our proposed method can be extended if there are more intermediate categories. In our work, we classify the run-time knowledge as the more-related and the less-related. However, if we have more fine-grained supervision or supervision detection tools, e.g., the product type the current profile is about, we can design more categories for the run-time knowledge and extend our method accordingly.  Thank you for this constructive question, and we will explore this in our future study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The newly proposed method, based on QA-AVE approach, is well grounded in current literature and addresses drawbacks of previous work.\n2. The experiment comparing the proposed method with SOTA models validates the performance gain of KSelF and the value of the pre-training corpus.", "reasons_to_reject": "1. The pre-training method relies on psudo-labeling based on a naive approach (extract string match) and is not a novel contribution per se.\n2. The benchmark contribution is compared to a relatively old and smaller dataset, while a newer, larger, and cleaner open dataset is available.", "questions_to_authors": "Question A: This paper pointed out that attribute values for the same attribute on products from different categories are in relatively disjoint space as a motivation for selecting knowledge when performing query expansion (i.e., categorizing values into more and less related). However, since product profiles are often associated with categories (or can be classified if not), the same can be done by retrieving values for query expansion within products of the same category. Are there any other properties of product profiles that KSelF could disambiguate to improve performance?", "paper_topic_and_main_contributions": "This paper presents methods to improve QA-based models for the Attribute Value Extraction (AVE) task, especially for rare attributes. The main contributions of this paper include:\n1. a Knowledge-Selective Framework (KSelF) that can use training data more efficiently, and a pre-training method for QA-based AVE models to improve model performance on rare and unseen attributes,\n2. EC-AVE, a new well-structured benchmark for the AVE task, and\n3. quantitative experimental results showing that the proposed methods outperform SOTA.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Li Yang, Qifan Wang, Zac Yu, Anand Kulkarni, Sumit Sanghai, Bin Shu, Jon Elsas, and Bhargav Kanagal. 2022. MAVE: A Product Dataset for Multi-source Attribute Value Extraction. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM '22). Association for Computing Machinery, New York, NY, USA, 1256–1265. https://doi.org/10.1145/3488560.3498377"}}, {"review_id": "qYfGe2iaav", "reviewer": "Reviewer_Z4E8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n## Comment 1\n\n> Comment 1.1: \"The experiment section would have been more robust if they included MAVE ([1]) dataset as well.\"\n\nFirstly, we construct and release a new benchmark dataset EC-AVE which is larger and cleaner than the AE-pub dataset. Experiments on both the datasets show that our method is effective.\n\nSecondly, the goal of the MAVE dataset is different from ours. MAVE aims to retrieve attributes from multiple sources like product descriptions, which is very different from our setting where the product profile is the product title.  AVE on product titles is not easy as the product titles are more concise and informative, and sometimes the titles are not coherent natural language. Harvesting attributes/values from product titles is a more challenging task compared to the setting in MAVE dataset. Meanwhile, as reported in the MAVE paper, the performance of the BERT-QA baseline on the MAVE dataset is 98.14, which is very saturated (some researchers have also reported that the MAVE dataset is noisy and too simple), while the Micro-F1 scores achieved by BERT-QA on the two datasets adopted in this paper are 89.42 and 89.11 respectively. The lower numbers further show that AVE on product titles is more challenging than the setting in the MAVE dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Authors proposed an pre-training corpus creation strategy for AVE task and showed the effectiveness of such corpus and pre-training on BERT and KSelF models.\n2. Although overall performance improvement is not very high on AE-pub or EC-AVE datasets, improvements on the low-frequency attributes and unseen attributes are quite significant on these two datasets.\n", "reasons_to_reject": "1. The experiments section of the paper may be considered little weak as the AE-pub dataset is noisy as mentioned by the authors. The experiment section would have been more robust if they included MAVE ([1]) dataset as well. \n\n\n[1] MAVE: A Product Dataset for Multi-source Attribute Value Extraction", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a knowledge selection (KS) framework and pre-training strategy for attribute-value extraction task. They use the KS framework to expand the query more accurately. They created a pre-training corpus for this task from E-commerce sites. They pre-train the BERT on this corpus along with KS framework and during fine-tuning on task specific dataset also, they use this KS framework. Experiments on the noise AE-pub and their own EC-AVE datasets show the effectiveness of the proposed work. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for attribute value extraction that addresses the limitations of existing methods by leveraging pre-training and query expansion. The experiments on two benchmark datasets show it achieves state-of-the-art performance on both datasets and outperforms existing methods on rare and unseen attributes.

This paper makes some marginal but valuable contributions that are explained through a long rebut process. However, such contributions are still just incremental to existing literature and they still leave the paper lacking in novelty.
Decision: Accept-Findings