AVEN-GR: Attribute Value Extraction and Normalization using product GRaphs
Year: 2023

Abstract:
Getting a good understanding of the user intent is vital for e-commerce applications to surface the right product to a given customer query. Query Understanding (QU) systems are essential for this purpose, and many e-commerce providers are working on complex solutions that need to be data efficient and able to capture early emerging market trends. Query Attribute Understanding (QAU) is a sub-component of QU that involves extracting named attributes from user queries and linking them to existing e-commerce entities such as brand, material, color, etc. While extracting named entities from text has been extensively explored in the literature, QAU requires specific attention due to the nature of the queries, which are often short, noisy, ambiguous, and constantly evolving. This paper makes three contributions to QAU. First, we propose a novel end-to-end approach that jointly solves Named Entity Recognition (NER) and Entity Linking (NEL) and enables open-world reasoning for QAU. Second, we introduce a novel method for utilizing product graphs to enhance the representation of query entities. Finally, we present a new dataset constructed from public sources that can be used to evaluate the performance of future QAU systems.

heading: 1 Introduction
text: Search queries are the main point of interaction between the customer and the search system. As such, extracting information from the queries is pivotal in surfacing the relevant products, making the task directly responsible for the quality of the overall customer experience. Query Understanding (QU) not only inherits all the challenges of standard natural language understanding but poses additional difficulties: queries are short and lack context, which makes them challenging to understand. They often contain implicit knowledge that
*These authors contributed equally to this work
is difficult to capture without external reference. For example, the query "M2 laptop" refers to Apple laptops since M2 processors are only sold by Apple. Furthermore, customers do not have technical writing skills, which can result in queries that are noisy or use inappropriate search terms.
In this work, we focus on the task of Query Attribute Understanding (QAU), which aims to extract the attribute values from the queries and make them usable for other downstream applications in the Search Engine (see fig. 1). QAU is related to another important task, Document Attribute Understanding (DAU), which aims to extract attributes from product descriptions. DAU has received significant attention from the community in the past years ((Zheng et al., 2018; Xu et al., 2019; Dong et al., 2020; Karamanolakis et al., 2020)) and does not suffer from the difficulties mentioned above and that are specific to queries. Both QAU and DAU are specific instances of Named Entity Recognition and Linking (NER/NEL), which aims to extract typed mentions from text. However, in contrast to classic NER, which usually handles fewer attribute types (such as Person, Location, and Organization), QAU and DAU deal with a larger number of attribute types (which can reach thousands in e-commerce as noted in (Xu et al., 2019)).
We claim that three critical elements need to be addressed to get a practical solution to QAU. Firstly, named entity recognition should be performed jointly with entity linking, in order to map the detected entities to our knowledge base. Solving these tasks separately is not practical in an industrial context, as it leads to error propagation (linking module cannot make up for a wrong attribute prediction by the NER module) and more generally hidden technical debt (see (Sculley et al., 2015)). Furthermore, separating the tasks precludes the possibility of inductive transfer, which has been shown to be crucial in related tasks (Zhang and Yang, 2021; Caruana, 1997; Ruder, 2017).
126
Secondly, product graphs (PG) are becoming a new standard to represent e-commerce concepts and the relations between searchable products. Therefore, QAU systems should be able to leverage this new source of knowledge to improve their performance. Finally, QAU systems should always be designed with an "open-world" setup in mind to deal dynamically with new concepts. For instance, if we consider the query ‘Sony A95K TV’, we should be able to detect that ‘A95K’ is a mention representing a product line even if this product does not exist in our knowledge base.
Note that extreme classification (Jain et al., 2016) is a possible alternative to classic NER/NER stacking, but it does not consider the coarse-grained nature of attributes (entities belong to different attribute types) and does not easily take into account the open-world nature of the task. Users can search for attribute values that are not yet in the knowledge base or not associated with any known product, making it difficult to predict normalized attribute values directly.

heading: Overview of our approach
text: To overcome the aforementioned limitations of existing approaches, we propose an end-to-end multitask approach that jointly predicts mentions, attribute types and entities. We build a shared representation of the text spans via a pre-trained transformer architecture (Liu et al., 2019). The shared span representation is used to determine the probability of the span being a mention, containing a particular attribute type, and representing a specific entity instance of that attribute. Our method can handle an open-world scenario where an attribute value does not have a matching entity in the knowledge base. In such cases, the model can still predict the attribute type of the value. Note
that this approach is also data-efficient and can effectively utilize weakly labeled data points without entity annotations. Importantly, the end-to-end approach avoids error propagation since the entitylevel prediction is conditioned but not solely reliant on the attribute-level information. Additionally, our approach can handle overlapping spans without requiring additional adjustments. Finally, if the entities are structured in a knowledge graph, our approach can leverage its topology to enrich the entity embeddings.
In order for our approach to be tested in scenarios with varying difficulties we need a dataset of queries of controllable complexity, along with a knowledge graph involving the entities there mentioned. To this end, we propose leveraging the products in the Amazon Berkeley Objects dataset (Collins et al., 2022) to construct a knowledge graph consisting of products related to their attribute values by relations encoding the attribute type. The product graph is used both as knowledge base for the approach and as starting artifact to generate a dataset of public synthetic queries. As public, non-confidential resources, we aim to release both artifacts for reproducibility and to encourage research in the field. Summarizing, our contributions are three-fold:
1. we propose AVEN, a novel end-to-end method that can effectively solve QAU in an open world setting;
2. we propose a way to use Product Graph to enrich the representation of the entities
3. we present a novel evaluation that combines a public product graph with a set of synthetic queries involving associated entities, aimed at promoting research on knowledge-based methods for QAU.

heading: 2 Related work
text: Document Attribute Understanding As previously noted, Query Attribute Understanding (QAU) shares similarities with Document Attribute Understanding (DAU), which has been previously addressed in the literature. (Zheng et al., 2018) proposed an early solution based on a classic NER pipeline that assigns each attribute type with a set of BIO (Beginning, Inside, Outside) tags. However, this approach suffers from scalability issues when dealing with a large number of attributes, and also
hinders data sharing between head attributes (such as color) and tail attributes (such as glass color).
To solve this issue, several approaches (Xu et al., 2019; Dong et al., 2020) based on Question Answering were pushed in the subsequent years. These approaches consider each attribute as a separate question to be answered leveraging the product description. The main advantage of Question Answering approaches is that they do not require a specific set of BIO tags for each attribute and are therefore more scalable. However, they are also harder to train and highly depend on the semantic representations of the attribute types. In practical cases in which the detected entity mentions must also be linked to normalized entities, Entity Linking is performed independently over the output of the NER step. While all these works consider an attribute value to be just a span of unstructured text, we aim to directly obtain normalized entities as attribute values, hence requiring performing Entity Linking over the detected spans.
Entity Linking Entity Linking has been mostly studied in scenarios involving long documents with lot of context, while only few works exist for short sentences like queries. Most relevant to our work is ELQ (Li et al., 2020), in which a bi-encoder is employed to jointly perform mention detection and EL in a multi-task setup. Analogously, in Oliya et al. (2021) mention detection and entity linking are coupled with question answering in an end-to-end pipeline. We take inspiration from both works to tackle AVEN by injecting a new stage in the end-toend mention detection and entity linking pipeline, responsible for classifying the span attribute.
Query Attribute Understanding While it may be tempting to view Query Attribute Understanding (QAU) as a simplified version of Document Attribute Understanding (DAU), this assumption overlooks the unique challenges posed by queries, such as their inherent noisiness, lack of context, and ambiguity. To the best of our knowledge, the only existing work that deals with both attribute value extraction and subsequent entity linking is QUEACO (Zhang et al., 2021). Differently from our approach, QUEACO is a fragmented model that stacks a user-behavior based normalization module over a NER pipeline. While we use user behavior in the data collection, we don’t require it for the training and inference pipelines.

heading: 3 Data
text: In order to have a controlled ground for experimentation, we need (i) a dataset of user queries, and (ii) a Knowledge Graph containing most entities involved in the user queries. Knowledge Graphs involving products and relative information are usually called product graphs.

heading: 3.1 Product Graphs
text: A Product Graph is a Knowledge Graph involving a set of products and their corresponding attributes. Formally, it is a bipartite graph consisting of a vertex set V = (P ∪ A) containing products P and attribute values A connected by edges E = R1 ∪ R2 ∪ · · · ∪ Rm, where R1, R2, . . . , Rm are set of edges for the different m attribute types. In practice, a triple (p, r, a) relates a product p with an attribute value a through an attribute relation r.

heading: 3.2 Synthetic data
text: Given the lack of a public Product Graph, we constructed one by leveraging the Amazon Berkeley Objects (ABO)1 dataset (Collins et al., 2022). The constructed graph not only lends itself to the overall inference pipeline, but can also be used to generate a set of synthetic queries that involve the entities of interest by construction. The generation procedure simply constructs queries as bag of attributes by starting from product nodes and walking the relations related to the attributes of interest, then discarding the product node in the final query and only keeping its attribute values along with the attribute type annotations. The generation pipeline is formalized in appendix C. To increase the complexity of the dataset, we also replace product types with synonyms found in the same WordNet synsets (Fellbaum, 1998).

heading: 3.3 Real user queries
text: Given the huge number of possible attribute values, manual annotation of user queries with attribute and entity-level labels is unfeasible. For this reason, we leverage a pre-trained NER model to obtain the attribute-level labels and employ a deterministic heuristic to label the corresponding attribute values with entity-level annotations. Let P be a set of purchased items, and Q be the queries that led to the purchase. First, we create a Product Graph PG
1https://amazon-berkeleyobjects.s3.amazonaws.com/index.html
from P by creating a triple (p, r, a) for each product p connected to an attribute value a through attribute type r. Then, for each query q ∈ Q, we iterate over each NER-annotated span (r, v, s), where span s holds value v for attribute type r. We now want to annotate the span s with two annotations, one at the attribute level and one at the entity level. For the former, we can keep the one detected from NER r. For the entity-level annotation instead, we choose to annotate s with the entity a such that (p, r, a) ∈ PG. In other words, given that NER has predicted the span to refer to an attribute type r, we annotate the span with the entity corresponding to the attribute value for r of the product that the user bought after searching for the query q. Assume for instance that an user looked for ‘red Nike shoes’ and eventually bought some product p referring to a specific pair of shoes that are, in fact, red. In this case, the span s0,1 with value ‘red’ can be annotated to be a color as predicted by NER, while the entity label will be that of the value for p for the attribute color, which is the node corresponding to the value ‘red’ in the knowledge base. Of course, the user may also have eventually bought a black pair of shoes instead: in this case, the heuristic makes a mistake, and therefore the annotation is expected to be noisy. Nevertheless, assuming the query keywords to encode strong preferences when present, these cases are expected to be rare enough for the model to eventually learn to discard them as noise.

heading: 4 Approach
text: The overall architecture of AVEN contains three different sub-modules, each responsible for a different task: (i) a mention detection module; (ii) an attribute classification module; (iii) an entity disambiguation module.
The three modules are learnt jointly as shown in fig. 2 and each of them contributes to the final loss. The latter is obtained as a weighted sum of the three losses. While the coefficients are currently set to 1 for all the three tasks, we aim to eventually use GradNorm (Chen et al., 2018) to tune the loss weights.
More formally, let us define q = q1, . . . , qn as an input query with n tokens/words. We denote by s[i,j] the sub-span qiqi+1 . . . qj . We are interested in three different quantities: Mij refers to span s[i,j] being a mention, Aaij refers to the same span being an attribute value for attribute a, and finally
Eeij refers to s[i,j] being an instance of entity e. In the next sections, we will review the three different components.

heading: Mention Detection
text: For a span s[i,j], we denote the span embedding by sij = fθ(s[i,j]). A simple version of fθ(s[i,j]) is the mean of the RoBERTa (Liu et al., 2019) embeddings of the tokens in s[i,j]. We can define the probability of span s[i,j] being an actual mention to be
P (Mij) = σ (gµ(sij)) ,
where σ is the sigmoid function and gµ(·) is a parametric function taking in input the span representation and returning an unnormalized score. In our current implementation, this is realized as a MultiLayer Perceptron (MLP). Note that we employ the sigmoid as we assume that the probability of a span s[i,j] to be a mention does not depend on the probability of another span s[k,l] to be a mention. Note that, this assumption is questionable, especially as soon as s[i,j] and s[k,l] have a non-null intersection. Nevertheless, this choice allows the model to detect overlapping spans when faced with cases such as those exemplified in section 1. Note that it’s always possible to add a Non-Maximum Suppression (NMS) step if we want to avoid producing overlapping annotations. The mention detector is trained by minimizing a Binary Cross Entropy loss ℓMD.

heading: Attribute classification
text: We are now interested in the probability that a span s[i,j] has attribute type a knowing that it is a mention
P ( A (a) i,j |Mij ) =
exp ( h (a) ν (si,j) )
∑ a′∈A exp ( h (a′) ν (si,j) ) ,
where h(·)ν () is a parametric function taking into input the span representation. As for the mention detector, we employ a MLP. Note that we adopt a multi-task approach where we use the exact same span representation for the three different tasks, fostering information transfer among the latter. The attribute classifier is trained with a simple cross entropy loss ℓAC and only considers actual groundtruth mentions at train time.

heading: Entity disambiguation
text: In the entity disambiguation module, our goal is to estimate the probability P ( E (e) ij |Mij ) . Given the
fact that each entity e is associated with an unique attribute type a = type (e), we can argue that this is actually equivalent to estimating the joint probability P ( E (e) ij , A (a) ij |Mij ) Since the probability of a span to be type a is already given by the attribute classifier, we can just estimate for each possible attribute a
P ( E (e) ij | A (a) ij ,Mij ) =
exp ( v (a) ξ (si,j , e) )
∑ e′∈E exp ( v (a) ξ (si,j , e ′) ) ,
where v(a)ξ (·) is a parametric function taking into input the span representation and the entity e to be scored. The main advantage of this last expression is that it allows us to adopt a divide-and-conquer approach since for each attribute a, we only have to consider its compatible entities. Similarly to the attribute classifier, the entity disambiguator is learnt with a simple cross entropy loss ℓED on actual groundtruth mentions. Our first implementation of v(a)ξ (·) is a simple similarity scorer between the span representation and the embedding of the considered entity. Entity embeddings are computed by embedding a textual representation of their neighborhood in the knowledge graph, as illustrated in Figure 3.

heading: Inference
text: To compute the probability of each span s[i,j] being a mention of entity e at inference time, we simply multiply the mention probability by the entity classification score. To improve efficiency, we exclude all spans s[i,j] with a mention probability P (Mij) lower than a pre-defined threshold pmin, such as 0.5 in our experiments.

heading: Advantages
text: Our approach shares the span representation across all three tasks: mention detection, attribute classification, and entity disambiguation, benefiting from the effectiveness of multi-task learning (Caruana, 1997; Ruder, 2017) in transferring knowledge between similar tasks. This is particularly relevant for our method as the tasks require different levels of label details: mention detection only requires weak labeling, while the attribute/entity tasks rely on associations between mentions and knowledge graph entities. Sharing the representation allows the entity disambiguation module to leverage weakly-labeled mention data, leading to a more data-efficient approach.

heading: 5 Experimental Results
text: In this section, we present experimental results on two datasets described in section 3.2 and section 3.3. We provide a brief overview of the protocol used in both cases.

heading: 5.1 Considered metrics

heading: Mention Detection
text: We report both (micro) Precision and Recall for the mention detection task to validate the performance of the mention detector. The percentage of recalled mention will be a natural upper bound for the following metrics on attribute classification. Indeed, if we are not able to retrieve a mention, we will consider that we cannot be right at the subsequent tasks.

heading: Attribute Classification
text: We report the multiclass Accuracy for the attribute classification task; This metric is computed on the set of ground-truths mentions and thus ignoring
wrongly detected mentions (for which no attribute exists). We also present a complementary version of this metric, which focuses exclusively on groundtruth mentions that contain previously unseen "unknown" entities. This metric is only applicable to the second, more realistic dataset that includes novel entities in the test set.

heading: Entity Disambiguation
text: We report the multiclass Accuracy for the entity disambiguation task; This one is computed only on the subset of ground-truth mentions containing entities seen at train time.

heading: 5.2 Baselines
text: We consider the following models (i) NER+Dict: A RoBERTa-based NER baseline with dictionary lookup over the detected attributes. (ii) NER+NN: A RoBERTa-based NER baseline with nearest neighbor between detected attribute embeddings and entity embeddings. (iii) AVEN/AVEN-NC/AVEN-GR: Our end– to-end approach in three different flavours: with plain entity embedding, with plain entity embedding and no contextual span embedding and with product-graph based embeddings.

heading: 5.3 Results
text: We report in fig. 4 (resp. fig. 5) the results from the synthetic dataset described in section 3.2 (resp. the actual user queries described in section 3.3). Overall, our AVEN- methods outperform the "stacked" methods (NER + separate entity linker), particularly on the task of known entity classification. Among our methods, AVEN-NC has a lower mention recall due to the lack of contextual span embedding. However, our methods are effective in predicting the attribute type of unseen entities, as
evidenced by their performance on this task. It is worth noting that the attribute classification performance is lower for unseen attributes, which is expected.

heading: 6 Conclusions and future directions
text: In this paper, we introduced a novel approach to tackle QAU in a multi-task fashion. We demonstrated its effectiveness on two datasets, compared to some simple baselines. However, further ablation studies on more datasets / baselines (e.g. Ayoola et al. 2022) are necessary to assess its generalization power. Additionally, future work will focus on improving the multitasking efficiency of AVEN, for instance by implementing (Chen et al., 2018).

heading: A Limitations
text: Despite the promising results achieved by our approach, some limitations must be acknowledged. First, the use of product graphs as a knowledge source is a double-edged sword. Indeed, while it provides a valuable resource to exploit, the constant evolution of product graphs may create a strong coupling between the algorithm and the knowledge source, thus reducing the method’s robustness over time. Second, our method’s span-based approach makes it computationally expensive, requiring setting a maximum span size to circumvent this issue

heading: B Ethics Statement
text: Our approach aims to boost the effectiveness of ecommerce search engines. However, by jointly optimizing multiple tasks, we run the risk of creating a less transparent system that could be susceptible to biases. These biases may lead to certain less frequent entities being overlooked or misclassified as more common ones, thereby reducing the overall fairness and accuracy of the system.

heading: C Synthetic queries generation
text: Algorithm 1 outlines the synthetic query generation procedure.
Algorithm 1 Synthetic queries generation. 1: procedure GENERATE QUERIES(pg: Product-
Graph) 2: P ← pg.products 3: Acons ← considered attributes 4: Q← [] ▷ queries 5: for all product p in P do 6: Ap ← [] ▷ attributes for the product 7: T ← all triples (p, ∗, ∗) in pg 8: for all triple (p, a, r) in T do 9: if a in Acons then 10: Ap ← Ap ∪ a ▷ attribute values 11: Rp ← Rp ∪ r ▷ attribute types 12: end if 13: end for 14: shuffle Ap and Rp accordingly 15: qtext = str(Ap) ▷ query is a bag of
attribute values 16: qann = Rp ▷ annotations 17: end for 18: return Q 19: end procedure

heading: D Prediction inspection
text: We present in fig. 6, an example of our qualitative evaluation within the QAU framework we have presented.

heading: Future Work
text: Additionally, future work will focus on improving the multitasking efficiency of AVEN, for instance by implementing (Chen et al. , 2018)