SlowGuess/ACL_2020_OCR huggingface dataset:
Giannis Karamanolakis*
New York, NY 10027, USA
gkaraman@cs.columbia.edu
Jun Ma, Xin Luna Dong
Seattle, WA 98109, USA
{junmaa，lunadong}@amazon.com
Extracting structured knowledge from product profiles is crucial for various applications in e-Commerce. State-of-the-art approaches for knowledge extraction were each designed for a single category of product, and thus do not apply to real-life e-Commerce scenarios, which often contain thousands of diverse categories. This paper proposes TXtract, a taxonomy-aware knowledge extraction model that applies to thousands of product categories organized in a hierarchical taxonomy. Through category conditional self-attention and multi-task learning, our approach is both scalable, as it trains a single model for thousands of categories, and effective, as it extracts category-specific attribute values. Experiments on products from a taxonomy with 4,000 categories show that TXtract outperforms state-of-the-art approaches by up to  $10\%$  in F1 and  $15\%$  in coverage across all categories.
Real-world e-Commerce platforms contain billions of products from thousands of different categories, organized in hierarchical taxonomies (see Figure 1). Knowledge about products can be represented in structured form as a catalog of product attributes (e.g., flavor) and their values (e.g., "strawberry"). Understanding precise values of product attributes is crucial for many applications including product search, recommendation, and question answering. However, structured attributes in product catalogs are often sparse, leading to unsatisfactory search results and various kinds of defects. Thus, it is invaluable if such structured information can be extracted from product profiles such as product titles and descriptions. Consider for instance the "Ice Cream" product of Figure 1. The corresponding title can potentially
Figure 1: A hierarchical taxonomy with various product categories and the public webpage of a product assigned to "Ice Cream" category.
be used to extract values for attributes, such as "Ben & Jerry's" for brand, "Strawberry Cheese-cake" for flavor, and "16 oz" for capacity.
State-of-the-art approaches for attribute value extraction (Zheng et al., 2018; Xu et al., 2019; Rezk et al., 2019) have employed deep learning to capture features of product attributes effectively for the extraction purpose. However, they are all designed without considering the product categories and thus cannot effectively capture the diversity of categories across the product taxonomy. Categories can be substantially different in terms of applicable attributes (e.g., a "Camera" product should not have flavor), attribute values (e.g., "Vitamin" products may have "fruit" flavor but "Banana" products should not) and more generally, text patterns used to describe the attribute values (e.g., the phrase "infused with" is commonly followed by a scent value such as "lavender" in "Hair Care" products but not in "Mattresses" products).
In this paper, we consider attribute value extraction for real-world hierarchical taxonomies with thousands of product categories, where directly
* Work performed during internship at Amazon.
8489
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8489-8502 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
applying previous approaches presents limitations. On the one extreme, ignoring the hierarchical structure of categories in the taxonomy and assuming a single "flat" category for all products does not capture category-specific characteristics and, as we will show in Section 5, is not effective. On the other extreme, training a separate deep neural network for each category in the product taxonomy is prohibitively expensive, and can suffer from lack of training data on small categories.
To address the limitations of previous approaches under this challenging setting, we propose a framework for category-specific attribute value extraction that is both efficient and effective. Our deep neural network, TXtract, is taxonomy-aware: it leverages the hierarchical taxonomy of product categories and extracts attribute values for a product conditional to its category, such that it automatically associates categories with specific attributes, valid attribute values, and category-specific text patterns. TXtract is trained on all categories in parallel and thus can be applied even on small categories with limited labels.
The key question we need to answer is how to condition deep sequence models on product categories. Our experiments suggest that following previous work to append category-specific artificial tokens to the input sequence, or concatenate category embeddings to hidden neural network layers is not adequate. There are two key ideas behind our solution. First, we use the category information as context to generate category-specific token embeddings via conditional self-attention. Second, we conduct multi-task training by meanwhile predicting product category from profile texts; this allows us to get token embeddings that are discriminative of the product categories and further improve attribute extraction. Multi-task training also makes our extraction model more robust towards wrong category assignment, which occurs often in real e-Commerce websites.
To the best of our knowledge, TXtract is the first deep neural network that has been applied to attribute value extraction for hierarchical taxonomies with thousands of product categories. In particular, we make three contributions.
1. We develop TXtract, a taxonomy-aware deep neural network for attribute value extraction from product profiles for multiple product categories. In TXtract, we capture the hierarchical relations between categories into category embeddings, which in turn we use as context to generate category-specific token embeddings via conditional self-attention.
2. We improve attribute value extraction through multi-task learning: TXtract jointly extracts attribute values and predicts the product's categories by sharing representations across tasks.
3. We evaluate TXtract on a taxonomy of 4,000 product categories and show that it substantially outperforms state-of-the-art models by up to  $10\%$  in F1 and  $15\%$  in coverage across all product categories.
Although this work focuses on e-Commerce, our approach to leverage taxonomies can be applied to broader domains such as finance, education, and biomedical/clinical research. We leave experiments on these domains for future work.
The rest of this paper is organized as follows. Section 2 discusses related work. Section 3 presents background and formally defines the problem. Section 4 presents our solution and Section 5 describes experimental results. Finally, Section 6 concludes and suggests future work.
Here, we discuss related work on attribute value extraction (Section 2.1), and multi-task learning/meta-learning (Section 2.2).
2.1 Attribute Value Extraction from Product Profiles
Attribute value extraction was originally addressed with rule-based techniques (Nadeau and Sekine, 2007; Vandic et al., 2012; Gopalakrishnan et al., 2012) followed by supervised learning techniques (Ghani et al., 2006; Putthividhya and Hu, 2011; Ling and Weld, 2012; Petrovski and Bizer, 2017; Sheth et al., 2017). Most recent techniques consider open attribute value extraction: emerging attribute values can be extracted by sequence tagging, similar to named entity recognition (NER) (Putthividhya and Hu, 2011; Chiu and Nichols, 2016; Lample et al., 2016; Yadav and Bethard, 2018). State-of-the-art methods employ deep learning for sequence tagging (Zheng et al.,
Examples: (1) an ethernet cable assigned under the "Hair Brushes": https://www.amazon.com/dp/B012AE5EP4; (2) an eye shadow product assigned under "Travel Cases": https://www.amazon.com/dp/B07BBM5B33. Screenshots of these product profiles are taken in 12/2019 and available at the Appendix.
8490
2018; Xu et al., 2019; Rezk et al., 2019). However, all previous methods can be adapted to a small number of categories and require many labeled datapoints per category. Even the Active Learning method of Zheng et al. (2018) requires humans to annotate at least hundreds of carefully selected examples per category. Our work differs from previous approaches as we consider thousands of product categories organized in a hierarchical taxonomy.
2.2 Multi-Task/Meta-Learning
Our framework is related to multi-task learning (Caruana, 1997) as we train a single model simultaneously on all categories (tasks). Traditional approaches consider a small number of different tasks, ranging from 2 to 20 and employ hard parameter sharing (Alonso and Plank, 2017; Yang et al., 2017; Ruder, 2019): the first layers of neural networks are shared across all tasks, while the separate layers (or "heads") are used for each individual task. In our setting with thousands of different categories (tasks), our approach is efficient as we use a single (instead of thousands) head and effective as we distinguish between categories through low-dimensional category embeddings.
Our work is also related to meta-learning approaches based on task embeddings (Finn et al., 2017; Achille et al., 2019; Lan et al., 2019): the target tasks are represented in a low-dimensional space that captures task similarities. However, we generate category embeddings that reflect the already available, hierarchical structure of product categories in the taxonomy provided by experts.
3 Background and Problem Definition
We now provide background on open attribute value extraction (Section 3.1) and define our problem of focus (Section 3.2).
3.1 Open Attribute Value Extraction
Most recent approaches for attribute value extraction rely on the open-world assumption to discover attribute values that have never been seen during training (Zheng et al., 2018). State-of-the-art approaches address extraction with deep sequence tagging models (Zheng et al., 2018; Xu et al.,
Input	Ben	&	Jerry's	black	cherry	cheesecake	ice	cream
Output	O	O	O	B	I	E	O	O
Table 1: Example of input/output tag sequences for the "flavor" attribute of an ice cream product.
2019; Rezk et al., 2019): each token of the input sequence  $x = (x_{1},\ldots ,x_{T})$  is assigned a separate tag from  $\{\mathrm{B,I,O,E}\}$ , where "B," "I," "O," and "E" represent the beginning, inside, outside, and end of an attribute, respectively. (Not extracting any values corresponds to a sequence of "O"-only tags.) Table 1 shows an input/output example of flavor value extraction from (part of) a product title. Given this output tag sequence, "black cherry cheesecake" is extracted as a flavor for the ice cream product.
We represent the product taxonomy as a tree  $C$ , where the root node is named "Product" and each taxonomy node corresponds to a distinct product category:  $c \in C$ . A directed edge between two nodes represents the category-to-subcategory relationship. A product is assigned to a category node in  $C$ . In practice, there are often thousands of nodes in a taxonomy tree and the category assignment of a product may be incorrect. We now formally define our problem as follows.
DEFINITION: Consider a product from a category  $c$  and the sequence of tokens  $x = (x_{1},\ldots ,x_{T})$  from its profile, where  $T$  is the sequence length. Let  $a$  be a target attribute for extraction. Attribute extraction identifies subsequences of tokens from  $x$ , each sub-sequence representing a value for  $a$ .
For instance, given (1) a product title  $x =$  "Ben & Jerry's Strawberry Cheesecake Ice Cream 16 oz," (2) a product category  $c =$  "Ice Cream," and (3) a target attribute  $\alpha =$  flavor, we would like to extract "Strawberry Cheesecake" as a flavor for this product. Note that we may not see all valid attribute values during training.
4 TXtract Model: Taxonomy-Aware Knowledge Extraction
In this paper, we address open attribute value extraction using a taxonomy-aware deep sequence tagging model, TXtract. Figure 2 shows the model architecture, which contains two key components: attribute value extraction and product category
$^{2}$ Zheng et al. (2018) considered 3 categories: "Dog Dood," "Cameras," and "Detergent." Xu et al. (2019) consider 1 category: "Sports & Entertainment." Rezk et al. (2019) considered 21 categories and trained a separate model for each category.
8491
Figure 2: TXtract architecture: tokens  $(x_{1},\ldots ,x_{T})$  are classified to BIOE attribute tags  $(y_{1},\ldots ,y_{T})$  by conditioning to the product's category embedding  $e_c$ . TXtract is jointly trained to extract attribute values and assign a product to taxonomy nodes.
prediction, accounting for the two tasks in multitask training. Both components are taxonomy aware, as we describe next in detail.
4.1 Taxonomy-Aware Attribute Value Extraction
TXtract leverages the product taxonomy for attribute value extraction. The underlying intuition is that knowing the product category may help infer attribute applicability and associate the product with a certain range of valid attribute values. Our model uses the category embedding in conditional self-attention to guide the extraction of category-specific attribute values.
4.1.1 Product Encoder
The product encoder ("ProductEnc") represents the text tokens of the product profile  $(x_{1},\ldots ,x_{T})$  as low-dimensional, real-valued vectors:
$$
h _ {1}, \dots h _ {T} = \operatorname {P r o d u c t E n c} \left(\mathrm {x} _ {1}, \dots , \mathrm {x} _ {\mathrm {T}}\right) \in \mathbb {R} ^ {d}. \tag {1}
$$
To effectively capture long-range dependencies between the input tokens, we use word embeddings followed by bidirectional LSTMs (BiLSTMs), similar to previous state-of-the-art approaches (Zheng et al., 2018; Xu et al., 2019).
4.1.2 Category Encoder
Our category encoder ("CategoryEnc") encodes the hierarchical structure of product categories
such that TXtract understands expert-defined relations across categories, such as "Lager" is a subcategory of "Beer". In particular, we embed each product category  $c$  (taxonomy node) into a low-dimensional latent space:
$$
e _ {c} = \operatorname {C a t e g o r y E n c} (c) \in \mathbb {R} ^ {m}. \tag {2}
$$
To capture the hierarchical structure of the product taxonomy, we embed product categories into the  $m$ -dimensional Poincaré ball (Nickel and Kiela, 2017), because its underlying geometry has been shown to be appropriate for capturing both similarity and hierarchy.
4.1.3 Category Conditional Self-Attention
The key component for taxonomy-aware value extraction is category conditional self-attention ("CondSelfAtt"). CondSelfAtt generates category-specific token embeddings  $(\tilde{h}_i \in \mathbb{R}^d)$  by conditioning on the category embedding  $e_c$ :
$$
\tilde {h} _ {1}, \dots \tilde {h} _ {T} = \operatorname {C o n d S e l f A t t} \left(\left(h _ {1}, \dots , h _ {T}\right), e _ {c}\right). \tag {3}
$$
To leverage the mutual interaction between all pairs of token embeddings  $h_t$ ,  $h_{t'}$  and the category embedding  $e_c$  we use self-attention and compute pairwise sigmoid attention weights:
$$
\alpha_ {t, t ^ {\prime}} = \sigma \left(w _ {\alpha} ^ {T} g _ {t, t ^ {\prime}} + b _ {\alpha}\right), \quad t, t ^ {\prime} = 1.. T. \tag {4}
$$
We compute scores  $g_{t,t'}$  using both the token embeddings  $h_t, h_{t'}$  and the category embedding  $e_c$ :
8492
$$
g _ {t, t ^ {\prime}} = \tanh  \left(W _ {1} h _ {t} + W _ {2} h _ {t ^ {\prime}} + W _ {3} e _ {c} + b _ {g}\right), \tag {5}
$$
where  $W_{1}\in \mathbb{R}^{p\times d}$ $W_{2}\in \mathbb{R}^{p\times d}$ $W_{3}\in \mathbb{R}^{p\times m}$ $w_{\alpha}\in \mathbb{R}^{p}$  are trainable attention matrices and  $b_{g}\in \mathbb{R}^{p},b_{\alpha}\in \mathbb{R}$  , are trainable biases. The  $T\times T$  attention matrix  $A = a_{t,t^{\prime}}$  stores the pairwise attention weights. The contextualized token embeddings are computed as:
$$
\tilde {h} _ {t} = \sum_ {t ^ {\prime} = 1} ^ {T} \alpha_ {t, t ^ {\prime}} \cdot h _ {t ^ {\prime}}. \tag {6}
$$
4.1.4 CRF Layer
We feed the contextualized token representations  $\tilde{h} = (\tilde{h}_1,\dots ,\tilde{h}_T)$  to CRFs to get the sequence of BIOE tags with the highest probability:
$$
y _ {1}, \dots , y _ {T} = \operatorname {C R F} \left(\tilde {h} _ {1}, \dots , \tilde {h} _ {t}\right). \tag {7}
$$
We then extract attribute values as valid subsequences of the input tokens  $(x_{1},\ldots ,x_{T})$  with B/I/E tags (see Section 3.1).
4.1.5 Training for Attribute Value Extraction
Our training objective for attribute value extraction is to minimize the negative conditional log-likelihood of the model parameters on  $N$  training products  $x_{i}$  with ground truth labels  $\hat{y}_{i1},\dots ,\hat{y}_{iT}$ :
$$
L _ {a} = - \sum_ {i = 1} ^ {N} \log P r \left(\hat {y} _ {i 1}, \dots , \hat {y} _ {i T} \mid x _ {i}, c _ {i}\right) \tag {8}
$$
We train our model on all categories in parallel, thus leveraging for a given category products from related categories. To generate training sequence labels from the corresponding attribute values, we use the distant supervision framework of Mintz et al. (2009), similar to Xu et al. (2019), by generating tagging labels according to existing (sparse) values in the Catalog.
4.2 Taxonomy-Aware Product Category Prediction
We now describe how we train TXtract for the auxiliary task of product category prediction through multi-task learning. Our main idea is that by encouraging TXtract to predict the product categories using only the product profile, the model will learn token embeddings that are discriminative of the product categories. Thus, we introduce an inductive bias for more effective category-specific attribute value extraction.
4.2.1 Attention Layer
Our attention component ("Att") represents the product profile  $(x_{1},\ldots ,x_{T})$  as a single vector  $h\in \mathbb{R}^n$  computed through the weighted combination of the ProductEnc's embeddings  $(h_1,\dots ,h_T)$  ..
$$
h = \sum_ {t = 1} ^ {T} \beta_ {t} \cdot h _ {t}. \tag {9}
$$
This weighted combination allows tokens that are more informative for a product's category to get higher "attention weights"  $\beta_{1},\ldots ,\beta_{T}\in [0,1]$ . For example, we expect  $x_{t} =$  "frozen" to receive a relatively high  $\beta_{t}$  for the classification of a product to the "Ice Cream" category. We compute the attention weights as:
$$
\beta_ {t} = \operatorname {s o f t m a x} \left(u _ {c} ^ {T} \tanh  \left(W _ {c} h _ {t} + b _ {c}\right)\right), \tag {10}
$$
where  $W_{c}\in \mathbb{R}^{q\times d},b_{c}\in \mathbb{R}^{q},u_{c}\in \mathbb{R}^{q}$  are trainable attention parameters.
4.2.2 Category Classifier
Our category classifier ("CategoryCLF") classifies the product embedding  $h$  to the taxonomy nodes. In particular, we use a sigmoid classification layer to predict the probabilities of the taxonomy nodes:
$$
p _ {1}, \dots , p _ {| C |} = \operatorname {s i g m o i d} \left(W _ {d} h + b _ {d}\right), \tag {11}
$$
where  $W_{d} \in \mathbb{R}^{|C| \times d}$  and  $b_{d} \in \mathbb{R}^{|C|}$  are trainable parameters. We compute sigmoid (instead of softmax) node probabilities because we treat category prediction as multi-label classification, as we describe next.
4.2.3 Training for Category Prediction
Training for "flat" classification of products to thousands of categories is not effective because the model is fully penalized if it does not predict the exact true category  $\hat{c}$  while at the same time ignores parent-children category relations. Here, we conduct "hierarchical" classification by incorporating the hierarchical structure of the product taxonomy into a taxonomy-aware loss function.
The insight behind our loss function is that a product assigned under  $\hat{c}$  could also be assigned under any of the ancestors of  $\hat{c}$ . Thus, we consider hierarchical multi-label classification and encourage TXtract to assign a product to all nodes in the path from  $\hat{c}$  to the root, denoted by  $(\hat{c}_K,\hat{c}_{K - 1},\dots ,\hat{c}_1)$ , where  $K$  is the level of the
8493
node  $\hat{c}$  in the taxonomy tree. The model is thus encouraged to learn the hierarchical taxonomy relations and will be penalized less if it predicts high probabilities for ancestor nodes (e.g., "Beer" instead of "Lager" in Figure 1).
Our minimization objective is the weighted version of the binary cross-entropy (instead of un-weighted categorical cross-entropy) loss:3
$$
L _ {b} = \sum_ {c \in C} w _ {c} \left(y _ {c} \cdot \log p _ {c} + \left(1 - y _ {c}\right) \cdot \log \left(1 - p _ {c}\right)\right), \tag {12}
$$
For the nodes in the path from  $\hat{c}$  to the root  $(\hat{c}_K, \hat{c}_{K-1}, \dots, \hat{c}_1)$ , we define positive labels  $y_c = 1$  and weights  $w_c$  that are exponentially decreasing  $(w^0, w^1, \dots, w^{K-1})$ , where  $0 < w \leq 1$  is a tunable hyper-parameter. The remaining nodes in  $C$  receive negative labels  $y_c = 0$  and fixed weight  $w_c = w^{K-1}$ .
4.3 Multi-task Training
We jointly train TXtract for attribute value extraction and product category prediction by combining the loss functions of Eq. (8) and Eq. (12):
$$
L = \gamma \cdot L _ {a} + (1 - \gamma) \cdot L _ {b}, \tag {13}
$$
where  $\gamma \in [0,1]$  is a tunable hyper-parameter. Here, we employ multi-task learning, and share ProductEnc across both tasks.
We empirically evaluated TXtract and compared it with state-of-the-art models and strong baselines for attribute value extraction on 4000 product categories. TXtract leads to substantial improvement across all categories, showing the advantages of leveraging the product taxonomy.
Dataset: We trained and evaluated TXtract on products from public web pages of Amazon.com. We randomly selected 2 million products from 4000 categories under 4 general domains (sub-trees) in the product taxonomy: Grocery, Baby product, Beauty product, and Health product.
Experimental Setup: We split our dataset into training (60%), validation (20%), and test (20%) sets. We experimented with extraction of flavor, scent, and brand values from product titles, and
with ingredient values from product titles and descriptions. For each attribute, we trained TXtract on the training set and evaluated the performance on the held-out test set.
Evaluation Metrics: For a robust evaluation of attribute value extraction, we report several metrics. For a test product, we consider as true positive the case where the extracted values match at least one of the ground truth values (as some of the ground truth values may not exist in the text) and do not contain any wrong values. $^{4}$  We compute Precision (Prec) as the number of "matched" products divided by the number of products for which the model extracts at least one attribute value; Recall (Rec) as the number of "matched" products divided by the number of products associated with attribute values; and  $F1$  score as the harmony mean of Prec and Rec. To get a global picture of the model's performance, we consider micro-average scores ( $\mathrm{Mi}^*$ ), which first aggregates products across categories and computes Prec/Rec/F1 globally. To evaluate per-category performance we consider macro-average scores ( $\mathrm{Ma}^*$ ), which first computes Prec/Rec/F1 for each category and then aggregates per-category scores. To evaluate the capability of our model to discover (potentially new) attribute values, we also report the Value vocabulary (Vocab) as the total number of unique attribute values extracted from the test set (higher number is often better); and Coverage (Cov), as the number of products for which the model extracted at least one attribute value, divided by the total number of products.
For product category (multi-label) classification we reported the area under Precision-Recall curve (AUPR), Precision, Recall, and F1 score.
Model Configuration: We implemented our model in Tensorflow (Abadi et al., 2016) and Keras. For a fair comparison, we consider the same configuration as OpenTag for the ProductEnc (BiLSTM) and CRF components. For model configuration details see the appendix.
Model Comparison: We compared our model with state-of-the-art models in the literature and
4For example, if the ground-truth is  $[v_{1}]$  but the system extracts  $[v_{1},v_{2},v_{3}]$  , the extraction is considered as incorrect. 5https://keras.io/