Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E-Commerce Products (2024)

Abstract: Attribute value extraction involves identifying the value spans of predetermined attributes in product texts. This area of research has traditionally operated under a closed-world assumption, focusing on products from a static set of categories and their associated attributes. However, products in e-commerce stores are ever-increasing and evolving, calling for lifelong learning. If continuously trained on the fast-increasing products and attributes, most existing solutions not only struggle for parameter efficiency but also endure foreseeable defects due to data contamination, catastrophic forgetting, etc. As a remedy, we propose and study a new task, which aims to effectively maintain a strong single model for many domains in a lifelong learning fashion, without jeopardizing the model performance and parameter efficiency. We introduce factorization into the model and make it domain-aware by decoupling the modeling of product type and attribute, as a way to promote de-contamination and parameter efficiency while scaling up. Tuning the model with distillation prevents forgetting historical knowledge and enables continuous learning from emerging domains. Experiments on hundreds of domains showed that our model attains the near state-of-the-art performance with affordable parameter size, the least historical knowledge forgetting, and the greatest robustness against noises, whilst adding only a few parameters per domain when compared with competitive baselines.

heading: 1 Introduction
text: Attribute Value Extraction (AVE) (Nadeau and Sekine, 2007), similar to Named Entities Recognition (NER) (Chiticariu et al., 2010), aims to extract surface-form names (e.g. red/vanilla/plastic) of pre-defined product attributes (e.g. Color/Scent/Material) from product texts (e.g. product title/description). More specifically, we define each domain as the
combination of one product category/type (PT, e.g. Laundry_Detergent) and one attribute (Attr, e.g. Scent) – extracting scent values from laundry detergent products is an example of a domain.
Training one model per domain (One-for-Each) means we need to maintain tens of thousands of models, which is not scalable and does not fully leverage the correlation between multiple domains jointly. Therefore, training a single model for all domains (All-in-One) attracts intensive studies (Zheng et al., 2018; Xu et al., 2019; Karamanolakis et al., 2020; Wang et al., 2020; Yan et al., 2021; Yang et al., 2022). These works aim to train a scalable all-in-one model that absorbs stacked data from all domains in a closed-world fashion. They achieve decent performance while assuming a fixed list of pre-defined domains is available. However, products in the real world keep evolving in an openworld fashion. New attributes, new values, and new product types necessitate effective knowledge transfer and life-long adaptation, without the need to retrain the entire model from scratch.
Effective parameter sharing and parameter efficiency serve critical roles in such knowledge transfer and scalable adaptation across many domains. Intuitively, similar PTs can share the same attributes and values. For example, Laundry_Detergent and Hand_Sanitizer may both have cedarwood as the Scent value. The approach of fully shared parameters does not always result in the desired knowledge transfer and can sometimes negatively impact performance, especially when dealing with complex and evolving data. For example, different PTs possess drastically different values on the same attribute: Size for Shoes can be 7/7.5/8 and Size for Shirt takes values such as XS/S/M/L/XL. Also, the same value can be adopted by different domains: Orange can be a typical Flavor value for Drink products, while the same value is usually mentioned in Umbrella products as a Color. Although existing works (Xu et al., 2019;
8631
Yan et al., 2021; Yang et al., 2022) started to adopt various techniques to add additional inputs to make the model better aware of the domain it is handling and therefore alleviate domain contamination and facilitate knowledge transfer, we argue that because model parameters are essentially accessible to all domains, such fine-grained, domain-specific knowledge may find it hard to be stored organically in the form of loosely organized model parameters, exactly according to how the knowledge distributes and evolves.
For the first time, we study the open-world lifelong AVE task that requires model capability on both parameter efficiency for scale-up extraction and life-long extraction of evolving new data. We developed a unified solution, DALLA - DomainAware Life-Long Adapter. To alleviate the domain contamination without sacrificing parameter efficiency, we propose a factorized adapter to decouple the product type and the attribute by assigning them with smaller sub-matrices after performing low-rank decomposition on model weight matrices. Factorization not only facilitates parameter savings but, more importantly, decouples the PT/Attr aspects, allowing them to operate independently while maintaining the model’s awareness of the entire domain. Weight reconstruction only utilizes specific product type and attribute sub-matrices. This enables parameter sharing on a fine-grained (PT/Attr) level instead of shared across all domains, and strengthens domain-relevant knowledge transfers. Furthermore, we introduce a life-long distillation scheme based on the factorized adapter, to effectively boost underperforming or totally new domains while avoiding catastrophic forgetting of existing well-performing domains.
Our main contributions are summarized as: • To our knowledge, DALLA is the first work to
study the open-world life-long AVE task and the first time to introduce model parameter saving for scale-up and life-long attribute value extraction. • We propose the domain-aware factorized adapter module, along with its distillation strategy, to enable an Attribute Value Extraction (AVE) model that is scalable on vast and expanding data. The proposed model is robust against catastrophic forgetting, domain contamination, and noisy data, while also achieving performance on continually unseen data. • We conduct extensive experiments to show our superiority in parameter efficiency and stateof-the-art robustness performance on the open-
world AVE task compared with prevalent and latest baselines.

heading: 2 Domain-Aware Life-long Adapter
text: Problem formulation: We address the attribute value extraction (AVE) problem by regarding it as a sequential tagging problem, as a popular choice. Based on the BIO schema , AVE assigns a sequence of labels Y = [y1, ..., yNtoken ] to a given product profile sentence X = [x1, ..., xNtoken ] with Ntoken tokens, given a product type (PT) and an attribute (Attr). We take the combination of a product type and an attribute as a domain (PT,Attr). Given the domain, an AVE model should mark out all the entities in the product profile as attribute values. An entity is a span of tokens e = [xi, ...xj ](1 ≤ i ≤ j ≤ Ntoken). Specifically, the first token of an entity mentioned in the sentence is labeled as B, the other tokens inside that entity mention are labeled as I , and the non-entity tokens are labeled as O. Open-World Life-Long AVE Task: Previous works (Wang et al., 2020; Yan et al., 2021; Yang et al., 2022) show remarkable performances on closed-world data but fail to address the concern of life-long learning capabilities on the open-world data, which keeps evolving on either attribute value or domain set. Similar to most previous works, directly retraining with open-world data leads to catastrophic forgetting.
Unlike closed-world settings, for open-world life-long AVE task, we are inspired by (Ke and Liu, 2022) and seek to incrementally learn a model f with T sequential domains D = {Di}Ti=1: f : X ×D → Y , namely a sequential learning process (f(X,D1) → Y D1 , ..., f(X,DT ) → Y DT ). Each domain is regarded as a unit dataset absorbed by f for training and testing.
An ideal solution for the open-world, life-long AVE task requires: (1) Scalability: the model can provide cost-effective training on large-sized and number-increasing domains. We seek parameter efficiency by a domain-aware factorized adapter, depicted in Section 2.1. (2) Robustness: the model can overcome catastrophic forgetting and domain contamination during evolving training. We decouple sub-matrices for PTs and Attr for conditional parameter sharing and allow each PT/Attr sub-matrix to update independently to avoid contamination and propagation, detailed in Section 2.1. Furthermore, we distill knowledge through
the factorized sub-matrices for knowledge forgetting prevention and similar PTs/Attrs association, presented in Section 2.2. (3) Generalizability: the model is applicable to unseen data. We will use domain-aware sub-matrices to enable unseen domain extraction in Section 2.2.

heading: 2.1 Parameter Efficiency via Domain-Aware Factorized Adapter
text: ALBERT (Lan et al., 2019) is a compressed BERT (Devlin et al., 2018). ALBERT employs compression techniques, including (1) vocabulary embedding layer factorization; and (2) training one transformer encoder layer, and subsequently sharing the encoder parameters among the remaining layers. To compensate for the performance sacrifice of ALBERT after compression, we choose ALBERTxlarge as the basic model, wedged the proposed domain-aware factorized adapter, shown in the right part of Figure 2. Therefore, DALLA can retain ALBERT’s compression strategy as well as enjoy expressive contextual embedding compatible with BERT. Furthermore, to alleviate the model training burden, we follow the adapter tuning (Houlsby et al., 2019; Mahabadi et al., 2021) by freezing the original model while only tuning the adapter module.
Factorization for Parameter Saving: Weight factorization plays an important role in model compression (Panahi et al., 2021; Chen et al., 2021), which uses singular value decomposition (SVD) to compress the learned model weights. With SVD, the learned matrix is factorized into three matrices (U, S, V ), where S is usually a non-trainable identity matrix and absent during training. The multiplication of these smaller matrices will approximate the original one with fewer total parameters to achieve the model parameter saving. Given a matrix W ∈ RHi×Ho , we use the low-rank approximation via SVD: W ≈ UV ⊤, where U ∈ RHi×r,
V ∈ Rr×Ho , and r is the rank of matrix W , Hi and Ho are the input hidden embedding size and output hidden embedding size. The reduced size will be HiHo − (Hir + Hor). For all domains included in the model training dataset, we construct sub-matrices for all mentioned PT and Attr as UPT = {UPTi}Ni=1 and VAttr = {VAttrj}Mj=1, where N and M are the numbers of PTs and Attrs. For a single domain (PTi, Attrj), the reconstruction of W (shown in Figure 1), denoted as W ′ = UPTiV ⊤ Attrj
, plays the same role with W in the original network.
If the model is facing scale-up domains, we can maintain existing sub-matrices for well-performing domains or add new sub-matrices corresponding to the new domains. The creation of multiple PTs and Attrs is shown in Figure 1. We first construct and randomly initialize sub-matrices for all PTs (e.g. sofa, shoes, mugs) and Attrs (e.g. materials, size, color). When taking in the sample {“sofa”, “color”, profile}, the model only learns sub-matrix Usofa and Vcolor rather than the original parameter layer W . Moreover, the sub-matrices of the other PTs and Attrs are kept untouched until the model deals with samples associated with them. Factorized Adapter Integration: To strike a balance between the efforts invested in model training and the need for rapid adaptation to various tasks, researchers (Houlsby et al., 2019; Mahabadi et al., 2021) have proposed the integration of small modules, known as adapter layers, into a pre-trained model. These adapter layers serve as a middle ground between full fine-tuning and keeping the pre-trained weights fixed. The weights of the original network are untouched, whilst the new adapter layers are initialized at random. In adapter-tuning,
the parameters of the original network are frozen and therefore shared. This means that the total model size grows relatively slowly when more domains are added. As shown in the left part of Figure 2, the trainable parameters are illustrated in yellow. When faced with a given sample, only the factorized adapter consisting of the corresponding sub-matrix and layer norm of its PT and Attr is trainable.
As shown in Figure 2, we conduct SVD on the single adapter layer, W , and replace it with domainaware sub-matrices, (UPT ,VAttr) for domainaware W ′ restoration. UPT , and VAttr are a set of weights to squeeze domain-aware information through a small-size intermediate layer. As discussed in the previous works (Choi et al., 2020; Liu et al., 2021), layer norms in the pre-trained model store the mean and variance of the layer input, which can be utilized as a proxy to associate historical knowledge for life-long learning. To capture fine-grained association, we introduce the factorized norm layer sets NPT = {NPTi}Ni=1 and NAttr = {NAttrj}Mj=1 for UPT , and VAttr, respectively. NPTi is stacked right after UPTi to record the PTi’s input statistics, so as to NAttrj and VAttrj .

heading: 2.2 Life-Long Learning via Adaptive Distillation
text: We cast DALLA’s training strategy on open-world life-long AVE into a two-stage process, as shown
in Figure 3: (1) closed-world training stage, where we first feed the domain-aware factorized adapter with a number of seen domains that are pre-defined in a closed-world fashion; (2) open-world life-long training stage, where we conduct the adaptive distillation on unseen domains in an open-world fashion to mimic the evolving domains. To avoid catastrophic forgetting and benefit from similar PTs/Attrs associations, we adopt knowledge distillation based on the corresponding PTs/Attrs submatrix, which can further alleviate the cold start issue faced with new domains. The distillation allows parameter sharing conditioned on specific PT/Attr which prevents the domain contamination by only allowing relevant sub-matrix updates while leaving the remaining sub-matrices untouched.
We denote fθ̂ as the model learned in closedworld training stage by
Lt(D, X, Y ) = Entropy(Y, fθ̂(UPT ,VAttr)(X)), (1) where D = (PT,Attr). θ̂(UPT , VAttr) means parameter updates on PT and Attr sub-matrices related to the domain (PT,Attr). For the openworld life-long training stage, we initialize the teacher and the student model as fθtea = fθstu = fθ̂ = argminθ̂ Lt.
To better understand the association and distinction on both coarse-grained (domain) level and fine-grained (PT/Attr) level, the distillation loss
contains two parts: (1) domain feature transfer:
Lfea(X) = 1
N
N∑
n=1
−fθtea,n(X) log fθstu,n(X),
(2) where X ∈ X , containing N entities. f·,n(X) = f·(UPT ,VAttr),n(X) means the output of entity n given a domain (PT,Attr); (2) factorized norm layer transfer:
Lnorm(Σx,Σ′x) = (µx − µ′x)2 + σ2x
2σ′2x − log σx σ′x − 1 2 ,
(3) which is the Kullback-Leibler (KL) divergence of two Gaussian distributions to match the statistics Σ′ = (µ′, σ′) stored in the norm layers of the teacher and the statistics Σ = (µ, σ) computed for the student output at the same norm layers of the teacher, where x ∈ {NPT
⋃NAttr}. The updated student model is θ̂stu = argminθstu αLt+β1Lfea+ β2Lnorm, where α, β1, and β2 are coefficients.
When considering the new data for life-long learning, we first distinguish four different cases of unseen domains, as shown in the right part of Figure 3. We adopt different distillation schemes adaptively for each case: (1) (PTold, Attrold), a seen domain with new values of existing domains or a new combination of existing PT/Attr. Distillation associates the historical knowledge with new values, or help the model adapt to this new combination, through losses denoted in Eq. 2 and 3. (2) (PTnew, Attrnew), an unseen domain with a new PT and a new Attr. New PT and Attr submatrices are created and updated without overriding any already learned knowledge using the objective in Eq. 1. (3) (PTnew, Attrold), an unseen domain with a new PT and an existing Attr. Since we should create a new sub-matrix for the PT, we cannot directly distill the knowledge from the teacher but train the student with the objective denoted in Eq. 1. However, sharing the Attr sub-matrix can alleviate cold start problem even though we randomly initialize the new PT sub-matrix. (4) (PTold, Attrnew), an unseen domain with a new Attr and an existing PT. Similar to the case above, we employ parameter sharing conditioned on PT to expedite model optimization. The model only needs to put effort into learning the new Attr plus subtle updates on the PT sub-matrix.

heading: 3 Experimental Settings
text: Dataset and Baselines Details about dataset are interpreted in Appendix A. We compare our model with two groups of baselines, including the All-inOne and the One-for-Each. All baselines and the implementation details are presented in Appendix C and Appendix B. We additionally discuss the parameter saving estimation after implementing SVD on the different model components in Appendix D.

heading: 4 Evaluation of Parameter Efficiency
text: Model Effectiveness and Efficiency We evaluate the model parameter efficiency in Table 1 via discussion on three comparisons: (1) Different backbone model (ALBERT, BERT, ALBERTxlarge) comparison necessitates the trade-off between parameter saving and decent performance; (2) Whether and where to wedge the factorized layer (BERT vs. BERTft and ALBERT vs. ALBERTft♯ , BERTft vs. ALBERTft♯) raise the concern that the position for SVD indeed affects the parameter efficiency because single factorized adapter layer in BERTft surpasses BERT, ALBERT and ALBERTft♯ ; (3) Scale-up techniques comparison (Adapter in BERTft vs. Attention in SUOpenTag) reveals the superiority of factorized adapter. From all of the above comparisons, we observe that techniques employed in DALLA(ALBERTxlarge backbone model with single factorized adapter layer) address both the trade-off concern and scale-up advantage effectively.
From Table 1, DALLA attains almost the best on most of the evaluation metrics, especially for the Marcro-F1 and Micro-F1 with great margin against
the second best to be 1.6% and 1.3%. Apart from showing superiority among the All-in-One group, our model exhibits competitive performances to the two methods from the One-for-Each group, which are regarded as the upper bounds. Compared with BERT and ALBERTxlarge, better achievements in DALLA and the BERTft indicate the benefits from factorization to decouple PT and Attr allowing conditional parameter sharing. Therefore, the association and distinction among domains can be well captured and uncovered through the update or distillation of the PT-specific/Attr-specific submatrices. The observed increase in parameters of ALBERTft♯ is an intentional outcome, as excessive use of factorization can indeed lead to issues. As depicted in Table 6, when factorization is applied to Query, Key, Value, Dense, and FFN components in the transformer encoder, the parameter size actually expands rather than contracts.
Figure 4 reflects the trade-off between parameter efficiency (performance over per trainable parameter) and performance achievements across different model variations. All models in One-for-Each are located in the bottom-right corner, which implies they are good at achieving decent performance since model training is tailored for each domain. In the group of All-in-One, ALBERT seems to have a great advantage over the others in parameter efficiency due to its model compression strategy. However, it sacrifices too much performance (the worst F1 in Table 1). DALLA and BERTft are the first and the second best models obtaining the trade-off between performance and parameter efficiency in Figure 4. They all are qualified to be scale-up models. But under the setting of openworld life-long AVE, evolving data challenges the stability of most models in the All-in-One, because of a series of issues including weight overriding, catastrophic forgetting, and domain contamination when a single model is shared by all domains.

heading: 5 Evaluation of Life-Long Adaptation
text: Alleviating Catastrophic Forgetting We first study DALLA’s capability of overcoming catastrophic forgetting and domain contamination. To evaluate catastrophic forgetting alleviation, we split the domains into seen and unseen domains. Initially, we train the model on the seen domains during the closed-world training stage. Subsequently, we proceed with open-world training by conducting n=3 consecutive training sessions on expanding
sets of disjoint unseen domains. We then record the performance drops on seen domains after each set of open-world training sessions. There are two settings in the open-world training, one is when no additional seen domain data is available (NR = 0%, no additional seen samples; unseen data only). We aim to use this setting to test the direct knowledge transfer capability to unseen domains. The other setting also includes new values from seen domains (NR = 10%, 10% additional seen samples). We aim to use this setting to test if the model can be assertive in learning new knowledge while not forgetting the knowledge on the existing seen domains.
From the left part of Table 2, DALLA shows its superiority in controlling the forgetting score after several runs on unseen domains with the lowest scores in average forgetting. Although BERTft shows comparable performance with DALLA in terms of parameter efficiency, BERTft along with other baselines suffer from severe catastrophic forgetting because a great portion of parameters is shared. When we increasingly add new domains, DALLA still maintains decent performance on seen domains, which shows its advantage in overcoming domain contamination, compared with the others. When comparing the performance of models with NR = 0% and NR = 10%, it is important to note that for the latter, all models tackle the forgetting issue by retrieving their knowledge from seen domain samples. However, DALLA surpasses the other models and achieves the best performance in terms of preserving performance. Handling Various Unseen Cases We evaluate the model’s life-long adaptation ability on unseen domains, as shown in the right part of Table 2. Based on the four types of unseen domains, discussed
in Figure 3, we present two groups of the results: direct transfer to the unseen (NR = 0%) and transfer on both unseen domains and new values of seen domains (NR = 10%). For the first group results, DALLA shows the second-best performance among other baselines, because MAVEQA, originated from where the dataset comes, achieves nearly 99% on all attributes as reported in the paper (Yang et al., 2022). We discussed more details about MAVEQA in the Appendix C.
The second group exhibits not only the macrof1 scores but also the performance changes considering the effect of adding in a small portion of seen domain examples. DALLA improves the performance while the other baselines suffer from decreased performance in the case of (PTold, Attrold). This suggests that the other baselines suffer from contamination from the seen domain samples, attempting to maintain performance, let alone strengthen desired knowledge transfer to enhance the performance of the domains that consist of existing PTs and Attrs. We see BERT and MAVEQA loss score in the case of (PTnew, Attrnew). The others use their proposed domain-aware component to build specific knowledge for the totally new domains, while they directly concatenate PT and Attr with the model input is insufficient to capture the current domain well. Compared with BERTft and SUOpenTag, DALLA still shows the superiority of domain-aware ability due to independent updates on PT/Attr sub-matrices. In the remaining two cases, (PTnew, Attrold), and (PTold, Attrnew), we find learning on new PTs with existing Attrs is
harder than new Attrs with existing PTs according to smaller gains and performance drop shown in the case of (PTnew, Attrold). In fact, the value form for the same Attr among different PTs can be dramatically different. e.g. Size for Shoes should be numerical like 7/7.5/8, while Size for Shirt takes values in characters such as XS/S/M/L/XL. Moreover, we observe from the data that many attributes can be shared by tens to hundreds of product types but much sparser for product types to attributes, which implies learning well on Attr sub-matrices is harder than that on PT sub-matrices. If the Attr sub-matrix is underfitting, directly adopting it as initialization still requires huge efforts in model training instead of providing a beneficial warm-up.

heading: 6 Analysis
text: Robustness to Noises We further study a problem on model robustness to noisy training values in Table 3. We conduct noise manipulation on the dataset, including randomly removing values and sliding BIOE tags (with a window size of 1) on both 10% seen and 10% unseen data. Compared with SUOpenTag on both seen, unseen, and overall domains, DALLA shows 0.9% - 1.91% lower performance drop, even double efforts in minimizing the error propagation. Sub-Matrix Visualization We visualize PT/Attr sub-matrices to uncover the associations among PTs/Attrs. Note that those sub-matrices start with randomly initialized parameters and are learned in a purely data-driven fashion, rather than being assigned manually or under any assistance of their PT/Attr name semantics. From Figure 5(top), the
clusters marked in the green circle indicate welllearned attribute type semantics (e.g. "Materials", "Pan Materials", and "Core Materials"), via learning to extract their values. In Figure 5(bottom), we find that similar PTs are in proximity to each other (e.g. a cluster of food PTs in the lower right corner), which is very encouraging given no explicit prior knowledge of PT name semantics. We also observe that some irrelevant PTs in the red circle are close to each other (e.g. "Baby and Toddler Outfits" and "Drawing and Painting Kits"), mostly due to random initialization and lack of proper training on those PTs when very few data points are available. We expect more discriminative PT embeddings could be obtained with more incoming data, given our life-long learning setting.

heading: 6.1 Ablation study on adaptive distillation
text: In this section, we discuss the importance of distillation and what should be distilled by ablation study. As shown in Table 4, we test the distillation ability by measuring forgetting that occurs on average in the seen domains after we distill differ-
ent parts (norm layer, sentence feature embedding, encoder logits, and the weight sub-matrices) to distill. The impact of distilled knowledge from feature embedding and norm layer on the performance is outstanding among all alternatives, which indicates their importance in incremental open-world AVE. Therefore, we test the effectiveness of their combination and report the best score in the last row of Table 4.

heading: 6.2 Error Analysis
text: After ranking all domains’ performance, we observe two major error types. First, since training samples are imbalanced among domains, some domains consist of a limited number of samples, resulting in their relevant PT and Attr sub-matrices being inadequately learned. These domains usually suffer overwriting by other predominant domains. Second, attribute complexity affects the performance, because some attributes participate in multiple domains, such as "size" and "type". Since we maintain a uniform sub-matrix size for each attribute, those attributes with diverse surface names across different PTs present challenges. Learning a versatile Attr sub-matrix capable of accommodating all variations of surface names proves to be difficult, particularly for attributes like "size" and "type".
We conduct experiments to show even suffering errors DALLA still can control them compared with other baselines, as shown in Table 5 and Table 6, DALLA exhibit its superiority with lowest performance drop when handling rare or complex domains on open-world scenarios.

heading: 7 Related Work
text: Attribute Value Extraction Sequence tagging is one of the popular choices for attribute value extraction, where existing works are mostly improving scalability and effectiveness when faced with large-volume data. SUOpenTag (Xu et al., 2019) employs BERT as the encoder to capture profile embedding and attributes. TXtract (Karamanolakis et al., 2020) encodes the hierarchical embedding of product type for pt-specific extraction. AdaTag (Yan et al., 2021) proposes a mixture of expert and hypernetwork to capture the attribute-aware features. AVEQA (Wang et al., 2020) regards AVE as a question-answering task extracting the answer span. MAVE (Yang et al., 2022) devises the globallocal connection to enable the model to be aware of different parts of the profile. Shinzato (Shinzato et al., 2022) designs to expand queries for effectively improving rare attributes’ performance. Even though these models can handle huge data, it is still underneath the iceberg to maintain model stability and avoid catastrophic forgetting when faced with incremental data in an open-world fashion. Adapter for Parameter Efficiency Incorporating adapter layers into neural networks (Houlsby et al., 2019; Mahabadi et al., 2021; Hu et al., 2021) has garnered significant attention, particularly in the era of large language models (Touvron et al., 2023; Chowdhery et al., 2023; Ye et al., 2023). As the size of model parameters experiences exponential growth with the evolution of language models, employing an over-parameterized neural network for downstream tasks necessitates leveraging advantageous low-rank properties during the training pro-
cess to facilitate rapid adaptation. The low-rank factorized adapter can save trainable parameter size to speed up model training. We further adjust the usage of the low-rank matrix by implementing taskspecific semantic associations (PTs and Attrs) to scale up model capacity and foster life-long adaptability.

heading: 8 Conclusion
text: In this paper, we study the open-world life-long attribute value extraction task and devise a factorized life-long adapter module to enable model scalability and robustness. Factorized adapter decouples product type and attribute, allowing domain interaction on either PT-level or Attr-level. Therefore, the extraction model is domain-aware but updated independently for de-contamination. Moreover, we endow the model with life-long adaptation ability by distilling the decoupled information to avoid catastrophic forgetting. DALLA can achieve decent performance on the benchmark with the ability to control forgetting and error propagation and only consume affordable parameters.

heading: 9 Limitations
text: After decoupling PT and Attr, we can obtain domain associations through parameter sharing conditioned on PT/Attr to warm up model training on unseen domains that have associated PTs/Attrs. To learn solid associations, we expect each sub-matrix to be well learned by sufficient data points. Otherwise, the knowledge transfer and performance on unseen domains with associated PTs/Attrs towards worse.
Furthermore, even if the model is unaware of the specific names of PTs (Product Types) and Attrs (Attributes), DALLA is capable of capturing the relationships inside PTs and Attrs. To alleviate the learning burden on the model, in future work, we investigate a hypernetwork approach consistent with lifelong learning. This approach encodes the semantic meaning of PTs and Attrs into the creation of sub-matrices, enabling the model to leverage this encoded information during open-world life-long training.

heading: 10 Acknowledgement
text: We thank the anonymous reviewers for their constructive feedback which we incorporated in the final version of this manuscript. This work is supported in part by NSF under grant III-2106758.

heading: A Datasets
text: We evaluate our model on the public Multi-source Attribute Value Extraction (MAVE) dataset (Yang et al., 2022), which is created from raw values from Amazon Review pages (Ni et al., 2019). MAVE contains over 2.2 million products distributed into 1257 different product categories, with 3 million attribute-value annotations. We further filter domains with a frequency above 100, resulting in a dataset of 997 domains encompassing 448 PTs and 365 Attrs.
B Implementation Details
We split the dataset by separating samples under each domain into train:eval:test = 8:1:1,(The ratio is referred from MAVE (Yang et al., 2022)) We conduct experiments with two-stage trainers: closed-world training on only seen domains and open-world training on either only unseen domains or unseen+seen domains. For Table 1, we split the data into 500/400 seen/unseen domains. We make a split on 500 seen domain data with a ratio 9:1. 90% samples are used for closed-world training while 10% samples are used in open-world training as the new value cases of the domain type, (PTold, Attrold). 400 unseen domains comprise the training samples of the other three new domain types. The evaluation is performaned on all test samples of each domain.
Slightly different in Table 2, we only have 400 seen domains to ensure sufficient unseen domains participating in the three incremental trainings (from 100 to 300 unseen domains) at the openworld stage. And we also set apart 10% of seen domains for NR = 10%. Another difference is the
testing set, containing only the seen domain testing samples to evaluate forgetting scores.
We experimentally use the ALBERTxlarge as the backbone. We optimize using SparseAdam (Makhzani and Frey, 2013) to enable a single submatrix update whose learning rate is increased linearly over the first 10% of the steps and then decayed linearly to zero. All runs are conducted on 4 V100s with a batch size of 32. For each dataset and algorithm, we set the hyperparameters (α = 0.6, β1 = 0.4, β2 = 0.3) according to the best model according to Macro-F1 on the validation set. The sub-matrix hidden size is r = 32. Because, after evaluating a list value of r, 32 is the one that balances between efficient training and SOTA performance achievements. All the results are the average score after 5 runs.

heading: C Baselines
text: We compare our model with two groups of baselines. If the model trains multiple times each time targeting a single domain, we group these methods into the One-for-Each group. Otherwise, if trained on all domains at once, the models fall into the All-in-One group.
The One-for-Each training strategy is applied to the following: (1) BERTAda, an implementation of the work in Houlsby et al. (2019). (2) ALBERTAda, similar implementation of BERTAda with the modification that we wedge the adapter into the single encoder for replication in ALBERT (Lan et al., 2019).
The All-in-One group includes: (1) ALBERT: a compressed BERT (Devlin et al., 2018) taking in the concatenation of domain names and the product text profile. (2) BERT: vanilla baselines varying in model size, fed with the concatenation of domain and profile. Empirically, we concatenate the last two layers as the final features for classification. (3) ALBERTft♯ : we apply the weight factorization on all key components (Query, Key, Value, Dense, Feed-forward layers) forming the transformer encoder. During the life-long training, we only employ Lt and Lfea and remove Lnorm considering the model training expense. (4) BERTft: we conduct the factorization on the linear layer above BERT and before the label classification layer. The input to this linear layer is the concatenation of the last two feature layers in BERT. This ensures the same backbone setting with the baseline BERT and provides a comparable size feature representa-
tion with DALLA, which utilizes ALBERTxlarge with a hidden size of 2048. (5) SUOpenTag (Xu et al., 2019): enable the model to be aware of the attention of attributes to the profile. The model shows its scalability on the large-volume dataset with well-learned attention between attributes and values’ context. (6) MAVE (Yang et al., 2022): a multi-source attribute value extraction model via question answering, modeling the products with structure and long profiles. Compared with its preliminary work AVEQA (Wang et al., 2020), it addresses mutual attention of the local separate parts (product type, attribute, title, profile) in the long text. However, MAVE dataset is constructed by AVEQA and annotation rules. That is why the evaluations of AVEQA and MAVEQA on MAVE are in high scores approaching 100% as reported, and we only adopt it as a baseline when evaluating catastrophic forgetting rather than performance on the extraction task.
Our model belongs to the All-in-One group. The comparison with these two groups of baselines is to verify the effectiveness of model de-contamination when performances are approaching the One-forEach model, and model life-long learning ability with high parameter efficiency when superiority is exhibited among All-in-One models.

heading: D Parameter Saving
text: We describe the parameter-saving computation details. Each adapter layer has two sets of projection matrices. One is UPT ∈ RN×Hi×r, the other is VAttr ∈ RM×r×Ho . N and M are the number of PTs and Attrs included for training. 2r + 2Ho parameters are used for factorized norm layer NPT and NAttr. The total number of parameters for factorized Adapters across training domains is N ·(Hi ·r+2r)+(r ·Ho+2Ho) ·M , which scales linearly with the hidden layer size r times the number of PTs or Attrs. However during each run for a specific domain, the trainable parameter of the factorized adapter is 1·(Hi ·r+2r)+(r ·Ho+2Ho)·1. The rest are unavailable for gradient modification, which contributes to the training efficiency of DALLA.
We provide an example of how to save model size through factorization in Figure 6. We select 10 PTs and 10 Attrs, which can be combined to form a minimum of 10 and a maximum of 100 domains. We conduct SVD on different model components, including the linear layer, transformer encoder, and adapter. All of these components are integrated into a backbone model based on cased BERT, which consists of 12 layers and a hidden size of 768. After integrating these components, we conduct model
training using three different strategies: All-in-One, One-for-Each, and Domain-aware. If we apply SVD on a linear layer or a transformer encoder, the domain-aware model, which implements the parameter factorization, results in significant parameter reduction. This reduction in parameters is considerably less than that observed in the One-forEach model and approaches the parameter count of the All-in-One model. Excessive factorization can have a detrimental impact on performance, as evidenced by the degradation of parameter savings when applying SVD to all key components in an encoder, compared to applying it solely to a linear layer. Table 1 reveals that ALBERTft exhibits the poorest performance, whereas BERTft ranks as the second-best performer. Factorized Adapter demonstrates a nearly equivalent parameter size to that of the linear layer after implementing SVD. However, it excels in overcoming the challenge of catastrophic forgetting, as evidenced by the results presented in Table 2.

heading: Future Work
text: To alleviate the learning burden on the model, in future work, we investigate a hypernetwork approach consistent with lifelong learning. This approach encodes the semantic meaning of PTs and Attrs into the creation of sub-matrices, enabling the model to leverage this encoded information during open-world life-long training