COMAVE: Contrastive Pre-training with Multi-scale Masking for Attribute Value Extraction
Year: 2023

Abstract:
Attribute Value Extraction (AVE) aims to automatically obtain attribute value pairs from product descriptions to aid e-commerce. Despite the progressive performance of existing approaches in e-commerce platforms, they still suffer from two challenges: 1) difficulty in identifying values at different scales simultaneously; 2) easy confusion by some highly similar fine-grained attributes. This paper proposes a pre-training technique for AVE to address these issues. In particular, we first improve the conventional token-level masking strategy, guiding the language model to understand multi-scale values by recovering spans at the phrase and sentence level. Second, we apply clustering to build a challenging negative set for each example and design a pre-training objective based on contrastive learning to force the model to discriminate similar attributes. Comprehensive experiments show that our solution provides a significant improvement over traditional pretrained models in the AVE task, and achieves state-of-the-art on four benchmarks1.

heading: 1 Introduction
text: Product features are crucial components of ecommerce platforms and are widely used in applications such as product recommendation (Cao et al., 2018), product retrieval (Magnani et al., 2019), and product question answering (Yih et al., 2015; Chen et al., 2021b). Each product feature typically consists of an attribute and one or more values, providing detailed product descriptions to help customers make purchasing decisions. In recent years, Attribute Value Extraction (AVE) (Xu et al., 2019; Zhu et al., 2020; Yan et al., 2021) methods have received increasing attention because they can automatically extract product features from a massive amount of unstructured product text, with impressive results in e-commerce platforms, such as Amazon, AliExpress, and JD.
1https://github.com/ygxw0909/CoMave
However, as e-commerce grows, some emerging domains, such as finance, insurance, and healthcare, bring two new challenges: a) Multi-scale values. Unlike normal products (e.g., clothing) with only short values (e.g., color: red), insurance products can have a value of a longer phrase or even multiple sentences. For example, the value of attribute renewal rule in Figure 1 contains more than 25 words (in green), rendering it impractical to retrieve them using related techniques such as Name Entity Recognition (NER) (Li et al., 2020; Yang et al., 2021). b) Fine-grained divisions of attributes. Compared with the coarse division of attributes in traditional e-commerce (e.g., color, size, and material), the division in insurance products is more refined, resulting in different attributes often having similar types. For instance, in the insurance clauses in Figure 1, maximum insurance age and maximum renewal age are both ages, and grace period and hesitation period are both periods. This fine-grained division makes the distinction be-
6007
tween the different attributes subtle, thus increasing the difficulty to distinguish between them.
Although recent pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and ROBERTA (Liu et al., 2019) achieve tremendous success on a spectrum of NLP tasks, including AVE, we argue that they are not sufficient for the challenges above. First, the conventional Masking Language Model (MLM) focuses on token-level recovery and does not consider multi-scale values. Second, there is still a gap between the unsupervised general objectives and the downstream AVE in terms of task form, such that the model cannot benefit from pre-training when retrieving attributes, let alone distinguishing between fine-grained similar attributes..
In this paper, we propose COMAVE, a novel PLM for AVE tasks. Relying on the large-scale corpus of triples ⟨text, attribute, value⟩ collected by distant supervision, we propose three pre-training objectives to address the challenges: a) MultiScale Masked Language Model (MSMLM). We extend token-level recovery to the phrase as well as the sentence level, using different masking mechanisms to force the model to perceive spans of various lengths, thus providing a basis for identifying values at different scales. b) Contrastive Attribute Retrieval (CAR). To adapt the model to the fine-grained division of attributes, we require it to retrieve the correct attributes from a challenging candidate set of semantically similar attributes. The candidates are mainly collected by clustering and a contrastive loss is designed to help the model perceive the subtle differences between them. c) Value Detection (VD). To close the gap between pre-training and downstream AVE and further enhance the model’s perception of values extraction, we let the model recognize all values without considering the corresponding attribute. To fully evaluate our pre-trained COMAVE, we construct a new challenging benchmark INS. It consists of financial and medical texts from real scenarios and the corresponding manual annotations and is full of the two challenges we mentioned. Comprehensive experiments on four AVE datasets including INS demonstrate that, equipped with only a simple fine-tuning output layer, our COMAVE not only achieves state-of-the-art results on the hardest INS but also outperforms all the compared methods on existing benchmarks.
Our contributions are summarized as follows:
• We release an advanced pre-trained language model, namely COMAVE, for solving common challenges in AVE tasks. To the best of our knowledge, this is the first pre-training model aimed at AVE tasks.
• We propose three novel pre-training objectives: Multi-Scale MLM allows the model to adapt to values span of different scales, CAR uses contrastive loss to force the model to perceive subtle differences in similar attributes, and VD bridges the gap between pre-training and downstream tasks.
• Our method obtains state-of-the-art results on four AVE benchmarks, achieving significant improvements compared to existing PLMs.

heading: 2 Preliminaries
text: Given a natural language text T and a set of candidate attributes set A = {a1, a2, ..., a|A|}, where ai is an attribute, the goal of AVE is to extract a set Y = {(a∗1,V1), ..., (a∗n,Vn)}, where a∗i ∈ A and Vi is the set of values belonging to a∗i . For simplicity, each value v ∈ Vi is defined as a span of T . In general, T is collected from a large number of product-related documents or other data sources, and A is a collection of attributes for various products in different categories.
Note that although formally AVE is similar to NER, the two still have significant differences, as we mentioned in section 1. First, the division of attributes is more fine-grained than the division of entity types (e.g., location and person). Second, the scale of entities is generally shorter, while that of values varies from token level to sentence level. Therefore, conventional NER methods are difficult to directly port to AVE tasks.

heading: 3 Methodology

heading: 3.1 Pre-training Corpus Construction
text: The pre-training procedure of COMAVE requires a large-scale corpus C = {(Ti,Ai,Yi)}M containing tens of millions of data. Manual annotation of such a large corpus is obviously impractical, thus we designed an automatic method to construct C. In brief, we first collect the triples ⟨subject, predicate, object⟩ from several existing open-domain knowledge graphs, including DBpedia (Lehmann et al., 2015), Yago (Tanon et al., 2020), WikiData (Vrandecic and Krötzsch, 2014), and OpenKG (Chen et al., 2021a). Then, we regard
each predicate and object as the attribute ai and the value vi, respectively, thereby building a seed set {(ai, Vi)}N by aligning and merging the attributes. Finally, we use this set as a distant supervision to mine the corresponding texts from the web data, thus building the pre-training corpus.

heading: 3.2 Pre-training COMAVE
text: Since ROBERTA (Liu et al., 2019) has been shown to be promising and robust on multiple NLP tasks, we use it to initialize COMAVE. We then further pre-train COMAVE on our corpus C. As shown in Figure 2, we flatten each pair (T ,A) into a sequence X with the 
 token,

, x1, x2, ..., xn, 
, 
, x a 1,1, x a 1,2, xa1,|a1|, 
, x a 2,1, ..., x a m,|am|, 
, (1)
then COMAVE converts the each token of X into a semantic vector,
h
,hT1 ,h T 2 , ...,h T n ,h 
,h
,ha1,1,h a 1,2, ha1,|a1|,h 
,ha2,1, ...,h a m,|am|,h

, (2)
where hTi ∈ Rd and haj,k ∈ Rd denote the vector of xi and xaj,k, respectively, and h

 ∈ Rd is regarded as the global semantic vector of X . Considering the above challenges, we design three objectives to pre-train COMAVE as follows.

heading: Multi-Scale Masked Language Model
text: The most common objective of pre-training is to employ MLM to guide the model to perform extensively. Unlike BERT or ROBERTA which focuses on token-level recovery, we prefer COMAVE to be aware of various values, regardless of their scales. Consequently, we design two parallel masking mechanisms, namely phrase-level and sentencelevel masking. During pre-training, each T is performed by only one of the two mechanisms, and the probabilities are set as ρ and 1-ρ, respectively. Furthermore, we empirically find that an appropriate masking percentage is a prerequisite for MLM to be effective. We denote this budget percentage by µp and µs, respectively, and try to make the masking result close to it for both mechanisms.
In phrase-level masking, we are inspired by SpanBERT (Joshi et al., 2020) and randomly mask a short span of tokens for each selected T until the budget µp is spent. The probability distribution of the masking length, denoted by l ∈ [1, lmax], is:
Pphrase(l) = σ−l + γ
∑lmax l′=1 σ−l ′ + γ , (3)
where both σ and γ are hyper-parameters. This distribution ensures that the masking probability of each span decreases smoothly as its length in-
creases, while also preventing long spans from being rarely selected. Note that we make sure that each masked span is formed by complete words.
In sentence-level masking, we mask only one sentence for each selected T because recovering a sentence requires sufficient context. In this way, it is more difficult to make the total number of masked tokens approach µs compared to masked phrases since the length of different sentences can vary significantly. To achieve this goal, we propose a simple but effective strategy to dynamically control the masking probability of each sentence. Specifically, assuming that the current sentence masking rate of µc = ∑ |Tmask|∑ l(T ) , where Tmask is the tokens that has been masked. If µc < µs, it means that the current masking rate is less than the standard value, so we should pay more attention to longer sentences Slong = {s|l(s) > l(T ) ∗ µs}, giving higher masking probabilities. Otherwise, we should focus on short ones Sshort = {s|l(s) ≤ l(T ) ∗ µs}.
Following BERT (Devlin et al., 2019), we replace 80% of the masked tokens with <mask>, 10% with the random tokens in the corpus, and leave the remaining 10% unchanged.

heading: Contrastive Attribute Retrieval
text: We expect to adapt COMAVE to the subtle differences between attributes in the pre-training phase. To this end, for each training text T and its ground truth attributes A+, a challenging negative set A− = A−c ∪ A−g is built to confuse the model. Here, each ac ∈ A−c is sampled using clustering to guarantee it is highly similar to A+ (see below for details), and each ag ∈ A−g is a random one from the total attribute pool to maintain the diversity of the negative examples. If T has no ground truth, then A− = A−g . During pre-training, COMAVE is required to retrieve each correct attribute a+ ∈ A+ by scoring all a ∈ A+ ∪ A− with
P(T , a) = sigmoid(h
 ∗W CAR), (4)
where W CAR ∈ Rd∗η is a trainable parameter and η denotes the maximum of |A|.
To make the score of A+ higher than that of each negative example, i.e., ∀a− ∈ A−, P(T , a+) > P(T , a−), where a+ ∈ A+. Inspired by (Khosla et al., 2020), we define a Margin Ranking Loss to better leverage contrastive learning and strengthen
the distinction between fine-grained attributes,
LCRA = |A|∑
i=1
|A|∑
j=i+1
(1− z) ∗ |Pi − Pj |+
z ∗max(0, λ− |Pi − Pj |), (5)
where Pi is short for P(T , ai), and λ is the margin. If both ai and aj are positive or negative examples, z = 0, otherwise z = 1.
The key to this training objective is how to collect A−c that is highly similar to A+. Clustering has been proven to have a natural advantage in retrieving similar instances, so we used the widely used K-medoids (Park and Jun, 2009) clustering method to construct A−c . Concretely, the distance between two attributes is
d(ai, aj) = ω∗d̃(ai, aj)+(1−ω)∗d̃(Vi,Vj), (6)
d̃(zi, zj) = τ(ft(zi, zj)) + τ(fs(zi, zj)), (7)
where ft and fs denote Levenshtein distance and Euclidean metric, respectively. z ∈ Rd is the ROBERTA (Liu et al., 2019) pooling vector of z. τ denotes the score normalization to ensure balance. The distance considers both the literal and semantic features of the attributes and associated values.

heading: Value Detection
text: To further cross the gap between the pre-training and downstream AVE tasks, we also add a training objective of detecting values. For (T ,A), wherein each positive attribute a+i ∈ A+ corresponds to one or more extractable values Vi = {v1, v2, ..., vn} in T . The model needs to classify each token xi ∈ T , according to whether it is a part values of V:
P (xi|T ,A) = softmax(hTi ·W VD), (8)
where W VD ∈ Rd is trainable parameter. We define “V” and “O” as labels to represent that xi ∈ V and xi /∈ V , respectively. Note that each token does not need to be classified to the exactly belonged attribute.

heading: 3.3 Fine-tuning
text: To fully evaluate the effectiveness of our pretraining for downstream tasks, we add the following two output layers to fine-tune our COMAVE, respectively.

heading: Sequence Tagging Layer
text: In this setting, T and all candidate attributes A are first fed to COMAVE, as in the pre-training. Then, according to the output hT , a Conditional Random Field (CRF) generates a sequence Y = {y1, y2, ..., yn}. Here n is the length of T and each yi ∈ ⋃|A| k=1{Bk, Ik, O} is a tag indicating whether the token xi ∈ T is the beginning (Bk), inside (Ik) and outside (Ok) of a value in the attribute ak ∈ A.

heading: Machine Reading Comprehension Layer
text: In this case, COMAVE takes each (T , ai) as input and predicts the span of target values belonging to ai ∈ A in T . Here, we follow a representative work (Li et al., 2020) that consists of two steps. First, the candidate start and end indexes of the span are predicted using the binary classification of each token separately. Subsequently, a matching score is performed for each candidate index pair of start and end. Finally, the pairs with scores above the threshold are retained as the results.

heading: 4 Experiments

heading: Datasets
text: To comprehensively evaluate our method, we used the following four datasets covering both English and Chinese: 1) INS is a Chinese AVE dataset which is collected from the real product data of Alipay2 platform. It contains various types of large-scale insurance products from real scenarios, including wealth insurance, health insurance, travel insurance, life insurance, etc. From each product document, the attributes and values are manually annotated. There are 29 global attributes and the samples are divided into 9112/1138/1138 for Train/Val/Test, respectively. Table 1 gives several groups of similar attributes and the number of their corresponding examples. Table 2 shows the distribution of different value scales. They reveal that the two challenges we focus on are prevalent in INS. 2) MEPAVE (Zhu et al., 2020) is a Chinese AVE dataset with examples from the JD e-commerce platform 3, containing 26 global attributes and 87,194 samples. Most of the text is mainly from the product titles. We randomly divided the dataset into three parts of Train/Val/Test in the ratio of 8:1:1 according to (Zhu et al., 2020) for experiments. 3) AE-Pub (Xu et al., 2019) is an English AVE dataset with 110,484 samples and
2https://www.alipay.com/ 3https://www.jd.com
over 2400 attributes obtained from AliExpress4. In order to make a fair comparison with previous models that could not handle a large number of attributes, we selected 4 frequent attributes (i.e. BrandName, Material, Color, Category) and divided the relevant instances randomly by 7:1:2, referring to the dataset publisher. 4) MAE (IV et al., 2017) is an English multi-modal AVE dataset that contains 200 million samples and 2000 attributes. Following (Zhu et al., 2020), we built an MAEtext dataset to focus on the textual modal. Same as AE-Pub, we also selected the 20 most frequent attributes from Train/Val/Test sets.

heading: Evaluation Metrics
text: In most experiments, we used Mirco-F1 scores as the main evaluation metric. We followed the criterion of exact matching, where the complete sequence of predicted attributes and extracted values must be correct. Accuracy was also used as another evaluation in the detailed analysis.

heading: Methods for Comparison
text: We compared the proposed method with notable AVE methods, including BiLSTM+CRF (Ma and Hovy, 2016), OpenTag (Zheng et al., 2018),
4https://www.aliexpress.com
SUOpenTag (Xu et al., 2019), JAVE (Zhu et al., 2020), AVEQA (Wang et al., 2020), and UIE (Lu et al., 2022). In addition, the existing representative PLMs are involved in comparison to showing the improvement of our PLM on the AVE task, including BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), SpanBERT (Joshi et al., 2020), MacBERT (Cui et al., 2020), and ELECTRA (Clark et al., 2020).

heading: Implementation Details
text: Our method ran on Tesla A100 GPUs. All the pretrained models used in our experiments were large versions by default. Chinese and English versions of COMAVE were pre-trained respectively for evaluation in two languages. The hyper-parameters in pre-training were set as follows: (1) The batch size and the learning rate were set to 256 and 1e-5. (2) In the CAR task, η, λ, and ω were set to 12, 2, and 0.4, respectively. The ratio for A+, A−g , and A−c was 1:1:1 (3) In the MSMLM task, ρ, σ, γ, ℓmax, µp, and µs were separately set to 0.2, 1.20, 2e-4, 20, 15%, 10%. In phrase-level masking, ℓmax was set to 20, and ℓmean was approximately equal to 5.87. In the fine-tuning stage, the batch size and the learning rate were set to 80 and 2e-5, respectively.

heading: 4.1 Overall Results

heading: Comparison with AVE Baselines
text: We first compared with the baselines. To ensure fairness in the number of parameters, we replaced BERT-Base with ROBERTA-Large in the evaluations of Chinese datasets, and the distilled context layer of AVEQA is also replaced by ROBERTALarge in all evaluations. The results are shown in Table 3. Our proposed COMAVE equipped with the MRC layer achieves state-of-the-art on all four benchmarks. Most baselines perform poorly on INS because they focus on traditional e-commerce
products and cannot handle the two challenges mentioned in Section 1. Unlike them, UIE achieves competitive results, especially on Attr, as it is a generic approach pre-trained by multiple information extraction tasks. However, limited by its weak multi-scale value extraction capability, it still cannot handle INS. The performance on Attr and Val demonstrates that our method brings significant improvements in both attribute retrieval and value extraction, thus outperforming all baselines on Over. Benefiting from the pre-training, our COMAVE outperforms all the baselines by adding only a simple fine-tuning output layer.

heading: Comparison with PLMs
text: To further evaluate the contribution of our pretraining methods, we compared COMAVE with several common PLMs. All the models were guaranteed to be equipped with the same output layer
when compared. The results are shown in Table 4. Here SpanBERT and MacBERT have no results on some datasets because of lacking a corresponding language version. SpanBERT achieves almost the same results as ROBERTA with half the number of parameters because it excels in span representation. ELECTRA adopts creative adversarial pretraining and therefore performs well. Compared to the pre-training backbone ROBERTA, our further pre-trained COMAVE gains a significant improvement. Moreover, regardless of the simple finetuning layer used, our model outperforms all the other PLMs, which indicates that our pre-training effectively alleviates the challenges of AVE tasks.

heading: 4.2 Ablation Tests
text: To evaluate the contributions of each training objective, we considered the following settings:
• − MSMLM: Removing the training objective of Multi-Scale Masked Language Model. • MSMLM − PhraM: Only Using the sentencelevel masking mechanism. • MSMLM − SentM: Only Using the phraselevel masking mechanism. • − CAR: Removing the training objective of Contrastive Attribute Retrieval. • CAR − MRL: Replacing the Margin Ranking Loss LCAR with the Cross Entropy Loss. • CAR − CS: Cluster sampling is not used in CAR, i.e., A− = A−g . • − VD: Removing the training objective of Value Detection.
Here, MRC was uniformly selected as the output layer for all the settings due to its better performance. Table 5 shows the results on four datasets. CAR, MSMLM, and VD all bring obviously improvements, proving the effectiveness and necessity of our pre-training objectives for the AVE tasks.
The result indicates that the contribution of CAR is the most pronounced among the three objectives. The final performance of the model decreases significantly when either the clustering sampling or the contrast loss is removed. In addition, we find that the combination of phrase-level and sentencelevel masking is more effective than using only one of them. VD also delivers a promising improvement which proves the benefit for AVE tasks.

heading: 4.3 Tests on Fine-Grained Attribute Groups
text: To further validate the effectiveness of our method in discriminating fine-grained similar attributes, we evaluated the performance of the model on the finegrained attribute groups mentioned in Table 1. The experimental results are shown in the upper part of Figure 3. Our COMAVE equipped with all components achieves the best results on all attribute groups. When the CAR training objective is removed, the overall performance shows a dramatic decrease in all the fine-grained attribute groups. This reveals that contrastive learning in a challenging set during pre-training contributed significantly to enhancing the capability of discriminating similar attributes in downstream tasks.

heading: 4.4 Performance on Multi-Scale Values
text: We also tested the performance of the model for extracting values at different scales, and the results are shown in the lower part of Figure 3. As we expected, the contribution of phrase-level masking
is greater when dealing with shorter values (number of tokens less than 10). The improvement of sentence-level masking becomes significant when the length of value gradually grows to more than 20. This proves the reasonableness of our combination of phrase-level and sentence-level masking. Moreover, we find that even pre-training without MSMLM, COMAVE still performs better than pure ROBERTA. This demonstrates the boost from objective VD and the expected external knowledge of the pre-training corpus.

heading: 4.5 Few-Shot Tests
text: To further fit realistic applications, we also tested the performance of the model in few-shot scenarios. We adopted the N -WAY K-SHOT setup, i.e., the few-shot training set has N attributes, and each attribute has corresponding K training samples randomly selected. Here, we let N equal the number of attributes per dataset and focused on testing two sets of 5-SHOT and 10-SHOT settings.
Table 6 shows the experimental results. Under the stringent condition of using only 5 training samples for each attribute, COMAVE scores Mirco-F1 over 50% on all datasets. When the training set is expanded to 10-SHOT, the performance reaches approximately 65% on the other three datasets, except for the challenging INS. The smaller the sample size, the greater the improvement of COMAVE. Due to pre-training with a large-scale AVE corpus collected, COMAVE is more capable than ROBERTA in handling the few-shot AVE task.

heading: 5 Related Work
text: With the development of e-commerce, Attribute Value Extraction which aims to retrieve the attributes and extract the values from the target data resource in order to obtain the structured information of the products recently attracts lots of at-
tention. Several previous methods (Zheng et al., 2018; Xu et al., 2019) employ traditional sequence tagging models. Furthermore, AVEQA (Wang et al., 2020) first tries to use MRC based method to handle the task, but it can not be applied when each attribute has several different values. JAVE (Zhu et al., 2020) designs a multi-task model which divides the task into two sub-task: attributes prediction and value extraction. AdaTag (Yan et al., 2021) uses a hyper-network to train experts’ parameters for each attribute in order to build an adaptive decoder. QUEACO (Zhang et al., 2021) adopts a teacher-student network to leverage weakly labeled behavior data to improve performance. MAVEQA (Yang et al., 2022) mixes multisource information with a novel global and local attention mechanism. However, none of the existing methods pay attention to the two challenges mentioned in section 1.
Language model pre-training (Devlin et al., 2019; Liu et al., 2019) and task-specific fine-tuning achieve significant improvement on many NLP tasks. Recently, some work (Joshi et al., 2020; Clark et al., 2020; Cui et al., 2020; Sanh et al., 2019) further modified the MLM to achieve better results. In information extraction tasks, UIE (Lu et al., 2022) is proposed as a universal pre-training model for several extraction tasks by generation, it is generic but lacks further fitting for different extraction tasks. Currently, there is no task-specific pre-training model for attribute value extraction.

heading: 6 Conclusion
text: In this paper, we presented a new pre-training model for attribute value extraction, called COMAVE which is pre-trained by three novel objectives with a large-scale corpus. Multi-Scale Masked Language Model is designed to force the model to understand multi-scale values by recovering masked spans at both the phrase and sentence levels. Contrastive Attribute Retrieval improves the discrimination of fine-grained attributes based on contrastive learning. Meanwhile, Value Detection is adopted to reinforce the value extraction and further benefit downstream AVE tasks. Extensive experiments indicate that COMAVE achieves stateof-the-art results on four benchmarks compared with the existing baselines and PLMs. In future work, we will expand our work on more scenarios and industries, and also explore the optimization of the downstream fine-tune model.

heading: 7 Limitations
text: This paper proposed a novel pre-training model COMAVE which aims at textual AVE tasks, while in this field, multi-modal AVE tasks also widely exist in many e-commerce platforms. We expect that the following works can leverage COMAVE as a powerful word embedding pre-training model for text encoding combined with image feature representation in multi-modal AVE tasks in the future. Meanwhile, the same as the previous AVE works, we assume that each T is an independent extraction object, without considering the context-dependent of the whole data resources, such as long documents and instructions, which exceeds the length of an allowable single input.

heading: Acknowledgements
text: This work is supported by the NSFC (Grant No. U21A20488, 62006040), the Project for the Doctor of Entrepreneurship and Innovation in Jiangsu Province (Grant No. JSSCBS20210126), the Fundamental Research Funds for the Central Universities, and ZhiShan Young Scholar Program of Southeast University.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7: Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1: Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts?

heading: Left blank.
text: B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4: Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4: Experiments, Implementation Details
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4: Experiments, Implementation Details

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4: Experiments
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The packages used in our code are listed in GitHub.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4: Experiments, Datasets. We build a hand-labeled dataset called INS for evaluation.
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We build the manually labeled dataset INS for evaluation, while there is no human participation in other parts.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We build the manually labeled dataset INS for evaluation, while there is no human participation in other parts.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The dataset is collected from the platform of our affiliation, and the source is not discussed in this submission for anonymity.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Our dataset contains the product information of the e-commerce platform, not the information of humans, and humans only participate in labeling the dataset.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Our dataset contains the product information of the e-commerce platform, without the information of humans, and humans only participate in labeling the dataset.

heading: Future Work
text: We expect that the following works can leverage COMAVE as a powerful word embedding pre-training model for text encoding combined with image feature representation in multi-modal AVE tasks in the future. Meanwhile, the same as the previous AVE works, we assume that each T is an independent extraction object, without considering the context-dependent of the whole data resources, such as long documents and instructions, which exceeds the length of an allowable single input