GenAI4ELab/papercli-papers huggingface dataset:
Large Scale Generative Multimodal Attribute Extraction for E-commerce Attributes
Authors: Anant Khandelwal; Happy Mittal; Shreyas Kulkarni; Deepak Gupta
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: E-commerce websites (e.g. Amazon, Alibaba) have a plethora of structured and unstructured information (text and images) present on the product pages. Sellers often don’t label or mislabel values of the attributes (e.g. color, size etc.) for their products. Automatically identifying these attribute values from an eCommerce product page that contains both text and images is a challenging task, especially when the attribute value is not explicitly mentioned in the catalog. In this paper, we present a scalable solution for this problem where we pose attribute extraction problem as a question-answering task, which we solve using MXT, that consists of three key components: (i) MAG (Multimodal Adaptation Gate), (ii) Xception network, and (iii) T5 encoder-decoder. Our system consists of a generative model that generates attribute-values for a given product by using both textual and visual characteristics (e.g. images) of the product. We show that our system is capable of handling zero-shot attribute prediction (when attribute value is not seen in training data) and value-absent prediction (when attribute value is not mentioned in the text) which are missing in traditional classification-based and NER-based models respectively. We have trained our models using distant supervision, removing dependency on human labeling, thus making them practical for real-world applications. With this framework, we are able to train a single model for 1000s of (product-type, attribute) pairs, thus reducing the overhead of training and maintaining separate models. Extensive experiments on two real world datasets (total 57 attributes) show that our framework improves the absolute recall@90P by 10.16% and 6.9 from the existing state of the art models. In a popular e-commerce store, we have productionized our models that cater to 12K (product-type, attribute) pairs, and have extracted 150MM attribute values.
Consistent Text Categorization using Data Augmentation in e-Commerce
Authors: Noa Avigdor; Guy Horowitz; Ariel Raviv; Stav Yanovsky Daye
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: The categorization of massive e-Commerce data is a crucial, well-studied task, which is prevalent in industrial settings. In this work, we aim to improve an existing product categorization model that is already in use by a major web company, serving multiple applications. At its core, the product categorization model is a text classification model that takes a product title as an input and outputs the most suitable category out of thousands of available candidates. Upon a closer inspection, we found inconsistencies in the labeling of similar items. For example, minor modifications of the product title pertaining to colors or measurements majorly impacted the model’s output. This phenomenon can negatively affect downstream recommendation or search applications, leading to a sub-optimal user experience. To address this issue, we propose a new framework for consistent text categorization. Our goal is to improve the model’s consistency while maintaining its production-level performance. We use a semi-supervised approach for data augmentation and presents two different methods for utilizing unlabeled samples. One method relies directly on existing catalogs, while the other uses a generative model. We compare the pros and cons of each approach and present our experimental results.
An efficient method for Natural Language Querying on Structured Data
Authors: Hanoz Bhathena; Aviral Joshi; Prateek Singh
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: We present an efficient and reliable approach to Natural Language Querying (NLQ) on databases (DB) which is not based on text-to-SQL type semantic parsing. Our approach simplifies the NLQ on structured data problem to the following “bread and butter” NLP tasks: (a) Domain classification, for choosing which DB table to query, whether the question is out-of-scope (b) Multi-head slot/entity extraction (SE) to extract the field criteria and other attributes such as its role (filter, sort etc) from the raw text and (c) Slot value disambiguation (SVD) to resolve/normalize raw spans from SE to format suitable to query a DB. This is a general purpose, DB language agnostic approach and the output can be used to query any DB and return results to the user. Also each of these tasks is extremely well studied, mature, easier to collect data for and enables better error analysis by tracing problems to specific components when something goes wrong.
Boosting Transformers and Language Models for Clinical Prediction in Immunotherapy
Authors: Zekai Chen; Mariann Micsinai Balan; Kevin Brown
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Clinical prediction is an essential task in the healthcare industry. However, the recent success of transformers, on which large language models are built, has not been extended to this domain. In this research, we explore the use of transformers and language models in prognostic prediction for immunotherapy using real-world patients’ clinical data and molecular profiles. This paper investigates the potential of transformers to improve clinical prediction compared to conventional machine learning approaches and addresses the challenge of few-shot learning in predicting rare disease areas. The study benchmarks the efficacy of baselines and language models on prognostic prediction across multiple cancer types and investigates the impact of different pretrained language models under few-shot regimes. The results demonstrate significant improvements in accuracy and highlight the potential of NLP in clinical research to improve early detection and intervention for different diseases.
EvolveMT: an Ensemble MT Engine Improving Itself with Usage Only
Authors: Kamer Yüksel; Ahmet Gunduz; Mohamed Al-badrashiny; Hassan Sawaf
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: This work proposes a method named EvolveMT for the efficient combination of multiple machine translation (MT) engines. The method selects the output from one engine for each segment, using online learning techniques to predict the most appropriate system for each translation request. A neural quality estimation metric supervises the method without requiring reference translations. The method’s online learning capability enables it to adapt to changes in the domain or MT engines dynamically, eliminating the requirement for retraining. The method selects a subset of translation engines to be called based on the source sentence features. The degree of exploration is configurable according to the desired quality-cost trade-off. Results from custom datasets demonstrate that EvolveMT achieves similar translation accuracy at a lower cost than selecting the best translation of each segment from all translations using an MT quality estimator. To the best of our knowledge, EvolveMT is the first MT system that adapts itself after deployment to incoming translation requests from the production environment without needing costly retraining on human feedback.
A Static Evaluation of Code Completion by Large Language Models
Authors: Hantian Ding; Varun Kumar; Yuchen Tian; Zijian Wang; Rob Kwiatkowski; Xiaopeng Li; Murali Krishna Ramanathan; Baishakhi Ray; Parminder Bhatia; Sudipta Sengupta
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the other hand, static analysis tools such as linters, which can detect errors without running the program, haven’t been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.
Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems
Authors: Sarthak Ahuja; Mohammad Kachuee; Fatemeh Sheikholeslami; Weiqing Liu; Jaeyoung Do
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Off-Policy reinforcement learning has been the driving force for the state-of-the-art conversational AIs leading to more natural human-agent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.
MobileNMT: Enabling Translation in 15MB and 30ms
Authors: Ye Lin; Xiaohui Wang; Zhexi Zhang; Mingxuan Wang; Tong Xiao; Jingbo Zhu
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on devices is challenging with limited storage, memory, computation, and power consumption. Existing work either only focuses on a single metric such as FLOPs or general engine which is not good at auto-regressive decoding. In this paper, we present MobileNMT, a system that can translate in 15MB and 30ms on devices. We propose a series of principles for model compression when combined with quantization. Further, we implement an engine that is friendly to INT8 and decoding. With the co-design of model and engine, compared with the existing system, we speed up 47.0x and save 99.5% of memory with only 11.6% loss of BLEU. Our code will be publicly available after the anonymity period.
Multi-doc Hybrid Summarization via Salient Representation Learning
Authors: Min Xiao
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Multi-document summarization is gaining more and more attention recently and serves as an invaluable tool to obtain key facts among a large information pool. In this paper, we proposed a multi-document hybrid summarization approach, which simultaneously generates a human-readable summary and extracts corresponding key evidences based on multi-doc inputs. To fulfill that purpose, we crafted a salient representation learning method to induce latent salient features, which are effective for joint evidence extraction and summary generation. In order to train this model, we conducted multi-task learning to optimize a composited loss, constructed over extractive and abstractive sub-components in a hierarchical way. We implemented the system based on a ubiquiotously adopted transformer architecture and conducted experimental studies on multiple datasets across two domains, achieving superior performance over the baselines.
SaFER: A Robust and Efficient Framework for Fine-tuning BERT-based Classifier with Noisy Labels
Authors: Zhenting Qi; Xiaoyu Tan; Chao Qu; Yinghui Xu; Yuan Qi
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Learning on noisy datasets is a challenging problem when pre-trained language models are applied to real-world text classification tasks. In numerous industrial applications, acquiring task-specific datasets with 100% accurate labels is difficult, thus many datasets are accompanied by label noise at different levels. Previous work has shown that existing noise-handling methods could not improve the peak performance of BERT on noisy datasets, and might even deteriorate it. In this paper, we propose SaFER, a robust and efficient fine-tuning framework for BERT-based text classifiers, combating label noises without access to any clean data for training or validation. Utilizing a label-agnostic early-stopping strategy and self-supervised learning, our proposed framework achieves superior performance in terms of both accuracy and speed on multiple text classification benchmarks. The trained model is finally fully deployed in several industrial biomedical literature mining tasks and demonstrates high effectiveness and efficiency.
Chemical Language Understanding Benchmark
Authors: Yunsoo Kim; Hyuk Ko; Jane Lee; Hyun Young Heo; Jinyoung Yang; Sungsoo Lee; Kyu-hwang Lee
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: In this paper, we introduce the benchmark datasets named CLUB (Chemical Language Understanding Benchmark) to facilitate NLP research in the chemical industry. We have 4 datasets consisted of text and token classification tasks. As far as we have recognized, it is one of the first examples of chemical language understanding benchmark datasets consisted of tasks for both patent and literature articles provided by industrial organization. All the datasets are internally made by chemists from scratch. Finally, we evaluate the datasets on the various language models based on BERT and RoBERTa, and demonstrate the model performs better when the domain of the pretrained models are closer to chemistry domain. We provide baselines for our benchmark as 0.8054 in average, and we hope this benchmark is used by many researchers in both industry and academia.
HyperT5: Towards Compute-Efficient Korean Language Modeling
Authors: Dongju Park; Soonwon Ka; Kang Min Yoo; Gichang Lee; Jaewook Kang
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Pretraining and fine-tuning language models have become the standard practice in industrial natural language processing (NLP), but developing and deploying general-purpose language models without the abundant computation or data resources is a real-world issue faced by smaller organizations or communities whose main focus is languages with less accessible resources (e.g., non-English). This paper explores the sequence-to-sequence (seq2seq) language model architecture as a more practical and compute-efficient alternative to the decoder-oriented approach (e.g., GPT-3), accompanied by novel findings in compute-optimality analyses. We successfully trained billion-scale Korean-language seq2seq language models that strongly outperform other competitive models in Korean benchmarks. Moreover, we demonstrate that such language models can be more efficiently utilized by employing a heavy pre-finetuning strategy, by showcasing a case study on dialog-task adaptation. Our case study shows that adopting language models with more readily available domain-specific unlabeled data greatly improves fine-tuning data efficiency in low-resource settings.
Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings
Authors: Jong Myoung Kim; Young-jun Lee; Sangkeun Jung; Ho-jin Choi
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Ambiguity is a major obstacle to providing services based on sentence classification. However, because of the structural limitations of the service, there may not be sufficient contextual information to resolve the ambiguity. In this situation, we focus on ambiguity detection so that service design considering ambiguity is possible. We utilize similarity in a semantic space to detect ambiguity in service scenarios and training data. In addition, we apply task-specific embedding to improve performance. Our results demonstrate that ambiguities and resulting labeling errors in training data or scenarios can be detected. Additionally, we confirm that it can be used to debug services
Reliable and Interpretable Drift Detection in Streams of Short Texts
Authors: Ella Rabinovich; Matan Vetzler; Samuel Ackerman; Ateret Anaby Tavor
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Data drift is the change in model input data that is one of the key factors leading to machine learning models performance degradation over time. Monitoring drift helps detecting these issues and preventing their harmful consequences. Meaningful drift interpretation is a fundamental step towards effective re-training of the model. In this study we propose an end-to-end framework for reliable model-agnostic change-point detection and interpretation in large task-oriented dialog systems, proven effective in multiple customer deployments. We evaluate our approach and demonstrate its benefits with a novel variant of intent classification training dataset, simulating customer requests to a dialog system. We make the data publicly available.
Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding
Authors: Jonathan Hueser; Judith Gaspers; Thomas Gueudre; Chandana Prakash; Jin Cao; Daniil Sorokin; Quynh Do; Nicolas Anastassacos; Tobias Falke; Turan Gojayev
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: Leveraging representations from pre-trained transformer-based encoders achieves state-of-the-art performance on numerous NLP tasks. Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines).We evaluate using a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU compared to using smaller 17M parameter BERT encoders with language-, domain- and task-decoupled finetuning.Running inference with a larger shared encoder on GPU is latency neutral and reduces infrastructure cost compared to running inference for decoupled smaller encoders on CPU machines. The larger shared encoder reduces semantic error rates by 4.62% for test sets representing user requests to voice-controlled devices and 5.79% on the tail of the test sets on average across four languages.
Annotating Research Infrastructure in Scientific Papers: An NLP-driven Approach
Authors: Seyed Amin Tabatabaei; Georgios Cheirmpos; Marius Doornenbal; Alberto Zigoni; Veronique Moore; Georgios Tsatsaronis
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: In this work, we present a natural language processing (NLP) pipeline for the identification, extraction and linking of Research Infrastructure (RI) used in scientific publications. Links between scientific equipment and publications where the equipment was used can support multiple use cases, such as evaluating the impact of RI investment, and supporting Open Science and research reproducibility. These links can also be used to establish a profile of the RI portfolio of each institution and associate each equipment with scientific output. The system we are describing here is already in production, and has been used to address real business use cases, some of which we discuss in this paper. The computational pipeline at the heart of the system comprises both supervised and unsupervised modules to detect the usage of research equipment by processing the full text of the articles. Additionally, we have created a knowledge graph of RI, which is utilized to annotate the articles with metadata. Finally, examples of the business value of the insights made possible by this NLP pipeline are illustrated.
Event-Centric Query Expansion in Web Search
Authors: Yanan Zhang; Weijie Cui; Yangfan Zhang; Xiaoling Bai; Zhe Zhang; Jin Ma; Xiang Chen; Tianhua Zhou
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track))

Abstract: In search engines, query expansion (QE) is a crucial technique to improve search experience. Previous studies often rely on long-term search log mining, which leads to slow updates and is sub-optimal for time-sensitive news searches. In this work, we present Event-Centric Query Expansion (EQE), the QE system used in a famous Chinese search engine. EQE utilizes a novel event retrieval framework that consists of four stages, i.e., event collection, event reformulation, semantic retrieval and online ranking, which can select the best expansion from a significant amount of potential events rapidly and accurately. Specifically, we first collect and filter news headlines from websites. Then we propose a generation model that incorporates contrastive learning and prompt-tuning techniques to reformulate these headlines to concise candidates. Additionally, we fine-tune a dual-tower semantic model to serve as an encoder for event retrieval and explore a two-stage contrastive training approach to enhance the accuracy of event retrieval. Finally, we rank the retrieved events and select the optimal one as QE, which is then used to improve the retrieval of event-related documents. Through offline analysis and online A/B testing, we observed that the EQE system has significantly improved many indicators compared to the baseline. The system has been deployed in a real production environment and serves hundreds of millions of users.