Reviewerly/exHarmony huggingface dataset:
Image Manipulation via Multi-Hop Instructions - A New Dataset and Weakly-Supervised Neuro-Symbolic Approach

Harman Singh, Poorva Garg, Mohit Gupta, Kevin Shah, Ashish Goswami, Satyam Modi, Arnab Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks

Mixture-of-Experts (MoE) based sparse architectures can significantly increase model capacity with sublinear computational overhead, which are hence widely used in massively multilingual neural machine translation (MNMT). However, they are prone to overfitting on low-resource language translation. In this paper, we propose a modularized MNMT framework that is able to flexibly assemble dense and MoE-based sparse modules to achieve the best of both worlds. The training strategy of the modularized MNMT framework consists of three stages: (1) Pre-training basic MNMT models with different training objectives or model structures, (2) Initializing modules of the framework with pre-trained couterparts (e.g., encoder, decoder and embedding layers) from the basic models and (3) Fine-tuning the modularized MNMT framework to fit modules from different models together. We pre-train three basic MNMT models from scratch: a dense model, an MoE-based sparse model and a new MoE model, termed as MoE-LGR that explores multiple Language-Group-specifc Routers to incorporate language group knowledge into MNMT. The strengths of these pre-trained models are either on low-resource language translation, high-resource language translation or zero-shot translation. Our modularized MNMT framework attempts to incorporate these advantages into a single model with reasonable initialization and fine-tuning. Experiments on widely-used benchmark datasets demonstrate that the proposed modularized MNMT framwork substantially outperforms both MoE and dense models on high- and low-resource language translation as well as zero-shot translation. Our framework facilitates the combination of different methods with their own strengths and recycling off-the-shelf models for multilingual neural machine translation. Codes are available at https://github.com/lishangjie1/MMNMT.
Controlling Pre-trained Language Models for Grade-Specific Text Simplification

Text simplification systems rewrite text to make it more readable while preserving its content. However, what makes a text easy to read depends on the intended readers. Recent work has shown that pre-trained language models can simplify text using a wealth of techniques to control output simplicity, ranging from specifying only the desired reading grade level, to directly specifying low-level edit operations. Yet it remains unclear how to set these control parameters in practice. Existing approaches set them at the corpus level, disregarding the complexity of individual inputs and considering only one level of output complexity. In this work, we conduct an empirical study to understand how different control mechanisms impact the adequacy and simplicity of text simplification systems. Based on these insights, we introduce a simple method that predicts the edit operations required for simplifying a text for a specific grade level on an instance-per-instance basis. This approach improves the quality of the simplified outputs over corpus-level search-based heuristics.
Time-Aware Language Modeling for Historical Text Dating

Automatic text dating(ATD) is a challenging task since explicit temporal mentions usually do not appear in texts. Existing state-of-the-art approaches learn word representations via language models, whereas most of them ignore diachronic change of words, which may affect the efforts of text modeling. Meanwhile, few of them consider text modeling for long diachronic documents. In this paper, we present a time-aware language model named TALM, to learn temporal word representations by transferring language models of general domains to those of time-specific ones. We also build a hierarchical modeling approach to represent diachronic documents by encoding them with temporal word representations. Experiments on a Chinese diachronic corpus show that our model effectively captures implicit temporal information of words, and outperforms state-of-the-art approaches in historical text dating as well.
Ideology Takes Multiple Looks: A High-Quality Dataset for Multifaceted Ideology Detection

Ideology detection (ID) is important for gaining insights about peoples’ opinions and stances on our world and society, which can find many applications in politics, economics and social sciences. It is not uncommon that a piece of text can contain descriptions of various issues. It is also widely accepted that a person can take different ideological stances in different facets. However, existing datasets for the ID task only label a text as ideologically left- or right-leaning as a whole, regardless whether the text containing one or more different issues. Moreover, most prior work annotates texts from data resources with known ideological bias through distant supervision approaches, which may result in many false labels. With some theoretical help from social sciences, this work first designs an ideological schema containing five domains and twelve facets for a new multifaceted ideology detection (MID) task to provide a more complete and delicate description of ideology. We construct a MITweet dataset for the MID task, which contains 12,594 English Twitter posts, each annotated with a Relevance and an Ideology label for all twelve facets. We also design and test a few of strong baselines for the MID task under in-topic and cross-topic settings, which can serve as benchmarks for further research.
FLEEK: Factual Error Detection and Correction with Evidence Retrieved from External Knowledge

Farima Fatahi Bayat, Kun Qian, Benjamin Han, Yisi Sang, Anton Belyy, Samira Khorshidi, Fei Wu, Ihab Ilyas, Yunyao Li. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023.
Tagging-Assisted Generation Model with Encoder and Decoder Supervision for Aspect Sentiment Triplet Extraction

ASTE (Aspect Sentiment Triplet Extraction) has gained increasing attention. Recent advancements in the ASTE task have been primarily driven by Natural Language Generation-based (NLG) approaches. However, most NLG methods overlook the supervision of the encoder-decoder hidden representations and fail to fully utilize the semantic information provided by the labels to enhance supervision. These limitations can hinder the extraction of implicit aspects and opinions. To address these challenges, we propose a tagging-assisted generation model with encoder and decoder supervision (TAGS), which enhances the supervision of the encoder and decoder through multiple-perspective tagging assistance and label semantic representations. Specifically, TAGS enhances the generation task by integrating an additional sequence tagging task, which improves the encoder’s capability to distinguish the words of triplets. Moreover, it utilizes sequence tagging probabilities to guide the decoder, improving the generated content’s quality. Furthermore, TAGS employs a self-decoding process for labels to acquire the semantic representations of the labels and aligns the decoder’s hidden states with these semantic representations, thereby achieving enhanced semantic supervision for the decoder’s hidden states. Extensive experiments on various public benchmarks demonstrate that TAGS achieves state-of-the-art performance.
CLAIR: Evaluating Image Captions with Large Language Models

The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
Angel: Enterprise Search System for the Non-Profit Industry

Non-profit industry need a system for accurately matching fund-seekers (e.g., AMERICAN NATIONAL RED CROSS) with fund-givers (e.g., BILL AND MELINDA GATES FOUNDATION) aligned in cause (e.g., cancer) and target beneficiary group (e.g., children). In this paper, we create an enterprise search system “ANGEL” for the non-profit industry that takes a fund-giver’s mission description as input and returns a ranked list of fund-seekers as output, and vice-versa. ANGEL employs ColBERT, a neural information retrieval model, which we enhance by exploiting the two techniques of (a) Syntax-aware local attention (SLA) to combine syntactic information in the mission description with multi-head self-attention and (b) Dense Pseudo Relevance Feedback (DPRF) for augmentation of short mission descriptions. We create a mapping dictionary “non-profit-dict” to curate a “non-profit-search database” containing information on 594K fund-givers and 194K fund-seekers from IRS-990 filings for the non-profit industry search engines . We also curate a “non-profit-evaluation” dataset containing scored matching between 463 fund-givers and 100 fund-seekers. The research is in collaboration with a philanthropic startup that identifies itself as an “AI matching platform, fundraising assistant, and philanthropy search base.” Domain experts at the philanthropic startup annotate the non-profit evaluation dataset and continuously evaluate the performance of ANGEL. ANGEL achieves an improvement of 0.14 MAP@10 and 0.16 MRR@10 over the state-of-the-art baseline on the non-profit evaluation dataset. To the best of our knowledge, ours is the first effort at building an enterprise search engine based on neural information retrieval for the non-profit industry.
BLESS: Benchmarking Large Language Models on Sentence Simplification

Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, Matthew Shardlow. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks

Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. However, how to select new tasks to improve the performance and generalizability of IT models remains an open question. Training on all existing tasks is impractical due to prohibiting computation requirements, and randomly selecting tasks can lead to suboptimal performance. In this work, we propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks. We represent the informativeness of new tasks with the disagreement of the current model outputs over perturbed prompts. Our experiments on NIV2 and Self-Instruct datasets demonstrate that our method consistently outperforms other baseline strategies for task selection, achieving better out-of-distribution generalization with fewer training tasks. Additionally, we introduce a task map that categorizes and diagnoses tasks based on prompt uncertainty and prediction probability. We discover that training on ambiguous (prompt-uncertain) tasks improves generalization while training on difficult (prompt-certain and low-probability) tasks offers no benefit, underscoring the importance of task selection for instruction tuning.
Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the GeNTE Corpus

Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.
ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Jaap Jumelet, Michael Hanna, Marianne de Heer Kloots, Anna Langedijk, Charlotte Pouw, Oskar van der Wal. Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. 2023.
Implementing Natural Image Quality Evaluator for Performance Indicator on Noise Artefacts Recovery in CT Scan

The two most common evaluators for CT scan denoising are Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM). This paper offers an alternative evaluator by utilizing of Natural Image Quality Evaluator (NIQE) assessment to determine the performance of denoising work on noise artefact. The noise artefact was obtained during the cancer screening process and had a particular noise density pattern across the image. NIQE is one of the blind image assessments which rely on the measurable deviation of image patch as a reference; it can determine the improved quality of denoising image. Due to the method of comparison in NIQE, the two parameters: patch size and sharpness threshold, will play an essential part in getting the score compared with the result from the other evaluators (PSNR and SSIM).
A comparison study on creating simulated patient data for individuals suffering from chronic coronary disorders

An emerging area in data science that has lately gained attention is the virtual population (VP) and synthetic data generation. This field has the potential to significantly affect the healthcare industry by providing a means to augment clinical research databases that have a shortage of subjects. The current study provides a comparative analysis of five distinct approaches for creating virtual data populations from real patient data. The data set utilized for the current analyses involved clinical data collected among patients scheduled for elective coronary artery bypass graft surgery (CABG). To that end, the five computational techniques employed to augment the given dataset were: (i) Tabular Preset, (ii) Gaussian Copula Model (iii) Generative Adversarial Network based (GAN) Deep Learning data synthesizer (CTGAN), (iv) a variation of the CTGAN Model (Copula GAN), and (v) VAE-based Deep Learning data synthesizer (TVAE). The performance of these techniques was assessed against their effectiveness in producing high-quality virtual data. For this purpose, dataset correlation matrices, cosine similarity distance, density histograms, and kernel density estimation are employed to perform a comparative analysis of each attribute and the respective synthetic equivalent. Our findings demonstrate that Gaussian Copula Model prevails in creating virtual data with consistent distributions (Kolmogorov-Smirnov (KS) and Chi-Squared (CS) tests equal to 0.9 and 0.98, respectively) and correlation patterns (average cosine similarity equals to 0.95).Clinical Relevance— It has been shown that the use of a VP can increase the predictive performance of a ML model, i.e., above using a smaller non-augmented population.
A Classification Model Utilizing Facial Landmark Tracking to Determine Sentence Types for American Sign Language Recognition

The deaf and hard of hearing community relies on American Sign Language (ASL) as their primary mode of communication, but communication with others who do not know ASL can be difficult, especially during emergencies where no interpreter is available. As an effort to alleviate this problem, research in computer vision based real time ASL interpreting models is ongoing. However, most of these models are hand shape (gesture) based and lack the integration of facial cues, which are crucial in ASL to convey tone and distinguish sentence types. Thus, the integration of facial cues in computer vision based ASL interpreting models has the potential to improve performance and reliability. In this paper, we introduce a simple, computationally efficient facial expression based classification model that can be used to improve ASL interpreting models. This model utilizes the relative angles of facial landmarks with principal component analysis and a Random Forest Classification tree model to classify frames taken from videos of ASL users signing a complete sentence. The model classifies the frames as statements or assertions. The model was able to achieve an accuracy of 86.5%.
Filling in the Gaps, Doing What We Have Always Done in TRAIL

The Technical Report Archive &amp; Image Library (TRAIL) identifies, acquires, catalogs, digitizes, and provides unrestricted access to US government agency technical reports. Technical reports describe the process of engineering or scientific research and often include in-depth details including raw data. TRAIL currently consists of over four-dozen member institutions whose annual membership fees and volunteered staff time further the efforts of the project. The mission of TRAIL is to ensure preservation, discoverability, and persistent open access to government technical publications regardless of form or format. TRAIL has been in existence for nearly seventeen years and has made 93,514 technical reports openly accessible in that time. This article covers the work of TRAIL since its tenth anniversary in 2016. For the work and history of TRAIL up to that point, please read the article summarizing the first ten years of TRAIL in Documents to the People vol. 44, no. 2 ( https://www.journals.ala.org/index.php/dttp/article/view/6070/7795 ). Additional resources include the TRAIL History page at https://www.crl.edu/grn/trail/about-trail/history-trail and the page on publications and presentations about TRAIL at https://www.crl.edu/grn/trail/current-activities/PapersPresentations .
SwarMer: A Decentralized Localization Framework for Flying Light Specks

Swarm-Merging, SwarMer, is a decentralized framework to localize Flying Light Specks (FLSs) to render 2D and 3D shapes. An FLS is a miniature sized drone equipped with one or more light sources to generate different colors and textures with adjustable brightness. It is battery powered, network enabled with storage and processing capability to implement a decentralized algorithm such as SwarMer. An FLS is unable to render a shape by itself. SwarMer uses the inter-FLS relationship effect of its organizational framework to compensate for the simplicity of each individual FLS, enabling a swarm of cooperating FLSs to render complex shapes. SwarMer is resilient to network packet loss, FLSs failing, and FLSs leaving to charge their battery. It is fast, highly accurate, and scales to remain effective when a shape consists of a large number of FLSs.
CV classification in a big data environment using BERT and alignment learning

Abstract Today, text classification has always been a crucial discipline in the field of text data processing. However, with the advent of Big Data, text classification has reached new heights in terms of the volume of data to be processed and the complexity of the tasks. This task is of great importance in many fields, such as spam detection, sentiment analysis, document categorization, content recommendation and many more. In this work, we focus specifically on resume classification, an area that presents unique challenges due to diverse formats, ambiguous language, and variations in applicants' work experiences. We propose an innovative approach for distributed classification of CVs using contextual alignment techniques with the job offer and the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) which constitutes a major advance in this field , and using RNN (Recurrent Neural Network) to solve text classification problems. We use a distributed processing approach to leverage the parallel computing power of Spark, enabling large volumes of data to be processed efficiently. The main objective of our study is to improve the relevance of CV classification by leveraging the pre-trained language models and distributed processing power of Spark.
Continuous Risk Estimation of Acute Kidney Failure with Dense Temporal Data for ICU Patients

Acute kidney failure is a dangerous complication for ICU patients, and it is difficult to identify at early stage with conventional medical analysis. In recent years, machine learning approaches have been applied to tackle medical diagnosis tasks with great performance. In this work, we deploy machine learning models for early detection of acute kidney failure that can handle static, temporal, sparse and dense data of ICU patients. We investigate different pre-processing methods for patient data to achieve higher prediction performance and how they influence the contribution of different physiological signals in the prediction process.
Power Optimized Smartwatch-Earbuds Multimodal System for Monitoring Activities of Daily Living

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance— The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.