Reviewerly/exHarmony huggingface dataset:
Noisy Self-Training with Synthetic Queries for Dense Retrieval

Although existing neural retrieval models reveal promising results when training data is abundant and the performance keeps improving as training data increases, collecting high-quality annotated data is prohibitively costly. To this end, we introduce a novel noisy self-training framework combined with synthetic queries, showing that neural retrievers can be improved in a self-evolution manner with no reliance on any external models. Experimental results show that our method improves consistently over existing methods on both general-domain (e.g., MS-MARCO) and out-of-domain (i.e., BEIR) retrieval benchmarks. Extra analysis on low-resource settings reveals that our method is data efficient and outperforms competitive baselines, with as little as 30% of labelled training data. Further extending the framework for reranker training demonstrates that the proposed method is general and yields additional gains on tasks of diverse domains.
When Do Decompositions Help for Machine Reading?

Answering complex questions often requires multi-step reasoning in order to obtain the final answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments on decompositions in machine reading to unify recent work in this space, using a range of models and datasets. We find that decompositions can be helpful in zero or limited-data settings, giving several points of improvement in exact match. However, we also show that when models are given access to around a few hundred or more examples, decompositions are not helpful (and can actually be detrimental). Thus, our analysis implies that models can learn decompositions implicitly even with limited data.
Empowering a Metric with LLM-assisted Named Entity Annotation: HW-TSC’s Submission to the WMT23 Metrics Shared Task

Zhanglin Wu, Yilun Liu, Min Zhang, Xiaofeng Zhao, Junhao Zhu, Ming Zhu, Xiaosong Qiao, Jingfei Zhang, Ma Miaomiao, Zhao Yanqing, Song Peng, Shimin Tao, Hao Yang, Yanfei Jiang. Proceedings of the Eighth Conference on Machine Translation. 2023.
Descriptive Knowledge Graph in Biomedical Domain

We present a novel system that automatically extracts and generates informative and descriptive sentences from the biomedical corpus and facilitates the efficient search for relational knowledge. Unlike previous search engines or exploration systems that retrieve unconnected passages, our system organizes descriptive sentences as a relational graph, enabling researchers to explore closely related biomedical entities (e.g., diseases treated by a chemical) or indirectly connected entities (e.g., potential drugs for treating a disease). Our system also uses ChatGPT and a fine-tuned relation synthesis model to generate concise and reliable descriptive sentences from retrieved information, reducing the need for extensive human reading effort. With our system, researchers can easily obtain both high-level knowledge and detailed references and interactively steer to the information of interest. We spotlight the application of our system in COVID-19 research, illustrating its utility in areas such as drug repurposing and literature curation.
Blackbird Language Matrices Tasks for Generalization

To develop a system with near-human language capabilities, we need to understand current systems’ generalisation and compositional abilities. We approach this by generating compositional, structured data, inspired from visual intelligence tests, that depend on the problem-solvers being able to disentangle objects and their absolute and relative properties in a sequence of images. We design an analogous task and develop the corresponding datasets that capture specific linguistic phenomena and their properties. Solving each problem instance depends on detecting the relevant linguistic objects and generative rules of the problem. We propose two datasets modelling two linguistic phenomena – subject-verb agreement in French, and verb alternations in English. The datasets can be used to investigate how LLMs encode linguistic objects, such as phrases, their grammatical and semantic properties, such as number or semantic role, and how such information is combined to correctly solve each problem. Specifically generated error types help investigate the behaviour of the system, which important information it is able to detect, and which structures mislead it.
MEE4 and XLsim : IIIT HYD’s Submissions’ for WMT23 Metrics Shared Task

This paper presents our contributions to the WMT2023 shared metrics task, consisting of two distinct evaluation approaches: a) Unsupervised Metric (MEE4) and b) Supervised Metric (XLSim). MEE4 represents an unsupervised, reference-based assessment metric that quantifies linguistic features, encompassing lexical, syntactic, semantic, morphological, and contextual similarities, leveraging embeddings. In contrast, XLsim is a supervised reference-based evaluation metric, employing a Siamese Architecture, which regresses on Direct Assessments (DA) from previous WMT News Translation shared tasks from 2017-2022. XLsim is trained using XLM-RoBERTa (base) on English-German reference and mt pairs with human scores.
ViGPTQA - State-of-the-Art LLMs for Vietnamese Question Answering: System Overview, Core Models Training, and Evaluations

Large language models (LLMs) and their applications in low-resource languages (such as in Vietnamese) are limited due to lack of training data and benchmarking datasets. This paper introduces a practical real-world implementation of a question answering system for Vietnamese, called ViGPTQA, leveraging the power of LLM. Since there is no effective LLM in Vietnamese to date, we also propose, evaluate, and open-source an instruction-tuned LLM for Vietnamese, named ViGPT. ViGPT demonstrates exceptional performances, especially on real-world scenarios. We curate a new set of benchmark datasets that encompass both AI and human-generated data, providing a comprehensive evaluation framework for Vietnamese LLMs. By achieving state-of-the-art results and approaching other multilingual LLMs, our instruction-tuned LLM underscores the need for dedicated Vietnamese-specific LLMs. Our open-source model supports customized and privacy-fulfilled Vietnamese language processing systems.
Empower Nested Boolean Logic via Self-Supervised Curriculum Learning

Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method Curriculum Logical Reasoning (Clr), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.
Learning the Visualness of Text Using Large Vision-Language Models

Visual text evokes an image in a person’s mind, while non-visual text fails to do so. A method to automatically detect visualness in text will enable text-to-image retrieval and generation models to augment text with relevant images. This is particularly challenging with long-form text as text-to-image generation and retrieval models are often triggered for text that is designed to be explicitly visual in nature, whereas long-form text could contain many non-visual sentences. To this end, we curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. We also propose a fine-tuning strategy that adapts large vision-language models like CLIP by modifying the model’s contrastive learning objective to map text identified as non-visual to a common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E.
Automating Behavioral Testing in Machine Translation

Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine Translation (MT) is currently restricted to largely handcrafted tests covering a limited range of capabilities and languages. To address this limitation, we propose to use Large Language Models (LLMs) to generate a diverse set of source sentences tailored to test the behavior of MT models in a range of situations. We can then verify whether the MT model exhibits the expected behavior through matching candidate sets that are also generated using LLMs. Our approach aims to make behavioral testing of MT systems practical while requiring only minimal human effort. In our experiments, we apply our proposed evaluation framework to assess multiple available MT systems, revealing that while in general pass-rates follow the trends observable from traditional accuracy-based metrics, our method was able to uncover several important differences and potential bugs that go unnoticed when relying only on accuracy.
A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data

This paper describes the AST submission to the WMT23 Shared Task on Parallel Data Curation. We experiment with two approaches for curating data from the provided web-scraped texts. We use sentence alignment to identify document alignments in the data and extract parallel sentence pairs from the aligned documents. All other sentences, not aligned in that step, are paired based on cosine similarity before we apply various different filters. For filtering, we use language detection, fluency classification, word alignments, cosine distance as calculated by multilingual sentence embedding models, and Bicleaner AI. Our best model outperforms the baseline by 1.9 BLEU points on average over the four provided evaluation sets.
Summarization-based Data Augmentation for Document Classification

Despite the prevalence of pretrained language models in natural language understanding tasks, understanding lengthy text such as document is still challenging due to the data sparseness problem. Inspired by that humans develop their ability of understanding lengthy text form reading shorter text, we propose a simple yet effective summarization-based data augmentation, SUMMaug, for document classification. We first obtain easy-to-learn examples for the target document classification task by summarizing the input of the original training examples, while optionally merging the original labels to conform to the summarized input. We then use the generated pseudo examples to perform curriculum learning. Experimental results on two datasets confirmed the advantage of our method compared to existing baseline methods in terms of robustness and accuracy. We release our code and data at https://github.com/etsurin/summaug.
nerblackbox: A High-level Library for Named Entity Recognition in Python

We present **nerblackbox**, a python library to facilitate the use of state-of-the-art transformer-based models for named entity recognition. It provides simple-to-use yet powerful methods to access data and models from a wide range of sources, for fully automated model training and evaluation as well as versatile model inference. While many technical challenges are solved and hidden from the user by default, **nerblackbox** also offers fine-grained control and a rich set of customizable features. It is thus targeted both at application-oriented developers as well as machine learning experts and researchers.
VER: Unifying Verbalizing Entities and Relations

Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, we need to understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning a new scientific term, people usually start by reading its definition in dictionaries or encyclopedias. To know the relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER: a unified model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates a sentence to represent entities and relations. Extensive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning.
A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports

Table of contents (ToC) extraction centres on structuring documents in a hierarchical manner. In this paper, we propose a new dataset, ESGDoc, comprising 1,093 ESG annual reports from 563 companies spanning from 2001 to 2022. These reports pose significant challenges due to their diverse structures and extensive length. To address these challenges, we propose a new framework for Toc extraction, consisting of three steps: (1) Constructing an initial tree of text blocks based on reading order and font sizes; (2) Modelling each tree node (or text block) independently by considering its contextual information captured in node-centric subtree; (3) Modifying the original tree by taking appropriate action on each tree node (Keep, Delete, or Move). This construction-modelling-modification (CMM) process offers several benefits. It eliminates the need for pairwise modelling of section headings as in previous approaches, making document segmentation practically feasible. By incorporating structured information, each section heading can leverage both local and long-distance context relevant to itself. Experimental results show that our approach outperforms the previous state-of-the-art baseline with a fraction of running time. Our framework proves its scalability by effectively handling documents of any length.
Conversing with databases: Practical Natural Language Querying

In this work, we designed, developed and released in production DataQue – a hybrid NLQ (Natural Language Querying) system for conversational DB querying. We address multiple practical problems that are not accounted for in public Text-to-SQL solutions – numerous complex implied conditions in user questions, jargon and abbreviations, custom calculations, non-SQL operations, a need to inject all those into pipeline fast and to have guaranteed parsing results for demanding users, cold-start problem. The DataQue processing pipeline for Text-to-SQL translation consists of 10-15 model-based and rule-based components that allows to tightly control the processing.
Supervising the Centroid Baseline for Extractive Multi-Document Summarization

The centroid method is a simple approach for extractive multi-document summarization and many improvements to its pipeline have been proposed. We further refine it by adding a beam search process to the sentence selection and also a centroid estimation attention model that leads to improved results. We demonstrate this in several multi-document summarization datasets, including in a multilingual scenario.
THE SECULAR CREATION OF COMPOSER GHEORGHE DANGA

Gheorghe Danga is a well-known composer in our church for his religious works. A large part of his creațion includes many secular and patriotic compozițions. This article aims to detail this aspect, starting with the compozitions of his teenage years, since he begun to compose since the age of 16-17. He composed many reference folklore works that are sung a lot in our country and even in othersat various festivals or choral competitions. Also, patriotic compozitions had a special place in his work, the most important being the „Choral suite 190”, the largest among his creațions, a work who captures the events of the peasant uprising of 1907. Of course, to be complete as a composer, he wrote some songs for children, and also instrumental and vocal-instrumental works, and choirs with orchestra. So I consider that it is absolutely needed that his entire compozitional work to be known, in order to have a complete image of the Composer Gheorghe Danga.
Online Accreditation Portal of North Luzon Philippines State College

The Coronavirus disease pandemic has accelerated the development and adoption of web-based systems. As higher education institutions were forced to adopt their operations online, schools needed a way to provide their clienteles with access to information and services. Web portals were an ideal solution, as they could be easily accessed from anywhere with an internet connection. These provide a centralized location for information and services, and they can help improve communication and collaboration. As North Luzon Philippines State College which is now Ilocos Sur Polytechnic State College – Main campus by virtue of the Republic Act 11755, continues to provide quality and affordable education, its academic programs undergo accreditation through the Accrediting Agency of Chartered Colleges and Universities in the Philippines (AACCUP). Since in-person accreditation processes are prohibited due to mobility restrictions to mitigate the risk of COVID-19 transmission, the institution must devise a way to still pursue its program accreditations. This study developed an Online Accreditation Portal, as a solution to this challenge. The developed system provides a secure environment for storing accreditation-related data, readily accessible to accreditors. The study analyzed the system’s level of acceptability which is evaluated by 16 respondents using the ISO/IEC 25010:2011 questionnaire. The system's overall acceptability is very high as indicated by a mean rating of 4.42, which makes the developed system potential for implementation for the institution’s accreditation activities. A campus-wide integration of the developed system was recommended to benefit all the other campuses of the institution.
Topography Effects on Rainfall Characteristics in Bandung City and Cilacap Regency for the 2016-2020 Period

Topography and rainfall correlate with each other. Therefore, this study aims to determine the effect of topography on rainfall characteristicss in the Cilacap and Bandung areas. The research method used is a descriptive statistical analysis of average monthly, seasonal, and annual rainfall. In addition, time series and regression tests were carried out using the SPSS application. The data used is the daily rainfall data period of 2016–2020. The results of the descriptive analysis show that the variability of monthly rainfall in Bandung is higher than in Cilacap, with a value range that is 113.6–643.82 mm/month for Bandung and 92.1-355.56 mm/month for Cilacap. Global climate dynamics for instance apparent sun motion, Monsoon, ITCZ, ENSO, and IOD, as well as local weather systems like valley winds, also affect seasonal and annual rainfall variability in Bandung and Cilacap. The rainfall time series test produces values that are not stationary concerning the variance but stationary concerning the average. A simple linear regression test shows the effect of a topography of 55.2% on rainfall. The purpose of simple linear regression is to understand and model the relationship between two variables: the independent variable (topography) and the dependent variable (rainfall characteristics).
Rancang Bangun Kursi Roda Bermotor Elektrik Dengan Sistem Pengendalian Gestur Untuk Penyandang Tetraplegic

Improving medical services is continuously prioritized for the sake of achieving the welfare of all people in the world, and by providing good services in the medical field, people's needs will be increasingly fulfilled. Limitations of movement in someone with a disability, such as persons with disabilities tetraplegic in terms of mobility, are often obstacles. Therefore, a media-assisting movement is needed. This study designed an electric wheelchair that can help users with disabilities tetraplegic in moving without the need for too much force as in a standard wheelchair. This electric wheelchair uses automation by utilizing Accelerometer, which can detect vectors or so-called gyroscopes and connect with Arduino as the processor is continued to the DC motor as the driving force. Based on the test data, it was found that the design of this wheelchair moves as desired. This can be seen from the change in speed of the driving motor to changes in the tilt of the accelerometer mounted on the head. In the forward motion test, the maximum motor speed is 35 RPM when the accelerometer angle is at a slope of 64.7o positive axes. Meanwhile, the backward movement of the wheelchair occurs when the accelerometer is at a slope of -21.1o and produces a speed of 11.2 RPM on wheels. The rider's maximum load is designed to be 50 Kg.
The Relationship between Job Stress on Organizational Effectiveness and Intention for Career Change from Elementary School Sports Instructors

The purpose of this study examined the analysis of relationship between job stress, organizational effectiveness and intention for career change of elementary school instructors. Through convenience sampling method, selected 301 survey questionnaires were used for final analysis. Data were statistically processed using SPSS WIN 24.0. First, there were differences in job stress, organizational effectiveness and intention for career change to the personal characteristics of elementary school sports instructors. Second, the job stress had negative effect on organizational effectiveness. Third, the parental stress had positive effect on intention for career change, the colleague and boss stress had negative effect on intention for career change. Fourth, the organizational effectiveness had negative effect on intention for career change. Fifth, the organizational effectiveness partial mediating effect relationship between job stress and intention for career change. These results suggest that the job stress of elementary school instructors is involved in organizational effectiveness and intention for career, the higher organizational effectiveness the lower intention for career change.