Miracle-dz/UniFAR-FTUs huggingface dataset:
Learning to Generate Questions by Learning to Recover Answer-containing Sentences

To train a question answering model based on machine reading comprehension (MRC), signiﬁcant effort is required to prepare annotated training data composed of questions and their answers from contexts. Recent research has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model to augment the training data. In light of this research direction, we propose a novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences. We evaluate our method against existing ones in terms of the quality of generated questions, and ﬁne-tuned MRC model accuracy after training on the data synthetically generated by our method. We consistently improve the question generation capability of existing models such as T5 and UniLM, and achieve state-of-the-art results on MS MARCO and NewsQA, and comparable results to the state-of-the-art on SQuAD. Additionally, the data synthetically generated by our approach is beneﬁcial for boosting up the downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, KorQuAD and BioASQ, without any modiﬁcation to the existing MRC models. Furthermore, our method shines especially when a limited amount of pre-training or downstream MRC data is given.
MergeDistill: Merging Language Models using Pre-trained Distillation

Pre-trained multilingual language models (LMs) have achieved state-of-the-art results in cross-lingual transfer, but they often lead to an inequitable representation of languages due to limited capacity, skewed pre-training data, and sub-optimal vocabularies. This has prompted the creation of an ever-growing pre-trained model universe, where each model is trained on large amounts of language or domain specific data with a carefully curated, linguistically informed vocabulary. However, doing so brings us back full circle and prevents one from leveraging the benefits of multilinguality. To address the gaps at both ends of the spectrum, we propose MergeDistill, a framework to merge pre-trained LMs in a way that can best leverage their assets with minimal dependencies, using task-agnostic knowledge distillation. We demonstrate the applicability of our framework in a practical setting by leveraging pre-existing teacher LMs and training student LMs that perform competitively with or even outperform teacher LMs trained on several orders of magnitude more data and with a fixed model capacity. We also highlight the importance of teacher selection and its impact on student model performance.
Insertion-based Tree Decoding

Sequences are typically decoded in a left-to-right fashion, requiring as many decoding steps as there are tokens in the sequence. Recently, several works have proposed non-autoregressive decoders that are sub-linear, allowing to decode a sequence using fewer decoding steps than the length of the sequence, and thus substantially speed up inference. In contrast, non-autoregressive decoding of trees is less well-analysed, even though trees are used in important applications like semantic parsing and code generation. In this work, we present a novel general-purpose partially autoregressive tree decoder that uses tree-based insertion operations to generate trees in sub-linear time. We evaluate our approach on semantic parsing and compare it against strong baselines, including an insertion-based sequence decoder. The results demonstrate that the partially autoregressive tree decoder reaches competitive accuracies while clearly reducing the number of decoding steps.
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification

With pre-trained models, such as BERT, gaining more and more attention, plenty of research has been done to further promote their capabilities, from enhancing the experimental procedures (Sun et al., 2019) to improving the mathematical principles. In this paper, we pro-pose a concise method for improving BERT’s performance in text classiﬁcation by utilizing a label embedding technique while keeping almost the same computational cost. Experimental results on six text classiﬁcation benchmark datasets demonstrate its effectiveness.
Eficiência de Telas Termorefletoras e de Sombreamento em Ambiente Protegido Tipo Telado sob Temperaturas Elevadas

EFFICIENCY OF TERMO-REFLECTIVE SCREENS AND SHADING IN PROTECTED ENVIRONMENTAL SUBMITTED TO HIGH TEMPERATURES Cultivation in protected environments allows for crop production in seasons that normally would not be suitable for open field production. This study sought to evaluate the variations in luminosity, air and soil temperature and relative humidity of the air in different environmental conditions during the winter and summer, in Cáceres-MT, using screens with mesh of 30, 40 and 50% shading, thermo-reflective screens with 30, 40 and 50% shading and an open field. The experimental design utilized consisted of randomized blocks with 10 replications, represented by 10 days of collection. It was observed that the screened environments reduced the minimum, average and maximum values of the luminosity and air temperature attributes, and resulted in an increase in relative humidity compared to the open field environment. The black screens with 40% and 50% shading and the 50% thermo-reflective screen presented greatest efficiency to reduce luminosity and air temperature. The screen with 50% shading was that which resulted in the greatest indices of relative humidity of the air among the treatments.
Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization

Abstractive multi-document summarization aims to generate a comprehensive summary covering salient content from multiple input documents. Compared with previous RNN-based models, the Transformer-based models employ the self-attention mechanism to capture the dependencies in input documents and can generate better summaries. Existing works have not considered key phrases in determining attention weights of self-attention. Consequently, some of the tokens within key phrases only receive small attention weights. It can affect completely encoding key phrases that convey the salient ideas of input documents. In this paper, we introduce the Highlight-Transformer, a model with the highlighting mechanism in the encoder to assign greater attention weights for the tokens within key phrases. We propose two structures of high-lighting attention for each head and the multi-head highlighting attention. The experimental results on the Multi-News dataset show that our proposed model signiﬁcantly outperforms the competitive baseline models.
Transformer-Exclusive Cross-Modal Representation for Vision and Language

Ever since the advent of deep learning, cross-modal representation learning has been dominated by the approaches involving convolutional neural networks for visual representation and recurrent neural networks for language representation. Transformer architecture, however, has rapidly taken over the recurrent neural networks in natural language processing tasks, and it has also been shown that vision tasks can be handled with trans-former architecture, with compatible performance to convolutional neural networks. Such results naturally lead to speculation upon the possibility of tackling cross-modal representation for vision and language exclusively with transformer. This paper examines transformer-exclusive cross-modal representation to explore such possibility, demonstrating its potentials as well as discussing its current limitations and its prospects.
Unsupervised Domain Extension for Nighttime Semantic Segmentation in Urban Scenes

: This paper deals with the problem of semantic image segmentation of street scenes at night, as the recent advances in semantic image segmentation are mainly related to daytime images. We propose a method to extend the learned domain of daytime images to nighttime images based on an extended version of the CycleGAN framework and its integration into a self-supervised learning framework. The aim of the method is to reduce the cost of human annotation of night images by robustly transferring images from day to night and training the segmentation network to make consistent predictions in both domains, allowing the usage of completely unlabelled images in training. Experiments show that our approach signiﬁcantly improves the performance on nighttime images while keeping the performance on daytime images stable. Furthermore, our method can be applied to many other problem formulations and is not speciﬁcally designed for semantic segmentation.
Is the Lottery Fair? Evaluating Winning Tickets Across Demographics

Recent studies have suggested that weight pruning, e.g. using lottery ticket extraction techniques (Frankle and Carbin, 2018), comes at the risk of compromising the group fairness of machine learning models (Paganini, 2020; Hooker et al., 2020), but to the best of our knowledge, no one has empirically evaluated this hypothesis at scale in the context of natural language processing. We present experiments with two text classiﬁcation datasets annotated with demographic information: the Trustpilot Corpus (sentiment) and CivilComments (toxicity). We evaluate the fairness of lottery ticket extraction through layer-wise and global weight pruning across three lan-guages and two tasks. Our results suggest that there is a small increase in group disparity, which is most pronounced at high pruning rates and correlates with instability. The fairness of models trained with distributionally robust optimization objectives is sometimes less sensitive to pruning, but results are not consistent. The code for our experiments is available at https://github. com/vpetren/fairness_lottery .
Counter-Argument Generation by Attacking Weak Premises

Text generation has received a lot of attention in computational argumentation research as of recent. A particularly challenging task is the generation of counter-arguments. So far, approaches primarily focus on rebutting a given conclusion, yet other ways to counter an argument exist. In this work, we go beyond previous research by exploring argument undermining, that is, countering an argument by attacking one of its premises. We hypothesize that identifying the argument's weak premises is key to effective countering. Accordingly, we propose a pipeline approach that first assesses the premises' strength and then generates a counter-argument targeting the weak ones. On the one hand, both manual and automatic evaluation proves the importance of identifying weak premises in counter-argument generation. On the other hand, when considering correctness and content richness, human annotators favored our approach over state-of-the-art counter-argument generation.
Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder

Recent work has proposed several efficient approaches for generating gradient-based adversarial perturbations on embeddings and proved that the model's performance and robustness can be improved when they are trained with these contaminated embeddings. While they paid little attention to how to help the model to learn these adversarial samples more efficiently. In this work, we focus on enhancing the model's ability to defend gradient-based adversarial attack during the model's training process and propose two novel adversarial training approaches: (1) CARL narrows the original sample and its adversarial sample in the representation space while enlarging their distance from different labeled samples. (2) RAR forces the model to reconstruct the original sample from its adversarial representation. Experiments show that the proposed two approaches outperform strong baselines on various text classification datasets. Analysis experiments find that when using our approaches, the semantic representation of the input sentence won't be significantly affected by adversarial perturbations, and the model's performance drops less under adversarial attack. That is to say, our approaches can effectively improve the robustness of the model. Besides, RAR can also be used to generate text-form adversarial samples.
HacRED: A Large-Scale Relation Extraction Dataset Toward Hard Cases in Practical Applications

Relation extraction (RE) is an essential topic in natural language processing and has at-tracted extensive attention. Current RE approaches achieve fantastic results on common datasets, while they still struggle on practical applications. In this paper, we analyze the above performance gap, the underlying reason of which is that practical applications intrinsically have more hard cases. To make RE models more robust on such practical hard cases, we propose a case-oriented construction framework to build a H ard C ase R elation E xtraction D ataset ( HacRED ). The proposed HacRED consists of 65,225 relational facts annotated from 9,231 documents with sufficient and diverse hard cases. Notably, HacRED is one of the largest Chinese document-level RE datasets and achieves a high 96% F1 score on data quality. Furthermore, we apply the state-of-the-art RE models on this dataset and conduct a thorough evaluation. The results show that the performance of these models is far lower than humans, and RE applying on practical hard cases still requires further efforts. Ha-cRED is publicly available
A Comparative Analysis of Classic and Deep Learning Models for Inferring Gender and Age of Twitter Users

In order for social scientists to use social media as a source for understanding human behavior and public opinion, they need to understand the demographic characteristics of the population participating in the conversation. What proportion are female? What proportion are young? While previous literature has investigated this problem, this work presents a larger scale study that investigates inference techniques for predicting age and gender using Twitter data. We consider classic text features used in previous work and introduce new ones. Then we use a range of learning approaches from classic machine learning models to deep learning ones to understand the role of different language representations for demographic inference. On a data set created from Wikidata, we compare the value of different feature sets with different algorithms. In general, we find that classic models using statistical features and unigrams perform well. Neural networks also perform well, particularly models using sentence embeddings, e.g. a Siamese network configuration with attention to tweets and user biographies. The differences are marginal for age, but more significant for gender. In other words, it is reasonable to use simpler, interpretable models for some demographic inference tasks (like age). However, using richer language model is important for gender, highlighting the varying role language plays for demographic inference on social media.
A Multi-Task Learning Framework for Multi-Target Stance Detection

Multi-target stance detection aims to identify the stance taken toward a pair of different targets from the same text, and typically, there are multiple target pairs per dataset. Existing works generally train one model for each target pair. However, they fail to learn target-speciﬁc representations and are prone to overﬁtting. In this paper, we propose a new training strategy under the multi-task learning setting by training one model on all target pairs, which helps the model learn more universal representations and alleviate overﬁtting. Moreover, in order to extract more accurate target-speciﬁc representations, we propose a multi-task learning network which can jointly train our model with a stance (dis)agreement detection task that is designed to identify agreement and disagreement between stances in paired texts. Experimental results demonstrate that our proposed model outperforms the best-performing baseline by 12.39% in macro-averaged F1-score. Our resources are publicly available on GitHub. 1
LUX (Linguistic aspects Under eXamination): Discourse Analysis for Automatic Fake News Classification

The democratization/decentralization of both the production and consumption of information has resulted in a subjective and often misleading depiction of facts known as Fake News - a phenomenon that is effectively shaping the perception of reality for many individuals. Manual fact-checking is time-consuming and cannot scale and although automatic fact-checking, vis a vis machine learning holds promise, it is signiﬁcantly hindered by a deﬁcit of suitable training data. We present both a novel dataset, VERITAS(VERIfying Textual Aspects), a collection of fact-checked claims, containing their original documents and LUX(Language Under eXamination), a text classiﬁer that makes use of an extensive linguistic analysis to infer the likelihood of the input being a piece of fake-news.
Hypernym Discovery via a Recurrent Mapping Model

Hypernym discovery aims to identify all possible hypernyms of a given term. The most recent hypernym discovery models exploit multiple mapping functions to project a term to different semantic spaces and then aggregate these embeddings to a general representation for further classiﬁcation. We refer to this model as a parallel style model. In this work, we observe that there are hierarchical relations between a target terms’ hypernyms. However, these hierarchical relations were not suf-ﬁciently considered in the previous parallel style model. To leverage the hierarchical relations, we propose a sequential style model that recurrently maps the query words to their hy-pernyms, starting from the most speciﬁc ones to the less speciﬁc ones. Empirical studies on SemEval-2018 Task 9 conﬁrm the effectiveness of the presented model.
The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English

Recent papers have introduced methods to incorporate gazetteer features and entity segmentation techniques in neural named entity recognition models. These papers rely on different resources and include features not related to the use of gazetteers, rendering impossible the comparison of the relative effectiveness of the approaches. Here, we provide a comprehensive overview of methods for incorporating gazetteers and for entity segmentation. We evaluate representative methods from each in similar settings for a fair comparison and identify the ones that are consistently better across datasets and input representations. We further show that gazetteers improve entity segmentation and not just entity typing. Hence, we explore their utility in recognizing long entities, a problem for which entity segmentation techniques were developed. Our work explains the mechanisms via which gazetteers improve the performance of neural NER models.
Evolution of the Political Agenda during the COVID-19 Pandemic: Analysis of the Seimas Debates Transcripts

This paper analyses debates of the Lithuanian Parliament (the Seimas) during the first and the second wave of the COVID-19 pandemic (covering the period from January 2020 to January 2021), attempting to establish how the previous and the newly elected Parliament discussed certain aspects of the pandemic and its containment. For this purpose, the authors automatically extracted the transcripts of all the debates that took place in the Seimas during the period under study (https://e-seimas.lrs.lt/portal/documentSearch/lt) and devised a content analytic dictionary covering two broad themes: "COVID-19 pandemic"and "human rights. "In addition, the first theme was divided into subtopics covering: 1) general references to the pandemic, 2) references to public life restrictions, 3) references to medical measures of fighting the disease, and 4) references to distance learning/working. The results after an automatic and expert analysis showed that the COVID-19 pandemic outbreak and mass protests after the presidential elections in Belarus resonated strongly on the Lithuanian parliament's agenda, as substantial increases of search terms related to "pandemic"and 'human rights"topics were detected in the floor debates. On the more specific level, the study revealed that the Parliament's debate during the first wave of the COVID-19 pandemic was dominated by the subtopic of "public life restrictions"and the subtopic "pandemic in general"was at the centre of Parliament's interest during the second wave of the COVID-19 pandemic. © 2021 Copyright for this paper by its authors.
UAV Inspection of Large Components: Indoor Navigation Relative to Structures

: The inspection of large structures is increasingly carried out with the help of Unmanned Aerial Vehicles (UAVs). When navigating relative to the structure, multiple data sources can be used to determine the position of the UAV. Examples include track data from an installed camera and sensor data from the orientation sensors of the UAV. This paper deals with the fusion of this data and its use for navigation alongside the structure. For the sensor fusion, a concept is developed using a Kalman ﬁlter and evaluated simulatively in a prototype. The calculated position data are also fed into a vector ﬂight control system, which dynamically calculates and ﬂies a trajectory along the component using the potential ﬁeld method. This is done taking into account obstacles detected by the onboard sensors of the UAV. The established concept is then implemented with the Robot Operating System (ROS) and evaluated simulatively.
Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations

While research on explaining predictions of open-domain QA systems (ODQA) is gaining momentum, most works do not evaluate whether these explanations improve user trust. Furthermore, many users interact with ODQA using voice -assistants, yet prior works exclusively focus on visual displays, risking (as we also show) incorrectly extrapolating the effectiveness of explanations across modalities. To better understand the effectiveness of ODQA explanations strategies in the wild, we conduct user studies that measure whether explanations help users correctly decide when to accept or reject an ODQA system’s answer. Unlike prior work, we control for explanation modality , i.e. , whether they are communicated to users through a spoken or visual interface, and contrast effectiveness across modalities. We show that explanations derived from retrieved evidence can outperform strong base-lines across modalities but the best explanation strategy varies with the modality. We show common failure cases of current explanations, emphasize end-to-end evaluation of explanations, and caution against evaluating them in proxy modalities that differ from deployment.