Reviewerly/exHarmony huggingface dataset:
A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Human Activity Recognition (HAR) is the process of automatic recognition of Activities of Daily Living (ADL) from human motion data captured in various data modalities by wearable and ambient sensors. Advances in Deep Learning, especially Convolutional Neural Networks (CNNs) and sequential models have revolutionalized HAR from video data sources. Although these models are incredibly accurate and utilize both spatial and temporal information, they require huge computation and memory resources — making them unsuitable for edge or wearable applications. Tiny Machine Learning (TinyML) aims to optimize these models in terms of compute and memory requirements – aiming to make them suitable for always-on resource constrained devices – leading to a reduction in communication latency and network traffic for HAR frameworks. In this paper, we propose a two-stream multi-resolution fusion architecture for HAR from video data modality. The context stream takes a resized image as input and the fovea stream takes the cropped center portion of the resized image as input, reducing the overall dimensionality. We tested two quantization methods: Post-Training Quantization (PTQ) and Quantization Aware Training (QAT) to optimize these models for deployment in edge devices and tested the performance in two challenging video datasets: KTH and UCF11. We performed ablation studies to validate the two-stream model performance. We deployed the proposed architecture in commercial resource constrained devices and monitored their performance in terms of inference latency and power consumption. The results indicate that the proposed architecture clearly outperforms other relevant single-stream models tested in this work in terms of accuracy, precision, recall, and F1-Score while also reducing the overall model size.
Innovating Industrial Training with Immersive Metaverses: A Method for Developing Cross-Platform Virtual Reality Environments

The metaverse has garnered significant attention for its potential to provide engaging and social experiences in virtual reality. Despite substantial investment and interest from industry, there remains a lack of academic research on the development and implementation of metaverses for industrial training. Notably, research indicates that virtual reality training is, on average, four times faster than classroom-based training. This study proposes a method for developing immersive metaverses for industrial training, leveraging specialized tools like Epic Games’ Unreal Engine software version 4.27.2. To assess the efficacy of this method, a cross-platform metaverse was developed, and a questionnaire was administered to game developers. The results indicate that even junior developers and those with limited experience can comprehend the method, suggesting that it is possible to develop immersive virtual worlds with an emphasis on professional training even without prior experience with 3D modeling or third-party licensing.
Mawqif: A Multi-label Arabic Dataset for Target-specific Stance Detection

Social media platforms are becoming inherent parts of people's daily life to express opinions and stances toward topics of varying polarities.Stance detection determines the viewpoint expressed in a text toward a target.While communication on social media (e.g., Twitter) takes place in more than 40 languages, the majority of stance detection research has been focused on English.Although some efforts have recently been made to develop stance detection datasets in other languages, no similar efforts seem to have considered the Arabic language.In this paper, we present MAWQIF, the first Arabic dataset for target-specific stance detection, composed of 4,121 tweets annotated with stance, sentiment, and sarcasm polarities.MAWQIF, as a multi-label dataset, can provide more opportunities for studying the interaction between different opinion dimensions and evaluating a multi-task model.We provide a detailed description of the dataset, present an analysis of the produced annotation, and evaluate four BERT-based models on it.Our best model achieves a macro-F 1 of 78.89%, which shows that there is ample room for improvement on this challenging task.We publicly release our dataset, the annotation guidelines, and the code of the experiments. 1
Distilled Dual-Encoder Model for Vision-Language Understanding

On vision-language understanding (VLU) tasks, fusion-encoder vision-language models achieve superior results but sacrifice efficiency because of the simultaneous encoding of images and text.On the contrary, the dual-encoder model that separately encodes images and text has the advantage in efficiency, while failing on VLU tasks due to the lack of deep cross-modal interactions.To get the best of both worlds, we propose DIDE 1 , a framework that distills the knowledge of the fusion-encoder teacher model into the dual-encoder student model.Since the cross-modal interaction is the key to the superior performance of teacher model but is absent in the student model, we encourage the student not only to mimic the predictions of teacher, but also to calculate the cross-modal attention distributions and align with the teacher.Experimental results demonstrate that DIDE is competitive with the fusion-encoder teacher model in performance (only a 1% drop) while enjoying 4× faster inference.Further analyses reveal that the proposed cross-modal attention distillation is crucial to the success of our framework.
Making Language Models Better Reasoners with Step-Aware Verifier

Few-shot learning is a challenging task that requires language models to generalize from limited examples.Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems.To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate.In this paper, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models.DIVERSE has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain.We evaluate DIVERSE on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% → 83.2%).
Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better

While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out that without artificially encouraging models to hallucinate, previously existing methods fall short and even the standard sequence log-probability is more informative. It means that internal characteristics of the model can give much more information than we expect, and before using external models and measures, we first need to ask: how far can we go if we use nothing but the translation model itself ? We propose to use a method that evaluates the percentage of the source contribution to a generated translation. Intuitively, hallucinations are translations "detached" from the source, hence they can be identified by low source contribution. This method improves detection accuracy for the most severe hallucinations by a factor of 2 and is able to alleviate hallucinations at test time on par with the previous best approach that relies on external models. Next, if we move away from internal model characteristics and allow external tools, we show that using sentence similarity from cross-lingual embeddings further improves these results. We release the code of our experiments.
Faithfulness Tests for Natural Language Explanations

Explanations of neural models aim to reveal a model's decision-making process for its predictions.However, recent work shows that current methods giving explanations such as saliency maps or counterfactuals can be misleading, as they are prone to present reasons that are unfaithful to the model's inner workings.This work explores the challenging question of evaluating the faithfulness of natural language explanations (NLEs).To this end, we present two tests.First, we propose a counterfactual input editor for inserting reasons that lead to counterfactual predictions but are not reflected by the NLEs.Second, we reconstruct inputs from the reasons stated in the generated NLEs and check how often they lead to the same predictions.Our tests can evaluate emerging NLE models, proving a fundamental tool in the development of faithful NLEs.Test Original Instance Instance After Test Intervention Counterfactual test ( §2) Premise: Man in a black suit, white shirt and black bowtie playing an instrument with the rest of his symphony surrounding him.Hypothesis: A tall person in a suit.Prediction: neutral NLE: Not all men are tall.Premise: Man in a black suit, white shirt and black bowtie
MetaASSIST: Robust Dialogue State Tracking with Meta Learning

Existing dialogue datasets contain lots of noise in their state annotations.Such noise can hurt model training and ultimately lead to poor generalization performance.A general framework named ASSIST has recently been proposed to train robust dialogue state tracking (DST) models.It introduces an auxiliary model to generate pseudo labels for the noisy training set.These pseudo labels are combined with vanilla labels by a common fixed weighting parameter to train the primary DST model.Notwithstanding the improvements of ASSIST on DST, tuning the weighting parameter is challenging.Moreover, a single parameter shared by all slots and all instances may be suboptimal.To overcome these limitations, we propose a meta learning-based framework MetaASSIST to adaptively learn the weighting parameter.Specifically, we propose three schemes with varying degrees of flexibility, ranging from slot-wise to both slot-wise and instance-wise, to convert the weighting parameter into learnable functions.These functions are trained in a meta-learning manner by taking the validation set as meta data.Experimental results demonstrate that all three schemes can achieve competitive performance.Most impressively, we achieve a state-of-the-art joint goal accuracy of 80.10% on MultiWOZ 2.4.
OpenCQA: Open-ended Question Answering with Charts

Charts are very popular to analyze data and convey important insights. People often analyze visualizations to answer open-ended questions that require explanatory answers. Answering such questions are often difficult and time-consuming as it requires a lot of cognitive and perceptual efforts. To address this challenge, we introduce a new task called OpenCQA, where the goal is to answer an open-ended question about a chart with descriptive texts. We present the annotation process and an in-depth analysis of our dataset. We implement and evaluate a set of baselines under three practical settings. In the first setting, a chart and the accompanying article is provided as input to the model. The second setting provides only the relevant paragraph(s) to the chart instead of the entire article, whereas the third setting requires the model to generate an answer solely based on the chart. Our analysis of the results show that the top performing models generally produce fluent and coherent text while they struggle to perform complex logical and arithmetic reasoning.
Label-Driven Denoising Framework for Multi-Label Few-Shot Aspect Category Detection

Multi-Label Few-Shot Aspect Category Detection (FS-ACD) is a new sub-task of aspectbased sentiment analysis, which aims to detect aspect categories accurately with limited training instances.Recently, dominant works use the prototypical network to accomplish this task, and employ the attention mechanism to extract keywords of aspect category from the sentences to produce the prototype for each aspect.However, they still suffer from serious noise problems: (1) due to lack of sufficient supervised data, the previous methods easily catch noisy words irrelevant to the current aspect category, which largely affects the quality of the generated prototype; (2) the semanticallyclose aspect categories usually generate similar prototypes, which are mutually noisy and confuse the classifier seriously.In this paper, we resort to the label information of each aspect to tackle the above problems, along with proposing a novel Label-Driven Denoising Framework (LDF).Extensive experimental results show that our framework achieves better performance than other state-of-the-art methods.
Knowledge-Rich Self-Supervision for Biomedical Entity Linking

Entity linking faces significant challenges such as prolific variations and prevalent ambiguities, especially in high-value domains with myriad entities.Standard classification approaches suffer from the annotation bottleneck and cannot effectively handle unseen entities.Zero-shot entity linking has emerged as a promising direction for generalizing to new entities, but it still requires example gold entity mentions during training and canonical descriptions for all entities, both of which are rarely available outside of Wikipedia.In this paper, we explore Knowledge-RIch Self-Supervision (KRISS) for biomedical entity linking, by leveraging readily available domain knowledge.In training, it generates self-supervised mention examples on unlabeled text using a domain ontology and trains a contextual encoder using contrastive learning.For inference, it samples self-supervised mentions as prototypes for each entity and conducts linking by mapping the test mention to the most similar prototype.Our approach can easily incorporate entity descriptions and gold mention labels if available.We conducted extensive experiments on seven standard datasets spanning biomedical literature and clinical notes.Without using any labeled information, our method produces KRISSBERT, a universal entity linker for over three million UMLS entities that attains new state of the art, outperforming prior self-supervised methods by as much as 20 absolute points in accuracy.We released KRISSBERT at https://aka.ms/krissbert.
Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Hateful memes are a growing menace on social media.While the image and its corresponding text in a meme are related, they do not necessarily convey the same meaning when viewed individually.Hence, detecting hateful memes requires careful consideration of both visual and textual information.Multimodal pretraining can be beneficial for this task because it effectively captures the relationship between the image and the text by representing them in a similar feature space.Furthermore, it is essential to model the interactions between the image and text features through intermediate fusion.Most existing methods either employ multimodal pre-training or intermediate fusion, but not both.In this work, we propose the Hate-CLIPper architecture, which explicitly models the cross-modal interactions between the image and text representations obtained using Contrastive Language-Image Pre-training (CLIP) encoders via a feature interaction matrix (FIM).A simple classifier based on the FIM representation is able to achieve state-of-the-art performance on the Hateful Memes Challenge (HMC) dataset with an AUROC of 85.8, which even surpasses the human performance of 82.65.Experiments on other meme datasets such as Propaganda Memes and TamilMemes also demonstrate the generalizability of the proposed approach.Finally, we analyze the interpretability of the FIM representation and show that cross-modal interactions can indeed facilitate the learning of meaningful concepts.The code for this work is available at https://github.com/gokulkarthik/hateclipper.
Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer

In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step).However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens' attributions.Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions.In this work, we propose an interpretability method that tracks input tokens' attributions for both contexts.Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models.We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.
Structure-Unified M-Tree Coding Solver for Math Word Problem

As one of the challenging NLP tasks, designing math word problem (MWP) solvers has attracted increasing research attention for the past few years.In previous work, models designed by taking into account the properties of the binary tree structure of mathematical expressions at the output side have achieved better performance.However, the expressions corresponding to a MWP are often diverse (e.g., n 1 +n 2 ×n 3 -n 4 , n 3 ×n 2 -n 4 +n 1 , etc.), and so are the corresponding binary trees, which creates difficulties in model learning due to the non-deterministic output space.In this paper, we propose the Structure-Unified M-Tree Coding Solver (SUMC-Solver), which applies a tree with any M branches (M-tree) to unify the output structures.To learn the M-tree, we use a mapping to convert the M-tree into the M-tree codes, where codes store the information of the paths from tree root to leaf nodes and the information of leaf nodes themselves, and then devise a Sequence-to-Code (seq2code) model to generate the codes.Experimental results on the widely used MAWPS and Math23K datasets have demonstrated that SUMC-Solver not only outperforms several state-of-the-art models under similar experimental settings but also performs much better under low-resource conditions 1 .
Competence Development in an Undergraduate Physiotherapy Internship Program during the COVID-19 Pandemic: A Blended Learning Approach

Introduction: The COVID-19 pandemic has led to the creation of different teaching adaptations shared in health degrees, such as blended teaching methods, including face-to-face learning combined with online learning. Objectives: One objective was to compare the academic performance scores aligned to worked competencies during physiotherapy internships between two groups: one exposed to a blended learning (b-learning) educational model and another exposed to a face-to-face teaching system during internships. Another objective was to study the correlation of the marks of the following year’s internship subject. Design: Analytical study of retrospective cohorts. Settings: Physiotherapy university degree program. Participants: Three hundred students working towards attaining Physiotherapy degrees. Methods: Two groups were studied: the Non-Exposed Cohort, which had 100% face-to-face attendance at the clinical center, and the Exposed Cohort, which experienced b-learning and had 50% attendance at the clinical practice center and 50% completion of asynchronous online complementary training during lockdown. Results: The results show that both in the Exposed Cohort and in the Non-Exposed Cohort, the qualifications and, therefore, the competence development achieved by the students were similar. A stronger correlation with skills activities developed in the Exposed Cohort in relation to the final mark of the following year’s internship subject has been found. Discussion and Conclusions: For all the competencies developed during CS II, participants obtained the same grades in both groups, so it is concluded that the online activities implemented during confinement mixed with face-to-face teaching were useful for the achievement of these skills. Thus, b-learning is a good method for developing clinical practice competencies in physiotherapy students. Contribution of the Paper: Blended teaching (b-teaching) is useful for developing internship competencies. It is necessary to align the activities developed in internships with the evaluated skills.
Anamorphic Signatures: Secrecy from a Dictator Who Only Permits Authentication!

The goal of this research is to raise technical doubts regarding the usefulness of the repeated attempts by governments to curb Cryptography (aka the "Crypto Wars"), and argue that they, in fact, cause more damage than adding effective control. The notion of Anamorphic Encryption was presented in Eurocrypt'22 for a similar aim. There, despite the presence of a Dictator who possesses all keys and knows all messages, parties can arrange a hidden "anamorphic" message in an otherwise indistinguishable from regular ciphertexts (wrt the Dictator). In this work, we postulate a stronger cryptographic control setting where encryption does not exist (or is neutralized) since all communication is passed through the Dictator in, essentially, cleartext mode (or otherwise, when secure channels to and from the Dictator are the only confidentiality mechanism). Messages are only authenticated to assure recipients of the identity of the sender. We ask whether security against the Dictator still exists, even under such a strict regime which allows only authentication (i.e., authenticated/ signed messages) to pass end-to-end, and where received messages are determined by/ known to the Dictator, and the Dictator also eventually gets all keys to verify compliance of past signing. To frustrate the Dictator, this authenticated message setting gives rise to the possible notion of anamorphic channels inside signature and authentication schemes, where parties attempt to send undetectable secure messages (or other values) using signature tags which are indistinguishable from regular tags. We define and present implementation of schemes for anamorphic signature and authentication; these are applicable to existing and standardized signature and authentication schemes which were designed independently of the notion of anamorphic messages. Further, some cornerstone constructions of the foundations of signatures, in fact, introduce anamorphism. The extended version of this paper, including extra results and proofs, is available as [17].