Reviewerly/exHarmony huggingface dataset:
Dense Attention: A Densely Connected Attention Mechanism for Vision Transformer

Recently, Vision Transformer has demonstrated its impressive capability in image understanding. The multi-head self-attention mechanism is fundamental to its formidable performance. However, self-attention has the drawback of high computational effort, which makes the training of the model require powerful computational resources or more time. This paper designs a novel and efficient attention mechanism Dense Attention to overcome the above problem. Dense attention aims to focus on features from multiple views through a dense connection paradigm. Benefiting from the attention of comprehensive features, dense attention can i) remarkably strengthen the image representation of the model, and ii) partially replace the multi-head self-attention mechanism to allow model slimming. To verify the effectiveness of dense attention, we implement it in the prevalent Vision Transformer models, including non-pyramid architecture DeiT and pyramid architecture Swin Transformer. The experimental results on ImageNet classification show that dense attention indeed contributes to performance improvement, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$+\mathbf{1.8}/\mathbf{1.3}\%$</tex> for DeiT-T/s and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$+\mathbf{0.7}/\!\!+\mathbf{1.2}\%$</tex> for Swin-T/s, respectively. Dense attention also demonstrates its transferability on CIFAR10 and CIFAR100 recognition benchmarks with classification accuracy of 98.9% and 89.6% respectively. Furthermore, dense attention can weaken the performance sacrifice caused by the pruning in the number of heads. Code and pre-trained models will be available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/koala719/Dense-ViT.
MFGCL: Light graph contrast learning with multi-feature information for recommendation

Collaborative filtering(CF) uses user history data to predict user preferences and recommend items that users may be interested in. Although there are many ways to use graph convolutional networks to extend traditional CF, effective information cannot be used to fully express user preferences. At the same time, the sparse interaction data between users and items in the scenario is also an important factors that affect the recommendation effect. To address the above issues, we propose a light multi-feature information-enhanced graph contrastive learning model for recommendation scenarios (MFGCL). Based on LightGCN, when propagating neighborhood information, it not only propagates the interaction information between users and items but also learns the relationship between the user and item features. This aggregation method more comprehensively represents user preferences. Secondly, to solve the problem of data sparsity, a three-channel self-supervised learning framework is constructed to generate comparative views by adding uniform noise and extracting effective information from sparse data to the greatest extent, which can improve the expressiveness and scalability of the model. Experimental results on three public datasets demonstrate the effectiveness of the proposed method, which promotes the precision and accuracy of recommendations greatly. The code can be available at https://github.com/mintZYJ/MFGCL
Meta-Learning from Multimodal Task Distributions Using Multiple Sets of Meta-Parameters

Meta-Iearning or learning to learn involves training a model on various learning tasks in a way that allows it to quickly learn new tasks from the same distribution using only a small amount of training data (i.e., few-shot learning). Current meta-learning methods implicitly assume that the distribution over tasks is unimodal and consists of tasks belonging to a common domain, which significantly reduces the variety of task distributions they can handle. However, in real-world applications, tasks are often very diverse and come from multiple different domains, making it challenging to meta-learn common knowledge shared across the entire task distribution. In this paper, we propose a method for meta-learning from a multimodal task distribution. The proposed method learns multiple sets of meta-parameters (acting as different initializations of a neural network model) and uses a task encoder to select the best initialization to fine-tune for a new task. More specifically, with a few training examples from a task sampled from an unknown mode, the proposed method predicts which set of meta-parameters (i.e., model's initialization) would lead to a fast adaptation and a good post-adaptation performance on that task. We evaluate the proposed method on a diverse set of few-shot regression and image classification tasks. The results demonstrate the superiority of the proposed method compared to other state-of-the-art meta-learning methods and the benefit of learning multiple model initializations when tasks are sampled from a multimodal task distribution.
Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) is a promising paradigm to address the exploration-exploitation dilemma in reinforcement learning. It decomposes the source task into sub goal conditional subtasks and conducts exploration and exploitation in the subgoal space. The effectiveness of GCHRL heavily relies on sub goal representation functions and sub goal selection strategy. However, existing works often overlook the temporal coherence in GCHRL when learning latent sub goal representations and lack an efficient sub goal selection strategy that balances exploration and exploitation. This paper proposes HIerarchical reinforcement learning via dynamically building Latent Landmark graphs (HILL) to overcome these limitations. HILL learns latent subgoal representations that satisfy temporal coherence using a contrastive representation learning objective. Based on these representations, HILL dynamically builds latent landmark graphs and employs a novelty measure on nodes and a utility measure on edges. Finally, HILL develops a subgoal selection strategy that balances exploration and exploitation by jointly considering both measures. Experimental results demonstrate that HILL outperforms state-of-the-art baselines on continuous control tasks with sparse rewards in sample efficiency and asymptotic performance. Our code is available at https://github.com/papercode2022/HILL.
A Deep News Headline Generation Model with REINFORCE Filter

Generating accurate and concise headlines based on news content can help people filter out interesting content and improve the quality of life. News headline generation, as the sub-application area of text summarization, has many methodological commonalities but with higher requirements for generated text quality, which is more challenging. In this paper, we propose a new model to generate news headline, named RADGen (REIN-FORCE Aided Deep Generator for news headline), which combines a Transformer-based generator with a sentence-selecting filter based on REINFORCE algorithm. We perform experiments and assessments on Chinese news headline dataset, which achieve 25.71, 8.26 and 23.58 for f-score of ROUGE-1, ROUGE-2 and ROUGE-L respectively, to demonstrate the effectiveness of the proposed model. In addition, ablation experiments show the positive roll of reinforcement learning filter in news headline generation.
Modeling Global-Local Subtopic Distribution with Hypergraph to Diversify Search Results

Search result diversification aims to balance the relevance and diversity of retrieved documents to satisfy the different information needs of users. Three types of approaches have proliferated: explicit models that are based on explicit features (e.g., subtopic coverage), implicit models that are based on implicit features (e.g., the novelty of documents), and ensemble models that utilize both implicit and explicit features. However, the subtopics used by most explicit and ensemble models are usu-ally mined from queries (e.g., using Google Search Suggestions), which may not match the subtopics covered by the candidate documents. Besides, the implicit features used by most implicit models are formulated as either the similarity of documents or the intent of documents. The former cannot directly reflect the relationships of documents at the subtopic level, while the latter cannot capture non-pairwise relationships among documents. To tackle these issues, we propose a novel model that dynamically mines subtopics from the candidate documents and leverages thehypergraph structure to model the diversity of candidate documents, named HGDIV. Specifically, we dynamically mine subtopics from the candidate documents, rather than mining subtopics from queries or using static subtopics as existing methods do. More importantly, we introduce the hypergraph structure to model the diversity of candidate documents for search result diversification, which can capture the non-pairwise relationships among documents. Furthermore, we innovatively model the global and local subtopic distributions to extract the diversity of candidate documents. Experimental results on the public diversity benchmark TREC datasets demonstrate the superiority of our model over state-of-the-art models.
Zero-Shot Chinese Character Recognition with Stroke- and Radical-Level Decompositions

Zero-shot Chinese character recognition has attracted rising attention in recent years. Existing methods for this problem are mainly based on either certain low-level stroke-based decomposition or medium-level radical-based decomposition. Considering that the stroke- and radical-level decompositions can provide different levels of information, we propose an effective zero-shot Chinese character recognition method by combining them. The proposed method consists of a training stage and an inference stage. In the training stage, we adopt two similar encoder-decoder models to yield the estimates of stroke and radical encodings, which together with the true encodings are then used to formalize the associated stroke and radical losses for training. A similarity loss is introduced to regularize stroke and radical encoders to yield features of the same characters with high correlation. In the inference stage, two key modules, i.e., the stroke screening module (SSM) and feature matching module (FMM) are introduced to tackle the deterministic and confusing cases respectively. In particular, we introduce an effective stroke rectification scheme in FMM to enlarge the candidate set of characters for final inference. Numerous experiments over three benchmark datasets covering the handwritten, printed artistic and street view scenarios are conducted to demonstrate the effectiveness of the proposed method. Numerical results show that the proposed method outperforms the state-of-the-art methods in both character and radical zero-shot settings, and maintains competitive performance in the traditional seen character setting.
Adaptive Consensus Optimization Method for GANs

We propose a second order gradient based method with ADAM and RMSprop for the training of generative adversarial networks. The proposed method is fastest to obtain similar accuracy when compared to prominent second order methods. Unlike state-of-the-art recent methods, it does not require solving a linear system, or it does not require additional mixed second derivative terms. We derive the fixed point iteration corresponding to proposed method, and show that the proposed method is convergent. The proposed method produces better or comparable inception scores, and comparable quality of images compared to other recently proposed state-of-the-art second order methods. Compared to first order methods such as ADAM, it produces significantly better inception scores. The proposed method is compared and validated on popular datasets such as FFHQ, LSUN, CIFAR10, MNIST, and Fashion MNIST for image generation tasks.
Underwater Image Enhancement with Phase Transfer and Attention

Underwater pictures typically suffer from substantial deterioration due to the refraction and absorption of light by water, including color cast, hazy blur, and limited visibility. Such degradation in visibility eventually reduces the effectiveness of marine applications installed on autonomous underwater vehicles. Hence, an efficient pre-processing step is required for the significant performance of these applications. As a solution, underwater image enhancement (UIE) mainly focuses on enhancing the visibility of degraded images along with restoring crucial details. Existing methods generally utilize (a) complex cascaded architectures, (b) different degradation-prone color spaces, and (c) direct skip connections that pass irrelevant content. In light of this, we propose a lightweight transformer network with 1.7M parameters (1/6 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> of the existing state-of-the-art method) consisting of the proposed gray-scale attention and phase transformer block for UIE. A gray-scale attention block is proposed for the effective extraction of non-contaminated features. Further, a phase transfer block is proposed for effectively restoring the structural information in the outputs by propagating most relevant and undegraded features from the inputs. A comprehensive evaluation of the proposed method on synthetic (EUVP, UIEB) and real-world (UIEB, UCCS) image datasets as well as extensive ablation studies confirm its effectiveness over existing state-of-the-art approaches. The source code is provided at: https://github.com/Mdraqibkhan/UIEPTA.
Argument Mining with Modular BERT and Transfer Learning

We introduce BERT–MINUS, a modular, feature-enriched and transfer learning enabled model for Argument Mining. BERT–MINUS consists of: 1) a joint module which embeds the paragraph text, and 2) a dedicated module, consisting of three customized BERT models, which contextualize the argument markers, argument components and additional features given as text, respectively. BERT–MINUS implements two kinds of transfer learning – auto-transfer (transfer from a task to itself) and cross-transfer (classical transfer from one task to another) – via a novel Selective Fine-tuning mechanism. BERT–MINUS achieves state-of-the-art results on the Link Identification task and competitive results on the Argument Type Classification task. The synergy between the Features as Text and Selective Fine-tuning mechanisms significantly improves the performance of the model. Our work reveals the importance and potential of transfer learning via selective fine-tuning for modular Language Models. Moreover, this study dovetails naturally into the Prompt Engineering paradigm in NLP.
Class-Specific Variational Auto-Encoder for Content-Based Image Retrieval

Using a discriminative representation obtained by supervised deep learning methods showed promising results on diverse Content-Based Image Retrieval (CBIR) problems. However, existing methods exploiting labels during training try to discriminate all available classes, which is not ideal in cases where the retrieval problem focuses on a class of interest. In this paper, we propose a regularized loss for Variational Auto-Encoders (VAEs) forcing the model to focus on a given class of interest. As a result, the model learns to discriminate the data belonging to the class of interest from any other possibility, making the learnt latent space of the VAE suitable for class-specific retrieval tasks. The proposed Class-Specific Variational Auto-Encoder (CS-VAE) is evaluated on three public and one custom datasets, and its performance is compared with that of three related VAE-based methods. Experimental results show that the proposed method outperforms its competition in both in-domain and out-of-domain retrieval problems.
Bringing Carbon Awareness to Multi-cloud Application Delivery

Data centers consume roughly 1--2% of the world's electricity, with the majority of it attributed to compute, making the computing industry a substantial source of greenhouse gas emissions. Resources in data centers typically focus on providing high performance and availability, but the question of sustainability in managing these distributed resources often goes unnoticed over these other metrics. This problem will only exacerbate as the data center computing demand continues to increase.
Measuring Robustness of Deep Neural Networks from the Lens of Statistical Model Checking

Measuring robustness of deep neural networks (DNNs) is an important topic for trustworthy AI. Existing methods for verifying local robustness of DNNs usually face the scalability problem and have difficulties to deal with non-linear activation functions and complex semantic perturbations. Existing methods for measuring global robustness usually rely on large datasets, so it is difficult for users without a large dataset to compare the global robustness of different networks. In this paper, we propose two algorithms to measure the local and global robustness of DNNs from the lens of statistical model checking. Compared with the local robustness estimation method using the Okamoto bound, our method can provide the same theoretical guarantee using only 48.6%-74.1% of running time. Our global robustness estimation algorithm can provide high-quality estimation using only 10 images for CIFAR-10 and 50 images for ImageNet, thus can be used in a wider range of scenarios.
AfriWOZ: Corpus for Exploiting Cross-Lingual Transfer for Dialogue Generation in Low-Resource, African Languages

Dialogue generation is an important NLP task fraught with many challenges. The challenges become more daunting for low-resource African languages. To enable the creation of dialogue agents for African languages, we contribute the first high-quality dialogue datasets for 6 African languages: Swahili, Wolof, Hausa, Nigerian Pidgin English, Kinyarwanda & Yorùbá. There are a total of 9,000 turns, each language having 1,500 turns, which we translate from a portion of the English multi-domain MultiWOZ dataset. Subsequently, we benchmark by investigating & analyzing the effectiveness of modelling through transfer learning by utilziing state-of-the-art (SoTA) deep monolingual models: DialoGPT and BlenderBot. We compare the models with a simple seq2seq baseline using perplexity. Besides this, we conduct human evaluation of single-turn conversations by using majority votes and measure inter-annotator agreement (IAA). We find that the hypothesis that deep monolingual models learn some abstractions that generalize across languages holds. We observe human-like conversations, to different degrees, in 5 out of the 6 languages. The language with the most transferable properties is the Nigerian Pidgin English, with a human-likeness score of 78.1%, of which 34.4% are unanimous. We freely provide the datasets and host the model checkpoints/demos on the HuggingFace hub for public access.
On Fine-Tuned Deep Features for Unsupervised Domain Adaptation

Prior feature transformation based approaches to Unsupervised Domain Adaptation (UDA) employ the deep features extracted by pre-trained deep models without fine-tuning them on the specific source or target domain data for a particular domain adaptation task. In contrast, end-to-end learning based approaches optimise the pre-trained backbones and the customised adaptation modules simultaneously to learn domain-invariant features for UDA. In this work, we explore the potential of combining fine-tuned features and feature transformation based UDA methods for improved domain adaptation performance. Specifically, we integrate the prevalent progressive pseudo-labelling techniques into the fine-tuning framework to extract fine-tuned features which are subsequently used in a state-of-the-art feature transformation based domain adaptation method SPL (Selective Pseudo-Labeling). Thorough experiments with multiple deep models including ResNet-50/101 and DeiT-small/base are conducted to demonstrate the combination of fine-tuned features and SPL can achieve state-of-the-art performance on several benchmark datasets.
Explainable exclusion in the life insurance using multi-label classifier

To reduce manual tasks and to minimise the risks from a customer, many insurance companies have applied ar-tificial intelligence (AI) solutions, including, but not limited to, machine learning (ML) and deep learning (DL). Exclusion anal-ysis is one of the primary tasks in terms of minimising the risks a customer imposes on a life insurance company. Although a few research studies have made this the primary focus, they have yet to provide explainable research for the exclusion analysis to assist the underwriting process (UP) using ML/DL methods. Therefore, this paper makes the process of exclusion classification, along with its explainability, its primary concentration to assist the underwriters in understanding the underwriting data taken from the customer disclosure information. First, we explore this problem by applying a set of four multi-label classifiers (named binary relevance, classifier chains, label powerset, and ensemble learning) blended with five ML techniques (named multinomial Naive Bayes, support vector classifier, logistic regression, random forest, decision tree), using the data provided by one of the leading insurance companies in Australia. Then, we consider the best-performing model's classification probability and feature importance as input for the explainable ML system - Shapley additive explanations and introduced explainability outcome - as a quality assurance report (QAR). This paper offers an extensive empirical evaluation by comparing different metrics and human underwriters' reviews. Finally, the result demonstrates that the binary relevance algorithm combined with the decision tree classifier outperforms other existing methods for explainable exclusion, providing a better overview of the customer's risk profile.
Online Probabilistic Model Identification Using Adaptive Recursive MCMC

Although the Bayesian paradigm offers a formal framework for estimating the entire probability distribution over uncertain parameters, its online implementation can be challenging due to high computational costs. We suggest the Adaptive Recursive Markov Chain Monte Carlo (ARMCMC) method, which eliminates the shortcomings of conventional online techniques while computing the entire probability density function of model parameters. The limitations to Gaussian noise, the application to only linear in the parameters (LIP) systems, and the persistent excitation (PE) needs are some of these drawbacks. In ARMCMC, a temporal forgetting factor (TFF)-based variable jump distribution is proposed. The forgetting factor can be presented adaptively using the TFF in many dynamical systems as an alternative to a constant hyperparameter. By offering a trade-off between exploitation and exploration, the specific jump distribution has been optimised towards hybrid/multi-modal systems that permit inferences among modes. These trade-off are adjusted based on parameter evolution rate. We demonstrate that ARMCMC requires fewer samples than conventional MCMC methods to achieve the same precision and reliability. We demonstrate our approach using parameter estimation in a soft bending actuator and the Hunt-Crossley dynamic model, two challenging hybrid/multi-modal benchmarks. Additionally, we compare our method with recursive least squares and the particle filter, and show that our technique has significantly more accurate point estimates as well as a decrease in tracking error of the value of interest.