Reviewerly/exHarmony huggingface dataset:
Explainability for Clustering Models

The field of Artificial Intelligence is growing at a very high pace. Application of bigger and complex algorithms have become commonplace, thus making them harder to understand. The explainability of the algorithms and models in practice has become a necessity as these models are being widely adopted to make significant and consequential decisions. It makes it even more important for us to keep our understanding of the decisions and results of AI up to date. Explainable AI methods are currently addressing the interpretability, explainability, and fairness in supervised learning methods. There has been very less focus on explaining the results of unsupervised learning methods. This paper proposes an extension of the supervised explainability methods to deal with the unsupervised methods as well. We have researched and experimented with widely used clustering models to show the applicability of the proposed solution on most practiced unsupervised problems. We also have thoroughly investigated the methods to validate the results of both supervised and unsupervised explainability modules.
Federated Learning with Class Balanced Loss Optimized by Implicit Stochastic Gradient Descent

Federated learning is a paradigm for distributed machine learning in which a central server interacts with a large number of remote devices to create the optimal global model. System and data heterogeneity are now the two largest impediments to federated learning. This work suggests a federated learning strategy based on stochastic gradient descent optimization as a solution to the problem of heterogeneity-induced slow convergence, or even non-convergence, of the global model. This work estimates the average global gradient using locally uploaded model parameters without computing the first derivative or updating global model parameters through gradient descent. Allowing the global model to be used with fewer communication rounds. Obtain faster and more reliable convergence results. In experiments simulating varying degrees of heterogeneous settings, the strategy proposed in this work delivered faster and more stable convergence than FedProx and FedAvg. This work offers a strategy that decreases the number of communication cycles on highly heterogeneous synthetic datasets by around 50% compared to FedProx, therefore considerably enhancing the stability and durability of federated learning.
Discovering Popular Topics of Sarawak Gazette (SaGa) from Twitter Using Deep Learning

The emergence of social media as an information-sharing platform is progressively increasing. With the progress of artificial intelligence, it is now feasible to analyze historical document from social media. This study aims to understand more about how people use their social media to share the content of the Sarawak Gazette (SaGa), one of the valuable historical documents of Sarawak. In the study, a short text of Tweet corpus relating to SaGa was built (according to some keyword search criteria). The Tweet corpus will then be analyzed to extract the topic based on a topic modeling, specifically, Latent Dirichlet Allocation (LDA). Then, the topics will be further classified with Convolutional Neural Network (CNN) classifier.
Fault Diagnosis Methods of Deep Convolutional Dynamic Adversarial Networks

A DCDAN is proposed for intelligent fault diagnosis to address the issue that it is easy to obtain a large amount of labeled fault-type data in a laboratory environment but difficult or impossible to obtain a large amount of labeled data under actual working conditions. This method transfers the fault diagnosis knowledge acquired in the laboratory environment to the actual engineering equipment, obtains more comprehensive fault information by fusing the time domain and frequency domain data, employs the residual network to deeply extract fault features in the feature extraction layer, and makes use of the extracted fault features to improve fault diagnosis. To achieve unsupervised transfer learning, the marginal distributions and conditional probability distributions of the source and target domains are aligned by maximizing the domain classification loss, while the failure classification of mechanical equipment is achieved by minimizing the class prediction loss. The experimental results demonstrate that this model has a high classification accuracy in the unlabeled target data set and can effectively solve the problem of the lack of labels in the data set, i.e., realize intelligent mechanical fault diagnosis, under certain conditions.
Multi-source Heterogeneous Data Fusion Algorithm Based on Federated Learning

With the rapid advancement of science and technology, the number of edge devices with computing and storage capabilities, as well as the generated data traffic, continues to increase, making it difficult for the centralized processing mode centered on cloud computing to efficiently process the data generated by edge devices. Moreover, multimodal data is abundant due to the diversity of edge network devices and the ongoing improvement of data representation techniques. To make full use of heterogeneous data on edge devices and to tackle the “data communication barrier” problem caused by data privacy in edge computing, a multi-source heterogeneous data fusion technique based on Tucker decomposition is developed. The algorithm introduces tensor Tucker decomposition theory and realizes federated learning by constructing a high-order tensor with heterogeneous spatial dimension characteristics to capture the high-dimensional features of heterogeneous data and solve the fusion problem of heterogeneous data assuming no interaction. Combining and remembering. This method, unlike its predecessors, can efficiently integrate multi-source heterogeneous data without data transfer, therefore overcoming privacy and security-related data communication concerns. Finally, the effectiveness of the technique is confirmed using the MOSI data set, and the shortcomings of the original federated learning algorithm are solved by the updated federated weighted average algorithm. Subjective criteria are used to calculate the data quality for the improved federated average approach based on AHP. We present Influence, an improved federated weighted average method for dealing with multi-source data from a data quality perspective.
Understanding the Benefits of Forgetting When Learning on Dynamic Graphs

In order to solve graph-related tasks such as node classification, recommendation or community detection, most machine learning algorithms are based on node representations, also called embeddings, that allow to capture in the best way possible the properties of these graphs. More recently, learning node embeddings for dynamic graphs attracted significant interest due to the rich temporal information that they provide about the appearance of edges and nodes in the graph over time. In this paper, we aim to understand the effect of taking into account the static and dynamic nature of graph when learning node representations and the extent to which the latter influences the success of such learning process. Our motivation to do this stems from empirical results presented in several recent papers showing that static methods are sometimes on par or better than methods designed specifically for learning on dynamic graphs. To assess the importance of temporal information, we first propose a similarity measure between nodes based on the time distance of their edges with an explicit control over the decay of forgetting over time. We then devise a novel approach that combines the proposed time distance with static properties of the graph when learning temporal node embeddings. Our results on 3 different tasks (link prediction, node and edge classification) and 6 real-world datasets show that finding the right trade-off between static and dynamic information is crucial for learning good node representations and allows to significantly improve the results compared to state-of-the-art methods.
Learnable Masked Tokens for Improved Transferability of Self-supervised Vision Transformers

Vision transformers have recently shown remarkable performance in various visual recognition tasks specifically for self-supervised representation learning. The key advantage of transformers for self supervised learning, compared to their convolutional counterparts, is the reduced inductive biases that makes transformers amenable to learning rich representations from massive amounts of unlabelled data. On the other hand, this flexibility makes self-supervised vision transformers susceptible to overfitting when fine-tuning them on small labeled target datasets. Therefore, in this work, we make a simple yet effective architectural change by introducing new learnable masked tokens to vision transformers whereby we reduce the effect of overfitting in transfer learning while retaining the desirable flexibility of vision transformers. Through several experiments based on two seminal self-supervised vision transformers, SiT and DINO, and several small target visual recognition tasks, we show consistent and significant improvements in the accuracy of the fine-tuned models across all target tasks.
FROB: Few-Shot ROBust Model for Joint Classification and Out-of-Distribution Detection

Classification and Out-of-Distribution (OoD) detection in the few-shot setting remain challenging aims, but are important for devising critical systems in security where samples are limited. OoD detection requires that classifiers are aware of when they do not know and avoid setting high confidence to OoD samples away from the training data distribution. To address such limitations, we propose the Few-shot ROBust (FROB) model with its key contributions being (a) the joint classification and few-shot OoD detection, (b) the sample generation on the boundary of the support of the normal class distribution, and (c) the incorporation of the learned distribution boundary as OoD data for contrastive negative training. FROB finds the boundary of the support of the normal class distribution, and uses it to improve the few-shot OoD detection performance. We propose a self-supervised learning methodology for sample generation on the normal class distribution confidence boundary based on generative and discriminative models, including classification. FROB implicitly generates adversarial samples, and forces samples from OoD, including our boundary, to be less confident by the classifier. By including the learned boundary, FROB reduces the threshold linked to the model's few-shot robustness in the number of few-shots, and maintains the OoD performance approximately constant, independent of the number of few-shots. The low- and few-shot robustness evaluation of FROB on different image datasets and on One-Class Classification (OCC) data shows that FROB achieves competitive performance and outperforms baselines in terms of robustness to the OoD few-shot population and variability.
Resisting Graph Adversarial Attack via Cooperative Homophilous Augmentation

Recent studies show that Graph Neural Networks (GNNs) are vulnerable and easily fooled by small perturbations, which has raised considerable concerns for adapting GNNs in various safety-critical applications. In this work, we focus on the emerging but critical attack, namely, Graph Injection Attack (GIA), in which the adversary poisons the graph by injecting fake nodes instead of modifying existing structures or node attributes. Inspired by findings that the adversarial attacks are related to the increased heterophily on perturbed graphs (the adversary tends to connect dissimilar nodes), we propose a general defense framework CHAGNN against GIA through cooperative homophilous augmentation of graph data and model. Specifically, the model generates pseudo-labels for unlabeled nodes in each round of training to reduce heterophilous edges of nodes with distinct labels. The cleaner graph is fed back to the model, producing more informative pseudo-labels. In such an iterative manner, model robustness is then promisingly enhanced. We present the theoretical analysis of the effect of homophilous augmentation and provide the guarantee of the proposal’s validity. Experimental results empirically demonstrate the effectiveness of CHAGNN in comparison with recent state-of-the-art defense methods on diverse real-world datasets.
Bi-matching Mechanism to Combat Long-tail Senses of Word Sense Disambiguation

The long-tail phenomenon of word sense distribution in linguistics causes Word Sense Disambiguation (WSD) to face both head senses with a large number of samples and tail senses with only a few samples. Traditional recognition methods are suitable for head senses with sufficient training samples, but they cannot effectively deal with tail senses. Inspired by the diverse memory and recognition abilities of children’s linguistic behavior, we propose a bi-matching mechanism approach for WSD. Considering that tail senses are often presented in the form of fixed collocations, a collocation feature matching method suitable for tail senses is designed; the traditional definition matching method is used for head senses; finally, the two matching methods are combined to construct a WSD model with the bi-matching mechanism (called Bi-MWSD). Bi-MWSD can effectively combat the difficulty of identifying the tail senses due to insufficient training samples. The experiments are implemented in the standard English all-words WSD evaluation framework and the training data augmented evaluation framework. The experimental results outperform the baseline models and achieve state-of-the-art performance under the data augmentation evaluation framework.
Enhance Temporal Knowledge Graph Completion via Time-Aware Attention Graph Convolutional Network

Previous works on knowledge graph representation learning focus on static knowledge graph and get fully developed. However, task on temporal knowledge graph is far from consummation because of its late start. Recent researches have shifted to the temporal knowledge graph relying on the extension of static ones. Most of these methods seek approaches to incorporate temporal information but neglect the potential adjacent impact merged in temporal knowledge graphs. Meanwhile, different temporal information of involved facts evoke impact with different extent on the concerned entity, which is always overlooked in the previous works. In our paper, we propose a Time-aware Attention Graph Convolutional Network, named TAGCN, for temporal knowledge graph completion. Entity completion can be turned into interactions between entity and associated neighborhood. We utilize a graph convolutional network with a novel temporal attention layer to obtain neighboring information at all timestamps to avoid diachronic sparsity. We conduct extensive experiments on various datasets to evaluate our model performance. The results illustrate that our model outperforms the state-of-the-art baselines on entity prediction.
Supervised Contrastive Learning for Few-Shot Action Classification

In a typical few-shot action classification scenario, a learner needs to recognize unseen video classes with only few labeled videos. It is critical to learn effective representations of video samples and distinguish their difference when they are sampled from different action classes. In this work, we propose a novel supervised contrastive learning framework for few-shot video action classification based on spatial-temporal augmentations over video samples. Specifically, for each meta-training episode, we first obtain multiple spatial-temporal augmentations for each video sample, and then define the contrastive loss over the augmented support samples by extracting positive and negative sample pairs according to their class labels. This supervised contrastive loss is further combined with the few-shot classification loss defined over a similarity score regression network for end-to-end episodic meta-training. Due to its high flexibility, the proposed framework can deploy the latest contrastive learning approaches for few-shot video action classification. The extensive experiments on several action classification benchmarks show that the proposed supervised contrastive learning framework achieves state-of-the-art performance.
Summarizing Labeled Multi-graphs

Real-world graphs can be difficult to interpret and visualize beyond a certain size. To address this issue, graph summarization aims to simplify and shrink a graph, while maintaining its high-level structure and characteristics. Most summarization methods are designed for homogeneous, undirected, simple graphs; however, many real-world graphs are ornate; with characteristics including node labels, directed edges, edge multiplicities, and self-loops. In this paper we propose TG-sum, a versatile yet rigorous graph summarization model that (to the best of our knowledge, for the first time) can handle graphs with all the aforementioned characteristics (and any combination thereof). Moreover, our proposed model captures basic sub-structures that are prevalent in real-world graphs, such as cliques, stars, etc. Experiments demonstrate that TG-sum facilitates the visualization of real-world complex graphs, revealing interpretable structures and high-level relationships. Furthermore, TG-sum achieves better trade-off between compression rate and running time, relative to existing methods (only) on comparable settings.
Scalable Adversarial Online Continual Learning

Adversarial continual learning is effective for continual learning problems because of the presence of feature alignment process generating task-invariant features having low susceptibility to the catastrophic forgetting problem. Nevertheless, the ACL method imposes considerable complexities because it relies on task-specific networks and discriminators. It also goes through an iterative training process which does not fit for online (one-epoch) continual learning problems. This paper proposes a scalable adversarial continual learning (SCALE) method putting forward a parameter generator transforming common features into task-specific features and a single discriminator in the adversarial game to induce common features. The training process is carried out in meta-learning fashions using a new combination of three loss functions. SCALE outperforms prominent baselines with noticeable margins in both accuracy and execution time.
SAViR-T: Spatially Attentive Visual Reasoning with Transformers

We present a novel computational model, SAViR-T, for the family of visual reasoning problems embodied in the Raven’s Progressive Matrices (RPM). Our model considers explicit spatial semantics of visual elements within each image in the puzzle, encoded as spatio-visual tokens, and learns the intra-image as well as the inter-image token dependencies, highly relevant for the visual reasoning task. Token-wise relationship, modeled through a transformer-based SAViR-T architecture, extract group (row or column) driven representations by leveraging the group-rule coherence and use this as the inductive bias to extract the underlying rule representations in the top two row (or column) per token in the RPM. We use this relation representations to locate the correct choice image that completes the last row or column for the RPM. Extensive experiments across both synthetic RPM benchmarks, including RAVEN, I-RAVEN, RAVEN-FAIR, and PGM, and the natural image-based “V-PROM” demonstrate that SAViR-T sets a new state-of-the-art for visual reasoning, exceeding prior models’ performance by a considerable margin.
NE-WNA: A Novel Network Embedding Framework Without Neighborhood Aggregation

Graph Neural Networks (GNNs) are powerful tools in representation learning for graphs. Most GNNs use the message passing mechanism to obtain a distinguished feature representation. However, due to this message passing mechanism, most existing GNNs are inherently restricted by over-smoothing and poor robustness. Therefore, we propose a simple yet effective Network Embedding framework Without Neighborhood Aggregation (NE-WNA). Specifically, NE-WNA removes the neighborhood aggregation operation from the message passing mechanism. It only takes node features as input and then obtains node representations by a simple autoencoder. We also design an enhanced neighboring contrastive (ENContrast) loss to incorporate the graph structure into the node representations. In the representation space, the ENContrast encourages low-order neighbors to be closer to the target node than high-order neighbors. Experimental results show that NE-WNA enjoys high accuracy on the node classification task and high robustness against adversarial attacks.
Self-distilled Pruning of Deep Neural Networks

Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel self-distillation based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed cross-correlation objective for self-distilled pruning implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.
Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks

Using task-specific components within a neural network in continual learning (CL) is a compelling strategy to address the stability-plasticity dilemma in fixed-capacity models without access to past data. Current methods focus only on selecting a sub-network for a new task that reduces forgetting of past tasks. However, this selection could limit the forward transfer of relevant past knowledge that helps in future learning. Our study reveals that satisfying both objectives jointly is more challenging when a unified classifier is used for all classes of seen tasks–class-Incremental Learning (class-IL)–as it is prone to ambiguities between classes across tasks. Moreover, the challenge increases when the semantic similarity of classes across tasks increases. To address this challenge, we propose a new CL method, named AFAF (Code is available at: https://github.com/GhadaSokar/AFAF. ), that aims to Avoid Forgetting and Allow Forward transfer in class-IL using fix-capacity models. AFAF allocates a sub-network that enables selective transfer of relevant knowledge to a new task while preserving past knowledge, reusing some of the previously allocated components to utilize the fixed-capacity, and addressing class-ambiguities when similarities exist. The experiments show the effectiveness of AFAF in providing models with multiple CL desirable properties, while outperforming state-of-the-art methods on various challenging benchmarks with different semantic similarities.