Reviewerly/exHarmony huggingface dataset:
Concept-Guided Interpretable Federated Learning

Interpretable federated learning is an emerging challenge to identify explainable characteristics of each client-specific personalized model in a federated learning system. This paper proposes a novel federated concept bottleneck (FedCBM) method by introducing human-friendly concepts for client-wise model interpretation. Specifically, given a set of pre-defined concepts, all clients will collaboratively train shared Concept Activation Vectors (CAVs) in federated settings. The shared concepts will be the information carrier to align client-specific representations, and also be applied to enhance the model's accuracy under a supervised learning loss. The effectiveness of our method and concept-level reasoning is demonstrated in our experimental analysis.
Toward a Unified Framework for RGB and RGB-D Visual Navigation

In object-goal navigation, an agent is steered toward a target object based on its observations. The solution pipeline is usually composed of scene representation learning and navigation policy learning: the former reflects the agent observation, and the latter determines the navigation action. To this end, this article proposes a unified visual navigation framework, dubbed VTP, which can employ either RGB or RGB-D observations for object-goal visual navigation. Using a unified Visual Transformer Navigation network (VTN), the agent analyzes image areas in relation to specific objects, producing visual representations that capture both instance-to-instance relationships and instance-to-region relationships. Meanwhile, we utilize depth maps to explore the spatial relationship between instances and the agent. Additionally, we develop a pre-training scheme to associate visual representations with navigation signals. Furthermore, we adopt Tentative Policy Learning (TPL) to guide an agent to escape from deadlocks. When an agent is detected as being in deadlock states in training, we utilize Tentative Imitation learning (TIL) to provide the agent expert demonstrations for deadlock escape and such demonstrations are learned in a separate Tentative Policy Network (TPN). In testing, under deadlocks, estimated expert demonstrations are given to the policy network to find an escape action. Our system outperforms other methods in both iTHOR and RoboTHOR.
Story Sifting Using Object Detection Techniques

Our focus is on the problem of story sifting (or story recognition), which is the automated sifting of interesting stories that emerge from the interactions between virtual characters in virtual storyworld environments. To date, approaches to story sifting have been either: manual, with the burden of authoring sifting patterns; or automated, but of limited efficiency and scalability. In this paper, we address these shortcomings via a novel approach that recasts the problem as one of object detection. We demonstrate how an object detection model can be trained to detect story arcs of prominent story types emerging from an interactive virtual storyworld which can occur anywhere in the storyworld's timeline. We evaluate our approach using synthetic virtual story environments that show our approach is able to: detect story arcs anywhere in the storyworld's timeline with a high degree of accuracy and more efficiently than the state-of-the-art Arc Sift, making it scalable in real-time.
Gemini: A Dual-Task Co-training Model for Partial Label Learning

Partial-Label Learning (PLL) is an important weakly supervised learning task that assumes each training instance is annotated with a set of candidate labels. In recent years, self-training PLL models, which learn label confidence vectors and train the models with them iteratively, have attracted increasing attention and achieve state-of-the-art performance. However, they suffer from error accumulation problem due to that mistakenly disambiguated instances could mislead the model with false positive labels and causes performance degradation. To address this issue, we propose a novel dual-task co-training PLL model named Gemini that trains two structurally identical networks by optimizing two distinct tasks. Specifically, one is trained by optimizing PLL task with the original partially labeled data, and the other is trained by optimizing the noisy label learning (NLL) task with noisy pseudo labels generated according to the disambiguated label confidence. Moreover, we interrupt the error accumulation via a carefully-designed cooperation mechanism between the two networks, which comprises prediction alignment and correctly-disambiguated instance selection. Extensive experiments on benchmark datasets demonstrate the effectiveness and robustness of Gemini on both uniform and instance-dependent partially labeled data. The code is available at https://github.com/libeibeics/Gemini .
Cooperative Multi-Agent Reinforcement Learning with Dynamic Target Localization: A Reward Sharing Approach

Cooperation in multi-agent reinforcement learning (MARL) facilitates the acquisition of complex problem-solving skills and promotes more efficient and effective decision-making among agents. Numerous strategies for cooperative learning in MARL exist, including joint action learning, task decomposition, role assignment, and communication protocols. However, deploying these strategies in a complex and dynamic environment remains challenging. To address such challenges, we propose a technique that uses reward sharing to enhance cooperation in partially observable multi-agent environments. As an extension of reward shaping, reward sharing allows agents to work together towards a global objective while still pursuing their local objectives. This approach can foster cooperation and reduce competition between agents without explicit communication, ultimately leading to faster learning and better performance. This study compares three different reward sharing techniques: the Performance Incentive (PI), the Observer's Share (OS), and the Synergy Achievement (SA) in the context of dynamic target localization, focusing on simulation studies. Thereafter, the proposed reward sharing techniques are evaluated under the effects of objective prioritization, various agent counts, and a variety of map sizes. The research reveals that the proposed reward sharing techniques enhance agent performance, scaling the number of agents leads to higher rewards, and demonstrates a negative correlation between map size and average rewards.
Leaving the NavMesh: An Ablative Analysis of Deep Reinforcement Learning for Complex Navigation in 3D Virtual Environments

Expanding non-player character (NPC) navigation behavior in video games has the potential to induce novel player experiences. Current industry standards represent traversable world geometry by utilizing a Navigation Mesh (NavMesh); however NavMesh complexity scales poorly with additional navigation abilities (e.g. jumping, wall-running, jet-packs, etc.) and increasing world scale. Deep Reinforcement Learning (DRL) allows for an NPC agent to learn how to navigate environmental obstacles with any navigation ability without NavMesh dependence. Despite the promise of DRL navigation, adoption in industry remains low due to the required expert knowledge in agent design and the poor training efficiency of DRL algorithms. In this work, we utilize the off-policy Soft-Actor Critic (SAC) DRL algorithm to investigate the importance of different local observation types and agent scalar information to agent performance across three topologically distinct environments. We implement a truncated n-step returns method for minibatch sampling which improves early training efficiency by up to 75% by reducing inaccurate off-policy bias. We empirically evaluate environment partial observability with observation stacking where we find that 4–8 observation stacks renders the environments sufficiently Markovian.
Multiple Teacher Model for Continual Test-Time Domain Adaptation

Test-time adaptation (TTA) without accessing the source data provides a practical means of addressing distribution changes in testing data by adjusting pre-trained models during the testing phase. However, previous TTA methods typically assume a static, independent target domain, which contrasts with the actual scenario of the target domain changing over time. Using previous TTA methods for long-term adaptation often leads to problems of error accumulation or catastrophic forgetting, as it relies on the capability of a single model, leading to performance degradation. To address these challenges, we propose a multiple teacher model approach (MTA) for continual test-time domain adaptation. Firstly, we reduce error accumulation and leverage the robustness of multiple models by implementing a weighted and averaged multiple teacher model that provides pseudo-labels for enhanced prediction accuracy. Then, we mitigate catastrophic forgetting by logging mutation gradients and randomly restoring some parameters to the weights of the pre-trained model. Our comprehensive experiments demonstrate that MTA outperforms other state-of-the-art methods in continual time adaptation.
Bloating Reduction in Symbolic Regression Through Function Frequency-Based Tree Substitution in Genetic Programming

Genetic programming (GP) is an evolutionary machine learning method that can be used to address a wide range of both classification and regression conundrums. However, traditional GP algorithms can lead to unnecessary code growth known as bloating. This can slow down the convergence time, lead to over-fitting, and increase the computational cost required by the algorithm. The main focus of this paper is to control bloating caused by symbolic regression in GP trees. To address the bloating issue, this paper introduces a novel tree substitution method to reduce the tree size while increasing the exploring ability of the GP algorithm. The proposed method incorporates a comprehensive analysis to detect bloating in parent trees. When a bloated tree is detected, a new, smaller tree is generated, leveraging the function frequency of the identified bloated tree. A set of regression experiments have been conducted on six real-world datasets. Results showed that the proposed GP method obtains a reduction in the size of the best individual while maintaining similar performance as standard GP with a tree height limit.
3RE-Net: Joint Loss-REcovery and Super-REsolution Neural Network for REal-Time Video

Real-time video over the Internet suffers from packet loss and low network bandwidth. The receiving side may receive down-sampled video with damaged frames. In this work, we are motivated to enhance the quality of video by joint loss recovery and super-resolution. We propose Joint Loss-REcovery and Super-REsolution Neural Network for REal-time Video (3RE-Net), to recover the loss and super-resolve a damaged frame. 3RE-Net has two unprecedented advantages: (1) It only utilizes preceding frames and the current frame as input, as waiting for future frames causes additional delay, which is not suitable for real-time video streaming. (2) 3RE-Net induces small inference delay, which is applaudable for real-time videos. To mitigate computational workload, we jointly process the loss recover and super-resolve by reusing motions and features beneficial for both super-resolution and loss recovery. The design of 3RE-Net can be summarized as follows: It first extracts motions and features from the frames, and propagates the extracted motions and features through warping, synthesis, feature-level alignment, and deep detail refinement modules. Through this way, we can first obtain a set of warped candidate frames, which are later used to generate spatio-temporal consistent feature maps through synthesis and alignment. The output frame can be reconstructed by the feature maps in high resolution and loss free. We conduct experiments to compare 3RE-Net with state-of-the-art benchmark schemes. Results demonstrate that 3RE-Net outperforms all existing benchmarks in terms of both quality and delay.
A Group Genetic Algorithm for Energy-Efficient Resource Allocation in Container-Based Clouds with Heterogeneous Physical Machines

Containers are quickly gaining popularity in cloud computing environments due to their scalable and lightweight characteristics. However, the problem of Resource Allocation in Container-based clouds (RAC) is much more challenging than the Virtual Machines (VMs)-based clouds because RAC includes two levels of allocation problems: allocating containers to VMs and allocating VMs to Physical Machine (PMs). In this paper, we proposed a novel Group Genetic Algorithm (GGA) with energy-aware crossover, Best-Fit-Decreasing Insert (BFDI), and Local Search based Unpack (LSU) operator to solve RAC problems. Meanwhile, we apply an energy model with heterogeneous PMs that accurately captures the energy consumption of cloud data centers. Compared to state-of-the-art methods, experiments show that our method can significantly reduce the energy consumption on a wide range of test datasets.
SAR2EO: A High-Resolution Image Translation Framework with Denoising Enhancement

Synthetic Aperture Radar (SAR) to electro-optical (EO) image translation is a fundamental task in remote sensing that can enrich the dataset by fusing information from different sources. Recently, many methods have been proposed to tackle this task, but they are still difficult to complete the conversion from low-resolution images to high-resolution images. Thus, we propose a framework, SAR2EO, aiming at addressing this challenge. Firstly, to generate high-quality EO images, we adopt the coarse-to-fine generator, multi-scale discriminators, and improved adversarial loss in the pix2pixHD model to increase the synthesis quality. Secondly, we introduce a denoising module to remove the noise in SAR images, which helps to suppress the noise while preserving the structural information of the images. To validate the effectiveness of the proposed framework, we conduct experiments on the dataset of the Multi-modal Aerial View Imagery Challenge (MAVIC), which consists of large-scale SAR and EO image pairs. The experimental results demonstrate the superiority of our proposed framework, and we win the first place in the MAVIC held in CVPR PBVS 2023.
No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling

Vision Transformers (ViTs) have demonstrated outstanding performance in computer vision tasks, yet their high computational complexity prevents their deployment in computing resource-constrained environments. Various token pruning techniques have been introduced to alleviate the high computational burden of ViTs by dynamically dropping image tokens. However, some undesirable pruning at early stages may result in permanent loss of image information in subsequent layers, consequently hindering model performance. To address this problem, we propose IdleViT, a dynamic token-idle-based method that achieves an excellent trade-off between performance and efficiency. Specifically, in each layer, IdleViT selects a subset of the image tokens to participate in computations while keeping the rest of the tokens idle and directly passing them to this layer's output. By allowing the idle tokens to be re-selected in the following layers, IdleViT mitigates the negative impact of improper pruning in the early stages. Furthermore, inspired by the normalized graph cut, we devise a token cut loss on the attention map as regularization to improve IdleViT's token selection ability. Our method is simple yet effective and can be extended to pyramid ViTs since no token is completely dropped. Extensive experimental results on various ViT architectures have shown that IdleViT can diminish the complexity of pretrained ViTs by up to 33% with no more than 0.2% accuracy decrease on ImageNet, after finetuning for only 30 epochs. Notably, when the keep ratio is 0.5, IdleViT outperforms the state-of-the-art EViT on DeiT-S by 0.5% higher accuracy and even faster inference speed. The source code is available in the supplementary material.
Impact of Fidelity and Robustness of Machine Learning Explanations on User Trust

EXplainable machine learning (XML) has recently emerged as a promising approach to address the inherent opacity of machine learning (ML) systems by providing insights into their reasoning processes. This paper explores the relationships among user trust, fidelity, and robustness within the context of ML explanations. To investigate these relationships, a user study is implemented within the context of predicting students' performance. The study is designed to focus on two scenarios: (1) fidelity-based scenario—exploring dynamics of user trust across different explanations at varying fidelity levels and (2) robustness-based scenario—examining dynamics of in user trust concerning robustness. For each scenario, we conduct experiments based on two different metrics, including self-reported trust and behaviour-based trust metrics. For the fidelity-based scenario, we find that users trust both high and low-fidelity explanations compared to without-fidelity explanations (no explanations) based on the behaviour-based trust results, rather than relying on the self-reported trust results. We also obtain consistent findings based on different metrics, indicating no significant differences in user trust when comparing different explanations across fidelity levels. Additionally, for the robustness-based scenario, we get contrasting results from the two metrics. The self-reported trust metric does not demonstrate any variations in user trust concerning robustness levels, whereas the behaviour-based trust metric suggests that user trust tends to be higher when robustness levels are higher.
AdaptMatch: Adaptive Consistency Regularization for Semi-supervised Learning with Top-k Pseudo-labeling and Contrastive Learning

Semi-supervised learning has been established as a very effective paradigm for utilizing unlabeled data in order to reduce dependency on large labeled datasets. Most of the semi-supervised learning (SSL) methods proposed recently rely on a predefined and extremely high threshold to select unlabeled data that contribute to the training, thus failing to consider different learning statuses of the model and feature learning from unlabeled data. To address this issue, we propose AdaptMatch, an adaptive learning approach to leverage unlabeled data using Top-k pseudo-labeling and contrastive learning according to the model's learning status. The core of AdaptMatch is to adaptively adjust rules for different learning phases to allow informative unlabeled data and their pseudo-labels. If we cannot get high-confidence pseudo-labels from unlabeled data, contrastive learning can help the model learn more common features within the class. AdaptMatch outperforms or equals the state-of-the-art performance on a range of SSL benchmarks, exceptionally superior when the labeled data are extremely limited or imbalanced. For example, AdaptMatch reaches 91.56% and 97.44% accuracy with 4 labeled examples per class on CIFAR-10 and SVHN respectively, substantially improving over the previously best 88.70% and 96.66% accuracy achieved by FixMatch and ReMixMatch. Meanwhile, AdaptMatch also improves the accuracy of FixMatch in CIFAR10-LT with a performance gain of up to 2.3%.
Sequence Unlearning for Sequential Recommender Systems

Sequential recommender systems, leveraging clients' sequential product browsing history, have become an essential tool in delivering personalized product recommendations. As data protection regulations come into focus, certain clients may demand the removal of their data from the training sets used by these systems. In this paper, we focus on the problem of how specific client information can be efficiently removed from a pre-trained sequential recommender system without the need for retraining, particularly when the change to the data set is not substantial. We propose a novel sequence unlearning method for sequential recommender systems by leveraging label noise injection. Intuitively, our method promotes data unlearning by encouraging the system to produce random predictions for the sequences aiming to unlearn. To further prevent the model from overfitting an incorrect label, which could lead to substantial changes in its parameters, our method incorporates a dynamic process wherein the incorrect label is continually altered during the learning phase. This effectively encourages the model to lose confidence in the original label, while also discouraging it from fitting to a specific incorrect label. To the best of our knowledge, this is the first work to tackle the unlearning problem in sequential recommender systems without accessing the remaining data. Our approach is general and can work with any sequential recommender system. Empirically, we demonstrate that our method effectively helps different recommender systems unlearn specific sequential data while maintaining strong generalization performance on the remaining data across different datasets.
Multi-graph Laplacian Feature Mapping Incorporating Tag Information for Image Annotation

Automatic image annotation has exerted a tremendous fascination of many researchers with the development of multimedia and computer vision. However, most methods employ graph learning omit to combine the label information with the manifold structure between samples or utilize the single graph when computing the sample graph. Furthermore, the visual content is ignored in the process of computing the manifold structure between labels. These drawbacks lead to incomplete and inaccurate manifold information. To this end, we propose a Multi-graph Laplacian Feature Mapping Incorporating Tag Information method for image annotation. Our method firstly combines the label information with the Laplacian eigenmaps, and multi-graphs are utilized to maintain the local geometric structure of samples. Then the visual content is taken into account to obtain tag correlations. Afterward, a sea of empirical evaluations is conducted on three benchmark datasets to prove the effectiveness of the proposed method.
Estimation of Unmasked Face Images Based on Voice and 3DMM

Facemasks have become common due to the COVID-19 pandemic. They have begun to affect security and identification systems because they cover almost half of the face. Current state-of-the-art methods have been applied to estimate unmasked faces from masked face images. They are successful in improving the quality of the face texture by 3D Morphable Model (3DMM) as intermediate representations. However, their performance in restoring the face shapes is insufficient, and some of generated faces lack identities. In this study, we focus on voice, which has a particularly high correlation with the shape of the mouth and nose, which are obscured by masks. We propose a multimodal method using 3DMM and voice for face shape estimation under masks. Experimental results show that the proposed method qualitatively and quantitatively improves the quality of shape restoration of a face compared to the baseline method without considering voice.