OpenRaiser/Intern-Atlas huggingface dataset:
FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
Venue: arXiv cs.LG 2024
Year: 2024

In recent years, federated learning (FL) has emerged as a promising technique for training machine learning models in a decentralized manner while also preserving data privacy. The non-independent and identically distributed (non-i.i.d.) nature of client data, coupled with constraints on client or edge devices, presents significant challenges in FL. Furthermore, learning across a high number of communication rounds can be risky and potentially unsafe for model exploitation. Traditional FL approaches may suffer from these challenges. Therefore, we introduce FedSiKD, which incorporates knowledge distillation (KD) within a similarity-based federated learning framework. As clients join the system, they securely share relevant statistics about their data distribution, promoting intra-cluster homogeneity. This enhances optimization efficiency and accelerates the learning process, effectively transferring knowledge between teacher and student models and addressing device constraints. FedSiKD outperforms state-of-the-art algorithms by achieving higher accuracy, exceeding by 25\% and 18\% for highly skewed data at $α= {0.1,0.5}$ on the HAR and MNIST datasets, respectively. Its faster convergence is illustrated by a 17\% and 20\% increase in accuracy within the first five rounds on the HAR and MNIST datasets, respectively, highlighting its early-stage learning proficiency. Code is publicly available and hosted on GitHub (https://github.com/SimuEnv/FedSiKD)
Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs
Venue: arXiv cs.CV 2024
Year: 2024

Facial video inpainting plays a crucial role in a wide range of applications, including but not limited to the removal of obstructions in video conferencing and telemedicine, enhancement of facial expression analysis, privacy protection, integration of graphical overlays, and virtual makeup. This domain presents serious challenges due to the intricate nature of facial features and the inherent human familiarity with faces, heightening the need for accurate and persuasive completions. In addressing challenges specifically related to occlusion removal in this context, our focus is on the progressive task of generating complete images from facial data covered by masks, ensuring both spatial and temporal coherence. Our study introduces a network designed for expression-based video inpainting, employing generative adversarial networks (GANs) to handle static and moving occlusions across all frames. By utilizing facial landmarks and an occlusion-free reference image, our model maintains the user's identity consistently across frames. We further enhance emotional preservation through a customized facial expression recognition (FER) loss function, ensuring detailed inpainted outputs. Our proposed framework exhibits proficiency in eliminating occlusions from facial videos in an adaptive form, whether appearing static or dynamic on the frames, while providing realistic and coherent results.
Wireless Crowd Detection for Smart Overtourism Mitigation
Venue: arXiv cs.CY 2024
Year: 2024

Overtourism occurs when the number of tourists exceeds the carrying capacity of a destination, leading to negative impacts on the environment, culture, and quality of life for residents. By monitoring overtourism, destination managers can identify areas of concern and implement measures to mitigate the negative impacts of tourism while promoting smarter tourism practices. This can help ensure that tourism benefits both visitors and residents while preserving the natural and cultural resources that make these destinations so appealing. This chapter describes a low-cost approach to monitoring overtourism based on mobile devices' wireless activity. A flexible architecture was designed for a smart tourism toolkit to be used by Small and Medium-sized Enterprises (SMEs) in crowding management solutions, to build better tourism services, improve efficiency and sustainability, and reduce the overwhelming feeling of pressure in critical hotspots. The crowding sensors count the number of surrounding mobile devices, by detecting trace elements of wireless technologies, mitigating the effect of MAC address randomization. They run detection programs for several technologies, and fingerprinting analysis results are only stored locally in an anonymized database, without infringing privacy rights. After that edge computing, sensors communicate the crowding information to a cloud server, by using a variety of uplink techniques to mitigate local connectivity limitations, something that has been often disregarded in alternative approaches. Field validation of sensors has been performed on Iscte's campus. Preliminary results show that these sensors can be deployed in multiple scenarios and provide a diversity of spatio-temporal crowding data that can scaffold tourism overcrowding management strategies.
Design Space of Visual Feedforward And Corrective Feedback in XR-Based Motion Guidance Systems
Venue: arXiv cs.HC 2024
Year: 2024

Extended reality (XR) technologies are highly suited in assisting individuals in learning motor skills and movements -- referred to as motion guidance. In motion guidance, the "feedforward" provides instructional cues of the motions that are to be performed, whereas the "feedback" provides cues which help correct mistakes and minimize errors. Designing synergistic feedforward and feedback is vital to providing an effective learning experience, but this interplay between the two has not yet been adequately explored. Based on a survey of the literature, we propose design space for both motion feedforward and corrective feedback in XR, and describe the interaction effects between them. We identify common design approaches of XR-based motion guidance found in our literature corpus, and discuss them through the lens of our design dimensions. We then discuss additional contextual factors and considerations that influence this design, together with future research opportunities for motion guidance in XR.
DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers
Venue: arXiv cs.CV 2024
Year: 2024

Full-body avatar presence is crucial for immersive social and environmental interactions in digital reality. However, current devices only provide three six degrees of freedom (DOF) poses from the headset and two controllers (i.e. three-point trackers). Because it is a highly under-constrained problem, inferring full-body pose from these inputs is challenging, especially when supporting the full range of body proportions and use cases represented by the general population. In this paper, we propose a deep learning framework, DivaTrack, which outperforms existing methods when applied to diverse body sizes and activities. We augment the sparse three-point inputs with linear accelerations from Inertial Measurement Units (IMU) to improve foot contact prediction. We then condition the otherwise ambiguous lower-body pose with the predictions of foot contact and upper-body pose in a two-stage model. We further stabilize the inferred full-body pose in a wide range of configurations by learning to blend predictions that are computed in two reference frames, each of which is designed for different types of motions. We demonstrate the effectiveness of our design on a large dataset that captures 22 subjects performing challenging locomotion for three-point tracking, including lunges, hula-hooping, and sitting. As shown in a live demo using the Meta VR headset and Xsens IMUs, our method runs in real-time while accurately tracking a user's motion when they perform a diverse set of movements.
Efficient One-stage Video Object Detection by Exploiting Temporal Consistency
Venue: arXiv cs.CV 2024
Year: 2024

Recently, one-stage detectors have achieved competitive accuracy and faster speed compared with traditional two-stage detectors on image data. However, in the field of video object detection (VOD), most existing VOD methods are still based on two-stage detectors. Moreover, directly adapting existing VOD methods to one-stage detectors introduces unaffordable computational costs. In this paper, we first analyse the computational bottlenecks of using one-stage detectors for VOD. Based on the analysis, we present a simple yet efficient framework to address the computational bottlenecks and achieve efficient one-stage VOD by exploiting the temporal consistency in video frames. Specifically, our method consists of a location-prior network to filter out background regions and a size-prior network to skip unnecessary computations on low-level feature maps for specific frames. We test our method on various modern one-stage detectors and conduct extensive experiments on the ImageNet VID dataset. Excellent experimental results demonstrate the superior effectiveness, efficiency, and compatibility of our method. The code is available at https://github.com/guanxiongsun/vfe.pytorch.
Momentum Approximation in Asynchronous Private Federated Learning
Venue: arXiv cs.LG 2024
Year: 2024

Asynchronous protocols have been shown to improve the scalability of federated learning (FL) with a massive number of clients. Meanwhile, momentum-based methods can achieve the best model quality in synchronous FL. However, naively applying momentum in asynchronous FL algorithms leads to slower convergence and degraded model performance. It is still unclear how to effective combinie these two techniques together to achieve a win-win. In this paper, we find that asynchrony introduces implicit bias to momentum updates. In order to address this problem, we propose momentum approximation that minimizes the bias by finding an optimal weighted average of all historical model updates. Momentum approximation is compatible with secure aggregation as well as differential privacy, and can be easily integrated in production FL systems with a minor communication and storage cost. We empirically demonstrate that on benchmark FL datasets, momentum approximation can achieve $1.15 \textrm{--}4\times$ speed up in convergence compared to naively combining asynchronous FL with momentum.
Computational Complexity of Preferred Subset Repairs on Data-Graphs
Venue: arXiv cs.DB 2024
Year: 2024

Preferences are a pivotal component in practical reasoning, especially in tasks that involve decision-making over different options or courses of action that could be pursued. In this work, we focus on repairing and querying inconsistent knowledge bases in the form of graph databases, which involves finding a way to solve conflicts in the knowledge base and considering answers that are entailed from every possible repair, respectively. Without a priori domain knowledge, all possible repairs are equally preferred. Though that may be adequate for some settings, it seems reasonable to establish and exploit some form of preference order among the potential repairs. We study the problem of computing prioritized repairs over graph databases with data values, using a notion of consistency based on GXPath expressions as integrity constraints. We present several preference criteria based on the standard subset repair semantics, incorporating weights, multisets, and set-based priority levels. We show that it is possible to maintain the same computational complexity as in the case where no preference criterion is available for exploitation. Finally, we explore the complexity of consistent query answering in this setting and obtain tight lower and upper bounds for all the preference criteria introduced.
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) are now commonplace in conversation applications. However, their risks of misuse for generating harmful responses have raised serious societal concerns and spurred recent research on LLM conversation safety. Therefore, in this survey, we provide a comprehensive overview of recent studies, covering three critical aspects of LLM conversation safety: attacks, defenses, and evaluations. Our goal is to provide a structured summary that enhances understanding of LLM conversation safety and encourages further investigation into this important subject. For easy reference, we have categorized all the studies mentioned in this survey according to our taxonomy, available at: https://github.com/niconi19/LLM-conversation-safety.
Machine Learning for Stochastic Parametrisation
Venue: arXiv cs.LG 2024
Year: 2024

Atmospheric models used for weather and climate prediction are traditionally formulated in a deterministic manner. In other words, given a particular state of the resolved scale variables, the most likely forcing from the sub-grid scale processes is estimated and used to predict the evolution of the large-scale flow. However, the lack of scale-separation in the atmosphere means that this approach is a large source of error in forecasts. Over recent years, an alternative paradigm has developed: the use of stochastic techniques to characterise uncertainty in small-scale processes. These techniques are now widely used across weather, sub-seasonal, seasonal, and climate timescales. In parallel, recent years have also seen significant progress in replacing parametrisation schemes using machine learning (ML). This has the potential to both speed up and improve our numerical models. However, the focus to date has largely been on deterministic approaches. In this position paper, we bring together these two key developments, and discuss the potential for data-driven approaches for stochastic parametrisation. We highlight early studies in this area, and draw attention to the novel challenges that remain.
Pearson Correlations on Networks: Corrigendum
Venue: arXiv cs.SI 2024
Year: 2024

Recently, the first author proposed a measure to calculate Pearson correlations for node values expressed in a network, by taking into account distances or metrics defined on the network. In this technical note, we show that using an arbitrary choice of distances might result in imaginary or unbounded correlation values, which is undesired. We prove that this problem is solved by restricting to a special class of distances: negative type metrics. We also discuss two natural classes of negative type metrics on graphs, for which the network correlations are properly defined.
Zak-OTFS and LDPC Codes
Venue: arXiv cs.IT 2024
Year: 2024

Orthogonal Time Frequency Space (OTFS) is a framework for communications and active sensing that processes signals in the delay-Doppler (DD) domain. It is informed by 6G propagation environments, where Doppler spreads measured in kHz make it more and more difficult to estimate channels, and the standard model-dependent approach to wireless communication is starting to break down. We consider Zak-OTFS where inverse Zak transform converts information symbols mounted on DD domain pulses to the time domain for transmission. Zak-OTFS modulation is parameterized by a delay period $τ_{p}$ and a Doppler period $ν_{p}$, where the product $τ_{p}ν_{p}=1$. When the channel spread is less than the delay period, and the Doppler spread is less than the Doppler period, the Zak-OTFS input-output relation can be predicted from the response to a single pilot symbol. The highly reliable channel estimates concentrate around the pilot location, and we configure low-density parity-check (LDPC) codes that take advantage of this prior information about reliability. It is advantageous to allocate information symbols to more reliable bins in the DD domain. We report simulation results for a Veh-A channel model where it is not possible to resolve all the paths, showing that LDPC coding extends the range of Doppler spreads for which reliable model-free communication is possible. We show that LDPC coding reduces sensitivity to the choice of transmit filter, making bandwidth expansion less necessary. Finally, we compare BER performance of Zak-OTFS to that of a multicarrier approximation (MC-OTFS), showing LDPC coding amplifies the gains previously reported for uncoded transmission.
Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer
Venue: arXiv cs.LG 2024
Year: 2024

In Chaos, a minor divergence between two initial conditions exhibits exponential amplification over time, leading to far-away outcomes, known as the butterfly effect. Thus, the distant future is full of uncertainty and hard to forecast. We introduce Group Reservoir Transformer to predict long-term events more accurately and robustly by overcoming two challenges in Chaos: (1) the extensive historical sequences and (2) the sensitivity to initial conditions. A reservoir is attached to a Transformer to efficiently handle arbitrarily long historical lengths, with an extension of a group of reservoirs to reduce the sensitivity to the initialization variations. Our architecture consistently outperforms state-of-the-art models in multivariate time series, including TimeLLM, GPT2TS, PatchTST, DLinear, TimeNet, and the baseline Transformer, with an error reduction of up to -59\% in various fields such as ETTh, ETTm, and air quality, demonstrating that an ensemble of butterfly learning can improve the adequacy and certainty of event prediction, despite of the traveling time to the unknown future.
Combatting deepfakes: Policies to address national security threats and rights violations
Venue: arXiv cs.CR 2024
Year: 2024

This paper provides policy recommendations to address threats from deepfakes. First, we provide background information about deepfakes and review the harms they pose. We describe how deepfakes are currently used to proliferate sexual abuse material, commit fraud, manipulate voter behavior, and pose threats to national security. Second, we review previous legislative proposals designed to address deepfakes. Third, we present a comprehensive policy proposal that focuses on addressing multiple parts of the deepfake supply chain. The deepfake supply chain begins with a small number of model developers, model providers, and compute providers, and it expands to include billions of potential deepfake creators. We describe this supply chain in greater detail and describe how entities at each step of the supply chain ought to take reasonable measures to prevent the creation and proliferation of deepfakes. Finally, we address potential counterpoints of our proposal. Overall, deepfakes will present increasingly severe threats to global security and individual liberties. To address these threats, we call on policymakers to enact legislation that addresses multiple parts of the deepfake supply chain.
WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
Venue: arXiv cs.LG 2024
Year: 2024

A common phenomena confining the representation quality in Self-Supervised Learning (SSL) is dimensional collapse (also known as rank degeneration), where the learned representations are mapped to a low dimensional subspace of the representation space. The State-of-the-Art SSL methods have shown to suffer from dimensional collapse and fall behind maintaining full rank. Recent approaches to prevent this problem have proposed using contrastive losses, regularization techniques, or architectural tricks. We propose WERank, a new regularizer on the weight parameters of the network to prevent rank degeneration at different layers of the network. We provide empirical evidence and mathematical justification to demonstrate the effectiveness of the proposed regularization method in preventing dimensional collapse. We verify the impact of WERank on graph SSL where dimensional collapse is more pronounced due to the lack of proper data augmentation. We empirically demonstrate that WERank is effective in helping BYOL to achieve higher rank during SSL pre-training and consequently downstream accuracy during evaluation probing. Ablation studies and experimental analysis shed lights on the underlying factors behind the performance gains of the proposed approach.
Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation
Venue: arXiv cs.CV 2024
Year: 2024

Test-time adaptation (TTA) refers to adapting a trained model to a new domain during testing. Existing TTA techniques rely on having multiple test images from the same domain, yet this may be impractical in real-world applications such as medical imaging, where data acquisition is expensive and imaging conditions vary frequently. Here, we approach such a task, of adapting a medical image segmentation model with only a single unlabeled test image. Most TTA approaches, which directly minimize the entropy of predictions, fail to improve performance significantly in this setting, in which we also observe the choice of batch normalization (BN) layer statistics to be a highly important yet unstable factor due to only having a single test domain example. To overcome this, we propose to instead integrate over predictions made with various estimates of target domain statistics between the training and test statistics, weighted based on their entropy statistics. Our method, validated on 24 source/target domain splits across 3 medical image datasets surpasses the leading method by 2.9% Dice coefficient on average.
Multi-Fidelity Methods for Optimization: A Survey
Venue: arXiv cs.LG 2024
Year: 2024

Real-world black-box optimization often involves time-consuming or costly experiments and simulations. Multi-fidelity optimization (MFO) stands out as a cost-effective strategy that balances high-fidelity accuracy with computational efficiency through a hierarchical fidelity approach. This survey presents a systematic exploration of MFO, underpinned by a novel text mining framework based on a pre-trained language model. We delve deep into the foundational principles and methodologies of MFO, focusing on three core components -- multi-fidelity surrogate models, fidelity management strategies, and optimization techniques. Additionally, this survey highlights the diverse applications of MFO across several key domains, including machine learning, engineering design optimization, and scientific discovery, showcasing the adaptability and effectiveness of MFO in tackling complex computational challenges. Furthermore, we also envision several emerging challenges and prospects in the MFO landscape, spanning scalability, the composition of lower fidelities, and the integration of human-in-the-loop approaches at the algorithmic level. We also address critical issues related to benchmarking and the advancement of open science within the MFO community. Overall, this survey aims to catalyze further research and foster collaborations in MFO, setting the stage for future innovations and breakthroughs in the field.
GeoBotsVR: A Robotics Learning Game for Beginners with Hands-on Learning Simulation
Venue: arXiv cs.HC 2024
Year: 2024

This article introduces GeoBotsVR, an easily accessible virtual reality game that combines elements of puzzle-solving with robotics learning and aims to cultivate interest and motivation in robotics, programming, and electronics among individuals with limited experience in these domains. The game allows players to build and customize a two-wheeled mobile robot using various robotic components and use their robot to solve various procedurally-generated puzzles in a diverse range of environments. An innovative aspect is the inclusion of a repair feature, requiring players to address randomly generated electronics and programming issues with their robot through hands-on manipulation. GeoBotsVR is designed to be immersive, replayable, and practical application-based, offering an enjoyable and accessible tool for beginners to acquaint themselves with robotics. The game simulates a hands-on learning experience and does not require prior technical knowledge, making it a potentially valuable resource for beginners to get an engaging introduction to the field of robotics.