OpenRaiser/Intern-Atlas huggingface dataset:
Federated Aggregation of Mallows Rankings: A Comparative Analysis of Borda and Lehmer Coding
Venue: arXiv cs.LG 2024
Year: 2024

Rank aggregation combines multiple ranked lists into a consensus ranking. In fields like biomedical data sharing, rankings may be distributed and require privacy. This motivates the need for federated rank aggregation protocols, which support distributed, private, and communication-efficient learning across multiple clients with local data. We present the first known federated rank aggregation methods using Borda scoring and Lehmer codes, focusing on the sample complexity for federated algorithms on Mallows distributions with a known scaling factor $ϕ$ and an unknown centroid permutation $σ_0$. Federated Borda approach involves local client scoring, nontrivial quantization, and privacy-preserving protocols. We show that for $ϕ\in [0,1)$, and arbitrary $σ_0$ of length $N$, it suffices for each of the $L$ clients to locally aggregate $\max\{C_1(ϕ), C_2(ϕ)\frac{1}{L}\log \frac{N}δ\}$ rankings, where $C_1(ϕ)$ and $C_2(ϕ)$ are constants, quantize the result, and send it to the server who can then recover $σ_0$ with probability $\geq 1-δ$. Communication complexity scales as $NL \log N$. Our results represent the first rigorous analysis of Borda's method in centralized and distributed settings under the Mallows model. Federated Lehmer coding approach creates a local Lehmer code for each client, using a coordinate-majority aggregation approach with specialized quantization methods for efficiency and privacy. We show that for $ϕ+ϕ^2<1+ϕ^N$, and arbitrary $σ_0$ of length $N$, it suffices for each of the $L$ clients to locally aggregate $\max\{C_3(ϕ), C_4(ϕ)\frac{1}{L}\log \frac{N}δ\}$ rankings, where $C_3(ϕ)$ and $C_4(ϕ)$ are constants. Clients send truncated Lehmer coordinate histograms to the server, which can recover $σ_0$ with probability $\geq 1-δ$. Communication complexity is $\sim O(N\log NL\log L)$.
JaxLife: An Open-Ended Agentic Simulator
Venue: arXiv cs.AI 2024
Year: 2024

Human intelligence emerged through the process of natural selection and evolution on Earth. We investigate what it would take to re-create this process in silico. While past work has often focused on low-level processes (such as simulating physics or chemistry), we instead take a more targeted approach, aiming to evolve agents that can accumulate open-ended culture and technologies across generations. Towards this, we present JaxLife: an artificial life simulator in which embodied agents, parameterized by deep neural networks, must learn to survive in an expressive world containing programmable systems. First, we describe the environment and show that it can facilitate meaningful Turing-complete computation. We then analyze the evolved emergent agents' behavior, such as rudimentary communication protocols, agriculture, and tool use. Finally, we investigate how complexity scales with the amount of compute used. We believe JaxLife takes a step towards studying evolved behavior in more open-ended simulations. Our code is available at https://github.com/luchris429/JaxLife
Equitable Skin Disease Prediction Using Transfer Learning and Domain Adaptation
Venue: arXiv cs.CV 2024
Year: 2024

In the realm of dermatology, the complexity of diagnosing skin conditions manually necessitates the expertise of dermatologists. Accurate identification of various skin ailments, ranging from cancer to inflammatory diseases, is paramount. However, existing artificial intelligence (AI) models in dermatology face challenges, particularly in accurately diagnosing diseases across diverse skin tones, with a notable performance gap in darker skin. Additionally, the scarcity of publicly available, unbiased datasets hampers the development of inclusive AI diagnostic tools. To tackle the challenges in accurately predicting skin conditions across diverse skin tones, we employ a transfer-learning approach that capitalizes on the rich, transferable knowledge from various image domains. Our method integrates multiple pre-trained models from a wide range of sources, including general and specific medical images, to improve the robustness and inclusiveness of the skin condition predictions. We rigorously evaluated the effectiveness of these models using the Diverse Dermatology Images (DDI) dataset, which uniquely encompasses both underrepresented and common skin tones, making it an ideal benchmark for assessing our approach. Among all methods, Med-ViT emerged as the top performer due to its comprehensive feature representation learned from diverse image sources. To further enhance performance, we conducted domain adaptation using additional skin image datasets such as HAM10000. This adaptation significantly improved model performance across all models.
The Impact of Generative AI-Powered Code Generation Tools on Software Engineer Hiring: Recruiters' Experiences, Perceptions, and Strategies
Venue: arXiv cs.SE 2024
Year: 2024

The rapid advancements in Generative AI (GenAI) tools, such as ChatGPT and GitHub Copilot, are transforming software engineering by automating code generation tasks. While these tools improve developer productivity, they also present challenges for organizations and hiring professionals in evaluating software engineering candidates' true abilities and potential. Although there is existing research on these tools in both industry and academia, there is a lack of research on how these tools specifically affect the hiring process. Therefore, this study aims to explore recruiters' experiences and perceptions regarding GenAI-powered code generation tools, as well as their challenges and strategies for evaluating candidates. Findings from our survey of 32 industry professionals indicate that although most participants are familiar with such tools, the majority of organizations have not adjusted their candidate evaluation methods to account for candidates' use/knowledge of these tools. There are mixed opinions on whether candidates should be allowed to use these tools during interviews, with many participants valuing candidates who can effectively demonstrate their skills in using these tools. Additionally, most participants believe that it is important to incorporate GenAI-powered code generation tools into computer science curricula and mention the key risks and benefits of doing so.
Geno-Weaving: Low-Complexity Capacity-Achieving DNA Storage
Venue: arXiv cs.IT 2024
Year: 2024

As a possible implementation of data storage using DNA, multiple strands of DNA are stored in a liquid container so that, in the future, they can be read by an array of DNA readers in parallel. These readers will sample the strands with replacement to produce a random number of noisy reads for each strand. An essential component of such a data storage system is how to reconstruct data out of these unsorted, repetitive, and noisy reads. It is known that if a single read can be modeled by a substitution channel $W$, then the overall capacity can be expressed by the "Poisson-ization" of $W$. In this paper, we lay down a rateless code along each strand to encode its index; we then lay down a capacity-achieving block code at the same position across all strands to protect data. That weaves a low-complexity coding scheme that achieves DNA's capacity.
Semantically Controllable Augmentations for Generalizable Robot Learning
Venue: arXiv cs.RO 2024
Year: 2024

Generalization to unseen real-world scenarios for robot manipulation requires exposure to diverse datasets during training. However, collecting large real-world datasets is intractable due to high operational costs. For robot learning to generalize despite these challenges, it is essential to leverage sources of data or priors beyond the robot's direct experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. These generative models encompass a broad range of real-world scenarios beyond a robot's direct experience and can synthesize novel synthetic experiences that expose robotic agents to additional world priors aiding real-world generalization at no extra cost. In particular, our approach leverages pre-trained generative models as an effective tool for data augmentation. We propose a generative augmentation framework for semantically controllable augmentations and rapidly multiplying robot datasets while inducing rich variations that enable real-world generalization. Based on diverse augmentations of robot data, we show how scalable robot manipulation policies can be trained and deployed both in simulation and in unseen real-world environments such as kitchens and table-tops. By demonstrating the effectiveness of image-text generative models in diverse real-world robotic applications, our generative augmentation framework provides a scalable and efficient path for boosting generalization in robot learning at no extra human cost.
Enhancing Privacy in Federated Learning: Secure Aggregation for Real-World Healthcare Applications
Venue: arXiv cs.CR 2024
Year: 2024

Deploying federated learning (FL) in real-world scenarios, particularly in healthcare, poses challenges in communication and security. In particular, with respect to the federated aggregation procedure, researchers have been focusing on the study of secure aggregation (SA) schemes to provide privacy guarantees over the model's parameters transmitted by the clients. Nevertheless, the practical availability of SA in currently available FL frameworks is currently limited, due to computational and communication bottlenecks. To fill this gap, this study explores the implementation of SA within the open-source Fed-BioMed framework. We implement and compare two SA protocols, Joye-Libert (JL) and Low Overhead Masking (LOM), by providing extensive benchmarks in a panel of healthcare data analysis problems. Our theoretical and experimental evaluations on four datasets demonstrate that SA protocols effectively protect privacy while maintaining task accuracy. Computational overhead during training is less than 1% on a CPU and less than 50% on a GPU for large models, with protection phases taking less than 10 seconds. Incorporating SA into Fed-BioMed impacts task accuracy by no more than 2% compared to non-SA scenarios. Overall this study demonstrates the feasibility of SA in real-world healthcare applications and contributes in reducing the gap towards the adoption of privacy-preserving technologies in sensitive applications.
Experimental Analysis of Freehand Multi-Object Selection Techniques in Virtual Reality Head-Mounted Displays
Venue: arXiv cs.HC 2024
Year: 2024

Object selection is essential in virtual reality (VR) head-mounted displays (HMDs). Prior work mainly focuses on enhancing and evaluating techniques for selecting a single object in VR, leaving a gap in the techniques for multi-object selection, a more complex but common selection scenario. To enable multi-object selection, the interaction technique should support group selection in addition to the default pointing selection mode for acquiring a single target. This composite interaction could be particularly challenging when using freehand gestural input. In this work, we present an empirical comparison of six freehand techniques, which are comprised of three mode-switching gestures (Finger Segment, Multi-Finger, and Wrist Orientation) and two group selection techniques (Cone-casting Selection and Crossing Selection) derived from prior work. Our results demonstrate the performance, user experience, and preference of each technique. The findings derive three design implications that can guide the design of freehand techniques for multi-object selection in VR HMDs.
3D Priors-Guided Diffusion for Blind Face Restoration
Venue: arXiv cs.CV 2024
Year: 2024

Blind face restoration endeavors to restore a clear face image from a degraded counterpart. Recent approaches employing Generative Adversarial Networks (GANs) as priors have demonstrated remarkable success in this field. However, these methods encounter challenges in achieving a balance between realism and fidelity, particularly in complex degradation scenarios. To inherit the exceptional realism generative ability of the diffusion model and also constrained by the identity-aware fidelity, we propose a novel diffusion-based framework by embedding the 3D facial priors as structure and identity constraints into a denoising diffusion process. Specifically, in order to obtain more accurate 3D prior representations, the 3D facial image is reconstructed by a 3D Morphable Model (3DMM) using an initial restored face image that has been processed by a pretrained restoration network. A customized multi-level feature extraction method is employed to exploit both structural and identity information of 3D facial images, which are then mapped into the noise estimation process. In order to enhance the fusion of identity information into the noise estimation, we propose a Time-Aware Fusion Block (TAFB). This module offers a more efficient and adaptive fusion of weights for denoising, considering the dynamic nature of the denoising process in the diffusion model, which involves initial structure refinement followed by texture detail enhancement. Extensive experiments demonstrate that our network performs favorably against state-of-the-art algorithms on synthetic and real-world datasets for blind face restoration. The Code is released on our project page at https://github.com/838143396/3Diffusion.
DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning
Venue: arXiv cs.CL 2024
Year: 2024

In recent years, Large Language Models (LLMs) have demonstrated significant improvements across a variety of tasks, one of which is the long-context capability. The key to improving long-context performance lies in effective data organization and management strategies that integrate data from multiple domains and optimize the context window during training. Through extensive experimental analysis, we identified three key challenges in designing effective data management strategies that enable the model to achieve long-context capability without sacrificing performance in other tasks: (1) a shortage of long documents across multiple domains, (2) effective construction of context windows, and (3) efficient organization of large-scale datasets. To address these challenges, we introduce DataSculpt, a novel data management framework designed for long-context training. We first formulate the organization of training data as a multi-objective combinatorial optimization problem, focusing on attributes including relevance, homogeneity, integrity, and efficiency. Specifically, our approach utilizes a coarse-to-fine methodology to optimize training data organization both efficiently and effectively. We begin by clustering the data based on semantic similarity (coarse), followed by a multi-objective greedy search within each cluster to score and concatenate documents into various context windows (fine). Our comprehensive evaluations demonstrate that DataSculpt significantly enhances long-context training performance, resulting in improvements of 18.09% in retrieval augmentation, 21.23% in summarization, 21.27% in reading comprehension, and a 3.81% increase in code completion, while also maintaining overall model proficiency with a 4.88% improvement.
Annealing-inspired training of an optical neural network with ternary weights
Venue: arXiv cs.ET 2024
Year: 2024

Artificial neural networks (ANNs) represent a fundamentally connectionnist and distributed approach to computing, and as such they differ from classical computers that utilize the von Neumann architecture. This has revived research interest in new unconventional hardware to enable more efficient implementations of ANNs rather than emulating them on traditional machines. In order to fully leverage the capabilities of this new generation of ANNs, optimization algorithms that take into account hardware limitations and imperfections are necessary. Photonics represents a particularly promising platform, offering scalability, high speed, energy efficiency, and the capability for parallel information processing. Yet, fully fledged implementations of autonomous optical neural networks (ONNs) with in-situ learning remain scarce. In this work, we propose a ternary weight architecture high-dimensional semiconductor laser-based ONN. We introduce a simple method for achieving ternary weights with Boolean hardware, significantly increasing the ONN's information processing capabilities. Furthermore, we design a novel in-situ optimization algorithm that is compatible with, both, Boolean and ternary weights, and provide a detailed hyperparameter study of said algorithm for two different tasks. Our novel algorithm results in benefits, both in terms of convergence speed and performance. Finally, we experimentally characterize the long-term inference stability of our ONN and find that it is extremely stable with a consistency above 99\% over a period of more than 10 hours, addressing one of the main concerns in the field. Our work is of particular relevance in the context of in-situ learning under restricted hardware resources, especially since minimizing the power consumption of auxiliary hardware is crucial to preserving efficiency gains achieved by non-von Neumann ANN implementations.
Towards Split Learning-based Privacy-Preserving Record Linkage
Venue: arXiv cs.CR 2024
Year: 2024

Split Learning has been recently introduced to facilitate applications where user data privacy is a requirement. However, it has not been thoroughly studied in the context of Privacy-Preserving Record Linkage, a problem in which the same real-world entity should be identified among databases from different dataholders, but without disclosing any additional information. In this paper, we investigate the potentials of Split Learning for Privacy-Preserving Record Matching, by introducing a novel training method through the utilization of Reference Sets, which are publicly available data corpora, showcasing minimal matching impact against a traditional centralized SVM-based technique.
Poster: Developing an O-RAN Security Test Lab
Venue: arXiv cs.CR 2024
Year: 2024

Open Radio Access Networks (ORAN) is a new architectural approach, having been proposed only a few years ago, and it is an expansion of the current Next Generation Radio Access Networks (NG-RAN) of 5G. ORAN aims to break this closed RAN market that is controlled by a handful of vendors, by implementing open interfaces between the different Radio Access Networks (RAN) components, and by introducing modern technologies to the RAN like machine learning, virtualization, and disaggregation. However, the architectural design of ORAN was recently causing concerns and debates about its security, which is considered one of its major drawbacks. Several theoretical risk analyses related to ORAN have been conducted, but to the best of our knowledge, not even a single practical one has been performed yet. In this poster, we discuss and propose a way for a minimal, future-proof deployment of an ORAN 5G network, able to accommodate various hands-on security analyses for its different elements.
Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information
Venue: arXiv cs.CV 2024
Year: 2024

With the advancement of large-scale language modeling techniques, large multimodal models combining visual encoders with large language models have demonstrated exceptional performance in various visual tasks. Most of the current large-scale multimodal models achieve this by mapping visual features obtained from the visual encoder into a large language model and using them as inputs alongside text for downstream tasks. Therefore, the number of visual tokens directly affects the training and inference speed of the model. There has been significant work on token pruning for visual transformers, but for large multimodal models, only relying on visual information for token pruning or compression may lead to significant loss of important information. On the other hand, the textual input in the form of a question may contain valuable information that can aid in answering the question, providing additional knowledge to the model. To address the potential oversimplification and excessive pruning that can occur with most purely visual token pruning methods, we propose a text information-guided dynamic visual token recovery mechanism that does not require training. This mechanism leverages the similarity between the question text and visual tokens to recover visually meaningful tokens with important text information while merging other less important tokens. Experimental results demonstrate that our proposed method achieves comparable performance to the original approach while compressing the visual tokens to an average of 10% of the original quantity. Our source code will be made publicly available following acceptance.
OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model
Venue: arXiv cs.CV 2024
Year: 2024

Variational Autoencoder (VAE), compressing videos into latent representations, is a crucial preceding component of Latent Video Diffusion Models (LVDMs). With the same reconstruction quality, the more sufficient the VAE's compression for videos is, the more efficient the LVDMs are. However, most LVDMs utilize 2D image VAE, whose compression for videos is only in the spatial dimension and often ignored in the temporal dimension. How to conduct temporal compression for videos in a VAE to obtain more concise latent representations while promising accurate reconstruction is seldom explored. To fill this gap, we propose an omni-dimension compression VAE, named OD-VAE, which can temporally and spatially compress videos. Although OD-VAE's more sufficient compression brings a great challenge to video reconstruction, it can still achieve high reconstructed accuracy by our fine design. To obtain a better trade-off between video reconstruction quality and compression speed, four variants of OD-VAE are introduced and analyzed. In addition, a novel tail initialization is designed to train OD-VAE more efficiently, and a novel inference strategy is proposed to enable OD-VAE to handle videos of arbitrary length with limited GPU memory. Comprehensive experiments on video reconstruction and LVDM-based video generation demonstrate the effectiveness and efficiency of our proposed methods.
Saying goodbyes to rotating your phone: Magnetometer calibration during SLAM
Venue: arXiv cs.RO 2024
Year: 2024

While Wi-Fi positioning is still more common indoors, using magnetic field features has become widely known and utilized as an alternative or supporting source of information. Magnetometer bias presents significant challenge in magnetic field navigation and SLAM. Traditionally, magnetometers have been calibrated using standard sphere or ellipsoid fitting methods and by requiring manual user procedures, such as rotating a smartphone in a figure-eight shape. This is not always feasible, particularly when the magnetometer is attached to heavy or fast-moving platforms, or when user behavior cannot be reliably controlled. Recent research has proposed using map data for calibration during positioning. This paper takes a step further and verifies that a pre-collected map is not needed; instead, calibration can be done as part of a SLAM process. The presented solution uses a factorized particle filter that factors out calibration in addition to the magnetic field map. The method is validated using smartphone data from a shopping mall and mobile robotics data from an office environment. Results support the claim that magnetometer calibration can be achieved during SLAM with comparable accuracy to manual calibration. Furthermore, the method seems to slightly improve manual calibration when used on top of it, suggesting potential for integrating various calibration approaches.