OpenRaiser/Intern-Atlas huggingface dataset:
Hierarchical Upper Confidence Bounds for Constrained Online Learning
Venue: arXiv cs.LG 2024
Year: 2024

The multi-armed bandit (MAB) problem is a foundational framework in sequential decision-making under uncertainty, extensively studied for its applications in areas such as clinical trials, online advertising, and resource allocation. Traditional MAB formulations, however, do not adequately capture scenarios where decisions are structured hierarchically, involve multi-level constraints, or feature context-dependent action spaces. In this paper, we introduce the hierarchical constrained bandits (HCB) framework, which extends the contextual bandit problem to incorporate hierarchical decision structures and multi-level constraints. We propose the hierarchical constrained upper confidence bound (HC-UCB) algorithm, designed to address the complexities of the HCB problem by leveraging confidence bounds within a hierarchical setting. Our theoretical analysis establishes sublinear regret bounds for HC-UCB and provides high-probability guarantees for constraint satisfaction at all hierarchical levels. Furthermore, we derive a minimax lower bound on the regret for the HCB problem, demonstrating the near-optimality of our algorithm. The results are significant for real-world applications where decision-making processes are inherently hierarchical and constrained, offering a robust and efficient solution that balances exploration and exploitation across multiple levels of decision-making.
Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss
Venue: arXiv cs.CV 2024
Year: 2024

Contrastive loss is a powerful approach for representation learning, where larger batch sizes enhance performance by providing more negative samples to better distinguish between similar and dissimilar data. However, scaling batch sizes is constrained by the quadratic growth in GPU memory consumption, primarily due to the full instantiation of the similarity matrix. To address this, we propose a tile-based computation strategy that partitions the contrastive loss calculation into arbitrary small blocks, avoiding full materialization of the similarity matrix. Furthermore, we introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems, employing ring-based communication at the GPU level to optimize synchronization and fused kernels at the CUDA core level to reduce I/O overhead. Experimental results show that the proposed method scales batch sizes to unprecedented levels. For instance, it enables contrastive training of a CLIP-ViT-L/14 model with a batch size of 4M or 12M using 8 or 32 A800 80GB without sacrificing any accuracy. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed. The code will be made publicly available.
Inference with K-means
Venue: arXiv cs.LG 2024
Year: 2024

This thesis aims to invent new approaches for making inferences with the k-means algorithm. k-means is an iterative clustering algorithm that randomly assigns k centroids, then assigns data points to the nearest centroid, and updates centroids based on the mean of assigned points. This process continues until convergence, forming k clusters where each point belongs to the closest centroid. This research investigates the prediction of the last component of data points obtained from a distribution of clustered data using the online balanced k-means approach. Through extensive experimentation and analysis, key findings have emerged. It is observed that a larger number of clusters or partitions tends to yield lower errors while increasing the number of assigned data points does not significantly improve inference errors. Reducing losses in the learning process does not significantly impact overall inference errors. Indicating that as learning is going on inference errors remain unchanged. Recommendations include the need for specialized inference techniques to estimate better data points derived from multi-clustered data and exploring methods that yield improved results with larger assigned datasets. By addressing these recommendations, this research advances the accuracy and reliability of inferences made with the k-means algorithm, bridging the gap between clustering and non-parametric density estimation and inference.
Evaluation of Systems Programming Exercises through Tailored Static Analysis
Venue: arXiv cs.CY 2024
Year: 2024

In large programming classes, it takes a significant effort from teachers to evaluate exercises and provide detailed feedback. In systems programming, test cases are not sufficient to assess exercises, since concurrency and resource management bugs are difficult to reproduce. This paper presents an experience report on static analysis for the automatic evaluation of systems programming exercises. We design systems programming assignments with static analysis rules that are tailored for each assignment, to provide detailed and accurate feedback. Our evaluation shows that static analysis can identify a significant number of erroneous submissions missed by test cases.
Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models
Venue: arXiv cs.IR 2024
Year: 2024

Large-scale industrial recommendation models predict the most relevant items from catalogs containing millions or billions of options. To train these models efficiently, a small set of irrelevant items (negative samples) is selected from the vast catalog for each relevant item (positive example), helping the model distinguish between relevant and irrelevant items. Choosing the right negative sampling method is a common challenge. We address this by implementing and comparing various negative sampling methods - random, popularity-based, in-batch, mixed, adaptive, and adaptive with mixed variants - on modern sequential recommendation models. Our experiments, including hyperparameter optimization and 20x repeats on three benchmark datasets with varying popularity biases, show how the choice of method and dataset characteristics impact key model performance metrics. We also reveal that average performance metrics often hide imbalances across popularity bands (head, mid, tail). We find that commonly used random negative sampling reinforces popularity bias and performs best for head items. Popularity-based methods (in-batch and global popularity negative sampling) can offer balanced performance at the cost of lower overall model performance results. Our study serves as a practical guide to the trade-offs in selecting a negative sampling method for large-scale sequential recommendation models. Code, datasets, experimental results and hyperparameters are available at: https://github.com/apple/ml-negative-sampling.
Enhancing Real-Time Master Data Management with Complex Match and Merge Algorithms
Venue: arXiv cs.DC 2024
Year: 2024

Master Data Management (MDM) ensures data integrity, consistency, and reliability across an organization's systems. I introduce a novel complex match and merge algorithm optimized for real-time MDM solutions. The proposed method accurately identifies duplicates and consolidates records in large-scale datasets by combining deterministic matching, fuzzy matching, and machine learning-based conflict resolution. I implemented it using PySpark and Databricks; the algorithm benefits from distributed computing and Delta Lake for scalable and reliable data processing. Comprehensive performance evaluations demonstrate a 90% accuracy on datasets of up to 10 million records while maintaining low latency and high throughput, significantly improving upon existing MDM approaches. The method shows strong potential in domains such as healthcare and finance, with an overall 30% improvement in latency compared to traditional MDM systems.
Two Classes of Quantum MDS Codes with Large Minimal Distance
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, two classes of quantum MDS codes are constructed. The main tools are multiplicative structures on finite fields. Carefully choosing different cosets can make the corresponding generalized Reed-Solomon codes Hermitian self-orthogonal, which yields quantum MDS codes by utilizing Hermitian type CSS construction. In some cases, these codes have minimal distances larger than $q/2$. Also, the parameters of these codes have never been reported before.
DeLLiriuM: A large language model for delirium prediction in the ICU using structured EHR
Venue: arXiv cs.AI 2024
Year: 2024

Delirium is an acute confusional state that has been shown to affect up to 31% of patients in the intensive care unit (ICU). Early detection of this condition could lead to more timely interventions and improved health outcomes. While artificial intelligence (AI) models have shown great potential for ICU delirium prediction using structured electronic health records (EHR), most of them have not explored the use of state-of-the-art AI models, have been limited to single hospitals, or have been developed and validated on small cohorts. The use of large language models (LLM), models with hundreds of millions to billions of parameters, with structured EHR data could potentially lead to improved predictive performance. In this study, we propose DeLLiriuM, a novel LLM-based delirium prediction model using EHR data available in the first 24 hours of ICU admission to predict the probability of a patient developing delirium during the rest of their ICU admission. We develop and validate DeLLiriuM on ICU admissions from 104,303 patients pertaining to 195 hospitals across three large databases: the eICU Collaborative Research Database, the Medical Information Mart for Intensive Care (MIMIC)-IV, and the University of Florida Health's Integrated Data Repository. The performance measured by the area under the receiver operating characteristic curve (AUROC) showed that DeLLiriuM outperformed all baselines in two external validation sets, with 0.77 (95% confidence interval 0.76-0.78) and 0.84 (95% confidence interval 0.83-0.85) across 77,543 patients spanning 194 hospitals. To the best of our knowledge, DeLLiriuM is the first LLM-based delirium prediction tool for the ICU based on structured EHR data, outperforming deep learning baselines which employ structured features and can provide helpful information to clinicians for timely interventions.
On the use of Large Language Models in Model-Driven Engineering
Venue: arXiv cs.SE 2024
Year: 2024

Model-Driven Engineering (MDE) has seen significant advancements with the integration of Machine Learning (ML) and Deep Learning (DL) techniques. Building upon the groundwork of previous investigations, our study provides a concise overview of current Language Large Models (LLMs) applications in MDE, emphasizing their role in automating tasks like model repository classification and developing advanced recommender systems. The paper also outlines the technical considerations for seamlessly integrating LLMs in MDE, offering a practical guide for researchers and practitioners. Looking forward, the paper proposes a focused research agenda for the future interplay of LLMs and MDE, identifying key challenges and opportunities. This concise roadmap envisions the deployment of LLM techniques to enhance the management, exploration, and evolution of modeling ecosystems. By offering a compact exploration of LLMs in MDE, this paper contributes to the ongoing evolution of MDE practices, providing a forward-looking perspective on the transformative role of Language Large Models in software engineering and model-driven practices.
SigCLR: Sigmoid Contrastive Learning of Visual Representations
Venue: arXiv cs.CV 2024
Year: 2024

We propose SigCLR: Sigmoid Contrastive Learning of Visual Representations. SigCLR utilizes the logistic loss that only operates on pairs and does not require a global view as in the cross-entropy loss used in SimCLR. We show that logistic loss shows competitive performance on CIFAR-10, CIFAR-100, and Tiny-IN compared to other established SSL objectives. Our findings verify the importance of learnable bias as in the case of SigLUP, however, it requires a fixed temperature as in the SimCLR to excel. Overall, SigCLR is a promising replacement for the SimCLR which is ubiquitous and has shown tremendous success in various domains.
Interação entre robôs humanoides: desenvolvendo a colaboração e comunicação autônoma
Venue: arXiv cs.RO 2024
Year: 2024

This study investigates the interaction between humanoid robots NAO and Pepper, emphasizing their potential applications in educational settings. NAO, widely used in education, and Pepper, designed for social interactions, of er new opportunities for autonomous communication and collaboration. Through a series of programmed interactions, the robots demonstrated their ability to communicate and coordinate actions autonomously, highlighting their potential as tools for enhancing learning environments. The research also explores the integration of emerging technologies, such as artificial intelligence, into these systems, allowing robots to learn from each other and adapt their behavior. The findings suggest that NAO and Pepper can significantly contribute to both technical learning and the development of social and emotional skills in students, of ering innovative pedagogical approaches through the use of humanoid robotics.
Bauplan: zero-copy, scale-up FaaS for data pipelines
Venue: arXiv cs.DB 2024
Year: 2024

Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness
GDDA: Semantic OOD Detection on Graphs under Covariate Shift via Score-Based Diffusion Models
Venue: arXiv cs.LG 2024
Year: 2024

Out-of-distribution (OOD) detection poses a significant challenge for Graph Neural Networks (GNNs), particularly in open-world scenarios with varying distribution shifts. Most existing OOD detection methods on graphs primarily focus on identifying instances in test data domains caused by either semantic shifts (changes in data classes) or covariate shifts (changes in data features), while leaving the simultaneous occurrence of both distribution shifts under-explored. In this work, we address both types of shifts simultaneously and introduce a novel challenge for OOD detection on graphs: graph-level semantic OOD detection under covariate shift. In this scenario, variations between the training and test domains result from the concurrent presence of both covariate and semantic shifts, where only graphs associated with unknown classes are identified as OOD samples (OODs). To tackle this challenge, we propose a novel two-phase framework called Graph Disentangled Diffusion Augmentation (GDDA). The first phase focuses on disentangling graph representations into domain-invariant semantic factors and domain-specific style factors. In the second phase, we introduce a novel distribution-shift-controlled score-based generative diffusion model that generates latent factors outside the training semantic and style spaces. Additionally, auxiliary pseudo-in-distribution (InD) and pseudo-OOD graph representations are employed to enhance the effectiveness of the energy-based semantic OOD detector. Extensive empirical studies on three benchmark datasets demonstrate that our approach outperforms state-of-the-art baselines.
Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact
Venue: arXiv cs.CL 2024
Year: 2024

Multilingual Large Language Models (MLLMs) represent a pivotal advancement in democratizing artificial intelligence across linguistic boundaries. While theoretical foundations are well-established, practical implementation guidelines remain scattered. This work bridges this gap by providing a comprehensive end-to-end framework for developing and deploying MLLMs in production environments. We make three distinctive contributions: First, we present an actionable pipeline from data pre-processing through deployment, integrating insights from academic research and industrial applications. Second, using Llama2 as a case study, we provide detailed optimization strategies for enhancing multilingual capabilities, including curriculum learning approaches for balancing high-resource and low-resource languages, tokenization strategies, and effective sampling methods. Third, we offer an interdisciplinary analysis that considers technical, linguistic, and cultural perspectives in MLLM development. Our findings reveal critical challenges in supporting linguistic diversity, with 88.38% of world languages categorized as low-resource, affecting over a billion speakers. We examine practical solutions through real-world applications in customer service, search engines, and machine translation. By synthesizing theoretical frameworks with production-ready implementation strategies, this survey provides essential guidance for practitioners and researchers working to develop more inclusive and effective multilingual AI systems.
Improving Connectivity of RIS-Assisted UAV Networks using RIS Partitioning and Deployment
Venue: arXiv cs.IT 2024
Year: 2024

Reconfigurable intelligent surface (RIS) is pivotal for beyond 5G networks in regards to the surge demand for reliable communication in unmanned aerial vehicle (UAV) networks. This paper presents an innovative approach to maximize connectivity of UAV networks using RIS deployment and virtual partitioning, wherein an RIS is deployed to assist in the communications between an user-equipment (UE) and blocked UAVs. Closed-form (CF) expressions for signal-to-noise ratio (SNR) of the two-UAV setup are derived and validated. Then, an optimization problem is formulated to maximize network connectivity by optimizing the 3D deployment of the RIS and its partitioning subject to predefined quality-of-service (QoS) constraints. To tackle this problem, we propose a method of virtually partitioning the RIS given a fixed 3D location, such that the partition phase shifts are configured to create cascaded channels between the UE and the blocked two UAVs. Then, simulated-annealing (SA) method is used to find the 3D location of the RIS. Simulation results demonstrate that the proposed joint RIS deployment and partitioning framework can significantly improve network connectivity compared to benchmarks, including RIS-free and RIS with a single narrow-beam link.
Predicting 30-Day Hospital Readmission in Medicare Patients: Insights from an LSTM Deep Learning Model
Venue: arXiv cs.LG 2024
Year: 2024

Readmissions among Medicare beneficiaries are a major problem for the US healthcare system from a perspective of both healthcare operations and patient caregiving outcomes. Our study analyzes Medicare hospital readmissions using LSTM networks with feature engineering to assess feature contributions. We selected variables from admission-level data, inpatient medical history and patient demography. The LSTM model is designed to capture temporal dynamics from admission-level and patient-level data. On a case study on the MIMIC dataset, the LSTM model outperformed the logistic regression baseline, accurately leveraging temporal features to predict readmission. The major features were the Charlson Comorbidity Index, hospital length of stay, the hospital admissions over the past 6 months, while demographic variables were less impactful. This work suggests that LSTM networks offers a more promising approach to improve Medicare patient readmission prediction. It captures temporal interactions in patient databases, enhancing current prediction models for healthcare providers. Adoption of predictive models into clinical practice may be more effective in identifying Medicare patients to provide early and targeted interventions to improve patient outcomes.
Differentially Private Learning Needs Better Model Initialization and Self-Distillation
Venue: arXiv cs.LG 2024
Year: 2024

Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. We introduce DPRefine, a three-phase method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, applies DP finetuning on private data, and performs self-distillation to refine outputs. This approach significantly outperforms vanilla DPSGD, with AlpacaEval preferring DPRefine's generations in 78.4% of cases across all datasets. Our analysis reveals that DPRefine reduces linguistic errors in generated text by 84.0%, mitigating grammar and spelling errors, commonly associated with DPSGD. It also reduces inconsistencies of non-private models, such as hallucinated details and misattributed quotes. We find that small models like GPT-2 can be effective for initialization and distillation, highlighting their potential in enabling scalable and efficient deployment of privacy-preserving language.
Exploring Tokenization Methods for Multitrack Sheet Music Generation
Venue: arXiv cs.SD 2024
Year: 2024

This study explores the tokenization of multitrack sheet music in ABC notation, introducing two methods--bar-stream and line-stream patching. We compare these methods against existing techniques, including bar patching, byte patching, and Byte Pair Encoding (BPE). In terms of both computational efficiency and the musicality of the generated compositions, experimental results show that bar-stream patching performs best overall compared to the others, which makes it a promising tokenization strategy for sheet music generation.
Pillow Box Design
Venue: arXiv cs.CG 2024
Year: 2024

This paper focuses on packaging design using origami techniques, specifically designs incorporating curves, known as pillow boxes. While conventional paper packaging boxes are typically cuboid, pillow box designs include curved surfaces, offering both aesthetic and practical advantages. This study analyzes the specific curved folds of pillow boxes, clarifying the fundamental geometric condition these curves must meet. Additionally, it proposes new design variations for pillow boxes based on the condition. The relationship between the shape of the folds and the volume of the final three-dimensional shape is also explored. This research extends the boundaries of functionality and aesthetics in origami design and explores new possibilities in packaging solutions.