OpenRaiser/Intern-Atlas huggingface dataset:
DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documents
Venue: arXiv cs.AI 2024
Year: 2024

Information extraction from handwritten documents involves traditionally three distinct steps: Document Layout Analysis, Handwritten Text Recognition, and Named Entity Recognition. Recent approaches have attempted to integrate these steps into a single process using fully end-to-end architectures. Despite this, these integrated approaches have not yet matched the performance of language models, when applied to information extraction in plain text. In this paper, we introduce DANIEL (Document Attention Network for Information Extraction and Labelling), a fully end-to-end architecture integrating a language model and designed for comprehensive handwritten document understanding. DANIEL performs layout recognition, handwriting recognition, and named entity recognition on full-page documents. Moreover, it can simultaneously learn across multiple languages, layouts, and tasks. For named entity recognition, the ontology to be applied can be specified via the input prompt. The architecture employs a convolutional encoder capable of processing images of any size without resizing, paired with an autoregressive decoder based on a transformer-based language model. DANIEL achieves competitive results on four datasets, including a new state-of-the-art performance on RIMES 2009 and M-POPP for Handwriting Text Recognition, and IAM NER for Named Entity Recognition. Furthermore, DANIEL is much faster than existing approaches. We provide the source code and the weights of the trained models at \url{https://github.com/Shulk97/daniel}.
The Magic XRoom: A Flexible VR Platform for Controlled Emotion Elicitation and Recognition
Venue: arXiv cs.HC 2024
Year: 2024

Affective computing has recently gained popularity, especially in the field of human-computer interaction systems, where effectively evoking and detecting emotions is of paramount importance to enhance users experience. However, several issues are hindering progress in the field. In fact, the complexity of emotions makes it difficult to understand their triggers and control their elicitation. Additionally, effective emotion recognition requires analyzing multiple sensor data, such as facial expressions and physiological signals. These factors combined make it hard to collect high-quality datasets that can be used for research purposes (e.g., development of emotion recognition algorithms). Despite these challenges, Virtual Reality (VR) holds promise as a solution. By providing a controlled and immersive environment, VR enables the replication of real-world emotional experiences and facilitates the tracking of signals indicative of emotional states. However, controlling emotion elicitation remains a challenging task also within VR. This research paper introduces the Magic Xroom, a VR platform designed to enhance control over emotion elicitation by leveraging the theory of flow. This theory establishes a mapping between an individuals skill levels, task difficulty, and perceived emotions. In the Magic Xroom, the users skill level is continuously assessed, and task difficulty is adjusted accordingly to evoke specific emotions. Furthermore, user signals are collected using sensors, and virtual panels are utilized to determine the ground truth emotional states, making the Magic Xroom an ideal platform for collecting extensive datasets. The paper provides detailed implementation information, highlights the main properties of the Magic Xroom, and presents examples of virtual scenarios to illustrate its abilities and capabilities.
Inference Optimization of Foundation Models on AI Accelerators
Venue: arXiv cs.AI 2024
Year: 2024

Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI across various industries. Industry and research community have witnessed a large number of new applications, based on those foundation models. Such applications include question and answer, customer services, image and video generation, and code completions, among others. However, as the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. As a result, the demand for cost-effective and fast inference using AI accelerators is ever more higher. To this end, our tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators. Beginning with an overview of basic Transformer architectures and deep learning system frameworks, we deep dive into system optimization techniques for fast and memory-efficient attention computations and discuss how they can be implemented efficiently on AI accelerators. Next, we describe architectural elements that are key for fast transformer inference. Finally, we examine various model compression and fast decoding strategies in the same context.
Decentralized multi-agent reinforcement learning algorithm using a cluster-synchronized laser network
Venue: arXiv cs.LG 2024
Year: 2024

Multi-agent reinforcement learning (MARL) studies crucial principles that are applicable to a variety of fields, including wireless networking and autonomous driving. We propose a photonic-based decision-making algorithm to address one of the most fundamental problems in MARL, called the competitive multi-armed bandit (CMAB) problem. Our numerical simulations demonstrate that chaotic oscillations and cluster synchronization of optically coupled lasers, along with our proposed decentralized coupling adjustment, efficiently balance exploration and exploitation while facilitating cooperative decision-making without explicitly sharing information among agents. Our study demonstrates how decentralized reinforcement learning can be achieved by exploiting complex physical processes controlled by simple algorithms.
AI-Powered Immersive Assistance for Interactive Task Execution in Industrial Environments
Venue: arXiv cs.HC 2024
Year: 2024

Many industrial sectors rely on well-trained employees that are able to operate complex machinery. In this work, we demonstrate an AI-powered immersive assistance system that supports users in performing complex tasks in industrial environments. Specifically, our system leverages a VR environment that resembles a juice mixer setup. This digital twin of a physical setup simulates complex industrial machinery used to mix preparations or liquids (e.g., similar to the pharmaceutical industry) and includes various containers, sensors, pumps, and flow controllers. This setup demonstrates our system's capabilities in a controlled environment while acting as a proof-of-concept for broader industrial applications. The core components of our multimodal AI assistant are a large language model and a speech-to-text model that process a video and audio recording of an expert performing the task in a VR environment. The video and speech input extracted from the expert's video enables it to provide step-by-step guidance to support users in executing complex tasks. This demonstration showcases the potential of our AI-powered assistant to reduce cognitive load, increase productivity, and enhance safety in industrial environments.
Simple Linear Loops: Algebraic Invariants and Applications
Venue: arXiv cs.CC 2024
Year: 2024

The automatic generation of loop invariants is a fundamental challenge in software verification. While this task is undecidable in general, it is decidable for certain restricted classes of programs. This work focuses on invariant generation for (branching-free) loops with a single linear update. Our primary contribution is a polynomial-space algorithm that computes the strongest algebraic invariant for simple linear loops, generating all polynomial equations that hold among program variables across all reachable states. The key to achieving our complexity bounds lies in mitigating the blowup associated with variable elimination and Gröbner basis computation, as seen in prior works. Our procedure runs in polynomial time when the number of program variables is fixed. We examine various applications of our results on invariant generation, focusing on invariant verification and loop synthesis. The invariant verification problem investigates whether a polynomial ideal defining an algebraic set serves as an invariant for a given linear loop. We show that this problem is coNP-complete and lies in PSPACE when the input ideal is given in dense or sparse representations, respectively. In the context of loop synthesis, we aim to construct a loop with an infinite set of reachable states that upholds a specified algebraic property as an invariant. The strong synthesis variant of this problem requires the construction of loops for which the given property is the strongest invariant. In terms of hardness, synthesising loops over integers (or rationals) is as hard as Hilbert's Tenth problem (or its analogue over the rationals). When the constants of the output are constrained to bit-bounded rational numbers, we demonstrate that loop synthesis and its strong variant are both decidable in PSPACE, and in NP when the number of program variables is fixed.
Weakly-supervised Autism Severity Assessment in Long Videos
Venue: arXiv cs.CV 2024
Year: 2024

Autism Spectrum Disorder (ASD) is a diverse collection of neurobiological conditions marked by challenges in social communication and reciprocal interactions, as well as repetitive and stereotypical behaviors. Atypical behavior patterns in a long, untrimmed video can serve as biomarkers for children with ASD. In this paper, we propose a video-based weakly-supervised method that takes spatio-temporal features of long videos to learn typical and atypical behaviors for autism detection. On top of that, we propose a shallow TCN-MLP network, which is designed to further categorize the severity score. We evaluate our method on actual evaluation videos of children with autism collected and annotated (for severity score) by clinical professionals. Experimental results demonstrate the effectiveness of behavioral biomarkers that could help clinicians in autism spectrum analysis.
iMIV: in-Memory Integrity Verification for NVM
Venue: arXiv cs.AR 2024
Year: 2024

Non-volatile Memory (NVM) could bridge the gap between memory and storage. However, NVMs are susceptible to data remanence attacks. Thus, multiple security metadata must persist along with the data to protect the confidentiality and integrity of NVM-resident data. Persisting Bonsai Merkel Tree (BMT) nodes, critical for data integrity, can add significant overheads due to need to write large amounts of metadata off-chip to the bandwidth-constrained NVMs. We propose iMIV for low-overhead, fine-grained integrity verification through in-memory computing. We argue that memory-intensive integrity verification operations (BMT updates and verification) should be employed close to the NVM to limit off-chip data movement. We design iMIV based on typical NVDIMM designs that have an onboard logic chip with a trusted encryption engine, separate from the untrusted storage media. iMIV reduces the performance overheads from 205% to 55% when integrity verification operations are offloaded to NVM compared to when all the security operations are employed at the memory controller.
Generative Models for Synthetic Urban Mobility Data: A Systematic Literature Review
Venue: arXiv cs.CR 2024
Year: 2024

Although highly valuable for a variety of applications, urban mobility data is rarely made openly available as it contains sensitive personal information. Synthetic data aims to solve this issue by generating artificial data that resembles an original dataset in structural and statistical characteristics, but omits sensitive information. For mobility data, a large number of corresponding models have been proposed in the last decade. This systematic review provides a structured comparative overview of the current state of this heterogeneous, active field of research. A special focus is put on the applicability of the reviewed models in practice.
Optimized Federated Multitask Learning in Mobile Edge Networks: A Hybrid Client Selection and Model Aggregation Approach
Venue: arXiv cs.NI 2024
Year: 2024

We propose clustered federated multitask learning to address statistical challenges in non-independent and identically distributed data across clients. Our approach tackles complexities in hierarchical wireless networks by clustering clients based on data distribution similarities and assigning specialized models to each cluster. These complexities include slower convergence and mismatched model allocation due to hierarchical model aggregation and client selection. The proposed framework features a two-phase client selection and a two-level model aggregation scheme. It ensures fairness and effective participation using greedy and round-robin methods. Our approach significantly enhances convergence speed, reduces training time, and decreases energy consumption by up to 60%, ensuring clients receive models tailored to their specific data needs.
Cool URIs for FAIR Knowledge Graphs
Venue: arXiv cs.DL 2024
Year: 2024

This guide is for everyone who seeks advice for creating stable, secure, and persistent Uniform Resource Identifiers (URIs) in order to publish their data in accordance to the FAIR principles. The use case does not matter. It could range from publishing the results of a small research project to a large knowledge graph at a big corporation. The FAIR principles apply equally and this is why it is important to put extra thought into the URI selection process. The title aims to extend the tradition of "Cool URIs don't change" and "Cool URIs for the Semantic Web". Much has changed since the publication of these works and we would like to revisit some of the principles. Many still hold today, some had to be reworked, and we could also identify new ones
FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
Venue: arXiv cs.NI 2024
Year: 2024

Addressing the challenges of deploying large language models in wireless communication networks, this paper combines low-rank adaptation technology (LoRA) with the splitfed learning framework to propose the federated split learning for large language models (FedsLLM) framework. The method introduced in this paper utilizes LoRA technology to reduce processing loads by dividing the network into client subnetworks and server subnetworks. It leverages a federated server to aggregate and update client models. As the training data are transmitted through a wireless network between clients and both main and federated servers, the training delay is determined by the learning accuracy and the allocation of communication bandwidth. This paper models the minimization of the training delay by integrating computation and communication optimization, simplifying the optimization problem into a convex problem to find the optimal solution. Additionally, it presents a lemma that describes the precise solutions to this problem. Simulation results demonstrate that the proposed optimization algorithm reduces delays by an average of 47.63% compared to unoptimized scenarios.
Provable Privacy Advantages of Decentralized Federated Learning via Distributed Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Federated learning (FL) emerged as a paradigm designed to improve data privacy by enabling data to reside at its source, thus embedding privacy as a core consideration in FL architectures, whether centralized or decentralized. Contrasting with recent findings by Pasquini et al., which suggest that decentralized FL does not empirically offer any additional privacy or security benefits over centralized models, our study provides compelling evidence to the contrary. We demonstrate that decentralized FL, when deploying distributed optimization, provides enhanced privacy protection - both theoretically and empirically - compared to centralized approaches. The challenge of quantifying privacy loss through iterative processes has traditionally constrained the theoretical exploration of FL protocols. We overcome this by conducting a pioneering in-depth information-theoretical privacy analysis for both frameworks. Our analysis, considering both eavesdropping and passive adversary models, successfully establishes bounds on privacy leakage. We show information theoretically that the privacy loss in decentralized FL is upper bounded by the loss in centralized FL. Compared to the centralized case where local gradients of individual participants are directly revealed, a key distinction of optimization-based decentralized FL is that the relevant information includes differences of local gradients over successive iterations and the aggregated sum of different nodes' gradients over the network. This information complicates the adversary's attempt to infer private data. To bridge our theoretical insights with practical applications, we present detailed case studies involving logistic regression and deep neural networks. These examples demonstrate that while privacy leakage remains comparable in simpler models, complex models like deep neural networks exhibit lower privacy risks under decentralized FL.
Towards Personalised Patient Risk Prediction Using Temporal Hospital Data Trajectories
Venue: arXiv cs.AI 2024
Year: 2024

Quantifying a patient's health status provides clinicians with insight into patient risk, and the ability to better triage and manage resources. Early Warning Scores (EWS) are widely deployed to measure overall health status, and risk of adverse outcomes, in hospital patients. However, current EWS are limited both by their lack of personalisation and use of static observations. We propose a pipeline that groups intensive care unit patients by the trajectories of observations data throughout their stay as a basis for the development of personalised risk predictions. Feature importance is considered to provide model explainability. Using the MIMIC-IV dataset, six clusters were identified, capturing differences in disease codes, observations, lengths of admissions and outcomes. Applying the pipeline to data from just the first four hours of each ICU stay assigns the majority of patients to the same cluster as when the entire stay duration is considered. In-hospital mortality prediction models trained on individual clusters had higher F1 score performance in five of the six clusters when compared against the unclustered patient cohort. The pipeline could form the basis of a clinical decision support tool, working to improve the clinical characterisation of risk groups and the early detection of patient deterioration.
Open-Canopy: Towards Very High Resolution Forest Monitoring
Venue: arXiv cs.CV 2024
Year: 2024

Estimating canopy height and its changes at meter resolution from satellite imagery is a significant challenge in computer vision with critical environmental applications. However, the lack of open-access datasets at this resolution hinders the reproducibility and evaluation of models. We introduce Open-Canopy, the first open-access, country-scale benchmark for very high-resolution (1.5 m) canopy height estimation, covering over 87,000 km$^2$ across France with 1.5 m resolution satellite imagery and aerial LiDAR data. Additionally, we present Open-Canopy-$Δ$, a benchmark for canopy height change detection between images from different years at tree level-a challenging task for current computer vision models. We evaluate state-of-the-art architectures on these benchmarks, highlighting significant challenges and opportunities for improvement. Our datasets and code are publicly available at https://github.com/fajwel/Open-Canopy.
GazeRace: Revolutionizing Remote Piloting with Eye-Gaze Control
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents GazeRace, a novel system that leverages eye-tracking technology for intuitive drone control. Using the MediaPipe library, the system translates eye movements into precise drone commands, enabling effective remote piloting. In testing, GazeRace demonstrated an 18% reduction in drone trajectory length while maintaining competitive speed with traditional controls. The results suggest that this approach enhances control accuracy and reduces user frustration, offering a significant advancement in the field of human-computer interaction and drone navigation.
Interactive Coding with Unbounded Noise
Venue: arXiv cs.DS 2024
Year: 2024

Interactive coding allows two parties to conduct a distributed computation despite noise corrupting a certain fraction of their communication. Dani et al.\@ (Inf.\@ and Comp., 2018) suggested a novel setting in which the amount of noise is unbounded and can significantly exceed the length of the (noise-free) computation. While no solution is possible in the worst case, under the restriction of oblivious noise, Dani et al.\@ designed a coding scheme that succeeds with a polynomially small failure probability. We revisit the question of conducting computations under this harsh type of noise and devise a computationally-efficient coding scheme that guarantees the success of the computation, except with an exponentially small probability. This higher degree of correctness matches the case of coding schemes with a bounded fraction of noise. Our simulation of an $N$-bit noise-free computation in the presence of $T$ corruptions, communicates an optimal number of $O(N+T)$ bits and succeeds with probability $1-2^{-Ω(N)}$. We design this coding scheme by introducing an intermediary noise model, where an oblivious adversary can choose the locations of corruptions in a worst-case manner, but the effect of each corruption is random: the noise either flips the transmission with some probability or otherwise erases it. This randomized abstraction turns out to be instrumental in achieving an optimal coding scheme.
Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecasting
Venue: arXiv cs.RO 2024
Year: 2024

Deep learning-based trajectory prediction models for autonomous driving often struggle with generalization to out-of-distribution (OOD) scenarios, sometimes performing worse than simple rule-based models. To address this limitation, we propose a novel framework, Adaptive Prediction Ensemble (APE), which integrates deep learning and rule-based prediction experts. A learned routing function, trained concurrently with the deep learning model, dynamically selects the most reliable prediction based on the input scenario. Our experiments on large-scale datasets, including Waymo Open Motion Dataset (WOMD) and Argoverse, demonstrate improvement in zero-shot generalization across datasets. We show that our method outperforms individual prediction models and other variants, particularly in long-horizon prediction and scenarios with a high proportion of OOD data. This work highlights the potential of hybrid approaches for robust and generalizable motion prediction in autonomous driving. More details can be found on the project page: https://sites.google.com/view/ape-generalization.
GPTutor: Great Personalized Tutor with Large Language Models for Personalized Learning Content Generation
Venue: arXiv cs.HC 2024
Year: 2024

We developed GPTutor, a pioneering web application designed to revolutionize personalized learning by leveraging the capabilities of Generative AI at scale. GPTutor adapts educational content and practice exercises to align with individual students' interests and career goals, enhancing their engagement and understanding of critical academic concepts. The system uses a serverless architecture to deliver personalized and scalable learning experiences. By integrating advanced Chain-of-Thoughts prompting methods, GPTutor provides a personalized educational journey that not only addresses the unique interests of each student but also prepares them for future professional success. This demo paper presents the design, functionality, and potential of GPTutor to foster a more engaging and effective educational environment.