OpenRaiser/Intern-Atlas huggingface dataset:
Graph Sparsification via Mixture of Graphs
Venue: arXiv cs.LG 2024
Year: 2024

Graph Neural Networks (GNNs) have demonstrated superior performance across various graph learning tasks but face significant computational challenges when applied to large-scale graphs. One effective approach to mitigate these challenges is graph sparsification, which involves removing non-essential edges to reduce computational overhead. However, previous graph sparsification methods often rely on a single global sparsity setting and uniform pruning criteria, failing to provide customized sparsification schemes for each node's complex local context. In this paper, we introduce Mixture-of-Graphs (MoG), leveraging the concept of Mixture-of-Experts (MoE), to dynamically select tailored pruning solutions for each node. Specifically, MoG incorporates multiple sparsifier experts, each characterized by unique sparsity levels and pruning criteria, and selects the appropriate experts for each node. Subsequently, MoG performs a mixture of the sparse graphs produced by different experts on the Grassmann manifold to derive an optimal sparse graph. One notable property of MoG is its entirely local nature, as it depends on the specific circumstances of each individual node. Extensive experiments on four large-scale OGB datasets and two superpixel datasets, equipped with five GNN backbones, demonstrate that MoG (I) identifies subgraphs at higher sparsity levels ($8.67\%\sim 50.85\%$), with performance equal to or better than the dense graph, (II) achieves $1.47-2.62\times$ speedup in GNN inference with negligible performance drop, and (III) boosts ``top-student'' GNN performance ($1.02\%\uparrow$ on RevGNN+\textsc{ogbn-proteins} and $1.74\%\uparrow$ on DeeperGCN+\textsc{ogbg-ppa}).
Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis
Venue: arXiv cs.CL 2024
Year: 2024

Training LLMs for low-resource languages usually utilizes data augmentation from English using machine translation (MT). This, however, brings a number of challenges to LLM training: there are large costs attached to translating and curating huge amounts of content with high-end machine translation solutions; the translated content carries over cultural biases; and if the translation is not faithful and accurate, data quality degrades causing issues in the trained model. In this work, we investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the open NLLB-3B MT model. We train a number of story generation models of size 1M-33M parameters using this data. We identify a number of quality and task-specific issues in the resulting models. To rectify these issues, we further pre-train the models with a small dataset of synthesized high-quality Arabic stories generated by a capable LLM, representing 1% of the original training data. We show, using GPT-4 as a judge and Dictionary Learning Analysis from mechanistic interpretability, that the suggested approach is a practical means to resolve some of the machine translation pitfalls. We illustrate the improvements through case studies of linguistic and cultural bias issues.
Tuning-free Universally-Supervised Semantic Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

This work presents a tuning-free semantic segmentation framework based on classifying SAM masks by CLIP, which is universally applicable to various types of supervision. Initially, we utilize CLIP's zero-shot classification ability to generate pseudo-labels or perform open-vocabulary segmentation. However, the misalignment between mask and CLIP text embeddings leads to suboptimal results. To address this issue, we propose discrimination-bias aligned CLIP to closely align mask and text embedding, offering an overhead-free performance gain. We then construct a global-local consistent classifier to classify SAM masks, which reveals the intrinsic structure of high-quality embeddings produced by DBA-CLIP and demonstrates robustness against noisy pseudo-labels. Extensive experiments validate the efficiency and effectiveness of our method, and we achieve state-of-the-art (SOTA) or competitive performance across various datasets and supervision types.
Focus Anywhere for Fine-grained Multi-page Document Understanding
Venue: arXiv cs.CV 2024
Year: 2024

Modern LVLMs still struggle to achieve fine-grained document understanding, such as OCR/translation/caption for regions of interest to the user, tasks that require the context of the entire page, or even multiple pages. Accordingly, this paper proposes Fox, an effective pipeline, hybrid data, and tuning strategy, that catalyzes LVLMs to focus anywhere on single/multi-page documents. We introduce a novel task to boost the document understanding by making LVLMs focus attention on the document-level region, such as redefining full-page OCR as foreground focus. We employ multiple vision vocabularies to extract visual hybrid knowledge for interleaved document pages (e.g., a page containing a photo). Meanwhile, we render cross-vocabulary vision data as the catalyzer to achieve a full reaction of multiple visual vocabularies and in-document figure understanding. Further, without modifying the weights of multiple vision vocabularies, the above catalyzed fine-grained understanding capabilities can be efficiently tuned to multi-page documents, enabling the model to focus anywhere in both format-free and page-free manners. Besides, we build a benchmark including 9 fine-grained sub-tasks (e.g., region-level OCR/summary, color-guided OCR) to promote document analysis in the community. The experimental results verify the superiority of our model.
Smooth Pseudo-Labeling
Venue: arXiv cs.LG 2024
Year: 2024

Semi-Supervised Learning (SSL) seeks to leverage large amounts of non-annotated data along with the smallest amount possible of annotated data in order to achieve the same level of performance as if all data were annotated. A fruitful method in SSL is Pseudo-Labeling (PL), which, however, suffers from the important drawback that the associated loss function has discontinuities in its derivatives, which cause instabilities in performance when labels are very scarce. In the present work, we address this drawback with the introduction of a Smooth Pseudo-Labeling (SP L) loss function. It consists in adding a multiplicative factor in the loss function that smooths out the discontinuities in the derivative due to thresholding. In our experiments, we test our improvements on FixMatch and show that it significantly improves the performance in the regime of scarce labels, without addition of any modules, hyperparameters, or computational overhead. In the more stable regime of abundant labels, performance remains at the same level. Robustness with respect to variation of hyperparameters and training parameters is also significantly improved. Moreover, we introduce a new benchmark, where labeled images are selected randomly from the whole dataset, without imposing representation of each class proportional to its frequency in the dataset. We see that the smooth version of FixMatch does appear to perform better than the original, non-smooth implementation. However, more importantly, we notice that both implementations do not necessarily see their performance improve when labeled images are added, an important issue in the design of SSL algorithms that should be addressed so that Active Learning algorithms become more reliable and explainable.
Estimating the Expected Social Welfare and Cost of Random Serial Dictatorship
Venue: arXiv cs.GT 2024
Year: 2024

We consider the assignment problem, where $n$ agents have to be matched to $n$ items. Each agent has a preference order over the items. In the serial dictatorship (SD) mechanism the agents act in a particular order and pick their most preferred available item when it is their turn to act. Applying SD using a uniformly random permutation as agent ordering results in the well-known random serial dictatorship (RSD) mechanism. Accurate estimates of the (expected) efficiency of its outcome can be used to assess whether RSD is attractive compared to other mechanisms. In this paper, we explore whether such estimates are possible by sampling a (hopefully) small number of agent orderings and applying SD using them. We consider a value setting in which agents have values for the items as well as a metric cost setting where agents and items are assumed to be points in a metric space, and the cost of an agent for an item is equal to the distance of the corresponding points. We show that a (relatively) small number of samples is enough to approximate the expected social welfare of RSD in the value setting and its expected social cost in the metric cost setting despite the #P-hardness of the corresponding exact computation problems.
SmartCS: Enabling the Creation of ML-Powered Computer Vision Mobile Apps for Citizen Science Applications without Coding
Venue: arXiv cs.CY 2024
Year: 2024

It is undeniable that citizen science contributes to the advancement of various fields of study. There are now software tools that facilitate the development of citizen science apps. However, apps developed with these tools rely on individual human skills to correctly collect useful data. Machine learning (ML)-aided apps provide on-field guidance to citizen scientists on data collection tasks. However, these apps rely on server-side ML support, and therefore need a reliable internet connection. Furthermore, the development of citizen science apps with ML support requires a significant investment of time and money. For some projects, this barrier may preclude the use of citizen science effectively. We present a platform that democratizes citizen science by making it accessible to a much broader audience of both researchers and participants. The SmartCS platform allows one to create citizen science apps with ML support quickly and without coding skills. Apps developed using SmartCS have client-side ML support, making them usable in the field, even when there is no internet connection. The client-side ML helps educate users to better recognize the subjects, thereby enabling high-quality data collection. We present several citizen science apps created using SmartCS, some of which were conceived and created by high school students.
LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision
Venue: arXiv cs.CV 2024
Year: 2024

Prototypical parts networks combine the power of deep learning with the explainability of case-based reasoning to make accurate, interpretable decisions. They follow the this looks like that reasoning, representing each prototypical part with patches from training images. However, a single image patch comprises multiple visual features, such as color, shape, and texture, making it difficult for users to identify which feature is important to the model. To reduce this ambiguity, we introduce the Lucid Prototypical Parts Network (LucidPPN), a novel prototypical parts network that separates color prototypes from other visual features. Our method employs two reasoning branches: one for non-color visual features, processing grayscale images, and another focusing solely on color information. This separation allows us to clarify whether the model's decisions are based on color, shape, or texture. Additionally, LucidPPN identifies prototypical parts corresponding to semantic parts of classified objects, making comparisons between data classes more intuitive, e.g., when two bird species might differ primarily in belly color. Our experiments demonstrate that the two branches are complementary and together achieve results comparable to baseline methods. More importantly, LucidPPN generates less ambiguous prototypical parts, enhancing user understanding.
Expert exploranation for communicating scientific methods -- A case study in conflict research
Venue: arXiv cs.CY 2024
Year: 2024

Science communication aims at making key research insights accessible to the broad public. If explanatory and exploratory visualization techniques are combined to do so, the approach is also referred to as exploranation. In this context, the audience is usually not required to have domain expertise. However, we show that exploranation can not only support the communication between researchers and a broad audience, but also between researchers directly. With the goal of communicating an existing method for conducting causal inference on spatio-temporal conflict event data, we investigated how to perform exploranation for experts, i.e., expert exploranation. Based on application scenarios of the inference method, we developed three versions of an interactive visual story to explain the method to conflict researchers. We abstracted the corresponding design process and evaluated the stories both with experts who were unfamiliar with the explained method and experts who were already familiar with it. The positive and extensive feedback from the evaluation shows that expert exploranation is a promising direction for visual storytelling, as it can help to improve scientific outreach, methodological understanding, and accessibility for researchers new to a field.
Retrieval-Augmented Mining of Temporal Logic Specifications from Data
Venue: arXiv cs.LG 2024
Year: 2024

The integration of cyber-physical systems (CPS) into everyday life raises the critical necessity of ensuring their safety and reliability. An important step in this direction is requirement mining, i.e. inferring formally specified system properties from observed behaviors, in order to discover knowledge about the system. Signal Temporal Logic (STL) offers a concise yet expressive language for specifying requirements, particularly suited for CPS, where behaviors are typically represented as time series data. This work addresses the task of learning STL requirements from observed behaviors in a data-driven manner, focusing on binary classification, i.e. on inferring properties of the system which are able to discriminate between regular and anomalous behaviour, and that can be used both as classifiers and as monitors of the compliance of the CPS to desirable specifications. We present a novel framework that combines Bayesian Optimization (BO) and Information Retrieval (IR) techniques to simultaneously learn both the structure and the parameters of STL formulae, without restrictions on the STL grammar. Specifically, we propose a framework that leverages a dense vector database containing semantic-preserving continuous representations of millions of formulae, queried for facilitating the mining of requirements inside a BO loop. We demonstrate the effectiveness of our approach in several signal classification applications, showing its ability to extract interpretable insights from system executions and advance the state-of-the-art in requirement mining for CPS.
Optimal Whole Body Trajectory Planning for Mobile Manipulators in Planetary Exploration and Construction
Venue: arXiv cs.RO 2024
Year: 2024

Space robotics poses unique challenges arising from the limitation of energy and computational resources, and the complexity of the environment and employed platforms. At the control center, offline motion planning is fundamental in the computation of optimized trajectories accounting for the system's constraints. Smooth movements, collision and forbidden areas avoidance, target visibility and energy consumption are all important factors to consider to be able to generate feasible and optimal plans. When mobile manipulators (terrestrial, aerial) are employed, the base and the arm movements are often separately planned, ultimately resulting in sub-optimal solutions. We propose an Optimal Whole Body Planner (OptiWB) based on Discrete Dynamic Programming (DDP) and optimal interpolation. Kinematic redundancy is exploited for collision and forbidden areas avoidance, and to improve target illumination and visibility from onboard cameras. The planner, implemented in ROS (Robot Operating System), interfaces 3DROCS, a mission planner used in several programs of the European Space Agency (ESA) to support planetary exploration surface missions and part of the ExoMars Rover's planning software. The proposed approach is exercised on a simplified version of the Analog-1 Interact rover by ESA, a 7-DOFs robotic arm mounted on a four wheels non-holonomic platform.
EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Venue: arXiv cs.DC 2024
Year: 2024

Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively. Experiments of Llama2 serial models on a heterogeneous physical prototype demonstrate that EdgeShard achieves up to 50% latency reduction and 2x throughput improvement over baseline methods.
Reliable Trajectory Prediction and Uncertainty Quantification with Conditioned Diffusion Models
Venue: arXiv cs.LG 2024
Year: 2024

This work introduces the conditioned Vehicle Motion Diffusion (cVMD) model, a novel network architecture for highway trajectory prediction using diffusion models. The proposed model ensures the drivability of the predicted trajectory by integrating non-holonomic motion constraints and physical constraints into the generative prediction module. Central to the architecture of cVMD is its capacity to perform uncertainty quantification, a feature that is crucial in safety-critical applications. By integrating the quantified uncertainty into the prediction process, the cVMD's trajectory prediction performance is improved considerably. The model's performance was evaluated using the publicly available highD dataset. Experiments show that the proposed architecture achieves competitive trajectory prediction accuracy compared to state-of-the-art models, while providing guaranteed drivable trajectories and uncertainty quantification.
Explainable Few-shot Knowledge Tracing
Venue: arXiv cs.AI 2024
Year: 2024

Knowledge tracing (KT), aiming to mine students' mastery of knowledge by their exercise records and predict their performance on future test questions, is a critical task in educational assessment. While researchers achieved tremendous success with the rapid development of deep learning techniques, current knowledge tracing tasks fall into the cracks from real-world teaching scenarios. Relying heavily on extensive student data and solely predicting numerical performances differs from the settings where teachers assess students' knowledge state from limited practices and provide explanatory feedback. To fill this gap, we explore a new task formulation: Explainable Few-shot Knowledge Tracing. By leveraging the powerful reasoning and generation abilities of large language models (LLMs), we then propose a cognition-guided framework that can track the student knowledge from a few student records while providing natural language explanations. Experimental results from three widely used datasets show that LLMs can perform comparable or superior to competitive deep knowledge tracing methods. We also discuss potential directions and call for future improvements in relevant topics.
Qualifying and Quantifying the Benefits of Mindfulness Practices for IT Workers
Venue: arXiv cs.SE 2024
Year: 2024

The well-being and productivity of IT workers are crucial for both individual success and the overall prosperity of the organisations they serve. This study proposes mindfulness to alleviate stress and improve mental well-being for IT workers. During an 8-week program, IT workers learn about mindfulness, coupled with breathing practices. This study investigates the potential effects of these practices by analysing participants' reflections through thematic analysis and daily well-being ratings. The analysis showcased an increase in mental well-being and perceived productivity. It also indicated a change in the participants' perception, which showed increased self-awareness. The study recommends continuing the program in the industry to see its impact on work outputs.
Large Language Models for Explainable Decisions in Dynamic Digital Twins
Venue: arXiv cs.AI 2024
Year: 2024

Dynamic data-driven Digital Twins (DDTs) can enable informed decision-making and provide an optimisation platform for the underlying system. By leveraging principles of Dynamic Data-Driven Applications Systems (DDDAS), DDTs can formulate computational modalities for feedback loops, model updates and decision-making, including autonomous ones. However, understanding autonomous decision-making often requires technical and domain-specific knowledge. This paper explores using large language models (LLMs) to provide an explainability platform for DDTs, generating natural language explanations of the system's decision-making by leveraging domain-specific knowledge bases. A case study from smart agriculture is presented.
GeoFaaS: An Edge-to-Cloud FaaS Platform
Venue: arXiv cs.DC 2024
Year: 2024

The massive growth of mobile and IoT devices demands geographically distributed computing systems for optimal performance, privacy, and scalability. However, existing edge-to-cloud serverless platforms lack location awareness, resulting in inefficient network usage and increased latency. In this paper, we propose GeoFaaS, a novel edge-to-cloud Function-as-a-Service (FaaS) platform that leverages real-time client location information for transparent request execution on the nearest available FaaS node. If needed, GeoFaaS transparently offloads requests to the cloud when edge resources are overloaded, thus, ensuring consistent execution without user intervention. GeoFaaS has a modular and decentralized architecture: building on the single-node FaaS system tinyFaaS, GeoFaaS works as a stand-alone edge-to-cloud FaaS platform but can also integrate and act as a routing layer for existing FaaS services, e.g., in the cloud. To evaluate our approach, we implemented an open-source proof-of-concept prototype and studied performance and fault-tolerance behavior in experiments.