OpenRaiser/Intern-Atlas huggingface dataset:
TumblerBots: Tumbling Robotic sensors for Minimally-invasive Benthic Monitoring
Venue: arXiv cs.RO 2024
Year: 2024

Robotic systems show significant promise for water environmental sensing applications such as water quality monitoring, pollution mapping and biodiversity data collection. Conventional deployment methods often disrupt fragile ecosystems, preventing depiction of the undisturbed environmental condition. In response to this challenge, we propose a novel framework utilizing a lightweight tumbler system equipped with a sensing unit, deployed via a drone. This design minimizes disruption to the water habitat by maintaining a slow descent. The sensing unit is detached once on the water surface, enabling precise and non-invasive data collection from the benthic zone. The tumbler is designed to be lightweight and compact, enabling deployment via a drone. The sensing pod, which detaches from the tumbler and descends to the bottom of the water body, is equipped with temperature and pressure sensors, as well as a buoyancy system. The later, activated upon task completion, utilizes a silicon membrane inflated via a chemical reaction. The reaction generates a pressure of 70 kPa, causing the silicon membrane to expand by 30\%, which exceeds the 5.7\% volume increase required for positive buoyancy. The tumblers, made from ecofriendly materials to minimize environmental impact when lost during the mission, were tested for their gliding ratio and descent rate. They exhibit a low descent rate, in the range of 0.8 to 2.5 meters per seconds, which minimizes disturbance to the ecosystem upon water landing. Additionally, the system demonstrated robustness in moderate to strong wind conditions during outdoor tests, validating the overall framework.
PIP-MM: Pre-Integrating Prompt Information into Visual Encoding via Existing MLLM Structures
Venue: arXiv cs.CV 2024
Year: 2024

The Multimodal Large Language Models (MLLMs) have activated the capabilitiesof Large Language Models (LLMs) in solving visual-language tasks by integratingvisual information. The prevailing approach in existing MLLMs involvesemploying an image encoder to extract visual features, converting thesefeatures into visual tokens via an adapter, and then integrating them with theprompt into the LLM. However, because the process of image encoding isprompt-agnostic, the extracted visual features only provide a coarsedescription of the image, impossible to focus on the requirements of theprompt. On one hand, it is easy for image features to lack information aboutthe prompt-specified objects, resulting in unsatisfactory responses. On theother hand, the visual features contain a large amount of irrelevantinformation, which not only increases the burden on memory but also worsens thegeneration effectiveness. To address the aforementioned issues, we propose\textbf{PIP-MM}, a framework that \textbf{P}re-\textbf{I}ntegrates\textbf{P}rompt information into the visual encoding process using existingmodules of MLLMs. Specifically, We utilize the frozen LLM in the MLLM tovectorize the input prompt, which summarizes the requirements of the prompt.Then, we input the prompt vector into our trained Multi-Layer Perceptron (MLP)to align with the visual input requirements, and subsequently replace the classembedding in the image encoder. Since our model only requires adding atrainable MLP, it can be applied to any MLLM. To validate the effectiveness ofPIP-MM, we conducted experiments on multiple benchmarks. Automated evaluationmetrics and manual assessments demonstrate the strong performance of PIP-MM.Particularly noteworthy is that our model maintains excellent generationresults even when half of the visual tokens are reduced.
Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanisms
Venue: arXiv cs.AI 2024
Year: 2024

We present Public Domain 12M (PD12M), a dataset of 12.4 million high-quality public domain and CC0-licensed images with synthetic captions, designed for training text-to-image models. PD12M is the largest public domain image-text dataset to date, with sufficient size to train foundation models while minimizing copyright concerns. Through the Source.Plus platform, we also introduce novel, community-driven dataset governance mechanisms that reduce harm and support reproducibility over time.
Energy-Efficient Intra-Domain Network Slicing for Multi-Layer Orchestration in Intelligent-Driven Distributed 6G Networks: Learning Generic Assignment Skills with Unsupervised Reinforcement Learning
Venue: arXiv cs.NI 2024
Year: 2024

Since the 6th Generation (6G) of wireless networks is expected to provide a new level of network services and meet the emerging expectations of the future, it will be a complex and intricate networking system. 6Gs sophistication and robustness will be accompanied by complexities, which will require novel strategies to tackle them. This research work focuses on decentralized and multi-level system models for 6G networks and proposes an energy efficient automation strategy for edge domain management and Network Slicing (NS) with the main objective of reducing the networks complexity by leveraging scalability, efficiency, and generalization. Accordingly, we propose a pre-train phase to discover useful assignment skills in network edge domains by utilizing unsupervised Reinforcement Learning (unsupervised RL). The suggested technique does not depend on the domain specifications and thus is applicable to all the edge domains. Our proposed approach not only enables scalability and decentralization, but it also delivers efficiency by assisting domain controllers to provide various service types. We implemented the pre-training phase, and monitored that the discovered assignment skills span the entire interval of possible resource assignment portions for every service type.
ReasoningRec: Bridging Personalized Recommendations and Human-Interpretable Explanations through LLM Reasoning
Venue: arXiv cs.IR 2024
Year: 2024

This paper presents ReasoningRec, a reasoning-based recommendation framework that leverages Large Language Models (LLMs) to bridge the gap between recommendations and human-interpretable explanations. In contrast to conventional recommendation systems that rely on implicit user-item interactions, ReasoningRec employs LLMs to model users and items, focusing on preferences, aversions, and explanatory reasoning. The framework utilizes a larger LLM to generate synthetic explanations for user preferences, subsequently used to fine-tune a smaller LLM for enhanced recommendation accuracy and human-interpretable explanation. Our experimental study investigates the impact of reasoning and contextual information on personalized recommendations, revealing that the quality of contextual and personalized data significantly influences the LLM's capacity to generate plausible explanations. Empirical evaluations demonstrate that ReasoningRec surpasses state-of-the-art methods by up to 12.5\% in recommendation prediction while concurrently providing human-intelligible explanations. The code is available here: https://github.com/millenniumbismay/reasoningrec.
HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. SSL approaches typically work through some invariance term to ensure consistency between similar samples and a regularization term to prevent global dimensional collapse. Dimensional collapse refers to data representations spanning a lower-dimensional subspace. Recent work has demonstrated that the representation space of these algorithms gradually reflects a semantic hierarchical structure as training progresses. Data samples of the same hierarchical grouping tend to exhibit greater dimensional collapse locally compared to the dataset as a whole due to sharing features in common with each other. Ideally, SSL algorithms would take advantage of this hierarchical emergence to have an additional regularization term to account for this local dimensional collapse effect. However, the construction of existing SSL algorithms does not account for this property. To address this, we propose an adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively. This decomposition is based on an adaptive threshold that gradually lowers to reflect the emerging hierarchical structure of the representation space throughout training. It is based on an analysis of the cosine similarity distribution of samples in a batch. We demonstrate that this hierarchical emergence exploitation (HEX) approach can be integrated across a wide variety of SSL algorithms. Empirically, we show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.
ELMGS: Enhancing memory and computation scaLability through coMpression for 3D Gaussian Splatting
Venue: arXiv cs.CV 2024
Year: 2024

3D models have recently been popularized by the potentiality of end-to-end training offered first by Neural Radiance Fields and most recently by 3D Gaussian Splatting models. The latter has the big advantage of naturally providing fast training convergence and high editability. However, as the research around these is still in its infancy, there is still a gap in the literature regarding the model's scalability. In this work, we propose an approach enabling both memory and computation scalability of such models. More specifically, we propose an iterative pruning strategy that removes redundant information encoded in the model. We also enhance compressibility for the model by including in the optimization strategy a differentiable quantization and entropy coding estimator. Our results on popular benchmarks showcase the effectiveness of the proposed approach and open the road to the broad deployability of such a solution even on resource-constrained devices.
Attribute-to-Delete: Machine Unlearning via Datamodel Matching
Venue: arXiv cs.LG 2024
Year: 2024

Machine unlearning -- efficiently removing the effect of a small "forget set" of training data on a pre-trained machine learning model -- has recently attracted significant research interest. Despite this interest, however, recent work shows that existing machine unlearning techniques do not hold up to thorough evaluation in non-convex settings. In this work, we introduce a new machine unlearning technique that exhibits strong empirical performance even in such challenging settings. Our starting point is the perspective that the goal of unlearning is to produce a model whose outputs are statistically indistinguishable from those of a model re-trained on all but the forget set. This perspective naturally suggests a reduction from the unlearning problem to that of data attribution, where the goal is to predict the effect of changing the training set on a model's outputs. Thus motivated, we propose the following meta-algorithm, which we call Datamodel Matching (DMM): given a trained model, we (a) use data attribution to predict the output of the model if it were re-trained on all but the forget set points; then (b) fine-tune the pre-trained model to match these predicted outputs. In a simple convex setting, we show how this approach provably outperforms a variety of iterative unlearning algorithms. Empirically, we use a combination of existing evaluations and a new metric based on the KL-divergence to show that even in non-convex settings, DMM achieves strong unlearning performance relative to existing algorithms. An added benefit of DMM is that it is a meta-algorithm, in the sense that future advances in data attribution translate directly into better unlearning algorithms, pointing to a clear direction for future progress in unlearning.
Geometrically predictable micro fabricated continuum robot
Venue: arXiv cs.RO 2024
Year: 2024

Compared to the micro continuum robots that use traditional manufacturing technology, the micro fabricated continuum robots are different in terms of the application of smart materials, additive manufacturing process, and physical field control. However, the existing geometrical prediction models of the micro continuum robots still follow the model frameworks designed for their larger counterparts, which is inconsistent with the real geometrical transformation principle of micro fabricated continuum robots. In this paper, we present a universal geometrical prediction method for the geometry transformation of the micro fabricated continuum robots based on their material properties and the displacement of the stress points. By discretizing of the micro fabricated continuum structure and applying force constraints between adjacent points to simulate material properties, formulations and simulations are demonstrated to prove the feasibility and effectiveness of the proposed method. Three micro fabricated continuum robots driven through different external field forces are investigated to show two superiorities: the geometrical deformation of a micro fabricated continuum robot under external disturbances can be predicted, and a targeted geometry can be shaped by predicting the sequence and directions of external forces. This pioneer research has contributed to promote understanding and operation of micro fabricated continuum robots and their deformation both from theoretical aspect and real experimental operations.
Improving Image Data Leakage Detection in Automotive Software
Venue: arXiv cs.CR 2024
Year: 2024

Data leakage is a very common problem that is often overlooked during splitting data into train and test sets before training any ML/DL model. The model performance gets artificially inflated with the presence of data leakage during the evaluation phase which often leads the model to erroneous prediction on real-time deployment. However, detecting the presence of such leakage is challenging, particularly in the object detection context of perception systems where the model needs to be supplied with image data for training. In this study, we conduct a computational experiment on the Cirrus dataset from our industrial partner Volvo Cars to develop a method for detecting data leakage. We then evaluate the method on another public dataset, Kitti, which is a popular and widely accepted benchmark dataset in the automotive domain. The results show that thanks to our proposed method we are able to detect data leakage in the Kitti dataset, which was previously unknown.
Domain-decomposed image classification algorithms using linear discriminant analysis and convolutional neural networks
Venue: arXiv cs.CV 2024
Year: 2024

In many modern computer application problems, the classification of image data plays an important role. Among many different supervised machine learning models, convolutional neural networks (CNNs) and linear discriminant analysis (LDA) as well as sophisticated variants thereof are popular techniques. In this work, two different domain decomposed CNN models are experimentally compared for different image classification problems. Both models are loosely inspired by domain decomposition methods and in addition, combined with a transfer learning strategy. The resulting models show improved classification accuracies compared to the corresponding, composed global CNN model without transfer learning and besides, also help to speed up the training process. Moreover, a novel decomposed LDA strategy is proposed which also relies on a localization approach and which is combined with a small neural network model. In comparison with a global LDA applied to the entire input data, the presented decomposed LDA approach shows increased classification accuracies for the considered test problems.
Multilingual Vision-Language Pre-training for the Remote Sensing Domain
Venue: arXiv cs.CV 2024
Year: 2024

Methods based on Contrastive Language-Image Pre-training (CLIP) are nowadays extensively used in support of vision-and-language tasks involving remote sensing data, such as cross-modal retrieval. The adaptation of CLIP to this specific domain has relied on model fine-tuning with the standard contrastive objective, using existing human-labeled image-caption datasets, or using synthetic data corresponding to image-caption pairs derived from other annotations over remote sensing images (e.g., object classes). The use of different pre-training mechanisms has received less attention, and only a few exceptions have considered multilingual inputs. This work proposes a novel vision-and-language model for the remote sensing domain, exploring the fine-tuning of a multilingual CLIP model and testing the use of a self-supervised method based on aligning local and global representations from individual input images, together with the standard CLIP objective. Model training relied on assembling pre-existing datasets of remote sensing images paired with English captions, followed by the use of automated machine translation into nine additional languages. We show that translated data is indeed helpful, e.g. improving performance also on English. Our resulting model, which we named Remote Sensing Multilingual CLIP (RS-M-CLIP), obtains state-of-the-art results in a variety of vision-and-language tasks, including cross-modal and multilingual image-text retrieval, or zero-shot image classification.
Temporal Fair Division
Venue: arXiv cs.GT 2024
Year: 2024

We study temporal fair division, whereby a set of agents are allocated a (possibly different) set of goods on each day for a period of days. We study this setting, as well as a number of its special cases formed by the restrictions to two agents, same goods on each day, identical preferences, or combinations thereof, and chart out the landscape of achieving two types of fairness guarantees simultaneously: fairness on each day (per day) and fairness over time (up to each day, or the weaker version, overall). In the most general setting, we prove that there always exists an allocation that is stochastically-dominant envy-free up to one good (SD-EF1) per day and proportional up to one good (PROP1) overall, and when all the agents have identical preferences, we show that SD-EF1 per day and SD-EF1 overall can be guaranteed. For the case of two agents, we prove that SD-EF1 per day and EF1 up to each day can be guaranteed using an envy balancing technique. We provide counterexamples for other combinations that establish our results as among the best guarantees possible, but also leaving open some tantalizing questions.
Stepping Out of the Shadows: Reinforcement Learning in Shadow Mode
Venue: arXiv cs.LG 2024
Year: 2024

Reinforcement learning (RL) is not yet competitive for many cyber-physical systems, such as robotics, process automation, and power systems, as training on a system with physical components cannot be accelerated, and simulation models do not exist or suffer from a large simulation-to-reality gap. During the long training time, expensive equipment cannot be used and might even be damaged due to inappropriate actions of the reinforcement learning agent. Our novel approach addresses exactly this problem: We train the reinforcement agent in a so-called shadow mode with the assistance of an existing conventional controller, which does not have to be trained and instantaneously performs reasonably well. In shadow mode, the agent relies on the controller to provide action samples and guidance towards favourable states to learn the task, while simultaneously estimating for which states the learned agent will receive a higher reward than the conventional controller. The RL agent will then control the system for these states and all other regions remain under the control of the existing controller. Over time, the RL agent will take over for an increasing amount of states, while leaving control to the baseline, where it cannot surpass its performance. Thus, we keep regret during training low and improve the performance compared to only using conventional controllers or reinforcement learning. We present and evaluate two mechanisms for deciding whether to use the RL agent or the conventional controller. The usefulness of our approach is demonstrated for a reach-avoid task, for which we are able to effectively train an agent, where standard approaches fail.
Dynamic Information Sub-Selection for Decision Support
Venue: arXiv cs.LG 2024
Year: 2024

We introduce Dynamic Information Sub-Selection (DISS), a novel framework of AI assistance designed to enhance the performance of black-box decision-makers by tailoring their information processing on a per-instance basis. Blackbox decision-makers (e.g., humans or real-time systems) often face challenges in processing all possible information at hand (e.g., due to cognitive biases or resource constraints), which can degrade decision efficacy. DISS addresses these challenges through policies that dynamically select the most effective features and options to forward to the black-box decision-maker for prediction. We develop a scalable frequentist data acquisition strategy and a decision-maker mimicking technique for enhanced budget efficiency. We explore several impactful applications of DISS, including biased decision-maker support, expert assignment optimization, large language model decision support, and interpretability. Empirical validation of our proposed DISS methodology shows superior performance to state-of-the-art methods across various applications.
A Committee Based Optimal Asynchronous Byzantine Agreement Protocol W.P. 1
Venue: arXiv cs.DC 2024
Year: 2024

Multi-valued Byzantine agreement (MVBA) protocols are essential for atomic broadcast and fault-tolerant state machine replication in asynchronous networks. Despite advances, challenges persist in optimizing these protocols for communication and computation efficiency. This paper presents a committee-based MVBA protocol (cMVBA), a novel approach that achieves agreement without extra communication rounds by analyzing message patterns in asynchronous networks with probability 1.
On Enforcing Satisfiable, Coherent, and Minimal Sets of Dyadic Relation Constraints in MatBase
Venue: arXiv cs.DB 2024
Year: 2024

This paper rigorously and concisely defines, in the context of our (Elementary) Mathematical Data Model ((E)MDM), the mathematical concepts of dyadic relation, reflexivity, irreflexivity, symmetry, asymmetry, transitivity, intransitivity, Euclideanity, inEuclideanity, equivalence, acyclicity, connectivity, the properties that relate them, and the corresponding corollaries on the coherence and minimality of sets made of such dyadic relation properties viewed as database constraints. Its main contribution is the pseudocode algorithm used by MatBase, our intelligent database management system prototype based on both (E)MDM, the relational, and the entity-relationship data models, for enforcing dyadic relation constraint sets. We proved that this algorithm guarantees the satisfiability, coherence, and minimality of such sets, while being very fast, solid, complete, and minimal. In the sequel, we also presented the relevant MatBase user interface as well as the tables of its metacatalog used by this algorithm.