OpenRaiser/Intern-Atlas huggingface dataset:
Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance Fields
Venue: arXiv cs.CV 2024
Year: 2024

Neural radiance fields (NeRFs) have exhibited potential in synthesizing high-fidelity views of 3D scenes but the standard training paradigm of NeRF presupposes an equal importance for each image in the training set. This assumption poses a significant challenge for rendering specific views presenting intricate geometries, thereby resulting in suboptimal performance. In this paper, we take a closer look at the implications of the current training paradigm and redesign this for more superior rendering quality by NeRFs. Dividing input views into multiple groups based on their visual similarities and training individual models on each of these groups enables each model to specialize on specific regions without sacrificing speed or efficiency. Subsequently, the knowledge of these specialized models is aggregated into a single entity via a teacher-student distillation paradigm, enabling spatial efficiency for online render-ing. Empirically, we evaluate our novel training framework on two publicly available datasets, namely NeRF synthetic and Tanks&Temples. Our evaluation demonstrates that our DaC training pipeline enhances the rendering quality of a state-of-the-art baseline model while exhibiting convergence to a superior minimum.
Agile Effort Estimation: Comparing the Accuracy and Efficiency of Planning Poker, Bucket System, and Affinity Estimation methods
Venue: arXiv cs.SE 2024
Year: 2024

Published studies on agile effort estimation predominantly focus on comparisons of the accuracy of different estimation methods, while efficiency comparisons, i.e. how much time the estimation methods consume was not in the forefront. However, for practical use in software development, the time required can be a very important cost factor for enterprises, especially when the accuracy of different agile effort estimations is similar. In this study, we thus try to advance the current standard accuracy comparison between methods by introducing efficiency i.e. time it takes to use a method as an additional dimension of comparison. We conduct this comparison between three agile effort estimation methods that were not yet compared in the literature, namely Planning Poker, Bucket System and Affinity Estimation. For the comparison, we used eight student teams with 29 students that had to use all the effort estimation methods during the course where they had to finish a programming project in 3 weeks. The results indicate that after the students get used to using the different methods the accuracy between them is not statistically significantly different, however, the efficiency is. On average Bucket System and Affinity Estimation methods take half as much time as Planning Poker.
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Venue: arXiv cs.CL 2024
Year: 2024

Mobile device agent based on Multimodal Large Language Models (MLLM) is becoming a popular application. In this paper, we introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. Based on the perceived vision context, it then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step. Different from previous solutions that rely on XML files of Apps or mobile system metadata, Mobile-Agent allows for greater adaptability across diverse mobile operating environments in a vision-centric way, thereby eliminating the necessity for system-specific customizations. To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations. Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent. The experimental results indicate that Mobile-Agent achieved remarkable accuracy and completion rates. Even with challenging instructions, such as multi-app operations, Mobile-Agent can still complete the requirements. Code and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
Venue: arXiv cs.CV 2024
Year: 2024

Instruction finetuning on a variety of image-text instruction data is the key to obtaining a versatile Multimodal Large Language Model (MLLM), and different configurations of the instruction data can lead to finetuned models with different capabilities. However, we have discovered that data conflicts are inevitable when mixing instruction data from distinct domains, which can result in performance drops for tasks of a specific domain. To address this issue, we propose to apply an efficient Mixture of Experts (MoE) design, which is a sparse Mixture of LoRA Experts (MoLE) for instruction finetuning MLLMs. Within the Transformer layers, we extend the popular Low-Rank Adaption (LoRA) method by creating a set of LoRA experts specifically for the MLP layer, and route each token to the top-1 expert based on a routing function, allowing adaptive choices for tokens from different domains. Since the LoRA experts are sparsely activated, the training and inference cost are kept roughly constant compared to the original LoRA method. By replacing the plain-LoRA of LLaVA-1.5 with our MoE design, our final model is named LLaVA-MoLE. Extensive experiments proved that LLaVA-MoLE effectively mitigates the data conflict issue when mixing multiple distinct instruction datasets with various configurations, and achieves consistent performance gains over the strong plain-LoRA baselines. Most importantly, on the mixed datasets, LLaVA-MoLE can even outperform the plain-LoRA baseline trained with twice the samples.
Graph Neural Network-based Joint Equalization and Decoding
Venue: arXiv cs.IT 2024
Year: 2024

This paper proposes to use graph neural networks (GNNs) for equalization, that can also be used to perform joint equalization and decoding (JED). For equalization, the GNN is build upon the factor graph representations of the channel, while for JED, the factor graph is expanded by the Tanner graph of the parity-check matrix (PCM) of the channel code, sharing the variable nodes (VNs). A particularly advantageous property of the GNN is the robustness against cycles in the factor graphs which is the main problem for belief propagation (BP)-based equalization. As a result of having a fully deep learning-based receiver, joint optimization instead of individual optimization of the components is enabled, so-called end-to-end learning. Furthermore, we propose a parallel flooding schedule that further reduces the latency, which turns out to improve also the error correcting performance. The proposed approach is analyzed and compared to state-of-the-art baselines in terms of error correcting capability and latency. At a fixed low latency, the flooding GNN for JED demonstrates a gain of 2.25 dB in bit error rate (BER) compared to an iterative Bahl--Cock--Jelinek--Raviv (BCJR)-BP baseline.
Foundations of Work-Systems Modeling
Venue: arXiv cs.OH 2024
Year: 2024

In 2006, the course "Modeling of Organizations" is taught for the third time. This third time will be the second time we will use the new lecture notes "Work Systems Modelling" from the DA VINCI series. These lecture notes, however, will be evolved further hand-in-hand with the actual process of lecturing. In the academic year 2005/2006, a second incarnation of these lecture notes will be created, where the aim is to deliver these lecture notes in three increments. An important step that will be taken in this academic year is the integration of the ICIS Work Systems Modelling lecture notes with the NICI course on Organisational Dynamics. The first results of this integration will start to appear in the second and third trimester.
DAEDALUS: Defense Against Firmware ROP Exploits Using Stochastic Software Diversity
Venue: arXiv cs.CR 2024
Year: 2024

This paper presents DAEDALUS, a software diversity-based framework designed to resist ROP attacks on Linux-based IoT devices. DAEDALUS generates unique, semantically equivalent but syntactically different rewrites of IoT firmware, disrupting large-scale replication of ROP attacks. DAEDALUS employs STOKE, a stochastic optimizer for x86 binaries, as its core diversity engine but introduces significant extensions to address unique IoT firmware challenges. DAEDALUS's effectiveness is evaluated using DDoSim, a published botnet DDoS attack simulation testbed. Results demonstrate that DAEDALUS successfully neutralizes ROP payloads by diversifying critical basic blocks in the firmware, preventing attackers from compromising multiple devices for DDoS attacks via memory error vulnerabilities. The findings indicate that DAEDALUS not only mitigates the impact of ROP attacks on individual IoT devices through probabilistic protection but also thwarts large-scale ROP attacks across multiple devices.
Alternating Minimization for Wideband Multiuser IRS-aided MIMO Systems under Imperfect CSI
Venue: arXiv cs.IT 2024
Year: 2024

This work focuses on wideband intelligent reflecting surface (IRS)-aided multiuser MIMO systems. One of the major challenges of this scenario is the joint design of the frequency-dependent base station (BS) precoder and user filters, and the IRS phase-shift matrix which is frequency flat and common to all the users. In addition, we consider that the channel state information (CSI) is imperfect at both the transmitter and the receivers. A statistical model for the imperfect CSI is developed and exploited for the system design. A minimum mean square error (MMSE) approach is followed to determine the IRS phase-shift matrix, the transmit precoders, and the receiving filters. The broadcast (BC)- multiple access channel (MAC) duality is used to solve the optimization problem following an alternating minimization approach. Numerical results show that the proposed approach leads to substantial performance gains with respect to baseline strategies that neglect the inter-user interference and do not optimize the IRS phase-shift matrix. Further performance gains are obtained when incorporating into the system design the statistical information of the channel estimation errors.
A.I. In All The Wrong Places
Venue: arXiv cs.CY 2024
Year: 2024

This text describes experiences gained across a two-year test period during which two generations of Generative Artificial Intelligence (A.I.) systems were incorporated into an interdisciplinary, university level course on A.I. for art and design practices. The text uses the results from the courses to reflect on new opportunities for generative systems in art and design, while considering traps and limits.
Rethinking the Producer-Consumer Relationship in Modern DRAM-Based Systems
Venue: arXiv cs.AR 2024
Year: 2024

Generational improvements to commodity DRAM throughout half a century have long solidified its prevalence as main memory across the computing industry. However, overcoming today's DRAM technology scaling challenges requires new solutions driven by both DRAM producers and consumers. In this paper, we observe that the separation of concerns between producers and consumers specified by industry-wide DRAM standards is becoming a liability to progress in addressing scaling-related concerns. To understand the problem, we study four key directions for overcoming DRAM scaling challenges using system-memory cooperation: (i) improving memory access latencies; (ii) reducing DRAM refresh overheads; (iii) securely defending against the RowHammer vulnerability; and (iv) addressing worsening memory errors. We find that the single most important barrier to advancement in all four cases is the consumer's lack of insight into DRAM reliability. Based on an analysis of DRAM reliability testing, we recommend revising the separation of concerns to incorporate limited information transparency between producers and consumers. Finally, we propose adopting this revision in a two-step plan, starting with immediate information release through crowdsourcing and publication and culminating in widespread modifications to DRAM standards.
Dual feature-based and example-based explanation methods
Venue: arXiv cs.LG 2024
Year: 2024

A new approach to the local and global explanation is proposed. It is based on selecting a convex hull constructed for the finite number of points around an explained instance. The convex hull allows us to consider a dual representation of instances in the form of convex combinations of extreme points of a produced polytope. Instead of perturbing new instances in the Euclidean feature space, vectors of convex combination coefficients are uniformly generated from the unit simplex, and they form a new dual dataset. A dual linear surrogate model is trained on the dual dataset. The explanation feature importance values are computed by means of simple matrix calculations. The approach can be regarded as a modification of the well-known model LIME. The dual representation inherently allows us to get the example-based explanation. The neural additive model is also considered as a tool for implementing the example-based explanation approach. Many numerical experiments with real datasets are performed for studying the approach. The code of proposed algorithms is available.
Momentary Stressor Logging and Reflective Visualizations: Implications for Stress Management with Wearables
Venue: arXiv cs.HC 2024
Year: 2024

Commercial wearables from Fitbit, Garmin, and Whoop have recently introduced real-time notifications based on detecting changes in physiological responses indicating potential stress. In this paper, we investigate how these new capabilities can be leveraged to improve stress management. We developed a smartwatch app, a smartphone app, and a cloud service, and conducted a 100-day field study with 122 participants who received prompts triggered by physiological responses several times a day. They were asked whether they were stressed, and if so, to log the most likely stressor. Each week, participants received new visualizations of their data to self-reflect on patterns and trends. Participants reported better awareness of their stressors, and self-initiating fourteen kinds of behavioral changes to reduce stress in their daily lives. Repeated self-reports over 14 weeks showed reductions in both stress intensity (in 26,521 momentary ratings) and stress frequency (in 1,057 weekly surveys).
Creative Telescoping for Hypergeometric Double Sums
Venue: arXiv cs.SC 2024
Year: 2024

We present efficient methods for calculating linear recurrences of hypergeometric double sums and, more generally, of multiple sums. In particular, we supplement this approach with the algorithmic theory of contiguous relations, which guarantees the applicability of our method for many input sums. In addition, we elaborate new techniques to optimize the underlying key task of our method to compute rational solutions of parameterized linear recurrences.
On Achievable Rates for the Shotgun Sequencing Channel with Erasures
Venue: arXiv cs.IT 2024
Year: 2024

In shotgun sequencing, the input string (typically, a long DNA sequence composed of nucleotide bases) is sequenced as multiple overlapping fragments of much shorter lengths (called \textit{reads}). Modelling the shotgun sequencing pipeline as a communication channel for DNA data storage, the capacity of this channel was identified in a recent work, assuming that the reads themselves are noiseless substrings of the original sequence. Modern shotgun sequencers however also output quality scores for each base read, indicating the confidence in its identification. Bases with low quality scores can be considered to be erased. Motivated by this, we consider the \textit{shotgun sequencing channel with erasures}, where each symbol in any read can be independently erased with some probability $δ$. We identify achievable rates for this channel, using a random code construction and a decoder that uses typicality-like arguments to merge the reads.
Cross-Modal Coordination Across a Diverse Set of Input Modalities
Venue: arXiv cs.CV 2024
Year: 2024

Cross-modal retrieval is the task of retrieving samples of a given modality by using queries of a different one. Due to the wide range of practical applications, the problem has been mainly focused on the vision and language case, e.g. text to image retrieval, where models like CLIP have proven effective in solving such tasks. The dominant approach to learning such coordinated representations consists of projecting them onto a common space where matching views stay close and those from non-matching pairs are pushed away from each other. Although this cross-modal coordination has been applied also to other pairwise combinations, extending it to an arbitrary number of diverse modalities is a problem that has not been fully explored in the literature. In this paper, we propose two different approaches to the problem. The first is based on an extension of the CLIP contrastive objective to an arbitrary number of input modalities, while the second departs from the contrastive formulation and tackles the coordination problem by regressing the cross-modal similarities towards a target that reflects two simple and intuitive constraints of the cross-modal retrieval task. We run experiments on two different datasets, over different combinations of input modalities and show that the approach is not only simple and effective but also allows for tackling the retrieval problem in novel ways. Besides capturing a more diverse set of pair-wise interactions, we show that we can use the learned representations to improve retrieval performance by combining the embeddings from two or more such modalities.
Mixed-Order Meshes through rp-adaptivity for Surface Fitting to Implicit Geometries
Venue: arXiv cs.MS 2024
Year: 2024

Computational analysis with the finite element method requires geometrically accurate meshes. It is well known that high-order meshes can accurately capture curved surfaces with fewer degrees of freedom in comparison to low-order meshes. Existing techniques for high-order mesh generation typically output meshes with same polynomial order for all elements. However, high order elements away from curvilinear boundaries or interfaces increase the computational cost of the simulation without increasing geometric accuracy. In prior work, we have presented one such approach for generating body-fitted uniform-order meshes that takes a given mesh and morphs it to align with the surface of interest prescribed as the zero isocontour of a level-set function. We extend this method to generate mixed-order meshes such that curved surfaces of the domain are discretized with high-order elements, while low-order elements are used elsewhere. Numerical experiments demonstrate the robustness of the approach and show that it can be used to generate mixed-order meshes that are much more efficient than high uniform-order meshes. The proposed approach is purely algebraic, and extends to different types of elements (quadrilaterals/triangles/tetrahedron/hexahedra) in two- and three-dimensions.
Learning logic programs by finding minimal unsatisfiable subprograms
Venue: arXiv cs.LG 2024
Year: 2024

The goal of inductive logic programming (ILP) is to search for a logic program that generalises training examples and background knowledge. We introduce an ILP approach that identifies minimal unsatisfiable subprograms (MUSPs). We show that finding MUSPs allows us to efficiently and soundly prune the search space. Our experiments on multiple domains, including program synthesis and game playing, show that our approach can reduce learning times by 99%.
Green Adaptation of Real-Time Web Services for Industrial CPS within a Cloud Environment
Venue: arXiv cs.AR 2024
Year: 2024

Managing energy efficiency under timing constraints is an interesting and big challenge. This work proposes an accurate power model in data centers for time-constrained servers in Cloud computing. This model, as opposed to previous approaches, does not only consider the workload assigned to the processing element, but also incorporates the need of considering the static power consumption and, even more interestingly, its dependency with temperature. The proposed model has been used in a multi-objective optimization environment in which the Dynamic Voltage and Frequency Scaling (DVFS) and workload assignment have been efficiently optimized.
LLM4SecHW: Leveraging Domain Specific Large Language Model for Hardware Debugging
Venue: arXiv cs.AR 2024
Year: 2024

This paper presents LLM4SecHW, a novel framework for hardware debugging that leverages domain specific Large Language Model (LLM). Despite the success of LLMs in automating various software development tasks, their application in the hardware security domain has been limited due to the constraints of commercial LLMs and the scarcity of domain specific data. To address these challenges, we propose a unique approach to compile a dataset of open source hardware design defects and their remediation steps, utilizing version control data. This dataset provides a substantial foundation for training machine learning models for hardware. LLM4SecHW employs fine tuning of medium sized LLMs based on this dataset, enabling the identification and rectification of bugs in hardware designs. This pioneering approach offers a reference workflow for the application of fine tuning domain specific LLMs in other research areas. We evaluate the performance of our proposed system on various open source hardware designs, demonstrating its efficacy in accurately identifying and correcting defects. Our work brings a new perspective on automating the quality control process in hardware design.
AI in Energy Digital Twining: A Reinforcement Learning-based Adaptive Digital Twin Model for Green Cities
Venue: arXiv cs.LG 2024
Year: 2024

Digital Twins (DT) have become crucial to achieve sustainable and effective smart urban solutions. However, current DT modelling techniques cannot support the dynamicity of these smart city environments. This is caused by the lack of right-time data capturing in traditional approaches, resulting in inaccurate modelling and high resource and energy consumption challenges. To fill this gap, we explore spatiotemporal graphs and propose the Reinforcement Learning-based Adaptive Twining (RL-AT) mechanism with Deep Q Networks (DQN). By doing so, our study contributes to advancing Green Cities and showcases tangible benefits in accuracy, synchronisation, resource optimization, and energy efficiency. As a result, we note the spatiotemporal graphs are able to offer a consistent accuracy and 55% higher querying performance when implemented using graph databases. In addition, our model demonstrates right-time data capturing with 20% lower overhead and 25% lower energy consumption.