OpenRaiser/Intern-Atlas huggingface dataset:
Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization
Venue: arXiv cs.LG 2024
Year: 2024

Recently, sharpness-aware minimization (SAM) has emerged as a promising method to improve generalization by minimizing sharpness, which is known to correlate well with generalization ability. Since the original proposal of SAM, many variants of SAM have been proposed to improve its accuracy and efficiency, but comparisons have mainly been restricted to the i.i.d. setting. In this paper we study SAM for out-of-distribution (OOD) generalization. First, we perform a comprehensive comparison of eight SAM variants on zero-shot OOD generalization, finding that the original SAM outperforms the Adam baseline by $4.76\%$ and the strongest SAM variants outperform the Adam baseline by $8.01\%$ on average. We then provide an OOD generalization bound in terms of sharpness for this setting. Next, we extend our study of SAM to the related setting of gradual domain adaptation (GDA), another form of OOD generalization where intermediate domains are constructed between the source and target domains, and iterative self-training is done on intermediate domains, to improve the overall target domain error. In this setting, our experimental results demonstrate that the original SAM outperforms the baseline of Adam on each of the experimental datasets by $0.82\%$ on average and the strongest SAM variants outperform Adam by $1.52\%$ on average. We then provide a generalization bound for SAM in the GDA setting. Asymptotically, this generalization bound is no better than the one for self-training in the literature of GDA. This highlights a further disconnection between the theoretical justification for SAM versus its empirical performance, with recent work finding that low sharpness alone does not account for all of SAM's generalization benefits. For future work, we provide several potential avenues for obtaining a tighter analysis for SAM in the OOD setting.
Spatially-Adaptive Hash Encodings For Neural Surface Reconstruction
Venue: arXiv cs.CV 2024
Year: 2024

Positional encodings are a common component of neural scene reconstruction methods, and provide a way to bias the learning of neural fields towards coarser or finer representations. Current neural surface reconstruction methods use a "one-size-fits-all" approach to encoding, choosing a fixed set of encoding functions, and therefore bias, across all scenes. Current state-of-the-art surface reconstruction approaches leverage grid-based multi-resolution hash encoding in order to recover high-detail geometry. We propose a learned approach which allows the network to choose its encoding basis as a function of space, by masking the contribution of features stored at separate grid resolutions. The resulting spatially adaptive approach allows the network to fit a wider range of frequencies without introducing noise. We test our approach on standard benchmark surface reconstruction datasets and achieve state-of-the-art performance on two benchmark datasets.
"If it has an exclamation point, I step away from it, I need facts, not excited feelings": Technologically Mediated Parental COVID Uncertainty
Venue: arXiv cs.HC 2024
Year: 2024

As a novel virus, COVID introduced considerable uncertainty into the daily lives of people all over the globe since late 2019. Relying on twenty-three semi-structured interviews with parents whose children contracted COVID, we analyzed how the use of social media moderated parental uncertainty about the symptoms, prognosis, long-term potential health ramifications of infection, vaccination, and other issues. We framed our findings using Mishel's Uncertainty in Illness theory. We propose new components to the theory that account for technological mediation in uncertainty. We also propose design recommendations to help parents cope with health uncertainty using social media.
LinVT: Empower Your Image-level Large Language Model to Understand Videos
Venue: arXiv cs.CV 2024
Year: 2024

Large Language Models (LLMs) have been widely used in various tasks, motivating us to develop an LLM-based assistant for videos. Instead of training from scratch, we propose a module to transform arbitrary well-trained image-based LLMs into video-LLMs (after being trained on video data). To better adapt image-LLMs for processing videos, we introduce two design principles: linear transformation to preserve the original visual-language alignment and representative information condensation from redundant video content. Guided by these principles, we propose a plug-and-play Linear Video Tokenizer(LinVT), which enables existing image-LLMs to understand videos. We benchmark LinVT with six recent visual LLMs: Aquila, Blip-3, InternVL2, Mipha, Molmo and Qwen2-VL, showcasing the high compatibility of LinVT. LinVT-based LLMs achieve state-of-the-art performance across various video benchmarks, illustrating the effectiveness of LinVT in multi-modal video understanding.
Automated, Unsupervised, and Auto-parameterized Inference of Data Patterns and Anomaly Detection
Venue: arXiv cs.SE 2024
Year: 2024

With the advent of data-centric and machine learning (ML) systems, data quality is playing an increasingly critical role in ensuring the overall quality of software systems. Data preparation, an essential step towards high data quality, is known to be a highly effort-intensive process. Although prior studies have dealt with one of the most impacting issues, data pattern violations, these studies usually require data-specific configurations (i.e., parameterized) or use carefully curated data as learning examples (i.e., supervised), relying on domain knowledge and deep understanding of the data, or demanding significant manual effort. In this paper, we introduce RIOLU: Regex Inferencer auto-parameterized Learning with Uncleaned data. RIOLU is fully automated, automatically parameterized, and does not need labeled samples. RIOLU can generate precise patterns from datasets in various domains, with a high F1 score of 97.2%, exceeding the state-of-the-art baseline. In addition, according to our experiment on five datasets with anomalies, RIOLU can automatically estimate a data column's error rate, draw normal patterns, and predict anomalies from unlabeled data with higher performance (up to 800.4% improvement in terms of F1) than the state-of-the-art baseline, even outperforming ChatGPT in terms of both accuracy (12.3% higher F1) and efficiency (10% less inference time). A variant of RIOLU, with user guidance, can further boost its precision, with up to 37.4% improvement in terms of F1. Our evaluation in an industrial setting further demonstrates the practical benefits of RIOLU.
Preprocessing is All You Need: Boosting the Performance of Log Parsers With a General Preprocessing Framework
Venue: arXiv cs.SE 2024
Year: 2024

Log parsing has been a long-studied area in software engineering due to its importance in identifying dynamic variables and constructing log templates. Prior work has proposed many statistic-based log parsers (e.g., Drain), which are highly efficient; they, unfortunately, met the bottleneck of parsing performance in comparison to semantic-based log parsers, which require labeling and more computational resources. Meanwhile, we noticed that previous studies mainly focused on parsing and often treated preprocessing as an ad hoc step (e.g., masking numbers). However, we argue that both preprocessing and parsing are essential for log parsers to identify dynamic variables: the lack of understanding of preprocessing may hinder the optimal use of parsers and future research. Therefore, our work studied existing log preprocessing approaches based on Loghub, a popular log parsing benchmark. We developed a general preprocessing framework with our findings and evaluated its impact on existing parsers. Our experiments show that the preprocessing framework significantly boosts the performance of four state-of-the-art statistic-based parsers. Drain, the best statistic-based parser, obtained improvements across all four parsing metrics (e.g., F1 score of template accuracy, FTA, increased by 108.9%). Compared to semantic-based parsers, it achieved a 28.3% improvement in grouping accuracy (GA), 38.1% in FGA, and an 18.6% increase in FTA. Our work pioneers log preprocessing and provides a generalizable framework to enhance log parsing.
A Machine Learning-Based Approach For Detecting Malicious PyPI Packages
Venue: arXiv cs.SE 2024
Year: 2024

Background. In modern software development, the use of external libraries and packages is increasingly prevalent, streamlining the software development process and enabling developers to deploy feature-rich systems with little coding. While this reliance on reusing code offers substantial benefits, it also introduces serious risks for deployed software in the form of malicious packages - harmful and vulnerable code disguised as useful libraries. Aims. Popular ecosystems, such PyPI, receive thousands of new package contributions every week, and distinguishing safe contributions from harmful ones presents a significant challenge. There is a dire need for reliable methods to detect and address the presence of malicious packages in these environments. Method. To address these challenges, we propose a data-driven approach that uses machine learning and static analysis to examine the package's metadata, code, files, and textual characteristics to identify malicious packages. Results. In evaluations conducted within the PyPI ecosystem, we achieved an F1-measure of 0.94 for identifying malicious packages using a stacking ensemble classifier. Conclusions. This tool can be seamlessly integrated into package vetting pipelines and has the capability to flag entire packages, not just malicious function calls. This enhancement strengthens security measures and reduces the manual workload for developers and registry maintainers, thereby contributing to the overall integrity of the ecosystem.
Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
Venue: arXiv cs.CV 2024
Year: 2024

4D driving simulation is essential for developing realistic autonomous driving simulators. Despite advancements in existing methods for generating driving scenes, significant challenges remain in view transformation and spatial-temporal dynamic modeling. To address these limitations, we propose a Spatial-Temporal simulAtion for drivinG (Stag-1) model to reconstruct real-world scenes and design a controllable generative network to achieve 4D simulation. Stag-1 constructs continuous 4D point cloud scenes using surround-view data from autonomous vehicles. It decouples spatial-temporal relationships and produces coherent keyframe videos. Additionally, Stag-1 leverages video generation models to obtain photo-realistic and controllable 4D driving simulation videos from any perspective. To expand the range of view generation, we train vehicle motion videos based on decomposed camera poses, enhancing modeling capabilities for distant scenes. Furthermore, we reconstruct vehicle camera trajectories to integrate 3D points across consecutive views, enabling comprehensive scene understanding along the temporal dimension. Following extensive multi-level scene training, Stag-1 can simulate from any desired viewpoint and achieve a deep understanding of scene evolution under static spatial-temporal conditions. Compared to existing methods, our approach shows promising performance in multi-view scene consistency, background coherence, and accuracy, and contributes to the ongoing advancements in realistic autonomous driving simulation. Code: https://github.com/wzzheng/Stag.
RISC-V Word-Size Modular Instructions for Residue Number Systems
Venue: arXiv cs.AR 2024
Year: 2024

Residue Number Systems (RNS) are parallel number systems that allow the computation on large numbers. They are used in high performance digital signal processing devices and cryptographic applications. However, the rigidity of instruction set architectures of the market-dominant microprocessors limits the use of such number systems in software applications. This article presents the impact of word-size modular arithmetic specific RISC-V instructions on the software implementation of Residue Number Systems. We evaluate this impact on several RNS modular multiplication sequential algorithms. We observe that the fastest implementation uses the Kawamura et. al. base extension. Simulations of architectures with GEM5 simulator show that RNS modular multiplication with Kawamura's base extension is 2.76 times faster using specific word-size modular arithmetic instructions than pseudo-Mersenne moduli for In Order processors. It is more than 3 times for Out of Order processors. Compared to x86 architectures, RISC-V simulations show that using specific instructions requires 4.5 times less cycles in In Order processors and 8 less in Out of Order ones.
The Tiny Median Filter: A Small Size, Flexible Arbitrary Percentile Finder Scheme Suitable for FPGA Implementation
Venue: arXiv cs.AR 2024
Year: 2024

This document reports the design, implementation and testing of a small silicon resource usage, very flexible arbitrary percentile finding scheme called the Tiny Median Filter. It can be used not only as a median filter in image processing with square filtering windows, but also for applications of any percentile filter or maximum or minimum finder with any size of data set as long as the number of bits of the data is finite. It opens possibilities for image processing tasks with non-square or irregular filter windows. In this scheme, data swapping or data bit manipulating are avoided and high functional efficiency of the logic components is applied to save silicon resources. Some logic functions are absorbed into other functions to further reduce the complexity. The combinational logic paths are designed to be sufficiently short so that the firmware can be compiled to the maximum operating frequency allowed by the block memories of the FPGA devices. The Tiny Median Filter receives, processes and output data in non-stop manner with no irregular timing which helps to simplify design of surrounding stages.
Automated Dynamic Image Analysis for Particle Size and Shape Classification in Three Dimensions
Venue: arXiv cs.CV 2024
Year: 2024

We introduce OCULAR, an innovative hardware and software solution for three-dimensional dynamic image analysis of fine particles. Current state-of-the art instruments for dynamic image analysis are largely limited to two-dimensional imaging. However, extensive literature has demonstrated that relying on a single two-dimensional projection for particle characterisation can lead to inaccuracies in many applications. Existing three-dimensional imaging technologies, such as computed tomography, laser scanning, and orthophotography, are limited to static objects. These methods are often not statistically representative and come with significant post-processing requirements, as well as the need for specialised imaging and computing resources. OCULAR addresses these challenges by providing a cost-effective solution for imaging continuous particle streams using a synchronised array of optical cameras. Particle shape characterisation is achieved through the reconstruction of their three-dimensional surfaces. This paper details the OCULAR methodology, evaluates its reproducibility, and compares its results against X-ray micro computed tomography, highlighting its potential for efficient and reliable particle analysis.
CALICO: Conversational Agent Localization via Synthetic Data Generation
Venue: arXiv cs.CL 2024
Year: 2024

We present CALICO, a method to fine-tune Large Language Models (LLMs) to localize conversational agent training data from one language to another. For slots (named entities), CALICO supports three operations: verbatim copy, literal translation, and localization, i.e. generating slot values more appropriate in the target language, such as city and airport names located in countries where the language is spoken. Furthermore, we design an iterative filtering mechanism to discard noisy generated samples, which we show boosts the performance of the downstream conversational agent. To prove the effectiveness of CALICO, we build and release a new human-localized (HL) version of the MultiATIS++ travel information test set in 8 languages. Compared to the original human-translated (HT) version of the test set, we show that our new HL version is more challenging. We also show that CALICO out-performs state-of-the-art LINGUIST (which relies on literal slot translation out of context) both on the HT case, where CALICO generates more accurate slot translations, and on the HL case, where CALICO generates localized slots which are closer to the HL test set.
From Voice to Value: Leveraging AI to Enhance Spoken Online Reviews on the Go
Venue: arXiv cs.HC 2024
Year: 2024

Online reviews help people make better decisions. Review platforms usually depend on typed input, where leaving a good review requires significant effort because users must carefully organize and articulate their thoughts. This may discourage users from leaving comprehensive and high-quality reviews, especially when they are on the go. To address this challenge, we developed Vocalizer, a mobile application that enables users to provide reviews through voice input, with enhancements from a large language model (LLM). In a longitudinal study, we analysed user interactions with the app, focusing on AI-driven features that help refine and improve reviews. Our findings show that users frequently utilized the AI agent to add more detailed information to their reviews. We also show how interactive AI features can improve users self-efficacy and willingness to share reviews online. Finally, we discuss the opportunities and challenges of integrating AI assistance into review-writing systems.
Multi-Armed Bandit Approach for Optimizing Training on Synthetic Data
Venue: arXiv cs.LG 2024
Year: 2024

Supervised machine learning methods require large-scale training datasets to perform well in practice. Synthetic data has been showing great progress recently and has been used as a complement to real data. However, there is yet a great urge to assess the usability of synthetically generated data. To this end, we propose a novel UCB-based training procedure combined with a dynamic usability metric. Our proposed metric integrates low-level and high-level information from synthetic images and their corresponding real and synthetic datasets, surpassing existing traditional metrics. By utilizing a UCB-based dynamic approach ensures continual enhancement of model learning. Unlike other approaches, our method effectively adapts to changes in the machine learning model's state and considers the evolving utility of training samples during the training process. We show that our metric is an effective way to rank synthetic images based on their usability. Furthermore, we propose a new attribute-aware bandit pipeline for generating synthetic data by integrating a Large Language Model with Stable Diffusion. Quantitative results show that our approach can boost the performance of a wide range of supervised classifiers. Notably, we observed an improvement of up to 10% in classification accuracy compared to traditional approaches, demonstrating the effectiveness of our approach. Our source code, datasets, and additional materials are publically available at https://github.com/A-Kerim/Synthetic-Data-Usability-2024.
EC-Chain: Cost-Effective Storage Solution for Permissionless Blockchains
Venue: arXiv cs.CR 2024
Year: 2024

Permissionless blockchains face considerable challenges due to increasing storage demands, driven by the proliferation of Decentralized Applications (DApps). This paper introduces EC-Chain, a cost-effective storage solution for permissionless blockchains. EC-Chain reduces storage overheads of ledger and state data, which comprise blockchain data. For ledger data, EC-Chain refines existing erasure coding-based storage optimization techniques by incorporating batch encoding and height-based encoding. We also introduce an easy-to-implement dual-trie state management system that enhances state storage and retrieval through state expiry, mining, and creation procedures. To ensure data availability in permissionless environments, EC-Chain introduces a network maintenance scheme tailored for dynamism. Collectively, these contributions allow EC-Chain to provide an effective solution to the storage challenges faced by permissionless blockchains. Our evaluation demonstrates that EC-Chain can achieve a storage reduction of over \(90\%\) compared to native Ethereum Geth.
Dynamic Digital Twins of Blockchain Systems: State Extraction and Mirroring
Venue: arXiv cs.CR 2024
Year: 2024

Blockchain adoption is reaching an all-time high, with a plethora of blockchain architectures being developed to cover the needs of applications eager to integrate blockchain into their operations. However, blockchain systems suffer from the trilemma trade-off problem, which limits their ability to scale without sacrificing essential metrics such as decentralisation and security. The balance of the trilemma trade-off is primarily dictated by the consensus protocol used. Since consensus protocols are designed to function well under specific system conditions, and consequently, due to the blockchain's complex and dynamic nature, systems operating under a single consensus protocol are bound to face periods of inefficiency. The work presented in this paper constitutes part of an effort to design a Digital Twin-based blockchain management framework to balance the trilemma trade-off problem, which aims to adapt the consensus process to fit the conditions of the underlying system. Specifically, this work addresses the problems of extracting the blockchain system and mirroring it in its digital twin by proposing algorithms that overcome the challenges posed by blockchains' decentralised and asynchronous nature and the fundamental problems of global state and synchronisation in such systems. The robustness of the proposed algorithms is experimentally evaluated.
GAQAT: gradient-adaptive quantization-aware training for domain generalization
Venue: arXiv cs.CV 2024
Year: 2024

Research on loss surface geometry, such as Sharpness-Aware Minimization (SAM), shows that flatter minima improve generalization. Recent studies further reveal that flatter minima can also reduce the domain generalization (DG) gap. However, existing flatness-based DG techniques predominantly operate within a full-precision training process, which is impractical for deployment on resource-constrained edge devices that typically rely on lower bit-width representations (e.g., 4 bits, 3 bits). Consequently, low-precision quantization-aware training is critical for optimizing these techniques in real-world applications. In this paper, we observe a significant degradation in performance when applying state-of-the-art DG-SAM methods to quantized models, suggesting that current approaches fail to preserve generalizability during the low-precision training process. To address this limitation, we propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG. Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization, where the task loss and smoothness loss induce conflicting gradients for the scaling factors of quantizers, with certain layers exhibiting opposing gradient directions. This conflict renders the optimization of quantized weights highly unstable. To mitigate this, we further introduce a mechanism to quantify gradient inconsistencies and selectively freeze the gradients of scaling factors, thereby stabilizing the training process and enhancing out-of-domain generalization. Extensive experiments validate the effectiveness of the proposed GAQAT framework. On PACS, our 3-bit and 4-bit models outperform direct DG-QAT integration by up to 4.5%. On DomainNet, the 4-bit model achieves near-lossless performance compared to full precision, with improvements of 1.39% (4-bit) and 1.06% (3-bit) over the SOTA QAT baseline.