OpenRaiser/Intern-Atlas huggingface dataset:
Sharpness-Aware Black-Box Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Black-box optimization algorithms have been widely used in various machine learning problems, including reinforcement learning and prompt fine-tuning. However, directly optimizing the training loss value, as commonly done in existing black-box optimization methods, could lead to suboptimal model quality and generalization performance. To address those problems in black-box optimization, we propose a novel Sharpness-Aware Black-box Optimization (SABO) algorithm, which applies a sharpness-aware minimization strategy to improve the model generalization. Specifically, the proposed SABO method first reparameterizes the objective function by its expectation over a Gaussian distribution. Then it iteratively updates the parameterized distribution by approximated stochastic gradients of the maximum objective value within a small neighborhood around the current solution in the Gaussian distribution space. Theoretically, we prove the convergence rate and generalization bound of the proposed SABO algorithm. Empirically, extensive experiments on the black-box prompt fine-tuning tasks demonstrate the effectiveness of the proposed SABO method in improving model generalization performance.
Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) have shown remarkable capabilities in natural language processing but exhibit significant performance gaps among different languages. Most existing approaches to address these disparities rely on pretraining or fine-tuning, which are resource-intensive. To overcome these limitations without incurring significant costs, we propose Inference-Time Cross-Lingual Intervention (INCLINE), a novel framework that enhances LLM performance on low-performing (source) languages by aligning their internal representations with those of high-performing (target) languages during inference. INCLINE initially learns alignment matrices using parallel sentences from source and target languages through a Least-Squares optimization, and then applies these matrices during inference to transform the low-performing language representations toward the high-performing language space. Extensive experiments on nine benchmarks with five LLMs demonstrate that INCLINE significantly improves performance across diverse tasks and languages, compared to recent strong baselines. Our analysis demonstrates that INCLINE is highly cost-effective and applicable to a wide range of applications. In addition, we release the code to foster research along this line: https://github.com/weixuan-wang123/INCLINE.
RADS-Checker: Measuring Compliance with Right of Access by the Data Subject in Android Markets
Venue: arXiv cs.CR 2024
Year: 2024

The latest data protection regulations worldwide, such as the General Data Protection Regulation (GDPR), have established the Right of Access by the Data Subject (RADS), granting users the right to access and obtain a copy of their personal data from the data controllers. This clause can effectively compel data controllers to handle user personal data more cautiously, which is of significant importance for protecting user privacy. However, there is currently no research systematically examining whether RADS has been effectively implemented in mobile apps, which are the most common personal data controllers. In this study, we propose a compliance measurement framework for RADS in apps. In our framework, we first analyze an app's privacy policy text using NLP techniques such as GPT-4 to verify whether it clearly declares offering RADS to users and provides specific details on how the right can be exercised. Next, we assess the authenticity and usability of the identified implementation methods by submitting data access requests to the app. Finally, for the obtained data copies, we further verify their completeness by comparing them with the user personal data actually collected by the app during runtime, as captured by Frida Hook. We analyzed a total of 1,631 apps in the American app market G and the Chinese app market H. The results show that less than 54.50% and 37.05% of apps in G and H, respectively, explicitly state in their privacy policies that they can provide users with copies of their personal data. Additionally, in both app markets, less than 20% of apps could truly provide users with their data copies. Finally, among the obtained data copies, only about 2.94% from G pass the completeness verification.
Stability properties for subgroups generated by return words
Venue: arXiv cs.DM 2024
Year: 2024

Return words are a classical tool for studying shift spaces with low factor complexity. In recent years, their projection inside groups have attracted some attention, for instance in the context of dendric shift spaces, of generation of pseudorandom numbers (through the welldoc property), and of profinite invariants of shift spaces. Aiming at unifying disparate works, we introduce a notion of stability for subgroups generated by return words. Within this framework, we revisit several existing results and generalize some of them. We also study general aspects of stability, such as decidability or closure under certain operations.
LLM-based Translation Inference with Iterative Bilingual Understanding
Venue: arXiv cs.CL 2024
Year: 2024

The remarkable understanding and generation capabilities of large language models (LLMs) have greatly improved translation performance. However, incorrect understanding of the sentence to be translated can degrade translation quality. To address this issue, we proposed a novel Iterative Bilingual Understanding Translation (IBUT) method based on the cross-lingual capabilities of LLMs and the dual characteristics of translation tasks. The cross-lingual capability of LLMs enables the generation of contextual understanding for both the source and target languages separately. Furthermore, the dual characteristics allow IBUT to generate effective cross-lingual feedback, iteratively refining contextual understanding, thereby reducing errors and improving translation performance. Experimental results showed that the proposed IBUT outperforms several strong comparison methods, especially being generalized to multiple domains (e.g., news, commonsense, and cultural translation benchmarks).
Exploring Plural Perspectives in Self-Tracking Technologies: Trust and Reflection in Self Tracking Practices
Venue: arXiv cs.HC 2024
Year: 2024

Contemporary self-tracking technologies (STTs), such as smartwatches and smartphone apps, allow people to become self-aware through the datafication of their everyday lives. However, concerns are emerging over the global north/Western portrayal of the self in the envisionment of STTs. Given the call to diversify participant samples in HCI knowledge building, we see it timely in understanding the influence of ubiquitous STTs in global south societies. We conduct a between-group analysis of 156 and 121 participants from Global North and South through two iterative surveys, respectively. We uncover significant differences in perceived trust with their STTs and reflection practices between the groups. We provide an empirical understanding on advocating for inclusive design strategies that recognize diverse interpretations of STTs and highlight the need to prioritize local values and flexibility in tracking to foster deeper reflection across cultures. Lastly, we discuss our findings in relation to the existing literature and highlight design recommendations for future research.
Investigating Sensitive Directions in GPT-2: An Improved Baseline and Comparative Analysis of SAEs
Venue: arXiv cs.LG 2024
Year: 2024

Sensitive directions experiments attempt to understand the computational features of Language Models (LMs) by measuring how much the next token prediction probabilities change by perturbing activations along specific directions. We extend the sensitive directions work by introducing an improved baseline for perturbation directions. We demonstrate that KL divergence for Sparse Autoencoder (SAE) reconstruction errors are no longer pathologically high compared to the improved baseline. We also show that feature directions uncovered by SAEs have varying impacts on model outputs depending on the SAE's sparsity, with lower L0 SAE feature directions exerting a greater influence. Additionally, we find that end-to-end SAE features do not exhibit stronger effects on model outputs compared to traditional SAEs.
Leveraging Augmented Reality for Improved Situational Awareness During UAV-Driven Search and Rescue Missions
Venue: arXiv cs.RO 2024
Year: 2024

In the high-stakes domain of search-and-rescue missions, the deployment of Unmanned Aerial Vehicles (UAVs) has become increasingly pivotal. These missions require seamless, real-time communication among diverse roles within response teams, particularly between Remote Operators (ROs) and On-Site Operators (OSOs). Traditionally, ROs and OSOs have relied on radio communication to exchange critical information, such as the geolocation of victims, hazardous areas, and points of interest. However, radio communication lacks information visualization, suffers from noise, and requires mental effort to interpret information, leading to miscommunications and misunderstandings. To address these challenges, this paper presents VizCom-AR, an Augmented Reality system designed to facilitate visual communication between ROs and OSOs and their situational awareness during UAV-driven search-and-rescue missions. Our experiments, focus group sessions with police officers, and field study showed that VizCom-AR enhances spatial awareness of both ROs and OSOs, facilitate geolocation information exchange, and effectively complement existing communication tools in UAV-driven emergency response missions. Overall, VizCom-AR offers a fundamental framework for designing Augmented Reality systems for large scale UAV-driven rescue missions.
Adaptive Prompt Learning with SAM for Few-shot Scanning Probe Microscope Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

The Segment Anything Model (SAM) has demonstrated strong performance in image segmentation of natural scene images. However, its effectiveness diminishes markedly when applied to specific scientific domains, such as Scanning Probe Microscope (SPM) images. This decline in accuracy can be attributed to the distinct data distribution and limited availability of the data inherent in the scientific images. On the other hand, the acquisition of adequate SPM datasets is both time-intensive and laborious as well as skill-dependent. To address these challenges, we propose an Adaptive Prompt Learning with SAM (APL-SAM) framework tailored for few-shot SPM image segmentation. Our approach incorporates two key innovations to enhance SAM: 1) An Adaptive Prompt Learning module leverages few-shot embeddings derived from limited support set to learn adaptively central representatives, serving as visual prompts. This innovation eliminates the need for time-consuming online user interactions for providing prompts, such as exhaustively marking points and bounding boxes slice by slice; 2) A multi-source, multi-level mask decoder specifically designed for few-shot SPM image segmentation is introduced, which can effectively capture the correspondence between the support and query images. To facilitate comprehensive training and evaluation, we introduce a new dataset, SPM-Seg, curated for SPM image segmentation. Extensive experiments on this dataset reveal that the proposed APL-SAM framework significantly outperforms the original SAM, achieving over a 30% improvement in terms of Dice Similarity Coefficient with only one-shot guidance. Moreover, APL-SAM surpasses state-of-the-art few-shot segmentation methods and even fully supervised approaches in performance. Code and dataset used in this study will be made available upon acceptance.
FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion
Venue: arXiv cs.CV 2024
Year: 2024

Benefiting from the revolutionary advances in large language models (LLMs) and foundational vision models, large vision-language models (LVLMs) have also made significant progress. However, current benchmarks focus on tasks that evaluating only a single aspect of LVLM capabilities (e.g., recognition, detection, understanding). These tasks fail to fully demonstrate LVLMs' potential in complex application scenarios. To comprehensively assess the performance of existing LVLMs, we propose a more challenging task called the Flow Text with Image Insertion task (FTII). This task requires LVLMs to simultaneously possess outstanding abilities in image comprehension, instruction understanding, and long-text interpretation. Specifically, given several text paragraphs and a set of candidate images, as the text paragraphs accumulate, the LVLMs are required to select the most suitable image from the candidates to insert after the corresponding paragraph. Constructing a benchmark for such a task is highly challenging, particularly in determining the sequence of flowing text and images. To address this challenge, we turn to professional news reports, which naturally contain a gold standard for image-text sequences. Based on this, we introduce the Flow Text with Image Insertion Benchmark (FTII-Bench), which includes 318 high-quality Chinese image-text news articles and 307 high-quality English image-text news articles, covering 10 different news domains. Using these 625 high-quality articles, we construct problems of two different types with multiple levels of difficulty. Furthermore, we establish two different evaluation pipelines based on the CLIP model and existing LVLMs. We evaluate 9 open-source and 2 closed-source LVLMs as well as 2 CLIP-based models. Results indicate that even the most advanced models (e.g., GPT-4o) face significant challenges when tackling the FTII task.
Parsing Akkadian Verbs with Prolog
Venue: arXiv cs.CL 2024
Year: 2024

This paper describes a parsing/generation system for finite verbal forms in Akkadian, with the possible addition of suffixes, implemented in Prolog. The work described provides the framework and engine to interpret the D, N, and G stems along with accusative, dative and ventive endings.
Faster Algorithms for Growing Collision-Free Convex Polytopes in Robot Configuration Space
Venue: arXiv cs.RO 2024
Year: 2024

We propose two novel algorithms for constructing convex collision-free polytopes in robot configuration space. Finding these polytopes enables the application of stronger motion-planning frameworks such as trajectory optimization with Graphs of Convex Sets [1] and is currently a major roadblock in the adoption of these approaches. In this paper, we build upon IRIS-NP (Iterative Regional Inflation by Semidefinite & Nonlinear Programming) [2] to significantly improve tunability, runtimes, and scaling to complex environments. IRIS-NP uses nonlinear programming paired with uniform random initialization to find configurations on the boundary of the free configuration space. Our key insight is that finding near-by configuration-space obstacles using sampling is inexpensive and greatly accelerates region generation. We propose two algorithms using such samples to either employ nonlinear programming more efficiently (IRIS-NP2 ) or circumvent it altogether using a massively-parallel zero-order optimization strategy (IRIS-ZO). We also propose a termination condition that controls the probability of exceeding a user-specified permissible fraction-in-collision, eliminating a significant source of tuning difficulty in IRIS-NP. We compare performance across eight robot environments, showing that IRIS-ZO achieves an order-of-magnitude speed advantage over IRIS-NP. IRISNP2, also significantly faster than IRIS-NP, builds larger polytopes using fewer hyperplanes, enabling faster downstream computation. Website: https://sites.google.com/view/fastiris
Physics-Informed Learning for the Friction Modeling of High-Ratio Harmonic Drives
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents a scalable method for friction identification in robots equipped with electric motors and high-ratio harmonic drives, utilizing Physics-Informed Neural Networks (PINN). This approach eliminates the need for dedicated setups and joint torque sensors by leveraging the roboťs intrinsic model and state data. We present a comprehensive pipeline that includes data acquisition, preprocessing, ground truth generation, and model identification. The effectiveness of the PINN-based friction identification is validated through extensive testing on two different joints of the humanoid robot ergoCub, comparing its performance against traditional static friction models like the Coulomb-viscous and Stribeck-Coulomb-viscous models. Integrating the identified PINN-based friction models into a two-layer torque control architecture enhances real-time friction compensation. The results demonstrate significant improvements in control performance and reductions in energy losses, highlighting the scalability and robustness of the proposed method, also for application across a large number of joints as in the case of humanoid robots.
Reconfiguring homomorphisms to reflexive graphs via a simple reduction
Venue: arXiv cs.DM 2024
Year: 2024

Given a graph $G$ and two graph homomorphisms $α$ and $β$ from $G$ to a fixed graph $H$, the problem $H$-Recoloring asks whether there is a transformation from $α$ to $β$ that changes the image of a single vertex at each step and keeps a graph homomorphism throughout. The complexity of the problem depends among other things on the presence of loops on the vertices. We provide a simple reduction that, using a known algorithmic result for $H$-Recoloring for square-free irreflexive graphs $H$, yields a polynomial-time algorithm for $H$-Recoloring for square-free reflexive graphs $H$. This generalizes all known algorithmic results for $H$-Recoloring for reflexive graphs $H$. Furthermore, the construction allows us to recover some of the known hardness results. Finally, we provide a partial inverse of the construction for bipartite instances.
Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in diffusion models trained on large-scale data have enabled the generation of indistinguishable human-level images, yet they often produce harmful content misaligned with human values, e.g., social bias, and offensive content. Despite extensive research on Large Language Models (LLMs), the challenge of Text-to-Image (T2I) model alignment remains largely unexplored. Addressing this problem, we propose LiVO (Lightweight Value Optimization), a novel lightweight method for aligning T2I models with human values. LiVO only optimizes a plug-and-play value encoder to integrate a specified value principle with the input prompt, allowing the control of generated images over both semantics and values. Specifically, we design a diffusion model-tailored preference optimization loss, which theoretically approximates the Bradley-Terry model used in LLM alignment but provides a more flexible trade-off between image quality and value conformity. To optimize the value encoder, we also develop a framework to automatically construct a text-image preference dataset of 86k (prompt, aligned image, violating image, value principle) samples. Without updating most model parameters and through adaptive value selection from the input prompt, LiVO significantly reduces harmful outputs and achieves faster convergence, surpassing several strong baselines and taking an initial step towards ethically aligned T2I models.
Comparative Analysis of Extrinsic Factors for NER in French
Venue: arXiv cs.CL 2024
Year: 2024

Named entity recognition (NER) is a crucial task that aims to identify structured information, which is often replete with complex, technical terms and a high degree of variability. Accurate and reliable NER can facilitate the extraction and analysis of important information. However, NER for other than English is challenging due to limited data availability, as the high expertise, time, and expenses are required to annotate its data. In this paper, by using the limited data, we explore various factors including model structure, corpus annotation scheme and data augmentation techniques to improve the performance of a NER model for French. Our experiments demonstrate that these approaches can significantly improve the model's F1 score from original CRF score of 62.41 to 79.39. Our findings suggest that considering different extrinsic factors and combining these techniques is a promising approach for improving NER performance where the size of data is limited.
The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
Venue: arXiv cs.LG 2024
Year: 2024

Model merging aims to efficiently combine the weights of multiple expert models, each trained on a specific task, into a single multi-task model, with strong performance across all tasks. When applied to all but the last layer of weights, existing methods -- such as Task Arithmetic, TIES-merging, and TALL mask merging -- work well to combine expert models obtained by fine-tuning a common foundation model, operating within a "local" neighborhood of the foundation model. This work explores the more challenging scenario of "non-local" merging, which we find arises when an expert model changes significantly during pretraining or where the expert models do not even share a common foundation model. We observe that standard merging techniques often fail to generalize effectively in this non-local setting, even when accounting for permutation symmetries using standard techniques. We identify that this failure is, in part, due to "variance collapse", a phenomenon identified also in the setting of linear mode connectivity by Jordan et al. (2023). To address this, we propose a multi-task technique to re-scale and shift the output activations of the merged model for each task, aligning its output statistics with those of the corresponding task-specific expert models. Our experiments demonstrate that this correction significantly improves the performance of various model merging approaches in non-local settings, providing a strong baseline for future research on this problem.
Identification of crowds using mobile crowd detection (MCS) and visualization with the DBSCAN algorithm for a Smart Campus environment
Venue: arXiv cs.CY 2024
Year: 2024

Multidisciplinary research, in conjunction with artificial intelligence (AI), the Internet of Things (IoT), Blockchain and Big Data analysis, has lowered barriers and made companies more productive, in other words, the joint work of these areas has promoted digital transformation in all areas, for example Artificial intelligence (AI) has made it possible to automate processes, and the Internet of Things (IoT) has connected devices and physical objects, enabling real-time data collection and analysis. Blockchain has provided a secure and transparent way to transact and store data. Big Data analysis has allowed companies to obtain valuable insights from large amounts of data. As these technologies continue to evolve, we can expect to see even more innovations and benefits in the future. This paper explores the feasibility of using Mobile Crowd Sensing (MCS) and visualization algorithms to detect crowding on a university campus. A survey was conducted to evaluate the university community's perception of a mobile application that provides information about crowds, and a detection scenario was simulated using randomly generated data and the DBSCAN algorithm for visualization. Preliminary results suggest that the system is viable and could be a useful tool for the prevention of accidents due to crowding and for the management of public spaces. The limitations of the study are discussed and future lines of research are proposed, such as crowd prediction, data privacy, and visualization optimization.