OpenRaiser/Intern-Atlas huggingface dataset:
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal language models (MLMs) still face challenges in fundamental visual perception tasks where specialized models excel. Tasks requiring reasoning about 3D structures benefit from depth estimation, and reasoning about 2D object instances benefits from object detection. Yet, MLMs can not produce intermediate depth or boxes to reason over. Finetuning MLMs on relevant data doesn't generalize well and outsourcing computation to specialized vision tools is too compute-intensive and memory-inefficient. To address this, we introduce Perception Tokens, intrinsic image representations designed to assist reasoning tasks where language is insufficient. Perception tokens act as auxiliary reasoning tokens, akin to chain-of-thought prompts in language models. For example, in a depth-related task, an MLM augmented with perception tokens can reason by generating a depth map as tokens, enabling it to solve the problem effectively. We propose AURORA, a training method that augments MLMs with perception tokens for improved reasoning over visual inputs. AURORA leverages a VQVAE to transform intermediate image representations, such as depth maps into a tokenized format and bounding box tokens, which is then used in a multi-task training framework. AURORA achieves notable improvements across counting benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench, outperforming finetuning approaches in generalization across datasets. It also improves on relative depth: over +6% on BLINK. With perception tokens, AURORA expands the scope of MLMs beyond language-based reasoning, paving the way for more effective visual reasoning capabilities.
PaliGemma 2: A Family of Versatile VLMs for Transfer
Venue: arXiv cs.CV 2024
Year: 2024

PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning. The resulting family of base models covering different model sizes and resolutions allows us to investigate factors impacting transfer performance (such as learning rate) and to analyze the interplay between the type of task, model size, and resolution. We further increase the number and breadth of transfer tasks beyond the scope of PaliGemma including different OCR-related tasks such as table structure recognition, molecular structure recognition, music score recognition, as well as long fine-grained captioning and radiography report generation, on which PaliGemma 2 obtains state-of-the-art results.
FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes
Venue: arXiv cs.CV 2024
Year: 2024

We propose FreeSim, a camera simulation method for autonomous driving. FreeSim emphasizes high-quality rendering from viewpoints beyond the recorded ego trajectories. In such viewpoints, previous methods have unacceptable degradation because the training data of these viewpoints is unavailable. To address such data scarcity, we first propose a generative enhancement model with a matched data construction strategy. The resulting model can generate high-quality images in a viewpoint slightly deviated from the recorded trajectories, conditioned on the degraded rendering of this viewpoint. We then propose a progressive reconstruction strategy, which progressively adds generated images of unrecorded views into the reconstruction process, starting from slightly off-trajectory viewpoints and moving progressively farther away. With this progressive generation-reconstruction pipeline, FreeSim supports high-quality off-trajectory view synthesis under large deviations of more than 3 meters.
Improving Tool Retrieval by Leveraging Large Language Models for Query Generation
Venue: arXiv cs.IR 2024
Year: 2024

Using tools by Large Language Models (LLMs) is a promising avenue to extend their reach beyond language or conversational settings. The number of tools can scale to thousands as they enable accessing sensory information, fetching updated factual knowledge, or taking actions in the real world. In such settings, in-context learning by providing a short list of relevant tools in the prompt is a viable approach. To retrieve relevant tools, various approaches have been suggested, ranging from simple frequency-based matching to dense embedding-based semantic retrieval. However, such approaches lack the contextual and common-sense understanding required to retrieve the right tools for complex user requests. Rather than increasing the complexity of the retrieval component itself, we propose leveraging LLM understanding to generate a retrieval query. Then, the generated query is embedded and used to find the most relevant tools via a nearest-neighbor search. We investigate three approaches for query generation: zero-shot prompting, supervised fine-tuning on tool descriptions, and alignment learning by iteratively optimizing a reward metric measuring retrieval performance. By conducting extensive experiments on a dataset covering complex and multi-tool scenarios, we show that leveraging LLMs for query generation improves the retrieval for in-domain (seen tools) and out-of-domain (unseen tools) settings.
Leveraging Large Language Models for Generating Labeled Mineral Site Record Linkage Data
Venue: arXiv cs.IR 2024
Year: 2024

Record linkage integrates diverse data sources by identifying records that refer to the same entity. In the context of mineral site records, accurate record linkage is crucial for identifying and mapping mineral deposits. Properly linking records that refer to the same mineral deposit helps define the spatial coverage of mineral areas, benefiting resource identification and site data archiving. Mineral site record linkage falls under the spatial record linkage category since the records contain information about the physical locations and non-spatial attributes in a tabular format. The task is particularly challenging due to the heterogeneity and vast scale of the data. While prior research employs pre-trained discriminative language models (PLMs) on spatial entity linkage, they often require substantial amounts of curated ground-truth data for fine-tuning. Gathering and creating ground truth data is both time-consuming and costly. Therefore, such approaches are not always feasible in real-world scenarios where gold-standard data are unavailable. Although large generative language models (LLMs) have shown promising results in various natural language processing tasks, including record linkage, their high inference time and resource demand present challenges. We propose a method that leverages an LLM to generate training data and fine-tune a PLM to address the training data gap while preserving the efficiency of PLMs. Our approach achieves over 45\% improvement in F1 score for record linkage compared to traditional PLM-based methods using ground truth data while reducing the inference time by nearly 18 times compared to relying on LLMs. Additionally, we offer an automated pipeline that eliminates the need for human intervention, highlighting this approach's potential to overcome record linkage challenges.
OKG: On-the-Fly Keyword Generation in Sponsored Search Advertising
Venue: arXiv cs.IR 2024
Year: 2024

Current keyword decision-making in sponsored search advertising relies on large, static datasets, limiting the ability to automatically set up keywords and adapt to real-time KPI metrics and product updates that are essential for effective advertising. In this paper, we propose On-the-fly Keyword Generation (OKG), an LLM agent-based method that dynamically monitors KPI changes and adapts keyword generation in real time, aligning with strategies recommended by advertising platforms. Additionally, we introduce the first publicly accessible dataset containing real keyword data along with its KPIs across diverse domains, providing a valuable resource for future research. Experimental results show that OKG significantly improves keyword adaptability and responsiveness compared to traditional methods. The code for OKG and the dataset are available at https://github.com/sony/okg.
Reinforced Symbolic Learning with Logical Constraints for Predicting Turbine Blade Fatigue Life
Venue: arXiv cs.LG 2024
Year: 2024

Accurate prediction of turbine blade fatigue life is essential for ensuring the safety and reliability of aircraft engines. A significant challenge in this domain is uncovering the intrinsic relationship between mechanical properties and fatigue life. This paper introduces Reinforced Symbolic Learning (RSL), a method that derives predictive formulas linking these properties to fatigue life. RSL incorporates logical constraints during symbolic optimization, ensuring that the generated formulas are both physically meaningful and interpretable. The optimization process is further enhanced using deep reinforcement learning, which efficiently guides the symbolic regression towards more accurate models. The proposed RSL method was evaluated on two turbine blade materials, GH4169 and TC4, to identify optimal fatigue life prediction models. When compared with six empirical formulas and five machine learning algorithms, RSL not only produces more interpretable formulas but also achieves superior or comparable predictive accuracy. Additionally, finite element simulations were conducted to assess mechanical properties at critical points on the blade, which were then used to predict fatigue life under various operating conditions.
A Survey on E-Commerce Learning to Rank
Venue: arXiv cs.IR 2024
Year: 2024

In e-commerce, ranking the search results based on users' preference is the most important task. Commercial e-commerce platforms, such as, Amazon, Alibaba, eBay, Walmart, etc. perform extensive and relentless research to perfect their search result ranking algorithms because the quality of ranking drives a user's decision to purchase or not to purchase an item, directly affecting the profitability of the e-commerce platform. In such a commercial platforms, for optimizing search result ranking numerous features are considered, which emerge from relevance, personalization, seller's reputation and paid promotion. To maintain their competitive advantage in the market, the platforms do no publish their core ranking algorithms, so it is difficult to know which of the algorithms or which of the features is the most effective for finding the most optimal search result ranking in e-commerce. No extensive surveys of ranking to rank in the e-commerce domain is also not yet published. In this work, we survey the existing e-commerce learning to rank algorithms. Besides, we also compare these algorithms based on query relevance criterion on a large real-life e-commerce dataset and provide a quantitative analysis. To the best of our knowledge this is the first such survey which include an experimental comparison among various learning to rank algorithms.
Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

Procedural Knowledge is the know-how expressed in the form of sequences of steps needed to perform some tasks. Procedures are usually described by means of natural language texts, such as recipes or maintenance manuals, possibly spread across different documents and systems, and their interpretation and subsequent execution is often left to the reader. Representing such procedures in a Knowledge Graph (KG) can be the basis to build digital tools to support those users who need to apply or execute them. In this paper, we leverage Large Language Model (LLM) capabilities and propose a prompt engineering approach to extract steps, actions, objects, equipment and temporal information from a textual procedure, in order to populate a Procedural KG according to a pre-defined ontology. We evaluate the KG extraction results by means of a user study, in order to qualitatively and quantitatively assess the perceived quality and usefulness of the LLM-extracted procedural knowledge. We show that LLMs can produce outputs of acceptable quality and we assess the subjective perception of AI by human evaluators.
Soft-Output Successive Cancellation List Decoding
Venue: arXiv cs.IT 2024
Year: 2024

We introduce an algorithm for approximating the codebook probability that is compatible with all successive cancellation (SC)-based decoding algorithms, including SC list (SCL) decoding. This approximation is based on an auxiliary distribution that mimics the dynamics of decoding algorithms with an SC decoding schedule. Based on this codebook probability and SCL decoding, we introduce soft-output SCL (SO-SCL) to generate both blockwise and bitwise soft-output (SO). Using that blockwise SO, we first establish that, in terms of both block error rate (BLER) and undetected error rate (UER), SO-SCL decoding of dynamic Reed-Muller (RM) codes significantly outperforms the CRC-concatenated polar codes from 5G New Radio under SCL decoding. Moreover, using SO-SCL, the decoding misdetection rate (MDR) can be constrained to not exceed any predefined value, making it suitable for practical systems. Proposed bitwise SO can be readily generated from blockwise SO via a weighted sum of beliefs that includes a term where SO is weighted by the codebook probability, resulting in a soft-input soft-output (SISO) decoder. Simulation results for SO-SCL iterative decoding of product codes and generalized LDPC (GLDPC) codes, along with information-theoretical analysis, demonstrate significant superiority over existing list-max and list-sum approximations.
Resilience and Criticality: Brothers in Arms for 6G
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we develop the first comprehensive tutorial on designing future 6G networks that synergistically integrate notions of resilience and criticality from the ground up. While resilience refers to the ability to absorb, adapt to, and recover from adversarial or challenging conditions, criticality indicates the degree of importance or urgency assigned to a particular service or component. Despite a spiking interest in designing resilient wireless networks, most prior works do not provide a unified resilience definition, nor harness the intricate interplay between resilience and criticality. In order to fill this gap, in this paper, we highlight the importance of a criticality-aware approach as a key enabler for providing reliable and resilient service functionality. Moreover, we delve into the unique challenges and opportunities of the envisioned 6G features pertaining to resilience and (mixed) criticality. After reviewing resilience definitions, we present a core resilience strategy, a unified resilience metric, different criteria for service criticality, and prioritization frameworks, that augment the 6G resilience prospects. Afterwards, we explore the opportunities presented by promising technologies that enable a resilient 6G network design from a radio access network protocol stack perspective. We briefly revisit state-of-the-art network architectures, establish a rough connection to the Open-RAN Alliance vision, and discuss opportunities, existing techniques, and promising enabling mechanisms for 6G at each layer. Finally, the article discusses important research directions and open problems concerning resilience and criticality in 6G.
Hyperparameter Tuning Through Pessimistic Bilevel Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Automated hyperparameter search in machine learning, especially for deep learning models, is typically formulated as a bilevel optimization problem, with hyperparameter values determined by the upper level and the model learning achieved by the lower-level problem. Most of the existing bilevel optimization solutions either assume the uniqueness of the optimal training model given hyperparameters or adopt an optimistic view when the non-uniqueness issue emerges. Potential model uncertainty may arise when training complex models with limited data, especially when the uniqueness assumption is violated. Thus, the suitability of the optimistic view underlying current bilevel hyperparameter optimization solutions is questionable. In this paper, we propose pessimistic bilevel hyperparameter optimization to assure appropriate outer-level hyperparameters to better generalize the inner-level learned models, by explicitly incorporating potential uncertainty of the inner-level solution set. To solve the resulting computationally challenging pessimistic bilevel optimization problem, we develop a novel relaxation-based approximation method. It derives pessimistic solutions with more robust prediction models. In our empirical studies of automated hyperparameter search for binary linear classifiers, pessimistic solutions have demonstrated better prediction performances than optimistic counterparts when we have limited training data or perturbed testing data, showing the necessity of considering pessimistic solutions besides existing optimistic ones.
Securing RC Based P2P Networks: A Blockchain-based Access Control Framework utilizing Ethereum Smart Contracts for IoT and Web 3.0
Venue: arXiv cs.CR 2024
Year: 2024

Ensuring security for highly dynamic peer-to-peer (P2P) networks has always been a challenge, especially for services like online transactions and smart devices. These networks experience high churn rates, making it difficult to maintain appropriate access control. Traditional systems, particularly Role-Based Access Control (RBAC), often fail to meet the needs of a P2P environment. This paper presents a blockchain-based access control framework that uses Ethereum smart contracts to address these challenges. Our framework aims to close the gaps in existing access control systems by providing flexible, transparent, and decentralized security solutions. The proposed framework includes access control contracts (ACC) that manage access based on static and dynamic policies, a Judge Contract (JC) to handle misbehavior, and a Register Contract (RC) to record and manage the interactions between ACCs and JC. The security model combines impact and severity-based threat assessments using the CIA (Confidentiality, Integrity, Availability) and STRIDE principles, ensuring responses are tailored to different threat levels. This system not only stabilizes the fundamental issues of peer membership but also offers a scalable solution, particularly valuable in areas such as the Internet of Things (IoT) and Web 3.0 technologies.
A Water Efficiency Dataset for African Data Centers
Venue: arXiv cs.LG 2024
Year: 2024

AI computing and data centers consume a large amount of freshwater, both directly for cooling and indirectly for electricity generation. While most attention has been paid to developed countries such as the U.S., this paper presents the first-of-its-kind dataset that combines nation-level weather and electricity generation data to estimate water usage efficiency for data centers in 41 African countries across five different climate regions. We also use our dataset to evaluate and estimate the water consumption of inference on two large language models (i.e., Llama-3-70B and GPT-4) in 11 selected African countries. Our findings show that writing a 10-page report using Llama-3-70B could consume about \textbf{0.7 liters} of water, while the water consumption by GPT-4 for the same task may go up to about 60 liters. For writing a medium-length email of 120-200 words, Llama-3-70B and GPT-4 could consume about \textbf{0.13 liters} and 3 liters of water, respectively. Interestingly, given the same AI model, 8 out of the 11 selected African countries consume less water than the global average, mainly because of lower water intensities for electricity generation. However, water consumption can be substantially higher in some African countries with a steppe climate than the U.S. and global averages, prompting more attention when deploying AI computing in these countries. Our dataset is publicly available on \href{https://huggingface.co/datasets/masterlion/WaterEfficientDatasetForAfricanCountries/tree/main}{Hugging Face}.
Language Model Meets Prototypes: Towards Interpretable Text Classification Models through Prototypical Networks
Venue: arXiv cs.CL 2024
Year: 2024

Pretrained transformer-based Language Models (LMs) are well-known for their ability to achieve significant improvement on NLP tasks, but their black-box nature, which leads to a lack of interpretability, has been a major concern. My dissertation focuses on developing intrinsically interpretable models when using LMs as encoders while maintaining their superior performance via prototypical networks. I initiated my research by investigating enhancements in performance for interpretable models of sarcasm detection. My proposed approach focuses on capturing sentiment incongruity to enhance accuracy while offering instance-based explanations for the classification decisions. Later, I developed a novel white-box multi-head graph attention-based prototype network designed to explain the decisions of text classification models without sacrificing the accuracy of the original black-box LMs. In addition, I am working on extending the attention-based prototype network with contrastive learning to redesign an interpretable graph neural network, aiming to enhance both the interpretability and performance of the model in document classification.
Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning
Venue: arXiv cs.LG 2024
Year: 2024

The exploration \& exploitation dilemma poses significant challenges in reinforcement learning (RL). Recently, curiosity-based exploration methods achieved great success in tackling hard-exploration problems. However, they necessitate extensive hyperparameter tuning on different environments, which heavily limits the applicability and accessibility of this line of methods. In this paper, we characterize this problem via analysis of the agent behavior, concluding the fundamental difficulty of choosing a proper hyperparameter. We then identify the difficulty and the instability of the optimization when the agent learns with curiosity. We propose our method, hyperparameter robust exploration (\textbf{Hyper}), which extensively mitigates the problem by effectively regularizing the visitation of the exploration and decoupling the exploitation to ensure stable training. We theoretically justify that \textbf{Hyper} is provably efficient under function approximation setting and empirically demonstrate its appealing performance and robustness in various environments.
Speech Recognition-based Feature Extraction for Enhanced Automatic Severity Classification in Dysarthric Speech
Venue: arXiv cs.SD 2024
Year: 2024

Due to the subjective nature of current clinical evaluation, the need for automatic severity evaluation in dysarthric speech has emerged. DNN models outperform ML models but lack user-friendly explainability. ML models offer explainable results at a feature level, but their performance is comparatively lower. Current ML models extract various features from raw waveforms to predict severity. However, existing methods do not encompass all dysarthric features used in clinical evaluation. To address this gap, we propose a feature extraction method that minimizes information loss. We introduce an ASR transcription as a novel feature extraction source. We finetune the ASR model for dysarthric speech, then use this model to transcribe dysarthric speech and extract word segment boundary information. It enables capturing finer pronunciation and broader prosodic features. These features demonstrated an improved severity prediction performance to existing features: balanced accuracy of 83.72%.