OpenRaiser/Intern-Atlas huggingface dataset:
Web Retrieval Agents for Evidence-Based Misinformation Detection
Venue: arXiv cs.IR 2024
Year: 2024

This paper develops an agent-based automated fact-checking approach for detecting misinformation. We demonstrate that combining a powerful LLM agent, which does not have access to the internet for searches, with an online web search agent yields better results than when each tool is used independently. Our approach is robust across multiple models, outperforming alternatives and increasing the macro F1 of misinformation detection by as much as 20 percent compared to LLMs without search. We also conduct extensive analyses on the sources our system leverages and their biases, decisions in the construction of the system like the search tool and the knowledge base, the type of evidence needed and its impact on the results, and other parts of the overall process. By combining strong performance with in-depth understanding, we hope to provide building blocks for future search-enabled misinformation mitigation systems.
Evolving Text Data Stream Mining
Venue: arXiv cs.IR 2024
Year: 2024

A text stream is an ordered sequence of text documents generated over time. A massive amount of such text data is generated by online social platforms every day. Designing an algorithm for such text streams to extract useful information is a challenging task due to unique properties of the stream such as infinite length, data sparsity, and evolution. Thereby, learning useful information from such streaming data under the constraint of limited time and memory has gained increasing attention. During the past decade, although many text stream mining algorithms have proposed, there still exists some potential issues. First, high-dimensional text data heavily degrades the learning performance until the model either works on subspace or reduces the global feature space. The second issue is to extract semantic text representation of documents and capture evolving topics over time. Moreover, the problem of label scarcity exists, whereas existing approaches work on the full availability of labeled data. To deal with these issues, in this thesis, new learning models are proposed for clustering and multi-label learning on text streams.
Could Micro-Expressions be Quantified? Electromyography Gives Affirmative Evidence
Venue: arXiv cs.HC 2024
Year: 2024

Micro-expressions (MEs) are brief, subtle facial expressions that reveal concealed emotions, offering key behavioral cues for social interaction. Characterized by short duration, low intensity, and spontaneity, MEs have been mostly studied through subjective coding, lacking objective, quantitative indicators. This paper explores ME characteristics using facial electromyography (EMG), analyzing data from 147 macro-expressions (MaEs) and 233 MEs collected from 35 participants. First, regarding external characteristics, we demonstrate that MEs are short in duration and low in intensity. Precisely, we proposed an EMG-based indicator, the percentage of maximum voluntary contraction (MVC\%), to measure ME intensity. Moreover, we provided precise interval estimations of ME intensity and duration, with MVC\% ranging from 7\% to 9.2\% and the duration ranging from 307 ms to 327 ms. This research facilitates fine-grained ME quantification. Second, regarding the internal characteristics, we confirm that MEs are less controllable and consciously recognized compared to MaEs, as shown by participants responses and self-reports. This study provides a theoretical basis for research on ME mechanisms and real-life applications. Third, building on our previous work, we present CASMEMG, the first public ME database including EMG signals, providing a robust foundation for studying micro-expression mechanisms and movement dynamics through physiological signals.
Analysis of nonlocal smart beams following fractional-order constitutive relations
Venue: arXiv cs.CE 2024
Year: 2024

In this study, we develop a fractional-calculus based constitutive model for capturing nonlocal interactions over the multiphysics response in solids. More specifically, we develop constitutive relations for nonlocal piezoelectricity incorporating fractional-order kinematic relations to capture the long-range interactions over electrical and mechanical field variables. This study breaks new ground by developing fractional-order constitutive models for a two-way multiphysics (electro-mechanical) coupling, specifically the direct and converse piezoelectric effect. It is expected that long-range interactions over each field variable (elastic and electrical) can be leveraged to develop metastructures with enhanced multiphysics coupling. To better illustrate this, we choose the example of a smart beam composed of a nonlocal substrate and a piezoelectric layer. We establish the analytical and numerical framework to analyze nonlocal smart beams based on variational principles. The fractional-Finite Element (f-FE) numerical solver, facilitating multiphysics coupling, undergoes comprehensive validation through multiple case studies. Finally, detailed studies point towards tuning the multiphysics coupling possible via nonlocal interactions across the domain.
Estimation and Visualization of Isosurface Uncertainty from Linear and High-Order Interpolation Methods
Venue: arXiv cs.GR 2024
Year: 2024

Isosurface visualization is fundamental for exploring and analyzing 3D volumetric data. Marching cubes (MC) algorithms with linear interpolation are commonly used for isosurface extraction and visualization. Although linear interpolation is easy to implement, it has limitations when the underlying data is complex and high-order, which is the case for most real-world data. Linear interpolation can output vertices at the wrong location. Its inability to deal with sharp features and features smaller than grid cells can lead to an incorrect isosurface with holes and broken pieces. Despite these limitations, isosurface visualizations typically do not include insight into the spatial location and the magnitude of these errors. We utilize high-order interpolation methods with MC algorithms and interactive visualization to highlight these uncertainties. Our visualization tool helps identify the regions of high interpolation errors. It also allows users to query local areas for details and compare the differences between isosurfaces from different interpolation methods. In addition, we employ high-order methods to identify and reconstruct possible features that linear methods cannot detect. We showcase how our visualization tool helps explore and understand the extracted isosurface errors through synthetic and real-world data.
Leveraging Internet of Things Network Metadata for Cost-Effective Automatic Smart Building Visualization
Venue: arXiv cs.HC 2024
Year: 2024

In recent years, the building sector has experienced an increasing legislative pressure to reduce the energy consumption. This has created a global need for affordable building management systems (BMS) in areas such as lighting-, temperature-, air quality monitoring and control. BMS uses 2D and 3D building representations to visualize various aspects of building operations. Today the creation of these visual building representations relies on labor-intensive and costly computer-aided design (CAD) processes. Hence, to create affordable BMS there is an urgent need to develop methods for cost-effective automatic creation of visual building representations. This paper introduces an automatic, metadata-driven method for constructing building visualizations using metadata from existing smart building infrastructure. The method presented in this study utilizes a Velocity Verlet integration-based physics particle simulation that uses metadata to define the force dynamics within the simulation. This process generates an abstract point cloud representing the organization of BMS components into building zones. The developed system was tested in two buildings of respectively 2,560 m2 and 18,000 m2. The method successfully produced visual building representations based on the available metadata, demonstrating its feasibility and cost-effectiveness.
An alternative formulation of attention pooling function in translation
Venue: arXiv cs.CL 2024
Year: 2024

The aim of this paper is to present an alternative formulation of the attention scoring function in translation tasks. Generally speaking, language is deeply structured, and this is reflected in the attention scoring matrix. We exploit this property to define the attention pooling function, taking this aspect into account. In the first chapters, we introduce the attention mechanism in mathematical terms and explain its limitations and alternative formulations. Next, we focus on the experimental session that led to the alternative formulation. Essentially, we guide queries and keys to interact in a specific manner, encoding the distinct roles of attention heads and directing values on where to seek context. In mathematical terms, we can think of this formula as projecting the attention scores matrix, say $H$, onto the space of band matrices with fixed bandwidth. This convex subspace is clearly finite-dimensional and therefore closed. As a consequence, the projection on this space is well-posed and unique. However, at the price of losing the uniqueness of the projection (i.e., the best approximation for $H$), we defined a new space consisting of band matrices plus error sparse matrices. We prove that this is a compact subspace which guarantees the existence of a matrix that best approximates $H$. We conclude the thesis by validating the new formula, namely calculating how well the new formula for attention scores approximates the original one. Additionally, we explore the impact of different parameters such as w (context windows) and num-pos (number of relevant words in a sentence). These analyses provide deeper insights into how languages are processed and translated, revealing nuances in the roles of context and word relevance.
Generative-Adversarial Networks for Low-Resource Language Data Augmentation in Machine Translation
Venue: arXiv cs.CL 2024
Year: 2024

Neural Machine Translation (NMT) systems struggle when translating to and from low-resource languages, which lack large-scale data corpora for models to use for training. As manual data curation is expensive and time-consuming, we propose utilizing a generative-adversarial network (GAN) to augment low-resource language data. When training on a very small amount of language data (under 20,000 sentences) in a simulated low-resource setting, our model shows potential at data augmentation, generating monolingual language data with sentences such as "ask me that healthy lunch im cooking up," and "my grandfather work harder than your grandfather before." Our novel data augmentation approach takes the first step in investigating the capability of GANs in low-resource NMT, and our results suggest that there is promise for future extension of GANs to low-resource NMT.
Towards Battery-Free Wireless Sensing via Radio-Frequency Energy Harvesting
Venue: arXiv cs.NI 2024
Year: 2024

Diverse Wi-Fi-based wireless applications have been proposed, ranging from daily activity recognition to vital sign monitoring. Despite their remarkable sensing accuracy, the high energy consumption and the requirement for customized hardware modification hinder the wide deployment of the existing sensing solutions. In this paper, we propose REHSense, an energy-efficient wireless sensing solution based on Radio-Frequency (RF) energy harvesting. Instead of relying on a power-hungry Wi-Fi receiver, REHSense leverages an RF energy harvester as the sensor and utilizes the voltage signals harvested from the ambient Wi-Fi signals to enable simultaneous context sensing and energy harvesting. We design and implement REHSense using a commercial-off-the-shelf (COTS) RF energy harvester. Extensive evaluation of three fine-grained wireless sensing tasks (i.e., respiration monitoring, human activity, and hand gesture recognition) shows that REHSense can achieve comparable sensing accuracy with conventional Wi-Fi-based solutions while adapting to different sensing environments, reducing the power consumption by 98.7% and harvesting up to 4.5mW of power from RF energy.
Classification of Safety Events at Nuclear Sites using Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

This paper proposes the development of a Large Language Model (LLM) based machine learning classifier designed to categorize Station Condition Records (SCRs) at nuclear power stations into safety-related and non-safety-related categories. The primary objective is to augment the existing manual review process by enhancing the efficiency and accuracy of the safety classification process at nuclear stations. The paper discusses experiments performed to classify a labeled SCR dataset and evaluates the performance of the classifier. It explores the construction of several prompt variations and their observed effects on the LLM's decision-making process. Additionally, it introduces a numerical scoring mechanism that could offer a more nuanced and flexible approach to SCR safety classification. This method represents an innovative step in nuclear safety management, providing a scalable tool for the identification of safety events.
A Hybrid Framework for Spatial Interpolation: Merging Data-driven with Domain Knowledge
Venue: arXiv cs.LG 2024
Year: 2024

Estimating spatially distributed information through the interpolation of scattered observation datasets often overlooks the critical role of domain knowledge in understanding spatial dependencies. Additionally, the features of these data sets are typically limited to the spatial coordinates of the scattered observation locations. In this paper, we propose a hybrid framework that integrates data-driven spatial dependency feature extraction with rule-assisted spatial dependency function mapping to augment domain knowledge. We demonstrate the superior performance of our framework in two comparative application scenarios, highlighting its ability to capture more localized spatial features in the reconstructed distribution fields. Furthermore, we underscore its potential to enhance nonlinear estimation capabilities through the application of transformed fuzzy rules and to quantify the inherent uncertainties associated with the observation data sets. Our framework introduces an innovative approach to spatial information estimation by synergistically combining observational data with rule-assisted domain knowledge.
Emerging Vulnerabilities in Frontier Models: Multi-Turn Jailbreak Attacks
Venue: arXiv cs.CR 2024
Year: 2024

Large language models (LLMs) are improving at an exceptional rate. However, these models are still susceptible to jailbreak attacks, which are becoming increasingly dangerous as models become increasingly powerful. In this work, we introduce a dataset of jailbreaks where each example can be input in both a single or a multi-turn format. We show that while equivalent in content, they are not equivalent in jailbreak success: defending against one structure does not guarantee defense against the other. Similarly, LLM-based filter guardrails also perform differently depending on not just the input content but the input structure. Thus, vulnerabilities of frontier models should be studied in both single and multi-turn settings; this dataset provides a tool to do so.
Learning-Based Finite Element Methods Modeling for Complex Mechanical Systems
Venue: arXiv cs.LG 2024
Year: 2024

Complex mechanic systems simulation is important in many real-world applications. The de-facto numeric solver using Finite Element Method (FEM) suffers from computationally intensive overhead. Though with many progress on the reduction of computational time and acceptable accuracy, the recent CNN or GNN-based simulation models still struggle to effectively represent complex mechanic simulation caused by the long-range spatial dependency of distance mesh nodes and independently learning local and global representation. In this paper, we propose a novel two-level mesh graph network. The key of the network is to interweave the developed Graph Block and Attention Block to better learn mechanic interactions even for long-rang spatial dependency. Evaluation on three synthetic and one real datasets demonstrates the superiority of our work. For example, on the Beam dataset, our work leads to 54.3\% lower prediction errors and 9.87\% fewer learnable network parameters.
Facilitating phenotyping from clinical texts: the medkit library
Venue: arXiv cs.CL 2024
Year: 2024

Phenotyping consists in applying algorithms to identify individuals associated with a specific, potentially complex, trait or condition, typically out of a collection of Electronic Health Records (EHRs). Because a lot of the clinical information of EHRs are lying in texts, phenotyping from text takes an important role in studies that rely on the secondary use of EHRs. However, the heterogeneity and highly specialized aspect of both the content and form of clinical texts makes this task particularly tedious, and is the source of time and cost constraints in observational studies. To facilitate the development, evaluation and reproductibility of phenotyping pipelines, we developed an open-source Python library named medkit. It enables composing data processing pipelines made of easy-to-reuse software bricks, named medkit operations. In addition to the core of the library, we share the operations and pipelines we already developed and invite the phenotyping community for their reuse and enrichment. medkit is available at https://github.com/medkit-lib/medkit
Text2Tradition: From Epistemological Tensions to AI-Mediated Cross-Cultural Co-Creation
Venue: arXiv cs.HC 2024
Year: 2024

This paper introduces Text2Tradition, a system designed to bridge the epistemological gap between modern language processing and traditional dance knowledge by translating user-generated prompts into Thai classical dance sequences. Our approach focuses on six traditional choreographic elements from No. 60 in Mae Bot Yai, a revered Thai dance repertoire, which embodies culturally specific knowledge passed down through generations. In contrast, large language models (LLMs) represent a different form of knowledge--data-driven, statistically derived, and often Western-centric. This research explores the potential of AI-mediated systems to connect traditional and contemporary art forms, highlighting the epistemological tensions and opportunities in cross-cultural translation.
Social MediARverse Investigating Users Social Media Content Sharing and Consuming Intentions with Location-Based AR
Venue: arXiv cs.HC 2024
Year: 2024

Augmented Reality (AR) is evolving to become the next frontier in social media, merging physical and virtual reality into a living metaverse, a Social MediARverse. With this transition, we must understand how different contexts (public, semi-public, and private) affect user engagement with AR content. We address this gap in current research by conducting an online survey with 110 participants, showcasing 36 AR videos, and polling them about the content's fit and appropriateness. Specifically, we manipulated these three spaces, two forms of dynamism (dynamic vs. static), and two dimensionalities (2D vs. 3D). Our findings reveal that dynamic AR content is generally more favorably received than static content. Additionally, users find sharing and engaging with AR content in private settings more comfortable than in others. By this, the study offers valuable insights for designing and implementing future Social MediARverses and guides industry and academia on content visualization and contextual considerations.
Unveiling Processing--Property Relationships in Laser Powder Bed Fusion: The Synergy of Machine Learning and High-throughput Experiments
Venue: arXiv cs.LG 2024
Year: 2024

Achieving desired mechanical properties in additive manufacturing requires many experiments and a well-defined design framework becomes crucial in reducing trials and conserving resources. Here, we propose a methodology embracing the synergy between high-throughput (HT) experimentation and hierarchical machine learning (ML) to unveil the complex relationships between a large set of process parameters in Laser Powder Bed Fusion (LPBF) and selected mechanical properties (tensile strength and ductility). The HT method envisions the fabrication of small samples for rapid automated hardness and porosity characterization, and a smaller set of tensile specimens for more labor-intensive direct measurement of yield strength and ductility. The ML approach is based on a sequential application of Gaussian processes (GPs) where the correlations between process parameters and hardness/porosity are first learnt and subsequently adopted by the GPs that relate strength and ductility to process parameters. Finally, an optimization scheme is devised that leverages these GPs to identify the processing parameters that maximize combinations of strength and ductility. By founding the learning on larger easy-to-collect and smaller labor-intensive data, we reduce the reliance on expensive characterization and enable exploration of a large processing space. Our approach is material-agnostic and herein we demonstrate its application on 17-4PH stainless steel.
MAPWise: Evaluating Vision-Language Models for Advanced Map Queries
Venue: arXiv cs.CV 2024
Year: 2024

Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing 1000 questions. Our benchmark incorporates 43 diverse question templates, requiring nuanced understanding of relative spatial relationships, intricate map features, and complex reasoning. It also includes maps with discrete and continuous values, encompassing variations in color-mapping, category ordering, and stylistic patterns, enabling comprehensive analysis. We evaluate the performance of multiple VLMs on this benchmark, highlighting gaps in their abilities and providing insights for improving such models.
AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning
Venue: arXiv cs.CV 2024
Year: 2024

All-Weather Image Restoration (AWIR) under adverse weather conditions is a challenging task due to the presence of different types of degradations. Prior research in this domain relies on extensive training data but lacks the utilization of additional contextual information for restoration guidance. Consequently, the performance of existing methods is limited by the degradation cues that are learnt from individual training samples. Recent advancements in visual in-context learning have introduced generalist models that are capable of addressing multiple computer vision tasks simultaneously by using the information present in the provided context as a prior. In this paper, we propose All-Weather Image Restoration using Visual In-Context Learning (AWRaCLe), a novel approach for AWIR that innovatively utilizes degradation-specific visual context information to steer the image restoration process. To achieve this, AWRaCLe incorporates Degradation Context Extraction (DCE) and Context Fusion (CF) to seamlessly integrate degradation-specific features from the context into an image restoration network. The proposed DCE and CF blocks leverage CLIP features and incorporate attention mechanisms to adeptly learn and fuse contextual information. These blocks are specifically designed for visual in-context learning under all-weather conditions and are crucial for effective context utilization. Through extensive experiments, we demonstrate the effectiveness of AWRaCLe for all-weather restoration and show that our method advances the state-of-the-art in AWIR.