OpenRaiser/Intern-Atlas huggingface dataset:
iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

Most available data is unstructured, making it challenging to access valuable information. Automatically building Knowledge Graphs (KGs) is crucial for structuring data and making it accessible, allowing users to search for information effectively. KGs also facilitate insights, inference, and reasoning. Traditional NLP methods, such as named entity recognition and relation extraction, are key in information retrieval but face limitations, including the use of predefined entity types and the need for supervised learning. Current research leverages large language models' capabilities, such as zero- or few-shot learning. However, unresolved and semantically duplicated entities and relations still pose challenges, leading to inconsistent graphs and requiring extensive post-processing. Additionally, most approaches are topic-dependent. In this paper, we propose iText2KG, a method for incremental, topic-independent KG construction without post-processing. This plug-and-play, zero-shot method is applicable across a wide range of KG construction scenarios and comprises four modules: Document Distiller, Incremental Entity Extractor, Incremental Relation Extractor, and Graph Integrator and Visualization. Our method demonstrates superior performance compared to baseline methods across three scenarios: converting scientific papers to graphs, websites to graphs, and CVs to graphs.
Bringing the RT-1-X Foundation Model to a SCARA robot
Venue: arXiv cs.RO 2024
Year: 2024

Traditional robotic systems require specific training data for each task, environment, and robot form. While recent advancements in machine learning have enabled models to generalize across new tasks and environments, the challenge of adapting these models to entirely new settings remains largely unexplored. This study addresses this by investigating the generalization capabilities of the RT-1-X robotic foundation model to a type of robot unseen during its training: a SCARA robot from UMI-RTX. Initial experiments reveal that RT-1-X does not generalize zero-shot to the unseen type of robot. However, fine-tuning of the RT-1-X model by demonstration allows the robot to learn a pickup task which was part of the foundation model (but learned for another type of robot). When the robot is presented with an object that is included in the foundation model but not in the fine-tuning dataset, it demonstrates that only the skill, but not the object-specific knowledge, has been transferred.
AI data transparency: an exploration through the lens of AI incidents
Venue: arXiv cs.CY 2024
Year: 2024

Knowing more about the data used to build AI systems is critical for allowing different stakeholders to play their part in ensuring responsible and appropriate deployment and use. Meanwhile, a 2023 report shows that data transparency lags significantly behind other areas of AI transparency in popular foundation models. In this research, we sought to build on these findings, exploring the status of public documentation about data practices within AI systems generating public concern. Our findings demonstrate that low data transparency persists across a wide range of systems, and further that issues of transparency and explainability at model- and system- level create barriers for investigating data transparency information to address public concerns about AI systems. We highlight a need to develop systematic ways of monitoring AI data transparency that account for the diversity of AI system types, and for such efforts to build on further understanding of the needs of those both supplying and using data transparency information.
Estimating Indoor Scene Depth Maps from Ultrasonic Echoes
Venue: arXiv cs.SD 2024
Year: 2024

Measuring 3D geometric structures of indoor scenes requires dedicated depth sensors, which are not always available. Echo-based depth estimation has recently been studied as a promising alternative solution. All previous studies have assumed the use of echoes in the audible range. However, one major problem is that audible echoes cannot be used in quiet spaces or other situations where producing audible sounds is prohibited. In this paper, we consider echo-based depth estimation using inaudible ultrasonic echoes. While ultrasonic waves provide high measurement accuracy in theory, the actual depth estimation accuracy when ultrasonic echoes are used has remained unclear, due to its disadvantage of being sensitive to noise and susceptible to attenuation. We first investigate the depth estimation accuracy when the frequency of the sound source is restricted to the high-frequency band, and found that the accuracy decreased when the frequency was limited to ultrasonic ranges. Based on this observation, we propose a novel deep learning method to improve the accuracy of ultrasonic echo-based depth estimation by using audible echoes as auxiliary data only during training. Experimental results with a public dataset demonstrate that our method improves the estimation accuracy.
Identification of non-causal systems with arbitrary switching modes
Venue: arXiv cs.IT 2024
Year: 2024

We consider the identification of non-causal systems with arbitrary switching modes (NCS-ASM), a class of models essential for describing typical power load management and department store inventory dynamics. The simultaneous identification of causal-and-anticausal subsystems, along with the presence of possibly random switching sequences, however, make the overall identification problem particularly challenging. To this end, we develop an expectation-maximization (EM) based system identification technique, where the E-step proposes a modified Kalman filter (KF) to estimate the states and switching sequences of causal-and-anticausal subsystems, while the M-step consists in a switching least-squares algorithm to estimate the parameters of individual subsystems. We establish the main convergence features of the proposed identification procedure, also providing bounds on the parameter estimation errors under mild conditions. Finally, the effectiveness of our identification method is validated through two numerical simulations.
Journalists are most likely to receive abuse: Analysing online abuse of UK public figures across sport, politics, and journalism on Twitter
Venue: arXiv cs.CY 2024
Year: 2024

Engaging with online social media platforms is an important part of life as a public figure in modern society, enabling connection with broad audiences and providing a platform for spreading ideas. However, public figures are often disproportionate recipients of hate and abuse on these platforms, degrading public discourse. While significant research on abuse received by groups such as politicians and journalists exists, little has been done to understand the differences in the dynamics of abuse across different groups of public figures, systematically and at scale. To address this, we present analysis of a novel dataset of 45.5M tweets targeted at 4,602 UK public figures across 3 domains (members of parliament, footballers, journalists), labelled using fine-tuned transformer-based language models. We find that MPs receive more abuse in absolute terms, but that journalists are most likely to receive abuse after controlling for other factors. We show that abuse is unevenly distributed in all groups, with a small number of individuals receiving the majority of abuse, and that for some groups, abuse is more temporally uneven, being driven by specific events, particularly for footballers. We also find that a more prominent online presence and being male are indicative of higher levels of abuse across all 3 domains.
Game On: Towards Language Models as RL Experimenters
Venue: arXiv cs.AI 2024
Year: 2024

We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents. To do so, it leverages a VLM to perform some of the capabilities normally required of a human experimenter, including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute - enabling our system to build automated curricula for learning. We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning. We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation. For this, we use a standard Gemini model, without additional fine-tuning, to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm, in order to steer data collection so as to aid learning new skills. Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain. Additional examination of the ability of the system to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents.
Continuous risk assessment in secure DevOps
Venue: arXiv cs.SE 2024
Year: 2024

DevOps (development and operations), has significantly changed the way to overcome deficiencies for delivering high-quality software to production environments. Past years witnessed an increased interest in embedding DevOps with cybersecurity in an approach dubbed secure DevOps. However, as the practices and guidance mature, teams must consider them within a broader risk context. We argue here how secure DevOps could profit from engaging with risk related activities within organisations. We focus on combining Risk Assessment (RA), particularly Threat Modelling (TM) and apply security considerations early in the software life-cycle. Our contribution provides a roadmap for enacting secure DevOps alongside risk objectives, devising informed ways to improve TM and establishing effective security underpinnings in organisations focusing on software products and services. We aim to outline proven methods over the literature on the subject discussing case studies, technologies, and tools. It presents a case study for a real-world inspired organisation employing the proposed approach with a discussion. Enforcing these novel mechanisms centred on security requires investment, training, and stakeholder engagement. It requires understanding the actual benefits of automation in light of Continuous Integration/Continuous Delivery settings that improve the overall quality of software solutions reaching the market.
Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
Venue: arXiv cs.CL 2024
Year: 2024

The advancement of Large Language Models (LLMs) for domain applications in fields such as materials science and engineering depends on the development of fine-tuning strategies that adapt models for specialized, technical capabilities. In this work, we explore the effects of Continued Pretraining (CPT), Supervised Fine-Tuning (SFT), and various preference-based optimization approaches, including Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO), on fine-tuned LLM performance. Our analysis shows how these strategies influence model outcomes and reveals that the merging of multiple fine-tuned models can lead to the emergence of capabilities that surpass the individual contributions of the parent models. We find that model merging leads to new functionalities that neither parent model could achieve alone, leading to improved performance in domain-specific assessments. Experiments with different model architectures are presented, including Llama 3.1 8B and Mistral 7B models, where similar behaviors are observed. Exploring whether the results hold also for much smaller models, we use a tiny LLM with 1.7 billion parameters and show that very small LLMs do not necessarily feature emergent capabilities under model merging, suggesting that model scaling may be a key component. In open-ended yet consistent chat conversations between a human and AI models, our assessment reveals detailed insights into how different model variants perform and show that the smallest model achieves a high intelligence score across key criteria including reasoning depth, creativity, clarity, and quantitative precision. Other experiments include the development of image generation prompts based on disparate biological material design concepts, to create new microstructures, architectural concepts, and urban design based on biological materials-inspired construction principles.
MOBIUS: Towards the Next Generation of Query-Ad Matching in Baidu's Sponsored Search
Venue: arXiv cs.IR 2024
Year: 2024

Baidu runs the largest commercial web search engine in China, serving hundreds of millions of online users every day in response to a great variety of queries. In order to build a high-efficiency sponsored search engine, we used to adopt a three-layer funnel-shaped structure to screen and sort hundreds of ads from billions of ad candidates subject to the requirement of low response latency and the restraints of computing resources. Given a user query, the top matching layer is responsible for providing semantically relevant ad candidates to the next layer, while the ranking layer at the bottom concerns more about business indicators (e.g., CPM, ROI, etc.) of those ads. The clear separation between the matching and ranking objectives results in a lower commercial return. The Mobius project has been established to address this serious issue. It is our first attempt to train the matching layer to consider CPM as an additional optimization objective besides the query-ad relevance, via directly predicting CTR (click-through rate) from billions of query-ad pairs. Specifically, this paper will elaborate on how we adopt active learning to overcome the insufficiency of click history at the matching layer when training our neural click networks offline, and how we use the SOTA ANN search technique for retrieving ads more efficiently (Here ``ANN'' stands for approximate nearest neighbor search). We contribute the solutions to Mobius-V1 as the first version of our next generation query-ad matching system.
Tyche: Collateral-Free Coalition-Resistant Multiparty Lotteries with Arbitrary Payouts
Venue: arXiv cs.CR 2024
Year: 2024

We propose Tyche, a family of protocols for performing practically (as well as asymptotically) efficient multiparty lotteries, resistant against aborts and majority coalitions. Our protocols are based on a commit-and-reveal approach, requiring only a collision-resistant hash function. All our protocols use a blockchain as a public bulletin board and for buy-in collection and payout settlement. Importantly though, they do not rely on it or any other third party for providing randomness. Also, participants are not required to post any collateral beyond their buy-in. Any honest participant can eventually settle the lottery, and dishonest behavior never reduces the winning probability of any honest participant. Further, we adapt all three protocols into anonymous lotteries, where (under certain conditions) the winner is unlinkable to any particular participant. We show that our protocols are secure, fair, and some preserve the participants' privacy. Finally, we evaluate the performance of our protocols, particularly in terms of transaction fees, by implementing them on the Sui blockchain. There we see that per user transaction fees are reasonably low and our protocols could potentially support millions of participants.
Hemispherical Antenna Array Architecture for High-Altitude Platform Stations (HAPS) for Uniform Capacity Provision
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we present a novel hemispherical antenna array (HAA) designed for high-altitude platform stations (HAPS). A significant limitation of traditional rectangular antenna arrays for HAPS is that their antenna elements are oriented downward, resulting in low gains for distant users. Cylindrical antenna arrays were introduced to mitigate this drawback; however, their antenna elements face the horizon leading to suboptimal gains for users located beneath the HAPS. To address these challenges, in this study, we introduce our HAA. An HAA's antenna elements are strategically distributed across the surface of a hemisphere to ensure that each user is directly aligned with specific antenna elements. To maximize users minimum signal-to-interference-plus-noise ratio (SINR), we formulate an optimization problem. After performing analog beamforming, we introduce an antenna selection algorithm and show that this method achieves optimality when a substantial number of antenna elements are selected for each user. Additionally, we employ the bisection method to determine the optimal power allocation for each user. Our simulation results convincingly demonstrate that the proposed HAA outperforms the conventional arrays, and provides uniform rates across the entire coverage area. With a $20~\mathrm{MHz}$ communication bandwidth, and a $50~\mathrm{dBm}$ total power, the proposed approach reaches sum rates of $14~\mathrm{Gbps}$.
Submodularity of Mutual Information for Multivariate Gaussian Sources with Additive Noise
Venue: arXiv cs.IT 2024
Year: 2024

Sensor placement approaches in networks often involve using information-theoretic measures such as entropy and mutual information. We prove that mutual information abides by submodularity and is non-decreasing when considering the mutual information between the states of the network and a subset of $k$ nodes subjected to additive white Gaussian noise. We prove this under the assumption that the states follow a non-degenerate multivariate Gaussian distribution.
Enabling Practical and Privacy-Preserving Image Processing
Venue: arXiv cs.CR 2024
Year: 2024

Fully Homomorphic Encryption (FHE) enables computations on encrypted data, preserving confidentiality without the need for decryption. However, FHE is often hindered by significant performance overhead, particularly for high-precision and complex data like images. Due to serious efficiency issues, traditional FHE methods often encrypt images by monolithic data blocks (such as pixel rows), instead of pixels. However, this strategy compromises the advantages of homomorphic operations and disables pixel-level image processing. In this study, we address these challenges by proposing and implementing a pixel-level homomorphic encryption approach, iCHEETAH, based on the CKKS scheme. To enhance computational efficiency, we introduce three novel caching mechanisms to pre-encrypt radix values or frequently occurring pixel values, substantially reducing redundant encryption operations. Extensive experiments demonstrate that our approach achieves up to a 19-fold improvement in encryption speed compared to the original CKKS, while maintaining high image quality. Additionally, real-world image applications such as mean filtering, brightness enhancement, image matching and watermarking are tested based on FHE, showcasing up to a 91.53% speed improvement. We also proved that our method is IND-CPA (Indistinguishability under Chosen Plaintext Attack) secure, providing strong encryption security. These results underscore the practicality and efficiency of iCHEETAH, marking a significant advancement in privacy-preserving image processing at scale.
Reimagining Data Visualization to Address Sustainability Goals
Venue: arXiv cs.HC 2024
Year: 2024

Information visualization holds significant potential to support sustainability goals such as environmental stewardship, and climate resilience by transforming complex data into accessible visual formats that enhance public understanding of complex climate change data and drive actionable insights. While the field has predominantly focused on analytical orientation of visualization, challenging traditional visualization techniques and goals, through critical visualization research expands existing assumptions and conventions in the field. In this paper, I explore how reimagining overlooked aspects of data visualization, such as engagement, emotional resonance, communication, and community empowerment, can contribute to achieving sustainability objectives. I argue that by focusing on inclusive data visualization that promotes clarity, understandability, and public participation, we can make complex data more relatable and actionable, fostering broader connections and mobilizing collective action on critical issues like climate change. Moreover, I discuss the role of emotional receptivity in environmental data communication, stressing the need for visualizations that respect diverse cultural perspectives and emotional responses to achieve impactful outcomes. Drawing on insights from a decade of research in public participation and community engagement, I aim to highlight how data visualization can democratize data access and increase public involvement in order to contribute to a more sustainable and resilient future.
Limited but consistent gains in adversarial robustness by co-training object recognition models with human EEG
Venue: arXiv cs.LG 2024
Year: 2024

In contrast to human vision, artificial neural networks (ANNs) remain relatively susceptible to adversarial attacks. To address this vulnerability, efforts have been made to transfer inductive bias from human brains to ANNs, often by training the ANN representations to match their biological counterparts. Previous works relied on brain data acquired in rodents or primates using invasive techniques, from specific regions of the brain, under non-natural conditions (anesthetized animals), and with stimulus datasets lacking diversity and naturalness. In this work, we explored whether aligning model representations to human EEG responses to a rich set of real-world images increases robustness to ANNs. Specifically, we trained ResNet50-backbone models on a dual task of classification and EEG prediction; and evaluated their EEG prediction accuracy and robustness to adversarial attacks. We observed significant correlation between the networks' EEG prediction accuracy, often highest around 100 ms post stimulus onset, and their gains in adversarial robustness. Although effect size was limited, effects were consistent across different random initializations and robust for architectural variants. We further teased apart the data from individual EEG channels and observed strongest contribution from electrodes in the parieto-occipital regions. The demonstrated utility of human EEG for such tasks opens up avenues for future efforts that scale to larger datasets under diverse stimuli conditions with the promise of stronger effects.
A Fused Large Language Model for Predicting Startup Success
Venue: arXiv cs.LG 2024
Year: 2024

Investors are continuously seeking profitable investment opportunities in startups and, hence, for effective decision-making, need to predict a startup's probability of success. Nowadays, investors can use not only various fundamental information about a startup (e.g., the age of the startup, the number of founders, and the business sector) but also textual description of a startup's innovation and business model, which is widely available through online venture capital (VC) platforms such as Crunchbase. To support the decision-making of investors, we develop a machine learning approach with the aim of locating successful startups on VC platforms. Specifically, we develop, train, and evaluate a tailored, fused large language model to predict startup success. Thereby, we assess to what extent self-descriptions on VC platforms are predictive of startup success. Using 20,172 online profiles from Crunchbase, we find that our fused large language model can predict startup success, with textual self-descriptions being responsible for a significant part of the predictive power. Our work provides a decision support tool for investors to find profitable investment opportunities.