OpenRaiser/Intern-Atlas huggingface dataset:
From Alignment to Entailment: A Unified Textual Entailment Framework for Entity Alignment
Venue: arXiv cs.CL 2023
Year: 2023

Entity Alignment (EA) aims to find the equivalent entities between two Knowledge Graphs (KGs). Existing methods usually encode the triples of entities as embeddings and learn to align the embeddings, which prevents the direct interaction between the original information of the cross-KG entities. Moreover, they encode the relational triples and attribute triples of an entity in heterogeneous embedding spaces, which prevents them from helping each other. In this paper, we transform both triples into unified textual sequences, and model the EA task as a bi-directional textual entailment task between the sequences of cross-KG entities. Specifically, we feed the sequences of two entities simultaneously into a pre-trained language model (PLM) and propose two kinds of PLM-based entity aligners that model the entailment probability between sequences as the similarity between entities. Our approach captures the unified correlation pattern of two kinds of information between entities, and explicitly models the fine-grained interaction between original entity information. The experiments on five cross-lingual EA datasets show that our approach outperforms the state-of-the-art EA methods and enables the mutual enhancement of the heterogeneous information. Codes are available at https://github.com/OreOZhao/TEA.
When SAM Meets Shadow Detection
Venue: arXiv cs.CV 2023
Year: 2023

As a promptable generic object segmentation model, segment anything model (SAM) has recently attracted significant attention, and also demonstrates its powerful performance. Nevertheless, it still meets its Waterloo when encountering several tasks, e.g., medical image segmentation, camouflaged object detection, etc. In this report, we try SAM on an unexplored popular task: shadow detection. Specifically, four benchmarks were chosen and evaluated with widely used metrics. The experimental results show that the performance for shadow detection using SAM is not satisfactory, especially when comparing with the elaborate models. Code is available at https://github.com/LeipingJie/SAMSh.
Contextualized Word Vector-based Methods for Discovering Semantic Differences with No Training nor Word Alignment
Venue: arXiv cs.CL 2023
Year: 2023

In this paper, we propose methods for discovering semantic differences in words appearing in two corpora based on the norms of contextualized word vectors. The key idea is that the coverage of meanings is reflected in the norm of its mean word vector. The proposed methods do not require the assumptions concerning words and corpora for comparison that the previous methods do. All they require are to compute the mean vector of contextualized word vectors and its norm for each word type. Nevertheless, they are (i) robust for the skew in corpus size; (ii) capable of detecting semantic differences in infrequent words; and (iii) effective in pinpointing word instances that have a meaning missing in one of the two corpora for comparison. We show these advantages for native and non-native English corpora and also for historical corpora.
RECIPE: How to Integrate ChatGPT into EFL Writing Education
Venue: arXiv cs.HC 2023
Year: 2023

The integration of generative AI in the field of education is actively being explored. In particular, ChatGPT has garnered significant interest, offering an opportunity to examine its effectiveness in English as a foreign language (EFL) education. To address this need, we present a novel learning platform called RECIPE (Revising an Essay with ChatGPT on an Interactive Platform for EFL learners). Our platform features two types of prompts that facilitate conversations between ChatGPT and students: (1) a hidden prompt for ChatGPT to take an EFL teacher role and (2) an open prompt for students to initiate a dialogue with a self-written summary of what they have learned. We deployed this platform for 213 undergraduate and graduate students enrolled in EFL writing courses and seven instructors. For this study, we collect students' interaction data from RECIPE, including students' perceptions and usage of the platform, and user scenarios are examined with the data. We also conduct a focus group interview with six students and an individual interview with one EFL instructor to explore design opportunities for leveraging generative AI models in the field of EFL education.
MIDI-Draw: Sketching to Control Melody Generation
Venue: arXiv cs.SD 2023
Year: 2023

We describe a proof-of-principle implementation of a system for drawing melodies that abstracts away from a note-level input representation via melodic contours. The aim is to allow users to express their musical intentions without requiring prior knowledge of how notes fit together melodiously. Current approaches to controllable melody generation often require users to choose parameters that are static across a whole sequence, via buttons or sliders. In contrast, our method allows users to quickly specify how parameters should change over time by drawing a contour.
Designing architectured ceramics for transient thermal applications using finite element and deep learning
Venue: arXiv cs.CE 2023
Year: 2023

Topologically interlocking architectures can generate tough ceramics with attractive thermo-mechanical properties. This concept can make the material design pathway a challenging task, since modeling the whole design space is neither effective nor feasible. We propose an approach to design high-performance architectured ceramics using machine learning (ML) with data from finite element analysis (FEA). Convolutional neural networks (CNNs) and Multilayer Perceptrons (MLPs) are used as the deep learning approaches. A limited set of FEA simulation data containing a variety of architectural design parameters is used to train our neural networks, including learning how independent and dependent design parameters are related. A trained network is then used to predict the optimum structure from the configurations. A FEA simulation is run on the best predictions of both MLP and CNN algorithms to evaluate the performance of our networks. Although a limited amount of simulation data are available, our networks are effective in predicting the transient thermo-mechanical responses of possible panel designs. For example, the optimal design after using CNN prediction resulted in $\approx \! 30\%$ improvement in terms of edge temperature.
Channel Cycle Time: A New Measure of Short-term Fairness
Venue: arXiv cs.IT 2023
Year: 2023

This paper puts forth a new metric, dubbed channel cycle time (CCT), to measure the short-term fairness of communication networks. CCT characterizes the average duration between two consecutive successful transmissions of a user, during which all other users successfully accessed the channel at least once. In contrast to existing short-term fairness measures, CCT provides more comprehensive insight into the transient dynamics of communication networks, with a particular focus on users' delays and jitter. To validate the efficacy of our approach, we analytically characterize the CCTs for two classical communication protocols: slotted Aloha and CSMA/CA. The analysis demonstrates that CSMA/CA exhibits superior short-term fairness over slotted Aloha. Beyond its role as a measurement metric, CCT has broader implications as a guiding principle for the design of future communication networks by emphasizing factors like fairness, delay, and jitter in short-term behaviors.
V2X-Boosted Federated Learning for Cooperative Intelligent Transportation Systems with Contextual Client Selection
Venue: arXiv cs.LG 2023
Year: 2023

Machine learning (ML) has revolutionized transportation systems, enabling autonomous driving and smart traffic services. Federated learning (FL) overcomes privacy constraints by training ML models in distributed systems, exchanging model parameters instead of raw data. However, the dynamic states of connected vehicles affect the network connection quality and influence the FL performance. To tackle this challenge, we propose a contextual client selection pipeline that uses Vehicle-to-Everything (V2X) messages to select clients based on the predicted communication latency. The pipeline includes: (i) fusing V2X messages, (ii) predicting future traffic topology, (iii) pre-clustering clients based on local data distribution similarity, and (iv) selecting clients with minimal latency for future model aggregation. Experiments show that our pipeline outperforms baselines on various datasets, particularly in non-iid settings.
ViDaS Video Depth-aware Saliency Network
Venue: arXiv cs.CV 2023
Year: 2023

We introduce ViDaS, a two-stream, fully convolutional Video, Depth-Aware Saliency network to address the problem of attention modeling ``in-the-wild", via saliency prediction in videos. Contrary to existing visual saliency approaches using only RGB frames as input, our network employs also depth as an additional modality. The network consists of two visual streams, one for the RGB frames, and one for the depth frames. Both streams follow an encoder-decoder approach and are fused to obtain a final saliency map. The network is trained end-to-end and is evaluated in a variety of different databases with eye-tracking data, containing a wide range of video content. Although the publicly available datasets do not contain depth, we estimate it using three different state-of-the-art methods, to enable comparisons and a deeper insight. Our method outperforms in most cases state-of-the-art models and our RGB-only variant, which indicates that depth can be beneficial to accurately estimating saliency in videos displayed on a 2D screen. Depth has been widely used to assist salient object detection problems, where it has been proven to be very beneficial. Our problem though differs significantly from salient object detection, since it is not restricted to specific salient objects, but predicts human attention in a more general aspect. These two problems not only have different objectives, but also different ground truth data and evaluation metrics. To our best knowledge, this is the first competitive deep learning video saliency estimation approach that combines both RGB and Depth features to address the general problem of saliency estimation ``in-the-wild". The code will be publicly released.
Distortion Under Public-Spirited Voting
Venue: arXiv cs.GT 2023
Year: 2023

A key promise of democratic voting is that, by accounting for all constituents' preferences, it produces decisions that benefit the constituency overall. It is alarming, then, that all deterministic voting rules have unbounded distortion: all such rules - even under reasonable conditions - will sometimes select outcomes that yield essentially no value for constituents. In this paper, we show that this problem is mitigated by voters being public-spirited: that is, when deciding how to rank alternatives, voters weigh the common good in addition to their own interests. We first generalize the standard voting model to capture this public-spirited voting behavior. In this model, we show that public-spirited voting can substantially - and in some senses, monotonically - reduce the distortion of several voting rules. Notably, these results include the finding that if voters are at all public-spirited, some voting rules have constant distortion in the number of alternatives. Further, we demonstrate that these benefits are robust to adversarial conditions likely to exist in practice. Taken together, our results suggest an implementable approach to improving the welfare outcomes of elections: democratic deliberation, an already-mainstream practice that is believed to increase voters' public spirit.
Marginalized Beam Search Algorithms for Hierarchical HMMs
Venue: arXiv cs.LG 2023
Year: 2023

Inferring a state sequence from a sequence of measurements is a fundamental problem in bioinformatics and natural language processing. The Viterbi and the Beam Search (BS) algorithms are popular inference methods, but they have limitations when applied to Hierarchical Hidden Markov Models (HHMMs), where the interest lies in the outer state sequence. The Viterbi algorithm can not infer outer states without inner states, while the BS algorithm requires marginalization over prohibitively large state spaces. We propose two new algorithms to overcome these limitations: the greedy marginalized BS algorithm and the local focus BS algorithm. We show that they approximate the most likely outer state sequence with higher performance than the Viterbi algorithm, and we evaluate the performance of these algorithms on an explicit duration HMM with simulation and nanopore base calling data.
Cross-Lingual Supervision improves Large Language Models Pre-training
Venue: arXiv cs.CL 2023
Year: 2023

The recent rapid progress in pre-training Large Language Models has relied on using self-supervised language modeling objectives like next token prediction or span corruption. On the other hand, Machine Translation Systems are mostly trained using cross-lingual supervision that requires aligned data between source and target languages. We demonstrate that pre-training Large Language Models on a mixture of a self-supervised Language Modeling objective and the supervised Machine Translation objective, therefore including cross-lingual parallel data during pre-training, yields models with better in-context learning abilities. As pre-training is a very resource-intensive process and a grid search on the best mixing ratio between the two objectives is prohibitively expensive, we propose a simple yet effective strategy to learn it during pre-training.
Monte-Carlo Search for an Equilibrium in Dec-POMDPs
Venue: arXiv cs.AI 2023
Year: 2023

Decentralized partially observable Markov decision processes (Dec-POMDPs) formalize the problem of designing individual controllers for a group of collaborative agents under stochastic dynamics and partial observability. Seeking a global optimum is difficult (NEXP complete), but seeking a Nash equilibrium -- each agent policy being a best response to the other agents -- is more accessible, and allowed addressing infinite-horizon problems with solutions in the form of finite state controllers. In this paper, we show that this approach can be adapted to cases where only a generative model (a simulator) of the Dec-POMDP is available. This requires relying on a simulation-based POMDP solver to construct an agent's FSC node by node. A related process is used to heuristically derive initial FSCs. Experiment with benchmarks shows that MC-JESP is competitive with exisiting Dec-POMDP solvers, even better than many offline methods using explicit models.
MaGIC: Multi-modality Guided Image Completion
Venue: arXiv cs.CV 2023
Year: 2023

Vanilla image completion approaches exhibit sensitivity to large missing regions, attributed to the limited availability of reference information for plausible generation. To mitigate this, existing methods incorporate the extra cue as a guidance for image completion. Despite improvements, these approaches are often restricted to employing a single modality (e.g., segmentation or sketch maps), which lacks scalability in leveraging multi-modality for more plausible completion. In this paper, we propose a novel, simple yet effective method for Multi-modal Guided Image Completion, dubbed MaGIC, which not only supports a wide range of single modality as the guidance (e.g., text, canny edge, sketch, segmentation, depth, and pose), but also adapts to arbitrarily customized combination of these modalities (i.e., arbitrary multi-modality) for image completion. For building MaGIC, we first introduce a modality-specific conditional U-Net (MCU-Net) that injects single-modal signal into a U-Net denoiser for single-modal guided image completion. Then, we devise a consistent modality blending (CMB) method to leverage modality signals encoded in multiple learned MCU-Nets through gradient guidance in latent space. Our CMB is training-free, thereby avoids the cumbersome joint re-training of different modalities, which is the secret of MaGIC to achieve exceptional flexibility in accommodating new modalities for completion. Experiments show the superiority of MaGIC over state-of-the-art methods and its generalization to various completion tasks. Our project with code and models is available at yeates.github.io/MaGIC-Page/.
AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
Venue: arXiv cs.CY 2023
Year: 2023

This paper presents a community-centered study of cultural limitations of text-to-image (T2I) models in the South Asian context. We theorize these failures using scholarship on dominant media regimes of representations and locate them within participants' reporting of their existing social marginalizations. We thus show how generative AI can reproduce an outsiders gaze for viewing South Asian cultures, shaped by global and regional power inequities. By centering communities as experts and soliciting their perspectives on T2I limitations, our study adds rich nuance into existing evaluative frameworks and deepens our understanding of the culturally-specific ways AI technologies can fail in non-Western and Global South settings. We distill lessons for responsible development of T2I models, recommending concrete pathways forward that can allow for recognition of structural inequalities.
Predictive Wand: a mathematical interface design for operations with delays
Venue: arXiv cs.HC 2023
Year: 2023

Action-feedback delay during operation reduces both task performance and sense of agency (SoA). In this study, using information-theoretic free energy, we formalized a novel mathematical model for explaining the influence of delay on both task performance and SoA in continuous operations. Based on the mathematical model, we propose a novel interface design called Predictive Wand for predicting future outcomes to prevent task performance and SoA degradation resulting from response delays. Model-based simulations and operational experiments with participants confirmed that operational delay considerably reduces both task performance and SoA. Furthermore, the proposed Predictive Wand mitigates these problems. Our findings support the model-based interface design for continuous operations with delay to prevent task performance and SoA degradation.
User Experience Considered Harmful (for the Planet)
Venue: arXiv cs.HC 2023
Year: 2023

Great user experience is killing us (more or less)! My argument in this provocation is that the excessive focus on user experience (UX) by the tech industry and academic community has a negative impact on the sustainability of ICT devices. I will argue based on two examples, that we need new metrics or extend current UX metrics to also include third order effects and sustainability perspectives. Lastly, I would like us - the (Sustainable) HCI community - to increase our focus on solving the problems that result from our very own creations.
Hyper-automation-The next peripheral for automation in IT industries
Venue: arXiv cs.HC 2023
Year: 2023

The extension of legacy business process automation beyond the bounds of specific processes is known as hyperautomation. Hyperautomation provides automation for nearly any repetitive action performed by business users by combining AI tools with RPA. It automates complex IT business processes that a company's top brains might not be able to complete. This is an end-to-end automation of a standard business process deployment. It enables automation to perform task digitalization by combining a brain computer interface (BCI) with AI and RPA automation tools. BCI, in conjunction with automation tools, will advance the detection and generation of automation processes to the next level. It allows enterprises to combine business intelligence systems, address complex requirements, and enhance human expertise and automation experience. Hyperautomation and its importance in today's environment are briefly discussed in this paper. The article then goes on to discuss how BCI and sensors might aid Hyperautomation. The specific sectors of solicitations were examined using a variety of flexible technologies associated to this concept, as well as dedicated workflow techniques, which are also diagrammatically illustrated. Hyperautomation is being utilized to improve the efficiency, accuracy, and human enhancement of automated tasks dramatically. It incorporates a number of automated tools in its discovery, implementation, and automation phases. As a result, it's well-suited to integrating cutting-edge technologies and experimenting with new methods of working. Keywords- Hyperautomation, Brain computer Interface (BCI), Technology, Used case, Sensors, Industries.
Sequential Best-Arm Identification with Application to Brain-Computer Interface
Venue: arXiv cs.HC 2023
Year: 2023

A brain-computer interface (BCI) is a technology that enables direct communication between the brain and an external device or computer system. It allows individuals to interact with the device using only their thoughts, and holds immense potential for a wide range of applications in medicine, rehabilitation, and human augmentation. An electroencephalogram (EEG) and event-related potential (ERP)-based speller system is a type of BCI that allows users to spell words without using a physical keyboard, but instead by recording and interpreting brain signals under different stimulus presentation paradigms. Conventional non-adaptive paradigms treat each word selection independently, leading to a lengthy learning process. To improve the sampling efficiency, we cast the problem as a sequence of best-arm identification tasks in multi-armed bandits. Leveraging pre-trained large language models (LLMs), we utilize the prior knowledge learned from previous tasks to inform and facilitate subsequent tasks. To do so in a coherent way, we propose a sequential top-two Thompson sampling (STTS) algorithm under the fixed-confidence setting and the fixed-budget setting. We study the theoretical property of the proposed algorithm, and demonstrate its substantial empirical improvement through both synthetic data analysis as well as a P300 BCI speller simulator example.
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
Venue: arXiv cs.CL 2023
Year: 2023

Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) -- languages for which NLP re-search is particularly far behind in meeting user needs -- it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text-only, multi-modal (vision, audio, and text),supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models