OpenRaiser/Intern-Atlas huggingface dataset:
Efficiently Learning the Graph for Semi-supervised Learning
Venue: arXiv cs.LG 2023
Year: 2023

Computational efficiency is a major bottleneck in using classic graph-based approaches for semi-supervised learning on datasets with a large number of unlabeled examples. Known techniques to improve efficiency typically involve an approximation of the graph regularization objective, but suffer two major drawbacks - first the graph is assumed to be known or constructed with heuristic hyperparameter values, second they do not provide a principled approximation guarantee for learning over the full unlabeled dataset. Building on recent work on learning graphs for semi-supervised learning from multiple datasets for problems from the same domain, and leveraging techniques for fast approximations for solving linear systems in the graph Laplacian matrix, we propose algorithms that overcome both the above limitations. We show a formal separation in the learning-theoretic complexity of sparse and dense graph families. We further show how to approximately learn the best graphs from the sparse families efficiently using the conjugate gradient method. Our approach can also be used to learn the graph efficiently online with sub-linear regret, under mild smoothness assumptions. Our online learning results are stated generally, and may be useful for approximate and efficient parameter tuning in other problems. We implement our approach and demonstrate significant ($\sim$10-100x) speedups over prior work on semi-supervised learning with learned graphs on benchmark datasets.
Strategic Communication and Deliberation on Climate Change of different Actor Groups using Twitter
Venue: arXiv cs.SI 2023
Year: 2023

Strategic communication in Twitter is compared between different actor groups with regard to the topic of climate change. The main hypothesis is that different actor groups will be more or less central in the reply-trees depending on their strategic interests based on their profession or organizational affiliation.
Prompt-based Extraction of Social Determinants of Health Using Few-shot Learning
Venue: arXiv cs.CL 2023
Year: 2023

Social determinants of health (SDOH) documented in the electronic health record through unstructured text are increasingly being studied to understand how SDOH impacts patient health outcomes. In this work, we utilize the Social History Annotation Corpus (SHAC), a multi-institutional corpus of de-identified social history sections annotated for SDOH, including substance use, employment, and living status information. We explore the automatic extraction of SDOH information with SHAC in both standoff and inline annotation formats using GPT-4 in a one-shot prompting setting. We compare GPT-4 extraction performance with a high-performing supervised approach and perform thorough error analyses. Our prompt-based GPT-4 method achieved an overall 0.652 F1 on the SHAC test set, similar to the 7th best-performing system among all teams in the n2c2 challenge with SHAC.
The Effect of Masking Strategies on Knowledge Retention by Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Language models retain a significant amount of world knowledge from their pre-training stage. This allows knowledgeable models to be applied to knowledge-intensive tasks prevalent in information retrieval, such as ranking or question answering. Understanding how and which factual information is acquired by our models is necessary to build responsible models. However, limited work has been done to understand the effect of pre-training tasks on the amount of knowledge captured and forgotten by language models during pre-training. Building a better understanding of knowledge acquisition is the goal of this paper. Therefore, we utilize a selection of pre-training tasks to infuse knowledge into our model. In the following steps, we test the model's knowledge retention by measuring its ability to answer factual questions. Our experiments show that masking entities and principled masking of correlated spans based on pointwise mutual information lead to more factual knowledge being retained than masking random tokens. Our findings demonstrate that, like the ability to perform a task, the (factual) knowledge acquired from being trained on that task is forgotten when a model is trained to perform another task (catastrophic forgetting) and how to prevent this phenomenon. To foster reproducibility, the code, as well as the data used in this paper, are openly available.
Cybersecurity Training for Users of Remote Computing
Venue: arXiv cs.CR 2023
Year: 2023

End users of remote computing systems are frequently not aware of basic ways in which they could enhance protection against cyber-threats and attacks. In this paper, we discuss specific techniques to help and train users to improve cybersecurity when using such systems. To explain the rationale behind these techniques, we go into some depth explaining possible threats in the context of using remote, shared computing resources. Although some of the details of these prescriptions and recommendations apply to specific use cases when connecting to remote servers, such as a supercomputer, cluster, or Linux workstation, the main concepts and ideas can be applied to a wider spectrum of cases.
A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
Venue: arXiv cs.CL 2023
Year: 2023

Large language models such as BERT and the GPT series started a paradigm shift that calls for building general-purpose models via pre-training on large datasets, followed by fine-tuning on task-specific datasets. There is now a plethora of large pre-trained models for Natural Language Processing and Computer Vision. Recently, we have seen rapid developments in the joint Vision-Language space as well, where pre-trained models such as CLIP (Radford et al., 2021) have demonstrated improvements in downstream tasks like image captioning and visual question answering. However, surprisingly there is comparatively little work on exploring these models for the task of multimodal machine translation, where the goal is to leverage image/video modality in text-to-text translation. To fill this gap, this paper surveys the landscape of language-and-vision pre-training from the lens of multimodal machine translation. We summarize the common architectures, pre-training objectives, and datasets from literature and conjecture what further is needed to make progress on multimodal machine translation.
Differential Game Analysis for Cooperation Models in Automotive Supply Chain under Low-Carbon Emission Reduction Policies
Venue: arXiv cs.GT 2023
Year: 2023

In the context of reducing carbon emissions in the automotive supply chain, collaboration between vehicle manufacturers and retailers has proven to be an effective measure for enhancing carbon emission reduction within the enterprise. This study aims to evaluate the effectiveness of such collaboration by constructing a differential game model that incorporates carbon trading and consumer preferences for low-carbon products. The model examines the decision-making process of an automotive supply chain comprising a vehicle manufacturer and multiple retailers. By utilizing the Hamilton-Jacobi-Bellman equation, we analyze the equilibrium strategies of the participants under both a decentralized model and a Stackelberg leader-follower game model. In the decentralized model, the vehicle manufacturer optimizes its carbon emission reduction efforts, while each retailer independently determines its low-carbon promotion efforts and vehicle retail price. In the Stackelberg leader-follower game model, the vehicle manufacturer cooperates with the retailers by offering them a subsidy. Consequently, the manufacturer plays as the leader, making decisions on carbon emission reduction efforts and the subsidy rate, while the retailers, as followers, compute their promotion efforts and retail prices accordingly. Through theoretical analysis and numerical experiments considering the manufacturer's and retailers' efforts, the low-carbon reputation of vehicles, and the overall system profits under both models, we conclude that compared to the decentralized model, where each party pursues individual profits, the collaboration in the Stackelberg game yields greater benefits for both parties. Furthermore, this collaborative approach promotes the long-term development of the automotive supply chain.
Kalman Filter Auto-tuning through Enforcing Chi-Squared Normalized Error Distributions with Bayesian Optimization
Venue: arXiv cs.RO 2023
Year: 2023

The nonlinear and stochastic relationship between noise covariance parameter values and state estimator performance makes optimal filter tuning a very challenging problem. Popular optimization-based tuning approaches can easily get trapped in local minima, leading to poor noise parameter identification and suboptimal state estimation. Recently, black box techniques based on Bayesian optimization with Gaussian processes (GPBO) have been shown to overcome many of these issues, using normalized estimation error squared (NEES) and normalized innovation error (NIS) statistics to derive cost functions for Kalman filter auto-tuning. While reliable noise parameter estimates are obtained in many cases, GPBO solutions obtained with these conventional cost functions do not always converge to optimal filter noise parameters and lack robustness to parameter ambiguities in time-discretized system models. This paper addresses these issues by making two main contributions. First, we show that NIS and NEES errors are only chi-squared distributed for tuned estimators. As a result, chi-square tests are not sufficient to ensure that an estimator has been correctly tuned. We use this to extend the familiar consistency tests for NIS and NEES to penalize if the distribution is not chi-squared distributed. Second, this cost measure is applied within a Student-t processes Bayesian Optimization (TPBO) to achieve robust estimator performance for time discretized state space models. The robustness, accuracy, and reliability of our approach are illustrated on classical state estimation problems.
No Free Lunch: The Hazards of Over-Expressive Representations in Anomaly Detection
Venue: arXiv cs.LG 2023
Year: 2023

Anomaly detection methods, powered by deep learning, have recently been making significant progress, mostly due to improved representations. It is tempting to hypothesize that anomaly detection can improve indefinitely by increasing the scale of our networks, making their representations more expressive. In this paper, we provide theoretical and empirical evidence to the contrary. In fact, we empirically show cases where very expressive representations fail to detect even simple anomalies when evaluated beyond the well-studied object-centric datasets. To investigate this phenomenon, we begin by introducing a novel theoretical toy model for anomaly detection performance. The model uncovers a fundamental trade-off between representation sufficiency and over-expressivity. It provides evidence for a no-free-lunch theorem in anomaly detection stating that increasing representation expressivity will eventually result in performance degradation. Instead, guidance must be provided to focus the representation on the attributes relevant to the anomalies of interest. We conduct an extensive empirical investigation demonstrating that state-of-the-art representations often suffer from over-expressivity, failing to detect many types of anomalies. Our investigation demonstrates how this over-expressivity impairs image anomaly detection in practical settings. We conclude with future directions for mitigating this issue.
Ethical Considerations of AR Applications in Smartphones; A Systematic Literature Review of Consumer Perspectives
Venue: arXiv cs.HC 2023
Year: 2023

This study focuses on the ethical considerations that a consumer perceives with augmented reality (AR) in the context of smartphone applications. Through a systematic review, this research can provide an understanding and ability for developers, product managers, digital marketers and associated business professionals to effectively implement and deploy mobile AR related applications and campaigns, with consideration to the perceptions of the ethical considerations that consumers have of this growing technology. The rise in digital transformation and new technologies paved this research agenda. Trends in the data revealed two overarching factors of 'Benefits' and 'Ethical Considerations'. Within these two factors, several consumer perceived themes were identified with regards to AR applications and their association categorised either positive, negative or neutral. 'Benefits' revealed 3 consistent themes of personalisation, interactivity and information acquisition. 'Ethical Considerations' revealed consistent patterns of educational awareness, privacy, transparency and security. From identifying the consumer perceptions, business professionals can strategically address and or challenge the inherent limitations and their associations during AR application development, product adoption strategies or marketing purposes.
Value function estimation using conditional diffusion models for control
Venue: arXiv cs.LG 2023
Year: 2023

A fairly reliable trend in deep reinforcement learning is that the performance scales with the number of parameters, provided a complimentary scaling in amount of training data. As the appetite for large models increases, it is imperative to address, sooner than later, the potential problem of running out of high-quality demonstrations. In this case, instead of collecting only new data via costly human demonstrations or risking a simulation-to-real transfer with uncertain effects, it would be beneficial to leverage vast amounts of readily-available low-quality data. Since classical control algorithms such as behavior cloning or temporal difference learning cannot be used on reward-free or action-free data out-of-the-box, this solution warrants novel training paradigms for continuous control. We propose a simple algorithm called Diffused Value Function (DVF), which learns a joint multi-step model of the environment-robot interaction dynamics using a diffusion model. This model can be efficiently learned from state sequences (i.e., without access to reward functions nor actions), and subsequently used to estimate the value of each action out-of-the-box. We show how DVF can be used to efficiently capture the state visitation measure for multiple controllers, and show promising qualitative and quantitative results on challenging robotics benchmarks.
Lost in Translation: Large Language Models in Non-English Content Analysis
Venue: arXiv cs.CL 2023
Year: 2023

In recent years, large language models (e.g., Open AI's GPT-4, Meta's LLaMa, Google's PaLM) have become the dominant approach for building AI systems to analyze and generate language online. However, the automated systems that increasingly mediate our interactions online -- such as chatbots, content moderation systems, and search engines -- are primarily designed for and work far more effectively in English than in the world's other 7,000 languages. Recently, researchers and technology companies have attempted to extend the capabilities of large language models into languages other than English by building what are called multilingual language models. In this paper, we explain how these multilingual language models work and explore their capabilities and limits. Part I provides a simple technical explanation of how large language models work, why there is a gap in available data between English and other languages, and how multilingual language models attempt to bridge that gap. Part II accounts for the challenges of doing content analysis with large language models in general and multilingual language models in particular. Part III offers recommendations for companies, researchers, and policymakers to keep in mind when considering researching, developing and deploying large and multilingual language models.
Probing Quantifier Comprehension in Large Language Models: Another Example of Inverse Scaling
Venue: arXiv cs.CL 2023
Year: 2023

With their increasing size, large language models (LLMs) are becoming increasingly good at language understanding tasks. But even with high performance on specific downstream task, LLMs fail at simple linguistic tests for negation or quantifier understanding. Previous work on quantifier understanding in LLMs show inverse scaling in understanding few-type quantifiers. In this paper, we question the claims of of previous work and show that it is a result of inappropriate testing methodology. We also present alternate methods to measure quantifier comprehension in LLMs and show that LLMs are able to better understand the difference between the meaning of few-type and most-type quantifiers as their size increases, although they are not particularly good at it. We also observe inverse scaling for most-type quantifier understanding, which is contrary to human psycho-linguistic experiments and previous work, where the model's understanding of most-type quantifier gets worse as the model size increases. We do this evaluation on models ranging from 125M-175B parameters, which suggests that LLMs do not do as well as expected with quantifiers. We also discuss the possible reasons for this and the relevance of quantifier understanding in evaluating language understanding in LLMs.
Enhancing Topic Extraction in Recommender Systems with Entropy Regularization
Venue: arXiv cs.CL 2023
Year: 2023

In recent years, many recommender systems have utilized textual data for topic extraction to enhance interpretability. However, our findings reveal a noticeable deficiency in the coherence of keywords within topics, resulting in low explainability of the model. This paper introduces a novel approach called entropy regularization to address the issue, leading to more interpretable topics extracted from recommender systems, while ensuring that the performance of the primary task stays competitively strong. The effectiveness of the strategy is validated through experiments on a variation of the probabilistic matrix factorization model that utilizes textual data to extract item embeddings. The experiment results show a significant improvement in topic coherence, which is quantified by cosine similarity on word embeddings.
Connecting continuum poroelasticity with discrete synthetic vascular trees for modeling liver tissue
Venue: arXiv cs.CE 2023
Year: 2023

Computational simulations have the potential to assist in liver resection surgeries by facilitating surgical planning, optimizing resection strategies, and predicting postoperative outcomes. The modeling of liver tissue across multiple length scales constitutes a significant challenge, primarily due to the multiphysics coupling of mechanical response and perfusion within the complex multiscale vascularization of the organ. In this paper, we present a modeling framework that connects continuum poroelasticity and discrete vascular tree structures to model liver tissue across disparate levels of the perfusion hierarchy. The connection is achieved through a series of modeling decisions, which include source terms in the pressure equation to model inflow from the supplying tree, pressure boundary conditions to model outflow into the draining tree, and contact conditions to model surrounding tissue. We investigate the numerical behaviour of our framework and apply it to a patient-specific full-scale liver problem that demonstrates its potential to help assess surgical liver resection procedures
Gender-Inclusive Grammatical Error Correction through Augmentation
Venue: arXiv cs.CL 2023
Year: 2023

In this paper we show that GEC systems display gender bias related to the use of masculine and feminine terms and the gender-neutral singular "they". We develop parallel datasets of texts with masculine and feminine terms and singular "they" and use them to quantify gender bias in three competitive GEC systems. We contribute a novel data augmentation technique for singular "they" leveraging linguistic insights about its distribution relative to plural "they". We demonstrate that both this data augmentation technique and a refinement of a similar augmentation technique for masculine and feminine terms can generate training data that reduces bias in GEC systems, especially with respect to singular "they" while maintaining the same level of quality.
Psychological Aspects of Pair Programming
Venue: arXiv cs.SE 2023
Year: 2023

With the recent advent of artificially intelligent pairing partners in software engineering, it is interesting to renew the study of the psychology of pairing. Pair programming provides an attractive way of teaching software engineering to university students. Its study can also lead to a better understanding of the needs of professional software engineers in various programming roles and for the improvement of the concurrent pairing software. Objective: This preliminary study aimed to gain quantitative and qualitative insights into pair programming, especially students' attitudes towards its specific roles and what they require from the pairing partners. The research's goal is to use the findings to design further studies on pairing with artificial intelligence. Method: Using a mixed-methods and experimental approach, we distinguished the effects of the pilot, navigator, and solo roles on (N = 35) students' intrinsic motivation. Four experimental sessions produced a rich data corpus in two software engineering university classrooms. It was quantitatively investigated using the Shapiro-Wilk normality test and one-way analysis of variance (ANOVA) to confirm the relations and significance of variations in mean intrinsic motivation in different roles. Consequently, seven semi-structured interviews were conducted with the experiment's participants. The qualitative data excerpts were subjected to the thematic analysis method in an essentialist way. Results: The systematic coding interview transcripts elucidated the research topic by producing seven themes for understanding the psychological aspects of pair programming and for its improvement in university classrooms. Statistical analysis of 612 self-reported intrinsic motivation inventories confirmed that students find programming in pilot-navigator roles more interesting and enjoyable than programming simultaneously.