Reviewerly/exHarmony huggingface dataset:
Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset. We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that ``optimal'' subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.
SCP: Scene Completion Pre-training for 3D Object Detection

3D object detection using LiDAR point clouds is a fundamental task in the fields of computer vision, robotics, and autonomous driving. However, existing 3D detectors heavily rely on annotated datasets, which are both time-consuming and prone to errors during the process of labeling 3D bounding boxes. In this paper, we propose a Scene Completion Pre-training (SCP) method to enhance the performance of 3D object detectors with less labeled data. SCP offers three key advantages: (1) Improved initialization of the point cloud model. By completing the scene point clouds, SCP effectively captures the spatial and semantic relationships among objects within urban environments. (2) Elimination of the need for additional datasets. SCP serves as a valuable auxiliary network that does not impose any additional efforts or data requirements on the 3D detectors. (3) Reduction of the amount of labeled data for detection. With the help of SCP, the existing state-of-the-art 3D detectors can achieve comparable performance while only relying on 20% labeled data.
Predicting Routine Object Usage for Proactive Robot Assistance

Proactivity in robot assistance refers to the robot's ability to anticipate user needs and perform assistive actions without explicit requests. This requires understanding user routines, predicting consistent activities, and actively seeking information to predict inconsistent behaviors. We propose SLaTe-PRO (Sequential Latent Temporal model for Predicting Routine Object usage), which improves upon prior state-of-the-art by combining object and user action information, and conditioning object usage predictions on past history. Additionally, we find some human behavior to be inherently stochastic and lacking in contextual cues that the robot can use for proactive assistance. To address such cases, we introduce an interactive query mechanism that can be used to ask queries about the user's intended activities and object use to improve prediction. We evaluate our approach on longitudinal data from three households, spanning 24 activity classes. SLaTe-PRO performance raises the F1 score metric to 0.57 without queries, and 0.60 with user queries, over a score of 0.43 from prior work. We additionally present a case study with a fully autonomous household robot.
Risk-Aware Reinforcement Learning through Optimal Transport Theory

In the dynamic and uncertain environments where reinforcement learning (RL) operates, risk management becomes a crucial factor in ensuring reliable decision-making. Traditional RL approaches, while effective in reward optimization, often overlook the landscape of potential risks. In response, this paper pioneers the integration of Optimal Transport (OT) theory with RL to create a risk-aware framework. Our approach modifies the objective function, ensuring that the resulting policy not only maximizes expected rewards but also respects risk constraints dictated by OT distances between state visitation distributions and the desired risk profiles. By leveraging the mathematical precision of OT, we offer a formulation that elevates risk considerations alongside conventional RL objectives. Our contributions are substantiated with a series of theorems, mapping the relationships between risk distributions, optimal value functions, and policy behaviors. Through the lens of OT, this work illuminates a promising direction for RL, ensuring a balanced fusion of reward pursuit and risk awareness.
The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.
OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation

Temporal action segmentation is typically achieved by discovering the dramatic variances in global visual descriptors. In this paper, we explore the merits of local features by proposing the unsupervised framework of Object-centric Temporal Action Segmentation (OTAS). Broadly speaking, OTAS consists of self-supervised global and local feature extraction modules as well as a boundary selection module that fuses the features and detects salient boundaries for action segmentation. As a second contribution, we discuss the pros and cons of existing frame-level and boundary-level evaluation metrics. Through extensive experiments, we find OTAS is superior to the previous state-of-the-art method by $41\%$ on average in terms of our recommended F1 score. Surprisingly, OTAS even outperforms the ground-truth human annotations in the user study. Moreover, OTAS is efficient enough to allow real-time inference.
Quantum memories for squeezed and coherent superpositions in a driven-dissipative nonlinear oscillator

Quantum oscillators with nonlinear driving and dissipative terms have gained significant attention due to their ability to stabilize cat-states for universal quantum computation. Recently, superconducting circuits have been employed to realize such long-lived qubits stored in coherent states. We present a generalization of these oscillators, which are not limited to coherent states, in the presence of different nonlinearities in driving and dissipation, exploring different degrees. Specifically, we present an extensive analysis of the asymptotic dynamical features and of the storage of squeezed states. We demonstrate that coherent superpositions of squeezed states are achievable in the presence of a strong symmetry, thereby allowing for the storage of squeezed cat-states. In the weak symmetry regime, accounting for linear dissipation, we investigate the potential application of these nonlinear driven-dissipative resonators for quantum computing and quantum associative memory and analyze the impact of squeezing on their performance.
Preliminary Results from a U.S. Demographic Analysis of SMiSh Susceptibility

As adoption of mobile phones has skyrocketed, so have scams involving them. The text method is called SMiShing, (aka SMShing, or smishing) in which a fraudster sends a phishing link via Short Message Service (SMS) text to a phone. However, no data exists on who is most vulnerable to SMiShing. Prior work in phishing (its e-mail cousin) indicates that this is likely to vary by demographic and contextual factors. In our study, we collect this data from N=1007 U.S. adult mobile phone users. Younger people and college students emerge in this sample as the most vulnerable. Participants struggled to correctly identify legitimate messages and were easily misled when they knew they had an account with the faked message entity. Counterintuitively, participants with higher levels of security training and awareness were less correct in rating possible SMiSH. We recommend next steps for researchers, regulators and telecom providers.
Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering

Robustness in Natural Language Processing continues to be a pertinent issue, where state of the art models under-perform under naturally shifted distributions. In the context of Question Answering, work on domain adaptation methods continues to be a growing body of research. However, very little attention has been given to the notion of domain generalization under natural distribution shifts, where the target domain is unknown. With drastic improvements in the quality and access to generative models, we answer the question: How do generated datasets influence the performance of QA models under natural distribution shifts? We perform experiments on 4 different datasets under varying amounts of distribution shift, and analyze how "in-the-wild" generation can help achieve domain generalization. We take a two-step generation approach, generating both contexts and QA pairs to augment existing datasets. Through our experiments, we demonstrate how augmenting reading comprehension datasets with generated data leads to better robustness towards natural distribution shifts.
Down the Toxicity Rabbit Hole: Investigating PaLM 2 Guardrails

This paper conducts a robustness audit of the safety feedback of PaLM 2 through a novel toxicity rabbit hole framework introduced here. Starting with a stereotype, the framework instructs PaLM 2 to generate more toxic content than the stereotype. Every subsequent iteration it continues instructing PaLM 2 to generate more toxic content than the previous iteration until PaLM 2 safety guardrails throw a safety violation. Our experiments uncover highly disturbing antisemitic, Islamophobic, racist, homophobic, and misogynistic (to list a few) generated content that PaLM 2 safety guardrails do not evaluate as highly unsafe. We briefly discuss the generalizability of this framework across eight other large language models.
Unveiling the potential of large language models in generating semantic and cross-language clones

Semantic and Cross-language code clone generation may be useful for code reuse, code comprehension, refactoring and benchmarking. OpenAI's GPT model has potential in such clone generation as GPT is used for text generation. When developers copy/paste codes from Stack Overflow (SO) or within a system, there might be inconsistent changes leading to unexpected behaviours. Similarly, if someone possesses a code snippet in a particular programming language but seeks equivalent functionality in a different language, a semantic cross-language code clone generation approach could provide valuable assistance. In this study, using SemanticCloneBench as a vehicle, we evaluated how well the GPT-3 model could help generate semantic and cross-language clone variants for a given fragment.We have comprised a diverse set of code fragments and assessed GPT-3s performance in generating code variants.Through extensive experimentation and analysis, where 9 judges spent 158 hours to validate, we investigate the model's ability to produce accurate and semantically correct variants. Our findings shed light on GPT-3's strengths in code generation, offering insights into the potential applications and challenges of using advanced language models in software development. Our quantitative analysis yields compelling results. In the realm of semantic clones, GPT-3 attains an impressive accuracy of 62.14% and 0.55 BLEU score, achieved through few-shot prompt engineering. Furthermore, the model shines in transcending linguistic confines, boasting an exceptional 91.25% accuracy in generating cross-language clones
Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape

Stochastic Gradient Descent (SGD) is an out-of-equilibrium algorithm used extensively to train artificial neural networks. However very little is known on to what extent SGD is crucial for to the success of this technology and, in particular, how much it is effective in optimizing high-dimensional non-convex cost functions as compared to other optimization algorithms such as Gradient Descent (GD). In this work we leverage dynamical mean field theory to benchmark its performances in the high-dimensional limit. To do that, we consider the problem of recovering a hidden high-dimensional non-linearly encrypted signal, a prototype high-dimensional non-convex hard optimization problem. We compare the performances of SGD to GD and we show that SGD largely outperforms GD for sufficiently small batch sizes. In particular, a power law fit of the relaxation time of these algorithms shows that the recovery threshold for SGD with small batch size is smaller than the corresponding one of GD.
The development of proton-electron math e-comic to improve special needs students’ mathematical concepts understanding

Technology-based learning media is still lacking for special needs students in mathematics. Based on observations and interviews with Special School (SLB) teachers, the learning media used were still monotonous, such as picture books and writing using a blackboard. This research aims to develop e-comic math using the App Pixton application, which is valid and practical for children with special needs in SLB. The research method used is Plomp model development research in the prototype development phase: self-evaluation, expert review, and one-to-one evaluation. The data collection instruments used were a questionnaire and mathematical concepts on addition and subtraction topics for students with special needs. The research subjects were 21 students and 5 teachers from 5 different special schools in 5 West Sumatra, Indonesia cities. The study results show that the media developed is valid and practical for students with deaf and intellectual disabilities. This research implies that teachers must be able to develop other IT-based learning media to help students with special needs learn mathematics like normal children.
CR-SoRec: BERT driven Consistency Regularization for Social Recommendation

In the real world, when we seek our friends’ opinions on various items or events, we request verbal social recommendations. It has been observed that we often turn to our friends for recommendations on a daily basis. The emergence of online social platforms has enabled users to share their opinion with their social connections. Therefore, we should consider users’ social connections to enhance online recommendation performance. The social recommendation aims to fuse social links with user-item interactions to offer more relevant recommendations. Several efforts have been made to develop an effective social recommendation system. However, there are two significant limitations to current methods: First, they haven’t thoroughly explored the intricate relationships between the diverse influences of neighbours on users’ preferences. Second, existing models are vulnerable to overfitting due to the relatively low number of user-item interaction records in the interaction space. For the aforementioned problems, this paper offers a novel framework called CR-SoRec, an effective recommendation model based on BERT and consistency regularization. This model incorporates Bidirectional Encoder Representations from Transformer(BERT) to learn bidirectional context-aware user and item embeddings with neighbourhood sampling. The neighbourhood Sampling technique samples the most influential neighbours for all the users/ items. Further, to effectively use the available user-item interaction data and social ties, we leverage diverse perspectives via consistency regularization to harness the underlying information. The main objective of our model is to predict the next item that a user would interact with based on its interaction behaviour and social connections. Experimental results show that our model defines a new state-of-the-art on various datasets and outperforms previous work by a significant margin. Extensive experiments are also conducted to analyze the proposed method.
Collaborative filtering algorithms are prone to mainstream-taste bias

Collaborative filtering has been a dominant approach in the recommender systems community since the early 1990s. Collaborative filtering (and other) algorithms, however, have been predominantly evaluated by aggregating results across users or user groups. These performance averages hide large disparities: an algorithm may perform very well for some users (or groups) and poorly for others. We show that performance variation is large and systematic. In experiments on three large-scale datasets and using an array of collaborative filtering algorithms, we demonstrate large performance disparities across algorithms, datasets and metrics for different users. We then show that two key features that characterize users, their mean taste similarity and dispersion in taste similarity with other users, can systematically explain performance variation better than previously identified features. We use these two features to visualize algorithm performance for different users and we point out that this mapping can capture different categories of users that have been proposed before. Our results demonstrate an extensive mainstream-taste bias in collaborative filtering algorithms, which implies a fundamental fairness limitation that needs to be mitigated.
Tuning Stressful Experience in Virtual Reality Games

Dynamic game balancing using players’ affective state is a promising approach for creating immersive and engaging video games. We present here the results of StrEx, a system which modulates the stress level induced by a video game in a completely unobtrusive way. It has been tested on an ad-hoc developed Virtual Reality horror-survival game. The system collects motion behavioral data from Head-Mounted Display and its controllers and updates a dynamical model of the player stress level, that takes into account the interior affective state dynamics. Such model is used to guide the transitions of a Finite State Machine that controls the stress level induced by the game, to maintain an appropriate level of stress. The effectiveness of the system has been evaluated through an experimental study in which participants played the game with and without StrEx. Preliminary results show that the StrEx plugin can increase players’ perceived competence and decrease their tension and frustration, thus leading to an increased engagement in the game.
Harmonizing Epochs: Blockchain Technology at the Intersection of Friedman’s Monetary Policy and Keynesian Economic Principles

&lt;p&gt;The advent of blockchain technology has ushered in a new era of innovation, potentially reshaping the landscape of various industries, including the financial and insurance sectors. This study seeks to explore the intricate interplay between blockchain technology and two pivotal economic theories: Friedman’s monetary policy and Keynesian economic principles. Through meticulous analysis, this research aims to unravel the potential synergies and conflicts that emerge when modern technology meets established economic doctrines. &lt;/p&gt; &lt;p&gt;At the heart of this research is a comprehensive case study on Allianz, a prominent insurance company in Germany, which has embarked on a journey to integrate blockchain technology into its operations. This case study serves as a microcosm to understand the broader implications of blockchain technology in the context of Friedman’s emphasis on monetary stability and Keynesian advocacy for government intervention in economic management. &lt;/p&gt; &lt;p&gt;By dissecting Allianz’s initiatives, this study delves into the nuances of blockchain implementation, examining its potential to foster efficiency, transparency, and economic stimulation, facets that resonate well with the theories propounded by Friedman and Keynes. Furthermore, this research scrutinizes the regulatory challenges and opportunities that emerge in this transformative journey, offering a balanced perspective on the potential harmonization of these epochs. &lt;/p&gt; &lt;p&gt;Through this research, a nuanced dialogue is fostered that bridges the gap between technology and economic theory, paving the way for a harmonized future where innovation complements tradition. By harmonizing these epochs, a future is envisioned where blockchain technology not only revolutionizes industries but also aligns seamlessly with enduring economic principles, fostering a symbiotic relationship that propels society towards sustainable growth and development. &lt;/p&gt;