Reviewerly/exHarmony huggingface dataset:
Knowledge updating in real-world estimation: Connecting hindsight bias and seeding effects.

When people estimate the quantities of objects (e.g., country populations), are then presented with the objects' actual quantities, and subsequently asked to remember their initial estimates, responses are often distorted towards the actual quantities. This hindsight bias-traditionally considered to reflect a cognitive error-has more recently been proposed to result from adaptive knowledge updating. But how to conceptualize such knowledge-updating processes and their potentially beneficial consequences? Here, we provide a framework that conceptualizes knowledge updating in the context of hindsight bias in real-world estimation by connecting it with research on seedingeffects-improvements in people's estimation accuracy after exposure to numerical facts. This integrative perspective highlights a previously neglected facet of knowledge updating, namely, recalibration of metric domain knowledge, which can be expected to lead to transfer learning and thus improve estimation for objects from a domain more generally. We develop an experimental paradigm to investigate the association of hindsight bias with improved estimation accuracy. In Experiment 1, we demonstrate that the classical approach to induce hindsight bias indeed produces transfer learning. In Experiment 2, we provide evidence for the novel prediction that hindsight bias can be triggered via transfer learning; this establishes a direct link from knowledge updating to hindsight bias. Our work integrates two prominent but previously unconnected research programs on the effects of knowledge updating in real-world estimation and supports the notion that hindsight bias is driven by adaptive learning processes. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Efficient option pricing with a unary-based photonic computing chip and generative adversarial learning

In the modern financial industry system, the structure of products has become more and more complex, and the bottleneck constraint of classical computing power has already restricted the development of the financial industry. Here, we present a photonic chip that implements the unary approach to European option pricing, in combination with the quantum amplitude estimation algorithm, to achieve a quadratic speedup compared to classical Monte Carlo methods. The circuit consists of three modules: a module loading the distribution of asset prices, a module computing the expected payoff, and a module performing the quantum amplitude estimation algorithm to introduce speed-ups. In the distribution module, a generative adversarial network is embedded for efficient learning and loading of asset distributions, which precisely capture the market trends. This work is a step forward in the development of specialized photonic processors for applications in finance, with the potential to improve the efficiency and quality of financial services.
Text Is All You Need: Learning Language Representations for Sequential Recommendation

Sequential recommendation aims to model dynamic user behavior from historical interactions. Existing methods rely on either explicit item IDs or general textual features for sequence modeling to understand user preferences. While promising, these approaches still struggle to model cold-start items or transfer knowledge to new datasets. In this paper, we propose to model user preferences and item features as language representations that can be generalized to new items and datasets. To this end, we present a novel framework, named Recformer, which effectively learns language representations for sequential recommendation. Specifically, we propose to formulate an item as a "sentence" (word sequence) by flattening item key-value attributes described by text so that an item sequence for a user becomes a sequence of sentences. For recommendation, Recformer is trained to understand the "sentence" sequence and retrieve the next "sentence". To encode item sequences, we design a bi-directional Transformer similar to the model Longformer but with different embedding layers for sequential recommendation. For effective representation learning, we propose novel pretraining and finetuning methods which combine language understanding and recommendation tasks. Therefore, Recformer can effectively recommend the next item based on language representations. Extensive experiments conducted on six datasets demonstrate the effectiveness of Recformer for sequential recommendation, especially in low-resource and cold-start settings.
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with fullparameter fine-tuning.To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning" in Ding et al. ( 2022), which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed.However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hardcode specific delta tuning methods for each PTM.In this paper, we present OpenDelta 1 , an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods.Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs.OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.
Dialog-Post: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training

Dialogue representation and understanding aim to convert conversational inputs into embeddings and fulfill discriminative tasks.Compared with free-form text, dialogue has two important characteristics, hierarchical semantic structure and multi-facet attributes.Therefore, directly applying the pretrained language models (PLMs) might result in unsatisfactory performance.Recently, several work focused on the dialogue-adaptive post-training (Dial-Post) that further trains PLMs to fit dialogues.To model dialogues more comprehensively, we propose a DialPost method, DIALOG-POST, with multi-level self-supervised objectives and a hierarchical model.These objectives leverage dialogue-specific attributes and use selfsupervised signals to fully facilitate the representation and understanding of dialogues.The novel model is a hierarchical segment-wise self-attention network, which contains innersegment and inter-segment self-attention sublayers followed by an aggregation and updating module.To evaluate the effectiveness of our methods, we first apply two public datasets for the verification of representation ability.Then we conduct experiments on a newly-labelled dataset that is annotated with 4 dialogue understanding tasks.Experimental results show that our method outperforms existing SOTA models and achieves a 3.3% improvement on average. Token-level SSOsUtterance-level SSO Dialogue-level SSOs : 这个手机支持5G吗?
Learning to Generate Question by Asking Question: A Primal-Dual Approach with Uncommon Word Generation

Automatic question generation (AQG) is the task of generating a question from a given passage and an answer.Most existing AQG methods aim at encoding the passage and the answer to generate the question.However, limited work has focused on modeling the correlation between the target answer and the generated question.Moreover, unseen or rare word generation has not been studied in previous works.In this paper, we propose a novel approach which incorporates question generation with its dual problem, question answering, into a unified primal-dual framework.Specifically, the question generation component consists of an encoder that jointly encodes the answer with the passage, and a decoder that produces the question.The question answering component then re-asks the generated question on the passage to ensure that the target answer is obtained.We further introduce a knowledge distillation module to improve the model generalization ability.We conduct an extensive set of experiments on SQuAD and HotpotQA benchmarks.Experimental results demonstrate the superior performance of the proposed approach over several state-of-the-art methods.
PyHealth: A Deep Learning Toolkit for Healthcare Applications

Deep learning (DL) has emerged as a promising tool in healthcare applications. However, the reproducibility of many studies in this field is limited by the lack of accessible code implementations and standard benchmarks. To address the issue, we create PyHealth, a comprehensive library to build, deploy, and validate DL pipelines for healthcare applications. PyHealth supports various data modalities, including electronic health records (EHRs), physiological signals, medical images, and clinical text. It offers various advanced DL models and maintains comprehensive medical knowledge systems. The library is designed to support both DL researchers and clinical data scientists. Upon the time of writing, PyHealth has received 633 stars, 130 forks, and 15k+ downloads in total on GitHub.
Deep Span Representations for Named Entity Recognition

Span-based models are one of the most straightforward methods for named entity recognition (NER).Existing span-based NER systems shallowly aggregate the token representations to span representations.However, this typically results in significant ineffectiveness for long entities, a coupling between the representations of overlapping spans, and ultimately a performance degradation.In this study, we propose DSpERT (Deep Span Encoder Representations from Transformers), which comprises a standard Transformer and a span Transformer.The latter uses low-layered span representations as queries, and aggregates the token representations as keys and values, layer by layer from bottom to top.Thus, DSpERT produces span representations of deep semantics.With weight initialization from pretrained language models, DSpERT achieves performance higher than or competitive with recent state-ofthe-art systems on six NER benchmarks. 1 Experimental results verify the importance of the depth for span representations, and show that DSpERT performs particularly well on longspan entities and nested structures.Further, the deep span representations are well structured and easily separable in the feature space.
PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

Current advances in recommender systems have been remarkably successful in optimizing immediate engagement. However, long-term user engagement, a more desirable performance metric, remains difficult to improve. Meanwhile, recent reinforcement learning (RL) algorithms have shown their effectiveness in a variety of long-term goal optimization tasks. For this reason, RL is widely considered as a promising framework for optimizing long-term user engagement in recommendation. Though promising, the application of RL heavily relies on well-designed rewards, but designing rewards related to long-term user engagement is quite difficult. To mitigate the problem, we propose a novel paradigm, recommender systems with human preferences (or Preference-based Recommender systems), which allows RL recommender systems to learn from preferences about users' historical behaviors rather than explicitly defined rewards. Such preferences are easily accessible through techniques such as crowdsourcing, as they do not require any expert knowledge. With PrefRec, we can fully exploit the advantages of RL in optimizing long-term goals, while avoiding complex reward engineering. PrefRec uses the preferences to automatically train a reward function in an end-to-end manner. The reward function is then used to generate learning signals to train the recommendation policy. Furthermore, we design an effective optimization method for PrefRec, which uses an additional value function, expectile regression and reward model pre-training to improve the performance. We conduct experiments on a variety of long-term user engagement optimization tasks. The results show that PrefRec significantly outperforms previous state-of-the-art methods in all the tasks.
Debiasing Recommendation by Learning Identifiable Latent Confounders

Recommendation systems aim to predict users' feedback on items not exposed to them yet. Confounding bias arises due to the presence of unmeasured variables (e.g., the socio-economic status of a user) that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. However, they cannot guarantee the identification of counterfactual feedback, which can lead to biased predictions. In this work, we propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables (e.g., observed user features) to resolve the aforementioned non-identification issue. The proposed iDCF is a general deconfounded recommendation framework that applies proximal causal inference to infer the unmeasured confounders and identify the counterfactual feedback with theoretical guarantees. Extensive experiments on various real-world and synthetic datasets verify the proposed method's effectiveness and robustness.
SAMD: An Industrial Framework for Heterogeneous Multi-Scenario Recommendation

Industrial recommender systems usually need to serve multiple scenarios at the same time. In practice, there are various heterogeneous scenarios, since users frequently engage in scenarios with varying intentions and the items within each scenario typically belong to diverse categories. Existing works of multi-scenario recommendation mainly focus on modeling homogeneous scenarios which have similar data distributions. They equally transfer knowledge to each scenario without considering the diversity of heterogeneous scenarios. In this paper, we argue that the heterogeneity in multi-scenario recommendations is a key problem that needs to be solved. To this end, we propose an industrial framework named Scenario-Aware Model-Agnostic Meta Distillation (SAMD) for the multi-scenario recommendation. SAMD aims to provide scenario-aware and model-agnostic knowledge sharing across heterogeneous scenarios by modeling scenarios' relationship and conducting heterogeneous knowledge distillation. Specifically, SAMD first measures the comprehensive representation of each scenario and then proposes a novel meta distillation paradigm to conduct scenario-aware knowledge sharing. The meta network first establishes the potential scenarios' relationships and generates the strategies of knowledge sharing for each scenario. Then the heterogeneous knowledge distillation utilizes scenario-aware strategies to share knowledge across heterogeneous scenarios through intermediate features distillation without the restriction of the model architecture. In this way, SAMD shares knowledge across heterogeneous scenarios in a scenario-aware and model-agnostic manner, which addresses the problem of heterogeneity. Compared with other state-of-the-art methods, extensive offline experiments, and online A/B testing demonstrate the superior performance of the proposed SAMD framework, especially in heterogeneous scenarios.
Conditional Neural ODE Processes for Individual Disease Progression Forecasting: A Case Study on COVID-19

Time series forecasting, as one of the fundamental machine learning areas, has attracted tremendous attentions over recent years. The solutions have evolved from statistical machine learning (ML) methods to deep learning techniques. One emerging sub-field of time series forecasting is individual disease progression forecasting, e.g., predicting individuals' disease development over a few days (e.g., deteriorating trends, recovery speed) based on few past observations. Despite the promises in the existing ML techniques, a variety of unique challenges emerge for disease progression forecasting, such as irregularly-sampled time series, data sparsity, and individual heterogeneity in disease progression. To tackle these challenges, we propose novel Conditional Neural Ordinary Differential Equations Processes (CNDPs), and validate it in a COVID-19 disease progression forecasting task using audio data. CNDPs allow for irregularly-sampled time series modelling, enable accurate forecasting with sparse past observations, and achieve individual-level progression forecasting. CNDPs show strong performance with an Unweighted Average Recall (UAR) of 78.1%, outperforming a variety of commonly used Recurrent Neural Networks based models. With the proposed label-enhancing mechanism (i.e., including the initial health status as input) and the customised individual-level loss, CNDPs further boost the performance reaching a UAR of 93.6%. Additional analysis also reveals the model's capability in tracking individual-specific recovery trend, implying the potential usage of the model for remote disease progression monitoring. In general, CNDPs pave new pathways for time series forecasting, and provide considerable advantages for disease progression monitoring.
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions

Yeah, we're pretty lousy with pens around here, so knock yourself out.Really? Thanks.
DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation

State-of-the-art Automatic Speech Recognition (ASR) systems are known to exhibit disparate performance on varying speech accents.To improve performance on a specific target accent, a commonly adopted solution is to finetune the ASR model using accent-specific labeled speech.However, acquiring large amounts of labeled speech for specific target accents is challenging.Choosing an informative subset of speech samples that are most representative of the target accents becomes important for effective ASR finetuning.To address this problem, we propose DITTO (Data-efficient and faIr Targeted subseT selectiOn) that uses Submodular Mutual Information (SMI) functions as acquisition functions to find the most informative set of utterances matching a target accent within a fixed budget.An important feature of DITTO is that it supports fair targeting for multiple accents, i.e. it can automatically select representative data points from multiple accents when the ASR model needs to perform well on more than one accent.We show that DITTO is 3-5 times more label-efficient than other speech selection methods on the Indic-TTS and L2 datasets.
Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

Multilingual Vision-Language Pre-training (VLP) is a promising but challenging topic due to the lack of large-scale multilingual imagetext pairs.Existing works address the problem by translating English data into other languages, which is intuitive and the generated data is usually limited in form and scale.In this paper, we explore a more practical and scalable setting: weakly supervised multilingual VLP with only English image-text pairs and multilingual text corpora.We argue that the universal multilingual representation learned from texts allows the cross-modal interaction learned in English to be transferable to other languages.To this end, we propose a framework to effectively unify cross-lingual and cross-modal pre-training.For unified modeling on different data, we design an architecture with flexible modules to learn different interactions.Moreover, two unified tasks are introduced to efficiently guide the unified crosslingual cross-modal learning.Extensive experiments demonstrate that our pre-trained model learns universal multilingual multimodal representations, allowing effective cross-lingual transfer on multimodal tasks.Code and models are available at https://github.com/FudanDISC/weakly-supervised-mVLP.
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments compared to traditional metrics based on lexical overlap, such as BLEU.Yet neural metrics are, to a great extent, "black boxes" that return a single sentence-level score without transparency about the decisionmaking process.In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics.Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors.To ease future research, we release our code at https:
A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification

In recent years, large language models (LLMs) have achieved strong performance on benchmark tasks, especially in zero or few-shot settings.However, these benchmarks often do not adequately address the challenges posed in the real-world, such as that of hierarchical classification.In order to address this challenge, we propose refactoring conventional tasks on hierarchical datasets into a more indicative longtail prediction task.We observe LLMs are more prone to failure in these cases.To address these limitations, we propose the use of entailment-contradiction prediction in conjunction with LLMs, which allows for strong performance in a strict zero-shot setting.Importantly, our method does not require any parameter updates, a resource-intensive process and achieves strong performance across multiple datasets.
Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion

Temporal Knowledge graph completion (TKGC) is a crucial task that involves reasoning at known timestamps to complete the missing part of facts and has attracted more and more attention in recent years.Most existing methods focus on learning representations based on graph neural networks while inaccurately extracting information from timestamps and insufficiently utilizing the implied information in relations.To address these problems, we propose a novel TKGC model, namely Pre-trained Language Model with Prompts for TKGC (PPT).We convert a series of sampled quadruples into pre-trained language model inputs and convert intervals between timestamps into different prompts to make coherent sentences with implicit semantic information.We train our model with a masking strategy to convert TKGC task into a masked token prediction task, which can leverage the semantic information in pre-trained language models.Experiments on three benchmark datasets and extensive analysis demonstrate that our model has great competitiveness compared to other models with four metrics.Our model can effectively incorporate information from temporal knowledge graphs into the language models.