Reviewerly/exHarmony huggingface dataset:
Understanding Differential Search Index for Text Retrieval

The Differentiable Search Index (DSI) is a novel information retrieval (IR) framework that utilizes a differentiable function to generate a sorted list of document identifiers in response to a given query.However, due to the black-box nature of the end-to-end neural architecture, it remains to be understood to what extent DSI possesses the basic indexing and retrieval abilities.To mitigate this gap, in this study, we define and examine three important abilities that a functioning IR framework should possess, namely, exclusivity, completeness, and relevance ordering.Our analytical experimentation shows that while DSI demonstrates proficiency in memorizing the unidirectional mapping from pseudo queries to document identifiers, it falls short in distinguishing relevant documents from random ones, thereby negatively impacting its retrieval effectiveness.To address this issue, we propose a multi-task distillation approach to enhance the retrieval quality without altering the structure of the model and successfully endow it with improved indexing abilities.Through experiments conducted on various datasets, we demonstrate that our proposed method outperforms previous DSI baselines 1 .
Teaching Small Language Models to Reason

Chain of thought prompting successfully improves the reasoning capabilities of large language models, achieving state of the art results on a range of datasets.However, these reasoning capabilities only appear to emerge in models with at least tens of billions of parameters.In this paper, we explore the transfer of such reasoning capabilities to smaller models via knowledge distillation, also investigating model and dataset size trade-off.Specifically, we finetune a student model on the chain of thought outputs generated by a larger teacher model.Our experiments show that the proposed method improves task performance across arithmetic, commonsense and symbolic reasoning datasets.For example, the accuracy of T5 XXL on GSM8K improves from 8.11% to 21.99% and 18.42% when finetuned on PaLM 540B and GPT-3 175B generated chains of thought, respectively.
WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models

This paper describes our submission to the MEDIQA-Chat 2023 shared task for automatic clinical note generation from doctor-patient conversations.We report results for two approaches: the first fine-tunes a pre-trained language model (PLM) on the shared task data, and the second uses few-shot in-context learning (ICL) with a large language model (LLM).Both achieve high performance as measured by automatic metrics (e.g.ROUGE, BERTScore) and ranked second and first, respectively, of all submissions to the shared task.Expert human scrutiny indicates that notes generated via the ICL-based approach with GPT-4 are preferred about as often as human-written notes, making it a promising path toward automated note generation from doctor-patient conversations.
Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information.However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs.To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM).To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics.This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue.Extensive experimental results show that our TSCTM outperforms state-ofthe-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions. 1
GHAN: Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval

Text-video retrieval focuses on two aspects: cross-modality interaction and video-language encoding.Currently, the mainstream approach is to train a joint embedding space for multimodal interactions.However, there are structural and semantic differences between text and video, making this approach challenging for fine-grained understanding.In order to solve this, we propose an end-to-end graph-based hierarchical aggregation network for text-video retrieval according to the hierarchy possessed by text and video.We design a token-level weighted network to refine intra-modality representations and construct a graph-based message passing attention network for global-local alignment across modality.We conduct experiments on the public datasets MSR-VTT-9K, MSR-VTT-7K and MSVD, and achieve Recall@1 of 73.0%, 65.6%, and 64.0% , which is 25.7%, 16.5%, and 14.2% better than the current state-of-the-art model.
CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation

Practical dialog systems need to deal with various knowledge sources, noisy user expressions, and the shortage of annotated data.To better solve the above problems, we propose CGoDial 1 , a new challenging and comprehensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation.It contains 96,763 dialog sessions, and 574,949 dialog turns totally, covering three datasets with different knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with treeformed knowledge, and a retrieval-based dialog (RBD) dataset with candidate-formed knowledge.To bridge the gap between academic benchmarks and spoken dialog scenarios, we either collect data from real conversations or add spoken features to existing datasets via crowdsourcing.The proposed experimental settings include the combinations of training with either the entire training set or a few-shot training set, and testing with either the standard test set or a hard test subset, which can assess model capabilities in terms of general prediction, fast adaptability and reliable robustness.
Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding

Despite the great success of spoken language understanding (SLU) in high-resource languages, it remains challenging in low-resource languages mainly due to the lack of labeled training data.The recent multilingual codeswitching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zeroshot cross-lingual SLU.However, current codeswitching methods are limited to implicit alignment and disregard the inherent semantic structure in SLU, i.e., the hierarchical inclusion of utterances, slots, and words.In this paper, we propose to model the utterance-slot-word structure by a multi-level contrastive learning framework at the utterance, slot, and word levels to facilitate explicit alignment.Novel codeswitching schemes are introduced to generate hard negative examples for our contrastive learning framework.Furthermore, we develop a label-aware joint model leveraging label semantics to enhance the implicit alignment and feed to contrastive learning.Our experimental results show that our proposed methods significantly improve the performance compared with the strong baselines on two zero-shot crosslingual SLU benchmark datasets.
TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Base

Pre-trained language models (PLMs) have shown their effectiveness in multiple scenarios.However, KBQA remains challenging, especially regarding coverage and generalization settings.This is due to two main factors: i) understanding the semantics of both questions and relevant knowledge from the KB; ii) generating executable logical forms with both semantic and syntactic correctness.In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items.Moreover, constrained decoding is used to control the output space and reduce generation errors.Experiments over important benchmarks demonstrate the effectiveness of our approach.TIARA outperforms previous SOTA, including those using PLMs or oracle entity annotations, by at least 4.1 and 1.1 F1 points on GrailQA and WebQuestionsSP, respectively.Specifically on GrailQA, TIARA outperforms previous models in all categories, with an improvement of 4.7 F1 points in zero-shot generalization. 1
Power output forecasting of solar photovoltaic plant using LSTM

Renewable energy sources are gaining popularity, where solar photovoltaics (PV) being the most preferred option due to its cleanliness, affordability, and abundance. The energy output of solar PV is primarily based on temperature & irradiance. Therefore, a weather-based intelligent model is needed for estimating solar energy output to fulfil energy demand and decision making. Predicting PV power output is essential for energy management, security, and operation. In addition to enhancing the output efficiency of PV power plants, the power grid's stability can be enhanced by enhancing the efficacy of PV power plants' electricity generation. This work focuses on LSTM and BPNN for forecasting solar plant power output and it is observed that their findings are virtually compatible with realistic power production in terms of MAE, MAPE, RMSPE, and R2 score. LSTM model comparisons with different layers for each weather season are also analysed. Comparing the extent of errors in the LSTM and BPNN models reveals that LSTM provides more accurate predictions.
Compositional Zero-Shot Artistic Font Synthesis

Recently, many researchers have made remarkable achievements in the field of artistic font synthesis, with impressive glyph style and effect style in the results. However, due to less exploration in style disentanglement, it is difficult for existing methods to envision a kind of unseen style (glyph-effect) compositions of artistic font, and thus can only learn the seen style compositions. To solve this problem, we propose a novel compositional zero-shot artistic font synthesis gan (CAFS-GAN), which allows the synthesis of unseen style compositions by exploring the visual independence and joint compatibility of encoding semantics between glyph and effect. Specifically, we propose two contrast-based style encoders to achieve style disentanglement due to glyph and effect intertwining in the image. Meanwhile, to preserve more glyph and effect detail, we propose a generator based on hierarchical dual styles AdaIN to reorganize content-styles representations from structure to texture gradually. Extensive experiments demonstrate the superiority of our model in generating high-quality artistic font images with unseen style compositions against other state-of-the-art methods. The source code and data is available at moonlight03.github.io/CAFS-GAN/.
TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition

Text irregularities pose significant challenges to scene text recognizers. Thin-Plate Spline (TPS)-based rectification is widely regarded as an effective means to deal with them. Currently, the calculation of TPS transformation parameters purely depends on the quality of regressed text borders. It ignores the text content and often leads to unsatisfactory rectified results for severely distorted text. In this work, we introduce TPS++, an attention-enhanced TPS transformation that incorporates the attention mechanism to text rectification for the first time. TPS++ formulates the parameter calculation as a joint process of foreground control point regression and content-based attention score estimation, which is computed by a dedicated designed gated-attention block. TPS++ builds a more flexible content-aware rectifier, generating a natural text correction that is easier to read by the subsequent recognizer. Moreover, TPS++ shares the feature backbone with the recognizer in part and implements the rectification at feature-level rather than image-level, incurring only a small overhead in terms of parameters and inference time. Experiments on public benchmarks show that TPS++ consistently improves the recognition and achieves state-of-the-art accuracy. Meanwhile, it generalizes well on different backbones and recognizers. Code is at https://github.com/simplify23/TPS_PP.
Female participation in agriculture and economic development in 33 African Countries.

Women occupy a crucial position in the agricultural sector because; they participate in different forms, as entrepreneurs, labourers, and marketers among others. Despite the various responsibilities' women are involved in, such as family and child-care among others; they contribute up to 40% of agricultural GDP. Against this background, this study examined how female participation in agriculture contributes to economic development in selected African countries, which is in line with the United Nations (UN) 2030 Sustainable Development Goals (SDGs 5 & 8; to ensure gender equality, decent work and economic growth respectively). The study engaged a panel data of selected 33 African countries sourced from the World Development Indicators (WDI) and the Human Development Index (HDI) for the period of 2000 - 2018. The study applied the Pooled Ordinary Least Squares (POLS) and the fixed effects method based on Hausman specification result. Findings from the POLS and fixed effect were consistent across sub-regions, showing that, female participation in agriculture, though statistically significant, was negatively related to economic development. This means that a 1% increase in female participation in agriculture may reduce economic development by between 0.06% and 2.7%. On the other hand, across model, female education was found to be statistically significant and positively related to economic development. On the average, increased level of female education may increase economic development by 1.71%. The implication of this is that an increase in female participation in agriculture without the required level of education and training and access to agricultural materials may have a negative impact on economic development. Therefore, the study recommended that female education and training are required to improve economic development in Africa.
Look Before You Match: Instance Understanding Matters in Video Object Segmentation

Exploring dense matching between the current frame and past frames for long-range context modeling, memory-based methods have demonstrated impressive results in video object segmentation (VOS) recently. Nevertheless, due to the lack of instance understanding ability, the above approaches are oftentimes brittle to large appearance variations or viewpoint changes resulted from the movement of objects and cameras. In this paper, we argue that instance understanding matters in VOS, and integrating it with memory-based matching can enjoy the synergy, which is intuitively sensible from the definition of VOS task, i.e., identifying and segmenting object instances within the video. Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank. We employ the well-learned object queries from IS branch to inject instance-specific information into the query key, with which the instance-augmented matching is further performed. In addition, we introduce a multi-path fusion block to effectively combine the memory readout with multi-scale features from the instance segmentation decoder, which incorporates high-resolution instance-aware features to produce final segmentation results. Our method achieves state-of-the-art performance on DAVIS 2016/2017 val (92.6% and 87.1%), DAVIS 2017 test-dev (82.8%), and YouTube-VOS 2018/2019 val (86.3% and 86.3%), outperforming alternative methods by clear margins.
ResFormer: Scaling ViTs with Multi-Resolution Training

Vision Transformers (ViTs) have achieved overwhelming success, yet they suffer from vulnerable resolution scalability, i.e., the performance drops drastically when presented with input resolutions that are unseen during training. We introduce, ResFormer, a framework that is built upon the seminal idea of multi-resolution training for improved performance on a wide spectrum of, mostly unseen, testing resolutions. In particular, ResFormer operates on replicated images of different resolutions and enforces a scale consistency loss to engage interactive information across different scales. More importantly, to alternate among varying resolutions effectively, especially novel ones in testing, we propose a global-local positional embedding strategy that changes smoothly conditioned on input sizes. We conduct extensive experiments for image classification on ImageNet. The results provide strong quantitative evidence that ResFormer has promising scaling abilities towards a wide range of resolutions. For instance, ResFormer-B-MR achieves a Top-1 accuracy of 75.86% and 81.72% when evaluated on relatively low and high resolutions respectively (i.e., 96 and 640), which are 48% and 7.49% better than DeiT-B. We also demonstrate, moreover, ResFormer is flexible and can be easily extended to semantic segmentation, object detection and video action recognition.
WINNER: Weakly-supervised hIerarchical decompositioN and aligNment for spatio-tEmporal video gRounding

Spatio-temporal video grounding aims to localize the aligned visual tube corresponding to a language query. Existing techniques achieve such alignment by exploiting dense boundary and bounding box annotations, which can be prohibitively expensive. To bridge the gap, we investigate the weakly-supervised setting, where models learn from easily accessible video-language data without annotations. We identify that intra-sample spurious correlations among video-language components can be alleviated if the model captures the decomposed structures of video and language data. In this light, we propose a novel framework, namely WINNER, for hierarchical video-text understanding. WINNER first builds the language decomposition tree in a bottom-up manner, upon which the structural attention mechanism and top-down feature backtracking jointly build a multi-modal decomposition tree, permitting a hierarchical understanding of unstructured videos. The multi-modal decomposition tree serves as the basis for multi-hierarchy language-tube matching. A hierarchical contrastive learning objective is proposed to learn the multi-hierarchy correspondence and distinguishment with intra-sample and inter-sample video-text decomposition structures, achieving video-language decomposition structure alignment. Extensive experiments demonstrate the rationality of our design and its effectiveness beyond state-of-the-art weakly supervised methods, even some supervised methods.
MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers

In this paper, we propose Mixed and Masked AutoEncoder (MixMAE), a simple but efficient pretraining method that is applicable to various hierarchical Vision Transformers. Existing masked image modeling (MIM) methods for hierarchical Vision Transformers replace a random subset of input tokens with a special [MASK] symbol and aim at reconstructing original image tokens from the corrupted image. However, we find that using the [MASK] symbol greatly slows down the training and causes pretraining-finetuning inconsistency, due to the large masking ratio (e.g., 60% in SimMIM). On the other hand, MAE does not introduce [MASK] tokens at its encoder at all but is not applicable for hierarchical Vision Transformers. To solve the issue and accelerate the pretraining of hierarchical models, we replace the masked tokens of one image with visible tokens of another image, i.e., creating a mixed image. We then conduct dual reconstruction to reconstruct the two original images from the mixed input, which significantly improves efficiency. While MixMAE can be applied to various hierarchical Transformers, this paper explores using Swin Transformer with a large window size and scales up to huge model size (to reach 600M parameters). Empirical results demonstrate that MixMAE can learn high-quality visual representations efficiently. Notably, MixMAE with Swin-B/W14 achieves 85.1% top-1 accuracy on ImageNet-1K by pretraining for 600 epochs. Besides, its transfer performances on the other 6 datasets show that MixMAE has better FLOPs / performance tradeoff than previous popular MIM methods.
Neural superstatistics for Bayesian estimation of dynamic cognitive models

Abstract Mathematical models of cognition are often memoryless and ignore potential fluctuations of their parameters. However, human cognition is inherently dynamic. Thus, we propose to augment mechanistic cognitive models with a temporal dimension and estimate the resulting dynamics from a superstatistics perspective. Such a model entails a hierarchy between a low-level observation model and a high-level transition model. The observation model describes the local behavior of a system, and the transition model specifies how the parameters of the observation model evolve over time. To overcome the estimation challenges resulting from the complexity of superstatistical models, we develop and validate a simulation-based deep learning method for Bayesian inference, which can recover both time-varying and time-invariant parameters. We first benchmark our method against two existing frameworks capable of estimating time-varying parameters. We then apply our method to fit a dynamic version of the diffusion decision model to long time series of human response times data. Our results show that the deep learning approach is very efficient in capturing the temporal dynamics of the model. Furthermore, we show that the erroneous assumption of static or homogeneous parameters will hide important temporal information.
Non-Gaussian-state generation with time-gated photon detection

Non-Gaussian states of light, which are essential in fault-tolerant and universal optical quantum computation, are typically generated by a heralding scheme using photon detectors. Recently, it has been theoretically shown that the large timing jitter of the photon detectors deteriorates the purity of the generated non-Gaussian states [T. Sonoyama et al., Phys. Rev. A 105, 043714 (2022)]. Here, we circumvent this problem by time-gated photon detection and generate non-Gaussian states with Wigner negativity using continuous-wave light. We use a fast optical switch for time gating to effectively improve the timing jitter of a photon-number-resolving detector based on a transition edge sensor from 50 ns to 10 ns. As a result, we generate non-Gaussian states with Wigner negativity of \ensuremath{-}$0.011\ifmmode\pm\else\textpm\fi{}0.004$ $(\ensuremath{\hbar}=1)$, which cannot be observed without the time-gated photon detection method. These results confirm the effect of the timing jitter on non-Gaussian state generation experimentally for the first time. The proposed time-gated photon detection method would be useful for high-purity non-Gaussian state preparation in optical quantum information processing.