Reviewerly/exHarmony huggingface dataset:
BlendFilter: Advancing Retrieval-Augmented Large Language Models via  Query Generation Blending and Knowledge Filtering

Authors:
raw_author_name: Wang, Haoyu
raw_author_name: Zhao, Tuo
raw_author_name: Gao, Jing

Abstract: Retrieval-augmented Large Language Models (LLMs) offer substantial benefits in enhancing performance across knowledge-intensive scenarios. However, these methods often face challenges with complex inputs and encounter difficulties due to noisy knowledge retrieval, notably hindering model effectiveness. To address this issue, we introduce BlendFilter, a novel approach that elevates retrieval-augmented LLMs by integrating query generation blending with knowledge filtering. BlendFilter proposes the blending process through its query generation method, which integrates both external and internal knowledge augmentation with the original query, ensuring comprehensive information gathering. Additionally, our distinctive knowledge filtering module capitalizes on the intrinsic capabilities of the LLM, effectively eliminating extraneous data. We conduct extensive experiments on three open-domain question answering benchmarks, and the findings clearly indicate that our innovative BlendFilter surpasses state-of-the-art baselines significantly.
ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs

Authors:
raw_author_name: Li, Yuhan
raw_author_name: Yu, Jeffrey Xu
raw_author_name: Li, Jia

Abstract: With the development of foundation models such as large language models, zero-shot transfer learning has become increasingly significant. This is highlighted by the generative capabilities of NLP models like GPT-4, and the retrieval-based approaches of CV models like CLIP, both of which effectively bridge the gap between seen and unseen data. In the realm of graph learning, the continuous emergence of new graphs and the challenges of human labeling also amplify the necessity for zero-shot transfer learning, driving the exploration of approaches that can generalize across diverse graph data without necessitating dataset-specific and label-specific fine-tuning. In this study, we extend such paradigms to zero-shot transferability in graphs by introducing ZeroG, a new framework tailored to enable cross-dataset generalization. Addressing the inherent challenges such as feature misalignment, mismatched label spaces, and negative transfer, we leverage a language model to encode both node attributes and class semantics, ensuring consistent feature dimensions across datasets. We also propose a prompt-based subgraph sampling module that enriches the semantic information and structure information of extracted subgraphs using prompting nodes and neighborhood aggregation, respectively. We further adopt a lightweight fine-tuning strategy that reduces the risk of overfitting and maintains the zero-shot learning efficacy of the language model. The results underscore the effectiveness of our model in achieving significant cross-dataset zero-shot transferability, opening pathways for the development of graph foundation models. Especially, ZeroG, as a zero-shot method, can even achieve results comparable to those of semi-supervised learning on Pubmed.
PromptKD: Distilling Student-Friendly Knowledge for Generative Language  Models via Prompt Tuning

Authors:
raw_author_name: Kim, Gyeongman
raw_author_name: Yang, Eunho

Abstract: Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, research on KD for generative language models like LLMs is relatively sparse, and the approach of distilling student-friendly knowledge, which has shown promising performance in KD for classification models, remains unexplored in generative language models. To explore this approach, we propose PromptKD, a simple yet effective method that utilizes prompt tuning - for the first time in KD - to enable generative language models to transfer student-friendly knowledge. Unlike previous works in classification that require fine-tuning the entire teacher model for extracting student-friendly knowledge, PromptKD achieves similar effects by adding a small number of prompt tokens and tuning only the prompt with student guidance. Extensive experiments on instruction-following datasets using the GPT-2 model family show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher's parameters as prompts. Further analysis suggests that distilling student-friendly knowledge alleviates exposure bias effectively throughout the entire training process, leading to performance enhancements.
A New Chaos-Based Image Encryption Algorithm Based on Discrete Fourier Transform and Improved Joseph Traversal

Authors:
raw_author_name: Mingxu Wang
raw_affiliation_string: School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
raw_author_name: Xianping Fu
raw_affiliation_string: School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
raw_author_name: Xiaopeng Yan
raw_affiliation_string: School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
raw_author_name: Lin Teng
raw_affiliation_string: School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China

Abstract: To further enhance the security of image encryption, a new chaos-based image encryption algorithm (IEA) based on discrete Fourier transform and Joseph traversal is proposed to encrypt the plain image in both the frequency domain and space domain simultaneously. In the proposed IEA, the logistic map is used to generate the appropriate chaotic sequence, and the improved Joseph traversal is used to scramble the image in different starting positions and variable step sizes. Then, block diffusion is performed at the end. The main finding concerning the proposed IEA is that the combination of discrete Fourier transform and Joseph traversal can enhance the security of the image information, which has been validated by measuring the performance in resisting the common types of attacks.
Target Speech Extraction with Pre-trained Self-supervised Learning  Models

Authors:
raw_author_name: Peng, Junyi
raw_author_name: Delcroix, Marc
raw_author_name: Ochiai, Tsubasa
raw_author_name: Plchot, Oldrich
raw_author_name: Araki, Shoko
raw_author_name: Cernocky, Jan

Abstract: Pre-trained self-supervised learning (SSL) models have achieved remarkable success in various speech tasks. However, their potential in target speech extraction (TSE) has not been fully exploited. TSE aims to extract the speech of a target speaker in a mixture guided by enrollment utterances. We exploit pre-trained SSL models for two purposes within a TSE framework, i.e., to process the input mixture and to derive speaker embeddings from the enrollment. In this paper, we focus on how to effectively use SSL models for TSE. We first introduce a novel TSE downstream task following the SUPERB principles. This simple experiment shows the potential of SSL models for TSE, but extraction performance remains far behind the state-of-the-art. We then extend a powerful TSE architecture by incorporating two SSL-based modules: an Adaptive Input Enhancer (AIE) and a speaker encoder. Specifically, the proposed AIE utilizes intermediate representations from the CNN encoder by adjusting the time resolution of CNN encoder and transformer blocks through progressive upsampling, capturing both fine-grained and hierarchical features. Our method outperforms current TSE systems achieving a SI-SDR improvement of 14.0 dB on LibriMix. Moreover, we can further improve performance by 0.7 dB by fine-tuning the whole model including the SSL model parameters.
Probing Self-supervised Learning Models with Target Speech Extraction

Authors:
raw_author_name: Peng, Junyi
raw_author_name: Delcroix, Marc
raw_author_name: Ochiai, Tsubasa
raw_author_name: Plchot, Oldrich
raw_author_name: Ashihara, Takanori
raw_author_name: Araki, Shoko
raw_author_name: Cernocky, Jan

Abstract: Large-scale pre-trained self-supervised learning (SSL) models have shown remarkable advancements in speech-related tasks. However, the utilization of these models in complex multi-talker scenarios, such as extracting a target speaker in a mixture, is yet to be fully evaluated. In this paper, we introduce target speech extraction (TSE) as a novel downstream task to evaluate the feature extraction capabilities of pre-trained SSL models. TSE uniquely requires both speaker identification and speech separation, distinguishing it from other tasks in the Speech processing Universal PERformance Benchmark (SUPERB) evaluation. Specifically, we propose a TSE downstream model composed of two lightweight task-oriented modules based on the same frozen SSL model. One module functions as a speaker encoder to obtain target speaker information from an enrollment speech, while the other estimates the target speaker's mask to extract its speech from the mixture. Experimental results on the Libri2mix datasets reveal the relevance of the TSE downstream task to probe SSL models, as its performance cannot be simply deduced from other related tasks such as speaker verification and separation.
Analysis of Using Sigmoid Loss for Contrastive Learning

Authors:
raw_author_name: Sohn, Jy-yong

Abstract: Contrastive learning has emerged as a prominent branch of self-supervised learning for several years. Especially, CLIP, which applies contrastive learning to large sets of captioned images, has garnered significant attention. Recently, SigLIP, a variant of CLIP, has been proposed, which uses the sigmoid loss instead of the standard InfoNCE loss. SigLIP achieves the performance comparable to CLIP in a more efficient manner by eliminating the need for a global view. However, theoretical understanding of using the sigmoid loss in contrastive learning is underexplored. In this paper, we provide a theoretical analysis of using the sigmoid loss in contrastive learning, in the perspective of the geometric structure of learned embeddings. First, we propose the double-Constant Embedding Model (CCEM), a framework for parameterizing various well-known embedding structures by a single variable. Interestingly, the proposed CCEM is proven to contain the optimal embedding with respect to the sigmoid loss. Second, we mathematically analyze the optimal embedding minimizing the sigmoid loss for contrastive learning. The optimal embedding ranges from simplex equiangular-tight-frame to antipodal structure, depending on the temperature parameter used in the sigmoid loss. Third, our experimental results on synthetic datasets coincide with the theoretical results on the optimal embedding structures.
Quantum Embedding with Transformer for High-dimensional Data

Authors:
raw_author_name: Chen, Hao-Yuan
raw_author_name: Chang, Yen-Jui
raw_author_name: Liao, Shih-Wei
raw_author_name: Chang, Ching-Ray

Abstract: Quantum embedding with transformers is a novel and promising architecture for quantum machine learning to deliver exceptional capability on near-term devices or simulators. The research incorporated a vision transformer (ViT) to advance quantum significantly embedding ability and results for a single qubit classifier with around 3 percent in the median F1 score on the BirdCLEF-2021, a challenging high-dimensional dataset. The study showcases and analyzes empirical evidence that our transformer-based architecture is a highly versatile and practical approach to modern quantum machine learning problems.
Me LLaMA: Foundation Large Language Models for Medical Applications

Authors:
raw_author_name: Xie, Qianqian
raw_author_name: Chen, Qingyu
raw_author_name: Chen, Aokun
raw_author_name: Peng, Cheng
raw_author_name: Huang, Jimin
raw_author_name: Zhang, Jeffrey
raw_author_name: He, Huan
raw_author_name: Wu, Yonghui
raw_author_name: Xu, Hua
raw_author_name: Bian, Jiang

Abstract: Recent large language models (LLMs) like ChatGPT and LLaMA have shown great promise in many AI applications. However, their performance on medical tasks is suboptimal and can be further improved by training on large domain-specific datasets. This study introduces Me LLaMA, a medical LLM family including foundation models - Me LLaMA 13/70B and their chat-enhanced versions - Me LLaMA 13/70B-chat, developed through the continual pre-training and instruction tuning of LLaMA2 using large medical data. Our domain-specific data suite for training and evaluation, includes a large-scale continual pre-training dataset with 129B tokens, an instruction tuning dataset with 214k samples, and a medical evaluation benchmark (MIBE) across six tasks with 14 datasets. Our extensive evaluation using MIBE shows that Me LLaMA models surpass existing open-source medical LLMs in zero-shot and few-shot learning and outperform commercial giants like ChatGPT on 6 out of 8 datasets and GPT-4 in 3 out of 8 datasets. In addition, we empirically investigated the catastrophic forgetting problem, and our results show that Me LLaMA models outperform other medical LLMs. Me LLaMA is one of the first and largest open-source foundational LLMs designed for the medical domain, using both biomedical and clinical data. It exhibits superior performance across both general and medical tasks compared to other medical LLMs, rendering it an attractive choice for medical AI applications. All resources are available at: https://github.com/BIDS-Xu-Lab/Me-LLaMA.
Efektivitas Layanan Informasi dengan Format Klasikal Menggunakan Metode Small Group Discussion untuk Meningkatkan Self-Efficacy Akademik Siswa SMA Negeri 10 Padang

Authors:
raw_author_name: Ahmad Wahid
raw_author_name: Neviyarni Suhaili

Abstract: Academic self-efficacy is one of the many factors that can affect student learning achievement. Someone who has good academic self-efficacy, will feel that he is always ready and alert in completing his academic tasks without any doubts about his condition. in completing their academic tasks without any doubts about their own condition. This study aims to see the effectiveness of the implementation of information services with a classical format using the small group discussion method to improve students' academic self-efficacy. This research uses a quantitative approach, with the type of research Quasi Experiment design The Non-Equivalent Control Group. The population in this study were Phase E students of SMA Negeri 10 Padang, sampling using purposive sampling method, then obtained samples of Phase E 4 and Phase E 9 classes. The technique used in data collection is by distributing questionnaires, then analyzed using the Wilcoxon Signed Ranks Test, and Mann-Whitney U 2 Independent Sample Test. The results of this study indicate that the implementation of information services with a classical format using the small group discussion method is effective in increasing students' academic self-efficacy. Specifically, the use of this method increases better academic self-efficacy.
ELAD: Explanation-Guided Large Language Models Active Distillation

Authors:
raw_author_name: Zhang, Yifei
raw_author_name: Pan, Bo
raw_author_name: Ling, Chen

Abstract: The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve efficient sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model's reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLM knowledge distillation.
General multi-attack detection for continuous-variable quantum key distribution with local local oscillator

Authors:
raw_author_name: Wei-Qi Liu
raw_author_name: Jin Qi
raw_author_name: Chen He

Abstract: Abstract Continuous-variable quantum key distribution with a local local oscillator(LLO CVQKD) has been extensively researched due to its simplicity and security. For the practical security of the LLO CVQKD system, there are two main attack modes referred to as reference pulse attack and polarization attack presently. However, there is currently no general defense strategy against such attacks and the security of the system needs further investigation. Here, we employ a deep learning framework called generative adversarial networks (GANs) to detect both attacks. We first analyze the data in different cases, derive a feature vector as input to a GAN model, and then show the training and testing process of the GAN model for attack classification. The proposed model has two parts, a discriminator and a generator, both of which employ a convolutional neural network (CNN) to improve accuracy. Simulation results show that the proposed scheme can detect and classify attacks without reducing the secret key rate and the maximum transmission distance. It only establishes a detection model by monitoring features of the pulse without adding additional devices.
A Bibliometric Analysis of Industry-Education Integration Research Based on Web of Science

Authors:
raw_author_name: Fang Huang
raw_affiliation_string: Xiangsihu College of Guangxi Minzu University, Nanning, China
raw_author_name: Yanxin Ye
raw_affiliation_string: Guangxi Modern Polytechnic College, Hechi, China

Abstract: This study aims to gain a deeper understanding of the research dynamics and trends in industry-education integration through a quantitative analysis of the literature in this field. Eight hundred ninety-five relevant articles were retrieved from the Web of Science Core Collection database, and the research focused on the annual publication trends, prominent authors, institutions and countries, highly cited articles, keywords, and collaborative networks. The study found that the field of industry-education integration has undergone three stages. Secondly, researchers from China and the United States ranked first and second in the number of publications in this field. Thirdly, highly cited articles have a particular influence on industry-education integration, but the overall citation count is relatively limited. Fourthly, keywords such as “cooperative education,” “students,” and “performance” are hot topics in this field. Finally, the analysis of collaborative networks revealed the existence of multiple communities, but the overall scale of collaboration is relatively small, and the influence is limited. This study provides profound insights into the academic research in industry-education integration. It helps scholars, policymakers, and practitioners better understand this field’s development trends.
Supplemental Material: Genetic association between carbonates and gold precipitation mechanisms in the Jinshan deposit, eastern Jiangnan orogen

Authors:
raw_author_name: Qingxiang Wang
raw_author_name: Teng Deng
raw_author_name: None et al.

Abstract: &lt;p&gt;Thermodynamic database for Geochemist's Workbench (GWB).&lt;/p&gt;
GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a  Gradient-Aware Mask and Semantic Constraints

Authors:
raw_author_name: Yang, Zhiyuan
raw_author_name: Zhu, Haiyue
raw_author_name: Mao, Kezhi

Abstract: Self-supervised depth estimation has evolved into an image reconstruction task that minimizes a photometric loss. While recent methods have made strides in indoor depth estimation, they often produce inconsistent depth estimation in textureless areas and unsatisfactory depth discrepancies at object boundaries. To address these issues, in this work, we propose GAM-Depth, developed upon two novel components: gradient-aware mask and semantic constraints. The gradient-aware mask enables adaptive and robust supervision for both key areas and textureless regions by allocating weights based on gradient magnitudes.The incorporation of semantic constraints for indoor self-supervised depth estimation improves depth discrepancies at object boundaries, leveraging a co-optimization network and proxy semantic labels derived from a pretrained segmentation model. Experimental studies on three indoor datasets, including NYUv2, ScanNet, and InteriorNet, show that GAM-Depth outperforms existing methods and achieves state-of-the-art performance, signifying a meaningful step forward in indoor depth estimation. Our code will be available at https://github.com/AnqiCheng1234/GAM-Depth.
Global Safe Sequential Learning via Efficient Knowledge Transfer

Authors:
raw_author_name: Duennbier, Olaf
raw_author_name: Toussaint, Marc
raw_author_name: Rakitsch, Barbara
raw_author_name: Zimmer, Christoph

Abstract: Sequential learning methods such as active learning and Bayesian optimization select the most informative data to learn about a task. In many medical or engineering applications, the data selection is constrained by a priori unknown safety conditions. A promissing line of safe learning methods utilize Gaussian processes (GPs) to model the safety probability and perform data selection in areas with high safety confidence. However, accurate safety modeling requires prior knowledge or consumes data. In addition, the safety confidence centers around the given observations which leads to local exploration. As transferable source knowledge is often available in safety critical experiments, we propose to consider transfer safe sequential learning to accelerate the learning of safety. We further consider a pre-computation of source components to reduce the additional computational load that is introduced by incorporating source data. In this paper, we theoretically analyze the maximum explorable safe regions of conventional safe learning methods. Furthermore, we empirically demonstrate that our approach 1) learns a task with lower data consumption, 2) globally explores multiple disjoint safe regions under guidance of the source knowledge, and 3) operates with computation comparable to conventional safe learning methods.
Unveiling Linguistic Regions in Large Language Models

Authors:
raw_author_name: Zhang, Zhihao
raw_author_name: Zhao, Jun
raw_author_name: Zhang, Qi
raw_author_name: Gui, Tao
raw_author_name: Huang, Xuanjing

Abstract: Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs' cross-lingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve cross-lingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependency, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct regions exist for different monolingual families, and disruption to these specific regions substantially reduces the LLMs' proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common occurrence observed during further pre-training of LLMs. Overall, exploring the LLMs' functional regions provides insights into the foundation of their intelligence.