Reviewerly/exHarmony huggingface dataset:
Path-Specific Counterfactual Fairness for Recommender Systems

Recommender systems (RSs) have become an indispensable part of online platforms. With the growing concerns of algorithmic fairness, RSs are not only expected to deliver high-quality personalized content, but are also demanded not to discriminate against users based on their demographic information. However, existing RSs could capture undesirable correlations between sensitive features and observed user behaviors, leading to biased recommendations. Most fair RSs tackle this problem by completely blocking the influences of sensitive features on recommendations. But since sensitive features may also affect user interests in a fair manner (e.g., race on culture-based preferences), indiscriminately eliminating all the influences of sensitive features inevitably degenerate the recommendations quality and necessary diversities. To address this challenge, we propose a path-specific fair RS (PSF-RS) for recommendations. Specifically, we summarize all fair and unfair correlations between sensitive features and observed ratings into two latent proxy mediators, where the concept of path-specific bias (PS-Bias) is defined based on path-specific counterfactual inference. Inspired by Pearl's minimal change principle, we address the PS-Bias by minimally transforming the biased factual world into a hypothetically fair world, where a fair RS model can be learned accordingly by solving a constrained optimization problem. For the technical part, we propose a feasible implementation of PSF-RS, i.e., PSF-VAE, with weakly-supervised variational inference, which robustly infers the latent mediators such that unfairness can be mitigated while necessary recommendation diversities can be maximally preserved simultaneously. Experiments conducted on semi-simulated and real-world datasets demonstrate the effectiveness of PSF-RS.
Domain-Guided Spatio-Temporal Self-Attention for Egocentric 3D Pose Estimation

Vision-based ego-centric 3D human pose estimation (ego-HPE) is essential to support critical applications of xR-technologies. However, severe self-occlusions and strong distortion introduced by the fish-eye view from the head mounted camera, make ego-HPE extremely challenging. To address these challenges, we propose a domain-guided spatio-temporal transformer model that leverages information specific to ego-views. Powered by this domain-guided transformer, we build Egocentric Spatio-Temporal Self-Attention Network (Ego-STAN), which uses 2D image representations and spatio-temporal attention to address both distortions and self-occlusions in ego-HPE. Additionally, we introduce a spatial concept called feature map tokens (FMT) which endows Ego-STAN with the ability to draw complex spatio-temporal information encoded in ego-centric videos. Our quantitative evaluation on the contemporary xR-EgoPose dataset, achieves a 38.2% improvement on the highest error joints against the SOTA ego-HPE model, while accomplishing a 22% decrease in the number of parameters. Finally, we also demonstrate the generalization capabilities of our model to real-world HPE tasks beyond ego-views achieving 7.7% improvement on 2D human pose estimation with the Human3.6M dataset. Our code is also made available at: https://github.com/jmpark0808/Ego-STAN
Certified Edge Unlearning for Graph Neural Networks

The emergence of evolving data privacy policies and regulations has sparked a growing interest in the concept of "machine unlearning", which involves enabling machine learning models to forget specific data instances. In this paper, we specifically focus on edge unlearning in Graph Neural Networks (GNNs), which entails training a new GNN model as if certain specified edges never existed in the original training graph. Unlike conventional unlearning scenarios where data samples are treated as independent entities, edges in graphs exhibit correlation. Failing to carefully account for this data dependency would result in the incomplete removal of the requested data from the model. While retraining the model from scratch by excluding the specific edges can eliminate their influence, this approach incurs a high computational cost. To overcome this challenge, we introduce CEU, a Certified Edge Unlearning framework. CEU expedites the unlearning process by updating the parameters of the pre-trained GNN model in a single step, ensuring that the update removes the influence of the removed edges from the model. We formally prove that CEU offers a rigorous theoretical guarantee under the assumption of convexity on the loss function. Our empirical analysis further demonstrates the effectiveness and efficiency of CEU for both linear and deep GNNs - it achieves significant speedup gains compared to retraining and existing unlearning methods while maintaining comparable model accuracy to retraining from scratch.
Learning to Schedule in Diffusion Probabilistic Models

Recently, the field of generative models has seen a significant advancement with the introduction of Diffusion Probabilistic Models (DPMs). The Denoising Diffusion Implicit Model (DDIM) was designed to reduce computational time by skipping a number of steps in the inference process of DPMs. However, the hand-crafted sampling schedule in DDIM, which relies on human expertise, has its limitations in considering all relevant factors in the sampling process. Additionally, the assumption that all instances should have the same schedule is not always valid. To address these problems, this paper proposes a method that leverages reinforcement learning to automatically search for an optimal sampling schedule for DPMs. This is achieved by a policy network that predicts the next step to visit based on the current state of the noisy image. The optimization of the policy network is accomplished using an episodic actor-critic framework, which incorporates reinforcement learning. Empirical results demonstrate the superiority of our approach over various datasets with different timesteps. We also observe that the trained sampling schedule has a strong generalization ability across different DPM baselines.
Single image noise level estimation by artificial noise

In this paper, we propose a new method for noise level estimation from a single noisy image contaminated by additive white Gaussian noise and gamma noise, which usually appear due to different physical imaging systems. Our method is motivated by the key observation that the curves of small eigenvalues of the patch covariance matrices corresponding to the original noisy image and the artificial noisy image are nearly parallel. Based on some assumptions, we derive a proportional relation equation between eigenvalues and variances of the noisy images in the redundant dimensions. The solution of this equation gives the basic noise estimation formula. However, the assumptions are not strictly satisfied for natural images with textures and fine details. To overcome this drawback, we modified the basic noise estimation formula by introducing a correction parameter. The proposed method is also extended to multiplicative Gamma noise estimation. Numerical results demonstrate that the proposed noise level estimation method outperforms the existing state-of-the-art approaches in accuracy for various scenes. In particular, the decreasing of Root Mean Square Error (RMSE) is an average of over 10% for the testing databases. We further demonstrate that the proposed noise estimation method can be used for blind denoising.
Recognition of multivariate geochemical anomalies using a geologically-constrained variational autoencoder network with spectrum separable module – A case study in Shangluo District, China

This study has developed a novel variational autoencoder architecture by incorporating the spectrum separable module, termed SSM-VAE, so as to recognize the multi-mineral-species geochemical patterns over the Shangluo district, central China, and then facilitate a better understanding of the regional metallogenesis. The primary advantage of SSM-VAE is its ability to explore the interlayer correlations and then integrate them into the reconstructed features, which greatly improved the geological interpretability of the model products. In the training process, a hybrid loss function with a geologically-constrained term derived from the fractal analysis was designed to guide the model to focus on anomalies related to mineralization. Afterward, the factor analysis together with the EM-MML thresholding algorithm were integrated as a post-processing tool. Finally, the mineral-spot identification rate (δ) versus the size of the anomalous area (S) are used to validate the ore-bearing potential of different anomaly divisions. Comparing to other state-of-the-art models without regard to the interlayer correlations, SSM-VAE can produce well-zoned, mineral species-specific, and more metalliferous anomalous patches. To be specific, we separated out 10 zoned anomaly divisions, and on average, when S = 1% there are 8.5013 mineral spots falling within.
Perturbation-Based Self-Supervised Attention for Attention Bias in Text Classification

In text classification, the traditional attention mechanisms usually focus too much on frequent words, and need extensive labeled data in order to learn. This paper proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overhead. Specifically, we add as much noise as possible to all the words in the sentence without changing their semantics and predictions. We hypothesize that words that tolerate more noise are less significant, and we can use this information to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly improve the performance of current attention-based models, and is more effective than existing self-supervised methods. We also provide a visualization analysis to verify the effectiveness of our approach.
Physical Time-Varying Transfer Function as Generic Low-Overhead Power-SCA Countermeasure

Mathematically secure cryptographic algorithms leak significant side-channel information through their power supplies when implemented on a physical platform. These side-channel leakages can be exploited by an attacker to extract the secret key of an embedded device. The existing state-of-the-art countermeasures mainly focus on power balancing, gate-level masking, or signal-to-noise (SNR) reduction using noise injection and signature attenuation, all of which suffer either from the limitations of high power/area overheads, throughput degradation or are not synthesizable. In this article, we propose a generic low-overhead digital-friendly power SCA countermeasure utilizing a physical Time-Varying Transfer Function (TVTF) by randomly shuffling distributed switched capacitors to significantly obfuscate the traces in the time domain. We evaluate our proposed technique utilizing a Matlab-based system-level simulation. Finally, we implement a 65nm CMOS prototype IC and evaluate our technique against power side-channel attacks (SCA). System-level simulation results of the TVTF-AES show ~5000× minimum traces to disclosure (MTD) improvement over the unprotected implementation with ~1.25× power and ~1.2× area overheads, and without any performance degradation. SCA evaluation with the prototype IC shows 3.4M MTD which is 500× greater than the unprotected solution.
Parallel Synthesis for Autoregressive Speech Generation

Autoregressive neural vocoders have achieved outstanding performance and are widely used in speech synthesis tasks such as text-to-speech and voice conversion. An autoregressive vocoder predicts a sample at some time step conditioned on those at previous time steps. Though it can generate highly natural human speech, the iterative generation inevitably makes the synthesis time proportional to the utterance length, leading to low efficiency. Many works were dedicated to generating the whole speech time sequence in parallel and then proposed GAN-based, flow-based, and score-based vocoders. This paper proposed a new thought for the autoregressive generation. Instead of iteratively predicting samples in a time sequence, the proposed model performs frequency-wise autoregressive generation (FAR) and bit-wise autoregressive generation (BAR) to synthesize speech. In FAR, a speech utterance is first split into different frequency subbands. The proposed model generates a subband conditioned on the previously generated one. A full-band speech can then be reconstructed from these generated subbands. Similarly, in BAR, an 8-bit quantized signal is generated iteratively from the first bit. By redesigning the autoregressive method to compute in domains other than the time domain, the number of iterations in the proposed model is no longer proportional to the utterance's length but to the number of subbands/bits. The inference efficiency is hence significantly increased. Besides, a post-filter is employed to sample audio signals from output posteriors, and its training objective is designed based on the characteristics of the proposed autoregressive methods. The experimental results show that the proposed model can synthesize speech faster than real-time without GPU acceleration. Compared with the baseline autoregressive and non-autoregressive vocoders, the proposed model achieves better MUSHRA results and shows good generalization ability while synthesizing 44 kHz speech or utterances from unseen speakers.
Multi-Branch Cascade Receptive Field Residual Network

Deep convolutional neural networks (CNNs) have significantly enhanced image classification in the past decade. This paper proposes Multi-branch Cascade Receptive Field Residual Networks (MCRF-ResNets) based on the original Residual Network (ResNet) architecture for classification and object detection. MCRF-ResNets incorporate multiple branches with different receptive field (RF) sizes to improve image classification performance. Each MCRF residual block contains a 5 × 5 and a 3 × 3 RF-sized convolution block. The 5 × 5 and 3 × 3 RF-sized blocks recognize larger and smaller objects, respectively. Group convolutions are used to reduce redundancy and balance feature extraction and parameter usage in the multiple branches. The proposed model approach demonstrates significant improvement in model performance on CIFAR-10 and CIFAR-100 datasets and a subset of the ImageNet 2012 dataset. The model achieves a 2.8% increase in top-1 accuracy compared with the baseline ResNet-50 model with a similar number of parameters on the subset of the ImageNet 2012 dataset. In addition, the proposed method shows good object detection on Pascal VOC and MS COCO datasets.
Guide the Many-to-One Assignment: Open Information Extraction via IoU-aware Optimal Transport

Open Information Extraction (OIE) seeks to extract structured information from raw text without the limitations of close ontology.Recently, the detection-based OIE methods have received great attention from the community due to their parallelism.An essential step of those models is how to assign ground truth labels to the parallelly generated tuple proposals, which remains under-exploited.The commonly utilized Hungarian algorithm for this procedure is restricted to handling one-to-one assignment among the desired tuples and tuple proposals, which ignores the correlation between proposals and affects the recall of the models.To solve this problem, we propose a dynamic many-to-one label assignment strategy named IOT.Concretely, the label assignment process in OIE is formulated as an Optimal Transport (OT) problem.We leverage the intersectionover-union (IoU) as the assignment quality measurement, and convert the problem of finding the best assignment solution to the one of solving the optimal transport plan by maximizing the IoU values.To further utilize the knowledge from the assignment, we design an Assignment-guided Multi-granularity (AM) loss by simultaneously considering word-level and tuple-level information.Experiment results show the proposed method outperforms the state-of-the-art models on three benchmarks.
Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations

Unlike empathetic dialogues, the system in emotional support conversations (ESC) is expected to not only convey empathy for comforting the help-seeker, but also proactively assist in exploring and addressing their problems during the conversation.In this work, we study the problem of mixed-initiative ESC where the user and system can both take the initiative in leading the conversation.Specifically, we conduct a novel analysis on mixed-initiative ESC systems with a tailor-designed schema that divides utterances into different types with speaker roles and initiative types.Four emotional support metrics are proposed to evaluate the mixed-initiative interactions.The analysis reveals the necessity and challenges of building mixed-initiative ESC systems.In the light of this, we propose a knowledge-enhanced mixedinitiative framework (KEMI) for ESC, which retrieves actual case knowledge from a largescale mental health knowledge graph for generating mixed-initiative responses.Experimental results on two ESC datasets show the superiority of KEMI in both content-preserving evaluation and mixed initiative related analyses.* The work described in this paper is substantially sup-
Towards L2-friendly pipelines for learner corpora: A case of written production by L2-Korean learners

We introduce the Korean-Learner-Morpheme (KLM) corpus, a manually annotated dataset consisting of 129,784 morphemes from second language (L2) learners of Korean, featuring morpheme tokenization and part-of-speech (POS) tagging.We evaluate the performance of four Korean morphological analyzers in tokenization and POS tagging on the L2-Korean corpus.Results highlight the analyzers' reduced performance on L2 data, indicating the limitation of advanced deep-learning models when dealing with L2-Korean corpora.We further show that fine-tuning one of the models with the KLM corpus improves its accuracy of tokenization and POS tagging on L2-Korean dataset.
Ranger: A Toolkit for Effect-Size Based Multi-Task Evaluation

In this paper, we introduce Ranger -a toolkit to simplify the utilization of effect-size-based meta-analysis for multi-task evaluation in NLP and IR.We observed that our communities often face the challenge of aggregating results over incomparable metrics and scenarios, which makes conclusions and take-away messages less reliable.With Ranger, we aim to address this issue by providing a task-agnostic toolkit that combines the effect of a treatment on multiple tasks into one statistical evaluation, allowing for comparison of metrics and computation of an overall summary effect.Our toolkit produces publication-ready forest plots that enable clear communication of evaluation results over multiple tasks.Our goal with the ready-to-use Ranger toolkit is to promote robust, effect-size based evaluation and improve evaluation standards in the community.We provide two case studies for common IR and NLP settings to highlight Ranger's benefits.
Towards Robust Ranker for Text Retrieval

A neural ranker plays an indispensable role in the de facto 'retrieval & rerank' pipeline, but its training still lags behind due to the weak negative mining during contrastive learning.Compared to retrievers boosted by selfadversarial (i.e., in-distribution) negative mining, the ranker's heavy structure suffers from query-document combinatorial explosions, so it can only resort to the negative sampled by the fast yet out-of-distribution retriever.Thereby, the moderate negatives compose ineffective contrastive learning samples, becoming the main barrier to learning a robust ranker.To alleviate this, we propose a multi-adversarial training strategy that leverages multiple retrievers as generators to challenge a ranker, where i) diverse hard negatives from a joint distribution are prone to fool the ranker for more effective adversarial learning and ii) involving extensive out-of-distribution label noises renders the ranker against each noise distribution, leading to more challenging and robust contrastive learning.To evaluate our robust ranker (dubbed R 2 ANKER), we conduct experiments in various settings on the passage retrieval benchmarks, including BM25-reranking, full-ranking, retriever distillation, etc.The empirical results verify the new state-of-the-art effectiveness of our model.
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks.However, it has yet to be explored for vision and multimodal tasks.In this work, we introduce MUL-TIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-toseq format covering 10 broad categories.The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expertwritten instructions.We take OFA (Wang et al., 2022a) as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale NATURAL INSTRUCTIONS dataset (Mishra et al., 2022).Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset.We also design a new evaluation metric -Sensitivity, to evaluate how sensitive the model is to the variety of instructions.Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task 1 .
Mitigating Label Biases for In-context Learning

Various design settings for in-context learning (ICL), such as the choice and order of the incontext examples, can bias the model's predictions.While many studies discuss these design choices, there have been few systematic investigations into categorizing them and mitigating their impact.In this work, we define a typology for three types of label biases in ICL for text classification: vanilla-label bias, contextlabel bias, and domain-label bias (which we conceptualize and detect for the first time).Our analysis demonstrates that prior label bias calibration methods fall short of addressing all three types of biases.Specifically, domainlabel bias restricts LLMs to random-level performance on many tasks regardless of the choice of in-context examples.To mitigate the effect of these biases, we propose a simple bias calibration method that estimates a language model's label bias using random indomain words from the task corpus.After controlling for this estimated bias when making predictions, our novel domain-context calibration significantly improves the ICL performance of GPT-J and GPT-3 on a wide range of tasks.The gain is substantial on tasks with large domain-label bias (up to 37% in Macro-F1).Furthermore, our results generalize to models with different scales, pretraining methods, and manually-designed task instructions, showing the prevalence of label biases in ICL.