OpenRaiser/Intern-Atlas huggingface dataset:
Temporal Convolutional Explorer Helps Understand 1D-CNN's Learning Behavior in Time Series Classification from Frequency Domain
Authors: Junru Zhang, Lang Feng, Yang He, Yuhan Wu, Yabo Dong
Venue: CIKM
Year: 2023

While one-dimensional convolutional neural networks (1D-CNNs) have been empirically proven effective in time series classification tasks, we find that there remain undesirable outcomes that could arise in their application, motivating us to further investigate and understand their underlying mechanisms. In this work, we propose a Temporal Convolutional Explorer (TCE) to empirically explore the learning behavior of 1D-CNNs from the perspective of the frequency domain. Our TCE analysis highlights that deeper 1D-CNNs tend to distract the focus from the low-frequency components leading to the accuracy degradation phenomenon, and the disturbing convolution is the driving factor. Then, we leverage our findings to the practical application and propose a regulatory framework, which can easily be integrated into existing 1D-CNNs. It aims to rectify the suboptimal learning behavior by enabling the network to selectively bypass the specified disturbing convolutions. Finally, through comprehensive experiments on widely-used UCR, UEA, and UCI benchmarks, we demonstrate that 1) TCE's insight into 1D-CNN's learning behavior; 2) our regulatory framework enables state-of-the-art 1D-CNNs to get improved performances with less consumption of memory and computational overhead.
3MN: Three Meta Networks for Multi-Scenario and Multi-Task Learning in Online Advertising Recommender Systems
Authors: Yifei Zhang, Hua Hua, Hui Guo, Shuangyang Wang, Chongyu Zhong, Shijie Zhang
Venue: CIKM
Year: 2023

Recommender systems are widely applied on web. For example, online advertising systems rely on recommender systems to accurately estimate the value of display opportunities, which is critical to maximize the profits of advertisers. To reduce computational resource consumption, the core tactic of Multi-Scenario Multi-Task Learning (MSMTL) is to devise a single recommder system that is adapted to all contexts instead of implementing multiple scenario-oriented or task-oriented recommender systems. However, MSMTL is challenging because there are complicated task-task, scenario-scenario, and task-scenario interrelations; the characteristic of different tasks in different scenarios also largely varies; and samples of each context are often unevenly distributed. Previous MSMTL solutions focus on applying scenario knowledge to improve the performance of multi-task learning, while neglecting the complicated interrelations among tasks and scenarios. Moreover, samples derived from different scenarios are transferred into the latent embedding with the same dimension. This static embedding strategy impedes the practicality of model expressiveness, since the scenarios with sufficient samples are underrepresented and those with insufficient samples are over-represented. In this paper, we propose a novel three meta networks-based solution (3MN) to MSMTL that addresses all the limitations discussed above. Specifically, we innovatively bind the meta network with scenario-related input in bottom embedding layer, so that the embedding layer is capable of learning the scenario-related knowledge explicitly. To counteract the imbalanced scenario-related data distributions, our flexible embedding layer adaptively learns the representation of samples. This innovative embedding layer is also able to boost other solutions as a plug-in. Moreover, to fully capture the interrelations among scenarios and tasks, we enforce the task and scenario information into the other two meta networks, and transfer the resulted meta-knowledge into the top components (i.e., backbone network and classifier) of the recommender system, respectively. These three meta networks contribute to the superiority of our 3MN solution over state-of-the-art MSMTL solutions, which is demonstrated by extensive offline experiments. 3MN has been successfully deployed in our industrial online advertising system.
Time-aware Graph Structure Learning via Sequence Prediction on Temporal Graphs
Authors: Haozhen Zhang, Xueting Han, Xi Xiao, Jing Bai
Venue: CIKM
Year: 2023

Temporal Graph Learning, which aims to model the time-evolving nature of graphs, has gained increasing attention and achieved remarkable performance recently. However, in reality, graph structures are often incomplete and noisy, which hinders temporal graph networks (TGNs) from learning informative representations. Graph contrastive learning uses data augmentation to generate plausible variations of existing data and learn robust representations. However, rule-based augmentation approaches may be suboptimal as they lack learnability and fail to leverage rich information from downstream tasks. To address these issues, we propose a Time-aware Graph Structure Learning (TGSL) approach via sequence prediction on temporal graphs, which learns better graph structures for downstream tasks through adding potential temporal edges. In particular, it predicts time-aware context embedding based on previously observed interactions and uses the Gumble-Top-K to select the closest candidate edges to this context embedding. Additionally, several candidate sampling strategies are proposed to ensure both efficiency and diversity. Furthermore, we jointly learn the graph structure and TGNs in an end-to-end manner and perform inference on the refined graph. Extensive experiments on temporal link prediction benchmarks demonstrate that TGSL yields significant gains for the popular TGNs such as TGAT and GraphMixer, and it outperforms other contrastive learning methods on temporal graphs. We release the code at https://github.com/ViktorAxelsen/TGSL.
Efficient Exact Minimum k-Core Search in Real-World Graphs
Authors: Qifan Zhang, Shengxin Liu
Venue: CIKM
Year: 2023

The k-core, which refers to the induced subgraph with a minimum degree of at least k, is widely used in cohesive subgraph discovery and has various applications. However, the k-core in real-world graphs tends to be extremely large, which hinders its effectiveness in practical applications. This challenge has motivated researchers to explore a variant of the k-core problem known as the minimum k-core search problem. This problem has been proven to be NP-Hard, and most of the existing studies naturally either deal with approximate solutions or suffer from inefficiency in practice. In this paper, we focus on designing efficient exact algorithms for the minimum k-core search problem. In particular, we develop an iterative-based framework that decomposes an instance of the minimum k-core search problem into a list of problem instances on another well-structured graph pattern. Based on this framework, we propose an iterative-based branch-and-bound algorithm, namely IBB, with additional pruning and reduction techniques. We show that, with a n-vertex graph, IBB runs in cn nO(1) time for some c < 2, achieving better theoretical performance than the trivial bound of 2n nO(1). Finally, our experiments on real-world graphs demonstrate that IBB is up to three orders of magnitude faster than the state-of-the-art algorithms on real-world datasets.
Task Relation Distillation and Prototypical Pseudo Label for Incremental Named Entity Recognition
Authors: Duzhen Zhang, Hongliu Li, Wei Cong, Rongtao Xu, Jiahua Dong, Xiuyi Chen
Venue: CIKM
Year: 2023

Incremental Named Entity Recognition (INER) involves the sequential learning of new entity types without accessing the training data of previously learned types. However, INER faces the challenge of catastrophic forgetting specific for incremental learning, further aggravated by background shift (i.e., old and future entity types are labeled as the non-entity type in the current task). To address these challenges, we propose a method called task Relation Distillation and Prototypical pseudo label (RDP) for INER. Specifically, to tackle catastrophic forgetting, we introduce a task relation distillation scheme that serves two purposes: 1) ensuring inter-task semantic consistency across different incremental learning tasks by minimizing inter-task relation distillation loss, and 2) enhancing the model's prediction confidence by minimizing intra-task self-entropy loss. Simultaneously, to mitigate background shift, we develop a prototypical pseudo label strategy that distinguishes old entity types from the current non-entity type using the old model. This strategy generates high-quality pseudo labels by measuring the distances between token embeddings and type-wise prototypes. We conducted extensive experiments on ten INER settings of three benchmark datasets (i.e., CoNLL2003, I2B2, and OntoNotes5). The results demonstrate that our method achieves significant improvements over the previous state-of-the-art methods, with an average increase of 6.08% in Micro F1 score and 7.71% in Macro F1 score.
CoSaR: Combating Label Noise Using Collaborative Sample Selection and Adversarial Regularization
Authors: Xiaobo Zhang, Yutao Liu, Hao Wang, Wei Wang, Panpan Ni, Ji Zhang
Venue: CIKM
Year: 2023

Learning with noisy labels is nontrivial for deep learning models. Sample selection is a widely investigated research topic for handling noisy labels. However, most existing methods face challenges such as imprecise selection, a lack of global selection capabilities, and the need for tedious hyperparameter tuning. In this paper, we propose CoSaR (Collaborative Selection and adversarial Regularization ), a twin-networks based model that performs globally adaptive sample selection to tackle label noise. Specifically, the collaborative selection estimates the average distribution distances between predictions and generation labels through the collaboration of two networks to address the bias of the average distribution distances and the manual tuning of hyperparameters. Adversarial regularization is integrated into CoSaR to restrict the network's tendency to fit and memorize noisy labels, thereby enhancing its collaborative selection capability. In addition, we employ a label smoothing regularization and two types of data augmentation to enhance the robustness of the model further. Extensive experiments on both synthetic and real-world noisy datasets demonstrate that the proposed model outperforms baseline methods remarkably, with an accuracy improvement ranging between +0.56% and +15.14%.
Closed-form Machine Unlearning for Matrix Factorization
Authors: Shuijing Zhang, Jian Lou, Li Xiong, Xiaoyu Zhang, Jing Liu
Venue: CIKM
Year: 2023

Matrix factorization (MF) is a fundamental model in data mining and machine learning, which finds wide applications in diverse application areas, including recommendation systems with user-item rating matrices, phenotype extraction from electronic health records, and spatial-temporal data analysis for check-in records. The "right to be forgotten" has become an indispensable privacy consideration due to the widely enforced data protection regulations, which allow personal users having contributed their data for model training to revoke their data through a data deletion request. Consequently, it gives rise to the emerging task of machine unlearning for the MF model, which removes the influence of the matrix rows/columns from the trained MF factors upon receiving the deletion requests from the data owners of these rows/columns. The central goal is to effectively remove the influence of the rows/columns to be forgotten, while avoiding the computationally prohibitive baseline approach of retraining from scratch. Existing machine unlearning methods are either designed for single-variable models and not compatible with MF that has two factors as coupled model variables, or require alternative updates that are not efficient enough. In this paper, we propose a closed-form machine unlearning method. In particular, we explicitly capture the implicit dependency between the two factors, which yields the total Hessian-based Newton step as the closed-form unlearning update. In addition, we further introduce a series of efficiency-enhancement strategies by exploiting the structural properties of the total Hessian. Extensive experiments on five real-world datasets from three application areas as well as synthetic datasets validate the efficiency, effectiveness, and utility of the proposed method.
Multi-gate Mixture-of-Contrastive-Experts with Graph-based Gating Mechanism for TV Recommendation
Authors: Cong Zhang, Dongyang Liu, Lin Zuo, Junlan Feng, Chao Deng, Jian Sun, Haitao Zeng, Yaohong Zhao
Venue: CIKM
Year: 2023

With the rapid development of smart TV, TV recommendation is attracting more and more users. TV users usually distribute in multiple regions with different cultures and hence have diverse TV program preferences. From the perspective of engineering practice and performance improvement, it's very essential to model users from multiple regions with one single model. In previous work, Multi-gate Mixture-of-Expert (MMoE) has been widely adopted in multi-task and multi-domain recommendation scenarios. In practice, however, we first observe the embeddings generated by experts tend to be homogeneous which may result in high semantic similarities among embeddings that reduce the capability of Multi-gate Mixture-of-Expert (MMoE) model. Secondly, we also find there are lots of commonalities and differences between multiple regions regarding user preferences. Therefore, it's meaningful to model the complicated relationships between regions. In this paper, we first introduce contrastive learning to overcome the expert representation degeneration problem. The embeddings of two augmented samples generated by the same experts are pushed closer to enhance the alignment, and the embeddings of the same samples generated by different experts are pushed away in vector space to improve uniformity. Then we propose a Graph-based Gating Mechanism to empower typical Multi-gate Mixture-of-Experts. Graph-based MMoE is able to recognize the commonalities and differences among multiple regions by introducing a Graph Neural Network (GNN) with region similarity prior. We name our model Multi-gate Mixture-of-Contrastive-Experts model with Graph-based Gating Mechanism (MMoCEG). Extensive offline experiments and online A/B tests on a commercial TV service provider over 100 million users and 2.3 million items demonstrate the efficacy of MMoCEG compared to the existing models.
Unleashing the Power of Large Language Models for Legal Applications
Authors: Dell Zhang, Alina Petrova, Dietrich Trautmann, Frank Schilder
Venue: CIKM
Year: 2023

The use of Large Language Models (LLMs) is revolutionizing the legal industry. In this technical talk, we would like to explore the various use cases of LLMs in legal tasks, discuss the best practices, investigate the available resources, examine the ethical concerns, and suggest promising research directions.
POSPAN: Position-Constrained Span Masking for Language Model Pre-training
Authors: Zhenyu Zhang, Lei Shen, Yuming Zhao, Meng Chen, Xiaodong He
Venue: CIKM
Year: 2023

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.
Towards Dynamic and Reliable Private Key Management for Hierarchical Access Structure in Decentralized Storage
Authors: Yifang Zhang, Mingyue Wang, Yu Guo, Fangda Guo
Venue: CIKM
Year: 2023

With the widespread development of decentralized storage, it is increasingly popular for users to store their data to the decentralized database systems for the well-understood benefits of outsourced storage. To ensure the data privacy, systems commonly require users to securely keep their private keys. Thus, the secure storage of private keys is an important issue in these systems. However, existing key-management schemes commonly rely on a Trusted Third Party (TTP), which raises critical security concerns such as the single point of failure and Distributed Denial of Service (DDoS) attacks. In this paper, we propose HasDPSS, a secure and efficient blockchain-based key-management scheme for decentralized storage systems. It uses secret sharing, a lightweight cryptographic technique, to build the decentralized key-management scheme. Considering that the reliability of managing participants has inherent heterogeneity, we introduce the hierarchical access structure to achieve fine-grained key management. Meanwhile, to adapt the node churn of decentralized key management, HasDPSS enables a dynamic management committee to provide reliable services with a proactive refresh mechanism while protecting the integrity and security of private keys. In our design, we use the dimension switch method of polynomials in the evolving process to achieve the committee change of the hierarchical access structure. The reliability of participants is guaranteed by the customized commitment protocol and the immutable property of the blockchain. We thoroughly analyze security strengths and conduct extensive experiments to demonstrate the practicality of our design.
Build Faster with Less: A Journey to Accelerate Sparse Model Building for Semantic Matching in Product Search
Authors: Jiong Zhang, Yau-Shian Wang, Wei-Cheng Chang, Wei Li, Jyun-Yu Jiang, Cho-Jui Hsieh, Hsiang-Fu Yu
Venue: CIKM
Year: 2023

The semantic matching problem in product search seeks to retrieve all semantically relevant products given a user query. Recent studies have shown that extreme multi-label classification~(XMC) model enjoys both low inference latency and high recall in real-world scenarios. These XMC semantic matching models adopt TF-IDF vectorizers to extract query text features and use mainly sparse matrices for the model weights. However, limited availability of libraries for efficient parallel sparse modules may lead to tediously long model building time when the problem scales to hundreds of millions of labels. This incurs significant hardware cost and renders the semantic model stale even before it is deployed. In this paper, we investigate and accelerate the model building procedures in a tree-based XMC model. On a real-world semantic matching task with 100M labels, our enhancements achieve over 10 times acceleration (from 3.1 days to 6.7 hours) while reducing hardware cost by 25%.
FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
Authors: Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng, Zhongfang Zhuang, Wei Zhang
Venue: CIKM
Year: 2023

Sequential tabular data is one of the most commonly used data types in real-world applications. Different from conventional tabular data, where rows in a table are independent, sequential tabular data contains rich contextual and sequential information, where some fields aredynamically changing over time and others arestatic. Existing transformer-based approaches analyzing sequential tabular data overlook the differences between dynamic and static fields by replicating and filling static fields into each record, and ignore temporal information between rows, which leads to three major disadvantages: (1) computational overhead, (2) artificially simplified data for masked language modeling pre-training task that may yield less meaningful representations, and (3) disregarding the temporal behavioral patterns implied by time intervals. In this work, we propose FATA-Trans, a model with two field transformers for modeling sequential tabular data, where each processes static and dynamic field information separately. FATA-Trans isfield - andtime -aware for sequential tabular data. Thefield -type embedding in the method enables FATA-Trans to capture differences between static and dynamic fields. Thetime -aware position embedding exploits both order and time interval information between rows, which helps the model detect underlying temporal behavior in a sequence. Our experiments on three benchmark datasets demonstrate that the learned representations from FATA-Trans consistently outperform state-of-the-art solutions in the downstream tasks. We also present visualization studies to highlight the insights captured by the learned representations, enhancing our understanding of the underlying data. Our codes are available at https://github.com/zdy93/FATA-Trans.
Assessing the Continuous Causal Responses of Typhoon-related Weather on Human Mobility: An Empirical Study in Japan
Authors: Zhiwen Zhang, Hongjun Wang, Zipei Fan, Ryosuke Shibasaki, Xuan Song
Venue: CIKM
Year: 2023

To understand human mobility following the typhoon, analyzing the causal impact of extreme typhoon weather on human mobility is important for disaster emergency management. However, the unobserved confounders (e.g., the characteristic of each region) correlate with the strength of typhoon weather and also affect human mobility during typhoon, which may generate biased influences on the causal analysis process. Besides, these confounders may be time-varying following the dynamic movements of typhoon. In this work, we develop a neural network-based continuous causal effect estimation framework to mitigate the interference from (unobserved) confounders and assess the continuous causal responses of typhoon-related weather (treatment) on several types of human mobility (outcome) across different counties at any given period. To this end, we integrate the big data from two huge typhoons in Japan (i.e., Typhoon Faxai and Hagibis) and leverage multiple sources of covariates (i.e., residents' vigilance and basic mobility patterns) from different counties to learn the representations of time-varying confounders. The experimental results indicate the effectiveness of our proposed framework in capturing the confounders for quantifying the causal impact of extreme weather during the typhoon process, compared with several existing causal studies.