OpenRaiser/Intern-Atlas huggingface dataset:
CANCN-BERT: A Joint Pre-Trained Language Model for Classical and Modern Chinese
Authors: Zijing Ji, Xin Wang, Yuxin Shen, Guozheng Rao
Venue: CIKM
Year: 2021

Pre-Trained Models (PTMs) can learn general knowledge representations and perform well in Natural Language Processing (NLP) tasks. For the Chinese language, several PTMs are developed, however, most existing methods concentrate on modern Chinese and are not ideal for processing classical Chinese due to the differences in grammars and semantics between these two forms. In this paper, in order to process two forms of Chinese uniformly, we propose a novel Classical and Modern Chinese pre-trained language model (CANCN-BERT), with the advantage of effectively processing both classical and modern Chinese, which is an extension of BERT. Form-aware pre-training tasks are elaborately designed to train our model, so as to better adapt it to classical and modern Chinese corpus. Moreover, we define a joint model, proposing dedicated optimization methods through different paths with the control of the switch mechanism. Our model merges characteristics of both classical and modern Chinese, which can adequately and efficiently enhance the representation ability for both forms. Extensive experiments show that our model outperforms baseline models on processing classical and modern Chinese and achieves significant and consistent improvements. Also, the results of ablation experiments demonstrate the effectiveness of each module.
Complex Temporal Question Answering on Knowledge Graphs
Authors: Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, Gerhard Weikum
Venue: CIKM
Year: 2021

Question answering over knowledge graphs (KG-QA) is a vital topic in IR. Questions with temporal intent are a special class of practical importance, but have not received much attention in research. This work presents EXAQT, the first end-to-end system for answering complex temporal questions that have multiple entities and predicates, and associated temporal conditions. EXAQT answers natural language questions over KGs in two stages, one geared towards high recall, the other towards precision at top ranks. The first step computes question-relevant compact subgraphs within the KG, and judiciously enhances them with pertinent temporal facts, using Group Steiner Trees and fine-tuned BERT models. The second step constructs relational graph convolutional networks (R-GCNs) from the first step's output, and enhances the R-GCNs with time-aware entity embeddings and attention over temporal relations. We evaluate EXAQT on TimeQuestions, a large dataset of 16k temporal questions we compiled from a variety of general purpose KG-QA benchmarks. Results show that EXAQT outperforms three state-of-the-art systems for answering complex questions over KGs, thereby justifying specialized treatment of temporal QA.
HAO Unity: A Graph-based System for Unifying Heterogeneous Data
Authors: Fei Jie, Yanxiang Huang, Qiangwei Bai, Xindong Wu
Venue: CIKM
Year: 2021

Many real-world applications have to face the problem of diversity in data formats and semantics. Currently, how to deal with heterogeneous data effectively is still a big challenge. With the rise of knowledge graphs, more and more applications are built upon graph-like data models, which benefit from flexible schemas and convenient support for relationship queries. We propose a graph-based unifying system for heterogeneous data unification, which helps to (1) transform data in many other formats into graphs, or conversely, from graph to other formats, (2) integrate graph data based on HAO intelligence, which achieves schema integration and entity consolidation, and (3) explore data at different levels via querying the integrated graphs. In this paper, we introduce the overall system architecture, explain in detail the implementation, and display the usage in two practical scenarios.
ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning
Authors: Ming Jin, Yixin Liu, Yu Zheng, Lianhua Chi, Yuan-Fang Li, Shirui Pan
Venue: CIKM
Year: 2021

Anomaly detection on graphs plays a significant role in various domains, including cybersecurity, e-commerce, and financial fraud detection. However, existing methods on graph anomaly detection usually consider the view in a single scale of graphs, which results in their limited capability to capture the anomalous patterns from different perspectives. Towards this end, we introduce a novel graph anomaly detection framework, namely ANEMONE, to simultaneously identify the anomalies in multiple graph scales. Concretely, ANEMONE first leverages a graph neural network backbone encoder with multi-scale contrastive learning objectives to capture the pattern distribution of graph data by learning the agreements between instances at the patch and context levels concurrently. Then, our method employs a statistical anomaly estimator to evaluate the abnormality of each node according to the degree of agreement from multiple perspectives. Experiments on three benchmark datasets demonstrate the superiority of our method.
The Primacy of Data in Deep Learning NLP for Conversational AI
Authors: Mark Johnson
Venue: CIKM
Year: 2021

Computational Linguistics and Natural Language Processing have changed considerably in the past few decades. Early research focused on representing and using linguistic knowledge in computational processes such as parsers, while these days the field focuses on practically-useful tasks such as information retrieval and chatbots. Currently our Deep Learning models have little to do with linguistic theory For example, the Oracle Digital Assistant is built on top of generic "Foundation" Deep Learning models. An intermediate Focusing step adapts these models to specific enterprise domains. Transfer Learning is used to refocus these models onto specific customer-oriented tasks such as Intent Classification, Named Entity Recognition, as well as more advanced models such as text-to-SQL sequence-to-sequence models. These technologies have revolutionised the application of NLP to practical problems with commercial relevance, enabling us to build better systems faster and cheaper than ever before. Linguistic insights aren't gone from the field, however; they play a critical role in data manufacturing and evaluation. This talk explain how we use hundreds of different evaluations to understand the strengths and weaknesses of our models in the Oracle Digital Assistant, and how we automatically use this in hyper-parameter tuning. It also describes areas where additional research is still required before we can claim that NLP has become an engineering field.
Dynamic Early Exit Scheduling for Deep Neural Network Inference through Contextual Bandits
Authors: Weiyu Ju, Wei Bao, Liming Ge, Dong Yuan
Venue: CIKM
Year: 2021

Recent advances in Deep Neural Networks (DNNs) have dramatically improved the accuracy of DNN inference, but also introduce larger latency. In this paper, we investigate how to utilize early exit, a novel method that allows inference to exit at earlier exit points at the cost of an acceptable amount of accuracy. Scheduling the optimal exit point on a per-instance basis is challenging because the realized performance (i.e., confidence and latency) of each exit point is random and the statistics vary in different scenarios. Moreover, the performance has dependencies among the exit points, further complicating the problem. Therefore, the optimal exit scheduling decision cannot be known in advance but should be learned in an online fashion. To this end, we propose Dynamic Early Exit (DEE), a real-time online learning algorithm based on contextual bandit analysis. DEE observes the performance at each exit point as context and decides whether to exit or keep processing. Unlike standard contextual bandit analyses, the rewards of the decisions in our problem are temporally dependent. Furthermore, the performances of the earlier exit points are inevitably explored more compared to the later ones, which poses an unbalance exploration-exploitation trade-off. DEE addresses the aforementioned challenges, where its regret per inference asymptotically approaches zero. We compare DEE with four benchmark schemes in the real-world experiment. The experiment result shows that DEE can improve the overall performance by up to 98.1% compared to the best benchmark scheme.
RABERT: Relation-Aware BERT for Target-Oriented Opinion Words Extraction
Authors: Taegwan Kang, Minwoo Lee, Nakyeong Yang, Kyomin Jung
Venue: CIKM
Year: 2021

Targeted Opinion Word Extraction (TOWE) is a subtask of aspect-based sentiment analysis, which aims to identify the correspondingopinion terms for given opinion targets in a review. To solve theTOWE task, recent works mainly focus on learning the target-aware context representation that infuses target information intocontext representation by using various neural networks. However,it has been unclear how to encode the target information to BERT,a powerful pre-trained language model. In this paper, we proposea novel TOWE model, RABERT (Relation-Aware BERT), that canfully utilize BERT to obtain target-aware context representations.To introduce the target information into BERT layers clearly, wedesign a simple but effective encoding method that adds targetmarkers indicating the opinion targets to the sentence. In addi-tion, we find that the neighbor word information is also importantfor extracting the opinion terms. Therefore, RABERT employs thetarget-sentence relation network and the neighbor-aware relationnetwork to consider both the opinion target and the neighbor wordsinformation. Our experimental results on four benchmark datasetsshow that RABERT significantly outperforms the other baselinesand achieves state-of-the-art performance. We also demonstrate theeffectiveness of each component of RABERT in further analysis
Question Answering using Web Lists
Authors: Anoop R. Katti, Kai Hui, Adrià de Gispert, Hagen Fürstenau
Venue: CIKM
Year: 2021

There are many natural questions that are best answered with a list. We address the problem of answering such questions using lists that occur on the Web, i.e. List Question Answering (ListQA). The diverse formats of lists on the Web makes this task challenging. We describe state-of-the-art methods for list extraction and ranking, that also consider the text surrounding the lists as context. Due to the lack of realistic public datasets for ListQA, we present three novel datasets that together are realistic, reproducible and test out-of-domain generalization. We benchmark the above steps on these datasets, with and without context. On the hardest setting (realistic and out-of-domain), we achieve an end-to-end Precision@1 of 51.28% and HITs@5 of 79.38%, effectively demonstrating the difficulty of the task and quantifying the immediate opportunity for improvement. We highlight some future directions through error analysis and release the datasets for further research.
Norma: A Hybrid Feature Alignment for Class-Aware Unsupervised Domain Adaptation
Authors: Mahsa Keramati, Zahra Zohrevand, Uwe Glässer
Venue: CIKM
Year: 2021

Unsupervised domain adaptation is the problem of transferring extracted knowledge from a labeled source domain to an unlabeled target domain. To achieve discriminative domain adaptation recent studies take advantage of target sample pseudo-labels to impose class-aware distribution alignment across the source and target domains. Still, they have some shortcomings such as making decisions based on inaccurate pseudo-labeled samples that mislead the adaptation process. In this paper, we propose a progressive deep feature alignment, called Norma, to tackle class-aware unsupervised domain adaptation for image classification by enforcing inter-class compactness and intra-class discrepancy through a hybrid learning process. To this end, Norma's optimization process is defined based on a novel triplet loss which not only addresses soft prototype alignment but also pushes away multiple negative centroids. Also, to extract maximum discriminative domain knowledge per iteration, we propose a joint positive and negative learning procedure along with an uncertainty-guided progressive pseudo-labeling on the basis of prototype-based clustering and conditional probability. Our experimental results on several benchmarks demonstrate that Norma outperforms the state-of-the-art methods.
Semantic Concept Annotation for Tabular Data
Authors: Udayan Khurana, Sainyam Galhotra
Venue: CIKM
Year: 2021

Determining the semantic concepts of columns in tabular data is of use for many applications ranging from data integration, cleaning, search to feature engineering and model building in machine learning. Several prior works have proposed supervised learning-based or heuristic-based approaches to semantic type annotation. These techniques suffer from poor generalizability over a large number of concepts or examples. Recent neural network based supervised learning methods generalize to different datasets but require large amounts of curated training data and also present scalability issues. Furthermore, none of the known methods works well for numerical data. We present C2, a system that maps each column to a concept based on a maximum likelihood estimation approach through ensembles. It is able to effectively utilize vast amounts of, albeit somewhat noisy, openly available table corpora in addition to two popular knowledge graphs (Wikidata and DBpedia), to perform effective and efficient concept annotation for tabular data. Specifically, we utilize a collection of 32 million openly available webtables from several sources. We also present efficient indexing techniques for categorical string, numeric and mixed-type data, and novel techniques for table context utilization. We demonstrate the effectiveness and efficiency of C2 over available techniques on 9 real-world datasets containing a wide variety of concepts.
Self-supervised Fine-tuning for Efficient Passage Re-ranking
Authors: Meoungjun Kim, Youngjoong Ko
Venue: CIKM
Year: 2021

Passage retrievers based on neural language models have recently achieved significant performance improvements in ranking tasks. Such ranking models have the advantage of finding the contextual features of queries and documents better than traditional keyword based methods. However, these deep learning-based models are limited by the large amounts of training data required. We propose a new fine-tuning method based on a masked language model (MLM) that is typically used in pre-trained language models. Our model improves the ranking performance using the MLM while efficiently utilizing less training data via data augmentation. The proposed approach applies self-supervised learning to information retrieval without needing additional expensive labeled data. In addition, because masking important terms during the fine-tuning stage can undermine ranking performance, the importance values of each term and sentence in a passage are calculated using the BM25 scheme and applied to the fine-tuning task such that the more important terms are masked less often. Our model is trained with dataset from MS MARCO re-ranking leaderboard and achieves the state-of-the-art MRR@10 performance in the leaderboard except for the ensemble-based method.
ALADDIN: Asymmetric Centralized Training for Distributed Deep Learning
Authors: Yun-Yong Ko, Kibong Choi, Hyunseung Jei, Dongwon Lee, Sang-Wook Kim
Venue: CIKM
Year: 2021

To speed up the training of massive deep neural network (DNN) models, distributed training has been widely studied. In general, a centralized training, a type of distributed training, suffers from the communication bottleneck between a parameter server (PS) and workers. On the other hand, a decentralized training suffers from increased parameter variance among workers that causes slower model convergence. Addressing this dilemma, in this work, we propose a novel centralized training algorithm, ALADDIN, employing "asymmetric" communication between PS and workers for the PS bottleneck problem and novel updating strategies for both local and global parameters to mitigate the increased variance problem. Through a convergence analysis, we show that the convergence rate of ALADDIN is O(1 ønk ) on the non-convex problem, where n is the number of workers and k is the number of training iterations. The empirical evaluation using ResNet-50 and VGG-16 models demonstrates that (1) ALADDIN shows significantly better training throughput with up to 191% and 34% improvement compared to a synchronous algorithm and the state-of-the-art decentralized algorithm, respectively, (2) models trained by ALADDIN converge to the accuracies, comparable to those of the synchronous algorithm, within the shortest time, and (3) the convergence of ALADDIN is robust under various heterogeneous environments.
UQJG: Identifying Transactions that Collaborate to Violate an SQL Assertion
Authors: Toon Koppelaars, Xavier Oriol, Ernest Teniente, Sergi Curto, Eduard Pujol
Venue: CIKM
Year: 2021

An SQL assertion is a declarative statement about data that must always be satisfied in any database state. Assertions were introduced in the SQL92 standard but no commercial DBMS has implemented them so far. Some approaches have been proposed to incrementally determine whether a transaction violates an SQL assertion, but they assume that transactions are applied in isolation, hence not considering the problem of concurrent transaction executions that collaborate to violate an assertion. This is the main stopper for its commercial implementation. To handle this problem, we have developed a technique for efficiently serializing concurrent transactions that might interact to violate an SQL assertion.
Understanding Job Seeker Funnel for Search and Discovery Personalization
Authors: Nagaraj Kota, Venkatesh Duppada, Ashvini Kumar Jindal, Mohit Wadhwa
Venue: CIKM
Year: 2021

The search and discovery process of a job-seeker towards realizing the dream opportunity can be very complex. Given the dynamic nature of job postings, churn-rate of skills, and gaps in intent matches, professionals often find it hard to discover the right opportunity. Most often, they need guidance on the right search queries to issue or next-steps in the job-seeking funnel to reach a target job-apply. In this work, we experiment with job-sessions dataset from LinkedIn, to understand and represent user's job-seeking behavior. In particular using action sequences unified from various search and discovery channels, we pre-train language models, e.g. BERT (Bidirectional Encoder Representations from Transformers) to model user's activities. We further fine-tune the BERT based contextual session embeddings towards predicting entities from target sessions, in an eXtreme Multi-Label (XML) classification setting. We hypothesize that XML fine-tuning task enables dense-representation, and predicted entities to be used in multiple downstream tasks of job-search query recommendation, job-search ranking, job recommendation retrieval, and job-notification expansion, as shown in experiments. We demonstrate significant improvements in accuracy across tasks leading to reduced time to reach a given job-apply, as well as increase in total job-applies in the system. We also share the learning from deploying these models in production. To the best of our knowledge, this is the first work to efficiently model cross-channel activities at scale using self-attention mechanisms, leading to statistically significant improvement in job-seeker experience.
SPARQL-vision: A Platform for Querying, Visualising and Exploring SPARQL endpoints
Authors: Maria Krommyda, Verena Kantere
Venue: CIKM
Year: 2021

The wide adaptation of the Semantic Web and the Resource Description Framework (RDF) has made available many important datasets. The SPARQL query language facilitates the exploration of this information, which is available in a semi-structured way that does not comply with relational data models, deviating from exploration techniques that most researchers are familiar with. Usually, only people with training and extensive knowledge of the RDF model can explore and understand it in depth. We present here a platform that supports the users with querying, exploring and visualizing information available in SPARQL endpoints. A dedicated visualization module, built upon a knowledge database, allows us to provide case-specific visualization solutions for SPARQL query results. The selection is based exclusively on features extracted from the result, without any knowledge about the structure, content and characteristics of the underlying dataset.
Exploratory Search of GANs with Contextual Bandits
Authors: Ivan Kropotov, Alan Medlar, Dorota Glowacka
Venue: CIKM
Year: 2021

Interactive image retrieval involves users searching a collection of images to satisfy their subjective information needs. However, even large image collections are finite and therefore may not be able to satisfy users. An alternate approach would be to explore a generative adversarial network (GAN) and model users' search intents directly in terms of the latent space used by the GAN to generate images. In this article, we present a simulation study exploring the performance of Gaussian Process bandits in the context of interactive GAN exploration. We used recent advances in interpretable GAN controls to investigate the scalability of different approaches in terms of image space dimensionality. While we present several experiments with promising results, none of the approaches tested scale sufficiently well to explore the entire GAN image space.
Asterisk-Shaped Features for Tabular Data
Authors: Yuki Kurauchi, Yoshiaki Takimoto, Shuhei Yamamoto, Shunichi Seko, Hiroyuki Toda
Venue: CIKM
Year: 2021

Data often accumulates in tabular format with many attribute items, and prediction using machine learning adds value to data for business. However, studies on machine learning for tabular data only input attribute values, which reduces accuracy. Therefore, we propose an inference method that inputs attribute values and values from aggregated tabular data that has varying attribute values for each attribute item. In an experiment, we compared our proposed method with AutoGluon-Tabular using AutoML benchmark datasets. Our proposed method achieved the highest accuracy for 21 out of 39 datasets.
Attention Based Subgraph Classification for Link Prediction by Network Re-weighting
Authors: Darong Lai, Zheyi Liu, Junyao Huang, Zhihong Chong, Weiwei Wu, Christine Nardini
Venue: CIKM
Year: 2021

Supervised link prediction aims at finding missing links in a network by learning directly from the data suitable criteria for classifying link types into existent or non-existent. Recently, along this line, subgraph-based methods learning a function that maps subgraph patterns to link existence have witnessed great successes. However, these approaches still have drawbacks. First, the construction of the subgraph relies on an arbitrary nodes selection, often ineffective. Second, the inability of such approaches to evaluate adaptively nodes importance reduces flexibility in nodes features aggregation, an important step in subgraph classification. To address these issues, a novel graph-classification based link-prediction model is proposed: Attention and Re-weighting based subgraph Classification for Link prediction (ARCLink). ARCLink first extracts a subgraph around the two nodes whose link should be predicted, by network reweighting, i.e. attributing a weight in the range 0-1 to all links of the original network, and then learns a function to map the subgraph to a continuous vector for classification, thus revealing the nature (non-existence/existence) of the unknown link. For leaning the mapping function, ARCLink generates a vector representation of the extracted subgraph by hierarchically aggregating nodes features according to nodes importance. In contrast to previous studies that either fully ignore or use fixed schemes to compute nodes importance, ARCLink instead learns nodes importance adaptively by employing attention mechanism. Through extensive experiments, ARCLink was validated on a series of real-world networks against state-of-the-art link prediction methods, consistently demonstrating its superior performances