OpenRaiser/Intern-Atlas huggingface dataset:
Evaluating Mixed-initiative Conversational Search Systems via User Simulation
Authors: Ivan Sekulic, Mohammad Aliannejadi, Fabio Crestani
Venue: WSDM
Year: 2022

Clarifying the underlying user information need by asking clarifying questions is an important feature of modern conversational search system. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In this paper, we propose a conversational User Simulator, called USi, for automatic evaluation of such conversational search systems. Given a description of an information need, USi is capable of automatically answering clarifying questions about the topic throughout the search session. Through a set of experiments, including automated natural language generation metrics and crowdsourcing studies, we show that responses generated by USi are both inline with the underlying information need and comparable to human-generated answers. Moreover, we make the first steps towards multi-turn interactions, where conversational search systems asks multiple questions to the (simulated) user with a goal of clarifying the user need. To this end, we expand on currently available datasets for studying clarifying questions, i.e., Qulac and ClariQ, by performing a crowdsourcing-based multi-turn data acquisition. We show that our generative, GPT2-based model, is capable of providing accurate and natural answers to unseen clarifying questions in the single-turn setting and discuss capabilities of our model in the multi-turn setting. We provide the code, data, and the pre-trained model to be used for further research on the topic.
Finding a Concise, Precise, and Exhaustive Set of Near Bi-Cliques in Dynamic Graphs
Authors: Hyeonjeong Shin, Taehyung Kwon, Neil Shah, Kijung Shin
Venue: WSDM
Year: 2022

A variety of tasks on dynamic graphs, including anomaly detection, community detection, compression, and graph understanding, have been formulated as problems of identifying constituent (near) bi-cliques (i.e., complete bipartite graphs). Even when we restrict our attention to maximal ones, there can be exponentially many near bi-cliques, and thus finding all of them is practically impossible for large graphs. Then, two questions naturally arise: (Q1) What is a ''good'' set of near bi-cliques? That is, given a set of near bi-cliques in the input dynamic graph, how should we evaluate its quality? (Q2) Given a large dynamic graph, how can we rapidly identify a high-quality set of near bi-cliques in it? Regarding Q1, we measure how concisely, precisely, and exhaustively a given set of near bi-cliques describes the input dynamic graph. We combine these three perspectives systematically on the Minimum Description Length principle. Regarding Q2, we propose CutNPeel, a fast search algorithm for a high-quality set of near bi-cliques. By adaptively re-partitioning the input graph, CutNPeel reduces the search space and at the same time improves the search quality. Our experiments using six real-world dynamic graphs demonstrate that CutNPeel is (a) High-quality: providing near bi-cliques of up to 51.2% better quality than its state-of-the-art competitors, (b) Fast: up to 68.8X faster than the next-best competitor, and (c) Scalable: scaling to graphs with 134 million edges. We also show successful applications of CutNPeel to graph compression and pattern discovery.
Using Conjunctions for Faster Disjunctive Top-k Queries
Authors: Michal Siedlaczek, Antonio Mallia, Torsten Suel
Venue: WSDM
Year: 2022

While current search engines use highly complex ranking functions with hundreds of features, they often perform an initial candidate generation step that uses a very simple ranking function to identify a limited set of promising candidates. A common approach is to use a disjunctive top-k query for this step. There are many methods for disjunctive top-k computation, but they tend to be slow for the required values of k, which are in the hundreds to thousands. We propose a new approach to safe disjunctive top-k computation that, somewhat counterintuitively, uses precomputed conjunctions of inverted lists to speed up disjunctive queries. The approach is based on a generalization of the well-known MaxScore algorithm, and utilizes recent improvements in threshold estimation techniques as well as new ideas to obtain significant improvements in performance. Our algorithms are implemented as an extension of the PISA framework for search-engine query processing, and available as open-source to support replication and follow-up work.
Attributed Graph Modeling with Vertex Replacement Grammars
Authors: Satyaki Sikdar, Neil Shah, Tim Weninger
Venue: WSDM
Year: 2022

Recent work at the intersection of formal language theory and graph theory has explored graph grammars for graph modeling. However, existing models and formalisms can only operate on homogeneous (i.e., untyped or unattributed) graphs. We relax this restriction and introduce the Attributed Vertex Replacement Grammar (AVRG), which can be efficiently extracted from heterogeneous (i.e., typed, colored, or attributed) graphs. Unlike current state-of-the-art methods, which train enormous models over complicated deep neural architectures, the AVRG model is unsupervised and interpretable. It is based on context-free string grammars and works by encoding graph rewriting rules into a graph grammar containing graphlets and instructions on how they fit together. We show that the AVRG can encode succinct models of input graphs yet faithfully preserve their structure and assortativity properties. Experiments on large real-world datasets show that graphs generated from the AVRG model exhibit substructures and attribute configurations that match those found in the input networks.
Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-shelf
Authors: Michael Soprano, Kevin Roitero, Francesco Bombassei De Bona, Stefano Mizzaro
Venue: WSDM
Year: 2022

Due to their relatively low cost and ability to scale, crowdsourcing based approaches are widely used to collect a large amount of human annotated data. To this aim, multiple crowdsourcing platforms exist, where requesters can upload tasks and workers can carry them out and obtain payment in return. Such platforms share a task design and deploy workflow that is often counter-intuitive and cumbersome. To address this issue, we propose Crowd\_Frame, a simple and complete framework which allows to develop and deploy diverse types of complex crowdsourcing tasks in an easy and customizable way. We show the abilities of the proposed framework and we make it available to researchers and practitioners.
Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning
Authors: Dusan Stamenkovic, Alexandros Karatzoglou, Ioannis Arapakis, Xin Xin, Kleomenis Katevas
Venue: WSDM
Year: 2022

Since the inception of Recommender Systems (RS), the accuracy of the recommendations in terms of relevance has been the golden criterion for evaluating the quality of RS algorithms. However, by focusing on item relevance, one pays a significant price in terms of other important metrics: users get stuck in a "filter bubble" and their array of options is significantly reduced, hence degrading the quality of the user experience and leading to churn. Recommendation, and in particular session-based/sequential recommendation, is a complex task with multiple - and often conflicting objectives - that existing state-of-the-art approaches fail to address. In this work, we take on the aforementioned challenge and introduce Scalarized Multi-Objective Reinforcement Learning (SMORL) for the RS setting, a novel Reinforcement Learning (RL) framework that can effectively address multi-objective recommendation tasks. The proposed SMORL agent augments standard recommendation models with additional RL layers that enforce it to simultaneously satisfy three principal objectives: accuracy, diversity, and novelty of recommendations. We integrate this framework with four state-of-the-art session-based recommendation models and compare it with a single-objective RL agent that only focuses on accuracy. Our experimental results on two real-world datasets reveal a substantial increase in aggregate diversity, a moderate increase in accuracy, reduced repetitiveness of recommendations, and demonstrate the importance of reinforcing diversity and novelty as complementary objectives.
An Interactive Knowledge Graph Based Platform for COVID-19 Clinical Research
Authors: Juntao Su, Edward T. Dougherty, Shuang Jiang, Fang Jin
Venue: WSDM
Year: 2022

Since the first identified case of COVID-19 in December 2019, a plethora of pharmaceuticals and therapeutics have been tested for COVID-19 treatment. While medical advancements and breakthroughs are well underway, the sheer number of studies, treatments, and associated reports makes it extremely challenging to keep track of the rapidly growing COVID-19 research landscape. While existing scientific literature search systems provide basic document retrieval, they fundamentally lack the ability to explore data, and in addition, do not help develop a deeper understanding of COVID-19 related clinical experiments and findings. As research expands, results do so as well, resulting in a position that is complicated and overwhelming. To address this issue, we present a named entity recognition based framework that accurately extracts COVID-19 related information from clinical test results articles, and generates an efficient and interactive visual knowledge graph. This knowledge graph platform is user friendly, and provides intuitive and convenient tools to explore and analyze COVID-19 research data and results including medicinal performances, side effects and target populations.
Automating ETL and Mining of Ethereum Blockchain Network
Authors: Voon Hou Su, Sourav Sen Gupta, Arijit Khan
Venue: WSDM
Year: 2022

The popularity of blockchain technology led to the development of many web platforms with different functionalities. Ethereum, a decentralized, open-source blockchain featuring smart contracts, introduces an interesting ecosystem of human users and autonomous agents (the contracts). It is the most actively used blockchain platform, hosting ether, the second largest cryptocurrency by market capitalization, as its native store of value. The Ethereum blockchain contains a vast amount of user-to-user, user-to-contract, contract-to -user, and contract-to-contract interactions that can be modeled as complex networks. To mine these interactions as graphs through a preferred analytics toolbox, an end-user has to extract, transform, and load (ETL) the required data into the desired network format. However, it is costly and time-consuming to manage the ETL pipeline for the massive and complex blockchain data. To support research in this domain, we develop an end-to-end, automated tool - EtherNet, which performs ETL tasks from a single source of truth (Google BigQuery), and provides graph equivalent representations for visualization and mining on the entire Ethereum blockchain network.
Efficient Reachability Query with Extreme Labeling Filter
Authors: Zhixiang Su, Di Wang, Xiaofeng Zhang, Lizhen Cui, Chunyan Miao
Venue: WSDM
Year: 2022

Being a fundamental graph operator, reachability query has been widely studied by the data mining community in the past decades. In a directed acyclic graph (DAG), one vertex is reachable by another if there exists a chain of directed edges connecting the two vertexes. The state-of-the-art (SOTA) reachability query methods mostly first index all the vertexes in the underlying DAG and assign them with different labels, and then use these indexes and/or labels to efficiently filter out as many unreachable queries as possible. Thus, because a large portion of unreachable queries can be identified without evoking any tedious path-finding process, the overall time taken by a huge number of queries is much shortened with a tolerable compensation on the additional index and/or label preprocessing time and space. In this paper, we propose the Extreme Labeling Filter (ELF), which is a novel generic filter that can be applied to existing reachability query methods to additionally identify a large number of unreachable queries. Based on the analysis of the given DAG in a systematic and autonomous manner, ELF first determines whether to use predecessors or successors to label the vertexes. Based on such self-determined labels, ELF is then able to identify a large number of unreachable queries with a low time complexity of O(1). To evaluate the performance of ELF, we apply it on 4 reachability query methods (1 conventional and 3 SOTA, all designated for reachability query in DAGs) and conduct experiments on 17 datasets of different sizes. The experimental results show that by applying ELF, all methods significantly shorten the query time.
Graph Few-shot Class-incremental Learning
Authors: Zhen Tan, Kaize Ding, Ruocheng Guo, Huan Liu
Venue: WSDM
Year: 2022

The ability to incrementally learn new classes is vital to all real-world artificial intelligence systems. A large portion of high-impact applications like social media, recommendation systems, E-commerce platforms, etc. can be represented by graph models. In this paper, we investigate the challenging yet practical problem,Graph Few-shot Class-incremental (Graph FCL) problem, where the graph model is tasked to classify both newly encountered classes and previously learned classes. Towards that purpose, we put forward a Graph Pseudo Incremental Learning paradigm by sampling tasks recurrently from the base classes, so as to produce an arbitrary number of training episodes for our model to practice the incremental learning skill. Furthermore, we design a Hierarchical-Attention-based Graph Meta-learning framework, HAG-Meta from an optimization perspective. We present a task-sensitive regularizer calculated from task-level attention and node class prototypes to mitigate overfitting onto either novel or base classes. To employ the topological knowledge, we add a node-level attention module to adjust the prototype representation. Our model not only achieves greater stability of old knowledge consolidation, but also acquires advantageous adaptability to new knowledge with very limited data samples. Extensive experiments on three real-world datasets, including Amazon-clothing, Reddit, and DBLP, show that our framework demonstrates remarkable advantages in comparison with the baseline and other related state-of-the-art methods.
Uncovering Causal Effects of Online Short Videos on Consumer Behaviors
Authors: Ziqi Tan, Shengyu Zhang, Nuanxin Hong, Kun Kuang, Yifan Yu, Jin Yu, Zhou Zhao, Hongxia Yang, Shiyuan Pan, Jingren Zhou, Fei Wu
Venue: WSDM
Year: 2022

In recent years, online short videos have become more popular, especially as an online advertising intermediary. To better understand their effects as advertisements, it is essential to analyze the causal relations of online short videos on consumer behaviors. Our study is based on fine-grained consumer behavior data from a world-leading e-commerce platform, i.e., Taobao.com. We first decompose the total causal effects into informative effects and persuasive effects following a common practice in the economic literature. Moreover, we extract the subjectivity scores of short videos through a dictionary-based subjectivity analysis model and evaluate the correlation between the subjectivity scores and each causal effect. The findings of this paper are as follows: First, both causal effects (i.e., informative and persuasive effects) are significant. Second, these effects have a strong correlation with the short videos' subjectivity scores. Third, the signs of these correlations vary with the prices of the products. Our results not only shed light on the research of how short videos exert influence on online consumers, but also give sellers advice on better video design and recommendation.
Predicting Users' Gender and Age based on Mobile Tasks
Authors: Yuan Tian
Venue: WSDM
Year: 2022

Demographic attributes are a key factor in marketing products and services, which enable a business owner to find the ideal customer. Users' app usage behaviors could reveal rich clues regarding their personal attributes since they always determine what apps to use depending on their personal needs and interests. Prior studies [1, 2] have tried to predict users' gender and age through their app usage behavior. However, most of the existing methods for users' demographic prediction are straightforward, simply using popular used apps or app usage frequency as features, without considering the internal semantic relationship of apps usage. Recently, mobile tasks [3] have been identified from mobile app usage logs, representing a more accurate unit for capturing users' goals and behavioral insights, where a "mobile task" can be thought of as a group of related used apps to accomplish a single discrete task. For example, to plan dinner with friends, multiple apps (e.g., WhatsApp, Yelp, Uber and Google Maps) might be accessed for completing the task. In this talk, I will introduce how we leverage the fine-grained task units for generating user representation aims at predicting users' gender and age. We analyzed the effectiveness of using tasks to infer users' demographics, especially when compared to only treating apps independently. We explored different approaches for constructing users' representation and models with both mobile apps and tasks. Finally, we validated that the two-level hierarchical structure of "apps to tasks" and "tasks to users" is an important factor that should be taken into consideration for improving mobile user modelling. This work shed light on whether and how the extracted mobile tasks could be effectively applied. We believe that the task-based representations could be further explored for improving many other applications.
Towards a Cross-domain Semantically Interoperable Ecosystem
Authors: Milenko Tosic, Fábio André Coelho, Barry Nouwt, David Emanuel Rua, Aleksandar Tomcic, Sasa Pesic
Venue: WSDM
Year: 2022

The increasing number of IoT devices and digital services offers cross-domain sensing and control opportunities to a growing set of stakeholders. The provision of cross-domain digital services requires interoperability as a key enabler to bridge domain specifics, while inferring knowledge and allowing new data-driven services. This work addresses H2020 InterConnect project's Interoperability Framework, highlighting the use of semantic web technologies. The interoperability framework layering is presented, particularly addressing the Semantic Interoperability layer as its cornerstone to build an interoperable ecosystem of cross-domain digital services via a federation of distributed knowledge bases. Departing from a generic, ontology-agnostic approach that can fit any cross-domain use case, it validates the approach by considering the SAREF family of ontologies, showcasing an IoT and energy cross-domain use case.
DAME: Domain Adaptation for Matching Entities
Authors: Mohamed Trabelsi, Jeff Heflin, Jin Cao
Venue: WSDM
Year: 2022

Entity matching (EM) identifies data records that refer to the same real-world entity. Despite the effort in the past years to improve the performance in EM, the existing methods still require a huge amount of labeled data in each domain during the training phase. These methods treat each domain individually, and capture the specific signals for each dataset in EM, and this leads to overfitting on just one dataset. The knowledge that is learned from one dataset is not utilized to better understand the EM task in order to make predictions on the unseen datasets with fewer labeled samples. In this paper, we propose a new domain adaptation-based method that transfers the task knowledge from multiple source domains to a target domain. Our method presents a new setting for EM where the objective is to capture the task-specific knowledge from pretraining our model using multiple source domains, then testing our model on a target domain. We study the zero-shot learning case on the target domain, and demonstrate that our method learns the EM task and transfers knowledge to the target domain. We extensively study fine-tuning our model on the target dataset from multiple domains, and demonstrate that our model generalizes better than state-of-the-art methods in EM.
Challenges in Data Production for AI with Human-in-the-Loop
Authors: Dmitry Ustalov
Venue: WSDM
Year: 2022

Today, successful Artificial Intelligence applications rely on three pillars: machine learning algorithms, hardware for running them, and data for training and evaluating models. Although algorithms and hardware have already become commodities, obtaining up-to-date and high-quality data at scale is still challenging-but possible by building hybrid human-computer pipelines called human-in-the-loop. This talk will show how to make a significant business impact using human-in-the-loop pipelines that combine machine learning with crowdsourcing. We will share the experience of one of the world's largest search engines, Yandex. After a brief introduction to human-in-the-loop, we will describe two insightful case studies with a significant business impact at Yandex. First, we will show how to use human-in-the-loop with subjective human opinions to gather training data for learning-to-rank models in the online setting, crucial for the recommendation, e-commerce, and search applications. Second, we will show how human-in-the-loop combined with spatial crowdsourcing enables keeping information on brick-and-mortar businesses up-to-date and transformed into structured data, essential for social impactful applications like online maps and directories. Then, we will present the practical challenges of deploying human-in-the-loop pipelines, focusing on common issues with task design and quality control. We will demonstrate the end-to-end task design techniques that better fit for open-ended and subjective questions compared to widely-used classification tasks. We will present our recent advances in this field, including the use of large-scale language models (like BART and T5) for sequence aggregation. Also, we will show the new evaluation datasets for textual and subjective annotation, which are publicly available at https://toloka.ai/datasets. We will discuss the problem of reliable quality control in crowdsourcing by describing the relevant computational methods for aggregation, quality estimation, and model selection. Finally, we will demonstrate Crowd-Kit, an open-source library that offers battle-tested and platform-agnostic implementations of all the above-described methods in Python: https://github.com/Toloka/crowd-kit. Overall, we will share our experience in running impactful human-in-the-loop pipelines in production while overcoming the common practical challenges using the available and reliable open-source technologies, datasets, and tools.