OpenRaiser/Intern-Atlas huggingface dataset:
Knowledge-enhanced Multi-View Graph Neural Networks for Session-based Recommendation
Authors: Qian Chen, Zhiqiang Guo, Jianjun Li, Guohui Li
Venue: SIGIR
Year: 2023

Session-based recommendation (SBR) has received increasing attention to predict the next item via extracting and integrating both global and local item-item relationships. However, there still exist some deficiencies in current works when capturing these two kinds of relationships. For global item-item relationships, the global graph constructed by most SBR is a pseudo-global graph, which may cause redundant mining of sequence relationships. For local item-item relationships, conventional SBR only mines the sequence patterns while ignoring the feature patterns, which may introduce noise when learning users' interests. To address these problems, we propose a novel Knowledge-enhanced Multi-View Graph Neural Network (KMVG) by constructing three views, namely knowledge view, session view, and pairwise view. Specifically, benefiting from the rich semantic information in the knowledge graph (KG), we build a genuine global graph that is sequence-independent based on KG to mine the global item-item relationships in the knowledge view. Then, a session view is utilized to capture the contextual transitions among items as the sequence patterns of local item-item relationships, and a pairwise view is used to explore the feature commonality within a session as the feature patterns of the local item-item relationships. Extensive experiments on three real-world public datasets demonstrate the superiority of KMVG, showing that it outperforms the state-of-the-art baselines. Further analysis also reveals the effectiveness of KMVG in exploiting the item-item relationships under multiple views.
Practice and Challenges in Building a Business-oriented Search Engine Quality Metric
Authors: Nuo Chen, Donghyun Park, Hyungae Park, Kijun Choi, Tetsuya Sakai, Jinyoung Kim
Venue: SIGIR
Year: 2023

One of the most challenging aspects of operating a large-scale web search engine is to accurately evaluate and monitor the search engine's result quality regardless of search types. From a business perspective, in the face of such challenges, it is important to establish a universal search quality metric that can be easily understood by the entire organisation. In this paper, we introduce a model-based quality metric using Explainable Boosting Machine as the classifier and online user behaviour signals as features to predict search quality. The proposed metric takes into account a variety of search types and has good interpretability. To examine the performance of the metric, we constructed a large dataset of user behaviour on search engine results pages (SERPs) with SERP quality ratings from professional annotators. We compared the performance of the model in our metric to those of other black-box machine learning models on the dataset. We also share a few experiences within our company for the org-wide adoption of this metric relevant to metric design.
Using Entropy for Group Sampling in Pairwise Ranking from implicit feedback
Authors: Yujie Chen, Runlong Yu, Qi Liu, Enhong Chen, Zhenya Huang
Venue: SIGIR
Year: 2023

In recent years, pairwise methods, such as Bayesian Personalized Ranking (BPR), have gained significant attention in the field of collaborative filtering for recommendation systems. Group BPR is an extension of BPR that incorporates user groups to relax the strict assumption of independence between two users. However, the reliability of its user groups may be compromised as they only focus on a few behavioral similarities. To address this problem, this paper proposes a new entropy-weighted similarity measure for implicit feedback to quantify the relation between two users and sample like-minded user groups. We first introduce the group preference into several pairwise ranking algorithms and then utilize the entropy-weighted similarity to sample groups to further improve these algorithms. Unlike other approaches that rely solely on common item ratings, our method incorporates global information into the similarity measure, resulting in a more reliable approach to group sampling. We conducted experiments on two real-world datasets and evaluated our method using different metrics. The results show that our method can construct better user groups from sparse data and produce more accurate recommendations. Our approach can be applied to a wide range of recommendation systems, and this can significantly improve the performance of pairwise ranking algorithms, making it an effective tool for pairwise ranking.
How Significant Attributes are in the Community Detection of Attributed Multiplex Networks
Authors: Junwei Cheng, Chaobo He, Kunlin Han, Wenjie Ma, Yong Tang
Venue: SIGIR
Year: 2023

Existing community detection methods for attributed multiplex networks focus on exploiting the complementary information from different topologies, while they are paying little attention to the role of attributes. However, we observe that real attributed multiplex networks exhibit two unique features, namely, consistency and homogeneity of node attributes. Therefore, in this paper, we propose a novel method, called ACDM, which is based on these two characteristics of attributes, to detect communities on attributed multiplex networks. Specifically, we extract commonality representation of nodes through the consistency of attributes. The collaboration between the homogeneity of attributes and topology information reveals the particularity representation of nodes. The comprehensive experimental results on real attributed multiplex networks well validate that our method outperforms state-of-the-art methods in most networks.
Fairness for both Readers and Authors: Evaluating Summaries of User Generated Content
Authors: Garima Chhikara, Kripabandhu Ghosh, Saptarshi Ghosh, Abhijnan Chakraborty
Venue: SIGIR
Year: 2023

Summarization of textual content has many applications, ranging from summarizing long documents to recent efforts towards summarizing user generated text (e.g., tweets, Facebook or Reddit posts). Traditionally, the focus of summarization has been to generate summaries which can best satisfy the readers. In this work, we look at summarization of user-generated content as a two-sided problem where satisfaction of both readers and authors is crucial. Through three surveys, we show that for user-generated content, traditional evaluation approach of measuring similarity between reference summaries and algorithmic summaries cannot capture author satisfaction. We propose an author satisfaction-based evaluation metric CROSSEM which, we show empirically, can potentially complement the current evaluation paradigm. We further propose the idea of inequality in satisfaction, to account for individual fairness amongst readers and authors. To our knowledge, this is the first attempt towards developing a fair summary evaluation framework for user generated content, and is likely to spawn lot of future research in this space.
PEPO: Petition Executing Processing Optimizer Based on Natural Language Processing
Authors: Yin-Wei Chiu, Hsiao-Ching Huang, Cheng-Ju Lee, Hsun-Ping Hsieh
Venue: SIGIR
Year: 2023

In this paper, we propose "Petition Executing Process Optimizer (PEPO)," an AI-based petition processing system that features three components, (a) Department Classification, (b) Importance Assessment, and (c) Response Generation for improving the Public Work Bureau (PWB) 1999 Hotline petitions handling process in Taiwan. Our Department Classification algorithm has been evaluated with NDCG, achieving an impressive score of 86.48%, while the Important Assessment function has an accuracy rate of 85%. Besides, Response Generation enhances communication efficiency between the government and citizens. The PEPO system has been deployed as an online web service for the Public Works Bureau of the Tainan City Government. With PEPO, the PWB benefits greatly from the effectiveness and efficiency of handling citizens' petitions.
One Stop Shop for Question-Answering Dataset Selection
Authors: Chang Nian Chuy, Qinmin Vivian Hu, Chen Ding
Venue: SIGIR
Year: 2023

In this paper, we offer a new visualization tool -- Dataset Statistical View (DSV), to lower the barrier of research entry by providing easy access to the question-answering (QA) datasets that researchers can build their work upon. Our target users are new researchers to the QA domain with no prior knowledge nor programming skills. The system is populated with multiple QA datasets, which covers a wide range of QA tasks. It allows researchers to explore and compare existing QA datasets at a one-stop website. The system shows statistical graphs for each QA dataset to offer an overview and a visual comparison between datasets. Although this paper focuses mainly at the syntactic level comparison, integrating bias and semantic level analysis is our ongoing work. We believe our DSV system is a valuable contribution to the advancement of the QA field, as it provides a solid starting point for new researchers and practitioners. An overview of the framework is demonstrated in this paper and the introduction of the application system is available at https://cnchuy.github.io/images/demo.mp4.
Exploiting Cluster-Skipping Inverted Index for Semantic Place Retrieval
Authors: Enes Recep Cinar, Ismail Sengor Altingovde
Venue: SIGIR
Year: 2023

Semantic place retrieval aims to find the top-k place entities, which are both textually relevant and spatially close to a given query, from a knowledge graph. In this work, our contribution toward improving the efficiency of semantic place retrieval is two-fold. First, we show that by applying an ad hoc yet intuitive restriction on the depth of search on the knowledge graph, it is possible to adopt IR-tree indexing scheme [7], which has been introduced for processing spatial keyword queries, for the semantic place retrieval scenario. Secondly, as a novel solution to this problem, we adapt the idea of cluster-skipping inverted index (CS-IIS) [1, 4], which has been originally proposed for retrieval over topically clustered document collections. Our experiments show that CS-IIS is comparable to IR-tree in terms of CPU time, while it yields substantial efficiency gains in terms of I/O time during query processing.
An Effective, Efficient, and Scalable Confidence-based Instance Selection Framework for Transformer-Based Text Classification
Authors: Washington Cunha, Celso França, Guilherme Fonseca, Leonardo Rocha, Marcos André Gonçalves
Venue: SIGIR
Year: 2023

Transformer-based deep learning is currently the state-of-the-art in many NLP and IR tasks. However, fine-tuning such Transformers for specific tasks, especially in scenarios of ever-expanding volumes of data with constant re-training requirements and budget constraints, is costly (computationally and financially) and energy-consuming. In this paper, we focus on Instance Selection (IS) - a set of methods focused on selecting the most representative documents for training, aimed at maintaining (or improving) classification effectiveness while reducing total time for training (or fine-tuning). We propose E2SC-IS -- Effective, Efficient, and Scalable Confidence-Based IS -- a two-step framework with a particular focus on Transformers and large datasets. E2SC-IS estimates the probability of each instance being removed from the training set based on scalable, fast, and calibrated weak classifiers. E2SC-IS also exploits iterative heuristics to estimate a near-optimal reduction rate. Our solution can reduce the training sets by 29% on average while maintaining the effectiveness in all datasets, with speedup gains up to 70%, scaling for very large datasets (something that the baselines cannot do).
Neuro-Symbolic Representations for Information Retrieval
Authors: Laura Dietz, Hannah Bast, Shubham Chatterjee, Jeffrey Dalton, Jian-Yun Nie, Rodrigo Nogueira
Venue: SIGIR
Year: 2023

This tutorial will provide an overview of recent advances on neuro-symbolic approaches for information retrieval. A decade ago, knowledge graphs and semantic annotations technology led to active research on how to best leverage symbolic knowledge. At the same time, neural methods have demonstrated to be versatile and highly effective. From a neural network perspective, the same representation approach can service document ranking or knowledge graph reasoning. End-to-end training allows to optimize complex methods for downstream tasks. We are at the point where both the symbolic and the neural research advances are coalescing into neuro-symbolic approaches. The underlying research questions are how to best combine symbolic and neural approaches, what kind of symbolic/neural approaches are most suitable for which use case, and how to best integrate both ideas to advance the state of the art in information retrieval. Materials are available online: https://github.com/laura-dietz/neurosymbolic-representations-for-IR
Rows or Columns? Minimizing Presentation Bias When Comparing Multiple Recommender Systems
Authors: Patrik Dokoupil, Ladislav Peska, Ludovico Boratto
Venue: SIGIR
Year: 2023

Going beyond accuracy in the evaluation of a recommender system is an aspect that is receiving more and more attention. Among the many perspectives that can be considered, the impact of presentation bias is of central importance. Under presentation bias, the attention of the users to the items in a recommendation list changes, thus affecting their possibility to be considered and the effectiveness of a model. Page-wise within-subject studies are widely employed in the recommender systems literature to compare algorithms by displaying their results in parallel. However, no study has ever been performed to assess the impact of presentation bias in this context. In this paper, we characterize how presentation bias affects different layout options, which present the results in column- or row-wise fashion. Concretely, we present a user study where six layout variants are proposed to the users in a page-wise within-subject setting, so as to evaluate their perception of the displayed recommendations. Results show that presentation bias impacts users clicking behavior (low-level feedback), but not so much the perceived performance of a recommender system (high-level feedback). Source codes and raw results are available at https://tinyurl.com/PresBiasSIGIR2023.
Dimension-Prompts Boost Commonsense Consolidation
Authors: Jiazhan Feng, Chongyang Tao, Tao Shen, Chang Liu, Dongyan Zhao
Venue: SIGIR
Year: 2023

Neural knowledge models emerged and advanced common-sense-centric knowledge grounding. They parameterize a small seed curated commonsense knowledge graph (CS-KG) in a language model to generalize more. A current trend is to scale the seed up by directly mixing multiple sources of CS-KG (e.g., ATOMIC, ConceptNet) into one model. But, such brute-force mixing inevitably hinders effective knowledge consolidation due to i) ambiguous, polysemic, and/or inconsistent relations across sources and ii) knowledge learned in an entangled manner despite distinct types (e.g., causal, temporal). To mitigate this, we adopt a concept of commonsense knowledge dimension and propose a brand-new dimension-disentangled knowledge model (D2KM) learning paradigm with multiple sources. That is, a generative language model with dimension-specific soft prompts is trained to disentangle knowledge acquisitions along with different dimensions and facilitate potential intra-dimension consolidation across CS-KG sources. Experiments show our knowledge model outperforms its baselines in both standard and zero-shot scenarios.
Wisdom of Crowds and Fine-Grained Learning for Serendipity Recommendations
Authors: Zhe Fu, Xi Niu, Li Yu
Venue: SIGIR
Year: 2023

Serendipity is a notion that means an unexpected but valuable discovery. Due to its elusive and subjective nature, serendipity is difficult to study even with today's advances in machine learning and deep learning techniques. Both ground truth data collecting and model developing are the open research questions. This paper addresses both the data and the model challenges for identifying serendipity in recommender systems. For the ground truth data collecting, it proposes a new and scalable approach by using both user generated reviews and a crowd sourcing method. The result is a large-scale ground truth data on serendipity. For model developing, it designed a self-enhanced module to learn the fine-grained facets of serendipity in order to mitigate the inherent data sparsity problem in any serendipity ground truth dataset. The self-enhanced module is general enough to be applied with many base deep learning models for serendipity. A series of experiments have been conducted. As the result, a base deep learning model trained on our collected ground truth data, as well as with the help of the self-enhanced module, outperforms the state-of-the-art baseline models in predicting serendipity.
AutoDPQ: Automated Differentiable Product Quantization for Embedding Compression
Authors: Xin Gan, Yuhao Wang, Xiangyu Zhao, Wanyu Wang, Yiqi Wang, Zitao Liu
Venue: SIGIR
Year: 2023

Deep recommender systems typically involve numerous feature fields for users and items, with a large number of low-frequency features. These low-frequency features would reduce the prediction accuracy with large storage space due to their vast quantity and inadequate training. Some pioneering studies have explored embedding compression techniques to address this issue of the trade-off between storage space and model predictability. However, these methods have difficulty compacting the embedding of low-frequency features in various feature fields due to the high demand for human experience and computing resources during hyper-parameter searching. In this paper, we propose the AutoDPQ framework, which automatically compacts low-frequency feature embeddings for each feature field to an adaptive magnitude. Experimental results indicate that AutoDPQ can significantly reduce the parameter space while improving recommendation accuracy. Moreover, AutoDPQ is compatible with various deep CTR models by improving their performance significantly with high efficiency.
Searching for Reliable Facts over a Medical Knowledge Base
Authors: Fabio Giachelle, Stefano Marchesin, Gianmaria Silvello, Omar Alonso
Venue: SIGIR
Year: 2023

This work presents CoreKB, a Web platform for searching reliable facts over gene expression-cancer associations Knowledge Base (KB). It provides search capabilities over an RDF graph using natural language queries, structured facets, and autocomplete. CoreKB is designed to be intuitive and easy to use for healthcare professionals, medical researchers, and clinicians. The system offers the user a comprehensive overview of the scientific evidence supporting a medical fact. It provides a quantitative comparison between the possible gene-cancer associations a particular fact can reflect.
PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting
Authors: Zixin Guo, Tzu-Jui Julius Wang, Selen Pehlivan, Abduljalil Radman, Jorma Laaksonen
Venue: SIGIR
Year: 2023

Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models over a wide range of VL downstream tasks, especially for cross-modal retrieval. However, it hinges on a huge amount of image-text pairs, which requires tedious and costly curation. On the contrary,weakly-supervised VLP (W-VLP) explores means with object tags generated by a pre-trained object detector (OD) from images. Yet, they still require paired information, i.e. images and object-level annotations, as supervision to train an OD. To further reduce the amount of supervision, we propose Prompts-in-The-Loop (PiTL) that prompts knowledge from large language models (LLMs) to describe images. Concretely, given a category label of an image, e.g.refinery, the knowledge, e.g.a refinery could be seen with large storage tanks, pipework, and ..., extracted by LLMs is used as the language counterpart. The knowledge supplements, e.g. the common relations among entities most likely appearing in a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of 14K categories from ImageNet21K with PiTL. Empirically, the VL models pre-trained with PiTL-generated pairs are strongly favored over other W-VLP works on image-to-text (I2T) and text-to-image (T2I) retrieval tasks, with less supervision. The results reveal the effectiveness of PiTL-generated pairs for VLP.
Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
Authors: Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, Harrie Oosterhuis
Venue: SIGIR
Year: 2023

Since its inception, the field of unbiased learning to rank (ULTR) has remained very active and has seen several impactful advancements in recent years. This tutorial provides both an introduction to the core concepts of the field and an overview of recent advancements in its foundations along with several applications of its methods. The tutorial is divided into four parts: Firstly, we give an overview of the different forms of bias that can be addressed with ULTR methods. Secondly, we present a comprehensive discussion of the latest estimation techniques in the ULTR field. Thirdly, we survey published results of ULTR in real-world applications. Fourthly, we discuss the connection between ULTR and fairness in ranking. We end by briefly reflecting on the future of ULTR research and its applications. This tutorial is intended to benefit both researchers and industry practitioners who are interested in developing new ULTR solutions or utilizing them in real-world applications.
ErrorCLR: Semantic Error Classification, Localization and Repair for Introductory Programming Assignments
Authors: Siqi Han, Yu Wang, Xuesong Lu
Venue: SIGIR
Year: 2023

Programming education at scale increasingly relies on automated feedback to help students learn to program. An important form of feedback is to point out semantic errors in student programs and provide hints for program repair. Such automated feedback depends essentially on solving the tasks of classification, localization and repair of semantic errors. Although there are datasets for the tasks, we observe that they do not have the annotations supporting all three tasks. As such, existing approaches for semantic error feedback treat error classification, localization and repair as independent tasks, resulting in sub-optimal performance on each task. Moreover, existing datasets either contain few programming assignments or have few programs for each assignment. Therefore, existing approaches often leverage rule-based methods and evaluate them with a small number of programming assignments. To tackle the problems, we first describe the creation of a new dataset COJ2022 that contains 5,914 C programs with semantic errors submitted to 498 different assignments in an introductory programming course, where each program is annotated with the error types and locations and is coupled with the repaired program submitted by the same student. We show the advantages of COJ2022 over existing datasets on various aspects. Second, we treat semantic error classification, localization and repair as dependent tasks, and propose a novel two-stage method ErrorCLR to solve them. Specifically, in the first stage we train a model based on graph matching networks to jointly classify and localize potential semantic errors in student programs, and in the second stage we mask error spans in buggy programs using information of error types and locations and train a CodeT5 model to predict correct spans. The predicted spans replace the error spans to form repaired programs. Experimental results show that ErrorCLR remarkably outperforms the comparative methods for all three tasks on COJ2022 and other public datasets. We also conduct a case study to visualize and interpret what is learned by the graph matching network in ErrorCLR. We have released the source code and COJ2022 at https://github.com/DaSESmartEdu/ErrorCLR.