Reviewerly/exHarmony huggingface dataset:
Implicit Query Parsing at Amazon Product Search

Query Parsing aims to extract product attributes, such as color, brand, and product type, from search queries. These attributes play a crucial role in search engines for tasks such as matching, ranking, and recommendation. There are two types of attributes: explicit attributes that are mentioned explicitly in the search query, and implicit attributes that are mentioned implicitly. Existing works on query parsing do not differentiate between explicit query parsing and implicit query parsing, which limits their performance in product search engines. In this work, we demonstrate the critical importance of implicit attributes in real-world product search engines. We then present our solution for implicit query parsing at Amazon Search, which is a unified framework combining recent advancements in knowledge graph technologies and customer behavior analysis. We demonstrate the effectiveness of our proposal through offline experiments on Amazon search log data. We also show how to deploy and use the framework on Amazon search to improve customers' shopping experiences.
Hierarchical Type Enhanced Negative Sampling for Knowledge Graph Embedding

Knowledge graph embedding aims at modeling knowledge by projecting entities and relations into a low-dimensional semantic space. Most of the works on knowledge graph embedding construct negative samples by negative sampling as knowledge graphs typically only contain positive facts. Although substantial progress has been made by dynamic distribution based sampling methods, selecting plausible and prior information-engaged negative samples still poses many challenges. Inspired by type constraint methods, we propose Hierarchical Type Enhanced Negative Sampling (HTENS) which leverages hierarchical entity type information and entity-relation cooccurrence information to optimize the sampling probability distribution of negative samples. The experiments performed on the link prediction task demonstrate the effectiveness of HTENS. Additionally, HTENS shows its superiority in versatility and can be integrated into scalable systems with enhanced negative sampling.
Force-Feedback and Music: Five Decades of Research and Development at ACROE: An Interview with Claude Cadoz (ACROE, Grenoble, France)

Recorded on March 23 and 2023 [...]
Recipe-MPR: A Test Collection for Evaluating Multi-aspect Preference-based Natural Language Retrieval

The rise of interactive recommendation assistants has led to a novel domain of natural language (NL) recommendation that would benefit from improved multi-aspect reasoning to retrieve relevant items based on NL statements of preference. Such preference statements often involve multiple aspects, e.g., "I would like meat lasagna but I'm watching my weight". Unfortunately, progress in this domain is slowed by the lack of annotated data. To address this gap, we curate a novel dataset which captures logical reasoning over multi-aspect, NL preference-based queries and a set of multiple-choice, multi-aspect item descriptions. We focus on the recipe domain in which multi-aspect preferences are often encountered due to the complexity of the human diet. The goal of publishing our dataset is to provide a benchmark for joint progress in three key areas: 1) structured, multi-aspect NL reasoning with a variety of properties (e.g., level of specificity, presence of negation, and the need for commonsense, analogical, and/or temporal inference), 2) the ability of recommender systems to respond to NL preference utterances, and 3) explainable NL recommendation facilitated by aspect extraction and reasoning. We perform experiments using a variety of methods (sparse and dense retrieval, zero- and few-shot reasoning with large language models) in two settings: a monolithic setting which uses the full query and an aspect-based setting which isolates individual query aspects and aggregates the results. GPT-3 results in much stronger performance than other methods with 73% zero-shot accuracy and 83% few-shot accuracy in the monolithic setting. Aspect-based GPT-3, which facilitates structured explanations, also shows promise with 68% zero-shot accuracy. These results establish baselines for future research into explainable recommendations via multi-aspect preference-based NL reasoning.
NeuralKG-ind: A Python Library for Inductive Knowledge Graph Representation Learning

Since the dynamic characteristics of knowledge graphs, many inductive knowledge graph representation learning (KGRL) works have been proposed in recent years, focusing on enabling prediction over new entities. NeuralKG-ind is the first library of inductive KGRL as an important update of NeuralKG library. It includes standardized processes, rich existing methods, decoupled modules, and comprehensive evaluation metrics. With NeuralKG-ind, it is easy for researchers and engineers to reproduce, redevelop, and compare inductive KGRL methods. The library, experimental methodologies, and model re-implementing results of NeuralKG-ind are all publicly released at https://github.com/zjukg/NeuralKG/tree/ind https://github.com/zjukg/NeuralKG/tree/ind.
Threat Construction for Dynamic Enemy Status in a Platformer Game using Classical Genetic Algorithm

Digital game genres such as Action-Platformer is widely popular among buyers on a platform like Steam. The non-playable character enemies in the game are important in action games. Unfortunately, they usually have static attributes like health point, damage, and enemy movement. Using procedural content generation with a classical genetic algorithm, we drive the threat value of a platform to construct the enemy status resulting in more dynamic enemies. We use the threat value as an input parameter calculated from the enemies’ stats in every platform, such as total damage that the enemy might produce, the player’s health point, and the enemy’s movement speed. We conclude that using a classical genetic algorithm may produce dynamic enemy status through desired threat or danger set by the game designer as an input parameter. Moreover, the game designer may limit the generation with constraints.
The 1st International Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT'23)

The first edition of the Implicit Author Characterization from Texts for Search and Retrieval (IACT'23) aims at bringing to the forefront the challenges involved in identifying and extracting from texts implicit information about authors (e.g., human or AI) and using it in IR tasks. The IACT workshop provides a common forum to consolidate multi-disciplinary efforts and foster discussions to identify the wide-ranging issues related to the task of extracting implicit author-related information from the textual content, including novel tasks and datasets. We will also discuss the ethical implications of implicit information extraction. In addition, we announce a shared task focused on automatically determining the literary epochs of written books.
ReNeuIR at SIGIR 2023: The Second Workshop on Reaching Efficiency in Neural Information Retrieval

Multifaceted, empirical evaluation of algorithmic ideas is one of the central pillars of Information Retrieval (IR) research. The IR community has a rich history of studying the effectiveness of indexes, retrieval algorithms, and complex machine learning rankers and, at the same time, quantifying their computational costs, from creation and training to application and inference. As the community moves towards even more complex deep learning models, questions on efficiency have once again become relevant with renewed urgency. Indeed, efficiency is no longer limited to time and space; instead it has found new, challenging dimensions that stretch to resource-, sample- and energy-efficiency with ramifications for researchers, users, and the environment alike. Examining algorithms and models through the lens of holistic efficiency requires the establishment of standards and principles, from defining relevant concepts, to designing metrics, to creating guidelines for making sense of the significance of new findings. The second iteration of the ReNeuIR workshop aims to bring the community together to debate these questions, with the express purpose of moving towards a common benchmarking framework for efficiency.
Balanced Topic Aware Sampling for Effective Dense Retriever: A Reproducibility Study

Knowledge distillation plays a key role in boosting the effectiveness of rankers based on pre-trained language models (PLMs); this is achieved using an effective but inefficient large model to teach a more efficient student model. In the context of knowledge distillation for a student dense passage retriever, the balanced topic-aware sampling method has been shown to provide state-of-the-art effectiveness. This method intervenes in the creation of the training batches by creating batches that contain positive-negative pairs of passages from the same topic, and balancing the pairwise margins of the positive and negative passages.
RHB-Net: A Relation-aware Historical Bridging Network for Text2SQL Auto-Completion

Test2SQL, a natural language interface to database querying, has seen considerable improvement, in part due to advances in deep learning. However, despite recent improvement, existing Text2SQL proposals allow only input in the form of complete questions. This leaves behind users who struggle to formulate complete questions, e.g., because they lack database expertise or are unfamiliar with the underlying database schema. To address this shortcoming, we study the novel problem of Text2SQL Auto-Completion (TSAC) that extends Text2SQL to also take partial or incomplete questions as input. Specifically, the TSAC problem is to predict the complete, executable SQL query. To solve the problem, we propose a novel Relation-aware Historical Bridging Network (RHB-Net) that consists of a relation-aware union encoder and an extraction-generation sensitive decoder. RHB-Net models relations between questions and database schemas and predicts the ambiguous intents expressed in partial queries. We also propose two optimization strategies: historical query bridging that fuses historical database queries, and a dynamic context construction that prevents repeated generation of the same SQL elements. Extensive experiments with real-world data offer evidence that RHB-Net is capable of outperforming baseline algorithms.
A Preference Judgment Tool for Authoritative Assessment

Preference judgments have been established as an effective method for offline evaluation of information retrieval systems with advantages to graded or binary relevance judgments. Graded judgments assign each document a pre-defined grade level, while preference judgments involve assessing a pair of items presented side by side and indicating which is better. However, leveraging preference judgments may require a more extensive number of judgments, and there are limitations in terms of evaluation measures. In this study, we present a new preference judgment tool called JUDGO, designed for expert assessors and researchers. The tool is supported by a new heap-like preference judgment algorithm that assumes transitivity and allows for ties. An earlier version of the tool was employed by NIST to determine up to the top-10 best items for each of the 38 topics for the TREC 2022 Health Misinformation track, with over 2,200 judgments collected. The current version has been applied in a separate research study to collect almost 10,000 judgments, with multiple assessors completing each topic. The code and resources are available at https://judgo-system.github.io.
Linked-DocRED - Enhancing DocRED with Entity-Linking to Evaluate End-To-End Document-Level Information Extraction Pipelines

Information Extraction (IE) pipelines aim to extract meaningful entities and relations from documents and structure them into a knowledge graph that can then be used in downstream applications. Training and evaluating such pipelines requires a dataset annotated with entities, coreferences, relations, and entity-linking. However, existing datasets either lack entity-linking labels, are too small, not diverse enough, or automatically annotated (that is, without a strong guarantee of the correction of annotations). Therefore, we propose Linked-DocRED, to the best of our knowledge, the first manually-annotated, large-scale, document-level IE dataset. We enhance the existing and widely-used DocRED dataset with entity-linking labels that are generated thanks to a semi-automatic process that guarantees high-quality annotations. In particular, we use hyperlinks in Wikipedia articles to provide disambiguation candidates. We also propose a complete framework of metrics to benchmark end-to-end IE pipelines, and we define an entity-centric metric to evaluate entity-linking. The evaluation of a baseline shows promising results while highlighting the challenges of an end-to-end IE pipeline. Linked-DocRED, the source code for the entity-linking, the baseline, and the metrics are distributed under an open-source license and can be downloaded from a public repository.
The BETTER Cross-Language Datasets

The IARPA BETTER (Better Extraction from Text Through Enhanced Retrieval) program held three evaluations of information retrieval (IR) and information extraction (IE). For both tasks, the only training data available was in English, but systems had to perform cross-language retrieval and extraction from Arabic, Farsi, Chinese, Russian, and Korean. Pooled assessment and information extraction annotation were used to create reusable IR test collections. These datasets are freely available to researchers working in cross-language retrieval, information extraction, or the conjunction of IR and IE. This paper describes the datasets, how they were constructed, and how they might be used by researchers.
Bridging Quantitative and Qualitative Digital Experience Testing

Digital user experiences are a mainstay of modern communication and commerce; multi-billion dollar industries have arisen around optimizing digital design. Usage analytics and A/B testing solutions allow growth hackers to quantitatively compute conversion over key user journeys, while user experience (UX) testing platforms enable UX researchers to qualitatively analyze usability and brand perception. Although these workflows are in pursuit of the same objective - producing better UX - the gulf between quantitative and qualitative testing is wide: they involve different stakeholders, and rely on disparate methodologies, budget, data streams, and software tools. This gap belies the opportunity to create a single platform that optimizes digital experiences holistically: using quantitative methods to uncover what and how much and qualitative analysis to understand why.
Multi-lingual Semantic Search for Domain-specific Applications: Adobe Photoshop and Illustrator Help Search

Search has become an integral part of Adobe products and users rely on it to learn about tool usage, shortcuts, quick links, and ways to add creative effects and to find assets such as backgrounds, templates, and fonts. Within applications such as Photoshop and Illustrator, users express domain-specific search intents via short text queries. In this work, we leverage sentence-BERT models fine-tuned on Adobe's HelpX data to perform multi-lingual semantic search on help and tutorial documents. We used behavioral data (queries, clicks, and impressions) and additional annotated data to train several BERT-based models for scoring query-document pairs for semantic similarity. We benchmarked the keyword-based production system against semantic search. Subsequent AB tests demonstrate that this approach improves engagement for longer queries while reducing null results significantly.
Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and Transformer

In addition to relevance, diversity is an important yet less studied performance metric of cross-modal image retrieval systems, which is critical to user experience. Existing solutions for diversity-aware image retrieval either explicitly post-process the raw retrieval results from standard retrieval systems or try to learn multi-vector representations of images to represent their diverse semantics. However, neither of them is good enough to balance relevance and diversity. On the one hand, standard retrieval systems are usually biased to common semantics and seldom exploit diversity-aware regularization in training, which makes it difficult to promote diversity by post-processing. On the other hand, multi-vector representation methods are not guaranteed to learn robust multiple projections. As a result, irrelevant images and images of rare or unique semantics may be projected inappropriately, which degrades the relevance and diversity of the results generated by some typical algorithms like top-k. To cope with these problems, this paper presents a new method called CoLT that tries to generate much more representative and robust representations for accurately classifying images. Specifically, CoLT first extracts semantics-aware image features by enhancing the preliminary representations of an existing one-to-one cross-modal system with semantics-aware contrastive learning. Then, a transformer-based token classifier is developed to subsume all the features into their corresponding categories. Finally, a post-processing algorithm is designed to retrieve images from each category to form the final retrieval result. Extensive experiments on two real-world datasets Div400 and Div150Cred show that CoLT can effectively boost diversity, and outperforms the existing methods as a whole (with a higher F1 score).
Graph Collaborative Signals Denoising and Augmentation for Recommendation

Graph collaborative filtering (GCF) is a popular technique for capturing high-order collaborative signals in recommendation systems. However, GCF's bipartite adjacency matrix, which defines the neighbors being aggregated based on user-item interactions, can be noisy for users/items with abundant interactions and insufficient for users/items with scarce interactions. Additionally, the adjacency matrix ignores user-user and item-item correlations, which can limit the scope of beneficial neighbors being aggregated.
Adversarial Meta Prompt Tuning for Open Compound Domain Adaptive Intent Detection

Intent detection plays an essential role in dialogue systems. This paper takes the lead to study open compound domain adaptation (OCDA) for intent detection, which brings the advantage of improved generalization to unseen domains. OCDA for intent detection is indeed a more realistic domain adaptation setting, which learns an intent classifier from labeled source domains and adapts it to unlabeled compound target domains containing different intent classes with the source domains. At inference time, we test the intent classifier in open domains that contain previously unseen intent classes. To this end, we propose an Adversarial Meta Prompt Tuning method (called AMPT) for open compound domain adaptive intent detection. Concretely, we propose a meta prompt tuning method, which utilizes language prompts to elicit rich knowledge from large-scale pre-trained language models (PLMs) and automatically finds better prompt initialization that facilitates fast adaptation via meta learning. Furthermore, we leverage a domain adversarial training technique to acquire domain-invariant representations of diverse domains. By taking advantage of the collaborative effect of meta learning, prompt tuning, and adversarial training, we can learn an intent classifier that can effectively generalize to unseen open domains. Experimental results on two benchmark datasets (i.e., HWU64 and CLINC) show that our model can learn substantially better-generalized representations for unseen domains compared with strong competitors.
A Topic-aware Summarization Framework with Different Modal Side Information

Automatic summarization plays an important role in the exponential document growth on the Web. On content websites such as CNN.com and WikiHow.com, there often exist various kinds of side information along with the main document for attention attraction and easier understanding, such as videos, images, and queries. Such information can be used for better summarization, as they often explicitly or implicitly mention the essence of the article. However, most of the existing side-aware summarization methods are designed to incorporate either single-modal or multi-modal side information, and cannot effectively adapt to each other. In this paper, we propose a general summarization framework, which can flexibly incorporate various modalities of side information. The main challenges in designing a flexible summarization model with side information include: (1) the side information can be in textual or visualformat, and the model needs to align and unify it with the document into the same semantic space, (2) the side inputs can contain information from variousaspects, and the model should recognize the aspects useful for summarization. To address these two challenges, we first propose a unified topic encoder, which jointly discovers latent topics from the document and various kinds of side information. The learned topics flexibly bridge and guide the information flow between multiple inputs in a graph encoder through a topic-aware interaction. We secondly propose a triplet contrastive learning mechanism to align the single-modal or multi-modal information into a unified semantic space, where thesummary quality is enhanced by better understanding thedocument andside information. Results show that our model significantly surpasses strong baselines on three public single-modal or multi-modal benchmark summarization datasets.
Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning

Multi-task learning (MTL) has been widely applied in online advertising systems. To address the negative transfer issue, recent optimization methods emphasized the gradient alignment of directions or magnitudes. Since prior studies have proven that the shared modules contain both general and specific knowledge, overemphasizing on gradient alignment may crowd out task-specific knowledge. In this paper, we propose a transference-driven approach CoGrad that adaptively maximizes knowledge transference via Coordinated Gradient modification. We explicitly quantify the transference as loss reduction from one task to another, and optimize it to derive an auxiliary gradient. By incorporating this gradient into original task gradients, the model automatically maximizes inter-task transfer and minimizes individual losses, leading to general and specific knowledge harmonization. Besides, we introduce an efficient approximation of the Hessian matrix, making CoGrad computationally efficient. Both offline and online experiments verify that CoGrad significantly outperforms previous methods.