OpenRaiser/Intern-Atlas huggingface dataset:
Mell: Large-scale extensible user intent classification for dialogue systems with meta lifelong learning
Authors: Chengyu Wang, Haojie Pan, Yuan Liu, Kehan Chen, Minghui Qiu, Wei Zhou, Jun Huang, Haiqing Chen, Wei Lin, Deng Cai
Venue: Conference KDD 2021
Year: 2021

User intent detection is vital for understanding their demands in dialogue systems. Although the User Intent Classification (UIC) task has been widely studied, for large-scale industrial applications, the task is still challenging. This is because user inputs in distinct domains may have different text distributions and target intent sets. When the underlying application evolves, new UIC tasks continuously emerge in a large quantity. Hence, it is crucial to develop a framework for large-scale extensible UIC that continuously fits new tasks and avoids catastrophic forgetting with an acceptable parameter growth rate. In this paper, we introduce the Meta Lifelong Learning (MeLL) framework to address this task. In MeLL, a BERT-based text encoder is employed to learn robust text representations across tasks, which is slowly updated for lifelong learning. We design global and local memory networks to capture the cross-task prototype representations of different classes, treated as the meta-learner quickly adapted to different tasks. Additionally, the Least Recently Used replacement policy is applied to manage the global memory such that the model size does not explode through time. Finally, each UIC task has its own task-specific output layer, with the attentive summarization of various features. We have conducted extensive experiments on both open-source and real industry datasets. Results show that MeLL improves the performance compared with strong baselines and also reduces the number of total parameters. We have also deployed MeLL on a real-world e-commerce dialogue system AliMe and observed significant improvements in terms of both F1 and the resources usage.
On the streaming complexity of computing local clustering coefficients
Authors: Konstantin Kutzkov, Rasmus Pagh
Venue: Conference WSDM 2013
Year: 2013

Due to a large number of applications, the problem of estimating the number of triangles in graphs revealed as a stream of edges, and the closely related problem of estimating the graph's clustering coefficient, have received considerable attention in the last decade. Both efficient algorithms and impossibility results have shed light on the computational complexity of the problem. Motivated by applications in Web mining, Becchetti et al.~presented new algorithms for the estimation of the local number of triangles, i.e., the number of triangles incident to individual vertices. The algorithms are shown, both theoretically and experimentally, to efficiently handle the problem. However, at least two passes over the data are needed and thus the algorithms are not suitable for real streaming scenarios.
Characterizing Search Intent Diversity into Click Models
Authors: Botao Amber Hu, Yuchen Zhang, Weizhu Chen, Gang Wang, Qiang Yang
Venue: Conference WWW 2011
Year: 2011

Modeling a user's click-through behavior in click logs is a challenging task due to the well-known position bias problem. Recent advances in click models have adopted the examination hypothesis which distinguishes document relevance from position bias. In this paper, we revisit the examination hypothesis and observe that user clicks cannot be completely explained by relevance and position bias. Specifically, users with different search intents may submit the same query to the search engine but expect different search results. Thus, there might be a bias between user search intent and the query formulated by the user, which can lead to the diversity in user clicks. This bias has not been considered in previous works such as UBM, DBN and CCM. In this paper, we propose a new intent hypothesis as a complement to the examination hypothesis. This hypothesis is used to characterize the bias between the user search intent and the query in each search session. This hypothesis is very general and can be applied to most of the existing click models to improve their capacities in learning unbiased relevance. Experimental results demonstrate that after adopting the intent hypothesis, click models can better interpret user clicks and achieve a significant NDCG improvement.
Global or Local: Constructing Personalized Click Models for Web Search
Authors: Junqi Zhang, Yiqun Liu, Jiaxin Mao, Xiaohui Xie, Min Zhang, Shaoping Ma, Qi Tian
Venue: Conference WWW 2022
Year: 2022

Click models are widely used for user simulation, relevance inference, and evaluation in Web search. Most existing click models implicitly assume that users’ relevance judgment and behavior patterns are homogeneous. However, previous studies have shown that different users interact with search engines in rather different ways. Therefore, a unified click model can hardly capture the heterogeneity in users’ click behavior. To shed light on this research question, we propose a Click Model Personalization framework (CMP) that adaptively selects from global and local models for individual users. Different adaptive strategies are designed to personalize click behavior modeling only for specific users and queries. We also reveal that capturing personalized behavior patterns is more important than modeling personalized relevance assessments in constructing personalized click models. To evaluate the performance of the proposed CMP framework, we build a large-scale practical Personalized Web Search (PWS) dataset, which consists of the search logs of 1,249 users from a commercial search engine over six months. Experimental results show that the proposed CMP framework achieves significant performance improvements than the non-personalized click models in click prediction.
DeepFEC: Energy consumption prediction under real-world driving conditions for smart cities
Authors: Sayda Elmi, Kian-Lee Tan
Venue: Conference WWW 2021
Year: 2021

The status of air pollution is serious all over the world. Analysing and predicting vehicle energy consumption becomes a major concern. Vehicle energy consumption depends not only on speed but also on a number of external factors such as road topology, traffic, driving style, etc. Obtaining the cost for each link (i.e., link energy consumption) in road networks plays a key role in energy-optimal route planning process. This paper presents a novel framework that identifies vehicle/driving environment-dependent factors to predict energy consumption over a road network based on historical consumption data for different vehicle types. We design a deep-learning-based structure, called DeepFEC, to forecast accurate energy consumption in each and every road in a city based on real traffic conditions. A residual neural network and recurrent neural network are employed to model the spatial and temporal closeness, respectively. Static vehicle data reflecting vehicle type, vehicle weight, engine configuration and displacement are also learned. The outputs of these neural networks are dynamically aggregated to improve the spatially correlated time series data forecasting. Extensive experiments conducted on a diverse fleet consisting of 264 gasoline vehicles, 92 Hybrid Electric Vehicles, and 27 Plug-in Hybrid Electric Vehicles/Electric Vehicles drove in Michigan road network, show that our proposed deep learning algorithm significantly outperforms the state-of-the-art prediction algorithms. To make the results reproductible, the code, the used data and details of the experimental setup are made available online at https://github.com/ElmiSay/DeepFEC.
Personalized long-distance fuel-efficient route recommendation through historical trajectories mining
Authors: Zhan Wang, Zhaohui Peng, Senzhang Wang, Qiao Song
Venue: Conference WSDM 2022
Year: 2022

Finding fuel-efficient routes for drivers has increasingly important value in terms of saving energy, protecting the environment and saving expenses. Previous studies basically adopt simple fuel consumption calculation or prediction methods to recommend the fuel-efficient routes within a city, which have two major limitations. First, the effect of drivers' driving behavior preferences (e.g. acceleration, frequency of clutch use, etc.) on fuel consumption is not fully studied and utilized. Second, existing methods mainly focus on short-distance route recommendation. Due to the difference in the road network structure and route composition, it is not effective to directly apply the route recommendation methods designed for short-distance travel within a city on the scenario of long-distance travel among cities. In this paper, we propose a novel model PLd-FeRR for the Personalized Long-distance Fuel-efficient Route Recommendation. Specifically, we first identify the features reflecting the user's driving behavior preference based on the user's historical driving trajectory, and then extract the potential factors that can affect long-distance fuel consumption. As transformer can effectively capture the temporal features for long sequence data, the extracted personalized driving preference features and long-distance fuel consumption features are input into a transformer-based fuel consumption prediction model. Next, the prediction model is combined with a genetic algorithm to further improve the performance of recommending fuel-efficient routes. Extensive evaluations are conducted on the large real-world dataset, and the results show the effectiveness of our proposal.
User-event graph embedding learning for context-aware recommendation
Authors: Dugang Liu, Mingkai He, Jinwei Luo, Jiangxu Lin, Meng Wang 0009, Xiaolian Zhang, Weike Pan, Zhong Ming 0001
Venue: Conference KDD 2022
Year: 2022

Most methods for context-aware recommendation focus on improving the feature interaction layer, but overlook the embedding layer. However, an embedding layer with random initialization often suffers in practice from the sparsity of the contextual features, as well as the interactions between the users (or items) and context. In this paper, we propose a novel user-event graph embedding learning (UEG-EL) framework to address these two sparsity challenges. Specifically, our UEG-EL contains three modules: 1) a graph construction module is used to obtain a user-event graph containing nodes for users, intents and items, where the intent nodes are generated by applying intent node attention (INA) on nodes of the contextual features; 2) a user-event collaborative graph convolution module is designed to obtain the refined embeddings of all features by executing a new convolution strategy on the user-event graph, where each intent node acts as a hub to efficiently propagate the information among different features; 3) a recommendation module is equipped to integrate some existing context-aware recommendation model, where the feature embeddings are directly initialized with the obtained refined embeddings. Moreover, we identify a unique challenge of the basic framework, that is, the contextual features associated with too many instances may suffer from noise when aggregating the information. We thus further propose a simple but effective variant, i.e., UEG-EL-V, in order to prune the information propagation of the contextual features. Finally, we conduct extensive experiments on three public datasets to verify the effectiveness and compatibility of our UEG-EL and its variant.
Concet: Entity-aware topic classification for open-domain conversational agents
Authors: Ali Ahmadvand, Harshita Sahijwani, Jason Ingyu Choi, Eugene Agichtein
Venue: Conference CIKM 2019
Year: 2019

Identifying the topic (domain) of each user's utterance in open-domain conversational systems is a crucial step for all subsequent language understanding and response tasks. In particular, for complex domains, an utterance is often routed to a single component responsible for that domain. Thus, correctly mapping a user utterance to the right domain is critical. This is a challenging task: users could mention entities like actors, singers or locations to implicitly indicate the domain, which requires extensive domain knowledge to interpret. To address this problem, we introduce ConCET: a Concurrent Entity-aware conversational Topic classifier, which incorporates entity type information together with the utterance content features. Specifically, ConCET utilizes entity information to enrich the utterance representation, combining character, word, and entity type embeddings into a single representation. However, for rich domains with millions of available entities, unrealistic amounts of labeled training data would be required. To complement our model, we propose a simple and effective method for generating synthetic training data, to augment the typically limited amounts of labeled training data, using commonly available knowledge bases as to generate additional labeled utterances. We extensively evaluate ConCET and our proposed training method first on an openly available human-human conversational dataset called Self-Dialogue, to calibrate our approach against previous state-of-the-art methods; second, we evaluate ConCET on a large dataset of human-machine conversations with real users, collected as part of the Amazon Alexa Prize. Our results show that ConCET significantly improves topic classification performance on both datasets, reaching 8-10% improvements compared to state-of-the-art deep learning methods. We complement our quantitative results with detailed analysis of system performance, which could be used for further improvements of conversational agents.
ASPIRE: Air Shipping Recommendation for E-Commerce Products via Causal Inference Framework
Authors: Abhirup Mondal, Anirban Majumder, Vineet Chaoji
Venue: Conference KDD 2022
Year: 2022

Speed of delivery is critical for the success of e-commerce platforms. Faster delivery promise to the customer results in increased conversion and revenue. There are typically two mechanisms to control the delivery speed - a) replication of products across warehouses, and b) air-shipping the product. In this paper, we present a machine learning based framework to recommend air-shipping eligibility for products. Specifically, we develop a causal inference framework (referred to as Air Shipping Recommendation or ASPIRE) that balances the trade-off between revenue or conversion and delivery cost to decide whether a product should be shipped via air. We propose a doubly-robust estimation technique followed by an optimization algorithm to determine air eligibility of products and calculate the uplift in revenue and shipping cost.
Hierarchical training: Scaling deep recommendation models on large cpu clusters
Authors: Yuzhen Huang, Xiaohan Wei, Xing Wang, Jiyan Yang, Bor-Yiing Su, Shivam Bharuka, Dhruv Choudhary, Zewei Jiang, Hai Zheng, Jack Langman
Venue: Conference KDD 2021
Year: 2021

Neural network based recommendation models are widely used to power many internet-scale applications including product recommendation and feed ranking. As the models become more complex and more training data is required during training, improving the training scalability of these recommendation models becomes an urgent need. However, improving the scalability without sacrificing the model quality is challenging. In this paper, we conduct an in-depth analysis of the scalability bottleneck in existing training architecture on large scale CPU clusters. Based on these observations, we propose a new training architecture called Hierarchical Training, which exploits both data parallelism and model parallelism for the neural network part of the model within a group. We implement hierarchical training with a two-layer design: a tagging system that decides the operator placement and a net transformation system that materializes the training plans, and integrate hierarchical training into existing training stack. We propose several optimizations to improve the scalability of hierarchical training including model architecture optimization, communication compression, and various system-level improvements. Extensive experiments at massive scale demonstrate that hierarchical training can speed up distributed recommendation model training by 1.9x without model quality drop.
Using survival models to estimate user engagement in online experiments
Authors: Praveen Chandar, Brian St. Thomas, Lucas Maystre, Vijay Pappu, Roberto Sanchis-Ojeda, Tiffany Wu, Ben Carterette, Mounia Lalmas, Tony Jebara
Venue: Conference WWW 2022
Year: 2022

Online controlled experiments, in which different variants of a product are compared based on an Overall Evaluation Criterion (OEC), have emerged as a gold standard for decision making in online services. It is vital that the OEC is aligned with the overall goal of stakeholders for effective decision making. However, this is a challenge when the overall goal is not immediately observable. For instance, we might want to understand the effect of deploying a feature on long-term retention, where the outcome (retention) is not observable at the end of an A/B test. In this work, we examine long-term user engagement outcomes as a time-to-event problem and demonstrate the use of survival models for estimating long-term effects. We then discuss the practical challenges in using time-to-event metrics for decision making in online experiments. We propose a simple churn-based time-to-inactivity metric and describe a framework for developing & validating modeled metrics using survival models for predicting long-term retention. Then, we present a case study and provide practical guidelines on developing and evaluating a time-to-churn metric on a large scale real-world dataset of online experiments. Finally, we compare the proposed approach to existing alternatives in terms of sensitivity and directionality.
Studying Long-Term User Behaviour Using Dynamic Time Warping for Customer Retention
Authors: Harsha Gwalani
Venue: Conference WSDM 2022
Year: 2022

Dynamic Time Warping (DTW) is an asynchronous alignment algorithm used to measure similarities between temporal sequences. DTW is advantageous for comparing sequences when the shape of the pattern is more important than the speed of events. It has been widely used for automated speech recognition, pattern detection in stock pricing data, studying energy consumption patterns for appliances, etc. In this presentation, we discuss the use of dynamic time warping for understanding long-term usage patterns on Twitter. Time series data are more useful to understand repeat user behavior and build user narratives than aggregated and/or snapshot user features. We utilize the time series of different user metrics as temporal signals and cluster them to identify specific usage and engagement patterns for churning users (users who become inactive). This approach led to more accurate opportunity sizing for the different user personas which in turn helped prioritize interventions for customer retention. We will discuss the implementation of this approach for large-scale data, understanding the time series clusters in a human-understandable manner, and challenges associated with multi-dimensional time-series data in the presentation.
Numerical Feature Representation with Hybrid N-ary Encoding
Authors: Bo Chen, Huifeng Guo, Weiwen Liu, Yue Ding, Yunzhe Li, Wei Guo, Yichao Wang, Zhicheng He, Ruiming Tang, Rui Zhang
Venue: Conference CIKM 2022
Year: 2022

Numerical features (e.g., statistical features) are widely used in recommender systems and online advertising. Existing approaches for numerical feature representation in industry are primarily based on discretization. However, hard-discretization based methods (e.g., Equal Distance Discretization) are deficient in continuity while soft-discretization based methods (e.g., AutoDis) lack discriminability. To emphasize both continuity and discriminability for numerical features, we propose an end-to-end representation learning framework named NaryDis. Specifically, NaryDis first leverages hybrid n-ary encoding as an automatic discretization module to generate hybrid-grained discretization results (multiple encoded sequences). Each position of the encoded sequence is assigned with a positional embedding and an intra-ary attention network is leveraged to aggregate the positional embeddings for obtaining ary-wise representations. Then an inter-ary attention is adopted to assemble these representations, which are further constrained by a self-supervised regularization module. Comprehensive experiments on two public datasets are conducted to show the superiority and compatibility of NaryDis. Besides, we deeply investigate the properties of continuity and discriminability. Moreover, we further verify the effectiveness of NaryDis on a large-scale industrial advertisement dataset.
Augmenting Log-based Anomaly Detection Models to Reduce False Anomalies with Human Feedback
Authors: Tong Jia, Ying Li 0012, Yong Yang 0011, Gang Huang 0001, Zhonghai Wu
Venue: Conference KDD 2022
Year: 2022

With the increasing complexity of modern software systems, it is essential yet hard to detect anomalies and diagnose problems precisely. Existing log-based anomaly detection approaches rely on a few key assumptions on system logs and perform well in some experimental systems. However, real-world industrial systems are often with poor logging quality, in which system logs are noisy and often violate the assumptions of existing approaches. This makes these approaches inefficient. This paper first conducts a comprehensive study on the system logs of three large-scale industrial software systems. Through the study, we identify four typical anti-patterns that affect the detection results the most. Based on these patterns, we propose HiLog, an effective human-in-the-loop log-based anomaly detection approach that integrates human knowledge to augment anomaly detection models. With little human labeling effort, our approach can significantly improve the effectiveness of existing models. Experiment results on three large-scale industrial software systems show that our method improves over 50% precision rate on average.
A Comparison of Classifiers and Document Representations for the Routing Problem
Authors: Hinrich Schütze, David A. Hull, Jan O. Pedersen
Venue: Conference SIGIR 1995
Year: 1995

In this paper, we compare learning techniques based on statistical classification to traditional methods of relevance feedback for the document routing problem. We consider three classification techniques which have decision rules that are derived via explicit error minimization linear discriminant analysis, logistic regression, and neuraf networks. We demonstrate that the classifiers perform 10-15% better than relevance feedback via Rocchio expansion for the TREC-2 and TREC-3 routing tasks.
Building structures from classifiers for passage reranking
Authors: Aliaksei Severyn, Massimo Nicosia, Alessandro Moschitti
Venue: Conference CIKM 2013
Year: 2013

This paper shows that learning to rank models can be applied to automatically learn complex patterns, such as relational semantic structures occurring in questions and their answer passages. This is achieved by providing the learning algorithm with a tree representation derived from the syntactic trees of questions and passages connected by relational tags, where the latter are again provided by the means of automatic classifiers, i.e., question and focus classifiers and Named Entity Recognizers. This way effective structural relational patterns are implicitly encoded in the representation and can be automatically utilized by powerful machine learning models such as kernel methods.