Miracle-dz/UniFAR-FTUs huggingface dataset:
External Evaluation of Ranking Models under Extreme Position-Bias

Implicit feedback from users behavior is a natural and scalable source for training and evaluating ranking models in human-interactive systems. However, inherent biases such as the position bias are key obstacles to its effective usage. This is further accentuated in cases of extreme bias, where behavioral feedback can be collected exclusively on the top ranked result. In fact, in such cases, state-of-art debiasing methods cannot be applied. A prominent use case of extreme position bias is the voice shopping medium, where only a small amount of information can be presented to the user during a single interaction, resulting in user behavioral signals that are almost exclusively limited to the top offer. There is no way to know how the user would have reacted to a different offer than the top one he was actually exposed to. Thus, any new ranker we wish to evaluate with respect to a behavioral metric, requires online experimentation. We propose a novel approach, based on anexternal estimator model, for accurately predicting offline the performance of a new ranker. The accuracy of our solution is proven theoretically, as well as demonstrated by a line of experiments. In these experiments, we focus on the use case of purchase prediction, and show that our estimator can accurately predict offline the purchase rate of different rankers over a segment of voice shopping traffic. Our prediction is validated online, as being compared to the actual performance obtained by each ranker when being exposed to users.
Doctoral Consortium of WSDM'22: Exploring the Bias of Adversarial Defenses

Deep neural networks (DNNs) have achieved extraordinary accomplishments on various machine learning tasks. However, the existence of adversarial attacks still raise great concerns when they are adopted to safety-critical tasks. As countermeasures to protect DNN models against adversarial attacks, there are various defense strategies proposed. However, we find that the robustness ("safety'') provided by the robust training algorithms usually result unequal performance either among classes or sub-populations across the whole data distribution. For example, the model can achieve extremely low accuracy / robustness on certain groups of data. As a result, the safety of the model is still under great threats. As a summary, our project is about to study the bias problems of robust trained neural networks from different perspectives, which aims to build eventually reliable and safe deep learning models. We propose to present our research works in the Doctoral Consortium in WSDM'22 and gain opportunities to share our contribution to the relate problems.
Using Conjunctions for Faster Disjunctive Top-k Queries

While current search engines use highly complex ranking functions with hundreds of features, they often perform an initial candidate generation step that uses a very simple ranking function to identify a limited set of promising candidates. A common approach is to use a disjunctive top-k query for this step. There are many methods for disjunctive top-k computation, but they tend to be slow for the required values of k, which are in the hundreds to thousands. We propose a new approach to safe disjunctive top-k computation that, somewhat counterintuitively, uses precomputed conjunctions of inverted lists to speed up disjunctive queries. The approach is based on a generalization of the well-known MaxScore algorithm, and utilizes recent improvements in threshold estimation techniques as well as new ideas to obtain significant improvements in performance. Our algorithms are implemented as an extension of the PISA framework for search-engine query processing, and available as open-source to support replication and follow-up work.
DAME: Domain Adaptation for Matching Entities

Entity matching (EM) identifies data records that refer to the same real-world entity. Despite the effort in the past years to improve the performance in EM, the existing methods still require a huge amount of labeled data in each domain during the training phase. These methods treat each domain individually, and capture the specific signals for each dataset in EM, and this leads to overfitting on just one dataset. The knowledge that is learned from one dataset is not utilized to better understand the EM task in order to make predictions on the unseen datasets with fewer labeled samples. In this paper, we propose a new domain adaptation-based method that transfers the task knowledge from multiple source domains to a target domain. Our method presents a new setting for EM where the objective is to capture the task-specific knowledge from pretraining our model using multiple source domains, then testing our model on a target domain. We study the zero-shot learning case on the target domain, and demonstrate that our method learns the EM task and transfers knowledge to the target domain. We extensively study fine-tuning our model on the target dataset from multiple domains, and demonstrate that our model generalizes better than state-of-the-art methods in EM.
Differential Query Semantic Analysis: Discovery of Explicit Interpretable Knowledge from E-Com Search Logs

We present a novel strategy for analyzing E-Com search logs called Differential Query Semantic Analysis (DQSA) to discover explicit interpretable knowledge from search logs in the form of a semantic lexicon that makes context-specific mapping from a query segment (word or phrase) to the preferred attribute values of a product. Evaluation on a set of size-related query segments and attribute values shows that DQSA can effectively discover meaningful mappings of size-related query segments to their preferred specific attributes and attributes values in the context of a product type. DQSA has many uses including improvement of E-Com search accuracy by bridging the vocabulary gap, comparative analysis of search intent, and alleviation of the problem of tail queries and products.
Evaluation of spectrum based fault localization tools

Software Fault localization (SFL) is the first step in any program debugging process. For more than three decades, researchers have aggressively studied, evaluated, and proposed numerous automatic SFL techniques spanning across various families of methods such as spectrum-based, slice-based, mutation-based, etc. Another facet contributed by researchers is the practical implementation of the above techniques in the form of open-source tools, IDE plugins, extensions, etc. Examples include (but are not limited to) GZoltzar, Jaguar, and iFL4Eclipse. Previous research has established the metrics and threshold values for the adoption of SFL techniques in real-life software development. Several attempts have been made to evaluate automatic fault repair tools, and Information Retrieval (IR) based fault localization tools. Whilst Spectrum Based Fault Localization (SBFL) remains the most contributed family of SFL methods, no studies have been found which evaluate the existing SBFL tools. This paper presents a comparative theoretical assessment of selected SBFL tools by understanding the developmental dynamics involved in implementing them and establishing results that would guide the same in the future. Our research steps can briefly be summarized as systematic collection and filtering of SBFL tools and their research papers, developing a historical timeline for the same, and comparative theoretical analysis through the lens of software engineering. We theoretically determined that there is a lack of rigorous testing for the scalability and correctness of SBFL tools. While some tools are extensible with respect to the underlying algorithm used for computation, none provide flexibility in choosing the coverage collection framework. While Open-source tools have been more successful, there is a general lack in maintenance and development post initial publication. A natural progression of this work is a large-scale empirical assessment of the SBFL tools.
Localizing Faults Using Verification Technique

In model-based fault diagnosis, it is assumed that a correct model of each program being diagnosed is available. That is, these models serve as the oracles of the corresponding programs. Differences between the behaviors of a model and the actual observed behaviors of the program are then used to find bugs in the program. However, these techniques normally require test cases and user-specified assertions to localize the fault. In this paper, we aim to localize faults in a faulty program without user-specified assertions and without executing the programs, and therefore, without using test cases. Instead, given the faulty and the correct versions of a program, a product code is automatically constructed and assertions are automatically generated. This is a fully automatic, model-based static approach to fault localization. The proposed method reduces the fault search space by removing equivalent regions from the product code using verification techniques. We demonstrate the effectiveness of the proposed method using the Siemens TCAS benchmark. We observe that the method can successfully localize the wrong safety check bug produced by LLVM compiler.
Learning Relevant Questions for Conversational Product Search using Deep Reinforcement Learning

We propose RelQuest, a conversational product search model based on reinforcement learning to generate questions from product descriptions in each round of the conversation, directly maximizing any desired metrics (i.e., the ultimate goal of the conversation), objectives, or even an arbitrary user satisfaction signal. By enabling systems to ask questions about user needs, conversational product search has gained increasing attention in recent years. Asking the right questions through conversations helps the system collect valuable feedback to create better user experiences and ultimately increase sales. In contrast, existing conversational product search methods are based on an assumption that there is a set of effectively pre-defined candidate questions for each product to be asked. Moreover, they make strong assumptions to estimate the value of questions in each round of the conversation. Estimating the true value of questions in each round of the conversation is not trivial since it is unknown. Experiments on real-world user purchasing data show the effectiveness of RelQuest at generating questions that maximize standard evaluation measures such as NDCG.
Structure Meets Sequences: Predicting Network of Co-evolving Sequences

Co-evolving sequences are ubiquitous in a variety of applications, where different sequences are often inherently inter-connected with each other. We refer to such sequences, together with their inherent connections modeled as a structured network, as network of co-evolving sequences (NoCES). Typical NoCES applications include road traffic monitoring, company revenue prediction, motion capture, etc. To date, it remains a daunting challenge to accurately model NoCES due to the coupling between network structure and sequences. In this paper, we propose to modeling \pname\ with the aim of simultaneously capturing both the dynamics and the interplay between network structure and sequences. Specifically, we propose a joint learning framework to alternatively update the network representations and sequence representations as the sequences evolve over time. A unique feature of our framework lies in that it can deal with the case when there are co-evolving sequences on both network nodes and edges. Experimental evaluations on four real datasets demonstrate that the proposed approach (1) outperforms the existing competitors in terms of prediction accuracy, and (2) scales linearly w.r. t. the sequence length and the network size.
MtCut: A Multi-Task Framework for Ranked List Truncation

Ranked list truncation aims to cut the ranked results in short considering user-defined objectives, which balances the overall utility and user efforts over retrieval results. The exact selection of an optimal cut-off position brings potential benefits in various real-world applications, such as patent search and legal search. However, there is significant retrieval bias in the ranked list. The result scores and the disorder of document sequences cause difficulties in judging the relevance between the queries and documents -- alleviating the existing methods' performance improvement. In this work, we investigate the characteristics of retrieval bias on altering truncation and propose a multi-task truncation model, MtCut. It employs two auxiliary tasks to make complementary for the retrieval bias. As a practical evaluation, we explore its performance on two datasets, and the results show that MtCut outperforms the state-of-the-art methods on both F1-score and DCG metrics.
A Neighborhood-Attention Fine-grained Entity Typing for Knowledge Graph Completion

Knowledge graph (KG) entity typing focuses on inferring possible entity type instances, which is a significant subtask of knowledge graph completion (KGC). Existing entity typing methods usually exploit the entity representation to model the transmission between entities and their types, which cannot fully explore the fine-grained entity typing on identifying the semantic type of an entity. To address these issues, we propose Neighborhood-Attention Neural Fine-Grained Entity Typing (AttEt), which considers the neighborhood information of the entities from KGs to bridge entities and their types together. In this paper, AttEt first develops a type-specific attention mechanism to aggregate the neighborhood knowledge of the given entity with type-specific weights. These weights are beneficial to capture various characteristics for different types of the entity, and further imply the complex correlation among these fine-grained types. Then, AttEt adaptively integrates the aggregated neighbor-level representation with entity inherent embedding to calculate the matching score between the entity and its candidate type. Besides, many entities are sparse in their relations with other entities in KGs, which makes the entity typing task more challenging. To solve this problem, we present a smooth strategy on relation-sparsity entities to improve the robustness of the model. Extensive experiments on two real-world datasets (Freebase and YAGO) show that AttEt significantly outperforms state-of-the-art baselines in the HITS@1 by 2.11% on Freebase and by 8.42% on YAGO, respectively.
AngHNE: Representation Learning for Bipartite Heterogeneous Networks with Angular Loss

Real-world networks often show heterogeneity. A frequently encountered type is the bipartite heterogeneous structure, in which two types of nodes and three types of edges exist. Recently, much attention has been devoted to representation learning in these networks. One of the essential differences between heterogeneous and homogeneous learning is that the former structure requires methods to possess awareness to node and edge types. Most existing methods, including metapath-based, proximity-based and graph neural network-based, adopt inner product or vector norms to evaluate the similarities in embedding space. However, these measures either violates the triangle inequality, or show severe sensitivity to scaling transformation. The limitations often hinder the applicability to real-world problems. In view of this, in this paper, we propose a novel angle-based method for bipartite heterogeneous network representation. Specifically, we first construct training sets by generating quintuples, which contain both positive and negative samples from two different parts of networks. Then we analyze the quintuple-based problem from a geometry perspective, and transform the comparisons between preferred and non-preferred samples to the comparisons of angles. In addition, we utilize convolution modules to extract node features. A hinge loss, as the final objective, is proposed to relax the angular constraint for learning. Extensive experiments for two typical tasks show the efficacy of the proposed method, comparing with eight competitive methods.
AI-enabled Project Initiation: An approach based on RFP Response Document

Project planning starts with the project initiation phase in which high-level project objectives, commitments, requirements, risks etc. are identified. Typically, the Project Manager involves multiple stakeholders such as HR, Admin, Infrastructure team for the initiation phase to understand and outline the requirements for each group. Current industry practice largely relies on the project manager’s experience to carry out the project initiation activities keeping in view the customer context and commitments made. Many times, important information is missed during the transfer of information from sales to delivery resulting in not meeting customer expectations and delivery slippages. Here we propose an AI-enabled approach to automatically extract and classify project initiation relevant information from Request For Proposal (RFP) response document using a combination of NLP and ML-based techniques. The approach is validated with real life RFP response documents for five customers. Overall, 76% accuracy was observed for question classification and 41% information was found to be relevant for project initiation from the RFP response documents. In this paper, we share details of our approach, its implementation, early results, and lessons learnt.
Evidence and perceptions on GUI test automation - An Exploratory Study

Context: Software testing requires substantial effort in maintaining GUI test scripts, leading to rework or waste. Therefore, it is important to identify the factors leading to waste in GUI-based automation testing. Objective: To identify testability requirements and factors associated with waste from literature and practice. Methods: We conducted a literature review to identify the state of knowledge and a two-phased interview with practitioners to capture their subjective opinions and gather their views on the state of knowledge. Results: We identified 16 testability requirements and 14 factors associated with waste in GUI test automation. We classified each of them into SUT-related, test-process-related, test-tool-related, human and organizational, environment and cross-cutting. Conclusions: Our results indicate that the generic testability requirements identified in the literature are also valid for GUI automation testing. Our study identifies new findings in addition to the existing literature.
Identifying Cost-effective Debunkers for Multi-stage Fake News Mitigation Campaigns

Online social networks have become a fertile ground for spreading fake news. Methods to automatically mitigate fake news propagation have been proposed. Some studies focus on selecting top k influential users on social networks as debunkers, but the social influence of debunkers may not translate to wide mitigation information propagation as expected. Other studies assume a given set of debunkers and focus on optimizing intensity for debunkers to publish true news, but as debunkers are fixed, even if with high social influence and/or high intensity to post true news, the true news may not reach users exposed to fake news and therefore mitigation effect may be limited. In this paper, we propose the multi-stage fake news mitigation campaign where debunkers are dynamically selected within budget at each stage. We formulate it as a reinforcement learning problem and propose a greedy algorithm optimized by predicting future states so that the debunkers can be selected in a way that maximizes the overall mitigation effect. We conducted extensive experiments on synthetic and real-world social networks and show that our solution outperforms state-of-the-art baselines in terms of mitigation effect.
Outside In: Market-aware Heterogeneous Graph Neural Network for Employee Turnover Prediction

As an emerging initiative of proactive human resource management, employee turnover prediction is critically important for employers to retain talents and avoid the loss of intellectual capital. While considerable research efforts have been made in this direction, most of them only focus on modeling the within-company career trajectories of employees where the influence of external job market has been largely neglected. To this end, in this paper, we propose an enhanced framework of employee turnover prediction by jointly modeling the turnover clues from both internal and external views. Specifically, from the external-market view, we construct a heterogeneous graph which connects the employees with external job markets through shared skills. In this way, we can capture the potential popularity of employees in external markets specific to skills. Meanwhile, from the internal-company view, we design a graph convolutional network with hierarchical attention mechanism to capture the influence of organizational structure (e.g., superiors, subordinates, and peers) and colleagues with similar skills. Furthermore, both modules are modeled with Bidirectional LSTM and survival analysis to learn effective and dynamic representations of employee turnover prediction. Finally, we conduct extensive experiments on a large-scale real-world talent dataset with state-of-the-art methods, which clearly demonstrate the effectiveness of our approach as well as some interesting findings that could help us understand the employee turnover patterns, such as different impacts of external systems and collaborators from different groups.
Portable design of ADPLL for TV applications

Local oscillators are key buildings in radio communication system. They are considered as intensively analog blocks and then more sensitive to internal noise than digital blocks. This noise is up-converted to phase noise around the carrier frequency at the output of the oscillator. High performance of lo in terms of low phase noise is paid by high surface circuit or by high power consumption. One solution, proposed in the literature, is the all digital phase locked loops ADPLL. Within, digital blocks replace the analog blocks, such as the phase comparator and the loop filter. In this thesis, down to top behavior model of ADPLL is developed using VHDL-AMS. This model includes oscillator phase noise and TDC quantification noise. The modeling of the DCO is based on the linear time variant LTV model of phase noise that is based on the impulse sensitivity function ISF of the oscillator. We are interested in this work by the design and modeling of the DCO and of the TDC because of their noisy behavior. We are interested in this work in the design of digital controlled ring oscillator. We opt for nine-stage ring interpolative oscillator made by nor cells for its larger tuning range than inverter based ring oscillators. We propose an ISF based design strategy that can optimize the design variables of ring oscillators for minimum phase noise. In order to minimize the circuit area and the power consumption, we propose to merge DCO and TDC architectures. TDC uses the ring status to determinate the position of the rising or the falling edge of the oscillator waveform that is propagating in the ring at the rising edge of the reference.
Elicitation of Personal Data Categories for Implementing Data Protection: An Exploratory Study in an Educational Institution

Protecting the privacy of natural persons has become very important in today's data economy. Countries around the world are in the process of legislating data protection regulations to prevent breaches of personal data. However, the implementation of data protection is very much domain and context-specific. The data categories that are to be protected in the domain of implementation need to be discovered. We propose a human-centric approach for eliciting such data categories that cause privacy concerns to stakeholders in an educational institution. A study was conducted to understand the privacy concerns of the stakeholders, related to different data categories to be protected. A combination of surveys and in-depth interviews of the different stakeholders were used to gain insights into the privacy and data protection requirements that need to be incorporated into the associated information system design.