OpenRaiser/Intern-Atlas huggingface dataset:
Readsre: Retrieval-augmented distantly supervised relation extraction
Authors: Yue Zhang 0086, Hongliang Fei, Ping Li 0001
Venue: Conference SIGIR 2021
Year: 2021

Distant supervision (DS) has been widely used to automatically construct (noisy) labeled data for relation extraction (RE). To address the noisy label problem, most models have adopted the multi-instance learning paradigm by representing entity pairs as a bag of sentences. However, this strategy depends on multiple assumptions (e.g., all sentences in a bag share the same relation), which may be invalid in real-world applications. Besides, it cannot work well on long-tail entity pairs which have few supporting sentences in the dataset. In this work, we propose a new paradigm named retrieval-augmented distantly supervised relation extraction (ReadsRE), which can incorporate large-scale open-domain knowledge (e.g., Wikipedia) into the retrieval step. ReadsRE seamlessly integrates a neural retriever and a relation predictor in an end-to-end framework. We demonstrate the effectiveness of ReadsRE on the well-known NYT10 dataset. The experimental results verify that ReadsRE can effectively retrieve meaningful sentences (i.e., denoise), and relieve the problem of long-tail entity pairs in the original dataset through incorporating external open-domain corpus. Through comparisons, we show ReadsRE outperforms other baselines for this task.
The positioning matters: Estimating geographical bias in the multilingual record of biographies on wikipedia
Authors: Pablo Beytía
Venue: Conference WWW 2020
Year: 2020

This article proposes that an appropriate assessment of the geographical bias in multilingual Wikipedia's content should consider not only the number of articles linked to places, but also their internal positioning –i.e. their location in different languages and their centrality in the network of references between articles–. This idea is studied empirically, systematically evaluating the geographic concentration in the biographical coverage of globally recognized individuals (those whose biographies are found in more than 25 language versions of Wikipedia). Considering the internal positioning levels of these biographies, only 5 countries account for more than 62% of Wikipedia's biographical coverage. In turn, the inequality in coverage between countries reaches very high levels, estimated with a Gini coefficient of .84 and a Palma ratio of 207. In all the tests carried out, the inclusion of the linguistic and/or relational positioning of the articles increases the estimate of inequality in biographical coverage. This suggests that previous estimates of geographical bias, which do not consider differences in internal positioning, have underestimated the degree of inequality in the distribution of information.
Biosnowball: automated population of wikis
Authors: Xiaojiang Liu, Zaiqing Nie, Nenghai Yu, Ji-Rong Wen
Venue: Conference KDD 2010
Year: 2010

Internet users regularly have the need to find biographies and facts of people of interest. Wikipedia has become the first stop for celebrity biographies and facts. However, Wikipedia can only provide information for celebrities because of its neutral point of view (NPOV) editorial policy. In this paper we propose an integrated bootstrapping framework named BioSnowball to automatically summarize the Web to generate Wikipedia-style pages for any person with a modest web presence. In BioSnowball, biography ranking and fact extraction are performed together in a single integrated training and inference process using Markov Logic Networks (MLNs) as its underlying statistical model. The bootstrapping framework starts with only a small number of seeds and iteratively finds new facts and biographies. As biography paragraphs on the Web are composed of the most important facts, our joint summarization model can improve the accuracy of both fact extraction and biography ranking compared to decoupled methods in the literature. Empirical results on both a small labeled data set and a real Web-scale data set show the effectiveness of BioSnowball. We also empirically show that BioSnowball outperforms the decoupled methods.
Progressively optimized bi-granular document representation for scalable embedding based retrieval
Authors: Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Yingxia Shao, Defu Lian, Chaozhuo Li, Hao Sun, Denvy Deng, Liangjie Zhang, Qi Zhang, Xing Xie
Venue: Conference WWW 2022
Year: 2022

Ad-hoc search calls for the selection of appropriate answers from a massive-scale corpus. Nowadays, the embedding-based retrieval (EBR) becomes a promising solution, where deep learning based document representation and ANN search techniques are allied to handle this task. However, a major challenge is that the ANN index can be too large to fit into memory, given the considerable size of answer corpus. In this work, we tackle this problem with Bi-Granular Document Representation, where the lightweight sparse embeddings are indexed and standby in memory for coarse-grained candidate search, and the heavyweight dense embeddings are hosted in disk for fine-grained post verification. For the best of retrieval accuracy, a Progressive Optimization framework is designed. The sparse embeddings are learned ahead for high-quality search of candidates. Conditioned on the candidate distribution induced by the sparse embeddings, the dense embeddings are continuously learned to optimize the discrimination of ground-truth from the shortlisted candidates. Besides, two techniques: the contrastive quantization and the locality-centric sampling are introduced for the learning of sparse and dense embeddings, which substantially contribute to their performances. Thanks to the above features, our method effectively handles massive-scale EBR with strong advantages in accuracy: with up to +4.3% recall gain on million-scale corpus, and up to +17.5% recall gain on billion-scale corpus. Besides, Our method is applied to a major sponsored search platform with substantial gains on revenue (+1.95%), Recall (+1.01%) and CTR (+0.49%). Our code is available at https://github.com/microsoft/BiDR.
Gradmask: Gradient-guided token masking for textual adversarial example detection
Authors: Han Cheol Moon, Shafiq R. Joty, Xu Chi
Venue: Conference KDD 2022
Year: 2022

We present GradMask, a simple adversarial example detection scheme for natural language processing (NLP) models. It uses gradient signals to detect adversarially perturbed tokens in an input sequence and occludes such tokens by a masking process. GradMask provides several advantages over existing methods including improved detection performance and an interpretation of its decision with a only moderate computational cost. Its approximated inference cost is no more than a single forward- and back-propagation through the target model without requiring any additional detection module. Extensive evaluation on widely adopted NLP benchmark datasets demonstrates the efficiency and effectiveness of GradMask. Code and models are available at https://github.com/Han8931/grad_mask_detection
A cooperative neural information retrieval pipeline with knowledge enhanced automatic query reformulation
Authors: Xiangsheng Li, Jiaxin Mao, Weizhi Ma, Zhijing Wu, Yiqun Liu, Min Zhang, Shaoping Ma, Zhaowei Wang, Xiuqiang He
Venue: Conference WSDM 2022
Year: 2022

This paper presents a neural information retrieval pipeline that integrates cooperative learning of query reformulation and neural retrieval models. Our pipeline first exploits an automatic query reformulator to reformulate the user-issued query and then submits the reformulated query to the neural retrieval model. We simultaneously optimize the quality of reformulated queries and ranking performance with an alternate training strategy where query reformulator and neural retrieval model learn from the feedback of each other. Besides, we incorporate knowledge information into automatic query reformulation. The reformulated queries are further improved and contribute to a better ranking performance of the following neural retrieval model. We study two representative neural retrieval models KNRM and BERT in our pipeline. Experiments on two datasets show that our pipeline consistently improves the retrieval performance of the original neural retrieval models while only increases negligible time on automatic query reformulation.
Having your cake and eating it too: Training neural retrieval for language inference without losing lexical match
Authors: Vikas Yadav, Steven Bethard, Mihai Surdeanu
Venue: Conference SIGIR 2020
Year: 2020

We present a study on the importance of information retrieval (IR) techniques for both the interpretability and the performance of neural question answering (QA) methods. We show that the current state-of-the-art transformer methods (like RoBERTa) encode poorly simple information retrieval (IR) concepts such as lexical overlap between query and the document. To mitigate this limitation, we introduce a supervised RoBERTa QA method that is trained to mimic the behavior of BM25 and the soft-matching idea behind embedding-based alignment methods. We show that fusing the simple lexical-matching IR concepts in transformer techniques results in improvement a) of their (lexical-matching) interpretability, b) retrieval performance, and c) the QA performance on two multi-hop QA datasets. We further highlight the lexical-chasm gap bridging capabilities of transformer methods by analyzing the attention distributions of the supervised RoBERTa classifier over the context versus lexically-matched token pairs.
Selecting discriminative terms for relevance model
Authors: Dwaipayan Roy, Sumit Bhatia, Mandar Mitra
Venue: Conference SIGIR 2019
Year: 2019

Pseudo-relevance feedback based on the relevance model does not take into account the inverse document frequency of candidate terms when selecting expansion terms. As a result, common terms are often included in the expanded query constructed by this model. We propose three possible extensions of the relevance model that address this drawback. Our proposed extensions are simple to compute and are independent of the base retrieval model. Experiments on several TREC news and web collections show that the proposed modifications yield significantly better MAP, precision, NDCG, and recall values than the original relevance model as well as its two recently proposed state-of-the-art variants.
Neural relation extraction on wikipedia tables for augmenting knowledge graphs
Authors: Erin MacDonald, Denilson Barbosa
Venue: Conference CIKM 2020
Year: 2020

Knowledge Graph Augmentation is the task of adding missing facts to an incomplete knowledge graph to improve its effectiveness in applications such as web search and question answering. State-of-the-art methods rely on information extraction from running text, leaving rich sources of facts such as tables behind. We help close this gap with a neural method that uses contextual information surrounding a table in a Wikipedia article to extract relations between entities appearing in the same row of a table or between the entity of said article and entities appearing in the table. We trained and tested our method on a much larger dataset compared to previous work which we have made public and observed experimentally that our method is very promising for the task.
Boltzrank: learning to maximize expected ranking gain
Authors: Maksims Volkovs, Richard S. Zemel
Venue: Conference ICML 2009
Year: 2009

Ranking a set of retrieved documents according to their relevance to a query is a popular problem in information retrieval. Methods that learn ranking functions are difficult to optimize, as ranking performance is typically judged by metrics that are not smooth. In this paper we propose a new listwise approach to learning to rank. Our method creates a conditional probability distribution over rankings assigned to documents for a given query, which permits gradient ascent optimization of the expected value of some performance measure. The rank probabilities take the form of a Boltzmann distribution, based on an energy function that depends on a scoring function composed of individual and pairwise potentials. Including pairwise potentials is a novel contribution, allowing the model to encode regularities in the relative scores of documents; existing models assign scores at test time based only on individual documents, with no pairwise constraints between documents. Experimental results on the LETOR3.0 data set show that our method out-performs existing learning approaches to ranking.
CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos
Authors: Shengyao Zhuang, Guido Zuccon
Venue: Conference SIGIR 2022
Year: 2022

Current dense retrievers are not robust to out-of-domain and outlier queries, i.e. their effectiveness on these queries is much poorer than what one would expect. In this paper, we consider a specific instance of such queries: queries that contain typos. We show that a small character level perturbation in queries (as caused by typos) highly impacts the effectiveness of dense retrievers. We then demonstrate that the root cause of this resides in the input tokenization strategy employed by BERT. In BERT, tokenization is performed using the BERT's WordPiece tokenizer and we show that a token with a typo will significantly change the token distributions obtained after tokenization. This distribution change translates to changes in the input embeddings passed to the BERT-based query encoder of dense retrievers. We then turn our attention to devising dense retriever methods that are robust to such queries with typos, while still being as performant as previous methods on queries without typos. For this, we use CharacterBERT as the backbone encoder and an efficient yet effective training method, called Self-Teaching (ST), that distills knowledge from queries without typos into the queries with typos. Experimental results show that CharacterBERT in combination with ST achieves significantly higher effectiveness on queries with typos compared to previous methods. Along with these results and the open-sourced implementation of the methods, we also provide a new passage retrieval dataset consisting of real-world queries with typos and associated relevance assessments on the MS MARCO corpus, thus supporting the research community in the investigation of effective and robust dense retrievers. Code, experimental results and dataset are made available at https://github.com/ielab/CharacterBERT-DR.
Spatially coherent latent topic model for concurrent segmentation and classification of objects and scenes
Authors: Liangliang Cao, Li Fei-Fei 0001
Venue: Conference ICCV 2007
Year: 2007

We present a novel generative model for simultaneously recognizing and segmenting object and scene classes. Our model is inspired by the traditional bag of words representation of texts and images as well as a number of related generative models, including probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA). A major drawback of the pLSA and LDA models is the assumption that each patch in the image is independently generated given its corresponding latent topic. While such representation provides an efficient computational method, it lacks the power to describe the visually coherent images and scenes. Instead, we propose a spatially coherent latent topic model (Spatial-LTM). Spatial-LTM represents an image containing objects in a hierarchical way by over-segmented image regions of homogeneous appearances and the salient image patches within the regions. Only one single latent topic is assigned to the image patches within each region, enforcing the spatial coherency of the model. This idea gives rise to the following merits of Spatial-LTM: (1) Spatial-LTM provides a unified representation for spatially coherent bag of words topic models; (2) Spatial-LTM can simultaneously segment and classify objects, even in the case of occlusion and multiple instances; and (3) Spatial-LTM can be trained either unsupervised or supervised, as well as when partial object labels are provided. We verify the success of our model in a number of segmentation and classification experiments.
Table search using a deep contextualized language model
Authors: Zhiyu Chen 0001, Mohamed Trabelsi 0003, Jeff Heflin, Yinan Xu 0005, Brian D. Davison 0001
Venue: Conference SIGIR 2020
Year: 2020

Pretrained contextualized language models such as BERT have achieved impressive results on various natural language processing benchmarks. Benefiting from multiple pretraining tasks and large scale training corpora, pretrained models can capture complex syntactic word relations. In this paper, we use the deep contextualized language model BERT for the task of ad hoc table retrieval. We investigate how to encode table content considering the table structure and input length limit of BERT. We also propose an approach that incorporates features from prior literature on table retrieval and jointly trains them with BERT. In experiments on public datasets, we show that our best approach can outperform the previous state-of-the-art method and BERT baselines with a large margin under different evaluation metrics.
Ask to know more: Generating counterfactual explanations for fake claims
Authors: Shih-Chieh Dai, Yi-Li Hsu, Aiping Xiong, Lun-Wei Ku
Venue: Conference KDD 2022
Year: 2022

Automated fact checking systems have been proposed that quickly provide veracity prediction at scale to mitigate the negative influence of fake news on people and on public opinion. However, most studies focus on veracity classifiers of those systems, which merely predict the truthfulness of news articles. We posit that effective fact checking also relies on people's understanding of the predictions. In this paper, we propose elucidating fact checking predictions using counterfactual explanations to help people understand why a specific piece of news was identified as fake. In this work, generating counterfactual explanations for fake news involves three steps: asking good questions, finding contradictions, and reasoning appropriately. We frame this research question as contradicted entailment reasoning through question answering (QA). We first ask questions towards the false claim and retrieve potential answers from the relevant evidence documents. Then, we identify the most contradictory answer to the false claim by use of an entailment classifier. Finally, a counterfactual explanation is created using a matched QA pair with three different counterfactual explanation forms. Experiments are conducted on the FEVER dataset for both system and human evaluations. Results suggest that the proposed approach generates the most helpful explanations compared to state-of-the-art methods.
Finsbd-2021: the 3rd shared task on structure boundary detection in unstructured text in the financial domain
Authors: Willy Au, Abderrahim Ait-Azzi, Juyeon Kang
Venue: Conference WWW 2021
Year: 2021

Document processing is a foundational pre-processing task in natural language application applied in the financial domain. In this paper, we present the result of FinSBD-3, the 3rd shared task on Structure Boundary Detection in unstructured text in the financial domain. The shared task is organized as part of the 1st Workshop on Financial Technology on the Web. Participants were asked to create system detecting the boundaries of elements in unstructured text extracted from financial PDF. This edition extends the previous shared tasks by adding boundaries of visual elements such as tables, figures, page headers and page footers; on top of sentences, lists and list items which were already present in previous edition of the shared tasks.
HiText: Text reading with dynamic salience marking
Authors: Qian Yang, Gerard de Melo, Yong Cheng, Sen Wang
Venue: Conference WWW 2017
Year: 2017

The staggering amounts of content readily available to us via digital channels can often appear overwhelming. While much research has focused on aiding people at selecting relevant articles to read, only few approaches have been developed to assist readers in more efficiently reading an individual text. In this paper, we present HiText, a simple yet effective way of dynamically marking parts of a document in accordance with their salience. Rather than skimming a text by focusing on randomly chosen sentences, students and other readers can direct their attention to sentences determined to be important by our system. For this, we rely on a deep learning-based sentence ranking method. Our experiments show that this results in marked increases in user satisfaction and reading efficiency, as assessed using TOEFL-style reading comprehension tests.
Transparent, scrutable and explainable user models for personalized recommendation
Authors: Krisztian Balog, Filip Radlinski, Shushan Arakelyan
Venue: Conference SIGIR 2019
Year: 2019

Most recommender systems base their recommendations on implicit or explicit item-level feedback provided by users. These item ratings are combined into a complex user model, which then predicts the suitability of other items. While effective, such methods have limited scrutability and transparency. For instance, if a user's interests change, then many item ratings would usually need to be modified to significantly shift the user's recommendations. Similarly, explaining how the system characterizes the user is impossible, short of presenting the entire list of known item ratings. In this paper, we present a new set-based recommendation technique that permits the user model to be explicitly presented to users in natural language, empowering users to understand recommendations made and improve the recommendations dynamically. While performing comparably to traditional collaborative filtering techniques in a standard static setting, our approach allows users to efficiently improve recommendations. Further, it makes it easier for the model to be validated and adjusted, building user trust and understanding.
Jit2r: A joint framework for item tagging and tag-based recommendation
Authors: Xu Chen 0017, Changying Du, Xiuqiang He 0001, Jun Wang 0012
Venue: Conference SIGIR 2020
Year: 2020

Predicting tags for a given item and leveraging tags to assist item recommendation are two popular research topics in the field of recommender system. Previous studies mostly focus only one of them to make contributions. However, we believe that these tasks are inherently correlated with each other: tags can provide additional information to profile items for more accurate recommendation; user behaviors can help to infer item relationships to benefit the item tagging process. In order to take the advantages of such mutually influential signals, we propose to integrate item tagging and tag-based recommendation into a unified model. We firstly design a basic framework, where the user-item interaction signals are leveraged to supervise the item tagging process. Then we extend the basic model with a bootstrapping technique to circulate such mutual improvements between different tasks. We conduct extensive experiments based on real-word datasets to demonstrate our model's superiorities.