OpenRaiser/Intern-Atlas huggingface dataset:
Text-based measures of document diversity
Authors: Kevin Bache, David Newman, Padhraic Smyth
Venue: Conference KDD 2013
Year: 2013

Quantitative notions of diversity have been explored across a variety of disciplines ranging from conservation biology to economics. However, there has been relatively little work on measuring the diversity of text documents via their content. In this paper we present a text-based framework for quantifying how diverse a document is in terms of its content. The proposed approach learns a topic model over a corpus of documents, and computes a distance matrix between pairs of topics using measures such as topic co-occurrence. These pairwise distance measures are then combined with the distribution of topics within a document to estimate each document's diversity relative to the rest of the corpus. The method provides several advantages over existing methods. It is fully data-driven, requiring only the text from a corpus of documents as input, it produces human-readable explanations, and it can be generalized to score diversity of other entities such as authors, academic departments, or journals. We describe experimental results on several large data sets which suggest that the approach is effective and accurate in quantifying how diverse a document is relative to other documents in a corpus.
Starsum: A simple star graph for multi-document summarization
Authors: Mohammed Al-Dhelaan
Venue: Conference SIGIR 2015
Year: 2015

Graph-based approaches for multi-document summarization have been widely used to extract top sentences for a summary. Traditionally, the documents' cluster is modeled as a graph of the cluster's sentences only which might limit the ability of recognizing topically discriminative sentences in regard to other clusters. In this paper, we propose StarSum a star bipartite graph which models sentences and their topic signature phrases. The approach ensures sentence similarity and content importance from the graph structure. We extract sentences in an approach that guarantees diversity and coverage which are crucial for multi-document summarization. Regardless of the simplicity of the approach in ranking, a DUC experiment shows the effectiveness of StarSum compared to different baselines.
Multi-document hyperedge-based ranking for text summarization
Authors: Abdelghani Bellaachia, Mohammed Al-Dhelaan
Venue: Conference CIKM 2014
Year: 2014

In a multi-document settings, graph-based extractive summarization approaches build a similarity graph out of sentences in each cluster of documents then use graph centrality approaches to measure the importance of sentences. The similarity is computed between each pair of sentences. However, it is not clear if such approach captures high-order relations among more than two sentences or can differentiate between descriptive sentences of the cluster in comparison with other clusters. In this paper, we propose to model sentences as hyperedges and words as vertices using a hypergraph and combine it with topic signatures to differentiate between descriptive sentences and non-descriptive sentences. To rank sentences, we propose a new random walk over hyperedges that will prefer descriptive sentences of the cluster when measuring their centrality scores. Our approach outperform a number of baseline in the DUC 2001 dataset using the ROUGE metric.
defend: A system for explainable fake news detection
Authors: Limeng Cui, Kai Shu, Suhang Wang, Dongwon Lee, Huan Liu
Venue: Conference CIKM 2019
Year: 2019

Despite recent advancements in computationally detecting fake news, we argue that a critical missing piece be the explainability of such detection--i.e., why a particular piece of news is detected as fake--and propose to exploit rich information in users' comments on social media to infer the authenticity of news. In this demo paper, we present our system for an explainable fake news detection called dEFEND, which can detect the authenticity of a piece of news while identifying user comments that can explain why the news is fake or real. Our solution develops a sentence-comment co-attention sub-network to exploit both news contents and user comments to jointly capture explainable top-k check-worthy sentences and user comments for fake news detection. The system is publicly accessible.
Cyberbullying detection: a step toward a safer internet yard
Authors: Maral Dadvar, Franciska de Jong
Venue: Conference WWW 2012
Year: 2012

As a result of the invention of social networks friendships, relationships and social communications have all gone to a new level with new definitions. One may have hundreds of friends without even seeing their faces. Meanwhile, alongside this transition there is increasing evidence that online social applications have been used by children and adolescents for bullying. State-of-the-art studies in cyberbullying detection have mainly focused on the content of the conversations while largely ignoring the users involved in cyberbullying. We propose that incorporation of the users' information, their characteristics, and post-harassing behaviour, for instance, posting a new status in another social network as a reaction to their bullying experience, will improve the accuracy of cyberbullying detection. Cross-system analyses of the users' behaviour - monitoring their reactions in different online environments - can facilitate this process and provide information that could lead to more accurate detection of cyberbullying.
Cyberbullying ends here: Towards robust detection of cyberbullying in social media
Authors: Mengfan Yao, Charalampos Chelmis, Daphney-Stavroula Zois
Venue: Conference WWW 2019
Year: 2019

The potentially detrimental effects of cyberbullying have led to the development of numerous automated, data-driven approaches, with emphasis on classification accuracy. Cyberbullying, as a form of abusive online behavior, although not well-defined, is a repetitive process, i.e., a sequence of aggressive messages sent from a bully to a victim over a period of time with the intent to harm the victim. Existing work has focused on harassment (i.e., using profanity to classify toxic comments independently) as an indicator of cyberbullying, disregarding the repetitive nature of this harassing process. However, raising a cyberbullying alert immediately after an aggressive comment is detected can lead to a high number of false positives. At the same time, two key practical challenges remain unaddressed: (i) detection timeliness, which is necessary to support victims as early as possible, and (ii) scalability to the staggering rates at which content is generated in online social networks. In this work, we introduce CONcISE, a novel approach for timely and accurate Cyberbullying detectiON on Instagram media SEssions. We propose a sequential hypothesis testing formulation that seeks to drastically reduce the number of features used in classifying each comment while maintaining high classification accuracy. CONcISE raises an alert only after a certain number of detections have been made. Extensive experiments on a real-world Instagram dataset with ~ 4M users and ~ 10M comments demonstrate the effectiveness, scalability, and timeliness of our approach and its benefits over existing methods.
Learning to suggest: a machine learning framework for ranking query suggestions
Authors: Umut Ozertem, Olivier Chapelle, Pinar Donmez, Emre Velipasaoglu
Venue: Conference SIGIR 2012
Year: 2012

We consider the task of suggesting related queries to users after they issue their initial query to a web search engine. We propose a machine learning approach to learn the probability that a user may find a follow-up query both useful and relevant, given his initial query. Our approach is based on a machine learning model which enables us to generalize to queries that have never occurred in the logs as well. The model is trained on co-occurrences mined from the search logs, with novel utility and relevance models, and the machine learning step is done without any labeled data by human judges. The learning step allows us to generalize from the past observations and generate query suggestions that are beyond the past co-occurred queries. This brings significant gains in coverage while yielding modest gains in relevance. Both offline (based on human judges) and online (based on millions of user interactions) evaluations demonstrate that our approach significantly outperforms strong baselines.
Multi-instance tree learning
Authors: Hendrik Blockeel, David Page, Ashwin Srinivasan 0001
Venue: Conference ICML 2005
Year: 2005

We introduce a novel algorithm for decision tree learning in the multi-instance setting as originally defined by Dietterich et al. It differs from existing multi-instance tree learners in a few crucial, well-motivated details. Experiments on synthetic and real-life datasets confirm the beneficial effect of these differences and show that the resulting system outperforms the existing multi-instance decision tree learners.
Knowledge-enhanced ensemble learning for word embeddings
Authors: Lanting Fang, Yong Luo, Kaiyu Feng, Kaiqi Zhao, Aiqun Hu
Venue: Conference WWW 2019
Year: 2019

Representing words as embeddings in a continuous vector space has been proven to be successful in improving the performance in many natural language processing (NLP) tasks. Beyond the traditional methods that learn the embeddings from large text corpora, ensemble methods have been proposed to leverage the merits from pre-trained word embeddings as well as external semantic sources. In this paper, we propose a knowledge-enhanced ensemble method to combine both knowledge graphs and pre-trained word embedding models. Specifically, we interpret relations in knowledge graphs as linear translation from one word to another. We also propose a novel weighting scheme to further distinguish edges in the knowledge graph with same type of relation. Extensive experiments demonstrate that our proposed method is up to 20% times better than state-of-the-art in word analogy task and up to 16% times better than state-of-the-art in word similarity task.
Reply With: Proactive Recommendation of Email Attachments
Authors: Christophe Van Gysel, Bhaskar Mitra, Matteo Venanzi, Roy Rosemarin, Grzegorz Kukla, Piotr Grudzien, Nicola Cancedda
Venue: Conference CIKM 2017
Year: 2017

Email responses often contain items---such as a file or a hyperlink to an external document---that are attached to or included inline in the body of the message. Analysis of an enterprise email corpus reveals that 35% of the time when users include these items as part of their response, the attachable item is already present in their inbox or sent folder. A modern email client can proactively retrieve relevant attachable items from the user's past emails based on the context of the current conversation, and recommend them for inclusion, to reduce the time and effort involved in composing the response. In this paper, we propose a weakly supervised learning framework for recommending attachable items to the user. As email search systems are commonly available, we constrain the recommendation task to formulating effective search queries from the context of the conversations. The query is submitted to an existing IR system to retrieve relevant items for attachment. We also present a novel strategy for generating labels from an email corpus---without the need for manual annotations---that can be used to train and evaluate the query formulation model. In addition, we describe a deep convolutional neural network that demonstrates satisfactory performance on this query formulation task when evaluated on the publicly available Avocado dataset and a proprietary dataset of internal emails obtained through an employee participation program.
Personal Knowledge Base Construction from Text-based Lifelogs
Authors: An-Zi Yen, Hen-Hsen Huang, Hsin-Hsi Chen
Venue: Conference SIGIR 2019
Year: 2019

Previous work on lifelogging focuses on life event extraction from image, audio, and video data via wearable sensors. In contrast to wearing an extra camera to record daily life, people are used to log their life on social media platforms. In this paper, we aim to extract life events from textual data shared on Twitter and construct personal knowledge bases of individuals. The issues to be tackled include (1) not all text descriptions are related to life events, (2) life events in a text description can be expressed explicitly or implicitly, (3) the predicates in the implicit events are often absent, and (4) the mapping from natural language predicates to knowledge base relations may be ambiguous. A joint learning approach is proposed to detect life events in tweets and extract event components including subjects, predicates, objects, and time expressions. Finally, the extracted information is transformed to knowledge base facts. The evaluation is performed on a collection of lifelogs from 18 Twitter users. Experimental results show our proposed system is effective in life event extraction, and the constructed personal knowledge bases are expected to be useful to memory recall applications.
Attitude detection for one-round conversation: Jointly extracting target-polarity pairs
Authors: Zhaohao Zeng, Ruihua Song, Pingping Lin, Tetsuya Sakai
Venue: Conference WSDM 2019
Year: 2019

We tackle Attitude Detection, which we define as the task of extracting the replier's attitude, i.e., a target-polarity pair, from a given one-round conversation. While previous studies considered Target Extraction and Polarity Classification separately, we regard them as subtasks of Attitude Detection. Our experimental results show that treating the two subtasks independently is not the optimal solution for Attitude Detection, as achieving high performance in each subtask is not sufficient for obtaining correct target-polarity pairs. Our jointly trained model AD-NET substantially outperforms the separately trained models by alleviating the target-polarity mismatch problem. Moreover, we proposed a method utilising the attitude detection model to improve retrieval-based chatbots by re-ranking the response candidates with attitude features. Human evaluation indicates that with attitude detection integrated, the new responses to the sampled queries from are statistically significantly more consistent, coherent, engaging and informative than the original ones obtained from a commercial chatbot.
Sentiment analysis of peer review texts for scholarly papers
Authors: Ke Wang, Xiaojun Wan
Venue: Conference SIGIR 2018
Year: 2018

Sentiment analysis has been widely explored in many text domains, including product reviews, movie reviews, tweets, and so on. However, there are very few studies trying to perform sentiment analysis in the domain of peer reviews for scholarly papers, which are usually long and introducing both pros and cons of a paper submission. In this paper, we for the first time investigate the task of automatically predicting the overall recommendation/decision (accept, reject, or sometimes borderline) and further identifying the sentences with positive and negative sentiment polarities from a peer review text written by a reviewer for a paper submission. We propose a multiple instance learning network with a novel abstract-based memory mechanism (MILAM) to address this challenging task. Two evaluation datasets are constructed from the ICLR open reviews and evaluation results verified the efficacy of our proposed model. Our model much outperforms a few existing models in different experimental settings. We also find the generally good consistency between the review texts and the recommended decisions, except for the borderline reviews.
Time expression recognition using a constituent-based tagging scheme
Authors: Xiaoshi Zhong, Erik Cambria
Venue: Conference WWW 2018
Year: 2018

We find from four datasets that time expressions are formed by loose structure and the words used to express time information can differentiate time expressions from common text. The findings drive us to design a learning method named TOMN to model time expressions. TOMN defines a time-related tagging scheme named TOMN scheme with four tags, namely \tomnT,\tomnO, \tomnM,and \tomnN, indicating the constituents of time expression, namely \tomnT ime token, \tomnM odifier, \tomnN umeral, and the words \tomnO utside time expression. In modeling, TOMN assigns a word with a TOMN tag under conditional random fields with minimal features. Essentially, our constituent-based TOMN scheme overcomes the problem of inconsistent tag assignment that is caused by the conventional position-based tagging schemes (\eg BIO scheme and BILOU scheme). Experiments show that TOMN is equally or more effective than state-of-the-art methods on various datasets, and much more robust on cross-datasets. Moreover, our analysis can explain many empirical observations in other works about time expression recognition and named entity recognition.
How to invest my time: Lessons from human-in-the-loop entity extraction
Authors: Shanshan Zhang, Lihong He, Eduard C. Dragut, Slobodan Vucetic
Venue: Conference KDD 2019
Year: 2019

Recognizing entities that follow or closely resemble a regular expression (regex) pattern is an important task in information extraction. Common approaches for extraction of such entities require humans to either write a regex recognizing an entity or manually label entity mentions in a document corpus. While human effort is critical to build an entity recognition model, surprisingly little is known about how to best invest that effort given a limited time budget. To get an answer, we consider an iterative human-in-the-loop (HIL) framework that allows users to write a regex or manually label entity mentions, followed by training and refining a classifier based on the provided information. We demonstrate on 5 entity recognition tasks that classification accuracy improves over time with either approach. When a user is allowed to choose between regex construction and manual labeling, we discover that (1) if the time budget is low, spending all time for regex construction is often advantageous, (2) if the time budget is high, spending all time for manual labeling seems to be superior, and (3) between those two extremes, writing regexes followed by manual labeling is typically the best approach. Our code and data is available at https://github.com/nymph332088/HILRecognizer.
Auto-annotation for voice-enabled entertainment systems
Authors: Wenyan Li 0001, Ferhan Türe
Venue: Conference SIGIR 2020
Year: 2020

Voice-activated intelligent entertainment systems are prevalent in modern TVs. These systems require accurate automatic speech recognition (ASR) models to transcribe voice queries for further downstream language understanding tasks. Currently, labeling audio data for training is the main bottleneck in deploying accurate machine learning ASR models, especially when these models require up-to-date training data to adapt to the shifting customer needs. We present an auto-annotation system, which provides high quality training data without any hand-labeled audios by detecting speech recognition errors and providing possible fixes. Through our algorithm, the auto-annotated training data reaches an overall word error rate (WER) of 0.002; furthermore, we obtained a reduction of 0.907 in WER after applying the auto-suggested fixes.
What do viewers say to their TVs? an analysis of voice queries to entertainment systems
Authors: Jinfeng Rao, Ferhan Türe, Jimmy Lin
Venue: Conference SIGIR 2018
Year: 2018

A recently-introduced product of Comcast, a large cable company in the United States, is a "voice remote" that accepts spoken queries from viewers. We present an analysis of a large query log from this service to answer the question: "What do viewers say to their TVs?" In addition to a descriptive characterization of queries and sessions, we describe two complementary types of analyses to support query understanding. First, we propose a domain-specific intent taxonomy to characterize viewer behavior: as expected, most intents revolve around watching programs---both direct navigation as well as browsing---but there is a non-trivial fraction of non-viewing intents as well. Second, we propose a domain-specific tagging scheme for labeling query tokens, that when combined with intent and program prediction, provides a multi-faceted approach to understand voice queries directed at entertainment systems.
Yelling at your TV: An analysis of speech recognition errors and subsequent user behavior on entertainment systems
Authors: Raphael Tang, Ferhan Türe, Jimmy Lin
Venue: Conference SIGIR 2019
Year: 2019

Millions of consumers issue voice queries through television-based entertainment systems such as the Comcast X1, the Amazon Fire TV, and Roku TV. Automatic speech recognition (ASR) systems are responsible for transcribing these voice queries into text to feed downstream natural language understanding modules. However, ASR is far from perfect, often producing incorrect transcriptions and forcing users to take corrective action. To better understand their impact on sessions, this paper characterizes speech recognition errors as well as subsequent user responses. We provide both quantitative and qualitative analyses, examining the acoustic as well as lexical attributes of the utterances. This work represents, to our knowledge, the first analysis of speech recognition errors from real users on a widely-deployed entertainment system.
KLOOS: KL divergence-based out-of-scope intent detection in human-to-machine conversations
Authors: Eyup Halit Yilmaz, Cagri Toraman
Venue: Conference SIGIR 2020
Year: 2020

User intent is not restricted in human-to-machine conversations, and sometimes overshoots the scope of a designed system. Many tasks for understanding conversations require the elimination of such out-of-scope queries. We propose an out-of-scope intent detection method, called KLOOS, based on a novel feature extraction mechanism that incorporates the information accumulation of sequential word processing. Information is accumulated by KL divergence between the intent distributions of consecutive words. The performance of our approach is compared with the conventional classifiers and state-of-the-art language models fine-tuned for out-of-scope detection on three spoken query collections. The results show that KLOOS statistically significantly improves out-of-scope sensitivity in all cases, while the overall performance is not deteriorated in most cases.
Degree- Aware Alignment for Entities in Tail
Authors: Weixin Zeng, Xiang Zhao 0002, Wei Wang 0011, Jiuyang Tang, Zhen Tan
Venue: Conference SIGIR 2020
Year: 2020

Entity alignment (EA) is to discover equivalent entities in knowledge graphs (KGs), which bridges heterogeneous sources of information and facilitates the integration of knowledge. Existing EA solutions mainly rely on structural information to align entities, typically through KG embedding. Nonetheless, in real-life KGs, only a few entities are densely connected to others, and the rest majority possess rather sparse neighborhood structure. We refer to the latter as long-tail entities, and observe that such phenomenon arguably limits the use of structural information for EA. To mitigate the issue, we revisit and investigate into the conventional EA pipeline in pursuit of elegant performance. For pre-alignment, we propose to amplify long-tail entities, which are of relatively weak structural information, with entity name information that is generally available (but overlooked) in the form of concatenated power mean word embeddings. For alignment, under a novel complementary framework of consolidating structural and name signals, we identify entity's degree as important guidance to effectively fuse two different sources of information. To this end, a degree-aware co-attention network is conceived, which dynamically adjusts the significance of features in a degree-aware manner. For post-alignment, we propose to complement original KGs with facts from their counterparts by using confident EA results as anchors via iterative training. Comprehensive experimental evaluations validate the superiority of our proposed techniques.