OpenRaiser/Intern-Atlas huggingface dataset:
Twanchor text: a preliminary study of the value of tweets as anchor text
Authors: Gilad Mishne, Jimmy Lin
Venue: SIGIR
Year: 2012

It is well known that anchor text plays an important role in search, providing signals that are often not present in the source document itself. The paper reports results of a preliminary investigation on the value of tweets and tweet conversations as anchor text. We show that using tweets as anchors improves significantly over using HTML anchors, and significantly increases recall of news item retrieval.
Aspect-based opinion mining from product reviews
Authors: Samaneh Moghaddam, Martin Ester
Venue: SIGIR
Year: 2012

"What other people think" has always been an important piece of information for most of us during the decision-making process. Today people tend to make their opinions available to other people via the Internet. As a result, the Web has become an excellent source of consumer opinions. There are now numerous Web resources containing such opinions, e.g., product reviews forums, discussion groups, and blogs. But, it is really difficult for a customer to read all of the reviews and make an informed decision on whether to purchase the product. It is also difficult for the manufacturer of the product to keep track and manage customer opinions. Also, focusing on just user ratings (stars) is not a sufficient source of information for a user or the manufacturer to make decisions. Therefore, mining online reviews (opinion mining) has emerged as an interesting new research direction. Extracting aspects and the corresponding ratings is an important challenge in opinion mining. An aspect is an attribute or component of a product, e.g. 'zoom' for a digital camera. A rating is an intended interpretation of the user satisfaction in terms of numerical values. Reviewers usually express the rating of an aspect by a set of sentiments, e.g. 'great zoom'. In this tutorial we cover opinion mining in online product reviews with the focus on aspect-based opinion mining. This problem is a key task in the area of opinion mining and has attracted a lot of researchers in the information retrieval community recently. Several opinion related information retrieval tasks can benefit from the results of aspect-based opinion mining and therefore it is considered as a fundamental problem. This tutorial covers not only general opinion mining and retrieval tasks, but also state-of-the-art methods, challenges, applications, and also future research directions of aspect-based opinion mining.
Detecting quilted web pages at scale
Authors: Marc Najork
Venue: SIGIR
Year: 2012

Web-based advertising and electronic commerce, combined with the key role of search engines in driving visitors to ad-monetized and e-commerce web sites, has given rise to the phenomenon of web spam: web pages that are of little value to visitors, but that are created mainly to mislead search engines into driving traffic to target web sites. A large fraction of spam web pages is automatically generated, and some portion of these pages is generated by stitching together parts (sentences or paragraphs) of other web pages. This paper presents a scalable algorithm for detecting such "quilted" web pages. Previous work by the author and his collaborators introduced a sampling-based algorithm that was capable of detecting some, but by far not all quilted web pages in a collection. By contrast, the algorithm presented in this work identifies all quilted web pages, and it is scalable to very large corpora. We tested the algorithm on the half-billion page English-language subset of the ClueWeb09 collection, and evaluated its effectiveness in detecting web spam by manually inspecting small samples of the detected quilted pages. This manual inspection guided us in iteratively refining the algorithm to be more efficient in detecting real-world spam.
Improving tweet stream classification by detecting changes in word probability
Authors: Kyosuke Nishida, Takahide Hoshide, Ko Fujimura
Venue: SIGIR
Year: 2012

We propose a classification model of tweet streams in Twitter, which are representative of document streams whose statistical properties will change over time. Our model solves several problems that hinder the classification of tweets; in particular, the problem that the probabilities of word occurrence change at different rates for different words. Our model switches between two probability estimates based on full and recent data for each word when detecting changes in word probability. This switching enables our model to achieve both accurate learning of stationary words and quick response to bursty words. We then explain how to implement our model by using a word suffix array, which is a full-text search index. Using the word suffix array allows our model to handle the temporal attributes of word n-grams effectively. Experiments on three tweet data sets demonstrate that our model offers statistically significant higher topic-classification accuracy than conventional temporally-aware classification models.
Top-k learning to rank: labeling, ranking and evaluation
Authors: Shuzi Niu, Jiafeng Guo, Yanyan Lan, Xueqi Cheng
Venue: SIGIR
Year: 2012

In this paper, we propose a novel top-k learning to rank framework, which involves labeling strategy, ranking model and evaluation measure. The motivation comes from the difficulty in obtaining reliable relevance judgments from human assessors when applying learning to rank in real search systems. The traditional absolute relevance judgment method is difficult in both gradation specification and human assessing, resulting in high level of disagreement on judgments. While the pairwise preference judgment, as a good alternative, is often criticized for increasing the complexity of judgment from O(n) to (n log n). Considering the fact that users mainly care about top ranked search results, we propose a novel top-k labeling strategy which adopts the pairwise preference judgment to generate the top k ordering items from n documents (i.e. top-k ground-truth) in a manner similar to that of HeapSort. As a result, the complexity of judgment is reduced to O(n log k). With the top-k ground-truth, traditional ranking models (e.g. pairwise or listwise models) and evaluation measures (e.g. NDCG) no longer fit the data set. Therefore, we introduce a new ranking model, namely FocusedRank, which fully captures the characteristics of the top-k ground-truth. We also extend the widely used evaluation measures NDCG and ERR to be applicable to the top-k ground-truth, referred as κ-NDCG and κ-ERR, respectively. Finally, we conduct extensive experiments on benchmark data collections to demonstrate the efficiency and effectiveness of our top-k labeling strategy and ranking models.
A visual tool for bayesian data analysis: the impact of smoothing on naive bayes text classifiers
Authors: Giorgio Maria Di Nunzio, Alessandro Sordoni
Venue: SIGIR
Year: 2012

Naive-Bayes (NB) classifiers are simple probabilistic classifiers still widely used in supervised learning due to their tradeoff between efficient model training and good empirical results. One of the drawbacks of these classifiers is that in situations of data sparsity (i.e. when the size of training set is small) the maximum likelihood estimation of the probability of unseen features in these situations is equal to zero causing arithmetic anomalies. To prevent this undesirable behavior, a number of smoothing techniques have been proposed. Among these, the Bayesian approach incorporates smoothing in terms of prior knowledge about the parameters of the model usually called hyper-parameters. Our research question is: can a visualization tool help researchers to quickly assess the goodness of the performance of NB classifiers by setting optimal smoothing parameters?
Entity sentiment extraction using text ranking
Authors: John O&apos, Neil
Venue: SIGIR
Year: 2012

Entity extraction and sentiment classification are among the most common types of information derived from documents, but the problem of directly associating entities and sentiment has received less attention. We use TextRank on a graph linking entities and sentiment-laden words and phrases. We extract from the resulting eigenvector the final sentiment weights of the entities. We then explore the algorithm's performance and accuracy, compared to a baseline.
Social annotations: utility and prediction modeling
Authors: Patrick Pantel, Michael Gamon, Omar Alonso, Kevin Haas
Venue: SIGIR
Year: 2012

Social features are increasingly integrated within the search results page of the main commercial search engines. There is, however, little understanding of the utility of social features in traditional search. In this paper, we study utility in the context of social annotations, which are markings indicating that a person in the social network of the user has liked or shared a result document. We introduce a taxonomy of social relevance aspects that influence the utility of social annotations in search, spanning query classes, the social network, and content relevance. We present the results of a user study quantifying the utility of social annotations and the interplay between social relevance aspects. Through the user study we gain insights on conditions under which social annotations are most useful to a user. Finally, we present machine learned models for predicting the utility of a social annotation using the user study judgments as an optimization criterion. We model the learning task with features drawn from web usage logs, and show empirical evidence over real-world head and tail queries that the problem is learnable and that in many cases we can predict the utility of a social annotation.
Efficient estimation of aspect weights
Authors: Jon Parker, Andrew Yates, Nazli Goharian, Wai Gen Yee
Venue: SIGIR
Year: 2012

Many websites encourage people to submit reviews of various products and services. We present and evaluate a novel approach to efficiently model and analyze the text within user reviews to estimate how much reviewers care about different aspects of a product (i.e., amenities, food, location, room, etc. of a hotel). Our approach performs statistically quite similar to the best existing method. However, our method for computing aspect weights is a linear time method while the current state of the art solution requires cubic time at best.
BReK12: a book recommender for K-12 users
Authors: Maria Soledad Pera, Yiu-Kai Ng
Venue: SIGIR
Year: 2012

Ideally, students in K-12 grade levels can turn to book recommenders to locate books that match their interests. Existing book recommenders, however, fail to take into account the readability levels of their users, and hence their recommendations may be unsuitable for the users. To address this issue, we introduce BReK12, a recommender that targets K-12 users and prioritizes the reading level of its users in suggesting books of interest. Empirical studies conducted using the Bookcrossing dataset show that BReK12 outperforms a number of existing recommenders (developed for general users) in identifying books appealing to K-12 users.
Adversarial content manipulation effects
Authors: Fiana Raiber
Venue: SIGIR
Year: 2012

We address a question that has been somewhat overlooked throughout the transition from classical ad hoc retrieval to Web search: how is the performance of classical retrieval approaches affected by the presence of content manipulation? Our initial experiments have shown that the relative performance patterns of some classical retrieval strategies might change in the transition from non-manipulated to manipulated corpora. A natural future venue to explore is how to mix these strategies and make (some of) them more robust under presumed content manipulation conditions.
Lightweight contrastive summarization for news comment mining
Authors: Gobaan Raveendran, Charles L. A. Clarke
Venue: SIGIR
Year: 2012

We develop and discuss a news comment miner that presents distinct viewpoints on a given theme or event. Given a query, the system uses metasearch techniques to find relevant news articles. Relevant articles are then scraped for both article content and comments. Snippets from the comments are sampled and presented to the user, based on theme popularity and contrastiveness to previously selected snippets. The system design focuses on being quicker and more lightweight than recent topic modelling approaches, while still focusing on selecting orthogonal snippets.
On real-time ad-hoc retrieval evaluation
Authors: Stephen E. Robertson, Evangelos Kanoulas
Venue: SIGIR
Year: 2012

Lab-based evaluations typically assess the quality of a retrieval system with respect to its ability to retrieve documents that are relevant to the information need of an end user. In a real-time search task however users not only wish to retrieve the most relevant items but the most recent as well. The current evaluation framework is not adequate to assess the ability of a system to retrieve both recent and relevant items, and the one proposed in the recent TREC Microblog Track has certain flaws that quickly became apparent to the organizers. In this poster, we redefine the experiment for a real-time ad-hoc search task, by setting new submission requirements for the submitted systems/runs, proposing metrics to be used in evaluating the submissions, and suggesting a pooling strategy to be used to gather relevance judgments towards the computation of the described metrics. The proposed task can indeed assess the quality of a retrieval system with regard to retrieving both relevant and timely information.
IR models: foundations and relationships
Authors: Thomas Roelleke
Venue: SIGIR
Year: 2012

In IR research it is essential to know IR models. Research over the past years has consolidated the foundations of IR models. Moreover, relationships have been reported that help to use and position IR models. Knowing about the foundations and relationships of IR models can significantly improve building information management systems.
CloudSearch and the democratization of information retrieval
Authors: Daniel E. Rose
Venue: SIGIR
Year: 2012

Amazon CloudSearch is a new hosted search service, built on top of many cloud-based AWS services, and based on the same technology that powers search on Amazon's retail sites. Because of its ease of configuration and scalability, CloudSearch represents the next step in the democratization of information retrieval. This democratization process, increasing access to search for both end users and potential search providers, has continued over several decades, through technologies like early online metered search services, enterprise search software, web search, and open source search tools. CloudSearch further reduces barriers to entry, allowing a person or organization to basically say "make my content searchable" and have it happen automatically. CloudSearch may also offer an opportunity to overcome the stagnation that has occurred in search user experiences over the past 15 years. When you no longer need to be a search expert to make your content available, you're not stuck with ten blue links. Instead, you can focus on providing the kind of interaction that makes sense for your application and your users. CloudSearch enables a flowering of search applications that need not be tied to the web, and an opportunity to explore new ways of interacting with information retrieval technology.
Estimating the magic barrier of recommender systems: a user study
Authors: Alan Said, Brijnesh J. Jain, Sascha Narr, Till Plumbaum, Sahin Albayrak, Christian Scheel
Venue: SIGIR
Year: 2012

Recommender systems are commonly evaluated by trying to predict known, withheld, ratings for a set of users. Measures such as the Root-Mean-Square Error are used to estimate the quality of the recommender algorithms. This process does however not acknowledge the inherent rating inconsistencies of users. In this paper we present the first results from a noise measurement user study for estimating the magic barrier of recommender systems conducted on a commercial movie recommendation community. The magic barrier is the expected squared error of the optimal recommendation algorithm, or, the lowest error we can expect from any recommendation algorithm. Our results show that the barrier can be estimated by collecting the opinions of users on already rated items.
Towards zero-click mobile IR evaluation: knowing what and knowing when
Authors: Tetsuya Sakai
Venue: SIGIR
Year: 2012

In this poster, we propose two evaluation tasks for mobile information access. The first task evaluates the system's ability to guess what the user's query should be given a context ("Knowing What"). The second task evaluates the system's ability to decide when to proactively deploy a given query ("Knowing When"). We conduct a preliminary manual analysis of a mobile query log to limit the space of possible queries so as to design feasible and practical evaluation tasks.
Collaborative information seeking: art and science of achieving 1+1>2 in IR
Authors: Chirag Shah
Venue: SIGIR
Year: 2012

The assumption of information seekers being independent and IR problem being individual has been challenged often in the recent past, with an argument that the next big leap in search and retrieval will come through incorporating social and collaborative aspects of information seeking. This half-day tutorial will introduce the student to theories, methodologies, and tools that focus on information retrieval/seeking in collaboration. The student will have an opportunity to learn about the social aspect of IR with a focus on collaborative information seeking (CIS) situations, systems, and evaluation techniques. The course is intended for those interested in social and collaborative aspects of IR (from both academia and industry), and requires only a general understanding of IR systems and evaluation.
Time-sensitive query auto-completion
Authors: Milad Shokouhi, Kira Radinsky
Venue: SIGIR
Year: 2012

Query auto-completion (QAC) is a common feature in modern search engines. High quality QAC candidates enhance search experience by saving users time that otherwise would be spent on typing each character or word sequentially.
Parallelizing ListNet training using spark
Authors: Shilpa Shukla, Matthew Lease, Ambuj Tewari
Venue: SIGIR
Year: 2012

As ever-larger training sets for learning to rank are created, scalability of learning has become increasingly important to achieving continuing improvements in ranking accuracy. Exploiting independence of "summation form" computations, we show how each iteration in ListNet gradient descent can benefit from parallel execution. We seek to draw the attention of the IR community to use Spark, a newly introduced distributed cluster computing system, for reducing training time of iterative learning to rank algorithms. Unlike MapReduce, Spark is especially suited for iterative and interactive algorithms. Our results show near linear reduction in ListNet training time using Spark on Amazon EC2 clusters.
Retrieving similar discussion forum threads: a structure based approach
Authors: Amit Singh, Deepak P, Dinesh Raghu
Venue: SIGIR
Year: 2012

Online forums are becoming a popular way of finding useful information on the web. Search over forums for existing discussion threads so far is limited to keyword-based search due to the minimal effort required on part of the users. However, it is often not possible to capture all the relevant context in a complex query using a small number of keywords. Example-based search that retrieves similar discussion threads given one exemplary thread is an alternate approach that can help the user provide richer context and vastly improve forum search results. In this paper, we address the problem of finding similar threads to a given thread. Towards this, we propose a novel methodology to estimate similarity between discussion threads. Our method exploits the thread structure to decompose threads in to set of weighted overlapping components. It then estimates pairwise thread similarities by quantifying how well the information in the threads are mutually contained within each other using lexical similarities between their underlying components. We compare our proposed methods on real datasets against state-of-the-art thread retrieval mechanisms wherein we illustrate that our techniques outperform others by large margins on popular retrieval evaluation measures such as NDCG, MAP, [email protected] and MRR. In particular, consistent improvements of up to 10% are observed on all evaluation measures.
Time to judge relevance as an indicator of assessor error
Authors: Mark D. Smucker, Chandra Prakash Jethani
Venue: SIGIR
Year: 2012

When human assessors judge documents for their relevance to a search topic, it is possible for errors in judging to occur. As part of the analysis of the data collected from a 48 participant user study, we have discovered that when the participants made relevance judgments, the average participant spent more time to make errorful judgments than to make correct judgments. Thus, in relevance assessing scenarios similar to our user study, it may be possible to use the time taken to judge a document as an indicator of assessor error. Such an indicator could be used to identify documents that are candidates for adjudication or reassessment.
Multi-aspect query summarization by composite query
Authors: Wei Song, Qing Yu, Zhiheng Xu, Ting Liu, Sheng Li, Ji-Rong Wen
Venue: SIGIR
Year: 2012

Conventional search engines usually return a ranked list of web pages in response to a query. Users have to visit several pages to locate the relevant parts. A promising future search scenario should involve: (1) understanding user intents; (2) providing relevant information directly to satisfy searchers' needs, as opposed to relevant pages. In this paper, we present a search paradigm to summarize a query's information from different aspects. Query aspects could be aligned to user intents. The generated summaries for query aspects are expected to be both specific and informative, so that users can easily and quickly find relevant information. Specifically, we use a Composite Query for Summarization" method, where a set of component queries are used for providing additional information for the original query. The system leverages the search engine to proactively gather information by submitting multiple component queries according to the original query and its aspects. In this way, we could get more relevant information for each query aspect and roughly classify information. By comparative mining the search results of different component queries, it is able to identify query (dependent) aspect words, which help to generate more specific and informative summaries. The experimental results on two data sets, Wikipedia and TREC ClueWeb2009, are encouraging. Our method outperforms two baseline methods on generating informative summaries.
Short text classification using very few words
Authors: Aixin Sun
Venue: SIGIR
Year: 2012

We propose a simple, scalable, and non-parametric approach for short text classification. Leveraging the well studied and scalable Information Retrieval (IR) framework, our approach mimics human labeling process for a piece of short text. It first selects the most representative and topical-indicative words from a given short text as query words, and then searches for a small set of labeled short texts best matching the query words. The predicted category label is the majority vote of the search results. Evaluated on a collection of more than 12K Web snippets, the proposed approach achieves comparable classification accuracy with the baseline Maximum Entropy classifier using as few as 3 query words and top-5 best matching search hits. Among the four query word selection schemes proposed and evaluated in our experiments, term frequency together with clarity gives the best classification accuracy.