OpenRaiser/Intern-Atlas huggingface dataset:
Relevance as a subjective and situational multidimensional concept
Authors: Carsten Eickhoff
Venue: SIGIR
Year: 2012

Relevance is the central concept of information retrieval. Although its important role is unanimously accepted among researchers, numerous different definitions of the term have emerged over the years. Considerable effort has been put into creating consistent and universally applicable descriptions of relevance in the form of relevance frameworks. Across these various formal systems of relevance, a wide range of relevance criteria has been identified. The probably most frequently used single criterion, that in some applications even becomes a synonym for relevance, is topicality. It expresses a document's topical overlap with the user's information need. For textual resources, it is often estimated based on term co-occurrences between query and document. There is, however, a significant number of further noteworthy relevance criteria. Prominent specimen are: (Currency) determines how recent and up to date the document is. Outdated information may have become invalid over time.
Quality through flow and immersion: gamifying crowdsourced relevance assessments
Authors: Carsten Eickhoff, Christopher G. Harris, Arjen P. de Vries, Padmini Srinivasan
Venue: SIGIR
Year: 2012

Crowdsourcing is a market of steadily-growing importance upon which both academia and industry increasingly rely. However, this market appears to be inherently infested with a significant share of malicious workers who try to maximise their profits through cheating or sloppiness. This serves to undermine the very merits crowdsourcing has come to represent. Based on previous experience as well as psychological insights, we propose the use of a game in order to attract and retain a larger share of reliable workers to frequently-requested crowdsourcing tasks such as relevance assessments and clustering. In a large-scale comparative study conducted using recent TREC data, we investigate the performance of traditional HIT designs and a game-based alternative that is able to achieve high quality at significantly lower pay rates, facing fewer malicious submissions.
RDF Xpress: a flexible expressive RDF search engine
Authors: Shady Elbassuoni, Maya Ramanath, Gerhard Weikum
Venue: SIGIR
Year: 2012

We demonstrate RDF Xpress, a search engine that enables users to effectively retrieve information from large RDF knowledge bases or Linked Data Sources. RDF Xpress provides a search interface where users can combine triple patterns with keywords to form queries. Moreover, RDF Xpress supports automatic query relaxation and returns a ranked list of diverse query results.
Confidence-aware graph regularization with heterogeneous pairwise features
Authors: Yuan Fang, Bo-June Paul Hsu, Kevin Chen-Chuan Chang
Venue: SIGIR
Year: 2012

Conventional classification methods tend to focus on features of individual objects, while missing out on potentially valuable pairwise features that capture the relationships between objects. Although recent developments on graph regularization exploit this aspect, existing works generally assume only a single kind of pairwise feature, which is often insufficient. We observe that multiple, heterogeneous pairwise features can often complement each other and are generally more robust in modeling the relationships between objects. Furthermore, as some objects are easier to classify than others, objects with higher initial classification confidence should be weighed more towards classifying related but more ambiguous objects, an observation missing from previous graph regularization techniques. In this paper, we propose a Dirichlet-based regularization framework that supports the combination of heterogeneous pairwise features with confidence-aware prediction using limited labeled training data. Next, we showcase a few applications of our framework in information retrieval, focusing on the problem of query intent classification. Finally, we demonstrate through a series of experiments the advantages of our framework on a large-scale real-world dataset.
Task-aware search assistant
Authors: Henry Allen Feild, James Allan
Venue: SIGIR
Year: 2012

When users search the web, their goal is to accomplish something, which we call a search task. Previous studies have observed an average of 2.6–3.3 queries per search task [2, 3, 4] and Jones and Klinkner [2] found 16% of tasks were revisited over a three-day span of Yahoo! logs. These statistics suggest that a task-aware search assistant may aid users in continuing previous search tasks as well as new ones. There are several existing tools that are related or similar in nature to a task-aware search assistant, the most basic of which are built-in browser histories and extensions that make them easier to navigate. Both Google and Bing provide search histories over their respective services. However, these lack a task-oriented view of searches. The most closely related work that we are aware of is Yahoo’s Search Pad, which attempts to automatically identify that a user is conducting research (e.g., planning a vacation). It then notifies the user and begins tracking queries and result clicks in an interface, to which the user can add and remove content. Two drawbacks are its lack of support for non-research tasks and its restriction to Yahoo! search only. Our approach to aiding users during search—both for new and continuing tasks—is to present them with an easy to use sidebar implemented as a browser extension. The interface consists of a collapsible sidebar in the browser with multiple panes that can be paged through using streamlined navigation buttons. The primary pane displays three key groupings: (1) a list of the most recent search tasks, (2) a list of related queries entered in the past, and (3) a list of related tasks conducted in the past. Note that a task consists of a cluster of queries and visited documents. This primary
Retrieval evaluation on focused tasks
Authors: Besnik Fetahu, Ralf Schenkel
Venue: SIGIR
Year: 2012

Ranking of retrieval systems for focused tasks requires large number of relevance judgments. We propose an approach that minimizes the number of relevance judgments, where the performance measures are approximated using a Monte-Carlo sampling technique. Partial measures are taken using relevance judgments, whereas the remaining part of passages are annotated using a generated relevance probability distribution based on result rank. We define two conditions for stopping the assessment procedure when the ranking between systems is stable.
Scheduling queries across replicas
Authors: Ana Freire, Craig Macdonald, Nicola Tonellotto, Iadh Ounis, Fidel Cacheda
Venue: SIGIR
Year: 2012

For increased efficiency, an information retrieval system can split its index into multiple shards, and then replicate these shards across many query servers. For each new query, an appropriate replica for each shard must be selected, such that the query is answered as quickly as possible. Typically, the replica with the lowest number of queued queries is selected. However, not every query takes the same time to execute, particularly if a dynamic pruning strategy is applied by each query server. Hence, the replica's queue length is an inaccurate indicator of the workload of a replica, and can result in inefficient usage of the replicas. In this work, we propose that improved replica selection can be obtained by using query efficiency prediction to measure the expected workload of a replica. Experiments are conducted using 2.2k queries, over various numbers of shards and replicas for the large GOV2 collection. Our results show that query waiting and completion times can be markedly reduced, showing that accurate response time predictions can improve scheduling accuracy and attesting the benefit of the proposed scheduling algorithm.
Opinion influence and diffusion in social network
Authors: Dehong Gao
Venue: SIGIR
Year: 2012

Nowadays, more and more people tend to make decisions based on the opinion information from the Internet, in addition to recommendations from offline friends or parents. For example, we may browse the resumes and comments on election candidates to determine if one candidate is qualified, or consult the consumer reports or reviews on special e-commercial websites to decide which brand of computer is suitable for one's needs. Though opinion information is rich on the Internet, [2] points out that 58% of American Internet users deem that online information is irretrievable, confusing, or conflicting with each other. Early works on opinion mining help to classify opinion polarity, to extract specific opinions and to summarize opinion texts. However, all these works are usually based on plain texts (reviews, comments or news articles). With the explosion of Web 2.0 applications, especially social network applications like blogs, discussion forums, micro-blogs, the massive individual users go to the major media websites, which leads to much more opinion materials posted on the Internet by user-shared experiences or views [3]. These opinion-rich and social network-based applications bring new perspectives for opinion mining as well. First, in addition to plain texts (reviews, newswire) in traditional opinion mining, we see new types of cyber-based text, like personal diary blogs, cyber-SMS tweets. Second, if we regard the opinions in plain text as static, the dynamic change of opinions in the social network is a new promising area, and catch increasing attention of worldwide researchers. In the social network, the opinion held by one individual is not static, but changes, which can be influenced by others. A serial of changes among different users forms the opinion propagation or diffusion in the network.
Large-scale graph mining and learning for information retrieval
Authors: Bin Gao, Taifeng Wang, Tie-Yan Liu
Venue: SIGIR
Year: 2012

For many information retrieval applications, we need to deal with the ranking problem on very large scale graphs. However, it is non-trivial to perform efficient and effective ranking on them. On one aspect, we need to design scalable algorithms. On another aspect, we also need to develop powerful computational infrastructure to support these algorithms. This tutorial aims at giving a timely introduction to the promising advances in the aforementioned aspects in recent years, and providing the audiences with a comprehensive view on the related literature.
GLASE 0.1: eyes tell more than mice
Authors: Viktors Garkavijs, Mayumi Toshima, Noriko Kando
Venue: SIGIR
Year: 2012

This paper proposes a prototype system called Gaze-Learning-Access-and-Search-Engine 0.1 (GLASE), which can perform image relevance ranking based on gaze data and within-session learning. We developed a search user interface that uses an eye-tracker as an input device and employed a relevance re-ranking algorithm based on the gaze length. The preliminary experimental results showed that using our gaze-driven system reduced the task completion time an average of 13.7% in a search session.
Improving e-discovery using information retrieval
Authors: Kripabandhu Ghosh
Venue: SIGIR
Year: 2012

E-discovery is the requirement that the documents and information in electronic form stored in corporate systems be produced as evidence in litigation. It has posed great challenges for legal experts. Legal searchers have always looked to find "any and all" evidence for a given case. Thus, a legal search system would essentially be a recall-oriented system. It has been a common practice among expert searchers to formulate Boolean queries to represent their information need. We want to work on three basic problems: Boolean query formulation - Our primary goal is to study Boolean query formulation in the light of the E-discovery task. This will include automatic Boolean query generation, expansion and learning the effect of proximity operators in Boolean searches. Data fusion - We would also like to explore the effectiveness of data fusion techniques in improving recall. Error modeling - Finally, we will work on error modeling methods for noisy legal documents.
Ousting ivory tower research: towards a web framework for providing experiments as a service
Authors: Tim Gollub, Benno Stein, Steven Burrows
Venue: SIGIR
Year: 2012

With its close ties to the Web, the IR community is destined to leverage the dissemination and collaboration capabilities that the Web provides today. Especially with the advent of the software as a service principle, an IR community is conceivable that publishes experiments executable by anyone over the Web. A review of recent SIGIR papers shows that we are far away from this vision of collaboration. The benefits of publishing IR experiments as a service are striking for the community as a whole, and include potential to boost research profiles and reputation. However, the additional work must be kept to a minimum and sensitive data must be kept private for this paradigm to become an accepted practice. To foster experiments as a service in IR, we present a Web framework for experiments that addresses the outlined challenges and possesses a unique set of compelling features in comparison to existing solutions. We also describe how our reference implementation is already used officially as an evaluation platform for an established international plagiarism detection competition.
Medical information retrieval: an instance of domain-specific search
Authors: Allan Hanbury
Venue: SIGIR
Year: 2012

Due to an explosion in the amount of medical information available, search techniques are gaining importance in the medical domain. This tutorial discusses recent results on search in the medical domain, including the outcome of surveys on end user requirements, research relevant to the field, and current medical and health search applications available. Finally, the extent to which available techniques meet user requirements are discussed, and open challenges in the field are identified.
Optimizing positional index structures for versioned document collections
Authors: Jinru He, Torsten Suel
Venue: SIGIR
Year: 2012

Versioned document collections are collections that contain multiple versions of each document. Important examples are Web archives, Wikipedia and other wikis, or source code and documents maintained in revision control systems. Versioned document collections can become very large, due to the need to retain past versions, but there is also a lot of redundancy between versions that can be exploited. Thus, versioned document collections are usually stored using special differential (delta) compression techniques, and a number of researchers have recently studied how to exploit this redundancy to obtain more succinct full-text index structures.
Learning to rank social update streams
Authors: Liangjie Hong, Ron Bekkerman, Joseph Adler, Brian D. Davison
Venue: SIGIR
Year: 2012

As online social media further integrates deeper into our lives, we spend more time consuming social update streams that come from our online connections. Although social update streams provide a tremendous opportunity for us to access information on-the-fly, we often complain about its relevance. Some of us are flooded with a steady stream of information and simply cannot process it in full. Ranking the incoming content becomes the only solution for the overwhelmed users. For some others, in contrast, the incoming information stream is pretty weak, and they have to actively search for relevant information which is quite tedious. For these users, augmenting their incoming content flow with relevant information from outside their first-degree network would be a viable solution. In that case, the problem of relevance becomes even more prominent. In this paper, we start an open discussion on how to build effective systems for ranking social updates from a unique perspective of LinkedIn -- the largest professional network in the world. More specifically, we address this problem as an intersection of learning to rank, collaborative filtering, and clickthrough modeling, while leveraging ideas from information retrieval and recommender systems. We propose a novel probabilistic latent factor model with regressions on explicit features and compare it with a number of non-trivial baselines. In addition to demonstrating superior performance of our model, we shed some light on the nature of social updates on LinkedIn and how users interact with them, which might be applicable to social update streams in general.
An uncertainty-aware query selection model for evaluation of IR systems
Authors: Mehdi Hosseini, Ingemar J. Cox, Natasa Milic-Frayling, Milad Shokouhi, Emine Yilmaz
Venue: SIGIR
Year: 2012

We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods.
Genre classification for million song dataset using confidence-based classifiers combination
Authors: Yajie Hu, Mitsunori Ogihara
Venue: SIGIR
Year: 2012

We proposed a method to classify songs in the Million Song Dataset according to song genre. Since songs have several data types, we trained sub-classifiers by different types of data. These sub-classifiers are combined using both classifier authority and classification confidence for a particular instance. In the experiments, the combined classifier surpasses all of these sub-classifiers and the SVM classifier using concatenated vectors from all data types. Finally, the genre labels for the Million Song Dataset are provided.
Integrative online research-data management
Authors: Michael Huggett, Edie Rasmussen
Venue: SIGIR
Year: 2012

In support of our research projects in information retrieval, we have developed an integrated multi-process software system that shepherds research data from induction through aggregation, analysis, and presentation. We combine public-domain code libraries with our own software to provide a flexible, easily- configured modular system that exposes data online for easier collaboration. The goal is to create a single online infrastructure that allows colleagues to submit, process, analyze and visualize data, and discuss and prioritize issues through a single integrated interface. We demonstrate our system within the context of the large data set provided by the Indexer's Legacy project [1].
Clarity re-visited
Authors: Shay Hummel, Anna Shtok, Fiana Raiber, Oren Kurland, David Carmel
Venue: SIGIR
Year: 2012

We present a novel interpretation of Clarity [5], a widely used query performance predictor. While Clarity is commonly described as a measure of the "distance" between the language model of the top-retrieved documents and that of the collection, we show that it actually quantifies an additional property of the result list, namely, its diversity. This analysis, along with empirical evaluation, helps to explain the low prediction quality of Clarity for large-scale Web collections.
AspecTiles: tile-based visualization of diversified web search results
Authors: Mayu Iwata, Tetsuya Sakai, Takehiro Yamamoto, Yu Chen, Yi Liu, Ji-Rong Wen, Shojiro Nishio
Venue: SIGIR
Year: 2012

A diversified search result for an underspecified query generally contains web pages in which there are answers that are relevant to different aspects of the query. In order to help the user locate such relevant answers, we propose a simple extension to the standard Search Engine Result Page (SERP) interface, called AspecTiles. In addition to presenting a ranked list of URLs with their titles and snippets, AspecTiles visualizes the relevance degree of a document to each aspect by means of colored squares ("tiles"). To compare AspecTiles with the standard SERP interface in terms of usefulness, we conducted a user study involving 30 search tasks designed based on the TREC web diversity task topics as well as 32 participants. Our results show that AspecTiles has some advantages in terms of search performance, user behavior, and user satisfaction. First, AspecTiles enables the user to gather relevant information significantly more efficiently than the standard SERP interface for tasks where the user considers several different aspects of the query to be important at the same time (multi-aspect tasks). Second, AspecTiles affects the user's information seeking behavior: with this interface, we observed significantly fewer query reformulations, shorter queries and deeper examinations of ranked lists in multi-aspect tasks. Third, participants of our user study found AspecTiles significantly more useful for finding relevant information and easy to use than the standard SERP interface. These results suggest that simple interfaces like AspecTiles can enhance the search performance and search experience of the user when their queries are underspecified.
Towards alias detection without string similarity: an active learning based approach
Authors: Lili Jiang, Jianyong Wang, Ping Luo, Ning An, Min Wang
Venue: SIGIR
Year: 2012

Entity aliases commonly exist and accurately detecting these aliases plays a vital role in various applications. In this paper, we use an active-learning-based method to detect aliases without string similarity. To minimize the cost on pairwise comparison, a subset-based method restricts the alias selection within a small-scale entity set. Within each generated entity set, an active learning based logistic regression classifier is employed to predict whether a candidate is the alias of a given entity. The experimental results on three datasets clearly demonstrate that our proposed approach can effectively detect this kind of entity aliases.
Finding readings for scientists from social websites
Authors: Jiepu Jiang, Zhen Yue, Shuguang Han, Daqing He
Venue: SIGIR
Year: 2012

Current search systems are designed to find relevant articles, especially topically relevant ones, but the notion of relevance largely depends on search tasks. We study the specific task that scientists are searching for worth-reading articles beneficial for their research. Our study finds: users' perception of relevance and preference of reading are only moderately correlated; current systems can effectively find readings that are highly relevant to the topic, but 36% of the worth-reading articles are only marginally relevant or even non-relevant. Our system can effectively find those worth-reading but marginally relevant or non-relevant articles by taking advantages of scientists' recommendations in social websites.