OpenRaiser/Intern-Atlas huggingface dataset:
A simple and efficient sampling method for estimating AP and NDCG
Authors: Emine Yilmaz, Evangelos Kanoulas, Javed A. Aslam
Venue: SIGIR
Year: 2008

We consider the problem of large scale retrieval evaluation. Recently two methods based on random sampling were proposed as a solution to the extensive effort required to judge tens of thousands of documents. While the first method proposed by Aslam et al. [1] is quite accurate and efficient, it is overly complex, making it difficult to be used by the community, and while the second method proposed by Yilmaz et al., infAP [14], is relatively simple, it is less efficient than the former since it employs uniform random sampling from the set of complete judgments. Further, none of these methods provide confidence intervals on the estimated values.
Non-greedy active learning for text categorization using convex transductive experimental design
Authors: Kai Yu, Shenghuo Zhu, Wei Xu, Yihong Gong
Venue: SIGIR
Year: 2008

In this paper we propose a non-greedy active learning method for text categorization using least-squares support vector machines (LSSVM). Our work is based on transductive experimental design (TED), an active learning formulation that effectively explores the information of unlabeled data. Despite its appealing properties, the optimization problem is however NP-hard and thus--like most of other active learning methods--a greedy sequential strategy to select one data example after another was suggested to find a suboptimum. In this paper we formulate the problem into a continuous optimization problem and prove its convexity, meaning that a set of data examples can be selected with a guarantee of global optimum. We also develop an iterative algorithm to efficiently solve the optimization problem, which turns out to be very easy-to-implement. Our text categorization experiments on two text corpora empirically demonstrated that the new active learning algorithm outperforms the sequential greedy algorithm, and is promising for active text categorization applications.
Relational distance-based collaborative filtering
Authors: Wei Zhang
Venue: SIGIR
Year: 2008

In this paper, we present a novel hybrid recommender system called RelationalCF, which integrate content and demographic information into a collaborative filtering framework by using relational distance computation approaches without the effort of form transformation and feature construction. Our experiments suggest that the effective combination of various kinds of information based on relational distance approaches provides improved accurate recommendations than other approaches.
Learning with support vector machines for query-by-multiple-examples
Authors: Dell Zhang, Wee Sun Lee
Venue: SIGIR
Year: 2008

We explore an alternative Information Retrieval paradigm called Query-By-Multiple-Examples (QBME) where the information need is described not by a set of terms but by a set of documents. Intuitive ideas for QBME include using the centroid of these documents or the well-known Rocchio algorithm to construct the query vector. We consider this problem from the perspective of text classification, and find that a better query vector can be obtained through learning with Support Vector Machines (SVMs). For online queries, we show how SVMs can be learned from one-class examples in linear time. For offline queries, we show how SVMs can be learned from positive and unlabeled examples together in linear or polynomial time. The effectiveness and efficiency of the proposed approaches have been confirmed by our experiments on four real-world datasets.
A generation model to unify topic relevance and lexicon-based sentiment for opinion retrieval
Authors: Min Zhang, Xingyao Ye
Venue: SIGIR
Year: 2008

Opinion retrieval is a task of growing interest in social life and academic research, which is to find relevant and opinionate documents according to a user's query. One of the key issues is how to combine a document's opinionate score (the ranking score of to what extent it is subjective or objective) and topic relevance score. Current solutions to document ranking in opinion retrieval are generally ad-hoc linear combination, which is short of theoretical foundation and careful analysis. In this paper, we focus on lexicon-based opinion retrieval. A novel generation model that unifies topic-relevance and opinion generation by a quadratic combination is proposed in this paper. With this model, the relevance-based ranking serves as the weighting factor of the lexicon-based sentiment ranking function, which is essentially different from the popular heuristic linear combination approaches. The effect of different sentiment dictionaries is also discussed. Experimental results on TREC blog datasets show the significant effectiveness of the proposed unified model. Improvements of 28.1% and 40.3% have been obtained in terms of MAP and [email protected] respectively. The conclusion is not limited to blog environment. Besides the unified generation model, another contribution is that our work demonstrates that in the opinion retrieval task, a Bayesian approach to combining multiple ranking functions is superior to using a linear combination. It is also applicable to other result re-ranking applications in similar scenario.
A topical PageRank based algorithm for recommender systems
Authors: Liyan Zhang, Kai Zhang, Chunping Li
Venue: SIGIR
Year: 2008

In this paper, we propose a Topical PageRank based algorithm for recommender systems, which aim to rank products by analyzing previous user-item relationships, and recommend top-rank items to potentially interested users. We evaluate our algorithm on MovieLens dataset and empirical experiments demonstrate that it outperforms other state-of-the-art recommending algorithms.
SOPING: a Chinese customer review mining system
Authors: Chao Zhou, Guang Qiu, Kangmiao Liu, Jiajun Bu, Mingcheng Qu, Chun Chen
Venue: SIGIR
Year: 2008

With the booming development of the Web, popular Chinese forums enable people to find experienced customers' reviews for products. In order to get an all-around opinion about one product, users need to go through plenty of web pages, which is time-consuming and inefficient. Consequently, automatic review mining and summarization has become a hot research topic recently. However, previous approaches are not applicable for mining Chinese customer reviews. In this paper, we introduce SOPING, a Chinese customer review mining system that mines reviews from forums. Specifically, we propose a novel search-based approach to extract product features and a feature-oriented sentence orientation determination method. Our experimental results show that our proposed techniques are highly effective.
Learning to rank with ties
Authors: Ke Zhou, Gui-Rong Xue, Hongyuan Zha, Yong Yu
Venue: SIGIR
Year: 2008

Designing effective ranking functions is a core problem for information retrieval and Web search since the ranking functions directly impact the relevance of the search results. The problem has been the focus of much of the research at the intersection of Web search and machine learning, and learning ranking functions from preference data in particular has recently attracted much interest. The objective of this paper is to empirically examine several objective functions that can be used for learning ranking functions from preference data. Specifically, we investigate the roles of ties in the learning process. By ties, we mean preference judgments that two documents have equal degree of relevance with respect to a query. This type of data has largely been ignored or not properly modeled in the past. In this paper, we analyze the properties of ties and develop novel learning frameworks which combine ties and preference data using statistical paired comparison models to improve the performance of learned ranking functions. The resulting optimization problems explicitly incorporating ties and preference data are solved using gradient boosting methods. Experimental studies are conducted using three publicly available data sets which demonstrate the effectiveness of the proposed new methods.
On profiling blogs with representative entries
Authors: Jinfeng Zhuang, Steven C. H. Hoi, Aixin Sun
Venue: SIGIR
Year: 2008

With an explosive growth of blogs, information seeking in blogosphere becomes more and more challenging. One example task is to find the most relevant topical blogs against a given query or an existing blog. Such a task requires concise representation of blogs for effective and efficient searching and matching. In this paper, we investigate a new problem of profiling a blog by choosing a set of m most representative entries from the blog, where m is a predefined number that is application-dependent. With the set of selected representative entries, applications on blogs avoid handling hundreds or even thousands of entries (or posts) associated with each blog, which are updated frequently and often noisy in nature. To guide the process of selecting the most representative entries, we propose three principles, i.e., anomaly, representativeness, and diversity. Based on these principles, a greedy yet very efficient entry selection algorithm is proposed. To evaluate the entry selection algorithms, an extrinsic evaluation methodology from document summarization research is adapted. Specifically, we evaluate the proposed entry selection algorithms by examining their blog classification accuracies. By evaluating on a number of different classification methods, our empirical results showed that comparable classification accuracy could be achieved by using fewer than 20 representative entries for each blog compared to that of engaging all entries.
Query-drift prevention for robust query expansion
Authors: Liron Zighelnic, Oren Kurland
Venue: SIGIR
Year: 2008

Pseudo-feedback-based automatic query expansion yields effective retrieval performance on average, but results in performance inferior to that of using the original query for many information needs. We address an important cause of this robustness issue, namely, the query drift problem, by fusing the results retrieved in response to the original query and to its expanded form. Our approach posts performance that is significantly better than that of retrieval based only on the original query and more robust than that of retrieval using the expanded query.
Exploiting correlated keywords to improve approximate information filtering
Authors: Christian Zimmer, Christos Tryfonopoulos, Gerhard Weikum
Venue: SIGIR
Year: 2008

Information filtering, also referred to as publish/subscribe, complements one-time searching since users are able to subscribe to information sources and be notified whenever new documents of interest are published. In approximate information filtering only selected information sources, that are likely to publish documents relevant to the user interests in the future, are monitored. To achieve this functionality, a subscriber exploits statistical metadata to identify promising publishers and index its continuous query only in those publishers. The statistics are maintained in a directory, usually on a per-keyword basis, thus disregarding possible correlations among keywords. Using this coarse information, poor publisher selection may lead to poor filtering performance and thus loss of interesting documents.1
Relevance criteria for e-commerce: a crowdsourcing-based experimental analysis
Authors: Omar Alonso, Stefano Mizzaro
Venue: SIGIR
Year: 2009

We discuss the concept of relevance criteria in the context of e-Commerce search. A vast body of research literature describes the beyond-topical criteria used to determine the relevance of the document to the need. We argue that in an e-Commerce scenario there are some differences, and novel and different criteria can be used to determine relevance. We experimentally validate this hypothesis by means of Amazon Mechanical Turk using a crowdsourcing approach.
Modeling uncertainty in video retrieval: a retrieval model for uncertain semantic representations of videos
Authors: Robin Aly
Venue: SIGIR
Year: 2009

The amount of multimedia content increases at a tremendous speed every day -- enabled by ever simpler production and processing techniques. Content based Multimedia Information Retrieval should help the user find information in these growing collections. However, the retrieval performance often prohibits real life application. The reason is the mismatch between the user's information need and the available low-level features. A lot of research aims to detect more meaningful representations of the multimedia objects, such as the occurrences of concepts or spoken words. However, the detection results are still far from being perfect. On the other hand, for most detectors it is possible to obtain a probability distribution over the possible representations of the object. This paper proposes the Expected Probability of Relevance Ranking Framework. A relationship between the objects representation and relevance is established by the probability of relevance given the representation, abstracting from the uncertain knowledge of this representation. The objects are then ranked by these probabilities weighted by the probability that the object has this representation, the expected probability of relevance. After introducing the framework we present previous work which estimates the probability of relevance given the binary occurrences of concepts as a reperesentation without prior relevance information. Then, following extensions are proposed: (1) a way to integrate the spoken content in the representation, (2) ways to improve the knowledge over the representation of individual shots and (3) a way to simulate detectors of a given performance to see what detector quality is needed.
The wisdom of the few: a collaborative filtering approach based on expert opinions from the web
Authors: Xavier Amatriain, Neal Lathia, Josep M. Pujol, Haewoon Kwak, Nuria Oliver
Venue: SIGIR
Year: 2009

Nearest-neighbor collaborative filtering provides a successful means of generating recommendations for web users. However, this approach suffers from several shortcomings, including data sparsity and noise, the cold-start problem, and scalability. In this work, we present a novel method for recommending items to users based on expert opinions. Our method is a variation of traditional collaborative filtering: rather than applying a nearest neighbor algorithm to the user-rating data, predictions are computed using a set of expert neighbors from an independent dataset, whose opinions are weighted according to their similarity to the user. This method promises to address some of the weaknesses in traditional collaborative filtering, while maintaining comparable accuracy. We validate our approach by predicting a subset of the Netflix data set. We use ratings crawled from a web portal of expert reviews, measuring results both in terms of prediction accuracy and recommendation list precision. Finally, we explore the ability of our method to generate useful recommendations, by reporting the results of a user-study where users prefer the recommendations generated by our approach.
Multiclass VisualRank: image ranking method in clustered subsets based on visual features
Authors: Mitsuru Ambai, Yuichi Yoshida
Venue: SIGIR
Year: 2009

This paper proposes Multiclass VisualRank, a method that expands the idea of VisualRank into more than one category of images. Multiclass VisualRank divides images retrieved from search engines into several categories based on distinctive patterns of visual features, and gives ranking within the category. Experimental results show that our method can extract several different image categories relevant to given keyword and gives good ranking scores to retrieved images.
The ESA retrieval model revisited
Authors: Maik Anderka, Benno Stein
Venue: SIGIR
Year: 2009

Among the retrieval models that have been proposed in the last years, the ESA model of Gabrilovich and Markovitch received much attention. The authors report on a significant improvement in the retrieval performance, which is explained with the semantic concepts introduced by the document collection underlying ESA. Their explanation appears plausible but our analysis shows that the connections are more involved and that the "concept hypothesis" does not hold. In our contribution we analyze several properties that in fact affect the retrieval performance. Moreover, we introduce a formalization of ESA, which reveals its close connection to existing retrieval models.
Modeling facial expressions and peripheral physiological signals to predict topical relevance
Authors: Ioannis Arapakis, Ioannis Konstas, Joemon M. Jose, Ioannis Kompatsiaris
Venue: SIGIR
Year: 2009

By analyzing explicit & implicit feedback information retrieval systems can determine topical relevance and tailor search criteria to the user's needs. In this paper we investigate whether it is possible to infer what is relevant by observing user affective behaviour. The sensory data employed range between facial expressions and peripheral physiological signals. We extract a set of features from the signals and analyze the data using classification methods, such as SVM and KNN. The results of our initial evaluation indicate that prediction of relevance is possible, to a certain extent, and implicit feedback models can benefit from taking into account user affective behavior.
Has adhoc retrieval improved since 1994?
Authors: Timothy G. Armstrong, Alistair Moffat, William Webber, Justin Zobel
Venue: SIGIR
Year: 2009

Evaluation forums such as TREC allow systematic measurement and comparison of information retrieval techniques. The goal is consistent improvement, based on reliable comparison of the effectiveness of different approaches and systems. In this paper we report experiments to determine whether this goal has been achieved. We ran five publicly available search systems, in a total of seventeen different configurations, against nine TREC adhoc-style collections, spanning 1994 to 2005. These runsets were then used as a benchmark for reassessing the relative effectiveness of the original TREC runs for those collections. Surprisingly, there appears to have been no overall improvement in effectiveness for either median or top-end TREC submissions, even after allowing for several possible confounds. We therefore question whether the effectiveness of adhoc information retrieval has improved over the past decade and a half.
Search engine predilection towards news media providers
Authors: Leif Azzopardi, Ciaran Owens
Venue: SIGIR
Year: 2009

In this poster paper, we present a preliminary study on the predilection of web search engines towards various online news media provider sites using an access based measure.
Efficiency trade-offs in two-tier web search systems
Authors: Ricardo Baeza-Yates, Vanessa Murdock, Claudia Hauff
Venue: SIGIR
Year: 2009

Search engines rely on searching multiple partitioned corpora to return results to users in a reasonable amount of time. In this paper we analyze the standard two-tier architecture for Web search with the difference that the corpus to be searched for a given query is predicted in advance. We show that any predictor better than random yields time savings, but this decrease in the processing time yields an increase in the infrastructure cost. We provide an analysis and investigate this trade-off in the context of two different scenarios on real-world data. We demonstrate that in general the decrease in answer time is justified by a small increase in infrastructure cost.
Undergraduates' evaluations of assigned search topics
Authors: Earl W. Bailey, Diane Kelly, Karl Gyllstrom
Venue: SIGIR
Year: 2009

This paper evaluates undergraduate students' knowledge, interests and experiences with 20 topics from the TREC Robust Track collection. The goal is to characterize these topics along several dimensions to help researchers make more informed decisions about which topics are most appropriate to use in experimental IIR evaluations with undergraduate student subjects.
Learning to rank for quantity consensus queries
Authors: Somnath Banerjee, Soumen Chakrabarti, Ganesh Ramakrishnan
Venue: SIGIR
Year: 2009

Web search is increasingly exploiting named entities like persons, places, businesses, addresses and dates. Entity ranking is also of current interest at INEX and TREC. Numerical quantities are an important class of entities, especially in queries about prices and features related to products, services and travel. We introduce Quantity Consensus Queries (QCQs), where each answer is a tight quantity interval distilled from evidence of relevance in thousands of snippets. Entity search and factoid question answering have benefited from aggregating evidence from multiple promising snippets, but these do not readily apply to quantities. Here we propose two new algorithms that learn to aggregate information from multiple snippets. We show that typical signals used in entity ranking, like rarity of query words and their lexical proximity to candidate quantities, are very noisy. Our algorithms learn to score and rankquantity intervals directly, combining snippet quantity and snippet text information. We report on experiments using hundreds of QCQs with ground truth taken from TREC QA, Wikipedia Infoboxes, and other sources, leading to tens of thousands of candidate snippets and quantities. Our algorithms yield about 20% better MAP and NDCG compared to the best-known collective rankers, and are 35% better than scoring snippets independent of each other.
From networks to human behavior
Authors: Albert-László Barabási
Venue: SIGIR
Year: 2009

Highly interconnected networks with amazingly complex topology describe systems as diverse as the World Wide Web, our cells, social systems or the economy. Recent studies indicate that these networks are the result of self-organizing processes governed by simple but generic laws, resulting in architectural features that makes them much more similar to each other than one would have expected by chance. I will discuss the amazing order characterizing our interconnected world and its implications to network robustness and spreading processes. Finally, most of these networks are driven by the temporal patterns characterizing human activity. I will use communication and web browsing data to show that there is deep order in the temporal domain of human dynamics, and discuss the different ways to understand and model the emerging patterns.
Context transfer in search advertising
Authors: Hila Becker, Andrei Z. Broder, Evgeniy Gabrilovich, Vanja Josifovski, Bo Pang
Venue: SIGIR
Year: 2009

We define and study the process of context transfer in search advertising, which is the transition of a user from the context of Web search to the context of the landing page that follows an ad-click. We conclude that in the vast majority of cases, the user is shown one of three types of pages, which can be accurately distinguished using automatic text classification.
A study of inter-annotator agreement for opinion retrieval
Authors: Adam Bermingham, Alan F. Smeaton
Venue: SIGIR
Year: 2009

Evaluation of sentiment analysis, like large-scale IR evaluation, relies on the accuracy of human assessors to create judgments. Subjectivity in judgments is a problem for relevance assessment and even more so in the case of sentiment annotations. In this study we examine the degree to which assessors agree upon sentence-level sentiment annotation. We show that inter-assessor agreement is not contingent on document length or frequency of sentiment but correlates positively with automated opinion retrieval performance. We also examine the individual annotation categories to determine which categories pose most difficulty for annotators.