OpenRaiser/Intern-Atlas huggingface dataset:
When is query performance prediction effective?
Authors: Claudia Hauff, Leif Azzopardi
Venue: SIGIR
Year: 2009

The utility of Query Performance Prediction (QPP) methods is commonly evaluated by reporting correlation coefficients to denote how well the methods perform at predicting the retrieval performance of a set of queries. However, a quintessential question remains unexplored: how strong does the correlation need to be in order to realize an increase in retrieval performance? In this work, we address this question in the context of Selective Query Expansion (SQE) and perform a large-scale experiment. The results show that to consistently and predictably improve retrieval effectiveness in the ideal SQE setting, a Kendall's Tau correlation of tau>=0.5 is required, a threshold which most existing query performance prediction methods fail to reach.
Topic structure for information retrieval
Authors: Jiyin He
Venue: SIGIR
Year: 2009

In my research, I propose a coherence measure, with the goal of discovering and using topic structures within and between documents, of which I explore its extensions and applications in information retrieval.
Fitting score distribution for blog opinion retrieval
Authors: Ben He, Jie Peng, Iadh Ounis
Venue: SIGIR
Year: 2009

Current blog opinion retrieval approaches cannot be applied if the topic relevance and opinion score distributions by rank are dissimilar. This problem severely limits the feasibility of these approaches. We propose to tackle this problem by fitting the distribution of opinion scores, which replaces the original topic relevance score distribution with the simulated one. Our proposed score distribution fitting method markedly enhances the feasibility of a state-of-the-art dictionary-based opinion retrieval approach. Evaluation on a standard TREC blog test collection shows significant improvements over high quality topic relevance baselines.
Annotation of URLs: more than the sum of parts
Authors: Max Hinne, Wessel Kraaij, Stephan Raaijmakers, Suzan Verberne, Theo P. van der Weide, Maarten van der Heijden
Venue: SIGIR
Year: 2009

Recently a number of studies have demonstrated that search engine logfiles are an important resource to determine the relevance relation between URLs and query terms. We hypothesized that the queries associated with a URL could also be presented as useful URL metadata in a search engine result list, e.g. for helping to determine the semantic category of a URL. We evaluated this hypothesis by a classification experiment based on the DMOZ dataset. Our method can also annotate URLs that have no associated queries.
A classification-based approach to question answering in discussion boards
Authors: Liangjie Hong, Brian D. Davison
Venue: SIGIR
Year: 2009

Discussion boards and online forums are important platforms for people to share information. Users post questions or problems onto discussion boards and rely on others to provide possible solutions and such question-related content sometimes even dominates the whole discussion board. However, to retrieve this kind of information automatically and effectively is still a non-trivial task. In addition, the existence of other types of information (e.g., announcements, plans, elaborations, etc.) makes it difficult to assume that every thread in a discussion board is about a question. We consider the problems of identifying question-related threads and their potential answers as classification tasks. Experimental results across multiple datasets demonstrate that our method can significantly improve the performance in both question detection and answer finding subtasks. We also do a careful comparison of how different types of features contribute to the final result and show that non-content features play a key role in improving overall performance. Finally, we show that a ranking scheme based on our classification approach can yield much better performance than prior published methods.
A study on performance volatility in information retrieval
Authors: Mehdi Hosseini
Venue: SIGIR
Year: 2009

A common practice in comparative evaluation of information retrieval (IR) systems is to create a test collection comprising a set of topics (queries), a document corpus, and relevance judgments, and to monitor the performance of retrieval systems over such a collection. A typical evaluation of a system involves computing a performance metric, e.g., Average Precision (AP), for each topic and then using the average performance metric, e.g., Mean Average Precision (MAP) to express the overall system performance. However, averages do not capture all the important aspects of system performance, and used alone may not thoroughly express system effectiveness, i.e., average of performance can mask large variance in individual topic effectiveness. The author hypothesis is that, in addition to the average of overall performance, attention needs to be paid to how a system performance varies across topics. This variability can be measured by calculating the standard deviation (SD) of individual performance scores. We refer to this performance variation as Volatility.
A bayesian learning approach to promoting diversity in ranking for biomedical information retrieval
Authors: Xiangji Huang, Qinmin Hu
Venue: SIGIR
Year: 2009

In this paper, we propose a Bayesian learning approach to promoting diversity for information retrieval in biomedicine and a re-ranking model to improve retrieval performance in the biomedical domain. First, the re-ranking model computes the maximum posterior probability of the hidden property corresponding to each retrieved passage. Then it iteratively groups the passages into subsets according to their properties. Finally, these passages are re-ranked from the subsets as our output. There is no need for our proposed method to use any external biomedical resource. We evaluate our Bayesian learning approach by conducting extensive experiments on the TREC 2004-2007 Genomics data sets. The experimental results show the effectiveness of the proposed Bayesian learning approach for promoting diversity in ranking for biomedical information retrieval on four years TREC data sets.
The importance of manual assessment in link discovery
Authors: Darren Wei Che Huang, Andrew Trotman, Shlomo Geva
Venue: SIGIR
Year: 2009

Using a ground truth extracted from the Wikipedia, and a ground truth created through manual assessment, we show that the apparent performance advantage seen in machine learning approaches to link discovery are an artifact of trivial links that are actively rejected by manual assessors.
Using dynamic markov compression to detect vandalism in the wikipedia
Authors: Kelly Y. Itakura, Charles L. A. Clarke
Venue: SIGIR
Year: 2009

We apply the Dynamic Markov Compression model to detect spam edits in the Wikipedia. The method appears to outperform previous efforts based on compression models, providing performance comparable to methods based on manually constructed rules.
Cluster-based query expansion
Authors: Inna Gelfer Kalmanovich, Oren Kurland
Venue: SIGIR
Year: 2009

We demonstrate the merits of using document clusters that are created offline to improve the overall effectiveness and performance robustness of a state-of-the-art pseudo-feedback-based query expansion method -- the relevance model.
Exploiting temporal information in retrieval of archived documents
Authors: Nattiya Kanhabua
Venue: SIGIR
Year: 2009

In a text retrieval community, many researchers have shown a good quality of searching a current snapshot of the Web. However, only a small number have demonstrated a good quality of searching a long-term archival domain, where documents are preserved for a long time, i.e., ten years or more. In such a domain, a search application is not only applicable for archivists or historians, but also in a context of national library and enterprise search (searching document repositories, emails, etc.). In the rest of this paper, we will explain three problems of searching document archives and propose possible approaches to solve these problems. Our main research question is: How to improve the quality of search in a document archive using temporal information?
Using wikipedia categories for ad hoc search
Authors: Rianne Kaptein, Marijn Koolen, Jaap Kamps
Venue: SIGIR
Year: 2009

In this paper we explore the use of category information for ad hoc retrieval in Wikipedia. We show that techniques for entity ranking exploiting this category information can also be applied to ad hoc topics and lead to significant improvements. Automatically assigned target categories are good surrogates for manually assigned categories, which perform only slightly better.
Who said what to whom?: capturing the structure of debates
Authors: Rianne Kaptein, Maarten Marx, Jaap Kamps
Venue: SIGIR
Year: 2009

Transcripts of meetings are a document genre characterized by a complex narrative structure. The essence is not only what is said, but also by who and to whom. This paper investigates whether we can use semantic annotations like the speaker in order to capture this debate structure, as well as the related content of the debate. The structure is visualized in a graph, while the content is condensed into word clouds, that are created using a parsimonious language model. Evaluation shows that both tools adequately capture the structure and content of the debate at an aggregated level.
Towards methods for the collective gathering and quality control of relevance assessments
Authors: Gabriella Kazai, Natasa Milic-Frayling, Jamie Costello
Venue: SIGIR
Year: 2009

Growing interest in online collections of digital books and video content motivates the development and optimization of adequate retrieval systems. However, traditional methods for collecting relevance assessments to tune system performance are challenged by the nature of digital items in such collections, where assessors are faced with a considerable effort to review and assess content by extensive reading, browsing, and within-document searching. The extra strain is caused by the length and cohesion of the digital item and the dispersion of topics within it. We propose a method for the collective gathering of relevance assessments using a social game model to instigate participants' engagement. The game provides incentives for assessors to follow a predefined review procedure and makes provisions for the quality control of the collected relevance judgments. We discuss the approach in detail, and present the results of a pilot study conducted on a book corpus to validate the approach. Our analysis reveals intricate relationships between the affordances of the system, the incentives of the social game, and the behavior of the assessors. We show that the proposed game design achieves two designated goals: the incentive structure motivates endurance in assessors and the review process encourages truthful assessment.
Dynamicity vs. effectiveness: studying online clustering for scatter/gather
Authors: Weimao Ke, Cassidy R. Sugimoto, Javed Mostafa
Venue: SIGIR
Year: 2009

We proposed and implemented a novel clustering algorithm called LAIR2, which has constant running time average for on-the-fly Scatter/Gather browsing [4]. Our experiments showed that when running on a single processor, the LAIR2 on-line clustering algorithm was several hundred times faster than a parallel Buckshot algorithm running on multiple processors [11]. This paper reports on a study that examined the effectiveness of the LAIR2 algorithm in terms of clustering quality and its impact on retrieval performance. We conducted a user study on 24 subjects to evaluate on-the-fly LAIR2 clustering in Scatter/Gather search tasks by comparing its performance to the Buckshot algorithm, a classic method for Scatter/Gather browsing [4]. Results showed significant differences in terms of subjective perceptions of clustering quality. Subjects perceived that the LAIR2 algorithm produced significantly better quality clusters than the Buckshot method did. Subjects felt that it took less effort to complete the tasks with the LAIR2 system, which was more effective in helping them in the tasks. Interesting patterns also emerged from subjects' comments in the final open-ended questionnaire. We discuss implications and future research.
Blog distillation using random walks
Authors: Mostafa Keikha, Mark James Carman, Fabio Crestani
Venue: SIGIR
Year: 2009

This paper addresses the blog distillation problem. That is, given a user query find the blogs most related to the query topic. We model the blogosphere as a single graph that includes extra information besides the content of the posts. By performing a random walk on this graph we extract most relevant blogs for each query. Our experiments on the TREC'07 data set show 15% improvement in MAP and 8% improvement in [email protected] over the Language Modeling baseline.
Temporal collaborative filtering with adaptive neighbourhoods
Authors: Neal Lathia, Stephen Hailes, Licia Capra
Venue: SIGIR
Year: 2009

Collaborative Filtering aims to predict user tastes, by minimising the mean error produced when predicting hidden user ratings. The aim of a deployed recommender system is to iteratively predict users' preferences over a dynamic, growing dataset, and system administrators are confronted with the problem of having to continuously tune the parameters calibrating their CF algorithm. In this work, we formalise CF as a time-dependent, iterative prediction problem. We then perform a temporal analysis of the Netflix dataset, and evaluate the temporal performance of two CF algorithms. We show that, due to the dynamic nature of the data, certain prediction methods that improve prediction accuracy on the Netflix probe set do not show similar improvements over a set of iterative train-test experiments with growing data. We then address the problem of parameter selection and update, and propose a method to automatically assign and update per-user neighbourhood sizes that (on the temporal scale) outperforms setting global parameters.
An improved markov random field model for supporting verbose queries
Authors: Matthew Lease
Venue: SIGIR
Year: 2009

Recent work in supervised learning of term-based retrieval models has shown significantly improved accuracy can often be achieved via better model estimation. In this paper, we show retrieval accuracy with Metzler and Croft's Markov random field (MRF) approach can be similarly improved via supervised learning. While the original MRF method estimates a parameter for each of its three feature classes from data, parameters within each class are set via a uniform weighting scheme adopted from the standard unigram. We conjecture greater MRF retrieval accuracy should be possible by better estimating within-class parameters, particularly for verbose queries employing natural language terms. Retrieval experiments with these queries on three TREC document collections show our improved MRF consistently out-performs both the original MRF and supervised unigram baselines. Additional experiments using blind-feedback and evaluation with optimal weighting demonstrate both the immediate value and further potential of our method.
Tag-based object similarity computation using term space dimension reduction
Authors: Yong Ki Lee, Sung Jun Lee, Jonghun Park
Venue: SIGIR
Year: 2009

In this paper, we propose a novel approach for measuring similarity between web objects. Our similarity measure is defined based on the representation of a web object as a collection of tags. Precisely, we first construct a vector space in which multiple terms are mapped into a single dimension by using information available from Open Directory Project and Delicious.com. Then we position web objects in the vector space and apply the traditional cosine measure for similarity computation. We demonstrate that the proposed similarity computation method is able to overcome the limitation of traditional vector space approach while at the same time require less computational cost compares to LSI (Latent Semantic Indexing).
Finding advertising keywords on video scripts
Authors: Jung-Tae Lee, Hyungdong Lee, Hee-Seon Park, Young-In Song, Hae-Chang Rim
Venue: SIGIR
Year: 2009

A key to success to contextual in-video advertising is finding advertising keywords on video contents effectively, but there has been little literature in the area so far. This paper presents some preliminary results of our learning-based system that finds relevant advertising keywords on particular scene of video contents using their scripts. The system is trained with not only features proven useful in earlier studies but novel features that reflect the situation of a targeted scene. Experimental results show that the new features are potentially helpful for enhancing the accuracy of keyword extraction for contextual in-video advertising.
Comparing both relevance and robustness in selection of web ranking functions
Authors: Fan Li, Xin Li, Shihao Ji, Zhaohui Zheng
Venue: SIGIR
Year: 2009

In commercial search engines, a ranking function is selected for deployment mainly by comparing the relevance measurements over candidates. In this paper we suggest to select Web ranking functions according to both their relevance and robustness to the changes that may lead to relevance degradation over time. We argue that the ranking robustness can be effectively measured by taking into account the ranking score distribution across Web pages. We then improve NDCG with two new metrics and show their superiority in terms of stability to ranking score turbulence and stability in function selection.
A probabilistic topic-based ranking framework for location-sensitive domain information retrieval
Authors: Huajing Li, Zhisheng Li, Wang-Chien Lee, Dik Lun Lee
Venue: SIGIR
Year: 2009

It has been observed that many queries submitted to search engines are location-sensitive. Traditional search techniques fail to interpret the significance of such geographical clues and as such are unable to return highly relevant search results. Although there have been efforts in the literature to support location-aware information retrieval, critical challenges still remain in terms of search result quality and data scalability. In this paper, we propose an innovative probabilistic ranking framework for domain information retrieval where users are interested in a set of location-sensitive topics. Our proposed method recognizes the geographical distribution of topic influence in the process of ranking documents and models it accurately using probabilistic Gaussian Process classifiers. Additionally, we demonstrate the effectiveness of the proposed ranking framework by implementing it in a Web search service for NBA news. Extensive performance evaluation is performed on real Web document collections, which confirms that our proposed mechanism works significantly better (around 29.7% averagely using DCG20 measure) than other popular location-aware information retrieval techniques in ranking quality.
Knowledge transformation for cross-domain sentiment classification
Authors: Tao Li, Vikas Sindhwani, Chris H. Q. Ding, Yi Zhang
Venue: SIGIR
Year: 2009

With the explosion of user-generated web2.0 content in the form of blogs, wikis and discussion forums, the Internet has rapidly become a massive dynamic repository of public opinion on an unbounded range of topics. A key enabler of opinion extraction and summarization is sentiment classification: the task of automatically identifying whether a given piece of text expresses positive or negative opinion towards a topic of interest. Building high-quality sentiment classifiers using standard text categorization methods is challenging due to the lack of labeled data in a target domain. In this paper, we consider the problem of cross-domain sentiment analysis: can one, for instance, download rated movie reviews from rottentomatoes.com or IMBD discussion forums, learn linguistic expressions and sentiment-laden terms that generally characterize opinionated reviews and then successfully transfer this knowledge to the target domain, thereby building high-quality sentiment models without manual effort? We outline a novel sentiment transfer mechanism based on constrained non-negative matrix tri-factorizations of term-document matrices in the source and target domains. We report some preliminary results with this approach.
Simultaneously modeling semantics and structure of threaded discussions: a sparse coding approach and its applications
Authors: Chen Lin, Jiang-Ming Yang, Rui Cai, Xin-Jing Wang, Wei Wang, Lei Zhang
Venue: SIGIR
Year: 2009

The huge amount of knowledge in web communities has motivated the research interests in threaded discussions. The dynamic nature of threaded discussions poses lots of challenging problems for computer scientists. Although techniques such as semantic models and structural models have been shown to be useful in a number of areas, they are inefficient in understanding threaded discussions due to three reasons: (I) as most of users read existing messages before posting, posts in a discussion thread are temporally dependent on the previous ones; It causes the semantics and structure to be coupled with each other in threaded discussions; (II) in online discussion threads, there are a lot of junk posts which are useless and may disturb content analysis; and (III) it is very hard to judge the quality of a post. In this paper, we propose a sparse coding-based model named SMSS to Simultaneously Model Semantics and Structure of threaded discussions. The model projects each post into a topic space, and approximates each post by a linear combination of previous posts in the same discussion thread. Meanwhile, the model also imposes two sparse constraints to force a sparse post reconstruction in the topic space and a sparse post approximation from previous posts. The sparse properties effectively take into account the characteristics of threaded discussions. Towards the above three problems, we demonstrate the competency of our model in three applications: reconstructing reply structure of threaded discussions, identifying junk posts, and finding experts in a given board/sub-board in web communities. Experimental results show encouraging performance of the proposed SMSS model in all these applications.
Using computational community interest as an indicator for ranking
Authors: Xiaozhong Liu
Venue: SIGIR
Year: 2009

Ranking documents in response to users' information needs is a challenging task, due, in part, to the dynamic nature of users' interests with respect to a query. I hypothesize that the interests of a given user are similar to the interests of the broader community of which he is a part and propose an innovative method that uses social media to characterize the interests of the community and use this characterization to improve future rankings. By generating a community interest vector (CIV) for a given query, we use community interest to alter the ranking score of individual documents retrieved by the query. The CIV is based on a continuously updated set of recent (daily or past few hours) user-oriented text data. The user-oriented data can be user blogs or user comment tagged news. Preliminary evaluation shows that the new ranking method significantly improves ranking performance.
Personalizing information retrieval using task features, topic knowledge, and task product
Authors: Jingjing Liu
Venue: SIGIR
Year: 2009

Personalization of information retrieval tailors search towards individual users to meet their particular information needs. Personalization systems obtain additional information about users and their contexts beyond the queries they submit to the systems, and use this information to bring the desired documents to top ranks. The additional information can come from various sources: user preferences, user behaviors, contexts, etc. [1] To avoid users taking extra effort in providing explicit preferences, most personalization approaches have adopted an implicit strategy to obtain users' interests from their behaviors and/or contexts, such as query history, browsing history, and so on.