OpenRaiser/Intern-Atlas huggingface dataset:
Re-ranking search results using document-passage graphs
Authors: Michael Bendersky, Oren Kurland
Venue: SIGIR
Year: 2008

We present a novel passage-based approach to re-ranking documents in an initially retrieved list so as to improve precision at top ranks. While most work on passage-based document retrieval ranks a document based on the query similarity of its constituent passages, our approach leverages information about the centrality of the document passages with respect to the initial document list. Passage centrality is induced over a bipartite document-passage graph, wherein edge weights represent document-passage similarities. Empirical evaluation shows that our approach yields effective re-ranking performance. Furthermore, the performance is superior to that of previously proposed passage-based document ranking methods.
Topic based language models for OCR correction
Authors: Anurag Bhardwaj, Faisal Farooq, Huaigu Cao, Venu Govindaraju
Venue: SIGIR
Year: 2008

Despite several decades of research in document analysis, recognition of unconstrained handwritten documents is still considered a challenging task. Previous research in this area has shown that word recognizers produce reasonably clean output when used with a restricted lexicon. But in absence of such a restricted lexicon, the output of an unconstrained handwritten word recognizer is noisy. The objective of this research is to process noisy recognizer output and eliminate spurious recognition choices using a topic based language model. We construct a topic based language model for every document using a training data which is manually categorized. A topic categorization sub-system based on Maximum Entropy model is also trained which is used to generate the topic distribution of a test document. A given test word image is processed by the recognizer and its word recognition likelihood is refined by incorporating topic distribution of the document and topic based language model probability. The proposed method is evaluated on a publicly available IAM dataset and experimental results show significant improvement in the word recognition accuracy from 32% to 40% over a test set consisting of 4033 word images extracted from 70 handwritten document images.
Talking the talk vs. walking the walk: salience of information needs in querying vs. browsing
Authors: Mikhail Bilenko, Ryen W. White, Matthew Richardson, G. Craig Murray
Venue: SIGIR
Year: 2008

Traditional information retrieval models assume that users express their information needs via text queries (i.e., their "talk"). In this poster, we consider Web browsing behavior outside of interactions with retrieval systems (i.e., users' "walk") as an alternative source of signal describing users' information needs, and compare it to the query-expressed information needs on a large dataset. Our findings demonstrate that information needs expressed in different behavior modalities are largely non-overlapping, and that past behavior in each modality is the most accurate predictor of future behavior in that modality. Results also show that browsing data provides a stronger source of signal than search queries due to its greater volume, which explains previous work that has found implicit behavioral data to be a valuable source of information for user modeling and personalization.
Reorganizing compressed text
Authors: Nieves R. Brisaboa, Antonio Fariña, Susana Ladra, Gonzalo Navarro
Venue: SIGIR
Year: 2008

Recent research has demonstrated beyond doubts the benefits of compressing natural language texts using word-based statistical semistatic compression. Not only it achieves extremely competitive compression rates, but also direct search on the compressed text can be carried out faster than on the original text; indexing based on inverted lists benefits from compression as well.Such compression methods assign a variable-length codeword to each different text word. Some coding methods (Plain Huffman and Restricted Prefix Byte Codes) do not clearly mark codeword boundaries, and hence cannot be accessed at random positions nor searched with the fastest text search algorithms. Other coding methods (Tagged Huffman, End-Tagged Dense Code, or (s, c)-Dense Code) do mark codeword boundaries, achieving a self-synchronization property that enables fast search and random access, in exchange for some loss in compression effectiveness.In this paper, we show that by just performing a simple reordering of the target symbols in the compressed text (more precisely, reorganizing the bytes into a wavelet-treelike shape) and using little additional space, searching capabilities are greatly improved without a drastic impact in compression and decompression times. With this approach, all the codes achieve synchronism and can be searched fast and accessed at arbitrary points. Moreover, the reordered compressed text becomes an implicitly indexed representation of the text, which can be searched for words in time independent of the text length. That is, we achieve not only fast sequential search time, but indexed search time, for almost no extra space cost.We experiment with three well-known word-based compression techniques with different characteristics (Plain Huffman, End-Tagged Dense Code and Restricted Prefix Byte Codes), and show the searching capabilities achieved by reordering the compressed representation on several corpora. We show that the reordered versions are not only much more efficient than their classical counterparts, but also more efficient than explicit inverted indexes built on the collection, when using the same amount of space.
Proximity-aware scoring for XML retrieval
Authors: Andreas Broschart, Ralf Schenkel
Venue: SIGIR
Year: 2008

Proximity-aware scoring functions lead to significant effectiveness improvements for text retrieval. For XML IR, we can sometimes enhance the retrieval quality by exploiting knowledge about the document structure combined with established text IR methods. This paper introduces modified proximity scores that take the document structure into account and demonstrates the effect for the INEX benchmark.
Towards personalized distributed information retrieval
Authors: Mark James Carman, Fabio Crestani
Venue: SIGIR
Year: 2008

Our aim is to investigate if and how the performance of Distributed Information Retrieval (DIR) systems can be improved through personalization. Toward this aim we are building a testbed of document collections and corresponding personalized relevance judgments. In this paper we discuss our intended approach for personalizing the three different phases of the DIR process. We also describe the test collection we are building and discuss our methodology for evaluating personalized DIR using relevance information taken from social bookmarking data.
Evaluation measures for preference judgments
Authors: Ben Carterette, Paul N. Bennett
Venue: SIGIR
Year: 2008

There has been recent interest in collecting user or assessor preferences, rather than absolute judgments of relevance, for the evaluation or learning of ranking algorithms. Since measures like precision, recall, and DCG are defined over absolute judgments, evaluation over preferences will require new evaluation measures that explicitly model them. We describe a class of such measures and compare absolute and preference measures over a large TREC collection.
Evaluation over thousands of queries
Authors: Ben Carterette, Virgiliu Pavlu, Evangelos Kanoulas, Javed A. Aslam, James Allan
Venue: SIGIR
Year: 2008

Information retrieval evaluation has typically been performed over several dozen queries, each judged to near-completeness. There has been a great deal of recent work on evaluation over much smaller judgment sets: how to select the best set of documents to judge and how to estimate evaluation measures when few judgments are available. In light of this, it should be possible to evaluate over many more queries without much more total judging effort. The Million Query Track at TREC 2007 used two document selection algorithms to acquire relevance judgments for more than 1,800 queries. We present results of the track, along with deeper analysis: investigating tradeoffs between the number of queries and number of judgments shows that, up to a point, evaluation over more queries with fewer judgments is more cost-effective and as reliable as fewer queries with more judgments. Total assessor effort can be reduced by 95% with no appreciable increase in evaluation errors.
Information retrieval on bug locations by learning co-located bug report clusters
Authors: Ing-Xiang Chen, Hojun Jaygarl, Cheng-Zen Yang, Ping-Jung Wu
Venue: SIGIR
Year: 2008

Bug locating usually involves intensive search activities and incurs unpredictable cost of labor and time. An issue of information retrieval on bug locations is particularly addressed to facilitate identifying bugs from software code. In this paper, a novel bug retrieval approach with co-location shrinkage (CS) is proposed. The proposed approach has been implemented in open-source software projects collected from real-world repositories, and consistently improves the retrieval accuracy of a state-of-the-art Support Vector Machine (SVM) model.
Task detection for activity-based desktop search
Authors: Sergey Chernov
Venue: SIGIR
Year: 2008

The desktop search tools provide powerful query capabilities and result presentation techniques. However, they do not take the user context into account. We propose to exploit collected information about user activities with desktop files and applications for activity-based desktop search. When I prepare for a project review and type in a search box the name of a colleague, I expect to find her last deliverable draft, but not her email with a paper review or our joint conference presentation. Ideally, the desktop search system should be able to infer my current task from the logs of my previous activities and present task-specific search results.
Novelty and diversity in information retrieval evaluation
Authors: Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, Ian MacKinnon
Venue: SIGIR
Year: 2008

Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.
Detecting synonyms in social tagging systems to improve content retrieval
Authors: Maarten Clements, Arjen P. de Vries, Marcel J. T. Reinders
Venue: SIGIR
Year: 2008

Collaborative tagging used in online social content systems is naturally characterized by many synonyms, causing low precision retrieval. We propose a mechanism based on user preference profiles to identify synonyms that can be used to retrieve more relevant documents by expanding the user's query. Using a popular online book catalog we discuss the effectiveness of our method over usual similarity based expansion methods.
A scalable assistant librarian: hierarchical subject classification of books
Authors: Steven P. Crain, Jian Huang, Hongyuan Zha
Venue: SIGIR
Year: 2008

In this paper, we discuss our work in progress towards a scalable hierarchical classification system for books using the Library of Congress subject hierarchy. We examine the characteristics of this domain which make the problem very challenging, and we look at several appropriate performance measurements. We show that both Hieron and Hierarchical Support Vector Machines perform moderately well.
Social recommendations at work
Authors: Tom Crecelius, Mouna Kacimi, Sebastian Michel, Thomas Neumann, Josiane Xavier Parreira, Ralf Schenkel, Gerhard Weikum
Venue: SIGIR
Year: 2008

Online communities have become popular for publishing and searching content, and also for connecting to other users. User-generated content includes, for example, personal blogs, bookmarks, and digital photos. Items can be annotated and rated by different users, and users can connect to others that are usually friends and/or share common interests.
A flexible extension of XPath to improve XML querying
Authors: Ernesto Damiani, Stefania Marrara, Gabriella Pasi
Venue: SIGIR
Year: 2008

This work presents a flexible XML selection language, FleXPath which allows the formulation of flexible constraints on both structure and content of XML documents. Some experimental results, obtained with a preliminary prototype, are described in order to show that the idea promises good results.
A comparative study of statistical features of language in blogs-vs-splogs
Authors: Soumya Datta, Sudeshna Sarkar
Venue: SIGIR
Year: 2008

Language usage in Blogs deviate from the language used in traditional corpora largely due to the noise from various causes like spelling errors, grammatical irregularity, overuse of abbreviations and symbolic characters like emoticons. Spam Blogs or Splogs comprise the subset of blogs, which are usually written to target specific audience for marketing promotions and are mostly generated by software that readily imitates Zipfian distribution of words. Therefore it becomes a difficult task to separate splogs from non-splogs using only frequentist distribution of unigrams. In this detailed comparative study we present and highlight several additional statistical features of language, which are hard to imitate and serve as good discriminator between splogs and blogs.
A method for transferring retrieval scores between collections with non-overlapping vocabularies
Authors: Fernando D. Diaz
Venue: SIGIR
Year: 2008

We present a method for projecting retrieval scores across two corpora with a shared, parallel corpus.
Theoretical bounds on and empirical robustness of score regularization to different similarity measures
Authors: Fernando D. Diaz
Venue: SIGIR
Year: 2008

We present theoretical bounds and empirical robustness of score regularization given changes in the similarity measure.
Posterior probabilistic clustering using NMF
Authors: Chris H. Q. Ding, Tao Li, Dijun Luo, Wei Peng
Venue: SIGIR
Year: 2008

We introduce the posterior probabilistic clustering (PPC), which provides a rigorous posterior probability interpretation for Nonnegative Matrix Factorization (NMF) and removes the uncertainty in clustering assignment. Furthermore, PPC is closely related to probabilistic latent semantic indexing (PLSI).
Learning to rank with partially-labeled data
Authors: Kevin Duh, Katrin Kirchhoff
Venue: SIGIR
Year: 2008

Ranking algorithms, whose goal is to appropriately order a set of objects/documents, are an important component of information retrieval systems. Previous work on ranking algorithms has focused on cases where only labeled data is available for training (i.e. supervised learning). In this paper, we consider the question whether unlabeled (test) data can be exploited to improve ranking performance. We present a framework for transductive learning of ranking functions and show that the answer is affirmative. Our framework is based on generating better features from the test data (via KernelPCA) and incorporating such features via Boosting, thus learning different ranking functions adapted to the individual test queries. We evaluate this method on the LETOR (TREC, OHSUMED) dataset and demonstrate significant improvements.
Personal vs non-personal blogs: initial classification experiments
Authors: Erik Elgersma, Maarten de Rijke
Venue: SIGIR
Year: 2008

We address the task of separating personal from non-personal blogs, and report on a set of baseline experiments where we compare the performance on a small set of features across a set of five classifiers. We show that with a limited set of features a performance of up to 90% can be obtained.
Search effectiveness with a breadth-first crawl
Authors: Dennis Fetterly, Nick Craswell, Vishwa Vinay
Venue: SIGIR
Year: 2008

Previous scalability experiments found that early precision improves as collection size increases. However, that was under the assumption that a collection's documents are all sampled with uniform probability from the same population. We contrast this to a large breadth-first web crawl, an important scenario in real-world Web search, where the early documents have quite different characteristics from the later documents.
Cross-lingual search over 22 european languages
Authors: Blaz Fortuna, Jan Rupnik, Bostjan Pajntar, Marko Grobelnik, Dunja Mladenic
Venue: SIGIR
Year: 2008

In this paper we present a system for cross-lingual information retrieval, which can handle tens of languages and millions of documents. Functioning of the system is demonstrated on corpus of European Legislation (22 languages, more than 400,000 documents per language). The system uses an interactive web-interface, which can take advantage of a predefined thesaurus allowing the user to dynamically re-rank the retrieval results based on the mapping onto a predefined thesaurus.
Summarization of compressed text images: an experience on Indic script documents
Authors: Utpal Garain
Venue: SIGIR
Year: 2008

Automatic summarization of JBIG2 coded textual images is discussed. Compressed images are partially decompressed to compute relevant features. The feature extraction method is free from using any character recognition module. Summary sentences are ranked. Experiment considers documents in Indic scripts that lack in having any efficient OCR systems. Script independent aspect of the approach is highlighted through use of two most popular Indic scripts. Sentence selection efficiency of about 61% is achieved when judged against man-made summarization. A nonparametric (distribution-free) rank statistic shows a correlation coefficient of 0.33 as a measure of the (minimum) strength of the associations between sentence ranking by machine and human.
On document splitting in passage detection
Authors: Nazli Goharian, Saket S. R. Mengle
Venue: SIGIR
Year: 2008

Passages can be hidden within a text to circumvent their disallowed transfer. Such release of compartmentalized information is of concern to all corporate and governmental organization. We explore the methodology to detect such hidden passages within a document. A document is divided into passages using various document splitting techniques, and a text classifier is used to categorize such passages. We present a novel document splitting technique called dynamic windowing, which significantly improves precision, recall and F1 measure.
Learning to rank with SoftRank and Gaussian processes
Authors: John Guiver, Edward Lloyd Snelson
Venue: SIGIR
Year: 2008

In this paper we address the issue of learning to rank for document retrieval using Thurstonian models based on sparse Gaussian processes. Thurstonian models represent each document for a given query as a probability distribution in a score space; these distributions over scores naturally give rise to distributions over document rankings. However, in general we do not have observed rankings with which to train the model; instead, each document in the training set is judged to have a particular relevance level: for example "Bad", "Fair", "Good", or "Excellent". The performance of the model is then evaluated using information retrieval (IR) metrics such as Normalised Discounted Cumulative Gain (NDCG). Recently Taylor et al. presented a method called SoftRank which allows the direct gradient optimisation of a smoothed version of NDCG using a Thurstonian model. In this approach, document scores are represented by the outputs of a neural network, and score distributions are created artificially by adding random noise to the scores. The SoftRank mechanism is a general one; it can be applied to different IR metrics, and make use of different underlying models. In this paper we extend the SoftRank framework to make use of the score uncertainties which are naturally provided by a Gaussian process (GP), which is a probabilistic non-linear regression model. We further develop the model by using sparse Gaussian process techniques, which give improved performance and efficiency, and show competitive results against baseline methods when tested on the publicly available LETOR OHSUMED data set. We also explore how the available uncertainty information can be used in prediction and how it affects model performance.
A unified and discriminative model for query refinement
Authors: Jiafeng Guo, Gu Xu, Hang Li, Xueqi Cheng
Venue: SIGIR
Year: 2008

This paper addresses the issue of query refinement, which involves reformulating ill-formed search queries in order to enhance relevance of search results. Query refinement typically includes a number of tasks such as spelling error correction, word splitting, word merging, phrase segmentation, word stemming, and acronym expansion. In previous research, such tasks were addressed separately or through employing generative models. This paper proposes employing a unified and discriminative model for query refinement. Specifically, it proposes a Conditional Random Field (CRF) model suitable for the problem, referred to as Conditional Random Field for Query Refinement (CRF-QR). Given a sequence of query words, CRF-QR predicts a sequence of refined query words as well as corresponding refinement operations. In that sense, CRF-QR differs greatly from conventional CRF models. Two types of CRF-QR models, namely a basic model and an extended model are introduced. One merit of employing CRF-QR is that different refinement tasks can be performed simultaneously and thus the accuracy of refinement can be enhanced. Furthermore, the advantages of discriminative models over generative models can be fully leveraged. Experimental results demonstrate that CRF-QR can significantly outperform baseline methods. Furthermore, when CRF-QR is used in web search, a significant improvement of relevance can be obtained.
Some thoughts on failure analysis for noisy data
Authors: Donna Harman
Venue: SIGIR
Year: 2008

A critical piece of evaluation is doing a failure analysis of the experimental results. This is necessary to improve results, but equally important, it is necessary to understand where the problems REALLY are, as opposed to just following conventional wisdom. The talk will present some current failure analysis techniques and examine how these techniques could be extended to retrieval from noisy data.
Personalized active learning for collaborative filtering
Authors: Abhay Harpale, Yiming Yang
Venue: SIGIR
Year: 2008

Collaborative Filtering (CF) requires user-rated training examples for statistical inference about the preferences of new users. Active learning strategies identify the most informative set of training examples through minimum interactions with the users. Current active learning approaches in CF make an implicit and unrealistic assumption that a user can provide rating for any queried item. This paper introduces a new approach to the problem which does not make such an assumption. We personalize active learning for the user, and query for only those items which the user can provide rating for. We propose an extended form of Bayesian active learning and use the Aspect Model for CF to illustrate and examine the idea. A comparative evaluation of the new method and a well-established baseline method on benchmark datasets shows statistically significant improvements with our method over the performance of the baseline method that is representative for existing approaches which do not take personalization into account. 1.