OpenRaiser/Intern-Atlas huggingface dataset:
Evaluating relevant in context: document retrieval with a twist
Authors: Jaap Kamps, Mounia Lalmas, Jovan Pehcevski
Venue: SIGIR
Year: 2007

The Relevant in Context retrieval task is document or article retrieval with a twist, where not only the relevant articles should be retrieved but also the relevant information within each article (captured by a set of XML elements) should be correctly identified. Our main research question is: how to evaluate the Relevant in Context task? We propose a generalized average precision measure that meets two main requirements: i) the score reflects the ranked list of articles inherent in the result list, and at the same time ii) the score also reflects how well the retrieved information per article (i.e., the set of elements) corresponds to the relevant information. The resulting measure was used at INEX 2006.
Management of keyword variation with frequency based generation of word forms in IR
Authors: Kimmo Kettunen
Venue: SIGIR
Year: 2007

This paper presents a new management method for morphological variation of keywords. The method is called FCG, Frequent Case Generation. It is based on the skewed distributions of word forms in natural languages and is suitable for languages that have either fair amount of morphological variation or are morphologically very rich. The proposed method has been evaluated so far with four languages, Finnish, Swedish, German and Russian, which show varying degrees of morphological complexity.
An effective snippet generation method using the pseudo relevance feedback technique
Authors: Youngjoong Ko, Hongkuk An, Jungyun Seo
Venue: SIGIR
Year: 2007

A (page or web) snippet is document excerpts allowing a user to understand if a document is indeed relevant without accessing it. This paper proposes an effective snippet generation method. The pseudo relevance feedback technique and text summarization techniques are applied to salient sentences extraction for generating good quality snippets. In the experimental results, the proposed method showed much better performance than other methods including Google and Naver.
Creativity support: information discovery and exploratory search
Authors: Eunyee Koh, Andruid Kerne, Rodney Hill
Venue: SIGIR
Year: 2007

We are developing support for creativity in learning through information discovery and exploratory search. Users engage in creative tasks, such as inventing new products and services. The system supports evolving information needs. It gathers and presents relevant information visually using images and text. Users are able to search, browse, and explore results from multiple queries and interact with information elements by manipulating design and expressing interest. A field study was conducted to evaluate the system in an undergraduate class. The results demonstrated the efficacy of our system for developing creative ideas. Exposure to diverse information in visual and interactive forms is shown to support students engaged in invention tasks.
Retrieval of discussions from enterprise mailing lists
Authors: Maheedhar Kolla, Olga Vechtomova
Venue: SIGIR
Year: 2007

Mailing list archives in an enterprise are a valuable source for employees to dig into the past proceedings of the organization that could be relevant to their present task. Going through the proceedings of discussions about certain topics might be cumbersome and regular search techniques might not work in this context due to the genre that the documents belong to. In this paper, we propose methods, based on theory of subjectivity, to retrieve email messages that could contain argumentative discussions about the topic that the user is interested in.
Interesting nuggets and their impact on definitional question answering
Authors: Kian-Wei Kor, Tat-Seng Chua
Venue: SIGIR
Year: 2007

Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets. This is insufficient as they do not address the novelty factor that a definitional nugget must also possess. This paper proposes to address the deficiency by building a "Human Interest Model" from external knowledge. It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic. We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
Babel: a machine transliteration workbench
Authors: A. Kumaran, Tobias Kellner
Venue: SIGIR
Year: 2007

Machine Transliteration deals with the conversion of text strings from one orthography to another, while preserving the phonetics of the strings in the two languages. Transliteration is an important problem in machine translation or cross-lingual information retrieval, as most proper names and generic iconic terms are out-of-vocabulary words, and therefore need to be transliterated. In this demo, we present Babel, a transliteration workbench, with generic statistical learning algorithms and a scripting engine to model the transliteration process. We demonstrate quick assembly of necessary components – algorithmic modules and training scripts – for systematic experimentation of transliteration tasks in a given pair of languages.
Using clustering to enhance text classification
Authors: Antonia Kyriakopoulou, Theodore Kalamboukis
Venue: SIGIR
Year: 2007

This paper addresses the problem of learning to classify textsby exploiting information derived from clustering both training and testing sets. The incorporation of knowledge resulting from clustering into the feature space representation of the texts is expected to boost the performance of a classifier. Experiments conducted on several widely used datasets demonstrate the effectiveness of the proposed algorithm especially for small training sets.
IDF revisited: a simple new derivation within the Robertson-Spärck Jones probabilistic model
Authors: Lillian Lee
Venue: SIGIR
Year: 2007

There have been a number of prior attempts to theoretically justify the effectiveness of the inverse document frequency (IDF). Those that take as their starting point Robertson and Sparck Jones's probabilistic model are based on strong or complex assumptions. We show that a more intuitively plausible assumption suffices. Moreover, the new assumption, while conceptually very simple, provides a solution to an estimation problem that had been deemed intractable by Robertson and Walker (1997).
Resource monitoring in information extraction
Authors: Jochen L. Leidner
Venue: SIGIR
Year: 2007

It is often argued that in information extraction (IE), certain machine learning (ML) approaches save development time over others, or that certain ML methods (e.g. Active Learning) require less training data than others, thus saving development cost. However, such development cost claims are not normally backed up by controlled studies which show that such development cost savings actually occur. This situation in Language Engineering is contrasted with Software Engineering in general, where a lot of studies investigating system development cost have been carried out. We argue for the need of controlled studies that measure actual system development time in language engineering. To this end, we carry out an experiment in resource monitoring for an IE task: three named entity taggers for the same "surprise" domain are developed in parallel, using competing methods. Their human development time is accounted forusing a logging facility.We report development cost results for parallel implementations of a named entity tagger and present a breakdown of the development time for the three alternative methods. We are not aware of detailed previous parallel studies that detail how system development time is spent when creating a named entity tagger.
Detecting, categorizing and clustering entity mentions in Chinese text
Authors: Wenjie Li, Donglei Qian, Qin Lu, Chunfa Yuan
Venue: SIGIR
Year: 2007

The work presented in this paper is motivated by the practical need for content extraction, and the available data source and evaluation benchmark from the ACE program. The Chinese Entity Detection and Recognition (EDR) task is of particular interest to us. This task presents us several language-independent and language-dependent challenges, e.g. rising from the complication of extraction targets and the problem of word segmentation, etc. In this paper, we propose a novel solution to alleviate the problems special in the task. Mention detection takes advantages of machine learning approaches and character-based models. It manipulates different types of entities being mentioned and different constitution units (i.e. extents and heads) separately. Mentions referring to the same entity are linked together by integrating most-specific-first and closest-first rule based pairwise clustering algorithms. Types of mentions and entities are determined by head-driven classification approaches. The implemented system achieves ACE value of 66.1 when evaluated on the EDR 2005 Chinese corpus, which has been one of the top-tier results. Alternative approaches to mention detection and clustering are also discussed and analyzed.
Applying ranking SVM in query relaxation
Authors: Ciya Liao, Thomas Chang
Venue: SIGIR
Year: 2007

We propose an approach QRRS (Query Relaxative Ranking SVM) that divides a ranking function into different relaxation steps, so that only cheap features are used in Ranking SVM of early steps for query efficiency. We show search quality in the approach is improved compared to conventional Ranking SVM.
High-dimensional visual vocabularies for image retrieval
Authors: João Magalhães, Stefan M. Rüger
Venue: SIGIR
Year: 2007

In this paper we formulate image retrieval by text query as a vector space classification problem. This is achieved by creating a high-dimensional visual vocabulary that represents the image documents in great detail. We show how the representation of these image documents enables the application of well known text retrieval techniques such as Rocchio tf-idf and naíve Bayes to the semantic image retrieval problem. We tested these methods on a Corel images subset and achieve state-of-the-art retrieval performance using the proposed methods.
Effects of highly agreed documents in relevancy prediction
Authors: Andrés R. Masegosa, Hideo Joho, Joemon M. Jose
Venue: SIGIR
Year: 2007

Finding significant contextual features is a challenging task in the development of interactive information retrieval (IR) systems. This paper investigated a simple method to facilitate such a task by looking at aggregated relevance judgements of retrieved documents. Our study suggested that the agreement on relevance judgements can indicate the effectiveness of retrieved documents as the source of significant features. The effect of highly agreed documents gives us practical implication for the design of adaptive search models in interactive IR systems.
A study of Poisson query generation model for information retrieval
Authors: Qiaozhu Mei, Hui Fang, ChengXiang Zhai
Venue: SIGIR
Year: 2007

Many variants of language models have been proposed for information retrieval. Most existing models are based on multinomial distribution and would score documents based on query likelihood computed based on a query generation probabilistic model. In this paper, we propose and study a new family of query generation models based on Poisson distribution. We show that while in their simplest forms, the new family of models and the existing multinomial models are equivalent. However, based on different smoothing methods, the two families of models behave differently. We show that the Poisson model has several advantages, including naturally accommodating per-term smoothing and modeling accurate background more efficiently. We present several variants of the new model corresponding to different smoothing methods, and evaluate them on four representative TREC test collections. The results show that while their basic models perform comparably, the Poisson model can out perform multinomial model with per-term smoothing. The performance can be further improved with two-stage smoothing.
VideoReach: an online video recommendation system
Authors: Tao Mei, Bo Yang, Xian-Sheng Hua, Linjun Yang, Shi-Qiang Yang, Shipeng Li
Venue: SIGIR
Year: 2007

This paper presents a novel online video recommendation system called VideoReach, which alleviates users' efforts on finding the most relevant videos according to current viewings without a sufficient collection of user profiles as required in traditional recommenders. In this system, video recommendation is formulated as finding a list of relevant videos in terms of multimodal relevance (i.e. textual, visual, and aural relevance) and user click-through. Since different videos have different intra-weights of relevance within an individual modality and inter-weights among different modalities, we adopt relevance feedback to automatically find optimal weights by user click-though, as well as an attention fusion function to fuse multimodal relevance. We use 20 clips as the representative test videos, which are searched by top 10 queries from more than 13k online videos, and report superior performance compared with an existing video site.
Hits hits TREC: exploring IR evaluation results with network analysis
Authors: Stefano Mizzaro, Stephen Robertson
Venue: SIGIR
Year: 2007

We propose a novel method of analysing data gathered fromTREC or similar information retrieval evaluation experiments. We define two normalized versions of average precision, that we use to construct a weighted bipartite graph of TREC systems and topics. We analyze the meaning of well known - and somewhat generalized - indicators fromsocial network analysis on the Systems-Topics graph. We apply this method to an analysis of TREC 8 data; amongthe results, we find that authority measures systems performance, that hubness of topics reveals that some topics are better than others at distinguishing more or less effective systems, that with current measures a system that wants to be effective in TREC needs to be effective on easy topics, and that by using different effectiveness measures this is no longer the case.
An analysis of peer-to-peer file-sharing system queries
Authors: Linh Thai Nguyen, Dongmei Jia, Wai Gen Yee, Ophir Frieder
Venue: SIGIR
Year: 2007

Many studies focus on the Web, but yet, few focus on peer-to-peer file-sharing system queries despite their massive scale in terms of Internet traffic. We analyzed several million queries collected on the Gnutella network and differentiated our findings from those of Web queries.
Ranking by community relevance
Authors: Lan Nie, Brian D. Davison, Baoning Wu
Venue: SIGIR
Year: 2007

A web page may be relevant to multiple topics; even when nominally on a single topic, the page may attract attention (and thus links) from multiple communities. Instead of indiscriminately summing the authority provided by all pages, we decompose a web page into separate subnodes with respect to each community pointing to it. By considering the relevance of these communities, we are able to better model the query-specific reputation for each potential result. We apply a total of 125 queries to the TREC .GOV dataset to demonstrate how the use of community relevance can improve ranking performance.
Winnowing wheat from the chaff: propagating trust to sift spam from the web
Authors: Lan Nie, Baoning Wu, Brian D. Davison
Venue: SIGIR
Year: 2007

The Web today includes many pages intended to deceive search engines, and attain an unwarranted result ranking. Since the links among web pages are used to calculate authority, ranking systems would benefit from knowing which pages contain content to be trusted and which do not. We propose and compare various trust propagation methods to estimate the trustworthiness of each page. We find that a non-trust-preserving propagation method is able to achieve close to a fifty percent improvement over TrustRank in separating spam from non-spam pages.
Focused ranking in a vertical search engine
Authors: Philip O&apos, Brien, Tony Abou-Assaleh
Venue: SIGIR
Year: 2007

Since the debut of PageRank and HITS, hyperlink-induced Web document ranking has come a long way. The Web has become increasingly vast and topically diverse. Such vastness has led many into the area of topic-sensitive ranking and its variants. We address the high dimensionality of the Web by providing tools for focused search. A focused search engine is one which seeks coverage over a subset of topics of the Web and presents users with relevant search results in a known domain. This demonstration will introduce readers to the GenieKnows.com Vertical Search Engine.
Document layout and color driven image retrieval
Authors: Pere Obrador
Venue: SIGIR
Year: 2007

This paper presents a contribution to image indexing applied to the document creation task. The presented method ranks a set of photographs based on how well they aesthetically work within a predefined document. Color harmony, document visual balance and image quality are taken into consideration. A user study conducted on people with a range of expertise in document creation helped gather the right visual features to consider by the algorithm. This shows some benefits for the traditional document creation task, as well as for the case of ever-changing web page banner colors and layout.
Improving text classification for oral history archives with temporal domain knowledge
Authors: J. Scott Olsson, Douglas W. Oard
Venue: SIGIR
Year: 2007

This paper describes two new techniques for increasing the accuracy oftopic label assignment to conversational speech from oral history interviews using supervised machine learning in conjunction with automatic speech recognition. The first, time-shifted classification, leverages local sequence information from the order in which the story is told. The second, temporal label weighting, takes the complementary perspective by using the position within an interview to bias label assignment probabilities. These methods, when used in combination, yield between 6% and 15% relative improvements in classification accuracy using a clipped R-precision measure that models the utility of label sets as segment summaries in interactive speech retrieval applications.
A multi-criteria content-based filtering system
Authors: Gabriella Pasi, Gloria Bordogna, Robert Villa
Venue: SIGIR
Year: 2007

In this paper we present a novel filtering system, based on a new model which reshapes the aims of content-based filtering. The filtering system has been developed within the EC project PENG, aimed at providing news professionals, such as journalists, with a system supporting both filtering and retrieval capabilities. In particular, we suggest that in tackling the problem of information overload, it is necessary for filtering systems to take into account multiple aspects of incoming documents in order to estimate their relevance to a user's profile, and in order to help users better understand documents, as distinct from solely attempting to either select relevant material from a stream, or block inappropriate material. Aiming to so this, a filtering model based on multiple criteria has been defined, based on the ideas gleamed in the project requirements stage. The filtering model is briefly described in this paper.
Context sensitive stemming for web search
Authors: Fuchun Peng, Nawaaz Ahmed, Xin Li, Yumao Lu
Venue: SIGIR
Year: 2007

Traditionally, stemming has been applied to Information Retrieval tasks by transforming words in documents to the their root form before indexing, and applying a similar transformation to query terms. Although it increases recall, this naive strategy does not work well for Web Search since it lowers precision and requires a significant amount of additional computation.
Incorporating term dependency in the dfr framework
Authors: Jie Peng, Craig Macdonald, Ben He, Vassilis Plachouras, Iadh Ounis
Venue: SIGIR
Year: 2007

Term dependency, or co-occurrence, has been studied in language modelling, for instance by Metzler & Croft who showed that retrieval performance could be significantlyenhanced using term dependency information. In this work, weshow how term dependency can be modelled within the Divergence From Randomness (DFR) framework. We evaluate our term dependency model on the two adhoc retrieval tasks using the TREC .GOV2 Terabyte collection. Furthermore, we examine the effect of varying the term dependency window size on the retrieval performance of the proposed model. Our experiments show that term dependency can indeed besuccessfully incorporated within the DFR framework.
Understanding the relationship of information need specificity to search query length
Authors: Nina Phan, Peter Bailey, Ross Wilkinson
Venue: SIGIR
Year: 2007

When searching, people's information needs flowthrough to expressing an information retrieval request posed to asearch engine. We hypothesise that the degree of specificity of anIR request might correspond to the length of a search query. Ourresults show a strong correlation between decreasing query lengthand increasing broadness or generality of the IR request. We foundan average cross-over point of specificity from broad to narrow of 3words in the query. These results have implications for searchengines in responding to queries of differing lengths.
An effective method for finding best entry points in semi-structured documents
Authors: Eugen Popovici, Pierre-Francois Marteau, Gildas Ménier
Venue: SIGIR
Year: 2007

Focused structured document retrieval employs the concept of best entry point (BEP), which is intended to provide optimal starting-point from which users can browse to relevant document components [4]. In this paper we describe and evaluate a method for finding BEPs in XML documents. Experiments conducted within the framework of INEX 2006 evaluation campaign on the Wikipedia XML collection [2] shown the effectiveness of the proposed approach.
Wikipedia in the pocket: indexing technology for near-duplicate detection and high similarity search
Authors: Martin Potthast
Venue: SIGIR
Year: 2007

We develop and implement a new indexing technology which allows us to use complete (and possibly very large) documents as queries, while having a retrieval performance comparable to a standard term query. Our approach aims at retrieval tasks such as near duplicate detection and high similarity search. To demonstrate the performance of our technology we have compiled the search index "Wikipedia in the Pocket", which contains about 2 million English and German Wikipedia articles.1 This index--along with a search interface--fits on a conventional CD (0.7 gigabyte). The ingredients of our indexing technology are similarity hashing and minimal perfect hashing.
Quantify query ambiguity using ODP metadata
Authors: Guang Qiu, Kangmiao Liu, Jiajun Bu, Chun Chen, Zhiming Kang
Venue: SIGIR
Year: 2007

Query ambiguity prevents existing retrieval systems from returning reasonable results for every query. As there is already lots of work done on resolving ambiguity, vague queries could be handled using corresponding approaches separately if they can be identified in advance. Quantification of the degree of (lack of) ambiguity laysthe groundwork for the identification. In this poster, we propose such a measure using query topics based on the topic structure selected from the Open Directory Project (ODP) taxonomy. We introduce clarity score to quantify the lack of ambiguity with respect to data sets constructed from the TREC collections and the rank correlation test results demonstrate a strong positive association between the clarity scores and retrieval precisions for queries.