OpenRaiser/Intern-Atlas huggingface dataset:
Leveraging knowledge across media for spammer detection in microblogging
Authors: Xia Hu, Jiliang Tang, Huan Liu
Venue: SIGIR
Year: 2014

While microblogging has emerged as an important information sharing and communication platform, it has also become a convenient venue for spammers to overwhelm other users with unwanted content. Currently, spammer detection in microblogging focuses on using social networking information, but little on content analysis due to the distinct nature of microblogging messages. First, label information is hard to obtain. Second, the texts in microblogging are short and noisy. As we know, spammer detection has been extensively studied for years in various media, e.g., emails, SMS and the web. Motivated by abundant resources available in the other media, we investigate whether we can take advantage of the existing resources for spammer detection in microblogging. While people accept that texts in microblogging are different from those in other media, there is no quantitative analysis to show how different they are. In this paper, we first perform a comprehensive linguistic study to compare spam across different media. Inspired by the findings, we present an optimization formulation that enables the design of spammer detection in microblogging using knowledge from external media. We conduct experiments on real-world Twitter datasets to verify (1) whether email, SMS and web spam resources help and (2) how different media help for spammer detection in microblogging.
The role of network distance in linkedin people search
Authors: Shih-Wen Huang, Daniel Tunkelang, Karrie Karahalios
Venue: SIGIR
Year: 2014

LinkedIn is the world's largest professional network, with over 300 million members. One of the primary activities on the site is people search, for which LinkedIn members are both the users and the corpus. This paper presents insights about people search behavior on LinkedIn, based on a log analysis and a user study. In particular, it examines the role that network distance plays in name searches and non-name searches. For name searches, users primarily click on only one of the results, and closer network distance leads to higher click-through rates. In contrast, for non-name searches, users are more likely to click on multiple results that are not in their existing connections, but with whom they have shared connections. The results show that, while network distance contributes significantly to LinkedIn search engagement in general, its role varies dramatically depending on the type of search query.
A tag-based personalized item recommendation system using tensor modeling and topic model approaches
Authors: Noor Ifada
Venue: SIGIR
Year: 2014

This research falls in the area of enhancing the quality of tag-based item recommendation systems. It aims to achieve this by employing a multi-dimensional user profile approach and by analyzing the semantic aspects of tags. Tag-based recommender systems have two characteristics that need to be carefully studied in order to build a reliable system. Firstly, the multi-dimensional correlation, called as tag assignment (user, item, tag), should be appropriately modelled in order to create the user profiles [1]. Secondly, the semantics behind the tags should be considered properly as the flexibility with their design can cause semantic problems such as synonymy and polysemy [2]. This research proposes to address these two challenges for building a tag-based item recommendation system by employing tensor modeling as the multi-dimensional user profile approach, and the topic model as the semantic analysis approach. The first objective is to optimize the tensor model reconstruction and to improve the model performance in generating quality recommendation. A novel Tensor-based Recommendation using Probabilistic Ranking (TRPR) method [3] has been developed. Results show this method to be scalable for large datasets and outperforming the benchmarking methods in terms of accuracy. The memory efficient loop implements the n-mode block-striped (matrix) product for tensor reconstruction as an approximation of the initial tensor. The probabilistic ranking calculates the probability of users to select candidate items using their tag preference list based on the entries generated from the reconstructed tensor. The second objective is to analyse the tag semantics and utilize the outcome in building the tensor model. This research proposes to investigate the problem using topic model approach to keep the tags nature as the "social vocabulary" [4]. For the tag assignment data, topics can be generated from the occurrences of tags given for an item. However there is only limited amount of tags available to represent items as collection of topics, since an item might have only been tagged by using several tags. Consequently, the generated topics might not able to represent the items appropriately. Furthermore, given that each tag can belong to any topics with various probability scores, the occurrence of tags cannot simply be mapped by the topics to build the tensor model. A standard weighting technique will not appropriately calculate the value of tagging activity since it will define the context of an item using a tag instead of a topic.
The search duel: a response to a strong ranker
Authors: Peter Izsak, Fiana Raiber, Oren Kurland, Moshe Tennenholtz
Venue: SIGIR
Year: 2014

How can a search engine with a relatively weak relevance ranking function compete with a search engine that has a much stronger ranking function? This dual challenge, which to the best of our knowledge has not been addressed in previous work, entails an interesting bi-modal utility function for the weak search engine. That is, the goal is to produce in response to a query a document result list whose effectiveness does not fall much behind that of the strong search engine; and, which is quite different than that of the strong engine. We present a per-query algorithmic approach that leverages fundamental retrieval principles such as pseudo-feedback-based relevance modeling. We demonstrate the merits of our approach using TREC data.
Extending test collection pools without manual runs
Authors: Gaya K. Jayasinghe, William Webber, Mark Sanderson, J. Shane Culpepper
Venue: SIGIR
Year: 2014

Information retrieval test collections traditionally use a combination of automatic and manual runs to create a pool of documents to be judged. The quality of the final judgments produced for a collection is a product of the variety across each of the runs submitted and the pool depth. In this work, we explore fully automated approaches to generating a pool. By combining a simple voting approach with machine learning from documents retrieved by automatic runs, we are able to identify a large portion of relevant documents that would normally only be found through manual runs. Our initial results are promising and can be extended in future studies to help test collection curators ensure proper judgment coverage is maintained across complete document collections.
Evaluating non-deterministic retrieval systems
Authors: Gaya K. Jayasinghe, William Webber, Mark Sanderson, Lasitha Sandamali Dharmasena, J. Shane Culpepper
Venue: SIGIR
Year: 2014

The use of sampling, randomized algorithms, or training based on the unpredictable inputs of users in Information Retrieval often leads to non-deterministic outputs. Evaluating the effectiveness of systems incorporating these methods can be challenging since each run may produce different effectiveness scores. Current IR evaluation techniques do not address this problem. Using the context of distributed information retrieval as a case study for our investigation, we propose a solution based on multivariate linear modeling. We show that the approach provides a consistent and reliable method to compare the effectiveness of non-deterministic IR algorithms, and explain how statistics can safely be used to show that two IR algorithms have equivalent effectiveness.
Necessary and frequent terms in queries
Authors: Jiepu Jiang, James Allan
Venue: SIGIR
Year: 2014

Vocabulary mismatch has long been recognized as one of the major issues affecting search effectiveness. Ineffective queries usually fail to incorporate important terms and/or incorrectly include inappropriate keywords. However, in this paper we show another cause of reduced search performance: sometimes users issue reasonable query terms, but systems cannot identify the correct properties of those terms and take advantages of the properties. Specifically, we study two distinct types of terms that exist in all search queries: (1) necessary terms, for which term occurrence alone is indicative of document relevance; and (2) frequent terms, for which the relative term frequency is indicative of document relevance within the set of documents where the term appears. We evaluate these two properties of query terms in a dataset. Results show that only 1/3 of the terms are both necessary and frequent, while another 1/3 only hold one of the properties and the final third do not hold any of the properties. However, existing retrieval models do not clearly distinguish terms with the two properties and consider them differently. We further show the great potential of improving retrieval models by treating terms with distinct properties differently.
Searching, browsing, and clicking in a search session: changes in user behavior by task and over time
Authors: Jiepu Jiang, Daqing He, James Allan
Venue: SIGIR
Year: 2014

There are many existing studies of user behavior in simple tasks (e.g., navigational and informational search) within a short duration of 1--2 queries. However, we know relatively little about user behavior, especially browsing and clicking behavior, for longer search session solving complex search tasks. In this paper, we characterize and compare user behavior in relatively long search sessions (10 minutes; about 5 queries) for search tasks of four different types. The tasks differ in two dimensions: (1) the user is locating facts or is pursuing intellectual understanding of a topic; (2) the user has a specific task goal or has an ill-defined and undeveloped goal. We analyze how search behavior as well as browsing and clicking patterns change during a search session in these different tasks. Our results indicate that user behavior in the four types of tasks differ in various aspects, including search activeness, browsing style, clicking strategy, and query reformulation. As a search session progresses, we note that users shift their interests to focus less on the top results but more on results ranked at lower positions in browsing. We also found that results eventually become less and less attractive for the users. The reasons vary and include downgraded search performance of query, decreased novelty of search results, and decaying persistence of users in browsing. Our study highlights the lack of long session support in existing search engines and suggests different strategies of supporting longer sessions according to different task types.
Enhancing sketch-based sport video retrieval by suggesting relevant motion paths
Authors: Ihab Al Kabary, Heiko Schuldt
Venue: SIGIR
Year: 2014

Searching for scenes in team sport videos is a task that recurs very often in game analysis and other related activities performed by coaches. In most cases, queries are formulated on the basis of specific motion characteristics the user remembers from the video. Providing sketching interfaces for graphically specifying query input is thus a very natural user interaction for a retrieval application. However, the quality of the query (the sketch) heavily depends on the memory of the user and her ability to accurately formulate the intended search query by transforming this 3D memory of the known item(s) into a 2D sketch query. In this paper, we present an auto-suggest search feature that harnesses spatiotemporal data of team sport videos to suggest potential directions containing relevant data during the formulation of a sketch-based motion query. Users can intuitively select the direction of the desired motion query on-the-fly using the displayed visual clues, thus relaxing the need for relying heavily on memory to formulate the query. At the same time, this significantly enhances the accuracy of the results and the speed at which they appear. A first evaluation has shown the effectiveness and efficiency of our approach.
Skewed partial bitvectors for list intersection
Authors: Andrew Kane, Frank Wm. Tompa
Venue: SIGIR
Year: 2014

This paper examines the space-time performance of in-memory conjunctive list intersection algorithms, as used in search engines, where integers represent document identifiers. We demonstrate that the combination of bitvectors, large skips, delta compressed lists and URL ordering produces superior results to using skips or bitvectors alone. We define semi-bitvectors, a new partial bitvector data structure that stores the front of the list using a bitvector and the remainder using skips and delta compression. To make it particularly effective, we propose that documents be ordered so as to skew the postings lists to have dense regions at the front. This can be accomplished by grouping documents by their size in a descending manner and then reordering within each group using URL ordering. In each list, the division point between bitvector and delta compression can occur at any group boundary. We explore the performance of semi-bitvectors using the GOV2 dataset for various numbers of groups, resulting in significant space-time improvements over existing approaches. Semi-bitvectors do not directly support ranking. Indeed, bitvectors are not believed to be useful for ranking based search systems, because frequencies and offsets cannot be included in their structure. To refute this belief, we propose several approaches to improve the performance of ranking-based search systems using bitvectors, and leave their verification for future work. These proposals suggest that bitvectors, and more particularly semi-bitvectors, warrant closer examination by the research community.
Investigating users' query formulations for cognitive search intents
Authors: Makoto P. Kato, Takehiro Yamamoto, Hiroaki Ohshima, Katsumi Tanaka
Venue: SIGIR
Year: 2014

This study investigated query formulations by users with {\it Cognitive Search Intents} (CSIs), which are users' needs for the cognitive characteristics of documents to be retrieved, {\em e.g. comprehensibility, subjectivity, and concreteness. Our four main contributions are summarized as follows (i) we proposed an example-based method of specifying search intents to observe query formulations by users without biasing them by presenting a verbalized task description;(ii) we conducted a questionnaire-based user study and found that about half our subjects did not input any keywords representing CSIs, even though they were conscious of CSIs;(iii) our user study also revealed that over 50\% of subjects occasionally had experiences with searches with CSIs while our evaluations demonstrated that the performance of a current Web search engine was much lower when we not only considered users' topical search intents but also CSIs; and (iv) we demonstrated that a machine-learning-based query expansion could improve the performances for some types of CSIs.Our findings suggest users over-adapt to current Web search engines,and create opportunities to estimate CSIs with non-verbal user input.
Evaluating answer passages using summarization measures
Authors: Mostafa Keikha, Jae Hyun Park, W. Bruce Croft
Venue: SIGIR
Year: 2014

Passage-based retrieval models have been studied for some time and have been shown to have some benefits for document ranking. Finding passages that are not only topically relevant, but are also answers to the users' questions would have a significant impact in applications such as mobile search. To develop models for answer passage retrieval, we need to have appropriate test collections and evaluation measures. Making annotations at the passage level is, however, expensive and can have poor coverage. In this paper, we describe the advantages of document summarization measures for evaluating answer passage retrieval and show that these measures have high correlation with existing measures and human judgments.
Choices and constraints: research goals and approaches in information retrieval (part 1)
Authors: Diane Kelly, Filip Radlinski, Jaime Teevan
Venue: SIGIR
Year: 2014

All research projects begin with a goal, for instance to describe search behavior, to predict when a person will enter a second query, or to discover which IR system performs the best. Different research goals suggest different research approaches, ranging from field studies to lab studies to online experimentation. This tutorial will provide an overview of the different types of research goals, common evaluation approaches used to address each type, and the constraints each approach entails. Participants will come away with a broad perspective of research goals and approaches in IR, and an understanding of the benefits and limitations of these research approaches. The tutorial will take place in two independent, but interrelated parts, each focusing on a unique set of research approaches but with the same intended tutorial outcomes. These outcomes will be accomplished by deconstructing and analyzing our own published research papers, with further illustrations of each technique using the broader literature. By using our own research as anchors, we will provide insight about the research process, revealing the difficult choices and trade-offs researchers make when designing and conducting IR studies.
Choices and constraints: research goals and approaches in information retrieval (part 2)
Authors: Diane Kelly, Filip Radlinski, Jaime Teevan
Venue: SIGIR
Year: 2014

All research projects begin with a goal, for instance to describe search behavior, to predict when a person will enter a second query, or to discover which IR system performs the best. Different research goals suggest different research approaches, ranging from field studies to lab studies to online experimentation. This tutorial will provide an overview of the different types of research goals, common evaluation approaches used to address each type, and the constraints each approach entails. Participants will come away with a broad perspective of research goals and approaches in IR, and an understanding of the benefits and limitations of these research approaches. The tutorial will take place in two independent, but interrelated parts, each focusing on a unique set of research approaches but with the same intended tutorial outcomes. These outcomes will be accomplished by deconstructing and analyzing our own published research papers, with further illustrations of each technique using the broader literature. By using our own research as anchors, we will provide insight about the research process, revealing the difficult choices and trade-offs researchers make when designing and conducting IR studies.
Using score differences for search result diversification
Authors: Sadegh Kharazmi, Mark Sanderson, Falk Scholer, David Vallet
Venue: SIGIR
Year: 2014

We investigate the application of a light-weight approach to result list clustering for the purposes of diversifying search results. We introduce a novel post-retrieval approach, which is independent of external information or even the full-text content of retrieved documents; only the retrieval score of a document is used. Our experiments show that this novel approach is beneficial to effectiveness, albeit only on certain baseline systems. The fact that the method works indicates that the retrieval score is potentially exploitable in diversity.
Improving offline and online web search evaluation by modelling the user behaviour
Authors: Eugene Kharitonov
Venue: SIGIR
Year: 2014

Measurements are fundamental to any empirical science and, similarly, search evaluation is a vital part of information retrieval (IR). Evaluation ensures the progressive development of approaches, tools, and methods studied in this field. Apart from the scientific perspective, the evaluation approaches are also important from the practical perspective. Indeed, the evaluation experiments enable commercial search engines to make data-driven decisions while developing new features and working on the quality of the user experience. Thus, it is not surprising that evaluation has gained a huge attention from the research community and such an interest spans almost fifty years of research [3]. The Cranfield experiments [3] evolved into the widely used offline system evaluation approach. Despite its convenience and popularity, the offline evaluation approach has several limitations [8]. These limitations resulted in the development and recent growth in popularity of the online user-based evaluation approaches such as interleaving and A/B testing.
Diversifying query suggestions based on query documents
Authors: Youngho Kim, W. Bruce Croft
Venue: SIGIR
Year: 2014

Many domain-specific search tasks are initiated by document-length queries, e.g., patent invalidity search aims to find prior art related to a new (query) patent. We call this type of search Query Document Search. In this type of search, the initial query document is typically long and contains diverse aspects (or sub-topics). Users tend to issue many queries based on the initial document to retrieve relevant documents. To help users in this situation, we propose a method to suggest diverse queries that can cover multiple aspects of the query document. We first identify multiple query aspects and then provide diverse query suggestions that are effective for retrieving relevant documents as well being related to more query aspects. In the experiments, we demonstrate that our approach is effective in comparison to previous query suggestion methods.
Comparing client and server dwell time estimates for click-level satisfaction prediction
Authors: Youngho Kim, Ahmed Hassan Awadallah, Ryen W. White, Imed Zitouni
Venue: SIGIR
Year: 2014

Click dwell time is the amount of time that a user spends on a clicked search result. Many previous studies have shown that click dwell time is strongly correlated with result-level satisfaction and document relevance. Accurate estimates of dwell time are therefore important for applications such as search satisfaction prediction and result ranking. However, dwell time can be estimated in different ways according to the information available about the search process. For example, a result reached for the query [Garfield] may involve 145s of "server-side" dwell time (observable to the search engine) and 40s of "client-side" dwell time (observable from the browser). Since search engines can only observe server-side actions (i.e., activity on the search engine result page), server-side dwell times are estimated by measuring the time between a search result click and the next search event (click or query). Conversely, more detailed information about page dwell times can be obtained via client-side methods such as Web browser toolbars. The client-side information enables the estimation of more accurate dwell times by measuring the amount of time that a user spends on pages of interest (either the landing page, or pages on the full navigation trail). In this paper, we define three different dwell times, i.e., server-side, client-side, and trail dwell time, and examine their effectiveness for predicting click satisfaction. For this, we collect toolbar and search engine logs from real users, and provide an analysis of dwell times for improving prediction performance. Moreover, we show further improvements in predicting click-level satisfaction by combining dwell times with other query features (e.g., query clarity).