OpenRaiser/Intern-Atlas huggingface dataset:
Adaptive near-duplicate detection via similarity learning
Authors: Hannaneh Hajishirzi, Wen-tau Yih, Aleksander Kolcz
Venue: Conference SIGIR 2010
Year: 2010

In this paper, we present a novel near-duplicate document detection method that can easily be tuned for a particular domain. Our method represents each document as a real-valued sparse k-gram vector, where the weights are learned to optimize for a specified similarity function, such as the cosine similarity or the Jaccard coefficient. Near-duplicate documents can be reliably detected through this improved similarity measure. In addition, these vectors can be mapped to a small number of hash-values as document signatures through the locality sensitive hashing scheme for efficient similarity computation. We demonstrate our approach in two target domains: Web news articles and email messages. Our method is not only more accurate than the commonly used methods such as Shingles and I-Match, but also shows consistent improvement across the domains, which is a desired property lacked by existing methods.
Improved robustness of signature-based near-replica detection via lexicon randomization
Authors: Aleksander Kolcz, Abdur Chowdhury, Joshua Alspector
Venue: Conference KDD 2004
Year: 2004

Detection of near duplicate documents is an important problem in many data mining and information filtering applications. When faced with massive quantities of data, traditional duplicate detection techniques relying on direct inter-document similarity computation (e.g., using the cosine measure) are often not feasible given the time and memory performance constraints. On the other hand, fingerprint-based methods, such as I-Match, are very attractive computationally but may be brittle with respect to small changes to document content. We focus on approaches to near-replica detection that are based upon large-collection statistics and present a general technique of increasing their robustness via multiple lexicon randomization. In experiments with large web-page and spam-email datasets the proposed method is shown to consistently outperform traditional I-Match, with the relative improvement in duplicate-document recall reaching as high as 40-60%. The large gains in detection accuracy are offset by only small increases in computational requirements.
Brute force and indexed approaches to pairwise document similarity comparisons with mapreduce
Authors: Jimmy Lin
Venue: Conference SIGIR 2009
Year: 2009

This paper explores the problem of computing pairwise similarity on document collections, focusing on the application of "more like this" queries in the life sciences domain. Three MapReduce algorithms are introduced: one based on brute force, a second where the problem is treated as large-scale ad hoc retrieval, and a third based on the Cartesian product of postings lists. Each algorithm supports one or more approximations that trade effectiveness for efficiency, the characteristics of which are studied experimentally. Results show that the brute force algorithm is the most efficient of the three when exact similarity is desired. However, the other two algorithms support approximations that yield large efficiency gains without significant loss of effectiveness.
Can social bookmarking improve web search?
Authors: Paul Heymann, Georgia Koutrika, Hector Garcia-Molina
Venue: Conference WSDM 2008
Year: 2008

Social bookmarking is a recent phenomenon which has the potential to give us a great deal of data about pages on the web. One major question is whether that data can be used to augment systems like web search. To answer this question, over the past year we have gathered what we believe to be the largest dataset from a social bookmarking site yet analyzed by academic researchers. Our dataset represents about forty million bookmarks from the social bookmarking site del.icio.us. We contribute a characterization of posts to del.icio. us: how many bookmarks exist (about 115 million), how fast is it growing, and how active are the URLs being posted about (quite active). We also contribute a characterization of tags used by bookmarkers. We found that certain tags tend to gravitate towards certain domains, and vice versa. We also found that tags occur in over 50 percent of the pages that they annotate, and in only 20 percent of cases do they not occur in the page text, backlink page text, or forward link page text of the pages they annotate. We conclude that social bookmarking can provide search data not currently provided by other sources, though it may currently lack the size and distribution of tags necessary to make a significant impact
Algorithmic mediation for collaborative exploratory search
Authors: Jeremy Pickens, Gene Golovchinsky, Chirag Shah, Pernilla Qvarfordt, Maribeth Back
Venue: Conference SIGIR 2008
Year: 2008

We describe a new approach to information retrieval: algorithmic mediation for intentional, synchronous collaborative exploratory search. Using our system, two or more users with a common information need search together, simultaneously. The collaborative system provides tools, user interfaces and, most importantly, algorithmically-mediated retrieval to focus, enhance and augment the team's search and communication activities. Collaborative search outperformed post hoc merging of similarly instrumented single user runs. Algorithmic mediation improved both collaborative search (allowing a team of searchers to find relevant information more efficiently and effectively), and exploratory search (allowing the searchers to find relevant information that cannot be found while working individually).
Exploiting query reformulations for web search result diversification
Authors: Rodrygo L. T. Santos, Craig Macdonald, Iadh Ounis
Venue: Conference WWW 2010
Year: 2010

When a Web user's underlying information need is not clearly specified from the initial query, an effective approach is to diversify the results retrieved for this query. In this paper, we introduce a novel probabilistic framework for Web search result diversification, which explicitly accounts for the various aspects associated to an underspecified query. In particular, we diversify a document ranking by estimating how well a given document satisfies each uncovered aspect and the extent to which different aspects are satisfied by the ranking as a whole. We thoroughly evaluate our framework in the context of the diversity task of the TREC 2009 Web track. Moreover, we exploit query reformulations provided by three major Web search engines (WSEs) as a means to uncover different query aspects. The results attest the effectiveness of our framework when compared to state-of-the-art diversification approaches in the literature. Additionally, by simulating an upper-bound query reformulation mechanism from official TREC data, we draw useful insights regarding the effectiveness of the query reformulations generated by the different WSEs in promoting diversity.
Inferring the impacts of social media on crowdfunding
Authors: Chun-Ta Lu, Sihong Xie, Xiangnan Kong, Philip S. Yu
Venue: Conference WSDM 2014
Year: 2014

Crowdfunding -- in which people can raise funds through collaborative contributions of general public (i.e., crowd) -- has emerged as a billion dollars business for supporting more than one million ventures. However, very few research works have examined the process of crowdfunding. In particular, none has studied how social networks help crowdfunding projects to succeed. To gain insights into the effects of social networks in crowdfunding, we analyze the hidden connections between the fundraising results of projects on crowdfunding websites and the corresponding promotion campaigns in social media. Our analysis considers the dynamics of crowdfunding from two aspects: how fundraising activities and promotional activities on social media simultaneously evolve over time, and how the promotion campaigns influence the final outcomes. From our investigation, we identify a number of important principles that provide a useful guide for devising effective campaigns. For example, we observe temporal distribution of customer interest, strong correlations between a crowdfunding project's early promotional activities and the final outcomes, and the importance of concurrent promotion from multiple sources. We then show that these discoveries can help predict several important quantities, including overall popularity and the success rate of the project. Finally, we show how to use these discoveries to help design crowdfunding sites.
Scaling RDF with time
Authors: Andrea Pugliese, Octavian Udrea, V. S. Subrahmanian
Venue: Conference WWW 2008
Year: 2008

The World Wide Web Consortium's RDF standard primarily consists of (subject, property, object) triples that specify the value that a given subject has for a given property. However, it is frequently the case that even for a fixed subject and property, the value varies with time. As a consequence, efforts have been made to annotate RDF triples with "valid time" intervals. However, to date, no proposals exist for efficient indexing of such temporal RDF databases. It is clearly beneficial to store RDF data in a relational DB - however, standard relational indexes are inadequately equipped to handle RDF's graph structure. In this paper, we propose the tGRIN index structure that builds a specialized index for temporal RDF that is physically stored in an RDBMS. Past efforts to store RDF in relational stores include Jena2 from HP, Sesame from OpenRDF.org, and 3store from the University of Southampton. We show that even when these efforts are augmented with well known temporal indexes like R+ trees, SR-trees, ST-index, and MAP21, the tGRIN index exhibits superior performance. In terms of index build time, tGRIN takes two thirds or less of the time used by any other system, and it uses a comparable amount of memory and less disk space than Jena, Sesame and 3store. More importantly, tGRIN can answer queries three to six times faster for average query graph patterns and five to ten times faster for complex queries than these systems.
The WWW (and an H) of Mobile Application Usage in the City: The What, Where, When, and How
Authors: Eduardo Graells-Garrido, Diego Caro, Omar Miranda, Rossano Schifanella, Oscar F. Peredo
Venue: Conference WWW 2018
Year: 2018

People fulfill their informational needs through smartphones, however, little is known regarding how the urban fabric and the activities that take place in it affect the usage of mobile applications. In this regard, starting from an anonymized dataset of Deep Packet Inspection (DPI) data from the largest telecommunications operator in Chile, we focus on the following questions: What are the most popular applications used in the city Where are they spatially clustered When does an application is more frequently used And How does the urban context and the mobility patterns relate to application usage As a result, we observed that specific applications present high spatial clustering, while the most popular services are geographically dispersed throughout the entire city. Clusters appear in places of high floating population; however, hotspots vary in space depending on the application. Interestingly, we found that commuting plays an important role, both in terms of rush hours and transportation infrastructure. We present a discussion on these results, focusing on how the physical space and the daily commuting routine affect the pattern of data consumption and represent an important aspect in mobile users behavioral studies.
A large-scale sentiment analysis for Yahoo! answers
Authors: Onur Kucuktunc, Berkant Barla Cambazoglu, Ingmar Weber, Hakan Ferhatosmanoglu
Venue: Conference WSDM 2012
Year: 2012

Sentiment extraction from online web documents has recently been an active research topic due to its potential use in commercial applications. By sentiment analysis, we refer to the problem of assigning a quantitative positive/negative mood to a short bit of text. Most studies in this area are limited to the identification of sentiments and do not investigate the interplay between sentiments and other factors. In this work, we use a sentiment extraction tool to investigate the influence of factors such as gender, age, education level, the topic at hand, or even the time of the day on sentiments in the context of a large online question answering site. We start our analysis by looking at direct correlations, e.g., we observe more positive sentiments on weekends, very neutral ones in the Science & Mathematics topic, a trend for younger people to express stronger sentiments, or people in military bases to ask the most neutral questions. We then extend this basic analysis by investigating how properties of the (asker, answerer) pair affect the sentiment present in the answer. Among other things, we observe a dependence on the pairing of some inferred attributes estimated by a user's ZIP code. We also show that the best answers differ in their sentiments from other answers, e.g., in the Business & Finance topic, best answers tend to have a more neutral sentiment than other answers. Finally, we report results for the task of predicting the attitude that a question will provoke in answers. We believe that understanding factors influencing the mood of users is not only interesting from a sociological point of view, but also has applications in advertising, recommendation, and search.
When Relevance Judgement is Happening?: An EEG-based Study
Authors: Marco Allegretti, Yashar Moshfeghi, Maria Hadjigeorgieva, Frank E. Pollick, Joemon M. Jose, Gabriella Pasi
Venue: Conference SIGIR 2015
Year: 2015

Relevance is a central notion in Information Retrieval, but it is considered to be a difficult concept to define. We analyse brain signals for the first 800 milliseconds (ms) of a relevance assessment process to answer the question "when relevance is happening in the brain?" with the belief that it will lead to better operational definitions of relevance. For this purpose, we devised a user study in which we captured the brain response of 20 participants. Using a 64-channel EEG device, we measured the electrophysiological activity of the brain while the subjects were in the phase of giving an explicit judgement about the relevance of presented images according to a given topic. Analyses were then performed over different time windows of the recorded EEG signals using repeated measures ANOVA. Data reveal significant variation between relevance and non-relevance within the EEG signals from the presentation of the image to 800 milliseconds afterwards. At an early stage these differences were located at frontal and posterior electrode sites. However, at later stages these differences were located in central, centro-parietal and centro-frontal areas.Our findings are an important step towards (i) a better understanding of the concept of relevance and (ii) a more effective implicit feedback systems.
NeuroIR 2015 - Neuro-Physiological Methods in IR Research
Authors: Jacek Gwizdka, Joemon M. Jose, Javed Mostafa, Max L. Wilson
Venue: Conference SIGIR 2015
Year: 2015

This Tutorial+Workshop will discuss opportunities and challenges involved in using neuro-physiological tools/techniques (such as fMRI, fNIRS, EEG, eye-tracking, GSR, HR, and facial expressions) and theories in information retrieval. The hybrid format will engage researchers and students at different levels of expertise, from those who are active in this area to those who are interested and want to learn more. The workshop will combine presentations, discussions and tutorial elements and consist of four segments (tutorial, completed research, work-in-progress, closing panel).
On Cognition, Emotion, and Interaction Aspects of Search Tasks with Different Search Intentions
Authors: Yashar Moshfeghi, Joemon M. Jose
Venue: Conference WWW 2013
Year: 2013

The complex and dynamic nature of search processes surrounding information seeking have been exhaustively studied. Recent studies have highlighted search processes with different intentions, such as those for entertainment purposes or re-finding a visited information object, are fundamentally different in nature to typical information seeking intentions. Despite the popularity of such search processes on the Web, they have not yet been thoroughly explored. Using a video retrieval system as a use case, we study the characteristics of four different search task types: seeking information, re-finding a particular information object, and two different entertainment intentions (i.e. entertainment by adjusting arousal level, and entertainment by adjusting mood). In particular, we looked at the cognition, emotion and action aspects of these search tasks at different phases of a search process. This follows the common assumption in the information seeking and retrieval community that a complex search process can be broken down into a relatively small number of activity phases. Our experimental results show significant differences in the characteristics of studied search tasks. Furthermore, we investigate whether we can predict these search tasks given user's interaction with the system. Results show that we can learn a model that predicts the search task types with reasonable accuracy. Overall, these findings may help to steer search engines to better satisfy searchers' needs beyond typically assumed information seeking processes.
Centrality measures on big graphs: Exact, approximated, and distributed algorithms
Authors: Francesco Bonchi, Gianmarco De Francisci Morales, Matteo Riondato
Venue: Conference WWW 2016
Year: 2016

Centrality measures allow to measure the relative importance of a node or an edge in a graph w.r.t.~other nodes or edges. Several measures of centrality have been developed in the literature to capture different aspects of the informal concept of importance, and algorithms for these different measures have been proposed. In this tutorial, we survey the different definitions of centrality measures and the algorithms to compute them. We start from the most common measures, such as closeness centrality and betweenness centrality, and move to more complex ones such as spanning-edge centrality. In our presentation, we begin from exact algorithms and then progress to approximation algorithms, including sampling-based ones, and to highly-scalable MapReduce algorithms for huge graphs, both for exact computation and for keeping the measures up-to-date on dynamic graphs where edges are inserted or removed over time. Our goal is to show how advanced algorithmic techniques and scalable systems can be used to obtain efficient algorithms for an important graph mining task, and to encourage research in the area by highlighting open problems and possible directions.
Algorithms for estimating relative importance in networks
Authors: Scott White, Padhraic Smyth
Venue: Conference KDD 2003
Year: 2003

Large and complex graphs representing relationships among sets of entities are an increasingly common focus of interest in data analysis---examples include social networks, Web graphs, telecommunication networks, and biological networks. In interactive analysis of such data a natural query is "which entities are most important in the network relative to a particular individual or set of individuals?" We investigate the problem of answering such queries in this paper, focusing in particular on defining and computing the importance of nodes in a graph relative to one or more root nodes. We define a general framework and a number of different algorithms, building on ideas from social networks, graph theory, Markov models, and Web graph analysis. We experimentally evaluate the different properties of these algorithms on toy graphs and demonstrate how our approach can be used to study relative importance in real-world networks including a network of interactions among September 11th terrorists, a network of collaborative research in biotechnology among companies and universities, and a network of co-authorship relationships among computer science researchers.
Almost linear-time algorithms for adaptive betweenness centrality using hypergraph sketches
Authors: Yuichi Yoshida
Venue: Conference KDD 2014
Year: 2014

Betweenness centrality measures the importance of a vertex by quantifying the number of times it acts as a midpoint of the shortest paths between other vertices. This measure is widely used in network analysis. In many applications, we wish to choose the k vertices with the maximum adaptive betweenness centrality, which is the betweenness centrality without considering the shortest paths that have been taken into account by already-chosen vertices. All previous methods are designed to compute the betweenness centrality in a fixed graph. Thus, to solve the above task, we have to run these methods $k$ times. In this paper, we present a method that directly solves the task, with an almost linear runtime no matter how large the value of k. Our method first constructs a hypergraph that encodes the betweenness centrality, and then computes the adaptive betweenness centrality by examining this graph. Our technique can be utilized to handle other centrality measures. We theoretically prove that our method is very accurate, and experimentally confirm that it is three orders of magnitude faster than previous methods. Relying on the scalability of our method, we experimentally demonstrate that strategies based on adaptive betweenness centrality are effective in important applications studied in the network science and database communities.
Measuring and maximizing group closeness centrality over disk-resident graphs
Authors: Junzhou Zhao, John C. S. Lui, Don Towsley, Xiaohong Guan
Venue: Conference WWW 2014
Year: 2014

As an important metric in graphs, group closeness centrality measures how close a group of vertices is to all other vertices in a graph, and it is used in numerous graph applications such as measuring the dominance and influence of a node group over the graph. However, when a large-scale graph contains hundreds of millions of nodes/edges which cannot reside entirely in computer's main memory, measuring and maximizing group closeness become challenging tasks. In this paper, we present a systematic solution for efficiently calculating and maximizing the group closeness for disk-resident graphs. Our solution first leverages a probabilistic counting method to efficiently estimate the group closeness with high accuracy, rather than exhaustively computing it in an exact fashion. In addition, we design an I/O-efficient greedy algorithm to find a node group that maximizes group closeness. Our proposed algorithm significantly reduces the number of random accesses to disk, thereby dramatically improving computational efficiency. Experiments on real-world big graphs demonstrate the efficacy of our approach.
Neural Query Performance Prediction using Weak Supervision from Multiple Signals
Authors: Hamed Zamani, W. Bruce Croft, J. Shane Culpepper
Venue: Conference SIGIR 2018
Year: 2018

Predicting the performance of a search engine for a given query is a fundamental and challenging task in information retrieval. Accurate performance predictors can be used in various ways, such as triggering an action, choosing the most effective ranking function per query, or selecting the best variant from multiple query formulations. In this paper, we propose a general end-to-end query performance prediction framework based on neural networks, called NeuralQPP. Our framework consists of multiple components, each learning a representation suitable for performance prediction. These representations are then aggregated and fed into a prediction sub-network. We train our models with multiple weak supervision signals, which is an unsupervised learning approach that uses the existing unsupervised performance predictors using weak labels. We also propose a simple yet effective component dropout technique to regularize our model. Our experiments on four newswire and web collections demonstrate that NeuralQPP significantly outperforms state-of-the-art baselines, in nearly every case. Furthermore, we thoroughly analyze the effectiveness of each component, each weak supervision signal, and all resulting combinations in our experiments.