Real Life, Real Users, and Real Needs: A Study and Analysis of User Queries on the Web

We analyzed transaction logs containing 51,473 queries posed by 18,113 users of Excite, a major Internet search service. We provide data on: (i) sessions changes in queries during a session, number of pages viewed, and use of relevance feedback, (ii) queries the number of search terms, and the use of logic and modifiers, and (iii) terms their rank/frequency distribution and the most highly used search terms. We then shift the focus of analysis from the query to the user to gain insight to the characteristic of the Web user. With these characteristics as a basis, we then conducted a failure analysis, identifying trends among user mistakes. We conclude with a summary of findings and a discussion of the implications of these finding. INTRODUCTION A panel session at the 1997 ACM Special Interest Group on Research Issues In Information Retrieval conference entitled “Real Life Information Retrieval: Commercial Search Engines” included representatives from several Internet search services. Doug Cutting represented Excite, one of the major services. Graciously, he offered to make available a set of user queries as submitted to his service for research. The analysis we present here on the nature of sessions, queries, and terms resulted from this offer. Interestingly, the authors expressed their interest independently of each other, then met via email, exchanged messages and data, and conducted collaborative research exclusively through the Internet, before ever meeting in person at a Rutgers conference in February 1998, when the results were first presented. In itself, this is an example how the Internet changed and is changing the conduct of research. We will argue in the conclusions that real life Internet searching is changing information retrieval (IR) as well. While Internet search engines are based on IR principles, Internet searching is very different then IR searching as traditionally practiced and researched. Internet IR is a different IR, with a number of implications that could portend changes in other areas of IR as well. With the phenomenal increase in usage of the Web, there has been a growing interest in the study of a variety of topics and issues related to use of the Web. For instance, on the hardware side, Crovella and Besravros (1996) studied client-side traffic; and Abdulla, et. al., (1997) analyzed server usage. On the software side, there have been many descriptive evaluations of Web search engines (e.g. Lynch 1997). Statistics of Web use appear regularly (e.g. Kehoe et. al. 1997, FIND/SVP, 1997), but as soon as they appear, they are out of date. The coverage of various Web search engine services was analyzed in several works. A recent article on this topic by Lawrence and Giles (1998) attracted a lot of attention. The pattern of Web surfing by users was analyzed as well (Huberman et. al. 1998). However, to date there has been no large-scale, quantitative or qualitative study of Web searching. How do they search the Web? What do they search for on the Web? These questions are addressed, as far as we can ascertain, for the first time in a large scale and academic manner in this study. Given the recent yearly exponential increase in the estimated number of Web users, this lack of scholarly research is surprising and disappointing. In contrast, there have been an abundance of user studies of on-line public access categories (OPAC) users. Many of these studies are reviewed in Peters (1997). Similarly, there are numerous studies of users of traditional IR systems. The combined proceedings of the International Conference on Research Issues in Information Retrieval present many of these studies. In the area of the Web users, however, there were only two narrow studies that we could find. One focused on the THOMAS system (Croft, Cook, and Wilder 1995) and contained some general information about users at that site. However, this study focused exclusively on the THOMAS Web site, did not attempt to characterize Web searching in a systematic way, and is devoted primarily to a description of the THOMAS system. The second paper was by Jones, Cunningham, and McNab (1998) and focused again on a single Web site, the New Zealand Digital Library, which contains computer science technical reports. Given the technical nature of this site, it is questionable whether these users represent Web users in general. Beyond these two articles, we could find no others. Obviously, there is a surprising lack of studies focusing on Web users, which is interesting given the numerous user studies in the OPAC and IR system areas and the current, almost ubiquitous nature of the Web. In this paper, we report results from a major and ongoing study of users’ searching behavior on the Web. We examined a set of transaction logs of users’ searches from Excite (http://www.excite.com). This study involved real users, using real queries, with real information needs, using a real search engine. The strength of this study is that it involved a real slice of life on the Web. The weakness is that it involved only a slice – an observable artifact of what the users actually did, without any information about the users themselves or about the results and uses. The users are anonymous, but we can identify one or a sequence of queries originating with a specific user. We know when they searched and what they searched for, but we do not know anything beyond that. We report on artifactual behavior, but without a context. However, the observation and analysis of such behavior provides for a fascinating, surprising, and sometimes unexpected insight about the interaction between users and the search engines on the Web. More importantly, this study provides detailed statistics on Web user behavior that is currently lacking. It also provides a basis for comparison with similar studies of user searching of more traditional IR and OPAC systems. The Web has a number of search engines. The approaches to searching, including algorithms, displays, modes of interaction and so on, vary from one search engine to another. Still, all Web search engines are IR tools for searching highly diverse and distributed information resources as found on the Web. They all follow the basic principles of IR and human-computer interaction. But by the nature of the Web resources, they are faced with different issues requiring different solutions than the search engines found in well organized systems, such as in DIALOG, or in lab experiments, such as in the Text Retrieval Conference (TREC). Moreover, from all that we know, Web users spans a vastly broader and thus probably different population of users (Spink, Bateman, and Jansen 1998) and information needs, which may greatly affect the queries, searches, and interactions. Thus, it is of considerable interest to examine the similarities and/or differences in Web searching compared to traditional IR systems. In either case, it is IR, but it is potentially a very different IR. Given the estimated size of the Web user population, the probable impact on other IR system user populations is high. The significance of this study is the same as all other related studies of IR interaction, queries and searching. By axiom and from lessons learned from experience and numerous studies: “The success or failure of any interactive system and technology is contingent on the extent to which user issues, the human factors, are addressed right from the beginning to the very end, right from theory, conceptualization, and design process to development, evaluation, and to provision of services. ” (Saracevic, 1997). RELATED IR STUDIES In this paper, we concentrate on users’ sessions, queries, and terms as key variables in IR interaction on the Web. While there are many papers that discuss many aspects of Web searching, most of those are descriptive, prescriptive, or commentary. Other than the two mentioned previously, we could not find any studies of Web searching similar to this one, containing data on searches, thus we have nothing to compare our study. However, there were several studies that included data on searching of existing, mostly commercial, IR systems, and we culled data from those to provide for some comparison between searches as done on the Web and those as done on IR systems outside the Web. A representative sample of such studies is reviewed. The studies cited below concentrated on different aspects and variables related to searching, using different methodologies, thus, they are difficult to compare. Still, each of them had data on the mean number of search terms in queries constructed by the searchers under study as follows: • Fenichel (1981): Novice searchers: 7.9. Moderately experienced: 9.6. Experienced: 14.4 • Hsieh-yee (1993): Familiar topics: Novices: 8.77. Experienced: 7.28. Non-familiar topics: Novices: 9.67. Experienced: 9.00 • Bates (1993): Humanities scholars: 14.95 • Spink & Saracevic (1997): Experienced searchers: 14.8. The studies indicated that searches by various populations contain a range of some 7 to 15 terms. As will be discussed below, this is a considerably higher range than the mean number of terms found in this study that concentrated on Web searches from the Excite search engine. BACKGROUND ON EXCITE AND DATA Founded in 1994, Excite, Inc. is a major Internet media public company which offers free Web searching and a variety of other services. The company and its services are described at its Web site, thus not repeated here. Only the search capabilities relevant to out results are summarized. Excite searches are based on the exact terms that a user enters in the query, however, capitalization is disregarded, with the exception of logical commands AND, OR, and AND NOT. Stemming is not available. An online thesaurus and concept linking method called Intelligent Concept Extraction (ICE) is used, to find related terms in addition to terms entered. Search results are provided in a ranked relevance order. A number of ad