Characterization of portuguese web searches

Nowadays the Web can be seen as a worldwide library, being one of the main access points to information. The large amount of information available on websites all around the world raises the need for mechanisms capable of searching and retrieving relevant information for the user. Information retrieval systems arise in this context, as systems capable of searching large amounts of information and retrieving relevant information in the user’s perspective. On the Web, search engines are the main information retrieval systems. The search engine returns a list of possible relevant websites for the user, according to his search, trying to fulfill his information need. The need to know what users search for in a search engine led to the development of methodologies that can answer this problem and provide statistical data for analysis. Many search engines store the information about all queries made in files called transaction logs. The information stored in these logs can vary, but most of them contain information about the user, query date and time and the content of the query itself. With the analysis of these logs, it is possible to get information about the number of queries made on the search engine, the mean terms per query, the mean session duration or the most common topics. This analysis is called Query Log Analysis. The results of this analysis can be compared with similar studies, to study eventual changes in users’ behavior due to factors like: time, culture, language or others. This kind of analysis brings many advantages for the search engine themselves and its users; knowing what users search for, it is possible to improve the search engine’s features anticipating and predicting user’s behavior. Analyzing a log from the Portuguese SAPO search engine covering a period of about six months in the year of 2010 some statistics were produced about user’s sessions, queries, terms and searched topics. The original log stored about 140 million queries and contained queries about human users and queries made by bots or other automatic processes. As this study should focus only on the analysis of human queries, these queries were removed. Around 30% of the original queries from the dataset were considered as bots and removed. Queries from the same user were delimited by sessions. Two queries from the same user belong to different sessions if they have an inactivity between them of at least 30 minutes. Sessions with more than 100 queries were considered to be made by bots and were removed. Having a majority of users from Portugal (90%) this study present a characterization of the Portuguese community and the similarities and differences with other studies. Users made more queries on the beginning of the week, and on almost all weekends the daily query traffic is below average. The highest hourly traffic is seen between 2:00 p.m. and 5:00 p.m. where the hourly query frequency is 50% above average. Oddly the hourly distributions of human queries and queries made by bots are very similar. The results show that SAPO’s users make short duration sessions, with 1 or i 2 queries writing between 1 and 3 terms per query. Around 65% of the sessions have a duration lower than 1 minute, and almost 88% last less than 15 minutes. The mean session duration is 5 minutes and 23 seconds. More than half of the sessions only had 1 query and almost 90% of the sessions (85.92%) had up to 5 queries. Only about 5% of the sessions had 10 or more queries. Too common words, function words, like adverbs, propositions or pronouns were removed as they have little lexical meaning and are not relevant for the analysis process. Around 90% of the queries have at most 3 terms and only 1% of the queries have 7 or more terms. The mean terms per query is 2.03. Few queries and terms are unique (19.59% and 3.03% respectively). Users rarely use advanced operators (only in 1.5% of the queries) and when modifying or refining a query the number of terms stays unchanged or 1 term is added/removed. Analyzing a random sample of 2,500 queries it was observed that the main topic of interest was Computers or Internet accounting for 26.88% of the analyzed queries. The second and third categories were People, Places or Things and Commerce, Travel, Employment or Economy with 22.64% and 16.56%. These three categories have a similar hourly distribution with the highest query traffic between 10:00 a.m. and 4:00 p.m. and a clear decreasing after 8:00 p.m. In many categories there is a clear downward tendency as the weekend approaches with some exceptions like Entertainment or Recreation which evidences an opossite tendency, rising until Friday dropping on Saturdays and rising again on Sundays.