An Efficient and Incremental System to Mine Contiguous Frequent Sequences

Mining frequent patterns is an important component of many p rediction systems. One common usage in web applications is the mining of users’ acce ss b havior for the purpose of predicting and hence pre-fetching the web pages that the u ser is likely to visit. Frequent sequence mining approaches in the literature are o ften based on the use of an Apriori-like candidate generation strategy, which typi cally requires numerous scans of a potentially huge sequence database. In this paper we inste ad introduce a more efficient strategy for discovering frequent patterns in sequence dat abases that requires only two scans of the database. The first scan obtains support counts f or subsequences of length two. The second scan extracts potentially frequent sequenc es of any length and represents them as a compressed frequent sequences tree structure (FStr e). Frequent sequence patterns are then mined from the FS-tree. Incremental and inter active mining functionalities are also facilitated by the FS-tree. As part of this work, we d eveloped the FS-Miner, an system that discovers frequent sequences from web log files. The FS-Miner has the ability to adapt to changes in users’ behavior over time, in the form o f new input sequences, and to respond incrementally without the need to perform full re -computation. Our system also allows the user to change the input parameters (e.g., mi nimum support and desired pattern size) interactively without requiring full re-com putation in most cases. We have tested our system using two different data sets, comp aring it against two other algorithms from the literature. Our experimental results s how that our system scales up linearly with the size of the input database. Furthermore, i t xhibits excellent adaptability to support threshold decreases. We also show that the increm ental update capability of the system provides significant performance advantages ove r full re-computation even for relatively large update sizes. Acknowledgements I would like to thank my thesis advisor Prof. Carolina Ruiz an d my thesis reader Prof. Elke Rundensteiner for their guidance and advise through ou t this work. I would also like to thank the department head Prof. Michael Gennert and m embers of the graduate committee Prof. Matthew Ward, Prof. Craig Wills, Prof. Mark Claypool, and Prof. Murali Mani and for their support. I would like to express my special gratitude to Prof. Elke Run de steiner for her guidance, encouragement, and support since I came to WPI. I would like also to thank Prof. Micha Hofri who gave me the chance to teach at WPI and to gain su ch a valuable experience. My deepest gratitude goes to my parents, who have been always behind every advance in my life, and also to my lovely wife Hanan, my lovely son Yous sef, and my lovely new baby Sief for all the love, support, and understanding. I must also express my appreciation for many people, who have supported me, including Mr. Mahmoud Ghoneim (may God bless his soul), Prof. S aid El-Neshokaty, Mahmoud Youssef, Prof. Carolina Ruiz, Prof. Stanley Selkow , Mrs. Glynis Hamel, members of the DSRG research group, and all my friends here in USA and back home in Egypt.