Operating system and file system monitoring: A comparison of passive network monitoring with full ke

System monitoring is the process by which information about computer systems and users of those systems is collected. It is carried out typically to assist in improving the operation of current systems and to aid in the development of future computer systems. This thesis compares and contrasts two system-monitoring techniques: full kernel instrumentation, where information is obtained through the placing of instrumentation code into the computer's operating system (kernel), and passive network monitoring where data associated with one or more computer systems attached to a network are extracted by monitoring the communications tra c exchanged by those computers with others on the network. In order to achieve this comparison and the contrast of the two system-monitoring techniques, two such systems were implemented, operated in tandem and the results then compared. A kernel-instrumentation system was ported from an earlier revision of the UNIX operating system to a more up-to-date version. This involved developing a working knowledge of the kernel system in general and the development of programs to process the records generated by the kernel instrumentation into a format that could be then be compared with the passive network monitoring system. The passive system was substantially complete as received although some support software was written. Processing software also had to be written for summarising the records of each technique and for producing a comprehensive analysis of the trace record. An understanding of the rule base of the post-processing software was achieved by thorough investigation including instrumentation and trial operation. A simpli ed cache simulator was constructed which, although it could not be integrated into the passive network monitoring software, aided in an understanding of the cache system. viii Full kernel instrumentation has been a popular technique because of its comprehensive nature but it su ers, among other things, from the necessity to have the kernel source-code available, from the need to make changes to the system being monitored and from the impact it can have on that system. In contrast, passive network monitoring can be conducted in a non-invasive, platform-independent manner which involves no changes to the operation of the monitored machine. Passive network monitoring has the potential to be used in place of full kernel instrumentation for many tasks, and even though it was unable to give results comparable with full kernel instrumentation in all cases, it is able to give good predictions of many values when compared with those derived from full kernel instrumentation. This was particularly true in areas related to the writing of data. This thesis also notes discrepancies between the result from the two system-monitoring methods and discusses ways in which those discrepancies can be reduced or eliminated. ix Acknowledgements Firstly, I would like to thank my parents, Alan and Dorothy Moore, and my brother, Nicholas, for their endless and unwavering love and encouragement. Mere words cannot begin to express my appreciation and love. I could not have had a better combination of mentors than my supervisors, Tony McGregor and Jim Breen, They are both true and steady friends who willingly give their time and guidance. This thesis would have been at least di erent and, perhaps, impossible without access to a number of programs. The starting point of this thesis was when Matt Blaze made the original rpcspy/nfstrace suite of software available to me. He then passed on changes and additions as well as words of advice and encouragement. I am more than grateful to him. Songnian Zhou and Chris Siebenmann gave me access to the original snooper code and gave unstintingly of their time to answer many questions and to o er advice. Michael Dahlin willingly assisted me by supplying changes his work group had made to rpcspy/nfstrace. My thanks to you all. I am grateful to Randy Appleton, Mary Baker, Charles Briggs, Dan Eaves, Rick Floyd, Simon Hill, Rick Macklem, John Ousterhout, Alan Rollow and Margo Seltzer for unhesitatingly giving of their enthusiasm, sought-after papers, ideas and the conrmation that the work of this thesis was worthwhile. I owe an eternal debt to the sta of the Department of Robotics and Digital Technology and to the users of the machines used for testing for putting up with crashing clients and low performance networks. x My thanks to Andrew Lysikatos for organising access to the Hewlett Packard network analyser and to Kathy Ching for giving so much of her time to assist in obtaining the much-needed source code which made this thesis possible. I extend thanks to my friends, in particular Michelle Judson, Cameron Blackwood, Rik Harris and Chris Beecroft, for making the past years both interesting and enjoyable and to Cameron and Chris for proof-reading. Finally, special thanks to Ralphe Neill, who willingly read through and commented on early revisions of this thesis.