Designing an online and reliable statistical anomaly detection framework for dealing with large high-speed network traffic

350 words maximum: (PLEASE TYPE) Despite a Network Anomaly Detection System (NADS) being capable of detecting existing and zero-day attacks, it is still not universally implemented in industry and real applications, with current systems producing high False Positive Rates (FPRs) and low Detection Rates (DRs). The challenges involved in designing a NADS architecture are 1) the methodology adopted for creating as comprehensive a profile as possible from diverse normal patterns and 2) in establishing an adaptive and lightweight Decision Engine (DE) which efficiently distinguishes between legitimate and anomalous activities at high speeds in large network environments. The need for such a method to be trained and validated on a decent dataset with the characteristics of current network environments is a significant challenge. This thesis provides substantial contributions to research on the building of a scalable, adaptive and lightweight NADS framework. It considers three aspects: a data source, relevant features and observations, and new DE approaches for achieving a reliable NADS architecture. The first key contribution is the creation of a new dataset called UNSW-NB15 that has a hybrid of realistic modern legitimate and synthetic malicious activities, with statistical analyses and evaluations of it fully explained. Also, its complexity is assessed using existing techniques to demonstrate the extent of current sophisticated types of anomalous events. The second core contribution is the development of a new theory for selecting important features and observations from network packets without redundancy to construct a legitimate profile from which any deviation is considered an attack, that is, establish an efficient NADS from analyses of the protocols and services of the OSI model. The third major contribution is the development of two scalable frameworks with two new DE techniques for successfully detecting malicious activities in less processing times than current methods. These techniques, called the Geometric Area Analysis-ADS (GAA-ADS) and Dirichlet Mixture Model-ADS (DMM-ADS), are based on mixture models for modelling all possible normal patterns and detecting abnormal events that deviate from them using new outlier approaches. Declaration relating to disposition of project thesis/dissertation I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only). Signature . Witness . Date The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research. FOR OFFICE USE ONLY Date of completion of requirements for Award: THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS