Project Summary : CI-SUSTAIN : Sustainable Tools for Analysis and Research on Darknet Unsolicited Traffic

For over a decade, the UCSD Network Telescope (UCSD-NT) instrumentation, which captures traffic sent to a large segment of unassigned IPv4 address space (darknet), has enabled global visibility into macroscopic Internet phenomena that few other data sources can provide. The relevance of this community research infrastructure for the study of the Internet is reflected in a vast literature in a broad set of sub-disciplines in CISE and beyond, from network and systems security and stability, to machine learning and big data processing techniques, and most recently to studies of cyberwarfare and political repression of communication. In 2011 we enhanced the Telescope instrumentation to enable access to raw and live telescope traffic data, which expanded the scope of possible research questions and the circle of researchers using the data. As of January 2017 we are aware of (a lower bound) over 100 publications – without UCSD co-authors – that used UCSD-NT data. Our exciting success with this project has triggered a new problem: the current state of this infrastructure is lagging behind the increasing demands in terms of storage, computing resources, and system administration. These issues hinder our ability to continue sharing UCSD-NT data with researchers, and have already required compromises that limit its utility to the community. We propose a project to enable sustained long-term operation of the UCSD-NT infrastructure, while also increasing its utility to the research community. We organize our proposed scope of work into three inter-related tasks. First, we will upgrade and modernize the current infrastructure (network data capture and communication hardware and storage) to stabilize operations. Second, we will transition the data analysis computing infrastructure to a sustainable model based on virtualization, leveraging NSF-funded HPC platforms at SDSC. This task consists of three connected sub-tasks: deploying cloud-compute support using new virtualization features of the Comet environment; redesigning the software pipeline to take advantage of this new capability for transferring live streams of data; and collaborating with HPC researchers at SDSC to develop tools to support dynamic provisioning of Spark and Hadoop clusters as needed. Third, we will introduce meta-data semantics that simplify many tasks researchers typically want to do with UCSD-NT data, leaving them more time (and available HPC resources) to focus on their specific scientific questions. While mainly targeting long-term sustainability, our proposed solutions will also improve the research utility of this data, further lowering barriers for and thus increasing diversity of the user base of this research infrastructure. Intellectual Merit. This project will result in an interdisciplinary collaboration between researchers from the field of computer networks and high-performance computing scientists and engineers to experiment with novel approaches leveraging HPC infrastructure for research on live traffic analysis. By upgrading the infrastructure to sustain predicted growth in use for many years, we are stabilizing and enhancing our ability to serve a diverse range of academic researchers, the vast majority of whom have no access to any other source of global Internet traffic data. Broader impacts. We will disseminate project results via conferences, web sites, archived video lectures, blogs, and the proposed workshops and will continue our engagement of faculty and graduate students in the use of our tools and data. Project results will contribute to advancing knowledge in diverse CISE disciplines, e.g., enabling the development of efficient strategies for early detection and mitigation of security-related events, providing macroscopic Internet performance and reliability assessments, and providing a new domain for the application of live streaming big data analysis, and in situ machine learning techniques.