THE USE , MISUSE , AND ABUSE OF STATISTICS IN INFORMATION SECURITY RESEARCH

Survey data on information security trends and concerns are used to justify increased expenditures on security tools and technologies. Students use the data to support term paper analyses. Government officials use these data to justify program initiatives and to berate companies for inadequate security. The numbers, however, are anecdotal, are not generalizable to the business level, and are reported in cumulative form. In a word, they are not useful for any of the purposes listed above. This paper examines this phenomenon, looking at survey data that has been published and the uses to which it has been put. Introduction In order for managers to know how to allocate scarce resources for information security in an environment, it is important to understand where the greatest likelihood for problems lie, how much damage can result from any particular attack or problem, and what benefit accrues from any particular technology or control. Managers need to be able to justify expenditures, through showing an avoidance of costly jeopardy, return on investment, or other managerial tools used to make rational decisions regarding enterprise resource allocations. These calculations must be driven by data that provide such things as the probability distributions of events, the expected loss from certain problems. The collection of data from the world at large can provide that kind of data for savvy managers to then use to assist in the difficult decision process of where to invest scarce resources. Many research efforts purport to provide that data to managers. A small sampling of recent headlines include the following: • "Cyber-Attack Costs Down, Says Survey" (Fisher, 2003) • "Business Not Prepared for E-Risks" (Computer Security Update, May 2003) • "The Sad And Increasingly Deplorable State Of Internet Security" (Piscitello and Kent, 2003) The data presented in these articles is sobering and thought provoking. Piscitello and Kent warn that "the security incident rate is doubling annually. " (Piscitello and Kent, 2003). The Computer Security Update article on E-Risks states that "19% of employers have battled lawsuits stemming from e-mail/Internet abuse, 31% have experienced loss of confidential information/intellectual property via e-mail, and 35% have terminated employees for e-mail/internet abuse. " (Computer Security Update, May 2003) Fisher reports some encouraging news, on the other hand: "The 530 organizations surveyed reported $201.8 million in losses this year; in 2002, 503 respondents lost $455.8 million. " (Fisher, 2003) Responsible managers reading these and other articles would certainly have a lot to think about in terms of their enterprises' approaches to security and protection of intellectual assets. What is lost in the stories of these various research efforts is the nuances and subtleties of the research methodologies used, the statistics applied, and the data reported. An in-depth study of survey data revealed a serious problem in all three of these areas. In many cases, the research methodologies were not sound (in some cases, the results were specifically identified as being unscientific). The statistical analyses were in some cases inappropriate and in general only partial results reported in the press (as might be expected). While these problems are not unexpected, there is a larger problem that the data is being taken from the popular press and used by policy and decision makers to guide resource allocations in security training, development, and technology applications. There is no doubt that security is of grave concern to the nation and to the business sector. The fact that the data driving the decision processes is in general fundamentally flawed in one or more of the phases of creation, manipulation and reporting is therefore of critical concern. The Surveys An analysis was performed on fourteen publicly available surveys on the state of information security practices and experience by business. The surveys chosen represented the most widely publicized surveys within a five year time period of 1995 to 2000. The surveys analyzed are shown in Exhibit 1. The majority of these surveys targeted information technology professionals and information security professionals at large companies through professional mailing lists or other professional contact databases. Half of the surveys were limited to the North American continent: four of the surveys covered only US firms and three covered firms in the US and Canada. Of the remaining seven, five were global in reach. Exhibit 1. Surveys Reviewed for Methodology and Results. Survey Number Of .