Combating Terrorism: The Challenge of Measuring Effectiveness
Released: 2007-03-12T00:00:00

Summary:
This report is designed to support efforts of the 110th Congress to understand and apply broad based objective criteria when evaluating progress in the nation's efforts to combat terrorism. It is not intended to define specific, in-depth, metrics for measuring progress against terrorism.
How one perceives and measures progress is central to formulating and implementing anti-terror strategy. Perception has a major impact, as well, on how nations prioritize and allocate resources. On the flip side, the parameters used to measure progress can set the framework for the measurement of failure. The measurement process is also inextricably linked to strategies. Progress is possible using diverse strategies, under very different approaches. The goals of terrorists and those who combat them are often diametrically opposed, but may also be tangential, with both sides achieving objectives and making progress according to their different measurement systems.
Within the context of these competing views and objectives, terrorist activity may be seen as a process which includes discrete, quantum-like changes or jumps often underscoring its asymmetric and nonlinear nature. An approach which looks at continuous metrics such as lower numbers of casualties may indicate success, while at the same time the terrorists may be redirecting resources towards vastly more devastating projects. Policymakers may face consideration of the pros and cons of reallocating more of the nation's limited resources away from ongoing defensive projects and towards preventing the next quantum jump of terrorism, even if this means accepting losses.
Measurement of progress, or lack thereof, may be framed in terms of incidents, attitudes and trends. A common pitfall of governments seeking to demonstrate success in anti-terrorist measures is overreliance on quantitative indicators, particularly those which may correlate with progress but not accurately measure it, such as the amount of money spent on anti-terror efforts. As terrorism is a complex multidimensional phenomenon, effective responses to terrorism may need to take into account, and to some degree be individually configured to respond to, the evolving goals, strategies, tactics and operating environment of different terrorist groups. Although terrorism's complex webs of characteristics—along with the inherent secrecy and compartmentalization of both terrorist organizations and government responses—limit available data, the formulation of practical, useful measurement criteria appears both tractable and ready to be addressed. This report will not be updated.

subsections: section_title: Introduction
subsections: section_title: Scope
paragraphs: This report is designed to help the 110 th Congress better understand the characteristics and importance of measurement of counter-terrorism activities, the dynamics of the phenomenon to be measured, i.e. terrorism, and what would generally be required from any entities—government or otherwise—tasked with establishing and evaluating measures of effectiveness. It is not intended to define counter-terrorism activities, nor to create a definitive, in-depth methodology for measuring progress against terrorism. Rather, it is designed to provide added tools and insights to support Congress in its efforts to coordinate, fund, and oversee the nation's anti-terrorism activities.
section_title: Importance of Measuring Progress in Anti-Terror Efforts
paragraphs: The Government Performance and Results Act (GPRA) requires agencies to set goals and objectives for their performance, and to measure progress against these goals and objectives. Under the law, anti-terror efforts are not exempt from these requirements, without which the objective evaluation of progress to the full satisfaction of Congress might be impeded.
Rising costs of anti-terrorism efforts have become an increasing problem. The vast land area of the United States and widespread U.S. interests abroad are impossible to protect entirely. Billions of dollars have been spent to develop anti-terror technologies, establish crisis management training and enhance security staffing throughout the country. Whether these expenditures are cost-effective, or whether the money would be better spent, for instance, building secular schools in Islamic countries or promoting public relations efforts aimed at young Muslims, remains an important policy question. Developing robust measurement criteria might assist government officials in answering such questions.
It is unclear just how much the United States spends overseas annually in non-military areas to combat terrorism, but the amounts are in the billions of dollars. For FY2004, the Government Accountability Office (GAO) put the figure at $11 billion. At home, dollar amounts spent on terrorism-related security by the 50 states are elusive as well, but FY2006 appropriations for the Department of Homeland Security topped $30 billion ($30.8 billion). Arguably, existing legislation requires government agencies to be held accountable for the cost-effectiveness of these mammoth expenditures on combating terrorism. Demonstrable, measurable, effective progress against terrorism is the desired goal.
section_title: Challenges in Measuring Progress
paragraphs: Among the various U.S. government agencies involved in anti-terrorism efforts, there is currently no common set of criteria for measuring success. Although over four years have elapsed since the tragic events of September 11, 2001, many agencies are still attempting to establish and define precise criteria and standards, without which they cannot measure organizational performance. Uncertainty with respect to both strategies and measurements makes it difficult to describe progress accurately and to demonstrate progress to the public or U.S. allies.
Different types of terrorist threats may carry different risks and potential impacts, and strategies may need to adapt accordingly. For instance, different strategies may be applicable to terrorism rooted in political or economic vs. cultural or theocratic origins. Such differences in threats and our response strategies make measurement of progress more difficult.
For the greatest success, the war on terrorism, like the war on drugs, must arguably be conducted and measured in a multi-faceted manner on many fronts, with sufficient resources allocated to each strategic element. It is usually not possible to achieve this approach due to funding limitations, and the result is vulnerabilities in certain areas, where acknowledgment of lack of progress may be politically unpopular.
Of concern to some is that efforts to gauge progress may be compromised should measurement criteria selectively target areas of apparent relative progress, such as increased airport security, while other areas, such as port security, may remain under-addressed. Another area of concern is that anti-terrorist actions might be undertaken for a wide range of reasons without being clearly linked to previously defined anti-terror goals; in such circumstances, caution would likely be warranted before characterizing results as progress.
A common misconception is that by increasing expenditures the nation is necessarily making good progress. As a practical matter, the nation cannot secure everything, everywhere. Terrorist operations are relatively inexpensive to organize and carry out, especially in comparison to the damage they may inflict, or the cost of trying to prevent them from happening. Consequently, spending more money may not necessarily increase security proportionally. Moreover, some suggest that the United States is bleeding itself dry economically, like the Soviet Union did in its attempts to match western military spending during the cold war.
Some contend that the biggest threat to democracy from terrorism is not destruction of property and life, but rather an inexorable erosion of civil liberties worldwide. Other concerns are loss of international unity due to policy differences, loss of opportunities due to budget and policy constraints, and reduction of U.S. stature and public relations image abroad. These and many other factors form a mosaic of measurements that highlight the complexities of analyzing progress.
section_title: Impact of Perceptions on Strategy
paragraphs: How one perceives and measures progress is central to formulating and implementing anti-terror strategy. The perception of progress has a major impact on establishing priorities and allocating resources. The parameters used to measure progress can also set the framework for measurement of failures. To better define the parameters of success, it is important to determine what both the terrorists, and those who fight them, see as their goals and priorities.
Critical to the measurement process is the realization that measurements are inextricably linked to strategies. Positive progress is possible using diverse strategies, which may employ very different tactics. For example, some nations take a hard line against radical Islamists, including targeted assassination, while other nations appear to have a more laissez-faire —if not conciliatory/accommodating—approach. Both strategies can claim progress, using different measurements. An important consideration in formulating measurable strategies is reduction of uncertainty through clear enunciation of policy.
section_title: Current Strategy and Cited Results
paragraphs: In a statement issued by President Bush on September 28, 2005 titled Fighting a Global War on Terror , he emphasized four core elements of America's strategy for victory in the war on terror: (1) fighting the enemy abroad; (2) denying terrorists state support and sanctuary; (3) denying terrorists access to weapons of mass destruction; and (4) spreading democracy. These elements generally echo Administration anti-terror strategy as set forth in its February 14, 2003, National Strategy for Combating Terrorism .
In the wake of 9/11, the Administration has pointed to the killing or capture of more than 2/3 of al Qaeda's top leadership and seizure of over $200 million in terrorist financing as examples of progress against terrorism. More recently, progress milestones cited by President Bush in his September 28, 2005 statement included (1) removal of brutal regimes in Afghanistan and Iraq that harbored terrorists; (2) moving forward in the "march" of democracy worldwide, noting Lebanon; (3) shutting down a major weapons of mass destruction [WMD] black market network originating in Pakistan, and Libya's rejoining a community of nations; and (4) capturing a number of key terrorists in Pakistan and Iraq, as well as capturing and killing hundreds of insurgents in Iraq. Disruption of al Qaeda terrorist plots and efforts to infiltrate the United States were subsequently cited as an additional indication of success by the President in his discussion of the war on terror at the National Endowment for Democracy on October 6, 2005.
section_title: Measurement Issues
paragraphs: Though some $200 million is said to have been have been confiscated, it is not clear how much damage has been done to the terrorist's ability to raise or transfer additional funds. Moreover, if one terminates 2/3 of the senior leadership of a particular terrorist organization, the ranks of the organization may grow and decentralize, similar to the impact of attacks on drug cartels, evolving into a more resilient adversary.
Another key issue is how one measures the impact of unintended consequences—or side effects and by-products—and other results, such as diverting scarce resources from one policy area to another, increasing spending and possibly adding to the budget deficit, or eroding civil liberties. These types of issues were often not addressed in the Patterns of Global Terrorism reports of previous years. However, in contrast, the successor to Patterns , titled Country Reports on Terrorism , now emphasizes in-depth, comprehensive analysis.
section_title: Describing and Measuring Progress Against Terrorism
subsections: section_title: Terrorism as Process
paragraphs: The phenomenon of terrorism can be seen as comprising human elements (supporters and hard core terrorists) and ideological elements. To the degree that terrorism is viewed as a process, the phenomenon is similar to a pipeline or factory assembly line with key stations along the way. The process includes ideological outreach, acquisition of funding and support, recruitment, organization, indoctrination, training, planning, targeting, attack, exploitation of results, financial rewards and other factors which lead to production of terrorist acts. Any such proposed anti-terrorism model would be adapted for specific terrorist groups, since such groups may operate differently.
A challenge facing those who seek to measure progress against terrorism is to identify critical, outcome-determining elements and assess how well they have been mitigated. Disrupting the process of terrorism as early as possible is vital, since it can eventually become an economic engine in its own right, with increasing numbers of individuals and businesses deriving financial benefits and developing vested interests.
The terrorists' initial search for ideological or financial supporters and physical recruits comes at the beginning of the process. Important here is the level of state support or opposition to such activities. At a subsequent phase, one might look at the organization or network and seek to measure its effectiveness and resiliency.
In addition to measuring the terrorism process itself, structural and environmental factors also require evaluation. Is the dependence on certain factors that terrorists or anti-terrorists can exploit as vulnerabilities increasing or decreasing? How seamlessly do terrorist organizations and networks interact? How seamlessly do new government anti-terrorist organizational structures and networks interact? Is the international operating environment becoming more or less inviting or restrictive for terrorists or for those combating them? Is it easier or harder today for terrorists to inflict the damage they seek to do?
Also important is the post-attack/recovery period. Since not all targets can be protected at all times, some will likely be hit. Hence the effectiveness of post-attack/incident recovery is a significant factor in measuring the success or failure of terrorist operations. Is the ability of the United States to recover from bombings or similar attacks stronger today than it was several years ago, or not? Increasingly, as terrorist groups seek to cause economic damage, the ability of governments to recover rapidly economically in the aftermath of terrorist attacks becomes an important indicator of progress in combating the phenomenon of terrorism.
Success at each stage of the process can be measured in various ways, including relatively continuous metrics such as the number of recruits, the dollars expended, the economic value of targets, the number of casualties inflicted, etc. Similar assessment categories can be developed for other pertinent factors, such as societal or environmental aspects of terrorism. Related factors may sometimes be grouped for convenience of discussion or to render them more amenable to certain mathematical treatment. Measurements may be compared at various points in time or otherwise analyzed inferentially.
It is natural to assume that decreases in terrorist activity, or even a slowing rate of increase, reflect progress in anti-terror efforts. Arguably, however, this type of measurement may underestimate the varied nature of terrorist actions. The often asymmetric, nonlinear nature of terrorist operations, frequently characterized by abrupt changes, increases the deadliness of the threat and necessitates measurements of progress that more accurately reflect this additional danger.
section_title: Measuring Terrorism as Quantum Change
paragraphs: It is important to recognize that terrorist activities (and concomitant anti-terrorist efforts) evolve as a set of actions, incidents, and other manifestations evidenced by quantum-like jumps and changes in state. This may make it more difficult to measure success, or failure, but this view is compatible with the complexities in this area.
For instance, once suicide bombing starts, it presents a qualitatively different environment than before and changes the nature of the threat. Likewise, small bombings may be on a continuum, but the attacks of 9/11 were a qualitatively different phenomenon. Once terrorism escalates to a new plateau, it becomes increasingly dangerous, like the mutation of a virus to a more virulent strain.
As terrorism mutates, so must an effective response designed to counter it. Once terrorism has escalated to a higher level, a previous response may be less effective. Hence, terrorism requires a pro-active and quickly malleable policy of prevention and mitigation.
One concern is that the phenomenon of terrorism, if not effectively challenged and disrupted, may at some point jump to become a regional or global pandemic of violence as an accepted modus operandi for social change. Yet another concern is that terrorism will become the ballot box for the dispossessed, if the gap between the "haves and have nots" continues to widen.
Arguably therefore, it is important not only to measure where terrorism is, but also how close terrorists are to the next quantum jump. The potential quantum jump currently of greatest concern to many would be to WMD (chemical, biological or radiological/nuclear). In this view, a focal point of measurement would be how close the terrorists are to this next level, what it would take for them to achieve it, and how well the nation is preventing them from getting there. What are possible indicators that a quantum jump is imminent?
section_title: Indicators of Quantum Change
paragraphs: Some of the possible indicators, experts look to, or might look to, as an indication that a terrorist group is about to move to another level include:
Intelligence . Ability of terrorists to ascertain specific knowledge critical to exploiting the nation's vulnerabilities or to thwarting anti-terror operations. Technology . Closeness of access to a critical quantum change-producing technology or equipment incorporating such technology. Impact on Society . Both material and psychological impact beyond a critical threshold, such as disruption of the banking system, or establishment of such pervasive fear that key civil liberties or moral principles underlying the national identity are set aside by the government in the interests of security. Targets and Their Protection . Shifts from individual targets to mass casualties; shifts from focus on high-visibility targets to coordinated attacks against multiple softer targets, yielding a domino effect with mega-impact (including mega-impact on morale); generally, dramatic shifts in the scale and brutality of attacks. Alliances . The emergence of new synergistic terrorist alliances and the willingness/ability of terrorist organizations to form alliances with other terrorist and criminal networks, and/or rogue states. This synergy could escalate terrorist operations in a nonlinear manner. Disruption . Large-impact, unexpected attacks which could force anti-terror operations into solely defensive posture. Amount of Unproductive Energy Expended . The degree to which terrorists force governments to expend a critical limit of funds or resources beyond which certain anti-terror efforts become unsustainable and must be curtailed. Sophistication of Effort . Hijacking an airplane to hold passengers hostage vs. doing so to use it as a missile; suicide bombings by lone individuals vs. suicide bombings by trucks with high explosives; hijacking of web pages vs. widespread disruption of communications networks using sophisticated computer viruses; use of conventional explosives vs. "dirty" bombs. Morale/Momentum . Ordinary recruitment of a small number of disenchanted fanatics vs. the critical mass of group dynamics needed to foment and propagate terrorism as a self-sustaining process. On the other hand, anti-terror efforts which dilute the process below a critical threshold may result in the eventual dissolution of the process.
section_title: Measuring Progress
paragraphs: Progress may be defined differently by the terrorists and those who oppose them. Hence both can claim progress, and both can be correct in their assessments. How can this be reconciled? How can measurements of progress be established which are not politicized or biased? In this regard, one must be cautious that success is not defined retrospectively, with goals reformulated after the fact to correspond with the known outcomes. Arguably, measurements of progress have greater validity if strategies are established before, and not after, taking action.
In a search for meaningful measurement criteria, the academic, engineering, scientific and actuarial communities may have much to offer government policymakers. Extensive mathematical tools exist to help define and validate proposed measurement systems, and indeed one might employ a variety of such systems used by different groups. As long as measurements are clearly defined and linked to goals and objectives, these differences need not be divisive.
The conduct of a survey of data on terrorism is an option. That is, a survey of what data on terrorism—especially data bases—currently exist, what categories and details are found within that data, and what the data can reasonably inform policymakers about. Existing methodologies for measuring progress in combating complex social phenomena such as drug trafficking and crime could contribute valuable insights.
section_title: Models for Measurement
subsections: section_title: Example Framework for Measurement
paragraphs: In designing metrics for measuring effectiveness of anti-terror efforts, one option might begin with three major categories: incidents, attitudes, and trends. This is one of many possible models. Analysis of the resulting data could address how well the process of terrorism is being disrupted, on both continuous and quantum scales.
section_title: Incidents
paragraphs: In the past, the number of terrorist attacks, or "incidents," were prominently displayed in publications such as the Department of State's annual Patterns of Global Terrorism reports. The reports also indicated how widespread the incidents were geographically, and how deadly they were in terms of persons killed or injured. But arguably neglected was the impact of such incidents—especially their effect on the macro-economy. Also, not fully clear is which attacks are important or meaningful enough from the standpoint of measurement to be considered "incidents."
In attempting to measure incidents, some in the United States tend to define success in familiar ways: body counts and numbers. In a western, science-and-technology-oriented society, many feel that if a problem can be quantified, it can be solved. However, a common pitfall is overreliance on quantitative data at the expense of its qualitative significance. In previous years' Patterns of Global Terrorism reports, incidents were counted equally without regard to their broader impact. To the degree that terrorist constituencies are not from western cultures, their mindsets may not necessarily place a premium on quantification metrics, but rather on other values such as religious precepts, or honor or revenge.
Western policymakers often tend to define success by the absence of attacks. When the shooting or bombing stops, for example, that is viewed as success. Yet terrorists sometimes define success in terms of making governments expend limited resources trying to defend an enormous number of potential targets. For terrorists, the absence of violent conflict may simply mean that they are focusing attention on economic, political, or social spheres, or just that they are in a "waiting period." Western policymakers often define success in terms of the amount of money confiscated from terrorist networks. Terrorists may define success in terms of the amount of money they force an opponent to squander to seize potentially insignificant amounts.
section_title: Attitudes
paragraphs: Attitudes drive both terrorism and the world's response to terrorism. How do attitudes affect political decisions and sentiments in countries to contain and defeat terrorism, or to support it? How long can democratic governments pursue policies that pressure terrorists if such policies are seen as bringing on terrorist retaliation? Similarly, how much increase in economic costs and reduction in civil liberties will public opinion tolerate? Shaping attitudes to break or weaken the political will to combat terrorism is a central terrorist goal and an important indicator of success or failure.
With regard to attitudes, terrorists often see success as breaking their opponents' will. They want to push the conflict into the political arena on the streets of Washington, London, Paris, Karachi, Moscow, and Madrid. They want the public to tire of the casualties caused by terror in places such as Baghdad, Chechnya, and wherever else they can strike a blow. They want the public to push governments to adopt policies of appeasement, or alternately, to force governments to spend beyond their means and to become increasingly oppressive and draconian towards their own populace. They may see public opinion concerning anti-terrorism policies as an Achilles heel, counting on protracted reaction of protest.
Other attitudinal criteria include (1) negative psychological or behavioral impact of terrorism on a society, (2) loss of public confidence in governments, or in their security measures, (3) the degree to which terrorists are able to radicalize and polarize Islam against the West and vice versa, (4) the level of anti-American or anti-Western sentiments, and (5) the level of religious bigotry in countries which are breeding grounds for terrorists.
Moreover, cultural differences may often also lead to different views concerning violence. Some societies have a warrior tradition and may not necessarily regard peaceful coexistence as a desirable goal. Certain theocracies may regard selective violence as being countenanced by scripture.
Attitudes are central to shaping of consensus, or lack thereof. To the extent that nations can reach a multilateral consensus concerning shared anti-terror strategies, goals and measurement criteria, the United States may be more successful in obtaining the support and assistance of other nations in anti-terror efforts.
section_title: Trends
paragraphs: Trends are changes of incidents, attitudes and other factors, over time. Measurement of trends is particularly relevant with regard to trends in terrorist infrastructure. Is their leadership being weakened; is their recruitment base, network, or target list growing? Relevant also are intentions (tactical and strategic goals). Have the intentions of a movement or group changed and if so are they more or less radical—more or less focused on causing widespread damage? Capabilities are important as well. What are the capabilities of a terrorist group to inflict serious damage? Are they increasing or decreasing?
Other trends that might be measured include are: (1) the number of governments that do not embrace appeasement policies, (2) the number of defectors from the terrorist ranks, (3) the terrorists' levels of Internet activity, (4) the amount of media coverage they receive, and (5) the number of supporters and recruits they gain.
A related issue here is how U.S. policies affect terrorists' popular support and recruiting. Also included here is the degree to which government bureaucratic institutions can work smoothly together, collectively adapting their strategies and tactics to keep up with and stay ahead of the methods utilized by individual terrorists and terrorist networks. Important as well is improvement in states' recovery capabilities and coping skills.
Analysts are quick to point out that the United States is engaged in an ongoing campaign; not a war in the traditional sense. Key here is the ability to sustain a long-term campaign. This takes international cooperation. Trends in international cooperation are important in measuring progress against terrorism. Past threats have generally united America with its allies. Today, the threat of terrorism appears to be dividing America and some traditional allies. Moreover, significant international dissent may signal that future progress will be more difficult and costly.
section_title: Conclusion
paragraphs: Effective responses to terrorism may need to take into account, and to some degree be individually configured to respond to, the evolving goals, strategies, tactics, and operating environment of different terrorist groups. Better understanding of the dynamics of terrorism allows for a more complete picture of the complexities involved in measuring success or failure and can assist the 110 th Congress as it coordinates, funds, and oversees anti-terrorism policy and programs. Although terrorism's complex webs of characteristics—along with the inherent secrecy and compartmentalization of both terrorist organizations and government responses—limit available data, the formulation of practical, useful measurement criteria appears both tractable and ready to be addressed.