Audit Result:

Ss

requirements evolve and impact upon equipment, software, and
operations. Maintaining current documentation will also help
minimize misunderstanding between parties involved in the
development of an IT project. The documentation process also permits
institutional learning to occur.

Corrective Actions Initiated by ATF

ATF has recognized the need to improve its IT process. For example,
a senior OST official believed additional senior level involvement in
very large projects such as the redesigned FTS is beneficial.
Involvement by senior level officials would help to enforce agreements
between developers and users regarding goals and objectives of the IT
projects and what the system should contain.

To help strengthen its IT process, ATF has initiated the following
steps, as outlined in its Information Services Division (ISD) FY 1999
Strategic Action Plan:

e Develop a policy and procedures to define a SDLC.

e Clearly articulate user requirements with an approval by the
division chief or higher level official.

e Document changes to requirements with sign-off at the division
chief or higher level.

e For new projects and the re-design of existing systems, (a) the
client office must appoint a full time project manager with a
supporting team to form a consensus concerning requirements
necessary for the project, and (b) ISD must appoint a full time
project leader.

An ISD official advised us that a policy and related procedures, which
would clearly define a SDLC, had not yet been completed because an
individual had not yet been hired for a key position with
responsibilities for this policy. ATF intends to develop and implement
this policy during FY 2000. We believe this policy should be followed
unless waived by the Assistant Director of OST and other Assistant
Directors involved in the project.

Additionally, the Strategic Action Plan recognized the need to institute
good standard repeatable processes for interaction and ensure good
internal communication and coordination. We believe this action is

OIG-00-093. BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 13


Appendix 4
MAJOR _ CONTRIBUTORS TO THIS REPORT

Chicago Field Office

Roberta N. Rickey Director
Robert K. Bronstrup Audit Manager
Patrick Nadon Auditor
Gary Wilk Auditor
Brad Mosher Auditor

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 34


Appendix 3
MANAGEMENT RESPONSE ©

~-2-

Assistant Inspector General for Audit

FINDING 1: ATF NEEDS TO ASSESS USER NEEDS AND PERFORM
| FORMAL REVIEWS OF INFORMATION TECHNOLOGY (IT)
PROJECTS

RECOMMENDATION: Document and implement a process for (a)
conducting Post-Implementation Reviews (PIRs) on IT
projects.. and (b) defining roles and responsibilities of
individuals who will address identified problems.

We concur with the recommendation. The Office of Science
and Technology (OST) will begin developing a directive for
conducting post implementation reviews on IT projects. This
process will include role and responsibility identification.
The directive will be completed during Fiscal Year 2000.

RECOMMENDATION: Review the results of the OIG survey to
determine what other issues may need to be addressed with
the FTS.

We concur with the recommendation. During this fiscal year,
OST's Software Management Branch (SMB) will work with the
National Tracing Center (NTC) to establish a team consisting
of user representatives from ATF field divisions, along with
the NTC's Tracing and Crime Gun Analysis Branches. The team
will review the survey and make recommendations on other
issues that need to be addressed within the FTS. The
results of this review will be presented to the ATF's
Information Resource Management Council.

RECOMMENDATION: Conduct PIRs on subsequent phases of the
FTS and other IT projects.

We concur with the recommendation. OST will develop
standard processes that include post implementation reviews
of all IT projects. The documentation will be completed
during Fiscal Year 2000. The audit report states that ATF's
Office of Inspection would be performing the PIRs. However,
further review of PIR requirements support OST coordinating
the post implementation reviews in conjunction with the
program offices. The finalized report would then be
submitted to the Office of Inspection.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 30


Audit Results

Information System Life Cycle Manual (TD P 84-01) states a PIR is
conducted to (1) ensure that the system functions as planned and
expected, (2) verify that the system cost is within the estimated
amount, and (3) verify that the intended benefits are derived as
projected. Normally, this one-time review occurs after a major system
implementation or after a major enhancement to a system. Similarly,
the National Institute of Standards and Technology (NIST) recognizes
TD P 84-01 as a source for agencies that use a system life cycle
methodology.

ATF, however, did not perform a formal review after implementation.
ATF did not perform this review because neither the ATF FY 1997-
2001 Operational Information Technology Plan (Plan) nor any other
document included a formal methodology for conducting PIRs. Asa
result, ATF did not allocate funds to perform a formal review of the
FTS after redesign.

An Information Services Division (ISD) official also stated that ATF did
not perform any formal review on the redesigned FTS because it met
deliverables that were finalized one month prior to implementation.
Further, users provided informal feedback to the official that they could
perform more traces with the redesigned FTS.

However, unless the needs of the FTS users are formally assessed,
ATF has less assurance that users are satisfied with the redesigned
FTS. Also, ATF risks not identifying user related issues on future IT
projects. A PIR can identify other areas that can be improved, such as
in planning IT projects and monitoring their implementation. To
illustrate, Findings 2 and 3 of this report discuss issues in ATF’s
planning and monitoring of the redesigned FTS that could also have

been identified in a PIR.

Accordingly, ATF needs to ensure that it implements a PIR process
that follows GAO and other Federal guidance. This guidance
recommends that an agency have a formal review process that clearly
explains and communicates the purpose of the PIR and states when the
PIRs are to be conducted. Additionally, PIRs should: (a) occur on a
regular and timely basis; (b) delineate roles, responsibilities, and
authorities for people and offices involved in conducting the PIRs;

(c) stipulate how conclusions and recommendations resulting from
PIRs are to be communicated to and reviewed by senior management;
and (d) include assessments of customer satisfaction and mission
impact.

Although ATF's Plan did not include a documented methodology for
conducting PIRs, officials recognized the need for them. The Plan

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 7


Audit Results

Performance and Other Reports Not Prepared

ATF employees, including the project team assigned to the FTS
redesign, exchanged considerable electronic mail regarding the project.
This electronic mail showed that employees and contractors recognized
certain risks and issues as the FTS redesign progressed. For example,
the electronic mail showed that:

e The project team, contractor, and FTS users did not always agree
on existing system capabilities and functionality.

e The FTS software was not sufficiently tested prior to being placed
into service in spite of risks to trace reporting.

These risks and other issues that became apparent during redesign
were documented and made available on a "hot list"? to project
participants. Although these issues were recognized, neither the
project manager or other team members prepared performance or other
status reports to show how these issues were addressed in a timely
manner. For example, the project team did not summarize weekly
contractor status reports, annotate task completion, or record risk
mitigation assessments.

Other Monitoring Activities Not Performed

In addition to the project team not preparing performance reports, the
COTR assigned to the project did not perform duties required by the
COTR Handbook for monitoring the contractor’s work.

Specifically the COTR did not: 1) provide technical surveillance,

2) audit project costs, 3) measure contractor work against statements of
work, and 4) monitor progress towards project completion and
financial status.

The COTR did not prepare reports and perform other monitoring
activities because of deficiencies in ATF guidance, information
systems, and management oversight. First, ATF lacked essential
guidance that defined the specific reports, and processes needed to
monitor IT projects. Second, as previously cited, ATF’s information
systems could not generate data on the total cost of IT projects. Third,
the COTR did not prepare other reports or perform required activities
because the COTR believed a supervisor was performing these
activities. The supervisor advised us, however, that he also had not
performed these activities. Senior management did not ensure that the
COTR was performing these functions.

? Also known as Release Notes, a listing of software problems encountered during pre-release development.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 17


Appendix 5
REPORT DISTRIBUTION

U.S. Department of the Treasury

Office of the Under Secretary of the Treasury for Enforcement

Office of the Assistant Secretary of the Treasury for Management/
Chief Financial Officer

Office of Strategic Planning and Evaluations, Departmental Offices
Office of Accounting and Internal Control, Departmental Offices
Office of Budget, Departmental Offices

Bureau of Alcohol, Tobacco and Firearms

Director

Assistant Director, Office of Inspections

Assistant Director, Science & Technology/CIO

Assistant Director, Management/CFO

Assistant Director, Liaison & Public Information
Assistant Director, Alcohol & Tobacco

Assistant Director, Firearms, Explosives & Arson
Assistant Director, Field Operations

Assistant Director, Training & Professional Development
Chief, Acquisitions & Property Management Division
Deputy Chief, Financial Officer/Financial Management Division
YCGII Director

Office of Management and Budget

Treasury Bureau Chief

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 35


EXECUTIVE DIGEST

Audit Results

ATF’s redesigned FTS has generally met users needs according to an
OIG survey. ATF however, needs to develop better policy and
procedures to implement a system development life cycle (SDLC) for
Information Technology (IT) projects. Without such a policy and
related procedures, ATF is less likely to determine whether (1) user
needs have been met on IT projects, and (2) planning and monitoring
of IT projects can be improved. Our review of the redesigned FTS
disclosed that ATF can strengthen its activities in these areas.

First, ATF did not perform a formal PIR to assess user satisfaction
with either the redesigned FTS or the agency’s IT process. As a
consequence of not performing these reviews, ATF increased the risk
that project specific issues would not be identified and resolved
quickly. Although most users are currently satisfied with the
redesigned FTS, 30 percent said the FTS worked without errors only to
some or a little extent immediately after redesign. Also, issues related
to the planning and monitoring of IT projects may not be identified,
thereby increasing the risk that the process may not have been cost
efficient. Further, projects may not meet expectations. Treasury and
other Federal guidance recommends agencies perform formal reviews
to identify and address issues with IT projects and to improve how the
organization selects, manages and uses its IT resources.

Second, although ATF developed a project plan in the early stages of
planning for the redesigned FTS, that plan was not updated to reflect
changes in ATF's new architecture', Asa result, ATF had less
assurance that the redesigned FTS complied with standard industry
practices thereby increasing the risk that the redesign would not
proceed efficiently. General Accounting Office (GAO) and Treasury
guidance states that agencies should ensure, for each IT project, that
the project's cost, schedule, and performance information is kept as
current and as accurate as possible. ATF did not follow the
recommended standard industry practices because of the need to
accelerate implementation of the redesigned FTS and to allow the
redesign to occur within ATF's architecture.

Third, ATF did not have data to show the total cost of the FTS
redesign. Managers also did not have other formal performance and
schedule reports that would have assisted them in monitoring the
project to help them hold employees accountable. Consequently,

' The Clinger-Cohen Act defines information technology architecture as an integrated framework for evolving or
maintaining existing IT and acquiring new IT to achieve the agency's strategic and information resources
management goals.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page ii


Appendix 2
Summary of Questionnaire Responses

17. Overall, to what extent are you satisfied with the redesigned FTS?
(F & G)

Little Or Some Moderate Great Very Great Total Responses
No Extent Extent Extent Extent Extent
2% 12% 26% 44% 16% 128

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 28


Audit Results

emma —

appropriate and should be formalized in a directive that can be
communicated to all parties involved in IT projects. Such action,
combined with our other recommendations, should help ensure the
effective implementation of future IT projects including subsequent
phases of the FTS.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 14


Audit Results

and implementation of our other recommendations will help officials
monitor IT projects more effectively.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 20


Audit Results

Details

In response to our third recommendation, ATF stated that it has taken
action to ensure all identified COTRs have had both the basic and any
appropriate refresher training. When a final version of the previous
OIG report is released, it will become the training portion of the next
Acquisition Branch meeting and the next COTR Refresher Training.

Additionally, an ISD staff member has been assigned to oversee all the
COTRs. The staff member has regular meetings with all the COTRs to
ensure contract compliance providing an additional level of
supervision. In addition, the team of COTRs also has routine meetings
with ATF's Contracting Officer.

We believe the above actions satisfy our recommendations and should
improve monitoring of IT projects including the development of
complete cost and performance reports.

ATF did not know the cost of the FTS redesign. Senior officials
advised us that they had meetings about the FTS redesign. However,
they did not have cost data, formal performance, or other schedule
reports to help them monitor the project and hold the project manager
and other employees accountable for performance. As summarized in
the following sections, neither the project manager nor other members
of the project team prepared these types of reports. Finally, the COTR
did not perform required activities and prepare other reports in
accordance with the requirements of the COTR Handbook.

Project Costs Unknown

Neither the project team, nor the COTR assigned to the FTS redesign
could provide data regarding the total cost of the FTS redesign.
Officials explained cost data was not available because ATF’s
information system could not generate data on the total cost of the IT
project. Accordingly, during the audit, we attempted to determine the
total cost of the FTS redesign. Our examination of the contract
records, YCGII budgetary information, and other informal
documentation shows that ATF had spent between $1.2 to $1.7 million
to implement the redesigned FTS. We were unable to determine,
however, the hardware costs associated with the FTS redesign and
other estimated costs associated with future enhancements that ATF
expects to make to the FTS.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 16


Appendix 3
MANAGEMENT RESPONSE

DEPARTMENT OF THE TREASURY
BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
WASHINGTON, D.C. 20226

| oinecron APR 25 200

903000:TPA:ss
8100

MEMORANDUM FOR: Assistant Inspector General for Audit
FROM: Director

SUBJECT: Draft Audit Report on the Bureau of
Alcohol, Tobacco and Firearms (ATF)
Conversion of the Firearms Tracing System

This serves to respond to your draft audit report dated
March 2, 2000, on the Firearms Tracing System (FTS). The
objective of the audit was to determine if ATF effectively
converted the FTS from a mainframe to a client server
operation. We always welcome this type of review that
results in constructive recommendations for improvement in
the services we provide to the public and efficiencies in
our use of the tax dollar. I wish to compliment you and
your staff for the thorough and professional manner in
which this audit was conducted.

The report is accurate and contains some very constructive
recommendations that will assist ATF in strengthening the
Information Technology (IT) process so that these types of
projects are effectively implemented. The Office of
Inspector General (OIG) audit team in this instance
provided an analysis of their results from a user
satisfaction questionnaire on the FTS, which we find quite
valuable. Your report overall presents three findings,
with a total of nine recommendations. ATF’s responses are
provided below, addressed in the same order as in the
draft. We agree with and will implement all
recommendations.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 29


Audit Results.

a report on the YCGII to the President. These needs combined with
the FTS mainframe deficiencies described on page 2 of this report,
resulted in the decision to accelerate implementation of the redesigned
FTS to April 27, 1998.

Because of the expedited April 27, 1998 implementation date, officials
did not update the project plan and related documentation such as the
requirements document. One official explained that planning
documents were revised but these revisions were discarded as the
redesign progressed. Additionally, planning documents could not be
revised for every change to add or delete a specific function because of
the need to implement the new system quickly.

In addition, a final functional specification document was not prepared
prior to the implementation date because of the need to accelerate the
project. Instead, the final functional specification document was
issued on July 9, 1998 after the redesigned FTS was implemented.
Users advised us that they were unaware the functional specification
document existed until the OIG presented it to them.

As a consequence of the above approach, there was confusion between FTS
users, ISD computer specialists, and contracted employees on what
functionality would be delivered by April 27, 1998. On April 15, 1998,
two weeks before the implementation date an OST official advised other
ATF senior officials that the NTC Director and other employees did not
have a complete understanding and agreement as to what functionality
would be available in the redesigned FTS. Specifically, NTC employees
were concerned that the redesigned FTS did not include the following
functions:

e Batch download process

e National Law Enforcement Telecommunications System
(NLETS) process

e Interstate Theft module
e Error report for supervisors
e Entry data field security

Additionally, NTC employees did not believe that testing had been
conducted in the most effective and efficient manner. To address these
concerns, a senior official had to convene a series of meetings between
all parties as to what action was needed to address these issues by
April 27, 1998.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 11


OBJECTIVES, SCOPE AND METHODOLOGY

The OIG included this review in its Office of Audit Annual Plan for FY
1999, because reducing violent crime committed by young people with
firearms has been a concern to the Administration and to Congress.
This review, is one of three reviews on ATF activities that are
associated with YCGII. The other two reviews focus on

(1) expenditures associated with the YCGII program and (2) efforts at
field offices and headquarters to work with other Federal, State, and
local law enforcement agencies to develop and implement
investigative and prosecutive strategies that target illegal gun
traffickers.

The overall objective of this review was to evaluate ATF’s redesigned
FTS. Our specific objectives were to determine whether ATF: (1) met
users needs in the redesigned FTS and conducted PIRs; (2) adequately
planned the redesign of the FTS, and (3) adequately monitored the
implementation of the project.

To accomplish our review, we conducted work at ATF Headquarters
where we interviewed officials from (1) OST and (2) FEA, which
includes the National Tracing Center. We also interviewed officials
from ATF’s Acquisitions Division in the Office of Management and
reviewed contracting files related to the redesigned FTS. We sent a
questionnaire to ATF and contract employees who use the FTS to help
us assess how well the redesigned FTS is working. Specifically, we
surveyed two groups of users that we consider "general" and
"functional" users. General users are data entry personnel, ATF agents
and other employees that use the FTS for investigations. Functional
users are usually supervisors, program analysts, computer specialist or
other system analysts who understand the functionality of the FTS.

Audit work was limited to the redesigned FTS. Our fieldwork was
performed from January 1999 through August 1999. Our work was
conducted in accordance with Government Auditing Standards issued
by the Comptroller General of the United States, and included such
audit tests as we determined necessary.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 3


Appendix 2
Summary of Questionnaire Responses

12. Were you involved in the FTS redesign? (F & G)

To determine user requirements?
Yes No Total Responses
11% 39% 133

In designing the system?
Yes No Total Responses

4% 96% 130

In testing the system?
Yes No Total Responses
20% 80% 132

To identify needed modifications?
Yes No Total Responses
20% 80% 132

13. Regardless of whether you were directly involved in redesigning the FTS, were you
provided the opportunity to express your needs or make recommendations to improve
the system before the conversion to either ATF management or the contractors who
were responsible for redesigning the FTS? (F & G)

Yes No Total Responses
52% 48% 133

14, Were all of the recommendations you proposed for the redesigned FTS incorporated?
(F& G

Yes : No Did Not Make Total
Recommendations Responses
21% 12% 67% 129

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 26


Appendix 2
Summary of Questionnaire Responses

2. To what extent is the FTS important to accomplishing your duties?
(F & G)

Little Or Some Moderate Great Very Great Total
No Extent Extent Extent Extent Extent Responses
9% 6% 12% 15% 58% 137

3. For what purpose do you use the FTS? (F & G)
To enter new FTS information?
Yes No Total Responses.

34% 46% 120

To update FTS information?
Yes No Total Responses
56% 44% 122

To view FTS information?
Yes No Total Responses
95% 5% 128

To perform maintenance on decode tables?
Yes No Total Responses

5% 95% 91

4. Did you participate in a PIR or any other studies conducted to show the success of the
FTS conversion? (F)

Yes No Total Responses

8% 92% 12

OIG-00-093 | BUREAU OF ALCOHOL, TOBACCO AND FIREARMS

CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 23


TABLE OF CONTENTS

EXECUTIVE DIGEST ....sescssssesssssssessessneseessnecsssieesssecssusecssssesssnecessvessssssaee

BACKGROUND

OBJECTIVES, SCOPE AND METHODOLOGY.

AUDIT RESULTS
Finding 1 ATF Needs to Assess User Needs and Perform Formal
Reviews Of IT Projects... ssssessessseessesssesseessecsesssessseessseesseeese
Finding 2 Planning Documents Should be Maintained and Kept Current

Finding 3 ATF Needs to Track IT Project Costs and Develop

APPENDICES

Appendix 1 Abbreviations...
Appendix 2 Summary of Questionnaire Responses........c.ccssssessssessessseeessessseeeseee
Appendix 3. Management Response Dated April 25, 2000..
Appendix 4 Major Contributors To This Report .......csesscsssesssesseessecssseessessseseees

Appendix 5 Report Distribution .......

Performance Reports........ssesessesssessssesseecseessessesssecsseesssesssneees

22

22D

34

OIG-00-093

BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page iv


EXECUTIVE DIGEST

management had less assurance that the project was implemented at
acceptable costs, within reasonable timeframes and included all
expected benefits. Federal statutes, Office of Management and Budget
(OMB) directives, and GAO guidance, require that agencies have a
defined, documented and repeatable process for monitoring IT
projects. The project team and the Contracting Officer’s Technical
Representative (COTR) did not prepare the reports and other analyses
because of inadequacies in ATF procedures and information systems.
Additionally, officials did not ensure that the COTR performed other
required duties.

Recommendations and Management Response

Our draft report, issued on March 2, 2000, made nine
recommendations to enhance ATF's oversight of IT projects. ATF
agreed with and will implement all recommendations. The details of
ATF's responses and actions initiated to address our findings and
recommendations are contained in the body of this report. We believe
that implementation of these recommendations will help strengthen
ATF's procedures and controls over current and future IT projects.

OIG-00-093

BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page iii


Audi

t Results

As described previously in our prior chapter, users had concerns about
the accuracy and efficiency of information immediately after
implementation. These issues could have been minimized had ATF
kept project documentation current, combined with sufficient reviews
and testing as the redesign effort progressed. As a result of not
revising or maintaining key planning documents, ATF had less
assurance that the redesign complied with standard industry practices.
This lack of assurance also increased the risk that the redesign did not
proceed in the most efficient manner possible.

Industry practices and TD P 84-01 recognize the need for entities to
maintain updated project file documentation and perform reviews after
each project phase to ensure each phase is completed successfully.
This process ensures that all products created during the life cycle meet
functional and performance requirements as outlined in all
requirements documentation. Current project file data together with
formal reviews can help to:

(1) ensure that project direction and goals remain consistent with the
organization's strategic plan and goals;

(2) provide an opportunity to terminate projects which fail to
demonstrate an adequate return on investment;

(3) measure the ongoing progress (i.e., budget, schedule and
deliverables) and identify potential problems for corrective actions;
and

(4) approve phase results and authorize further work.

As noted by GAO guidance and recent studies on other organizations'
IT processes, information in the project plan and related

documentation should be as up to date and as accurate as possible in
order for management reviews and analyses to be effective. In general,
senior management involvement should be greatest in the earlier
phases of systems development efforts.

Standard industry practices also dictate that the functional specification
document be provided to users and other parties prior to
implementation. This action is necessary because the document
specifies information about the (1) system requirements,

(2) operating environment, (3) design characteristics and (4) system
specifications for the redesigned FTS.

Although not every change to a system's requirements may be
documented due to time constraints or other factors, sufficient
documentation should be retained to show how the system

OIG-00-093

BUREAU OF ALCOHOL, TOBACCO AND FIREARMS

CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 12


BACKGRO

UND

As the volume of firearm traces increased in the mid 1990s, officials
recognized that the existing FTS, operating on an older mainframe
database, had the following deficiencies:

e Limited query capabilities

e Slow response times

e Frequent system breakdowns
e Lack of Year 2000 compliance

Accordingly, officials recognized a need to redesign the FTS and
eliminate the deficiencies. Additionally, ATF recognized a need for an
Enterprise Systems Architecture (ESA) which is a mix of hardware
and software to form a platform on which a suite of continually
evolving application services would be installed to support all ATF
activities. As part of this overall effort, ATF initiated a project to
redesign and improve the FTS while maintaining the same
functionality. The redesign was to decrease response times and
increase system reliability and improve query capabilities. New
features were also to be added to allow users to use the FTS more
efficiently. To accomplish this goal, the redesigned FTS was to
migrate the existing mainframe system to a relational database with a
Windows interface in a client-server environment.

The proposed redesigned FTS also was to provide a central database
for all trace-related information. In summary, the improvements to the
FTS would address the deficiencies listed above. A project team,
consisting of contractors working with employees from the Office of
Science and Technology (OST) and the Office of Firearms, Explosives
and Arson (FEA) implemented the redesigned FTS during 1997 and
1998. The redesigned FTS went online April 27, 1998.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 2


Audit Results

Without reports that track cost, schedule and performance data, senior
officials and the project team are less likely to identify issues quickly
and take timely corrective action. Officials will not have sufficient
information to show that the project is progressing at the lowest cost
and within expected timeframes. Additionally, they have less
assurance that the project includes all of the functions that were
planned for the project. Finally, management will be less able to hold
project managers and other employees accountable for meeting cost,
schedule and performance goals. Addressing these issues is important
because of the large number of IT projects that OST is working on.
OST officials advised us that they currently have 40 IT projects in
process.

Federal Statutes Require Performance, Schedule and Cost Analyses

To help ensure that Federal agencies monitor IT projects more
effectively, Congress has enacted several statutes during the 1990’s.
These statutes included (1) the Government Performance and Results
Act of 1993, (2) the Federal Acquisition Streamlining Act of 1994
(FASA), (3) the Paperwork Reduction Act of 1995 (PRA), and

(4) the Clinger Cohen Act of 1996. The statutes: reinforce financial
accountability; emphasize results oriented management; define cost
performance and schedule goals; and improve the acquisition of IT to
streamline federal programs. To illustrate, FASA requires agencies to
establish and track major acquisitions against cost, schedule, and
performance goals. Also, the head of each civilian agency is required
to approve or define the cost, performance, and schedule goals for
major acquisition programs of the agency.

To assist Federal agencies in implementing these statutes, OMB has
issued circulars and other guidance. OMB Circular A-130 (guidance
resulting from the PRA), for example, states that agencies must
perform various benefit-cost analyses to support management
oversight. These analyses are used to assess and document actual
versus estimated costs and benefits, identify causes for discrepancies,
and learn how to manage IT more effectively in the future.

Recent GAO guidance also describes how each agency can establish a
defined, documented, and repeatable process for monitoring and
reviewing IT projects. The GAO guidance states that project data
should consist of such items as comparisons of actual results achieved
to date versus estimates, and an assessment of benefits achieved.

Leading organizations that GAO studied also hold project managers
accountable for meeting cost, schedule, and performance goals. Some
organizations allow individual project managers to decide what

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 18


Final Report on the
Bureau of Alcohol, Tobacco and Firearms
Conversion of the Firearms Tracing System

OIG-00-093 June 16, 2000

This report has been reviewed for public dissemination by the Office of Counsel to

the Inspector General. Information requiring protection from public dissemination
has been redacted from this report in accordance with the Freedom of Information
Act, 5 U.S.C. section 552.

Office of Inspector General

sea aa

United States Department of the Treasury


Audit Results

Details

Gun Analysis Branches. The team will review the survey and make
recommendations on other issues that need to be addressed within the
FTS. The results of this review will be presented to the ATF's
Information Resource Management Council.

ATF management agreed with our third recommendation. OST will
develop standard processes that include post implementation reviews
of all IT projects. The documentation will be completed during FY
2000. ATF management stated that their further review of PIR
requirements supports OST coordination of post implementation
reviews in conjunction with the program offices. The finalized report
would then be submitted to the Office of Inspection.

We believe the above actions meet the intent of our recommendations
and should help ATF develop procedures to perform formal reviews of
IT projects.

ATF made no formal assessment after the implementation of the
redesigned FTS to determine whether user needs had been met.
Accordingly, to help us determine whether users needs were met, we
surveyed users to determine their satisfaction with the redesigned FTS.
The following chart summarizes current user satisfaction with the FTS.
(The entire questionnaire is summarized in Appendix 2)

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 5


Audit Results

Finding 1. ATF Needs to Assess User Needs and Perform Formal
Reviews of IT Projects

ATF had not performed a formal PIR of the redesigned FTS to assess
user satisfaction with either the redesigned FTS or the agency’s IT
process. As a consequence of not performing these reviews, the risk
that project specific issues will not be identified and resolved quickly
were increased. Although most users are currently satisfied with the
redesigned FTS, 30 percent said the FTS worked without errors only to
some or a little extent immediately after redesign. Also, issues related
to the planning and monitoring of IT projects may not be identified
thereby increasing the risk that the process will not be cost efficient. In
addition, further projects may not meet expectations. Treasury and
other Federal guidance requires agencies to perform formal reviews to
identify and address issues with IT projects and to improve how the
organization selects, manages and uses its IT resources.

Recommendations
We recommend the ATF Director ensures that officials:

1) Document and implement a process for (a) conducting PIRs on IT
projects using a standard methodology such as that outlined by the
General Accounting Office (GAO) and (b) defining the roles and
responsibilities of individuals who will address identified
problems.

2) Review the results of the OIG survey to determine what other
issues may need to be addressed with the FTS.

3) Conduct PIRs on subsequent phases of the FTS and other IT
projects.

Management Response and OIG Comments

ATF management agreed with the first recommendation and the OST
will begin developing a directive for conducting post implementation
reviews on IT projects. This process will include role and
responsibility identification. The directive will be completed during
FY 2000.

ATF management agreed with the second recommendation. During
this fiscal year, OST's Software Management Branch (SMB) will work
with the NTC to establish a team consisting of user representatives
from ATF field divisions, along with the NTC's Tracing and Crime

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 4


. Appendix 2
Summary of Questionnaire Responses
8. Are there any security deficiencies in the redesigned FTS? (F)
Yes No Total Responses
61% 33% 9

9. Are you able to perform the same functions now with the redesigned FTS, as you did
with the prior FTS operating on the mainframe? (F)

Yes No Total Responses

60% 40% 10

10. Were you involved in the testing of the redesigned FTS? (F & G)
Yes No Total Responses

22% 8% 136

If you were involved was the overall testing of the redesigned FTS system adequate?
Yes No Total Responses

80% 20% 30

11. Did you receive training on how to use the redesigned FTS? (F & G)
Yes No Total Responses
67% 33% 136

If you received training on the redesigned FTS, to what extent do you think it was
adequate?

Little Or Some Moderate Great Very Great Total
No Extent Extent Extent Extent Extent Responses
1% 11% 37% 36% 15% 87

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 25


Appendix 2
Summary of Questionnaire Responses

Using the following scale, please rate from I to 5 the extent of the information you
received from the redesigned FTS, when it was first put into production and today. (F
& G)

(5 = To a very great extent; 4 = To a great extent; 3 = To a moderate extent; 2 = To some extent; 1 = To little or
no extent)

immediately after

redesign 5 4 3 2 1 Total
Response

Complete 10% 23% 39% 17% 12% 101
Accurate 11% 25% 38% 15% 11% 102
Timely 10% 17% 39% 22% 12% 104
Currently 5 4 3 2 1

Complete 27% 45% 19% 1% 8% 7
Accurate 31% 46% 14% 2% TM% 118
Timely 26% 42% 23% 3% 6% 118

6. Did you observe any errors immediately? (F & G)
Yes No Total Responses

31% 69% 126

If you answered yes, how frequently did you observe errors?

Hourly Daily - Weekly Monthly Total
Responses
6% 39% 33% 22% 36

7. To what extent does the redesigned FTS provide the information you need to
accomplish your duties? (F & G)

Little Or Some Moderate Great Very Great Not Total
No Extent Extent Extent Extent Extent Applicable
Responses
2% 11% 13% 30% 37% TMM 136

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 24


BACKGROUND

The Bureau of the Alcohol, Tobacco and Firearms (ATF) is
responsible for the enforcement of the Gun Control Act and is the sole
agency responsible for tracing firearms used in crimes and recovered at
crime scenes. Toward that end, ATF has established the National

Tracing Center (NTC),
Firearms tracing 1s described as the systematic tracking of

irearms from manufacturer to purchaser for the purpose of aiding law
enforcement officials in (1) identifying suspects involved in criminal
violations, (2) establishing stolen status, and
(3) providing ownership.

The NTC receives trace requests from law enforcement agencies and
from ATF field offices through letters, faxes, telephone calls, the
National Law Enforcement Telecommunications System (NLETS) and
other electronic means. Data entry personnel then enter trace
information into the Firearms Tracing System (FTS). The collection of
information concerning thousands of trace requests and the NTC trace
response to those requests represent the core memory of the FTS.

The following summarizes the traces that have been processed by the
FTS during FYs 1996 through 1998:

Volume of Firearm Traces
Fiscal Year Number of Traces Performed

1996 114,223
1997 189,197
1998 189,482

In addition to its primary goal, ATF recognizes that a systematic
analysis of the FTS data can assist law enforcement agencies in
regulating firearms trafficking. ATF must often contact the Federal
Firearms Licensee (FFL) when conducting traces. When these
contacts are recorded as data in the FTS, the development of certain
patterns over time can be identified. Patterns of responses by FFLs to
trace requests and also patterns of purchasing, firearm recoveries, and
time intervals, can help identify illegal firearms trafficking. The cities
that are a part of Youth Crime Gun Interdiction Initiative (YCGII) are
to use this data to initiate investigations against criminals illegally
transferring firearms to juveniles, adult criminals and other prohibited
persons.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 1


Audit Results

Finding 3. ATF Needs to Track IT Project Costs and Develop
Performance Reports

ATF did not have data to determine the total cost of the FTS redesign.
Managers also did not have formal performance and schedule reports
that would assist them in monitoring the project to help them hold
employees accountable. Consequently, management had less
assurance that the project was implemented at acceptable costs, within
reasonable timeframes and included all expected benefits. Federal
statutes, and implementing OMB directives and GAO guidance,
require that agencies have a defined, documented and repeatable
process for monitoring IT projects. The project team and the COTR
did not prepare the reports and other analyses because of inadequacies
in ATF's procedures and information systems. Additionally, officials
did not ensure that the COTR performed other required duties.

Recommendations
We recommend the Director of ATF ensure that officials:

1. Establish and implement directives on monitoring IT project costs,
schedule and performance.

2. Hold the project manager(s) accountable for addressing issues and
risks that arise during the project.

w

Adequately supervise the COTR for compliance with the
responsibilities listed in the COTR Handbook.

Management Response and OIG Comments

ATF management concurred with our first recommendation and the
ISD is currently working on developing a methodology for monitoring
IT project costs, scheduling, and performance. OST has solicited
participation of program offices in monitoring IT projects. Cost,
schedule and performance monitoring will be implemented during FY
2001.

ATF management agreed with our second recommendation and stated
that as part of the new pay banding demonstration project, each project
manager will be accountable for, among other things, addressing all
issues and risks that arise during a project and will be evaluated on
their performance in this area.

OIG-00-093 | BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 15


Appendix 3
MANAGEMENT RESPONSE

-4-

Assistant Inspector General for Audit

FINDING 3: ATF NEEDS TO TRACK IT PROJECT COSTS AND DEVELOP
PERFORMANCE REPORTS

RECOMMENDATION: Establish and implement directives on
monitoring IT project costs, schedule, and performance.

We concur with the recommendation. The Information Services
Division (ISD) is currently working on developing a
methodology for monitoring IT project costs, scheduling, and
performance. This cost, schedule, and performance
monitoring will be implemented during Fiscal Year 2001. OST
has solicited the involvement of program offices in |
monitoring IT projects. To this end, we have reviewed all
new statements of work, and in some cases rewritten many so
that expectations of performance and scheduling of product
delivery are more clearly stated.

RECOMMENDATION: Hold project manager(s) accountable for
addressing issues and risks that arise during the project.

We concur with the recommendation. As part of the new pay
banding demonstration project, each project manager will be
accountable for, among other things, addressing all issues
and risks that arise during a project and will be evaluated
on their performance in this area.

RECOMMENDATION: Adequately supervise the COTR for
compliance with the COTR Handbook.

We concur with the recommendation. The OIG's concern about
contracting officers' technical representatives' (COTR)
adherence to the COTR Handbook was raised in an earlier
audit. As indicated in ATF's previous audit response, we
have taken action to ensure all identified COTRs have had
both the basic and any appropriate refresher training.
When the final version of the previous OIG report is
released, it will become the training portion of the next
Acquisition Branch meeting and the next COTR Refresher
Training. Relevant information from the final version of
the report will be provided to all ATF contracting
officers, procurement analysts, and COTRs.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM 5
Page 3


Appendix 3
MANAGEMENT RESPONSE

-5-

Assistant Inspector General for Audit

(Pinding Number Three Cont‘d)

Additionally, an ISD staff member has been assigned to
oversee all the COTRs. The staff member has regular
meetings with all the COTRs to ensure contract compliance
providing an additional level of supervision. In addition,
the team of COTRs also has routine meetings with ATF’s
Contracting Officer.

Again, I appreciate the thorough work of your team, and the
opportunity to provide comment on their findings. Should
you need any further information concerning the contents of
this document, please contact Terrence Austin, YCGII
Director, at (202) 927-8425 to facilitate your request.

['ehdetag @: [chlor —

Bradley A. Buckles

OIG-00-093. BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 33


Appendix |

ABBREVIATIONS _
ATF Bureau of Alcohol, Tobacco and Firearms
COTR Contracting Officer's Technical Representative
ESA Enterprise System Architecture

FASA Federal Acquisition Streamlining Act of 1994
FEA Office of Firearms, Explosives and Arson

FFL Federal Firearms Licensee

FTS Firearms Tracing System

FY Fiscal Year

GAO General Accounting Office

ISD Information Services Division

IT Information Technology

NLETS National Law Enforcement Telecommunications System
NTC National Tracing Center

OIG Office of Inspector General

OMB Office of Management and Budget

OST Office of Science and Technology

PIR Post-Implementation Review

Plan Operational Information Technology Plan
PRA Paperwork Reduction Act of 1995

SDLC System Development Life Cycle

SMB Software Management Branch

YCGII Youth Crime Gun Interdiction Initiative
OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS

CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 21


EXECUTIVE DIGEST

Overview

In July 1996, the Bureau of Alcohol, Tobacco and Firearms (ATF), at
the direction of the President, initiated the Youth Crime Gun
Interdiction Initiative (YCGID) to target gun traffickers who supply
firearms to young people. Studies showed that homicides committed
by youths with firearms had nearly tripled since 1985. To help
implement YCGII, ATF uses its Firearms Tracing System (FTS) to
trace firearms recovered in crimes in cities participating in this
initiative. This trace information provides ATF and other law
enforcement agencies with crucial investigative leads to target gun
traffickers supplying firearms to youths. This report focuses on ATF’s
efforts to redesign the FTS, which had developed several operating
deficiencies.

The Office of Inspector General (OIG) included a review of YCGII in
its Office of Audit Annual Plan for Fiscal Year (FY) 1999 because the
Administration and Congress are committed to reducing violent crime
committed by young people with firearms. This review of the FTS is
one in a series of three reviews that will cover ATF activities
associated with YCGII. The two other reviews focus on

(1) expenditures associated with YCGII and (2) results that YCGII has
achieved.

Objectives, Scope and Methodology

The overall objective of this review was to evaluate ATF’s redesigned
FTS. Our specific objectives were to determine whether ATF (1) met
users needs in the redesigned FTS and conducted post-implementation
reviews (PIR), (2) adequately planned the FTS redesign, and (3)
adequately monitored the implementation of the project.

To accomplish these objectives we reviewed and evaluated project
files related to the redesigned FTS. We also discussed the redesigned
FTS with officials and employees at ATF’s National Tracing Center
(NTC), Office of Science and Technology (OST), and the Acquisitions
Division in the Office of Management. Our review generally covered
the redesign efforts during FY 1997 and 1998. The redesigned FTS
went online April 27, 1998. Finally, we surveyed employees of ATF
who use the FTS to help us assess how well it is operating.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Pagei


Audit Results .

stated a review process should include a PIR because the benefits of a
PIR can determine what problems that ATF can avoid in future IT
projects. In response to our draft audit report, ATF stated that OST
will coordinate PIRs in conjunction with program offices. This action,
combined with our other recommendation, should help ensure a more
effective PIR process for IT projects.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 8


Audit Results

Details

performance expectation of future changes to the FTS and related
systems.

ATF management agreed with our third recommendation. The OST
will develop and implement a SDLC in FY 2001. Steps have already
been taken to improve involvement by client managers from the
appropriate program areas.

We believe the above actions satisfy our recommendations and should
ensure that planning and other key documents for IT projects are
maintained and kept current.

As early as 1995, ATF produced key project plan documentation to
redesign the FTS. ATF had prepared the following specific documents
for this effort:

= Business Process Reengineering of the Firearms Tracing Process
= Reengineering Alternatives Evaluation

= FTS Functional and Technical Assessment

* Requirements Document

These documents showed that ATF had considered various
assumptions, alternatives, costs, benefits and risks for redesigning the
FTS.

Subsequently, ATF's focus was redirected to modernizing all of ATF's
computer systems with an Enterprise System Architecture (ESA).
Specifically, in FY 1996, ATF developed a plan for the acquisition and
deployment of infrastructure equipment, integrated networks, and
operating and application software. This plan was initiated so ATF
could (1) achieve the goals of its core business strategies and (2) form
the foundation of an architectural IT vision into the next millennium.

As a result of this redirection, the FTS project plan and related
documentation developed during 1995-96 were no longer applicable to
ATF's efforts to redesign the FTS. Also, there were changes with ATF
officials and other key employees who had been assigned to the
project. Additionally, another contractor replaced the contractor who
initially worked on the project.

During this period of time, officials continued to recognize the need to
redesign the FTS. The FTS was running out of trace numbers to assign
to trace requests that were received from law enforcement agencies and
ATF field offices. Also, ATF had to produce accurate information for

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 10


Page 2

have been incorporated into the report and included in their
entirety as an appendix to the report.

We appreciate the courtesies and cooperation provided to our
staff during the audit. If you wish to discuss this report,
you may contact me at (202) 927-5400 or a member of your
staff may contact Ms. Roberta N. Rickey, Director, Program
Audit, at (312) 886-6300.

Attachment

cc: James E. Johnson, Under Secretary of the Treasury
for Enforcement

Richard Hankinson, Assistant Director
Office of Inspection


Appendix 3
MANAGEMENT RESPONSE

-3-

Assistant Inspector General for Audit

FINDING 2: PLANNING DOCUMENTS SHOULD BE MAINTAINED AND
KEPT CURRENT

RECOMMENDATION: Planning documentation for each IT project
be kept as current and as accurate as possible.

We concur with the recommendation. The SMB will ensure
that planning documentation for each IT project will be kept
as current and as accurate as possible. To ensure updated
and accurate documentation, a group consisting of users and
IT personnel will convene regularly as a formal
configuration management review team.

This team will submit requested changes to senior management
and to the Information Services Division (ISD) Enterprise
Configuration Management Control (ECMC) team. In addition,
application documentation will be included within the
Polytron Version Control System (PVCS) automated version
control management system.

| RECOMMENDATION: A senior official..is involved in future
| FTS phases and other major IT projects.

We concur with the recommendation. ISD has recently hired a
Software Management Branch Chief to help ensure that a
senior official at the appropriate level is involved in any
future significant IT projects. Currently, the Division
Chief, National Tracing Center; Director, Youth Crime Gun
Interdiction Initiative; and the Division Chief, Information
Services are jointly involved in discussions of
requirements, production delivery schedules, and performance
| expectations of future changes to the FTS and related
systems.

RECOMMENDATION: OST officials complete actions to develop
a policy for a System Development Life Cycle (SDLC). This
action should include...

We concur with the recommendation. OST will complete the
development of the System Development Life Cycle and
implement in Fiscal Year 2001. As noted in your report,
steps have already been taken to improve involvement by
client managers from the appropriate program areas.

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 31


Audit Results

management tools best meet their needs to monitor and track project
milestones and to identify cost and schedule variances from the project
plan. Given that flexibility, however, senior officials still need to
ensure that the project manager and other employees are held
accountable for adequately addressing issues that arise during the
implementation of IT projects. As part of that effort, officials also
need to ensure that the COTR receives sufficient direction from senior
management and follows the requirements in the COTR Handbook.

We discussed this issue with senior officials and they agreed that
improved reporting and data would enhance their monitoring of IT
projects. For example, the Deputy Assistant Director for FEA stated
that enhanced status and cost reports would assist senior officials in
monitoring IT projects, including future enhancements to the FTS. To
illustrate, upon implementation of the FTS redesign on April 27, 1998,
various other functions had not yet been included in the redesigned
FTS. The project team continued to work on FTS functions such as
the (1) FFL and Interstate Theft, (2) Queries Subsystem, and (3) Batch
Download Automation. By October 1, 1998, the project team had
implemented these functions in the FTS. Status reports to senior
officials would have helped ensure these enhancements progressed
timely. Such reports would also help to ensure future efforts to
incorporate security controls into the FTS are satisfactorily completed.
At the time we completed fieldwork, security controls had not been
fully incorporated.

Corrective Actions Initiated by ATF

During our review, the Director of OST advised us that ATF plans
include the following actions:

1. OST and the program offices will both be required to sign the
requisitions for the projects to approve the work and the funding;

2. ATF’s Information Resource Management Council will have
financial reports for its meetings that will show a project’s cost for
the past two years, current year, and future two years. The
financial reports will include detailed listings of requisitions for the
projects.

3. OST is soliciting the involvement of the program offices in the
monitoring of contractor performance.

These planned actions are appropriate; however, ATF should also
ensure that the new ATF financial system provides the data needed to
track total costs of IT projects. Completion of these proposed actions

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 19


DEPARTMENT OF THE TREASURY

WASHINGTON, D.c. 20220
June 16, 2000

me

OFFICE OF
INSPECTOR GENERAL

MEMORANDUM FOR BRADLEY A. BUCKLES, DIRECTOR
BUREAU OF ALCOHOL, ria. AND aa

FROM: Dennis S. Schindel bpm,
Assistant Inspector General for Audit

SUBJECT: Final Audit Report on the Bureau of Alcohol,
Tobacco and Firearms Conversion of the
Firearms Tracing System

This memorandum transmits the final report on the Office of
Inspector General's audit of the Bureau of Alcohol, Tobacco
and Firearms’ (ATF) Conversion of the Firearms Tracing
System (FTS). Our overall objective was to determine
whether ATF effectively converted the FTS from a mainframe
operation to a client server operation.

Our review showed that the converted FTS appears to be
meeting users needs; however, the conversion did not take
place in the most efficient manner. Our review showed that
ATF needs to:

e Develop procedures to perform formal reviews of
Information Technology (IT) projects;

e Ensure that planning and other key documents for IT
projects are maintained and kept current; and

e Improve monitoring of IT projects including the
development of complete cost and performance reports.

As a consequence of these conditions, there is an increased
risk that inefficiencies could occur in subsequent phases to
further improve the FTS or in other IT projects. Asa
result, projects may not be completed in a cost-effective
manner. Additionally, the projects may not fully achieve
their intended benefits.

Our draft report made nine recommendations that will help
strengthen the IT process. In ATF's response to the draft
report, ATF concurred with, and has begun or planned actions
to address our recommendations. ATF's official comments


Appendix 2
Summary of Questionnaire Responses

15. To what extent does the redesigned FTS meet those requirements defined in the
functional specification document? (F)

Little Or Some Moderate Great Very Great No Basis Total
No Extent Extent Extent Extent Extent to Judge Responses
0% 8% 17% 17% 0% 58% 12

16. Using the following scale, please rate from I to 5 the adequacy of system functions for
the redesigned FTS, when it was first put into production and today. (F & G)

(5 = To a very great extent; 4 = To a great extent; 3 = To a moderate extent; 2 = To some extent; | = To
little or no extent)

Immediately after redesign 5 4 3 2 1 Total
Responses

Screens are easy to

understand and use 16% 36% 27% 16% 5% 102

System works quickly

and efficiently % 30% 34% 21% 6% 102

System works without errors TM% 32% 32% 22% 8% 101

Online messages are

clearly presented TM% 42% 31% 16% 3% 97

Online messages adequately

explain commands and concepis TM% 37% 36% 14% 5% 97

Currently 5 ee 3 2 L Total
Responses

Screens are easy to

understand and use 35% 48% 13% 3% 0% 120

System works quickly

and efficiently 16% 53% 26% 5% 1% 121

System works without errors 16% 56% 21% TM% 1% 120

Online messages are
clearly presented 17% 58% 21% 3% 1% 116

Online messages adequately
explain commands and concepts 171% 57% 22% 3% 1% 116

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 27


: Appendix 2
Summary of Questionnaire Responses

The questionnaire on the ATF's redesigned FTS was sent to 299 FTS users. We received
questionnaire responses from 137 users, which represents 46 percent of those surveyed. The
questionnaire participants did not provide responses to all the questions and for some
questions they provided more than one response. As a result, the percentage shown in each
question may not total to 100 percent. In addition, we surveyed two groups of users that we
consider as "general" and "functional" users. General users are FTS data entry personnel and
ATF agents and employees that use the FTS for investigations. Functional users are defined
as users that know the functionality of the FTS, usually supervisors, program analysts,
computer specialists or system analysts.

The following abbreviations are presented with all survey questions to identify which user
group received each question:

e Functional User Group (F)

e General UserGroup (G)

QUESTIONNAIRE RESULTS
1, What FTS subsystem(s) do you use and how frequently? (F & G)
FTS Subsystem:

Sub- Multiple Suspect _ Interstate ce “Decode Total
system: Trace Sales Guns Theft FFL Theft Tables :

: : Responses
Users: 1% 59% 28% 16% 19% 20% 129

How Frequently:

Sub- ‘Multiple SuspectGuns Interstate : Decode —
system: Trace Sales Theft FFL Theft Tables
Daily 67% 58% 33% 43% 42% 62%
Weekly 23% 25% 28% 24% yi 19%

Monthly 10% 17% 39% 33% 38% 19%

OIG-00-093 BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM
Page 22


Audit Results

CHART 1
Current User Satisfaction with the FTS

OTo some or no —&
extent (14%)

OTo a moderate
extent (26%)

OTo a great and
very great
extent (60%)

Source: OIG Questionnaire responses

As the chart shows, about 60 percent of all users are currently satisfied
with the FTS to a great or very great extent. (See Question 17 of the
Summary of Questionnaire Responses in Appendix 2.)

Specific elements of the redesigned FTS, such as the ability to
understand and use screens, were also viewed favorably by the
respondents. A PIR can help show current user satisfaction with an IT
project like the FTS redesign. A PIR can also help identify issues
more quickly so officials can address them. For example, immediately
after redesign: (See Question 16 of the Summary of Questionnaire
Responses in Appendix 2.)

e 30 percent of the respondents stated the system worked without
errors only to some or to a little extent. An additional 32 percent
stated the system worked without errors to only a moderate extent.

e 27 percent of the respondents stated that the system worked
quickly and efficiently only to some or to a little extent. An
additional 34 percent said the system worked quickly and
efficiently only to a moderate extent.

The value of a formal PIR to assess issues in a timely manner, like
those cited above, has been recognized by Treasury and GAO guidance
and also by private industry. Specifically, Treasury Directive on the

OIG-00-093 © BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 6


Audit Results

Finding 2.

Planning Documents Should Be Maintained and Kept
Current

Although ATF developed a project plan in the early stages of planning
for the redesigned FTS, the plan was not updated to reflect changes in
ATF's new architecture. As a result, ATF had less assurance that the
redesigned FTS complied with standard industry practices thereby
increasing the risk that the redesign did not proceed efficiently. GAO
and Treasury guidance states that agencies should ensure, for each IT
project, that the project's cost, schedule, and performance information
be kept as current and as accurate as possible. ATF did not follow the
recommended standard industry practices because of the need to
accelerate implementation of the redesigned FTS and to allow the
redesign to occur on a new platform.

Recommendations

We recommend the ATF Director ensure that:

1. Planning documentation for each IT project be kept as current and
as accurate as possible.

2. A senior official at the appropriate level is involved in future FTS
phases and other major IT projects.

3. OST officials complete actions to develop a policy for a System
Development Life Cycle (SDLC). This action should include
procedures to implement the SDLC with standard processes for
interaction and good internal communication and coordination.

Management Response and OIG Comments

ATF management concurred with our first recommendation and the
SMB will ensure that planning documentation for each IT project will
be kept as current and as accurate as possible. To ensure updated and
accurate documentation, a group consisting of users and IT personnel
will convene regularly as a formal configuration management review
team.

ATF management agreed with our second recommendation and stated
that ISD has recently hired a SMB chief to help ensure that a senior
official at the appropriate level is involved in any future significant IT
projects. Currently, the Division Chief, NTC; Director, YCGII; and
the Division Chief, Information Services are jointly involved in
discussions of requirements, production delivery schedules, and

OIG-00-093 |= BUREAU OF ALCOHOL, TOBACCO AND FIREARMS
CONVERSION OF THE FIREARMS TRACING SYSTEM

Page 9