Office of Research and 

Development 

Washington, DC 20590 

Federal Railroad

Administration

U.S. Department

of Transportation

Quantitative Nondestructive Testing of 

Railroad Tank Cars Using the Probability 

of Detection Evaluation Approach

DOT/FRA/ORD-09/10 May 2009 

 Final Report 

This document is available to the

U.S. public through the National

Technical Information Service

Springfield, VA 22161.
Notice 

This document is disseminated under the sponsorship of the Department 

of Transportation in the interest of information exchange. The United 

States Government assumes no liability for its contents or use thereof. 

Notice 

The United States Government does not endorse products or 

manufacturers. Trade or manufacturers’ names appear herein solely 

because they are considered essential to the objective of this report. 
REPORT DOCUMENTATION PAGE Form approved 

OMB No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, 

searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send 

comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to 

Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 

22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0702-0288), Washington, D.C. 20503 

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 

May 2009 

3. REPORT TYPE AND DATES COVERED 

4. TITLE AND SUBTITLE 

Quantitative Nondestructive Testing of Railroad Tank Cars Using 

the Probability of Detection Evaluation Approach

5. FUNDING NUMBERS 

DTFR53-C-00012 

6. AUTHOR(S) Task Order 213

Gregory A. Garcia (TTCI), Ward Rummel (D&W Enterprises), 

Francisco Gonzalez (FRA)

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Transportation Technology Center, Inc. 

P.O. Box 11130 

Pueblo, CO 81001

8. PERFORMING ORGANIZATION 

REPORT NUMBERS

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING AGENCY 

U.S. Department of Transportation REPORT NUMBER 

Federal Railroad Administration 

Office of Research and Development, MS 20 FRA/ORD/DOT-09/10 

1120 Vermont Avenue, NW 

Washington, DC 20590

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILITY STATEMENT

This document is available through National Technical Information 

Service, Springfield, VA 22161

12b. DISTRIBUTION CODE

13. ABSTRACT

Through sponsorship by the Federal Railroad Administration and in cooperation with tank car industry 

representatives, Transportation Technology Center, Inc. (TTCI) has developed probability of detection (POD) 

curves for nondestructive testing (NDT) methods allowed under 49 CFR Sections 179 and 180 for use in structural 

integrity inspections. TTCI has also worked with FRA and industry representatives to establish baseline PODs for 

bubble leak and eddy current testing. 

As part of the same research program the Tank Requalification and Inspection Center, Tank Car Defect Library, 

and master gage development also continue to be developed. It is expected that through industry use of these 

resources, and implementation of quantified NDT processes and procedures, an increase in safety and reliability of 

tank car operations over revenue service track can be achieved. 

15. NUMBER OF PAGES

96 

14. SUBJECT TERMS

automated ultrasonic testing, damage tolerance analysis, direct visual testing, eddy 

current testing, liquid penetrant testing, magnetic particle testing, nondestructive 

evaluation, nondestructive inspection, nondestructive testing, probability of detection, 

radiographic testing, remote visual testing, ultrasonic testing, visual testing 

16. PRICE CODE

18. SECURITY 

CLASSIFICATION OF THIS 

PAGE 

UNCLASSIFIED

19. SECURITY 

CLASSIFICATION OF 

ABSTRACT 

UNCLASSIFIED

20. LIMITATION OF 

ABSTRACT 

17. SECURITY 

CLASSIFICATION 

UNCLASSIFIED

NSN 7540-01-280-5500 Standard Form 298 (Rec. 2-89) 

Prescribed by ANSI/NISO Std. 

239.18 

298-102

Table of Contents

EXECUTIVE SUMMARY................................................................................................. 1 

1.0 Introduction................................................................................................................ 5 

2.0 Objectives................................................................................................................... 7 

3.0 Butt Weld Panel Assessments.................................................................................... 9 

3.1 Summary of Phase I Work................................................................................ 9

3.2 Radiographic and Automated Ultrasonic Test Results on Butt Welds........... 12 

3.3 Automated Ultrasonic Inspection.................................................................... 13 

3.4 Design of Experiment and Protocol................................................................ 14

3.5. Gamma (γ) Radiography ................................................................................. 14 

3.6 Probability of Detection Analysis (RT and AUT) .......................................... 15 

3.7 Conclusions and Recommendations (RT and AUT)....................................... 23 

4.0 Fillet Weld Sample Preparation and Assessments................................................... 25 

4.1 Test Specimen Production .............................................................................. 25 

4.2 Master Gage Specimens.................................................................................. 30 

4.3 Test Specimen Documentation ....................................................................... 33 

4.4 Baseline Fillet Weld Inspections..................................................................... 34 

4.5 Inspection Protocol ......................................................................................... 34 

5.0 Industry Operator Assessment Results .................................................................... 37 

5.1 Probability of Detection Method Analysis...................................................... 37 

5.2 Probability of Detection Capability Results.................................................... 38 

6.0 Eddy Current POD Evaluations............................................................................... 55 

6.1 Eddy Current POD Evaluation Results........................................................... 55 

7.0 Bubble Leak Testing ................................................................................................ 61 

7.1 Bubble Leak Specimen Preparation................................................................ 61 

7.2 Bubble Leak Test Procedure Assessments...................................................... 66 

7.3 BLT Data Analysis and Results...................................................................... 66 

7.4 Bubble Leak Test Conclusions and Recommendations.................................. 69 

8.0 Summary of Phases I, II, and III........................................................................... 71 

9.0 Path Forward......................................................................................................... 73 

References......................................................................................................................... 75 

Appendix A. Presentation—Operator POD Briefing....................................................... 77 

Appendix B. Operator Profile Sheet ................................................................................ 87 

Acronyms .......................................................................................................................... 89 

iii
iv
List of Figures 

Figure 1. Combined NDT POD Comparison Showing Variability in Test Methods... 11 

Figure 2. NDT Method POD Comparison Showing Variability in Operators Using

Fluorescent Magnetic Particle....................................................................... 12 

Figure 3. Schematic View of a Pulse-Echo, Shear Wave Ultrasonic Inspection ......... 13 

Figure 4. AUT Setup on Butt Weld Test Panel at TTC................................................ 15 

Figure 5. X-Radiography Operator 1 POD Results...................................................... 17 

Figure 6. X-Radiography Operator 2 POD Results...................................................... 17 

Figure 7. X-Radiography Operator 3 POD Results...................................................... 18 

Figure 8. X-Radiography Operator 4 POD Results...................................................... 18 

Figure 9. X-Radiography Operator 5 POD Results...................................................... 19 

Figure 10. X-Radiography–POD Analysis of all Cracks Visible on X-Ray Film.......... 19 

Figure 11. Gamma Radiography Operator 1 POD Results ............................................ 20 

Figure 12. Gamma Radiography–Operators 4 and 5 Combined POD Results .............. 21 

Figure 13. AUT Shear Wave POD Curve Result........................................................... 22 

Figure 14. Comparison of Manual and Automated UT POD Results............................ 22 

Figure 15. POD Curve Comparison between Automated UT and 

Average of Manual UT ................................................................................. 23 

Figure 16. Instrumentation Used in the Setup for Tank Car Test Panel 

Dynamic Loading.......................................................................................... 26 

Figure 17. Tank Car Test Panel Setup for Dynamic Loading on the 200,000-Pound 

Load Frame ................................................................................................... 26 

Figure 18. Position of the Platen Adjacent to the Weld Prior to Dynamic Loading ...... 27 

Figure 19. Magnetic Particle Indication Showing an Artificially Induced 

Fatigue Crack at the Termination of a Tank Car Fillet Weld ....................... 28

Figure 20. Tank Car Fillet Weld Test Panel................................................................... 29 

Figure 21. Tank Car Fillet Weld Test Panel with Cover and Hinged ............................ 29 

Figure 22. Tank Car Fillet Weld Test Panel Undergoing Remote 

Visual Inspection........................................................................................... 30 

Figure 23. Tank Car Fillet Weld Master Gage with EDM Notches at 

Weld Terminations........................................................................................ 31 

Figure 24. Tank Car Fillet Weld Master Gage with Fatigue Cracks at 

Weld Terminations........................................................................................ 31 

Figure 25. Tank Car Butt Weld Master Gage with EDM Notches at Weld Toe............ 32 

Figure 26. Tank Car Butt Weld Master Gage with Fatigue Cracks at Weld Toe........... 32 

v
Figure 27. DVT POD Results Comparison for Four Industry Participants.................... 40 

Figure 28. Combined Average of DVT POD Results for Four Industry Participants.... 40 

Figure 29. MT POD Results Comparison for Four Industry Participants...................... 42 

Figure 30. Combined Average of MT POD Results for Four Industry Participants...... 43 

Figure 31. PT POD Results Comparison for Four Industry Participants....................... 45 

Figure 32. Combined Average of PT POD Results for Four Industry Participants....... 45 

Figure 33. Combined Average POD Comparison for the DVT, MT, and 

PT Methods................................................................................................... 48 

Figure 34. RVT POD Results Comparison for Four Industry Participants.................... 49 

Figure 35. Combined Average of RVT POD Results for Four Industry Participants.... 50 

Figure 36. Combined Average of DVT and RVT POD Results Comparison................ 51 

Figure 37. DVT and RVT POD Results Comparison for Operator 1 ............................ 52 

Figure 38. ET Butt Weld POD Operator Results Comparison for 

Industry Participants ..................................................................................... 56 

Figure 39. Combined Average of ET Butt Weld POD Operator Results....................... 56 

Figure 40. ET Fillet Weld POD Operator Results Comparison for 

Industry Participants ..................................................................................... 57 

Figure 41. Combined Average of ET Fillet Weld POD Operator Results..................... 58 

Figure 42. Comparison of ET Butt Weld and Fillet Weld POD Average Results......... 59 

Figure 43. Ring used for Flange BLT Evaluations Containing Known Leak Paths ...... 62 

Figure 44. Several Samples with Different Known Leak Path Sizes............................. 62 

Figure 45. Leak Test Capture Device for Leak Rate Calibration................................... 63 

Figure 46. Tank Car (Leak Test Mock-Up) Section Containing Multiple Ports............ 64 

Figure 47. Leak Test Master Gage Test Specimen......................................................... 64 

Figure 48. Leak Test Master Gage Test Specimen Showing Calibration Leak ............. 65 

Figure 49. Leak Test Master Gage Test Specimen Showing Air Hose Connection and 

Pressure Gage................................................................................................ 65 

Figure 50. Leak at Inspection Point................................................................................ 66 

Figure 51. Operator 1 BLT POD Results....................................................................... 67 

Figure 52. Operator 6 BLT POD Results....................................................................... 67 

Figure 53. Operator 3 BLT POD Results....................................................................... 68 

Figure 54. Operator 3 POD Results With Largest Leak Removed (missed by all)........ 68 

vi
List of Tables 

Table 1. Master Gage Totals for Butt and Fillet Weld Samples.................................... 32 

Table 2. Butt Weld Master Gage Flaw Information ...................................................... 33 

Table 3. Fillet Weld Master Gage Flaw Information..................................................... 33 

Table 4. POD Summary for Direct Visual Inspections.................................................. 41 

Table 5. POD Summary for Magnetic Particle Inspection ............................................ 43 

Table 6. POD Summary for Liquid Penetrant Inspections ............................................ 46 

Table 7. POD Summary for the Combined Average Comparison of 

DVT, MT, and PT............................................................................................ 48 

Table 8. POD Summary for Remote Visual Inspections............................................... 50 

Table 9. POD Summary Comparison Between DVT and RVT .................................... 53

Table 10. POD Summary for Eddy Current Inspections ............................................... 59 

vii
viii
EXECUTIVE SUMMARY 

Transportation Technology Center, Inc. (TTCI), under sponsorship from the Federal 

Railroad Administration (FRA), is working with the tank car industry to increase the 

reliability of railroad tank car structural integrity inspections. In support of the Tank 

Car Nondestructive Test (NDT) Program, TTCI researchers and industry participants 

have evaluated a variety of NDT methods used to inspect tank cars. Accomplishments 

to date include: 

• Baseline inspections of railroad tank cars 

• Validation of NDT methods 

• Development of baseline probability of detection (POD) curves 

− NDT methods per Code of Federal Regulations (CFR) liquid penetrant 

(PT), magnetic particle (MT), radiographic testing(RT), ultrasonic (UT), 

and visual (VT) 

− Other NDT methods: bubble leak testing (BLT), eddy current (ET) 

• Establishment of a tank car defect library 

• Development of master gages 

• Qualification of a BLT inspection procedure 

A rulemaking issued by the Department of Transportation (DOT) revises Hazardous 

Materials Regulations (HMR) to replace the hydrostatic pressure test, for regulation of 

tank cars, with appropriate NDT methods. HM-201 Requalification is a federal 

regulation governing the qualification of DOT & AAR tank cars. It eliminates the 

hydrostatic tank test previously used and uses NDT, which provides a better method of 

detecting defects and ensures tank car safety. The rulemaking also requires that the test 

methods used have been quantified to demonstrate the sensitivity and reliability of the 

inspection and test technique. The rule changes are located in Federal Register Title 49-

Transportation, Chapter I-Research and Special Programs Administration, Department 

of Transportation, PART 179-Specifications for Tank Cars and Part 180-Continuing 

Qualification and Maintenance of Packagings.1

The basis for assurance of the structural integrity and for life-cycle management of 

engineering structures based on material, loads, and nondestructive inspection (known 

as NDI) was established and is the primary basis for fleet management of aircraft 

structures. The well established principles and tools developed for aircraft applications 

have been adopted and applied to a wide range of engineering structures, components, 

and materials in the public domain and are the basis for the methodology that TTCI has 

applied to railroad tank car structures. 

Addressing the Revised HMRs 

CFR requirement under Section 179.7(b)(10) states: Procedures for evaluating the 

inspection and test technique employed, including the accessibility of the area and the 

sensitivity and reliability of the inspection and test technique and minimum detectable 

crack length. Section 180.509 of the CFR identifies Requirements for inspection and 

test of specification tank cars, paragraph (e) Structural integrity inspection tests. The 

1
CFR authorizes PT, MT, RT, UT, and VT as allowable NDT methods for structural 

integrity inspections. Alternative NDT methods may be allowed for railroad tank car 

inspections under special exemption issued by the FRA Office of Safety. 

Baseline NDT of Railroad Tank Cars 

NDT technicians from the railroad tank car industry assisted TTCI in identifying and 

documenting current industry practices. Industry representatives also performed a 

baseline inspection of four tank cars using the CFR authorized NDT methods along with 

the acoustic emissions NDT method. The technicians, who assisted in this effort, 

perform tank car inspections regularly as part of their job assignments for their 

respective companies. The areas of focus were the required inspection areas as 

identified in CFR – 180.509 including circumferential butt welds and longitudinal fillet 

welds. 

The tank cars used during the baseline inspections have been stored at the 

Transportation Technology Center (TTC) as part of the Tank Car Defect Library. The 

tank cars are available for future evaluations to provide capability comparisons as NDT 

technology is developed and implemented for tank car inspections. 

Validation of NDT Methods 

TTCI used information generated by the aerospace and nuclear industries to determine a 

methodology to validate railroad tank car NDT processes. A NDT process includes the 

NDT systems and procedures used for the inspection, as well as the NDT equipment, 

operator, inspection environment, and the object being inspected. 

Researchers performed the NDT method validation to assess the reliability and 

implementation costs associated with an NDT process. The use of a validation 

methodology to assess the applications, advantages, and limitations of NDT methods is 

a valuable tool to assure inspection reliability. 

Development of Baseline POD Curves 

The emergence of a damage tolerance approach to determine inspection intervals for an 

engineered structure—in this case a railroad tank car—requires the quantification of the 

detectable flaw size for the NDT methods used during inspection. Traditionally, NDT 

methods have not been quantified and assumed capabilities have often been found to be 

in error. Damage tolerance techniques have initiated an evolution in NDT 

understanding, methods, and requirements. National Transportation Safety Board 

(NTSB) safety recommendations R-92-21 through R-92-24 address the suggested 

process of performing reliable inspection of railroad tank cars based on a damage 

tolerance approach. Damage tolerance design and maintenance is expected to improve 

the reliability and confidence level of tank car acceptance and maintenance. NDT 

quantified using the POD approach, a key measure of NDT effectiveness, is integral to 

damage tolerance requirements.2

TTCI has worked with the FRA and the tank car industry to develop baseline POD 

curves for the allowed NDT methods. Initial evaluations were performed on the 

2
inspection of tank car circumferential butt welds. Subsequent efforts focused on both 

the butt welds and longitudinal fillet welds requiring inspection under the CFR. 

Tank Car Defect Library 

A defect library containing sample artifacts, such as railroad tank cars and sections of 

railroad tank cars, has been initiated by TTCI through FRA sponsorship. Samples 

include tank cars donated by the tank car industry and manufactured artifacts developed 

at TTC. Manufactured artifacts consist of test panels used for POD evaluations, along 

with master gages developed for inspection sensitivity verification. The combination of 

specimens contains discontinuities developed in service as well as manufactured flaws 

simulating location and type of discontinuities expected in service. 

The defect library was initiated to provide the tank car industry with resources similar to 

that established in the aerospace and nuclear industries. The primary benefits for 

establishing a defect library and validation center is to offer the industry a facility to 

perform comprehensive, independent, and quantitative evaluations of new and enhanced 

inspection, maintenance, and repair techniques. 

Master Gages 

Baseline PODs have been developed by TTCI using standard industry NDT procedures. 

This data is intended to provide a basis for design/life cycle maintenance assumptions 

for general nondestructive evaluation inspections. The data is to be anchored by 

application and response to tank car master gages. The PODs have been established to 

provide a capability that can be used for qualification of equivalent NDT procedures and 

for personnel skill demonstrations. 

The primary measure of reliability in NDT is repeatability and reproducibility. Master 

gages developed from the test tank cars are used as tools to perform a response 

comparison to calibration artifacts used in the field. The master gages are stored at TTC 

to preserve and periodically revalidate response linearity of the calibration artifacts. 

3
4
1.0 Introduction 

This report provides a review of Phase I results and documents those tasks performed during 

Phases II and III of the work that was initiated and reported in DOT/FRA/ORD-01/04, January 

2002 in development of baseline nondestructive methods and procedures to assess the structural 

integrity of rail tank cars.3

On September 21, 1995, the U.S. Department of Transportation (DOT) changed the federal 

regulations to require the use of nondestructive evaluations (NDE) to verify tank structural 

integrity. The National Transportation Safety Board (NTSB), based on previous accident 

experience, urged the DOT to seek a possible replacement of the test. Under Docket No. HM 

201, the FRA and the Pipeline Hazardous Material Safety Administration (PHMSA), formerly 

the Research and Special Programs Administration (RESPA), revised the Hazardous Metals 

Regulations (known as HMR) to replace the hydrostatic test with appropriate nondestructive 

testing (NDT) methods. The NDT methods increase the confidence to detect critical tank car 

defects, thereby enhancing safe transportation of hazardous materials. 

Docket No. HM 201 requires the development and implementation of quality assurance 

programs at facilities that build, repair, and inspect tank cars. The rule requires NDE in lieu of 

periodic hydrostatic pressure tests for fusion welded tank cars. 

The rule change was made to incorporate inspection methods that will: 

• More adequately detect critical cracks 

•

•

•

•

Require thickness measurements of tank cars 

Allow the continued use of tank cars with reduced shell thickness 

Revise the inspection and test intervals for tank cars 

Clarify the inspection requirements relating to tank cars prior to and during 

transportation 

These actions were deemed necessary to increase the confidence that critical tank car defects will 

be detected. The intended effect of these actions is to enhance the safe transportation of 

hazardous materials in tank cars. 

In support of Docket No. HM 201, the FRA Office of Research and Development contracted 

with the Transportation Technology Center, Inc. (TTCI), a subsidiary of the Association of 

American Railroads (AAR), to perform a joint government/industry evaluation of possible 

replacement tests/inspections for the prescribed hydrostatic test/visual inspection of tank cars. 

Under the guidance of the FRA, TTCI was directed to evaluate NDE techniques and determine 

how such techniques can best be applied for periodic testing and inspection of all tank cars that 

transport hazardous materials (NTSB R-92-94). Evaluations were performed at the TTC, 

Pueblo, Colorado. 

Within Phase I, a baseline capabilities study was performed on the CFR 49 accepted NDT 

methods that includes visual test (VT), liquid penetrate test (PT), magnetic particle test (MT), 

ultrasonic test (UT), and radiography test (RT). The Tank Car Defect Library of flawed and 

unflawed tank cars and tank car butt welded specimens was initiated. The probability of 

5 
detection (POD) methodology was established as a metric to quantify the capabilities of various 

accepted NDT methods, procedures, and personnel. 

Within Phase II, representative fillet weld specimens were added to the library, and initial NDT 

methods capabilities assessments were performed. Master gage specimens were developed and 

baseline characterization was conducted to supply references to provide and assess the 

reproducibility of various NDT methods, procedures, and personnel performance capabilities. 

Included in Phase III, BLT specimens were fabricated and added to the library. Baseline 

characterization of the leak test specimens was completed. Continuing assessment of the 

performance of CFR 49 accepted NDT methods was performed using the specimens in the Tank 

Car Defect Library. Quantification of performance capabilities using the POD approach was 

conducted for the ET method. 

6 
2.0 Objectives 

The objectives of the Tank Car NDE Program are to: 

• Observe, review, and document previously performed industry related work 

• Baseline current NDE processes allowed for use in railroad tank car inspection 

• Develop a validation methodology for the NDE processes 

• Introduce a standard process to determine the POD for the NDE methods 

• Establish the Tank Requalification and Inspection Center (TRIC) at TTC 

• Develop baseline reference artifacts that can be used to relate NDE procedures 

capabilities to established baselines 

Ultimately, the TRIC will be used to validate NDE processes for the inspection of tank cars 

similar to that which Sandia National Laboratories and the Federal Aviation Administration 

(FAA) have established at their Aging Aircraft Nondestructive Investigation Validation Center in 

Albuquerque, New Mexico. A report documenting the FAA-sponsored effort can be further 

reviewed in the National Aging Aircraft Research Program Plan dated October 1993. 4

7 
8 
3.0 Butt Weld Panel Assessments 

3.1 Summary of Phase I Work 

The focus of the Tank Car NDE project has been to provide direction and insight into the current 

capabilities of the railroad tank car industry in the use of the allowed NDE methods for tank car 

structural integrity inspections. Through government and industry cooperation, the 

accomplishments from this project should play a vital role in the continued assessment and 

improvements in the reliability of inspections. The current industry effort focuses on life-cycle 

management through the use of damage tolerance analysis (DTA) methods reliant on NDE 

procedures that are capable, reliable, and quantitative. The use of POD methods to quantify 

NDE capabilities provides a sound basis for the implementation of damage tolerance design and 

life-cycle management methods. 

During Phase I, the tank car structural integrity assessment task was addressed by performing: 

• A literature search 

• Surveying tank car producers and maintenance organizations 

• Initial task planning 

• Manufacture and characterization of butt welded test specimens 

• Development of a capabilities assessment protocol using the POD method 

• Initial/baseline assessments by industry users 

Phase I efforts focused on butt weld baseline, inspections using the VT, PT, MT and UT. The 

first and major step in baselining NDE capabilities was that of producing representative test 

specimens. A unique set of specimens was produced by cutting sections containing butt welds 

from retired tank cars, and initiating tightly closed fatigue cracks along the welds in the heat 

affected zone. Cracks were produced ranging in size from 0.020 to 3.5 inches. The specimens 

were cleaned, identified, and characterized before incorporating them into the baseline butt weld 

test set. This constituted the first addition to the Tank Car Defect Library. Tightly closed fatigue 

cracks were projected to be the most likely service induced defects and are representative of one 

of the most difficult to detect defects in new build butt welded components. 

After initial characterization, an inspection and data recording protocol was established. 

Inspectors from the tank car industry were invited to participate in baseline POD evaluations. 

Participants were asked to bring their own inspection procedures, equipment, and accessories to 

perform POD evaluations of the butt weld specimens using VT, MT, UT, and PT methods. 

The results were recorded as HIT/MISS and were tabulated for each inspection method and 

inspection sequence. Results were analyzed as individual inspection results and as a composite 

of combined inspection results using POD analysis metrics. The individual results are rigorous 

and were reported in the Phase I report. The composite results are less rigorous, but provide an 

indicator of baseline industry capabilities based on industry practices in place at the time of 

inspection. 

The initiation of the Tank Car Defect Library provides the railroad tank car industry with tank 

cars and tank car sections containing service and/or artificially induced discontinuities that can 

be used for operator or technology assessment and development. The baseline validation and 

POD methodologies developed can be used to assess and validate improvements in current and 

9 
new technologies introduced for inspection. Benefits to both industry and government that can 

be realized with the use of the artifacts available in the Tank Car Defect Library at TTC include: 

• Determining the reliability of inspections 

• Quantifying procedures and/or operators 

• Improving safety through technology development 

• Addressing industry needs in the areas of maintenance, inspection, and damage 

tolerance 

• Validating inspection technologies developed by government, academic, and 

commercial organizations 

• Developing validation models for probability of detection assessments 

• Performing cost benefit analysis 

• Promoting technology transfer 

As Figures 1 and 2 demonstrate, composite baseline POD evaluation results show variability in 

NDE methods, procedures, and operators. Such variation is expected and is representative of the 

state of field inspections. Results differ from assumed capabilities predicted (expected) by some 

of the participating NDE operators. 

The data now provides a common basis for analysis and communication. Detailed results for 

individual operators were reported in the Phase I report and demonstrate results of differing skill 

levels and specific procedures. The value of the specimens for quantification of operator skill 

level, progression in skill levels, quantification of NDE procedure capabilities, and 

improvements in NDE procedures is evident by the variations in the results obtained in the 

baseline assessments. 

10 
0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

Actual Crack Length - Inch

0

10

20

30

40

50

60

70

80

90

100

Data Set: Combined NDT by length

Test Object : Tank Car Butt Welds

Condition: As Cracked

Method: Fluorescent LP, Fluorescent MT, 

. Shear Wave UT, Optically Aided VT

Operator: All

Opportunities = 1264

Detected = 602

90% POD= MT 3.4 in. (86.36 mm), 

False Calls = Not Documented

 X HIT / MISS DATA 

Combined LP

Combined MT

Combined UT

Combined VT

UT 2.9 in. (73.66 mm), 

Probability of Detection (%) 

Figure 1. Combined NDT POD Comparison Showing Variability in Test Methods 

11 
0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

)

0

10

20

30

40

50

60

70

80

90

100

Figure 2. NDT Method POD Comparison Showing Variability in Operators Using 

Florescent Magnetic Particle 

3.2 Radiographic and Automated Ultrasonic Test Results on Butt Welds 

3.2.1 X-Radiography 

X-radiography (X-ray) is a widely used method of inspection and evaluation of welds during 

fabrication to detect flaws inherent to the welding processes and to provide a measure of 

confidence in structural integrity (fitness for service) of a weld joint. Tank car producers are 

familiar with X-ray methods and have experienced personnel with skills developed in application 

of the method. The procedure involves (1) placing a film or detector on one side of a weld and 

an x-ray source (tube) on the opposite side of the weld, (2) generating X-radiation at an 

optimized energy level and exposure time, (3) detecting the transmitted radiation on the 

film/detector side to an established image density level, (4) processing (developing) the film to 

produce an optimized image, and (5) “reading” the film (image) to detect and interpret variation 

in the image that may be related to an internal condition of the weld. 

Procedures for producing an X-ray image and image quality requirements are well defined for 

industrial processes. Interpretation of the images requires knowledge of the process, knowledge 

of the test object, and both skill and experience in relating the image characteristics to acceptable 

weld conditions. Since X-ray is sensitive to changes in material thickness and material density, 

it is well suited to detection of inclusions and porosity in welds. Cracks have little volume and 

detection depends on alignment of the X-radiation along the axis of the crack. Wide cracks and 

ACTUAL CRACK LENGTH - (Inch

Data Set: All, MP Comparison

Test Object : Tank Car Butt Welds

Condition: As Cracked

Method: Fluorescent Magnetic Particle

Operator: All

Opportunities = 79 each

Detected = OP1 (61), OP2 (52), OP3 (63),

 OP4 (45)

90% POD = OP3 1.08 in. (27.30 mm))

 OP4 2.80 in. (70.97 mm.)

False Calls = Not Documented

 X HIT / MISS DATA 

OP4

OP3

OP2

OP1

Probability of Detection (%) 

Actual Crack Length–Inch

12 
lack of fusion or lack of penetration may be detectable in production welds, but tightly closed 

fatigue cracks are detectable only at large crack sizes. In short, X-ray is not an optimum method 

for service induced crack detection. 

3.2.2 Gamma (γ) Radiography 

Gamma radiography is similar to X-ray; the difference being the use of a radioactive isotope as a 

source of generating penetrating radiation instead of an X-ray machine. Advantages of using a 

radioactive source is that it does not require a power source and is therefore more portable than 

an X-ray machine. Disadvantages of a radioactive source are that the radiation energy level is 

fixed (each source has a unique energy level and flux output) and the finite size of the source. 

Images produced using a source are generally less sharp due to the geometric unsharpness 

inherent to the size of the source and the nature of an image that is produced at a single energy 

level. In short, gamma (γ) radiography is often used in field applications due to its portability, 

but generally produces images of lesser quality than those produced by X-ray. 

3.3 Automated Ultrasonic Inspection 

Ultrasonic inspection is widely used in inspection of welds for inherent weld quality and for 

assessment of structural integrity (initial and continuing fitness for service). Ultrasonic weld 

inspection is generally applied in the pulse-echo, shear wave mode. In this mode, an ultrasonic 

energy pulse is generated at a moveable transducer source (usually a commercially available 

piezoelectric material excited by a commercially available ultrasonic instrument). The sound 

pulse is transmitted at an angle into the material adjacent to a weld. Anomalies (cracks, 

inclusions, and porosity) reflect the sound energy back to the transducer, and the reflected energy 

is displayed on an oscilloscope or cathode ray tube similar to the schematic shown in Figure 3. 

 

Figure 3. Schematic View of a Pulse-Echo, Shear Wave Ultrasonic Inspection 

Weld inspection may be accomplished by hand scanning or by automated scanning in the pulseecho mode. Hand scanning is commonly performed by the operator moving the transducer along 

the weld, back and forth to the weld, and at various angles with respect to the weld bead. Both 

scanning and interpretation of the screen display are highly dependent on operator experience 

and skill. Variance in detection capability may vary considerably between operators. The tank 

car butt weld specimens were previously inspected by hand scanning ultrasonic procedures used 

by the operators in service and the results reported in the Phase I report. 

13 
Automated inspection is often applied to reduce variability in inspections that are sometimes 

encountered with hand scanning. Automated inspection requires not only automation of the 

scanning motions, but considerable NDE engineering to establish scanning parameters, scan 

speeds, and reject levels. The automated procedure must then be characterized and validated to 

assure that the procedure meets structural integrity requirements. Additional attention must be 

given to transducer replacement, system calibration, and system maintenance in order to assure 

reproducible and consistent inspection. 

3.4 Design of Experiment and Protocol 

3.4.1 X-Radiography 

Assessment of X-ray inspection capabilities was initiated by X-ray inspection of all specimens 

using typical industry practices. Two sets of film images were produced to a standard industry 

image quality of military standard 453 at a 2-percent sensitivity level. One set of film was 

retained by TTCI for archive, and one set was sent to a variety of railroad tank car industry 

participants from tank car maintenance organizations. 

Participants were provided with instructions for read-out (interpretation of the film), data sheets 

for documentation of results, and operator profiles to document experience and skill level of 

operators. A single set of radiographs was used to focus on operator variance, because human 

factors continue to be cited as the cause of a failure to detect and is one of the main variables in 

the X-radiography process.5 Operators were instructed to independently read and document their 

finding using the viewing equipment, environmental conditions, and times that are typically used 

in each facility. Five operators at three facilities provided film read-out and documentation. 

3.5 Gamma (γ) Radiography 

In like manner, two sets of film were produced by gamma radiography; one set was retained by 

TTCI and the second set was sent to three different facilities and readout by five operators. 

3.5.1 Automated Ultrasonic Testing 

An existing production automated ultrasonic inspection system was brought to TTC and used for 

assessment of the test specimens. The system is designed for use on welds in a full tank 

configuration and some difficulty was experienced in inspection of the short panel segments. 

The system operated in an ultrasonic pulse-echo, shear wave mode and incorporates both 

automated scanning and automated discrimination/read-out. As Figure 4 shows, the system was 

operated in the configuration that is normally used at the production facility. No optimization or 

validation on the test specimens was provided because this was a quick-look assessment of 

current inspection options. Output from the system was recorded in a HIT/MISS form, and 

analysis was completed by the established POD methods. 

14 
Figure 4. AUT Setup on Butt Weld Test Panel at TTC 

3.6 Probability of Detection Analysis (Radiograhic and Automated Ultrasonic Testing) 

The FRA-sponsored method of NDE capabilities assessment is by the POD methodology. Since 

crack-to-crack variances as well as NDE process variances must be addressed, the POD method 

was developed as a probabilistic method of analysis. In short, the method assumes the result of 

any NDE method is discriminating between distributions of signal and noise analyses and the 

system process is consistent with a log logistics POD model. By fitting data on signal response 

as a function of flaw size to a log linear plot, a slope and intercept can be derived and input to the 

POD model. It is assumed that the log linear relationship can be reproduced by rigid NDE 

system calibration, and thus the POD/discrimination capability for an NDE system/procedure can 

be quantified.6 

By convention, the accepted discrimination level is at the point where the POD curve passes 

through the 90-percentile point. This single valued output is then input to structural analyses as 

the basic capability of NDE discrimination and acceptance. The POD method is an accepted 

metric for validation of the capability of an NDE procedure for comparison of capabilities of 

NDE procedures and for assessment of skill levels of NDE operators. It is the primary metric 

used in the assessment of capabilities addressed in this report.6 

The POD curve constitutes a plot of probability of detection as a function of increasing flaw size. 

The optimum form of a curve is a steep rise in detection to near 100 percent and then a flat line 

to increase flaw sizes. The smaller the flaw at the 90-percent detection levels, the better the 

detection and discrimination of the procedure. Some missed flaws are required to satisfy the 

form of the POD model, since the threshold of detection is the important characteristic. The 

additions of large flaws, that are easily detected, add no information to detection capability. For 

purposes of comparison, flaws that are missed are shown on the baseline (0-percent POD), and 

flaws that are detected are shown as points on the upper limit (100-percent POD). It is evident 

15 
that some flaws may be missed that are larger than the 90-percent POD threshold point. This is 

characteristic of the probabilistic nature of flaw-to-flaw variation and variation in the detection 

process.6

Curves that gradually increase and/or do not reach the 90-percent threshold level are indicative 

of a procedure that has poor detection/discrimination capability, and those procedures must be 

optimized in order to provide reliable inspections. POD results may also identify the limitations 

of the various NDT test methods imposed by the laws of physics. As an example, the sensitivity 

between the direct visual and other NDT methods previously shown in Figure 1 suggest that for 

tight fatigue cracks, located at the toe of circumferential butt welds in tank cars, the direct visual 

test (DVT) method detection results were lower than the other NDT methods evaluated. This 

result may have some influence from the procedures limitations, but the physical limitations of 

the test method such as dependency on the inspector’s visual acuity, the contrast at the inspection 

location due to lighting, the angle of inspection, geometry of the weld, and the tightness of the 

crack opening are likely the greater limiting factors in this case.6

3.6.1 X-Radiography Results 

Figures 5 through 10 show the results of the X-radiographic inspections. The results show 

considerable variation between operators (film readers and interpreters). A high false call rate 

indicated that not only was the discrimination process variant, but also that there was uncertainty 

in discrimination between cracks and manufacturing weld anomalies. Some of the operators 

were experienced primarily in new built inspection and had little experience in service induced 

crack detection. Operator variance was greater than was expected for the X-radiographic method 

and the added variance was evident due to specific experience in detecting and interpreting 

service induced cracks. 

Figure 10 is the result of viewing known locations for cracks and verifying that a crack image 

was present on the film. The results illustrate poor detection capabilities of the x-ray method for 

crack detection. Figures 5 through 9 are the combined result of the capability of the x-ray 

procedure and the capability of individual operators in detection. It is clear that Operators 4 and 

5 had more experience/skill in detecting service induced cracks. 

16 
Xray t1-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%) Probability of Detection (%) 

Actual Crack Length–Inch

Figure 5. X-Radiography Operator 1 POD Results 

Xray t2-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

P )

Probability of Detection (%) 

ROBABILITY OF DETECTION (%

Actual Crack Length–Inch

Figure 6. X-Radiography Operator 2 POD Results 

17 
Xray t3-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%) Probability of Detection (%) 

Actual Crack Length–Inch

Figure 7. X-Radiography Operator 3 POD Results 

Xray t4-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

PR ) Probability of Detection (%) OBABILITY OF DETECTION (%

Actual Crack Length–Inch

Figure 8. X-Radiography Operator 4 POD Results 

18 
Xray t5-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

Actual Crack Length–Inch

Figure 9. X-Radiography Operator 5 POD Results 

Xray g-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

P ) Probability of Detection (%) ROBABILITY OF DETECTION (%

Actual Crack Length–Inch

Figure 10. X-Radiography–POD Analysis of all Cracks Visible on X-Ray Film 

19 
3.6.2 Gamma (γ) Radiography Results 

Gamma (γ) radiography interpretation results were lower than results obtained from Xradiography. Figure 11 shows the results of Operator 1. Operators 2 and 3 were similar and the 

analysis model failed to converge due to the low detection rate. Figure 12 shows the combined 

results of Operators 4 and 5. It was necessary to combine those results to produce enough data 

for use of the model. The poor performance is indicative of both the poor image quality provided 

by the gamma (γ) radiographic procedure and the operator difficulty in interpretation. 

Gamma t1-only

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

ACTUAL CRACK LENGTH - (Inch)

P ) ROBABILITY OF DETECTION (% Probability of Detection (%) 

Actual Crack Length–Inch

Figure 11. Gamma [(γ)?] Radiography Operator 1 POD Results 

(Note: The curve produced is an estimate of performance 

provided by the model, due to no convergence of the data.) 

20 
Gamma Combined Minus t1, t2, t3 

0

10

20

30

40

50

60

70

80

90

100

0 0.511.522.533.5

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

ACTUAL CRACK LENGTH - (Inch) Actual Crack Length–Inch

Figure 12. Gamma Radiography–Operators 4 and 5 Combined POD Results 

3.6.3 Automated Ultrasonic Testing Results 

Figure 13 shows the results from the automated ultrasonic testing (AUT). The results show a 

capability that is less than expected industry values. The reduced capability results are attributed 

to the geometry difficulties, and a procedure that was not optimized and validated for application 

to the test samples. The results are thus the performance assessed and reported herein, but should 

not be used as representative of optimized capabilities. 

Figure 14 shows the comparison of manual and automated POD results. The comparison shows 

that the automated inspection performed at a higher POD for Operators 1 and 4 and a lower POD 

for Operators 2 and 3 at crack sizes of 0.50 and 1.00 in. The comparison also shows that at crack 

sizes greater than 2.25 in all of the manual inspections obtained a higher POD than the AUT 

approach. These results are also shown in Figure 15, which is the combined average of manual 

UT as compared to AUT. In the graph shown in Figure 15, AUT showed a higher POD up to 

approximately 1.10 in, and, manually, UT demonstrated a higher POD at all crack sizes greater 

than 1.10 in. 

21 
0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

0

10

20

30

40

50

60

70

80

90

100

Probability of Detection (%) 

Data Set: AUT by length

Test Object : Tank Car Butt Welds

Condition: As Cracked

Method: Automated Ultrasonic Shear Wave

Operator: NA

Opportunities = 70

Detected = 40

90% POD= Not Achieved

False Calls = Not Documented

 X HIT / MISS DATA 

ACTUAL CRACK LENGTH - (Inch) Actual Crack Length–Inch

Figure 13. AUT Shear Wave POD Curve Result 

0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

 

Figure 14. Comparison of Manual and Automated UT POD Results 

ACTUAL CRACK LENGTH - (Inch)

Data Set: All, UT Comparison

Test Object : Tank Car Butt Welds

Condition: As Cracked

Method: Ultrasonic Shear Wave

Operator: All

Opportunities = 79 each

Detected = OP1 (21), OP2 (47), OP3 (52),

 OP4 (25), AUT (40)

90% POD = OP1 1.87 in. (48.58 mm.)

 OP2 1.68 in. (42.78 mm.)

 OP3 2.16 in. (54.95 mm)

False Calls = Not Documented

 X HIT / MISS DATA 

OP1

OP4

OP3

OP2

AUT

Probability of Detection (%) 

Actual Crack Length–Inch

22 
0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

AC

 

Figure 15. POD Curve Comparison between 

Automated UT and Average of Manual UT 

3.7 Conclusions and Recommendations (RT and AUT) 

The results presented herein represent current industry capabilities for the NDE methods and 

procedures used. Conclusions are summarized as follows: 

1. Although X- and gamma radiography are widely used in production weld acceptance, 

the type of flaws/anomalies that are assessed are different from service induced 

cracks, and the detection capabilities may be considerably different. 

2. Neither X- nor gamma radiography are recommended for detection of service induced 

cracks in tank car butt welds that are represented by the samples assessed. 

3. Skill and experience with the specific hardware and type of anomaly to be detected 

have a significant effect on detection results. Operators 1–3 were experienced in 

weld anomaly detection but had little previous experience in service induced crack 

detection using the radiographic test method. 

4. Failure to image significant cracks on the film indicates that the radiographic 

inspection methods are not reliable for service induced crack detection. This is 

consistent with the physics of the inspection method and is not unique or specific to 

the experimental data produced. 

The AUT procedure applied was neither optimized nor effective in crack detection at a level that 

had been previously demonstrated in industry. The result should not be considered to be 

representative of the ultrasonic method, but that of a procedure and tooling that were not 

TUAL CRACK LENGTH - (Inch)

Data Set: UT AVG to AUT Comparison

Test Object : Tank Car Butt Welds

Condition: As Cracked

Method: Ultrasonic Shear Wave

Operator: All

Opportunities = 79 each

Detected = UT AVG (36), AUT (40)

90% POD = UT OP1, 1.87 in. (47.50 mm), )

OP 2 1.68 in. (42.67 mm),

 OP3, 2.16 in. (54.86 mm)

False Calls = Not Documented

 X HIT / MISS DATA

UT 

avg

AUT

Probability of Detection (%) 

Actual Crack Length–Inch

23 
optimized to the test objects. Additional assessment of automated ultrasonic procedures is 

strongly recommended, if this method is selected for general use in industry. Indeed an 

automated ultrasonic method is recommended for inspection of tank car circumferential welds 

and assessment should be repeated when procedures are optimized. 

Successes and shortfalls of NDT capabilities are frequently attributed to Human Factors and so a 

widely held solution to removing Human Factors and thus improve detection capability and 

reliability is automation. However, automation without attention to all application variables will 

not improve detection or reliability and may just be a mode of applying variable (uncontrolled) 

inspection faster. Capabilities of both manual and automated NDT procedures are dependent on 

the following variables: 

• Flaw (artifact) 

• Test object 

• NDT method 

• NDT materials 

• NDT equipment 

• NDT procedures 

• NDT process 

• Calibration 

• Acceptance criteria 

• Human factors 

24 
4.0 Fillet Weld Sample Preparation and Assessments 

Fillet welds are typically used at attachment points and in reinforcement areas. Cracks in fillet 

welds typically initiate and grow by fatigue at the end (start, stop, or termination) of a weld, but 

can initiate at other concentrated stress points or at sites of original weld inclusions. The 

objectives of the fillet weld characterization task were (1) to select and prepare a statistically 

significant number of test samples containing representative fatigue cracks at various sizes and 

located at representative locations, (2) to characterize and document in a database that the 

samples and cracks could be used for NDE assessments using a variety of NDE methods, and (3) 

to perform initial baseline NDE assessments to validate the test sample sets and baseline typical 

capabilities of the CFR allowed and other commonly applied NDE methods. 

4.1 Test Specimen Production 

Sections containing as-built fillet welds were cut from retired tank cars. Preparation and 

characterization of fillet weld specimens were completed by initiating and growing fatigue 

cracks in panels cut from the tank car sections. Tightly closed fatigue cracks were also projected 

to be the most likely service induced flaw in fillet welds. A review of literature, experience with 

revenue service tank cars, and stress concentration analyses indicated that the most likely 

location for service induced flaw (cracks) origin was at fillet weld terminations and at the radius 

of the weld near the juncture of the fillet with the base material. The initial challenge in 

designing the NDE assessment task was producing representative fatigue cracks in the 

established locations. 

To reduce the number of specimens and maximize inspection opportunities, portions of long 

fillet welds were removed by controlled grinding to produce a series of short weld sample 

locations, approximately 10-inches long, and thus provide a large number of weld terminations at 

the ends of the remaining weld locations. The fillet weld panels were mounted in the same load 

frame setup used to induce fatigue cracks in butt weld sections. Fatigue crack initiation and 

growth procedures were then developed for the fillet welds. 

Diamond scribe marks were placed at various locations near the end of the retained fillet weld 

ligaments to provide a starter notch for fatigue crack initiation. Crack growth was produced by 

securing a panel into a 200,000-pound load frame and dynamically loading in the area where the 

scribe mark was placed. Figures 16 and 17 show the setup for cracking the panels. 

25 
Figure 16. Instrumentation Used in the Setup for Tank Car 

Test Panel Dynamic Loading 

 

Figure 17. Tank Car Test Panel Setup for Dynamic Loading on the 

200,000-Pound Load Frame 

26 
The loading point used to produce the fatigue cracks was an oval tip approximately 0.19 × 0.38 

in (0.483 × 0.965 cm) welded to the top of the platen. The oval-shaped platen tip was designed 

to provide a point load at the opposite side of the test panel from where the diamond scribe 

marks were made. Figure 18 shows the placement of the platen adjacent to the butt weld before 

dynamic loading. The scribe marks on the test panels were generally in the range of 0.06 to 0.10 

in (0.15-0.25 cm) in length and were manually applied. The depth of the notches were not 

measured but were estimated to be about 0.02 to 0.03 in (0.05-0.08 cm). 

Figure 18. Position of the Platen Adjacent to the Weld 

Prior to Dynamic Loading 

The test panels were taken from retired tank cars and the material is representative of the 

American Society for Testing of Metals (ASTM) A515 Grade 70 steel used for tank car 

fabrication. The tank panel thickness and fillet welded reinforcement material is approximately 

0.44 in (1.12 cm). Mechanical properties for ASTM A515 Grade 70, as specified in Volume 1 of 

the ASM Metals Handbook, are: 

• Tensile strength–79 to 90 ksi (485 to 620 MPa) 

• Yield strength–38 ksi (260 MPa) 

• Minimum elongation in 2 inches (50 mm) is 21 percent 

The cracks were grown in bending under a maximum dynamic load of 25,000 lb. The mean load 

was set at 15,000 lb with a range of +10,000 lb. The load setting at a maximum dynamic load of 

25,000 lb was determined to be too high as the platen was indenting the material at the areas of 

point loading. It should be noted that although the setup samples were indenting, fatigue cracks 

did propagate from the scribe marks. The maximum dynamic load was then reduced to 17,000 lb 

with the load set at a mean of 12,000 lb with a range of +5,000 lb. The frequency during 

dynamic loading was set at 10 hertz. 

A 20× video camera was magnetically mounted to the test panel to monitor crack initiation and 

growth. The camera was electronically connected to both a video monitor and VHS recorder to 

27 
allow the technician to identify and record crack initiation and growth. Yellow paint was placed 

at the scribe mark to provide contrast during fatigue loading in order to determine crack 

propagation. A strobe light was attached to the load frame to illuminate the scribe area and 

provide a better contrast for the technician to identify any indication of crack initiation and 

growth. A magnetic rule was placed parallel to the scribe mark to provide a tool for the 

technician to estimate crack length during loading. Figure 19 shows a typical magnetic particle 

indication of a crack produced in this manner at the end of a fillet weld. 

Figure 19. Magnetic Particle Indication Showing an Artificially Induced 

Fatigue Crack at the Termination of a Tank Car Fillet Weld 

Cracks were induced at the end of the retained fillet weld ligaments and a significant number of 

ligament ends were left in the uncracked condition to minimize operator expectations and bias. 

Crack initiation and growth were monitored with a video camera mounted directly onto the 

panels being dynamically fatigued. A strobe light and yellow metal paint applied at the toe of 

the weld terminations were used to provide the required contrast and definition to determine and 

monitor crack growth. Several specimens were broken open to confirm that the cracks produced 

were representative of service induced cracks and to validate the crack initiation and growth 

procedures produced during dynamic loading. After cracking, excess material was removed 

from the panel sections to reduce the specimens to a size that could more easily be handled and 

moved. Panels were steam cleaned and dried in preparation for validation of crack sizes and 

locations. Figure 20 shows a typical fillet weld test panel. 

28 
Figure 20. Tank Car Fillet Weld Test Panel 

Hinges were installed to attach insulation and stand-off covers to enable assessment of remote 

visual detection capabilities with these sample panels. Figure 21 shows an open panel with 

insulation and cover installed. Figure 22 shows a specimen set up for remote visual inspection. 

Figure 21. Tank Car Fillet Weld Test Panel with Cover and Hinged 

29 
Figure 22. Tank Car Fillet Weld Test Panel 

Undergoing Remote Visual Inspection 

4.2 Master Gage Specimens 

TTCI, in support of FRA Task Order 213–Nondestructive Testing in Lieu of Hydrostatic Testing 

of DOT Specification Tank Car, manufactured butt weld (girth weld) and fillet weld master 

gages or calibration specimens that are intended to be used as tools during tank car NDT. These 

master gages, shown in Figures 23 through 26, were produced to provide inspectors with 

reference samples of tank car specimens and crack configurations. Master gages provide both 

inspection article familiarization and quantitative measurement references for those NDE 

methods that provide a scalar output. 

Panels with electro-discharge machined (EDM) slots and panels with induced fatigue cracks 

provided a common baseline for all operators and inspection assessments. The use of master 

gages with artificial and service type flaws allows inspectors to identify the indication and/or 

signal response variance between these types of flaws. 

Artificial flaws are usually more readily detectable due to the controlled geometry of the 

simulated crack. Fatigue cracks produced in master gages are tighter (narrow opening), which 

usually makes them harder to detect than EDM notches. Calibrating with the master gages 

allows the operator to establish the proper sensitivity for the differences in responses in order to 

provide a more reliable interrogation of the inspection article. 

30 
Figure 23. Tank Car Fillet Weld Master Gage with 

EDM Notches at Weld Terminations 

Figure 24. Tank Car Fillet Weld Master Gage with 

Fatigue Cracks at Weld Terminations 

31 
Figure 25. Tank Car Butt Weld Master Gage with EDM Notches at Weld Toe 

Figure 26. Tank Car Butt Weld Master Gage with Fatigue Cracks at Weld Toe 

Tables 1 through 3 provide specific information on master gages developed for use in support of 

tank car NDT. These master gages are samples of tank car specimens made from ASTM A515 

Grade 70 Steel and contain manufactured EDM notches and fatigue cracks. 

Table 1. Master Gage Totals for Butt and Fillet Weld Samples

Samples Butt Weld Fillet Weld 

Specimens Available 9 11 

EDM Notches 3 3 

Fatigue Cracks 6 8 

Weight (pounds) 36 34 

Length (inches) 24 17 

Width (inches) 12 12 

Thickness (inches) 0.46 .46 

Thickness with Pad (inches) NA 0.94 

32 
Table 2. Butt Weld Master Gage Flaw Information

Sample Identification 

Butt Weld 

Location 1 

(inches) 

Location 2 

(inches) 

Location 3 

(inches) 

TTCI-2 (EDM) toe of weld toe of weld toe of weld 

TTCI-3 (EDM) toe of weld toe of weld toe of weld 

TTCI-4 (EDM) toe of weld toe of weld toe of weld 

MG-6 (FS) toe of weld toe of weld toe of weld 

MG-7 (FS) toe of weld toe of weld toe of weld 

MG-8 (FS) toe of weld toe of weld toe of weld 

MG-13 (FS) toe of weld toe of weld toe of weld 

MG-16 (FS) toe of weld toe of weld toe of weld 

MG-18 (FS) toe of weld toe of weld toe of weld 

Table 3. Fillet Weld Master Gage Flaw Information

Sample Identification 

Fillet Weld 

Location 1 Location 2 

Left Right 

TTCI MGL-P2 (EDM) termination of weld termination of weld 

TTCI MGL-P3 (EDM) termination of weld termination of weld 

TTCI MGL-P4 (EDM) termination of weld termination of weld 

MGL-2 B (FS) termination of weld termination of weld 

MGL-3 A-B (FS) termination of weld termination of weld 

MGL-4 A-B (FS) termination of weld termination of weld 

MGL-5 A-B (FS) termination of weld termination of weld 

MGL-6 D-C (FS) termination of weld termination of weld 

MGL-9 A-B (FS) termination of weld termination of weld 

MGL-10 A-B (FS) termination of weld termination of weld 

MGL-10 D-C (FS) termination of weld termination of weld 

Note: EDM stands for electro-discharge machined notches and FS for fatigue sample. 

4.3 Test Specimen Documentation 

Induced crack lengths were documented on all panels during the crack growth cycle using strobe 

light illumination and video recording of crack growth along the surface. Crack depth could not 

always be assumed from the surface visual indications, and all crack lengths were validated 

using both visible and fluorescent magnetic particle assessments. The validated cracks were 

identified in the specimen database as a crack length. In some cases, the cracks curved around 

the exposed weld bead at the termination of the weld. Total crack length was recorded in the 

database. 

33 
4.4 Baseline Fillet Weld Inspections 

The fillet weld test specimen set was then subjected to inspection using applicable, CFR 

accepted, NDT methods applied during in-service tank car inspections. NDT methods included: 

• Direct visual or VT, 

• Remote VT, 

• Liquid penetrant or PT, and 

• Magnetic particle or MT. 

Inspectors participating in the POD evaluations included industry personnel certified in NDT 

who would normally develop and/or apply the various NDT methods on tank cars in service. 

Inspectors used their own inspection procedures, equipment, calibrated specimens, and 

inspection materials. No time limits were imposed, but inspection times for each sequence were 

recorded. Time limits were not imposed because these specimens contain more detection 

opportunities than would normally be encountered during in-service inspections. 

4.5 Inspection Protocol 

The same operational protocol was used for each inspector/inspection sequence. Appendix A 

contains the operator’s POD briefing protocol. Appendix B contains the operator profile sheet. 

The sequence was as follows: 

• Evaluations scheduled between Tuesday and Friday: 

 Monday was used as a logistics and sample preparation day for TTCI 

• A pretest meeting was conducted prior to evaluations: 

 Included TTCI Safety Manager, project engineer, and the industry NDE 

participant(s) 

 Addressed schedule and objectives for evaluations 

 Provided background information to the industry technician(s) pertaining to why 

the evaluations were being performed 

 Provided time to conduct an operator profile on the technician 

 Provided a forum to voice concerns or questions prior to testing 

• Evaluations performed after pretest meeting: 

 NDE technician performed all evaluations and flaw interpretations 

 Order of inspections: 

 VT 

 PT 

 MT 

 Remote VT 

• TTCI personnel documented all finds by technician 

• Video and photographs were taken during the evaluations 

• Post-test meeting conducted after all inspections were complete: 

 Opportunity to critique evaluations 

 Opportunity to identify areas for improvement 

34 
Each inspector was provided with a master gage or calibration specimen before starting an 

assessment sequence to become familiar with the test specimen configuration and responses from 

the induced cracks. For instrumented methods with a quantified output, the output response 

levels from the calibration/reference specimen cracks and slots were recorded before and after 

the inspection sequence and at interim break periods or when requested by the inspector. The 

inspector was requested to verbally identify the location and estimated crack size. Data was 

recorded in tabular form by TTCI and was subsequently entered into a computer database for 

analyses. All identification marks were removed, and the panels were cleaned between each 

inspection sequence. 

35 
36 
5.0 Industry Operator Assessment Results 

5.1 Probability of Detection Method Analysis 

Data analysis was completed using the POD method that was previously established and used in 

Phase I. This method was initially developed and used in the aerospace and nuclear industries 

and has evolved as a standard method for assessment of NDE detection capabilities in multiple 

industries throughout the world. The tank car test specimens, developed by researchers at TTCI, 

under FRA sponsorship, are available for use in characterization of additional NDT methods and 

procedures, for procedure qualification, for personnel qualification, and for joint regulatory and 

industry programs.∗ NDT methods not directly addressed for structural integrity inspections in 

the CFR have also been evaluated using the POD method. These methods include ET and BLT. 

NDT results consist of either HIT/MISS or detection with a scalar quantifier such as crack length 

or signal amplitude. VT, MT, PT, and film radiography methods involve pattern recognition and 

interpretation by inspectors to provide a HIT/MISS output decision. Instrumented methods such 

as ultrasonic or ET use a signal output level, and a HIT/MISS decision is made when the signal 

level exceeds a predetermined NDT threshold level. Both the decision and the signal level are 

recorded and used in POD analysis. 

To re-emphasize, the POD method of characterization and analysis was developed to address the 

variations in crack (defect) responses to various different inspection methods and procedures and 

to provide a statistically based sampling and confidence level in the combined crack-to-crack and 

inspection variances that are characteristic to NDI process applications (results). The method 

was developed to provide a basis for quantification of expected detection capabilities that are 

used as a basis for fracture and life-cycle service analyses. POD is thus an integral element in 

assessing structural integrity, defining service safety factors, risk analysis in use, and life-cycle 

structures management (service and maintenance). 

The goal of the POD assessment process is to provide a baseline capability and confidence level 

that the NDT procedure can be reliably applied and will produce some confidence in detecting 

cracks (or other target anomalies) in an online application. The elements of a reliable NDT 

procedure are: 

• Reproducibility–addressed by a rigorous calibration protocol, 

• Capability–addressed by the POD analysis method, and 

• Repeatability–addressed by rigor in NDT procedure application, process control, and 

personnel skills. 

The POD process, as listed below, is relatively simple but must be disciplined and documented 

for it to be representative of the inspection to be applied and to the intended application. 

• Generate a large number of cracks (or other test artifacts) that are representative of 

the cracks (condition) to be addressed in inspection and in materials, geometries, and 

environments that are representative of the inspection to be performed. 

 

∗

 The essential elements that are most often misunderstood are the requirements for procedure stability, rigid calibration, and 

operator skills. 

37 
• Calibration artifacts and a calibration test method are necessary to provide an 

inspection setup and system performance baseline that is reproducible within the 

bound of normal measurement variances. 

• A stable procedure (written and validated as capability of producing measurements 

that are significantly above the background noise signals) that has been shown to be 

capable for producing a reproducible signal from target cracks (artifacts) that are at or 

near the structural needs (lower limits) and acceptance criteria. 

• Stable equipment/measurement systems and operators who are skilled in the use of 

the equipment, the NDT procedure, and the intended procedure application. 

POD analysis is then completed using standardized statistical analysis software tools. 

The crack sample set used for assessment must represent the test condition and range of crack 

sizes that are expected in the service application. Analysis by a probabilistic method is required 

due to crack response variations from cracks of the same size and to variations in the response of 

the NDT inspector. It assumes that the NDT output response increases with increasing crack size 

(e.g., length and depth), and that the range of crack sizes tested is representative of the service 

application. 

For HIT/MISS data, the NDT decision results are fit to a response/crack size relationship using a 

maximum likelihood analysis. The response/crack size relationship is provided directly by those 

methods that provide a scalar/quantified output level. The characteristics of the response/size 

relationship is input to a model that provides a smooth curve output of POD as a function of 

crack size. The characteristic discrimination level for the NDT sequence is defined (by 

convention) as the point at which the POD curve reached the 90-percent POD level, and the 

crack size used for fracture analyses is the crack size at the 90-percent POD level. For more 

detailed information on the POD method, refer to Military Standard 823.6

5.2 Probability of Detection Capability Results 

This section includes the evaluation results for the test panels used in the Tank Car NDE project. 

The test panels contained a cumulative total of 104 cracks ranging from 0.080 to 6.00 inches (2 

to 152 mm) in length. 

Industry representatives using direct VT, visible dye penetrant, and MT inspection techniques 

evaluated the POD test panels containing manufactured flaws. TTCI researchers characterized 

and documented the test panels prior to the industry inspections. The crack sizes and locations 

were stored into a database, which was used to generate the POD curves in the subsequent 

sections of this report. 

During panel inspection, the inspectors identified and sized all defects using a magnetic rule, and 

reported the results of the inspection to a TTCI representative for documentation of the size and 

location of the flaw. A TTCI representative input the information into the POD database after all 

inspections were completed. 

The technicians, who participated in the panel evaluations, were scheduled for four days of 

onsite testing. Four companies were represented during the POD evaluations. The company 

representatives, who participated in the evaluations, are identified in this report as Operators 1, 5, 

38 
6, and 7 for those POD curves generated. Each of the industry representatives followed 

guidelines provided by TTCI. The guidelines were from generic procedures agreed upon by the 

AAR Tank Car NDE Steering Committee. In the absence of a documented industry-wide critical 

crack size, the steering committee focused the POD evaluations on the detection of 0.5-inch and 

1-inch crack lengths for circumferential butt welds and fillet welds respectively. 

The graphs shown for each of the inspection methods correspond to the POD curves generated 

for each operator. The first graph shown for each test method provides a comparison between 

operators; the second graph for each method shows the average achieved by combining the 

results from all four participating industry operators. A summary table is included at the end of 

each section of graphs to provide the POD percentages at various crack lengths for all operators. 

All graphs are presented in English units and all tables are presented in both English and metric 

units. 

5.2.1 Direct Visual Inspection POD Results 

The tank car panel evaluations using the direct VT method were performed in accordance with 

the AAR Manual of Standards and Recommended Practices, Section C–Part III, Specifications 

for Tank Cars, Specification M-1002, Appendix T, Part T8.00–Direct Visual Testing, October 1, 

2003. A VT procedure setup sheet was established, and the parameters identified on the sheet 

were verified by both the evaluating technician and a TTCI test representative prior to 

inspections. The parameters required identification and documentation of the VT equipment 

used to aid in the inspection (e.g., flashlight, magnifying glass, and mirrors) and the light 

intensity at and around the inspection surface.7

Figure 27 shows a comparison of the individual POD results for the operators. Figure 28 shows 

the results of the combined average of the four industry participants. Table 4 lists the actual 

percentage POD in 0.5-inch (12.7 mm) increments. 

39 
0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5

0

10

20

30

40

50

60

70

80

90

100

Figure 27. DVT POD Results Comparison for Four Industry Participants 

Figure 28. Combined Average of DVT POD Results for Four Industry Participants 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

ACTUAL CRACK LENGTH - (Inch)

Data Set: Tank Car

Test Object : Steel Fillet Weld 

Panels

Condition: Sand Blasted

Method: Direct Visual 

Operator: Average

Opportunities = 416 / 464 Sites

Detected = 188

90% POD = Not Achieved 

False Calls = 16

 X HIT / MISS DATA 

4

ACTUAL CRACK LENGTH - (Inch)

4.5 5 5.5 6

Data Set: Tank Car

Test Object : Steel Fillet Weld Panels

Condition: Sand Blasted

Method: Direct Visual 

Operator: All 4 Operators

Opportunities = 416/464 Sites

Detected = 188

90% POD = Not Achieved 

False Calls = 16

OP6

OP7

OP1

OP5

Probability of Detection (%) 

Actual Crack Length–Inch

Probability of Detection (%) 

Actual Crack Length–Inch

40 
Table 4. POD Summary for Direct Visual Inspections 

Crack Length DVT 

(in.) (cm) 

POD 

Average 

(percent) 

Operator 5

(percent) 

Operator 1

(percent) 

Operator 7 

(percent) 

Operator 6

(percent)

0.5 1.27 27 59 25 13 13 

1.0 2.54 36 69 32 23 18 

1.5 3.81 41 74 37 31 21 

2.0 5.08 45 78 40 38 24 

2.5 6.35 48 80 43 43 26 

3.0 7.62 51 82 45 48 28 

3.5 8.89 53 84 47 52 30 

4.0 10.16 55 85 48 55 31 

4.5 11.43 56 86 50 58 32 

5.0 12.70 58 86 51 61 34 

5.5 13.97 59 87 52 63 35 

6.0 15.24 61 88 53 65 36 

Analysis of the POD data generated for the direct VT method shows that a 90-percent POD is 

not achieved by any of the operators. On average, a 50-percent POD is not reached until crack 

lengths are greater than 2.5 in. Information from Figures 27 and 28 along with the data listed in 

Table 4 shows that at the 0.5- and 1-inch crack sizes, the average POD percentage between the 

four operators is 27 percent and 36 percent, respectively. Operator 5 achieved the maximum 

percentage of detection for both crack lengths. At the 0.5-inch crack length, Operator 5 reached 

59- and 69-percent POD at the 1.0-inch crack length. The minimum percentage POD of all 

operators at the 0.5-inch crack is 18-percent, and at the 1.0-inch crack two operators performed 

at 13-percent POD. The number of false calls per operator ranges from 0 to 4 during the direct 

visual inspections. 

Variance in inspection methodology was observed during inspections although a standard set of 

procedures was given to each operator. Such variances included speed of inspections, position of 

inspectors (sitting, kneeling, standing), and the use of allowable lights and magnifying glasses. 

No data was taken to quantitatively determine the affect of these variances on the POD results. 

5.2.2 Magnetic Particle Inspection POD Results 

Tank car panel evaluations using the MT method were performed in accordance with Procedure 

No.: TTCI/MPPOD.1, Magnetic Particle Inspections–Yoke Method, dated March 6, 1998. 

Required parameters were verified and documented by TTCI test representatives prior to testing. 

All inspections were performed using dry powder magnetic particles with a continuous 

longitudinal magnetic field. A portable AC/DC magnetic yoke was provided by TTCI to 

minimize the affects of varying equipment. Before each test, the operators were required to 

check the functionality of the yoke using a 10-pound lift block and an image quality indicator 

41 
(pie gage). Operators were also allowed magnifying glasses and a light source with a 50-foot 

candle minimum. 

Figure 29 compares the results of all four operators. Results for the combined average POD for 

all operators is shown in Figure 30, and the actual percentage POD is listed in Table 5. 

 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2

0

10

20

30

40

50

60

70

80

90

100

Probability of Detection (%) 

Data Set: Tank Car

Test Object : Fillet Weld Panels

Figure 29. MT POD Results Comparison for Four Industry Participants 

2.5 3 3.5 4 4.5 5 5.5 6

ACTUAL CRACK LENGTH - (Inch)

Condition: Sand Blasted

Method: Magnetic Particle

Operator: All Operators

Opportunities = 416/ 464 Sites

Detected = 324

90% POD = OP1 and 7 5.85 inch 

 . (148.49 mm)

False Calls = 14

 X HIT / MISS DATA 

OP6 OP7

OP5 OP1

. OP 5 2.95 inch

 . (74.93 mm) 

Actual Crack Length–Inch

42 
Figure 30. Combined Average of MT POD Results for Four Industry Participants 

Table 5. POD Summary for Magnetic Particle Inspection 

MT Crack Length 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Probability of Detection (%) 

(in.) (cm) 

POD 

Average

 (percent)

Operator 5 

(percent) 

Operator 1 

(percent) 

Operator 7 

(percent) 

Operator 6 

(percent) 

0.5 1.27 63 75 69 55 53 

1.0 2.54 73 82 77 67 66 

1.5 3.81 78 85 80 74 72 

2.0 5.08 81 88 83 78 76 

2.5 6.35 83 89 85 81 79 

3.0 7.62 85 90 86 83 81 

3.5 8.89 86 91 87 85 83 

4.0 10.16 87 92 88 86 84 

4.5 11.43 88 92 88 87 85 

5.0 12.70 89 93 89 88 86 

5.5 13.97 90 93 89 89 87 

6.0 15.24 90 93 90 90 88 

ACTUAL CRACK LENGTH - (Inch)

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Magnetic Particle 

Operator: Average

Opportunities = 416 / 464 Sites

Detected = 324

False Calls = 14

 X HIT / MISS DATA 

90% POD = OP1 and 7 5.85 inch 

 . (148.49 mm)

. OP 5 2.95 inch . (74.93 mm) 

Actual Crack Length–Inch

43 
Test panel evaluations for the MT method show that three of the four operators achieved 90-

percent POD at varying crack lengths. Operator 5 accomplished 90-percent POD at a 3.0-inch 

crack length while Operators 1 and 7 accomplished the same percentage at a 6.0-inch crack 

length. Information from Figures 29 and 30, along with data listed in Table 5, shows that at the 

0.5- and 1-inch crack sizes, the average POD percentage between the four operators is 63 and 73 

percent, respectively. Operator 5 achieved the maximum percentage of detection for both crack 

lengths. At the 0.5-inch crack length, Operator 5 reached 75- and 82-percent POD at the 1.0-

inch crack length. The minimum percentage POD of all operators at the 0.5-inch crack is 53 and 

66 percent at the 1.0-inch crack length. The number of false calls ranges from 0 to 4 using the 

MT technique. 

Again, variation in operator technique was observed during the inspections. Sources of variation 

were controlled by a written procedure provided to each of the operators prior to the inspections. 

The written procedure provided guidelines for surface preparation, type of particles to be used, 

calibration, technique, flux field application, application of particles, interpretation of 

indications, and post-inspection cleaning. Observable variances between operators include prod 

spacing and placement, yoke placement, particle application, and overall time of inspection. 

Differences between operators’ techniques should be recognized as a source of variation even 

though a quantitative number for operator-to-operator variances cannot be provided. 

5.2.3 Liquid Penetrant Inspection POD Results 

Tank car panel evaluations, using the PT method, were performed in accordance with Procedure 

No.: TTCI/LPPOD.1, Penetrant Inspections for Standard Temperatures (60–125 degrees 

Fahrenheit), March 6, 1998. Required parameters were verified and documented by TTCI test 

representatives prior to testing. Inspections were performed using water-washable/solvent 

removable penetrant. The inspections used red visible dye, which penetrates openings at the 

surface of the object being examined. TTCI provided the PT, cleaner/remover, developer, and 

cleaning supplies to ensure consistency between inspectors. Operators were also allowed 

magnifying glasses and a light source with a 50-foot candle minimum. 

Figure 31 shows results for each of the four operators. Figure 32 shows the results of the 

combined average POD for all operators, and Table 6 lists the actual percentage POD. 

44 
0

10

20

30

40

50

60

70

80

90

100

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0

ACTUAL CRACK LENGTH - (Inch)

0

10

20

30

40

50

60

70

80

90

100

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Visible Penetrant 

Operator: All Operators

Opportunities = 104 / 116 Sites

Detected = 73

90% POD = 5.40 inches 

False Calls = 2

 ------ PRED. POD

 X HIT / MISS DATA 

OP5

OP1

OP6

OP7

Probability of Detection (%) 

Actual Crack Length–Inch

Figure 31. PT POD Results Comparison for Four Industry Participants 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

A

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Visible Penetrant 

Operator: Average

Opportunities = 416 / 464 Sites

Detected = 74

90% POD = Not Achieved 

False Calls = 4

 ------ PRED. POD

 X HIT / MISS DATA 

CTUAL CRACK LENGTH - (Inch)

Figure 32. Combined Average of PT POD Results for Four Industry Participants 

Actual Crack Length–Inch

45 
Table 6. POD Summary for Liquid Penetrant Inspections 

Crack PT 

Length 

(in.) (cm) 

Average 

(percent)

Operator 5 

(percent) 

Operator 1 

(percent) 

Operator 7 

(percent) 

Operator 6 

(percent) 

0.5 1.27 49 54 51 40 52 

1.0 2.54 57 68 57 46 56 

1.5 3.81 61 75 60 49 59 

2.0 5.08 63 79 62 52 60 

2.5 6.35 65 82 64 53 61 

3.0 7.62 67 84 65 55 62 

3.5 8.89 68 86 66 56 63 

4.0 10.16 69 87 67 57 64 

4.5 11.43 70 88 68 58 65 

5.0 12.70 71 89 69 59 65 

5.5 13.97 71 90 69 60 66 

6.0 15.24 72 91 70 61 66 

Panel evaluations for the PT method show that only one operator achieved a 90-percent POD. 

Operator 5 reached 90-percent POD at a 5.5-inch crack length. Data from Table 6 shows that at 

the 0.5- and 1-inch crack sizes the average POD percentage for the four operators is 49 and 57 

percent, respectively. Operator 5 achieved the maximum percentage of detection for both crack 

lengths. At the 0.5-inch crack length, operator 5 reached 54- and 68-percent POD at the 1.0-inch 

crack length. The minimum percentage POD of all operators at the 0.5-inch crack is 40 and 46 

percent at the 1.0-inch crack length. Operator 5 achieved the maximum percentage, 91-percent 

POD, at a crack length of 6 inches. The number of false calls ranges from 2 to 4 using the PT 

inspection technique. 

Operator technique variation observed during the inspections was controlled by a written 

procedure provided to each of the operators prior to the PT inspections. The written procedure 

provided guidelines for surface preparation, penetrant application, dwell time, excess penetrant 

removal, developer application, and interpretation of results. The guidelines provided 

recommendations, which allowed for slight variations between operators. Some of these 

variations included differences in penetrant application tools, dwell times, and time to final 

evaluation. 

Some operators chose to apply the PT with a spray nozzle while others chose to apply the 

penetrant with a small brush. Dwell time for penetrant is specified as 5 to 15 minutes in 

TTCI/LPPOD.1, but on average the dwell time varied from 5 to 8 min between operators. 

Another apparent difference between operators was the method of removal for excess penetrant 

and the amount of excess penetrant removed. The removal of excess penetrant is critical to the 

46 
sensitivity of the method, as excess penetrant must be removed from the surface of the sample 

while assuring that as little penetrant as possible is being removed from possible discontinuities. 

Operator techniques should be recognized as a source of variation even though a quantitative 

number for operator-to-operator variances cannot be provided from this evaluation. TTCI 

employees monitored and controlled other sources of variation such as panel condition, 

temperature, and equipment. 

5.2.4 Comparison of DVT, MT, and PT POD Results 

Figure 33 and Table 7 show that only one of the three methods used during inspections reached a 

90-percent POD. The highest POD achieved is 93 percent at a 5.0-inch crack length by Operator 

5, using the MT technique. The highest POD results achieved for direct VT and PT techniques 

are 88 and 91 percent at a 6.0-inch crack length, respectively. A 50-percent POD is achieved by 

all three methods at crack lengths of approximately 3 in (7.62 cm) for direct visual, 0.25 in (0.64 

cm) MT, and 0.5 in (1.27 cm) for PT. 

At the 0.5- and 1.0-inch crack lengths, the maximum POD is achieved by Operator 5 in all three 

methods. The highest POD at the 0.5-inch crack length is 75 percent using the MT technique 

and the highest POD at the 1.0-inch crack length is 82 percent also using the MT technique. The 

average POD at the 0.5-inch crack length is 27 percent for direct visual, 63 percent for MT, and 

49 percent for penetrant testing. The average POD at the 1.0-inch crack length is 36 percent for 

direct visual, 73 percent for MT, and 49 percent for penetrant testing. 

Results from the POD evaluations show that of the three methods used during tank car panel 

fillet weld inspections performed at TTC, the dry powder MT inspection technique demonstrated 

the greatest probability of detection at both the 0.5-inch (1.27 cm) and 1.0-inch (2.54 cm) crack 

lengths. 

47 
0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : Steel Fillet Weld Panels

Condition: Sand Blasted

Method: DVT, MT, PT

Operator: Average Comparison

Opportunities = 1248 / 1392 Sites

Detected = 161 DVT, 309 MT, 254 PT

90% POD = Not Achieved

False Calls = 4 DVT, 3 MT, 5 PT 

 ------ PRED. POD

 X HIT / MISS DATA 

MT

PT

DVT

 

Figure 33. Combined Average POD Comparison for the DVT, MT, and PT Methods 

Table 7. POD Summary for the Combined Average Comparison of DVT, MT, and PT 

Crack 

Length 

Average 

(percent) 

Operator 5 

(percent) 

Operator 1 

(percent) 

Operator 7 

(percent) 

Operator 6 

(percent) 

(in.) (cm) 

DVT 

MT 

PT 

DVT 

MT 

PT 

DVT 

MT 

PT 

DVT 

MT 

PT 

DVT 

MT 

PT 

0.5 1.2 27 63 49 59 75 54 25 69 51 13 55 40 13 53 52 

1.0 2.5 36 73 57 69 82 68 32 77 57 23 67 46 18 66 56 

1.5 3.8 41 78 61 74 85 75 37 80 60 31 74 49 21 72 59 

2.0 5.0 45 81 63 78 88 79 40 83 62 38 78 52 24 76 60 

2.5 6.3 48 83 65 80 89 82 43 85 64 43 81 53 26 79 61 

3.0 7.6 51 85 67 82 90 84 45 86 65 48 83 55 28 81 62 

3.5 8.8 53 86 68 84 91 86 47 87 66 52 85 56 30 83 63 

4.0 10.1 55 87 69 85 92 87 48 88 67 55 86 57 31 84 64 

4.5 11.4 56 88 70 86 92 88 50 88 68 58 87 58 32 85 65 

5.0 12.7 58 89 71 86 93 89 51 89 69 61 88 59 34 86 65 

5.5 13.9 59 90 71 87 93 90 52 89 69 63 89 60 35 87 66 

6.0 15.2 61 90 72 88 93 91 53 90 70 65 90 61 36 88 66 

Actual Crack Length–Inch

Probability of Detection (%) 

48 
5.2.5 Remote Visual Inspection POD Results 

The tank car panel evaluations using the remote visual method were performed in accordance 

with the AAR Manual of Standards and Recommended Practices, Section C–Part III, 

Specifications for Tank Cars, Specification M-1002, Appendix T, Part T9.00–Remote Visual 

Testing, October 1, 2003.7

 A remote visual testing (RVT) procedure setup sheet was established, 

and the parameters identified on the sheet were verified by both the evaluating technician and a 

TTCI test representative prior to inspections. The parameters required identification and 

documentation of the RVT equipment used to aid in the inspection (e.g., flashlight, borescope, 

and mirrors) and the light intensity at and around the inspection surface. 

Figure 34 shows a comparison of the individual POD results for the operators. Figure 35 shows 

the results of the combined average of the four industry participants and Table 8 lists the actual 

percentage POD in 0.5-inch (12.7 mm) increments. 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

A

0

10

20

30

40

50

60

70

80

90

100

CTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Data Set: All Operators

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Remote Visual 

Opportunities = 416 / 464 Sites

Detected = 259

90% POD = Not Achieved

False Calls = 22

 ------ PRED. POD

 X HIT / MISS DATA 

Operator 1

Operator 2

Operator 3

Operator 4

Probability of Detection (%) 

Actual Crack Length–Inch

Figure 34. RVT POD Results Comparison for Four Industry Participants 

49 
0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

ACTUAL CRACK LENGTH - (Inch)

P

0

10

20

30

40

50

60

70

80

90

100

Probability of Detection (%) ROBABILITY OF DETECTION (%)

Data Set: Combined Average

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Remote Visual 

Operator: 4 Operators

Opportunities = 416 / 464 Sites

Detected = 259

90% POD = Not Achieved

False Calls = 22

 ------ PRED. POD

 X HIT / MISS DATA 

Figure 35. Combined Average of RVT POD Results for Four Industry Participants 

Table 8. POD Summary for Remote Visual Inspections 

Crack 

Length RVT 

Actual Crack Length–Inch

(in.) (cm) Average 

(percent) 

Operator 1 

(percent) 

Operator 2 

(percent) 

Operator 3 

(percent) 

Operator 4

(percent) 

0.5 1.27 55 27 63 88 40 

1.0 2.54 61 41 66 87 45 

1.5 3.81 64 50 68 87 48 

2.0 5.08 61 57 69 86 50 

2.5 6.35 67 62 70 86 52 

3.0 7.62 68 66 71 85 54 

3.5 8.89 70 69 71 85 55 

4.0 10.16 70 72 72 85 56 

4.5 11.43 71 74 72 85 57 

5.0 12.70 72 76 73 85 58 

5.5 13.97 72 77 73 84 58 

6.0 15.24 73 79 73 84 59 

50 
Analysis of the POD data generated for the remote visual method shows that a 90-percent POD 

is not achieved by any of the operators. On average, a 50-percent POD is not reached until crack 

lengths are greater than 0.25 in. Information from Figures 34 and 35, along with the data listed in 

Table 8, shows that at the 0.5- and 1-inch crack sizes, the average POD percentage between the 

four operators is 55 percent and 61 percent, respectively. Operator 3 achieved the maximum 

percentage of detection for both crack lengths. At the 0.5-inch crack length, Operator 3 reached 

88- and 87-percent POD at the 1.0-inch crack length. The minimum percentage POD of all 

operators at the 0.5-inch crack is 27 percent, and at the 1.0-inch crack, the minimum POD 

reached was at 41-percent POD. The number of false calls per operator ranges from three to 

seven during the remote visual inspections. 

Variance in inspection methodology was observed during inspections, although a standard set of 

procedures was given to each operator. Such variances included speed of inspections, position of 

inspectors (sitting, kneeling, standing), and the use of the bore scopes used during inspection. No 

data was taken to quantitatively determine the effect of these variances on the POD results. 

5.2.6 DVT and RVT POD Comparison Results 

Comparison of the DVT and RVT method shows a range in the POD between operators. The 

RVT results were generally higher than the DVT results. Figure 36 shows a comparison of the 

average of the four operators performing DVT and RVT. The averages show that for a 0.5-inch 

(1.27 cm) crack, the average DVT POD was 27 percent, whereas the average RVT result was 

55 percent. At the 1-inch crack size, the average DVT result was 36 percent, and the RVT result 

was 61. The average maximum POD achieved at the 6-inch crack size was 61 percent for DVT 

and 73 percent for RVT. 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

Data Set: Combined Average

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: DVT and RVT 

Comparison 

Operator: 4 Operators Each Method

Opportunities = 416 / 464 Sites

Detected = 161 DVT, 259 RVT

90% POD = Not Achieved

False Calls = 4 DVT, 22 RVT

 ------ PRED. POD

 X HIT / MISS DATA 

DVT

RVT

Figure 36. Combined Average of DVT and RVT POD Results Comparison 

Actual Crack Length–Inch

51 
Operator 1 is the only industry participant, thus far, that performed both DVT and RVT for the 

fillet weld panels. The comparison of Operator 1 results (Figure 37) showed that similar to the 

industry average, the RVT produced a higher POD than DVT. Operator 1 achieved a POD of 25 

percent at the 0.5-inch crack size for DVT and 27 percent for RVT. At the 1-inch crack size, the 

DVT POD was 32 percent for DVT and 41 percent for RVT. The results for Operator 1 also 

showed that as the crack size increased the difference in POD between RVT and DVT increased 

with the RVT method showing higher detection capability as the crack size increased. A 

suggested explanation for this phenomenon is the increased magnification provided with the 

RVT method which, in this case, provided an increase in interrogation capability. Table 9 lists 

the POD comparison between the average DVT and RVT results along with the results of DVT 

and RVT for Operator 1. 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

A

0

10

20

30

40

50

60

70

80

90

100

CTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: DVT and RVT 

Comparison 

Operator: Operator 1

Opportunities = 104 / 116 Sites

Detected = DVT 43, RVT 50

90% POD = Not Achieved

False Calls = 0 DVT, 3 RVT

 ------ PRED. POD

 X HIT / MISS DATA 

Data did not converge 

DVT

RVT

Probability of Detection (%) 

Figure 37. DVT and RVT POD Results Comparison for Operator 1 

Actual Crack Length–Inch

52 
Table 9. POD Summary Comparison Between DVT and RVT 

Crack 

Length 

(in.) (cm) 

DVT 

Avg. 

(percent) 

RVT 

Avg. 

(percent) 

Operator 1 

DVT 

(percent) 

Operator 1 

RVT 

(percent) 

0.5 1.27 27 55 59 27 

1.0 2.54 36 61 69 41 

1.5 3.81 41 64 74 50 

2.0 5.08 45 61 78 57 

2.5 6.35 48 67 80 62 

3.0 7.62 51 68 82 66 

3.5 8.89 53 70 84 69 

4.0 10.16 55 70 85 72 

4.5 11.43 56 71 86 74 

5.0 12.70 58 72 86 76 

5.5 13.97 59 72 87 77 

6.0 15.24 61 73 88 79 

53 
54 
6.0 Eddy Current POD Evaluations 

Eddy current testing, as defined in the American Society for Nondestructive Testing NDT 

Handbook–Volume 10, is an NDT method in which eddy current flow (electrical current induced 

in a conductor by a time varying magnetic field) is induced into a test object. Changes in the 

flow caused by variations in the test object are reflected into a nearby coil, coils, Hall effect 

device, or other magnetic flux sensor for subsequent analysis by suitable instrumentation and 

techniques.8 

The test specimens and test bed protocol available at TTC were used to perform a baseline 

evaluation of two eddy current systems. Researchers at TTCI monitored and documented the 

results of POD evaluations, using the eddy current test method, for both circumferential butt 

welds and fillet weld terminations. The industry participants performing the ET inspections were 

certified/experienced in the method and provided their own “eddy current inspection procedures 

and equipment” for the evaluations. The ET instrumentation used is commercially available. 

The two systems demonstrated reasonable reproducibility on calibration test panels made from 

actual tank car material (sections) and containing both EDM slots and fatigue cracks of varying 

sizes. The nature of the procedure was to “null” on a section of the weld that was away from the 

“calibration flaws” and to move over the flaws to produce both a response and an estimate of the 

depth of the flaw at peak signal output values. All measurements were made by the operator and 

all data was collected and recorded by TTCI personnel. 

Results of the preliminary data analyses and POD characterization indicated that the method was 

not effective for these applications. Before finalizing the data analyses, it was necessary to 

review the data collection files, reassess the condition of the test specimens (as they had been in 

storage for a considerable time period), and to re-verify the location and size of the largest cracks 

that were not detected by the NDT procedure. Panel condition, response to MT inspections, and 

large crack locations were all re-verified. The preliminary POD analyses and characterizations 

were re-verified. What was assumed to be a large 11.3-inch long crack was not detected in the 

fillet weld specimens and that miss negatively influenced the POD results. After re-evaluation of 

the panel data and panel flaw location itself, it was determined that this indication was in fact 

two separate cracks approximately 6 and 5 in, respectively, from two different weld terminations 

on the same test panel. Since that miss was discovered to be the sum of two cracks, the larger 

11.3-inch crack was deleted from the data, and the POD analysis was repeated. The results 

presented represent the final verification of flaw sizes and locations. 

6.1 Eddy Current POD Evaluation Results 

Tank car panel evaluations using the ET method were performed in accordance with procedures 

provided by the industry participants. Required parameters were verified and documented by a 

TTCI test representatives prior to testing. 

Figure 38 shows the results for the ET butt weld evaluations for each of the two participating 

operators. Figure 39 shows the results of the combined average butt weld POD for the operators. 

Figure 40 shows the results for the fillet weld evaluations for each of the two participating 

operators and Figure 41 shows the combined average fillet weld POD for the operators. Table 

10 lists the actual percentage POD. 

55 
0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5

ACTUAL CRACK LENGTH - (Inch)

0

10

20

30

40

50

60

70

80

90

100

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : Butt Weld Panels 

Condition: Sand Blasted

Method: Eddy Current

Operator: Operators 9 and 10 Comparison

Opportunities = 74

Detected = 62 - Op 9, 44 - Op 10

90% POD = 0.77 Op 9, Op 10 Not Achieved

False Calls = 4

 ------ PRED. POD

 X HIT / MISS DATA 

Op 10

Op 9 

Probability of Detection (%) 

Actual Crack Length–Inch

Figure 38. ET Butt Weld POD Operator Results Comparison 

for Industry Participants 

0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50

A

0

10

20

30

40

50

60

70

80

90

100

Probability of Detection (%) 

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : Butt Weld Panels 

Condition: Sand Blasted

Method: Eddy Current

Operator: Operators 9 and 10 Avg

Opportunities = 148

Detected = 106

90% POD = Avg at 2.49-inch

False Calls = 18

 ------ PRED. POD

 X HIT / MISS DATA 

Actual Crack Length–Inch CTUAL CRACK LENGTH - (Inch)

Figure 39. Combined Average of ET Butt Weld POD Operator Results 

56 
Results for the ET butt weld POD evaluations show that Operator 9 achieved an 85-percent POD 

at a crack length of 0.50 in and was at 92 percent at the 1-inch crack length. The maximum POD 

achieved by Operator 9, at a 3.5-inch crack length, was 98 percent. Operator 10 achieved a 37-

percent POD at 0.50 in, a 60-percent POD at the 1-inch crack length, and a maximum POD at 3.5 

inches of 89 percent and did not achieve 90 percent POD. The combined average POD for the 

ET evaluations at 0.50 in was 61 percent, 76 percent for 1 in, with a maximum of 93 percent at 

3.5 in. 

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

0

10

20

30

40

50

60

70

80

90

100

ACTUAL CRACK LENGTH - (Inch)

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Eddy Current Operator Comparison

Operator: Operators 9 and 10

Opportunities = 104 / 116 Sites

Detected = 81 Op 9, 82 Op 10

90% POD = Op 9 Not Achieved, 5.35-inch Op 10

False Calls = 1 - Op 9, 20 - Op 10

 ------ PRED. POD

 X HIT / MISS DATA 

Op 9 

Op 10

Actual Crack Length–Inch

Figure 40. ET Fillet Weld POD Operator Results Comparison 

for Industry Participants 

57 
0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

CTUAL CRACK LENGTH - (Inch)

0

10

20

30

40

50

60

70

80

90

100

PROBABILITY OF DETECTION (%)

Probability of Detection (%) 

Data Set: Tank Car

Test Object : Fillet Weld Panels

Condition: Sand Blasted

Method: Eddy Current Combined Average

Operator: Operators 9 and 10 

Opportunities = 208 / 232 Sites

Detected = 163

90% POD = 5.95

False Calls = 21

 ------ PRED. POD

 X HIT / MISS DATA 

Actual Crack Length–Inch A

Figure 41. Combined Average of ET Fillet Weld POD Operator Results 

Results for the ET fillet weld POD evaluations show that Operator 9 achieved a 70-percent POD 

at a crack length of 0.50 inch and was at 77 percent at the 1-inch crack length. The maximum 

POD achieved by Operator 9 at a 6.0-inch crack length was 89 percent and did not achieve 90-

percent POD. Operator 10 achieved a 54-percent POD at 0.50-inch crack, a 68-percent POD at 

the 1-inch crack length, and a maximum POD at 6.0 inches of 91 percent. The combined 

average POD for the ET evaluations at 0.50 in was 62 percent, 72 percent for 1-inch, with a 

maximum of 90 percent at 6.0 in. 

58 
0

10

20

30

40

50

60

70

80

90

100

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 5.50 6.00

ACTUAL CRACK LENGTH - (Inch

0

10

20

30

40

50

60

70

80

90

100

)

PROBABILITY OF DETECTION (%)

Data Set: Tank Car

Test Object : BW and FW Panels 

Condition: Sand Blasted

Method: Eddy Current

Operator: BW and FW Comb Avg

90% POD = 2.49-inch Butt Weld

 5.95-inch Fillet Weld

False Calls = 39 

Fillet Weld 

Avg POD

Butt Weld 

Avg POD

Figure 42. Comparison of ET Butt Weld and Fillet Weld POD Average Results 

Table 10. POD Summary for Eddy Current Inspections 

Crack 

Length POD Average Percentage 

(in.) (cm) 

BW 

Avg. 

(percent)

FW 

Avg. 

(percent) 

Operator 9 

BW 

(percent) 

Operator 10 

BW 

(percent) 

Operator 9 

FW 

(percent) 

Operator 10

FW 

(percent) 

0.5 1.27 61 62 85 37 70 54 

1.0 2.54 76 72 92 60 77 68 

1.5 3.81 83 78 95 72 80 75 

2.0 5.08 87 81 96 79 83 79 

2.5 6.35 90 83 97 83 84 82 

3.0 7.62 92 85 98 86 85 84 

3.5 8.89 93 86 98 89 86 86 

4.0 10.16 87 87 87 

4.5 11.43 88 88 89 

5.0 12.70 89 88 89 

5.5 13.97 89 89 90 

6.0 15.24 90 89 91 

Actual Crack Length–Inch

Probability of Detection (%) 

Gray area represents difference in longest crack size between butt-weld and fillet-weld test specimens. 

59 
A comparison of the combined industry average for tank car circumferential butt welds and fillet 

weld terminations show that the butt weld evaluations produced a higher average POD than did 

the fillet welds. A 90-percent POD was achieved at a 2.5-inch crack length for the butt weld 

evaluations and at approximately a 6-inch crack length for the fillet welds. 

The difference in detection results can be partially related to part geometry and flaw orientation. 

In the butt weld configuration, the eddy current field (eddies) are traveling across a transverse 

flaw that is oriented approximately 90 degrees to the current generation. The fillet weld 

configuration has the field traveling parallel to the flaw, thus producing less of a field change 

than the butt weld configuration. The weld beads for the fillet weld samples are more irregular 

in contour than the butt welds, and the eddy current probe did not readily fit up against the weld, 

which likely affects detection at those locations containing cracks. In addition, cracks at the toe 

of fillet welds are known to grow back under the toe, therefore making the target crack less 

accessible to the eddy current probe. Difficulties in probe fit up were noted in other inspections 

performed on these specimens. Since the test specimens are panels that were cut from retired 

tanks cars, the weld bead contour and irregularities are thought to be representative of tank cars 

in service. This condition must be addressed through eddy current probe and procedure 

optimization based on the suspected orientation of flaw(s) expected. 

The eddy current inspection procedure result showed greater variability than was expected based 

on the narrow bounds of responses obtained from the calibration specimen containing EDM slots 

of known length and depth. The instrument was capable of producing reproducible results, and 

the operators were skilled and experienced in the test method. 

60 
7.0 Bubble Leak Testing 

The tank car panel evaluations using the bubble leak test method were performed in accordance 

with the AAR Manual of Standards and Recommended Practices, Section C – Part III, 

Specifications for Tank Cars, Specification M-1002, Appendix T, Part T3.00–Leak Testing (LT), 

dated October 1, 2003.7

 A bubble leak testing (BLT) procedure setup sheet was established, and 

the parameters identified on the sheet were verified by both the evaluating technician and a TTCI 

test representative prior to inspections. The parameters required identification and 

documentation of the BLT equipment used to aid in the inspection (e.g., bubble leak fluid, 

flashlight, magnifying glass, and mirrors) and the light intensity at and around the inspection 

surface.7 

Leak testing is performed in industry by various methods ranging from visual detection of fluid 

escapement and time pressure loss to pressurized helium leak. The method employed depends 

on the materials involved and the size of the leak to be detected. Bubble (soap) leak testing is 

one of the simplest, most economical methods for a wide variety of applications. BLT is a 

standard testing method for tank car inspection and maintenance. It is applied to all tank wall 

penetrations such as valves, vents, and manways. The procedure involves: 

• Pressurizing the tank 

• Applying the BLT fluid to the interface being tested 

• Allowing the BLT fluid to dwell 

• Visually inspecting for the presence of bubbles 

• Documenting and reporting inspection results 

The rate and activity of bubble formation are indicators of the size of the leak. 

Although handbook data is often cited for BLT capabilities, no supporting quantitative data was 

found by a review of the literature. Further, the actual leak test procedures in use vary in the 

pressures (differential pressures) used, type of fluid used, dwell time, observation methods, 

lighting, direct visual or aided visual, and criteria for assessment. 

7.1 Bubble Leak Specimen Preparation 

An innovative approach to specimen design was crafted that involved cutting representative 

sections containing manways and other outlet hardware from retired tank cars. BLT is applied to 

tank car flanges, outlets, and manways that are pressure sealed by a gasket. All of the openings 

(ports) were built to accommodate variations in O-ring or molded gasket thickness and could 

accommodate double gaskets and an intervening test ring containing leak sites. 

For purpose of this test, sections of retired tank cars were cut out to utilize service flanges and 

manways. These test samples were arranged together such that multiple flanges were 

incorporated into a mock-up test section and each test port was provided with a separate pressure 

chamber with a relatively small volume. This enabled pressurizing and monitoring of each port 

separately. Leak paths were created by inserting a thin ring between two gaskets at each test 

location (see Figures 43 and 44). 

61 
Figure 43. Ring used for Flange BLT Evaluations Containing 

Known Leak Paths 

Figure 44. Several Samples with Different Known Leak Path Sizes 

The ring can either fit outside the bolt pattern or can be incorporated in the bolt pattern. The 

material can be soft copper, brass, or other corrosion resistant material (easy to drill and stable). 

This type of artifact can be built for a manway flange or any size to fit other flanges such as at 

62 
the ball valve sites. Holes may be randomly blind (no leak path) or through drilled. The ring 

may be retained within the bolt circle or may be fit to the bolt circle. The test rings were 

identified for use in specific locations on the mock-up test sections and could be changed out 

between tests to provide multiple test opportunities at each test section location within the mockup. 

The leak path was then created at selected holes. Several methods of producing fine leaks were 

assessed, and the method used was to plug the hole with a wood insert that could be grooved on 

the side to produce various size pin holes. Some problems occurred in using wood, but the 

method was inexpensive and timely for this test. It was necessary to “calibrate each leak” near 

the time that the test was conducted to assure that changes in the leak path had not occurred. 

Calibration of each leak was done by the positive displacement method. The flange was inserted 

into a test fixture and pressurized to the level to be used in the leak test. The leak rate was 

measured by sealing a tube over the leak that led to a burette tube as Figure 45 shows. Both 

temperature and pressure were precisely controlled during these measurements. The volume 

displaced in a given time interval was the basis for the leak rate for each leak site. 

Figure 45. Leak Test Capture Device for Leak Rate Calibration 

Precision of the measurement could be increased as required by extending the time for 

collection. Water temperature was held constant and pressure in the test fixture was also held 

constant. The calibration process was repeated for each leak artifact. 

The test rings were then mounted in the tank car (mock-up) sections and were changed out to 

provide the range of detection opportunities for detection capabilities assessments. Figure 46 

shows one of the tank car (mock-up) sections. A reference calibration specimen containing leaks 

of varying sizes was assessed before and after each programmed test sequence. This specimen, 

63 
shown in Figures 47 and 48, provided an indication of the reproducibility of the detection 

procedure and the operator. 

Figure 46. Tank Car (Leak Test Mock-Up) Section 

Containing Multiple Ports 

Figure 47. Leak Test Master Gage Test Specimen 

64 
Figure 48. Leak Test Master Gage Test Specimen 

Showing Calibration Leak 

Each inspection port location was connected to an individual small pressure chamber. Figure 49 

shows one of the chambers with the air hose connection and in-line pressure gage. Figure 50 

shows bubble formation and inspection at one of the ports. 

Figure 49. Leak Test Master Gage Test Specimen Showing 

Air Hose Connection and Pressure Gage 

65 
Figure 50. Leak at an Inspection Port 

7.2 Bubble Leak Test Procedure Assessments 

Inspectors from various rail car maintenance facilities were invited to participate in the baseline 

test assessments. They used their own test fluids, visual aids, and procedures. Test pressures 

and dwell times were adjusted to comply with each respective test procedure. Both 

commercially supplied leak test fluids and fluids from common household formulas were used 

by the inspectors. Optical and lighting aids were provided to each inspector in accordance with 

the requirements of their test procedures. 

The test-calibrated leak artifact rings were then installed in respective sites within the leak test 

specimens and were initially pressurized to assure sealing at each installed site. Specimens were 

then pressurized and NDT personnel from tank car service organizations performed leak tests 

using the materials and procedures that were used at their respective test facilities. Each 

inspector verbally identified the leak site position and provided a semi-quantitative assessment of 

leak size in terms of small, medium, and large. Inspectors had a personal preference for the type 

of leak test fluid used and method of application. Results of each assessment were documented 

by TTCI researchers and input to the leak test database. 

7.3 BLT Data Analysis and Results 

Test data was analyzed by the HIT/MISS method to provide a POD analysis in the same manner 

as used for other NDT methods and procedures. Plots were made in the form of probability of 

detection as a function of the leak rate (as measured by the positive displacement method). Initial 

results were somewhat surprising. The data showed that detection decreased as the leak rate 

increased. This was counterintuitive and data was re-analyzed to validate the initial results. 

Because few large leaks and few measurements on those leaks existed, the data was truncated at 

20 cm3

/min for final data analyses. Figures 51–54 show the results. 

66 
0

10

20

30

40

50

60

70

80

90

100

0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 0.180 0.200

Leak Rate - cc/min

PROBABILITY OF DETECTION (%)

Data Set: Tank Car Leak

Test Object : Calibrated Leaks in Flanges

Condition: Service

Method: Bubble Leak Test

Operator: 1

Opportunities = 79

Detected = 62

90% POD= Not Achieved

False Calls = 9

 X HIT / MISS DATA 

Figure 51. Operator 1 BLT POD Results 

0

10

20

30

40

50

60

70

80

90

100

0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 0.180 0.200

Leak Rate - cc/min

PROBABILITY OF DETECTION (%)

Data Set: Tank Car Leak

Test Object : Calibrated Leaks in Flanges

Condition: Service

Method: Bubble Leak Test

Operator: 3

Opportunities = 62

Detected = 45

90% POD= Not Achieved

False Calls = 6

 X HIT / MISS DATA 

Figure 52. Operator 3 BLT POD Results 

67 
0

10

20

30

40

50

60

70

80

90

100

0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 0.180 0.200

Leak Rate - cc/min

PROBABILITY OF DETECTION (%)

Data Set: Tank Car Leak

Test Object : Calibrated Leaks in Flanges

Condition: Service

Method: Bubble Leak Test

Operator: 6

Opportunities = 62

Detected = 48

90% POD= Not Achieved

False Calls = 6

 X HIT / MISS DATA 

Figure 53. Operator 6 BLT POD Results 

0

10

20

30

40

50

60

70

80

90

100

0.000 0.020 0.040 0.060 0.080 0.100 0.120 0.140 0.160 0.180 0.200

Leak Rate - cc/min

PROBABILITY OF DETECTION (%)

Data Set: Tank Car Leak

Test Object : Calibrated Leaks in Flanges

Condition: Service

Method: Bubble Leak Test

Operator: 6

Opportunities = 61

Detected = 48

90% POD= Not Achieved

False Calls = 6

 X HIT / MISS DATA 

REMOVED LARGEST 

LEAK WHICH WAS

MISSED BY ALL

Figure 54. Operator 6 POD Results With Largest Leak Removed (missed by all) 

68 
7.4 Bubble Leak Test Conclusions and Recommendations 

No prior quantitative data on BLT capabilities were found in the literature. This task has 

addressed an innovative approach to simulating leak detection opportunties on hardware taken 

from retired tank cars and modified to provide tank car (mock-up) section hardware containing 

multiple ports. Test ports are from retired tank cars and represent configurations that must be 

inspected in field operations. Thin, interchangeable test rings were installed with double gaskets 

at test locations to provide a test bed for multiple leaks and leak rates at all port locations. The 

calibration of individual leak rates by a positive displacement method provided an economical 

method of baseline measurements. 

The results show that detection of small leaks (within the range of the test) is strongly influenced 

by the operator and operator procedure. Small leak detection was in the range of 80 to 90 

percent based on extrapolation from the smallest crack in the data set. Results of detection on 

the master gage specimen showed a variation in detection of small leaks and large leaks. This is 

inherent to the inspection method and is not considered to be significantly different for different 

operators. 

Data is consistent in demonstrating that as the leak size increases, the detection rate decreases. 

This is counterintuitive but is consistent with field reports and experiences. On further 

consideration, large leaks are observed to blow away the leak test fluid and will result in failures 

in detection. BLT, therefore, must be considered to be a qualitative inspection that is applicable 

to many needs and requirements. Failure to detect a large leak by this method must be 

considered to be a limitation of the method and not that of the capability of the inspector. For 

critical leak applications, alternate methods must be considered for detection of both small and 

large leaks. 

69 
70 
8.0 Summary of Phases I, II, and III 

During Phase I, a test specimen library was established and baseline inspection capabilities for 

detection of cracks in butt welds using NDI methods that are commonly applied to railroad tank 

car inspection and in-service maintenance. A protocol for assessment and quantification of 

detection capabilities was established for use in assessment applications. The assessment 

methods are based on those that are well established in the aircraft industry as the basis for 

hardware component life extension and operational safety. 

During Phase II, master gage calibration specimens were fabricated and characterized to provide 

a capability for assessing the reproducibility of inspection methods and inspection system 

responses at various locations. Both EDM slots and fatigue cracks were induced in subscale test 

specimens in both butt welded and fillet welded panels. Slots and cracks were produced at three 

sizes that are typical of the sizes to be detected per safety and life extension goals. Three-point 

calibration provides a signature of inspection system responses and enables linking responses to 

baseline capabilities data, to provide confidence that the inspection system is performing 

consistently and is consistent with established capabilities. In addition, a fillet welded specimen 

test set was produced by inducing fatigue cracks of varying sizes in welds that were cut from 

retired tank cars. Cover shrouds were installed with insulation that is representative of service 

tank cars for use in assessing baseline remote visual inspections. Baseline data collection was 

initiated for the fillet welded specimens using direct VT, remote VT, MT, LT, and UT inspection 

methods. 

During Phase III, baseline capabilities for the fillet welded specimens were established by 

inspectors from various tank car maintenance organizations. The procedures and equipment 

used by operators in their maintenance facilities were applied to these specimens. In addition, 

test hardware (mock-up tank car sections) was fabricated, characterized, and used to assess 

baseline BLT detection capabilities. Multiple inspectors from tank car maintenance facilities 

used their own procedures, bubble check test fluids, and inspection aids to assess leak detection 

capabilities that closely simulate results that might be expected in field applications. 

TTCI, under sponsorship from FRA, has produced a unique set of representative test specimens 

and has used these specimens to establish baseline detection capabilities for NDI procedures that 

are used to assess tank car structural integrity and to provide a quantitative basis for life 

extension and a quantitative measure of service safety (and risks). The defect library, housed at 

TTC, includes girth weld, fillet weld, and BLT specimens. These inspection tools provide a 

resource for quantification of detection capabilities, a database to quantify the capabilities, and 

potential for application of new or alternative inspection procedures. The TRIC is also expected 

to assist in providing industry, regulatory, and academia with a common resource for 

demonstrating and implementing improvements in tank car operational safety and reliability. 

71 
72 
9.0 Path Forward 

The resources developed in support of this FRA sponsored program and the capabilities 

demonstrated are initial steps in providing quantitative data for extending and validating the 

detection capabilities of NDI methods, processes, and procedures. Development of these tools is 

also directed towards assisting in providing continuous improvements in operational tank car 

safety and reliability. The test bed can be used to support and provide quantitative data for the 

hardware and inspection methods assessed and can be extended to provide a common baseline 

for other railroad components and operational requirements. 

Results show variability in operators and procedures both of which can be influenced by 

training, experience, and how recent the operator has performed the inspection. Baselining these 

influences are expected to assist the industry in quantifying the capability of each of the 

inspection methods used for inspection of railroad tank cars. Through continued POD evaluation 

the determination of minimum detectable flaw size along with the POD for critical flaw sizes for 

each of the NDE methods can be achieved. 

Continuation of POD related efforts include: 

• Industry PODs on girth welds, fillet welds, and BLT samples 

• POD evaluations of allowed NDE methods along with evaluation of applicable NDE 

methods such as; 

o Ultrasonic phased arrays 

o Digital radiography 

o Thermography 

• Inspector training 

• Master gage and tank car sample development 

73 
74 
References 

1. Federal Register, Code of Federal Regulations, Sections 179 and 180, June 1996. 

2. National Transportation Safety Board, “Safety Recommendations,” R-92-21–R-92-24, 1992. 

3. Garcia, G.A., “Railroad Tank Car Nondestructive Methods Evaluation,” DOT/FRA/ORD01/04, January 2002. 

4. U.S. Department of Transportation, Federal Aviation Administration, “National Aging 

Aircraft Research Program Plan,” October 1993. 

5. Rummel, W.D., “Qualification and Validation of the Performance Capability (POD) for 

Nondestructive Inspection Procedures,” 16th World Conference on Nondestructive Testing, 

August 30–September 3, 2004. 

6. U.S. Department of Defense, Military Handbook 1823 (MIL-HDBK-1823), “Nondestructive 

Evaluation System Reliability Assessment,” April 1999. 

7. AAR Manual of Standards and Recommended Practices, Section C–Part III, Specifications 

for Tank Cars, Specification M-1002, Appendix T, October 1, 2003. 

8. American Society for Nondestructive Testing, Nondestructive Testing Handbook, 2nd Ed.: 

Vol. 10, “Nondestructive Testing Overview,” p.529, 1996. 

75 
76 
Appendix A. 

77 
78 
79 
80 
81 
82 
83 
84 
85 
86 
Appendix B. 

Operator Profile

Name: ______________________________ 

Date: _____________________________ 

Company Name: ____________________________

Job Title/ Position: __________________________

Training: _______________________________________________________________ 

 _______________________________________________________________ 

 _______________________________________________________________ 

Certifications: ___________________________________________________________ 

 

Number of Years with Company: __________________________________________ 

Number of Years in NDE: ________________________________________________ 

Number of years on Current Job: __________________________________________

Welding Background: ____________________________________________________ 

 ____________________________________________________ 

 ____________________________________________________ 

Tank Car Background: ___________________________________________________ 

 ___________________________________________________ 

 ___________________________________________________ 

 ___________________________________________________ 

Comfort Level with 

Inspections: ___________________________________________________ 

 ___________________________________________________ 

 ___________________________________________________ 

Additional Information: __________________________________________________ 

 __________________________________________________ 

 __________________________________________________ 

 __________________________________________________ 

 __________________________________________________ 

87 
88 
Acronyms 

ASTM American Society for Testing of Metals 

AUT automated ultrasonic testing 

BLT bubble leak testing 

CFR Code of Federal Regulations 

DOT Department of Transportation 

DTA damage tolerance analysis 

DVT direct visual testing 

EDM electrodischarge machine 

ET eddy current testing 

FAA Federal Aviation Administration 

FRA Federal Railroad Administration 

HMR Hazardous Materials Regulations 

LP, PT liquid penetrant testing 

MP, MT magnetic particle testing 

NDE nondestructive evaluation 

NDI nondestructive inspection 

NDT nondestructive testing 

NTSB National Transportation Safety Board 

POD probability of detection 

RT radiographic testing 

RVT remote visual testing 

TRIC Tank Requalification and Inspection Center 

TTC Transportation Technology Center (the Site) 

TTCI Transportation Technology Center, Inc. (the Company) 

UT ultrasonic testing 

VT visual testing 

89 
90