Configuration design for sensor integration S. F. El-Hakim and J.-A . Beraldin Institute for Information Technology , National Research Council , Ottawa , Ontario , Canada K1A 0R6 Abstract Due to the nature of many applications , it is difficult with present technology to use a single type of sensor to automatically , accurately , reliably , and completely measure or map , in three-dimensions , objects , sites , or scenes . Therefore , a combination of various sensor technologies is usually the obvious solution . There are several 3-D technologies , two of which ; digital photogrammetry and triangulation laser scanning , are dealt with in this paper . The final accuracy achieved by the combination of various sensors is a factor of many sensor parameters and the quality of the image coordinate measurements of each sensor . The effect of those parameters must be determined to maximize the overall quality within the constraints imposed by the requirements of the application . The parameters affecting the accuracy of measurement , the test laboratory , and test results using intensity and range data , are presented . The configuration design of intensity and range sensors is discussed based on the results presented here and in two previous papers . 1 . INTRODUCTION 1.1 . Previous Work The work presented in this paper is a follow-up on two previous papers . In the first1 we presented the rational behind integrating range and intensity data for dimensional measurement purposes . A procedure for integrating the data was also presented along with some results . In the second paper2 accuracy analysis based on extensive testing on various types of objects and features using different sensors were presented . The findings of the previous work are summarized as follows : l Range cameras using scanning light -LRB- laser -RRB- spot may produce erroneous results when sudden changes in surface height occur , on surfaces with large reflectance variations , and on rough surfaces . This is explained by the fact that in practice , the laser ray projected onto a scene is not infinitesimal in diameter . Hence , when the laser spot crosses a transition the imaged spot will lose most of its symmetry . As the laser spot crosses the reflectance transition , the centroid of the light distribution will shift to indicate for example longer distances between the camera and the object being inspected . The result is a small height bump in the range map near the transition . Similarly , when a height step is crossed , the laser spot on the position detector will provide erroneous range data . We have demonstrated that the use of the registered intensity image generated by a regular True centroid Spot distribution on position detector D p Scan direction Laser spot : Measured centroid High reflectance Low reflectance : True centroid Figure 1 : Edge error due to scanning spot . range camera can be used advantageously to alleviate the impact of erroneous range on edge measurements . l An external CCD camera or more may be useful for certain types of objects requiring special lighting , such as using backlight to improve edge extraction . It may also be desirable in cases where the requirements for the range image , such as ; speed , resolution , or pixel size , differ from those required for the intensity image in order to achieve good intensity data on all edges . Initial tests showed that , at similar resolutions , the same results are to be expected either from integrating the intensity and range from same range camera , or from integrating the intensity from a separate CCD camera . l A procedure for integrating both types of data for precise measurements on edges has been presented . The procedure takes advantage of the best properties , in terms of accuracy , of each type of data . Surface measurement and modeling are carried out using range data alone , while edge measurement employs both range and intensity . Initial tests confirmed that erroneous range data on edges could be corrected by the intensity data and the achieved accuracy is within the expected precision of the camera . l On well defined targets , the accuracy obtained by active range systems alone is lower than the one obtained by a passive photogrammetric system -LRB- ranges from 1 : 3000 to 1:7500 compared to 1:20000 or better -RRB- . Therefore , for applications allowing placement of targets on the object surfaces and requiring high accuracy , digital photogrammetric systems are the obvious choice . l On edges , photogrammetric systems still provide better accuracy than active range systems using intensity and range data -LRB- 1:15,000 or better compared to 1:7500 -RRB- . It should be noted , however , that the active systems provide a more complete 3-D data on all the visible edges in the scene while the passive system will be affected significantly by the ambient light and may require several camera arrangements to extract 3-D coordinates on all edges . More results are presented in this paper . l On untargeted surfaces , active range systems provide a complete 3-D map of the visible surfaces with about 1:3500 accuracy while passive systems may not be able to perform any measurement without distinguished features . Additional tests showing that improved accuracy can be achieved with active range sensors are presented here . l From the above short comparison , selection of a vision technology is largely dependent on the type of feature to be measured , the required accuracy , and the required density of data -LRB- completeness -RRB- . In some applications , when a variety of features are required to be measured , a combination of different technologies may be the answer . 1.2 . Scope The results presented in the previous papers were intended to help system designers in making the choice of a vision system or a combination of systems to suit their application . Once this selection is made , there are several configuration parameters that must be selected to guarantee the desired accuracy . In this paper we provide an evaluation of the effect of different system parameters on the final accuracy of dimensional measurement by various sensors and combination of different types of data . Since it is very difficult to theoretically investigate the effect of all the parameters and configurations , the analysis presented here is based on extensive laboratory testing under conditions similar to those expected in practice . The paper is organized as follows . In the next section we will identify the important parameters affecting the measurement accuracy of intensity-based and range-based vision systems . In the third section , the laboratory and tools used for the experiments are described . In section 4 , some definitions and standards followed in those experiments are defined . The results of the tests performed to study the effect of the various configurations and parameters are summarized in section 5 . In section 6 , based on the results presented in this paper and the previous two papers ,1,2 we present the possible configurations and how to select a particular design to meet the requirements of the application . Concluding remarks are given in section 7 . 2 . CHARACTERIZATION OF PARAMETERS There are several types of parameters that characterize a vision sensor or system . We will classify them , in a way that will guide the tests presented later , as follows -LRB- the reader will no doubt find other ways of classification elsewhere -RRB- : 1 - Constant internal sensor parameters -LSB- CISP -RSB- : Those are always the same and do not vary by different camera set ups . 2 - Variable internal sensor parameters -LSB- VISP -RSB- : Those may take different values depending on camera set up . 3 - External geometric configuration parameters -LSB- EGCP -RSB- : The geometric relationship between the sensors in the system . They are important for all triangulation-based systems . 4 - Application and environmental parameters -LSB- AEP -RSB- : Those are not related to the sensor itself but to the way it is used , such as quality of calibration , type of feature , quantization method , environment conditions , object range , and the number of repeated measurements . The first two classes of parameters are usually computed by sensor calibration . If we can identify the CISP , they can be calibrated in the laboratory , or by the sensor manufacturer , under the best possible conditions and then held fixed for this sensor . On the other hand , the VISP have to be recalibrated for every camera set up , either because they are unstable by nature or because they are user-adjustable . For standard CCD-based systems , the different types of parameters are well known and they can be easily classified according to the above criteria . Custom made sensors and those not extensively tested for measurement applications , such as many of the active range sensors , require an elaborate testing procedure to identify the different types of parameters and achieve the optimum performance of the sensor . In order to identify which parameters are CISP and which are VISP , we will make one assumption . Within the calibration zone -LRB- figure 2 -RRB- all internal sensor parameters are valid . This zone may be defined as the volume where reference calibration targets or objects have been placed . Outside this zone the VISP computed by calibration may not be valid . One of the test objectives is to determine which parameters can be retained outside the calibration zone and which ones must be recalibrated . In the next section , more specific identification of the system parameters are given for the two types of sensor addressed here . 2.1 . Passive intensity-based sensors Passive intensity-based systems , particularly those using CCD cameras , have been extensively tested for metrology purposes in photogrammetric literature,6-13 and to a much lesser extent in the computer vision literature .14 Parameters and sources of error are well understood , for example : CISP : VISP : EGCP : AEP : pixel size and resolution focal length and distortion parameters base-to-range ratio ambient light , in addition to the examples given above . camera Figure 2 : Calibration zone One objective of this paper is to similarly identify the classes of parameters for active range-based sensors . 2.2 . Active range-based sensors The parameters defined here are for the autosyncronous laser -LRB- ASL -RRB- range camera developed at NRC .3,4 Initially some obvious parameters can be identified : CISP : the fixed angles and spacing between the different internal components of the camera -LRB- shown in figure 3 -RRB- . Other parameters , such as the increment and origin of the scanning angles in x and y directions and the various distortion parameters are not obvious if they are CISP or VISP . We will try to identify the types of parameter in the tests presented below . The stability of all internal parameters is also affected by the quality of the camera components , thus constructing a sensor for precision measurements requires high quality components . For this type of sensor , all geometrical parameters are internal to the sensor , thus there are no EGCP . The AEP are the same as for passive systems , with one important exception . This sensor is not affected by the ambient light . However , the effect of temperature will be significant since any change in the internal dimensions between the components will produce erroneous results . For applications where significant change in temperature is expected , for example more than two degrees , the camera should be calibrated at a wide range of temperatures and the calibration parameters are selected according to the current application temperature . Testing at various temperatures is still in progress , however preliminary results indicate that variation of temperature up to 2oC does not result in significant changes in camera parameters . b p fo -LRB- x , z -RRB- Z X a a S d - dx d hy Y-mirror Figure 3 : Some internal parameters of the ASL camera 3 . THE CALIBRATION AND EVALUATION LABORATORY -LSB- CEL -RSB- 3.1 . General description A laboratory at the Institute for Information Technology of the National Research Council of Canada , has been dedicated to calibration and evaluation of machine vision sensors and systems . Specifically , the objectives of CEL are : 1 - Performing precise model-based calibration of various types of sensors and systems and provide internal precision numbers for the sensors . 2 - Monitoring sensor stability over time and under variable environment conditions such as temperature and ambient light . 3 - Evaluating system geometric measurement accuracy , with extensive statistical analysis , using a wide range of specially designed standard objects and high - precision positioning devices . 4 - Validating computer vision algorithms , such as target and edge measurement , multi-view registration , model-based recognition , and sensor fusion . The laboratory -LRB- figure 4 -RRB- is currently equipped with : 1 - Precise targets in various arrangements . 2 - Optical bench , with vibration isolators , and custom mounting devices . 3 - High precision positioning devices such as translation and rotation stages . 4 - Theodolites and electronic distance measurement -LRB- EDM -RRB- devices . 5 - Standard test objects , of different shapes and sizes , such as planes , spheres , cylinders , and straight and circular edges . 6 - PC and SGI workstation . 7 - Different light sources . 8 - General laboratory equipment such as thermometer , barometer , height gauges , and VCR . 9 - Software tools include , among others ; - Calibration based on sensor model , including added distortion parameters . - Measurement and inspection using 3-D data produced by various types of vision systems -LRB- figure 5 -RRB- - Display and manipulation of 3-D data files . - Statistical analysis packages . - Computer-aided design -LRB- CAD -RRB- . Figure 4 : Part of the CEL showing camera mounts , translation stage and targets 3.2 . The integrated vision measurement -LRB- IVM -RRB- system Figure 5 shows the general design of the integrated vision measurement -LRB- IVM -RRB- system . IVM SOFTWARE calibration PC statistical analysis Controlling Hardware & image storage Range Image Intensity Image input scene camera system measurement requirements Target Coordinates X , Y & Z Edge Parameters Circle Line Distances Splines ... . Surface Parameters Sphere Cylinder Plane Splines ... . output data Figure 5 : General design of the IVM system The IVM software , used for all the tests presented below , has the following capabilities : 1 - Performs rigorous camera calibration , according to the procedure described in section 4.2 , on CCD and various active range sensors developed at NRC , such as the ASL range camera . 2 - Converts raw data from an active range camera into corrected XYZ coordinates for every pixel . 3 - Performs low-level vision processing on intensity or range images , such as edge detection , target recognition , thresholding , and region segmentation . 4 - Integrates data from registered intensity and range images to improve measurements on edge points , according to the procedure described in an earlier paper .1 5 - Fits geometric models , such as spheres , cylinders , circles , and planes , to data points on edges and surfaces . 6 - Provides complete statistical analysis of the computed parameters and observations such as variance-covariance matrices and residual analysis . 4 . TEST PROCEDURE 4.1 . Standards for performance tests ANSI standards for automated vision systems-performance test-measurement of relative position of target features in two dimensional space5 , will be followed here , with changes to suit the 3-D space . Some of the definitions from those standards , which will be used in this paper , are : Accuracy : The degree of conformance between a measurement of an observable quantity and a recognized standard or specification that indicates the true value of the quantity . Field of Measurement -LSB- FOM -RSB- : The area within which targets or target clusters can be positioned . Field of View -LSB- FOV -RSB- : The area of space imaged at the focal plane of a camera . Repeatability : The degree to which repeated measurements of the same quantity vary about their mean . Target : A test object or objects having shape , size , position , or relative position characteristics that are known and traceable to recognized standard of measurement . Test Controller : The computer or other mechanism that initiates the actions , communications with the system under test , and records data necessary for performance of the defined test . According to the standards , the equipment required for the test are : a machine vision system ; a test controller ; a target with associated lighting ; and a translating table . All these equipment are part of CEL described above . Some of the notable standards are : l The accuracy of the known target dimensions or relationships between targets shall be at least three times the measured accuracy of the vision system for the test to be valid . This also applies to the accuracy of any translation device . All the test objects used for our tests were measured with a coordinate measuring machine -LRB- CMM -RRB- with better than 0.005 mm accuracy , which is well within this standard . l At each target location , the machine vision system shall take ten measurements . The number of distinct points measured shall be at least ninety . It is well known that averaging multiple independent images of the same modality improves the signal to noise ratio by a factor of n where n is the number of images . Therefore the images and the measurements in all the tests must be repeated a sufficient number of times until the standard deviation of the repeated measurements is within the noise level of the sensor . The repeated measurements must also be at varied conditions -LRB- as those expected in the application environment -RRB- where the object is placed at different random positions . l The basic procedure for testing shall be to move the target -LRB- s -RRB- to a random position and orientation and allow the machine vision system to take measurements . In the standards this procedure is intended for multi-point target plate . For our 3-D test objects , we will start with a random position of the object then rotate it by various angles . l For multi-point test , the accuracy is calculated by comparing the nominal distance to the measured distance between every pair of points . We now add one other test procedure : l For object surfaces and 3-D edges , the accuracy is calculated by comparing the given parameters of the surface or edgecurve function to the computed parameters from fitting the measured data to the function . 4.2 . The calibration procedure The calibration procedure for all the vision sensors in the CEL is described in details in earlier publications .1,4 A flat plate carrying targets at known locations -LRB- see figure 4 above -RRB- is imaged by the sensor in at least two positions covering the volume of interest . The positioning of the plate is accomplished by a precise linear stage . The known positions of the targets in the 3 - D object space and their extracted position on the sensor are used to solve for the camera parameters , including any modeled distortion parameters . Figure 6 summarizes this procedure for intensity and range sensors . When both CCD camera and the range sensor are calibrated simultaneously using the same target positions , the two sensors are in registration . A statistical analysis of the quality of the calibration is also provided to make sure that systematic errors larger than the noise level have been eliminated . l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l XYZ Coordinates i & p data 20 parameters Parameters Solution 3-D laser range camera p interpolation 15 parameters i data CCD camera Parameters Solution q segmentation q target extraction q measurement Intensity Processing Intensity Processing Figure 6 : The calibration procedure 5 . EFFECT OF SYSTEM PARAMETERS - TEST RESULTS 5.1 . Passive intensity-based systems Results of measurement on edges using our Vision Coordinate Measurement -LRB- VCM -RRB- system6 are presented . Results on targets are available from many photogrammetric publications,7-13 therefore no further tests on target measurements are presented here . 5.1.1 . Accuracy of edge measurements A test object consisting of a flat sheet metal and several circles -LRB- figure 7 -RRB- is used for the following tests on edge measurements . The `` true '' circle radii were measured with a coordinate measuring machine -LRB- CMM -RRB- to a 0.005 mm accuracy . Two CCD cameras with 512x512-pixels resolution were employed . Edge points were measured to a subpixel accuracy as described in a previous paper .6 The tests were performed at various camera configuration , FOV , and range . Table 1 displays a sample of the results . Accuracy values , represented by the bias as defined in section 4.1 , and the standard deviation of 5 repeated measurements -LRB- using only one image per measurement in order to clearly understand the behavior -RRB- are shown . At the closest range -LRB- table 1-A -RRB- , the accuracy was 0.011 mm , which is about 1:36,000 of the FOV . This FOV is found to be the optimum for the size of this object -LRB- the object is about 20cmx20cm -RRB- . In other FOV set ups , up to FOV equivalent to 3.5 times the object dimensions -LRB- FOM -RRB- , the relative accuracy was about 1:23,000 . At the largest of the three shown set ups , the standard deviation of the repeated measurements was significantly larger than those of the closer ranges -LRB- 0.035 mm compared to 0.021 at slightly closer range -RRB- . At the long ranges , the number of pixels having significant edge information -LRB- along the edge profile -RRB- is small and the edge quantization will vary significantly when small variation in lighting occurs , which is expected to be the case between the repeated measurements . This will be demonstrated in section 5.1.2 . circle # True Rad . Radius Bias 1 7.486 7.495 +0.009 2 10.000 10.007 +0.007 3 12.482 12.504 +0.022 4 14.990 14.986 - 0.004 5 19.998 20.003 +0.005 6 24.987 24.991 +0.004 -LSB- A -RSB- FOV : 40cm x 40cm - range : 200 cm - b/r : 0.40 RMS : 0.011 mm Relative accuracy : 1:36,000 ss : 0.015 mm -LRB- 5 measurements -RRB- circle # True Rad . Radius Bias 1 7.486 7.465 - 0.021 2 10.000 10.013 +0.013 3 12.482 12.522 +0.040 4 14.990 14.951 - 0.039 5 19.998 19.991 - 0.007 6 24.987 24.993 +0.007 -LSB- B -RSB- FOV : 60cm x 60cm - range : 300 cm - b/r : 0.27 RMS : 0.025 mm Relative accuracy : 1:24,000 ss : 0.021 mm -LRB- 5 measurements -RRB- 5 6 2 4 3 1 Figure 7 : Standard object for edge measurement circle # True Rad . Radius Bias 1 7.486 7.450 - 0.036 2 10.000 9.976 - 0.024 3 12.482 12.475 - 0.007 4 14.990 14.977 - 0.013 5 19.998 19.984 - 0.014 6 24.987 25.043 +0.056 -LSB- C -RSB- FOV : 70cm x 70cm - range : 350 cm - b/r : 0.23 RMS : 0.030 mm Relative accuracy : 1:23,000 ss : 0.035 mm -LRB- 5 measurements -RRB- Table 1 : Results of edge measurements of circles -LRB- figure 7 -RRB- - using VCM 5.1.2 . Effect of repeated measurements Figure 8 shows the bias of the measurement on circle 6 , case -LSB- C -RSB- , using the average of various number of repeated measurements . The case shown had the largest bias -LRB- 0.056 mm -RRB- when only one set of 5 images were used . Further repeating of the measurements gradually reduced the error . Although a bias lower than the precision was sometimes achieved , the standard deviation was large -LRB- 0.038 -RRB- . A bias of lower value than the standard deviation should not be used as a measure of accuracy since it is not statistically valid or practically guaranteed . This large variation is due to the long range -LRB- 3.5 m -RRB- and the random variation in ambient lighting throughout the experiment . The standard deviations in all the tests -LRB- tables 1 A , B , and C -RRB- are as large as the bias , thus one may conclude that the calibration of the system has resulted Number of repeated measurement sets 0 0.01 0.02 0.03 0.04 0.05 0.06 1 2 3 4 5 6 7 8 9 10 Figure 8 : Effect of Repeated Measurements - VCM in removing all systematic errors larger than the noise . We will refer to this calibration as perfect calibration . In this case , we may use the repeatability , or the precision , to represent the accuracy . Another important conclusion is that for the measurement to be within the expected system precision , in addition to the perfect calibration , at least four sets of repeated measurements , each has at least five averaged images , must be used . Bias computation should be evaluated from time to time -LRB- to check the stability of the calibration parameters -RRB- and also after every new calibration -LRB- to make sure that the perfect calibration has been achieved -RRB- . 5.2 . Active range-based systems Figure 9 shows the expected precision of the ASL camera as a function of the range . The curve has been computed from the camera model using the propagation of spot measurement error . The spot size becomes too-large at close range and the quantization error increases significantly thus resulting in larger range error at ranges smaller than 600 mm . When the range increases to more than 600 mm , the standard deviation of the measurements increases non-linearly and deteriorates at a faster rate at ranges higher than 3 m . This rate can be approximated by a straight line in a logarithmic graph . This dependency on range is much more significant than for passive intensity-based systems . In the latter , the increase in the standard deviation will remain linear over tens of meters in range . 0.1 1 10 0.01 0.1 1 10 Z Precision -LRB- mm -RRB- vs Z -LRB- m -RRB- Z -LRB- m -RRB- Figure 9 : Expected precision -LRB- z -RRB- for the ASL camera 5.2.1 . Effect of repeated measurements Repeatability has been determined in two different manners : 1 - The object is scanned several times in one position . The measurements are averaged and the standard deviation is computed . This represents the stability of the sensor . 2 - The object is moved to random positions in the FOV and scanned several times in each position . The average of the measurements in each location is used to compute the final average of the measurement and their standard deviation . In the first case , the standard deviations of the measurements on edges and surfaces were 0.010 mm and 0.005 mm , respectively . These values are very small and indicate that the sensor itself is very stable -LRB- object position is not a factor -RRB- . It is also much better than an intensity based system -LRB- which was between 0.015 mm and 0.035 mm -RRB- which is mainly due to the effect of lighting variations on the latter . When the object was repositioned at various locations in the FOV -LRB- case 2 -RRB- the standard deviation was much larger -LRB- 0.060 mm on edges and 0.057 mm on surfaces -RRB- and also larger than the intensity based system . However , it is still within the expected precision for this FOV . These results confirm that the repeatability , to be realistic , must be evaluated when the object is placed at different locations rather than repeating the scans while the object is at the same location . 5.2.2 . Effect of pixel resolution The ASL camera has a programmable pixel resolution of 128x128 pixels and higher . The laser beam can also be used in either a focused mode or a collimated mode . Figure 10 shows the maximum possible resolution for various ranges as computed from the spot size on the detector for the given range -LRB- using the beam diffraction -RRB- . As shown in the figure , the camera can achieve 4096x4096 pixels resolution at 1 m range in the focused mode and at 10 m and larger ranges in the collimated mode . All the presented tests were performed at a resolution of 512x512-pixels . We are currently in the process of repeating all the tests at 256x256-pixels resolution , and preliminary results showed that the RMS values on the test targets were larger by an average factor of 1.55 in X and Y and 1.3 in Z . More tests are required to be performed at other resolutions in order to completely evaluate the effect of this parameter . 0.1 1 10 100 100 1000 10000 X-Y Resolution : Collimated beam X-Y Resolution : Focused beam Z -LRB- m -RRB- 4096 Figure 10 : Maximum pixel resolution for various ranges 5.2.3 . Accuracy of edge measurements The tests performed here are continuation of the tests presented in a previous paper .2 The main objective of the following tests is to identify which parameters are CISP and which are VISP . Once the former are identified , they can be calibrated at the most favorable conditions and then held fixed when calibrating the VISP under the actual conditions . The test procedure is as follows . At the optimum range , which is 600 mm for this camera -LRB- figure 9 -RRB- , all the internal parameters are calibrated as free parameters . At the other ranges , we have two types of calibration . In the first , all the internal parameters will be left free . In the second , we will fix some of the parameters -LRB- the distances and angles between the components of the cameras , such as mirrors and galvanometers -RRB- which are expected to remain unchanged , at the values obtained at 600 mm range . The parameters related to the moving parts -LRB- the scanning increments and distortion parameters -RRB- will always be left free . We then use the calibrated parameters from the two types of calibration to measure test objects . The test object will remain at the same position from the camera -LRB- about 650 mm -RRB- thus the only variables are the values of the camera internal parameters . If the measurement accuracy improves when some parameters are fixed , this will confirm that they belong to the CISP category and should be calibrated only at the optimum range . The results displayed in table 2 indicate that much better accuracy is achieved when the distances and angles between the internal components -LRB- figure 3 -RRB- are fixed as obtained from the 600 mm calibration . For example at calibration range of 1600 mm , when these parameters were fixed , the accuracy was 0.055 mm compared to 0.138 mm when all the parameters were left free in the calibration . Units : mm Free Parameters Fixed CISP circle # True Rad . Radius Error Radius Error R = 600 RMS : 0.023 0.023 1 7.486 7.443 - 0.043 7.443 - 0.043 2 10.000 10.003 +0.003 10.003 +0.003 3 12.482 12.446 - 0.036 12.446 - 0.036 4 14.990 14.995 +0.005 14.995 +0.005 5 19.998 19.987 - 0.011 19.987 - 0.011 6 24.987 24.993 +0.006 24.993 +0.006 R = 850 RMS : 0.083 0.035 1 7.486 7.383 - 0.103 7.430 - 0.056 2 10.000 9.993 - 0.007 10.024 +0.024 3 12.482 12.364 - 0.118 12.459 - 0.023 4 14.990 15.024 +0.034 15.021 +0.031 5 19.998 19.871 - 0.127 20.020 +0.022 6 24.987 24.987 0.000 25.026 +0.039 R = 1100 RMS : 0.108 0.043 1 7.486 7.362 - 0.124 7.432 - 0.054 2 10.000 10.007 +0.007 10.032 +0.032 3 12.482 12.345 - 0.137 12.465 - 0.017 4 14.990 15.028 +0.038 15.028 +0.038 5 19.998 19.845 - 0.153 20.035 +0.037 6 24.987 25.081 +0.106 25.050 +0.063 R = 1350 RMS : 0.084 0.058 1 7.486 7.403 - 0.083 7.437 - 0.049 2 10.000 10.055 +0.055 9.995 - 0.005 3 12.482 12.407 +0.075 12.472 - 0.010 4 14.990 15.039 +0.049 15.041 +0.051 5 19.998 19.926 - 0.072 19.978 - 0.020 6 24.987 25.124 +0.137 25.109 +0.122 R = 1600 RMS : 0.138 0.055 1 7.486 7.485 - 0.002 7.464 - 0.021 2 10.000 10.052 +0.052 9.997 - 0.003 3 12.482 12.518 +0.036 12.484 +0.002 4 14.990 15.045 +0.055 15.000 +0.010 5 19.998 20.113 +0.115 20.015 +0.017 6 24.987 25.294 +0.307 25.119 +0.132 Table 2 : Results of edge measurement -LRB- object in figure 7 -RRB- 5.2.4 Accuracy of surface measurements The above test procedure was applied to the object shown in figure 11 which consists of three near-perfect spheres -LRB- billiard balls -RRB- . The test results are displayed in table 3 . As in the case of edge measurements , the same behavior was observed . When using all camera parameters as obtained from calibrations at other than the optimum range the accuracy deteriorates significantly . On the other hand , when the CISP were held fixed , as obtained from calibration at the optimum range , the accuracy decreased at a much slower rate . The advantage of knowing the CISP and the VISP is now clear . We can achieve good accuracy at long ranges , which means larger FOV and thus larger objects can be accurately measured . Another factor is studied in the next test -LRB- table 4 . -RRB- The same object is placed at various ranges and the best calibration for the range is used -LRB- i.e. the CISP are obtained from the 600 mm calibration and the VISP are obtained at the actual object range . -RRB- In this case we notice that the accuracy deteriorates at a rather fast rate , mainly because the number of pixels on the object decreases also at a fast rate . We can then conclude that the FOV should be selected optimally according to the size of the object . Our extensive testing showed that the full object , or the FOM defined in 4.1 , should be covering 50-70 % of the FOV . If it covers less than 50 % it risks that some of the object features will not be covered by sufficient number of pixels for high accuracy measurement . If it covers more than 70 % of the FOV , it risks having some of the features close to the perimeter of the image where scan errors and large distortions may occur . Units : mm Free Parameters Fixed CISP sphere # True Rad . Radius Error Radius Error R = 600 RMS : 0.030 0.030 1 28.289 28.316 +0.027 28.316 +0.027 2 28.283 28.242 - 0.041 28.242 - 0.041 3 28.378 28.393 +0.015 28.393 +0.015 R = 850 RMS : 0.113 0.056 1 28.289 28.205 - 0.084 28.281 - 0.008 2 28.283 28.155 - 0.128 28.279 - 0.004 3 28.378 28.255 - 0.123 28.282 - 0.096 R = 1100 RMS : 0.109 0.059 1 28.289 28.376 +0.087 28.213 - 0.076 2 28.283 28.154 - 0.129 28.286 +0.003 3 28.378 28.271 - 0.107 28.310 - 0.068 R = 1350 RMS : 0.267 0.008 1 28.289 28.306 +0.017 28.299 +0.010 2 28.283 28.560 +0.277 28.292 +0.009 3 28.378 28.748 +0.370 28.376 - 0.002 R = 1600 RMS : 0.459 0.084 1 28.289 28.995 +0.706 28.188 - 0.101 2 28.283 28.647 +0.364 28.355 +0.072 3 28.378 28.382 +0.004 28.303 - 0.075 Table 3 : Surface measurement on spheres -LRB- figure 11 -RRB- using different calibration parameters - object at 650 mm 1 2 3 Figure 11 : Object containing spherical surfaces sphere # True Rad . Radius Error R = 650 , FOV : 520x520 RMS : 0.030 1 28.289 28.316 +0.027 2 28.283 28.242 - 0.041 3 28.378 28.393 +0.015 R = 1200 , FOV : 950x950 RMS : 0.243 1 28.289 28.576 +0.287 2 28.283 28.692 +0.309 3 28.378 28.464 +0.086 R = 1500 , FOV : 1200x1200 RMS : 0.328 1 28.289 28.310 +0.022 2 28.283 28.833 +0.550 3 28.378 28.518 +0.140 Table 4 : Surface measurement on spheres -LRB- figure 11 -RRB- - object is at the indicated R values . 6 . SENSORS CONFIGURATION DESIGN There are several factors affecting the design of a vision system for a given measurement-application . These are : 1 - Accuracy , given the application parameters : a - Type of feature b - Object size 2 - Density of data -LRB- completeness -RRB- 3 - Cost effectiveness 4 - Speed 5 - Ease of use From all the results presented in this paper and the two previous papers ,1,2 we will now discuss some possible sensor configurations to help system designers select the appropriate configuration of sensors , and sensor parameters , for their application . Table 5 summarizes the characteristics of three sensor configurations . Since our test laboratory -LRB- CEL -RRB- and testing equipment are limited in size , the applications benefiting from our analysis are those involving objects placed at ranges up to 3 meters and object sizes up to about 1 cubic meters . For larger objects and objects at longer ranges , the placement of targets on the object surfaces and using configuration A -LRB- photogrammetry -RRB- are recommended in order to achieve high accuracy . The shading in the table reflects our recommendation of a configuration for a particular type of feature . The lighter the shading the higher the recommendation , taking the first three of the above factors into consideration . If the requirement for the application is the measurement of only targets , or targets , edges , and vertices , then configuration A is the best choice . If the requirement is to measure only surfaces , or surfaces , edges , and vertices , then configuration C is the best choice . If all types of feature are required , such as in reverse engineering and many of the inspection applications , then configuration B is highly recommended . This configuration takes advantage of the abilities of each type of sensor and provide the most complete measurements at a high accuracy . CCD camera Light sourceRange sensor Targets Edges & Vertices SurfacesConfiguration A B C most least Highest accuracy -1:20,000 - 80,000 Cost effective Not cost effective Not cost effective Lower accuracy - 1:2,500 - 7,500 Highest accuracy - 1:20,000 - 36,000 Incomplete data High accuracy - 1 : 7,500 - 20,000 Complete data Accuracy same as B Less complete data Not possible without targets / features Accuracy same as C Less cost effective Complete data Accuracy to 1:15,000 Recommended : Table 5 : Recommended configurations for objects at ranges up to 3 m The above configurations are for each object side . The object may be placed on a precise rotating table to present each side to the sensors , or several groups of registered sensors are to be placed each facing one of the sides required to be measured . 7 . CONCLUDING REMARKS In three papers , we have presented algorithms for calibrating and integrating intensity and range data . Extensive testing , using specially designed laboratory and software , has been carried out to evaluate the accuracy of the different sensors when measuring surfaces , edges , and targets . The different types of parameters affecting the accuracy of these sensors were experimentally studied . Our studies should help system designer select the best sensor configuration for their application and set up sensor parameters so that the best accuracy can be achieved . In addition to all the presented tests and analysis presented throughout this paper , one last conclusion is made here . To achieve the accuracy values presented in our evaluation , or to be within the expected theoretical accuracy of any vision sensor , the following conditions must be observed : 1 - Perfect calibration is to be performed and maintained by the sensor . This is accomplished when the RMS of the residuals remaining after calibration is within the noise level -LRB- or resolution -RRB- of the sensor . In this case the accuracy -LRB- bias -RRB- and repeatability are similar in value . 2 - Independent measurements are to be repeated a sufficient number of times -LRB- at least four -RRB- with the object at various random positions . In each position several images or scans -LRB- at least five in case of CCD cameras and less in case of stable active range sensors -RRB- are to be acquired and averaged . 3 - The geometric configuration of the vision sensor -LRB- s -RRB- is as recommended for best accuracy . Failure to observe the above three conditions means that the achievable accuracy -LRB- bias -RRB- will be unpredictable . 8 . ACKNOWLEDGMENT We would like to extend our appreciation to Luc Cournoyer for his technical assistance provided throughout the experiments . REFERENCES 1 . S.F. El-Hakim and J.-A . Beraldin , `` On the integration of range and intensity data to improve vision-based threedimensional measurements '' , in Videometrics III , Proc . SPIE 2350 , 306-321 , 1994 . 2 . S.F. El-Hakim , J.-A . Beraldin and F. Blais , `` A comparative evaluation of the performance of passive and active 3-D vision systems '' , in Proc. : St. Petersburg Great Lakes Conference on Digital Photogrammetry and Remote Sensing , St. Petersburg , Russia -LRB- in press , SPIE -RRB- , June 25-30 , 1995 . 3 . M. Rioux , ? Laser range finder based on synchronized scanners ? , Applied Optics , 23 , 3837-3844 , 1984 . 4 . J.-A . Beraldin , S.F. El-Hakim and L. Cournoyer , ? Practical range camera calibration ? , in Proc.Videometrics II , SPIE 2067 , 21-31 , 1993 . 5 . American National Standard for Automated Vision Systems - Performance Test - Measurement of Relative Position of Target Features in Two-Dimensional Space . ANSI/AVA A15 .05 / 1 - 1989 . 6 . El-Hakim , S.F. and N.Pizzi , ? Multicamera vision-based approach to flexible feature measurement for inspection and reverse engineering , ? Optical Engineering , 32 -LRB- 9 -RRB- , 2201-2215 , 1993 . 7 . H.A. Beyer , T.Kersten , and A.Streilein , `` Metric accuracy performance of solid-state camera systems '' , in Proc . Videometrics , SPIE-1820 , 103-110 , 1992 . 8 . S.F. El-Hakim , `` Application and performance evaluation of a vision-based automated measurement system '' , in Proc . Videometrics , SPIE-1820 , 181-195 , 1992 . 9 . C.S. Fraser , and M.R.Shortis , `` Vision metrology in industrial inspection : a practical evaluation '' , International Archives of Photogrammetry and Remote Sensing , XXX -LRB- 5 -RRB- ; 87-91 , 1994 . 10 . A.W. Gruen , ? Digital photogrammetric processing systems : current status and prospects ? , Photogrammetric Eng . and Remote Sensing , 55 -LRB- 5 -RRB- , 581 ? 586 , 1989 . 11 . P.C. Gustafson , `` An accuracy / repeatability test for a video photogrammetric measurement '' , in Proc . Industrial Vision Metrology , SPIE-1526 , 36-41 , 1991 . 12 . H.-G . Maas and T.P. Kersten , `` Experiences with a high resolution still video camera in digital photogrammetric applications on a shipyard '' , International Archives of Photogrammetry and Remote Sensing , XXX -LRB- 5 -RRB- ; 250-255 , 1994 . 13 . J. Peipe , C.-T . Schneider and K. Sinnreich , `` Performance of a PC based digital photogrammetric station '' , International Archives of Photogrammetry and Remote Sensing , XXX -LRB- 5 -RRB- ; 304-309 , 1994 . 14 . D. Petkovic , B. Dom , J. Sanz , and J. Mandville , `` Verifying accuracy of machine vision algorithms and systems '' , in Proc . 22nd ASILOMAR Conf . Signals , Systems and Computers , MAPLE Press , 955-961 , 1988 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O B O O O O O O O B O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O

accuracy
calibration
performance evaluation
sensor fusion