Inter Lab Comparison. How to plan one? What to do? Help understanding. - Page 1
Forum section: electronics / general

author: ttttrigg3r
author_rank: Contributor
timestamp: January 04, 2022, 11:24:22 pm
content: Good new year to all. I hope everyone is in good health for 2022.Calibration labs usually use a vendor for proficiency testing. You order a kit to your lab, take the measurement and uncertainty, then the vendor tells you the normalized error as compared to their reference measurement.What if you wanted to host an ILC with labs around the area? All the resources I've found so far tell you how to find normalized error, En, and accreditation body requirements such as how many PT and how often. I could not find a good source on how to plan an ILC, what artifact to use, etc. Perhaps someone can answer these questions on how to plan an ILC.1) What guidelines do you use to pick the artifact for a PT or ILC? What qualifying factors are required?I know that good stability is important, but how do you know when something is good enough to be used as the artifact? I see that the Fluke 732A is frequently used for PT. It's a 10V DC Voltage standard which generates 10V. If I had a random Agilent 10V power supply in my lab, could I use that as the artifact? I know it would not be as good(stable) as the 732A, but how do I quantify what is good enough?2) What is the minimum ratio between the reference uncertainty and the participating lab? I imagine that the reference uncertainty should be significantly lower than the participating lab uncertainty, but by what factor? If the uncertainty between ref and lab are similar, would that prevent a good ILC? Is there a ratio requirement between the stability or accuracy of the artifact to that of the reference uncertainty? I know to look for >4:1 as a rule of thumb when performing performance test; would that somehow apply here?Would it be crazy to pick up an Agilent 34401A and send it around to other labs and try to compare data of the measurements?Your expert help would certainly be appreciated. If you know any articles or references on this topic, that would be awesome.

author: jpanhalt
author_rank: Super Contributor
timestamp: January 05, 2022, 12:15:50 am
content: Quote from: ttttrigg3r on January 04, 2022, 11:24:22 pmGood startQuoteWhat regulation, if any, are you trying to follow? Is the regulation by a governmental or professional entity?In my profession (clinical laboratory testing), there were no universally accepted standards, as there are for voltage and so forth. We also had to deal with a matrix (e.g., serum, blood culture, throat swabs,...) of variable and unstable composition. We used "reference laboratories" to establish a first level of acceptability. Then we used the performance of the whole to determine whether the results from the reference laboratories were on base. We had a committee made of several representatives, some from the reference laboratories and some from the whole group of labs tested. There was thorough and open discussion before any grades were accessed. Later, the US Federal government entered the scene, and established some criteria for what could and could not be graded. The Clinical Laboratory Improvement Amendments, 1988 ("CLIA" and revisions) added criteria based on group performance. In one instance I can remember, the CLIA criteria would have led to the wrong result being considered correct, and the correct result being penalized. That is where free and open discussion by those interested in improvement, rather than grading, were essential. The challenge was not graded. It is not an inexpensive endeavor even though all of us served as volunteers. There were staff, transport, challenge preparation, and data processing to consider.QuoteI am not sure what you mean by "artifact." Please elaborate.QuoteSo far as I know, there is no single set standard for closeness to the mean/correct result in clinical laboratory work. In your area, there may be. I don't know.QuoteCrazy? I don't know, but how do you account for instability of that standard during shipping? What if two established vendors disagree? Hence, we had input from industry, private labs, and government in all of our discussions. You might want to look at methods used by the National Committee for Clinical Laboratory Standards (NCCLS) which has been renamed to the Clinical Laboratory Standards Institute (CLSI) (https://en.wikipedia.org/wiki/Clinical_and_Laboratory_Standards_Institute).Regards, John

author: ttttrigg3r
author_rank: Contributor
timestamp: January 05, 2022, 04:47:52 pm
content: I work in a calibration lab and the standard we have to adhere to is ISO 17025. We pay an accrediting body to audit us and they pretty much told us what we need to test for and at what interval. This keeps us in compliance with 17025.My question is more for calibration and equipment, and it looks like this is different from clinical lab testing. The concept is similar, but the thing being tested are different?For example, as a calibration lab, we claim we can measure 10V with uncertainty of 1ppm. The proficiency testing would test our performance every two years as compared to someone else. In my example above, the 732A is the artifact which contains the nominal 10V for us to measure. The artifact has some value undisclosed to the participants of the ILC, and the purpose is to measure this artifact and have the reference body compare values and uncertainty.What I want to do ultimately is to conduct this sort of comparison with labs, sort of unofficially. My company has a number of sister labs nearby that can participate, and I see an opportunity to compare performance.

author: Martin Miranda
author_rank: Regular Contributor
timestamp: January 05, 2022, 05:06:20 pm
content: subscribing. thanks!

author: jpanhalt
author_rank: Super Contributor
timestamp: January 05, 2022, 05:25:25 pm
content: Quote from: ttttrigg3r on January 05, 2022, 04:47:52 pmThe group I was associated with was the first in the US (late 1940's) to look at clinical labs not from the standpoint of calibration and whether standards were used, but in terms of real life results. Every lab does calibration, but how do you assess accuracy with random submissions? There's a recent thread on this blog about some suspicious "calibrations" done at big name labs.What those early pioneers did was to submit "blind" specimens to cooperating laboratories. That is, specimens with fake patient names simulated to be like real specimens. The real results were very disappointing. That grew into an world recognized effort that I was associated with from 1982 to the late 1990's. We helped write CLIA 1988.I can tell you from experience that many big name clinical labs have walls covered with calibrations charts and daily QC results, but when challenged with a blind unknown, they can fail. In other words, there is a difference between what a lab is capable of doing and what it does routinely.I misunderstood what you wanted to do. I thought you wanted to develop blind proficiency testing to determine how well calibration laboratories actually do versus what they are theoretically capable of doing. In my opinion, that needs to be done.EDIT: Here's one relevant thread:https://www.eevblog.com/forum/metrology/how-reliable-are-accredited-calibration-laboratories/25/Perhaps there are others.

author: alm
author_rank: Super Contributor
timestamp: January 05, 2022, 06:54:50 pm
content: Quote from: ttttrigg3r on January 04, 2022, 11:24:22 pmI would stay stability and ruggedness for shipping. It should be able to survive whatever your shipping method is, with possible shocks/vibration andfluctuation in environmental conditions, with a minimum change in value. So a power supply with an analog pot for value is not ideal, for example. You might characterize the drift of your artifact before to estimate what you expect in terms of drift you expect over the period of the comparison. You can validate the stability of the artifact you're shipping afterwards by measuring it at the same lab before and after.Herethey ship a Fluke 732A to another lab and use the before and after measurements to measure the drift during the comparison.Quote from: ttttrigg3r on January 04, 2022, 11:24:22 pmI would consider ILC a confidence check. It is not going to give you the same confidence as sending them to a higher tier lab, but it can verify that your procedures and standards are sane. You can calculate the uncertainty of the transfer afterwards. It will be a combination of the transfer accuracy from and to the artifact at both labs (impacted by the lab itself and by noise of the artifact), and the stability of the artifact.Quote from: ttttrigg3r on January 04, 2022, 11:24:22 pmI'd say if your goals are modest, that could work. The BIPM has done something similar with what's basically a very special DMM:https://www.bipm.org/documents/20126/45401061/APMP.EM-S12.pdf/df0647b7-d7da-c84f-5747-3116e52bd117You can find more similar comparisons of standards-lab level comparison on the BIPM website, like this one involving a transportable Josephson Junction Arrays:https://www.bipm.org/documents/20126/45254320/EUROMET.EM.BIPM-K10.a.pdf/2995e6d6-cfa4-acca-af18-15d513544ad7. They shipped the standard back to the initial lab every time to monitor drift.

author: mendip_discovery
author_rank: Super Contributor
timestamp: January 05, 2022, 07:00:10 pm
content: [fx=cracks_knuckles]I have done several interlabs. Mostly using a training contractor who also happens to be ex Marconi, and UKAS (UK 17025) and now is a contractor for them. He has so far brought a Resistance standard and a 6.5 digit Keysight meter. That was satisfactory for my audits. I got one of our frequency counters sent out for calibration and on its return I just tested a few ranges with my frequency generator and compared then entered the details (I made a point of not looking at the cert until I had done my own tests). I sent a copy of my report in PDF to the lab I got it calibrated with. I have since agreed to do a regular interlab with them but I would like to use something I know is rather stable and tests different elements, I have been thinking of a ADR 10 V reference and a VHP101 Resistor. For mechanical I have just an example plug gauge, thread gauge etc.Formula I have for calculating the ratio is =ABS((D17-B17)/(SQRT((E17*E17)+(C17*C17))))where;B17 = Lab A - MeasuredD17 = Lab B - MeasuredE17 = Lab B - UncertaintyC17 = Lab A - UncertaintyAnything less than 1 is what you are after. If testing engineers against each other in a blind test then the same calculator can be used but you are looking for 0.5 or less.This year before sending my Multimeter off for calibration I will calibrate it myself using my multifunction calibrator. Then compare the calibration certificates. Do the same the other way around (use DMM to calibrate MFC) and you are covering both sides of your accreditation.You could just send them a house hold DMM as its only to test that your uncertainties are valid.

author: mzzj
author_rank: Super Contributor
timestamp: January 05, 2022, 07:57:01 pm
content: Quote from: mendip_discovery on January 05, 2022, 07:00:10 pmI was about to mention this exact same thing.I see too often reported uncertainty that is either way optimistic or so pessimistic that there is a question if customer got what they needed."correct" uncertainty is bit of subjective matter but at least you can see if there are oddball labs that state totally different uncertainty than others.For example: DMM with 0.1v resolution and 0.02v calibration uncertainty smells fishydecent DMM with 0.001v resolution and 0.1v calibration uncertainty also smells fishyK-type thermocouple calibration with 0.03 cel calibration uncertainty smells fishy (even if cal lab would be able to measure it with 0.005cel uncertainty.)

author: mendip_discovery
author_rank: Super Contributor
timestamp: January 05, 2022, 08:19:24 pm
content: Quote from: mzzj on January 05, 2022, 07:57:01 pmI have always taken it that with the En Ratio, <1 Good, <0.5 Brilliant, <0.1 You need to check your budget as you may be over-inflating it, its safer but it also means the customer is getting barn door uncertainties.The Uncertainty doesn't need to take into account the resolution of the UUT, they are often just stating what they are accredited for on their schedule. Our certs have a statement that says please also add 1 LSD or 1/2 div to the stated uncertainties. I am going to query this with my auditor as I wasn't sure if it was put in by the previous HoL due to the auditor having an issue with something. Not seen it on certificates.Decision Rules are not popular, once you could go on a 4:1 or 5:1 ratio but now they like you to actually calculate it and tell the customer if it is Undetermined. It's a way of gently prodding labs to improve the kit and lower their uncertainties. Seehttps://www.isobudgets.com/statements-of-conformity-and-decision-rules/One of the problems I have so far got into with calibrating thermocouple displays is the CJ temp sensor has a direct correlation to the simulated temp, this meant I had to import a ±2°C into the budget. So far that is the one bit of kit that has caused no end of issues for me. Transmille fit a temp sensor into the connector, problem is it isn't very reliable and a few times has drifted off.

author: binary01
author_rank: Contributor
timestamp: January 06, 2022, 03:16:31 am
content: Our lab runs many measurement audits, round-robin PTs and intercomparisons commercially for accredited labs in the field of pressure calibration (we also conduct our own in electrical audits to support our own electrical accreditation).When approached to conduct a measurement audit, we consult with the lab about what area of measurement they would like to demonstrate/audit, and what uncertainty they are aiming to prove. For a very basic lab performing industrial testing, any reliable and suitably ranged pressure indicator would generally support the requested uncertainty over a short term (~1 month), but for high accuracy / low uncertainty labs we would carefully select a high resolution device with known stability to ensure it will be useful for the participant. We keep a range of artefacts just for this purpose.Perhaps it could be a little easier in electrical for most labs (?) where a good DMM could be a suitable artefact to cover many functions and ranges with a single device, but for lowest uncertainty in some higher level labs fixed artefacts like Fluke 732x might be required. Our accreditation body typically wants to see proficiency testing over the full range of our accreditation, so using fixed artefacts would be difficult.I wouldn't consider anything like a "TUR" during PT, but the basic rule on uncertainty that I consider to be very important is that the uncertainty on the performance of the artefact (i.e. our mesaurement uncertainty during testing of the artefact + its drift over the duration of the audit) is equal or better than the uncertainty that the paricipant is aiming to demonstrate. This is a requirement from our 17025 accreditation body (NATA), where the proficiency test can only be used to support an accreditation when the reference uncertainty is less than or equal to participant uncertainty. This makes good sense, of course, as there is no point having a very good lab trying to demonstrate proficiency by comparing with a lesser lab or instrument.Although technically En as high as 1 might still demonstrate some "agreement" / overlap between labs, the basic recommendation from our accreditation body is that an En > 0.7 should trigger an investigation. Of course, En closer to 0 is always desireable.ISO 17043 is the formal standard that covers proficiency testing, if you haven't read it already.I hope this helps.<Edit> With regards to frequency of audits, our accreditation body (NATA) states:Facilities are encouraged to participate in as broad a range of PT activities as practicable, but at least once every two years (different frequencies may be stated in the various NAC packages) for each major area of test, measurement or related activity covered by the scope of accreditation, where such programs are available.We are also required to do PT to support any requested change/addition to our accreditation.

author: mzzj
author_rank: Super Contributor
timestamp: January 06, 2022, 04:45:56 am
content: Quote from: mendip_discovery on January 05, 2022, 08:19:24 pmUKAS is ok with that? Hard to believe.I have never seen cal cert from accredited lab that would exclude the UUT resolution or stating that you need to add -+1 digit.I think it is pretty much against the idea of EA 4/02 and 17025

author: binary01
author_rank: Contributor
timestamp: January 06, 2022, 05:22:51 am
content: Quote from: mzzj on January 06, 2022, 04:45:56 amLikewise, it is not an approach that is used here. For any instance where you are taking a reading (i.e. from a DMM, DVM, pressure gauge), surely the uncertainty must always include how well you can read the instrument (its resolution/readability)?

author: mendip_discovery
author_rank: Super Contributor
timestamp: January 06, 2022, 08:48:54 am
content: Quote from: mzzj on January 06, 2022, 04:45:56 amOn our certs we mention 1lsd or 1/2 division. But majority of the certs I have seen don't say this but they just quote thier uncertainties as stated on the schedule. It may well be in the small print wording hence my mind set I don't need to account for UUT in my uncertainty calculations. It is somthing I am adding to my list of things to look into as I am sure I am covered but I like to be able to explain how I have it covered.

author: guenthert
author_rank: Frequent Contributor
timestamp: January 09, 2022, 06:18:17 pm
content: Quote from: mzzj on January 05, 2022, 07:57:01 pmWhy? Could you elaborate? I'd think it'll be permissible to repeat the measurement n times and thereby reduce uncertainty by n1/2, won't it?

author: mzzj
author_rank: Super Contributor
timestamp: January 09, 2022, 06:35:54 pm
content: Quote from: guenthert on January 09, 2022, 06:18:17 pmAveraging won't help if the readings are stable.resolution limits the uncertainty, in theory the result could be anywhere from -0.049 V to +0.049 V and the meter would still round it up to 0.0 V.You could test the switch-over points to determine more accurate value where the meter switches from digit to next but this is rarely done as it is lot more time consuming.(Some manufacturers also incorporate some amount of hysteresis to the readout so that the numbers wouldn't jump one way or another all the time. This makes it even slower to determine actual switch-over points)

author: alm
author_rank: Super Contributor
timestamp: January 10, 2022, 10:20:19 am
content: Or you could superimpose noise on the signal before averaging, which Datron did to implement a high resolution mode in the 1081, from 6.5 digits to 7.5 digits. But that all sounds way out of scope for a calibration unless specified that way by the manufacturer.

author: Conrad Hoffman
author_rank: Super Contributor
timestamp: January 10, 2022, 09:40:17 pm
content: I claim zero knowledge of how comparisons are done, but to have any certainty about something like voltage, you need three references. No fewer. You do a comparison between the three, then send one out. When it comes back, you do the comparison again. That gives you some confidence that nothing changed in shipment (or abuse) and nothing has changed at home. Or at least the odds are pretty low. A single 732C is great but what you really need is the set of three (or the 4 they sell as the 734C). No doubt the price is quite high- being a hobbyist I have the old 731 units.

author: mendip_discovery
author_rank: Super Contributor
timestamp: January 11, 2022, 07:24:32 pm
content: Quote from: mzzj on January 05, 2022, 07:57:01 pmQuote from: mzzj on January 05, 2022, 07:57:01 pmCan I add a meter in that could cause some fun? Digital Thermometer, Resolution 1°C, Specification for -50 to 300°C is 1°C. You might have a 0.05°C Uncertainty but due to taking the resolution into account, it becomes very hard to state a pass or fail with a decision rule. What would you do?Quote from: Conrad Hoffman on January 10, 2022, 09:40:17 pmWhat used to happen was you would pay the accrediting body to calibrate one of their meters, and it wasn't cheap. They would then compare your calibration results with several labs and then tell you if you needed to revisit your budget or not. These meters would not always be correct, some would have been tinkered with to have readings that were out of spec or just downright strange. It is about verifying your uncertainty budget is fit for purpose within the k=2 95% probability. Think of it like this, if you sent a resistor to several local "Join the Resistance" groups you would expect each of you to get roughly the same measurement within the specifications of your meters, if not then one of you has an issue and needs to look for the fault.

author: ttttrigg3r
author_rank: Contributor
timestamp: March 11, 2022, 05:37:15 pm
content: After doing a good amount of reading and exploring of the Inter Lab Comparison, I learned a great deal of it and can share my knowledge here in hopes of helping others.Here are some of my learning points.1. From the start, understanding what ILC and Proficiency Testing are. How are they different. Basically, ILC is comparing measurement results from two or more labs. Proficiency Testing is when you use/pay a third party provider and uses a reference laboratory to conduct the ILC, usually for accreditation purposes.Source:https://www.isobudgets.com/proficiency-testing-and-interlaboratory-comparisons/For what I was doing, it can be considered as hosting the ILC involving my lab and several other peer labs.2. When hosting the ILC, it's important to understand the end results and what it really means. One super common method of analysis is finding the Normalized Error (En). The formula can be found in the source above; I won't post it here.2a. Normalized Error (En). Compares one laboratory's measurement and uncertainty with the reference lab's meas and unc. In my case, my lab was hosting this project, so we designated it to be the "reference".A passing En result is: |En|<1. What this is basically saying is 'Does at least one of the measurement fall within the uncertainty of the other.Case 1: If the two lab's uncertainty are very similar AND the artifact is very stable, then the result is trustworthy (for lack of better word). This means when your En is failing, (En>1), one of the lab's processed is being influenced by systematic or random error. This is important to understand, because if your artifact isunstableand both lab's measurement uncertainty are very tight, then a failing En is almost surely.Case 2: One lab's uncertainty is tight and the other's uncertainty is huge. A dramatic example is if one lab is using an expensive Agilent 3458A multimeter and the other uses a Harbor Freight multimeter. A passing En is almost surely. In this case, you have to look at Pn which compares the uncertainty with the tolerance of the artifact.The important take-away for me is the stability of the artifact. It’s very important to know and test your artifact for stability if you want good results. The process of how to do that I still have to learn.There’s an article : A DISCUSSION ON ISSUES OF STABILITY AND HOMOGENEITY INPROFICIENCY TESTING FOR CALIBRATION LABORATORIES Jeff Gust, 2006.He talks about how to compensate for stability and drift. If you have a opening and closing measurement from your pivot lab, you can treat that as a rectangular distribution. “The estimate of uncertainty due to short term stability is usually determined by considering the measured deviation from pivot measurements completed before and after a participant. The most conservative analysis of this information would be to consider the opening and closing pivot measurements to be opposite ends of a rectangular distribution, so the standarduncertainty due to stability associated with the artifact is the difference between the opening and closing measurement divided by the square root of three.”For this ILC, I found that the Youden Analysis is a lot better visually and provides more information. To get a Youden Analysis, you have to double up on the artifact. I highly recommend the sources below for Youden analysis.https://www.intechopen.com/chapters/53144Very good source. Has formulas. Good explanationhttps://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fwww.nist.gov%2Fsystem%2Ffiles%2Fdocuments%2Fpml%2Fwmd%2Flabmetrology%2Fyouden-analysis-2.ppt&wdOrigin=BROWSELINKVery good ppt by Nist. Breakdown of graph with example.I don’t want to write too in depth; the sources above helped me greatly.I would also recommend Matlab instead of excel if you wish to graph the Youden. It worked out a lot better for me.

author: FransW
author_rank: Frequent Contributor
timestamp: March 18, 2022, 10:49:29 am
content: Round-Robin testing:https://en.wikipedia.org/wiki/Round_robin_test#:~:text=In%20experimental%20methodology%2C%20a%20round,variety%20of%20methods%20and%20equipment.Quote:A round robin test or interlaboratory comparison (ILC) is a type of performance test for accredited calibration laboratories. It is a method of external quality assurance for laboratories and their measurement procedures.UnquoteRegards, Frans