Bad Board - dealing with production issues? - Page 1
Forum section: electronics / general

author: phil from seattle
author_rank: Super Contributor
timestamp: January 06, 2021, 04:47:49 pm
content: I have a product I have been selling - a PCB - for about 6 months now. Sold over 200 and never had a problem with board quality until now. I have been doing 100% visual inspection upon receiving and have yet to find a single tombstone or other error during my inspections.Yet yesterday I had a customer complaint. He sent me a picture of the board and I was shocked, not 1 but 3 errors on the board. 2 tombstones and a small IC on its back and not on it's footprint. See the attached photo. It isn't the best quality but the errors are not subtle at all. I've circled them in red to be sure.So, I must have missed this one in inspection. My process is to, upon receiving from the vendor, visually inspect the board and then wrap with protective bubble film so I can fulfill an order quickly. How likely is this is just the only one? I still have about half of the production run the board came from and re-inspected them all. No errors were found.This PCB is very obviously bad - how was it possible to miss it? Short of going to AOI, is there a better way than just looking at the board under a low power microscope (approx 3X)? I'm way too small to be thinking AOI for receiving? I am not going to complain to the vendor - I failed to catch this in receiving. But I am going to be a lot more vigilant for the next batch (being assembled as we speak). Am I being too reliant on the PCB assembler's quality?Also, how likely is it there is only one board this bad in a production run? I guess with 3 errors on one board there must be others. Hard to believe the production run would only have one bad board. Yet, I have had no other complaints (knock on wood...) and the remaining half of the batch is clean. The only board out of 200 that was bad and I missed it? Pretty staggering odds, I would think. Waiting for more shoes to drop.And by the way, I shipped the customer a replacement immediately via upgraded shipping. I even sent him a photo of the actual shipped board. Not the end of the world - hopefully a positive and immediate response goes into the lemonade from lemons category. The customer responded positively.

author: lutkeveld
author_rank: Regular Contributor
timestamp: January 06, 2021, 05:35:41 pm
content: That should definitely be caught by an AOI. Maybe your board house only does AOI on the first PCB? Or not at all?I would always advise to check the boards before sending them out.Manual inspection with a good microsope takes only 20 seconds per board and can save you a lot of trouble.You could even use something likehttps://www.agnospcb.com/to automate it a bit more.

author: wraper
author_rank: Unknown
timestamp: January 06, 2021, 05:37:51 pm
content: I would not sell untested boards. I would add test points where applicable and make a test fixture.

author: kylehunter
author_rank: Regular Contributor
timestamp: January 06, 2021, 06:34:08 pm
content: Yeah, the only actual solution here is to functionally test every board. Even with a solid AOI, the only way to fully ensure the board is working, is to test it. For an end product that goes directly to customers, that's required on our end.

author: pisoiu
author_rank: Regular Contributor
timestamp: January 06, 2021, 07:09:57 pm
content: Visual inspection without anything else is a receipt for trouble. It depends on how tired is the person doing it, if the coffee had the correct parameters in the morning, etc. You cannot keep consistency over time. For my boards I use it only because I make full functional testing after it. On numerous occasion functional tests failed, I found the problem on the board then I asked myself 'why I did not see that on visual inspection?'

author: SVFeingold
author_rank: Frequent Contributor
timestamp: January 06, 2021, 07:18:47 pm
content: These should be caught at the vendors with AOI, and at either the vendor or your lab in a test fixture. Pretty much how all the companies do it. 100% testing. It's worth saying though that for tombstoned caps...it might still pass all functional tests unless every one of your caps is critical. Back to AOI.Giving it a once-over can do in a pinch but sooner or later you'll miss something, as you're experiencing now.QuoteAbsolutely you should. If they're running AOI this is a pretty big miss. If they aren't doing AOI, they should still know about it so that they can improve their process and reduce the likelihood of it happening again.QuoteFor 200 boards, yeah. Not worth it. But keep an eye on industrial auctions if you plan to scale up. Desktop AOI machines go for relatively cheap all the time, as in $2-4k for Mirtec AOIs.QuoteIt's a small run, with a 0.5% defect rate so far. That's already pretty bad even with just the one board, but definitely tolerable for small runs. Entirely possible they're all fine except this one. Maybe it got bumped...maybe the temperature profile wasn't quite right for that board. Who knows. Those things are for your PCB assembler to figure out. That expertise is what you pay them for after all.One method you might try is to set up a rig to capture clean photos of the boards. Get a picture of a clean, known good board. Your "golden" board. Every other board that comes in, take a picture of it then load it into Photoshop or some other program and do a "subtract" on the images to highlight differences. Camera/board should be in the same place relative to one another to make it easy. If it's worth setting something like this up it will help you catch such defects.

author: 48X24X48X
author_rank: Frequent Contributor
timestamp: January 07, 2021, 07:05:05 am
content: So, you don't test the board at all? Even the simplest breakout board should be tested. I test every single aspect of my products even the simplest thing like a push button. Make a pogo pin tester board/jig to access all those through hole connections. And it looks like a Teensy board sitting there so your tester should have one of that as well. One of the reason PCBA companies prefer to have to testing also included is to verify everything is perfect (so there's no finger pointing later). So, if they are not doing it, you should be doing it. First time hearing product shipped without testing.

author: 48X24X48X
author_rank: Frequent Contributor
timestamp: January 07, 2021, 07:08:51 am
content: Quote from: pisoiu on January 06, 2021, 07:09:57 pmThis happens to me from time to time. For sure our eyes will miss some of it and only during testing you will catch it. Most common are slight tombstone resistors (probably due to uneven reflow).

author: Mnyut
author_rank: Newbie
timestamp: January 07, 2021, 09:34:55 am
content: Quote from: phil from seattle on January 06, 2021, 04:47:49 pmThis can happen when you are distracted by something.... a phone call, someone talking to you...Quote from: phil from seattle on January 06, 2021, 04:47:49 pmI always say that I want to get feedback when a mistake slips through.Quote from: phil from seattle on January 06, 2021, 04:47:49 pmSometimes that happens, slipped by the vendor, slipped by you. Reports to him.Quote from: 48X24X48X on January 07, 2021, 07:08:51 amOr the mass of the pads is too differently or placed with too large an offset,...

author: asmi
author_rank: Super Contributor
timestamp: January 07, 2021, 03:52:14 pm
content: Yea, man, you've got to design a test jig and test each and every board, and do it quickly too - before you get a board with dud of a part and ship it to a client. Even if rarely now, this still does happen every once in a while.Also, it's much easier to miss a tombstone than you think. Just yesterday I've found a tombstone which I couldn't find even using multimeter, as probe pressed on a part enough to close the circuit during continuity testing. The only way I was able to find defect is because I do test each and every board before shipping to a customer.

author: olkipukki
author_rank: Frequent Contributor
timestamp: January 07, 2021, 04:44:58 pm
content: Quote from: 48X24X48X on January 07, 2021, 07:05:05 amNo, you "should not", perhaps "may be" - at least according some people I have spoken in the past.If we keep a moral etc. on a side at the moment and focus on OP's "dealing with production issues" - they can pass these to customers that actually doing right now.The pure logic is: they know a defective rate % (after all their efforts been in a place to maximize error-free output) and if dealing takes much less time/$ than do it properly initially, why bother?!The customer has complained - just send the replacement, that's all.

author: wraper
author_rank: Unknown
timestamp: January 07, 2021, 04:56:02 pm
content: Quote from: olkipukki on January 07, 2021, 04:44:58 pmUntil you ship 100% defective batch to hundreds of customers. If you do this crap, at least do sample testing.

author: SVFeingold
author_rank: Frequent Contributor
timestamp: January 07, 2021, 05:24:29 pm
content: Quote from: olkipukki on January 07, 2021, 04:44:58 pmBut they don't know the defect rate. Knowing the defect rate accurately enough to make these kind of judgments requires strict process control, large quantities, and extensive testing. Knowing you happened to get one bad board out of 200 doesn't mean you know the defect rate. The next 1,000 might be perfect. Or maybe 5% will be bad. There's not enough data to conclude either way.Replacing failed units that are in customers hands is fine...but it damages your reputation and irritates customers. It's not easy to put a dollar value on that if you are planning to scale up.

author: Matt-Brown
author_rank: Regular Contributor
timestamp: January 07, 2021, 05:27:08 pm
content: My company has many customers that do not want us to test the assemblies we build. Everyone assumes that their product will be 100% defect free all the time. This is not the case. You have to keep adding in manual and automated inspections to meet quality yields (based on solder joints).At my company we usually only do one AOI step and that is prior to reflow (why fix something after it is bad is the logic). This company may be doing the same, or they may not have an AOI at all and are only doing manual inspection ( 80% that a defect will be caught).Most of the time assemblers are given minimal data, no test information, no cleaning requirements etc, and yet everyone expects a miracle and 100% defect free. It doesnt happen!Give your assembler the feedback and ask questions. If they are worth their salt then they will improve their process. Give them as much infomation as you can and let them know what you want. They will ask questions and work with you.Good assembly houses want you to succeed. It means they also succceed in more business.

author: asmi
author_rank: Super Contributor
timestamp: January 07, 2021, 05:36:23 pm
content: Quote from: olkipukki on January 07, 2021, 04:44:58 pmIt depends on the kind of customers you have. If you ship 100's of boards to a single customer, than, yea, they probably expect that every once in a while they'll receive a dud, or at least they would be willing to work with you on get it RMA'ed once that happens. If, on the other hand, your average customer buys just a single board - then they likely expect it to work on arrival, especially if it's expensive board, and they will be much less willing to tolerate failures and deal with replacement, and will likely just request a refund and will go online and dump a lot of foul-smelling matter on you. And I don't think that you'd want THAT kind of publicity.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 07, 2021, 08:01:03 pm
content: Quote from: olkipukki on January 07, 2021, 04:44:58 pmBefore you start sending untested boards to the customers, look at it this way - when an assembly house sends you defective boards, that's exactly what they do. Worse yet, if they test it and find it defective, they will have to throw it away. If they ship it to you, chances are you will never request a replacement. Using your logic, by sending you defective boards they actually make more money. There's a little flaw though - your next order will go elsewhere.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: January 07, 2021, 08:07:18 pm
content: They say visual inspection is around 85% effective.Three passes -- by independent inspectors -- is therefore 99.7% effective. Maybe. Assuming they don't have overlapping gaps in observation (systematic error). If you can afford to hire two more people to do inspections, that would be an option.Or if you can afford the time to space out your inspections, so that you come at each board with fresh eyes, multiple times -- maybe that would work as well.Tim

author: Ice-Tea
author_rank: Super Contributor
timestamp: January 07, 2021, 08:12:47 pm
content: Apart from what and how you test I would still contact your board house and try to understand where it went wrong on their end. That should never have left their facility.