SAMD21: Unexpected port I/O failure mode [IDENTIFIED: ESD] - Page 1
Forum section: electronics / general

author: jrowberg
author_rank: Contributor
timestamp: March 03, 2021, 06:11:43 pm
content: Hi all,I have a problem that has only showed up after building 300 or so of a new custom flight yoke device, and so far I haven't been able to determine either the cause or a fix. TL;DR summary is that the MCU still runs and talks over USB, but GPIOs that should be in input + pulled high mode are seemingly driven to odd voltages like 0.2V, 1.4V, or 2.0V. Input registers can still detect the logic level when strongly driven externally, but something is internally wrong.Here's a short list of salient points about the current board design:ATSAMD21G18A microcontroller design w/external 32768Hz crystalSPX3819 3.3v regulator for main power rail, same as on many Adafruit boardsUSB type B connector for data + powerDownstream USB hub (USB2514) for low-power related peripherals(almost never used)External 5V supply connection in case downstream peripherals are more power-hungry(no customers have used this to date)MBR120 diodes to protect either USB or 5V in from feeding into the other oneMCU configured as a downstream device of the USB2514, but a physical switch allows rerouting MCU USB data lines directly to the USB B connector for dev/troubleshootingTwo ALS31300 hall effect sensors connected via I2C6x inputs comprising 2 momentary pushbuttons and a 4-way hat switch, all active LOW connecting GND to the pin through a 220-ohm safety resistor*All 6 button GPIOs are set to input + pulled high, all other GPIOs are left in their POR state and electrically unconnected*1 pin has no 220-ohm safety resistor, to allow it to be used as an optional VDD source for an LED in a different build variant, but this is only a standard button in the failing unitsThe circuit is not very complicated. I built about 30 of these boards myself (home stencil+paste, PnP, reflow) and did dozens of hours of firmware development on a couple of different workstations without any issues. Now, some units that have made it into the field (nearing 10% of those shipped) have failed, usually after many hours of perfect functionality. Buttons no longer respond correctly and sometimes remain in a "pressed" state all the time, rendering the yoke useless to the customer.I'm not the most knowledgeable electrical engineer in the world, but I have donemanydesigns around the SAMD21 and I've never seen this failure mode. The USB hub is the only departure from my typical designs, but I can't figure out why or how that could cause this kind of problem, especially since none of the failing units (to my knowledge) have involved the use of any downstream peripherals. I have not been able to reproduce the problem myself so far.As for troubleshooting, this is what I've found:No detectable physical issues such as solder bridges, flux residue, bad part placement, broken components, etc.Different sets of GPIOs seem to be affected, not always the same onesGPIOs on multiple ports (PORTA + PORTB) are affectedErase+reflash of firmware has no impactDisconnecting hall effect sensors (via ribbon cable) has no impactDisconnecting buttons/hat switch (via ribbon cable) has no impactRemoving 220 ohm safety resistors has no impactSwapping in a new MCU IC with hot air and flashing new firmware fixes everything immediatelyMonitoring various I/O pins, USB data lines, power nets, etc. with a DSO reveals nothing unexpected (e.g. spikes or dips), even through many repeated USB cable removal/insertion cyclesOne customer had two different units fail (now on his 3rd, still working)--suggests a possible customer-side problem given the odds?Searching online has not revealed a definitive answer to this. Not being able to reproduce the problem locally is limiting; "here's a custom hardware-tweaked replacement, see if this works for a few weeks" makes for an incredibly inefficient troubleshooting process. The majority of yokes are working fine, but every new device shipped is gambling on customer happiness.I'm asking here in case anyone has see this before, or knows better than I do what questions to ask--or who to pay for consulting, if necessary. About the only thing I've found so far is that there's no TVS protection on the USB input. But even in the case of USB subsystem damage, I wouldn't expect these symptoms.Any brave/knowledgeable EEVblog forum souls care to weigh in? I can share additional details if anyone needs to know something I haven't shown.

author: ataradov
author_rank: Super Contributor
timestamp: March 03, 2021, 06:36:33 pm
content: Do the affected pins work as outputs and can dive the set levels?Can you provide the schematic around the MCU? Just the power supply stuff would be sufficient.

author: JOEBOBSICLE
author_rank: Regular Contributor
timestamp: March 03, 2021, 06:46:35 pm
content: You're using internal pull up plus external pull down resistors?You might be sinking too much current via the internal pull up if all buttons are pressed.I'd press all buttons and look at the power supply current.

author: thm_w
author_rank: Super Contributor
timestamp: March 03, 2021, 10:15:53 pm
content: Quote from: JOEBOBSICLE on March 03, 2021, 06:46:35 pmThere is no external pull down, there is an external 220R to gnd when the button is pressed. But yeah schematic of the buttons would help... it also appears there is a capacitor on the button as well."Input registers can still detect the logic level when strongly driven externally, but something is internally wrong."This and swapping the chip fixes the random issue, makes me think ESD.Any idea what sort of strength is required to drive externally high, current wise?

author: jrowberg
author_rank: Contributor
timestamp: March 03, 2021, 10:44:03 pm
content: Thanks for your replies so far. I have attached a PDF for detail as well as a PNG of each page (two total). This is not the entire schematic, but it does have the bits that you have asked for.QuoteGood question, and something I hadn't tested until a moment ago. Output functionality is also broken, where the HIGH output state is the weird voltage instead of the expected 3.3v that the VDD rail has. Most strange is that one of the failing pins on the board I have on my desk now actually has LOW at 1.8v and HIGH at 2.1v. So the GND reference is borked as well...but just on that pin.QuoteThat's correct. The 220Rs are just to avoid a direct short between GND and VDD, but that's logically impossible with the way the firmware is written since the pins are only in input mode (ignoring my test above). The buttons do have 0.1uF caps for simple debouncing as well, though I don't think this has any impact on the failure in question.QuoteThe supply current on a working board hovers at about 70mA. Pressing all the buttons barely registers. The hat switch can only be in one direction at a time, meaning the max current is from three buttons held simultaneously. It's 2mA or less, empirically.QuoteFirst question: what's the easiest way for me to test this without spending a zillion dollars, if possible? +4kV neon sign power supply from Amazon strategically touched to the board?Second question: is it likely that I can fix this with add-on components or ground straps or something, even if I have to greenwire it? There's considerable $ sunk into the initial production batch, and it's going to be extremely painful if the entire batch is irredeemably unreliable.

author: ataradov
author_rank: Super Contributor
timestamp: March 03, 2021, 10:52:46 pm
content: So do they still work as outputs? Are output levels as expected?This is a really strange issue. I have never seen ESD do this.

author: thm_w
author_rank: Super Contributor
timestamp: March 03, 2021, 11:15:14 pm
content: The RC arrangement is a bit unusual, normally you have R first then the C on the GPIO side. With it reversed that means 0.1uF gets shorted through the switch when its closed (1uF is shown on the schematic). Which is not ideal, could possibly cause switch wear over time? Although I haven't seen that myself.But that shouldn't cause any pullup issues.Worst case if it is ESD you can bodge in some TVS in or put them on a daughter board. But I would confirm the IC damage a bit more first. Measure leakages from/into the pin, compare to other gpio?

author: jrowberg
author_rank: Contributor
timestamp: March 04, 2021, 12:12:55 am
content: Quote from: ataradov on March 03, 2021, 10:52:46 pmSorry, I wasn't really clear. I can put the pins into output mode and set them to HIGH or LOW states, but the actual voltages on the pads are wrong. Most lows are 0v, but that one pin I mentioned has a low voltage at 1.8v. The highs are all over the place--0.2v, 1.2v, 2.1v.Quote from: thm_w on March 03, 2021, 11:15:14 pmMy mistake, in too much of a hurry--yes, they're 1uF. The addition of the series resistors was a late design iteration and not added specifically for debouncing; rather for a combination of short protection and LED current-limiting in alternative build variants. I agree that it's not a standard RC debounce arrangement. Shorting through the switch is something I hadn't considered though. It should be low current, correct? They're simple momentary SPST, physically. I can't imagine that would cause any electromechanical stress.Quote from: thm_w on March 03, 2021, 11:15:14 pmApologies for asking, but is there a recommended way to do this while avoiding outside interference? Should the pins be tri-stated in firmware, or driven/pulled either way? Should I remove external components connected to each pin in question? Taking out the 220Rs is easy and would leave them electrically unconnected outside of the IC.One possibly relevant thing I noticed is that when the pins are in input mode and pulled high, if I externally short them to the opposite rail while in this state, current consumption increases about 5-10mA depending on the pin. As noted in an earlier reply, the expected current impact is extremely low, since the internal pull-up is only like 40k or something. So it seems like the internal GPIO peripherals are somehow damaged yet still semi-functional.Quote from: ataradov on March 03, 2021, 10:52:46 pmAgreed, but I'm willing to try anything to figure this out. Without going the ElectroBOOM zap gun route, is there a cheap DIY way to intentionally shock the board for test purposes?

author: ataradov
author_rank: Super Contributor
timestamp: March 04, 2021, 12:38:23 am
content: All this looks like output drivers are damaged. This may be some sort of ESD damage. One more thing to verify - there is a drive strength setting in the PORT. Check if behaviour is the same with both values.But also, in this configuration capacitors don't do anything and the pull-up resistors inside the MCU charge them when the button is released. I don't really know how that can be a problem, but who knows. I would just remove the capacitors.I don't think any of the homemade solutions would produce controllable result. You can zap the board, but chances of reproducing the same issue are low.

author: floobydust
author_rank: Super Contributor
timestamp: March 04, 2021, 12:56:53 am
content: To me this seems like ESD damaged I/O.When you sit down (generate several kV body charge) and touch the yoke, it may be discharging to switches which travel to the MCU port.You must have ESD protection for any MCU I/O that goes to field wiring and connectors people plug in. The PCB traces to the 1uF caps are miles away from the connectors so the trace inductance prevents them (caps) from really absorbing the hit, a fast rising impulse I think would kick up the port pin voltage. The 220R resistors may simply arc across.Get a BBQ lighter and hit some port pins, but know polarity needs to go both ways +ve, -ve during testing. You could make an in-line ESD clamp board just to prove the theory as well.The one site that is failing, get them to spray fabric softener Downy on the chair and floor to cut down on ESD and see what happens. They might have low humidity or a chair that makes a wallop.

author: wraper
author_rank: Unknown
timestamp: March 04, 2021, 01:19:43 am
content: You have MLCC directly on U2 output. SPX3819 datasheet only mentions Alu electrolytic or tantalum. That's a strong indication that it may have issues with low ESR on its output as many other LDO. Unless datasheet has ESR graph for capacitor where low ESR is fine or mentions that it's MLCC stable, you should be wary using them as it may cause LDO instability.

author: PCB.Wiz
author_rank: Super Contributor
timestamp: March 05, 2021, 12:28:18 am
content: Quote from: jrowberg on March 03, 2021, 06:11:43 pmThat does look like ESD or similar damage. You could check the impedance of those odd voltages, by adding parallel lowish value resistors Up then down, and see how much they deviate.That will indicate what the equivalent failure path resistance is inside the device.Strangest ESD? damage I ever saw on a MCU was one single pin somehow flipped logic, so a Hi drove low and vice versa.

author: thm_w
author_rank: Super Contributor
timestamp: March 05, 2021, 09:48:13 pm
content: Quote from: jrowberg on March 04, 2021, 12:12:55 amIf you have reasonable cable length going to the switch its probably OK.But essentially you are shorting 1uF across the switch contact so it could mean a current spike and reduced life down the line. 1uF is not much though, hard to say.QuoteWith your circuit there is no need to disconnect anything.But, if you are saying its on input mode and shorting causes 5-10mA draw that is huge, and seems to be damage (as others noted as well).

author: wraper
author_rank: Unknown
timestamp: March 05, 2021, 11:02:59 pm
content: Also 120uF C17 makes USB out of spec. Sink device must no have more than 10uF on VBUS according to USB specification.

author: nigelwright7557
author_rank: Frequent Contributor
timestamp: March 06, 2021, 01:42:01 am
content: My colleague used to design electronics for washing machines and they all had extra I think 100k or 10k resistor in series from input to reduce ESD.I can only wear cotton so dont have a problem with ESD but some man made clothes can be highly static generating.

author: jrowberg
author_rank: Contributor
timestamp: March 06, 2021, 03:15:13 am
content: Okay, 99.9% sure the problem is ESD due to static build-up and customer contact at this point. A customer email came in this afternoon specifically complaining about static electricity issues (the first time anyone has said so). I am putting together a static generator using a negative ion generator transformer a la NightHawkInLight's YouTube video:...so that I can force the issue and fry some of MCUs on purpose, then fix the design and not fry anymore MCUs.The main yoke chassis is made of steel, and the yoke handle is plastic attached to an aluminum square rod that fits into a linear rail system for movement. Although there are many exposed ground points on the PCB, lo and beholdnoneof them are actually strapped/screwed to the chassis. (Shame on me for not identifying this ahead of time, but at least I have the lame excuse that the chassis stuff was totally outside of my wheelhouse on this project. Live and learn.) So the electronics are just...floating, and any build-up coming from the outside of the device literallyhasto jump through the PCB somewhere.Quote from: wraper on March 04, 2021, 01:19:43 amThanks for this heads-up--I'll check into this as well.Quote from: wraper on March 05, 2021, 11:02:59 pmCan you point me to the relevant section? Not disagreeing, but genuinely interested. The big cap is for downstream devices that can be plugged into the USB-A connectors, since there is a hub in the design allowing for up to two external devices connected. My understanding is that a large cap is recommended to avoid downstream device insertion pulling down the voltage too much for other devices. The hub circuit is implemented as hub-powered, rather than self-powered, so VBUS is common to upstream and downstream.

author: SilverSolder
author_rank: Super Contributor
timestamp: March 06, 2021, 05:04:17 pm
content: Quote from: jrowberg on March 06, 2021, 03:15:13 amLOL that video looked a bit scary - not for testing pacemakers, presumably?

author: wraper
author_rank: Unknown
timestamp: March 06, 2021, 05:10:43 pm
content: Quote from: jrowberg on March 06, 2021, 03:15:13 amhttp://sdphca.ucsd.edu/lab_equip_manuals/usb_20.pdfpage 179QuotePage 177:Quote

author: jrowberg
author_rank: Contributor
timestamp: March 18, 2021, 02:13:45 am
content: So I've been consuming information on ESD prevention and troubleshooting for a few days now after this discussion, and I want to run a solution by you all to make sure it's at least reasonable and not entirely wrong-headed. My main goal is to avoid having to scrap 200+ assembled PCBs entirely, so bear in mind this will not be an ideal solution; however, through a fortuitous design choice early on (0.1" header holes for debug flexibility), I think it should be remarkably good given what we have to work with.To recap:We have ESD coming from users when the grab the flight yoke handle or throttle handle, and/or touch the metal chassis of the whole assembly. This ESD is frying the MCU in random ways.We suspect the ESD is coming principally through the button/hat switch input GPIO lines, since those are the functions that specifically stop working in virtually every case (not always the same ones, but the same general failure mode).The metal chassis itself is currentlynotstrapped to the PCB ground plane/pins/screw holes.The USB-B connector shield is connected directly to the PCB ground plane, and is the only electrical connection point back to PC/house/earth ground (i.e. there are no other power or other connections for ESD to follow).The design currently hasnoTVS components anywhere in the system. D'oh. Never again.The buttons and hat switch are simple SPST circuits that connect various MCU I/Os to GND, in most cases through a 220-ohm resistor to limit current. The buttons/switches are physically located inside the yoke handle, connected to the main PCB through ~10-inch multi-conductor cables into 1mm-pitch JST headers (visible in the photos earlier in this thread). Since I read that TVS protection in this situation should ideally be placed on the main PCB as close as possible to the connector, I have designed a very simple add-on PCB that incorporates 5x TITPD4E1U06(4-channel general purpose TVS diode array):The board uses the same 1mm JST headers to allow the original cables to plug in without modification/replacement. These pins are routed directly to the protection pins on the TVS ICs, using very short wide traces all on the same layer. The signals are then routed with narrower traces (naturally higher downstream impedance) through vias to the bottom side of the board and directly to the larger 0.1" headers, which will plug into the previously unused holes on the base PCB. There TVS GND pin goes to a dedicated earth ground trace and two large copper-exposed screw holes, which will be strapped to the metal chassis with a wire + ring terminals. The metal chassis will further be specifically tied to GND right next to the USB shield. For testing, I didnotconnect the earth ground side to the PCB ground side, but left header holes so I can jumper them if needed. I read that you don't necessarily want to shunt TVS discharge right into PCB ground; I'm not sure about the nuances of this warning, so I left either approach open. Obviously, the PCB ground eventually connects to chassis/earth ground anyway, but to manage the discharge path I figured keeping them separate right at/near the TVS ICs might be good.See the attached photo for board design detail.In short, it's a dedicated off-board ESD protection circuit sandwiched between the physical switches and the main PCB. In retrospect, although this approach costs more, this arrangementmighteven offer better protection than all the ESD circuitry were on the main board. But that's not an immediate option anyway, so I digress.My questions for interested and knowledgeable readers are these:Did I do anything horribly wrong so far?Are there additional things I might add to this add-on board to help further?Should I definitely either connect or separate PCB ground from TVS discharge/earth ground on the ESD protection board?

author: ataradov
author_rank: Super Contributor
timestamp: March 18, 2021, 04:37:48 am
content: It would be better to see the schematic rather than the layout.The layout looks ok, but most of the traces on the blue layer on the bottom can be routed in the red layer with no vias.

author: Monkeh
author_rank: Super Contributor
timestamp: March 18, 2021, 05:11:23 am
content: With your large 1uF caps on your switches, have you taken a scope and examined what the switch transition looks like? I've seen pins driven several volts below ground with such abuse before.

author: jrowberg
author_rank: Contributor
timestamp: March 18, 2021, 01:15:00 pm
content: Quote from: Monkeh on March 18, 2021, 05:11:23 amI just tested a few dozen presses on the generic 100MHz DSO I have, and I didn't see any significant dips below ground. In a few instances, maybe as much as -0.1v, but nothing worse and usually not that bad. Either way, taking those capacitors out entirely is not out of the question, at least for this batch of boards. New batches will have smaller values and a refactored RC debounce circuit anyway, per comments earlier in this thread.Quote from: ataradov on March 18, 2021, 04:37:48 amSee new attached image. PDF available as well, but it's simple enough that I imagine the image has enough detail.Quote from: ataradov on March 18, 2021, 04:37:48 amIn the interest of increasing impedance on the downstream side of the TVS diode (towards the protected IC), isn't having vias here acceptable, if not preferable?

author: ajb
author_rank: Super Contributor
timestamp: March 18, 2021, 05:34:29 pm
content: Having vias certainly doesn't hurt, but shouldn't be particularly valued, let alone relied on, for providing impedance here. You should probably throw some series resistors between the external connections and the TVS diode networks, which will reduce the stress on the diodes (which they're designed for, but doesn't hurt to give them a little extra cushion) and take a bit of the edge off of the incoming transient. As a bonus, series resistors will also limit the current that may flow through the diodes if an external voltage beyond the supply rails is applied to the input.It's also worth noting that you should retain the series resistors between the protection diodes and the MCU inputs when you respin the board. When the protection diodes begin conducting they will exhibit a voltage drop dependent on the current, so the clamped voltage can still wind up outside of your supply rails. This is particularly true of TVS diodes where there is a bit of a 'soft' region from the voltage where they begin to conduct until the current rises to the point where they hit their harder clamping voltage. You have to look at the TVS diode curves to make sure this is accounted for based on the fault conditions you're designing for. So the voltage at the MCU pin can reach a level where the internal protection diodes begin to conduct, and the series resistance between the external protection diodes and the IO pin will limit the current the MCU's protection diodes need to handle in that case. This is especially important with the TVS diode networks you've selected here where the upper diodes are only clamped to the built-in TVS diode, not to the supply rail, so the relationship between the clamped transient voltage and the supply voltage (which is what determines the effect on the MCU's internal diodes) is not as well controlled.

author: ataradov
author_rank: Super Contributor
timestamp: March 18, 2021, 05:39:59 pm
content: I agree. If you think you need extra inductance, add the components. I personally would value uninterrupted ground plane more than inductance from the vias. And yes, add the ground plane to your PCB.

author: jrowberg
author_rank: Contributor
timestamp: March 19, 2021, 02:45:45 am
content: Thanks for your continued input. This is all immediately helpful, not to mention instructive for future projects.Quote from: ajb on March 18, 2021, 05:34:29 pmHow critical would you say this is? The existing series resistance was chosen in part to provide proper brightness for 3 LEDs that are present in one of the variants that will need to use this ESD protection board. Is the extra protection for the TVS component just another safety net?Quote from: ajb on March 18, 2021, 05:34:29 pmWill do, in part for the reason noted above.Quote from: ajb on March 18, 2021, 05:34:29 pmThis is a component parameter I wasn't aware of. Should I be looking for a TVS array that includes supply-level clamping as well? Are there any sample P/Ns I could use as a jumping-off point for a parametric search?Quote from: ataradov on March 18, 2021, 05:39:59 pmSince this is the first attempt at a protection circuit, I'm not sure if we need extra inductance (or impedance). I'm hoping not.When you noted that most traces can be routed in the red layer with no vias, does what you're envisioning still maintain the trace path across the TVS pins before proceeding on to the downstream 0.1" headers? My imagination may be limited but I don't see how it's possible to avoid vias unless the traces proceed directly from the input headers to the output headers, without the TVS being physically between them. Maybe this doesn't matter since we're dealing with speed-of-light electron motion across a few centimeters. I'm happy to revise if there's no point in doing what I've done here.I've attached an image showing a small revision, with solid GND (not earth) polygons on both layers with plenty of edge stitching, plus I swapped out the PCB/earth ground jumpers for 1206 landing pads which will, for now, be populated with 0R jumpers.

author: ataradov
author_rank: Super Contributor
timestamp: March 19, 2021, 02:54:49 am
content: Path does not matter, the signal normally does not go "through" TVS. TVS is just sitting on the line and clamps it to a set voltage in case of an over-voltage. It does so by passing the current through itself.Clamping to both supply rails is better, but it is hard to tell if it is necessary here.

author: nigelwright7557
author_rank: Frequent Contributor
timestamp: May 22, 2021, 03:04:13 am
content: Usual problem with micros when ports dont work is port is set to analog.I know on PIC's you have to set ANSEL register accordingly.