Weird moody USB device stops talking, enumerates but denies BULK IN requests... - Page 1
Forum section: electronics / projects

author: berke
author_rank: Frequent Contributor
timestamp: February 02, 2024, 06:53:14 pm
content: A client has a PoS PoS (point of sale) equipment which is connected to an embedded PC running some custom Linux through USB. The equipment randomly decides that it doesn't want to talk anymore. It's USB 2.0 FS connection but it's some kind of bootleg USB implementation with a fantasy PID:VID, it's not compliant w.r.t. current consumption (starts consuming >100 mA before config, claims to be battery powered, etc.Here's what happens.It enumerates and then accepts its normal configuration, then you can talk to it with bulk transfers and a terrible protocol.Under some circumstances (not understood) it decides it won't talk anymore.When that happens, it will still enumerate and accept the first control mesage, but won't respond to bulk messages.Sniffing the physical bus I see that it's constantly responding NAK to BULK IN requests (at the usbmon level you don't see any activity, just a timeout).The ONLY way to get the device back is to power cycle it for example by physically unplugging and replugging it.- Rebooting the PC doesn't work.- Resetting the hub/USB/controller doesn't work.- The embedded PC's USB hub unfortunately is not switchable.Sometimes, the problem occurs if you reboot the PC, but not always.It happens on some machines. Some days. On some USB ports, not all of them. It seems to depend on what you had for breakfast.Actually, the easiest way to put the device in the stuck state is to deconfigure and reconfigure the USB hub it's connected to by writing 0/1 to bConfigurationValue. This causes the hub to reset, and then to re-enumerate the devices.AFAIK this shouldn't cause a problem and it doesn't cause a problem with the few other devices I've tested (keyboard, SATA adapters etc.).I looked at everything, from sniffing using usbmon to sniffing the physical lines with my scope and decoding them to checking the power consumption.The device has an NFC reader and I previously was able to crash the device using pulsed 13.56 MHz fields, this issue went away after a firmware upgrade- 80% crappy firmware, buggy USB stack- 10% internal power supply decoupling issue, causing glitches that cause faults under certain circumstances- 5% some kind of susceptibility to radiated EMI- 5% otherAny ideas welcome. What could be causing this? How do diagnose it? Did anyone experience something similar?In an ideal world, the manufacturer of the equipment would provide the full schematics and source code to the firmware, well actually they would send a couple engineers to fix the problem. Turns out the world is not ideal!(PS. Sorry I can't give full details about the brand name etc.)

author: selcuk
author_rank: Frequent Contributor
timestamp: February 02, 2024, 08:54:01 pm
content: It may be a firmware bug and you may not find it with hardware tests.Nevertheless, embedded devices usually don't have focus on supplying reliable power through USB. They are not like notebooks to which people connect their mobile phones to charge. So designers don't always bother. Can you try with an AC wall adapter powered USB hub between embedded linux and EUT?

author: berke
author_rank: Frequent Contributor
timestamp: February 02, 2024, 09:13:47 pm
content: Quote from: selcuk on February 02, 2024, 08:54:01 pmIt's quite possible, but the behaviour is so inconsistent across identical devices that I'm wondering if there is more than software at play.QuoteI did because I am suspecting a power issue. I used a powered TP-Link USB3 hub, then an unpowered hub; no change. I even inserted a small board to break out the USB lines, then I disconnected the EUT's VBUS from the PC and powered it from a bench supply instead, there was no change to device operation.I don't have permission yet to open the customer's device, but if I get it I could bypass the long custom USB cable and see if that helps.

author: magic
author_rank: Super Contributor
timestamp: February 02, 2024, 11:10:23 pm
content: Looks like a firmware bug, unfortunately.Tried other voodoo stuff like using different PCs or USB controllers?Quote from: berke on February 02, 2024, 06:53:14 pmMaybe it's the only way to trigger the bug and the devices are being reset randomly for some reason?What operating system, do you have any logs that could show if it's happening or not?

author: moffy
author_rank: Super Contributor
timestamp: February 03, 2024, 04:10:07 am
content: It sounds so like a memory leak/corruption of the embedded controller, notoriously hard to fault find and often very unpredictable.

author: berke
author_rank: Frequent Contributor
timestamp: February 03, 2024, 08:55:14 am
content: Quote from: magic on February 02, 2024, 11:10:23 pmYes it seems to be happier on newer hardware and on some ports. The EUT has USB2 and USB3 ports with different controllers (EHCI/XHCI), I thought at some point that the issue was only occurring on the USB2 ports, but like everything else further testing debunked that theory.So the main problem is the reproductibility. I fix the driver, run tests, it's OK. They test it, it's OK. They test it again, doesn't work.QuoteI do not see any bus reset or anomalous events during normal operation and those show up in the kernel logs.The BIOS enumerates the USB devices at boot, so I was thinking maybe that's how the bug occurs, but the BIOS does this everytime so how come the device works the first time?QuoteI wish I had logs from the device itself! The PC uses nothing special, it's a Linux generated using Yocto, with a 5.14 kernel. Actually I was sick of it and replaced it with a regular Debian to have normal tools, with no change in problem behaviour. The replacement driver I wrote is a userspace driver using usbdev. The kernel logs do not show the issue because the device still enumerates, it just doesn't respond to BULK IN requests afterwards, which devices are allowed to do. The manufacturer's original driver fares no better and will produce a kernel message complaining about the device not responding.I could post scope traces of D+/D- showing the USB transaction, but basically the host says BULK IN got any data? DUT sez "NAK".Quote from: moffy on February 03, 2024, 04:10:07 amYes, that's at the top of the list and would be totally unsurprising.

author: magic
author_rank: Super Contributor
timestamp: February 03, 2024, 09:40:57 am
content: Quote from: berke on February 03, 2024, 08:55:14 amI wonder if it really prefers some ports over others or if that too is a matter of insufficient statistics collected yet...I don't know, you could check USB supply voltage coming out of those ports but it's really desperate. The device probably regulates it to 3.3V internally anyway...Quote from: berke on February 03, 2024, 08:55:14 amNevermind, then. Linux doesn't log every BULK IN NAK of course, but it would log random resets or anything unusual.Quote from: berke on February 03, 2024, 08:55:14 amThe BIOS probably just asks for a few descriptors and decides that it doesn't care about this device.But your first post suggests that the device simply stops working mid-operation, right? It's very unclear what sort of bug in a USB device could have such effect later after being probed by the BIOS...Quote from: berke on February 03, 2024, 08:55:14 amThat's basically no data queued for the USB peripheral of the MCU. The task which generates the data got deadlocked or crashed, or is waiting for something that never happens (and USB reset is not it), or entered a completely invalid state somehow...Is this device known to work better under any particular other hardware/software environment, particularly one recommended or supplied by its manufacturer?Is the device meant to work for prolonged time, or are you trying to use 24/7 something that was envisioned to only be powered on for an hour at a time?

author: berke
author_rank: Frequent Contributor
timestamp: February 03, 2024, 10:11:39 am
content: Quote from: magic on February 03, 2024, 09:40:57 amVoltages are fine, there is some IR drop but it's well above 4.5V and like you said they're gonna be regulating it to 3.3V or lower anyway. However there are large and fast current peaks when doing some operations, which is why I'm wondering if it could have insufficient internal decoupling causing glitches and random crashes depending on the supply impedance.QuoteThe CPU doesn't even see those NAK's, the EHCI controller deals with them. You can see the NAKs by probing the wires.QuoteYes, but it certainly turns the hub controller on and maybe off.QuoteSometimes mid-operation, and sometimes across reboot (which keeps the device powered.) Turning the PC off (which is scheduled daily) doesn't unpower the USB ports unfortunately.QuoteYes that makes sense. But to make matters worse the manufacturer has its own firmware on top of which another firmware made by another company is running. It's an EFT (credit card) device.There could be multiple MCUs in the thing, one handling USB, another one handling the display, who knows...QuoteAllegedly the thing works fine under Windows 10. I'm three steps removed from the manufacturer...QuoteYes it's a payment terminal so it's meant to be on 24/7.

author: magic
author_rank: Super Contributor
timestamp: February 03, 2024, 10:51:26 am
content: If brownouts are suspected you could test with supply voltage raised to the maximum allowed. A USB device should work with up to 5.25V and recently this has been increased to 5.5V, though there is no guarantee that everybody got the memo.Quote from: berke on February 03, 2024, 10:11:39 amI would play with BIOS options, disable anything related to wakeups and USB support (except turning off the chipset controllers).Since PCIe only supplies 3.3V and 12V, some cheap PCIe USB cards take power from Molex/SATA connectors (no on-board buck). These obviously power cycle host-powered devices when the machine is power cycled. But some of them are rubbish, with no per-port overcurrent protection either...Quote from: berke on February 03, 2024, 10:11:39 amThis could be down to the driver being careful to avoid doing some things that upset the firmware, or maybe knowing some undocumented way to reset it.

author: berke
author_rank: Frequent Contributor
timestamp: February 03, 2024, 11:00:20 am
content: Quote from: magic on February 03, 2024, 10:51:26 amBoth good ideas. I did try the BIOS CSM settings. Problem one is that I can't get the thing to crash reliably except by reconfiguring the hub. Client has one device that shows the problem reliably, but client is in another part of the country. Problem two is that there already are many thousands of those things in the field and client is understandably cagey about attempting unattended remote BIOS upgrades on them, even if they could convince the PC supplier to convince the motherboard manufacturer to convince the BIOS supplier to generate a new BIOS with the right settings for them.I'm thinking of making an automated test rig by adding a switch to the USB port and writing some scripts to continuously test and power cycle.QuoteEDIT: And yes good call again, actually I did discover an unused "RESET" command in their Linux driver's header and it seemed to work a bit when I used it in my userspace driver, but like all other things, with more testing it turns out to not be as reliable. As a result my userspace driver has options for reset methods, and someone has to run tests on the n^p combinations of machines, ports, firmware versions and driver options.

author: magic
author_rank: Super Contributor
timestamp: February 03, 2024, 11:23:04 am
content: Do you mean that reconfiguring the hub is a reliable trigger?In such case it's not the same issue as on rebooting if rebooting doesn't crash the device every time.I wonder if it might be some race condition? Does it still crash if there is a delay between stopping and restarting the hub?

author: berke
author_rank: Frequent Contributor
timestamp: February 03, 2024, 11:45:19 am
content: Quote from: magic on February 03, 2024, 11:23:04 amYes, not a 100% but close.QuoteMaybe there is more than one bug, but the symptom is the same.QuoteThanks that's actually a good idea, I could try varying timings.The answer has to be yes as I was doing it initially by hand so there were a couple seconds of delay. The hub takes a second or so to restart anyway. Every time you restart the hub, it re-enumerates and Linux assigns increasing bus IDs to the device. I wondered if the device only works with one ID, or maybe with the first ID it got, so I wrote a script that reconfigures the bus to get an increasing sequence of numbers. The device didn't come back even after the numbers wrapped around.If it's a race condition it could be something related to the timing of the bus reset/enumerate sequence. In that case Windows could have different timings that don't trigger the bug.The general timings would be affected by what other devices are connected, causing all kinds of randomness. The BIOS messes with the flash NVRAM storage which has garbage collection cycles, so once in a while it could take a bit longer.

author: berke
author_rank: Frequent Contributor
timestamp: February 06, 2024, 06:15:15 pm
content: Thanks for all your suggestions.I made some progress. I was able to get the device to go mute by rebooting while sending some random bytes.It's almost as if some higher level protocol goes out of sync, but you'd expect that to reset after a bus enumeration. Maybe I can resync it.Figuring out what's going on across a reboot is a bit tricky. Even with segmented acquisition my scope's memory isn't deep enough to capture the whole boot process, but thankfully the device is only full speed so I will be able to sniff the bus with an FPGA dev board. Not sure if it's worth it though.I also found a Windows XP driver for this PoS. I'm not interested in messing around with doing an XP install (it's long and painful and I don't know my way around Windows), but if I get desperate I might try to disassemble it to get some clues, it's fairly small.Although it would be fun if it was some kind of EMI issue it's probably boring crappy FW issue.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 07, 2024, 03:42:58 am
content: When the first 3G/4G USB modems came out, many of them worked fine on desktops and servers, but had odd issues on embedded devices like Linux SBCs and routers. Turns out their current draw was very spiky, and while desktops and servers had either stiff 5V rails or lots of capacitance on the USB 5V output, embedded hosts did not. It is why Olimex came up withthis: 2mF of low-ESR bulk capacitance,completelyagainst the USB standards, but useful on certain embedded hosts to counter their weak and wobbly 5V USB rail.Similarly, on most desktop and server hosts the VBUS is almost always between 5.00V and 5.25V. If it is closer to 4.5V on the device end, and spiky, it is possible both the USB device and perhaps even the host USB chip (if powered from that particular rail) are experiencing very short brownouts, leading to the USB device locking up in confusion. Heck, even the cable or USB connectors might be oxidized, leading to unexpected resistance on the 5V rail, causing higher than expected voltage sag during those current spikes.If you can re-test with a power supply to power the USB device, consider adding bulk capacitance near the USB device end, with the power supply set at say 5.10V with a current limit at 2A (for the initial inrush, to charge the capacitors), to handle those current spikes locally without the 5V line spiking down. Maybe 1µF ceramic and a 470uF electrolytic, perhaps? The way I'd do this is to sacrifice a cable, removing the outer cover near the device end, cutting the 5V and GND lines, and splicing in the (isolated or floating!) supply GND to both ends of the GND line, and the positive supply to the device end of the 5V line, leaving the host end of the 5V line unconnected to anything.

author: magic
author_rank: Super Contributor
timestamp: February 07, 2024, 07:29:59 am
content: Device cable will likely nullify any ESR/ESL advantage that a ceramic cap may have over electrolytic. (This reminds me about one audiophool mod I have seen, where the guy installed an array of capacitors in decades from 1mF down to 1pF on adownstreamUSB port intended for audio DAC - I regret not saving those pics when they were still up there).I would go all out and use 5.2V, or whatever is the maximum specified by device vendor - if there is any spec at all

author: berke
author_rank: Frequent Contributor
timestamp: February 07, 2024, 11:13:29 am
content: Quote from: Nominal Animal on February 07, 2024, 03:42:58 amThat's very interesting thanks. The device has a custom .05" connector, I don't want to slice their custom cable but I could get a matching connector and put extra caps there.Note that the device doesn't even properly declare its current consumption in its USB decriptors. I've read the big thread about USB host current limits, but here the host doesn't do anything below 500 mA even though it "legally" could.Here is a current trace I had captured a while ago at the far end of the USB cable. Idle consumption is about 80 mA but when you do some things, such as if you insert a credit card you get a current spike. I eyeball about +250 mA in 100 µs which is 25µC which would be 5 µF.I opened the device.- The only large cap I see is probably a 1812 (~4.6 mm × 3 mm) MLCC, but there is a dozen smaller (~0805) MLCCs sprinkled around. No electrolytics or tantalums.- Locating the supply traces and probing the voltage during operation directly on the PCB should tell me if there's a brownout.- There is only one big IC, a Broadcom secure processor which ofcourse this being Broadcom doesn't have any kind of publicly available datasheet. No separate USB chip.There is a single chip and the device remains responsive to enumeration requests and control packets, I think that doesn't bode well for the brownout/EMI theory.

author: magic
author_rank: Super Contributor
timestamp: February 07, 2024, 11:27:17 am
content: It could be power supply related if there are multiple CPU cores on the chip and one responsible for "business logic" crashes while the other doing USB comms does not (maybe a separate supply rail, or just luck).But a firmware bug seems more likely. Software sucks, nothing new.Did you try strictly rate limiting commands sent to the device? It might be a matter of some undocumented timing requirements which happen to be met by vendor's Windows driver (by design or by accident), but not by your code. Or maybe vendor's driver doesn't work either and they just didn't try it for long enough...

author: berke
author_rank: Frequent Contributor
timestamp: February 07, 2024, 11:49:16 am
content: Quote from: magic on February 07, 2024, 11:27:17 amHmm... If what you say is true, maybe the USB portion crashes and reboots, so that it still responds to enumeration requests, but the comms with the rest of the chip is all messed up so you can't really talk to the application.QuoteYep looks like it.QuoteAnother good idea! Thanks. I will insert a configurable explicit delay around packets.There was a configurable maximum packet size that I had been playing with but it wasn't doing much.

author: berke
author_rank: Frequent Contributor
timestamp: February 21, 2024, 04:23:39 pm
content: After too many (billable) hours of investigation, turns out the device gets stuck if it receives a USB bus reconfiguration request while there is a pending command. It has a command-response protocol made using BULK OUT and BULK IN requests. In other words the problem occurs if you send a BULK OUT, it goes through and then the bus gets reconfigured before you query the response using a BULK IN.On EHCI ports the kernel deals with some low-level bookkeeping that is done by hardware on XHCI ports (such as address assignment). This can explain why it works better on certain ports or on certain machines and not so well on others, as that would affect timings. Timings are of course also affected by system load, and whatever other things are connected to the USB bus, hence the apparent unpredictability.It's most likely a firwmare bug, but maybe it's an unfixable silicon issue in the the device's controller. All I can do is avoid quitting the driver right after a BULK OUT, and sending the BULK IN ASAP, and maybe playing with process and RT priorities.Thanks for your suggestions!PS. I still think the electronics are crappy, even though the rails I've probed seemed stable I've seen a couple of bus errors (with the kernel complaining of a bad cable etc.) I believe checking those things wasn't completely uncalled for, since the device is obviously non-compliant electrically.

author: magic
author_rank: Super Contributor
timestamp: February 21, 2024, 05:39:20 pm
content: LOLThis may explain the reboot problems, but is it normal that the device sometimes receives configuration change requests during regular operation, or are those other crashes caused by something else you are yet to discover?Quote from: berke on February 21, 2024, 04:23:39 pmSilicon doesn't know or care about request-response protocols running over separate endpoints. Surely looks like a FW bug, unfixable simply because the vendor doesn't care.Quote from: berke on February 21, 2024, 04:23:39 pmIsn't it possible to queue the IN asynchronously before sending the OUT? AFAIK the host controller will be retrying the IN request until the device ACKs.

author: berke
author_rank: Frequent Contributor
timestamp: February 21, 2024, 10:46:12 pm
content: Quote from: magic on February 21, 2024, 05:39:20 pmAs far as I know there haven't been reconfigurations like that seen in the wild, but it's hard to get accurate data from the client. I'm forcing hub reconfigurations manually as a way of triggering the bug, which otherwise happens over reboots/power offs.QuoteYou have a good argument but I can't concede the point because the thing is a black box and it's not that hard to imagine some kind of buggy scatter gather DMA list processing logic that could be involved.QuoteInteresting idea, thanks. Yes I could do that, as I submit the URBs asynchronously with the usbdevfs ioctls. But is the controller going to send the IN request, get a NAK, then move on to the other endpoint? And how will the FW respond to that? Worth trying.

author: magic
author_rank: Super Contributor
timestamp: February 22, 2024, 06:09:48 am
content: My experience with device-side USB is limited to AVR and FX2, but I expect that others are similar. In these MCUs, the FW gives data buffers to the USB peripheral and then it's the peripheral's job to manage USB transactions. The FW wouldn't even know that the host polled some IN EP with no data queued and was given a NAK by the peripheral.A host receiving a NAK goes on to service other EPs or devices and retries later, as far as I understand.