does anyone actually use semihosting when it is soooo slow? - Page 1
Forum section: electronics / general

author: incf
author_rank: Frequent Contributor
timestamp: May 18, 2025, 11:32:19 am
content: "does anyone actually use semihosting when it is soooo slow?"Does it excel at certain tasks?Does it actually provide any facilities that are significantly more useful or easier to use than a uart? (named files or IO channels I suppose could be useful)I often find myself doing "real time" control tasks on a microcontroller. The slowness and blocking behavior of default semihosting implementations has always prevented me from taking the time to take semihost seriously as a debug/logging/configuration/etc. facility.Can it be less terrible and slow? (or rather, it is possible to prevent it from locking up the MCU while a small handful of IO operation take what feels like 15 seconds to complete)As a second note, I often find myself working on low power devices run from a battery. I halfway suspect that spitting data out of a UART can be far more friendly for monitoring the state of a system and maintaining low power than powering on a MCU's debug core.

author: ejeffrey
author_rank: Super Contributor
timestamp: May 18, 2025, 07:47:48 pm
content: It can be useful if you are using the microcontroller primarily for it's peripherals rather than real time capability. You can improve the speed by buffering to reduce the number of semihosting calls but ultimately it's never going to be fast so it's not worth putting a huge effort into it.I have primarily used it when I don't have an available debug uart for whatever reason. One nice alternative is the segger RTT API. It costs some memory, but IO calls are as fast as memcpy. The only downside is that if the host can't keep up data will just be dropped. But this also means that if you leave RTT compiled in your program will run fine even without the debug prove attached. The buffer will fill up and then future writes will be discarded.

author: jnk0le
author_rank: Regular Contributor
timestamp: May 19, 2025, 10:14:19 am
content: Depends on the probe/target.e.g. Stlink v2.1 (on nucleo h7) is much faster than the "chinalink" v2.0 (the one in aluminium case, requiring special openocd configs).Quotenope, and it goes into hardfault without debugger connected.Quoteonly way to speed up is to buffer more data into a single call.i.e. `_write()` (as in newlib) should do semihosting calls on whole buffers. (no need to copy into another buffer)Don't use the `stderr` or those "small"/"pico"/"tiny" libraries where `printf()` calls `_write()` byte by byte.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: May 20, 2025, 08:48:35 pm
content: No, I don't use it. I use Segger RTT.

author: westfw
author_rank: Super Contributor
timestamp: May 21, 2025, 07:54:21 pm
content: QuoteThere is frequent "debate" on the value of "debuggers" (in general) vs "print to UART" in debugging "common" problems.They are each "invasive" in different ways, producing their own sets of "HeisenBugs" (behaviors that change when "observed.")I've never used semihosting. From the descriptions I see, it sounds like it might be the worst of both worlds.Still, if you have only a debug connection, and are using all the other peripherals on the chip (no spare UART), it probably has its use cases.

author: SiliconWizard
author_rank: Super Contributor
timestamp: May 21, 2025, 08:13:54 pm
content: It might be interesting for automating tests, since it works while a debug connection is active, but if all you need is output some data, and you don't care about the debugger itself, using a UART or SPI or anything similar will usually be much more efficient.As a related note, I've seen too many people use ridiculously low baudrates for this purpose, for no reason other than because everybody does it since the 80's? Yes, back when computers had real RS322 ports with standard interface chips, you could rarely exceed 115200 bds or 230400 for the more advanced ones, but with USB adapters, this limitation is long gone.I routinely use 2 Mbds for my UART debug purposes, or something close, if peripheral/CPU clock allows. Most USB-UART adapters support baudrates at least as high as this, often higher. Just saying for those who would be sticking to 115200 bauds, or even worse, 9600 for their debugging purposes, just out of "habit". 2 Mbds is 17 times faster than 115200 and 208 times faster than 9600. Probably obvious for many, but mentioning it since I've seen it so many times. And they complain about "UART being slow".

author: incf
author_rank: Frequent Contributor
timestamp: May 21, 2025, 08:17:30 pm
content: Quote from: SiliconWizard on May 21, 2025, 08:13:54 pmSuppose you run that 2MHz signal 6 inches across the PCB to a pin header with nothing plugged in. And produce a TX signal with a less than 50ns rise time. (perhaps a bad idea)For some reason the idea of that makes me a bit uncomfortable.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 21, 2025, 08:31:36 pm
content: As a hobbyist, I've focused on microcontrollers with native USB, and plenty of endpoints. This lets me use an extra CDC ACM (USB Serial) endpoint for "debugging" information only. I've even used ELF object file shenanigans (specific sections) to minimize heisenbugs, with tiny command interfaces to manipulate internal data structures via such an extra serial interface.Although I do usegdb(and even write my own pretty-printers and access helpers for it in Python – seethis examplefor binary tree visualization from over a decade ago as an example), I rarely use it to examine code I've written myself. (I do examine the generated machine code in assembly form, sure; I just don't use a debugger to examine the same at run time.) I've been paid to write code in various programming languages (some userspace multimedia, mostly server side from kernels to user interfaces) for three decades now, and that just isn't part of my workflow anymore. I tend to verify the machine code separately, and track the internal state at run time using those separate comms channels, because I feel I am more productive and efficient that way.That said, I'm definitely not the fastest developer, and I am often blamed for being a "perfectionist", although I'm quite happy with good enough. It's just that I am not willing to leavesecurityanderror checkingfor "later when I have more time". I do expect others to have different preferences and experience, because mine is definitely not that typical. Still, I do find it intriguing how some other developers have drifted into similar solutions to mine from wildly different directions.

author: SiliconWizard
author_rank: Super Contributor
timestamp: May 21, 2025, 08:50:46 pm
content: Quote from: incf on May 21, 2025, 08:17:30 pmSo? We're talking about debugging here. I'm not suggesting to let debugging code active in production code, to begin with.That could cause some EMI issues, but you usually don't care in your lab. Also note that slew rate and communication speed are mostly unrelated. You could set 9600 bds and still have the same rise and fall times. Sure, if your UART adapter allows, a lower baudrate is an opportunity for setting a lower slew rate (if it's an option), but I've also seen most people not even care about it.If you absolutely have to let such signals active in production code and have to make them go to pin headers, you should probably rethink your approach. In such a case, don't use pin headers, because indeed they can act as nice antennas. Whether it matters would all depend on the density of toggling and slew rate rather than communication speed per se. So if you have short data bursts at high speeds, that can be negligible in terms of EMI. Worst case is of course constant toggling, such as a clock signal. Avoid outputting clock signals on pin headers, at least for production stuff.In lab for debugging, I've yet to see a single issue with that. And as a general rule, don't let debugging code run in production code unless you have a very specific and documented reason for doing so.

author: jnk0le
author_rank: Regular Contributor
timestamp: May 22, 2025, 07:55:02 am
content: Uart at 2+mbps can sometimes lose some data due to limited buffers in those usb-uart converters. Especially on windows.(that's not due to a noise as many belive)

author: Karel
author_rank: Super Contributor
timestamp: May 22, 2025, 08:13:04 am
content: Quote from: jnk0le on May 22, 2025, 07:55:02 amThat only happens when you don't use hardware flowcontrol (RTS & CTS).If you don't want to use these two extra connections for flowcontrol, you should never go faster than 9600 baud.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 23, 2025, 03:19:58 am
content: Quote from: SiliconWizard on May 21, 2025, 08:13:54 pmMe too.I've recently tested 80 MBaud UART with PIC32AK (although it's only rated for 50 MBaud) and it worked great - transferred 100 MiB in about 13 seconds. I bet it's faster than semihosting.

author: westfw
author_rank: Super Contributor
timestamp: May 23, 2025, 08:59:11 am
content: QuoteNonsense.

author: Karel
author_rank: Super Contributor
timestamp: May 23, 2025, 12:08:12 pm
content: Quote from: westfw on May 23, 2025, 08:59:11 amVery helpful response, very convincing as wellInstead of copying and pasting the same info, please have a look here:https://www.eevblog.com/forum/general-computing/the-death-of-xonxoff-flow-control/msg5641739/#msg5641739and here:https://www.eevblog.com/forum/general-computing/the-death-of-xonxoff-flow-control/msg5643133/#msg5643133Ofcourse, if you use a baudrate of, let's say 115K200 with no flowcontrol, and you are sending on average not more than approx. 1000 characters per second,there's no problem. The problem starts when your average dataspeed gets faster than that.

author: voltsandjolts
author_rank: Unknown
timestamp: May 23, 2025, 12:22:22 pm
content: Flowcontrol is only needed if their is a risk of rx buffer overflow at either end, due to small buffers and/or the buffer not being read quickly enough by the receiving cpu. It's rarely used these days.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 23, 2025, 02:17:14 pm
content: Quote from: Karel on May 23, 2025, 12:08:12 pmThis is complete non-sense. You should have adequate buffering in all stages of your process. For example, there might be a scheduling delay up to 15 ms, so there must be a buffer to hold all the characters that arrive within 15 ms period. If you don't have that, you'll lose data. The higher the speed, the bigger the buffer. For example, if you use 9600 baud rate, it's roughly 100 characters per sec, so 2-character buffer will be enough to survive for 15 ms. If you want to transfer 8 MBytes/sec, you'll need 128k, or better 256k buffer. If you have less, you'll lose characters. That's pure mathematics.As to the hardware sustained speed, a HID device can sustain traffic generated by roughly 500 kBaud UART, a USB FS device can sustain traffic from 8 MBaud UART, a USB HS device could sustain 300 MBaud UART if you were able to send it that fast. So, no problem here. And there are devices that actually use higher speeds. For example, Segger tracers can operate at up to 100 MHz on SWO line without losing anything.Of course, if your PC is swarming with JavaPythons consuming 90% of its resources desperately trying to protect you from a new nasty form of AI tuberculosis, then all bets are off.

author: SiliconWizard
author_rank: Super Contributor
timestamp: May 23, 2025, 02:45:10 pm
content: Yes. Of course, in an application where communication is critical and you can't predict what the receiving end can process at any given point, you need to use some flow control. Can also be purely part of a data protocol rather than done on a hardware level (some ACK mechanism). No question about it. Note again the *critical* adjective.But in the case we're talking about, which is basically "debug print", not only do you usually not care about an occasional loss of characters, but with a decent setup, it actually never happens. I often use Putty for this, there are of course many similar serial terminal applications - can also be just a dedicated program using your favorite language if you have some specific handling to do (a basic serial debug print terminal can be written with a couple lines of Python). If you can't handle a sustained ~ 200 kbytes/s this way with hardware that isn't 40 years old, you are probably doing something wrong. I recommend using proven serial-USB adapters if you're unsure - avoid random noname chinese stuff just to save a dollar. FT230X-based ones just work very well and are pretty cheap (as a note, it has RXTX buffers of 512 bytes each, which is plenty unless the receiving app is completely borked).I often use isolated adapters, they provide isolation, which adapts to any kind of power supply situation, and further act as level shifters (I think the ones I have handle about 1.5V to 5.5V).Regarding a physical UART + USB adapter compared to implementing a CDC interface if you use a MCU with a USB controller, sure the later can be convenient: no extra GPIO needed, no UART dedicated to this, and usually a faster yet time to send from the MCU's perspective. The downside is that it's usable, obviously, only if you have a MCU with embedded USB, if you are OK with adding a CDC interface for this purpose, and it adds a layer of complexity in your debugging setup: the USB "stack" itself - you may assume that it's 100% bug-free, but software is never bug-free. (For instance, I've used this approach using TinyUSB and the CDC class had some subtle bugs at the time.) Finally, it prevents logging any debug info *before* the USB device is initialized and enumerated, which may happen only at least one second after powering on your MCU (if it's USB-powered), making it impossible to send "debug prints" early.Just a few points.In terms of worrying about transmitting signals over simple wires at higher clock speeds for signal integrity reasons, you'll be surprised. People used to know this when they were hand-wiring CPU boards with a spaghetti of wires back in the days. I've used 50 MHz SPI for controlling displays over 15-20cm wires without a single missed byte. For prototyping of course.Again, let's put aside the EMI concern mentioned earlier. That's a different topic when dealing with debugging, at least usually. You may always think of one particular case where you had to do some debugging in a very sensitive board and would have to be very careful with EMI. We can come up with thousands of cases just to try and make a point.Likewise for control flow, again there are cases where it is required. And cases where it would be completely overkill, such as (usually) these debug print needs. (Someone may again come up with a particular case where it was very important. Use your engineering brain to determine what's required in which situaton. Otherwise, you're falling into the trap of cargo cult engineering. One should (in general) avoid extrapolating one approach that works in one given context to all other contexts - that's not engineering. Just my 2 cents.

author: Karel
author_rank: Super Contributor
timestamp: May 23, 2025, 03:51:24 pm
content: Quote from: NorthGuy on May 23, 2025, 02:17:14 pmOn windows, it's enough to quickly resize a window (by grabbing a corner with the mouse) for a couple of seconds and you'll loose incoming characters on the serial (usb) port when not using flowcontrol.Last time I checked this was with windows 7, maybe windows 10 has improved in this regard, I don't know.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 23, 2025, 04:15:49 pm
content: Quote from: SiliconWizard on May 23, 2025, 02:45:10 pmOften you cannot use flow control.If you're sending a piece of data which must be received in their entirety (such as uploading a file), then you can make sure that the file is transmitted without errors using flow control and re-tries. This comes at a price of extra time. The file will arrive, but because of pauses and retries it will take longer.If you're sending real-time data (such as transmitting a variable every microsecond to see how it changes), then you don't have extra time. As soon as the buffer on your MCU fills up, you have no choice - new data must go somewhere or be lost. Doesn't really matter if you use flow control or not. The only sensible thing you can do is to use a bigger buffer. This is, for example, what a scope is doing - that's why they all have such big buffers - they capture the data and then they may transmit it to the PC at their own pace. If you want to capture a limited amount of data on your MCU, you can do this as well. But only if your data is small enough to be stored on MCU. Otherwise, you need to use real time transfers.To avoid losing data in real-time situations you must ensure minimum bandwidth - each upstream unit must guarantee that it will move certain amount of data per unit of time, as opposed to flow control mechanism which will stop and inform the transmitter that it cannot get more data. Usually, the way to do it is to increase transmission speeds and maintain adequate buffers. Flow control is no substitute.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 24, 2025, 12:32:08 pm
content: Quote from: Karel on May 23, 2025, 03:51:24 pmIt depends on the driver as well as on the app which is reading the data. Does that happen when you resize any window, or just the window which belongs to the UART app?

author: HwAoRrDk
author_rank: Super Contributor
timestamp: May 24, 2025, 02:24:14 pm
content: Quote from: Karel on May 23, 2025, 03:51:24 pmI would be willing to bet this is more to do with how the application is written (poorly!) rather than Windows itself.Sounds like such an application is processing the reception of serial data within the window's main message loop, so when it starts getting a bunch of WM_RESIZE messages from the OS, and is repeatedly repainting the window, it has less time to spend on handling serial data. Handling the serial data in a separate thread would likely avoid the problem.

author: westfw
author_rank: Super Contributor
timestamp: May 25, 2025, 10:32:04 pm
content: QuoteWhen 56k modems were starting to appear in the consumer marketplace, at least one vendor started including a new UART card with each modem. Apparently, the "Old" serial cards with two bytes of buffering would not get serviced fast enough to prevent data loss. The new UART cards had deeper FIFOs, and that "fixed" it. Apparently including the new card was cheaper than handling all the customer complaints...Which of course has very little to do with how fast you can run traffic through a USB/Serial adapter, which is likely to have enough levels of buffering to make your head hurt...

author: Siwastaja
author_rank: Super Contributor
timestamp: May 26, 2025, 05:13:34 am
content: Quote from: Karel on May 22, 2025, 08:13:04 amYou are posting from 1995?UART flow control is dead. Because no one uses it, good luck finding non-broken (non-buggy) implementation you can depend on. Instead, large buffers became reality.UART between two hard real-time capable systems (e.g. microcontrollers) is easy to prove to not require separate flow control, even with buffer size of one byte (the data register of peripheral) - which is ironical, because those systems also could provide working flow control - but it is so much easier to just process the byte before the next one can appear (i.e.: just don't be slow), than to push the issue of processing slowness to the sender side (what shouldthatdo then? implement infinite buffer for real-time data?)On UART <-> general purpose PC, possibly USB adapter, flow control is unavailable or broken, and everybody depends on large bufferson the receiver side. And no one counts on having guaranteed 100% reliability - even at 9600bps. If you need that, you need something else, not UART.Flow control is nothing more than pushing responsibility of slownessof receiverto sender. The world figured out 20 years ago already it is almost always so much easier to do that on recipient side (fast processing, or buffering) that we stopped using flow control. As generic design pattern,managing somebody else's problems(in this case: sender managing receiver's slowness) ends up being complicated as it leads to distributed control system, and design teams having to anticipate problems outside their core responsibilities (also difficulty of testing).

author: Karel
author_rank: Super Contributor
timestamp: May 27, 2025, 10:18:39 am
content: I stand corrected.The comments from many people here made me start doubting and I decided to do some real testing today because my opinion was based on some experiences from 15 years ago.Today I did a test with a NUCLEO-L452RE which I connected to my workstation using an FTDI TTL-232R cable.The chip used in that cable is an FT232R which has a 256 byte transmit buffer and supports a max. baudrate of 3 Mbaud.The MCU of the Nucleo board sends continuously data to the serial port at 3 Mbaud using DMA.The data is created using xorshift32 in order to check for data corruption or missing bytes.The result is that I can send data at a speed of 3 Mbaud to my workstation (i9-13900K, openSUSE Leap 15.6) without flowcontrol and without data corruption.(I want to repeat this test with an older notebook and with a Raspberry Pi but that has to wait for now)The software at the receiver side reads from the serial port with an interval of approx. 10 millisec.When I increase this interval to 15 millisec I'll get data corruption.Somebody wrote that hardware flowcontrol wasn't reliable. I still don't agree with that. On the contrary, I found it extremely reliable.When I increased the read interval I was able to avoid any data corruption by using hardware flow control.Somebody wrote that flow control is something from the past and not used anymore. I still don't agree with that.When dealing with, for example, Classic Bluetooth modules (Serial Port Profile), it's practically impossible not to use hardware flow control and have a reliable connection.The same is valid for updating the firmware of these modules, you can't do that without hardware flow control.So, thanks to everybody for pointing out I was wrong, it's a valuable lesson.I'll report back here when I have done some testing with some older/slower computers.

author: peter-h
author_rank: Super Contributor
timestamp: May 27, 2025, 07:48:43 pm
content: The 32F4 still implements RTS/CTS hardware handshake, and it works (it is an option on my last product, user-selectable on one of the two RS232 ports). The FT232 also implements it. But there is no use for it normally, since all protocols in common use are packet based and the receiver allocates enough buffer for a whole packet.Hayes-type modems used hardware handshake, but there were reasons for it e.g. the data rate could be higher than the 9600 baud used for the RS232 interface. Old times... Some of us made a sh*itload of money in those good days3 megabits/sec is silly on RS232 because the normal interface chips like MAX232 can't run it. It is OK on 422/485 but only with the "10 mbps" chips.I googled semihosting. Never heard of it. Can't see any use for it. But for sure emitting debugs out of a UART is easier than running a target under a debugger. The SWV ITM debug channel can run very fast (megabit speeds) so is relatively nonintrusive, but is also unreliable.

author: Siwastaja
author_rank: Super Contributor
timestamp: May 28, 2025, 07:18:15 am
content: Quote from: peter-h on May 27, 2025, 07:48:43 pmUsing UART with 5V or 3.3V (or any other) CMOS (more rarely TTL) levels - although this use case is incorrectly often called "TTL" - is very common nowadays. Saves you the cost of RS232 transceiver IC (and the related speed limit as you mention). Of course, signal integrity is what it is, not good for long cables. Then again, no one would use good old RS232 for long cables either.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 28, 2025, 04:45:12 pm
content: Quote from: peter-h on May 27, 2025, 07:48:43 pmThese are very old interfaces created for totally different purposes. Modern communications, such as USB superspeed or Ethernat are well in Gigabits. If your MCU had an appropriate differential transmitter you could communicate much faster too. You still can build a parallel interface which is then converted to USB or Ethernet to get transfer rates of 50-100 MBytes/sec or even faster. It's actually very easy to accomplish with a small FPGA and FT601, for example.

author: westfw
author_rank: Super Contributor
timestamp: May 29, 2025, 04:45:13 am
content: QuoteI believe that the problem is with exactly when flow-control events happen, WRT things like hardware FIFOs, especially if the "hardware flow control" is implemented in software. It's similar to the problems one has determining when exactly all the bits have dribbled out of the the transmit shift register.Or you can have problems when the UART receiver ISR is "plenty quick enough", but the higher level software is out of space...

author: peter-h
author_rank: Super Contributor
timestamp: May 29, 2025, 06:02:13 am
content: IME, and having tested many, UARTs that implement hardware handshake, do it very well, and it works "instantly". It stops transfer from the holding register to the shift register, regardless of the FIFO size.QuoteFPGA? I think somebody else is paying for the bread on the table at your houseI used to do FPGA/ASIC design consultancy. This is the last thing I would spend time on. If I wanted asimplehigh speed serial link I would probably use SPI, for 10mbps and beyond. One is rapidly limited by the cable. That (a USART) is basically what ST use for their SWV ITM debug channel.

author: Karel
author_rank: Super Contributor
timestamp: May 29, 2025, 06:29:43 am
content: I repeated this testhttps://www.eevblog.com/forum/microcontrollers/does-anyone-actually-use-semihosting-when-it-is-soooo-slow/msg5927524/#msg5927524successfully on a simple, 12 year old, office pc running Linux Mint. No issues so far.

author: gamalot
author_rank: Super Contributor
timestamp: May 29, 2025, 07:48:05 am
content: I use UART or Segger RTT and never tried semihosting.

author: Siwastaja
author_rank: Super Contributor
timestamp: May 29, 2025, 11:19:05 am
content: Quote from: westfw on May 29, 2025, 04:45:13 amOr, simply that at least 90% of USB-UART adapters on the market (and hence, on engineers' drawers) do not implement HW flow control at all, especially the cheaper ones. The chip (FT or CH something) inside might implement it, but 5V/3.3V adapters do not bring the wires out at all, and RS232 versions save on transceiver cost by leaving HW flow control signals out.It's that people stop using flow control as they realize it is usually more work than doing without it, which then means there is less demand for HW and SW that supports it, and it will be tested less in practice, so it's more buggy as well.It's a nice idea in principle if you are designing a receiver only, until you realize that it's just pushing responsibility around, and now if you are designing the transmitter, it gives you more burden than it saved on receiver design.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 29, 2025, 01:50:37 pm
content: Also, USB CDC ACM, also known as USB serial, does not support flow control control.That is, if your USB dongle does support flow control, you must use specific drivers to enable flow control, because USB CDC ACM does not know about it.It is annoying as all hell to me, because there are unused status and control bits for it. I've ranted about this before...

author: peter-h
author_rank: Super Contributor
timestamp: May 29, 2025, 05:00:52 pm
content: QuoteIf we are talking about the same thing, USB VCP does support flow control. It is inherent in the way the Host polls the Client whether it has data or not, and whether it has read the last bit of data.This is one of several threads on ithttps://www.eevblog.com/forum/microcontrollers/32f417-how-does-usb-cdc-out-(pc-gt-target)-flow-control-work/msg4447447/#msg4447447It is possible that many coders of the Client device don't implement it...From my 2022 project notes:USB VCP (CDC)The USB VCP is linked to the serial comms functions, as Port 0.VCP Output (XXX to PC): originally there was no flow control implemented. Experimentally, it was determined thatthere is a call rate limit on CDC_Transmit_FS() of about 10kHzthe function has a max packet length of around 800 byteswaiting 200us between packets is another approachthe above values are likely to be system dependentThe original Cube MX generated code did not implement any flow control. This was workable if one put in delays and packet rate limits. Later it was determined that flow control is possible. This explains it:https://community.st.com/s/question/0D53W00001mfognSAA/32f4-usb-cdc-any-way-to-tell-if-host-has-retrieved-the-data-sent-to-it?t=1662361131149https://community.st.com/s/question/0D53W00001qnO9ESAU/32f417-how-does-usb-cdc-out-pc-target-flow-control-work?t=1664975189841However, this was not usable becauseTeraterm (or any other terminal) cannot quite cope with a lot of data at the full XXX speedfunny things happen if there is no VCP application running at allIf the PC did not have a USB VCP application running, consuming the data, serial tx on the XXX might be blocking. Actually this does not happen and windows chucks the data away, but every few minutes it goes to sleep so the XXX task would block, for maybe seconds. This is probably Host dependent. More herehttps://www.eevblog.com/forum/microcontrollers/32f417-how-does-usb-cdc-out-(pc-gt-target)-flow-control-work/msg4793321/#msg4793321Flow control is (optionally) implemented in CDC_Transmit_FS(). There is a timeout there to take care of the above “no VCP application running” condition.Detecting the lack of a USB VCP application running etc is apparently impossible without a custom driver.Now there is a TX buffer like there is for the serial ports, cdc_transmit_buffer, size CDC_TX_BUFFER_SIZE, and opqspace() returns a correct value based on this buffer. One does need a working opqspace() to write code which yields to RTOS if there is nothing to do, etc. This circular buffer is copied to CDC in the USB thread in main.c, at a rate of 256 bytes every 5ms.VCP Input (PC to XXX): The data is initially loaded by an ISR to cdc_receive_temp_buffer and an ISR then calls CDC_Receive_FS which copies that into a circular buffer cdc_receive_buffer. This is the flow...Flow control is implemented in CDC_Receive_FS().

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 30, 2025, 08:33:04 am
content: Quote from: peter-h on May 29, 2025, 05:00:52 pmThere is no "VCP" in USB standards. I believe that is some Windows-only thing you're talking about.I'm talking about the actual USB standard, currentlyClass definitions for Communication Devices 1.2 (2025-05-28); and thePSTN120.pdfin the combined ZIP. The ACM comes from "Abstract Control Model" subclass.If you look atPSTN120.pdfchapter 6.2.2 Abstract Control Model Requests, you can see the requests the host (computer) uses to control the actual bridge device. For example, 6.3.10 SetLineCoding (SET_LINE_CODING) is used to set the baud rate, number of stop bits (1, 1.5, or 2), parity, and number of data bits (5, 6, 7, 8, 16).There are two ways the USB CDC PSTN ACM could be extended to allow the host to control the flow control method on the USB bridge device.The minimal one is to use a some of the 14 unused bits in 6.3.12 SetControlLineState (SET_CONTROL_LINE_STATE).The full one is to use two of the currently reserved request numbers, for example 0x24 and 0x25, for SET_FLOW_CONTROL and GET_FLOW_CONTROL requests.These requests have a 16-bit value as a payload, plus optional data section; currently the longest sata section is seven bytes (for SET/GET_LINE_CODING).The full one could even support software flow control, by allowing the host to specify the XOFF and XON characters (two or four bytes), plus the flow control type enable bits. (I have heard of one device which actually required both RTS/CTS and XON/XOFF flow control at the same time.)As noted above, the device-to-host path is not problematic, as it is well enough buffered in well-written software anyway. It is the host-to-device path that really needs flow control, the device being able to tell the host when it is ready to receive more data. This control is trivially done in the bridge device; it simply needs to NAK any incoming USB data packets if it does not have enough internal buffer space for that much data. This works perfectly in all operating systems, because thisisthe standard USB flow control mechanism: the host is not allowed to discard the data before it receives an ACK from the target USB device. (Compare to TCP. Some vendors like Broadcom occasionally do fuck this up, and try to paper over it in software. Avoid those.)If the ACM subclass for PSTN devices in the USB CDC standard was extended with those two commands, we could have hardware flow control capable USB-serial dongles without requiring specific drivers; the same driver would work across all vendors.The funky thing is, requests like SET_CONTROL_LINE_STATE and SET/GET_LINE_CODING arealready optional. So, if SET_FLOW_CONTROL and GET_FLOW_CONTROL requests were added to the standard as optional, and OS drivers updated to use them,all current CDC ACM devices would continue to workwith those modified drivers; they simply would not support flow control! There is no question about backwards compatibility here, only about lack of interest or willingness to add the features to the standard. The only question is the exact contents of the parameter block (whether software flow control would be supported, whether flow control could be enabled in one direction only, et cetera), and pushing it into the standard. I don't know about the Windows or osX driver complexity, but I know for a fact that the Linux driver for this would be very, very straightforward and robust.Just think about it. You could use a cheap USB FS for your own serial-only devices as the bridge, or almost as cheap USB HS for up to 200-400 Mbit/s serial, and It Would Just Work, on all operating systems, without any kind of driver mess. (Of course, that is exactly why the large companies supporting USB-IF like FTDI are opposed to adding such to the CDC ACM class.)

author: NorthGuy
author_rank: Super Contributor
timestamp: May 30, 2025, 01:18:06 pm
content: Quote from: Nominal Animal on May 30, 2025, 08:33:04 amYou don't really need CTS/RTS settings to make CTS/RTS work. The converter may use CTS/RTS lines if you connect them, and have a pull-down on its CTS, so it would work without flow control if the CTS/RTS lines are disconnected.

author: peter-h
author_rank: Super Contributor
timestamp: May 30, 2025, 02:59:41 pm
content: QuoteThat's curious. I've been selling USB-RS232 and USB-RS485 converters, PL2303 and FT232 based, which have linux drivers. I am not a linux user but this must be creating some serial device within the OS. Under Windows it is a straight COM port.QuoteSure; if there are pullup resistors in there on the CTS input.

author: westfw
author_rank: Super Contributor
timestamp: May 31, 2025, 02:02:03 am
content: QuoteBut both of those have "custom" drivers (that presumanly DO provide the HW flow control capability), rather than relying on the "standard" CDC/ACM driver.The fact that they "present" as COM ports doesn't mean that they're CDC/ACM.

author: ejeffrey
author_rank: Super Contributor
timestamp: May 31, 2025, 03:37:34 am
content: Quote from: peter-h on May 30, 2025, 02:59:41 pmThose are not usb ACM. They are "proprietary" interfaces although they are quite simple and obvious, at least for the basic serial port.VCP is virtual com port, it's used at least for FTDI chips to distinguish two different modes of windows driver operation. VCP means it can be controlled by the standard windows COM port API. D2xx means it uses the libFTDI interface. This means you can't use it from standard terminal emulators but you can use other operating modes like SPI/I2C/JTAG, access extra GPIOs and do other things that don't fit in the serial port API.The FTDI chips actually had a EEPROM with a bit telling it whether to load the VCP or D2xx drivers. But the distinction is kind of irrelevant in Linux where you can detach the kernel serial port driver dynamically to use the low level calls.

author: peter-h
author_rank: Super Contributor
timestamp: May 31, 2025, 06:43:29 am
content: At the 32F4 end, it is CDC that's implemented. Then you have a VCP driver in the OS.The D2XX stuff is used to write to the EEPROM, which on the later FTDI chips is on-chip.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 31, 2025, 06:59:28 am
content: Quote from: peter-h on May 31, 2025, 06:43:29 amNo. VCP is a Windows-only thing. You have a "VCP driver" only in Windows, not in any other OS.Linux uses a FTDI-specific driver (drivers/usb/serial/ftdi_sio.c) written by well-known open source developers; FTDI only provided the protocol, USB requests used to control the bridge device. (Those are trivial to sniff anyway, so no reason to keep them secret.)In Linux, a "serial port" is any character device that presents a termios-capable interface (and optionally certain serial-port specific ioctls). You can even create them in software (pseudoterminals), and are very commonly used. Same applies to Android, of course; but also to BSDs, other Unix variants, and even mac OS. Basically everywhere except Windows (and some niche non-Unixy/POSIXy OSes).

author: peter-h
author_rank: Super Contributor
timestamp: May 31, 2025, 07:01:56 am
content: The story I was told was that when M$ were doing USB they implemented a keyboard, mouse, parallel port, etc, but forgot about a serial port, hence we have the VCP driver mess.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 31, 2025, 08:18:52 am
content: In Linux, there are two mechanisms for driver selection: one is matching USB vendor:product IDs, and the other is fallback to generic driver per endpoint, based on the endpoint USB class and subsystem (in its descriptor). This Just Works.It also means that if you do a composite device, with say a bulk interface for your "proprietary" communications, you can always add an USB CDC ACM endpoint, and that will appear as a serial port, no driver installation needed. Similarly for all standard USB HID interfaces (keyboards, mice, joystics, touchpads et cetera): they are supported by the generic drivers, so any combination of USB HID endpoints (and other standard USB devices like USB CDC ACM) will Just Work. Something like a keyboard+multimedia keyboard+joystick+serial+serial+serial is surprisingly useful for some self-made gadgets, and needs no drivers in Linux, Android, *BSDs, macOS.In Windows, you must only use SpEcIaL WiNdOwS descriptors, likethis, because Microsoft knows better than the USB-IF consortium, you see. (This originates from back when MS still vehemently opposed anything open or standard not controlled by them, and due to backwards compatibility requirements and the immense cost of reworking the entire input subsystem, cannot really be changed even if MS itself has mostly changed its stance.)In practice, this means that if you create a custom composite device by constructing your own USB descriptors, it does not suffice that you make themcorrect. Yes, any correct one will work fine in Linux, Android, BSDs, macOS, et cetera.. but Windows is special. For Windows, you need to do extra work, and verify different versions of Windows accept the descriptor and provides the expected devices; AFAIK, testing in practice is the only sure-fire way. Fortunately, with something like Teensy 4.0, with a little bit of prep work, you can construct any USB descriptor in the Arduino environment, so the testing cycle is quick –– if you have some different versions of Windows to test with; VMs will do fine, if you can assign an entire USB controller to the VM.

author: tru
author_rank: Regular Contributor
timestamp: May 31, 2025, 10:11:45 am
content: I would consider the Windows usbser.sys (USB CDC-ACM driver) as a generic driver, added in later Windows. It does work on all USB devices using the standard CDC-ACM descriptors and without any matching USB vendor:product ID.Checking out the serial device under the device manager will reveal which driver is being used, see attachment. If it doesn't show usbser.sys it is using "proprietary" mess.Btw, Linux does also need a driver for usb serial (CDC-ACM), but being better developed than Windows there is already one very long ago:https://github.com/torvalds/linux/blob/master/drivers/usb/class/cdc-acm.c

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 31, 2025, 11:08:42 am
content: Quote from: tru on May 31, 2025, 10:11:45 amThis was my understanding also, and obviously a good thing.Similarly for standard USB HID devices (keyboards, mice, joysticks, touchpads); those have a generic standard USB drivers also.(In Linux, you can also create any USB HID device in software, using theuinputmodule. This inserts HID events into the kernel, so it will work everywhere on that machine and all local applications and environments: X11, Wayland, even the terminal console. If you know anything about USB HID protocols, you'll find the interface quite easy to use.)Such USB standard drivers are quite straightforward, too. So much so that one can definitely support standard HID and serial devices even in ones own small microcontroller projects, if you have a native USB host ports (or OTG ports), and sufficient RAM available for endpoint buffers. Keyboards, gamepads, and mice in particular are easy to support, if you need only the standard features you get without installing any device-specific drivers or translation applications.What I do not know, is how well the Windows one works with composite devices. I do know that for example the Teensy/Teensyduinocomposite USB device profiles (with various USB serial and HID device endpoint combinations) do all work in Windows also, but if I understood correctly, it took quite a bit of testing by Paul Stoffregen to fine-tune the USB descriptors for everything to work well in all Windows versions too. So, itdefinitelyis possible to support those in Windows also, it's just that even if the USB descriptors work fine in other OSes, Windows tends to be pickier (for historical reasons and backward compatibility and technical design debt, not for any political reason nowadays).My first microcontroller project was an "Arcade plank", an arcade-style digital joystick + buttons combo, that generated keypresses, so I could play Flash puzzle platform games, based onTeensy 2.0++(Atmel AT90USB1286 with native USB 1.1 FS 12 Mbit/s) on "bare metal", avr-gcc and avr-libc, pre-Arduino. It worked just fine in Linux, although if I recall correctly, I did have something wonky (my fault!) in the USB descriptor, so the first time I used it in Windows, it didn't work.Quote from: tru on May 31, 2025, 10:11:45 amYep!For reading the Linux kernel sources, especially when comparing between different versions, I recommend usingelixir.bootlin.com/linux/; for exampledrivers/usb/class/cdc-acm.cfor the above. The Elixir Cross Referencer web interface makes all names links, so you can very easily check where one is defined and used, across the entire source tree. The canges you see between say 2.6.39.4 and 6.15 are due to kernel internal interface changes and devices added to the quirky list (acm_ids) identifying the workarounds needed for problematic devices.(In case anyone wonders, the_minorstuff is how Linux and Unixy/POSIXy systems differentiate character devices: each character device has an unique identifier consisting of major:minor, stored in the inode (seeman -s 3type statfor the information userspace sees), where major usually identifies device type, and minors are allocated dynamically as needed. Back in the olden days, they used to be fixed. Also, thettystuff is the device driver side for thetermiossupport, making the character device serial-port-like.)Similarly for official/upstream u-boot sources atelixir.bootlin.com/u-boot/. There are several projects similarly cross-referenced there; see the text at the very left edge of the window with the ▼, just below the full-window-wide bootlin title bar. Clicking on the text will pop up a list of the projects already cross-referenced.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 31, 2025, 03:43:54 pm
content: Quote from: Nominal Animal on May 31, 2025, 08:18:52 amWindows has built-in HID and CDC drivers, same as Linux. The "generic" USB devices are handled by usbfs in Linux, which is used, for example, by libusb. In Windows, they're handled by WinUsb driver which is very similar to usbfs.The special descriptors you're talking about are not necessary, but if they are present, Windows will associate the device with the WinUsb driver automatically. Without the descriptors you must establish the association manually (for example through a Device Manager or Zadig). The descriptors are actually a part of a broader mechanism which allows you to create your own classes of USB devices and supply your own drivers instead of WinUsb.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 31, 2025, 04:40:16 pm
content: Quote from: NorthGuy on May 31, 2025, 03:43:54 pmWe are talking about USB descriptor structures. The host asks these from each device during enumeration.Problem iscomposite devices, in the USB standard sense. Not all composite device combinations using perfectly correct USB device and endpoint descriptors work in Windows, even if all those endpoints use standard class drivers.As an example of preprocessor macro defined endpoint descriptor construction, seegithub.com/PaulStoffregen/cores/blob/master/teensy4/usb_desc.c. The USB standards allow these to be defined in various different but equally valid ways and orders, but Windows doesn't accept them all, unlike other OSes.Also, in Linux, there is no need for a tool like Zadig. The standard /sys interface contains pseudofiles "bind" and "unbind" (for example,/sys/bus/hid/drivers/hid-generic/bindand/sys/bus/hid/drivers/hid-generic/unbindfor USB HID class driver) that you write the kernel identifier to (seen in e.g. dmesg). You can essentially bind any USB device to any driver you want (as long as it is not listed as one to be ignored in the driver, which you would need to override using a kernel command line).

author: NorthGuy
author_rank: Super Contributor
timestamp: June 01, 2025, 12:04:48 am
content: Quote from: Nominal Animal on May 31, 2025, 04:40:16 pmGood to know. I would certainly never guess that by "SpEcIaL WiNdOwS descriptors" you would mean standard USB descriptors. Microsoft have their own descriptors too, which are not standard:https://learn.microsoft.com/en-us/windows-hardware/drivers/usbcon/microsoft-defined-usb-descriptorsThe don't replace standard descriptors, but complement them.Quote from: Nominal Animal on May 31, 2025, 04:40:16 pmComposite device is just a device with multiple interfaces. They work on Windows. I used many. All composite devices are taken by Usbccgp driver which creates separate devices for each interface. These devices are assigned their own drivers according to the USB interface classes or other criteria. If you assign a composite device to a different driver (instead of Usbccgp), it will no longer be treated as composite. It all works fine. I used this many times. If some of the composite devices don't work, you need to look at the particular devices and find what is the problem.See here for more details:https://learn.microsoft.com/en-us/windows-hardware/drivers/usbcon/enumeration-of-the-composite-parent-deviceQuote from: Nominal Animal on May 31, 2025, 04:40:16 pmWhat exactly it doesn't accept? Can you point to a particular descriptor you're having problems with?Quote from: Nominal Animal on May 31, 2025, 04:40:16 pmYou can do this with Windows too - DiInstallDevice() does exactly that. There's a command tool called "devcon" to do this from command line. Of course, in Windows driver associations are permanent (they persist across reboots until you change them), not ad hoc as in Linux.