Bakelite: A utility that makes it simple to communicate with your firmware - Page 1
Forum section: electronics / general

author: brendan0powers
author_rank: Newbie
timestamp: January 11, 2022, 02:05:23 pm
content: Hello! I've just released the first version of my new open source (MIT) library called Bakelite, and thought people might find it useful. It's intended to make building custom protocols for serial/USB/etc less tedious. It does this by providing a code generator and library that provides struct serialization, framing, and error detection. Currently there's a Python and C++ implementation, but I hope to add new languages in the future.Features:Compact, easy to understand data serialization formatSimple message passingBuilt in framing and error detectionEasy to integrate with Serial, USB, TCP, SPI, etc...Use only the parts you needCode generators for:C++ (header only, no STL or memory allocation)PythonDocumentationIf you're interested, you can find it here:https://github.com/brendan0powers/bakelite

author: dmendesf
author_rank: Frequent Contributor
timestamp: January 12, 2022, 01:13:25 pm
content: Let me ask you an unrelated question: What tool did you use to make the SVG diagram in your project page?

author: brendan0powers
author_rank: Newbie
timestamp: January 12, 2022, 02:29:15 pm
content: Google Drawings. It's pretty clunky, but it's free and was faster than searching for a real diagram tool.

author: voltsandjolts
author_rank: Unknown
timestamp: January 12, 2022, 02:47:58 pm
content: Cool.A code generator for plain C would be nice for us old-fashioned folks

author: FlyingDutch
author_rank: Regular Contributor
timestamp: January 12, 2022, 04:23:21 pm
content: Hello,maybe tomorrow I would like to try this project, it looks very interesting.Best Regards

author: brendan0powers
author_rank: Newbie
timestamp: January 12, 2022, 04:30:52 pm
content: QuoteYes, a pure C implementation is on the to-do list. It would be particularly useful in situations where you don't have a modern C++ toolchain available. Working with an 8051, for example. Is there a specific C standards version I should target?

author: voltsandjolts
author_rank: Unknown
timestamp: January 12, 2022, 05:44:09 pm
content: C89 would give maximum coverage I suppose, but if C99 makes things easier I would be happy with that (thinking of my older Microchip gcc compilers and sdcc for stm8/8051).

author: voltsandjolts
author_rank: Unknown
timestamp: January 13, 2022, 05:18:35 pm
content: At a glance, I think you are usingCOBSfor framingPlus options for CRC8/16/32 over the payload.Nice.Edit:Confirmed, just seen COBS mentioned in your docs. It's the best option for serial framing over uart.Perhaps another option would be no framing or crc, if using usb, and the rpc calls were always shorter than endpoint size (could raise error during bakelite build if not).

author: SiliconWizard
author_rank: Super Contributor
timestamp: January 13, 2022, 05:40:51 pm
content: A number of us have our own solution for this, but OTOH, communication protocols in general is a frequently asked question, so I'm sure it can be useful for some.I've never used COBS, but since it's mentioned, I'll have a look and see if/when the overhead is worth the trouble.

author: brendan0powers
author_rank: Newbie
timestamp: January 13, 2022, 05:54:34 pm
content: QuoteCOBS is a great choice for a framing algorithm. The biggest issue with it is that it requires at least a 255 byte buffer. At one point I was hoping to do away with the read/write buffers. I could still get rid of the read buffer, but the implementation gets a fair bit more complicated. In the meantime, if the maxLength is set to something under 255 bytes, I can assume I'll never need the full 255 byte buffer.For the USB use case, you can disable CRC, but you can't disable framing. The current API wouldn't even support this, there's no way to tell the decode that a frame has ended. I'll have to think about that use case a bit more.

author: voltsandjolts
author_rank: Unknown
timestamp: January 13, 2022, 07:46:37 pm
content: Here's a COBS topic I posted some time ago, with some python and C code, which might be useful for someone interested in learning about it.https://www.eevblog.com/forum/microcontrollers/implementing-uart-data-packets-with-consistent-overhead-byte-stuffing-(cobs)/Yeh, with usb providing structured serial, COBS would be unnecessary processing there.Maybe just a simple byte count at the start of the frame would be sufficient in that case.Google Protocol Buffersalways seemed a bit OTT for my needs.With a plain C code generator, this could be a much better fit.

author: poorchava
author_rank: Super Contributor
timestamp: January 22, 2022, 08:53:50 am
content: COBS makes sense, although I don't see the reason for constant length. Framing encoding in general makes life mich easier.I've been using ordinary byte stuffing (start, stop, escape, xormask) for a while and can't see any reason to change.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 22, 2022, 10:10:42 am
content: I prefer protocols that can be handled with minimal buffering.Like others, I use my own, additionally multiplexing a single channel for multiple separate data streams. (My Odroid-HC1 has only one 1.8V UART, so I developed a simple way to multiplex console with peripheral (GPIO) data. The UART is interfaced to a microcontroller with native USB, which can be connected to a separate machine providing two USB endpoints; one of which is the serial console, and the other is the peripheral/GPIO data.)Simple byte stuffing with a carefully chosen escape mechanism means I can switch between streams at almost any position, with a maximum buffer delay of a single byte, with only a few bytes (less than half a dozen, depending on the maximum number of data streams) of total overhead.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: January 22, 2022, 10:58:58 am
content: Quote from: dmendesf on January 12, 2022, 01:13:25 pmSpeaking of which, that diagram is partly illegible when using GitHub's dark mode, because the background is transparent.

author: brendan0powers
author_rank: Newbie
timestamp: January 22, 2022, 02:30:18 pm
content: QuoteThanks for pointing this out! I've updated the image with a white background.

author: voltsandjolts
author_rank: Unknown
timestamp: January 23, 2022, 10:13:15 am
content: Quote from: poorchava on January 22, 2022, 08:53:50 amYou've got that wrong, a cobs framecan be any length.QuoteSounds much likeslipwhich is ok but if your data happens to contain a lot of your chosen escape character, then the overhead goes to 100% (link bandwidth halved).

author: brendan0powers
author_rank: Newbie
timestamp: January 24, 2022, 02:56:31 pm
content: QuoteThis was the reason for going with COBS. No need to try and find an escape character that's unlikely to be used in your messages. On the other hand, the buffering requirements for COBS might make supporting SLIP worth it.

author: voltsandjolts
author_rank: Unknown
timestamp: January 24, 2022, 03:39:54 pm
content: Yeh, why not, users like configurable options.The buffering requirements for cobs can be minimal. Decoding received data is trivial, it can easily be done within the rx interrupt, without any additional buffering. With a little effort, transmission encoding can actually be done 'in-place' in the tx message buffer itself, as long as the buffer size can hold the few extra bytes required for overhead (see my thread linked above). Of course, if you're looking for more performance, then perhaps ping-pong buffers for DMA transmission would be used, which needs more ram but that's not a big issue on modern arm cortex. For cobs I think the juice is worth the squeeze!

author: Siwastaja
author_rank: Super Contributor
timestamp: January 24, 2022, 04:11:41 pm
content: I don't see the RAM use of buffering problematic, and for transferring large amounts of non-realtime (or "easy realtime") data I don't see the issue with delay, clearly it's an excellent encoding, especially recovery from byte loss/corruption (framing errors) in predictable time.However, am I missing something if I say adding 255-byte delay between thegenerationof data (you can't look ahead something you don't yet have no matter how much RAM you have at your disposal), to data hitting the target, is a massive turn-off exactly in realtime embedded work, I don't mean audio or video streaming, but communicating smaller amounts of structured data where you need to react to the inputs quickly. For data logging / streaming, this doesn't matter, but subject says "communicate" which is different I think.Also the claim that this delay can be made predictable and consistentin number of bytesis a red herring, because what actual matters is absolute time in seconds, and that is far from predictable, unless you send stuffing bytes all the time at the full maximum expected communication speed. Maybe this is exactly what is being done? But still, this all makes me think that this might not be the best possible protocol after all for generic microcontroller communication use.Me, I tend to use the classic magic + msgid + len + payload + crc paradigm which causes minimal delay and keeps the raw data stream intact and only looks in it for CRC calculation. But framing error is only caught at CRC step, after which scanning for a sync packet is required, making it lose some data during that wait, so not very good for unreliable links. But I also tend to write different implementation every time, because microcontroller projects are simple but tend to have some weird specific requirements.The only byte-delimited byte stream in MCU world I can think of is UART. UDP over Ethernet comes with packet delimitation. TCP over Ethernet comes with guarantee of no lost bytes so length + payload works perfectly. Raw ethernet frames come with packets. USB - packets. SPI - again, hardware delimited packets (despite the fact some don't understand this and do not use it in their advantage). Radio communication: packets.Using mediums that transfer packets - just to emulate byte streams over them - then to realize you needed packets not byte streams - just to add another layer of code emulating packets - seems unnecessary work. The total amount of buffering and delay is obviously then more than just needed for the upmost (COBS) layer.But I'm probably missing something, the intention is not to criticize, but to ask what I am missing.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 24, 2022, 05:15:30 pm
content: Not all packetized transports expose the packet boundaries to userspace; for example, USB Serial.Even when packet boundaries are exposed, it may not be possible to determine if two consecutive packets were sent in a single transmission or not.It is even possible for packet order to change, although this usually only happens on the internet.This means that if one relies on packets, each packet must be somewhat self-contained. If the packet length is determined by hardware –– for example, using UDP over IP you'll want to use 508-byte packets to maximize bandwidth and minimize overhead and the number of dropped packets; or say as in USB –– you may need to encapsulate the application-sized packets anyway, and distribute them over multiple hardware packets.SeeIP fragmentationfor a particularly relevant real world case.It is often more efficient (and sometimes necessary; e.g. with my Odroid HC1 which only exposes a single UART and no GPIO pins) to multiplex a single "physical" connection for one or more logical data connections.A particularly good example is TLS-encrypted TCP connections. The encryption itself is rather lightweight, compared to the computation needed during the two-way handshake. If you can use a single encrypted connection for all your different logical data or packet streams, you will need less computation (cryptographic math, especially public-key crypto using key lengths in the thousands of bits) and get better bandwidth on the same hardware, than when using a separate encrypted TLS connection for each logical data or packet stream.If those streams have different priorities, you either need to buffer each logical stream separately (which can lead to quite a bit of buffer space needed), or you need a way to interrupt/insert data from a new logical stream on the transport at basically any point.Even the TCP protocol itself has a notion ofout of band data("urgent, process me first, recipient, please and thank you"), although it is rarely used and may not be correctly implemented in the TCP/IP stack one uses.I myself don't always need additional error correction, because the underlying transports I use (UART, USB, and TCP over IPv4/v6) all have some error correction already built-in.I do not like COBS, because of the lookahead buffer size. I do not know how one could multiplex multiple COBS streams on top of one transport, without having a separate lookahead buffer for each logical stream; that amount of buffering (and buffer bloat) is too much in an embedded environment for me.I also do not need to know the packet size beforehand, as almost always the packet can only be processed when complete. In a very real sense, even text-based command streams like Gcode and HPGL can be considered delimited packets.

author: voltsandjolts
author_rank: Unknown
timestamp: January 25, 2022, 10:29:34 am
content: Quote from: Siwastaja on January 24, 2022, 04:11:41 pmAgreed, lets forget about streaming. We are discussing embedded mcu comms with uart here i.e. small amounts of structured data over an unstructured link. Lets say you have a block of data bytes in a buffer ready to send, the data can include any values and is any non-zero length. The cobs encoding removes all 0x00 bytes from the data so that 0x00 can be used as the framing character, marking the start and end of a cobs frame. The encoding is trivial but neat, in essence 0x00 data bytes are replaced with a cobs byte which is the distance to the next 0x00 data byte. It takes very little processing to encode the data bytes, essentially a loop over all input bytes and write encoded data to another buffer. So, microseconds of processing, which is a near irrelevant delay compared to the uart transmission itself.Quote from: Siwastaja on January 24, 2022, 04:11:41 pmWe are not talking about steaming. There is no delay. Get a block of data, cobs encode it in microseconds and send it with 0x00 start and end characters.The '255 byte delay' confusion stems from the maximum distance between cobs bytes. All zeroes are replaced with a cobs byte which is just the distance to the next cobs byte (i.e. the next 0x00 in data). If the distance is more than 0xFE, then a cobs byte of 0xFF must be inserted (creates some overhead), meaning no zero here, next cobs byte is 255 bytes ahead. When encoding you have to find the position of a zero, then go back to the last zero and write the distance there i.e. the cobs byte. Hence the buffer requirement for a 256 byte (or more) circular buffer forstreamingencoding.

author: voltsandjolts
author_rank: Unknown
timestamp: January 25, 2022, 10:41:16 am
content: Quote from: Nominal Animal on January 24, 2022, 05:15:30 pmLets forget about streams and instead focus on data packets.COBS simply provides a way of framing data packets, the length and content of data is entirely up to you.It provides bandwidth-efficient structure over unstructured serial links i.e. uart / tty / com ports.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 25, 2022, 11:00:48 am
content: Oh, I got it, thanks for the explanation. You can always send just one packet, and need only lookahead within this packet, or if packet is longer than 255 bytes, within that 255 bytes; whichever is smaller, and this is because the end-of-packet delimiter is added, and distance to this zero is what needs to be known during processing.

author: brendan0powers
author_rank: Newbie
timestamp: January 26, 2022, 05:29:24 pm
content: QuoteIf you really care about end-to-end latency being predictable, then COBS has a significant advantage. If you're in a situation where most of the time taken sending a message is transmitting the bytes over your serial link, then the number of bytes you transmit will determine the latency. COBS ensure that latency has a fixed upper bound. You never get into a situation where your message ends up taking twice as long to transmit because you got unlucky.In practice, I don't think that situation isn't all that common, at least in the systems I work with. If consistent latency isn't all that important to you, then byte stuffing works fine.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 26, 2022, 07:32:24 pm
content: Here is my practical use case example, showing the reason behind my "dislike" of COBS. (It's a hobby investigation, not a product; just something I do for fun.)I have a host SBC with a ~ 2Mbit/s UART connection to my microcontroller. I want to use that same UART connection for the serial console (including boot log), bulk data for displaying graphics and other information on a small display, feed back entropy from a hardware random number generator back to the Linux kernel, as well as provide some user interface events (key presses et cetera) back to the kernel.I have my own kernel driver for the UART connection, and my own firmware on the microcontroller.The serial console data from SBC to microcontroller should have maximum priority. This is especially important during bootup, to not slow down the bootup process (and to avoid having a larger buffer at boot time).The bulk data from SBC to microcontroller should have minimum priority; it should only be transmitted when there is nothing more important.The serial console events from the microcontroller back to the SBC should have equal priority with the user interface events reported.Feeding back some entropy from the microcontroller back to the SBC should have minimum priority; it should only be transmitted when there is nothing more important.As you can see, I kinda-sortahave totreat each of these as a stream. Packetizing them is really not feasible, since the bulk data could be huge, and I want the serial console data to 'interrupt' bulk data flow, and have priority.Byte stuffing, with specific byte values reserved as "serial console data follows", "display data follows", "event data follows", and "entropy data follows", means I can interrupt at any position (except between a stuffed "escape" marker and the payload byte). If I assume I end up with 8 reserved byte values, the escape one included, I can use 8+8×8=72 other byte values following the escape marker to encode the reserved byte values and any pair of reserved byte values, limiting the maximum overhead to 50%: when the original data stream has every other byte a reserved value, giving 2 to 3 expansion, or 150% encoded length compared to original.(If you consider the worst case in isolation, three bytes where the first and last are reserved values, that does expand to five bytes, giving 67% overhead. With additional implementation complexity, using 64 additional three-byte escape sequences (ESC PAIR mid -> FIRST mid SECOND) would fix that, though. Assuming 8-bit bytes, there is just enough unique byte values for this scheme to handle up to 10 reserved byte values, but no more.)Note that even the latency requirement/preference here is not related to the physical connection, but a direct result of the preferred priorities on what data gets transmitted first. I do not want to wait for a full packet to be transmitted before switching to the more important, higher priority data; I want to do it as soon as possible.I do believe that similar needs are relatively rare, though. I should have been clearer in my earlier posts: these aremypreferences due to the kind of problems I work with, not universal or directly applicable to others. For example, anyone using USB can use multiple endpoints, one for each sub-stream or channel.This can be useful in embedded environments that use TLS encrypted TCP connections, because the 50% overhead is much preferable to the work involved in TLS handshakes for using multiple TCP connections in parallel; basically, you save on both computation and memory requirements, and can even simplify the TLS+TCP code needed.Now, if I was doing something else, where consistent overhead was more important, and I didn't need to multiplex a single connection for multiple logical connections, I'd like COBS just fine.

author: voltsandjolts
author_rank: Unknown
timestamp: January 27, 2022, 09:51:41 am
content: The latency of access to a packet-multiplexed uart serial link is the time required to send the longest packet. To reduce latency I would limit packet length (split bulk data blocks) and the consistent low overhead of COBS packets would be very beneficial there. You can only reduce packet length so much until you are better off just switching to some stream protocol, such as you did. You are right that a COBS stream is not suited to being interrupted at short notice, so I understand your preference for reserved control chars and escaping in the scenario you describe. You have traded the efficiency of COBS to gain faster bus access.Anyway, we are going off topic a little.I think COBS is a good choice for packetising data in the RPC protocol generator being discussed here.

author: tooki
author_rank: Super Contributor
timestamp: January 27, 2022, 11:55:09 am
content: Quote from: dmendesf on January 12, 2022, 01:13:25 pmTry draw.io. I just used it for a project and am quite happy with it. Open source, both as a web app and local app, runs on everything.

author: brendan0powers
author_rank: Newbie
timestamp: February 02, 2022, 08:08:15 pm
content: QuoteDraw.io is great! thanks for the recommendation!

author: tooki
author_rank: Super Contributor
timestamp: February 03, 2022, 09:53:01 am
content: Quote from: brendan0powers on February 02, 2022, 08:08:15 pmGlad it worked out for you. I know I’ll be using it for more things in the future.