Framing algorithm for fixed packet sizes - Page 1
Forum section: electronics / projects

author: paeion
author_rank: Contributor
timestamp: March 18, 2023, 02:55:03 am
content: Hi all,I am working on an application that visualises sensor data in real time. An ESP32 collects the sensor data and sends 18-byte packets over bluetooth serial to the PC, using python and the Qt framework for the GUI. The ESP32 bluetooth latency is fairly inconsistent so I want to limit the packet overhead to achieve the required throughput. I am currently transmitting the data at 500Hz but hoping to increase that towards 1kHz. Using 2-bytes as start-of-frame and another 2 for end-of-frame, I have a crude way of synchronising and delimiting my packets but I still get framing errors. I am fairly certain ASCII-encoding is too much overhead for the ESP32.I have looked into COBS encoding but the 255 byte-lookahead is not ideal and I am not overly confident in implementing the buffer for that (C++). SLIP encoding also looks interesting but I am wondering if the non-constant packet sizes from byte stuffing will cause more troubles for the bluetooth serial communication than a constant-overhead communication protocol. I can live without a checksum/ CRC for what I am trying to do.What framing algorithm/ communication protocol would you recommend for transmitting fixed-length 18-byte packets at the desired 1000Hz?

author: ejeffrey
author_rank: Super Contributor
timestamp: March 18, 2023, 04:46:15 am
content: COBS only needs look ahead for the lesser of 254 bytes or 1 packet. If your messages are only 18 bytes, that is all the look ahead you need. That also means that there is always exactly one padding byte per message at the very beginning so you can easily do the null removal transformation in place.If you have longer messages you can also modify COBS to trade overhead for lookahead. Standard COBS adds a padding byte for every 254 consecutive non-zero bytes. If you instead are willing to add a padding byte for every 32 consecutive non-zero bytes, then you only need 32 byte lookahead.Variable packet sizes also shouldn't be a problem for a bluetooth serial interface, so a simple escaping protocol like the SLIP encoding should also work fine but the worst case overhead will be a lot higher. How much you care about that I don't know.

author: paeion
author_rank: Contributor
timestamp: March 19, 2023, 07:26:47 am
content: Thanks for your reply ejeffrey, I think you've cleared up my confusions about COBS encoding!It is probably worth the time implementing both encoding methods and testing out which one fits my requirements best.What exactly does "null removal transformation in place" mean if I were to COBS encode and send just 18-byte packages?

author: voltsandjolts
author_rank: Unknown
timestamp: March 19, 2023, 09:15:24 am
content: Quote from: paeion on March 19, 2023, 07:26:47 amIt means doing the COBS encoding in the buffer you already have, without copying to a new buffer. But for such small buffers it wouldn't really matter much which way you you choose to do it.Here is myCOBS threadwith example code for MCU in C, and PC code in Python.

author: paeion
author_rank: Contributor
timestamp: March 20, 2023, 10:31:00 pm
content: Very useful resources, thank you for the thread link!I have resolved my packet framing issues with COBS encoding now. I get a consistent 2-byte overhead as my data always has \x00 present and it has improved the serial communication latency significantly.I will have to experiment further with increasing the lookahead to reduce the packet framing overhead but for the moment its a good starting point