Protocols and formats for sending data between PC and MCU - Page 1
Forum section: electronics / projects

author: Twistx77
author_rank: Regular Contributor
timestamp: July 19, 2018, 06:18:06 pm
content: Hi everyone!For many years I have been working with devices I've made and usually they are connected to a PC either using UART, USB, BT, or WIFI (TCP-UDP). The thing is that I always made my own "protocol" or format for data over TCP, USB, ETC. The thing is that I was wondering if people usually does this or there is some kind of standard protocols or data formats that are usually used, kind of like JSON, MQTT, SWAP.Do you have any suggestions I could look into?Thanks

author: helius
author_rank: Super Contributor
timestamp: July 19, 2018, 06:24:48 pm
content: In the industrial electronics field, there are several high- and low-level protocols used for communication among test equipment (ATE) and manufacturing controls (SCADA). Take a look at SCPI and VISA, for instance.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 19, 2018, 06:55:12 pm
content: Using a standard makes sense when there is a standard which provides the needed functionality for that particular application in an effective manner.Since applications are case-by-case, and universal standards are difficult to impossible to design, you often roll your own - which is exactly why so many standards exist!There is no need to feel any guilt in doing your own design, and it's often best to ignore irrelevant people who demand you follow some - almost always totally irrelevant - trendy standard. Most often, they have no clue what they are talking about since they never get anything done.Lean, simple standards are best. And of course, sometimes it just happens there is an application-specific higher level standard that happens to suit your need directly, but sadly, this is rarely the case, and hence, you end up designing a yet another new standard.Some sane recurring patterns make sense, however. Things like how you provide framing to low levels that provide inadequate framing (such as UART with 8-bit HW framing, inadequately small for most practical messages). In this case, I think it's a sign of good design that these things only take a few dozen lines of code, are portable, and don't require a heavy-weight library or a complex parser. Think about a complete opposite to what XML is, for example.Beware of the "ASCII is easy" trap, often touted on these forums as well. I have seen countless of totally broken designs and complex-as-hell parser needed just because someone thought ASCII provides framing "the easy way". The fact that most terminal equipment, PCs and libraries corrupt and change line feed symbols alone breaks so many implementations that rely on these symbols for framing, in specific order. I often use ASCII anyway, but the parser is almost always at least 200-300 lines on both sides, and I don't want to lie to myself that it's "simple". Sometimes a simple binary format is one tenth of the complexity.

author: Twistx77
author_rank: Regular Contributor
timestamp: July 19, 2018, 07:00:20 pm
content: Quote from: helius on July 19, 2018, 06:24:48 pmThank you I will take a look at those to see if they fit well to my application.

author: Twistx77
author_rank: Regular Contributor
timestamp: July 19, 2018, 07:05:18 pm
content: Quote from: Siwastaja on July 19, 2018, 06:55:12 pmI have always thought exactly as you have explained here. Although since the project I am working on right now is focused towards education and open source, I wanted to see if there was something available and used widely or at least some kind of standard or data format with implemented libraries.I like the binary format since I can avoid using strings but is less flexible.On one hand binary would be more interesting for people who learns and on the other hand ASCII format is usually to understand, at least the messages for someone that have never work with binary data formats.

author: towlerg
author_rank: Contributor
timestamp: July 19, 2018, 09:35:13 pm
content: Personally it's horses for courses. If it makes sense to use an existing standard fine, elsewise make up one that is appropriate for the task in hand.

author: Jeroen3
author_rank: Super Contributor
timestamp: July 19, 2018, 09:49:49 pm
content: Protobuf.https://developers.google.com/protocol-buffers/Quote

author: Twistx77
author_rank: Regular Contributor
timestamp: July 20, 2018, 08:57:44 pm
content: Quote from: Jeroen3 on July 19, 2018, 09:49:49 pmThank you very much for that info, if I don't use it I might take ideas from iot.

author: djacobow
author_rank: Super Contributor
timestamp: July 21, 2018, 03:44:29 am
content: Folks have made some good points about the pain of using text-based protocols. I don't disagree, but if you have solved the problem for how to reliably transfer strings between your host and microcontrollers, let me suggest you just go with JSON. It's flexible and straightforward, and generally easy to debug. You can add elements to your JSON that the target doesn't understand and, as long as the string still fits in memory, the target won't care. This is nice for rolling out new features to new devices, maintaining backward compatibility to old ones. On the host side, there are libraries galore in every language, so that will be trivial.It does get a bit tricky on the controller side, especially if memory is scarce, but there are libraries for that, too. Check out Rapidjson, or JSMN for pretty minimalist JSON parsers. Or, for the ultimate in minimalist, you can try this one:https://github.com/djacobow/djsby your truly. Mine is "special" in that no buffer is required to hold the parsed tree at all. Instead, the code just 'finds' elements in the existing string. This means no memory overhead at all, at the expense of having to re-scan the string for each item you're after. I have not tested my code extensively, but I've used it successfully in a few projects.

author: voltsandjolts
author_rank: Unknown
timestamp: July 21, 2018, 08:18:13 am
content: Nice, I like the idea of using JSON and your library makes it more palatable on the embedded side.

author: towlerg
author_rank: Contributor
timestamp: July 22, 2018, 03:31:23 pm
content: It may well be easier but imagine how sloooow it is. If you don't care about throughput it sounds ok but if you want to minimize uC tx/rx time maybe not so much. I'd be curious but much to lazy, to compile it and see how big it is cause after all one mans minimal may not be anothers.

author: djacobow
author_rank: Super Contributor
timestamp: July 22, 2018, 03:38:09 pm
content: I think you said it earlier: horses for courses. Sometimes speed of parsing is critical, sometimes it isn't.What I posted is just an option.

author: dmills
author_rank: Super Contributor
timestamp: July 22, 2018, 04:59:51 pm
content: I would offer ASN.1 as an internally utterly horrible, but actually quite efficient choice that works surprisingly well.there are some very useful tools that build minimal validating parsers from a textual description of the data types and ranges, and the resulting programs do validate ranges, so if you have an integer with a range of 103 to 761 you can have that enforced by the parser which is nice.https://ttsiodras.github.io/asn1.htmldiscusses one example of the sorts of tools available.Regards, Dan.

author: Twistx77
author_rank: Regular Contributor
timestamp: July 22, 2018, 05:56:13 pm
content: Quote from: djacobow on July 21, 2018, 03:44:29 amSure JSON is a good idea I thought of using it at first. The thing is complexity of the parser and throughput but I don't know yet I will try to evaluate both aproaches.

author: Twistx77
author_rank: Regular Contributor
timestamp: July 22, 2018, 05:57:01 pm
content: Quote from: dmills on July 22, 2018, 04:59:51 pmI will take a look. Thank you.

author: voltsandjolts
author_rank: Unknown
timestamp: July 22, 2018, 06:04:20 pm
content: Unfortunately I don't think there is any free to use ASN1 compiler with C# or Python output, which I tend to use on the PC side.Anybody know of a free to use protocol compiler which spits out C (suitable for microcontrollers) and C#?

author: carljrb
author_rank: Contributor
timestamp: July 22, 2018, 08:44:59 pm
content: I also use C on the MCU side and C# on the PC side (primarily).They're working on adding that stuff to .NET Core:https://github.com/dotnet/corefx/issues/21833. In version 3, it'll finally support desktop applications:https://blogs.msdn.microsoft.com/dotnet/2018/05/07/net-core-3-and-support-for-windows-desktop-applications/.Meanwhile, I'd look at other options. Personally, I very much dislike JSON (it feels out of place in anything but javascript and web pages, and an unnecessary pain to deal with mostly everywhere else). As for what is "better", that totally depends on the application (e.g. test equipment could use USBTMC/SCPI/LXI/HISLIP) and the transport (USB, UART, Ethernet, etc). There's no shortage of existing protocols out there.For quick one-offs (that aren't networked), I make a simple binary message format (instead of parsing/deserializing text of an over-engineered protocol that will often have no real benefits), and send that encoded with COBShttps://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

author: voltsandjolts
author_rank: Unknown
timestamp: July 22, 2018, 08:55:20 pm
content: Yes, COBS is great for transport.There is probably a userbase for a simple protocol compiler to make a lean-mean binary message format with decoders/encoders in C# and C.

author: NiHaoMike
author_rank: Super Contributor
timestamp: July 22, 2018, 11:49:28 pm
content: I have used two approaches a lot regarding data transmission over UART links.One is to denote two byte values as reserved, ideally ones that rarely appear in the data. One value denotes a start of frame and the other is used as an escape character. Data bytes are sent as is except for the reserved values, which are replaced with the escape character plus the value XORed with a constant so that it doesn't conflict. Such a scheme is very efficient with bandwidth, unless some condition causes more than the usual number of reserved values to appear.The other is to convert the data bytes to hex characters, then use standard ASCII for out of band framing. The main advantage is that a standard terminal program can be used to view the traffic, but is very inefficient with respect to bandwidth.Something else to keep in mind is that a UART link might already support some form of out of band communication! Control lines are one form (subject to race conditions), as well as sending break signals or using 9 bit signaling if both ends support it. In some cases, simply waiting can be an acceptable way to get the other end into a known state.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: July 23, 2018, 12:54:10 am
content: Quote from: Siwastaja on July 19, 2018, 06:55:12 pmThe key to the ASCII approach is to design the protocol so that the parser is as simple as possible. Ideally, parser = small finite state machine, so that protocol = regular grammar. Write the protocol and the parser together. If the parser starts to get tricky, change the protocol.The advantage I find with this approach is speed of debugging: if something is going wrong you can simply log all of the traffic, and quickly scan it by eye or a text editor. The task of testing exceptional cases can reduce to: create a text file and "cat" it into a suitable pipe.

author: Jeroen3
author_rank: Super Contributor
timestamp: July 23, 2018, 05:51:56 am
content: Using readable text, because it's easy to debug, is bit of false hope. It won't make it easier in the long run.You're better of not exchanging readability for performance, while also spending some time on an analyzer.The bonus is that you then also have tests, since with a readable protocolyouare the test, and you are not automated.Readability by humans of what goes over the line should not be in the specifications.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 23, 2018, 08:05:48 am
content: Quote from: thermistor-guy on July 23, 2018, 12:54:10 amThis may result in a protocol with false hopes for human-writability, which will utterly fail as soon as someone else tries to communicate with it, finding all those nice little "design simplifications" that prevent it from working using anything else than your specific encoder.Well, when it's an actual human commanding it, we can at least use our fairly complexAI system to automatically adjust timing, sequencing, parsing... But it'll get fairly difficult when we need to write some code to do that. So, we end up looking at your parser code (if available), or documentation, and work it out "manually". In which case, ASCII becomes totally unimportant, and most likely only an extra complication. The only advantage left is human-readability. (Well, you can do worse: for example, use XML, and you won't have either human-writability or human-readability, yet you have hopes for both!)I spent days trying to initialize a LIDAR which has total of 7 bits of configuration and command space, which would fit inside one single octet without any framing issues, but instead it implements an "easy" ASCII interface, with tens of pages of documentation, with all kinds of nasty surprises regarding timing - for example, it just outright ignoressomecommands atsometimes (and doesn't give feedback on success/failure) and replies back unexpected things, so you need a fairly complicated parser in the FSM. The funniest part? After configuration and turn-on command (which could have been one freaking single byte), it switches to a simple binary interface (which just works) for the data, since the actual data would have too much bandwidth for ASCII overhead. So there is no human readability anyway.In simple projects, I tend to use this ASCII format:Commands (PC->MCU): everything must fit in one byte. If not, go for something else. For example, I have used these:c: start chargingd: start dischargings: stopOr in another project:o: open the doorc: close the doors: stop the doorStatus (MCU->PC): Everything is a label=value pair, separated with spacestime=100 curr_limit_setpoint=123 power=234 voltage=4200This is far from perfect, but there is certain simplicity in it; especially the fact that the MCU parser will be simple-and-robust as hell because there is no framing or any state in the parser.And, this is well controllable by both human and simple computer programs; and it's human readable (as long as one line of status fits nicely on the terminal screen line) - I tend to pad values with leading zeroes to force nice alignment. Haven't seen any problems parsing that - all atoi() and scanf() style implementations work with leading zeroes just fine.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: July 23, 2018, 10:35:37 am
content: My two cents:- I for one prefer human readable protocols, always, except when/if ultra high performance is paramount. Interactive terminal sessions are the best debugging tool there is, mission impossible with binary formats.- You want to be able to ping the other side to check it's ready. Sort of like the "ok" reply of the AT commands of the modems.- You want to be able to reset at any time the receiving end to a known state, the old school serial break, ctrl-c, or something else you choose. Very important.- You may want to know that the cmd you just sent was received correctly. Easy peasy way to achieve that is to return in hex a checksum or xor or a crc of the txt of the command. You can simply ignore it in interactive terminal sessions, or check it when it matters.- Or optionally embed the checksum, xor or crc at the end of the cmd line, like this ":xor=7f" so that the receiving end can check it's ok before executing, and abort if it's not: cmd:send missiles:chksum=deadbeef- You may need to send a series of commands to be executed all at once, or none at all. Think about that. If you send: open the tap - wait 1 minute - close the tap and for some reason the close the tap never makes it to the receiving end, you've got a problem. So devise a way to send a group of commands to be executed, not only individual ones.