Architecture neutral data interchange between devices? - Page 1
Forum section: electronics / general

author: e100
author_rank: Frequent Contributor
timestamp: June 03, 2019, 05:40:07 am
content: In the desktop world people use JSON or XML.Is there a more efficient equivalent for embedded devices that doesn't require everything to be sent as text strings?

author: ataradov
author_rank: Super Contributor
timestamp: June 03, 2019, 05:55:37 am
content: Protobufs is one option.But really, don't over-complicate it, just send raw binary data.

author: MosherIV
author_rank: Super Contributor
timestamp: June 03, 2019, 07:07:15 am
content: The biggest problem between Intel and Motorola and now sometime ARM is endianess.There is no way around this.You just have to specify which one to use and make sure bith sender and receiver both send in the same endianness.Eg. CAN protocol says you MUST specify big or little enduan when sending anything larger than a byte.

author: Someone
author_rank: Super Contributor
timestamp: June 03, 2019, 07:17:42 am
content: SCPI has binary blocks built around their well defined formats:https://en.wikipedia.org/wiki/Standard_Commands_for_Programmable_Instrumentsworks fine, but its best to have some sort of more open computer in the middle to translate all the messages.

author: Siwastaja
author_rank: Super Contributor
timestamp: June 03, 2019, 07:53:24 am
content: Yes, just use binary data. It's no less "architecture neutral" than any other communication. Any communication requires a conversion layer to be "architecture neutral", and for traditional binary data, this layer is the simplest, BY FAR.The reason for XML, JSON and so on to exist has actually very little to do with being architecture neutral: it's their ability to contain variable number and variable types of fields, specify templates/schemas that formally describe the fields, and the apparent easiness of writing high-level software around this concept of "anything might come in".If you don't need thisfeaturein its full-blown form, but have a limited number of message types and datatypes, known compile-time, simple binary data with simple header files tends to be the easiest - both to write and read -, by far. This is true, for example, for all low-level network protocols (think about TCP).Packed structs in C provide a way to access the data. Time-tested way to do it. Will be never out of fashion.Endianness is not a problem. You have to just choose either one for the communication. Of course, choose little endian, and you are done. If you later come up with a big endian device you need to communicate with - very unlikely - then finish your abstraction layer. It is about 10 lines of code (endianness swapping macros defined with #ifdef ... #else ... #endif), so you could (and arguably should) write it to begin with, but I won't usually bother.Be very careful about ASCII-based things that are delimited by line-feeds. People recommend them eagerly. Then I'm the one who needs to deal with all the broken devices. You can do it right, but everybody else tends to do it wrong, and they have a long history of not working. Many SCPI devices are like this. Was it \r\n or \n\r, or \r, or \n? Why they forgot to specify their command delimiter? Why does this shit error out randomly? What sequences do I need to give it to clear out of error states?It makes little sense to construct your own complex "parsed" protocol. If you need the features of JSON or protobuf, just use them. Chances are, you won't need. Then use raw binary, and most of your "parsers" just go away.

author: paulca
author_rank: Super Contributor
timestamp: June 03, 2019, 09:58:56 am
content: The data types in a C struct "should" be standardised with regards length and the packing "should" maintain alignment, but that's at least two "shoulds". I have encountered several architectural issues in the past.I'm not sure what happens if you pass a 64bit long into an 8 bit MC though. You have to be careful with reference and pointer types as well, you can't just memcpy the struct to the output buffer if it contains addresses for example.In stock exchanges, which re highly performance and latency sensitive we used both binary protocols using packed and unioned C structs as well as text based FIX messaging protocol. The advantage of the later is it doesn't need the structs to change to add or modify a custom field. So the customers don't need to update their serializer code.For a recent personal project (home automation data network) I use high level JSON based strings but I have no performance demands where that might be an issue and the data is going over TCP/UDP anyway. As mentioned JSON provides a concise, expandable mechanism to provide arbitrary or loosely defined data which was important in my use case. In my case the data is not defined in advance any device can send data and receive all data back as a response, only some basic structures like: id, value, type, unit, updateTimestamp. It's a mixture of real PCs, RaspberryPIs, arduino's and ESP8266's.

author: Fire Doger
author_rank: Regular Contributor
timestamp: June 03, 2019, 10:03:22 am
content: Packed structs with e few bytes of overhead like tcp and maybe reconstruction data. rock solid for me!

author: tggzzz
author_rank: Super Contributor
timestamp: June 03, 2019, 10:52:12 am
content: You could alwaysconsiderASN.1, which has been around a long time.

author: Siwastaja
author_rank: Super Contributor
timestamp: June 03, 2019, 11:14:04 am
content: Quote from: paulca on June 03, 2019, 09:58:56 amThis is why stdint.h (and inttypes.h which includes it) exist.QuoteIf you need to change theserializerpart when the message structs change, something's gone wrong.In the end, when writing theactual application codeon the top of whatever comms, the data has to be packaged in some kind of structure, anyway. The idea is to use this struct directly, so that the low-level comms doesn't need to understand what the struct fields actually mean. I mean, you can't avoid defining your messagessomewhere. It's most important they are written only once so that there is no manual syncing of code.

author: tszaboo
author_rank: Super Contributor
timestamp: June 03, 2019, 11:28:32 am
content: You have BSON, which is Binary JSON.Pyhton has Pickle, which has options for data compression. You can pickle a picture object with a string a library and a file and have it compressed.In C or C++ you are on your own, or you should look for a library to do it. Honestly, I would just stick to a standard.

author: amyk
author_rank: Super Contributor
timestamp: June 03, 2019, 11:30:26 am
content: Quote from: tggzzz on June 03, 2019, 10:52:12 amEspecially the PER and DER variants, which are still binary and thus efficient and relatively easy to parse. You can always use a similar encoding system even if you don't use the full spec.

author: NorthGuy
author_rank: Super Contributor
timestamp: June 03, 2019, 12:25:38 pm
content: Embedded devices rarely use data exchange schemes such as XML or JSON (or DER for that matter) unless there's a need to communicate with external world, such as implementing a Web server which regular browsers can access. Embedded networks and interchanges are typically much simpler.CAN is very common for inter-MCU peer-to-peer networking. There are several CAN networks in every car nowadays. The hardware part ensures transmission of small packets. The packets have ids. This is enough to build communications in the situation where you don't know what the other devices may connect. There are standard protocols such as J1939, or you can invent your own.

author: dmills
author_rank: Super Contributor
timestamp: June 03, 2019, 12:55:10 pm
content: I second the ASN.1 recommendation, only problem is the standard is written in the 'telecom' dialect of standardsese.There are some very good tools out there for taking a textual description of the data and producing validating parsers and generators in C, well worth the cost of admission as the same tools can produce database tables and detailed documentation.See for examplehttps://www.thanassis.space/asn1.htmlfor one very cool implementation.Or just go packed binary, but define the byte order and watch the fact that C compilers **will** place padding within structures to meet alignment requirements (Also in C, bitfields are horribly badly defined making them useless for this sort of thing).Regards, Dan.

author: e100
author_rank: Frequent Contributor
timestamp: June 03, 2019, 02:25:07 pm
content: Quote from: MosherIV on June 03, 2019, 07:07:15 amDo things get complicated when one of the parties is using a programming language that doesn't have a full set of equivalent c data types and needs to talk to a regular microcontroller programmed in c? For example, Java doesn't have unsigned ints as part of the core language.

author: paulca
author_rank: Super Contributor
timestamp: June 03, 2019, 04:14:31 pm
content: Quote from: Siwastaja on June 03, 2019, 11:14:04 amWhatever terminology you use we had to release a new version of the header files and the client had to recompile and re-release their application.With the FIX protocol you can add/remove fields as you see fit. Of course the application may need updating to use the new fields, but hey.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 03, 2019, 04:54:15 pm
content: Quote from: paulca on June 03, 2019, 09:58:56 amThose are both patently false and have never been even slightly true. So if by should you mean "I wish it did", then OK, but if you are saying there was ever an actual expectation that it worked that way it isn't true.The only thing C promises is that sequential fields in a struct have increasing address. It doesn't guarantee anything about word length or packing. That is specified by specific architecture and compiler ABIs.Using c99 fixed with types like uint32_t el al. will solve the width problem but it won't solve padding. Granted: almost all modern architectures will have consistent packing for types up to 32 bit but it isn't a guarantee and it is not true for 64 bit types (integer or floating point) which will sometimes be 4 byte aligned and sometimes 8 byte.

author: MosherIV
author_rank: Super Contributor
timestamp: June 03, 2019, 07:49:16 pm
content: QuoteDepends.If the unsigned is equivilant to the signed, then it us ok.Eg 126 is the same in uint8 and int8Once you get to 128, the int8 has a problem.In general though, data width should have been sized large enough to cope with the range needed.So, you should be ok.I agree with ejeffry last post.Things should work BUT there is no guarentee.

author: NorthGuy
author_rank: Super Contributor
timestamp: June 03, 2019, 08:18:27 pm
content: Quote from: e100 on June 03, 2019, 02:25:07 pmThere are thousands of different file formats. Programs written in different languages and running on different platforms can read them without problems.There are hundreds of different communication protocols. Programs written in different languages can access them.There's absolutely nothing complicated about this.

author: LapTop006
author_rank: Unknown
timestamp: June 04, 2019, 01:47:30 pm
content: Quote from: ataradov on June 03, 2019, 05:55:37 amAs much as I love protocol buffers (well, proto2 at least, proto3 less so) the generated code is a touch heavy weight. This may or may not be an issue depending on your target.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 04, 2019, 04:08:36 pm
content: Quote from: e100 on June 03, 2019, 02:25:07 pmWell, if you use a "real" deserializer it won't be a problem. It will either decode to an appropriate sized type for the data or raise an error if it can't or won't do so. For instance, a java implementation would probably have 8 or 16 bit integers unpack into a 32 bit variable, so they would have plenty of range. An unsigned 32 bit integer >= 2^31 could either unpack to a 64 bit integer or fail. Likewise, on the serialization end, if your interface description says "unsigned integer", a good serializer should just raise an error if you provide it a negative value or otherwise out of range for the wire format.If you are just doing bit casting using the closest available native type, you can run into problems. That is why serialization libraries exist: to solve these problems once so you don't have to.

author: fchk
author_rank: Frequent Contributor
timestamp: June 04, 2019, 07:44:45 pm
content: The eXternal Data Representation XDR aka RFC 1014, which is used to address exactly this problem in Sun/ONC RPC, is about 30 years old. There is no need to reinvent the wheel.fchk

author: T3sl4co1l
author_rank: Super Contributor
timestamp: June 04, 2019, 08:51:52 pm
content: Modbus comes to mind. Basically a lightweight layer providing access in an addressable-registers paradigm, like how you access memory-mapped I/O in a microcomputer system.Modbus is a higher level protocol that runs on any basic communication channel. Async. serial is typical (via RS-232, 485, etc.).Tim

author: e100
author_rank: Frequent Contributor
timestamp: June 05, 2019, 06:35:06 am
content: Quote from: T3sl4co1l on June 04, 2019, 08:51:52 pmIs there a more up to date version of Modbus that can send a 32 or 64 bit number in a single message instead of leaving it to the application layer to chop it up into a series of 16 bit messages?

author: gmb42
author_rank: Frequent Contributor
timestamp: June 05, 2019, 11:47:28 am
content: Quote from: e100 on June 05, 2019, 06:35:06 am