structure info and guidance - Page 1
Forum section: electronics / general

author: nikoso40@yahoo.com
author_rank: Newbie
timestamp: August 12, 2025, 12:52:22 pm
content: Hi all.i have made a project with few sensors and a pic16f18346. The sensors data are converting to strings and results via UART to the pc terminal. Now i want to send these data instead pc to another pi18f46k22 with the same protocol and nightmare starts. The only way i know is to send bytes so i create a tx buffer. What i have done until now is:1. i create a typedef struct to keep the data. The numbers the is row from the read functions of every sensor.2. then a void sending all data to a string3. send the stringCode:char tx_buf;typedef struct{unsigned short _light;uint16_t _co2;uint16_t _o3;uint16_t _co;uint16_t _temp;uint16_t _press;uint16_t _alt;uint16_t _dust;char _status;}results;results myResults = {0, 0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '0'}; // char 0 as status is to inform the receiver of transmition end./* this function assign the data to the string */void data_in_buffer(const results *data, uint8_t *_buffer){_buffer = data -> _light;_buffer = data ->_co2;_buffer = (uint8_t)(data ->_o3 >>8);_buffer = (uint8_t)(data ->_o3);_buffer = (uint8_t)(data ->_co >>8);_buffer = (uint8_t)(data ->_co);_buffer = (uint8_t)(data ->_temp >>8);_buffer = (uint8_t)(data ->_temp);_buffer = (uint8_t)(data ->_press >>8);_buffer = (uint8_t)(data ->_press);_buffer = (uint8_t)(data ->_alt >>8);_buffer = (uint8_t)(data ->_alt);_buffer = (uint8_t)(data ->_dust >>8);_buffer = (uint8_t)(data ->_dust);_buffer = (uint8_t)(data ->_status);}/* this function called from main once an hour to send data */void tx_SensorsData(const results *data){uint8_t _data[sizeof(results)];data_in_buffer(data, _data);for(size_t i=0; i<sizeof(results); i++){UART_send_char(_data);}}main(){if(flag){flag = 0;tx_SensorsData(const results *data);}}This ofcourse not working. Can someone help on that. Please keep patient because i am self learning.

author: brucehoult
author_rank: Super Contributor
timestamp: August 12, 2025, 09:36:24 pm
content: Why "of course"? There's not much wrong there.I only had to fix a couple of things to run it on a normal PC:- declare main as type "void" as required by C- add a global "int flag = 1"- changetx_SensorsData(const results *data);totx_SensorsData(&myResults);- add my ownUART_send_char()Then it compiled and ran.Just a couple more comments:- there are 19 bytes inresults, so you're allocating 19 bytes for_data, but you're only filling in 15 bytes in_data, but you're printing 19 bytes from_data, so the last four values are junk.- you're trying to assign floating point values (0.0) to integer variables. So they're getting set to the closest integer to 0.0 (which is 0, of course).

author: westfw
author_rank: Super Contributor
timestamp: August 13, 2025, 02:15:28 am
content: QuoteI don't see any conversion of the data into "string" form...

author: John Coloccia
author_rank: Super Contributor
timestamp: August 13, 2025, 03:33:48 am
content: style points (just my opinion)- get rid of the explicit numbering of your buffer. A simple i++ makes it so I only have to check the order of your structure, not the indices too.- status doesn't appear to have any purpose other than for the comm protocol. It doesn't belong in the structure, unless there's some other reason, or it's ever anything but '0'.- pack your structures- you're playing around with the endianess on everything except light and co2. You're just truncating those.- like westfw said, there is no ascii data here, other than by bad luckThere's certainly more elegant ways of doing this, but if that's all you're really doing, doesn't seem horrible. But it feels like it's probably not doing what you're intending.Since you're just sending stuff out once an hour, why not make it simple?- make everything ints, or uints if that works- write a function that takes a uint and puts it out in whatever format you want- pass the structure and size, and just loop over it as uintsNo real reason to have a separate function to move bits around, and treat everything separately. Just make it all the same.Code:enum {D0,D1,NUM_DATA};uint16_t data;int main(){data = 10;data = 20;// pretend this is your function for spitting out uints...for(int i = 0; i < NUM_DATA; i++) {printf("%u ", data);}printf("0\n");return 0;}One day, C might support flexible array members in unions, and then there would be a really slick solution. But I don't think that day will ever come.

author: nikoso40@yahoo.com
author_rank: Newbie
timestamp: August 13, 2025, 09:39:40 am
content: Hi. I must give some more information. The primary project is without all these are written in this topic. So the primary sending all data to pc. For that i take out the measurements from the sensors and using private funtions convert them to string and send the strings one by one to pc. Now instead of doing that i want to send them to another pic which has same architecture. As i am self learner of c i thought that i can just push the strings straight to the other pic. After some search no one is doing it that way so i figure out this method with struct. This is all story.

author: nikoso40@yahoo.com
author_rank: Newbie
timestamp: August 13, 2025, 09:41:22 am
content: i will tested and give feedback. Unfortunatelly i am working three days on the row night shift so it will take some time.

author: nikoso40@yahoo.com
author_rank: Newbie
timestamp: August 13, 2025, 09:44:18 am
content: Hi. Yes i understand your snippet and i will try it. I will come back with update.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 13, 2025, 11:32:32 am
content: Code:#define NUM_ELEM(_tbl_) (sizeof(_tbl_)==0?0:(sizeof(_tbl_)/sizeof(_tbl_)))typedef union{struct __attribute__((packed)){uint16_t thing;int32_t another_thing;float fancy_thing;};uint8_t u8[2+4+4];} my_type_t;void send_func(my_type_t* thing_to_send){for(int i = 0; i < NUM_ELEM(thing_to_send->u8); i++)uart_write(thing_to_send->u8);}Expect some backlash from people who don't trust using the "packed" compiler extension as a guarantee of correct unaligned access instructions; I stopped writing bitmasks and manually populating fields for stuff like this years ago and could not be happier. Works very well for me, with reduced amount of code to write and maintain.BTW, you might want to design some sort of packet delimiting thing if you are sending these structs over uart. A long enough time-delay between the packetsmightdo it, but even in that case adding a known magic "start" value in the beginning, and CRC code at the end, is a very good idea.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 13, 2025, 01:59:49 pm
content: Why not just make them all ints, and toss them in an array? Choose your units so that you don't need floats. Like instead of "12.054 meters", make it "12054mm".A frequent thing we do, especially with communication protocols, is just make everything the same. Unless there's some performance problem, and there almost never is because we're communicating so slowly anyway, it's OK to "waste" bytes. Save the brainpower for when you actually need to squeeze performance out of it.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 13, 2025, 02:22:18 pm
content: Quote from: Siwastaja on August 13, 2025, 11:32:32 amJust learn your platform and its limitations. There's no way around it. If unaligned access may be emitted by the compiler and not supported by the target, just either make sure that no struct field is unaligned (by reordering fields and/or padding) or make sure to acess them appropriately. That's not rocket science.But if you know that unaligned access is not a problem in your case, don't bother. Just document it above your struct definitions so that if the code ever needs to be ported to another platform, that would document what to be careful about.You can also just isolate this kind of "low-level" code and consider (document!) that's it's not portable.I'm all for making portable as much of the code as possible, but don't waste time trying to make the few low-level pieces (as long as properly isolated and identified) portable. In most cases, you'll find out that your code will never get ported to another target with different requirements, and for the rare cases where it is so, just porting the low-level pieces will be more efficient.The key is to make things modular enough so that porting the code base won't look like stirring tomato sauce in a spaghetti bowl.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 13, 2025, 06:21:21 pm
content: Quote from: SiliconWizard on August 13, 2025, 02:22:18 pmRemember the last time we tried to find any even remotely recent-ish version of GCC or clang onanyplatform (not supporting unaligned accesses) whichwouldcreate unaligned access with a packed struct - and we couldn't find any! It just worked on every platform. I'm still willing to bet that I'm not the only one whose code relies on this property - changing it would break a lot of code. Really the only way to break is to take a pointer of individual member of packed struct and dereference it, losing the packed attribute - luckily GCC has emitted a warning from doing this for a long time.Of course, exactly because it generates individual byte accesses and bitshifting internally, there is a significant performance (and code size) penalty to using the packed attribute - it basically does the same under the hood that you would manually do with shifting and masking, more or less. So it's worth identifying which is more important - communication time or CPU time. Communication through UART is thousands of times slower, so on wall clock it makes sense to squeeze every bit to absolute minimum storage, but you can then make the mistake of using the same struct - which is very appealing as that's the whole point of avoiding maintaining extra code - in some very timing critical algorithm. Or using it all over the codebase in gazillion of ways always forcing the compiler to generate excess bloat which slowly adds up to not insignificant code size. In such cases, separate datatypes for communication and internal storage are needed - and that requires writing pack/unpack functions that need to do the member assignments, so a single packed struct isn't so attractive anymore; you can as well write the bitshifts/masks into the byte buffer.Me - (1) I'm somewhat lazy, (2) I prefer automation (language doing what I want, instead of manual typing or copy-pasta), (3) I prefer data type driven design when possible, (4) to me reality with compilers suffices, compiler extensions are fine, and I'm fine with practical experience of how they work, (5) I want to minimize the amount of code to write, read and maintain, (6) I avoid premature optimization, so I use packed structs whenever communications between different systems is involved - including different architectures with different alignment rules - by default, and only switch to another internal storage format when there are special reasons for it.

author: nctnico
author_rank: Super Contributor
timestamp: August 13, 2025, 07:40:21 pm
content: But you have to consider why every modern day data communication protocol and data exchange is text based (human readable). SCPI, HTTP, JSON, etc are good examples. When I need to implement communication between different devices, I default to using a text based communication protocol. Another issue to consider is checking values. Just accepting any data and using it as-is can lead to problems.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 13, 2025, 08:29:31 pm
content: We mostly do text based too, CAN being the exception. Combination of positional fields and name/value pairs. Pairs are especially powerful. You can update interfaces, and if the user doesn't know about the new options yet, oh well. He just didn't use them and we continue with the old, defaullt behavior.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 13, 2025, 11:47:30 pm
content: Quote from: nctnico on August 13, 2025, 07:40:21 pmHTTP is especially modern. Just 35 years old or so. Even HTTP2, which BTW is no longer text based, is probably 10 years old give or take.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 14, 2025, 06:29:25 am
content: Yeah, imagine all those modern text-based data exchange formats like JPEG, MPEG-4, H.265 and MP3.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 14, 2025, 07:00:46 am
content: Text based formats require care to design, too. JSON is one of the best, with a clear syntax and structure and obvious delimiters, and not too much bloat, but SCPI and HTTP specifically fell into the trap of significant whitespace, like many other 1970's - 1980's protocols did, and with SCPI specifically I have spent too much times debugging poorly functioning devices because of disagreeing line feeds - four options exists; it ispossibleto design a system which is compatible with all four, but for example HP/Agilent is unable to do that, so what can we doIf interfacing with a human being is to be expected - or valuable for debugging - and amount of data is low, then of course choose a text based protocol. If you roll your own, I recommend using explicit delimiter characters like ';'. Upon receiving ;, process the buffer and clear it.If you need checksumming, which is a very good idea for data integrity, then text based format is PITA for everyone. Human being cannot use it because calculating those checksums while writing commands is PITA. So you need to write a specific tool which generates and parses the packets, calculating checksums - which is a great idea, never underestimate the value of creating simple set of custom tools - but that means the format could as well be in binary after all.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 14, 2025, 10:17:37 am
content: Quote from: Siwastaja on August 14, 2025, 07:00:46 amJust use a different end of message delimiter for messages that don't include a checksum, for debug purposes.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 14, 2025, 01:08:30 pm
content: Quote from: Siwastaja on August 13, 2025, 06:21:21 pmYes, of course. Now I definitely remember a couple cases that I encountered: the 16-bit PIC family with XC16 (such as PIC24), for which unaligned struct members would definitely not work. I don't know if it's "solved" by now with newer XC16 - maybe, maybe not. But it's pretty easy to test. The other case was with a aarch64 CPU with more recent GCC, but in this case, it was actually a GCC bug that I reported and it got fixed (and I ran into it in code that was running before the MMU was enabled - keep in mind that on aarch64, without the MMU enabled, unaligned accesses are not supported by the CPU itself.But in most cases, especially if people stick to ARM Cortex-M MCUs, RISC-V, or pretty much any desktop CPU, packed structures just work as intended. And as I said, they've been available almost forever in C compilers with the #pragma pack(1) directive. I've used it, I still use packed structs, a lot of people do.My point was: know your tools and do what's known to work with these. Even if packed structs work in a lot of cases as intended, you still need to know what could occasionally go wrong exactly, what would cause it and on which targets. That to avoid any cargo cult approach either way.I have not used any "text-based" approach for protocols in ages and would certainly never. These are utterly inefficient and prone to parsing errors. But again the factor that really irritates me is the cargo cult thing.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 14, 2025, 01:25:40 pm
content: Quote from: SiliconWizard on August 14, 2025, 01:08:30 pmIt is interesting how different perceptions can be. For nctnico, text-based protocols are easy to produce, parse and debug; safer so to speak. I prefer binary protocols not only for efficiency, but also because I find them easy and safe!

author: gamalot
author_rank: Super Contributor
timestamp: August 14, 2025, 01:34:41 pm
content: One suggestion, don't mix uint16_t with short or char, use only those integer types defined in stdint.h, e.g. use int8_t instead of char.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 14, 2025, 11:07:45 pm
content: Quote from: Siwastaja on August 13, 2025, 11:32:32 amI only suggested packing if you're going to loop over it as uint_16, for example.But in this case, it's seems so simple that I'd just have an array of ints (or whatever), and forget about putting it into a structure.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 14, 2025, 11:12:06 pm
content: Quote from: Siwastaja on August 14, 2025, 01:25:40 pmLike most things, I think it depends on context and how you intend to use the thing. I can think of many pros and cons of doing it both ways. So there's just no good right answer, any more than there's a good answer to "what kind of vehicle should you drive?" I dunno, depends what you're doing and what your goals are.

author: nctnico
author_rank: Super Contributor
timestamp: August 15, 2025, 09:28:42 am
content: Quote from: John Coloccia on August 14, 2025, 11:12:06 pmTrue. For some reason implimenting (and debugging) communication protocols is something I have been doing over and over. At some point I even made it to having the job title 'CTI protocol specialist'. That job title means writing protcol implementations for interfacing telecom switches to computers. I've seen protocols in all shapes and sizes (serial port, network, text, binary, DCOM). From that experience, I know that the text based ones are the easiest to implement. A big downside of binary protocols is that adding fields and remaining backwards compatible is difficult. So in some cases I had to resort to fingerprinting in order to figure out which protocol version I was dealing with. And not just in one occasion. In several cases I replaced binary protocols with text based ones just to make life easier in terms of flexibility / expanding the system. Especially when the communication needs to run across several layers or is spread over multiple devices. Typically the systems I design run a command line interface for debugging & diagnostics. So the text based interface is already there. The communication interface (which could wrap the text in packets with CRC and so on) also uses the command line interface to control a device. So this gives the opportunity to test commands from the local interface as if they where send from the controller unit to see if they have the desired effect. So if there is a problem, it is easy to isolate the problem without needing extra tooling or lots of debugging. I can tell a field engineer to connect a serial port to a device and produce logging while executing commands using a terminal emulator. So in my case it is not just about the protocol but also about the diagnostic and debugging ecosystem around it which needs next to nothing in terms of developing any special tooling.

author: xvr
author_rank: Frequent Contributor
timestamp: August 15, 2025, 12:45:57 pm
content: QuoteMay be Protobuf can be used in both cases? It is binary 'on wire', but can be automatically converted to/from text. Also it support extension (to some extent).

author: Siwastaja
author_rank: Super Contributor
timestamp: August 15, 2025, 01:30:27 pm
content: Extensibility is not an inherent feature of binary vs text protocol at all. It's just that a text based protocol isalwayscomplicated enough to generate and parse, such that when writing those parsers (e.g., key and value extraction), some extensibility usually comes as a side effect. Binary protocol allows super-simplistic design (as shown by OP or myself) but the cost of that simplicity is lack of extensibility.I have managed extending simple binary protocols, while keeping things simple, with the following principles:* In whatever headers you have, includelengthof the message. This allows skipping over unsupported message types.* Include record type identifier. Can be a simple number; for really large designs maybe 16 bits, but usually 8 suffices.* When extending existing record type, add new stuff at the end. When receiving, treat missing data as "sane defaults", and ignore extra data.* When changes are so large that such extension at end is not feasible, just create a new record type. You can of course keep handling both old format and new format messages, or to deprecate the old one when possible.It's not like a text based protocol magically solves compatibility. If it's key=value system, then all it does is allow random ordering of data, and widening of numbers (but you can't know what number ranges the receiver can accept anyway!). You can do that with binary as well, just add records with type identifier, length and data, then arbitrary positioning is possible. Parser looks quite similar to that of text-based system, in fact.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 15, 2025, 02:15:50 pm
content: IMHO, the single advantage of text-based protocols is the ability to view them using proverbial "terminal" software. If you want to observe other protocols, you need something different, such as USB packet viewer for USB, CAN analyzer for CAN, or your own software for custom protocols. Not to mention that observing your protocol in a binary form may be quite sufficient.For the ability to use the "terminal" you pay with 2 things - more work for encoding/decoding, and more traffic on the wire.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 15, 2025, 03:11:29 pm
content: Quote from: NorthGuy on August 15, 2025, 02:15:50 pmHyperterminal or similar only gets you so far. Somehow, I always end up adding more and more useful features to my custom tooling; in such case, whether the actual protocol ends up being text or binary becomes only a small detail, since a lot of generation, parsing, transformations are going on anyway - so binary is fine anyway and gives the performance / code size / transfer time benefits.Example of how my internal tools end up being after being in daily use for 2 years, mostly used by me so being a weird combination of sane design and some very ugly but quick-to-write hacks. In this case the protocols are binary and the tool allows arbitrary construction and parsing of binaries (not even byte alignment needed).Code:$ commanderUsage:commander [server_id/]<devid>/<port_id>/<instance>  (or [server_id/]<gw_devid>/gw/<dest_devid>/<port_id>/<instance>-v increase verbosity level (e.g. -vvv)-h <hostname> broker IP address or hostname-p <port> broker TCP/IP port-t <cafile|capath/>,<certfile>,<keyfile>Enable TLS; use cafile, certfile, keyfilecafile ending in / is treated as capath-R Reset and start fresh commands; but keep broker connection openAfter -R, everything except -v, -h, -p, -t, is cleared and re-evaluated, including devid/port/instanceUseful for doing many things quickly without the broker authentication spam-O Send CONFMIT (not sent if other operations, usually -c, specified and they fail)-T <id> choose transaction id; by default incremented by tmpfile, now 247-y don't check transaction id (quicker access without extra poll message)-m simulate: do not actually send the command-M modbus mode: swap endianness of 32-bit values and swap string char order-L csv print mode - only parsed values-LL csv print mode with fixed size fields - only parsed values-LLL same as -LLL but fewer decimals and narrower fields-B Print unicode bars-e print timestamp at beginning of the line (useful with -L or -LL)-E <modulo-seconds> (only with -F logfile): parse exactly one record between modulo-secondse.g. -E 60: print the first record of every minute (print nothing ifno messages durin the minute)Run rm commander_tmp_starttime first; then run the first logfile; then the restcalled with the same -E value will be synced to the first.-Z completely omit print of -z parse units (no empty field printed)Outgoing payload data format descriptions:-d Send /do message: start description of the /do messageOR-c Send /setconf message: start description of the /conf message-C<byteoffset> Do read-modify-write getconf-setconf starting at byte offsetTHEN-xN <value> Add N-bit value interpreted from hexadecimal input-uN <value> Add N-bit value interpreted from unsigned decimal input-iN <value> Add N-bit value interpreted from signed decimal input-bN <value> Add N-bit bitfield interpreted from string of 0 and 1; LSbs first-sN <string> Add C string padded with zeroes to N-BYTE length. N IS IN BYTES, NOT BITS-fN <filename> Add a file; contents of file are sent. N IS IN BYTES, NOT BITSIncoming payload data format descriptions:-n Poll info message and parse it; start description of info message-N Same, but only receive the next info message whenever it comes, don't poll for it-Q Same, but keep receiving and don't quit-P <string> Instead of polling/receiving messages over mqtt, parse this hexadecimal string-F <logfile|stdin> Instead of polling/receiving messages over mqtt, parse logfile (stdin treated as special name)compatible logfile format: mosquitto_sub -F %I\ %t\ %l\ %xOR-g Send getconf and parse the conf; start description of conf message:THEN-xN Add N-bit value shown as hexadecimal output-uN Add N-bit value shown unsigned decimal output-iN Add N-bit value shown as signed decimal output-bN Add N-bit bitfield shown as string of 0 and 1; LSbs first-sN Add N-BYTE string, shown as text. N IS IN BYTES, NOT BITS-fN <filename> Add a file; received contents are written to the file. N IS IN BYTES, NOT BITS-f0 <filename> Write the received conf to a file, to the length of received conf.-zN Ignore N bits, do not print-l <string> print explanatory string at this pointoptional server_id/ (single letter + /) before devid (defaults to P for prod) allows loading broker address, port and cert filesbased on the following environment variables:COMMANDER_BROKER_HOST_<server_id>COMMANDER_BROKER_PORT_<server_id>COMMANDER_BROKER_CA_<server_id>COMMANDER_BROKER_CRT_<server_id>COMMANDER_BROKER_KEY_<server_id>COMMANDER_BROKER_FCR_CA_<server_id> (for ports 3??)COMMANDER_BROKER_FCR_CRT_<server_id> (for ports 3??)COMMANDER_BROKER_FCR_KEY_<server_id> (for ports 3??)example: commander abcd1234/3/2 -b1 1equivalent: commander abcd1234/3/2 -b8 10000000equivalent: commander abcd1234/3/2 -b1 1 -b7 0equivalent: commander abcd1234/3/2 -u8 1turn relay 2 (3th relay) onWith -i and -u, one can add a floating point multiplier (and offset) used for parsing: e.g. -i16*4.2+5.0: parsed value multiplied by 4.2, then 5.0 added.Another hack: multiplier of -9 causes the number to be interpreted as ministamp and printed as local timeAnother hack: multiplier of -99 causes the number to be interpreted as unix timestamp and printed as local timewith -xN, -uN, -iN, -bN, if N is 0, previously parsed value will be reused and re-interpreted (e.g., with -i instead of -u, or different multiplier)FCR-related magics:-u32*-999 Interpret as unix timestamp and print as UTC time with FCR-log-compliant formatting; also store value internally, see below:-z999 At this point, increment last parsed timestamp by 0.1 seconds and print the incremented timestamp-z9999 At this point, increment last parsed timestamp by 1 second and print the incremented timestamp-iN*<mult>@ Saturate values (after applying multiplier) smaller than 0 to 0

author: xvr
author_rank: Frequent Contributor
timestamp: August 15, 2025, 03:35:21 pm
content: QuoteOf course, both text and binary protocols can be made extensible or not. Text parsing is a more complex task than directly reading a binary structure.But you can always use a standard protocol and get rid of the need to manually write parsers and solve extensibility problems.I already mentioned Protobuf (https://protobuf.dev/).It solves ALL the problems you mentioned.Only one problem with it could be absence of offitial C binding (and MCU targeted binding)Now it supports C++, C#, Dart, Go, Java, Kotlin, Objective-C, PHP, Python, Ruby, RustC binding also exists, but it maintained not by Google

author: nctnico
author_rank: Super Contributor
timestamp: August 15, 2025, 03:58:32 pm
content: Quote from: xvr on August 15, 2025, 03:35:21 pmNo. Because you'll always need to develop specific tooling which costs time and creates yet another project to maintain. Time = money. And the money needs to come from somewhere.And at the moment you are going to develop a record based binary protocol, you might as well do text parsing. Keep in mind string compare exits at the first mismatching character; it doesn't go through the entire string so testing message type versus command is not going to make a huge difference in processing speed.The same goes for size. Text based protocols allow to skip fields which don't change and only use enough characters to convey the value. In a binary protocol the field sizes are fixed so there is a lot redundant data in there. All in all there is very little difference in the amount of data to be transmitted.

author: xvr
author_rank: Frequent Contributor
timestamp: August 15, 2025, 04:14:51 pm
content: QuoteDid you know what Protobuf is?You don't need to 'develop specific tooling' - it already developed for any possible project.QuoteWhat to faster - compare one byte at record header or compare unknown number of bytes at begining of string?QuoteWho force to include records for unchanged data in binary protocol? Both of them can skip some data - it is not a question of text/binary, but question of data format/structure.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 15, 2025, 05:17:58 pm
content: Quote from: nctnico on August 15, 2025, 03:58:32 pmSure. Being able to skip fields means stateful communication, desynchronization of the data being shared, and implicit "last values" that A) you don't even know if they have been shared (e.g., system boot), B) what are the last values; unless specially queried.This is all more work, and higher risk for hard to find bugs.Often it does not matter, of course. Neither the extra overhead of transmitting every value every time, though.The latest binary protocol I designed simply updates the whole state of the subsystem every time, even when it's a bit wasteful, maybe just one field actually changed. That leaves a record in the database though what the instantaneous valueswere- even if they were the same. Simplicity is valuable, as you say.The other direction,commands, are prefixed with a transaction ID, and the reply indicates a copy of this ID, and success code, so that whoever did the command can check if the response was generatedafterthe command was processed, and that no other commands intervened.To you, I'm sure, these kind of details do not matter. You type the commands in Hyperterminal, and get a reply. If it does not work, you retry. Works with humans; you are subconsciously the parser, error checker and recovery system. With a machine and asynchronous link like UART, if tx is being buffered, you may be processing old input data as a reply to your command. Your protocol is text-based, and hence automagically Good, Easy and Bug-free.My experience on project-specific tooling is that it almost always saves time and money in the longer run. Using one-size-fits-all generic tools is inefficientevery timeyou use them. If you only need to do that randomly - not a problem. But for something you do 1000 times, spending 20 hours on a tool which saves 5 minutes every time, has saved you 63 hours. Of course, avoid the other extreme, NIH syndrome.In recent years, I have needed mostly CSV inputs and outputs formatted in a very specific ways, for testing, simulation and debug. Surprisingly quick to write in C, both ways, given that you have the binary data and struct definitions in header files!

author: nctnico
author_rank: Super Contributor
timestamp: August 15, 2025, 07:38:07 pm
content: It is not like a text protocol can't be used for automation. SCPI (to name just one example) has been made for automation. Many terminal emulators have the ability to send a command repeatedly and capture the output. Handy if you want to do a measurement over a longer period. No need to type commands manually over and over and no need to develop anythin.And if anything else fails, you can hack a Python script together quickly. Can make nice graphs as well with a few lines of code. No way I'm going to do that in C. I get that you want to use C for dealing with binary data because C sucks at dealing with text. But that is not a problem with using a text protocol, the problem is using C for tooling.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 15, 2025, 08:56:19 pm
content: Quote from: nctnico on August 15, 2025, 07:38:07 pmIf you roll out your own protocol, you already have the C code to parse it at the receiving end. What's the point in re-writing it in Python? Simply use the same code on PC or wherever you do your protocol debugging.

author: westfw
author_rank: Super Contributor
timestamp: August 15, 2025, 08:58:18 pm
content: Quote"text based" protocols are seen as less efficient than binary protocols, but when you combine improvements in technology (CPUs with fast multiple and divide instructions, better "dictionary" software algorithms (sometimes incorporated into programming languages!) with the desire for type-independent binary algorithms, I'm not sure that that's true any more.I would much rather use a text-based algorithm than go anywhere near ASN.1, for instance.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 16, 2025, 12:14:46 am
content: Quote from: westfw on August 15, 2025, 08:58:18 pmIt's all about your application and goals, right? Text based is actually simpler to parse in some ways, because you have all sorts of characters available to convey special meaning.I can think of a lot of times when text wouldn't work and I didn't even bother trying, simply because of latency or size, but the most of the time it just hasn't mattered. Yes, it's possibly way less efficient, but my processors and comm channels are mostly sitting around doing approximately nothing most of the time anyway. May as well use them to make our lives easier.

author: brucehoult
author_rank: Super Contributor
timestamp: August 16, 2025, 12:19:02 am
content: Quote from: nctnico on August 15, 2025, 07:38:07 pmG-Code for controlling machine tools, since 1963. And found new life in 3D printers.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 16, 2025, 03:08:59 am
content: Quote from: brucehoult on August 16, 2025, 12:19:02 amCNC operators often write G-code by hand.

author: John Coloccia
author_rank: Super Contributor
timestamp: August 16, 2025, 04:16:39 am
content: Quote from: brucehoult on August 16, 2025, 12:19:02 amI know you know this, but for those who don't, our gerbers basically g-code.