How to prevent the compiler messing with a structures allignment - Page 1
Forum section: electronics / general

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 01:55:00 pm
content: I am sending a 24 byte structure over can bus. It is sent in 3 parts, the middle message randomly compiles depending on my attempts to fix it with the first 32 bit variable after the last 4 1 byte variables, why? why can't it leave the damn thing alone?

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 02:40:58 pm
content: Lesson 1: don't send structs over communication busses / map structs onto buffers. Create a buffer and place the elements in there (disassemble). At the receiving side: re-assemble and do range / validity checking.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 02:44:46 pm
content: I send them all the time like that, just pack the structure. How exactly - depends on your compiler.Sure, it is technically non-standard behavior, but all major compilers support that and in the end it ends up being more efficient, since compiler will generate fast access code for members that ended up being aligned anyway.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 02:53:14 pm
content: Seems it did not like a variable crossing the 4 byte boundary so started making a mess.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 02:57:01 pm
content: Compilers will naturally align types on their size boundary. Even if underlying hardware supports unaligned access, it is typically slower than aligned access.Just pack the struct and compiler will generate code for unaligned access where necessary. Note that it may be slower as a result, but often it does not matter that much.Here is example where structure packing is used to describe all the USB stuff -https://github.com/ataradov/free-dap/blob/master/platform/samd11/usb/usb_std.hand usb_descriptors.h. Doing this manually would be super messy.For GCC all you have to do is add "__attribute__((packed))" to the struct definition.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: August 23, 2023, 03:52:23 pm
content: Another thing to be aware when sending struct data across a communication channel is endianness.Even if your struct is packed identically between nodes, if one is little-endian and one big-endian, the data is still not mutually intelligible. For example, one side has a big-endian uint16_t of 0x3039 (12,345), but if the other side is little-endian it'll see it as the wrong value, 0x3930 (14,640). So either ensure your nodes all have the same endianness, or pick one and translate appropriately.Things such as TCP/IP define the endianness as part of the packet specification, which is commonly known as "network order", as opposed to "host order" (which may actually be the same thing on some architectures).

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 04:04:54 pm
content: Well the can controller has no option on endianess so hopefully they managed to agree on what to do, forever.

author: djacobow
author_rank: Super Contributor
timestamp: August 23, 2023, 04:08:43 pm
content: You have a lot of options:* attribute packed: simple, cheap, but you have to deal with any difference between the sender machine and receiver machine (endianess, type sizes (use stdint.h types rather than "int", "short", etc))* protobufs (I use nanopb to make low resource embedded c implementation)* text: json, xml (ugh, but idiot proof, works over and channel, easy to log and for humans to interpret)

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 04:22:38 pm
content: Quote from: djacobow on August 23, 2023, 04:08:43 pmYou are forgetting the simple shift instructions to split up multi-byte words in single byte ones:buf= int16_var & 0xff;buf= (int16_var >> 8 ) & 0xff;This will always work on any architecture for any endianess. A clever compiler may even recognise and optimise. The code is easy to follow and it can't go wrong.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 04:52:48 pm
content: Quote from: djacobow on August 23, 2023, 04:08:43 pmWell if you can tell me exactly how it will be packed I may listen, this is the solution my google engineer apprentice keeps trying to make me use. But until I know how it behaves and if it is repeatable across systems then I am no ears.QuoteErr what?QuoteSo how does that work? Is it because it is 100% ascii so all 1 byte bits? quite the over complication for CAN bus that sends 8 bytes at a time with something like 40% data efficiency to the bus speed. That is why CAN open was invented I guess, where again randomly packed data in messages will wreak havoc.Sorry but for those that may have not noticed, the chip vendor has no problem with doing this for hardware registers, so why would it be a problem for any other structure? The answer that we have stumbled on ourselves is that the compiler is trying to optimize for 4 byte packs. So long as you write it as 4 byte units it will leave it alone, otherwise it starts to meddle.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 05:03:31 pm
content: Quote from: Simon on August 23, 2023, 04:52:48 pmIt is very simple. Normally all types are aligned at their size. With packing, they will be placed next to each other regardless of their size.So, "struct {uint8, uint32 }" would be 8 bytes in size. But the same struct packed would be 5 bytes in size. The downside is that uint32 member would be mis-aligned, so it can't be loaded/stored using a single CPU instruction.Quote from: Simon on August 23, 2023, 04:52:48 pmProtobufs are a somewhat universal way to serialize/de-serialize binary data.Quote from: Simon on August 23, 2023, 04:52:48 pmDoing what with registers? There are no misaligned registers in any devices, nothing would work that way.You may also want to provide a better description of what you are actually observing. We may be talking about different things. Provide some code snippets.

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 05:11:11 pm
content: Quote from: Simon on August 23, 2023, 04:52:48 pmGive him a kick in the nuts. The reason it works for registers is because these are aligned and have carefully defined types. If you dig deeper into how they are declared, you'll see these are packed structs which are tied into the architecture and the compiler.For general purpose code, it is better to not depend on compiler and platform/architecture specific solutions. It will come back to bite you as you have noticed. You won't be the first or last who trips over this. Better spend half an hour extra on writing code that is 100% guaranteed to work than 3 days looking for a bug.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 05:12:23 pm
content: As I am not at work I don't have any code to copy. But essentially sending data over CANbus. So the data that will be exchanged is set up as a structure. This allows it to be treated as one lump for transmission or reception but individual variables can be accessed.So as it is communication where knowing what is where is the only way to know what anything is, any automatic rearanging makes things very difficult. You also have to know that a different compiler for a different chip would still allign/pack the data in the same way. It's a nightmare. Imagine trying to do J1939 where the message ID tells you what is in the message. It would simply not work, in fact it would not work in any communication as the way communication works is to have a protocol that decides what means what and we all stick to that.In our case we had 3 1 byte variables followed by a 2 byte variable. At this point sections of 4 bytes reverse their positions. As soon as we rearranged it so that nothing crossed a 32 bit boundary it worked.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: August 23, 2023, 05:14:11 pm
content: It's worth reading this for some insight into alignment and padding:The Lost Art of Structure Packing.Quote from: ataradov on August 23, 2023, 05:03:31 pmI think he possibly meant why are vendor register definition structs not packed. And that's because registers are (typically) the same size as the CPU's native word size (e.g. 32-bit) and so are self-aligning, because there are no members in the struct that are smaller or larger than the word size.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 05:15:26 pm
content: Quote from: nctnico on August 23, 2023, 05:11:11 pmYou are missing the point. For communication to work you need the same reliability that is apparent in the structured declarations of registers. Communication works because you know what the received data is based on it's location in the message, unlike in one system where you can call anything by name and the system deals with where to get it.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 05:21:09 pm
content: Quote from: Simon on August 23, 2023, 05:15:26 pmI think you are missing something here.From what we gather, you are trying to cast a struct to an array of bytes and send it over CAN. This is fine, but the alignment of the members of this struct is implementation-defined according to the C standard. Most compilers will place members on their natural size boundary. In case of registers, designers of the register set did the same exact thing.Compiler alignment is 100% predictable, you jest need to take time to understand it instead of blaming everyone else.And nothing is stopping you from defining your protocol too have the same alignment for the members. Then you won't have to do anything else. Or just pack the structures.If compilers packed all the structs by default, you would be complaining that the code to read or write those members is slow and big and not atomic.

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 05:31:04 pm
content: Quote from: Simon on August 23, 2023, 05:15:26 pmI'm not missing the point. You are missing my point. And that is that you are trying to rely on the C compiler magically solving a problem for you. That is not going to happen. You'll either need to craft your structs carefully and tell the compiler how to treat them OR write code that tells any compiler in a unambiquous way (by using byte wise shifting) where you want which piece of information in the buffers you are sending across your communication channel.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 05:32:57 pm
content: Quote from: Simon on August 23, 2023, 05:12:23 pmSo the final layout of this struct would be {u8, u8, u8, dummy_u8, u16 }. If you add packed attribute, the dummy member would go away and extra code would be generated every time you access u16 member.Quote from: Simon on August 23, 2023, 05:12:23 pmIt is not a 32-bit boundary, it is any boundary that is multiple of the member size. If you only had one u8 member, then the struct would still be {u8, dummy_u8, u16 }. There is not even 32-bits here, yet alignment took place.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 05:48:30 pm
content: Quote from: ataradov on August 23, 2023, 05:21:09 pmI'm not complaining, I was under the impression that we had a correctly aligned and packed structure and was trying to understand why it was not doing as you have described.The issue was that because the structure was not properly aligned the compiler was doing this instead. This means that the data appeared in different places in the messages. Having found the problem and realigned the variables it compiled as expected.We ended up with a 32 byte structure at one point expecting to have a 24 byte one.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 05:50:31 pm
content: Quote from: nctnico on August 23, 2023, 05:31:04 pmI'm not missing the point. You are missing my point. And that is that you are trying to rely on the C compiler magically solving a problem for you. That is not going to happen. You'll either need to craft your structs carefully and tell the compiler how to treat them OR write code that tells any compiler in a unambiquous way (by using byte wise shifting) where you want which piece of information in the buffers you are sending across your communication channel.byte wise shifting? not heard of that before. What I would do if the structure was not to work would be to create an array of uint8_t and then setup pointers do define the variables, this would all be very tedious of course.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 05:52:13 pm
content: Quote from: Simon on August 23, 2023, 05:50:31 pmThis would not work. You will end up with pointers to u16 and u32 that are not aligned, causing hard faults. This is the reason for the compiler to bother with alignment in a first place.See Reply #8 for example of the most universal way of doing it. Also the most annoying way.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 05:59:08 pm
content: Oh, bit shifts. Yes. Well it sounds like if I align the structure data properly I should not have a problem.

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 06:04:25 pm
content: Quote from: Simon on August 23, 2023, 05:59:08 pmByte shifts that is (or smaller portions if necessary). But does the next person working on the project understand that? If you have difficulty with the concept yourself, then you should really ask yourself whether you want to go this route. I have stopped mapping structs onto buffers a long time ago because they are prone to all kinds of errors and/or people have trouble understanding the underlying concept because it is so tightly coupled with the hardware / architecture the software is running on.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 06:09:25 pm
content: People have trouble understanding a lot of things. If you target least understanding person, writing code would be miserable.I use directly mapped structs everywhere. It would be nice to have better and more standard ways of controlling the alignment or members, but simple packing works fine in practical situation. And in many cases doing things without that is a lot of work and I'd rather compiler do that work.Ultimately it is a matter of preference.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 06:28:28 pm
content: All I need is that it be repeatable. If people are not competent then they should not attempt the job. We originally set out to be 4 byte aligned, this also aids in the variables being updated as another "process" in the interrupt handler will update the variable, if it is in the middle of boing read then that may not be nice. but if it is all 4 byte aligned then the read/write will be one operation so no entrails.

author: madires
author_rank: Super Contributor
timestamp: August 23, 2023, 06:28:41 pm
content: Why we don't use structs as protocol datagrams:- padding- alignment- endiannessInstead we put everything into a datagram buffer or extract everything from a received datagram, using functions like htonl(), htons(), ntohl() and ntohs() for example.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 06:40:27 pm
content: You have reasonable and predicable control over the padding and alignment. Endianness is a bit harder, but also solvable in a much cleaner way.I don't understand this fear of packing the structures. It results in a much cleaner and easier to maintain code where you have exact description of the data in a language format instead of extracting bytes from a buffer and potentially messing up the offsets when the data format changes.

author: PlainName
author_rank: Super Contributor
timestamp: August 23, 2023, 07:06:11 pm
content: QuoteIME this will lead to tears, usually after you've forgotten why it has to be aligned and using a different system where that doesn't apply anyway, so the problem isn't immediately obvious. It takes a while to figure out what's happening and then a longer while to figure out why it's broken now and wasn't before.

author: madires
author_rank: Super Contributor
timestamp: August 23, 2023, 07:11:25 pm
content: Quote from: ataradov on August 23, 2023, 06:40:27 pmOne reason is portability between CPU/MCU architectures. Compilers can cause some fun too (other compiler, other version). Also, protocols have their own idea on how data is arranged which might be very different from the CPU/MCU architecture you're writing code for.

author: Simon
author_rank: Unknown
timestamp: August 23, 2023, 07:43:56 pm
content: Quote from: PlainName on August 23, 2023, 07:06:11 pmWell if you can explain how to "mutex" in a simple C environment. When you think about it it is actually a lot more complicated than "just set a flag to grant/deny access to variable" enter OSes and ya ya ya I'll just use a single board computer next for what it was not intended for.

author: nctnico
author_rank: Super Contributor
timestamp: August 23, 2023, 07:48:34 pm
content: Quote from: Simon on August 23, 2023, 06:28:28 pmThe problem is that when using packed structs, this atomic operation is not guaranteed. I have seen compilers emit code that reads & shifts a multi-byte struct member from a buffer one byte at a time which is not an atomic operation as you expect.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 23, 2023, 07:55:19 pm
content: It really depends on how you use it. If you use the packed structure only for serialization, its fine. If you use it for anything else, it can start to be a problem: pointers to members are a different data type than pointers to the native types, in general, packed structures are also unaligned which means potentially every access requires byte-wise manipulation by the compiler. As mentioned, if you use the structure from an ISR you can't access it atomically.One way to solve this is to have two versions of the struct, a packed and unpacked, and only use the packed one for serialization. You then do an element-wise copy from one to the other when serializing. It's probably the best way to handle this if you need to match a specific layout, and it's easy enough to add endian conversion at this point if needed. It also reliably does exactly the same as the manual byte shuffling code people advocate writing by hand, but possibly more efficiently.But I generally wouldn't do that unless I needed to match a pre-existing data type. If I am working on PCs exclusively, I would just use a purpose built serialization format: probobuf, cap'n proto, or json. On a microcontroller I typically just serialize native (non-packed) structs. These days, alignment rules are standard enough that if you use stdint fixed width types, everything will get the same alignment on every platform I care about. You can add explicit padding members for clarity or if you want it to be compatible with 8- or 16- bit ABIs that don't use natural alignment. This is certainly not standards guaranteed behavior, but it works well enough for me.The advantage here is that you get fast access and avoid any other issues with packed data types.If I only need microcontroller->PC, generating JSON (or YAML) actually isn't too bad, but I wouldn't want to parse JSON on a microcontroller.

author: PlainName
author_rank: Super Contributor
timestamp: August 23, 2023, 08:28:28 pm
content: QuoteDepends on what you're doing but it could be as simple as reading until you get two values the same. More 'proper' would be to use some variable with atomic access to act as the mutex - this would be in your BSP so is dependent on, and likely changes with, your hardware but the interface to it would be the same for all. Move to other hardware and all that should change is the BSP (which is why it's so named).

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 23, 2023, 09:10:19 pm
content: Quote from: nctnico on August 23, 2023, 07:48:34 pmNote that if you can use C11, you can use stdatomic - define the struct members that require atomic operation as an stdatomic data type and use the appropriate functions for access.Of course you still have to be aware of the added cost, if it matters.

author: DavidAlfa
author_rank: Super Contributor
timestamp: August 23, 2023, 11:04:21 pm
content: Use packed struct and then send using uint8_t* pointer with data size to avoid endian issues.Now I had a doubt. Will the compiler handle copying between same structures, one packed and other not?Code:typedef struct { ... } data_t;__attribute__((packed)) data_t data_packed;data_t data_unpacked;Would this work properly?Code:data_unpacked = data_packed;data_packed = data_unpacked;I don't think so! Never tried, I guess the compiler will spit some nasty warnings out.

author: Shonky
author_rank: Frequent Contributor
timestamp: August 23, 2023, 11:16:59 pm
content: I'd expect the compile to barf on that straight away as data_packed and data_unpacked are essentially not that same type.

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2023, 11:47:15 pm
content: You can't set packed attribute to individual variables, it is a type attribute. GCC will ignore it and issue a warning that it was ignored.But generally, if you do a lot of processing on the variables, it may make sense to have two versions and use one for the storage and one for the transmission. You would still have to assign all the members manually, but in the end it is still cleaner code than building a buffer by hand. It is also more efficient code is some members end up being aligned by chance in the packed structure. Plus is is the best place to add endian swap in a clean way if needed.

author: djacobow
author_rank: Super Contributor
timestamp: August 24, 2023, 03:44:39 am
content: Why do these types of questions always end up like this: a thread with people insulting each other.Here's the thing: lots of people through history have needed to move data over comms channels. It is, to a first approximation, "easy" ... and then invariably this gets effed up one way or another.If you are a clever, or just think you are, then by all means, just send packed structures, or do your own byte order swapping, or whatever you need for your source, channel, and destination, and be on your merry way.Protobufs were invented by smart people at early Google who found that despite being smart, the effed this up regularly. So they made a protocol to make it easy to serialize and deserialize complex data structures. Furthermore they made a tool to make it easy to access the structures in whatever language they were using on whichever end. There are other tools for this purpose out there, but you should ask yourself why do they exist at all if it's so simple.The next level up of ways to transfer data is to stringify it. When you have a lot of structured data to send, and the structs vary for example lists change length, and dictionaries have more keys, then json or xml or even csv can be appealing. And it can be much easier to debug. But no doubt it has a lot of overhead regarding the size of the data transmitted.Original point was only that there are options, and if your problem really is simple, and if you actually know what you're doing, then of course you can send packed structs. And, the easiest way to deal with endianess is to have a system where all the endpoints are the same endianess. Then you can do nothing! And in fact people get away with that for a long time, until some new device shows up and is different, and then their code breaks. This has happened before, and it will happen again. And again, and again, and again, forever.

author: nctnico
author_rank: Super Contributor
timestamp: August 24, 2023, 10:06:33 am
content: Quote from: ataradov on August 23, 2023, 11:47:15 pmIt is easy to create a few small functions that put a certain type at the end of a buffer and increment the buffer pointer. That way adding / deleting a member just works out of the box.something like put_int32(value, **buffer_pointer)It byte-shifts 'value' into 4 bytes of the buffer and increments the buffer pointer accordingly. Unless the formatting of the buffer needs to adhere to a specification, you don't have to care where each element is at.And the compiler being able to do aligned accesses on a buffer is far less likely to happen. How can the compiler know a buffer pointed to by a void pointer or byte type pointer is aligned during runtime? And how much do you actually gain from a single aligned access versus loading 4 bytes when processing a message every once in a while?

author: Siwastaja
author_rank: Super Contributor
timestamp: August 24, 2023, 12:04:18 pm
content: Simon, ignore everybody else except ataradov and just simply useCode:struct __attribute__((packed)) type_name {...};I will guarantee it works and this solves your problems without overthinking nonexistent issues. This also minimizes the risk of human errors, and dependencies on external work. Basically, you hand out all the low-level details like alignment etc. to the compiler, and pay a performance penalty for this. Premature optimization is evil; don't do it; instead, describe the data type as a data type and let the compiler handle the access.I have been doing exactly this in dozens and dozens of different projects, never any problem for the last decade. None. Zero. Believe me and ignore the naysayers.(This will be my only post in this thread. Either you get it, or you don't.)

author: JPortici
author_rank: Super Contributor
timestamp: August 24, 2023, 12:51:04 pm
content: Quote from: Siwastaja on August 24, 2023, 12:04:18 pm+1And i just want to add that if one of the entities receiving/sending the data is a windows PC with the software compiled with GCC/MinGW, before the definitions of the types you have to add#pragma pack(1)then after the definition of the types you have to add#pragma pack()or MinGW will happily ignore the packed attribute

author: madires
author_rank: Super Contributor
timestamp: August 24, 2023, 01:36:21 pm
content: Quote from: djacobow on August 24, 2023, 03:44:39 amyoungster: I use this shortcut. Works great!greybeard: Some words of wisdom ...youngster: Meh. It works and it's simpler.greybeard: Sooner or later you'll find out the hard way.youngster: I don't need your advise!greybeard: Ungrateful youngster!A never ending story.

author: AVI-crak
author_rank: Regular Contributor
timestamp: August 24, 2023, 01:43:58 pm
content: I don’t understand how you can discuss an abstract problem blindfolded ...

author: DavidAlfa
author_rank: Super Contributor
timestamp: August 24, 2023, 02:27:06 pm
content: Greybeard = wisdom? Sometimes, but very often it's just fixation/obsession/habits, never-happenning 1 in 10 million things, and "I always did that way " rule, as you age it gets worse and worse and some won't accept any other method except their's.Probably I'm getting roasted again, but I don't care!

author: gf
author_rank: Super Contributor
timestamp: August 24, 2023, 03:10:37 pm
content: Quote from: Siwastaja on August 24, 2023, 12:04:18 pmAnd in C++ with the Boost.Endian library you can even go further and write e.g.#include <boost/endian/arithmetic.hpp>using namespace boost::endian;struct S {big_uint16_t a;little_int32_t b;};which leads to a packed struct as well, and which does the proper endian conversion on the fly when you access the members.

author: Karel
author_rank: Super Contributor
timestamp: August 24, 2023, 03:39:04 pm
content: C++ is a horrible language. Way too complex and you don't know what it's doing under the hood...Thanks but I'll stick with C.

author: peter-h
author_rank: Super Contributor
timestamp: August 24, 2023, 03:39:29 pm
content: Data intended to go over a serial channel should always be assembled and sent byte-serially.Compose a buffer (defined as bytes e.g. uint8_t fred; ) and load it with your data.At this point you will need to think about which of say 4 bytes of a uint32_t goes out first. So e.g. if you haveuint32_t joe;you might dofred = joe & 0xff; // loads least significant byte of joe into 1st byte of bufferfred = (joe>>8 ) & 0xff;fred = (joe>>16) & 0xff;fred = (joe>>24) & 0xff; // loads most significant byte of joe into 4th byte of bufferProbably the "&0xff" is not required since the array fred is defined as bytes, but it makes it clear, and covers the "integer promotion" thingy in C.The >>24 shifts are just 1 clock cycle because an arm32 has a 32 bit barrel shifter.The other advantage of preparing a buffer is that you can breakpoint the code and examine it before it gets sent, and similarly at the other end.A lot of people love structures, typedef structures, structures of structures, and before long nobody else can follow what is going on.In your case you are probably stuck with some predefined protocol format definition, but an even more portable way would be to generate a decimal string, with sprintf ("%d etc. Or ltoa() etc. Then pick it up at the far end with atoi() or atol().

author: abeyer
author_rank: Frequent Contributor
timestamp: August 24, 2023, 08:00:35 pm
content: Quote from: SiliconWizard on August 23, 2023, 09:10:19 pmI'm curious, do implementations of stdatomic deal with unaligned access in practice? They're certainly not guaranteed to

author: cv007
author_rank: Frequent Contributor
timestamp: August 24, 2023, 08:18:36 pm
content: QuoteAn M0 (assuming a SAMC in use here) cannot make use of the atomic headers, in addition I believe you cannot use packed structs with atomic as it will assume all atomic vars are aligned. The compiler will emit instructions to get atomic and 'forget' what it knows about the packed struct (for example emits a ldrh even when packed struct alignment requires 2 ldrb's).The atomic headers for c/c++ are pretty much worthless for the mcu when you have an M0 in the mix of the mcu's you use, and want a universal solution. I assume that's why you seldom see any manufacturer's libraries using anything other than the 'old fashioned' interrupt protection.Simple example of interrupt protecting any scope, which in c++ ends up being nicer than the typical macros used in C-https://godbolt.org/z/WMG7MjvTYNot sure how good the example data structs are, but was just created on the fly to make the example a little more relevant.

author: Veteran68
author_rank: Frequent Contributor
timestamp: August 24, 2023, 08:33:35 pm
content: Quote from: Karel on August 24, 2023, 03:39:04 pmI disagree, but it's going to be a personal thing. The cult of programming language is one of the strongest forces known to man.A little career history. I was an assembler programmer before I was a C programmer, and looked down my young and arrogant nose at those "high level" C programmers who couldn't code in a "real language." Then I had to learn C (because you know, job) and quickly learned to appreciate the benefits it provided over assembler (even though I still have a soft spot for ASM and it still has it's place). For years I was an embedded systems engineer who wrote firmware code in C, ran it through the compiler to get an ASM dump, then walked through and hand-optimized the ASM to squeeze out every clock cycle I could before running it through the assembler to get a binary.I saw the C++ craze taking off, and once again, I looked down my nose saying "everything you do in C++ I can do in C." Then when I had to learn C++, the initial learning curve to "get it" was huge. I struggled. Then one day, the OOP light bulb went off and I was sold. I still wrote C from time to time for lighter stuff, and generally did prefer it for really tight code requirements, but as I wrote heavier, general/business computer software I appreciated the value of OOP and C++.Years later, when the Java craze took off, I once again frowned at it. However, due to a big project that was the first Java project in the company (we're predominantly Java these days), I had to learn it. I really fell in love with "write once, run anywhere" which I was skeptical of at first but quickly learned was 99% true. I was very familiar with writing C/C++ code that attempted to target multiple OSes and platforms (x86, 68K, 6502, 6809, and then DOS, Windows, Unix/AIX, Linux, Mac, IBM mainframe OS/390) and all the hassles that entailed no matter how much you stuck to "standard" C. Whereas with Java, I could compile one time and carry the bytecode around to any of those business platforms we had and run it as-is. Sure, Java has its drawbacks, mainly in startup time, but well-written Java once started up was quite performant. I had a demo codebase I used to demonstrate C++ performance and Java performance with the same algorithm, and with certain workloads, due to JVM optimizations, some Java code would actually meet or exceed C++ performance.Point being, I finally learned after several iterations of this life experience, that languages evolve and all bring their advantages (and disadvantages) depending on the use-case. For the past decade+ I've been on the management side and mainly write code these days only for my own benefit, but still consider myself a polyglot programmer who embraces new languages. I've always said that any software engineer worth his salt should be EXCITED to try new languages, and not resist them.I get that most on this forum are hardware guys that tolerate software as a necessary evil, so I get why you may not share my love for multiple languages. But I've always been surprised just how many in the software field (including many greybeards as previously mentioned) have the same attitude about new/different languages.

author: peter-h
author_rank: Super Contributor
timestamp: August 24, 2023, 08:40:47 pm
content: QuoteInteresting point... some arm32 handle unaligned, others do not, and even if the compiler knows the exact CPU and ensures packed structs are packed in a way which doesn't blow the code up, the code won't be portable. Even more fun if the CPU at one end of the link "is" and the one at the other end "isn't".

author: Simon
author_rank: Unknown
timestamp: August 24, 2023, 08:46:48 pm
content: people are getting confused. The structure is not sent as one message. It is sent as 3, being CAN bus knowing what is in each message is useful and where it is essential.I have already found that writing the same value to a variable constantly meants that now and then it shows up as 0 on the buss before the value is reasserted.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 25, 2023, 07:00:57 am
content: Quote from: Simon on August 24, 2023, 08:46:48 pmYou can do this:Code:typedef union{struct __attribute__((packed)){float thing;int32_t asdf;char qwerty;...}; // something that totals to 3*8 bytesuint8_t can_payload;uint64_t other_way;} the_message_type_t;This looks better than pointer casting or memcpying from some offsets. You can also the_message_type_t msg = {0}, no need for memset/sizeof stuff.Conversions through C datatypes are very handy, high-level and readable, especially if you allow yourself to use packed attribute which is compatible with all real-world actual compilers and thus creates no compatibility issues. For rare big-endian machines, you have to use endian swap macros, though, but this is rare and still less work than the suggested alternatives.

author: PlainName
author_rank: Super Contributor
timestamp: August 25, 2023, 07:26:39 am
content: QuoteSomething amiss there and, without evidence to the contrary, I'd go for a software issue. If the data is written out during the ISR then contention would be the first thing I looked at.

author: mikerj
author_rank: Super Contributor
timestamp: August 25, 2023, 08:01:19 am
content: Quote from: Simon on August 24, 2023, 08:46:48 pmIf you are using a packed struct then access to multi-byte words may no longer be atomic so you need to consider this if the struct is being accessed from multiple threads e.g. disabling interrupts whilst reading/writing.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 25, 2023, 10:15:19 am
content: Quote from: ataradov on August 23, 2023, 06:40:27 pmIt's not fear. It is just that if you intend to use the structure in for example Linux, the structure packing rules – specifically the order in which bitfields are allocated in bytes or words – vary depending on the ABI, and if you take care of all the variants based onpre-defined macros, you have a repeated mess of#if/#elif/#else/#endifin your hands that is extremely susceptible to bugs during editing.Before you get mad and point out we're talking about microcontrollers here, this is also about CAN bus and data interchange formats. It would be nice to have a consistent definition of that format across all systems.My own basic rule is that packed structures whose interpretation is defined separately at the bit level (in a comment or a separate document) are absolutely fine in a microcontroller or embedded environment; not so much in hosted applications, stuff running on top of a fully-featured OS, that should be intended to be portable by default. (That is,assumingit does not need to be portable is always an error; either you have a reason why it cannot be, or it should be assumed to need to be portable.)The main thing is that one must not define the on-wire data formatting based on some packed C structure: one must specify and document the actual bit sequence on the wire instead. And keep that spec with the code or header file that implements the packed C structure (or helper accessors), for future reference or if someone uses the code on an arch or ABI whose bitfield packing rules are incompatible.

author: HwAoRrDk
author_rank: Super Contributor
timestamp: August 25, 2023, 10:47:53 am
content: Quote from: Nominal Animal on August 25, 2023, 10:15:19 amI have encountered this in the wild, believe it or not. The official documentation for a serial communication protocol that had the message structure specified as a C struct. And naturally no supplementary information such as endianness or anything else. So I had to get the logic analyser out and figure out what was actually being passed on the wire anyway.

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2023, 11:39:28 am
content: Quote from: HwAoRrDk on August 25, 2023, 10:47:53 amExactly what happened to me with the Motorola CPU32EVS serial protocol.It's an custom BDM chip they made for some of their customers, it talks on the serial line.No additional information, not even in the manual except a C-like struct to describe fields.Not enough, I spend several hours on a logic analyzer to figure it out.At the end, cpu32 is BE, but the protocol has a lot of LE fields as it's assumed "for PC hosts", and x86 is LE.Plus, lot of fields padding here and there, all not mentioned in the documentation!I think, they made it this way to ask you money for the full documentation, or, better still, for a full implementation, which is usually closed source, or binary-blob.

author: Simon
author_rank: Unknown
timestamp: August 25, 2023, 02:00:29 pm
content: Quote from: mikerj on August 25, 2023, 08:01:19 amNo, it is not packed. I cannot pack it as that means that I have no control over the precise location of data in the messages which makes my particular use of CAN bus futile.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 25, 2023, 05:40:56 pm
content: This is very simple actually. If you design a protocol, use a structure where all members are naturally aligned (i.e. 2-byte members are at 2-byte boundary, 4-byte members are at 4-byte boundaries). Then you don't need to worry about packing or anything of that sort. All reasonable protocols do this.Show your structure, and we'll re-arrange it to make it naturally aligned.You will be hard-pressed to find a big-endian CPU nowadays. So, use everything little-endian and document this if you want to publish your protocol.If you use CAN, consider using CAN FD if you can - it can send longer messages and does it much faster.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 25, 2023, 06:00:40 pm
content: Quote from: abeyer on August 24, 2023, 08:00:35 pmFrom what I have seen no. And it should generate a warning as you are taking an address of an unaligned variable to pass to the atomic load/store functions. But I think that's a bit pointless. If you are doing much of any operations on a structure you shouldn't make it packed. Instead make a separate structure used for serialization only and mark that packed. You then just write a single function which copies the members element-wise before sending them -- adding endianness conversion if needed.It's simple, easy to understand after the fact, and at least as efficient as doing it manually. It's a tool the compiler gives you to create deterministic byte layouts, and you don't need to pretend it does something else.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 25, 2023, 09:48:18 pm
content: Quote from: peter-h on August 24, 2023, 08:40:47 pmI have used stdatomic on a number of different targets, but not Cortex-M ones so far. So, you made me curious.- First thing is that I haven't read anything regarding stdatomic and unaligned access in the C standard (both C11 and C23). I may have missed it though, but can't find it. It just doesn't talk about it. While C11 and later allow specifying alignment ('alignas' and siblings), the standard is pretty terse about all this. And "packed" structures are not part of the standard altogether, at least that I know of. I guess it makes sense that most of what it entails is completely implementation-defined.- GCC (in the form of arm-none-eabi-gcc, which is the variant that is used in all cases I've ever seen with Cortex-M MCUs) does come with stdatomic, and does support it for Cortex-M0 and upwards. I made some tests. (Possibly some vendors strip off stdatomic from the provided GCC distribution, although I haven't noticed it, but I only use the mainline GCC for Cortex-M targets anyway, so YMMV.)- Most Cortex-M (and even Cortex-A, may have missed some exceptions though?) do not support unaligned accesses by themselves, and this would raise a Hard Fault. Usually unaligned accesses are supported by the associated MPU when there's one available, and it is properly configured and enabled. (In particular, I did some baremetal coding on a RPi 4, aarch64 target, and until the MPU is enabled, unaligned accesses trigger a hard fault as well.) I have a doubt about the Cortex-M7. TBC.My tests show that:- You can absolutely use stdatomic on Cortex-M0 (and above) MCUs.- The compiler either emits 'dmb' instructions around the atomic access (Data Memory Barrier), or call '__atomic_load_x' functions from the GCC library. From what I can tell, dmb instructions are used up to 32 bits and __atomic_load_8 is used for 64-bit 'atomic' integers. I haven't looked at the code that implements '__atomic_load_x' functions. the 'dmb' instruction seems available on all Cortex-M starting with M0 though.- While unaligned accesses to (packed) struct members with non-atomic types are properly handled (byte-size instructions are used), with atomic types, GCC doesn't seem to care, so you will get an unaligned access - so a hard fault. In that latter case, you also get a compiler warning about taking the address of a packed member. Again the end result is completely dependent on the implementation and target. If you have access to a MPU, it should work fine.Below is a small example, compiled withCode:arm-none-eabi-gcc -S -Os -Wall -mthumb -mcpu=cortex-m0plus stdatomic_ex_1.cCode:#include <stdint.h>#include <stdatomic.h>typedef struct __attribute__((packed)){uint8_t n1;atomic_char32_t n2;uint16_t n3;atomic_uint_fast64_t n4;} Message_t;uint32_t Foo(Message_t *pMsg){return (uint32_t) atomic_load(&pMsg->n2);}uint64_t Foo2(Message_t *pMsg){return (uint64_t) atomic_load(&pMsg->n4);}uint16_t Foo3(Message_t *pMsg){return pMsg->n3;}Which yields the following assembly:Code:.cpu cortex-m0plus.arch armv6s-m.fpu softvfp.eabi_attribute 20, 1.eabi_attribute 21, 1.eabi_attribute 23, 3.eabi_attribute 24, 1.eabi_attribute 25, 1.eabi_attribute 26, 1.eabi_attribute 30, 4.eabi_attribute 34, 0.eabi_attribute 18, 4.file"stdatomic_ex_1.c".text.align1.globalFoo.syntax unified.code16.thumb_func.typeFoo, %functionFoo:@ args = 0, pretend = 0, frame = 0@ frame_needed = 0, uses_anonymous_args = 0@ link register save eliminated.dmbish@ sp neededaddsr0, r0, #1ldrr0, dmbishbxlr.sizeFoo, .-Foo.align1.globalFoo2.syntax unified.code16.thumb_func.typeFoo2, %functionFoo2:@ args = 0, pretend = 0, frame = 0@ frame_needed = 0, uses_anonymous_args = 0push{r4, lr}movsr1, #5@ sp neededaddsr0, r0, #7bl__atomic_load_8pop{r4, pc}.sizeFoo2, .-Foo2.align1.globalFoo3.syntax unified.code16.thumb_func.typeFoo3, %functionFoo3:@ args = 0, pretend = 0, frame = 0@ frame_needed = 0, uses_anonymous_args = 0@ link register save eliminated.ldrbr3, [r0, #5]ldrbr0, [r0, #6]@ sp neededlslsr0, r0, #8orrsr0, r3bxlr.sizeFoo3, .-Foo3

author: nctnico
author_rank: Super Contributor
timestamp: August 25, 2023, 10:09:34 pm
content: From my own experience I have to warn you that not all ARM platforms produce a hardfault on unaligned access but instead store garbage. I'm not going to discuss whether that is right or wrong. In an ideal world there wouldn't be crime either. Real world sucks some times, deal with it.Another problem with relying on platforms producing a hard fault in case you mess up code is that software which runs fine on a PC based test bench, doesn't work as expected on the actual platform. In a normal situation, the code may not take the unaligned code path very often and thus it is leading to an obscure bug that can take days to discover (not to mention the inconvenience it may cause to customers). Been there, done that. All in all, using stdatomic in combination with packed structs doesn't look like a good idea to me.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 25, 2023, 10:37:06 pm
content: Quote from: SiliconWizard on August 25, 2023, 09:48:18 pmAFAIU _atomic_load_8 uses a mutex. At least on hosted implementations. The typical implementation is to have an array of locks and pick the one to use based on the address of the atomic variable. I'm not sure what it does on freestanding implementations, probably calls a mutex stub function that the user needs to override. If that doesn't do anything because the platform doesn't implement locks you are just going to get non-atomic operations. In any case, you need to check that the operations are actually lock free if you want to use it in a signal handler, which I guess is the most common use of atomics on the smaller cortex-M processors.Quote"Fine" in the sense that it won't crash, but will it actually be atomic? Or will it be converted to a series of multiple aligned memory access? I think the latter, certainly you can't count on it working properly.None of that is anything bad about using packed structs, they are a perfectly fine way of marshaling data to send over a serial line. Putting atomics in a packed structure and/or accessing them concurrently from multiple threads or signal handlers is just a bit of a silly thought exercise, it's not something anyone should actually do.

author: cv007
author_rank: Frequent Contributor
timestamp: August 26, 2023, 12:57:34 am
content: QuoteYou never got to the link stage. See if you can get the linker to find these atomic functions for an M0.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 26, 2023, 01:26:54 am
content: Quote from: SiliconWizard on August 25, 2023, 09:48:18 pmalignas only allows making alignment stricter. The intent is to allow things like cache line alignment to prevent false sharing, align SIMD vectors, and page alignment when it matters. So it doesn't affect the validity of the objects placed with it since the location would be legal anyway.

author: radiolistener
author_rank: Super Contributor
timestamp: August 26, 2023, 02:23:35 am
content: using struct to format message to communicate between systems is a bad idea.The first issue here is different alignment setting on different platform/OS. It means that structure sent from one system to another will be broken. You can use alignment control attributes and compiler setting to fix that, but that is a half of the story...The second issue is that different platform/OS have different byte order for data representation in memory (see big-endian vs little-endian byte order). It means that structure sent from one system to another will be broken...So, the reliable way to format the message is to use raw bytes array and serialize data into that array byte-by-byte according to standard structure and byte order required for message exchange.Also note that using manually controlled alignment don't solve all issues, because there is also x16/x32/x64 compilation mode which should automatically adjust alignment, but if you use custom alignment it can lead to broken data sizes and alignment. This is because many type size depends on compilation mode, for example x16 mode has sizeof(int)=2, x32 mode has sizeof(int)=4, x64 mode has sizeof(int)=8.As you can see, this is a bad idea to use struct to format message for communication between systems.

author: ataradov
author_rank: Super Contributor
timestamp: August 26, 2023, 02:39:42 am
content: Everything is a bad idea if you do it wrong. Packed structures give you 100% control over alignment. Only packed structures should be even remotely considered for direct interfacing. And obviously don't use 'int' or any other variable size, only use fixed size types for protocol defining structs.There has not been a relevant BE system in decades, and it makes sense to define the protocol as LE. If someone is unfortunate enough to be working with a BE system, they would have to flip the bytes or do manual shifts. It it their problem not yours. And there won't be any new BE systems, since designers are not stupid.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 26, 2023, 03:19:37 am
content: Quote from: cv007 on August 26, 2023, 12:57:34 amDigging this a bit deeper, GCC implements atomic functions in libatomic:https://github.com/gcc-mirror/gcc/tree/master/libatomic, for anyone who'd like to have a look. Of course, given the large number of possible targets, figuring out how exactly it is implemented for a given target, or if it's implemented at all, is going to take a while browsing the gigantic code base.libatomic, at least most of it, doesn't seem to be implemented for Cortex-M targets in general. Some atomic functions may be available (due to them being easy to implement directly with assembly instructions on a given target), but you'll have to figure it out on your own. For the M0/M0+, all that seems available are plain read/write atomic operations for widths <= 32 bits, and GCC emits DMB instructions just to ensure the memory order. That's pretty much it. Even the useful atomic_fetch_add()/_sub() are not available, even for widths <= 32 bits.The main reason is that there aren't any facility in Cortex-M0/M0+, and pretty limited ones in M3, M4. (Would have to look at the M7), apart from DMB.The sucky part is that the C std requires implementations that do not support stdatomic to define the __STDC_NO_ATOMICS__ macro. GCC doesn't define it for any ARM Cortex-M target. So, you have partial support and maybe it's listed somewhere in GCC's docs. Dunno.Here's an implementation of libatomic for targets that don't implement it:https://github.com/iot-lab/riot-upstream/blob/master/core/atomic_c11.cIt unsurprisingly implements most operations disabling/reenabling interrupts. You could use this as a template to implement it for your own target, using the approach that works best in your use case. That would allow writing code that is portable using stdatomic, if that's a goal.With this said, using this paired with packed structs was indeed more out of curiosity than anything else. But actually there's nothing specific to stdatomic here - you'll get the same potential issues as you would while dereferencing a pointer to an unaligned struct member, stdatomic or not. As I mentioned, when the compiler has the context (you're accessing the member from the structure itself or a pointer thereof, rather than directly from a pointer to the member), it will properly access it byte by byte when the alignment is not guaranteed.Of course, if you have complete control over the structure itself and how the members are ordered, you can always arrange to define a packed struct with all members properly aligned, while minimizing the amount of padding needed (if any).And finally, if I mentioned stdatomic, it's because I find it very useful on platforms that actually support it. It's usually much more efficient (and portable) than using mutexes or other sync means when all you have to do can be achieved with the standard atomic functions.

author: peter-h
author_rank: Super Contributor
timestamp: August 26, 2023, 06:16:39 am
content: We had a thread on atomic variables before.https://www.eevblog.com/forum/microcontrollers/32f417-arm32-which-variables-are-atomic/There is some asm instruction on the M4 which does uint64_t atomically but GCC doesn't use it. Single floats are atomic, doubles are not.32F4xx supports unaligned accesses fine.QuoteI don't doubt it but it is a dumb way to do it if there is another instruction.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 26, 2023, 04:15:02 pm
content: I don't believe there is. SRTD and LDRD are not (guaranteed) atomic. They might be good enough on some systems and some use cases but the compiler has to be correct first.You can of course implement your own atomic_load_8 and related functions to do what you want. If using load/store double is good enough for you, go for it.

author: Simon
author_rank: Unknown
timestamp: August 26, 2023, 05:47:40 pm
content: Quote from: NorthGuy on August 25, 2023, 05:40:56 pmYes indeed, which is what this thread was about, how to arrange the data so that the compiler did not mess with it. We have that licked now.Not all devices support CAN FD and there is no point in using it just because we have exceeded the 8 byte limit. What about when we exceed the 64 byte limit? No we have to understand the problem not just use a bigger sledge hammer to crack the nut. the nut will just keep growing.Ultimately we need to work with CAN Open, I guess we could consider CAN FD to send messages between our own devices as a separate protocol that are not picked up by the slower CAN Open, for example transferring data logs/files but day to day comms will end up being CAN Open.

author: JPortici
author_rank: Super Contributor
timestamp: August 26, 2023, 08:38:30 pm
content: Quote from: Simon on August 26, 2023, 05:47:40 pmYes. As a matter of fact ECUs usually broadcast their data with a sequence of multiple frames. e.g.: 0x610, 0x611, 0x612, 0x613, 0x614 are all issued at the same time every (say) 10ms, the 40 byte data spread across the frames

author: Simon
author_rank: Unknown
timestamp: August 26, 2023, 08:51:59 pm
content: It depends on how you are using CAN bus. J1939 is all about lots of messages where the data contained is identified by the message ID and it's location in the message. So having the data sent properly is very important, you can't just rely on the fact that once the messages arrive they will go to the right places becasue the system encoding and decoding is running the same variant of code that randomizes the data arrangement. That would work in our case but is not guaranteed and I would like to spy on the bus with a USB CAN dongle and use a bus monitor program with a DBC file, makes life so much easier.In the case of CAN Open each message seems to carry just one piece of data. But now the data is identified by the data in the payload. From what I understand so far 24 bits are used to give the identification of the data carried. This means that again any mis communication unless identically interpreted at both ends which it won't be will be fatal.

author: DiTBho
author_rank: Super Contributor
timestamp: August 27, 2023, 10:13:18 am
content: Write critical modules in assembly and call them from C; this way you isolate the problem and you are 100% sure things are executed the right way.

author: DiTBho
author_rank: Super Contributor
timestamp: August 27, 2023, 10:26:44 am
content: Quote from: nctnico on August 25, 2023, 10:09:34 pmThis problem happens even with MIPS and PowerPC.MIPS ISA books say the CPU must always produce a hardfault on unaligned access, however from my own experiences I found CPUs declared "MIPS32 ISA" without any hardware circuit to issue an hardfault on unaligned access, so here you can understand why the company MIPS.inc hammered hard, suing anyone who called a CPU similar to their ISA, "MIPS-like" but not endorsed by them.Although it violates what is stated in the three IBM books that detail the ISA and give guidelines for implementations, same problem for PowerPC/Embedded. There are some softcore and hardcore implementing 4xx-cores that are missing hardware circuit to issue an hardfault on unaligned access as well as there are MIPS-like CPUs that do not even implement Ksegs.

author: Simon
author_rank: Unknown
timestamp: August 28, 2023, 10:48:00 am
content: Quote from: DiTBho on August 27, 2023, 10:13:18 amI doubt people writing architecture independent CAN Open libraries do this.

author: DiTBho
author_rank: Super Contributor
timestamp: August 28, 2023, 11:44:18 am
content: Quote from: Simon on August 28, 2023, 10:48:00 amsure, but look at the benefits: it's only a (edit: low level) module, it's deterministic, and it's the simplest and least painful way.

author: Jeroen3
author_rank: Super Contributor
timestamp: August 28, 2023, 12:31:00 pm
content: Because J1939 also support fractional bytes, such as values 5 or 11bytesbitsbig, on random start positions. You can't use any C structs to describe them, ever. Because eventually you'll run into an exception.Maybe C23 with the actual usable bitfields and arbitrary width vars, but I didn't read into that very much. Is it final yet?For this exact problem people invented generated serializer/deserializers such as protobufs. It doesn't really work for CAN because nanopb requires a stream interface and CAN has blocks of 8 bytes. So you have to make your own (de)serializers.The only way is to actually go in an do the shifts. Unfortunately. CAN bus is weird in that way.You can however read into SLOTs and create functions for those.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 28, 2023, 03:02:04 pm
content: Quote from: Jeroen3 on August 28, 2023, 12:31:00 pmYou probably mean "5 or 11bits".C has bitfields for that.If you deal with data which are not fixed structures (i.e. elements are not fixed at their positions), as if you wanted to decode BER (or Jason for that matter), you won't map your data to C structures. You do such mapping where it is beneficial for you, and you don't where it is impossible. Of course you need a little bit of a wisdom to tell such cases apart.

author: AVI-crak
author_rank: Regular Contributor
timestamp: August 28, 2023, 03:23:01 pm
content: Quote from: Simon on August 23, 2023, 01:55:00 pmWhen will your code be published? You can only part of the code - the subject of discussion of this topic.Are you sure you have an error?

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 29, 2023, 03:43:45 am
content: Quote from: peter-h on August 26, 2023, 06:16:39 amARM docs are not the easiest to browse. But from what I found out, unaligned accesses are supported on the Cortex M4 and M7, but not on the M0 (and probably M0+).But on the M4 and M7, unaligned access support can be enabled/disabled via a configuration register. The value upon reset may depend on the decision each MCU vendor has made (so could be enabled by default, or disabled by default.)Either way, you could change it in the startup code (or even anywhere else in your code if that made any sense.)When unaligned accesses are supported and enabled, they are not guaranteed atomic.Either way, GCC's default behavior depends on the target, and can be changed using the appropriate option flag.Quote

author: radiolistener
author_rank: Super Contributor
timestamp: August 29, 2023, 08:01:42 am
content: Quote from: ataradov on August 26, 2023, 02:39:42 amit give you almost 100% for a home pet projects. But if you're doing something more complicated, which needs cross platform support, then using structures can give you a lot of problem, especially if you're interact between systems with different byte order and different alignment.I know example of structures which can easily give you mismatch and there is no way to write it in such way to compile with no mismatch on different systems without manual edit... Interesting thing is that they are very popular and many peoples even don't understand why that code cannot works on x86 platform, but running ok on x64... And why their code works ok on one linux distro and cannot works on anotherThere are some hacks that allows to bypass some limitations of structure alignment, but these hacks working for a limited cases. Such kind of programming is named "kludges" and is not a good approach for a serious code.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 29, 2023, 10:44:57 am
content: Quote from: radiolistener on August 29, 2023, 08:01:42 amName one. A packed struct with fixed width types will give you perfectly reproducible layout on any platform.QuoteIt's incredibly common in "serious code". If you are serializing a raw binary blob it's by far the best way to do it. It has very good performance, is easily portable, and is much less error prone than manual byte shuffling.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 11:34:25 am
content: Quote from: ejeffrey on August 29, 2023, 10:44:57 amOnly when the structure does not contain any bit fields. Bit field packing order depends on ABI, not the hardware architecture: they can be packed either least significant bit first, or most significant bit first, and is not dependent or the same as byte order (endianness). It may even be modifiable using compiler options on some architectures (where there are ABIs with different bit field packing conventions, typically).Before you say you meant structures without bit fields, bit fieldscan, and should be used with fixed-width integer types, to ensure correct byte/word allocation pattern: a bit field will always be contained in a single unit of the base type, and never crosses the unit boundary. An unnamed zero-width bit field can be used to tell the compiler to skip the rest of the current unit, so that if followed by bit field of the same base type, it will start in a new unit.

author: mikerj
author_rank: Super Contributor
timestamp: August 29, 2023, 01:34:24 pm
content: Quote from: ejeffrey on August 29, 2023, 10:44:57 amA packed struct will still fail if the endianness of the target doesn't match the incomming data.

author: DavidAlfa
author_rank: Super Contributor
timestamp: August 29, 2023, 02:40:49 pm
content: Quote from: SiliconWizard on August 29, 2023, 03:43:45 amAlso Cortex-M3 supports it !I know that first hand because someone messed up part of the soldering firmare, somehow killing only STM32F0 devices (M0), while F1 series (M3) did just fine.After a hell of analyzing, the crash was indeed caused by unaligned access due bad pointer code.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 29, 2023, 03:18:26 pm
content: Quote from: mikerj on August 29, 2023, 01:34:24 pmDocument endianness and the order of bit fields, job done. I document the bit field ordereven ifI know that both ends would use the same ABI (e.g., the same microcontroller), because it's useful when parsing by eye (it's not obvious without a comment). All that's needed is comment on the first bitfield of a struct:Code:/*interface for module asdfquick guide of what asdf does and how it is used goes hereall multi-byte fields are little endian*/#pragma oncetypedef struct __attribute__((packed)) {// BYTE 0uint8_t enabled : 1; // lowest bit <--- this comment is pretty useful!uint8_t mode : 2;uint8_t reserved : 5; // write as 0// BYTE 1...} asdf_conf_t;

author: PlainName
author_rank: Super Contributor
timestamp: August 29, 2023, 04:22:37 pm
content: Perhaps we aren't all talking the same thing...There would be some protocol that defines the data, and that definition may be a struct (or not - so long as it's clear how the data is formatted that's what matters). Then you have a means of putting that data on the wire from your app, and if a packed format does that then why not? If you move the app to a different platform or there are changes to the protocol where the structure no longer works, then change it to whatever does work - that's the platform-dependent stuff.I think the arguments we are seeing here are because that protocol structure is then assumed to be being used in the application proper, and data used for, say, a display have no reason to use the same structure as the protocol even if it's the same stuff. By that I mean that the display data should be formatted in a way that suits the display, not the protocol. Maybe it turns out that the same structure could be used, but it shouldn't actually be the same structure - there should be a translation, or encode/decode, layer even if in one implementation it does nothing (easily compiled out, but it is logically there).So I am wondering if part of the argument here is because there is conflation of the inner workings and the low level workings and neither side is actually wrong, just discussing different things.(Of course, there is a strong imperative to just take the easiest route at the moment, particularly if it's a quicky and unlikely to be used on a different platform. Takes a few old projects getting a new life to get the lesson that quick and easy now may be a drag and a pain later.)

author: nctnico
author_rank: Super Contributor
timestamp: August 29, 2023, 05:56:00 pm
content: Quote from: ejeffrey on August 29, 2023, 10:44:57 amLot's of people smoke and/or drive a car after drinking a few beers. That something is commonly done doesn't make it a good idea. New insights take time to lead to new behaviour.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 06:35:56 pm
content: Quote from: Siwastaja on August 29, 2023, 03:18:26 pmYes, but:1) Not everyone agree that "lowest bit" = "the least significant bit". Old IBM Power documentation disagrees with that, IIRC, labeling bits starting at most significant bit as 1.2) I prefer a separate table in a comment before the structure, instead of using the structure members to document the underlying format. I've found that easier to keep up to date when patches are used to modify the structure, than when combining the underlying format and the structure definition.Thus, I would recommend more along the lines ofCode:#ifndef ASDF_H#define ASDF_H/* Interface for module asdfQuick guide of what asdf does and how it is used goes here.*//* asdf_conf_tExplanation where this packed struct is used.Multi-byte members are stored least significant byte first.Bit 0 is the least significant bit, and offset is in bytes:Offset Bit(s) Description-------- -------- --------------0 0 Enable foobar when set0 1-2 Mode0 3-7 Reserved, write zero: : :*/typedef struct __attribute__((packed)) {uint8_t enabled : 1; // Possible implementation details,uint8_t mode : 2; // like enum sets used for multi-bit fieldsuint8_t reserved : 5; // and such// ...} asdf_conf_t;#endif /* ASDF_H */but it is a matter of what one believes is easier to maintain and other humans to understand correctly.A build time test (run on the target) would then verify the packed structure size, fill it with a specific byte pattern, and verify the fields have expected values. The byte pattern must be such that at least one field will have an incorrect value if the endianness (byte order) and/or structure bit field ordering (whether the first bit field will include the least significant bit or the most significant bit in the unit) differ.On ELF-based toolchains, when the structure does not contain anonymous padding, one can do the inverse: create one or more read-only structures targeting a custom special section each. The source file is compiled to the target architecture (but no need to link), andobjdump/readelf(or similar ELF file tool) used to extract the special section(s) contents, which are then compared to the expected storage byte patterns. This can be done at build time, even when cross compiling, reliably.I use a very similar build time test to verify the implementation has an IEEE-754 Binary32 and/or Binary64 -compatiblefloatand/ordoublestorage implementation, too. I only really care about normals or finite values only, finite values being normals and subnormals. (For example, 0x2E3D4C5B corresponds to 0.000000000043041442f, if the byte order is the same for 32-bit (unsigned) integers andfloat. Some bi-endian architectures allow setting the byte order separately for floating-point values, so they're not necessarily/obviously/always the same; but a build-time check for the target architecture will tell.)

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 29, 2023, 08:48:01 pm
content: If you really want 100% failsafe and portable structs with the exact alignment, bit positions that you have defined, use ADA.Otherwise, packed structs in C are certainly usable as a number of us have said, but obviously not without a minimum of care and documentation.A couple additional tips apart from what has been said above:- Clearly separate the code that is not 100% portable from the code that is.- Document any non-portable code in appropriate ways. Writing a "porting guide" can be much more effective than just adding a few comments in the code itself, which some may ignore, either ouf of laziness or assuming de facto that code comments are false.And of course sometimes packed structs are perfectly viable, sometimes less so. Use whatever makes the most sense in a given situation. That's the difference between cargo cult and engineering.

author: radiolistener
author_rank: Super Contributor
timestamp: August 30, 2023, 04:01:16 am
content: Quote from: ejeffrey on August 29, 2023, 10:44:57 amIt was a structs from X11 headers used in cross platform software which tried to predict struct layout by knowing it's platform. It is not always possible, because even on the same platform the struct layout can depends on kernel compilation settings. So you cannot hardcode struct layout, because it depends on exact system.If you use packed struct with only fixed width types it may work, but again there is still some pit falls like byte order and platforms with specific alignment requirements.Recently I was seen data format which uses BE and LE byte order fields simultaneously in the same data structure. It was in Chinese oscilloscope...

author: DavidAlfa
author_rank: Super Contributor
timestamp: August 30, 2023, 01:55:17 pm
content: Those issues are 99% programmer's fault.A struct should not be written simultaneously by multiple threads, and if you want to send it somewhere you want a clean snapshot, using a mutex /flag to avoid anything modifying it, copying to a new buffer and releasing the lock.You have now a clean copy of the struct to do whatever you want without the rest of the program messing with it.

author: Simon
author_rank: Unknown
timestamp: August 31, 2023, 09:02:11 am
content: OK so I discovered that I had two conflicting pieces of code, one set it to 0 and I was trying to set it to 500, my fault.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 31, 2023, 11:01:24 am
content: Quote from: DavidAlfa on August 30, 2023, 01:55:17 pmAs an aside, one that can bite unwary programmers in POSIXy systems (Linux, MacOS, Android, BSDs):Mutexes and other locking primitives cannot be safely used in a POSIX signal handler, aside fromsem_post()(which increments/releases a semaphore). The workaround is to block the signals in all threads, and create one or more dedicated threads (with minimal stacks to minimise resource use) that wait for signals to occur usingsigwaitinfo(). To have these threads exit cleanly, use any POSIX realtime signal (SIGRTMIN+0toSIGRTMAX-0, inclusive), with the dedicated threads verifying thatinfo->si_pid == getpid() || info->si_pid == 0 || info->si_pid == 1(which correspond to sender being self, kernel, or init, respectively); and send such a signal usingsigqueue()orpthread_sigqueue()to each dedicated thread.It may sound complicated, but it is very simple in practice, and will open up completely new approaches, as POSIXy kernels can deliver all sorts of events via dedicated signals. Signal handling, however, is a bit tricky, because not everything in the standard C library can be used (seeman 7 signal-safety), but the above dedicated thread handling –– the same thread can handle any number of signals, noting that only realtime ones are queued –– avoids such limitations, and for those signals (that are blocked elsewhere) also ensures one won't getEINTR(interrupted system call) "errors". Add POSIX timers (I recommend in a simple binary min-heap keyed on real-world elapsing time), and you can do all sorts of timeouts robustly, with minimum overhead.Although POSIX threads are not completely portable in C, they are extremely powerful and surprisingly lightweight (except for the default stack size, which tends to be laughably large even on 32-bit systems).On microcontrollers – and this thread is in the microcontroller forum – there is a somewhat similar situation with interrupts: there aren't any universal mutex primitives one can safely use in an interrupt. Deadlock will occur, if code holding the same mutex the interrupt handler wants to grab is interrupted. On some architectures, one can use aload-link/store-conditionalmachine instruction for struct members that do not cross a native word boundary, with enough time in between to modify the native word in between. On others, readers and writers have to disable interrupts for the duration of the access. A yet another option is to simply use the interrupt to set a flag or post a message in a mailbox indicating the action needs to be done, with the action taken by the "main loop". (It is also a good option on multi-core microcontrollers.)Thus, it is pretty clear that if you want or need to modify the members from different threads or from interrupt contexts on a microcontroller, it is much better to use a non-packed structure, with member fields that can be modified separately. Both readers and writers still need to use some kind of synchronization primitive, which basically means accessor functions have to be used anyway.The main underlying problem is that the interrupt can occur while the interrupted code has the value to be modified already loaded/cached.The read accessor function is usually of a form that accesses the value atomically only once. No, you cannot just mark the membervolatileand have the compiler do the work for you; you want tocopythe value using a single access. (In C99 and later, a volatile-punned access in astatichelper accessor function that returns the read value, will work fine.) Some architectures have load instructions that do this, others may need to disable interrupts for the duration of the load, depending on the size of the "word" to be returned.The write accessor function varies. On some architectures for sufficiently small "words" it can be similar to the read accessor; on others, some kind of locking is needed. The architecture may provide LL/SC or compare-exchange primitives. One possibility is to have paired counters that write accessors increment in a specific order, and read accessors read in the opposite order, retrying until both counters read the same value (thus the data "word" read in between is "atomic"). These counters may be specific to the member, structure, or even global (i.e., foranydata accessed via these accessors). (If the writer accessor blocks other concurrent writers, for example by disabling interrupts, then it makes sense to keep the first counter value in a register, and update the second counter value by storing the incremented first counter value.)Thus: there are lots of ways to implement the mutex-like locking, but as DavidAlfa indicated, it is quite important in all cases that the lock (or equivalent) is held for the minimum duration possible. When the locking can fail, like above for readers in the dual counter case, a loop retrying until successful is basically required. Simply put, all of the available approaches have their upsides and downsides, which depend on the exact hardware architecture (and read-write pattern) at hand.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 31, 2023, 02:18:15 pm
content: Quote from: Nominal Animal on August 31, 2023, 11:01:24 amAlthough, quite often the members in a struct are somehow related to each other, for example current and voltage measurements from the same time, in which case one needs to mutex/disable interrupts around the access to the whole struct - e.g., copied during the critical section to a local copy.

author: tridac
author_rank: Regular Contributor
timestamp: August 31, 2023, 02:29:18 pm
content: Hi,typedef a union containing the structure and a U8 byte array, sizeof (struct). Then, you can send the structure by accessing that array, in several parts as need be. Compilers may pad out smaller than word size structure elements and that will depend on the compiler. You can force alignment to word size by including dummy elements, useful if you are mapping a machine peripheral register set onto a structure..Another way might be to just copy each structure element into an array, adding the size of element to the index for each, then send as required. The receiver gets the actual structure contents in size, with no compiler padding...Chris

author: gf
author_rank: Super Contributor
timestamp: September 01, 2023, 09:41:01 am
content: Quote from: Nominal Animal on August 31, 2023, 11:01:24 amDepending on the exact use case, it may even be necessary for them to reside in separate cache lines on multi-core systems, for performance reasons (-> avoid false sharing).

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 01, 2023, 01:55:42 pm
content: Quote from: gf on September 01, 2023, 09:41:01 amYup; I've been bitten by cacheline ping-pong before myself.In parallelized HPC simulations, it is often better to duplicate almost-constant data, and use explicit synchronization to distribute any changes, than let different threads access a single copy of the data. It is also a big reason why a process-parallel simulation/calculation written in C is often faster than a thread-parallel one: it is very difficult to understand and exploit all the cache effects (and nowadays the effects of speculative execution exploit mitigation), and the simpler rules of process parallelism just "guide" human developers in a better direction. Just like it is easier to write compiler-vectorized matrix computation in Fortran than it is in C, even though both are compiled to machine code: the Fortran (90 and later) idioms just match better the efficient approaches than the existing approaches possible in pure C.Essentially, to haveper-taskcopies of the almost-constant data needed during computation.In a microcontroller environment, this means that it may sometimes be necessary (but also useful) to have an interrupt modify acopyof some property –– say, a voltage or current measurement –– which is onlysynchronized(or cached) at specific point or points in the main loop.This is particularly true if the measurement is used multiple times in the main loop; say, to calculate instantaneous power, median power, median voltage, median current, and average voltage, current, and power. If the voltage or current was updated during one of those calculations, it is quite possible for the medians and averages to actually slowly drift apart! Unless you realize exactly how and why that is possible, trying to debug something like that is a bit of a nightmare, because it is not a code problem, but a modeling or design problem.

author: nctnico
author_rank: Super Contributor
timestamp: September 01, 2023, 02:04:10 pm
content: Quote from: tridac on August 31, 2023, 02:29:18 pmThis still doesn't take care of alignment issues which is one of the potential problems when mapping structs onto byte buffers. The root of the problem is that the data types of C where never designed/defined to be portable across platforms in their binary/platform native form.

author: NorthGuy
author_rank: Super Contributor
timestamp: September 01, 2023, 04:13:07 pm
content: Quote from: nctnico on September 01, 2023, 02:04:10 pmThe keyword here is "potential". You need a little bit of basic understanding to avoid the problems.People lacking such understanding, when they encounter problems, see such problems as inexplicable and consequently are makng an elephant of a fly.

author: nctnico
author_rank: Super Contributor
timestamp: September 01, 2023, 04:17:16 pm
content: Quote from: NorthGuy on September 01, 2023, 04:13:07 pmIn theory you are right, but the real world is different. The latter is why all modern data exchange formats and protocols are text based. There is no ambiguity about how to interpret c.q. use the data. But this has been explained at length several times already.

author: NorthGuy
author_rank: Super Contributor
timestamp: September 01, 2023, 04:44:45 pm
content: Quote from: nctnico on September 01, 2023, 04:17:16 pmHow come? HTTP/1 was text based, but HTTP/2 is binary.I haven't seen any text based CAN protocols though.

author: nctnico
author_rank: Super Contributor
timestamp: September 01, 2023, 05:09:20 pm
content: Quote from: NorthGuy on September 01, 2023, 04:44:45 pmI have

author: Siwastaja
author_rank: Super Contributor
timestamp: September 01, 2023, 05:27:55 pm
content: Quote from: nctnico on September 01, 2023, 04:17:16 pmYour trolling is getting too obvious. This "clicked" to me maybe a year or so ago, that you are some kind of artistic installment / interesting troll, but many have not realized it yet, but if you go this far, I think too many will get your true intentions and hence lose interest.The "packed struct causes unaligned access" lie is excellent, 90-95% just believe it, do not confirm it and you can do real harm with such lies, and since you repeat it ad nauseam, people like me lose interest in correcting this misinformation in every thread, multiple times per thread. But ALL modern data exchange based on text? Give me a break, nowthat'snot going to fly. No one will take this seriously.

author: nctnico
author_rank: Super Contributor
timestamp: September 01, 2023, 05:39:51 pm
content: Quote from: Siwastaja on September 01, 2023, 05:27:55 pmNo, you are just getting tired of being proven wrong all the time and now need to resort to some kind of petty vendetta. Just play the ball. Many of your contributions are usefull but don't degrade yourself with your vendetta. This is not the place.Based on what you wrote so far I probably have been doing engineering for a living for 1 or 2 decades longer than you have. Worked on a wider variety of systems with a wider variety of people. A lot of engineering problems do not stem from technical problems but human error. And then mix in being on a budget time & money wise. So part of the solutions I provide also attempt to mitigate human error and use solutions that have very little chance of leading to wasting time. You are simply not seeing that (yet). Likely because you rarely work within a team of people who have multiple disciplines and skill levels or are just stuck in a mode where all your co-workers should have a god-level skillset or can bugger off otherwise.

author: Siwastaja
author_rank: Super Contributor
timestamp: September 01, 2023, 06:06:20 pm
content: Yeah, I admit, I'm wrong. Really all modern data exchange formats are text-based. Good examples are The text-based h.264 video codec, The text-based JPEG2000 which replaced the obsolete binary JPEG format, the text-based Ogg Vorbis audio storage format which is superior to stupid legacy binary MPEG layer 3 format, not to talk about ZIP, the text-based data compression format. As we all know, IPv6 headers also transitioned into text format from the old legacy binary IPv4, and ext4 filesystem uses XML and stores the file contents encoded in base64 within that XML. What modernism!Sorry 'bout that. My shorter work experience prevented me from seeing how all modern data exchange formats and protocols are text based. They clearly are, because nctnico can't be wrong with such high work status.

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 01, 2023, 08:53:45 pm
content: Quote from: nctnico on September 01, 2023, 05:39:51 pmSo is he wrong all the time or are many of his contributions useful? Surely that wouldn't really add up. Unless being wrong is actually useful. Or something.Maybe one can design a text-based protocol out of this.

author: tridac
author_rank: Regular Contributor
timestamp: September 02, 2023, 12:34:30 am
content: It only works in a portable way if all the structure elements are U8, but if the union is defined the same at both ends of the comms, then it should work, assuming the same compiler is used. Copying structure elements, breaking down U16 / U32 into bytes while copying to the array is also useful. Have used both methods here in the past, but tends to be application specfic as to which is the best way. Perhaps best to organise the data to avoid having to send a structure in the first place. Comms works in byte streams, not structures...

author: Siwastaja
author_rank: Super Contributor
timestamp: September 02, 2023, 06:23:46 am
content: Quote from: SiliconWizard on September 01, 2023, 08:53:45 pmBingo! This is the key. Remember, wrong information, complex/difficult solutions, and especially adding confusion to things instead of trying to clarify them is an opportunity to some. When some lose, others win. Many make full careers out of this principle, and I totally loathe it.

author: PlainName
author_rank: Super Contributor
timestamp: September 02, 2023, 08:57:56 am
content: Quote from: SiliconWizard on September 01, 2023, 08:53:45 pmPerhaps we could calm down a little and revert to talking instead of being nitpickers. When we talk, as opposed to writing detailed specs, we often exaggerate or use hyperbole (or just talk 'in general') so there will be exceptions, different percentages and all that stuff. Easy to pick on if you want, but it's pointless and just stirs the flames (to mix metaphors). When I read "wrong all the time.... many useful" I don't read thatliterallybut understand what he is getting at - that the chap is wrong a lot, but that he also often comes out with good stuff.And please note I am writing what I think nctino is saying, not giving my opinion of Siwastaja.As for the protocol thing, treat it literally and you can easily prove it wrong, as you could pretty much any statement (and which members here do quite often!). The photo example is a good one. And yet... we do send photos as text in emails so that kind of thing isn't completely silly. And taken in the manner I said above, where we may use a bit of hyperbole, I think the perception is that we would use text-based protocols when possible. That doesn't mean it has to be every time, but if performance or whatever isn't an overriding factor then surely we would revert to something like JSON or XML or whatever the current flavour is.

author: Simon
author_rank: Unknown
timestamp: September 02, 2023, 01:39:17 pm
content: We seem to be getting confused between communication methods and data storage. Nothing is stored as text or transmitted as text. If I want the number 256 in my program it is stored in binary as 2 bytes, when it is transmitted, it will be transmitted as such.

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 02, 2023, 02:55:52 pm
content: Quote from: Simon on September 02, 2023, 01:39:17 pmIt can be useful to decouple the transmission format from the storage format, though.Some say packed structs are the way, some use accessor functions (or pack-unpack functions); both work, and have their upsides and downsides.Regardless of which approach you use, using a temporary structure to manage the multi-message transfer can be very useful. As an example, consider a case where you receive a chunk of information split in multiple messages whose order (or even source) may vary. You could have a bit mask in the temporary structure, one bit per message part, which is cleared when the temporary structure is copied to the variables or structure used by the main code. Packing this temporary structure is not a good idea, because you typically have only one, and having access to it as lightweight and "atomic" as possible will lead to more robust code.If there is a key or index in each message indicating which update each message belongs to, you'll also want to reset the bit mask when the key or index is newer than the one being constructed in the temporary structure. If the key or index order is unpredictable, youwillneed at leasttwotemporary structures, plus optionally a ring buffer of the few latest completed and/or discarded keys or indexes; with each structure having an extra (monotonic) counter. When a message with a key that does not match any of those in the temporary structures, you check the ring buffer if it is an old message. If it is not an old message, it is a new message, so you clear the older temporary structure, replacing the oldest key or index in the ring buffer with its key or index, and update the cleared temporary structure with the current message contents. If you have messages coming in for the same structure using different buses, this may benecessaryto properly receive all updates correctly.(It can be extremely informative to keep counters of the messages completely received, messages with keys/indices that had to be discarded, and duplicated messages ignored. Perhaps not during ordinary runtime, but definitely so when debugging.)When all parts of an update have been received, and the main code uses a structure, you can reduce latencies (assuming you use the "disable interrupts, update values, re-enable interrupts" scheme) by composing a temporary (and properly aligned) copy of that main-code structure locally, on stack. That way, the update simplifies to disable interrupt, fast copy of a fixed number of native words, re-enable interrupts, yielding the minimum time interrupts are disabled.Of course, when your main code uses fields of that structure, it too should disable interrupts, copy the entire structure to a local one (on stack), and re-enable interrupts, to ensure atomicity, and that the values do notunexpectedlychange during the main code (as things like average/mean voltage/current can drift from each other if that happens; they will only stay in sync as long as all statistically collected data involves the exact same data points).Of course, if you use an RTOS and not bare hardware, then you'd use the synchronization methods provided by that RTOS; but theprinciplesstill apply: temporary structures that can hold the necessary information, but not having the same format/alignment/packing that the transferred data or the structure used by the main code, can actually solve quite a few otherwise annoying and sticky problems. And minimizing exclusive lock duration is always a good thing, usually making a real difference with throughput and/or latencies.Hopefully, you can see how the posts by myself and others (before this text protocol nonsense) sketch out related issues and solutions to the binary-structured-data-via-serial-protocols general problem, those more usual in parallel and distributed processing under fully-featured OSes, but also applicable to microcontrollers (both under RTOSes and on bare metal) in this kind of situation.

author: slugrustle
author_rank: Frequent Contributor
timestamp: September 02, 2023, 03:06:19 pm
content: Quote from: Simon on September 02, 2023, 01:39:17 pmEmphasis mine.Possible exceptions areMICRcodes on checks (stored on the check as text, "transmitted" as text),fax("stored" as text on either end), and any storage or transmission method usingOCR. Then there are letters, books, and postcards, but I'm assuming the phrasecommunication methodshere implies communication between computers.I am only writing this to amuse myself and realize it is off topic.

author: NorthGuy
author_rank: Super Contributor
timestamp: September 02, 2023, 03:29:30 pm
content: Quote from: PlainName on September 02, 2023, 08:57:56 amThe email message format in the form which is used today was generally finalized in 1982 - you can read RFC 822. At that time, the majority of communication lines were 7-bit (ASCII is also 7-bit), therefore to transmit data over these lines, everything had to be converted to the text. That's why. Further, many of the mail programs of these days couldn't accept lines longer than 72-characters, so the text had to be broken down to short lines. This all was done out of necessity. This made email format somewhat convoluted, but there was no other choice.Later, in 1996, where official HTTP appeared, this was no longer the case, so they kept textual headers, but binary data (e.g. pictures) was transmitted as is. This was a leap forward.This was about 30 years ago. Now we have much better rockets than Coyote.

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2023, 04:06:40 am
content: Quote from: Siwastaja on September 02, 2023, 06:23:46 amso as logical choice to avoid being misunderstood and wasting your energy and time one'd better avoid to expose too much of personal professional knowledge and opinions based on personal working experience, instead of wasting them on imbeciles who then could potentially show no gratitude at best or accuse you of trolling at worst.

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2023, 04:34:08 am
content: Quote from: SiliconWizard on September 01, 2023, 08:53:45 pmhome automation sensors: text-basedtilt sensors: text-baseddebugger interfaces (gdb-like, trace32-like, ... ): text-basedactually it doesn't surprise me at all to read a similar answer, although I would NOT use it with canbus, but simply because the bandwidth is limited to 1Mbps and text-form halves the payload, plus, canbus should be used for fast event-processing, and transmitting text, while reducing ambiguity, costs more CPU cycles to decode.

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2023, 04:37:18 am
content: personally, I would investing time in the design and implementation of what Nominal Animal suggested in his last post.

author: Siwastaja
author_rank: Super Contributor
timestamp: September 04, 2023, 05:26:41 am
content: Quote from: DiTBho on September 04, 2023, 04:34:08 amI actually happen to professionally work with both home automation sensors and inertial measurement units (basically tilt sensors yes) and most I have seen have been binary protocols. Reasons are simple; home automation is often wireless* (mesh even) and packets have to be very small, so they can't afford text. Inertial measurement units and tilt sensors on the other hand tend to produce data at quite high data rates, and still be used in small embedded systems. With CAN, they easily hog the whole 1Mbaud/s bandwidth, or with RS485/422, they hog a 1Mbit/s UART. No one wants to really parse text in microcontrollers (except nctnico) when that can be avoided.*) and wired uses something like modbus rtu, still binaryWith home automation though, there often is another software layer (server, "data hub", "bridge") which might do conversion from binary to text format, so then you have both in one system. This is handy for example if you want to serialize the data for a web page to be accessed in JavaScript, in which case JSON makes a lot of sense.As usual, generalizations tend to go seriously wrong if they are made on feelings / cargo cult basis, instead of actual knowledge and experience.

author: JPortici
author_rank: Super Contributor
timestamp: September 04, 2023, 06:08:55 am
content: Quote from: Amelia Smith on September 04, 2023, 02:45:03 amnot.You usually employ two methods to broadcast data that has size larger than the data field1) you send multiple nodes, for example 0x4F0 0x4F1 0x4F2, at the same time, you do this to broadcast data with minimum latency2) You use transport protocols, such as CAN-TP described in ISO 15765-2, you do this do send a variable length amount of data from one producer to one consumer (the consumer has to acknowledge and control the transfer progress)i have also seen using a single node with one data byte representing the part of the message (simillar to CAN-TP, but not really), don't do this. It confuses the hell out of protocol analyzers unless you write your own that interprets the data.

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2023, 06:16:46 am
content: Quote from: Siwastaja on September 04, 2023, 05:26:41 amI think I said both bandwidth/rate and wired/wireless both matter.edit:instead of getting pissed, I tell a little anecdote:- - -Ironically, I'm working on converting some 3rd party tilt sensors sold as "text based" to "binary protocol" (canbus) sensors.It's 20hz 3 axes tilt, and it's sold as-is. RS232, text-based, in a plastic shield with a 4pin cable.With my colleagues, I'm developing a control motor for a cargo-bicycle based on a BOSH motor + SHIMANO 8SP internal gearbox coupled with a custom digital linear actuator that controls the gear.Nothing special, but when you stop at the traffic light going uphill with full load (200Kg!!!) the engine should know that you are going uphill to correctly select the gear and the crawl start procedure, as well as going downhill, the engine should know how to use the brake motor correctly in order to save energy and charge the supercapacitor and therefore the battery.This is because we would like to outperform the competition by offering something slightly better, especially in the mountains, where having a greater electric range means not having to use a lot of muscle traction or, worse, staying on foot because there is no way to recharge/replace the battery, and the load is too heavy for simple muscle traction.Anyway, cargobike are not a "new things", we are hacking some existing products, as the primary focus was the algorithm, which is "the" added value; the sensors, the protocols, the costs didn't matter for the first proof of concept.We worked with a prototype with "off the shelf" sensors, once the algorithm was designed and tested in a lab field. Once completed, analyzed, and verified, we can think of implementing our hardware with our own canbus inclination sensor, as required by our customers.September-October. It's time to think about how to design a little MPU board with a CanBus adapter and how it uses accelerometers, gyroscopes and sensor fusion math to bring out the artificial horizon. Design, implement it, debug and test it. A sub-project into the main project.Why binary, this time? Because it is the sensor that we produce for our own hardware, and therefore we can do it as we like. Still 20Hz, still very slow, but made binary it is better suited to CanBus because it reduces the payload and all messages + some meta can be packed and sent in one shot.

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 04, 2023, 08:06:38 am
content: Do we even agree to the distinction between "text" and "binary" formats? In my use, they correspond roughly to "human-readable/writable" and "non-human-readable", which is pretty vague and unhelpful. Here are the four format types thatdomatter to myself:Fixed formats, where the relevance of each byte is dictated by its positionChunked formats, where each chunk includes the exact length of said chunkStream formats, where control and formatting begins with a reserved value that is escaped or otherwise masked in the data streamHybrid formatsIt does not matter to me much if values are restricted in range or in a specific base: parsing big-endian multi-byte binary integers is not that different to parsing big-endian decimal numbers.For example, HTTP/1 is a hybrid format. Most of it is a stream format with CR LF (0x0D 0x0A) being a delimiter, but with chunked transfer encoding, it provides a length (decimal encoded, followed by a CR LF), then exactly that many raw bytes of payload (followed by an "extra" CR LF delimiter). It even supports gzip compression for the data payload, on top of chunked transfer encoding, in which case the length refers to the encoded length, not the decoded length.GIF images are an example of a fixed format. PNG and JFIF JPEG images are an example of a chunked format. HTML and XML are stream formats.Standard encoding of protobufs is a stream format, but netstrings are a chunked format.Many people suggest chunked formats, but they tend to require significant buffering (consider e.g. netstrings: to provide a value, you need to know its exact length in bytes before you can start emitting it, thus necessitating an output buffer of the maximum value length), making them annoying/problematic on small microcontrollers.I do have my own solution, a stream-based format that can be generated and parsed with minimal memory and processing, using a relatively simple state machine, supports lossless conversion from any XML-derived format with a lot less overhead than XML, can be used to replace protocols like HTTP, and can easily be extended to support multiple independent data streams on top of the same transport stream with minimal buffering (4-8 bytes). It can be expressed and defined in human-readable terms, but a binary representation (reserved byte values) tends to be more effective. With the multi-stream extension, the worst case overhead can be as high as 33%, but is neglible for typical data given a sensible choice of reserved byte values.I only bring it up because it fits both "binary" and "text" format categories, depending on the reserved byte value choices.(I really should try and publish it, because I think it would be useful to many if they just knew of the technique. Alas, I don't have thesocial werewithalto become a vocal proponent for it and push it to the relevant working groups and projects. Mentioning it here, like DiTBho does for their my-c, is near my limit.)It is not suitable for CAN bus, however, which has its own standard frame types with base and extended frames having 0 to 8 data payload bytes (and the number of payload bytes already specified in the data frame). Each payload segment is also so short that only a fixed format makes sense here. For up to a reasonable number of different logical payloads, different CAN bus identifiers can be used for each part/sub-message (each bus supporting 2048 unique identifiers), so multi-part messages do not need to use the payload bytes to identify the payload itself.Note that if the CAN bus message order cannot be absolutely controlled, you can useNtimes the number of CAN bus identifiers, in a cyclic round-robin scheme, forNdifferent logical messages in time. (You'd also want to have the additional bit mask I described, to determine when all parts of a logical message have been received.) This is also useful when using an RTOS and a mailbox scheme, because then you have (theoretically)N-1 logical message intervals to process the message in the main loop, before it is overwritten by a receive interrupt. (Obviously you can also use a queue for the logical messages to avoid losing any messages, but the queue primitives needed may not be available in an interrupt context – see my post about the problem of using a mutex in an interrupt context above.)

author: PlainName
author_rank: Super Contributor
timestamp: September 04, 2023, 01:01:40 pm
content: Did you allow for chunks not in a fixed order? I think that's one of the things that text lends itself to (apart from being readable by us): the data can be shuffled around and transferred in arbitrary order, which can make things both simpler to generate and trickier to parse. And, of course, arbitrary chunk lengths: text is defined by terminators and I think that's probably more robust than having the expected length noted before the actual data.

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2023, 01:57:25 pm
content: Like gps nmea

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 04, 2023, 02:07:23 pm
content: Quote from: PlainName on September 04, 2023, 01:01:40 pmSure, the order is often semi-arbitrary. For example, PNG files have an 8-byte fixed header, the first chunk is IHDR, the last chunk is IEND, but otherwise the chunk order may vary. IDAT chunks contain the image data, and can be split into two or more, it does not need to be a single chunk; the order of the IDAT chunks with respect to each other must be retained, however.That is, the order ofdifferenttypes of chunks is often arbitrary, but the order of the same specific type of chunk must be retained; and often one or two initial and sometimes a final chunk are mandatory.In fact, we could consider TCP (TCPv4 AKA TCP over IPv4, or TCPv6 AKA TCP over IPv6) protocol a chunked format, with their sequence number indicating the position in the stream. Even URGent TCP data is just a 16-bit offset into the packet payload (indicating the end of normal data and the start of URGent data), although TCP stack implementations (are supposed to) treat it as out-of-band data.Quote from: DiTBho on September 04, 2023, 01:57:25 pmYes, NMEA GPS is a stream format, where$is the separator (or new message begin marker), and comma,a field separator marker, and asterisk*the checksum-follows marker for each message.It does not matter that much that each field is technically human-readable; most of the data is numeric, andcouldbe nibble-encoded or even floating-point data, but those would be even more work to parse! The only downside is the relative low density storage for the numbers, but as the messages are quite short and the report interval is in the human scale (as opposed to say thousands of times per second), it does not matter when using typical current standard serial transport methods (as they have plenty of bandwidth for this).Edit: Not all text formats are stream formats, though. There are fixed-number-of-columns text formats, as well as text-based netstring-like encodings. And of course there are oddball variants between the four classes I showed, TCP in particular, where earlier bits in each chunk can modify the format of the current chunk (header fields). So it isn't a perfect classification system by any means, but it is more useful than "text" versus "binary".

author: nctnico
author_rank: Super Contributor
timestamp: September 04, 2023, 05:55:12 pm
content: Quote from: Nominal Animal on September 04, 2023, 08:06:38 amIf you have dedicated CAN IDs per producer / consumer pair, then the order of the CAN messages is perfectly controllable. Many CAN controllers also have multiple transmit and receive queues so a whole bunch of stream oriented CAN messages don't get in the way of higher priority CAN status broadcasts. But you'll still need to add something to the stream protocol that can detect missing / lost CAN packets.

author: julian1
author_rank: Frequent Contributor
timestamp: October 13, 2023, 10:52:24 pm
content: I am using bitfields in a packed struct,Gcc gives the following 'note' (not warning or error) related to a change between old compiler versions,In file included from myfile.c:nn:./myheader.h:nnn:n: note: offset of packed bit-field 'myfield' has changed in GCC 4.4Is there a way target/turnoff/suppress the 'note' with a pragma (preferred) or flag?Normally warnings can be handled with -W and -Wno flags etc, but I have not seen 'note' used as a reporting method before.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 14, 2023, 01:07:38 am
content: Quote from: julian1 on October 13, 2023, 10:52:24 pmYes: use-Wno-packed-bitfield-compat.

author: julian1
author_rank: Frequent Contributor
timestamp: October 14, 2023, 02:10:18 am
content: Thanks Nominal Animal. That works great.