Converting struct in to array  changing byte order - Page 1
Forum section: electronics / general

author: sangar
author_rank: Regular Contributor
timestamp: July 04, 2017, 05:39:15 am
content: Hello all,I am working on PIC18 (XC8 compiler) based Project which acquires 7 sensors values and send them to PC. Each sensor value is 2 bytes long basically 16-bit integer.so,totally 14 bytes.To frame all the sensor values in one packet, I use structure and convert it to array while send to PC.Problem:In the packet received at PC side, I am getting LSByte followed by MSByte instead MSByte followed by LSByte.EX:Actual value of Sensor_1 = 0x0155 , but in the RX side I am getting 0x5501.Here is the code:Code:typedef struct{uint16_t CR_sensor; //sensor 1uint16_t MR_sensor; //sensor 2uint16_t BC_sensor; //sensor 3uint16_t AR_sensor; //sensor 4uint16_t BP_sensor; //sensor 5uint16_t LVDT_1; //sensor 6uint16_t LVDT_2; //sensor 7}Packet_t;extern Packet_t packet;Uart_transmission:Code:UART_Write_Packet((uint8_t *)&packet,sizeof(Packet_t));Compiler: XC8 V1.36I read that this is the problem due to big endian format used by Compiler.Please suggest a solution.Thanks,Sangar

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 04, 2017, 05:47:56 am
content: correct the endianess in PC.. problem solved. or val = ((val & 255) << 8 ) + ((val >> 8 ) & 255) in mcu... edit: damned smiley code..

author: cv007
author_rank: Frequent Contributor
timestamp: July 04, 2017, 06:26:06 am
content: QuoteThe data is actually stored in little endian - the lower address has the lower byte. A uint16_t is stored LL HH (LL in the lower address, HH in the higher address), which means if sending out as bytes the lower address goes out first.If you want to send out HH LL, you have to reverse it somewhere.for( uint8_t i = 0; i < sizeof(Packet_t); i += 2){UART_Write_Packet((uint8_t *)&packet[i+1],1);//HHUART_Write_Packet((uint8_t *)&packet,1);//LL}or rewrite functionUART_Write_Packet(uint16_t* pkt, uint8_t sz){for( uint8_t i = 0; i < sz; i++){UART_write_byte(pkt>>8 );//HHUART_write_byte(pkt);//LL}}or many other options

author: agehall
author_rank: Frequent Contributor
timestamp: July 04, 2017, 06:27:20 am
content: Always use htons() and ntohs() for things like this!

author: AG6QR
author_rank: Frequent Contributor
timestamp: July 04, 2017, 06:32:42 am
content: Quote from: Mechatrommer on July 04, 2017, 05:47:56 amAt first I misread this as, "correct the endiannessofPC". IOW, avoid using anything with an Intel architecture or one of the other little-endian machines. I was going to shout out a big "Bravo!"But then I realized you probably meant to write the PC software such that it would flip the byte order. I suppose that's the most practical solution, and I will (somewhat reluctantly) endorse that.But something about fixing the PC to be big-endian still appeals to me.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 04, 2017, 06:37:40 am
content: Quote from: sangar on July 04, 2017, 05:39:15 amXC8 user guide page 149:-All integer values are represented inlittle endianformat with the Least Significant Byte (LSB) at the lower address.QuoteAnd that is a problem because?

author: IanB
author_rank: Super Contributor
timestamp: July 04, 2017, 07:28:55 am
content: Quote from: sangar on July 04, 2017, 05:39:15 amEither correct the byte order at the sending end or the receiving end. This question is a little bit like asking how to tie a shoelace, is it not?Another solution is to send the data in hex format, by literally sending "0155" over the link. A good reason for sending human readable data over the wire is that you can more readily sniff the messages and see what is being sent and received for diagnostic purposes.

author: Nusa
author_rank: Super Contributor
timestamp: July 04, 2017, 07:54:19 am
content: For portable code, the ideal solution is to avoid endianness issues entirely. I've written a lot of portable code in my day.One solution is to transmit data in text format. As a bonus, it's a human-readable data stream. If you're transmitting floating point data between unknown systems, I'd recommend this for generic situations. Native formats for floating point can be VERY different.Otherwise, my recommendation is properly define your transmission packets at the BYTE level. Then you know EXACTLY what to pack and unpack on each end, and the enddianness of the platform is irrelevant and doesn't byte you (pun intended).So instead of what you have, something like:typedef struct{uint8_t CR_sensor_msb; //sensor 1uint8_t CR_sensor_lsb; //sensor 1uint8_t MR_sensor_msb; //sensor 2uint8_t MR_sensor_lsb; //sensor 2uint8_t BC_sensor_msb; //sensor 3uint8_t BC_sensor_lsb; //sensor 3// etc...you get the idea}Packet_t;

author: alm
author_rank: Super Contributor
timestamp: July 04, 2017, 12:49:37 pm
content: While I agree that ASCII can be a good solution, I do not see the point of the latter suggestion. If you want to send binary integers > 8 bits, just define a byte order for the protocol and use something like htons() to convert. To split up the bytes in a struct still requires pretty much the same code as converting, and results in code that is harder to read.How is:Code:packet->CR_sensor_msb = cr_sensor >> 8;packet->CR_sensor_lsb = (uint8_t)cr_sensor;[...]UART_Write_Packet(packet, sizeof(packet));[...]cr_sensor = packet->CR_sensor_msb << 8 & packet->CR_sensor_lsb;More readable than:Code:packet->CR_sensor = htons(cr_sensor);[...]UART_Write_Packet(packet, sizeof(packet));[...]cr_sensor = ntohs(packet->CR_sensor);

author: NorthGuy
author_rank: Super Contributor
timestamp: July 04, 2017, 01:27:27 pm
content: How do you acquire data from sensors? The easiest way is to store the acquired bytes in the correct order (opposite to what you do now).

author: NivagSwerdna
author_rank: Super Contributor
timestamp: July 04, 2017, 02:05:58 pm
content: Quote from: Mechatrommer on July 04, 2017, 05:47:56 am+1These endian-ness issue crops up a lot when converting from internal representations to on-the-wire forms. In particular you find it when writing network protocols (if that's what you spent some of your life doing many decades ago)! The Berkley Socket functions htons (host to network short) functions were created to help out with this respect. On the plus side having a well defined network order at least allows the recipients to convert to their own host order which in a heterogeneous environment may differ (certainly did in the environment I was working.... but that's a longer story).

author: ealex
author_rank: Frequent Contributor
timestamp: July 04, 2017, 02:12:09 pm
content: Treating structure as byte arrays and sending them over between system with different endianes is a bad idea.It will work if you structure is homogeneous - same data type and it will break and make you waste hours when you change something in the structure - different data types or just structure packing.Specific case : 3 mcu's , different endianes, very little available resources ( 1K of RAM on one of them )complex structures had to go from 1'st to 3'rd where the 2'nd acted as a router + data modifier.The chain was : MCU 1 gets CAN frame, populates specific frame structure and sends it to MCU 2MCU 2 shifts endianess, changes something, shifts endianess again and sends it to MCU3now imagine what will happens if a compiler option changed on one of the mcu's to pack structures or when CAN structures change ...=> hours and hours and hours of comparisons, hunting bugs, etc=> the solution was to have common macros to shift / pack / unpack each structure to the serial bufferadded bonus : we could make a PC decoder and have some real-time debug

author: NorthGuy
author_rank: Super Contributor
timestamp: July 04, 2017, 02:17:11 pm
content: Quote from: ealex on July 04, 2017, 02:12:09 pmFYI: That is how Internet works

author: sokoloff
author_rank: Super Contributor
timestamp: July 04, 2017, 02:22:54 pm
content: Quote from: NorthGuy on July 04, 2017, 02:17:11 pmNot really. It treats structures as structures and uses hton and ntoh (host to network and network to host) convertors to put those structs into bytes on the network in network-byte-order.

author: NorthGuy
author_rank: Super Contributor
timestamp: July 04, 2017, 02:43:40 pm
content: Quote from: sokoloff on July 04, 2017, 02:22:54 pmPHY operates on the byte level. Drivers treat the sequences of bytes as packets (structures). It is up to the user program to apply hton/ntoh functions as necessary. In the end, "it" does all the job for you.When you're considering communications from MCU to PC through a serial line, you need to it all by yourself.

author: sokoloff
author_rank: Super Contributor
timestamp: July 04, 2017, 02:56:11 pm
content: Quote from: NorthGuy on July 04, 2017, 02:43:40 pmAgreed, but surely you don't consider PHY/layer 1 on a local LAN to be the same as "the Internet"

author: NorthGuy
author_rank: Super Contributor
timestamp: July 04, 2017, 02:59:55 pm
content: Quote from: sokoloff on July 04, 2017, 02:56:11 pmOf course not. Internet is all of the above and more ...

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 04, 2017, 03:17:39 pm
content: Quote from: agehall on July 04, 2017, 06:27:20 amBut XC8 doesn't have those functions built in so you have to write them yourself, and you are back to square one because the implementation is platform-specific.Luckily Microchip have a TCPIP library, which includes the following (in network.h):-Code:#define byteSwap16(a) ((((uint16_t)a & (uint16_t)0xFF00) >> 8) | (((uint16_t)a & (uint16_t)0x00FF) << 8))#define byteReverse32(a) ((((uint32_t)a&(uint32_t)0xff000000) >> 24) | \(((uint32_t)a&(uint32_t)0x00ff0000) >> 8) | \(((uint32_t)a&(uint32_t)0x0000ff00) << 8) | \(((uint32_t)a&(uint32_t)0x000000ff) << 24) )// host to network & network to host macros#define htons(a) byteSwap16(a)#define ntohs(a) byteSwap16(a)#define htonl(a) byteReverse32(a)#define ntohl(a) byteReverse32(a)

author: Ian.M
author_rank: Super Contributor
timestamp: July 04, 2017, 04:38:36 pm
content: Ditch the struct and code it in pure 'vanilla' ANSI C with *NO* assumptions about the endianness of the underlying architecture or compiler implementation.To do this store the sensor values in a uint16_t array so the transmit routine can loop through them as uint16_t values without type punning. Use an enum for the fixed index of each named sensor in the array - the sequence of the enumed names sets the sequence of the fields in the packet. Make the transmit routine responsible for the TX byte order of each uint16_t value, extracting the bytes with shifts and masks.

author: madires
author_rank: Super Contributor
timestamp: July 04, 2017, 04:48:31 pm
content: As we got a network byte order define a byte order for your protocol and convert to/from based on the endianness of the MCU or PC. For performance reasons I'd choose the MCU's endianness as byte order for my protocol.

author: Nusa
author_rank: Super Contributor
timestamp: July 04, 2017, 06:42:45 pm
content: Quote from: madires on July 04, 2017, 04:48:31 pmWhich is fine between KNOWN systems that are explicitly aware of underlying assumptions. Which are hopefully boldly documented in the code. There are so many programmers that aren't even AWARE of what endieness the current processor is using. It varies among micro families as well; it's not just a PC and intel-based Mac thing.My point is that the problem came up in the first place becausethe transmission packet was poorly defined, to the point of being different depending on the platform it was executing on. My example above was for illustration of a simple case. There are many variations on the theme. And yes, using the network byte order macros is one of those. Much of my career predates the existance of that standard, nor are they a total solution.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 05, 2017, 05:29:34 am
content: Quote from: AG6QR on July 04, 2017, 06:32:42 amwe cannot say either big or small endian is right or wrong, its inherent in the design of the hardware... btw, in avr chip there is register setting to store 10/16bits adc value as right or left aligned in adc register, setting this appropriately, and you dont have to change your code a thing. i'm not sure about pic18... otoh sending in ascii (of hex ascii) format have some good points as mentioned, but also bad points that needs to be aware of, the transmitted data will double the size (create some issue in large data transmission, slower etc) and have translation processing (binary to ascii) penalty, so not suited if higher speed acquisition application is required. ymmv...

author: legacy
author_rank: Super Contributor
timestamp: July 05, 2017, 09:43:12 am
content: OT:Code:uint32_t data=0x01234567;Little endian Big endian+----+----+----+----+ +----+----+----+----+|0x67|0x45|0x23|0x01| |0x01|0x23|0x45|0x67|+----+----+----+----+ +----+----+----+----+byte: 0 1 2 3 0 1 2 3Little Endian(e.g. how it appears on x86) vsBig Endian(e.g. how it appears on m68k and PowerPC)Recently a friend wanted to realize a 68K clone in Little EndianQuoteAnd people claimed thatQuotemore elegant? more sensible? more-UAT-W-h-a-t?My mind goes back when people programmed directly in machine codes and their program-lists where just a tabs of hex numbers. Oh, it's so clear to me WHY Motorola and IBM have always enjoyed the BigEndian format: because it makes it natural, since for human beings, numbers usually start with the higher part on the left.FSCKOFF Intel!!!And their silly decision to "make it different", reinventing the wheel in Little Endian.btw, now why on the why ARM wants to be LE(1)?!?!?Business decision? Marketing? Intel alliance?(as initially intel wanted to produce arm chips)Probably technical decisions are like blowing in the wind, but for sure we are still (and we forever will be) annoyed by their choiceedit:(1) yes, ARM-chips have a trick somewhere in the chip-configuration which allows them to switch from LE to BE. Some MIPS modern chips (e.g. MIPS32/R2) have a similar trick as well as the old MIPS-R3K used by SONY for their PlayStation1. The chip is configured as LE/BE (firmware choice) as the first instruction executed when the power on. But it's not the point.

author: nfmax
author_rank: Super Contributor
timestamp: July 05, 2017, 10:02:31 am
content: It seems people are missing the point here. The byte endian-ness is a property thatmustbe specified in the communication protocol between the two systems, and the implementation of the protocol in each systemmustbe responsible for converting to & from the native machine format. Just using 'text format' doesn't get round the problem - the native text encoding might be different in each system: 7 bit ASCII, UTF-8, UTF-16 or even EBCDIC! Even things like the bit order (for serial protocols) and polarityhaveto be specified.No-one seems to have picked up that when sendingstructs, the member alignment also has to be specified by the protocol.

author: helius
author_rank: Super Contributor
timestamp: July 05, 2017, 10:09:41 am
content: The reason probably has to do with little-endian I/O devices. Your processor can have the capability to switch between different endian modes for each I/O device and memory (every PPC since the 601 has had it), but it needs to be supported by software with the appropriate instructions. If device DMA goes straight to memory, you have additional problems.There are some algorithmic differences between big and little-endian. Certain algorithms (for searching or parsing) prefer a particular byte order.

author: Nusa
author_rank: Super Contributor
timestamp: July 05, 2017, 01:56:26 pm
content: Intel didn't reinvent endianness...they were microprocessor pioneers, just like the other early developers, and that's simply the choice they made. Others made different choices and things developed in parallel. Intel's dominence in the PC platform came much later.Nor are big and little the only results of history...there are also mixed-endian implementations, the most prominent of which is probably PDP-11 (0x12345678 stores as 0x34127856). Realize nothing was set in stone back then...not even the size of a byte. My first programs were for the CDC cyber series with 60 bit words, 12 bit bytes, and 6 bit characters.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 05, 2017, 07:08:52 pm
content: Quote from: Mechatrommer on July 05, 2017, 05:29:34 am8 bit PICs arestrictly8 bit. All instructions work on single Bytes only, so you can choose whichever endian you like. When I started using PICs I chose big-endian because that is what I was most familiar with (on 68000). However XC8 uses little-endian exclusively, so in this case thecompilerhas made the choice for you.If you code properly it's not a problem. But if you do stuff like this:-Code:UART_Write_Packet((uint8_t *)&packet,sizeof(Packet_t));then you shouldn't be surprised when it doesn't turn out quite like you expected...

author: Ian.M
author_rank: Super Contributor
timestamp: July 05, 2017, 07:26:03 pm
content: Strictly speaking, the 8 bit PIC hardware has a (mild) bias towards little-endian because there are various peripheral register pairs in little-endian order that are best treated as single 16 bit entities.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 05, 2017, 10:37:15 pm
content: Quote from: Ian.M on July 05, 2017, 07:26:03 pmSome may be, but others need to be treated as separate 'High' and 'Low' registers. For example in the PIC12F675 ADRESH is at 0x1E and ADRESL is at 0x9E - the same address but different banks. TMR1 is little-endian, but Microchip recommends reading and writing to TMR1H and TMR1L separately - in big-endian order.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 05, 2017, 11:29:50 pm
content: Quote from: Bruce Abbott on July 05, 2017, 10:37:15 pmthere is no atomic opcode in 8bit mcu for reading/writing /updating 16bit value (even if one thinks it is when coding in c) , and certain read/write order is needed due to how the 16bit 8bit register pair is non atomicness updated, not due to endianness. Avr may have different read order vs pic even if they have same memory layout... it could be, it could be not. afaik.

author: Ian.M
author_rank: Super Contributor
timestamp: July 06, 2017, 12:05:39 am
content: Yes. some registers e.g the TMR0H/L pair are latched, and you must either explicitly access them 8 bits at a time as per datasheet, or use the compiler's access macros for that register pair which are guaranteed to perform the reads or writes in the correct order. If the compiler's access order is ever changed, the compiler team will update the macros to match.

author: sangar
author_rank: Regular Contributor
timestamp: July 06, 2017, 07:18:54 am
content: Hi,XC8 compiler manual says that it does not support for big endian format.Please refer the attached image.Well.What I understood is that it's always best to correct the endianess at Our application in the PC. At PC side, I am going to use labview. I will do this correction there itself and get back to you.Thanks,Sangar

author: NorthGuy
author_rank: Super Contributor
timestamp: July 06, 2017, 01:34:16 pm
content: Quote from: sangar on July 06, 2017, 07:18:54 amOf course, doing things on PC is always better - much more computing power.However, you get two bytes from the sensor. It doesn't really make any difference in what order you save them. Why not to save them in little-endian order right away and avoid the problem altogether?

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 06, 2017, 02:55:20 pm
content: Quote from: sangar on July 06, 2017, 07:18:54 amFinally we get to find out what your 'problem' is! PC's generally use little-endian. Labview is an exception.Writing Binary Files with LabVIEW That Can Be Read by Other ApplicationsQuote

author: alm
author_rank: Super Contributor
timestamp: July 06, 2017, 03:03:33 pm
content: That applies to writing binary files. Does it also apply to sending/receiving binary data over RS-232?

author: cv007
author_rank: Frequent Contributor
timestamp: July 07, 2017, 01:27:47 am
content: Wouldn't it be fun if serial ports were bit-endian to whatever each hardware maker decided to use? (add one more parameter I guess- 8 N 1 L)It appears the OP is sending one 'packet' of the packet_t struct, but how would the pc side know where each 'packet' starts/stops? If the pc side is simply looking for x bytes or x amount of time between 'packets', then it could get 'out of sync' in any number of ways.I would guess that the next step after the trivial problem of byte order is taken care of, is a 'frame' to enclose the 'packet' so the pc side can tell where each value belongs.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 07, 2017, 01:34:20 am
content: Quote from: alm on July 06, 2017, 03:03:33 pmObviously, because when data is sent or received via RS232 the order of individual Bytes in it doesn't change (or at least they better not!). In this case the data being sent is a block of memory. This can be received by Labview as a binary 'string', then converted to big-endian numbers using the 'Unflatten' function.Quote from: cv007That's another problem to solve. Labview has a timeout function, so just having a large enough gap between transmissions might be enough.

author: John Coloccia
author_rank: Super Contributor
timestamp: July 09, 2017, 12:14:37 pm
content: Quote from: legacy on July 05, 2017, 09:43:12 amIn addition to little endian historically being more efficient to implement in hardware (especially math), big endian has a crazy pathology that the same variable, cast to different widths (i.e. 32 bits vs 16 bits) has a different pointer address. Totally crazy that to know where something starts, you need to know how big it is. The only advantage to big endian I can think of is that it's potentially more human readable. I say potentially because after a little while, you just read it as naturally as anything else.And this is coming from someone that's spent most of his embedded development on big endian machines. The real question in my head is why big endian ever existed in the first place, not the other way around.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 10, 2017, 04:28:56 pm
content: Quote from: John Coloccia on July 09, 2017, 12:14:37 pmOnly if the pointer points to thefirstbyte in the word rather than the last. But it's really not a big deal. Adding an offset to the address is a trivial detail that is taken care of by the compiler.Apart from debugging or doing crazy stuff with pointers, the programmer should not care about (or rely on) the internal representation of numbers.QuoteTotally crazy that know how big a number is, you need to know how 'big' it is.Quote.esle gnihtyna sa yllarutan sa ti daer tsuj uoy ,elihw elttil a retfa esuaceb yllaitnetop yas I .elbadaer namuh erom yllaitnetop s'ti taht si fo kniht nac I naidne gib ot egatnavda ylno ehT

author: John Coloccia
author_rank: Super Contributor
timestamp: July 11, 2017, 10:22:30 am
content: Quote from: Bruce Abbott on July 10, 2017, 04:28:56 pmA pointer pointing at anything but the lowest addressable byte would be even weirder.

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: July 12, 2017, 09:40:16 am
content: Quote from: John Coloccia on July 11, 2017, 10:22:30 amThe point is you shouldn't make any assumptions about how the individual Bits of a variable are stored in memory.Here's the allocation in memory of Words and Bytes for the TMS9995 (used in the Texas Instruments TI-99/4A home computer). Notice something weird?

author: Nusa
author_rank: Super Contributor
timestamp: July 12, 2017, 10:33:49 am
content: Quote from: Bruce Abbott on July 12, 2017, 09:40:16 am16-bit processor, big endian, 16-bit word-aligned memory on an 8-bit memory bus. Sounds typical for what it is, not weird. Modern processors may align on 32-bit or 64-bit boundaries for some data types.But his point is that block transfers of memory should not be considered compatible between unknown machines, even if the endianess is known to be the same. Compilers will pad structure definitions as required to maintain the alignment the processor requires.

author: nfmax
author_rank: Super Contributor
timestamp: July 12, 2017, 11:53:52 am
content: Ah yes, the numbering of the individual bits is back to front in TI devices. Brings back memories!