passing argument 1 of 'memcpy' discards 'volatile' qualifier from pointer target - Page 1
Forum section: electronics / general

author: Simon
author_rank: Unknown
timestamp: August 20, 2024, 03:48:30 pm
content: So, we had quite the discussion last time I meddled with memcpy(), I fixed my mess and things ran OK. So now I am rewriting my code with hindsight.This is can bus and frankly the only sensible way to move "data" around of which the type is unknown is with memcpy(). So I am still using the trusty SAMC micro controller which has 2 MTTCAN controllers from bosch. The standard header files from the CMSIS give type definitions of structures that represent the message and filter elements that are to be put into the RAM space. All of these types are qualified with __IO which basically is just a definition of volatile.So, the manufacturer has done this and I stick with their recommendation, ah, but! its that same old problem, ya can't memcpy() with volatile stuff, or at least the theory is that it's not such a good idea. So I even tried creating a buffer array and memcpy()'ing my data to there to then copy it variable to variable in the "volatile" buffer. Oh there was some other warning so I reasoned that I would go back to memcpy().Basically you just can't do this without upsetting the compiler to warning level. Now usually I take warnings seriously as they have in the past been an indicator of further actual errors that have not been completely unearthed by the compiler and come to light on trying to fix it.But in this case I have two options, remove the volatile qualifier or just live with the warning, I know it is OK because I know how the system works. The volatile qualifier seems to make sense in that all of this stuff is accessed either by hardware so its use is invisible to the compiler or during an interrupt.Now I did solve this issue on the last system by taking out volatile qualifiers using my own definitions of the ram buffers and the compiler did not optimize them out, but then if I just leave those in and this time and accept the pedantry of the compiler I have a better chance that nothing will go wrong in terms of mis optimizations.

author: IanB
author_rank: Super Contributor
timestamp: August 20, 2024, 04:20:27 pm
content: You can, usually, use a #pragma in the code to disable a specific warning instance if you have decided you are going to accept the code as written. Does your compiler have this?

author: tellurium
author_rank: Frequent Contributor
timestamp: August 20, 2024, 04:45:50 pm
content: memcpy() is general purpose, and may use optimisations based on size and alignmentHow about making a custom copy function that uses `volatile uint32_t *` as source/destination, instead of `void *` ?

author: ataradov
author_rank: Super Contributor
timestamp: August 20, 2024, 05:49:05 pm
content: Don't use memcpy() for volatiles. This warning is legit (as most warnings from modern compilers), ignoring it is just hiding the possible issue.Make a custom version that operates on volatile types.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 20, 2024, 05:57:00 pm
content: Quote from: ataradov on August 20, 2024, 05:49:05 pmYeah, this irritates me. Writing 100% equivalent custom version of something with just a differently qualified arguments feels so stupid, because there are many advantages to be had from using the standard one: compiler caninternallyimplement it, maybe without a function call at all, like taking alignment stuff into account "inline". Everyone knows the interface, and it is well tested. Even if library call is emitted, it is an optimized one. And you are already using that memcpy elsewhere in your code so need to spend the ROM twice for basically the same thing.Yet, if you just cast the volatile away (explicitly or implicitly), indeed it ispossiblethat the compiler does a wrong thing. By careful testing, or by looking at assembly listing, you could verify it happens to work, but then it's in risk of breaking when compiler is updated.I would want to have at least something like builtin_volatile_memcpy().

author: ataradov
author_rank: Super Contributor
timestamp: August 20, 2024, 06:45:03 pm
content: Quote from: Siwastaja on August 20, 2024, 05:57:00 pmBut this is exactly what you don't want in case of volatile. And you can't really make memcpy() to be a part of the compiler instead of a standard library.Quote from: Siwastaja on August 20, 2024, 05:57:00 pmI agree that it would be nice to have a standard volatile version. And a separate function would be the only way of doing this. But this is up to the C standard body, and they don't move very fast.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 20, 2024, 07:06:01 pm
content: I don't really get what the semantics of memcpy_volatile are supposed to be. Obviously load and store exactly once from each address in the source and destination. But: in normal memory order? Is reverse OK? Should it be 8, 16, or 32 bit loads and stores? If the buffer is not aligned can you do 8 bit loads and stores at the ends and 32 bit in the middle?If the answer to all this is "I don't care" then just write a 2 line naive bytewise memcpy_volatile implementation. If that's not OK either from a performance or correctness standpoint then it should be clear why such a function would be difficult to include in the standard library. In that case you should make a custom function anyway, and call it "copy_can_buffer" or something to make it clear it's a special purpose volatile copy.

author: ataradov
author_rank: Super Contributor
timestamp: August 20, 2024, 07:19:40 pm
content: This is a good point. I personally think that in-order byte by byte copy would be the most "correct". If this does not work in some case, then the custom version is a way to go.

author: bson
author_rank: Unknown
timestamp: August 20, 2024, 07:34:27 pm
content: If you discard the 'volatile' qualifier and do multiple memcpy's to the destination the compiler may drop all except the last if it can tell they exactly overlap. It can also reorder or change it if you later modify parts of it, or overwrite parts explicitly initialized before.You need two memcpy's: one where the compiler is free to optimize to its heart's content, and one which forms a memory barrier, where the compiler can't remove or reorder. Removing the volatileness is NOT a solution and can yield significant grief - other than as part of creating a wrapper to implement a volatile version which still uses the builtin underneath.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 20, 2024, 07:53:04 pm
content: Quote from: bson on August 20, 2024, 07:34:27 pmActually, if the data is for DMA buffers rather than direct memory mapped IO -- that is, reading or writing to them doesn't have immediate side effects, then the real solution is indeed to remove volatile. If you do an atomic load-acquire to fetch buffer pointer (or whatever you need to know you own that buffer for writing), memcpy the data into the buffer (using ordinary memcpy on non-volatile pointers), then do an atomic store-release to mark your transaction done and to start the IO. Then the compiler will know to do exactly the correct thing: memcpy with whatever optimizations it wants, but keep it in bounds of the load/acquire regions. It's even free to combine reads and writes, as long as it is combining them from outside the critical section to inside. It will also generate hardware memory barriers as needed, but even if those aren't needed, such as if the memory in question is IO memory that is not cached by the hardware then the atomic operations will direct the compiler to do what you actually want.

author: glenenglish
author_rank: Frequent Contributor
timestamp: August 20, 2024, 07:59:37 pm
content: Putting aside vagarities of memcpy().....If you are copying bytes into IO space, especially, an IO peripheral, you are well advised to do as the peripheral expects.Example - one microcontroller peripheral controller I used - this needed byte by byte access to the buffer, and so an optimized wide write using (16 bit architecture) 16 bit writes did not work to trigger to the send, it wanted to see a byte write to the last location in a buffer. The 16 bit (two byte) write at-a-time did not trigger the send. I ended up with 16 bit writes for all but the last byte pair.But, of course this is easily tested.In the op's case, I would use a customized copy routine to be sure the solution was peripheral compatible and only use memcpy for memory space..

author: Simon
author_rank: Unknown
timestamp: August 20, 2024, 08:46:16 pm
content: Quote from: glenenglish on August 20, 2024, 07:59:37 pmthing is this is a filter element, I only write once at the start and it's useless until it's written, then it's under the control of the hardware...... why it had to be volatile I do not know. But the next fun bit will be the buffers, sounds like I should write my own memcpy(), in the past I was told that this is a bad idea but I assume this hits the limit where needs must and it's for this specific purpose.

author: IanB
author_rank: Super Contributor
timestamp: August 20, 2024, 08:52:36 pm
content: Ultimately, as others have mentioned, I think it depends on whether it is a DMA buffer, or a set of I/O registers.If it is a DMA buffer, a suitable approach is to set the appropriate flag in the I/O controller to lock the buffer so that the DMA will not access it until you are ready, then use normal memcpy on the buffer. After you have read the buffer (or written to it), then you can clear the I/O flag and let DMA access proceed.If the memory addresses are I/O peripheral addresses, then you have no choice but to read and write to the registers in the manner that the peripheral demands.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 20, 2024, 08:58:40 pm
content: If it's a DMA buffer, why not use a DMA memory -> memory transaction to copy data to it? It will not require any CPU (apart from setting up the transfer) and is guaranteed to do what you are expecting. (Unless the source data is in a memory area that can't be used by the DMA controller.)Otherwise, yes, write a custom copy function. I think this has already been discussed in other threads in great lengths and it keeps creeping back.As it's going to be a custom function specific to a particular case, contrary to a full-fledged optimized memcpy() (that will try to optimize for all cases), you don't need to make it as convoluted in terms of optimization. Just copy by the largest width you can (depending on alignment and size) and done. A couple lines. A lot less work than even reading a thread that is likely to become tens of posts long.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 21, 2024, 03:48:38 am
content: Volatile only guarantees that the accesses to variables are performed (as opposed to optimised out) and preserves the order of accesses if such order is mandated by C standard. For example, if you write data to a DMA buffer, the compiler may notice that the data is never read back and optimize writes out. Technically, the compiler can optimize the whole memcpy() out in such case. Volatile qualifier prevents such optimization. Although, in practical terms, the optimization probably won't happen, not until the developers of C compilers reach the next level of optimization paranoia.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 21, 2024, 05:02:50 am
content: Quote from: SiliconWizard on August 20, 2024, 08:58:40 pmI don't know exactly how Simon's library works, but one reason would be if you need to add or modify the data (for instance add headers or checksums) before sending it to the DMA. Or if you just want to make a private copy of the data passed in before returning to the calling function so it doesn't stomp on the buffer during the DMA process. Sure, zero copy is great if you can do it, but it's not always worth the trouble.The other condition would be not really DMA, but if the frame buffer is actually in the peripheral, but it doesn't do anything until triggered by writing a control register with the frame length. Then, a non-volatile memcpy into that buffer followed by an atomic store to the control register might be a correct implementation.

author: ataradov
author_rank: Super Contributor
timestamp: August 21, 2024, 05:05:00 am
content: It does not make a lot of sense to use DMA for this. Those messages and filter elements are tiny, getting DMA going would take more time than just copy a few bytes.

author: cv007
author_rank: Frequent Contributor
timestamp: August 21, 2024, 05:44:16 am
content: It looks to me the compiler does see volatile even when casting it away-https://godbolt.org/z/v5ez78dqhIf so, it could be the solution is as simple as casting both arguments to void* provided one of them is a volatile type (struct type has to be volatile, not just its members).

author: ataradov
author_rank: Super Contributor
timestamp: August 21, 2024, 05:48:29 am
content: But the next version might not see it. And you will be debugging that stuff for ages. Just write correct programs, there is a lot of crap code out there already.The amount of text typed in this thread is enough to cover a lifetime supply of volatile memcpy()s.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 21, 2024, 06:03:39 am
content: Quote from: ataradov on August 20, 2024, 06:45:03 pmWut, why not? Of course compiler needs to respect the volatile qualifier, but that does not mean there is nothing to optimize. At very least, function call overhead can be optimized out. Especially when the size parameter isn't a volatile qualified variable but a compile-time constant value can be derived, compiler can e.g. unroll the loop if the copy length is constant and small.QuoteAhem, it already is, unless you prevent the compiler from using a builtin to replace it with command line flags. C standard says nothing about compilers having to emit linker symbols to external link-time libraries.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 21, 2024, 06:10:07 am
content: Quote from: ejeffrey on August 20, 2024, 07:06:01 pmYeah, there would be no one correct answer. In my opinion: memcpy_volatile must access the whole input and whole output range (possibly: exactly once), but size of access and order of access do not matter.If user has something super specific e.g. make 16-bit loads and 8-bit stores in reverse order, it makes no sense to write gazillion of different memcpy_volatile()'s, or add any configuration parameters; at that point, you just write the code yourself, probably just ad hoc in-the-place because then the code showswhat actually happens, and you of course add commentswhy: e.g.// SPI peripheral requires 16-bit writes to produce 16-bit transactions separated by nCS toggling (generated by hardware)But I still see use for generic memcpy_volatile, which simply guarantees reading all input and producing all outputwithin that callso that these memory operations are finished when the function returns.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 21, 2024, 06:25:28 am
content: Apart from doing exactly what you want, writing your own ad-hoc memcpy has the potential benefit (just define it static and/or inline) of getting inlined, while if you call memcpy() from the std library, compilers will usually only inline it if the size is known statically (at compile time) and it's small enough. Otherwise it will just call memcpy(). Not that it should make much difference in practice unless you call it repeatedly in a tight loop.As to a standard memcpy_volatile(), it may indeed not be the best naming, for reasons given above. I think what the OP (and probably many use cases) would like is a memcpy_something() taking volatile-qualified arguments, guaranteeing that the entire size of memory passed is actually copied in all cases, but not caring for order or specific alignment (as long as it uses valid accesses for a given target).There would still be cases for which you'd need an ad-hoc function.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 21, 2024, 06:48:39 am
content: Personally, I'd create an include file,memcopyv.h, containingCode:#ifndef MEMCOPYV_H#define MEMCOPYV_H#ifdef __cplusplusextern "C" {#endif// SPDX-License-Identifier: CC0-1.0#include <stddef.h>#include <stdint.h>static inline void memcopyv_asc8(volatile void *dst, const void *src, size_t bytes) {volatile char *d = dst;const char *s = src;const char *const z = s + bytes;while (s < z)*(d++) = *(s++);}static inline void memcopyv_asc16(volatile void *dst, const void *src, size_t bytes) {volatile uint16_t *d = dst;const uint16_t *s = src;const uint16_t *const z = s + bytes / sizeof (uint16_t);while (s < z)*(d++) = *(s++);}static inline void memcopyv_asc32(volatile void *dst, const void *src, size_t bytes) {volatile uint32_t *d = dst;const uint32_t *s = src;const uint32_t *const z = s + bytes / sizeof (uint32_t);while (s < z)*(d++) = *(s++);}static inline void memcopyv_asc64(volatile void *dst, const void *src, size_t bytes) {volatile uint64_t *d = dst;const uint64_t *s = src;const uint64_t *const z = s + bytes / sizeof (uint64_t);while (s < z)*(d++) = *(s++);}static inline void memcopyv_desc8(volatile void *dst, const void *src, size_t bytes) {volatile char *d = (volatile char *)dst + bytes;const char *s = (const char *)src + bytes;const char *const z = src;while (s-- > z)*(d--) = *s;}static inline void memcopyv_desc16(volatile void *dst, const void *src, size_t bytes) {volatile uint16_t *d = (volatile uint16_t *)dst + bytes / sizeof (uint16_t);const uint16_t *s = (const uint16_t *)src + bytes / sizeof (uint16_t);const uint16_t *const z = src;while (s-- > z)*(d--) = *s;}static inline void memcopyv_desc32(volatile void *dst, const void *src, size_t bytes) {volatile uint32_t *d = (volatile uint32_t *)dst + bytes / sizeof (uint32_t);const uint32_t *s = (const uint32_t *)src + bytes / sizeof (uint32_t);const uint32_t *const z = src;while (s-- > z)*(d--) = *s;}static inline void memcopyv_desc64(volatile void *dst, const void *src, size_t bytes) {volatile uint64_t *d = (volatile uint64_t *)dst + bytes / sizeof (uint64_t);const uint64_t *s = (const uint64_t *)src + bytes / sizeof (uint64_t);const uint64_t *const z = src;while (s-- > z)*(d--) = *s;}#ifdef __cplusplus}#endif#endif /* MEMCOPYV_H */and include it whenever I need one of thememcopyv_ascN()ormemcopyv_descN()functions.Theextern "C" {,}are only for us humans, to indicate the code works exactly the same when compiled with C or C++ compilers. It only affects linkage, which here is strictly local;static inlinehas the exact same results in both C and C++.Why do it this way?This expresses the developer intent to the C or C++ compiler in an unambiquous way. Whenever used, the function implementation is visible to the compiler, and it can tell exactly what is done to the target buffer (which isvolatile). In practice, all eight functions behave the same asmemcpy(dst,src,bytes), except that the_ascNvariants do so always in increasing order in units ofNbits, the_descNvariants do so always in decreasing order in units ofNbits, and if(bytes% (N/8)) != 0(i.e.,bytesis not a multiple ofN/8), the trailingbytes% (N/8)bytes are not copied.Moreover, it is commonly observed code, copyingconstdata to avolatilearray. It is unlikely that future compilers will optimize this wrong due to additions to the C or C++ standards, because it would break too much existing code.Optimally, we'd want a generic version,memcopyv(dst,src,bytes), ormemcopyv_asc()andmemcopyv_desc(), that do the right thing depending ondstandbytes. Unfortunately, the "right thing" depends on the compiler and on the target architecture. As written above, the best guess on 32-bit is to usememcopyv_asc32(), noting thatbytesmust be a multiple of 4 as trailing bytes will not be copied.If using GCC or Clang or compatible C or C++ compilers, one can use the__builtin_constant_p(expression)built-in to determine if the value ofexpressionis known at compile time. Then, one can assumesize_tis of native register size, and ascending order acceptable copying order, yieldingCode:#ifndef MEMCOPYVC_H#define MEMCOPYVC_H#ifdef __cplusplusextern "C" {#endif// SPDX-License-Identifier: CC0-1.0#include <stddef.h>#include <stdint.h>static inline void memcopyvc(volatile void *dst, const void *src, const size_t bytes) {if (__builtin_constant_p(bytes) && (bytes % sizeof (size_t)) == 0 &&__builtin_constant_p(dst) && ((size_t)dst % sizeof (size_t)) == 0 &&__builtin_constant_p(src) && ((size_t)src % sizeof (size_t)) == 0) {volatile size_t *d = dst;const size_t *s = src;const size_t *const z = s + bytes / sizeof (size_t);while (s < z)*(d++) = *(s++);} else {volatile char *d = dst;const char *s = src;const char *const z = s + bytes;while (s < z)*(d++) = *(s++);}}#ifdef __cplusplus}#endif#endif /* MEMCOPYVC_H */That generates acceptable code for various architectures, even for AVR (using GCC-5.4.0; newer versions of GCC generate lots of extra code within the inner loop for AVR). However, it still needs to bestatic inlineand included in a header file, so that the compiler will implement the correct version.For a library function version, we don't need any GCC/Clang specific built-ins, as we need to do the test anyway to see if "native word" sized copy is allowed, or whether it has to be done character by character. Since memcpy() direction is unspecified, we can assume increasing addresses is okay, arriving at something likeCode:// SPDX-License-Identifier: CC0-1.0#include <stddef.h>#include <stdint.h>#ifndef NATIVE_WORD_TYPE#define NATIVE_WORD_TYPE size_t#endifvoid memcopyvc(volatile void *dst, const void *src, const size_t bytes) {if ((((uintptr_t)bytes | (uintptr_t)dst | (uintptr_t)src) % sizeof (NATIVE_WORD_TYPE)) == 0) {volatile NATIVE_WORD_TYPE *d = dst;const NATIVE_WORD_TYPE *s = src;const NATIVE_WORD_TYPE *const z = s + bytes / sizeof (NATIVE_WORD_TYPE);while (s < z)*(d++) = *(s++);} else {volatile char *d = dst;const char *s = src;const char *const z = s + bytes;while (s < z)*(d++) = *(s++);}}which generates only a very short preamble to verify the arguments are valid, doing the actual copy in eithersize_torcharunits in ascending order.The trick in such a library function is to minimize the preamble cost. If we assumesizeof (NATIVE_WORD_TYPE)is a power of two (1 for 8-bit, 2 for 16-bit, 4 for 32-bit, and 8 for 64-bit architectures), then we can binary OR all three parameters together, and just inspect whether the resulting value has the relevant least significant bits zero. Writing the test this way the preamble simplifies to just 4-8 instructions.On most architectures,size_tshould be the native word size. For any other architecture, one can simply define theNATIVE_WORD_TYPEmacro (via e.g.-DNATIVE_WORD_TYPE=intcommand line option if using GCC or Clang) when compiling to override.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 21, 2024, 06:51:59 am
content: You may be able to make this generic using C11's _Generic, but I'm not 100% sure the qualifiers (like volatile) are part of the type as _Generic() discriminates against. Something I'll have to check in the standard.(So I'm not sure _Generic is able to distinguish 'volatile void *' from 'void *', for instance.)

author: Simon
author_rank: Unknown
timestamp: August 21, 2024, 08:10:33 am
content: So this is what I have come up with, it has the added advantage of already working on 4 bytes at a time:Code:static inline void memcpy32v(volatile void *dest, volatile void *src, size_t n){// Typecast src and dest addresses to (uint32_t *)uint32_t *csrc = (uint32_t *)src;uint32_t *cdest = (uint32_t *)dest;// Copy contents of src[] to dest[]for (uint32_t i=0; i<(n/4); i++)cdest = csrc;}

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 21, 2024, 08:26:48 am
content: Quote from: SiliconWizard on August 21, 2024, 06:51:59 amYes, because_Generic()matches types based on the"compatible type"criteria (C11 6.5.1.1p2); and for twoconst,volatile, or_Atomic-qualified types to be compatible, they need to have identical qualifiers (C11 6.7.3p11).In other words,const void *,const volatile void *,volatile void *, andvoid *are all separate types (not compatible types with each other) in_Generic()type association list.The thing to avoid is array types. The types in the type association list must be complete types (so unspecified-length arrays are out), and eitherlvalues, function designators, or void expressions (C11 6.5.1.1p4, so arrays in general are out because they're notlvalues). If the argument is an array type, the corresponding pointer type (same-qualified pointer to first element) will match it.Quote from: Simon on August 21, 2024, 08:10:33 amYou should not omitvolatile, and the casts (fromvoid *) are not needed in C. That is, consider usingCode:static inline void memcpy32v(volatile void *dest, volatile void *src, size_t n){volatile uint32_t *csrc = src;volatile uint32_t *cdest = dest;for (uint32_t i=0; i<(n/4); i++)cdest = csrc;}instead.

author: cv007
author_rank: Frequent Contributor
timestamp: August 21, 2024, 11:51:36 am
content: QuoteThis assumes any caller of the function will only use these void pointer arguments with vars that are 32bit aligned. I think I would make a more specific function for your specific purpose with a more descriptive name of what you are doing, and keep the specific mem copy code inside the function for this purpose only-void init_can_ram(){ /*code*/ }instead of having a version of memcpy floating around that can be misused, where your unaligned access may not show up until days later when the vars happen to move to a different address. The '32' in the name may not be much help as it may look like 'speed' and the alignment part will be forgotten.QuotePossibly, but there must be a reason why gcc is not optimizing away memcpy even when volatile is cast away. The use of builtin's are allowing the compiler to optimize, where if using -fno-builtin-memcpy you will get the library call no matter what.When the compiler uses the memcpy builtin you can prevent it from optimizing away any use by making either the underlying source or destination volatile as the compiler appears to still 'see' the volatile attribute after casting it away (to conform to the argument types) . Use -fno-builtin-memcpy and the compiler now has to call the library function, and you get the same result (not optimized away) because the compiler cannot optimize the use of the library function (and now no need to make either underlying vars volatile).https://godbolt.org/z/z5a63sve9(-fno-builtin-memcpy is used in this case, and no optimizing away in any case)

author: ejeffrey
author_rank: Super Contributor
timestamp: August 21, 2024, 01:36:59 pm
content: Quote from: Simon on August 21, 2024, 08:10:33 amThat's mostly fine as long as you inputs are aligned multiples of 4 bytes, but you cast away volatile in your function body! It _probanly_ will still work but why would you do that? Make csrc and cdest volatile uint32 *.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 21, 2024, 01:57:01 pm
content: In terms of alignment, the arm cortex (edit: M4) processors, like most modern processors, support misaligned access in hardware. However it will turn into multiple read and write cycles at the bus level. This may or may not be a problem for any specific application but its worth documenting.

author: cv007
author_rank: Frequent Contributor
timestamp: August 21, 2024, 02:20:31 pm
content: QuoteM0/M0+ does not.

author: ataradov
author_rank: Super Contributor
timestamp: August 21, 2024, 02:38:42 pm
content: When it comes to peripheral access, it does not even matter if your core supports misaligned access, since the peripheral dictates that. And many would be very particular as to what they support. So, you are not getting away with a generic universal function no matter what.

author: Simon
author_rank: Unknown
timestamp: August 21, 2024, 03:49:50 pm
content: the destination and source variables need to be word aligned, this is already done for the ram variable types definition.

author: Tation
author_rank: Regular Contributor
timestamp: August 21, 2024, 05:07:12 pm
content: I still cannot see the point of using memcpy() when accessing peripherals. OK, a memory mapped peripheral can be accessed using memory accesses, but it is not memory at all and may not behave as memory, so you better write your ad-hoc functions to access it.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 21, 2024, 06:52:42 pm
content: Quote from: ataradov on August 21, 2024, 02:38:42 pmNot for everything, yes, but I don't think there is need. For example, consider the case where doing a 16-bit bus write on SPI peripheral data register creates one 16-bit transaction (delimited by continuous active chip select). Thus, two 16-bit writes creates two transactions separated by chip select inactive pulse, whereas 32-bit write would create one continuous 32-bit transaction; a different behavior.In this case, assuming you want two 16-bit transactions, and further assuming you know there is enough room in SPI FIFO, you do not want to write:Code:memcpy(SPI->DR, spi_data, 4);because that removes the volatile qualifier and does not specify operation size, which likely would be incorrect except by luck (e.g. spi_data being aligned(2) but not aligned(4) by pure coincidence and some sort of optimized memcpy decicing to do 16-bit writes.)But similarly, you don't want to writeCode:memcpy_volatile_ascending_16bit(SPI->DR, spi_data, 4);because that is excess modularization. That one would probably never get reused.But, you would just write, like any sane person:Code:for(int i=0; i<2; i++)*((uint16_t*)&SPI->DR) = spi_data;or, even better, fix the manufacturer's header file from:Code:volatile struct{...uint32_t DR;...} SPI;to:Code:volatile struct{...union{uint32_t DR32;uint16_t DR16;uint8_t DR8;};...} SPI;after which code can become pretty writable and readable:Code:for(int i=0; i<2; i++)SPI->DR16 = spi_data;Now it is obvious to any reader what happens and in what order. All that is left is to add a tiny comment for those unfamiliar with the MCU peripheral:Code:// With 16-bit bus writes, 16-bit transactions separated by nCS inactive pulse are generated

author: ataradov
author_rank: Super Contributor
timestamp: August 21, 2024, 07:01:20 pm
content: I don't think anyone suggested to use memcpy for regular register writes. It is mostly useful for buffers. And in that case your last example is the explicit version of the memcpy(). And in practice this is short enough that there is really no need for a dedicated memcpy() function.

author: bson
author_rank: Unknown
timestamp: August 21, 2024, 08:04:24 pm
content: Quote from: Simon on August 20, 2024, 08:46:16 pmLikely because you never used the result. It's like assigning to a unused variable - the compiler will remove the assignment AND the variable, or it can move it to an inner scope where it's actually used (like say in a switch or a branch of an if-else statement). It generally can't remove globals, but it can often rearrange assignments to them. At least until you create a production build version where you concatenate all your sources into a single fat source and compile it with -fwhole-file, at which point the compiler gets a view of the entire codebase and will know when and how globals are used - and suddenly without volatile it no longer works.

author: Simon
author_rank: Unknown
timestamp: August 21, 2024, 08:42:20 pm
content: Quote from: Tation on August 21, 2024, 05:07:12 pmThe fact is that I am accessing memory. The tx buffers, rx buffers and filter elements are stored in the RAM memory. As each of the two controllers can address up to 16kB each for these functions it was thought a waste of money to build into a micro controller with 16kB of RAM another 32kB of memory for a function that may not use all of it.

author: ataradov
author_rank: Super Contributor
timestamp: August 21, 2024, 08:48:02 pm
content: While those buffers are located in the regular SRAM, they are shared with the peripheral, so should be treated as volatile.

author: peter-h
author_rank: Super Contributor
timestamp: August 21, 2024, 10:34:09 pm
content: The ST-supplied .h files define all GPIO and peripheral etc ports as volatile.I had other problems. I used memcpy in a boot block which has no access to stdlib soCode:// Local version__attribute__((optimize("O0")))// prevent replacement with memcpy()static void x_memcpy (void *dest, const void *src, size_t len){char *d = dest;const char *s = src;while (len--)*d++ = *s++;}// Local version__attribute__((optimize("O0")))// prevent replacement with memset()static void x_memset(void *s, uint8_t c, uint32_t len){uint8_t * p=s;while(len--){*p++ = c;}}The above also disables the compiler identifying the loop structure and messing with it.

author: cv007
author_rank: Frequent Contributor
timestamp: August 22, 2024, 12:31:32 am
content: QuoteYou can still 'call' the builtins.Code://startup.cpp...//linker script symbolsextern "C" u8 _etext []; //end of textextern "C" u8 _srelocate []; //data (initialized)extern "C" u8 _erelocate [];extern "C" u8 _sbss []; //bss (zeroed)extern "C" u8 _ebss [];static voidresetFunc (){__builtin_memcpy( _srelocate, _etext, _erelocate - _srelocate );__builtin_memset( _sbss, 0, _ebss - _sbss );...Even in cases using -fno-builtin/-fno-builtin-func where you do want the library call instead of the builtin, you can still use the builtin by name when wanted.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 22, 2024, 01:01:39 am
content: Quote from: cv007 on August 22, 2024, 12:31:32 ampeter-h has a freestanding C environment, where those implementations are not available. That is, there is no object code that implements either the built-ins or the standard functions, because there is no standard C library implementation linked in. Yet, the compiler can still optimize code to a call tomemset()/__builtin_memset()/memcpy()/__builtin_memcpy()/memmove()/__builtin_memmove(). Disabling optimization for those alternate memset/memcpy-like functions is the simple way out. Other options do exist, including implementingmemcpy()/memmove()/memset()oneself.

author: westfw
author_rank: Super Contributor
timestamp: August 22, 2024, 02:22:33 am
content: QuoteIs the compiler really allowed to optimize away memcpy() calls where it thinks the destination is not used?I mean, that's pretty much at odds with the definition of memcpy(), isn't it?Does every function that the compiler MIGHT optimize to 'something else' going to need explicit documentation about whether it actually does what the original function did? (This whole idea of the compiler "recognizing" library functions seems like a really slippery slope. if I say printf() or memcpy(), I want it to call those functions in whatever library I'm using, not decide that it should use putstring or not bother to do anything...)

author: ataradov
author_rank: Super Contributor
timestamp: August 22, 2024, 02:31:35 am
content: Quote from: westfw on August 22, 2024, 02:22:33 amYes, sure. This is not going to be a direct optimization in many cases, but though a chain of replacements. GCC can already replace short and fixed length memcpy() by a series of load/store instructions. If it further detects that those store instructions store into the locations that are not used, those stores can be optimized away.Quote from: westfw on August 22, 2024, 02:22:33 amNot really, since there are no observable effects of the copy. At least as far as the compiler can see. This is why volatile exists.C standard always speaks in terms of observable effects. And any optimizations that are not observable by a valid program are a fair game.There is no need to document this, since there is no way to observe the difference. The only way you can know that something was optimized is by examining the generated code.If you can observe those side effects, then your real machine does not behave like an abstract machine in the standard, and this is not the standard's problem.

author: westfw
author_rank: Super Contributor
timestamp: August 22, 2024, 05:03:46 am
content: QuoteI'm just used to function calls being a "visibility barrier" to the complier.This thing where "function calls" and "language features" are hard to tell apart used to be blissfully absent from C, although I guess present in many other languages (Pascal, Fortran)Is there a list of functions that C thinks it has intimate knowledge of? Everything in libc? (is that even well-defined?)What's next? "I noticed you were writing a file to /tmp, which doesn't survive anyway, so I just didn't do it!"?

author: Siwastaja
author_rank: Super Contributor
timestamp: August 22, 2024, 05:13:02 am
content: Quote from: westfw on August 22, 2024, 02:22:33 amOf course. C is a high level language, has always been, even the early standards describe it that way (even if compilers of the time did not fully utilize the possibilities). Code describesbehavior(within limits of the language), not a form of machine implementation. Compiler is free to implement that behavior in any way it sees fit. There are limits to how freely it can do this, but my point is, it's best to start with an assumption there are none, than the opposite, very dangerous assumption.This is all very logical and simple to understand once you grok it. And if you want to produce memory accesses for something outside of the visibility of language rules, you use volatile. (Not all languages have this kind of qualifier at all.)So you need to switch your thinking into logical, high level programming mindset and understand you are writing standardized language recipe for a "virtual" machine. If you don't like that, don't use C, use a lower level language, although most other languages available are even higher level, not lower, so this pretty much limits you to assembly.

author: ataradov
author_rank: Super Contributor
timestamp: August 22, 2024, 05:17:49 am
content: Everything in libc. And I guess more generally, everything described in the C standard. Those functions have well defined behavior in terms of the same abstract machine, so compilers are free to substitute them. There does not even need to be a separate C library as a standalone package, compiler authors are free to implement it all embedded.There will be no "/tmp" optimizations, since they are not part of the C spec.At the same time I had a very interesting case, which I unfortunately can't reproduce anymore. And I don't think I'm crazy, so that probably happened. I was working on some function that inadvertently had undefined behavior at the very beginning, which rendered the whole function to be UB. The compiler replaced the whole body with a single "udf" instruction. This is a technically correct optimization, since all that code was UB anyway. I really need to find a way to reproduce this again.And this was a tremendous help for debugging this, since I've got an exception, which made me look at the disassembly, which was not there. I really wish compilers were more aggressive with optimizations and substitutions like this. Something like FPGA tools that have no issues throwing away a ton of logic because they see it is completely useless.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 22, 2024, 06:51:41 am
content: And what would be even more useful yet, like FPGA tools too: a complete report of optimizations done by the compiler, in a readable way (with "verbosity" options so you could pick the level of info you want to see).

author: cv007
author_rank: Frequent Contributor
timestamp: August 22, 2024, 07:21:18 am
content: QuoteQuoteIs this all dependent on the choice the compiler makes as to which memcpy it uses? For example-Code:extern "C" unsigned char _sbss [];extern "C" unsigned char _ebss [];//test1 - normal- let compiler decide to optimize memsetmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, 1 ); //optimized to a movs/strbmemset( _sbss, 0, 1 ); //optimized away//test2 - normal- let compiler decide to optimize memsetmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, 1 ); //optimized awaymemset( _sbss, 0, 1 ); //optimized awaymemset( _sbss, 0, 2 ); //calls a memset function//-fno-builtin-memsetmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, _ebss-_sbss ); //calls a memset functionmemset( _sbss, 0, 1 ); //calls a memset functionmemset( _sbss, 0, 1 ); //calls a memset functionmemset( _sbss, 0, 2 ); //calls a memset functionwhich appears to indicate as soon as the compiler starts using a callable memset (from libc_nano.a in this case), it no longer optimizes away its use.Maybe someone can explain the relationship between builtins/libgcc/libc_nano(for example).

author: Simon
author_rank: Unknown
timestamp: August 22, 2024, 07:28:55 am
content: Quote from: ataradov on August 21, 2024, 08:48:02 pmYes, which is why I asked in the first place. I didn't want to remove the volatile qualifier, I assume it was put there for a reason, which I beleive is that the compiler may well optimize it all out.Is there another way to stop the compiler from optimizing out variables? It seems to me that volatile is having to serve two purposes. On the one hand it is saying that this variable will be externally changed by something not in this program so don't optimize it out, but on the other hand I think the compiler makes other assumptions about it?In this case the volatility itself is not an issue and the compiler does not I think need warning of this as the access to the data is managed, the filters are only written by the user and only read by the can bus controller. The data buffers are only written by the controller and only read by the user. The users access to the buffers in a way that prevents reading live changing data is managed by the put and get flags a mechanism that the compiler is unaware of. So to stop the compiler optimizing out the variable as it does not see it's use we make it "volatile" but other connotations of volatile do not apply.

author: peter-h
author_rank: Super Contributor
timestamp: August 22, 2024, 07:31:40 am
content: Doing one's own functions and disabling optimisations on those is a great way to make sure code will do what you expect. Assembler is the other way which is never optimised.With peripherals it should not matter if they are defined as volatile in the general .h file, as ST does. But I would still not use memcpy() there because you have no idea if it will do 1 byte at a time, 4 bytes, whatever. Actually the newlib libc one does just 1 byte at a time, which is sillyThe gotcha I found is the compiler replacing loop structures. The command line option -fno-tree-loop-distribute-patterns does not seem to work (GCC 11). Not sure where it came from; here I believe:https://stackoverflow.com/questions/46996893/gcc-replaces-loops-with-memcpy-and-memsetInteresting gotcha there: you cannot stop the compiler calling memcpy when copying structures. And if the destination structure appears unused, it might skip it. It's wonderful to be purist about this but it can do your head in. I've only been doing a lot of C for about 5 years and keep it very simple.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 02:20:49 pm
content: Quote from: Siwastaja on August 22, 2024, 05:13:02 amC started as a macro-assembler, and that what it still is. Plenty of high-level languages existed before C, but C became immensely popular exactly because it was similar to a macro-assembler, compared to real assembler, it made many things easier to write without giving up control at low level. BTW: That's why it is suitable for MCUs.The standard appeared long after the C birth and tried to describe C in virtual terms so that it could be used on any CPU. As a result C became a portable macro-assembler - to port it to a different CPU you only needed to re-write inline assembler parts (if any). That's how it is until present day.The unification allowed for controlled optimizations - the compiler could optimize register and storage assignments, program flow etc. This shaved from 20 to 50% of the code compared to un-optimized versions. This stage was complete in early 2000-th.By that time, there were many programmers who could use C, but could not use assembler. They didn't know what the macro-assembler is, and could not see the difference between C and other high level languages. Hence, the view of C started changing. C was thought of as a high level language, and consequently many useful macro-assembler features started looking unsafe and unneeded. Also, a safety frenzy, and optimization frenzy had began. This was accompanied by huge over-development in GCC which exacerbated problems.Surprisingly, the C standard withstood all that and you still (in 2024) can use C as a macro-assembler.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 22, 2024, 03:05:44 pm
content: Quote from: NorthGuy on August 22, 2024, 02:20:49 pmThis is a weird view of history, IMHO. In reality, the abstract machine concept, definitions of side effects, volatile qualifier (which would make no sense if the description of "macro assembler" was true) were already present in the first official standard, already in 1989.This is because, unlike you seem to think, C was primarily not developed for "embedded" development, microcontrollers etc. C was the language for writing Unix userland programs, text based and graphical. In fact usage of C in programming small embedded systems started gaining traction as late as in late 1990's. At that point, C had over a decade of use as a high-level abstract language.The fact that some of your practices work even worse on modern-day compilers does not prove that compilers are now somehow wrong, and your approach right. Your approach was wrong to begin with. Modern compilers do better job at broken code, e.g. as ataradov described, by replacing a broken piece of code by a single UDF instruction, so you can fix your code. And year after year, GCC (and especialy clang) has become better at emitting compile time warnings of non-standard-compliant or UB code , where old compilers just "did something" which might have worked or not, and often produced esoteric bugs.Quote from: westfw on August 22, 2024, 05:03:46 amYou don't have to fear - just RTFM.C has many advantages:* It is a relatively simple language, compared to monsters like C++. It is entirely possible for a normal human being to understand the whole language, and maybe remember 80-90% of the details by heart (and the rest are easy to look up when needed).* Unlike some others (Python, Rust), C has an official standard which you can refer to, and it is possible for a normal human being to read the whole standard if they so wish.* Unlike some others (Python, Rust), C does not change every other day.In fact, all of your "concerns" were already present in the language in 1989. So being a graybeard is no excuse.All you need to do is to correct your underlying misconceptions of C being some kind of "macro assembler", at least glance through the standard (or, at least read what those who have read it have to say, e.g. on this forum or stack overflow) and stop relying on undefined behavior in your programs.And I assure you, even in an embedded project 98% of the code is application logic, data management, user interfaces, all sort of that - nothing "hardware related" as such. Once you start treating C as a high level language, your code quality will also improve: you should not think how to instruct the CPU to work through C, you should describe what the program is supposed to do in C, and let compiler handle the implementation.Only a tiny part of your firmware actually deals with timing-critical stuff, or memory mapped IO, or similar. Using things likevolatilequalifier, you can still write these parts in C, of course. In some cases, you need to use a tiny bit of assembler; C compilers offer inline assembly for this reason.But when your misconception lives so deep, it is very very hard to admit and fix, and I don't expect it to happen, but at least we can help prevent others who are new to embedded programming from making this mistake from day one.Personally, I was able to weed out the misconception because I read useful comments on forums like this at the point where I had just programmed for less than a decade. And that dramatically helped me to stop wasting time in "strange problems". Some people on this forum complain how compilers are stupid and make their programming a difficult and tedious job. I suggest the problem is not in the compilers, but instead these people do something wrong. I don't have the same classes of problems since I dropped the concept of C being a "portable assembler".

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 04:06:15 pm
content: Quote from: Siwastaja on August 22, 2024, 03:05:44 pmC appeared first in 1972, which is 17 years before 1989. I, sort of, explained that in the part which you turned into "..." in the quotation of my post.Quote from: Siwastaja on August 22, 2024, 03:05:44 pmI didn't say such a thing. There was nothing embedded back in 1972 (not in reach of ordinary people anyway), and for a long time after that. But C is well suited for embedded development, so it was picked up when a need appeared. Also, BSD and Mac,Linux,Windows kernels all are written in C. This is because C is well suited for low level work.Quote from: Siwastaja on August 22, 2024, 03:05:44 pmYou know nothing about my practices. Probably not very wise to criticize what you don't know.

author: peter-h
author_rank: Super Contributor
timestamp: August 22, 2024, 04:21:27 pm
content: What would be more useful, I think, than a lecture on philosophy, would be examples of how to get bitten in the bum by this "C virtual machine can do what the hell it likes behind the scenes" concept.I have spent a lot of my time discovering these things, with the memcpy() memset() etc being one of the first. Fortunately those just crashed, so not too hard to track down.But optimisation isn't always thus. We had a thread on this recently, and constructs likeCode:uint32_t x_size = buf|(buf<<8)|(buf<<16)|(buf<<24);which would fail if the C integer promotion scheme was strictly implemented, are implemented "to work as expected".All this is a damn good reason tonotupgrade your tools unless absolutely necessary. And you can't anyway. There is a new GCC version once or twice a year (shipped with Cube IDE) and each one breaks something which needs time spent on it (mainly due to different defaults for command line options) and then somebody has to do a ton of regression testing which is near-impossible on a nontrivial product. GCC11 to GCC12 reduced my project by 1k on 450k so clearly the code is different... I have a life so went back to GCC11.QuoteProbably, but I can tell you (as a hw and sw designer in the 1980s, a colleague writing in IAR C and myself writing perf critical parts in asm, for IBM coax and twinax emulation boxes; google DP8344) that nothing less than completely obvious got optimised. So code ran as expected back then. It did do the integer promotion though, which crippled performance on a Z80 because everything got moved to BC DE HL so you were wasting your time trying to do byte based stuff, and together with some other crap, asm could be 10x faster.I'd say C introduced many more ways to bite one since those days. Maybe about 20 years ago? It is to be expected because GCC is a mature product, so the compiler devs can only move sideways, doing increasingly weird optimisations which have zero perf gain. How far sideways can they go while still within the C VM spec while screwing around with dev brains, I wonder. I watch the almost weekly MbedTLS project emails and they have been moving sideways for years.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 22, 2024, 04:54:05 pm
content: Quote from: peter-h on August 22, 2024, 04:21:27 pmC's problems are elsewhere. Like you mention, integer promotion rules are sometimes unintuitive, but these rules are from good old 1970's and this has nothing to do with the abstract machine, and even with all optimizations disabled and language forced to be this "macroassembler", those integer promotion rules still cause bugs.Same can be said about "implementation defined" things. Most have nothing to do with abstract machine.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 04:55:01 pm
content: Quote from: Siwastaja on August 22, 2024, 03:05:44 pmI always thought (and still think) that C is more of macro-assembler than a high level language. And I never had any strange problems. Judging by forum posts, people have "strange problems" when they don't know the syntax or didn't read the standard. I always write to the standard, and any compliant compiler must work. I haven't seen a non-compliant compiler for a long time, 20 years perhaps. Other than that, I can re-compile the code of a modern project with an old compiler (if it supports the part of course), and it will work just as well.Do you have an example of a problem you had because you thought of C as a macro-assembler?

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 05:07:59 pm
content: Quote from: peter-h on August 22, 2024, 04:21:27 pmIt should work as "expected" if buf is unsigned char and may not work as "expected" if buf is signed char. Assuming what you expect is:Code:uint32_t x_size = *(uint32_t*)(buf+4);

author: Siwastaja
author_rank: Super Contributor
timestamp: August 22, 2024, 05:11:59 pm
content: Quote from: NorthGuy on August 22, 2024, 04:55:01 pmI totally believe this; my comments of having problems were more directed to some others.QuoteSame kind of stuff peter-h discusses here repeatedly: having to adjust optimization settings pretty often to "make things work" - random behavioral changes when new variables are added or removed or other totally unrelated changes being made. I expected the compiler to kinda "directly" translate every line of C code into machine code, and was frustrated when this did not work out. It was helpful to stop thinking such correlation exists at all, and instead concentrate on writing code which is compliant with C standard, free of UB, and tries to describe intent in more generic way and let the compiler handle implementation details as much as it can.

author: peter-h
author_rank: Super Contributor
timestamp: August 22, 2024, 05:14:32 pm
content: QuoteYes of course one would not shift signed ints. Buf in this case is uint8_t.QuoteGetting personal is always a standard refuge of a certain type of personality

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 08:15:12 pm
content: Quote from: peter-h on August 22, 2024, 05:14:32 pmIt should work ok with uint8_t. The integer promotion only add zeroes to uint8_t (e.g. 0xb5 is promoted to 0x000000b5). Therefore the code you posted won't be affected by it.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 22, 2024, 08:33:03 pm
content: Quote from: Siwastaja on August 22, 2024, 05:11:59 pmIn my experience, such things happen when you have some code accessing wrong places (for example a stray pointer, or out-of-bounds access). When such code corrupts something, the result depends on what is corrupted. The effect may not be seen (for example corrupting unused memory is totally harmless), or may have severe consequences (if you corrupt important data or other pointers). When you change optimization levels, add or remove variables, or do other changes to your code, the memory map changes. The data being corrupted change accordingly, and therefore you see the problem come and go, or you see the symptoms change in various way.In such cases, if you can debug, you just select the case where the problem causes the worst damage and start debugging it without changing code (because if you start inserting new code the problem may disappear). It may be tedious, but it is always possible to track it back to the root cause.The worst thing you can do, is to select the case where there's no symptoms and decide that the problem is solved. Because there may be other effects which you don't see. And also because the problem may come back at any time. Therefore, the root cause of the bug must be found and fixed.

author: peter-h
author_rank: Super Contributor
timestamp: August 22, 2024, 08:34:32 pm
content: A couple of years ago, someone said thishttps://www.eevblog.com/forum/programming/why-is-loading-a-multi-byte-scalar-variable-from-some-address-so-complicated/msg4481104/#msg4481104QuoteThe issues I had were1) Higher levels would enable identification of loops and their replacement with e.g. memcpy, which blew up because my boot block module did not have stdlib. I don't think it is fair to take the piss out of somebody who got this. It could have happened to any expert programmer.2) I never actually got this problem but it is possible to violate things like /CS chip timings with faster code. When using e.g. an SPI FLASH, one drops /CS, runs the SPI, and when it is established that data is all shifted (best done by receiving own tx, if you are the master), raise /CS. Similar to 2-wire RS485 for example. Now, if your flash needs just say 20ns, you are extremely unlikely to violate this with a 168MHz CPU which is loosely coupled to a 21MHz SPI. And I check all this with a scope. But somebody careless, OR someone who (like everybody) lifted code from github, written 20 years ago for an 8MHz AVR, could find that code is very marginal or even dud. If the device needs a CS margin of say 1us (like some slow ADC or LED driver might need - STLED316?) then you can't use software delays; you have to do it properly, with an asm loop, using CYCCNT etc, but few coders know enough about hardware.Getting away from purist lectures (again) if you have a business selling some box, are you going to just tweak up the optimisation "because the C standard guarantees xyz" and ship 1000 boxes the next day??? NO. That would be stupid. It would be equally stupid to wind up optimisation levels on a product which is running fine and is selling. To do so would fly in the face of all business risk management, and nobody with a brain who has a stake in the company should do it.An employee might do it.

author: Tation
author_rank: Regular Contributor
timestamp: August 22, 2024, 09:09:50 pm
content: Quote from: Siwastaja on August 22, 2024, 03:05:44 pmIncidentally, this is, more or less, also my personal experience. Weird problems I encountered ended up being me hitting some UB because I expected C code being directly converted to assembly in some direct way, say: "a = b + c" must turn into "ADDS r0, r1, r2". Now I turned much more into the semantics of the code, not into some, guessed, assembly implementation of it. I suppose that learning and programming in ASM for years before learning C didn't helped me much...

author: peter-h
author_rank: Super Contributor
timestamp: August 22, 2024, 10:09:42 pm
content: Nobody, least of all an asm coder, should expect C to produce equivalent asm on a per-line basis. One should at least expect expressions to get folded.It almost does line for line if you turn off all opt, but then you pay for it in size - example:-O0 produces 491k-Og produces 342k <-- I use this, only-O1 produces 338k-Os produces 305k

author: cv007
author_rank: Frequent Contributor
timestamp: August 23, 2024, 01:54:30 am
content: QuoteNo one?Where do the builtins originate? They do not show up in a map file, and appear to be internal to the compiler. For example, the use of memset on a single byte (for example) will result in optimized code in the compiled object file (.o) IF neither -fno-builtin or -fno-builtin-memset are options specified to the compiler (whether you used 'memset' or '__builtin_memset').Like so, I think-switch( keyword)case "memset":if( -fno-builtin or -fno-builtin-memset ){ create a call to memset, linker provides address }else fallthrough to belowcase "__builtin_memset":if( looks_like_we_can_optimize_better_than_call ){ create optimized code }else { create a call to memset, linker provides address }Once the compiler decides to do the called version, the compiler cannot see what the called code may be so has to make the call and there can be no optimization later (excluding lto I suppose). So you can force the call via the no-builtin option, and you will get the library call 'no matter what'.It also appears you cannot get only the builtins to be used, because as soon as the compiler thinks it cannot optimize well enough (whatever that may be) it will generate a call (compiler has no idea there will be no such function available), and if the called symbol cannot be found the linker cannot link.Comments? Corrections?

author: ataradov
author_rank: Super Contributor
timestamp: August 23, 2024, 02:01:57 am
content: This is about right. Builtins are virtual functions that are implemented in the compiler itself. They don't actually exist as functions, just hints what code to generate. There are a lot of useful built-ins that don't correspond to the library functions.For example, __builtin_clz() will generate code to count the number of leading zeroes in the argument. On platforms that have a dedicated instruction for this, it will result in just one instruction. On other platforms, it will generate actual code. This will still be better than a native function, since that code is often hand optimized.And most of them are functions that already exist in the compiler for the purpose of the code generation, they are just exposed to the user.Many of them don't behave like normal function calls would. There are a lot of builtins that are oriented at specific sizes, so no integer promotion is performed, Compiler knows exactly what type was passed as an argument.

author: IanB
author_rank: Super Contributor
timestamp: August 23, 2024, 02:17:58 am
content: Quote from: cv007 on August 23, 2024, 01:54:30 amAs I understand it, the C standard library is part of the language specification.When you call a library function in your code, you can consider that to be like a keyword extension that the compiler recognizes, which has a defined behavior.The fact that such a library functionmightgenerate a call to an external library that gets linked in is one possible implementation, but such an implementation is not required, and if the compiler can replace it with inline code, it is allowed to do so.

author: cv007
author_rank: Frequent Contributor
timestamp: August 23, 2024, 02:54:41 am
content: QuoteAnd I assume this is why some of my earlier tests, where one of the arguments to memcpy was a volatile, subsequently cast away to conform to the memcpy argument type, ended up with the compiler still treating the argument as volatile and the builtin version was not optimized away (multiple consecutive statements). Although that does not hold true for memset, as it will optimize away multiple statements even if the first argument is a volatile.I use builtins for things such as avr delay cycles, swap, strlen, memcpy, va_arg, va_list, and so on. Probably most uses are simply to eliminate figuring out what header to include, but for library functions there is no other benefit to specifying the builtin until you use the -fno-builtin option and want to get an optimized version.I wonder what compiled code (sizes) would look like if they treated the builtins as something you have to specify to get. In other words no builtins as default. For something like an mcu where a small library is used it probably makes little difference- you will get to use any of the builtins when wanted, and for something like memcpy you no longer have to think about whether it gets optimized away.

author: brucehoult
author_rank: Super Contributor
timestamp: August 23, 2024, 04:32:11 am
content: Quote from: IanB on August 23, 2024, 02:17:58 amNo different to multiply or divide, or floating point. Or any other C built in operator that some CPU somewhere might not have as an instruction. Though those (which you don't need a#includebefore using) tend to be in libgcc, not libc.

author: peter-h
author_rank: Super Contributor
timestamp: August 23, 2024, 06:23:43 am
content: QuoteMaybe... the newlib libc for 32F4 was actually written in C(an earlier thread has the details). This practice has existed since for ever; IAR was doing it in 1980s, resulting in incredibly slow float ops. Today's 32F4 double floats (the CPU does hardware single floats only) is all C code.Similarly memcpy is unoptimised byte at a time, and I can't think of a more obvious candidate for not just the usual optimisations but hand written asm.

author: westfw
author_rank: Super Contributor
timestamp: August 23, 2024, 08:46:37 am
content: QuoteThere will be some hand-optimized assembler, depending on the exact target. Gnu Cortex-M3 has assembler floating point functions, but Cortex-m0 doesn't (last time I checked), and relies on the generic C float functions (rather bloated and slow.) (thus "adding insult to injury.")Tracking this sort of thing down by browsing the compiler source code is ... pretty painful. I suppose it gets easier if you do it a lot.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 23, 2024, 08:59:42 am
content: Quote from: peter-h on August 22, 2024, 08:34:32 pmI did. The "wrong conclusion" refers to hardware needing a barrel shifter being efficient. It is the optimizations a C compiler is allowed to make according to the abstract machine rules that makes it efficient. And, in that thread,bufwas explicitly defined asuint8_t buf;.Each term of thebufarray is indeed promoted tointbefore the shift, and that is why it works (on architectures with 32-bit or largerinttype). If you wanted portable C99-or-later code, assuminguint8_torunsigned charelements inbuf, it would need to be written as(uint32_t)(buf) | ((uint32_t)(buf) << 8) | ((uint32_t)(buf) << 16) | ((uint32_t)(buf) << 24)or the equivalent form, dropping the extra parentheses that I added to make it easier for us humans to read; relying on the C operator precendence rules:(uint32_t)buf | (uint32_t)buf << 8 | (uint32_t)buf << 16 | (uint32_t)buf << 24Using GCC, this is optimized to a single load at optimization level 2 and higher on architectures where unaligned 32-bit loads are "fast"; using Clang, at optimization level 1 and higher. I have not investigated exactly which optimization options affect this, because I habitually use-O2.Now, if we hadvolatile uint8_t buf[]orvolatile unsigned char buf[], both GCC and Clang will generate single-byte loads, to satisfy the abstract machine rules wrt.volatile; it is not allowed to optimize out any of the accesses, change the type of the access nor to merge the accesses.On ARM Cortex-M using GCC 10.5.0 or later, the expression is then best written as((uint32_t)buf | ((uint32_t)buf << 8))| (((uint32_t)buf | ((uint32_t)buf << 8)) << 16)so one gets the pretty-close-to-optimalCode:ldrb r3, [r0, #4] @ zero_extendqisi2ldrb r1, [r0, #5] @ zero_extendqisi2ldrb r2, [r0, #6] @ zero_extendqisi2ldrb r0, [r0, #7] @ zero_extendqisi2orr r3, r3, r1, lsl #8orr r0, r2, r0, lsl #8orr r0, r3, r0, lsl #16bx lrOn older versions of GCC, and on Clang, this and the previous expression produce pretty much the same somewhat longer code than this, often with separate left-shift operations.Now, if one hasvolatile uint8_t buf;, then e.g.memset((uint8_t *)buf, 0, sizeof buf)is actuallyUndefined Behaviour(C99 6.7.3p5, C11 6.7.3.p6, C17 6.7.3p7, C23 6.7.3p7):"If an attempt is made to refer to an object defined with a volatile-qualified type through use of an lvalue with non-volatile-qualified type, the behavior is undefined."If you've followed my posts on this subject, you'll notice that I haven't told anyone "don't do that, because its undefined behaviour". As to why, I'll explain more in detail after the break line below.Quote from: peter-h on August 22, 2024, 08:34:32 pmNo, it isn't fair, and yes, it tends to happen to even experts.Yet, you do need to acknowledge that there are other possible solutions, some of which have been suggested to you. The most common one is to split the boot block module into a separate build, so that you could include local implementations ofmemcpy(),memmove(),memset(),memcpy()used and referenced only within this block.Turning off optimizations is the option that requires the least amount of effort, that's all. Saying that it is a poor solution to the problem isfair; piss-taking is not.Currently, GCC requires freestanding implementations to providememcpy(),memmove(),memset(),memcmp(), and possiblyabort(); see theStandardschapter in the GCC documentation for details.C23 added most of<string.h>to what freestanding environments are required to support, so in the future, the set of functions needed will includememcpy(),memmove(),memset(),memalignment(),memccpy(),memcmp(),memchr(),strlen(),strcpy(),strncpy(),strcat(),strncat(),strcmp(),strncmp(),strchr(),strrchr(),strspn(),strcspn(),strstr(),strtok(),strpbrk(), a crapton of bit manipulation functions in<stdbit.h>starting withstdc_, plus support for typeva_argandva_start(),va_end(), andva_copy()functions. Hopefully the compilers will provide most of those inline, and keep updating the list of what they will emit for freestanding code that does not explicitly invoke said functions.Quote from: peter-h on August 22, 2024, 08:34:32 pmThat would becarelessof the code.It is not rare to implement hotspots in code using extended inline assembly (GCC, supported as-is in C and C++ by Clang also). One of these is a nanosecond-range "no-operation" macro that generates a suitable number of assembly no-operation mnemonics or a loop of such, to implement very short delays. (On Cortex-M's with ARM DWT, like Teensy 4.x running at 600 MHz+, I've used the cycle counter (ARM_DWT_CYCCNT); and on x86 and x86-64, the time stamp counter (__builtin_ia32_rdtsc()), although those require you to know the desired delay in number of cycles, i.e. divide the number of cycles per nanosecond by the number of nanoseconds to wait, and round up.)Reducing optimization level to avoid having to implement such (say, as macros, likeDELAY_NSECS(20)), is just foolish.Quote from: peter-h on August 22, 2024, 08:34:32 pmThis is exactly why I do all my development and testingby defaultat-O2, or at-Osfor very memory-constrained MCUs.What I object to, in other words, is considering compiler optimizations something you "wind up". To me, they should be by default at-O2/-Os, because it is a tool, not a judge or a simplistic filter or line-by-line "macro assembler".Personally, I don't like putting the standard over the observable reality, so instead ofrelyingon the compiler to follow/exploit the standard to the minutest details, I tend to examine the generated machine code instead, to verify it does what I intended it to.I consider this a practical approach: I do trust the standard that the compiler says it tries to follow, but I also verify.So, if there is a large bulk of code that expects specific behaviour for a code pattern that the standard says is Undefined Behaviour, any compiler that does not conform to the expected behaviour is going to see a lot of angry users whose code it breaks. If the code pattern is rare, then all bets are off regardless of what one or a few compiler versions might produce now.You see, from the early days of the C standard, up till C99, and now again starting at C23 (the C99-C17 interval being a bit of a "dark ages"), the C standard has been developedbased on existing implementations. That is, anything and everything codified in the C standard, has been in actual use and found useful or even necessary by the users – both compiler writers and the C language users using said compilers and standard C libraries. It isn't optimal, of course, but it does put the practical needs and use cases first, andthatis what I believe has kept it alive and evolving for decades now. Even the Abstract Machine grew out of necessity, representing the behaviour of computing machines without tying it into any (then-existing) architectural model.Some believe that the lack of a vision or visionary for C – unlike say Bjarne Stroustrup for C++, Guido van Rossum for Python, et cetera – makes the future of C muddy and non-inspiring. I disagree, because C sits firmly at the boundary between lowest-level assembly languages and higher-level more abstract programming languages, and often acts as a bridge between the two. It just does not need visionaries wanting to add high-level abstractions to it. Granted, I typically use C insystems programming, and like to do highest-level stuff like complex graphical user interfaces in scripting languages like Python, interfacing to my C code.In simple terms, we use C the same way we use integrated chips instead of individual logic gates or assembly language: to do things efficiently and effectively with what we have available as tools and resources right now.

author: peter-h
author_rank: Super Contributor
timestamp: August 23, 2024, 09:25:37 am
content: QuoteInteresting. Presumably because a correct implementation would require memset to use 8 bit 16 bit 32 bit or 64 bit writes, according to the type of the array.I must admit, if I had a memory mapped device with say 100 consecutively addressed ports, I would not use memset on it, but would write an explicit loop, and disable any optimisations on it. The reason is only because I know a clever memset could use 32 bit loads, and finishing off with byte loads at the start and end (if not 4-aligned) and that is likely to screw up that device. And to prevent the compiler spotting that loop and silently calling memset ()But, am I right that if that array was declared volatile, using memset on it would generate a compiler warning?On a related point, if you did declare that array thus (and it sits at a fixed address), if you then had some other code calling memset but referencing just that address (not the array name), would that also be picked up? IOW, does the compiler generate an internal map of volatile memory addresses?QuoteBut then if you need to use the debugger, it won't run well because the code will not correspond well enough with the C source, so you have to drop down to say -Og and the problem may go awayI find the difference between -Og and -O1/2/3 is negligible in overall performance terms, and in code size, and - on the basis that most CPUs spend 90% of their time running 10% (or 1%?) of the code - optimise individual functions with the -O3 attribute.The idea being to sell the product which you have been developing for x years, with no relevant changes. So, yeah, I don't use Cube's Debug and Release modes because they represent really dumb business risk management

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 23, 2024, 10:50:02 am
content: Quote from: IanB on August 23, 2024, 02:17:58 amWhoo boy, this is going to be a ride.Let's start by grabbing a copy ofN1570, the final public draft of C11 (which is right now the most widely supported version of C).The most important part to understand is chapter 4. Conformance paragraph 6, and the difference between afreestandingand ahostedenvironment. Simply put, in a hosted environment everything in the standard including the C standard library is available, but in a freestanding environment only a small subset is required to be available: those defined in header files<float.h>,<iso646.h>,<limits.h>,<stdalign.h>,<stdarg.h>,<stdbool.h>,<stddef.h>,<stdint.h>, and<stdnoreturn.h>. Note that these describe types and preprocessor macros, and do not declare any functions per se. However, any extensions or standard C library functions may be provided. The aforelinked-to PDF contains the functions et cetera defined in each header file in Annex B.Thus, when we develop microcontroller firmware or an operating system kernel in C, we have afreestanding environment. When you usenewlib, you basically have all of the standard C library available.TheC23 standardto be released any time now adds most of<string.h>to the above list, because of practical reasons. Indeed, if we look atStandardschapter in the GCC documentation, we see that it explicitly requires both freestanding and hosted implementations to implementmemcpy(),memmove(),memset(), andmemcmp(), plus optionallyabort(), andmemcpy()should work and not invoke undefined behaviour ifdest==src.The GCC documentation includes many different types of built-ins (both macros and functions). The most useful chapter is the6.63 Other Built-in Functions Provided by GCC. For the ones matching the functions in the standard C library, it explains that"These functions come in two forms: one whose names start with the__builtin_prefix, and the other without. Both forms have the same type (including prototype), the same address (when their address is taken), and the same meaning as the C library functions even if you specify the -fno-builtin option. Many of these functions are only optimized in certain cases; if they are not optimized in a particular case, a call to the library function is emitted."In other words, GCC provides those builtins, and may replace standard C library function calls, so that on architectures where it knows how to implement the particular use case in an optimized fashion, it can replace it with the optimized machine code.Otherwise, it will emit the standard C library call. And, as mentioned above, it internally may generate calls tomemset(),memcpy(),memmove(), andmemcmp().This means that GCC will never generate a function call to a__builtin_function()–– except when the user explicitly uses that built-in, and-fno-builtinor-fno-builtin-functioncompiler option is used.Thus, the proper view of built-in functions is that either they are implemented in inline machine code, or a call to the corresponding standard C function is generated. Clang is very similar at least on x86, x86-64, and various ARM architectures.A second group of "built-ins" are the functions GCC uses internally for example for arithmetic on architectures where they are not directly supported. These are listed in chapter4. The GCC low-level runtime libraryofGCC Internalsdocumentation. The most commonly encountered ones are listed in chapter4.1 Routines for integer arithmetic, and includes ones like__modsi3and__udivmoddi4. These are then found in thelibgcclibrary for the target architecture as part of GCC, and you can simply include that in your build toolchain, or build your own.ARM architectures have their own additional set of "built-ins", in the form of theARM Application Binary Interface(latest versions atgithub.com/ARM-software/abi-aa). You can find their implementations in ARM-specific C libraries, including theARM GNU Toolchain, which includesnewlibimplementation of almost all the standard C library functions and many POSIX and GNU extensions to it.Finally, any function matching a standard C library function name should have a compatible implementation, because the standard C library functions are part of the C language. As their behaviour is well specified, the functionality of the C program is not changed if the actual implementation replaces one function with another –– for example,memmove()withmemcpy()when the compiler knows the two arguments cannot overlap ––, so that kind of substitution is always allowed. In a freestanding environment, only some subset of them is available, but that subset is not limited by the standard at all.Thus, summarizing the above:We should never see linkages to__builtin_functionin object files, as they should be either implemented in inline machine code, or resolve to a standard C libraryfunctioncall, or a set of function calls available to whatever functions are known to be available in the current environment (hosted or freestanding).Some operations, like arithmetic operations on types not natively supported by the target architecture, can resolve to calls to GCC internal functions like__modsi3or__muldi3, which are found inlibgcc.GCC is allowed to replace any C standard library function call with a built-in one.GCC is also allowed to replace any C standard library function call with another standard library function call, as long as they perform the same function. This is whyprintf("Hello, world!\n");is often compiled toputs("Hello, world!");, andprintf("\n");toputc('\n');.Do note that while I've written C professionally for decades, I still occasionally err. The above is just my best current understanding.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 23, 2024, 12:00:51 pm
content: Quote from: peter-h on August 23, 2024, 09:25:37 amIt's simpler than that: casting awayvolatilequalifier and using the result is Undefined Behaviour, and none of the C standard library functions takevolatile-qualified parameters.In reality, GCC and Clang know that there is quite a lot of code that does that, and simply do thememset()/memcopy()/memmove().I haven't checked their intermediate representation to see if they also mark the state of the buffer unknown to the compiler somehow (ie. "Compiler, forget everything you know about the value or content of this data object"), but it would yield the same effect as remembering what objects were accessed via volatile accesses.Quote from: peter-h on August 23, 2024, 09:25:37 amNo. The device ports should be markedvolatile, so the explicit loop would suffice, at any optimization level.Quote from: peter-h on August 23, 2024, 09:25:37 amAgain, no. When the target is markedvolatile, the C compiler will not replace the accesses by a singlememset()or any other standard C library function call.The one strange case is when you have avolatilearray of structures, sayextern volatile struct mmdev somemmdev;or similar. (The exact address of such an array of structures can be set in the linker script, which is useful if you have variants where only the address varies.) Initializing the structures using e.g.struct mmdev initializer = {initializer-list};for (size_t i = 0; i < sizeof somemmdev / sizeof somemmdev; i++)somemmdev= initializer;does not necessarily copy structure members, but the entire storage as a whole, when the members ofstruct mmdevare notvolatilealso.That is, if the members arevolatile(recursively so for structure members), the above cannot generate anymemcpy()ormemmove()calls; their interface is simply incompatible with the above. But, if the members are non-volatile, we get into deep details of the C standard language which I'm not exactly sure of (specifically, the difference between avolatilearray of structures with non-volatilemembers, and avolatilearray of structures withvolatilemembers). GCC and Clang generate open-coded copy loops, but I just don't know if those can be replaced with call/calls tomemcpy()/memset()without violating the C standard Abstract Machine operation rules.The simple solution is to ensure that such structures have their members declaredvolatile, too. Then, there is no ambiquity, and setting such structures will not invokememcpy()/memmove()/memset(): the compiler will typically use an open coded loop of maximum-size general-purpose registers. On larger structures on x86-64, you see XMM registers (128-bit) used, for example. If you do need the members to be initialized in a specific order, or using specific-size accesses, then you do need to initialize the members instead of just assigning from an initialization structure –– but the members beingvolatilemeans that the compiler will do exactly that then, in the same order the code does it, without moving things around or replacing accesses.No need to play with the optimization level at all.Quote from: peter-h on August 23, 2024, 09:25:37 amHavingvolatile uint32_t buf;,memset(buf, 0, sizeof buf)will yield a warning, yes:"warning: passing argument 1 of 'memset' discards 'volatile' qualifier from pointer target type [-Wdiscarded-qualifiers]"using GCC,"warning: passing 'volatile uint8_t' (aka 'volatile unsigned char') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]"using Clang.You can disable the warnings (using-Wno-discarded-qualifierswith GCC,-Wno-incompatible-pointer-types-discards-qualifierswith Clang), but they are enabled by default, so you see those warnings even without specifying any-Woptions. I do prefer using-Wall, though: usually, the warnings are useful.Quote from: peter-h on August 23, 2024, 09:25:37 amI do not think it would be picked up, because the compiler would not know they match at compile time.No, I don't think the compiler generates an internal map of volatile memory addresses.Quote from: peter-h on August 23, 2024, 09:25:37 amBecause of my development practices, I rarely need a debugger or debugging symbols in my object files.I've described it before, but simply put, I do both top-down and bottom-up at the same time, in modular fashion, to minimize the set of changes between tests. Each module gets tested and verified separately, before integrating into the whole. (During experimental module/test case development, I do sometimes use a debugger, especially GDB with Python pretty-printers.) Additional tests are run after integrating each module. Thus, any issues after integration are either insufficient test coverage, or arise from complex interactions (like locking) between modules. Running the system under a debugger does not usually help with these, and instead verifying the state just before the problematic interaction is most likely to lead to the reason for the issue.For whatever reason, my Mark I eyeballs seem to be quite good at spotting bugs and issues, as long as I can sleep at least once between writing the code and examining it for bugs. I mean, I've already done so for e.g. GNU C library and the Linux kernel, which traditionally do get quite a lot of eyeballing-over in general. And when I do need to compare some state before and after, it is usually less intrusive to just copy that state to some reserved memory or push it down some fast enough pipe –– a variant of the "printf debugging approach" –– than attach a debugger to the device or application.Quote from: peter-h on August 23, 2024, 09:25:37 amTo me, it is more about what I'm ask the compiler to do, and avoiding pathological cases (both stupid code moving data between registers, which tends to be GCC's weak point, and also excess optimization like unrolling loops that are only run once or twice), than squeezing the last few percent of performance.Because of my development practices, I don't typically need debugging symbol support, either: I'm very familiar with ELF object files, and can find whatever I need in them fast enough. It's the complex interactions that take the most time to debug, and automated tools just don't help much with those.Those who deride "printf debugging" on principle, should take a look at the Linux kernellockdep. All it is, is a highly developed version of printf debugging, using data structures to store necessary data to verify runtime locking correctness. As long as I have a fast enough pipe to the target, or reserved memory I can asynchronously inspect, I can always add state tracking and verification that provides all relevant information in less intrusive form than hardware or software debugger. Then, ifthatadded short delay hides the problem, I know for sure I have a timing race window at hand, which makes me switch to a dedicated pessimal set of tests with the involved modules alone. Or, a Minimal Reproducible Example involving the modules and focusing on timing issues, in other words.Quote from: peter-h on August 23, 2024, 09:25:37 amMy experience in software and with Bit Rot in general, I'd say that is not very viable approach. I can fully understandno feature changes, but security patches (especiallywhen using libraries developed by others) would be the minimum for me.I do understand the idea of maximizing profit while minimizing development cost, I just don't agree with it as-is. Even with open source projects, their responsiveness and ability and willingness to address bugs and security issues is a major factor in choosing what I use; even more so for software I buy.(Although I currently do not use any paid proprietary software, that's just because I don't need any right now. I am very willing to pay, a lot, for any tools I need that are effective, and have done so in the past, including software. Knowing that there is no plan for providing fixes would turn me away instantly.)

author: peter-h
author_rank: Super Contributor
timestamp: August 23, 2024, 01:07:11 pm
content: QuoteIt isn't that. It is simply that one should not change anything for "quite a while" (during which a lot of feature testing etc takes place) before something is released.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 23, 2024, 02:27:46 pm
content: Quote from: peter-h on August 23, 2024, 01:07:11 pmI'm not sure I understand what you mean. I don't see testing as separate from development, and the way I write code, I don't require specific machine code to be generated, only that it performs as expected. I don't mind if a new compiler version might optimize the code differently.Obviously, there are specific sectors like medical, or anything certified at the machine code level, that need a different approach. However, my suggestion on those would be to use a more appropriate programming language, one with static and dynamic analysis tooling and a reputation for verifiability; say SPARK 2014 (based on Ada).

author: NorthGuy
author_rank: Super Contributor
timestamp: August 23, 2024, 03:30:28 pm
content: Quote from: Nominal Animal on August 23, 2024, 02:27:46 pmNo matter how hard you try, there may be bugs in your code, which may or may not manifest themselves depending on compiler settings and other circumstances. Therefore, before you ship anything in binary form, it should be tested. Typically, so called, regression tests are used which test the majority of the product functions on the exact binary being distributed. Some companies do very thorough tests which may take days to complete. Apparently, Crowdstrike didn't, which caused a little bit of troubles for the company.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 23, 2024, 03:50:07 pm
content: Quote from: Simon on August 22, 2024, 07:28:55 amI'd suggest trying to reframe your question. What you want is "how do I make sure that behavior I want is part of the observable behavior of the program." Variables and expressions are not themselves observable behavior, they are just names we assign to describe the program. They don't really have any meaning after compilation.Basically the two relevant ways to make a memory access have well defined behavior are volatile and C11 atomics. They work somewhat differently. voltaile access is directly a defined side effect of the program: the compiler must respect the order and contents of volatile accesses. There is some nuance there to do with sequence points and while the C standard doesn't require volatile accesses to be atomic, nor does it directly prevent hardware reordering or optimizations, individual platforms and ABIs do make guarantees needed to operate. For instance, memory mapped registers are generally configured by the platform to be non-cachable and non-reorderable by the CPU.C11 atomic operations are not strictly observable behavior, however they are how C11 defines it's memory model. Specifically, C11 guarantees "sequential consistency for race free programs". That means that if you use C11 atomic operations correctly to denote critical sections, the program will behave as if every operation happened strictly within the order as written, even in the face of multiple different views of "memory".Normally we think of volatile as used for IO and atomics as used for multi-core / multi-threaded communication, but it's possible to use atomics to enforce consistent memory view between peripherals and a processor core as well. This may actually be a good fit for what it sounds like you are trying to do. The nice thing is that the compiler is allowed to optimize the inside of a critical section however it likes, as long as memory is consistent at the end. It can also optimize by moving accesses into the critical section, but it can't move them out.There are definitely pitfalls to using C11 atomics as well, and for true IO, where the read or write has specific side effects, volatile is the correct tool. However for a buffer that just needs to be written to before an operation starts, atomics might work well.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 23, 2024, 05:31:22 pm
content: Quote from: NorthGuy on August 23, 2024, 03:30:28 pmObviously; so obviously in fact, that I consider that testing to be a natural part of the development. It is not something done "after the project is complete", but at all phases of the project, too.For example, seeing how users actually use the software, I typically realize new types of tests that can verify the system works as designed. I expect this, because no specification is truly complete (or evencorrect) up front, no matter how well written, and consider this too part of the development process.In practice, I look at development activity as a curve rather similar to typical product lifetimes. There is a big hump up front, then it tapers down slowly. Adding new features, new versions, refactoring, adds their own humps, and rewriting yields very much a new curve.It is because of this, because of how closely intertwined testing should be with development –– they should feed each other, continuously –– that I don't understand the idea of freezing the code base for a while. I understand freezing in the sense of not adding new features, and working on fixing bugs found in testing, but not leaving the code as is during testing.Quote from: ejeffrey on August 23, 2024, 03:50:07 pmAnother option is compiler memory barrier, usingasm volatile (""::"r"(buf):"memory"), wherebufis the buffer modified. It works with both GCC and Clang; the compiler will not move any memory accesses tobufover such a barrier, nor eliminate stores to it, before the barrier. (See relatedllvm bug (#15495) discussion.) It is used in the Linux kernel, via thebarrier_datamacro, for example to ensure that when callingmemzero_explicit(s,n), the buffer is truly cleared viamemset(s,0,n);and not eliminated because it is not accessed afterwards.I do suspect that such a compiler memory barrier would have worked for peter-h's use cases, but I'd need to know the precise details of the use case pattern to be sure. If the data involved is all in SRAM that does not get modified by hardware (other than possibly DMA initiated after the barrier), then it would suffice.Of course, while it conforms to the C11 memory model, it isnotstandard C at all, so whether it is a suitable solution worth considering depends on the situation. Assuming the target is a 32-bit ARM Cortex-M, then I would personally use a compiler memory barrier instead of volatile, when clearing or copying firmware-related stuff from the SRAM.

author: peter-h
author_rank: Super Contributor
timestamp: August 23, 2024, 06:32:22 pm
content: I don't think we actually disagree on anything, butQuotemeans you should be testing the version to be shipped "at all phases of the project"

author: Siwastaja
author_rank: Super Contributor
timestamp: August 23, 2024, 06:43:43 pm
content: Quote from: peter-h on August 23, 2024, 01:07:11 pmBeing careful withanychange during critical parts of operations is only natural and right. But we should be careful not to use it as an excuse to produce crappy code. I haven't had any large problems switching compiler versions, optimization settings etc. in a long long time. Of course I would not do that and then produce binaries without testing them. But then again, if adjusting compiler versions or optimization flags is something that everybody fears of becoming a "big project", something's seriously wrong in how the program is written.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 23, 2024, 07:56:59 pm
content: Quote from: ejeffrey on August 23, 2024, 03:50:07 pmI thought atomics were implemented with LDREX/STREX. In this case they cannot be use to synchronize memory for DMA.Anyway, "volatile" is only for C compiler - it guarantees that the compiler emits access instructions, it doesn't guarantee that the instructions do what you want them to do.If CPU uses caches, care must be taken to make sure the data make it to the memory (not just sitting in cache) by the time when you enable DMA.With out-of-order CPUs, you may need a memory barrier before you activate DMA because the desired writes may not yet been complete.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 24, 2024, 04:30:20 am
content: Quote from: NorthGuy on August 23, 2024, 07:56:59 pmIt depends on the platform and environment (which is a big pitfall with atomics), but generally LDREX/STREX and similar instructions are used only for compare exchange or other read/modify/write operations. atomic_load and atomic_store are just ordinary load/store with memory barriers, either separate barrier instructions or with dedicated load-acquire / store-release instructions. I think only 64 bit arm has these, and on 32 bit, an atomic load/store uses dmb.QuoteYep, that's what atomic operations do. Before an atomic store is visible (including to the IO peripheral), all of the preceding memory writes must be globally visible, and neither the compiler or the CPU is allowed to delay them past that point.Here is a simple example for arm32:https://godbolt.org/z/f9avoGeh6But whether this works for a given architecture and peripheral is not always clear, and even if it would work, platform headers are not usually set up for this.

author: peter-h
author_rank: Super Contributor
timestamp: August 24, 2024, 06:39:33 am
content: The dmb flushes the caches but that register load should be atomic anyway, surely (except for DMA writing to that address)?We did that herehttps://www.eevblog.com/forum/microcontrollers/32f417-arm32-which-variables-are-atomic/msg4781123/#msg4781123Interesting.

author: ataradov
author_rank: Super Contributor
timestamp: August 24, 2024, 02:57:32 pm
content: DMB does not flush the caches.LDR is atomic just for loads. LDREX/STREX are a pair of instructions that permit atomic exchange (load and store store that may fail as a pair if some other process loaded the same location before the store could complete).

author: ejeffrey
author_rank: Super Contributor
timestamp: August 24, 2024, 02:57:57 pm
content: Yes. The main point of atomic variables (despite the name) is that they create memory ordering constraints that apply both to the compiler and the processor. It's important that the accesses themselves be actually atomic, but as you say, machine word sized access is already atomic on every platform worth considering.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 24, 2024, 08:25:00 pm
content: Yes, that's why only memory barriers are required to be added.Of course, for the more involved atomic operations (like atomic add/sub), you either require specific instructions, or resort to disabling interrupts (which isn't very pretty but works).