Guaranteed storage of const variable in flash - Page 1
Forum section: electronics / general

author: ricko_uk
author_rank: Super Contributor
timestamp: March 17, 2021, 09:52:58 pm
content: Hi,I am using XC32 compiler on a PIC32MZ.I need to store various very large arrays in flash memory at compile time (their value will never be changed for the product's lifetime).I thought that declaring const would guarantee that to be the case but from reading XC32 manual that might not be the case. See page 169 here:https://ww1.microchip.com/downloads/en/DeviceDoc/50001686J.pdfThey need not to be auto type. I thought that all variables are automatically set as auto. Is that not the case? And how can I make sure a variable is definitely NOT auto?Thank you

author: DrG
author_rank: Super Contributor
timestamp: March 17, 2021, 09:57:32 pm
content: Quote from: ricko_uk on March 17, 2021, 09:52:58 pmSee section 9.3.2 Static Variables of the document you posted. Test it out, but it seems like that it what you are looking for.

author: ricko_uk
author_rank: Super Contributor
timestamp: March 17, 2021, 10:23:53 pm
content: Thank you DrG,yes that was it!

author: emece67
author_rank: Frequent Contributor
timestamp: March 18, 2021, 12:29:11 am
content: .

author: ricko_uk
author_rank: Super Contributor
timestamp: March 18, 2021, 01:38:23 am
content: Thank you emece67,is that the case even if I use extern in the other files where it would normally not be visible from to try to access it?Thank you

author: DavidAlfa
author_rank: Super Contributor
timestamp: March 18, 2021, 02:35:18 am
content: Then don't use static.Static is meant for that, or for storing the last value in reentrant functions.You can put a global static variable in every source file, named the same, ex. "static char Timer", totally isolated.

author: DrG
author_rank: Super Contributor
timestamp: March 18, 2021, 03:26:41 am
content: Taking a step back @ricko_uk, your main concern is: "I need to store various very large arrays in flash memory at compile time (their value will never be changed for the product's lifetime)."You noted that the variables should not be auto (according to the text) and static is not auto (also according to the text). But, there are some some issues with static that you may not like. - is that a fair statement?So, if you go back to page 169, you see:Note: For devices with internal Flash, a const-qualified variable combined with the default -membedded-data option can be used to place a constant in nonvolatile Flash memorySeems like what you are really interested in is making sure those variables go into flash. Based on that, you might be able to use const if you make sure -membedded-data option is in effect (I know it says default). That is a device-dependent option and you can add it to project / properties/ xc32-g++ / additional options /-membedded-data insuring that it is set.I don't know enough about PIC32MZ, but it seems to work on PIC32MX270 - at least it is not complaining. [edit: but I am flying blind here]I also don't know how Harmony deals with that stuff, but it may be something to try.

author: DrG
author_rank: Super Contributor
timestamp: March 18, 2021, 03:06:01 pm
content: So, I played around with this a little more and hoped to get some feedback. I mean, I hope the OP gets the situation sorted and I am happy to help if I can, but I would also like to know more about the issue.I am using an PIC32MX270F256B part with 256KB Flash memory (plus an additional 3 KB of Boot Flash) and 64K SRAM memory. I know that is not the same part as the OP is using.I am determining whether an array is going into SRAM or Flash by looking at this (MPLAB X IDE v5.40) after a build:Depending on how I declare the initialized array, I can see it going in or out of SRAM. For example,Using char xxarray={0,1,... I can see it in SRAM (that is when the pic was taken).If I use const char or static const char with the same array, it will NOT put it into SRAM.The device-dependent compiler option I mentioned earlier, does seem to be on by default as it's presence or absence does not change this behavior.Is this a correct approach with regard to seeing where it is going?I realize that the MX is not the MZ series and although the precise MZ chip is not specified, I can see "It also comes with dual-panel, live-update Flash (up to 2 MB), large RAM (512 KB) " in the generic description and I do not know what 'live update' Flash is.

author: ricko_uk
author_rank: Super Contributor
timestamp: March 18, 2021, 07:53:52 pm
content: Thank you DrG,I think that is a good way to check.Unfortunately I don't see those number change when I declare a second one or if I declare it as not cont. I do a full clean/build and also refresh that small window, but no changes.Should I do something else to update that window?Thank you

author: DrG
author_rank: Super Contributor
timestamp: March 18, 2021, 08:15:13 pm
content: Quote from: ricko_uk on March 18, 2021, 07:53:52 pmdunno - I was doing my testing with clean builds, just as you are - something device specific?

author: ricko_uk
author_rank: Super Contributor
timestamp: March 18, 2021, 08:26:04 pm
content: It might be, I will check this later and get back to you about it.Now the priority (rather urgent) is trying to sort out that other one, the nop issue in the other post.

author: westfw
author_rank: Super Contributor
timestamp: March 19, 2021, 11:52:03 pm
content: QuoteThere are CLI tools that make this sort of thing a lot easier, and it's a shame that they tend to go un-noticed in this day of GUI-based IDEs.The useful one in this case is "nm", which dumps symbols from a .elf file. In this case, probably .../xc32/v2.41/pic32mx/bin/nm (although apparently .elf files are standardized enough that an ARM copy of nm will adequtely dump info from a MIPS .elf file.Here, I compiled the sample uart_echo_interrupt Harmony project, which conveniently contains a startup message string. By default, this is declared as "char messageStart[] "string";", and nm (output filtered) says:Bash> /Applications/microchip/xc32/v2.41/pic32mx/bin/nm -SC --size-sort *.elf | grep message80000064 00000020 D messageError80000084000000a6DmessageStartthe "D" means that messageStart is off in the "data" section of the program at address 0x80000084 (which is presumably RAM on the chip in question.)If I add a "const" to the variable declaration, instead I get:Bash> /Applications/microchip/xc32/v2.41/pic32mx/bin/nm -SC --size-sort *.elf | grep message80000064 00000020 D messageError9d001150000000a6RmessageStartWhere the "R" means "Read-only data", and the address 0x9d001150 is in flash.MPLabX also produces a .map file which I guess contains similar information, but usually there's so much other stuff in there that it can be more confusing than enlightening....rodata 0x9d001150 0xa8.rodata 0x9d001150 0xa8 build/pic32mz_daa_sk/production/_ext/1360937237/main.o0x9d001150 messageStart

author: ricko_uk
author_rank: Super Contributor
timestamp: March 20, 2021, 04:48:49 pm
content: UPDATE...First of all thank you all for the suggestions and info, Always appreciated!westfw, that's very interesting and useful!After speaking with Microchip tech support:- all you need to do is declare a global var as const- ref seeing changes in the Dashboard window, even the engineer from Microchip could not get their window to change/update. He is looking into it and once I find out the reason I will post it hereThank you all again!

author: bson
author_rank: Unknown
timestamp: March 20, 2021, 10:14:28 pm
content: Plain old const works with just about any sensible toolchain.But with gcc you can explicitly put it in a section named .rodata-somethingorother:Code:#define __rodata(SECT) gnu::section(".rodata-" SECT)#define __weak gnu::weak// ...[[__rodata("sine_table"), __weak]] const int32_t sine_table= { ... values ...};This will put the variable sine_table in its own section, called ".rodata-sine_table". If it appears multiple times, because it's declared weak, you get only one copy instead of a compiler or linker error. If it's unused, the linker will remove it unless explicitly flagged as used (with the gnu::used attribute).Most ld command files already contain something like:Code:*(.rodata) /* all .rodata sections (constants, strings, etc.) */*(.rodata*) /* all .rodata* sections (constants, strings, etc.) */The second one matches any section name that begins with .rodata, including the one we added. If the linker script doesn't already have it it's easy to add (but I'm pretty sure it's standard for gnu ld).Edit: On second thought, the [] notation is somewhat modern C++, in C you may still have to use __attribute__((attr)).https://en.cppreference.com/w/cpp/language/attributes

author: SiliconWizard
author_rank: Super Contributor
timestamp: March 20, 2021, 10:35:15 pm
content: Yeah. Actually, all variable intializers will be held in flash (except for local variables with initializers "small" enough that initialization can be done with immediates in assembly). Of course, where else would they be stored?So the question isn't whether the data is stored in flash, but whether or not it will be copied to RAM upon startup, and accessed from RAM afterwards, instead of being directly accessed from FLASH. If said data is constant, this of course would waste RAM storage.As said above, normally, using a "const" qualifier will guarantee that. But... this also depends on the target architecture. Some "odd" architectures may require extra steps/or specific storage qualifiers, particularly when addressing Flash memory can't be done directly from an unified address space. Some PIC MCUs come to mind. Probably a lot of others, especially older ones. On those targets, compilers may or may not take care of it transparently.Now if you're using classic 32-bit stuff, such as ARM, RISC-V or MIPS-based CPUs, a const qualifier is all that's needed.

author: oliviasmithh901
author_rank: Newbie
timestamp: March 21, 2021, 03:58:19 am
content: So much to absorb in a single topic, I would go with simple option of EEPROM.

author: ricko_uk
author_rank: Super Contributor
timestamp: March 23, 2021, 12:34:23 am
content: Thank you all for the additional info. Always learning something new!

author: johnbeans
author_rank: Newbie
timestamp: August 28, 2022, 01:37:38 am
content: I'm sure you no longer care about this topic, but since I had the same problem and had to find a solution, I thought I would post it for others here.I'm using XC32 on a PIC32MM0256GPM036.Past usage with C32 had no problems, but suddenly my flash storage routines stopped working when I switched to XC32. They started acting like they were loaded at startup and never changed after using flash writing routines, despite using CONST at declaration.To fix it, I allocated the space as usual:const unsigned int __attribute__((aligned(4096), space(prog),address(0x9D00C000))) storedData;But the problem with this was that even after writing to these flash program locs during runtime, the values were always reported as 0x00000000, despite seeing in the debugger that the flash had been written (usingTo fix it, I created a global pointer value:int * flashData;Then at runtime I assigned it:flashData = storedData;After writing 0x55 to storedData, to the program both of the following statements were true:storedData = 0x00000000;flashData = 0x00000055;Weird, but there it is.

author: abyrvalg
author_rank: Frequent Contributor
timestamp: August 28, 2022, 06:53:36 am
content: If you were doing something like this:Code:if (storedData==0)puts(“it’s 0!”);FlashProgram(&storedData, 1234);if (storedData==0)puts(“it’s 0!”);and getting “it’s 0!” two times, there is nothing weird: the compiler could have optimized out the second access and reused the value obtained from the first one because it’s const. Or even have optimized out both accesses because the value is known to be 0 at compile time “officially”. And by using a non-const pointer you got a “non-const” behavior finally.

author: newbrain
author_rank: Super Contributor
timestamp: August 28, 2022, 08:29:49 am
content: As the array was declared const, the compiler has all the rights to assume its elements won't change in the lifetime of the program, hence it can optimise away any access to it or even the whole thing in some cases.If you want to make sure that this does not happen, it should be declared as volatile const, telling the compiler that some other agent can change the values behind its back.A common use for volatile const is e.g. for read only HW registers.Throwing away constness as you did is instead undefined behaviour, and, as any UB, not guaranteed to work.A good compiler should have also emitted a warning for that pointer assignment.

author: ricko_uk
author_rank: Super Contributor
timestamp: August 28, 2022, 08:32:09 pm
content: Interesting additional comments/feedback. Always welcomed!Thank you all as usual.

author: peter-h
author_rank: Super Contributor
timestamp: August 28, 2022, 08:33:54 pm
content: This has been quite useful. I've just gone over some fossilised code in my project and... by using "const" found another 1k of RAM

author: eugene
author_rank: Frequent Contributor
timestamp: August 29, 2022, 01:06:14 pm
content: Just as a point of discussion, suppose I have a lookup table that I want it to be copied to RAM for faster access. Perhaps someone can explain how to guarantee that.

author: peter-h
author_rank: Super Contributor
timestamp: August 29, 2022, 02:25:05 pm
content: constuint8_t fred  = { data } produces a table in FLASH only. It immediately follows the executable code (just checked this on my project). The link section is usually called TEXT.uint8_t fred  = { data } produces a table in RAM for which the "data" sits in FLASH and the startup code copies it from FLASH to RAM. The link section is usually called DATA, unless all the "data" is zero (0x00) in which case it goes into BSS which is not initialised but the BSS section is zeroed at startup*. So this is what you want. However, IME on the arm32, RAM resident stuff doesn't run faster than FLASH resident stuff - probably because the code cache works so well, being loaded 128 bytes at a time. I found delay loops run 20% faster from FLASH than in RAM resident code.There is some variation in the above section names but with ARM32 GCC, statics are categorised thus:int fred; goes into BSS (uninitialised data, used to go into COMMON in GCC versions before v10, now it is better because BSS gets zeroed)int fred=0; goes into BSS (which by definition is zeroed by startup code)int fred=1; goes into DATA (statics initialised to nonzero, copied to RAM at start)* therefore it is more efficient to haveint var=0;thanint var=5;because the former takes up no FLASH space (because zero-init variables get cleared in bulk when BSS is zeroed) whereas the latter takes up 4 bytes of FLASH, containing 0x00000005. Both take up 4 bytes of RAM.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 29, 2022, 06:39:39 pm
content: Yes to the above. Now don't forget that this is implementation-defined, so there is no hard guarantee that any compiler/linker for any target will behave this way. But this is indeed the most common on MCUs, at least the ones with non-exotic architectures.On architectures for which Flash and RAM are read in the same way, just at different locations, such as ARM and RISC-V, that const-qualified variables end up in Flash rather than RAM has a high probability (unless you specify otherwise with custom linker scripts in some way or play with odd compiler options), but don't completely exclude the possibility that for some particular target, the compiler may still decide to put some const variable in RAM (and copy initializer in startup code from Flash) just because accessing Flash would be much slower than RAM, for instance.So the "Guaranteed storage of const variable in flash" is hard to claim in the general case. It's just highly probable on common modern targets such as ARM Cortex.

author: eutectique
author_rank: Frequent Contributor
timestamp: August 29, 2022, 07:38:05 pm
content: Quote from: eugene on August 29, 2022, 01:06:14 pmDecorate your LUT with section attribute.Before:Code:static const uint32_t crc32_lut = {...}> arm-none-eabi-nm --print-size --size-sort --radix=x output/blahblah.elf | grep -i crc32_lut000010a0 00000040 t crc32_lut.5558After:Code:static const uint32_t crc32_lut __attribute__ ((section (".data"))) = {...}> arm-none-eabi-nm --print-size --size-sort --radix=x output/blahblah.elf | grep -i crc32_lut10000000 00000040 d crc32_lut.5558

author: newbrain
author_rank: Super Contributor
timestamp: August 30, 2022, 05:19:45 am
content: Quote from: peter-h on August 29, 2022, 02:25:05 pmint Fred; declared at file scope has static storage duration. Regardless of internal (if static is used) or external linkage (if not), and where the compiler decides to put it, it is NOT uninitialized data and never was for any vaguely standard compliant compiler.There is no such thing in C for static duration objects, if an initializer is not provided, it is as if =0 (or equivalent for aggregates) was written.Chapter and verse of the standard on request.No request, but I am (sometimes) a pedant.QuoteThis is an overgeneralization, might be true for your specific MCU (STM32F4xx, IIRC) but absolutely not true in general.EtA: Now I am at real keyboard:Proof of initialization to 0Standard ref, C11 but unchanged since, like, forever: 6.7.9 Initialization, §10:Quote

author: peter-h
author_rank: Super Contributor
timestamp: September 01, 2022, 05:50:16 am
content: QuoteI guess, it is possible (pre- GCC v10, when int fred; went into COMMON, not BSS) that your .s startup code didn't zero COMMON sections.

author: newbrain
author_rank: Super Contributor
timestamp: September 01, 2022, 01:09:47 pm
content: Quote from: peter-h on September 01, 2022, 05:50:16 amYes, it's possible to have a broken startup.s...A major change in gcc10 was finally stopping to silently accept multiple tentative definitions of the same object with (implicit) external linkage across translation units, and that's the reason for the section reshuffling.That code was already not compliant, falling afoul of 6.7.9, and UB - so yes, it would have somehow been justified to have them not initialized, still better than nasal demons. I would still consider a startup.s + linker script that would not take that into account irremediably broken.TBH it never affected me, as I (try to) write C, instead of something not totally unlike it, and use -std=c?? -Wall -Wextra -pedantic.

author: eutectique
author_rank: Frequent Contributor
timestamp: September 01, 2022, 04:15:03 pm
content: Quote from: newbrain on September 01, 2022, 01:09:47 pmTrue, but this is completely the responsibility of the developer, not the tool.

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 01, 2022, 06:30:01 pm
content: Quote from: eutectique on September 01, 2022, 04:15:03 pmWell, yeah. But even using the right tools and setting them up correctly is the responsibility of the developer anyway.

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 01, 2022, 06:33:16 pm
content: Quote from: newbrain on September 01, 2022, 01:09:47 pmNever affected me either, how hard was it to declare variables in one place only and then use 'extern' (of course then without any initializer) if you wanted access to them elsewhere? (Something that I do not particularly recommend these days anyway, I tend to prefer implementing accessor functions if I really need to access a global variable outside of where it's declared, and thus I'll usually declare them static, almost always.Enough with the spaghetti code. Spaghetti is good in your plate. That's pretty much it.