Does GCC LD EXCLUDE_FILE directive actually work? (32F4 ARM32 version) - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: July 02, 2023, 01:27:39 pm
content: Context:https://www.eevblog.com/forum/microcontrollers/a-question-on-gcc-ld-linker-script-syntax/msg4938223/#msg4938223Some of the directives in the generic LD manual do not work in this version (supplied with ST Cube IDE).This invocation is one of many formats I triedCode:/* This collects all other stuff, which gets loaded into FLASH */.code_constants_etc :{*(EXCLUDE_FILE(*b_loader.o) .text .text* .rodata .rodata* .data .data* ). = ALIGN(4);*(.text) /* .text sections (code) */*(.text*) /* .text* sections (code) */*(.rodata) /* .rodata sections (constants, strings, etc.) */*(.rodata*) /* .rodata* sections (constants, strings, etc.) */*(.glue_7) /* glue arm to thumb code */*(.glue_7t) /* glue thumb to arm code */*(.eh_frame)KEEP (*(.init))KEEP (*(.fini)). = ALIGN(4);_e_code_constants_etc = .; /* define a global symbol at end of code */} >FLASH_APPI've solved it by re-ordering stuff in the linkfile which based on the addresses in .map seems to have worked but there are side effects which I am trying to track down... spent hours on this so far.The problem is that the above section is collecting everything (as it should) but I need it tonotcollect b_loader.o stuff.Code:.b_loader_all :{. = ALIGN(4);_loader_ram_start = .;KEEP(*(.b_loader))*b_loader.o (.text .text* .rodata .rodata* .data .data*). = ALIGN(4);_loader_ram_end = .;} >RAM AT>FLASH_BOOT_loader_flash_start = LOADADDR(.b_loader_all);

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 02, 2023, 02:35:18 pm
content: Quote from: peter-h on July 02, 2023, 01:27:39 pmThis includes .text*, .rodata*, and .data* from all files except *b_loader.o, and then aligns to a 32-byte boundary. Then,Quote from: peter-h on July 02, 2023, 01:27:39 pmincludes the same from the excluded *b_loader.o.

author: peter-h
author_rank: Super Contributor
timestamp: July 02, 2023, 03:19:18 pm
content: Right; I get that, but I tried the excludeafterthe other stuff too.As I've posted in the other thread, I've given up on this and got it working by a re-ordering of the various blocks. I understand the basic ideas of collecting symbols (beeing doing embedded with text, data, bss, etc for 40 years) but these subtleties are way above my headAnd the documentation on LD is absolute total shit. The whole language is just circular. It's the worst ever, as evidenced by the vast number of posts online with questions on it./DISCARD/ should also be a way of doing it, but I found it dumps the specified stuff permanently; it cannot be picked up by a later section.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 02, 2023, 03:56:19 pm
content: I fear you are brute-forcing something that would be much easier to construct analytically.First step is realizing you need a paradigm switch. LD scripts are not configuration files, they are actual scripts the linker runs to generate an ELF file, written in the linker command language. While the syntax is funky, the underlying logic is quite straightforward, and very much reminds me ofawk, the record-and-field -based filtering and conversion tool and language.Second step is realizing you always need aplanbefore writing a script. You need to know what it does, or you'll end up adding features on top of features, making it into an unmaintainable spaghetti mess that has to be brute-forced to get to work as desired.What I recommend, is using a spreadsheet (like LibreOffice Calc, but anything goes) to create the actual memory map you want to achieve. I prefer to start with zero at the top, increasing addresses downwards. In separate columns, putMemory area name, like RAM, FLASH, BOOTFLASH.Output section (with descriptive names), to order things within each memory region on the device.Linker-defined symbol names (to identify specific addresses)Address, if fixedExcluded (or included) file name(s)Input section(s)With the complexity you have, you'll probably have a couple of hundred rows in that spreadsheet, but that's perfectly okay. Just make sure younevermake a change to the linker script that deviates from the spreadsheet; always edit the spreadsheet first.A plain text file (in UTF-8 with UnicodeBox Drawingcharacters for prettification) has the benefit that when using source control like git, even the changes are human-readable.Next, you read chapter 3.Linker Scriptsin the binutils LD manual. (You can check which version you actually use by running the linker command with--versionas the parameter; you should find the corresponding manual athttps://sourceware.org/binutils/docs-M.N/ld/, whereMis the major version (2), andNis the minor version (latest being 40, as of 2023-07-02).Your linker script should probably start with theMEMORYcommand. Its contents are determined by memory area name (1. column) and address (4. column) in the spreadsheet above.It is common to next define any linker symbols that depend only on the memory areas, like the start and end of RAM, start and end of FLASH, and so on. You can add and subtract addresses to calculate lengths. You often use theORIGIN(memory_area)andLENGTH(memory_area)for defining these symbols. Remember the syntax issymbol_name=expression;here.Next, you should use theSECTIONScommand. I'll expand on its contents below the dotted line.Finally, you add theENTRY(symbol)to set the ELF file entry address. You can have it at either the root level, or within the SECTIONS command. If the hardware has it hardwired, use that address. It is mostly a nicety, but it might be useful in some cases – if not for anything else, then for us humans to see without having to consult the microcontroller programming manual._ _ _ _ _Sections:This is the part that most reminds me of processing data with awk. Eachoutput sectioncommand defines one output section, its contents, and which memory area it is placed in. The contents are specified by one or moreinput section descriptions. You can also define symbols in each output section, but note that by default, the addresses (including.) is relative to the section, so you might wish to useABSOLUTE(.)to obtain the non-relocatable absolute address, or one of the otherbuilt-in functions, to manipulate the symbol values.You can also defineoverlays. You can use it for your code run in RAM during bootup before firmware/application is loaded to RAM, to ensure the linker uses correct addresses, or you can use position independent code._ _ _ _ _If you find yourself lost in the various terms, create your own crib sheet or example, with specific names so you can see the cross references. For example,MEMORY {FLASH(rx): ORIGIN = 0x08000000, LENGTH = 1024KRAM(rwx): ORIGIN = 0x20000000, LENGTH = 128KCCM(rwx): ORIGIN = 0x10000000, LENGTH = 64K}SECTIONS {. = 0;/* No offset */.text: {KEEP(*(.startup))/* Startup code */*(.text*)/* Program code */KEEP(*(.rodata*))/* Read only data */} >FLASH.data: ALIGN(16) {*(.data*) /* Initialized data */} >RAMAT>FLASH}Note how I put all names in italics, and used a different font for the comments? These are the cues I use to remind myself; use whatever works for you.For example, just looking at the.dataoutput section definition, I remind myself that>regionspecifies the run-time region, butAT>sregionwhere the contents are stored, and that my bootup needs to copy fromsregiontoregionfor this to work.Instead of making just one example to contain everything needed (like adding the symbols and expressions needed for the aforementioned copy), make separate examples or crib sheets, to reduce your cognitive load. Perhaps label them according to their use. You won't have that many of them, less than a dozen, anyway; linker scripts aren'tthatcomplicated, once yougrokhow they work.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 02, 2023, 04:08:17 pm
content: Quote from: peter-h on July 02, 2023, 03:19:18 pmThing is, you only need the exclude.For example,EXCLUDE_FILE(*foo.o *bar.o) *(.text .rodata)includes all.textand.rodatasections in all files excluding*foo.oand*bar.o.The syntax is precise; for example,EXCLUDE_FILE(*foo.o *bar.o) .text .rodataincludes all.rodatasections in all files, and all.textsections in all files excluding*foo.oand*bar.o.In other words,EXCLUDE_FILE(*foo.o *bar.o) *(.text .rodata)andEXCLUDE_FILE(*foo.o *bar.o) .text EXCLUDE_FILE(*foo.o *bar.o) .rodataare equivalent, including.textand.rodatasections from all files except for*foo.oand*bar.o.In summary, the syntax isEXCLUDE_FILE(exclude-glob-pattern(s))input-sectionEXCLUDE_FILE(exclude-glob-pattern(s))include-glob-pattern:input-sectionEXCLUDE_FILE(exclude-glob-pattern(s))include-glob-pattern:(input-section(s))whereinput-sectionis one ofsection-nameinclude-glob-pattern:section-nameinclude-glob-pattern:(section-name(s))where(s)indicates plural, i.e. one or more, whitespace-separated.Ifinclude-glob-patternis not specified, it defaults to*(all files).

author: peter-h
author_rank: Super Contributor
timestamp: July 02, 2023, 06:31:20 pm
content: Thank you so much for that amazing explanation!Some of that is just bizarre but I guess it has its uses. I just can't imagine getting back to it a year later and understanding itIt explains why the various examples online didn't work; people posted them without explanation of the very subtle syntax.My project has been as simple as I could get it:1) boot block - 32k size - asm init code does the data copy, and bss zeroing (this was the original ex ST MX demo project), stack top of 128k RAM (even if 32F437 used which has 192k RAM)2) boot loader* - 1.5k size - data is copied with the code; to somewhere high up in RAM, to an address after bss of 3), loader uses no bss, stack hard-coded in the CCM3) main code - from base+32k to the rest of the 1MB (this can also be an overlay supplied by someone else (I have a special Cube kit for building that), data copy and bss zeroing is done by a "main stub" (with C)* This was tricky to link because it is stored in the boot block and copied to RAM, so the code needs to be located to run from RAM, and for a long time the only way I could solve the linker complaints about section overlap was to lose that much from the usable RAM. I know one can also suppress the linker warnings. It isn't much RAM though. But then I had some problems and decided to clean up a lot of stuff, and placed the loader execution address after main code bss, and there it doesn't interfere (it is heap space which the linker doesn't know/care about).Other constraints are that 2) can call functions in 1), but 1) and 3) are totally separated at the base+32k boundary (main code is entered with an asm jmp to base+32k).Also the loader needs to be in RAM because I want to retain the option of updating the boot block too. Yes, there is a way whereby a 32F4 can program its own flash (it generates loads of wait states) but it can't program the addresses of the programming code, so one needs to switch to another bit of programming code then. But I never saw an example of that so probably it doesn't actually work. Another approach is to get the compiler to generate relocatable code but again I never found an example.If it wasn't for the loader, it would be trivialBut I need remote firmware upload, over http/https.Back to basics, why do C compilers generate sensible error messages? Real men need just "syntax error" and possibly, at a stretch, a line number. They go down with the ship while saluting the queen. This LD thing was written for real men, and probably the reason for the totally crap usability is because most people set it up once and never touch it again. And avoid touching it at all costs.

author: DiTBho
author_rank: Super Contributor
timestamp: July 02, 2023, 08:39:47 pm
content: Quote from: peter-h on July 02, 2023, 06:31:20 pmCompared to the my-c project linker script, which has been inspired by Avoget and Green Hills stuff, my personal opinion is that the GNU linker script is not very friendly because it is not well designed for the common user as it requires a lot of specific knowledge and competence, and I address the defect to the fact that designing things that look easy to be used requires more effort, which is not a GNU priority.

author: Nominal Animal
author_rank: Super Contributor
timestamp: July 02, 2023, 08:42:18 pm
content: Quote from: peter-h on July 02, 2023, 06:31:20 pmDomain-specific languages with a tiny set of users do tend to be, yeah.Quote from: peter-h on July 02, 2023, 06:31:20 pmI can't parse your description well enough to arrive to an actual memory map. You need that spreadsheet or a diagram!For example, starting with just the memory regions and output sections, ignoring what files define what, fill up the table of where things should end up at:MemoryregionOutputsectionOverlayLinkersymbolAddressDescriptionFLASH__flash_start0x08000000.boot_initExecutable in place, VMA=LMA.boot_loader_srcLMA, VMA=.boot_loader.fastrun_srcLMA, VMA=.fastrun, for RAM code.textExecutable in place, VMA=LMARAM__ram_start0x02000000.fastrunRAM1LMA, VMA=.fastrun_src, copied byfoo()in .boot_init.boot_loaderRAM1VMA, LMA=.boot_loader_src, copied bybar()in .boot_initCCM__ccm_start0x01000000Ignore/clear the Overlay column first. When you have every output section you need, you combine RAM and CCM output sections that can occupy the same address ranges since they never exist at the same time, by giving them a shared (but unique) overlay name. Above, I used RAM1 for .fastrun and .boot_loader, since you mentioned it has to be copied to RAM, but is not accessible to .text or .fastrun. For non-overlapped sections, just leave it empty.Then, and only then, will you add the column for input sections, and two columns for files: one used for when only specific files' sections are included, and the other when all except specific files' sections are included. When all files are included, both are empty. When only specific files' sections are included, only the first file column is used. When all but specific files' sections are included, only the second file column is used. I'd call the input section column "Input section", and the two file columns "in only files" and "in all files except", I think: it has to be crystal clear, we don't want some notation we forget in a month.With such a table or spreadsheet, constructing a linker script that does exactly what is wanted, is quite straightforward.If you fill the above, I'm pretty sure we can construct one that does exactly what the table or spreadsheet specifies.It is absolutely critical to keep the table or spreadsheet the master, and the linker script as the slave, because it can often be less work to rewrite the linker script based on the table or spreadsheet, than modify the existing script. However, with both, you can also re-learn what you have forgotten, by comparing the existing table or spreadsheet to the linker script itself, and check how each part has been configured.(While I believe it would be possible to write a Python script to parse CSV-formatted table with fixed-name fields into a linker script, it would go too far into code generation land for my taste. I do mention it, because the table or spreadsheet must contain all information, and therefore a translation is possible. For example, you might wish to add a column that states the required alignment for the output section or linker-defined symbol. And you probably want the linker to calculate the sizes for you (since it can), so you don't have to do the calculations at runtime, saving a few instructions here and there; I'd list these in the spreadsheet, before the actual table.)Quote from: peter-h on July 02, 2023, 06:31:20 pmIt's not that bad.Many of those porting a build toolchain to new hardware (i.e., most of those who ever write linker scripts in the first place) tend to have experience with weird domain-specific languages and tools, so they don't actually mind much. I've seen worse. The procedure I outlined earlier, and the experimentation with a trivial example source tree that defines at least one symbol for each section, is what most such people tend to end up with (because with weird tools, it is efficient way to truly find out). If they then look at ELF file capabilities, and start working on a better linker script command language, they discover they need the same features currently provided, with just abettersyntax. That set of people, being used to all sorts of syntaxes, just shrug and deal with it, because replacement is just not worth the effort to any of them.Is it a crappy language? I don't think so. The documentation is horrible, because it does not use terms consistently. When it talks about addresses, they can be load addresses (addresses in the image), virtual memory addresses (where the code or data will reside at runtime), or relative to the beginning of the current output section. Even the exact grammar isn't specified; that alone inaBNFformat would help a lot, because that way the sub-command sequences like the way input sections can be listed, would be clear (and is just sequential stuff; just like"a" varname "b"in Awk constructs a string from the contents of variablevarname, prefixed withaand suffixed withb).And the index doesn't even mention that linker script command language expressions can use the ternary operator, i.e.((expression) ?if-true-or-nonzero:if-false-or-zero), it's only mentioned inOperators listand in the description of theDEFINED(symbol) built-in!Quote from: DiTBho on July 02, 2023, 08:39:47 pmYes, but it's not just GNU, it is in all free/open source software, because designing things forothersto use is much harder than designing things foroneselfto use. In general, user-friendly design takes a lot of effort.Just compare how far KiCAD user interface has come in the last couple of major versions! Ilikeversion 7 a lot.