How to add in-field firmware upgrade - Page 1
Forum section: electronics / general

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 02, 2022, 08:46:05 am
content: Hey allSo I was wondering how you guys handle in-field firmware upgrade for your MCUs?I have a GD32 MCU which has USB Device, and ideally I would want the end-users to be able to simply connect to the product over USB and be able to run a program on their computers to upgrade the firmware.Unfortunately my MCU's bootloader only supports upgrade over UART and not USB. So what to do and how?ThanksSimon

author: GromBeestje
author_rank: Frequent Contributor
timestamp: July 02, 2022, 09:14:28 am
content: There are various project on github that might be usefullhttps://github.com/devanlai/dapboothttps://github.com/IntergatedCircuits/DfuBootloaderhttps://github.com/trueserve/stm32f103-bootloaderIf you are running a GD32F103, a project targetting the STM32F103 might work out of the box,otherwise some adjustments might be needed.In any case, I'd recommend going with something that implements the USB DFU standard as it allows for using existing tools to perform the update.

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 02, 2022, 09:54:59 am
content: That's just what I needed, thank you!But so the DFU code should be there together with the rest then?

author: GromBeestje
author_rank: Frequent Contributor
timestamp: July 02, 2022, 10:37:34 am
content: Yes, it will be below the firmware like this (The size of 8K in these examples is taken from dapboot)Code:[ Firmware ] [ DFU ] You will have to adjust the linker file of you firmware to move to make space for the bootloader.When your current linker file is likeCode:MEMORY{FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 128KRAM (xrw) : ORIGIN = 0x20000000, LENGTH = 20K}you will have to adjust likeCode:MEMORY{FLASH (rx) : ORIGIN = 0x08002000, LENGTH = 120KRAM (xrw) : ORIGIN = 0x20000000, LENGTH = 20K}

author: dkonigs
author_rank: Regular Contributor
timestamp: July 12, 2022, 01:49:08 am
content: One of the bigger problems with using a DFU bootloader is that there don't seem to be any good user-friendly applications out there to implement DFU from the host computer side. Something like dfu-util works pretty well for a technical user, but you'll need to write some sort of foolproof application to wrap it if you expect normal end users to install updates.This is why for my project, despite my MCU having DFU support built in, I actually threw together a UF2 bootloader. The advantage of UF2 is that your device basically just shows up as a USB mass storage device, and the user just has to drag-and-drop a file into it. That's all it takes. Very easy.I used the TinyUF2 project as a basis for my bootloader:https://github.com/adafruit/tinyuf2(However, I did a fair bit of work on top of that to better integrate with my platform and to provide the experience I wanted.)So I'm using the built-in DFU support as a way of installing my bootloader (or recovering from corrupted flash situations), then my bootloader uses UF2 to install the real device firmware.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 12, 2022, 05:05:23 am
content: Usually, writing to FLASH is not rocket science; just writing a few magic register values to enable access, trigger erase cycle, and then writing to memory.Thus, if you can write your actual firmware which communicates with user and does things, you definitely can write the flasher; as a part of the application, or as a separate bootloader project. The upside is, you can fully define the protocol as you wish, using any interface.As always, I will encourage you to open up the reference manual and do it yourself.If you write flash as part of the application, the only "weird" thing you need to remember is to make sure the flasher code (all of it; including compiler-generated memset/memcpy calls etc.) must be in RAM (or in another flash sector you are not erasing), so that you don't lose your flasher code while it is running. This will trigger the learning process of how to put code in RAM, which will be a valuable lesson in any case.

author: Peabody
author_rank: Super Contributor
timestamp: July 12, 2022, 02:27:50 pm
content: You could add a USB-to-UART adapter and a USB port to your board. Maybe the CP2102N. That would give you the UART input. But of course you would still need software for the computer to do the uploading.

author: tellurium
author_rank: Frequent Contributor
timestamp: July 12, 2022, 04:42:22 pm
content: This might help toohttps://github.com/esmil/gd32vf103inator

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 08:51:21 am
content: Thank you all for the repliesI have indeed given some thought about it since, and I think I am going to do it like @Siwastaja suggested - my product already has a USB-MIDI interface so actually I can save time in USB-dev time and simply get MIDI packages (called "Sysex" where you can basically send whatever) and write these to the flash. Because learning to implement USB DFU correctky will for sure take a lot longer than learning to write to flash.And actually Siwastaja I was literally thinking about what you mentioned - if I want to be able to write the whole flash I need to copy my code in RAM. How does that work? I imagine something like this:User selects "upgrade mode"The code finds a suitable area of the RAM (I was thinking right after the last global variable, as far away from the stack as possible)Code (or DMA) copies the whole USB code there (the hard part here will be to make sure I know what to copy...)Then I simply jump to the correct address in RAM - and that's it? (nothing special to do other than jumping?)

author: tellurium
author_rank: Frequent Contributor
timestamp: July 13, 2022, 09:12:23 am
content: Quote from: simonlasnier on July 13, 2022, 08:51:21 amIt's not that easy. Your code uses variables and calls functions. When it gets compiled, a compiler creates object files where those places - variables usage and function calls, marked as "placeholders", like "here I am calling function X". You can see those by running "nm my_object_file.o". Then a linker combines (links) all those object files together, it "resolves" those references by assigning addresses to those placeholders. All functions are put in the "code" region, and data is put in the "data" region. Where those regions are, is defined in a linker script. The "code" region usually resides in flash, and "data" region in RAM. Since everything, a code and data of your firmware, is kept on flash, then a startup code upon boot copies the data part to RAM.Now, your code calls other functions that reside in the code region - that means, their addresses are in the code region. Once you copy a part of your code to RAM, the very first function call would bring control back to the code region (flash). Another issue is that code/data references may be compiled using a distance from code's current position, i.e. using a relative offset from the PC register in which the very first call/variable usage would just crash the relocated code.

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 09:42:52 am
content: I'm aware of how an LD script and compiling worksI did not think about the function calls thoughActually the relative calls (i.e. PC+xxx) are great because these will work when I copy everything. But looking at the assembly listings (for the final linked file) all function calls seem to be absolute.So how to solve this? I guess the only way is to write a special bootloader function which is independent, not calling anything else, which I can copy to RAM? But that's a lot of duplicated code (the whole USB stack :/ ).

author: tellurium
author_rank: Frequent Contributor
timestamp: July 13, 2022, 09:47:57 am
content: Quote from: simonlasnier on July 13, 2022, 09:42:52 amCompiling to a position-independent code might help. I've never done that myself though. Maybe some extra steps would be required, like relocating ELF offset table manually.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 13, 2022, 10:10:13 am
content: I have just used linker script to make a section that goes to RAM (ITCM if available for high-performance functions throughout the program; or just normal RAM), and then just use the section attribute in every function called by the flasher code, including shared communication code.Something like this:Code:_TEXT_ITCM_I_BEGIN = LOADADDR(.text_itcm);.text_itcm :{_TEXT_ITCM_BEGIN = .;*(.text_itcm)_TEXT_ITCM_END = .;} >ram_itcm AT>romThen startup code which does the copy:Code:extern unsigned int _TEXT_ITCM_BEGIN;extern unsigned int _TEXT_ITCM_END;extern unsigned int _TEXT_ITCM_I_BEGIN;uint32_t* text_itcm_begin = (uint32_t*)&_TEXT_ITCM_BEGIN;uint32_t* text_itcm_end = (uint32_t*)&_TEXT_ITCM_END;uint32_t* text_itcm_i_begin = (uint32_t*)&_TEXT_ITCM_I_BEGIN;while(text_itcm_begin < text_itcm_end){*text_itcm_begin = *text_itcm_i_begin;text_itcm_begin++;text_itcm_i_begin++;}LD linker script command NOCROSSREFS is useful as it errors out on any function call from the ram section to flash section, including those nasty implicit memset/memcpy calls the compiler sometimes generates.

author: alexanderbrevig
author_rank: Frequent Contributor
timestamp: July 13, 2022, 10:25:57 am
content: SysEx is slow. 128k will - best case - be about 16 seconds if you send one byte per message. 13 seconds if you bit pack all 10 bits. I would say don't reinvent the wheel if you do not absolutely depend on it.As an owner of gear that uses SysEx DFU, I would strongly suggest you do not go that route.Good luck!

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 10:35:10 am
content: Quote from: Siwastaja on July 13, 2022, 10:10:13 amITCM - what does that mean?But ok I think I get it, the fact that you tell the linker that this section is in RAM means the calls will be to the RAM space (so branch to 0x2000xxxx instead of 0x0800xxxx), even though the actual code section will be placed somewhere in the flash - correct? Is that done by the "AT>rom" at the end?The copying code makes perfect sense.My problem with this is still the fact that I need a complete separate boatloader code which has to include the whole USB stack, while I was hoping reusing most of the already running code.Just thinking, maybe I can make it hybrid, i.e. putting the USB stack at the beginning of the flash (say till 0x08000FFF - first 4kB), and only erasing from 0x08001000. That way I'll be able to use the stack in flash and only need a little code in the RAM. Problem is then is the USB stack code cannot be upgraded.

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 10:37:27 am
content: Quote from: alexanderbrevig on July 13, 2022, 10:25:57 amHa ha thanks but how is that slow? My flash is only 128k, and the users will need an upgrade every... 6 months at most? They can wait for 20 seconds I think

author: Siwastaja
author_rank: Super Contributor
timestamp: July 13, 2022, 10:58:57 am
content: Quote from: simonlasnier on July 13, 2022, 10:35:10 amInstruction tightly coupled memory - it's optional fast RAM in the ARM core, so you can place code there so that code execution doesn't need to use the main SRAM bus.QuoteThe AT> syntax means the original location - i.e., the flash. > syntax tells the location where the things will be relocated.Syntax wise, it's exactly the same story as with initialized variables - the content (initialized values) are in the flash (AT>), but after the initial relocation copy, the code will access the > address.I recommend you look at some example linker scripts and documentation of GNU LD.But hey, I missed you use a GD32 MCU. I haven't used them, but big kids on the streets talked about these things shadowing the whole flash into RAM at boot and run out of RAM. If this is really the case, then you would not need to worry at all, just write code that does whatever with the flash.If you want safe end-user firmware update procedure with least risk of brickage, you should have a separate bootloader (with full communication stack) which can always write the application sector in case something goes wrong.

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 11:19:20 am
content: Thanks for the explanation - that makes sense.Quote from: Siwastaja on July 13, 2022, 10:58:57 amThis sounds crazy - never heard about this... You mean the MCU itself would do that? I've spenta lotof time going through many GD32 Manuals, never saw something like this.How would that even work since the RAM is much smaller (especially if your variables already use 90% of it)?Quote from: Siwastaja on July 13, 2022, 10:58:57 amFully agree with this. But then the bootloader will be in flash, never overwritten, correct? (i.e. what I said about only erasing from 0x08001000 f.x.)

author: pqass
author_rank: Super Contributor
timestamp: July 13, 2022, 12:27:43 pm
content: Quote from: simonlasnier on July 02, 2022, 08:46:05 amIf you can squeeze an spi flash onto your board, it can be used to hold one or more copies of a firmware update; keeping the bootloader code simple. Your application handles the user interaction and complex stacks (TCP/IP, USB, FAT, etc.) and just copies the new firmware to this spi flash. On reset, the bootloader checks if the current firmware version matches the spi flash and if they differ does a main flash update. No dedicated update software external to your board is needed.Mike explains it here:

author: Siwastaja
author_rank: Super Contributor
timestamp: July 13, 2022, 12:59:21 pm
content: Quote from: pqass on July 13, 2022, 12:27:43 pmWhat a weird idea, usually you do this with multiple FLASH pages/sectors - external chip offers no benefit. But of course there are microcontrollers (for example the STM32H750 "budget model" of H743) on the market with only one page/sector, and in those cases such board-level workaround might be still favorable over just using a suitable microcontroller.Even then, the external chip won't provide full safety against power-loss during erase-write cycle.But most MCUs do have flash sectioned into multiple pages because this is a very common requirement.

author: pqass
author_rank: Super Contributor
timestamp: July 13, 2022, 02:15:22 pm
content: Quote from: Siwastaja on July 13, 2022, 12:59:21 pmWeird? It does have a cost advantage.WRT STM32F103, going from 128KB to 512MB is a CA$7 difference. Whereas, a 512KB (4Mb) spi flash is CA$0.64 (See Digikey: W25X40CLSNIG)Safety from power loss is accomplished by the bootloader checking if the newest firmware in spi flash has a valid checksum (at a known offset in the binary) and if not, skipping it. Once in the app again, the user downloads and updates the spi flash and reboots to try to apply the update again. The trick is to make sure there is at least one valid version in sp flash. The bootloader can also check the MCU flash checksum and force a replace if that got corrupted. The bootloader itself never needs updating.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 13, 2022, 02:42:11 pm
content: Quote from: pqass on July 13, 2022, 02:15:22 pmOK, we are getting confused by mixing two totally orthogonal concepts here.STM32F103 with 128KB of flash divides the internal flash into 512 pages (if I read the reference manual correctly - so many parts; in any case, all have way more than 3 pages!)Hence, external flash chip does not offer anything regarding bootloader / firmware updating functionality which already isn't possible with the internal flash. You totally can erase and write only part of the flash within the bootloader.But now you are talking about not having enough FLASH - then obviously it becomes a cost/availability optimization between a MCU with too little memory + external expansion, vs. larger MCU with more memory. This is not weird.I find it quite rare to use a budget tier STM32F1 class device and run out of 128KB, even if you divide that to two copies for firmware switching. What makes the application so large? Sometimes it is bloat or poor design - and if this is the case, getting it down to more manageable size would enable ultimate cost reduction by being able to use a smaller microcontroller, without the complexity of adding external memory.

author: pqass
author_rank: Super Contributor
timestamp: July 13, 2022, 04:06:42 pm
content: @SiwastajaYes I'm aware that the user app can erase/update just part of its program flash. And therefore, one can apply the same technique I'd described soley with internal flash. However, I didn't see the technique described in earlier posts so I wanted to be more explicit (via the video and my description).Also, desire to keep the BOM to a minimum is understandable but even so, there's a CA$3 difference going to 256KB. It's OPs call.

author: ejeffrey
author_rank: Super Contributor
timestamp: July 13, 2022, 04:56:27 pm
content: Quote from: simonlasnier on July 13, 2022, 11:19:20 amThat is how I prefer to do it. It's possible to make the bootloader updatable but ideally that should not be necessary. It's much easier to avoid bricking if the bootloader is never overwritten.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 13, 2022, 05:22:02 pm
content: Quote from: ejeffrey on July 13, 2022, 04:56:27 pmOf course that is the way to do it.In a few projects, I have been pushing my luck by doing in-application flashers with no separate bootloader and no double firmware functionality. Of course I receive the whole new firmware in RAM (also a reason to make it small), do CRC check, and only after successful verification, do quick erase-write step, which cannot fail, and does not need any further communication. This only leaves around 0.5 second of critical time during which power loss bricks the thing. Coupled with the instruction of "do not power off", we have seen zero complaints on returns, over maybe ~100 units.But latest project to utilize such scheme did use STM32H750 with a single(!) flash page anyway, so there is no safe way to do it at all.

author: Saimoun
author_rank: Frequent Contributor
timestamp: July 13, 2022, 09:56:12 pm
content: Ha ha ha I was literally about to ask, how well does the instruction "DO NOT POWER OFF" work...But yea then that might just be easier, to keep the bootloader at the beginning of the flash. My MCU has pages of 1kB so it's all good.Also I had this idea, *if* ever one day I have no choice but to update the bootloader, I could:Make a special firmware which the user will load via the normal bootloaderThis special firmware, when run, will update the bootloader (after a press of a button f.x.) - that is critical but it will be very quick, just moving bytes from one part of the flash to the bootloader areaThen the user can load a new firmware or the original one using the updated bootloader.Anyways thank you all - super helpful asking here as always

author: ejeffrey
author_rank: Super Contributor
timestamp: July 14, 2022, 02:00:39 am
content: Yes that is the way to do it. You can reduce but not eliminate the window of vulnerability if you are careful. But the question is: if your bootloader is able to successfully load the special firmware why are you updating it?

author: Peabody
author_rank: Super Contributor
timestamp: July 14, 2022, 04:40:08 am
content: A while back I wrote an SD card bootloader for MSP430 processors. The bootloader was attached to a normal program flashed using JTAG with a Launchpad. Then when run, that program installed the bootloader at the beginning of flash memory, and set the startup vector to point to the bootloader. After that, on powerup if no SD card was present, it would jump to the first byte after the bootloader. But if a card was present, it would search the root directory for the latest file of the right size, then read in its contents and flash it, then shut down. Everything was done in assembler, and the bootloader used exactly 1K, which was a convenient size. I found that locating various pieces at the right places was easier in assembler without having to fight with linkers and such.Then I had to write a Windows program that would read in an IntelHex file prepared by an assembler or compiler, run various checks to make sure everything was right, then output the appropriate binary file that would be copied to the SD card. I was planning to distribute the firmware updates by sending users a new binary file, which they would copy to their SD cards, then install the update.It all worked, but I never got to use it in a project. I used the SD card because I wanted to be able to update in the field - literally - with no computer or mains power at the remote site. Just carry the SD card to the site, and you're in like Flynn.I think there are a number of ways this can be done. I have mixed feelings about assembler vs compiler. You don't want to have to learn assembler for a particular MCU, but on the other hand, I've never had much luck getting a compiler or linker to put things where I want them. It's probably just all in knowing how.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 14, 2022, 06:12:51 am
content: Quote from: ejeffrey on July 14, 2022, 02:00:39 amIndeed, plan the bootloader well and test is thoroughly and chances are very high you don't need to ever update it.Alternating between two application images makes the device non-brickable - it will always boot with valid application even if power is lost during flash erase-write cycle. This requires at least 3 flash sectors, and enough flash memory for (2*app) + bootloader. You can make the bootloader check which app has a valid checksum plus a later version number, and boot that one, for example.Often an acceptable solution is to have bootloader and one copy of application. With power loss during erase-write cycle, the application becomes non-bootable, but bootloader stays intact so trying flashing again will be possible. 2 flash sectors and enough memory for 1*app + bootloader is needed.1 sector + app only with flasher code integrated requires least memory, is easiest to develop and use, but carries risk of brickage. If you can't fit the whole fw image in RAM at once but need to communicate + write piece by piece, then this risk becomes way too high IMO. But if you can buffer the whole image, the risk might be manageable.

author: tellurium
author_rank: Frequent Contributor
timestamp: July 14, 2022, 02:50:50 pm
content: To add to what has been said already.In my company, we used to deal with OTA firmware updates a lot. We used a bootloader and 2 flash partitions for an app.In our case, customers also stored some configuration data on flash, which was part of an app partition.So, sometimes a new image was perfectly fine, all checksums are OK. But their custom configuration, for one or another reason, bricked a device. The same binary worked fine on 1000 devices, but bricked the 1001st.The solution for that was a to use a "blessed" marker, somewhere in the flash. A current firmware image is "blessed", a new one is not. The bootloader flashes a new app partition, verifies checksums, and reboots - into that NEW partition, which is not yet marked as blessed. At the same time, it starts a hardware timer that simply reboots the device in X amount of seconds. That X seconds should be enough to boot, initialize, verify the new functionality, and "bless" a new partition manually by some user action - so on the next reboot, a new firmware gets booted.If a device hangs, crashes, or whatever - a HW timer restarts a device, and it reboots back to the old "blessed" partition.This way, we used to OTA even very dodgy, bad firmwares in field, and that mechanism was always rebooting the good firmware back on any trouble - for example, if we lost connection with a device.

author: dkonigs
author_rank: Regular Contributor
timestamp: July 14, 2022, 03:16:55 pm
content: Quote from: Peabody on July 14, 2022, 04:40:08 amIt really comes down to learning how to muck with the linker configuration file. For simply defining where the code as a whole goes, this is often enough.If you want to place specific functions, data structures, or data blobs, then there's section attributes on C code and tools like GNU objcopy (if you need to reorganize the output). And of course, looking at your build process output with a hex editor to verify that it behaved the way you want. This is a bit frustrating, but with enough trial-and-error can be made to work.For one of my projects, I have a bootloader that installs firmware updates off a USB stick (device has a USB host port). Functionally similar to what you're probably doing with an SD card. For another project, where the device normally acts as a USB peripheral, I'm using UF2. But in either case, the higher level design is similar. Bootloader sits in a small area at the start of flash, and depending on how its started it'll either update the real firmware or simply jump to it.

author: Saimoun
author_rank: Frequent Contributor
timestamp: August 01, 2022, 09:43:11 am
content: Thank you all for the feedback - always good to hear what others have doneQuote from: ejeffrey on July 14, 2022, 02:00:39 amWell the whole point in this case *is* to update the bootloader. But yes, like Siwastaja said the idea is to plan it and test it well, and never have to update it (but it's nice to know it's still a possiblity!).Quote from: Siwastaja on July 14, 2022, 06:12:51 am1 sector for bootloader + 1 app only seems fine to me, if the upgrade fails the user can just try again since the bootloader will be intact, correct?But I am not sure what brickage is so maybe I'm missing something hereDo you mean if the app image is badly loaded? Then the checksum will fail and the user can simply do it again - or is the worry here that the checksum keeps failing so that the user can actually never get to use his device again?

author: Siwastaja
author_rank: Super Contributor
timestamp: August 01, 2022, 02:55:44 pm
content: Brickage is power failure during erase-write operation. The problem with typical flash is, erase is SLOW! Writing is of course slow too.For example, erasing and writing a sector on STM32H7 device seems to take around a second with maximum possible erase/write parallelism setting and all data available in RAM.Once you have started the erase, data is lost. Subsequent write must succeed. You can remove the risk of software bug by making the erase-write part extremely simple (a few lines of code running with interrupts disabled), and you will of course do CRC checkbeforestarting that erase. But this still leaves the power loss risk.Yes, 1 bootloader + 1 app (on different sectors* of course) can be made in such way that even if app update fails, bootloader stays intact so user will have a system which does not boot to the app, but still hassomeway of retrying the update, but maybe not through a menu you would usually use, if that part is within app.1 bootloader + 2 app has the advantage that the actual fully performing application can always boot even if update fails. This may or may not be important. I don't think it is, you can just give instructions with the firmware update, "in rare case of failure during update, do this special sequence to try again". Or maybe there is no special sequence at all, maybe the user always updates the firmware through a mechanism provided by the bootloader alone, simplest would be "insert media with the fw update file and reboot" or "start this PC software which floods update packets through interface X and reboot", bootloader then checking the availablity and correctness of image and performing the update.*) sector here meaning "smallest eraseable unit", sometimes called page

author: PCB.Wiz
author_rank: Super Contributor
timestamp: August 03, 2022, 08:36:24 pm
content: Quote from: simonlasnier on July 13, 2022, 11:19:20 amEven if some parts did do this, you need caution.IC vendors have used this approach on experimental/alpha parts, as the FAB sample costs for not-flash process are lower.So they expect multiple die iterations and choose the lowest iteration cost, then as the part is proven, they might change to a more normal true flash design.So you cannot guarantee a non documented feature, remains in the part.

author: peter-h
author_rank: Super Contributor
timestamp: August 03, 2022, 09:43:53 pm
content: I wrote a boot loader for the 32F417 and Simon you are welcome to the source. In fact I probably posted it here a year or so ago; I ran some very long threads on this topic.It isn't all that easy to do it. Quite a learning curve and most of the time when it doesn't work you have to reload everything with the debuggerApparently one can flash "from the flash" because the CPU supposedly doesn't crash (it supposedly gets a million wait states but it all still runs) but I didn't do that. I set up a 32k buffer in RAM and that enabled me to flash the "boot block" (bottom 32k in my product) in one go; it takes about 300ms and that is the entire "bricking time window".Important that the new firmware is stored locally and CRC validated, with a post build step in your devkit to add the CRC at the end of the firmware, so this can be checked. If you don't have enough local storage for the entire CPU FLASH then you are taking a risk.Other approaches involve never flashing the bottom 32k, etc and keeping all the relevant code, including the RAM-executed boot loader, in there. I spent a lot of time on this...Obviously you need to limit the risk of all your customers loading some broken firmware at the same time. That will likely sink your company. Phone companies minimise this by scheduling regional updates.