Saving state on power off - Page 1
Forum section: electronics / projects

author: Axk
author_rank: Regular Contributor
timestamp: April 13, 2019, 06:58:10 pm
content: Given a circuit with an FPGA and power disappearing how do you implement triggering of saving the current state?I'm thinking about adding a capacitor between the power supply and FPGA's power pins while having a GPIO pin monitor the power supply bypassing the capacitor.When the GPIO pin goes low then it would save the state. Assuming the capacitor will give it enough time to save the state before the FPGA loses power.Are there any standard approach(es) to dealing with "save on power off"?

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 13, 2019, 07:20:39 pm
content: That can certainly work and is done in quite a few popular 3D printers. Another solution is to constantly save the state to FRAM or SRAM with battery or supercap backup.

author: Ian.M
author_rank: Super Contributor
timestamp: April 13, 2019, 08:30:13 pm
content: It doesn't matter whether its a MCU, FPGA or other digital processor - the key ingredients are the same:* Some sort of non-volatile memory.* A power supervisor chip (or internal module) to either hold the processor in reset when its supply drops, or to de-assert the NVM Chip Enable so the processor doesn't corrupt the NVM with spurious writes as it drops below its min. operating voltage* Optionally, for write endurance limited NVMs like FLASH or EEPROM, or just to make maintaining data integrity easier, advance warning of power failure e.g loss of AC input, low battery detection, etc. to give enough time to save the state, or complete any pending writes, then the firmware locks out further writes until the incoming power is OK again (with or without compete powerdown in-between). If you don't have a power fail interrupt, you'll have to store multiple copies of the data with checksums as any copy of the data set may be corrupted by power loss while updating it.ultra-low-power MCUs can take an end-run around the problem: Maintain power to the CPU with a backup battery or supercap, and when incoming power-loss is detected, enter a low power sleep state that preserves internal RAM contents.Caution: if using FLASH or EEPROM NVM in mains powered or removable battery applications, its advisable to rate-limit NVM updates, otherwise a very noisy mains supply or intermittent battery contact due to vibration can cause rapidly repeated updates and chew through the NVM'ss expected write endurance lifetime in weeks or months rather than the expected years or decades. The trade-off is the possible loss of any state changes in the last X minutes before power loss, where X is the interval you chose to rate limit updates. This does not apply to effectively unlimited write endurance NVM, e.g. FRAM and battery backed static RAM

author: jbb
author_rank: Super Contributor
timestamp: April 13, 2019, 10:45:49 pm
content: Using an FPGA may make things more difficult, because you may be using internal block RAMs etc. Any technique which relies on pushing data into non-volatile (or battery backed) memory will need some hardware support to a) write the important data out periodically and b) get it back in again after a blackout.This will be easier if your FPGA has a microcontroller inside and all the relevant data is mapped to the micro’s memory space.If you need to maintainallof the state, you might need to simply deploy a backup battery large enough to keep the whole FPGA alive. (Note: clock gating in the FPGA could be very helpful here in terms of reducing power consumption / battery size.)

author: jbb
author_rank: Super Contributor
timestamp: April 13, 2019, 10:51:03 pm
content: Oh, yes. I forgot...In terms of system design, you can also use a specially designed power supply that can keep the DC rails up for a little while after the main supply drops out (i.e. a large capacitor somewhere). Then use a monitor circuit (with de glitching) on the power input to send a signal to the FPGA saying ‘power’s down, shut off loads and save critical states NOW.’I have seen a few systems which go so far as having a little LiPo or LiFePO4 cell to provide ‘last gasp’ features including state saving and even sending a power loss message via cellular modem.

author: DaJMasta
author_rank: Super Contributor
timestamp: April 14, 2019, 12:48:56 am
content: Similar technique, different implementation, you can do your power off monitoring on the mains side of your power supply, so that as long as you have enough cycles to save state on an interrupt, you get to use the capacitance in your power supply output without adding extra or trying to sense it very early on in the discharge cycle.It sort of comes down to how long you need to save state, though. If there's no capacitance or the chip may be in sleep or very slow operation, you probably want something that is just regularly saved or an additional supply (capacitor) to give it the time to save. If you keep your save requirements for the state to just a few cycles and the power sensor is quick and on a high priority interrupt, you may be able to get away with just the power sensor, no extra storage or hardware. With traditional power supply designs, you likely have at least 100uS, and probably have several ms before the power drops low enough to prevent operation, so you just have to be able to sense the fault and react in that window.

author: Axk
author_rank: Regular Contributor
timestamp: April 14, 2019, 11:54:03 am
content: I just need to save around 10 bytes of state and planning to save it into the FPGA's (LCXMO2-1200HC) internal flash.So I hope to get away with a couple of hundred of uF of capacitance to give it enough time (a couple of milliseconds at around 20mA max) to write to the flash.

author: mariush
author_rank: Super Contributor
timestamp: April 14, 2019, 01:17:45 pm
content: How often does that state change ?Maybe have a flag which is set if any of the bits in those 10 bytes changes, and start a timer at that point... when 5-10 seconds go by, dump the bytes to flash and reset the flag

author: Axk
author_rank: Regular Contributor
timestamp: April 14, 2019, 05:05:57 pm
content: up to 1.5Meg times a second

author: jbb
author_rank: Super Contributor
timestamp: April 14, 2019, 07:25:44 pm
content: Eek, that’s a lot. Now we can guess why you’re using an FPGAYou should check out the erase / write lifetime of the internal flash. As someone said earlier, you could get a lot of write cycles if the incoming AC power goes intermittent.

author: Axk
author_rank: Regular Contributor
timestamp: May 04, 2019, 01:43:20 pm
content: Thinking about the implementation details now.How do I detect the loss of power as close to the FPGA as possible, without going all the way up to the input of the circuit (which is 54V)?Putting a Schottky diode right before the backup capacitor and then sensing before the diode comes to mind (so that when the input current is lost the FPGA continues to work from the capacitor but sensing input doesn't see the voltage on the capacitor).Will this approach work?

author: Ian.M
author_rank: Super Contributor
timestamp: May 04, 2019, 02:03:52 pm
content: That depends on how much margin you've got between the normal FPGA operating voltage and the minimum voltage at which it can operate normally + write to FLASH. If you tap off earlier in the PSU, before the regulator, (possibly via a potential divider to a micropower comparator), you get a lot more warning time for a lot less capacitance as the input side of the regulator can drop far further below its normal voltage before the FPGA browns out. You'll probably still want a diode so the comparator doesn't sense the slowly dropping bulk cap voltage after the supply input goes away.

author: Axk
author_rank: Regular Contributor
timestamp: May 05, 2019, 01:09:41 pm
content: Thinking about it more, I already have voltage measurement before the regulator and the voltage divider (VSEN) that that is measured is disconnected when the circuit is switched off.So I suppose should be able to use this voltage measurement provided I add enough capacitance (not added in the schematic yet) to ground after the regulator to give it time to write to flash.

author: NorthGuy
author_rank: Super Contributor
timestamp: May 05, 2019, 01:47:03 pm
content: FPGA is likely to be high power which makes it difficult to power with a capacitor. I would use a small low power MCU instead of flash fed by a separate linear regulator. You would pass the state to the MCU continuously. MCU would detect voltage sag and then would store the state in the internal flash. You don't need a big capacitor to maintain an MCU and it would cost you the same as flash IC. You can also use the same MCU to power down your FPGA gracefully.

author: Axk
author_rank: Regular Contributor
timestamp: May 05, 2019, 02:03:40 pm
content: To write one page of flash it takes 0.1ms.I can erase the flash on power on, so I only need about 0.1ms to save the state.100uF (a couple of 1206 X5R 100uF ceramics considering derating) should be enough considering the 75mA total current draw of the circuit (3.3V).

author: Ian.M
author_rank: Super Contributor
timestamp: May 05, 2019, 06:01:24 pm
content: Its much more effective to add capacitance before the regulator. Assuming your regulator will work down to 25V (I'm guessing its a switching regulator as dissipating 3.8W in a linear regulator would need heatsinking), and starting from 10% low input voltage, so 48.6V, there's 868 Joules per Farad between those two voltages (from E=C*V2/2) You need 75mA, @3.3V, which if the regulator is 80% efficient is 0.31W input power. To hold the 3.3V rail up for 1 second only needs 356uF, or 0.356uF for 1ms.If you are using a linear regulator the maths is easier. Assuming it drops out at 8.6V, the input can fall 40V. From Q=CV, allowing 5mA quiescent current for the regulator and 75mA load current, it would tahe 2000uF to hold it up for 1 second, or 2uF for 1msThe tradeoff is the higher voltage caps required are larger and usually more expensive than low voltage ones. OTOH the 3.3V rail is maintained in spec. for the holdup time.

author: Axk
author_rank: Regular Contributor
timestamp: May 05, 2019, 09:10:48 pm
content: @Ian.M, thanks!I didn't think in terms of the regulator giving a lot of margin to how far a capacitor on the high side can discharge before the low side starts sinking.I though I would need an electrolytic on the high side, and I want the board to be low profile.So as you point out considering the margin I can use a couple of 100V chip ceramics instead.I'm planning to use a linear reg with an external TO-220 NPN with the whole board attached to a somewhat longer 2mm aluminum sheet and the transistor screwed to the sheet where it sticks out to keep it level with the board.I could use a switching converter but I want to keep things simple.

author: Axk
author_rank: Regular Contributor
timestamp: May 07, 2019, 05:40:50 pm
content: How do I determine if my PMOS switch can withstand the current surge needed to charge the 10-20uF of capacitance?The maximum instantaneous current is specified at 3A?Can it withstand more to charge the caps or should I use a current limiting circuit?I 20 ohm resistor in series with the capacitors?

author: Ian.M
author_rank: Super Contributor
timestamp: May 07, 2019, 08:58:06 pm
content: Choose a gate resistor to limit the PMOS turnon speed so the capacitor charging current doesn't exceed Id_peak. Some experimentation may be required with a current probe and a DSO,

author: RGD2
author_rank: Newbie
timestamp: June 09, 2022, 08:55:51 am
content: I'm just hitting this one myself: Needed the reminder about possible corrupt data writes.Signed up here just to chip in.My solution was a small FRAM chip, and I was going to also use the same sort of loss-of-unreg-input detection. Just a resistive divider to input, looking for the 5V rail to start dipping.So I was just going to use a trimpot, so as to bring the 5V down to just where the pin would register it as definitely a low if it dips.Will still give it a try, but the other circuits look more reliable (and reproducable). I'll report back in if it does.I decided to go with an FRAM chip - these write extremely fast, could fill a whole 2Mbit chip at 25MHz in only 84 ms or so, let alone a paltry few bytes.I was looking to try 'wear levelling' the thing, but it turns out you actually can't: FRAM's wear on both read and write cycles. Anything you try to do to wear-level therefore won't work: Every boot you would need to read to the 'last used' place somehow, and that just puts more wear over the front of the chip every time anyway.The reason I was hoping I could, was with 10^12 cycles (assuming basically only writes wear), I could just update on change, and keep appending-writing, with a 'only increments' counter tacked on, that would make the chip look like a circle buffer: recovery would be looking for the sudden downstep in that counter on boot to find the 'last' write, so as to pick up where it left off, and this would wear level nicely, and allow a small packet to gradually use up write cycles across the chip, thus allowing keeping a short section of critical data always up to date.But alas, can't do it: Reading also wears the chip, and you're reading as often as you write, or more so if you're trying to wear level with a scheme like that. Please anyone who can see how I'm wrong: show me that I'm wrong, I'm not 100% sure. Intuitively though it feels like wear levelling only works if reading is 'free' and only writing hurts. The problem is, on boot, how do you know the next spot you should wear, without causing wear? On eeprom that's easy - you can read without wearing. But if reading wears...So my solution is:Just write vector of critical data in a burst, packed to a multiple of 8 bits, when it is suspected that a power outage is about to occur, then disable the chip.Remember the 'write enable latch set' command first.This could go out at 20 - 40 MHz depending on which FRAM chip you get.Even with how 'high power' an FPGA is, this won't take long, since you don't have to wait for the slow writes of EEPROM with FRAM.On boot, just read from address zero your burst of bits.No wear levelling, and the chip will only last so many write cycles, so tune to where it doesn't write unexpectedly.Maybe count them and keep the number in your critical bits, and occasionally read/check how it's going? Just want to notice if it has started to hammer on the FRAM - an FPGA could easily wear it out unexpectedly in a few months. So long as you avoid that, life should be so very long that any wear levelling scheme is just not worth the effort.There was a guide to FRAMs for engineeers here:https://www.eetimes.com/an-engineers-guide-to-fram/. (Which of course I only though to look for after asking myself, "Hey, do reads maybe also wear FRAMs?"... yes!).

author: Ian.M
author_rank: Super Contributor
timestamp: June 09, 2022, 09:38:16 am
content: @RGD2,Welcome to the EEVblog forums. May I be the first to compliment you on the quality of your first post here.Yes, I think your wear levelling argument is valid - you can't do it for FRAM without an auxiliary NVM using different non-destructive read technology to hold at least some of the wear data.However even a humble 24C00 16 byte EEPROM or a few words of self-writable FLASH memory in your MCU*could let you do that - divide your FRAM into blocks, track each block's usage within itself, and once it reaches your chosen wear threshold, copy that block to the free block with the lowest wear, update root block pointers, and flip the corresponding bit in the auxiliary NVM from 1 to 0 to disable the worn out block. It goes a little differently if the root block wears out, as it has to be possible to find the root block from only the auxiliary NVM contents. You could store a pointer to the root block in the auxiliary NVM (which should change rarely enough that endurance for the pointer isn't an issue), or you could define the root block as the first block that hasn't worn out, and shuffle blocks to keep that so whenever it wears out.As each bit in the auxiliary NVM's block map only has to flip from 1 to 0 once, wear on it shouldn't be an issue. Some higher level checksums and error correction codes, and even tell-me-three-times redundancy may be required for the auxiliary NVM to handle cases where corruption is possible e.g. due to power loss during an auxiliary NVM write.* FPGAs aren't very efficient at implementing complex but slow logic directly as it typically requires as much resources as full-speed logic of comparable complexity. Once the complexity goes over a certain threshold, an 8 bit soft-core MCU to manage supervisory tasks may well be the best option.

author: AndyC_772
author_rank: Super Contributor
timestamp: June 09, 2022, 12:51:08 pm
content: Quote from: RGD2 on June 09, 2022, 08:55:51 amIt's all down to how often various things happen.Suppose you have to write a state at regular intervals. You can spread out the wear by treating the NV device as a circular buffer, divided up into blocks with a sequence number attached to each one. The last state is always the one just before the discontinuity in sequence number.You might write the state many times per second, and this wear levelling algorithm is effective at distributing the write cycles across the entire device.When the system boots up, it has to read the device - possibly the entire device - but that's OK. It's still only one access to each location per power cycle.Statistically, how often do power cycles occur compared to the number of times the circular write buffer wraps round? This gives an idea of the relative wear caused by the start-of-day search for the last written block, compared to the writing of the block themselves.

author: RGD2
author_rank: Newbie
timestamp: June 10, 2022, 03:10:55 am
content: QuoteGood point AndyC_772. Most of the wear will still be levelled, bootup access contribution should be negligable.In my app reboot is rare enough for it not to matter: the problem I need to cover is maintaining a performance guarentee of the system despite a short-duration power failure: the equipment being controlled won't tolerate a 'homing' sequence on boot: it will break something. There is only one open-loop stepper axis, and annoying, the 'safe' position is farthest from the only homeable endstop, so current position needs to be maintained despite power cycles. This whole requirement was one 'tacked on' to the end of everything else: The main job of the system is actually DI type fuel injection timing.

author: RGD2
author_rank: Newbie
timestamp: June 10, 2022, 05:47:11 am
content: @Ian.MThanks! You make a good point, and the scheme you detail certainly would solve that problem.Quote from: Ian.M on June 09, 2022, 09:38:16 amBy that point, I have previously sprung the on-FPGA resources for a swapforth/j1a core (or my personal j4a core): Almost full featured ANS forth compatible 16 bit core, that implements the Forth stack machine in hardware, not by 'stack is in ram, pointer in register', but 'both stacks exist in place of registers'.Runs easily up to about 60MHz, and that's one forth word call per clock, most of the time. Every indirect costs you a clock, but in general it executes standard 16 bit forth hilariously well. The hard-limited stack depths hurt less than you'd think, and it's a joy to use. (Seefor Jonathon's demo of swapforth j1a on a 1k gate ice40 chip).More recently, I'll just co-install an Arm SBC of some kind running some variety of debian. Then usually just communicate either by serial or USB (or both). Plain python is fine. Can then have stuff like tensorflow / openCL if really heavy lifting is needed: or just talk over the internet to a supercomputer.Certainly USB 2.0 HiSpeed does up to 30M bytes/sec reliably: So long as you have at least 8MiB of SDRAM on the FPGA side, at any rate. Through an SBC that does drop to only around 18MB/s or so, but I've found 16MB/s to be enough for my needs. This gets punted over the network, just using netcat on the SBC, and is 'caught' the same way elsewhere on a server with the space to put it to disk.I haven't looked into it deeply, but I do have a $20 FPGA board on my desk: A Colorlight 5A-75B, having a LFE5U-25F (25k gate) FPGA and 2x gigabit ethernet interfaces. There's already a open source RISCV soft core SoC project for this that works (including gigabit ethernet!) but I have yet to check what *sustainble* dataflow it can reliably manage, and whether the couple MB's sram it has is enough.Either way, the data capture pipe I tend keep completely separate from the 'control' FPGA: My business is basically getting evidence of running new equipment, and if it fails, we need to be able to learn from it. When it finally all works, the reports look totally boring, but that's what you want to see. Boring reliability. A whole bunch of pulses that look identical, apart from the natural noise. Better not to mess with what works, and has proven reliable so far. So the control FPGA is generally on a completely separate board, let alone chip. It's just easier that way.At most there's an additional link between them, which lets the control FPGA 'snoop' on selected live data.Most of the stream is somewhat of a black box: 8 channels of 12bit 0 to 3.3V data at 1MS/s, mostly watching hydraulic pressure sensors. Acts somewhat as a cross between an oscilloscope and and strip-chart recorder, but one which is more like a highspeed camera that is just running all day long: Really good for catching things breaking, really makes it easy to see what went wrong, since any 'triggering' can be done entirely post-process.You end up with some big files, but hard drives are really cheap these days. I use the 'snd' program to deal with them: works really well with arbitrary sample rate, bit depth, channel count and file size: Never crashes, just takes a while to read over the file sometimes, especially if you zoom out too far. But it comes back with a really neat overview, where you can find any glitch, no matter how small, and zoom in on it - it feels a bit like using google earth, because you can go from 'all day' to '100 microseconds' or so.But all the above is the recording / science side of the job: For controls, just putting the stuff that needs precision timing in the FPGA, with some dumb state machines, and setting up a low-latency link to where more complex code can run works well enough. The job is basically DI engine ECU, where I do DI injection and ignition timing, plus some other things as needed. The FPGA is just doing the job that the mechanical fuel system does - whilst being able to be a bit smarter about dealing with known injection latency, for example. This also does some correlation of deskewed pressure data with engine position, and having all that corrected for latency so it's collated in terms of observations at a give engine position make analysis much easier: separate data link and save flow for that.What really impresses me is how good the open source only FPGA tools are getting: At least for the chips they have good support for.The main thing, apart from being able keep all the toolchain as well as source code within the same box, is that it eliminates all the silly 'IT' failure modes. I've found such systems far more stable to bring out of mothballs after a couple of years. With everything else, there's another two battles: reinstalling the software, getting it relicensed (possibly repurchasing a time-limited license again) and then needing to fix any breakage that occurs because of the new versions.With the open source box, you can just turn it off, walk away, at let it sit for years. When you dust it off and turn it on, sure the OS is out of date, but it all just works, exactly as it was. Updating it is easy and quick, and usually painless.The OSS FPGA tools on a fast new PC are breathtakingly fast, especially compared to the older vendor tools. See for instance icestudio.io : It's like the arduino 'just an app' setup experiance, more or less, except going from changed verilog code to configured FPGA with blinking LED is literally seconds. Faster even than the same wait for a changed code on an Arduino platform design, or so it seems to me.But even on an embedded SBC, they're still about as fast as the vendor tools are on said fast PC: maybe 15 to 20 minutes, but doable, for that 8k gate embedded forth soft SoC FPGA recompile.With quicker turn around for development than anything but a live forth system, along with easy immunity to interprocess interlocks - which can be nasty intermittent problems to solve; I think even quite low end control cases should go to FPGA from microcontrollers, especially where servicability matters at all.It's quite feasible to now put the whole dev tools into an sd card, and have that card served in usable state via a web page running from a $4 esp microcontroller, where the user BYO's some wifi-connectable smart device which then provides all the RAM and compute needed to not only run a GUI, but also the whole toolchain including recompilation! I haven't done it yet, but it would be a good student project. esp-link , maybe some use of emscripten etc.That would save the cost of the SBC. Just stick a QR code in there to let whoever opens the door have their smart device connect up and point it at the entry web page. Can work entirely without internet access. Could use something like the cloud9 web-hosted IDE to make that quite comfortable too (I have hosted that from an embedded system, and it's fairly nice), or just use jupyterlab or so. IDK: not a web dev.At this time the FPGA tools are available already in architecture portable executables which run using WebASM (not in a browser).You can be up and running in a minute or two, on any platform. (YoWASP project). I do this using microsoft's WSL2 and it has been good. A little lower level / more flexible than icestudio.io, but very comfortable if you're comfortable in linux.As a side note: I've found dumb relay ladder logic, and dumb analog alarm relays (And a pulse-presence alarm relay) is all worthwhile for anything safety critical (especailly safety of people/environment, but also safety of plant). Should all be fail-safe and 'programmable' by turning a trimpot, with readily available E-stop buttons. No matter how 'smart' the programmable stuff works, day to day - safety is just a whole 'nother ball game! A couple threshold/alarm units, maybe some interlock switch-physical access locks go a long way to keeping absent-minded people safe.I did at first feel like I was going out on a bit of a limb, just using all open source toolchain stuff, but I've had excellent results doing so, and it's proven itself time and time again on some older apparatus that gets only intermittent use. It's easy to recommision, tailor to the new job, and be up and running in less time than it takes to just install/update the heavyweight windows software otherwise needed.projects Symbiflow, yosys, icestorm, trellis, apycula, YoWASP, etc are all very much like "gcc for FPGAs", and only improving with time. Probaby you'll get better design performance with vendor tools still, especially if you're chasing maximum clock rates or minimum space usage, but for what is typically otherwise 'arduino' territory, I think there's a very strong case for skipping over microcontrollers (including RTOS) entirely these days, in favour of a hybrid FPGA digital logic / Linux OS SBC approach.I expect eventually there'll be even smaller cheaper 'lower end' FPGA platforms, and there kind of is - c.f. Fomu FPGA.Support for some of the nicer older FPGAs: Xilinx spartans and machXO etc, is slowly appearing as well.Were I intel, I'd be trying to get on top of it.Anyway - I've got stuff to do. Sorry for the Rant, it's probably just I do have to deal with some colleages stuck in the 90's sometimes.

author: IDEngineer
author_rank: Super Contributor
timestamp: June 11, 2022, 08:20:31 pm
content: How does the FRAM "wear out"? If it fails to a known state, say all bits stick to 1's, you could accommodate a wear algorithm. Say you start at the top of the address space. The first valid byte going down in address is the first one not equal to 0xFF. As the current byte ages, eventually you write it to 0xFF and start using the next lowest byte. Further reads of the aged-out bytes will always read as 0xFF (in this example), either because that's the last value you wrote or because the cells are dying. Either way you will always read 1's and the algorithm works until you wear out the last byte, which should take a very long time.

author: OM222O
author_rank: Frequent Contributor
timestamp: June 12, 2022, 05:57:13 am
content: You don't really need to monitor the power at all. The "state" only chages from time to time, so save it every time it changes. If the "state" is updated infrequently (user settings, calibration data, etc.) save them on an EEPROM. If the the "state" changes frequently (FSM reovery after power outage) save the data to a SRAM. Then you can choose a large enough bulk capacitor on VCC so that the system can run for the time that is required to save the state (typically a few us or ms depending on what your storage device is).

author: jeremy
author_rank: Super Contributor
timestamp: June 12, 2022, 07:38:40 am
content: Maybe this can help you:https://www.microchip.com/en-us/products/memory/serial-eeram

author: RGD2
author_rank: Newbie
timestamp: June 14, 2022, 01:02:57 am
content: Yes, that's what I considered first up: Just update every time it chages, easy, already ready for an outage.Unfortunately, it is a process value (physical posiiton of an actuator) which can change quite often: The way the numbers turned out for me, it's basically necessary to only hit the FRAM when it matters - especially without a wear levelling mechanism.Sure, FRAMs last a long time, but to be readier for Murphy: I'll assume that when it does fail, it just starts reading out garbage data.Which will be a problem. So we'd rather it take at least years, preferably decades to get there. Just in case.My stuff is usually mostly prototype. But this particular project is getting a bit higher TRL now, so I do have to consider the typical expected lifespans in the application (dominated by the equipment it's attached to).I'd have just had the thing home into a stop on a reset: Except the physical stop I can do that with is at the end that would break the thing it's controlling, in the case of a unexpected power upset triggering a reset. And this is reasonably forseeable enough (remember: Prototype? But it is in an industrial environment, with high voltage things about, lots of vibration and heat, etc) that whilst a 'small issue', I do worry about it.My job title is "Experimental Scientist", and I can advise that you don't REALLY know Murphy's Law properly until that is also your job title!We have had this kind of 'unexpected FPGA reset' occur before, on another plant. When things were moving.Slightly different circumstances, but it did result in destruction of a sensor, which turned out to have become irreplacable, and so that whole project got cancelled, because it just became too hard to continue. It was an internally funded speculative project, so there were no external stakeholders to appease, but it would have been some really helpful fundamental empirical science to have gotten sorted out.That one also taught me to *never* to rely on anything programmable in part of a 'safety' loop: Have some dumb non-reconfigurable logic, preferably just relay logic that any electrician could understand on inspect, to back it up! Comes under the 'sufficiently simple' clause, if it's only 'programming' is setting a trimpot or two. Oh, and it should fail safe, of course. In that case there was no safety hazard: Except to that one extra-sensitive 'golden' sensor. (To its credit, it didn't physically leak or burst - it just deformed permanantly. As you'd expect with about 200MPa unexpectedly against a 10MPa sensor).Ok, Back on topic:One could, of course, just update the value 'occasionally'. But the problem there would be drift over time, power cycle on power cycle. It might actually work out ok for me, for now: I'm *mostly* only going to be power-cycling with steady data anyway.But... doing things that way might break other things unexpectedly down the track for someone else (I.E: future me). Particularly if it gets extended to track other, more often changing state, which is reasonably forseeable also. I'd rather not build in a pitfall like that.I was going to monitor power, and lock down the stepper driver the moment power failure was suspected: Until recovery above some threshold with some hysteresis at least, to accomodate brown-outs properly. The 'backup' would be triggered and take, for my clocks/data, only about 3 microseconds, long before any of the FPGA's power rails would even start to dip.I am planning to watch both the 5V board input rail, AND the 24V input which that supply in turn runs off: the former in case of funny business on that supply, which is distributed to other things, but the latter would be most likely to catch true outages.I have spare pins, so I don't mind using a couple per: One as the monitored input, with just an adjustible resistive divider. The other pin as an output to add hysteresis using analog feedback via another resistor: So it goes low if the 24V rail dips under say 19.2, but recovers if it gets up above say 23.Once this is proved out, I'm going to consider it a standard addition to all my FPGA deployments: I'll do at least a PCB module for it, and also advise it gets 'built into' the final product.Previously, I've 'saved state' by just never turning it off: This wasn't critical state though, it was just for convenience.But it would be nice to have a easy way to throw in a 'odometer' / 'hour meter' / 'lifetime cycle counter' on most of my work.

author: Bassman59
author_rank: Super Contributor
timestamp: June 14, 2022, 04:50:40 am
content: Quote from: IDEngineer on June 11, 2022, 08:20:31 pmIndeed, how does FRAM wear out? I haven't seen any evidence that it does, and this is with using them for parameter storage in a device which lives in a radiation environment.The only caveat with them -- or at least with the Cypress/Infineon parts I've been using -- is that you must adhere to the requirement that you want for at least one millisecond after power stabilizes before attempting to access them. If you try to do an access before that, they just lock up and you can never access the device until you cycle power again.

author: jeremy
author_rank: Super Contributor
timestamp: June 14, 2022, 05:23:53 am
content: I don't have any personal experience here, but at least fujitsu believes that wear-out is a problem:https://www.fujitsu.com/jp/group/fsm/en/products/fram/device/parallel-8m-mb85r8m2ta.htmlAnd they have obviously made some change in their process which they feel guarantees a longer lifetime, so the lifetime must not be infinite in their testing.

author: ajb
author_rank: Super Contributor
timestamp: June 14, 2022, 03:15:08 pm
content: Quote from: RGD2 on June 14, 2022, 01:02:57 amIt's not terribly uncommon in industrial machinery to handle this at the servo driver (or even encoder) level, with dedicated batteries specifically to maintain a record of an axis position. Keeping the encoder circuitry powered can even allow the driver to capture any externally-driven motion, like someone manually moving the thing when the machine power is off. But is there a reason you couldn't just add some other reference switch to the axis at a safe position? Or have a system monitor that locks down motion control immediately on power loss until the operator reconfirms the position? You have to establish the initial position of the actuator *somehow* anyway, right?

author: RGD2
author_rank: Newbie
timestamp: June 21, 2022, 12:49:31 am
content: Quote from: ajb on June 14, 2022, 03:15:08 pmBoss didn't let me 'complicate' the setup by adding a simple absolute position sensor. Ie, a switch.Given this axis is just actuating the mechanical fuel advance on a diesel mechanical DI pump, on a genset: It's not likely to actually move all that much. It's just bad if something glitches out during a run, especially if it 'homes' to the position which correlates to maximum advance! A homing switch to where I could home it to the least dangerous position would have been good.Anyway: Update: I have a simulation of the arrangement I used to monitor the +24V supply going to the stepper driver:http://falstad.com/circuit/circuitjs.html?ctz=CQAgjCAMB0l3BWcMBMcUHYMGZIA4UA2ATmIxAUgoqoQFMBaMMAKACcKNDwuRC88PblRRo47PgKGTBmFFBDZckCf0FheeONJFiVAcxkg5R3ABYFKgA5GN3NeDDyIVFQCUQeM4K2fvfC1cQMypsaGxLKGgECS8fKjjwQiCwQkoWAGVPbTs-dV4ggDMAQwAbAGc6BRRY-zBieUSTKlT0wyaMRv9zSxYNeQReXIQUfOEQcjA8aH8UQMtokAA1AHtSgBdi-ToWFeCAhRDiQSQYF3AWIA:(The URL encodes the circuit, which is why it's so long).The 1.7k resistors are just from the hysteresis output being configured to 2mA drive, but I ended up having them set for 4mA anyway. But the 22k/3k3 value were just 'handy' because I have a bunch of those set up for angle encoder inputs elsewhere: I can say that 9.1k for the feedback resistor is definitely too much conductance: It works, just the once during power down, but doesn't 'reset' if the rail comes back up. (which I did want it to, so as to cover for glitches).The 22k feeback setup I am using (But with CMOS3.3V output, DRIVE=4) gives a hysteresis of about 320mV on the scope. I have the hysteresis driven by a register, not just assynchronously fedback. The FSM it's in is part of what handles triggering the freeze-and-save state, so there's a fairly long delay built in while it waits for the FSM actually moving the data to report success: Less chance of oscillation. YMMV if you just feed it back assynch.The 'critical' data here is the accumulated stepper position, and the setpoint for it. The stepper FSM just runs linearly and quite slowly, and if power down is detected, it gets 'frozen' whereever it is - but will resume if the power comes good again. The sense is where if the stepper power fuse blows, this will be interpreted as loss of system power too - since the same supply ends up powering the FPGA via a 24V to 5V supply.I don't know whether I'll end up being able to post the code: I'll see if I can get it released under a copyleft license. It's not pretty though - typical 'scientist' code, certainly no better than at most a reference implementation for the real software engineers to start with.

author: RGD2
author_rank: Newbie
timestamp: June 21, 2022, 03:38:33 am
content: Quote from: Bassman59 on June 14, 2022, 04:50:40 amThanks you!Kept getting mangled data reads during recovery, waiting a bit and starting then, vs right after config, seems to have fixed that for me.

author: RGD2
author_rank: Newbie
timestamp: June 21, 2022, 03:58:31 am
content: Another update: Still not entirely proven to work reliably, so YMMV.Out of time to fix it for now, I can ameliorate the main part of the problem on a full hard-reset now, so this goes on the backburner for now, sorry people

author: ledtester
author_rank: Super Contributor
timestamp: June 21, 2022, 06:18:19 am
content: Use MRAM:https://www.edn.com/fast-high-capacity-and-endurant-mram-for-space-applications/This 32Kx8 SPI chip is in stock at Digikey:https://www.digikey.com/en/products/detail/MR25H256ACDF/819-1064-ND/8286370