SD Card advice please. - Page 1
Forum section: electronics / general

author: mikerj
author_rank: Super Contributor
timestamp: October 12, 2025, 01:12:52 pm
content: I have been developing project using a Teknic ClearCore and now need to regularly save small blocks of data to NV storage but the only option the Clearcore offers for this is an SD card slot. High level requirements/details are:1. Must survive unexpected power outage with either no data loss, or only the loss of a record being written during power out.2. I would very much like the data on the SD to be readable by PC, but binary files are fine.3. Record is 64bytes including CRC and I would like to save one record every minute.4. Happy to use the expensive industrial rated cards that claim to provide some level of protection against power loss.5. I intend to use the SDFat library.6. Time/date is available to the Clearcore via an HMI connected over Modbus RTU.My initial thoughts are to preallocate sufficient fixed length files to store data for 5 years and then write directly to the blocks within those files to avoid any issues with FAT tables being modified. Files would be named with sequential numbers so I can scan through at startup to determine the last one which has valid data and find the next block to be written.Questions:1) Can each 512 byte block in a file be treated as regular flash memory, e.g. if I have a 512 byte buffer in the ClearCore that I gradually fill with 64 byte records and I write that data to the same block after each record is added (i.e. 8 writes to each block) is the only flash wear from the bits changed, or does the SD demand an erase prior to every block write?2) If I get power loss during a write could this corrupt the entire 512 byte block, or would it be limited only to the changes made since the last write?3) Is it possible for any wear levelling algorithms built into the SD bite me in the rear or will this be transparent?4) Should I be considering a different scheme entirely?

author: Peabody
author_rank: Super Contributor
timestamp: October 12, 2025, 02:03:04 pm
content: I think your plan to create the files ahead of time is very good. The Arduino version of SdFat accomodates that, and lets you erase the entire card and then format and create the needed contiguous files. The Arm version probably does too.However, SD cards only write in blocks of 512 bytes. If you send data to the card, the card's controller writes the data to its ram buffer, and only writes another sector when the buffer is full. So that buffer is also at risk from a power loss. Of course you can collect 64 bytes, then send 512 bytes to the card, including 448 bytes of null or FF fill, and then come back and write again to the same sector after 128 bytes, and so forth, but each rewrite requires an erase first, or in modern cards the logical sector you're writing to probably just gets assigned to a different physical sector. But I would think that doing that over and over would be really abusing the card, and might reduce its life. So maybe you could think of 8 readings as being at risk from a power failure.But maybe the better approach would be to have a battery backup for your device. It wouldn't have to be a big battery, just enough to give you time to do a deliberate shutdown without loosing any data. You would just need a way to detect power loss. And you could have a mosfet that controls all power coming to the device from the battery, controlled by your processor, so after saving the last readings to the card, it could shut down power completely to prevent the battery from being completely discharged from an extended power loss.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 12, 2025, 02:15:39 pm
content: To minimize the probability of corruption, especially if you use a filesystem such as FAT, your best bet is to preallocate files with their max size and write in the preallocated clusters. Your data structures would contain an indication of their size so that the size of the file would not indicate what's in them. If you don't do that, the filesystem can very easily get corrupted if a write operation doesn't complete.Note that if using FAT16/FAT32, it's possible only if you have low-level access to the write functions. Otherwise, use exFAT if possible, as file preallocation is an integral feature of exFAT. I think FatFs supports exFAT if you enable it.It won't do wear-levelling on its own. Wear-levelling with a FAT filesystem is annoying. It would essentially consist in re-writing a file at a different location (so reallocating its clusters) everytime you write to it, and that increases the likelihood of corruption if the writes don't complete. I know some SD cards do implement wear-levelling with their embedded controller, but unfortunately it's a spec that is often hard to get from manufacturers.It's probably better to use something like LittleFs, which handles wear-levelling and protection against corruption.Sure some sufficient bulk capacitance will help increasing the odds of the writes to complete in case of power loss. But keep in mind a SD card typically draws more power than a single Flash or EEPROM chip.

author: nctnico
author_rank: Super Contributor
timestamp: October 12, 2025, 02:26:37 pm
content: Quote from: SiliconWizard on October 12, 2025, 02:15:39 pmFrom my testing, LittleFs is even worse for logging purposes. I ended up with over 100% of filesystem overhead and from accellerated testing it turns out LittleFs is prone to data corruption as well. After writing & deleting several thousand files, LittleFs also became very slow unlike FatFs. I decided to stick with FatFs.

author: radiolistener
author_rank: Super Contributor
timestamp: October 12, 2025, 03:22:47 pm
content: Quote from: nctnico on October 12, 2025, 02:26:37 pmThe slowdown is likely caused by accumulation of “dirty” blocks inside the SD card controller. You can run a TRIM operation (or reformat with full erase) to let the controller reclaim unused blocks — the speed should recover. However, running TRIM too often can shorten the flash lifetime, so it’s best to do it only occasionally or when performance noticeably drops.

author: nctnico
author_rank: Super Contributor
timestamp: October 12, 2025, 03:56:38 pm
content: Quote from: radiolistener on October 12, 2025, 03:22:47 pmFor clarity: this wasn't on an SD card but using flash memory directly.

author: mikerj
author_rank: Super Contributor
timestamp: October 12, 2025, 04:02:13 pm
content: Thanks all, I really appreciate the help.Quote from: Peabody on October 12, 2025, 02:03:04 pmThis was what I was trying to describe, writing the whole 512 block each time even though it would only have an additional 64 bytes of data each time. I was hoping the entire block did not need to be erased when the only bytes being changed were already in an erased state, as you could do with regular NAND type flash. In retrospect I could simply use one block per record, it feels wasteful but SD storage is so cheap even for industrial cards that it makes little difference. If I've done my sums right even 1GB would have sufficient space for 4 years worth of records and it gives me some future proofing.Quote from: SiliconWizard on October 12, 2025, 02:15:39 pmI can certainly use exFAT if that offers benefits and can also use FatFS with a little bit of work.Quote from: SiliconWizard on October 12, 2025, 02:15:39 pmAdding capacitance to to power rail is absolutely a possibility and the Clearcore does have an internal ADC channel to monitor the 24v incoming supply so I could check this prior to every write very easily.

author: rteodor
author_rank: Frequent Contributor
timestamp: October 12, 2025, 04:19:12 pm
content: SLC card recommended. Something likethis Endurance that for me outlived the Raspberry Pi's they ran on. So far I had one single logical issue once out of 4 of them running from 2014...2018.I am not sure that the erase size is 512 bytes. I vaguely remember some mention of being 4KB. Better check on that.As for write consistency you would have to give up one record in each 512B block. Then use that space to set flags (like its done in EEPROM emulation): mark the block as: used, not-used, write1 and write2. You will have last 2 blocks that would be both written with the same data until they fill up. write1 and write 2 will be marked with an additional flag that will beset before startingevery write andcleared after a writingis done. Be careful with block transitions and with recovery.Of course some big capacitors or supercapacitor will help but you could do without if you implement this properly.

author: amyk
author_rank: Super Contributor
timestamp: October 12, 2025, 11:32:30 pm
content: How much storage do you need? If it's what can fit in a big NOR flash or even serial EEPROM, use that instead for far better endurance and retention.If you write a 512-byte sector to an SD card, it'll write at least a page (several KB on most bulk capacity NAND these days). I don't know if they even make SD cards with small-block SLC anymore, but on those, a page is 512(528) bytes.

author: mikerj
author_rank: Super Contributor
timestamp: October 13, 2025, 02:51:32 pm
content: Quote from: amyk on October 12, 2025, 11:32:30 pmIf I could use serial EEPROM or FRAM I would do it a heartbeat but there is no facility to add this to the ClearCore, only an SD card slot. Initially I thought I could use a couple of pins on an unused Clearpath motor control ports for I2C but they are buffered and level shifted so that dashed my plans. I'd rather not butcher the inside of the ClearCore to do this.Quote from: amyk on October 12, 2025, 11:32:30 pmThat doesn't sound great, could a power out during a 512 byte sector write potentially corrupt the entire page? FYI SLC cards in smaller capacities are still made by Swissbit, but they are extremely pricey e.g. 512MB for $35