Help me select an embedded filesystem.... - Page 1
Forum section: electronics / general

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 01:45:26 pm
content: I need help....Thru the years I have been using commercial filesystems for embedded systems and products. Now has the time come to use "something" else for many reasons (the system we have used until now is pretty old and not optimum).What I'm looking for is something that will allow me to use NAND devices (f.ex MT29F1G01 etc), so it should include wear leveling etc. I have been looking at littlefs among others but I'm afraid that littlefs is "too little", there are a number of restrictions as I read it as well as some potential problems (f.ex changing the header of a large file etc). I have of course also used FATFs for projects, this is ok as long as one requires FAT support (on SD cards for example) but for embedded systems that must be resilient to sudden poweroff conditions, its a no-go.I have also looked at SPIFFS, but again I'm not sure this is perfect.Regarding memory footprint, I don't care if it occupies 20 or 30 KB, and maybe 10 KB of RAM, I'm not after super small sized. Important is that it is well tested, can handle power down situations and be reasonable easy to port (and that it can support different storage types, NOR, NAND etc)I know its a long shot, but perhapsyouhave the right suggestion ??

author: krho
author_rank: Regular Contributor
timestamp: January 11, 2019, 01:59:42 pm
content: F2FSfrom google.. but that's a linux filesystem, so this is probably not what you are after.

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 02:09:41 pm
content: Thanks krho,sorry I forgot to mention that, you are correct, what I'm after is "standalone" system, something that has no requirement for linux etc.I'm usually using a RTOS on my products, but I would rather find a system thats not relying on anythingI know its a tough one

author: legacy
author_rank: Super Contributor
timestamp: January 11, 2019, 02:23:40 pm
content: Quote from: cgroen on January 11, 2019, 01:45:26 pmwhy?

author: legacy
author_rank: Super Contributor
timestamp: January 11, 2019, 02:30:00 pm
content: a lot of time ago, I developedthistiny project, then I switched to a completely different designHave a look, and feel free to reuse.

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 02:43:31 pm
content: Quote from: legacy on January 11, 2019, 02:23:40 pmFAT is not very good at handling sudden powerdown, and FAT on a NAND device is also no good idea if I'm not mistaken (FAT table will be rewritten, wearing the flash)FAT with some sort of journaling is another possibility maybe...(funny little video and simulation for FATFs and littlefs here:https://os.mbed.com/blog/entry/littlefs-high-integrity-embedded-fs/)

author: legacy
author_rank: Super Contributor
timestamp: January 11, 2019, 02:54:58 pm
content: Quote from: cgroen on January 11, 2019, 02:43:31 pmwell, we are using FAT in avionics. You are probably talking about a weird implementation, but the design of the FAT filesystem itself is so easy that you can complete every transaction without the need of cashing anything.Novel NetWare is an example of FAT implementation where between a request and a IO-sync ... there might be up to 10 seconds (this was chosen in the 90s to speed up performances), which means ... you seriously needed a UPS.

author: legacy
author_rank: Super Contributor
timestamp: January 11, 2019, 02:58:03 pm
content: Quote from: cgroen on January 11, 2019, 02:43:31 pmif you need this, you'd better look into the MTD/FlashDevices (NAND/NOR) subtree of Linux.Anyway, you'd also better use a Linux router (10 Euro) as filesystem server, attached to the ethernet or USB-on-the-go so you can still use your RTOS without the problem of choosing and implementing a filesystem.Personally, it's exactly what I did for my project: I moved the problem to Linux, for which we are plenty of cheap boards.edit:LOLLL, the old Commodore C128 implemented something similar, in fact, the 5.25" harddrive is an external standalone minicomputer that accepts requests on a special custom cable and provides answers.Something likeC128: Hey? List all the files on the floppyFloppyDriver: ok, here they are .... list[]={ ... }C128: hey? Open this $filename, and move the pointer to the first blockFloppyDriver: ok, doneC128: hey? read 100 blocks of dataFloppyDriver: ok, here they are, block[]={block0...block98, EOL}, sorry there is no block#99C128: ah, ok! Close the file.and so onFunny, because even some AS/400 computers by IBM do something similar

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 03:21:40 pm
content: Its not an option to include a Linux router in these systems, its after all "embedded" (and some of the products are sold in 50K+ numbersSome of the products are battery driven and left alone for years, some are connected to the internet using lwIP (and 4G modems) etc etc. I know it might be a long shot to find something that will "cure all" problems....Regarding FAT, according to many sources, this is a no-go if you need something that can cope with powerloss (and still keep the filesystem intact _always_). At least I have not seen any that would do that, however I have seen a lot fail in this regard.There is also this one:https://www.freertos.org/FreeRTOS-Plus/Fail_Safe_File_System/Reliance_Edge_Fail_Safe_File_System.shtmlWhich is open source, and there is a possibility of a commercial license (which I have absolutely no problem in), it is more that I would like to be able to play around with it before handing over the money

author: SiliconWizard
author_rank: Super Contributor
timestamp: January 11, 2019, 03:31:55 pm
content: https://yaffs.net/

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 03:35:54 pm
content: Quote from: SiliconWizard on January 11, 2019, 03:31:55 pmThats not bad! Thanks for the pointer, I will have a very close look at that !EDIT: although the "low memory footprint" is somewhat "high":https://yaffs.net/archives/yaffs1-memory-footprint

author: JVR
author_rank: Regular Contributor
timestamp: January 11, 2019, 04:12:06 pm
content: The solution is easy, use NOR.I've used FAT systems on NAND, and paid the price. Unless there is a very specific need for large files, I'd much rather stick to smaller NOR devices. If you have to go NAND, read the documentation of the filesystem and wear leveling algorithms very closely, edgecases can bite you in the ass.I ended up writing a pseudo-static filesystem with dual redundancy on NOR, and the test was 200 units on a contactor that was reset randomly within 100s for a week, not a single device with the NOR failed, every single FAT device (with logging) was dead.

author: legacy
author_rank: Super Contributor
timestamp: January 11, 2019, 04:20:33 pm
content: well, supercapacitors are not so expensive nowadays and they can be used as a source of emergency energy to drive the MPU in order to correctly shutdown the filesystem.sort of: ops, the main power supply has just failed, don't panic and let's raise an interrupt with the highest priority possible so the MPU will stop everything it's doing and use the emergency energy (which is limited to a couple of seconds before the voltage drops on the supercapacitor) to correctly write things on the media, write things on the journal, flush things, and shut down the filesystem.why not?

author: JVR
author_rank: Regular Contributor
timestamp: January 11, 2019, 04:21:16 pm
content: True, and I've seen Linux based systems do this. I had the problem of not having the luxury of being able to solder to the ~9k devices installed in cars in the field. We used an expansion header to host the mezzanine board for the fix, so techs could just plug and play.You make a good point though, but I'd still stay away from NAND unless I had too. Random bits going dark FTL.

author: cgroen
author_rank: Unknown
timestamp: January 11, 2019, 04:44:31 pm
content: Thanks JVR,I'm (besides NAND) currently using Dataflash (atmel/adesto) with ok results (using a commercial filesystem). I can though see a demand for larger filesystems in the near future so I'm not sure that the NOR route is the right one to go, although the higher stability is a very nice bonus....The "super capacitor solution" is a no-go. Some of the devices this is going to be used in, has no room for stuff like that (especially size). I also have been bitten during the last 30 years a couple of times thinking that the hardware takes care of the powerfailure just perfectly, and then there was some corner case where it did not (temperature, batteries behaving strangely etc etc). I would rather have a filesystem that would not break regardless what the powerswitch was doing...

author: JVR
author_rank: Regular Contributor
timestamp: January 12, 2019, 08:55:15 pm
content: Dataflash behaves much the same as NOR, its stable, but small. You van get NOR up to around 64MiB for rather cheap these days. Well not cheap when compared to an SD card, but cheap for what you get.Do note that SD cards and eMMC are not the way out, they were not designed to work with small files that change often, the wear leveling will kill you. If you really need more than 64MiB, I'd second yaffs, and say have look at Micrium's file system, its very well written, but you do pay the price for it.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 12, 2019, 09:21:42 pm
content: If you're not going to move your storage device to other places, why do you ever need a filesystem. Just treat your device as a flat memory. You can use this memory to organize circular buffers, which is very handy for logs, or other storage schemes. If you have several tasks, divide your device into different areas and give each task its own area. This way there will be easy to make sure the data is intact in case of crash.

author: nctnico
author_rank: Super Contributor
timestamp: January 12, 2019, 09:39:53 pm
content: Quote from: cgroen on January 11, 2019, 02:43:31 pmYou have to make a distinction here. SD cards, CF cards, SSDs, eMMC, etc are managed flash devices which do internal wear leveling. Using FAT on these is perfectly OK. Although you'd still won't have any journalling and recovery besides that shadow FAT table which should allow to roll back a mishap to some extend.If you have an unmanaged (raw) flash device you'll need a layer which does error correction and wear levelling. You'd be looking at BCH error detection & correction and a wear levelling layer like UBI. On top of UBI you could use FAT or EXT4. There are several bootloaders (uboot, barebox) which have implementations of UBI and a fileystem in an embedded environment.And easy solution would be to use FATfs on an eMMC device.

author: cgroen
author_rank: Unknown
timestamp: January 13, 2019, 08:49:33 am
content: Thanks for all the ideas so farI need a "real filesystem" and not something that is too specialized. The products are used by customers as platforms for their own products (that they program them self) as well as for our own in-house products. Because of that I need a "normal" filesystem (open, close, write/read, subdirs etc).The eMMC I have looked at before, but if I remember correctly, it was a relatively expensive solution? yaffs2 seems like a candidate as does reliance from datalight. But maybe there are others "out there"....

author: nctnico
author_rank: Super Contributor
timestamp: January 13, 2019, 05:05:40 pm
content: How many units do you expect to sell? Getting the wear levelling layer and the filesystem on top of it running will take some effort which costs money (engineering hours). You'll also need to do extensive testing to see how a solution actually behaves. UBI combined with ext4 are quite resillient and have been thouroughly tested.

author: cgroen
author_rank: Unknown
timestamp: January 14, 2019, 08:13:19 am
content: Quote from: nctnico on January 13, 2019, 05:05:40 pmWe are talking 100K+ per year (products are currently selling at that rate, this is for a new line of products/projects). But, at the same time I would also like this to work on small projects, basically I need a system that can be used in as many places as possible. I will have a look at UBI also, thanks for pointing that out!

author: JVR
author_rank: Regular Contributor
timestamp: January 14, 2019, 08:19:12 am
content: What is the file read/write size and frequency? What is the total amount of data you expect to store?

author: cgroen
author_rank: Unknown
timestamp: January 14, 2019, 08:25:58 am
content: Quote from: JVR on January 14, 2019, 08:19:12 amThat is a good question! As the products are used by customers for their own application (on top of our API/operating system) it can be hard to say (some of the projects the products are used in we don't know the details of). But for the end products we do on the system ourself, it is typically a couple of updates (maybe 1 KByte) every 10 minutes or so. Space is typically in the 10 to 20 MBytes. But this can vary.

author: Jeroen3
author_rank: Super Contributor
timestamp: January 14, 2019, 09:11:09 am
content: I'm interested in what you will find out. FAT is generally a poor filesystem anyway. The only reason it's still around is for compatibility.Yaffs looks promising. But there are also ext projects out there for bare metal systems.

author: legacy
author_rank: Super Contributor
timestamp: January 14, 2019, 09:54:14 am
content: Quote from: Jeroen3 on January 14, 2019, 09:11:09 amFAT32 has the great advantage of having low complexity, and it's the reason why it's chosen for embedded stuff.

author: JVR
author_rank: Regular Contributor
timestamp: January 14, 2019, 10:46:55 am
content: Quote from: cgroen on January 14, 2019, 08:25:58 amI would be very weary of managed flash in that case. Most of their internal wear leveling algo's only really work if you write multi-sector amounts of data at once.I also came across this the other day, I have no experience with it though:https://os.mbed.com/blog/entry/littlefs-high-integrity-embedded-fs/

author: Jeroen3
author_rank: Super Contributor
timestamp: January 14, 2019, 01:32:59 pm
content: They "sell" LittleFS right. Generally mbed delivers working (highly abstracted) code. Licensing might be troublesome though.

author: cgroen
author_rank: Unknown
timestamp: January 14, 2019, 01:50:15 pm
content: Reading about littlefs it seems there are quite some shortcomings...I have it running on a RT1062 here (using some of its XIP QuadSPI flash). But reading on the github page for littlefs and some of the questions there, it seems that is not quite "there yet"...