FatFS library delay behavior - Page 1
Forum section: electronics / general

author: zer0c00l
author_rank: Contributor
timestamp: September 08, 2020, 06:28:19 pm
content: I am working on a project that logs data to an industrial grade 2GB microSD card. The way it works is that whenever it starts up it creates a new directory and then creates a new XML file within that directory, and then starts logging data to that file. I have noticed that the delay between creating a directory and starting to log data keeps on increasing as the number of files increases on the SD card. It starts from 5 seconds then keeps going up, but when it reaches 13 seconds it goes back to 5 seconds. The graph belong illustrates this. Delay is on Y-asis, and number of files is on X-axis. Is this characteristic of the FAT file system or a problem with the SD card?

author: T3sl4co1l
author_rank: Super Contributor
timestamp: September 08, 2020, 06:58:54 pm
content: Have you added profiling anywhere?Maybe it's a memory allocation thing. Maybe it's a directory traversal thing. Maybe it's a FAT re/allocation thing. Does it vary with the size and number of files or directories? (What if you change how much goes in each file.) Are you making many small writes, or large (buffered/cached) writes? Is it allocating extra file size, expanding the file as it goes, or writing at final size? How many FS operations are performed in the stack, is it minimal or is there a bug causing excessive (perhaps accidentally quadratic) API calls?(Afraid I don't know much about FAT offhand, whether this is a common symptom or anything.)Tim

author: Rick Law
author_rank: Super Contributor
timestamp: September 08, 2020, 08:11:44 pm
content: Myguessis:FAT file structure uses allocation clusters. Each time when you create a file in a directory, the process inlcudes (1) looks at current directory cluster and look for an empty location, (2) if none found, add a new cluster to the directory (folder), now try again.So, after a few file creation, your current directory cluster fills up - the file creation process now needs to add an allocation cluster to that directory and links it properly in the FAT table itself. So, (1) the process of adding a new allocation block takes time, (2) your directory's file table is fragmented even if you start with a newly formatted disc: since your directory's allocation clusters is dotted between clusters for the data files.The triangle graph is: each new file takes a little longer and longer to find the next empty slot (little bump up in the triangle)- until it founds none, now do the extra work to add a new allocation block to the directory (top of the triangle) - you are back to quickly finding an empty spot, but slows down again (new triangle climb) as this new allocation block fills up.To test if my guess is right:Use the same file name, you should see that extra delay disappear. If so, my guess is right.It is just how FAT do directories. Alternatively, you can pre-create say 1000 files so the directory already has slots for them, that triangle will be one big triangle.It that turns out to be the case, changing filename length (> 8 dot 3) will change the shape of the triangle as well.EDIT: adding this:Unlike directories, ROOT has a fixed length root-directly table. You wont have a triangle but once it fills up, you can't add new file(s) anymore. Pick your poison...

author: zer0c00l
author_rank: Contributor
timestamp: September 08, 2020, 10:25:20 pm
content: Quote from: T3sl4co1l on September 08, 2020, 06:58:54 pmI am profiling by starting a timer before directory creation, and after the directory creation function returns. So its upto the user how much he wants to write to the card. The delay looks to be connected to the number of directories/files on the sd card irrespective of how much data is written to each file.Quote from: Rick Law on September 08, 2020, 08:11:44 pmThanks for the great explaination. ANother thing I noticed was that if I use a standard SD card instead of an industrial grade the delay goes all the way till 30 seconds and never goes back down (or maybe I haven't noticed it go back down yet because I didn't let the experiment run long). The triangle only forms with an industrial grade card

author: maginnovision
author_rank: Super Contributor
timestamp: September 08, 2020, 11:16:28 pm
content: This doesn't seem right. I can create directories on a teensy 4 or 3 or a custom xmos board and it's a very very short time to do almost any process. What do the controller specific functions look like? How about the other functions calling the fatfs library? What sort of controller are you using? This is entirely uncharacteristic of what I've seen with my own projects. Typically things are ready in 20 milliseconds unless the sd card gets busy and even then you're only looking at maybe 2 seconds. It's possible to get worse but typically it shouldn't be like this. Are you using some sort of rtc slowing the thing down?

author: zer0c00l
author_rank: Contributor
timestamp: September 08, 2020, 11:32:12 pm
content: Quote from: maginnovision on September 08, 2020, 11:16:28 pmUsing an LPC2368 mcu. The directories are created in a short time, but as the number of directories increase, the amount of time required to create a new directory also increases. The mcu has its ownLow level disk I/O module which is being used by FatFS library to access the sd card. I am not using any RTC

author: maginnovision
author_rank: Super Contributor
timestamp: September 09, 2020, 12:42:55 am
content: Is there a link to the fatfs code? FatFS doesn't include the low level io functions or the time support code so all the related code would be good. The config file may also have some issue but I'm not aware of any that would cause this exact issue. I don't expect the mcu to cause this issue(without issue being detailed in errata), or the sd card specifically (despite you getting different times depending on class of card) but the libraries/code involved.

author: MosherIV
author_rank: Super Contributor
timestamp: September 09, 2020, 07:32:06 am
content: I slept on it. My thoery is that the write on the sd card is buffered. When the buffer fills, it progressively takes longer before more data can be loaded into the sd card to write. Every 13 seconds, the write for the buffer completes and the buffer is emptied, so it can start again.Try a different sd card to see if the same thing happens or you get a different pattern.

author: Jeroen3
author_rank: Super Contributor
timestamp: September 09, 2020, 08:36:34 am
content: What you should do is use code profiling tool to see which function it slow. Now you only have all the time it takes, and no details about low or high level.See if you can use coresight tracing to tell which functions take time. If that is wired on your hardware...

author: zer0c00l
author_rank: Contributor
timestamp: September 09, 2020, 04:55:08 pm
content: Quote from: maginnovision on September 09, 2020, 12:42:55 amI have attached the lowlevel diskio functions for this mcu, which are being used by FatFS

author: Jeroen3
author_rank: Super Contributor
timestamp: September 10, 2020, 06:44:47 am
content: I've removed all dead and commented code from the disk_write function. I suppose disk_write is only ever called for single sector writes?Since writing multiple sectors will only write one sector multiple times.And MCI_Write_Block is the actual function doing the transfer? So this is just a translation layer?I think you need to look at the mci.h code? This file isn't doing any arbitrary timings I think.Code:DRESULT disk_write (BYTE drv,/* Physical drive number (0..) */const BYTE *buff,/* Data to be written */DWORD sector,/* Sector number (LBA) */BYTE count)/* Sector count (1..255) */{DWORD i;void *WriteBlockptr = (void *)(DMA_SRC);if (drv || !count)return RES_PARERR;if (Stat & STA_NOINIT)return RES_NOTRDY;if (Stat & STA_PROTECT)return RES_WRPRT;if (count > 1) {memcpy(WriteBlockptr, buff, BLOCK_LENGTH);for(i = 0; i<count; i++) {if (MCI_Write_Block(sector) != TRUE) {return RES_ERROR;} else {/* When MCI_Block_End_Flag is clear, it indicates TX is done */while (MCI_Block_End_Flag == 1) { ; }}}} else {// Single Block write// TODO: avoid copymemcpy(WriteBlockptr, buff, BLOCK_LENGTH);if (MCI_Write_Block(sector) != TRUE) {return RES_ERROR;//break;} else {/* When MCI_Block_End_Flag is clear, it indicates TX is done */while (MCI_Block_End_Flag == 1) { ; }}}return RES_OK;}

author: maginnovision
author_rank: Super Contributor
timestamp: September 10, 2020, 03:15:19 pm
content: All the actual IO is done by outside functions(all those MCI_ calls)... FatFS is only the FS level stuff. Any waits are only based on the SD card. You have callbacks for read, write, send commands, checking status, initialization, time(for instance creating a folder with a correct time stamp). Without those specific functions I don't think it's possible to see if it is software. I definitely agree you should probably do some deeper profiling for these functions. I still think it's probably software but you do need all the code available and to see if/where it's hanging when it does. If you try a NEW formatted sd card and it reacts the same it's definitely software, if not it's possible your card is dying(or maybe it needs to be reformatted).

author: zer0c00l
author_rank: Contributor
timestamp: September 10, 2020, 04:22:51 pm
content: Quote from: Jeroen3 on September 10, 2020, 06:44:47 amQuote from: maginnovision on September 10, 2020, 03:15:19 pmThe MCI functions are part of NXP's library. I haven't modified them in any way. Attached.

author: Jeroen3
author_rank: Super Contributor
timestamp: September 11, 2020, 06:28:11 am
content: Can you emulate what fatfs is doing with the SD card on MCI directly?I think MCI lib is most likely to cause this. Code has the most opportunity to act like this. Scopes volatiles, inline delays andretries(is that sd spec??).