datalogging sampling frequency bottleneck - Page 1
Forum section: electronics / general

author: Skyland
author_rank: Contributor
timestamp: November 25, 2014, 01:37:02 pm
content: My datalogging system which consits of the Flora, RTC, SD card breakout board, LSM9DS0 sensor is shown in the picture below;https://www.dropbox.com/s/d8pwcn59nq11lzs/2014-11-25%2012.26.56.jpg?dl=0The sampling rate/s is capped at 30 when writing into the sd card. If I disable the sensor and only save the time it goes up to 65. Writing into the serial interface rather than the sd card makes the rate goes up even higher at 127. for my application I need at least 100 samples/s. I am thinking the sd card is the problem after making the tests mentioned. The flora clock at 16Mhz is surely able to provide a higher sampling frequency. After checking the sensor datasheet and library I haven't found any limitation either. any input on this issue?

author: Opticalworm
author_rank: Contributor
timestamp: November 25, 2014, 03:13:33 pm
content: Hiya Skyland,Without looking at your code or flow diagram of your code I can only assume that you are opening a file on the SD card (say Temp.dat) and writing directly into it and closing the file when you are done.If so, are you keeping the file open between samples or are you opening and closing it between samples?The second will add extra overhead to each sample you make which could explain the slower rate.-Ron

author: Jeroen3
author_rank: Super Contributor
timestamp: November 25, 2014, 03:25:03 pm
content: Looking into my glass orb or truth, this is most likely the cause:SD cards, because of their flash, prefer to be written in blocks equal to (or a number of times larger) than their minimum erase size. Mostly 512 bytes or entire sectors of a few kB.If you need to write 1 byte to something not 0xFF (erased status of flash), you'd actually need to erase an entire flash page before writing it again. And erasing takes a lot of time.If your filesystem/disk driver does not have a buffer for this, you're going to have bad time.Not to mention a lot of unnecessary writes to the FAT table (you're using fat right?) wearing out the card in turbo mode.

author: PA0PBZ
author_rank: Super Contributor
timestamp: November 25, 2014, 03:28:37 pm
content: +1Jeroen was just a bit ahead of meI'd do the sampling on an interrupt and store the samples in memory. Then in a seperate process check the size of the buffer and save to SD if you think the time has come to do so.But even then I would be worried about the SD card's life...

author: dannyf
author_rank: Super Contributor
timestamp: November 25, 2014, 03:37:44 pm
content: QuoteYou will have to figure out if it is burst of reads or sustained, and how much each read takes up in memory.The simplest solution would be to buffer the reads in ram and send / save it via interrupts / dma.If the bottleneck is the adc, you could use faster adc or multiple adc and sample in parallel. Some chips allow ping-pong styled adc and that helps a lot as well.If all failed, you could look to optimize your code.

author: Jeroen3
author_rank: Super Contributor
timestamp: November 25, 2014, 05:47:36 pm
content: While searching about that little mcu board wearable you have there. I noticed it has a ATmega32U4 on it.This has certain limitation.- float types. Not going to happen.- sprintf is slow.- sd card is in spi emulation mode. (only 1 of the 4 data channels)- There is no dma- Arduino does not have the most optimized pieces of code.Banish software delays from all code you are using. You have 4 timers, use them.