Looking for Storage Device for my Logger - Page 1
Forum section: electronics / general

author: kolbep
author_rank: Frequent Contributor
timestamp: October 28, 2013, 06:34:15 pm
content: Well, My Voltage/Current Logger based on a Pic Micro is well under way.What I have managed to accomplish so far (with the help of you guys):Get the Pic Talking to the LCD using the XLCD libraries.Getting the ADC to read a 0 to 5 volt range (using a pot between + and gnd)Converting the ADC output to a Float representing the VoltageConverting Float to StringOutputting the Voltage on the LCD.And, I am actually quite shocked, the Readout on the LCD is tracking Exactly with my DMMI know I do not have a good quality meter, but I am pleased that the ADC in the Pic can track along with it.So far I have been doing this all on a MCD Demo board.

author: kolbep
author_rank: Frequent Contributor
timestamp: November 02, 2013, 05:46:50 pm
content: Ok, now that I have gotten the Voltage reading part right (as well as Max/Min/Avg), reading from a transformer and voltage divider,I am getting a bit stumped on how to store that much data.I plan on adding a RTC Clock Chip as well.and logging the Current usage via a CT.Now, If I look at logging for a full 24 hours,That is 86400 Secondsx (10 Bit ADC (Voltage)+10 Bit ADC (Current)) = 20 Bits+ (SSMMHH DDMMYY) = 96 BitsSo the total storage I will need is 86400x116=1.2MBThe only EEProms I see are in the Kilobyte range.I was looking at writing to a SD Card, but the Library(Base+Writing) for C18 is too large for the flash I have free in my device.Is there any I2C or SPI chips that will meet my requirements?ThanksP

author: Kaptein QK
author_rank: Regular Contributor
timestamp: November 02, 2013, 05:51:37 pm
content: Any large enough SPI flash chip?

author: Unknown
author_rank: Unknown
timestamp: November 02, 2013, 05:55:21 pm
content: You might look at what used to be Atmel AT45DB* Dataflash (now taken over by some other company). These are SPI flash chips available in sizes up to a few Mbit for around $1. I believe there are plenty of other SPI flash chips in this size.SD card would be the obvious solution that would be easy to read, so I would argue you should probably picked the wrong micro for this application.You might also look at a system that Fluke uses in their logging meters, which records a sample every 5 minutes or so, but will record additional samples if the value changes by more than a few percent. If the noise is low enough, run-length encoding might accomplish a similar result.

author: kolbep
author_rank: Frequent Contributor
timestamp: November 03, 2013, 04:58:03 pm
content: Is it normal to be excited about using a new pic? Ithink I wont be able to sleep tonight. 500 something pages of datasheet to read.Based on the suggestions, I have decided to continue the sd card route, and with that, get a bigger pic uC.So I am going up from a 18f2420 to a 18f26j50.Larger data ram and program memory, usb support, rtcc, etc, and it is 3v3, so less hassles interfacing to the sd card., and is compatible with my pickit2 and is sdip for my breadboarding znd board.hopefully that new micro will serve me for the next years project..I am ordering my samples from Microchip tomorrow.

author: mariush
author_rank: Super Contributor
timestamp: November 04, 2013, 05:22:10 pm
content: Now, If I look at logging for a full 24 hours,That is 86400 Secondsx (10 Bit ADC (Voltage)+10 Bit ADC (Current)) = 20 Bits+ (SSMMHH DDMMYY) = 96 BitsSo the total storage I will need is 86400x116=1.2MBYou don't have to store ssmmhh ddmmyy with every record. You could store it separately, every few records.So you could have a record of 24 bits... 3 bytes.First bit 0, means the record is adc voltage +adc current with about 2 bits unused.First bit is 1, means you have a header which may contain date or timeFirst bit 1, Second bit is 1 , means you have the date yymmdd (dd goes up to 31, so it fits in 5 bits, there are 12 months so you have store up to 15 in 4 bits, year can be stored as a 0-1023 value so it fits in 10 bits so in total you have 19 bits)First bit 1, Second bit is 0 , means you have the time .. hh is 0-23 (0-31 is 5 bits), mm is 0-59 , ss is 0-59 0-63 is 6 bits so you have 6+6+5 = 17 bitsUse the rest of the bits (about 5 i think) to store the interval between measurements... 1s, 5s, 10s, 30s, 60s, etc or whatever you want.Next, if you log every second, you may have values that are pretty close together, just a few bits different between them.With a PIC32 you could also look into compression, it should be quite possible to write to eeprom data as soon as its measured AND keep the last 64-128 sets of measurements into memory,m then compress those into less bytes. There are some pic32 compatible libraries but I don't have links now.Something like LZO (but it requires 64KB of ram) or using something like LZJB (perhaps modifying it to use even less memory) which should use less ram if you modify it or place some restrictions on the maximum memory allocation. Considering you'd work with 768 bytes-1k (128 sets of 6 bytes) it shouldn't use a lot of memory.Seehttps://hg.java.net/hg/solaris~on-src/file/tip/usr/src/uts/common/os/compress.cThere's also the patented LZS compression mechanism, which works within 2 KB of ram to compress or decompress stuff.http://en.wikipedia.org/wiki/Lempel-Ziv-Storer-Szymanskihttp://tools.ietf.org/html/rfc3943Both of these should get you within 2 megabits of eeprom , actually even without compression you should probably fit a day's worth of logging in 2M.. 2 megabit is basically 256k x 8 bits 3 bytes per measurement gives you 85333 records.

author: kolbep
author_rank: Frequent Contributor
timestamp: November 04, 2013, 06:56:01 pm
content: The header rows every couple of samples is actually a good idea and a neat space saver. ThanksFor now I am going to give the Pic32 a miss. I am just getting the hang of c18 and the pic18sDont think I need the power of a pic32 yet..P

author: senso
author_rank: Frequent Contributor
timestamp: November 04, 2013, 09:46:17 pm
content: Get a SPI flash, 64Mbits for 70 to 80 cents, 8 pins SOIC8 package.

author: dexters_lab
author_rank: Unknown
timestamp: November 04, 2013, 10:35:39 pm
content: when i built my three channel k-type thermocouple datalogger (quite a few years ago now) i went through similar thoughts though i wasn't dealing with as much total data, i wanted around 30 mins of non-stop recording at the highest rate rather than hours, anything beyond that was a bonus. There was three 10 bit channels sampling at 5hz. uC was similar to yours, a pic18f4620.initially looked at writing direct in realtime to an sd card but was worried about long term endurance writing/updating data/fats that often, sd cards are not renowned for their longevity. There was also issues of write speed but that was really down to the compiler and libraries i was using.Opted in the end (i made 5 units i sold to friends) to write internally to a 512kbit i2c eeprom and then added in the user menus to dump the contents in one go to the sd card either as a binary dump or a fully formatted CSV file.The eeprom did not store timestamps etc, so you could only record -> save -> record. Starting recording again overwrote the old data from the beginning. But meant i only needed to store the logging rate (in my case one of 5 options) and the length of the recording (number of samples) so only needed a few bytes of the internal pic eeprom space for that leaving all the external eeprom for storage. My eeprom -> sdcard writing routines worked it all back to write out the correct HH:MM:SS.00 text into the CSV. There is no point storing data that you might be able to work back out in software.i'd echo what senso says, spi flash is probably a better bet now, bigger capacity & cheaper but i did find the eeprom was easy to work with. Buy a selection of different types and try them, though you need to read carefully the datasheets you can often spot little gotchas in there just to make your code more complicated!!