what is inside a data logger ? - Page 1
Forum section: electronics / beginners

author: alireza7
author_rank: Regular Contributor
timestamp: March 08, 2018, 10:40:01 pm
content: hiwhat is the physical layer of a data logger exactly ?my product should have areliable an robustdata logger . how should i Implement it?i it a good idea to connect SD card directly and simply to MCU ? or it should be a more complicated circuit?

author: Rick Law
author_rank: Super Contributor
timestamp: March 09, 2018, 01:52:45 am
content: Quote from: alireza7 on March 08, 2018, 10:40:01 pmI think the lack of reply here is because no one knows what you are trying to log, how is the log data given, how much you want to log, and how frequent you want to log... So on, so on, so on.The quality of answer depends on the quality of question. A general question like "should I buy a bike, an electric car, or a motorcycle" without any further info will result in a rather useless answer.So, if I may suggest, please try to give us more information here. Most likely, someone here will forgive you a good answer.

author: Brumby
author_rank: Unknown
timestamp: March 09, 2018, 02:47:50 am
content: Quote from: Rick Law on March 09, 2018, 01:52:45 am.... if any answer is offered.Yes, more information is needed.Things we need to know include:The type of data being logged- Temperature? Air Pressure? Hydraulic Pressure? Voltage - mV or kV? Fluid level? GPS location? Altitude? Vibration? .... Anything you can measure can be logged.The environment:Indoors/outdoors? Office? Aircraft? Weather balloon? Whale tracker? Jungle? Desert? Army Tank?All these are some basic examples of the sort of things that need to be stated ... and there will be more questions once these have been established.

author: Chris-IP5
author_rank: Contributor
timestamp: March 09, 2018, 04:27:45 am
content: I thought about how these devices remember what they logged. There's a Wiki page with lots of snippets of information. It points out the storage could be by means of: backed static random access memory, flash memory or EEPROM..."And of course remember those strip chart recorders? Many of those must be antiques now.The Wiki page ishttps://en.m.wikipedia.org/wiki/Data_loggerOP are you looking for IC part numbers and so on?

author: alireza7
author_rank: Regular Contributor
timestamp: March 09, 2018, 08:19:10 am
content: yes you are right i should give more informationactually my system is a tariff device. it measures water fluid and log it during day and night and it should work for a long time so it should be durable and also pass some environmental testsoperation temperature : -40 to 85 Celsius degreeand pass these tests:Radiated electromagnetic fieldConducted electromagnetic fieldDry Heat test @70ºCCold test @ -40ºCVibration testMecanical shockThe operation environment: outdoors (Jungle ? Desert , farm , ...)also i should mention that i want a chip or IC and implement my own data logger on the main pcb of me systemso what is the best storage element for my system?please inform me if there is other specifications that i show determine to make you able to help me

author: Rerouter
author_rank: Super Contributor
timestamp: March 09, 2018, 08:38:42 am
content: You failed giving any kind of indication of how much data are you logging, and how frequently,If its a tiny amount of data, e.g. 8 bytes every minute, you could go with a NOR flash memory chip, low speed but guarantee no dead cells. that memory would likely outlast the device.Equally to add to the math is how often you expect to download the devices, and how long you want them to hold the recordings for.You will also need to plan space for data verification, e.g. checksum, inverted copy of each record, or a combination.

author: capt bullshot
author_rank: Super Contributor
timestamp: March 09, 2018, 08:40:31 am
content: So in this case, my first question is:What kind of regulation / applicable standards are involved with this project? Metering in general is heavily regulated and depending on the target location (country specific), so at first you'd have to check which standards apply.Is the data logger part of a metering device, and may the logged data treated differently from the meter data?Will the logged data be used to turn mass flow into cash flow at some point?What amount of data is to be logged and kept for what time interval?What is the interface to read out the logged data?I guess, power consumption will be of great matter, such stuff usually is operated from a non-replaceable battery for it's lifetime.In general, you'll have to apply and verify some techniques to ensure data aren't lost and have some level of confidence. That'll work on any storage device of your choice, so usually the choice of the data storage is driven by economical reasons (lowest cost) and data integrity is shifted into development (usually SW and paperwork).1

author: alireza7
author_rank: Regular Contributor
timestamp: March 09, 2018, 09:33:31 am
content: Quote from: Rerouter on March 09, 2018, 08:38:42 amhow frequently : i don't need high data bandwidth, it is sufficient to be capable of read or write data every 30 second in the rate of 0.5kB/show much data are you logging : it is sufficient to be capable of storage 256-Kbitcan you have a discussion on flash memory vs EEPROM ? which one is more appropriate for my system?

author: alireza7
author_rank: Regular Contributor
timestamp: March 09, 2018, 09:42:25 am
content: Quote from: capt bullshot on March 09, 2018, 08:40:31 ami should pass standards like : ISO 4064-2 , ISO 16399 , ...the data logger is part of metering device.data should be kept at least for 20 yearsinterface between MCU and storage element is not my concern i with implement it.and yes power consumption is important

author: capt bullshot
author_rank: Super Contributor
timestamp: March 09, 2018, 10:40:24 am
content: Quote from: alireza7 on March 09, 2018, 09:33:31 amYou can get your work done with either one. My personal decision would be the EEPROM.EEPROMs are available in this size, and they have the great advantage of easiness to use (true random writing possible). They are readily available with way much better data retention specifications than Flash memories.Flash memory can be written once (some serial SPI ones more than once) per block, then a whole block must be erased at once to overwrite. You'll have to put more effort into your SW to organize the data, and you'll get less data retention and write cycles guaranteed than an EEPROM.Both should be available in ultra low power variants (ultra low power is very common with EEPROMs, not so with Flash) to fulfil your power consumption requirements.

author: Rerouter
author_rank: Super Contributor
timestamp: March 09, 2018, 10:45:49 am
content: Ok so 15 bytes per second,And the requirement to store for 20 years, Does this imply there are times where it is not logging?Just to give clarification, 15 bytes per second, or 0.5KB/30 seconds, is 10 megabit every day, In other words your 256Kb memory would last 25 days, or roughly 1 month,If you went with the maximum normal non nand flash size of 2Gb then it would last about 210 days, or just shy of a 2/3rd of a year.I chose NOR flash because of the size you will likely require, most data logger projects end up being limited only be memory technologies, EEPROM and even SRAM the other 2 fairly low power options generally cap out at 64Mb, while NAND's low reliability doesn't meet your specs,Can you store the data in a more efficient form? or sample less often, 20 years is difficult for your current requirements. which is closer to 10GB, now while an SD card may be able to manage it, I would expect you to struggle to make it both low power, and have a warrented reliability for that long.

author: alireza7
author_rank: Regular Contributor
timestamp: March 09, 2018, 11:04:29 am
content: Quote from: Rerouter on March 09, 2018, 10:45:49 amnot exactly! most of write operations just update previous data. in other word i need store accumulative water flow . and the other thing is that there is a hand held reader and we can extract data from tariff device periodically .20 years is just a confidence margin for data don't lose.

author: mikerj
author_rank: Super Contributor
timestamp: March 09, 2018, 11:31:45 am
content: Quote from: alireza7 on March 09, 2018, 11:04:29 amIf you are simply updating the integrated water flow rather than recording a history, then you could use a small EEPROM device. These typically have a write endurance of up to 1 million writes (though check the datasheet carefully, there are often some gotchas) and this could be extended greatly by using a wear leveling method e.g. having a circular buffer of structures in the EEPROM holding the accumulated flow, a timestamp and a checksum. Not only does this provide a large write endurance, it also gives you a little bit of near term historical data and provides fault resilience; if the last value gets corrupted the logger can start again from the last good value so you don't lose everything.

author: Rerouter
author_rank: Super Contributor
timestamp: March 09, 2018, 11:47:26 am
content: Ok, as an integrator, that greatly reduces the complexity, this could be accomplished in any old eeprom, I would still recommend you store a normal copy of your value, an inverted, and a checksum of some kind for each record, but from there, just attach it as a ring buffer,You would track in your micros RAM where the highest value is, and overwrite the next instance with your new maximum, should your device loose power, it runs through the eeprom finds the highest value, verifies its checksum and saves the location to RAM and continues on, If it encounters a bad checksum it checks the bit flipped copy, and either recovers the normal record, or finds them both faulty and reads back 1 location to verify that,

author: newbrain
author_rank: Super Contributor
timestamp: March 09, 2018, 01:29:20 pm
content: For greater write endurance and data retention, no need of wear levelling, and (possibly) low power consumption with respect to EEPROM, consider also ferroelectric RAM.http://www.ti.com/lit/ml/slat151/slat151.pdfhttps://www.fujitsu.com/us/Images/SPBG_FRAM_Overview_BR.pdfPower consumption (dis)advantage depends on the memory size and the amount of writing that is done:http://www.cypress.com/file/46746/download

author: paulca
author_rank: Super Contributor
timestamp: March 09, 2018, 01:40:57 pm
content: It's a bunch of little clerical daemons with notepads.(Terry Pratchett reference).

author: newbrain
author_rank: Super Contributor
timestamp: March 09, 2018, 01:54:58 pm
content: Quote from: paulca on March 09, 2018, 01:40:57 pmBut do they work inHex?Edit: reference added. Sorry for the OT.

author: paulca
author_rank: Super Contributor
timestamp: March 09, 2018, 01:58:20 pm
content: Quote from: newbrain on March 09, 2018, 01:54:58 pmGood question. If you ask them nicely they might give you the answer in hex.As long as it's safe hex, bata boom.

author: ebastler
author_rank: Super Contributor
timestamp: March 09, 2018, 03:09:47 pm
content: This whole discussion is a bit scary. Is this about a real product development? Do these really get started based on so little in terms of clues?!Please comfort me and say that this is a student assignment with a "mock" use case...

author: Kalvin
author_rank: Super Contributor
timestamp: March 09, 2018, 04:23:02 pm
content: Do not forget data integrity check and data recovery due to sudden power loss and intermittent resets etc. real-world events which may reset the device in the middle of data write/update.Edit: Plan for data loss ie. how much data is permitted be lost at maximum if the memory gets corrupted and how it will be detected and possible recovered so that the user will be able to detect that some data has been lost or corrupted.

author: alireza7
author_rank: Regular Contributor
timestamp: March 12, 2018, 01:41:01 pm
content: the endurance of a EEPROM is related to each byte or it's about whole device?

author: alireza7
author_rank: Regular Contributor
timestamp: March 12, 2018, 03:18:14 pm
content: Quote from: ebastler on March 09, 2018, 03:09:47 pmit's just an start up. we should start at a point any way , even a zero point . you can consider it as a student assignment but finally we want to make it a real product.i appreciate your concern anddemandwish you give me your advice to help me develop a real product.

author: paulca
author_rank: Super Contributor
timestamp: March 12, 2018, 03:23:54 pm
content: Quote from: alireza7 on March 12, 2018, 03:18:14 pmTypically "demanding" time or knowledge on a forum will only get your silence at best or a appropriately negative response at worst.

author: alireza7
author_rank: Regular Contributor
timestamp: March 12, 2018, 03:44:22 pm
content: Quote from: paulca on March 12, 2018, 03:23:54 pmoh sorry English is not my mother tongue and i just picked the wrong word "demanding".i think it is better to say i wish you help me

author: paulca
author_rank: Super Contributor
timestamp: March 12, 2018, 04:03:57 pm
content: Quote from: alireza7 on March 12, 2018, 03:44:22 pmNo problem. One way to say what you mean would be:"I would be grateful for your help and advice."

author: ebastler
author_rank: Super Contributor
timestamp: March 12, 2018, 07:23:31 pm
content: Quote from: alireza7 on March 12, 2018, 03:18:14 pmHmm, at least that is an interesting choice for a startup's first product!Capt. Bullshot has already provided a great list of questions to guide your design, in reply #10 above. Please start with these to make sure you are addressing the key requirements (or are even aware of them!). A couple of specific comments from my side:I would stress the power supply aspect in particular. The required operating lifetime of the meter, battery capacity (which may be lower than expected at cold temperatures), active/suspend duty cycle, and power consumption of your processor and sensor will determin much of your design.One aspect I would add to Capt. Bullshot's list is how to make your meter tamper-proof. Not all future users will be perfectly honest; some may have an interest in reducing the metered water consumption. And I would expect them to get quite creative about that!So you need to try and figure out all the ways how somebody could manipulate the meter, the sensor, or the connection between them, and develop ways to prevent or at least detect these manipulations.You had asked about SD card interfacing in your initial post. With a view to reliability and also tamper-proof implementation, I would recommend against a (removable) SD card for storing the data. I think you want an enclosure that remains permanently sealed, with non-removable storage inside, and an interface (RF or optical?) to download the data.Finally, a non-technical point: Have you figured out who you want to sell this to? Your customer would not be the end users (water consumer), but the water suppliers, right? Maybe smaller companies or not-for-profit initiatives in less developed areas? Have you actually started to reach out to such water suppliers? I strongly recommend that you do that as early as possible, certainly well before you hve what you think is your final product; and reach out to more than one of them. They may have needs and expectations which neither of us on the forum are aware of, and meeting their needs will be critical to your startup!Best of luck with your project -Juergen