Serial port monitor with CSV output and timestamp. - Page 1
Forum section: electronics / beginners

author: geggi1
author_rank: Frequent Contributor
timestamp: August 03, 2022, 06:56:14 pm
content: I'm looking for a serial port monitor / datalogger with timestamp that can store the streamed data to a CSV file.What can be reccomended?

author: wizard69
author_rank: Super Contributor
timestamp: August 03, 2022, 10:15:30 pm
content: Python.Of course that is write your own. If you had specifics a more in depth, targeted answer might be available. Even if that answer comes it si well worth the effort to learn Python for things like this.

author: geggi1
author_rank: Frequent Contributor
timestamp: August 03, 2022, 10:31:51 pm
content: Writing any software is way beyond my capabilities.I'm pretty sure that there are some software already made I just have to find it.I have played with a few serial consoles like Putty, Arduino serial monitor and Excel data streamer. But all of them are missing the timestamp function.

author: SmallCog
author_rank: Frequent Contributor
timestamp: August 03, 2022, 10:35:23 pm
content: Is this for your solar project?https://www.eevblog.com/forum/beginners/help-with-arduino-serial-print/Use any old terminal program that can save to a text fileWhen you're formatting your serial output on the arduino put the comma after the values

author: geggi1
author_rank: Frequent Contributor
timestamp: August 03, 2022, 11:07:27 pm
content: Yes its for my solar monitoring project.Its up end running now.But because I want to log the data for 24 hours or more there must be a timestamp.I just found the Realterm software that is basically working the way I want, but there is a limit for how many samples it will write to the file (1000).There is a walk-around for this by setting a longer pause after each time the Arduino sends data.I'm planing to have a sample each minute but this 1440 samples. Otherwise i need to sample every second minute that would be a possibility but I would like to get a better resolution on the measurements.There will be a schematics and code in the other post when everything is working.

author: SmallCog
author_rank: Frequent Contributor
timestamp: August 04, 2022, 03:15:21 am
content: Simplest solution would be to have the arduino write the time as well as the measurementsShould you ever decide to make it portable (and to my mind a device for measuring solar conditions would benefit from being portable) then you're only one step away from writing the same data to an SD card with its time stamp

author: kjr18
author_rank: Regular Contributor
timestamp: August 04, 2022, 06:29:12 am
content: How aboutTermite?It's small, supports plugins for timestamp and for data logging, should be able to work for you just right.

author: AndyC_772
author_rank: Super Contributor
timestamp: August 04, 2022, 06:40:39 am
content: Tera Term - latest version is 4.106 as of today.It won't add its own time stamps, and I'd be surprised if anything off-the-shelf did, but it will log to a file indefinitely.

author: ledtester
author_rank: Super Contributor
timestamp: August 04, 2022, 07:47:35 am
content: Quote from: wizard69 on August 03, 2022, 10:15:30 pmHave a look at this discussion, esepecially the response by Martin Evans:https://stackoverflow.com/questions/49080298/how-to-timestamp-data-collected-using-pyserial-and-export-to-csv

author: voltsandjolts
author_rank: Unknown
timestamp: August 04, 2022, 09:07:21 am
content: Quote from: geggi1 on August 03, 2022, 06:56:14 pmCan you give an example of the data coming in the serial port?Is there a newline or some other character to separate datasets?Also, an example of how you want it to appear in the csv file with the timestamp.

author: ledtester
author_rank: Super Contributor
timestamp: August 04, 2022, 01:46:19 pm
content: Quote from: geggi1 on August 03, 2022, 11:07:27 pmSo it appears it is for the project discussed here:https://www.eevblog.com/forum/beginners/help-with-arduino-serial-print/msg4337503/#msg4337503Here is a simple Python3 program to add timestamps to the lines coming into a serial port:Code:import serialimport timedef timestamp():date_now = time.strftime('%d/%m/%y')time_now = time.strftime('%H:%M:%S')return date_now+" "+time_nowOUTPUT_FILE = "output.csv"def main():ser = serial.Serial('COM3', 9600)with open(OUTPUT_FILE, "a") as f:while True:line = ser.readline().strip().decode("utf-8")now = timestamp()f.write(now+","+line+"\n")f.flush()print(now+","+line) # echo output to the terminalmain()Lines from the serial port are appended to the output file and also echoed to the terminal (via the last print statement).Note that this program simply prepends a timestamp plus a comma to the front of the line, so if the line from the serial port is CSV formatted the line placed in the output file will also be CSV formatted.

author: geggi1
author_rank: Frequent Contributor
timestamp: August 04, 2022, 08:09:20 pm
content: This is how i want the data.Timestamp 1, CH1, CH2, CH3, CH4,Timestamp 2, CH1, CH2, CH3, CH4,Timestamp 3, CH1, CH2, CH3, CH4,The timestamp should be with mili-second resolution.

author: iMo
author_rank: Super Contributor
timestamp: August 04, 2022, 08:20:19 pm
content: Quote from: AndyC_772 on August 04, 2022, 06:40:39 amTT does log the timestamp, at the beginning of each line received..Setup->Additional Settings->Log->Timestmap (select the type of the timestamp)

author: ledtester
author_rank: Super Contributor
timestamp: August 05, 2022, 02:07:36 am
content: Quote from: geggi1 on August 04, 2022, 08:09:20 pmChange / add these lines:Code:from datetime import datetimedef timestamp():return datetime.now().strftime('%Y/%m/%d %H:%M:%S.%f')[:-3]For more details, see:https://stackoverflow.com/questions/7588511/format-a-datetime-into-a-string-with-millisecondsand the python strftime documentation:https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior

author: wizard69
author_rank: Super Contributor
timestamp: August 06, 2022, 04:53:06 pm
content: Quote from: geggi1 on August 03, 2022, 10:31:51 pmThen how do you accomplish the rest of the project?QuoteYou might be able to do with with a shell script on Linux but I'd never go that route. I still think of Python first here. Even so some ideas with respect to timestamps and BASH:https://stackoverflow.com/questions/17066250/create-timestamp-variable-in-bash-scriptQuoteYou would be better off learning to program in Python. That might seem extreme but almost any solution I can think of would require a bit of scripting. You could look at Sigrok:https://sigrok.org/wiki/Main_Pagebut again you will not get anywhere without an uphill climb to learn that environment.As for serial port monitoring, there are so many such programs that if you look long enough you will find one that does what you want. Try these guys:https://www.electronic.us/or possibly:https://www.hhdsoftware.com/serial-port-monitor,if you want to pay for something. Otherwise just start going through the hundreds of apps that could potentially solve the problem.