Python - Reading and writing lines from file - Page 1
Forum section: electronics / general

author: mmoreau
author_rank: Contributor
timestamp: May 22, 2020, 10:40:23 am
content: I am learning python and am currently creating a project with a Raspberry Pi 0 W that will keep track of how many times me or my wife has fed and given my dog his medication per day(poor guy needs a lot of care). I am using a Waveshare 7x5 E Ink display and I have gotten the display to work and have two buttons working to record the data points.Currently, the writing to screen and the buttons (using callback) are in separate python files. I would like to store the data in a separate file(maybe txt but I'm open to suggestions). The data will be in the format of (date, #, #, #, #). I will parse it to and from a list. I would like the data in it's own file in case of power loss or error in the script. I plan on clearing the file about twice a year but that can still get kinda long and my job can keep me away from home for months on end.My idea is to first pull the last line to see if it is the same day then either overwrite or append a new line. Then, I would need to pull the last 5 lines to write to the screen.The methods I have seen so far either don't seem to be good for this or I might not have a full understanding of the function. I am open to any suggestions and alternate methods. Please point me in the right direction.

author: greenpossum
author_rank: Frequent Contributor
timestamp: May 22, 2020, 11:19:50 am
content: Read the input file line by line copying to an output file*. At the same time keep track of the last up to 5 lines read. You can do this with a list of strings that you append to the front and delete from the back when it exceeds 5.When you get to the end, you'll know whether to *copy the last line and append a newer line, or to replace the last line with a newer line.Rename the new file to the old name.

author: greenpossum
author_rank: Frequent Contributor
timestamp: May 22, 2020, 11:32:08 am
content: If there is no requirement to keep a complete history, only a partial history, and you don't mind the file being in reverse order, then a simpler way is:Read the first line and decide whether it is to be replaced or have a newer entry prepended to the front of the file. Write it or them to output file.Read enough lines to get 5 to display. Also copy to output file.Read as many lines to get to the total you want to keep, copying to the output file.Ignore the rest of the input. Close the files and rename the output file to be the current file.

author: mmoreau
author_rank: Contributor
timestamp: May 22, 2020, 12:11:52 pm
content: I am a little concerned that this might possibly lead to a loss of data if something went wrong. If possible, I would like a solution that makes recovery from a mishap a little more recoverable. I will give this a try and see how comfortable I am with it.

author: greenpossum
author_rank: Frequent Contributor
timestamp: May 22, 2020, 01:45:56 pm
content: You could flush and close the output file before reopening to add the new line to the remembered position which shrinks the window. If that goes well, then you can rename the original to a backup version, and the output file to the original name. You're always at the mercy of power out unless you devise an atomic update scheme using database techniques. As for software bugs created by the programmer, well the tools won't protect you from that.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: May 22, 2020, 02:11:24 pm
content: Append the last medication event to the end of a text file (with "\n") The file should stay permanent, even if the dog eats the PiRead the last 5 lines of the file into a sorted array, deleting the sixth entry , something like this:Code:list = []file = open("medication.dog", "r")for line in file:list.insert( 0 , line.rstrip() )if len(list) == 6:list.pop(5)file.close()for j in list:print(j)And then do your date compare on the first array element listYou can speed up file access using in memory files, but that's another tutorial.Hope your dog is okay.+++edit+++ a subsiduary thought. You can avoid pruning the file every six months by using a filename that contains the month. For example "medication_0920.dog".On the first of every month, have your script create a new file with the current MMYY. This keeps the monthly working file lean. Then just keep or delete the obsolete files.Use something like this:Code:filename = "medication_{0}.dog".format((date.today()).strftime("%m%y"))file = file.open(filename , "a")medication_0520.dogOpening the file in append mode will force the creation of the file if it does not already exist.

author: mmoreau
author_rank: Contributor
timestamp: May 23, 2020, 03:31:00 am
content: Thank you Syntax Error. This does sound like a very good solution. My original plan was to have txt files for each month but decided against it for simplicity of only having to pull from one file in case of the days showing being between two months. However, if the callback occurs on the first of the month then I can do a query of the file (and now it sounds like it would create the file if it doesn't exist, if I have that correct) then if it is null inside, I can just append the last four entries of the previous month and continue on. I would prefer to be able to look back over a month and see his trend in eating in case his vet asks.I have since found this method:Code:import oswith open('filename.txt', 'rb') as f:f.seek(-2, os.SEEK_END)while f.read(1) != b'\n':f.seek(-2, os.SEEK_CUR)last_line = f.readline().decode()From comments, I gather that it might be hard to understand what is actually happening but what's the worst that could happen if I give it a try? This also kinda makes sense to me for looking back and populating lists for those days line by line.Also, it is supposed to be quick but that might not be good for my case. The callback is triggered and I could not add the debounce argument to it. It said it does not allow a third argument even though the documentation I found said it should.Code:GPIO.add_event_callback(mb, mb_callback, bouncetime=200)I did also try the debounce function that VC Code tried to auto fill with no luck.The other option was to add it to event detect but it sounded like I can only do that if it was set to pull down. I could always change it to pull down and add a cap but I still don't know much so I don't want to change whats working. Adding sleep to the callback worked so if the function that I need to run is long enough in the callback, that might be all I need.Also, my dog Tyson is fine. We just have to feed him a special diet spread out through the day and meds twice a day. Poor fella is mad he can't get table scraps anymore.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: May 23, 2020, 09:29:32 pm
content: Your file read example is for a binary file. So are you using binary rather than char/text file? For robustness, you should perform a check to see if the file is seekable() first. The bounce period, should that just be the number 200?The correct way to handle file IO, is to have the read and write handled by an asynchronous coroutine structure. This is because file IO runs at the speed of the storage device, not the DMA bus. It's many times slower! If you want to learn about asyncio, check outhttps://docs.python.org/3/library/asyncio-task.htmlGo Tyson!

author: mmoreau
author_rank: Contributor
timestamp: May 24, 2020, 04:08:23 pm
content: I have been able to read and write the last line using this method. Now I am working on the reading only portion using this method. It is harder navigating the cursor in binary mode. So far I have been able to scroll to the beginning of the last line but I am having trouble finding a way to get the cursor to the line before that. I have tried scrolling back by using a set amount and have been able to get to the beginning of the line before the last but when I do it again it ends up being chopped up. I am now looking for a way to get the byte size of the current line (using len) but it just isn't working right now.Code:with open('list.txt', 'rb') as f:f.seek(-2, os.SEEK_END)while f.read(1) != b'\n':f.seek(-2, os.SEEK_CUR)last_line = f.readline().decode()ct = len(f.readline())ps = last_line.split('*')ds = psdsms = dnumb[int(ds)]eval(dsms).extend(ps)print(f.tell())f.seek(-ct, os.SEEK_CUR)print(f.tell())Lline = f.readline().decode()pss = Lline.split('*')ds = pssctt = len(f.readline())The line reads as 05*24*6*5*1*0* in the file. I'm not sure why each line would be different byte size. I started by assuming that each character would be the same size but it doesn't appear to be that way. Also, when I use the len method, tell reads as on the same line(117).

author: PlainName
author_rank: Super Contributor
timestamp: May 24, 2020, 07:11:17 pm
content: Doesn't f.readline affect the seek position? After that, you're at the end of the line you just read, so seeking to the previous EOL just gets you back to where you were before the f.readline. I think you need to store the seek position before doing the readline, then seek back to that before trying to find the start of the previous line.Note: I don't do python so this might be cobblers

author: mmoreau
author_rank: Contributor
timestamp: May 24, 2020, 10:01:43 pm
content: I figured it out. It took a few tries. Once I figured out how to go back byte by byte, I did the math and saw where everything was located. The end of the document has a disproportionate number of bytes than the lines. And apparently line break is one byte (and not physically written as \n?).I still have to clean up the code and once the screen is how I want it, I will implement the file naming system.Code:import loggingfrom waveshare_epd import epd7in5bcimport timefrom PIL import Image,ImageDraw,ImageFontimport tracebackfrom datetime import datefrom datetime import datetime#Date and Timetoday = date.today()makeMonth = today.strftime("%b")makeDay = today.strftime("%d")now = datetime.now()currentTime = now.strftime("%H")day = today.weekday()#Current Weekmm = [] #Month, day, daycode, food, med 1, med 2, dumped datatum = []wm = []thm = []fm = []sam = []sunm = []dnumb = ['mm','tum', 'wm', 'thm', 'fm', 'sam', 'sunm']# Pull data from file and write to listswith open('list.txt', 'rb') as f:f.seek(-2, os.SEEK_END)while f.read(1) != b'\n':f.seek(-2, os.SEEK_CUR)last_line = f.readline().decode()ps = last_line.split('*')ds = psdsms = dnumb[int(ds)]eval(dsms).extend(ps)loc = (f.tell() -29)# Grab 4 more daysCounttm = 0while Counttm < 4:f.seek(loc, os.SEEK_SET)Lline = f.readline().decode()pss = Lline.split('*')ds = pssdsmss = dnumb[int(ds)]eval(dsmss).extend(pss)loc = loc -15Counttm = Counttm +1f.close()# debug importingprint(mm)print(tum)print(wm)print(thm)print(fm)print(sam)print(sunm)Thanks for all the help guys!

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 26, 2020, 03:29:04 am
content: I would use a binary file for this. Thestructmodule contains the needed functions:pack(),unpack(), andcalcsize().For example, if you have just four small counter fields, each field between 0 and 255 inclusive, then format<H6Bwould work, and each day would use juststruct.calcsize("<H6B")bytes of data;8on all Linux systems – that means just 2920 bytes per year. To pack into abytesobject, usedata = struct.pack("<H6B", year, month, day, n1, n2, n3, n4); to unpack, useyear, month, day, n1, n2, n3, n4 = struct.unpack("<H6B", data). That way you can seek directly to the last record, read it, and either seek back by one record and overwrite the old record with updated data, or add a new record.(I always keep a conversion utility at hand also, for converting the binary data file to a text file.)A better option would be to use that same format,<H6B, but asdata = struct.pack("<H6B", year, month, day, hour, minute, second, event)so that you could track each event separately, for up to 256 events. Instead ofNdaily counts, you would have the time each event occurred. You could then display the number of occurrences of each event today, since midnight, or in the lastMhours/days. Each event would take that 8 bytes of storage space, so even if you had 30 events per day, it'd still be just 87,600 bytes per year. For statistics, you could draw a histogram of the intervals of each specific event.(That is, event 1 to 3 could be meds of different types and dosages, event 4 wet food, event 5 dry food, and so on.)If you'd like, I can show some example Python3 code.

author: mmoreau
author_rank: Contributor
timestamp: May 29, 2020, 08:27:00 pm
content: For the project as of now, The system I set up works fine and I think I will stick with it. The file will only have one month stored in it so 31 lines max.For future projects, I would like to use a binary file for it. I only used a text file since it's my first attempt and I wanted to keep it as simple as possible until I learn a little more. It looks like your method acts more like a database. I am a huge fan of databases. If I had thought of it before, I would have looked into SQL and see if I could use it in python. When I make version 2, I might need to because I want to write an app so we can check when we are away or just too lazy to walk over.I would appreciate some example code to get me started. I will make a copy of the project to try it out and who knows, I might keep it as the final version.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: May 29, 2020, 08:50:00 pm
content: Simple is good.First rule of good data design is only ever store that which you indend to retrieve. The second rule is, store data in a way that reflects the number of transactions (read write update) required. Third, don't get seduced by the latest database 'thing'.From your system, you found a simple flat text file is all that you needed. If you were building something like this web page, SQL is the way to go. As you're using Python, check out NODE.js which combines MySQL queries in a lightweight structure.https://www.w3schools.com/nodejs/nodejs_mysql.asp

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 29, 2020, 09:05:23 pm
content: Python3 includesSQLite3module, which provides an SQL interface to a local database file. (The 'connection' to the database is then more like a file handle, as there is no database process per se at all.)I'll cobble together an example for the event-based approach with a binary file, because I think knowing e.g. the intervals between the lastNevents of typeX– say, specific meds – would be useful even in a small interface.

author: mmoreau
author_rank: Contributor
timestamp: May 29, 2020, 09:14:52 pm
content: Some of my reasoning for going to a database is the fact that it is live for everything and multiple things can read and write it. This would also let me enter the data in a way that would allow me to store time things happen if I do decide to review our patterns or if the Vet asks but that information would be a byproduct. The screen writing function wouldn't need to know that info.One of the things I discovered on this project is when there is an error in the code(or power outage to the Pi), the file is obliterated. I kept a copy of the basic file to replace it easily when I was debugging. I know there are ways to mitigate this in code for the case of script ending in error but this idea makes me uncomfortable. It would be nicer having a safer way to do it.

author: mmoreau
author_rank: Contributor
timestamp: May 29, 2020, 09:32:19 pm
content: Just to be clear, I'm not looking to make Python a database but to have the python code query and write to the database. That way it's completely on it's own. The few things I have seen from posts seem to be using Python as the database and not just interact with it. But I might not be looking in the right places. Or I just don't understand that level yet.edit:It looks like both links you sent can be used that way. I usually jump into the code but this puts me right to the top for once. Maybe I'll learn it the right way this time.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 30, 2020, 02:43:20 am
content: Here's apetevent.pydefining a class for storing, retrieving, examining, and comparing events:Code:from struct import pack, unpack, calcsizefrom datetime import datetime, timezone, date, time, timedeltaimport os # For os.fsync()# This file is in public domain, or licensed under CC0-1.0, whichever you prefer.class PetEvent(bytes):"""Class describing an identifiable event (0-255) at a specific time"""def __new__(cls, *args, tz=None):"""PetEvent(bytes)Creates an event from 8 bytes of binary dataPetEvent(kind)Creates an event of type 'kind' for this local date and time,'kind' being between 0 and 255PetEvent(kind, year, month, day, hour, minute, second [, tz])Creates an event of type 'kind' for the specified date and time.If tz is not specified, local timezone is used."""# PetEvent(bytes)?if len(args) == 1 and isinstance(args, (bytes, bytearray)):if len(args) < 8:raise ValueError("Events need exactly 8 bytes, received %d." % len(args))else:return bytes.__new__(cls, (args)[0:8])elif len(args) < 1:raise ValueError("Cannot create an event out of nothing.")# First parameter must be the event kind.if (not isinstance(args, int)) or (args < 0) or (args > 255):raise ValueError("Invalid kind (%s); must be an integer between 0 and 255, inclusive." % str(type(args)))# PetEvent(kind)?if len(args) == 1:now = datetime.utcnow()return bytes.__new__(cls, pack('>H6B', now.year, now.month, now.day, now.hour, now.minute, now.second, args))# PetEvent(kind, year, month, day, hour, minute, second)?if len(args) == 7:# We'll let datetime handle the date and time verification.if tz is None:then = datetime(*args[1:7])else:then = datetime(*args[1:7], 0, tz)return bytes.__new__(cls, pack('>H6B', then.year, then.month, then.day, then.hour, then.minute, then.second, args))raise ValueError("Cannot construct an event from %s." % str(args))def __sub__(self, value):"""Subtracting two events yields their date and time difference as a timedelta object."""return self.datetime(tz=timezone.utc) - value.datetime(tz=timezone.utc)def __str__(self):"""String description of the event is 'YYYY-MM-DD HH:MM:SS UTC K', where K is the kind"""return '%04d-%02d-%02d %02d:%02d:%02d UTC %d' % self.tuple# Properties: tuple, localtuple, year, month, day, hour, minute, second, kind.@propertydef tuple(self):"""Returns a tuple(year, month, day, hour, minute, second, kind), in UTC."""return unpack('>H6B', self[0:8])@propertydef localtuple(self):"""Returns a tuple(year, month, day, hour, minute, second, kind), in local time."""data = unpack('>H6B', self[0:8])then = datetime(*data[0:6], 0, timezone.utc)return (then.year, then.month, then.day, then.hour, then.minute, then.second, data)@propertydef year(self):return unpack('>H', self[0:2])@propertydef month(self):return unpack('>B', self[2:3])@propertydef day(self):return unpack('>B', self[3:4])@propertydef hour(self):return unpack('>B', self[4:5])@propertydef minute(self):return unpack('>B', self[5:6])@propertydef second(self):return unpack('>B', self[6:7])@propertydef kind(self):"""Event type (0 to 255, inclusive)"""return unpack('>B', self[7:8])# date(), time(), and datetime() methodsdef date(self, tz=None):"""Event date as a datetime.date object (no time)"""if tz is None:return date(*unpack('>H3B', self[0:4]))else:return date(*unpack('>H3B', self[0:4]), tz)def time(self, tz=None):"""Event time as a datetime.time object (no date)"""if tz is None:return time(*unpack('>3B', self[4:7]), 0)else:return time(*unpack('>3B', self[4:7]), 0, tz)def datetime(self, tz=None):"""Event date and time as a datetime.datetime object"""if tz is None:return datetime(*unpack('>H5B', self[0:7]))else:return datetime(*unpack('>H5B', self[0:7]), 0, tz)def save(self, filename, sync=True):"""Append this event to a binary file.By default, this will return only after thechanges have been written to storage. If youdon't want that, use sync=False."""target = open(filename, 'ab')target.write(self)if sync:target.flush()try:os.fsync(target.fileno())except:passtarget.close()@classmethoddef load(cls, filename, latest=None, blocksize=-1, kind=None, kinds=None, before=None, after=None,years=None, months=None, weeks=None, days=None, hours=None, minutes=None, seconds=None):"""Load a sequence of events from a file.If you are only interested in the latest N events,specify latest=N. Then, the events are produced in time order.You can specify a single kind of event to get (kind=),or a list or iterable of kinds to get (kinds=).If you are only interested in events up to a specific moment,you can specify that (before=).If you are only interested in events since a specific moment,you can specify that too (after=).Alternatively, you can specify the number of weeks (weeks=),days (days=), hours (hours=), minutes (minutes=), or seconds(seconds=) prior to this moment to return the events for.If you want the events as a list, wrap the call around list().For constrained memory situations, supply blocksize witha reasonable power of two, say 16384. For better performance,use say 2097152. Default (-1) is for Python to use a heuristic."""# Prepare to filter for specific kinds of events only.keep_kinds = b''if kind is not None:keep_kinds += pack('>B', kind)if kinds is not None:for k in kinds:keep_kinds += pack('>B', k)# If there are no kinds specified, return all.if len(keep_kinds) < 1:keep_kinds = None# Keep only events on and before a specific date and time?if before is not None:minwhen = pack('>H5B', before.year, after.month, before.day,before.hour, before.minute, before.second)elif (years, months, weeks, days, hours, minutes, seconds).count(None) != 7:when = datetime.utcnow()if seconds is not None:when -= timedelta(seconds=seconds)if minutes is not None:when -= timedelta(minutes=minutes)if hours is not None:when -= timedelta(hours=hours)if days is not None:when -= timedelta(days=days)if weeks is not None:when -= timedelta(weeks=weeks)# Years are modified on calendar basisif isinstance(years, int) and years >= 0:y = yearselif years is not None:raise ValueError("years must be an integer")else:y = 0# As are monthsif isinstance(months, int) and months >= 0:m = monthselif months is not None:raise ValueError("months must be an integer")else:m = 0# Combine all 12 month periods into yearsif m >= 12:y = y + (m // 12) # // is integer divisionm = m % 12# Modify months firstif m != 0:if m < when.month:when.replace(month=when.month-m)else:# It'll be the previous year.y = y + 1when.replace(month=when.month+12-m)# And finally years.if y != 0:when.replace(year=when.year-y)# Now pack it up.minwhen = pack('>H5B', when.year, when.month, when.day,when.hour, when.minute, when.second)# and remove the datetime object from the local namespace.del whenelse:minwhen = b'\x00\x00\x00\x00\x00\x00\x00'# Keep only events on and after a specific date and time?if after is not None:maxwhen = pack('>H5B', after.year, after.month, after.day,after.hour, after.minute, after.second)else:maxwhen = b'\xFF\xFF\xFF\xFF\xFF\xFF\xFF'# No events read yet.events = []# Ready to read the binary file in chunks.with open(filename, 'rb', buffering=blocksize) as source:while True:chunk = source.read(8)if len(chunk) == 0:breakelif len(chunk) != 8:# Make sure we read a full chunk! Could be a pipe.while len(chunk) < 8:more = source.read(8 - len(chunk))if len(more) == 0:raise IOError("Partial event in file!")chunk += more# We always filter by datetime.# This comparison only works because of the order,# and when using '>H5B' to encode it!if (chunk[0:7] < minwhen) or (chunk[0:7] > maxwhen):continue# Filter by kind, if desired.if (keep_kinds is not None) and (chunk[7:8] not in keep_kinds):continueif latest is None:# Yield (as part of the sequence) this event.yield cls(chunk)else:# Add to the list.events.append(cls(chunk))# Need to drop any?if len(events) > latest:# Drop the oldest event in the list.events.sort()events.pop(0)if len(events) > 0:# Make sure they are in ascending order.events.sort()for event in events:yield eventIf you save it aspetevent.py, you can runpydoc3 peteventin the same directory to read its documentation.To use it in your own scripts, just put it in the same directory as your script, and in the beginning of your own code, addfrom petevent import PetEventEach PetEvent has a date and time, and a kind between 0 and 255, inclusive. You can access these fields using.year,.month,.day,.hour,.minute,.second, and.kindproperties. You can get these as a tuple in this order via the.tupleproperty, or via the.localtupleproperty if you want the date and time in local timezone. Substracting two PetEvents yields a datetime.timedelta object describing their difference in time; the kind is ignored. If you want the difference in seconds, use(ev1 - ev2).total_seconds().In the binary file and in the binary event objects, the date and time are stored in UTC, to avoid issues with daylight savings.The PetEvent itself is a subclass of bytes, and consist of 8-byte bytes objects, and functions to manipulate them. To ensure that they sort in time order (and in ascending kind for events at the same second), I used the>H6bpacking. That is, the year part is more significant byte first.To create an event, say of kind 1, and save it safely to a binary file nameddata.bin, you only need to doPetEvent(1).save("data.bin")Thesave()method opens and closes the file, and uses OS-specific fsync() if possible, to ensure the data hits the storage before the method returns. If you don't want it to do that, usePetEvent(1).save("data.bin", sync=False)instead.In particular, you shouldnotlose any data in the file if the Python process crashes. You shouldn't lose any data if the machine crashes either, but to be sure, I might add an fsync to the directory the data file is in. (The only issue with the machine crashing is if it crashes before the file metadata, including the size, is updated. The data will be on disk, but the metadata might not be, you see.)You can also create an event for a specific date and time, by default using the local timezone, usingevent = PetEvent(kind, year, month, day, hour, minute, second)or similarly for this moment,event = PetEvent(kind)and compare them or subtract them to get their time difference in datetime.timedelta units. (Use(A - B).total_seconds()to get the number of seconds between PetEventsAandB.)Let's say you save some test events into a file nameddata.bin.To print all events in it in UTC, (YYYY-MM-DD HH:MM:SS UTC kind), you just doCode:for event in PetEvent.load("data.bin"):print(event)For formatted output in local timezone, use e.g.Code:for event in PetEvent.load("data.bin"):fields = event.localtupleprint("%04d-%02d-%02d %02d:%02d:%02d kind %d" % fields)The loader has pretty advanced filtering built in. You can load only events of a specific type by supplyingkind=kind, or of some specific types usingkinds=(one,another). You can use datetime.datetime objects to specify what events you are interested in, by supplying them asafter=datetimeand/orbefore=datetime. If you are interested in events in the lastNyears, months, weeks, days, hours, minutes, or seconds, just supplyyears=N,months=N, and so on. (You can use more than one, saymonths=2, days=4for events in the last two months and four days, but not withbeforeorafter). If you only want the latestN, supplylatest=Ntoo.The loader produces a sequence, so if you want the events as a list, wrap it insidelist():events = list(PetEvent.load("data.bin", latest=15, kind=3))If you are only interested in how many events of say kind 5 and 3 have occurred in the last 24 hours, uselen(list(PetEvent.load("data.bin", kinds=(3,5)))).While the class isn't optimal, it does NOT read the entire file in memory: it is designed to work without using up too much memory. You can even passblocksize=16384to the loader functions, to ensure they don't use too much memory for file buffering, at the cost of more syscalls for very large files.Python I/O isn't that fast, and on my Core i5-4200U laptop, reading 365000 events (a 2.8 MiB file) takes about half a second. Appending a new event takes very little time, and does not depend on the file size. (This is true for both reading them all into a list, or just the lastNevents. The only difference there is the amount of RAM used by the Python process.)If you use e.g. Qt5 or any GUI toolkit, you can put the loader in a separate thread or process, and provide the data as a Queue to the main process. That way the main process stays responsive, and can show e.g. a "loading" icon, and display the latest events when they've been read from the file. Similarly, a thread/subprocess could be used to save the data, and provide a success message via a Queue to the QUI thread, while keeping the UI responsive. (That way, when the user presses a button, they'd see e.g. on a status line a "Saving.." icon, that turns to "Ok" whenever the data on disk and the metadata have been updated and sync'ed.)Finally, I just wrote this code, and tested it only lightly; it needs a check-over by another pair of eyes, before I'd really trust it. I did test most of the features, and it does seem to work, I'm just a bit paranoid and refuse to trust even myself without double-checking the code. I do make a lot of errors...

author: mrflibble
author_rank: Super Contributor
timestamp: May 30, 2020, 01:16:02 pm
content: Not sure if the log is supposed to have some limits on it in terms of history length, size, etc. If you want to do queries on it, sqlite will do the trick, as already mentioned. If you want logging with automatic log rotation and compression, you might want to use Ye Olde Syslog Facilities. And of course there's a python interface for it.https://docs.python.org/3/library/syslog.htmlJust use the "user-level messages" (Facility=1) and you're all set. <-- That's the default setting anyways, as per this bit from the python docs:"If the facility is not encoded in priority using logical-or (LOG_INFO | LOG_USER), the value given in the openlog() call is used."I notice theloggingmodule has also been mentioned. If you want to use that one, but still like the idea of automatic log rotation by using your systems syslog, you can use thelogging.handlers.SysLogHandler:https://docs.python.org/3/library/logging.handlers.html#logging.handlers.SysLogHandlerCouple of related links:https://devconnected.com/syslog-the-complete-system-administrator-guide/#a_What_are_Syslog_facility_levelshttps://tools.ietf.org/html/rfc5424Quick code snippet stolen from SO, because I am waaaay lazier than Nominal Animal:Code:# https://stackoverflow.com/questions/3968669/how-to-configure-logging-to-syslog-in-python/3969772#3969772import loggingimport logging.handlersmy_logger = logging.getLogger('MyLogger')my_logger.setLevel(logging.DEBUG)handler = logging.handlers.SysLogHandler(address = '/dev/log')my_logger.addHandler(handler)my_logger.debug('this is debug')my_logger.critical('this is critical')

author: mmoreau
author_rank: Contributor
timestamp: June 01, 2020, 06:45:52 pm
content: Thanks for all the help from everyone. I didn't expect that much code for saving to a binary file. I will have a project coming up where I will use that method instead. That is a great head start.https://ibb.co/HPpWtzzIt does what I need it to. Now to fix other problems but that will be a post in a different thread.Now I need to make the buttons on a perf board, design and 3D print the case for them and mount the display in a picture frame with the buttons on top.