solved: Linux: modifiying, updating, appending to live log files? - Page 1
Forum section: electronics / general

author: incf
author_rank: Frequent Contributor
timestamp: March 28, 2025, 02:36:41 pm
content: I have a situation where I have long running tests whose data is being saved to my computer in several log files.ie. some program has the file open and is writing to it constantlyI have another program that I run manually which can read non-corrupt log files and graph the data.This is good, but it has a fatal flaw. Sometimes a section of a logs get corrupted or missing due to factors outside of my control. The tests / data collection can't stop (can't be restarted). And the nature of the corruption is such that it needs to be manually edited out. (section of line numbers deleted)I can't edit the log files while they are being written to by other processes. I've tired things like trying to use the "ed" editor to delete a range of line numbers "in place", but it doesn't work.When these data corruptions happen, I have to resort to making a copy/snapshot of the log files, manually editing them, and then running them through my parsing/graphing software.I am looking for input on the process of automatically making copies of the logs, modifying them, and/or optionally updating them and then appending more recent log data.How would you do it?I think a bash script that creates copies of the files, and then fixes the copies from scratch every time it is run (deletes certain predefined ranges) it probably the simplest. But maybe utilities like rsync might be able to intelligently merge the new parts of the file while leaving the modified old part untouched?It would be nice to fix the file manually once, and them have a utility "remember" that fix without me having to manually enter in ranges of line numbers to be deleted each time de-corrupted log files are generated.

author: thephil
author_rank: Regular Contributor
timestamp: March 28, 2025, 03:19:15 pm
content: That sounds unpleasant...I would probably write a little script in python that keeps reading form the file so it can add new chunks every time there is a fresh chunk of data and continuously processes that and writes to another outfile.It would probably help if you could convince the process that creates the log to use unbuffered write so you have less chance of truncted data because the buffer is still retaining data.With respect to the corrections all I can think of is to really program a heuristic that detects the corrupted rows. "Learning" from a manual edit sounds a bit hard. maybe you can disclose a typical corruption?

author: thephil
author_rank: Regular Contributor
timestamp: March 28, 2025, 03:24:11 pm
content: And if manual editing is inevitable: Have the script write chunks of e.g. 100 rows each to separate files which you can edit manually and reassemble afterwards. Something like this:Code:log0000_0099.datlog0100_0199.datlog0200_0299.dat...Then edit. Then merge:Code:cat *.dat > assembledlogEdit: You can accomplish the chunks usingCode:split -a 4 -l 100 crazylogfile.log logAnd split even supports piping things to a filter command that does the processing if it can be automated.

author: BradC
author_rank: Super Contributor
timestamp: March 28, 2025, 03:28:35 pm
content: I'd run tail -f on the log file and pipe it through a script that recognises and drops the bad data. You could even have that break it up into nicely chunked files.Ideally of course you'd fix the software writing bogus log data.

author: ledtester
author_rank: Super Contributor
timestamp: March 28, 2025, 03:48:53 pm
content: Don't try to modify the original log files.Write another program to a create clean copy of the data for your graphing program which is something we can help you with.

author: langwadt
author_rank: Super Contributor
timestamp: March 28, 2025, 03:50:59 pm
content: Quote from: incf on March 28, 2025, 02:36:41 pmchange your parsing/graphing software to ignore the corrupt lines?

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 28, 2025, 05:38:17 pm
content: Standard approach for writing to log files continuously is to have the writer respond to a SIGUSR1 or SIGHUP signal by flushing its internal buffers to the file, optionallyfdatasync()'ing the file, closing it, and re-opening the log file (creating it if necessary) in append mode.You see, in Linux/POSIXy/UNIX/BSD systems you only use the filename for opening the underlyinginode. After you have a file open, renaming the file or even deleting does not matter. It is also the inode that contains the access mode, ownership, et cetera. It is only when the last open descriptor to the inode is closed that the inode (and thus contents on the filesystem) are deleted on storage. Without a filename, it cannot be opened anymore (although someone with a descriptor open can duplicate it and pass it or the duplicates to another process via Unix Domain socket message).The standard way torotate log filesis therefore to rename the current log file, then send the signal to the log writer, and wait until the new log file (with the old name) to appear. When it does, you can be sure that all pending writes have completed. (Re-opening the log file by the writer must not be delayed or postponed until the next record to be written; it must be done as soon as possible, in order to not unduly delay log rotation.fdatasync()is normally only used with privileged important records.)This is the way syslogd and other traditional logging services maintain log integrity, too. (I'm not sure systemd-based logging has any integrity guarantees.)Can you check if your log writer supports this? Do you have its sources so it can be added if it does not do so yet?If not, I could whip up an efficienttail -flogfile>newlogfilereplacement that does. You'd keep this helper running as long as the log generator is running –– perhaps run it as the parent of the log generator? –– and when you want to rip out the next set of logged data, you renamenewlogfile,killall -USR1helper, wait for the new file to appear (while [ ! -enewlogfile]; do sleep 0.1 ; done), and then use the renamed file. However, it won't know whether log records are complete or not; at best, it can split the files just after a newline.

author: xvr
author_rank: Frequent Contributor
timestamp: March 28, 2025, 06:05:56 pm
content: Take a look atlogrotate(https://linux.die.net/man/8/logrotate)It could resolve a lot of problems with growing logs.

author: bitwelder
author_rank: Super Contributor
timestamp: April 02, 2025, 10:14:21 am
content: If the application that write logs (e.g. to logfile.log) doesn't write continuously (but anyway keeps the logfile open)one option is to copy the current log to a safe location$ cp logfile.log /somewhere/safe/logN.logand then zero the original file without interrupting anything$ cp /dev/null logfile.logAnd then repeat after a while copying another piece of the log to a different logN.log.This way the application doesn't see any change, as the stream opened for writing will be still there,and it can keep appening content (to a file that has been truncated, effectively)As mentioned at the beginning, main limitation of this method is how to handle new logs entriesthat are written *while* the copy is in progress, or better between the first copy and the wipe.

author: madires
author_rank: Super Contributor
timestamp: April 02, 2025, 11:13:02 am
content: There's a great tool called awk for processing text files. Its a kind of scripting language specifically for text, but it will take some time to learn it. If the logging follows some syntax you could use awk to check for vaild log lines ond only pass those on.Ideally the applications generating the log files should be changed to support log rotation.

author: ledtester
author_rank: Super Contributor
timestamp: April 02, 2025, 05:53:22 pm
content: I would use this approach:- you have a program (log-converter) which runs periodically or on an as-needed basis.- the converter program reads from the original log file and makes a clean copy in another file- the program uses a separate file to keep track of the last file position in each log file that it has scanned- it uses that position file to know where to continue converting from and it updates that file when it is finished converting a file (i.e. it hits EOF)The idea is that you can just keep on running "log-convert some-log-file" manually or in a cron job and it will continue its work from where it left off.

author: JohanH
author_rank: Frequent Contributor
timestamp: April 03, 2025, 08:00:37 am
content: Sometimes a named pipe can be handy (create with mkfifo, tail the log to the pipe). I can recall having used that at some point. Great for concatenating multiple log files into one, or writing additional data, without storing the output on disk. Only one process can read from the pipe at the time, though, so that might be a limitation.

author: macboy
author_rank: Super Contributor
timestamp: April 03, 2025, 01:59:08 pm
content: I'd be tempted to use `tail -f` which will "watch" a file and print new lines to its STDOUT.Then I'd pipe that output to a script which can try to parse the lines and filter out the bad ones. The valid lines would then be concatenated to a "clean" version of the log file. Then your data consumer can do its thing on the clean log. This method avoids repeatedly copying the log file.

author: BradC
author_rank: Super Contributor
timestamp: April 03, 2025, 02:06:57 pm
content: I saw the title was changed, so go on then and tell us how you solved it ...

author: incf
author_rank: Frequent Contributor
timestamp: April 03, 2025, 02:17:01 pm
content: I actually didn't solve it. Ran out of time. I liked the solution of splitting the log files into 100 line chunks, deleting the bad section, and reassembling in order. But, I'll have to revisit it in a few weeks.I may be forced to make the parsing/conversion more robust.