Programm to show a filtered list of a csv file. - Page 1
Forum section: electronics / general

author: eTobey
author_rank: Super Contributor
timestamp: November 03, 2024, 12:28:37 pm
content: Hi,i need the ability, to filter data in a csv file in realtime. I wont work with libre office, as the filter is removed after each update on the data link.Anyone knows a program that could do some realtime filtering?

author: PA0PBZ
author_rank: Super Contributor
timestamp: November 03, 2024, 01:25:30 pm
content: Reading the title I immediately thought 'any spreadsheet can do that' but apparently that is not going to work for you.Care to explain a bit more, like for instance what do you do with the filtered results and what is the update rate?

author: brucehoult
author_rank: Super Contributor
timestamp: November 03, 2024, 07:09:40 pm
content: Grep, awk, perl, python, ruby, ... ?

author: radiolistener
author_rank: Super Contributor
timestamp: November 03, 2024, 08:17:16 pm
content: Basically almost any language can do that - octave, python, js, c, pascal, basic, etc...You can even connect it to ZX Spectrum and do it in Z80 asm

author: Ian.M
author_rank: Super Contributor
timestamp: November 03, 2024, 08:49:46 pm
content: Write a macro for LibreOffice Calc, that removes the filter, deletes the previous filtered data, imports the CSV file, sets up any named ranges required to match the number of lines of new data from the CSV, and reapplies the filter. Its *NOT* rocket science - any competent programmer, even one unfamiliar with LibreOffice, could adapt to their dialect of BASIC and knock out the code in under a day. Any MS VBA / Visual Basic programmer will find it even more familiar. Most of the stuff you need to know can be found by using the macro recorder to record the actions to perform the task manually, then looking at the code it generates to parameterise it so it takes the number of lines of old data from the current spreadsheet contents, and the number of lines of new data from the contents post CSV import.

author: eTobey
author_rank: Super Contributor
timestamp: November 03, 2024, 08:57:08 pm
content: Actually i dont want to write any more scripts... i want get to work on my microcontroller. I already wrote enough scripts/programs for the shortcomings of the SDS800X HD.I dont write another script for a program that checks another program that checks my original program/software that i want to develop. Its a rabbithole...

author: abeyer
author_rank: Frequent Contributor
timestamp: November 03, 2024, 11:14:56 pm
content: It would help if you'd actually reply to the request for more details about your requirements/use case, then, if you're looking for something that does exactly what you want out of the box.I'd take a look athttps://github.com/dbohdan/structured-text-tools?tab=readme-ov-file#csvand maybehttps://til.simonwillison.net/sqlite/one-line-csv-operations

author: Smokey
author_rank: Super Contributor
timestamp: November 03, 2024, 11:18:36 pm
content: this is exactly the type of job for chatgpt. paste your csv data format, and how you want it filtered, and it will generate a script in a language of your choosing.

author: SiliconWizard
author_rank: Super Contributor
timestamp: November 04, 2024, 12:38:02 am
content: Quote from: eTobey on November 03, 2024, 08:57:08 pmWell, then don't. Do as everyone does in the software industry these days, do not check much, ship fast and see what happens. It may not always work, but it's good money.

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 04, 2024, 02:30:18 am
content: Quote from: eTobey on November 03, 2024, 08:57:08 pmAh, copy that.I was going to post an example filter in Python, but that would have been a script, of course.I have also created sheets in LibreOffice/OpenOffice/Excel that filter the rows of another sheet using an arbitrary filter criteria, but that requires constructing some very crafty formulae and filling basically all cells of a sheet with them, so I think that's out too.Too bad. I guess your best bet is pasting the data to ChatGPT or similar, and hope it hallucinates something useful for you.

author: brucehoult
author_rank: Super Contributor
timestamp: November 04, 2024, 02:53:28 am
content: I guess you could hire someone in India or Congo or something to filter the data.You'd have to give them precise instructions on what exactly you mean by "filter" .. some sort of sequence of the things they'd need to do.

author: paulca
author_rank: Super Contributor
timestamp: November 04, 2024, 10:43:08 am
content: What do you mean by the terms:"Data link"and"Realtime"I'm not answering without this.

author: eTobey
author_rank: Super Contributor
timestamp: November 04, 2024, 05:42:16 pm
content: To clarify some details:Filter:I have many "id's" that also are "34" and "36". I want to filter them, meaning, they should not appear in the list anymore. Only the other "id's" should be in that list.Realtime:It should show the list from the data link (the data that is in a csv file and which that program should show), as it currently is, with a maximum delay of like 2 seconds.Libreoffice almost did that. Almost...Quote from: SiliconWizard on November 04, 2024, 12:38:02 amGood idea. Problem is: I want to use this product too, and it should never fail.chatgpt wants me to install stuff for this task. Not what i want...In the meantime i got my DSLogic Plus:Tried looking at the CAN decode list: Its a bloody useless! Every detail of the CAN frame is in one column.Now i had to create a script after all.Still its quite cumbersome to save the data every time.I am almost to a point to want to get rid of this hobby alltogether. Working on a bigger project is very frustrating, if the testequipment/software is not that useful (for the affordable ones). Writing a script for just so basic things every day...

author: PA0PBZ
author_rank: Super Contributor
timestamp: November 04, 2024, 07:09:42 pm
content: Quote from: eTobey on November 04, 2024, 05:42:16 pmThat's exactly where scripts are for, doing basic things all day that you don't want to do manually...

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 04, 2024, 07:23:27 pm
content: In Linux/Android/Mac/BSDs, this would be a trivial thing, combininginotifywait,fsnotifywait, orfswatch(documentation) and a simple Python script.The Python script would run the watcher in a subprocess, reporting changes to the target directory. Onclose_writeormoved_toevent affecting the source CSV file, the script re-reads it, saving the filtered results to a target CSV file. The latency depends on the total CPU and I/O load on the machine, but is a small fraction of a second in all but fully CPU or I/O loaded situations.Python 3has a built-incsv library, whoseexcelandexcel-tabdialects withcsv.QUOTE_NONNUMERICquoting is rock solid for Excel, LibreOffice, and OpenOffice. I've done this several times for various large datasets in real life.I'd need to write only about thirty lines of code to achieve this. Add another twenty or so for a minimal Qt/GTK/Tkinter GUI window for stopping the script, and maybe showing some info about the last modification seen; with a "filter now" button too. Essentially, as long as the script is running, any changes to the source CSV file will result in the filtered CSV file to be updated, with sub-second latency.For Windows, the issue is detecting the file-close events. You see, if the source CSV file is written to in-place, then each write to the file will affect the timestamp, and your script may read it before it has been fully written. You can work around this somewhat by adding a couple of seconds of latency, which is not needed on other operating systems. (EvenfswatchusesReadDirectoryChangesWin Windows, and that does not have a reliableclose_writedetection, unlike Linux/Android/OSX/BSD interfaces used.) One can useReadDirectoryChangesWvia thepywin32Python module, so for the few good programs that actually create a temporary file for the new contents, and only when completed rename/link/move it over the existing file, the extra latency is not needed.Personally, I don't use Windows nor have Windows installed even on a virtual machine, so I cannot help you there any further.Another approach would be to use an actual database, link the LibreOffice sheet to the database, and have the database read (or a script triggering data updates) the source CSV file when changed. I prefer to work with files myself, because then I can more easily verify and transfer the whole thing, not depending on an external server.

author: eTobey
author_rank: Super Contributor
timestamp: November 04, 2024, 09:36:24 pm
content: Quote from: PA0PBZ on November 04, 2024, 07:09:42 pmI spend many bucks to have a tool to give me some basic ouput like "ID, databytes, ackbit". I guess i have to spend even more bucks? Why are so often simple basic things not implemented/working? I want to write on my firmware, and not scripts. And thats not all: i wrote a script now, but still i have to do stuff by hand...This is my script, which converts a rather usless (in my case) list (why do the columns of each row even have no relation at all?):Code:import osimport globimport timeimport re# Define the pattern and the timeframe (in seconds)pattern = 'decoder--*.csv' # Change to your desired patterntimeframe = 1 # 1 hour in secondsoutput_file_path = 'currentFrames.csv'# Get the absolute path of the current scriptcurrent_script_path = os.path.abspath(__file__)# Get the base path (directory) of the current scriptbasepath = os.path.dirname(current_script_path)print("Script running")global newestFilewhile 1:# Get the current timecurrent_time = time.time()# Find files matching the patternfiles = glob.glob(pattern)newestFile = None# Filter for recently created filefor file in files:try:# Get the creation time (on Windows) or last modification time (on Unix-like systems)creation_time = os.path.getctime(f"{basepath}\\\\{file}") # Use os.path.getmtime(file) for Unix-like systemsexcept:print(f"EXEPT: {file} ")if current_time - creation_time <= timeframe:newestFile = open(file, 'r')if not newestFile:continue#wait until the file has been writtentime.sleep(3)# Open the output file for writingoutput_file = open(output_file_path, 'w')global linesstartPattern = r'[0-9]+,(.*),Start of frame'idPattern = r'^.+Identifier: [0-9]+ \((0x[0-9a-f]+)\)'lengthPattern = r'.+Data length code: ([0-9]+)'dataBytePattern = r'^.+Data byte [0-9]+: 0x([0-9a-f]+)'ackPattern = r'.*ACK slot: (.+)$'# Open the recent filelines = newestFile.readlines()global startTimeglobal dataBytesfor line in lines:timeMatch = re.search(startPattern, line)if timeMatch:startTime = float(timeMatch.group(1))/1000/1000sTimeString = (f"{startTime:.3f};").replace('.',',')output_file.write(sTimeString)dataBytes = NoneidMatch = re.search(idPattern, line)if idMatch:id = int(idMatch.group(1),16)output_file.write(f"{id};")lengthMatch = re.search(lengthPattern, line)if lengthMatch:length = lengthMatch.group(1)#print("")dataMatch = re.search(dataBytePattern, line)if dataMatch:dataByte = int(dataMatch.group(1),16)output_file.write(f"{dataByte};")ackMatch = re.search(ackPattern, line)if ackMatch:output_file.write(f"{ackMatch.group(1)}\n")output_file.close()The source file looked like this:Code:Id,Time,CAN-FD: Fields1,3756800.00,Start of frame2,3758800.00,Identifier: 36 (0x24)3,3782800.00,Remote transmission request: data frame4,3784800.00,Identifier extension bit: standard frame5,3786800.00,Flexible data format: 06,3790800.00,Data length code: 87,3798800.00,Data byte 0: 0x118,3816800.00,Data byte 1: 0xfd9,3834800.00,Data byte 2: 0xbe10,3852800.00,Data byte 3: 0xb411,3868800.00,Data byte 4: 0x6a12,3884800.00,Data byte 5: 0x3313,3900800.00,Data byte 6: 0x9514,3916800.00,Data byte 7: 0x5d15,3932800.00,CRC-15 sequence: 0x0c7016,3962800.00,CRC delimiter: 117,3965400.00,ACK slot: ACK18,3967400.00,ACK delimiter: 119,3969400.00,End of frameThis is what i wanted:

author: paulca
author_rank: Super Contributor
timestamp: November 05, 2024, 10:03:12 am
content: Quote from: eTobey on November 04, 2024, 05:42:16 pmIts okay, it's still a batch. Realtime would involve having the file open for read and write and reading rows while something else is inserting them. If you have completed and closed file, it's a batch and 100 times simpler.Depending on the particulars of the CSV file, such as "Does it contain any escaped comma?" I would either just pipe it through egrep or in the case of potential "full" implementation of CSV with escaping, quoted columns etc... a Python CSV reader.In the case of egrep a reg-exp should be fairly easy to construct which matches content in a set numbered column. This can be inverted of course.With a python CSV reader you will basically get a row iterator, you can do your filter in if statements, or you can usually wrap it in a collection operation like .filter(Predicate) and one line it.It sounds very simile to a "daily task" to me. Examples would be to find all log statements from the last test run that mention a certain string. If you have skills in bash, python, perl and know how to operate a commandline, you would typically not even save these, but just execute them as "onliners", knowing they will be in your command history for a few weeks anyway if needed again.

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 05, 2024, 02:14:00 pm
content: All DSView/PulseView protocol decoders have this format, because of theprotocol decoder API(seeHOWTOfor details). The exact columns vary –– not all use the second column for time in nanoseconds, for example ––, but they all tend to produce one row perfield, instead of one row perrecord.Parsing a CSV file is trivial in Python:import csvwith open(filename, 'r', newline='') as in_stream:input_csv = csv.reader(in_stream, dialect='excel')for fields in input_csv:#you havelen(fields)elements infieldsin current recordThe trick is to construct a state object or dictionary, filling in the members from different records, and writing the state out (if filtering says it is to be kept) in CSV format.For nicely formatted output, considerwith open(filename, 'x', newline='') as out_stream:output_csv = csv.writer(out_stream, dialect='excel', quoting=csv.QUOTE_NONNUMERIC)followed by aoutput_csv.writerow(fields)for each record, wherefieldsis the list (array) of fields in this record.