Utilities for extracting data from logs? - Page 1
Forum section: electronics / general

author: Stonent
author_rank: Super Contributor
timestamp: October 20, 2014, 04:15:44 am
content: I'm not even sure what you'd call it. I'm looking for some kind of gui utility (preferrably free) that can be used to extract data from raw text files or logs and then possibly turn it into XML or CSVWhat I'd like to be able to do is define things like "when you see Sun/Mon/Tues/Wed/Thu/Fri/Sat" this indicates a new record. Read that entire line and file it under the column "Date"When you see the phrase "blah" read everything after that and file it under the column "blah1"If you see "blah" again before a new record starts, read everything after that and file it under the column "blah2"If you encounter "xyz" but haven't already encountered "abc" store a null under the field abc and read the data associated with xyz.If you encounter "foo" jump to the next line and read that entire line and file it under the "foo" column.Certainly something like that exists, I'm just not sure what you call it.

author: Unknown
author_rank: Unknown
timestamp: October 20, 2014, 04:43:17 am
content: Could you provide a sample of the log data?You might be able to use Microsoft Excel.

author: Richard Crowley
author_rank: Super Contributor
timestamp: October 20, 2014, 04:50:33 am
content: Can you reveal the SOURCE of the log files? That could help us with your question.For example, are they from a web server? Which one?

author: Stonent
author_rank: Super Contributor
timestamp: October 20, 2014, 05:24:14 am
content: It's just data dumped using ipconfig, and tcpdump along with a WMI query. It's just old raw data I've collected over time that I want to move into a better format.The first part is something like this:Code:echo %date% %time% >> \\networkpath\to\logs\%computername%ipconfig /all | find /i "Host Name" >> \\networkpath\to\logs\%computername%ipconfig /all | find /i "Description" >> \\networkpath\to\logs\%computername%ipconfig /all | find /i "IPv4 Address >> \\networkpath\to\logs\%computername%ipconfig /all | find /i "Default Gateway" >> \\networkpath\to\logs\%computername%ipconfig /all | find /i "Physical Address" >> \\networkpath\to\logs\%computername%That generates something like this:Code:Sun 10/19/2014 20:48:46.61Host Name . . . . . . . . . . . . : COMPUTERABCD123Description . . . . . . . . . . . : Microsoft Virtual WiFi Miniport AdapterDescription . . . . . . . . . . . : Intel(R) Centrino(R) Advanced-N 6205Description . . . . . . . . . . . : Broadcom NetXtreme 57xx Gigabit ControllerIPv4 Address. . . . . . . . . . . : 123.213.123.213(Preferred)Default Gateway . . . . . . . . . : 123.213.213.123Physical Address. . . . . . . . . : EE-88-B4-31-A5-83Physical Address. . . . . . . . . : EE-88-B4-31-A5-82Physical Address. . . . . . . . . : EE-AC-6F-CA-DF-80Because of the way it was filtered you have to line up the first description with the first physical address and so on.Then it runs a tcpdump command that waits for a cisco CDP packet and filters for the lines that have information on the switchSince some of the computers have ethernet, wifi, and something else, this portion of the log is sometimes empty unless I change the parameters to check on nic 2, nic 3 etc.If it works correctly, it will dump something like this:Code:Device-ID (0x01), length: 10 bytes: 'SWITCH3B'Port-ID (0x03), length: 15 bytes: 'FastEthernet4/1'Platform (0x06), length: 14 bytes: 'cisco WS-C4506'Address (0x02), length: 13 bytes: IPv4 (1) 213.123.121.211Native VLAN ID (0x0a), length: 2 bytes: 244Duplex (0x0b), length: 1 byte: fullATA-186 VoIP VLAN request (0x0e), length: 3 bytes: app 1, vlan 344And the final line of the program isCode:wmic bios get serialnumberThat generates a line like this:Code:SerialNumberXYZABC123It's all raw like that because I had a specific need at the time to trace down a computer's location on the network when it couldn't be located. It ended up being useful so I decided to just keep gathering the data. I keep capturing the serial number and name because sometimes computers get renamed but the serial number is permanent, so it would make a good primary key in a database.Once I get my legacy data stored I'll work on rewriting the data collection script to wrap XML tags around individual items.

author: Stonent
author_rank: Super Contributor
timestamp: October 20, 2014, 05:26:18 am
content: I know about grep and have tried to figure out regexp, but never get it to work right. Especially stuff like "when you see this, ignore it, but display everything after it"

author: Unknown
author_rank: Unknown
timestamp: October 20, 2014, 05:33:18 am
content: It's fiddly but I did something similar with a bunch of nested IF statements, VLOOKUP/HLOOKUP and the use of LEFT/RIGHT functions in Excel. Correct me if I'm wrong but wouldn't Active Directory on the domain controller give you the same info?

author: nowlan
author_rank: Frequent Contributor
timestamp: October 20, 2014, 05:55:46 am
content: Perl would be able to do what you want. But I wouldnt recommend learning it.Possibly python if you want to learn proper scripting.Im sure there are inventory management tools around, rather than reinvent this.In fact, SNMP/WMI would fast way to poll your network.btw, mac addresses *should* be unique to each pc.

author: halexa
author_rank: Regular Contributor
timestamp: October 20, 2014, 06:23:17 am
content: Hi,I would recommend Python for this task. Easy to learn and script.

author: Rufus
author_rank: Super Contributor
timestamp: October 20, 2014, 06:33:28 am
content: "AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool."http://en.wikipedia.org/wiki/AWKGNU awk is a good implementationhttp://gnuwin32.sourceforge.net/packages/gawk.htmSyntax is C like so if you know C you are half way there.

author: PlainName
author_rank: Super Contributor
timestamp: October 20, 2014, 11:59:37 am
content: I wonder if Log File Viewer might be useful:http://www.log-file-tools.com/product/log-file-viewer-standard/index.htmlIt is a tricky one to describe, but essentially you tell it how to parse the log file and off it goes, displaying whatever fields you select in columnar format. Filtering is reasonably simple and you can export the results to CSV, etc. Things like awk might be more capable but you need to be a programmer or command line guru to get the best from them, and this one is more appropriate to a lazy GUI button pusher

author: Smokey
author_rank: Super Contributor
timestamp: October 20, 2014, 09:52:05 pm
content: Here are my recommendations in no particular order...1) Python....2) Python....3) Python........n) Python....Once you write your first script to handle something like this, you'll be hooked and find uses for tools like it all over the place. Its' time very well spent. Take a shot, and post up when you get stuck. We can help you out... but only if it's python

author: Bored@Work
author_rank: Super Contributor
timestamp: October 20, 2014, 11:39:37 pm
content: grepalone is probably too simple for the job. So eithergrepplus other basic Unix text processing tools (there are many), or a language with some more ompf. E.g. the already mentionedawk, or maybe evenperl.Quote from: Stonent on October 20, 2014, 05:24:14 amHere is a rough example how to do that kind of line up inawk. From memory, I didn't run it:Code:## Check if line starts with the name of a day.# If yes, reset counters, because a new record starts#/^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) / {# remember date information (not used later on in this example)d = $0# reset countersndesc = 0nphys = 0# empty arraydelete descnext}/Description/ {sub(/.*: /, "") # get rid of the dotted line and anything before it in the linedesc[ndesc++] = $0 # Store the remaining descriptionnext}/Physical Address/ {sub(/.*: /, "") # get rid of the dotted line and anything before it in the lineprint $0 ": " desc[nphys++] # print MAC address and corresponding descriptionnext}

author: dave_k
author_rank: Frequent Contributor
timestamp: October 21, 2014, 09:57:23 am
content: PERL = Practical Extraction & Reporting LanguageThat's what I'd be using ...

author: atferrari
author_rank: Frequent Contributor
timestamp: October 21, 2014, 11:56:24 am
content: I recall Don Lancaster (search tinaja, guru, liar) explaining how he retrieved data of accesses to his site.

author: Leiothrix
author_rank: Regular Contributor
timestamp: October 21, 2014, 09:25:25 pm
content: It sounds like you need to write a script -- Perl, PHP, Python, Ruby, whatever, to parse that data and store it in a database.If you're really keen, write a web frontend to the DB so you can query it easily from any machine.You can get all the GNU tools (grep, awk, etc), as well as the various scripting languages on Windows too if that's your thing.

author: grumpydoc
author_rank: Super Contributor
timestamp: October 21, 2014, 10:35:29 pm
content: Quote from: dave_k on October 21, 2014, 09:57:23 amAccording to the documentation Larry Wall admitted it is actually Pathologically Eclectic Rubbish ListerI'd probably use Perl as well.

author: ve7xen
author_rank: Super Contributor
timestamp: October 22, 2014, 03:51:38 am
content: To continue the tangent, it's Perl or perl and is a proper name, not an acronym. Many amusing backronyms have been devised, but the name of the language is just Perl.As far as the problem at hand, I would address this with a better tool on the machines that outputs a more sensible log than trying to parse that mess after the fact. Write some PowerShell or VBS to pull the data you need and write it to XML or submit it to a web service as JSON or whatever. Logs that are not either a) single line with leading timestamp and machine/process ID or b) structured format with well-defined grammar (ie. XML, JSON etc.) are useless and a nightmare to deal with.

author: ivaylo
author_rank: Frequent Contributor
timestamp: October 22, 2014, 07:27:45 am
content: There are even busineses nowadays around this need -https://www.loggly.comBut you do need log files in "standard" format...

author: Jeroen3
author_rank: Super Contributor
timestamp: October 22, 2014, 10:54:04 am
content: Some time ago I constructed (quick&dirty) a QT C++ tool that could read ARM Linker MAP files to sort-able tables.It was easy using regexps and known strings in the file.https://github.com/Jeroen6/MapViewer

author: grumpydoc
author_rank: Super Contributor
timestamp: October 22, 2014, 11:35:11 am
content: Quote from: ve7xenIt's Perl - to quote perl.org:"Perl" is the name of the language. Only the "P" is capitalized. Traditionally, the name of the program which runs the Perl script is "perl" with a lowercase "p". Most references to the language should use "Perl"(I know I put PERL in my post, it was left over from editing the post I was quoting and I forgot to change it).QuoteTo be fair Stonent did say he wanted the output to be in XML but if he already has "that mess" to parse I'd say a language which has powerful and flexible regular expression matching is going to be a good bet - I'd tend to go with Perl, expecially on a unix/linux platform.Given this is on Windows - dunno. Never used PowerShell so can't say. VB(S) in all its forms sucks so I'm not sure it would be my first choice for anything where it is not theonlychoice.The nice thing about Perl is there are so many ways of doing any given task.The awkward thing about Perl is that there aresomany ways of doing any given task.

author: ve7xen
author_rank: Super Contributor
timestamp: October 22, 2014, 05:00:43 pm
content: Quote from: grumpydoc on October 22, 2014, 11:35:11 amYes, I would too. My point was that it's probably easier to write a better script for the machines that outputs a sensible format than it is to parse an un-structured block of text like that. And a nicer solution to the problem, too. The relevant tools to parse the mess he has have already been mentioned, for the most part, no need to keep rehashing.QuoteThey were suggested because they are both installed by default on modern Windows and can be used for purposes such as this, widely deployed management scripts to extract information about the machine. Either has much better (and more parametric) access to system data like IP addresses and so on that he wants to extract than does a batch file. If all the machines are network accessible, it probably makes even more sense to use WMI or PowerShell remoting to do it. I think someone already mentioned there are commercial tools available to do this kind of thing in an automated, remote fashion.

author: Leiothrix
author_rank: Regular Contributor
timestamp: October 22, 2014, 09:16:33 pm
content: Quote from: ve7xen on October 22, 2014, 05:00:43 pmWe use a VBS file as part of the logon script (GPO would be fine too).The VBS script dumps a bunch of WMI & registry stuff to a text file in a standard format, then moves that file to a network file share for processing.A parser runs on the server which reads these text files and updates the database.There is a intranet frontend for the DB so our helpdesk/desktop guys can query it, and some people have direct access (read only) to the DB for ad-hoc reporting.We've tried a bunch of different methods, and this has worked the best in our environment.

author: djacobow
author_rank: Super Contributor
timestamp: October 22, 2014, 09:25:40 pm
content: Quote from: grumpydoc on October 21, 2014, 10:35:29 pmNot to beat a dead horse, but me, too. An added benefit for Perl is that you'll find modules ready to go for dumping a native data structure to <$format_of_your_choice>, be it XML (eh), JSON (yay), YAML (double yay!) or whatever. So really, you'll only have to write the parsing half of your script. The dumping part is done for you.OK, probably the same sitch in Python, but I have to give love to Perl since so many people hate on it these days.