Project: Vector (Observability and Analysis / Observability)
File: parsing-csv-logs-with-lua.md

title: Parsing CSV logs with Lua description: Parse structured application logs in CSV format using Lua transform authors: ["binarylogic"] domain: transforms transforms: ["lua"] weight: 4 tags: ["lua", "csv", "logs", "transform", "advanced", "guides", "guide"] {{< requirement title="Pre-requisites" >}} You understand the <a href="/docs/reference/configuration/transforms/lua">basic Lua concepts</a>. You understand the <a href="/docs/about/concepts">basic Vector concepts</a> and understand <a href="/docs/setup/quickstart/">how to set up a pipeline</a> {{< /requirement >}} Vector has many built-in for structured logs formats. However, when you need to ship logs in a custom or application-specific format, have got you covered. This guide walks through reading CSV logs using source and parsing them using the transform with a loadable Lua module. For certainty, it is assumed in the following that the logs to be read are produced by in PostgreSQL. For example, there might be the following log file: ```csv title="log.csv" 2020-04-09 12:48:49.661 UTC,,,1,,localhost.1,1,,2020-04-09 12:48:49 UTC,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"" 2020-04-09 12:48:49.669 UTC,,,27,,localhost.1b,1,,2020-04-09 12:48:49 UTC,,0,LOG,00000,"database system was shut down at 2020-04-09 12:48:25 UTC",,,,,,,,,"" 2020-04-09 12:48:49.683 UTC,,,1,,localhost.1,2,,2020-04-09 12:48:49 UTC,,0,LOG,00000,"database system is ready to accept connections",,,,,,,,,"" ``` Let us draft an initial version of the Vector's configuration file: ```toml title="vector.toml" data_dir = "." [sources.file] type = "file" include = ["*.csv"] startatbeginning = true [transforms.lua] inputs = ["file"] type = "lua" version = "2" hooks.process = """ function (event, emit) -- to be expanded emit(event) end """ [sinks.console] inputs = ["lua"] type = "console" encoding.codec = "json" ``` This config sets up a that reads log files, pipes them through the parsing transform (which currently is configured to just pass the events through), and displays the produced log events using sink. At this point, running `vector --config vector.toml` results in the following output: ```json {"file":"log.csv","host":"localhost","message":"2020-04-09 12:48:49.661 UTC,,,1,,localhost.1,1,,2020-04-09 12:48:49 UTC,,0,LOG,00000,\"ending log output to stderr\",,\"Future log output will go to log destination \"\"csvlog\"\".\",,,,,,,\"\"","timestamp":"2020-04-09T14:33:28Z"} {"file":"log.csv","host":"localhost","message":"2020-04-09 12:48:49.669 UTC,,,27,,localhost.1b,1,,2020-04-09 12:48:49 UTC,,0,LOG,00000,\"database system was shut down at 2020-04-09 12:48:25 UTC\",,,,,,,,,\"\"","timestamp":"2020-04-09T14:33:28Z"} {"file":"log.csv","host":"localhost","message":"2020-04-09 12:48:49.683 UTC,,,1,,localhost.1,2,,2020-04-09 12:48:49 UTC,,0,LOG,00000,\"database system is ready to accept connections\",,,,,,,,,\"\"","timestamp":"2020-04-09T14:33:28Z"} ``` In order to perform actual parsing, it is possible to leverage . Because it consists of a , it is possible to just download it to the same directory where `vector.toml` is stored: ```bash curl -o csv.lua https://raw.githubusercontent.com/geoffleyland/lua-csv/d20cd42d61dc52e7f6bcb13b596ac7a7d4282fbf/lua/csv.lua ``` Then it would be possible to load it by calling Lua function in the configuration section: ```toml source = """ csv = require("csv") """ ``` With this `source` the `csv` module is loaded when Vector is started up (or if the `lua` transform is added later and the config is automatically reloaded) and can be used through the global variable `csv`. With the `csv` module, the can be changed to the following: ```toml hooks.process = """ function (event, emit) fields = csv.openstring(event.log.message):lines()() -- parse the `message` field

= nil -- drop the `message` field column_names = { -- a sequence containing CSV column names -- ... } for column, value in ipairs(fields) do -- iterate over CSV columns columnname = columnnames[column] -- get column name event.log[column_name] = value -- set the corresponding field in the event end emit(event) -- emit the transformed event end """ ``` Note that the `column_names` can be created just once, in the `source` section instead to speed up processing. Putting it there and using the column names from the PostgreSQL documentation results in the following definition of the whole transform: ```toml title="vector.toml" [transforms.lua] inputs = ["file"] type = "lua" version = "2" source = """ csv = require("csv") -- load external module for parsing CSV column_names = { -- a sequence containing CSV column names "log_time", "user_name", "database_name", "process_id", "connection_from", "session_id", "sessionlinenum", "command_tag", "sessionstarttime", "virtualtransactionid", "transaction_id", "error_severity", "sqlstatecode", "message", "detail", "hint", "internal_query", "internalquerypos", "context", "query", "query_pos", "location", "application_name", } """ hooks.process = """ function (event, emit) fields = csv.openstring(event.log.message):lines()() -- parse the `message` field event.log.message = nil -- drop the `message` field for column, value in ipairs(fields) do -- iterate over CSV columns columnname = columnnames[column] -- get column name event.log[column_name] = value -- set the corresponding field in the event end emit(event) -- emit the transformed event end """ ``` Trying to run `vector --config vector.toml` with the same input file results in structured events being output: ```json {"applicationname":"","commandtag":"","connectionfrom":"","context":"","databasename":"","detail":"","errorseverity":"LOG","file":"log.csv","hint":"Future log output will go to log destination \"csvlog\".","host":"localhost","internalquery":"","internalquerypos":"","location":"","logtime":"2020-04-09 12:48:49.661 UTC","message":"ending log output to stderr","processid":"1","query":"","querypos":"","sessionid":"localhost.1","sessionlinenum":"1","sessionstarttime":"2020-04-09 12:48:49 UTC","sqlstatecode":"00000","timestamp":"2020-04-09T19:49:07Z","transactionid":"0","username":"","virtualtransactionid":""} {"applicationname":"","commandtag":"","connectionfrom":"","context":"","databasename":"","detail":"","errorseverity":"LOG","file":"log.csv","hint":"","host":"localhost","internalquery":"","internalquerypos":"","location":"","logtime":"2020-04-09 12:48:49.669 UTC","message":"database system was shut down at 2020-04-09 12:48:25 UTC","processid":"27","query":"","querypos":"","sessionid":"localhost.1b","sessionlinenum":"1","sessionstarttime":"2020-04-09 12:48:49 UTC","sqlstatecode":"00000","timestamp":"2020-04-09T19:49:07Z","transactionid":"0","username":"","virtualtransactionid":""} {"applicationname":"","commandtag":"","connectionfrom":"","context":"","databasename":"","detail":"","errorseverity":"LOG","file":"log.csv","hint":"","host":"localhost","internalquery":"","internalquerypos":"","location":"","logtime":"2020-04-09 12:48:49.683 UTC","message":"database system is ready to accept connections","processid":"1","query":"","querypos":"","sessionid":"localhost.1","sessionlinenum":"2","sessionstarttime":"2020-04-09 12:48:49 UTC","sqlstatecode":"00000","timestamp":"2020-04-09T19:49:07Z","transactionid":"0","username":"","virtualtransactionid":""} ``` Or, applying pretty formatting to one of the output events: ```json { "application_name": "", "command_tag": "", "connection_from": "", "context": "", "database_name": "", "detail": "", "error_severity": "LOG", "file": "log.csv", "hint": "Future log output will go to log destination \"csvlog\".", "host": "localhost", "internal_query": "", "internalquerypos": "", "location": "", "log_time": "2020-04-09 12:48:49.661 UTC", "message": "ending log output to stderr", "process_id": "1", "query": "", "query_pos": "", "session_id": "localhost.1", "sessionlinenum": "1", "sessionstarttime": "2020-04-09 12:48:49 UTC", "sqlstatecode": "00000", "timestamp": "2020-04-09T19:49:07Z", "transaction_id": "0", "user_name": "", "virtualtransactionid": "" } ``` After the task of parsing the CSV logs is accomplished, the following improvements can take place. CSV supports line breaks in strings. However, by default `file` source creates a separate event from each line. There are two options to deal with this: For simple cases it might be possible to use the configuration option in the `file` source. For more complex cases the messages from multiple events can be conditionally concatenated in the Lua code. See for more details on this. By default, all columns are parsed as strings. It is possible to convert them to other right in the Lua code using built-in functions, such as . Alternatively, it is possible to add the transform after the `lua` transform, for example, to .