Package: feedoo
feedooSponsored bySpartan conseilWhat is feedoo ?feedoo is an ETL, for Extract, Transform and Load. Basically, it gets data from files or database, process it thanks to pipelines and store data to a file or a database. It is very versatile and processing brick can be added without pain.The purpose of feedoo is generic :ETL to convert database to another oneAlerting like elastalertGather information from agent like FluentbitSIEM with correlation ruleIntrusion detection thanks AI...The feedoo's design is forSecurityOperationalCenter (SOC). But if you need to play with data, you need feedoo as friend :)Why ?They are many reasons why I decided to build feedoo.Firstly, I work withRethinkDBas main database. It is amazingly easy to use, with enough performance for my needs. But the main drawback is about community tools. Briefly, they are no connector to work with, especially with Fluentd, Fluentbit or a clone of Elastalert.Here we are: the second point ! I really appreciate Fluent family, especially Fluentbit fully written in C. Nevertheless, a drawback arrive when we talk about plugins or modifications. I worked many years with Fluentd and it can become painful when you need something was not shipped with.So asortofPythonversion a Fluent with rules and easy extension seems to me a good idea !InstallationThe installation is very easy :pip3installfeedooFirst runYou just install feedoo and you want to test ? Let's do a basic example !Create a file at/etc/feedoo/default.yamland copy-paste that :pipelines:"pipeline#1":-name:input_dummytag:"my_pypeline"data:{"log":"mylog"}-name:output_stdoutmatch:"*"Now execute feedoo :you@computer:>feedoo
my_pypeline[1607608082]:{'log':'my log'}It works ! :tada: You ran your firstpipeline.You said pipeline ?Keep in mind previous example, I will reuse it now.The heart of the processing is based onpipeline. It is similar topipeoperator in Unix system : every action do a basic operation and forward data to the following action :you@computer:>cat/var/log/auth.log|head|grep"sudo"feedoo do processing like this but add atagto data. This way following action can decide to process the data (if itmatch) or just forward it to the next action. Tag is added by the data producer ("my_pipeline" in input_dummy) and other action will try to match (" * " in output_stdout). In the feedoo context, we call dataEvent. Indeed diffent : Event contains data, called record (dict), an unix timestamp and the tag.Actions are categorized in four cases :inputoutputfilterparserInputInput produces events in the pipeline, including tag. If an input receives a event it forwards it to next action.input_dummyIt is useful for testing purpose and forward events base on dicts.Parameters :tag: events' tagdata: a dict or a list of dict with factrepeat=1: number of repeat of data. Set to negative number of infinit loop.Example:-name:input_dummytag:"my_pypeline"data:{"log":"mylog"}input_fileIt watch a path and load file if :The file exists on the startupThe file was create (written and closed)Parameters :tag: events' tagwatch_path: the path watched, typically a directorypath_pattern: provide a pattern which must match when a file is found with watch_pathremove=False: remove the file once readExample:-name:input_filetag:"logs"watch_path:/var/logpath_pattern:/var/log/stuff.*.loginput_forwardBased onFluentbit-server-py, it allows to received event from Fluentbit agent using the forward protocol.
It support authentication using shared key and TLS. No tag parameter is available since it is provided by the agent.Parameters :host="localhost": Used to bind socket serverport=24224: Port used to bind the servertls_enable=False: Used to enable TLS supportkey_file=None: if TLS enabled, path to the key filecrt_file=None: if TLS enabled, path to the certificate fileshared_key=None: defined a shared key between servers an nodes. If set, enable authenticationserver_hostname="": define the hostname, useful for shared_key authenticationbuffer_size=32768: define the incoming buffer sizequeue_size=1000: define the queue size.Warning: if you exposed the port to internet, use authentication and TLS. If you can, add firewall rule to decrease surface attack. BE CAREFUL !Example:-name:input_forwardport:24224host:0.0.0.0tls_enable:truekey_file:/etc/tls/foo.keycrt_file:/etc/tls/foo.crtshared_key:my_pr1vate_sh4red_K3yserver_hostname:foo.cominput_sqliteThis input use a SQLite database as input. It works with time serie : each table is base on time (ex : log_%Y%m%d) and a field is an
unix timestamp (int or float).Parameters :tag: tag used for generated eventwindow: time in second of processing windowtime_key: field use in database as timestamptable_name_match: Pattern (unix style) for find table (ex : log_*)filename: SQLite database pathfields: dict of field name and sql type as value (ie TEXT, FLOAT, INTEGER, BLOB)offset=0: Time in second of working time offset. Can be useful to process data in an other time zone or remove old tablesremove=False: If True, remove data once readreload_position=False: If True, reload data from the last known time (0 by default). Otherwise, start at current time.db_path=None: file path to store internal state. None means only RAM is used.Example:-name:input_sqlitetag:"log"table_name_match:"log_*"filename:"/my/path/test.db"fields:timestamp:INTEGERline:TEXTtime_key:timestampinput_rethinkdbThis input use a RethinkDB database as input. It works with time serie : each table is base on time (ex : log_%Y%m%d) and a field is an
unix timestamp (int or float).Parameters :tag: tag used for generated eventwindow: time in second of processing windowtimestamp_index: field use in database as timestamptable_name_match: Pattern (unix style) for find table (ex : log_*)ip=localhost: Rethinkdb instance IPport=28015: Rethinkdb instance portdatabase_name=test: Rethinkdb database namewait_connection=30: Wait time for database warm upoffset=0: Time in second of working time offset. Can be useful to process data in an other time zone or remove old tablesremove=False: If True, remove data once readreload_position=False: If True, reload data from the last known time (0 by default). Otherwise, start at current time.db_path=None: file path to store internal state. None means only RAM is used.Example:-name:input_rethinkdbtag:"log"table_name_match:"log_*"ip:foo.bar.comtimestamp_index:timestampwindow:60OutputIt exports events out of the pipeline. It can be file, database, etc.output_archiveIt stores in buffer event and it writes buffer in file.Parameters :match: pattern to match tagtime_key: used to extract time to interpolate path_templatepath_template: Template of the file pathbuffer_size=1000: Number of event stored before flushtimeout_flush=60: Flush buffer after timeout, in seconddb_path=None: file path to store internal state. None means only RAM is used.Example :-name:output_archivetime_key:timestamppath_template:"/archives/{source}/log-%Y%m%d.json"match:mylogNotes :if the example received the event contains record{"timestamp":1607618046, "source":foo, "data":"test"}, the path will be/archives/foo/log-20201210.jsonoutput_rethinkdbStore events inRethinkDBas time serie.Parametersmatch: pattern to match tagtime_key: used to extract time to interpolatetable_templatetable_template: Template of the table namebuffer_size=1000: Number of event stored before flushdatabase="test": database nameip="localhost": f set, change the database destination ipport=None: if set, change the database destination portwait_connection=30: used to wait the database warmuptimeout_flush=60: Flush buffer after timeout, in seconddb_path=None: file path to store internal state. None means only RAM is used.Example:-name:output_rethinkdbtime_key:timestamptable_template:"log-%Y%m%d"database:testip:rethink.comport:28015match:my_logoutput_stdoutDisplay event in stdout.Parametersmatch: pattern to match tagExample:-name:output_stdoutmatch:my_logoutput_sqliteStore events in Sqlite database. Only defined fields are stored in event, others are dropped. If a field is missing, the document is dropped.Parametersmatch: pattern to match tagtime_key: field use for time serie.table_template: template string used to generate table namesfields: dict of field name / Sqlite type. Sqlite types are REAL, TEXT, INTEGER or BLOB.filename: Sqlite filename. Can be :memory: to save in RAM.buffer_size=1000: Number of event stored before flushtimeout_flush=60: Flush buffer after timeout, in seconddb_path=None: file path to store internal state. None means only RAM is used.Example:-name:output_sqlitematch:my_logtable_template:"log_%Y%m%d"fields:timestamp:INTEGERline:TEXTfilename:"/tmp/test.db"db_path:"/tmp/test.pos.db"ParserA parser take an event et parse one field with a specific format :regex,json, etc.parser_jsonRead a field and add new fields to existing event.Parameters :match: pattern to match tagkey: Key to be parsedmode="merge": A string that can be "merge", "tree" or "add"Example of modes :merge:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "aaa":"bb"}add:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "value":'{"aaa": "bb"}', "aaa":"bb"}`tree:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "value":{"aaa":"bb"}}Example :-name:parser_jsonmatch:my_logkey:json_logmode:addparser_regexRead a field and add new fields to existing event.Parameters :match: pattern to match tagkey: Key to be parsedregex: define the behaviour. Use name group to create fieldmode="merge": A string that can be "merge", "tree" or "add"Example of modes :merge:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "aaa":"bb"}`add:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "value":'{"aaa": "bb"}', "aaa":"bb"}tree:  {"key":"Z", "value":'{"aaa": "bb"}'} -> {"key":"Z", "value":{"aaa":"bb"}}Example :-name:parser_regexmatch:my_logkey:linemode:mergeregex:".+?(?P<name>\\{.+\\})"FilterProcess event, may create or delete events.filter_changeThis action will monitor a certain field and match if that field changes. The field must change with respect to the last event with the same query_key.Parameters :match: pattern to match tagtag: tag used to generate new event on changealert: dict used to generate new event on changecompare_key: key monitored to find changequery_key: key used to group type of eventignore_null=True: ignore if compare_key is missing. If ignore_null if false, missing compare_key is a valid statedb_path=None: file path to store internal state. None means only RAM is used.Example :-name:filter_changematch:my_logtag:my_alertalert:title:The hostname change of statuspriority:2compare_key:statusquery_key:hostnameif events are :{"hostname":"foo.bar", "status":"on"}{"hostname":"foo.bar", "status":"off"}Then a new event will be created on second eventfilter_dateThis action performs a time parsing and allows to change the time format. Under the hook, it useChronyklibrary so feel free refere about time format.Parameters :match: pattern to match tagkey: define the key to be parsedformat=None: define the output format of key's value. None means unix timestampsExample :-name:filter_datematch:dateEvent likes{"date":"Fri, 11 Dec 2020 08:30:13 +0000"}become{"date":1607675413}It takes a date and convert it to timestampfilter_renameThis action rename fields. Useful to insert in database if the name is not compatible.Parameters :match: pattern to match tagkeys: dict of old name / new nameExample :-name:filter_renamematch:datekeys:date:timestampEvent likes{"date":1607675413}become{"timestamp":1607675413}filter_remove_keysThis action remove one or more key in event.Parameters :match: pattern to match tagkeys: on string or a list of string to describe keys to be removed.Example :-name:filter_remove_keysmatch:datekeys:-A-BEvent likes{"A":1, "B":2, "C":3}become{"C":3}filter_retagThis action change the event's tag with a value in event or with a constant value.Parameters :match: pattern to match tagvalue: New tag if key doesn't exist or if key=Nonekey=None: event value used to retag event. Use value parameter if missingExample :-name:filter_retagmatch:my_logvalue:generic_logkey:sourceIf event looks like{"source":"auth", "data":"xxx"}, the new tag will be "auth".
If event looks like{"data":"xxx"}, the new tag will be "generic_log".filter_frequencyThis action matches when there are at least a certain number of events in a given time frame. This may be counted on a per-query_key basis.Parameters :match: pattern to match tagtag: tag used to generate new event on changealert: dict used to generate new event on changenum_events: match if number of event during the time frame if higher or equal to this valuetimeframe: duration of the time windows in secondsquery_key=None: key used to group type of eventdb_path=None: file path to store internal state. None means only RAM is used.Example :-name:filter_frequencymatch:my_logtag:my_alertalert:title:The hostname change of status too oftenpriority:1query_key:hostnamenum_events:10timeframe:60filter_spikeThis action matches when the volume of events during a given time period is spike_height times larger or smaller than during the previous time period.
It uses two sliding windows to compare the current and reference frequency of events. We will call this two windows “reference” and “current”. A query
key and a field value can be defined.Parameters :match: pattern to match tagtag: tag used to generate new event on changealert: dict used to generate new event on changespike_height: define the factor between current and reference that create an eventspike_type: define the spike direction : up, down or bothtimeframe: duration of the time windows in secondsquery_key=None: key used to group type of eventfield_value=None: When set, uses the value of the field in the document and not the number of matching documents. This is useful to monitor for example a temperature sensor and raise an alarm if the temperature grows too fast. Note that the means of the field on the reference and current windows are used to determine if the spike_height value is reacheddb_path=None: file path to store internal state. None means only RAM is used.threshold_ref=10: minimum number of event in the reference frame to be evaluatedthreshold_cur=10: minimum number of event in the current frame to be evaluatedExample :-name:filter_spikematch:my_logtag:temperature_alertalert:title:The temperature rise to quicklypriority:1spike_height:2spike_type:upfield_value:temperaturetimeframe:60filter_flatlineThis action matches when the total number of events is under a given threshold for a time period.Parameters :match: pattern to match tagtag: tag used to generate new event on changealert: dict used to generate new event on changethreshold: minimum number of event in the frame (other wise, rise an alert)timeframe: duration of the time windows in secondsquery_key=None: key used to group type of eventforget_keys=True: set to false to keep tracking existing query key (emit alert for ever if no event come back).db_path=None: file path to store internal state. None means only RAM is used.Example :-name:filter_flalinematch:my_logtag:heartbeat_downalert:title:No more heartbeat !priority:3threshold:1field_value:sourcetimeframe:60filter_dnsblThis action useDNSBLmecanism to emit alert. To prevent negative false,
many domains can be used; This way can have a major drawback in term of performance. Nevertheless, we use multiprocessing approach to the the overhead low.Parameters :match: pattern to match tagtag: tag used to generate new event on changekey: key used to catch an ip value. If the value is not a valid ip, no processing is done.threshold_percent: minimum percent of trigged blacklist to rise an alertdomains: list of domains uses for DNSBLalert: dict used to generate new event on changedb_path=None: file path to store internal state. None means only RAM is used.timeout=3600: Duration in second of the cache. Prevent too many request on DNSBL.Example :-name:filter_dnsblmatch:netflowtag:ip_blacklistalert:title:you reach a bad serverpriority:3threshold:50key:destination_addressdomains:-aaa.dns.com-bbb.dns.com-ccc.dns.comtimeout:36000filter_matching_listThis action performs a multiple string matching using aho corasick algorithm. It can be used fordarklistorlightlist.Parameters :match: pattern to match tagtag: tag used to generate new event on changekey: key used to catch an ip value. If the value is not a valid ip, no processing is done.alert: dict used to generate new event on changemode: define the way the detection is performed.presentacts as darklist,absentacts as lightlist.matching_list: define the list of word to be detected (or not). Mutually exclusive withmatching_list_file.matching_list_file: define the filename containing the list of word to be detected (or not). Each line of the file is a word. Mutually exclusive withmatching_list.Example :-name:filter_matching_listmatch:netflowtag:bad_protocolalert:title:protocol is not allowedpriority:3key:protocolmode:absentmatching_list:-TCP-UDP-ARPiconsThanks to flaticon.com !Icons made byFreepikfromwww.flaticon.com