Package: dxql
Data Exploration Query Language (DXQL)Requires Python 3.7UsageImport dxql.search.Pipeline into your projectInstantiate a Pipeline using Pipeline.create_pipeline(query-string)Use the new pipeline to search over an iterable of dicts using pipeline.execute(events)Example:from dxql.search import Pipeline
pipeline = Pipeline.create_pipeline('search ip=192.168.1.10')
results = pipeline.execute(events)eventscan be any iterable. To search a file, just pass the opened file topipeline.execute(). Each line of the file will be considered an event.Example:# myfile.json is a file where each line is a JSON dictionary
with open('myfile.json') as file:
    results = pipeline.execute(file)SearchingSearching is inspired by Splunk's query language.Throughout the rest of this document, I will use the terms "search" and "query" interchangebly.A query can consist of multiple commands separated by a pipe (|). Imagine a multiple-command search as a "pipeline" where each command is applied to the data in turn, with the data being fed from one command to the next until the end of the pipeline.There are four commands available:1. searchThesearchcommand allows you to filter the data using key-value pairs and modifiers likeORandNOT. It must be the first command in the query.Usage:search <expression>...<expression><comparison-expression> | NOT <expression> | <expression> OR <expression><comparison-expression><field><operator><value><operator>= | != | < | <= | > | >=ExamplesRetrieving data from an indexThis search will return all data from thegeoipindex.search index=geoipRetrieving GeoIP data for specific IPsUse theORmodifier to specify multiple values for a field.search index=geoip ip=192.168.1.10 OR ip=192.168.1.11Retrieving GeoIP data for all IPs except onesearch index=geoip ip!=192.168.1.15orsearch index=geoip NOT ip=192.268.1.15Retrieving data for a specific IP from multiple indicesIt is not required to search by index.search ip=192.168.1.15The above search will return data withip=192.168.1.15from all indices (in this case, data from indicesgeoipandip_rdapwill be returned; events inrdapdo not contain anipfield).2. fieldsThefieldscommand allows you to display only the fields you want to see.Usagefields <field>...ExampleRemove all fields from the results except foripandcontinent_name:search index=geoip | fields ip continent_name3. joinThejoincommand allows you to join data together by a field (the "by-field"). Each event that shares the same value for the by-field is joined together under one event. This allows you to join data from two disparate data sources.Usagejoin BY <by-field>ExampleJoin an IP with its associated RDAP data using theip_rdapandrdapindices:search index=ip_rdap OR index=rdap | join BY handlehandleis the 'by-field', the field that is shared by the different kinds of data.4. prettyprintTheprettyprintcommand may only be used as the last command in the search. It allows you to print the result set in a prettier fashion than plain JSON blobs.Usageprettyprint format=<format><format>json | tableExamplesPrint results as pretty JSONUsingformat=jsonstill prints each result as JSON but with newlines and indentation.search index=rdap | prettyprint format=jsonPrint results as a tableUsingformat=tableprints the results as a formatted table.search index=rdap | prettyprint format=tableIf there are a lot of fields in the result set, the results will overflow onto the next line(s); therefore, it is recommended to pare down unwanted fields usingfieldsbefore usingprettyprint format=table. This happens expecially when joiningip_rdapandrdapdata together. Many IPs share the samerdapdata, so the IP values will become very long. I recommend specifying the IP(s) you are interested in before doing thejoin.