Package: tstat_transport
Client programs to read a tstat file hierarchy, parse the logs, convert
them to JSON objects, and send them to a remote archive (like a RabbitMQ
server). This tool was originally developed to support theNational
Science Foundation NetSage Project.Overviewtstat_sendWhen invoked, it crawls a tstat file hierarchy laid out like this:tstat/2016_02_17_12_53.out:
total 808
-rw-r--r--@ root users  135103 Feb 17 13:54 log_tcp_complete
-rw-r--r--@ root users  277154 Feb 17 13:54 log_udp_complete

tstat/2016_02_17_13_54.out:
total 792
-rw-r--r--@ root users  123801 Feb 17 14:54 log_tcp_complete
-rw-r--r--@ root users  278301 Feb 17 14:54 log_udp_completeSelected data are extracted from the logs, formatted into JSON objects,
and lists of the objects are sent to a remote server for archiving. The
objects from a single set of logs log are broken into a series of
smaller lists (up to 100 objects per list). Each “sub-list” gets sent to
the remote server so no one single send operation swamps the remote
server.When the logs in each directory have been successfully processed (the
data have been sent, delivery confirmations received, etc), a dotfile
named.processedwill be dropped in that directory. That marks that
directory as processed, and those logs will be ignored on subsequent
runs. Thetstat_cullutility similarly uses the .processed dotfiles
to prune old logs.It is not a persistent process and would be run periodically from cron
(for example) to periodically process logs on a “live” machine.Currently, the only “transport” that is supported is sending the JSON to
a RabbitMQ server, but it would be relatively straightforward to
implement other transports like using HTTP to send to a REST API.Usagetstat_send argumentsRequired–directoryPath to the “root” of the directory structure where tstat writes the
timestamped directories and logfiles. No default.–configPath to the .ini style config file used to pass configuration directives
to the underlying transport code.Default:./config.iniOptional–thresholdThe transfer threshold in megabytes. Any transfer below this threshold
below will be ignored.Default:1000–transportSpecify the underlying transport to send the JSON over. Currently, only
RabbitMQ is supported.Default:rabbit–sensorThe sensor_id element of the message metadata defaults tosocket.gethostname()- using this flag will set that value manually.–instanceThe instance_id element of the message metadata defaults to 0 - using
this flag will set that value manually. This makes it possible to
differentiate between instances on sensors that have multiple tstat
streams.–singleProcess a single “timestamped directory” of files, send JSON and exit.
This is primarily for development or debugging.–no-transportSkips sending the messages to the selected transport and dumps them to
standard out instead. Use standard shell redirection...--no-transport> file.jsonto save output to a file.–verbose and –debug--verbosetriggers additional log output.--debugchanges the
log level tologging.DEBUGin the transport module. This is
primarily for debugging connection problems with RabbitMQ, or to get
detailed output on the transactions with the remote server.Config filetstat_senduses an .ini style configuration file to pass options to
the underlying transport code. Doing this with command-line args would
be too ponderous. Example config file:[rabbit]
# host/port are required for all transport variants
host = localhost
# this is the rabbit ssl port, if not, use default 5672 or custom port
port = 5671
# these are required for some transports
username = esnet
password = some_mysterious_password
use_ssl = True
# these are rabbit specific - the exchange key is required
# even if you don't set the value to anything/use the default "".
vhost = netsage
queue = netsage_tstat
routing_key = netsage_tstat
exchange =

# This is an optional stanza. The key/value pairs
# will be passed to channel.queue_declare() as kwargs
# (ie: durable, exclusive, auto_delete, etc).
[rabbit_queue_options]

# This is an optional stanza. The key/value pairs
# will generate a dict to be passed as kwargs to ssl.wrap_socket()
# https://docs.python.org/2/library/ssl.html#ssl.wrap_socket
[ssl_options]The valueshostandportwill be required for all transport
variants. If they are not supplied, a configuration error occur.The rabbit transport requires theusernameandpasswordconfig values. They may also be enabled in other transport variants.vhost, queue, routing_key and exchangeshould be self-explanatory
RabbitMQ directives.Therabbit_queue_optionsstanza is optional and can be used to
pass additional kwargs toqueue_declare()if need be. By default
the code only passes thequeueargument with the name of the
queue.Thessl_optionsstanza is optional too. Only necessary if
additional args (paths to keyfiles, etc) need to be passed to the
underlyingssllibrary.Message formatEvery log line may generate zero, one or two JSON objects. This depends
on the threshold set with the--bitsflag and what kind of transfer
it is. The generated objects will be sub-divided into a series of lists
of up to 100 objects each. That way, each send operation is of a
manageable size rather than sending one huge list.All numeric values are being converted to “actual” numeric types. All
floating point values are being rounded to three decimal places to avoid
small values being rendered in scientific notation. Timestamps logged in
floating point ms are being converted to integer epoch seconds.UDP logsThis is the most basic/common format.{
    "interval": 600,
    "values": {
        "duration": 0.0,
        "num_bits": 544,
        "num_packets": 1
    },
    "meta": {
        "src_ip": "198.129.77.102",
        "src_port": 123,
        "dst_ip": "198.124.252.130",
        "dst_port": 123,
        "protocol": "udp"
    },
    "start": 1455745857,
    "end": 1455745857
},TCP logsThe TCP logs are identical, but have additional values in thevaluesstanza:{
    "interval": 600,
    "values": {
        "duration": 191.796,
        "num_bits": 22120,
        "num_packets": 14,
        "tcp_rexmit_bytes": 0,
        "tcp_rexmit_pkts": 0,
        "tcp_rtt_avg": 4.442,
        "tcp_rtt_min": 0.007,
        "tcp_rtt_max": 39.094,
        "tcp_rtt_std": 10.648,
        "tcp_pkts_rto": 0,
        "tcp_pkts_fs": 0,
        "tcp_pkts_reor": 0,
        "tcp_pkts_dup": 0,
        "tcp_pkts_unk": 0,
        "tcp_pkts_fc": 0,
        "tcp_pkts_unrto": 0,
        "tcp_pkts_unfs": 0,
        "tcp_cwin_min": 16,
        "tcp_cwin_max": 960,
        "tcp_out_seq_pkts": 0,
        "tcp_window_scale": 7,
        "tcp_mss": 1460,
        "tcp_max_seg_size": 960,
        "tcp_min_seg_size": 16,
        "tcp_win_max": 960,
        "tcp_win_min": 16,
        "tcp_initial_cwin": 21
    },
    "meta": {
        "src_ip": "198.128.14.246",
        "src_port": 58635,
        "dst_ip": "198.129.77.102",
        "dst_port": 22,
        "protocol": "tcp"
    },
    "start": 1455698490,
    "end": 1455698490
},Utility programststat_cullCrawls a tstat directory to cull processed logs from disk. Tstat can
generate a lot of output, this cleans up processed files, while leaving
a configurable “buffer” of time to keep processed logs. The default is
too keep 2 days worth of processed logs on disk, rather than deleting
them right after they have been processed.This script checks themtimeof the.processedstate file in a
directory of processed logs. If it is older than the--ttltime to
live in hours (default: 48), the directory and logs are removed.Required args–directoryThis is the path to the “root” of the directory structure where tstat
writes the timestamped directories and logfiles. No default.Optional args–ttlThe time to live in hours. Set this if you don’t want to use the default
of 48 hours.–dry-runDo a dry run. Just log the directories that will be deleted but don’t
delete them.Extending tstat_send with additional transportsAdding additional transports is fairly straightforward.See doc/extending.md in the source distribution.