# Presentation: 677940

## Dan Gunter
14 May 2004

- Dan Gunter
- 14 May 2004

- NetLogger Lite

## What Is NetLogger Lite?

- A simple, easy-to-use debugging, instrumentation, and logging toolkit
- Based on the NetLogger Toolkit

## Motivation

- NetLogger has grown:
- C API: 50,000 lines
- All APIs require the C API!
- Overkill for simple logging
- Makes it hard to incorporate into other packages

## Enter NetLogger “Lite”

- Single .c/.h for the C API
- Separate Python, Java API
- Use of C API is optional
- Much smaller, simpler
- Enhanced data model
- New data format (no more
- binary format (whew!))

## Rest of this talk

- Event data model
- Event format
- C API
- Dynamic log levels
- Python API (v0.1)
- Discussion:
- Feedback on above(?)
- DSD Projects that could benefit?

## Data Model Prologue

- Why bother with a data model?
- Makes it easier to write *generic* consumers of the data for analysis, visualization
- Relational databases need as much structure as they can get
- Elegance: if we can describe a general principle, we should!

## Event Data Model

- Logged event fields

- Target-dependent

- Timestamp:time

- Name:string

- Log level:enum

- Standard target fields

- User-defined:{string,
- number, time}

| Target | Field(s) |
| --- | --- |
| App | Host |
| Debug | Host, File, Line, Function |
| Node | Host, Val |
| Path | Src, Dst, Val |
| Other | -- |

## Event Record Format

**Event Record Format**

**t DATE: 2004-04-15T21:36:01.425059**

- Type code (_t_ime, _s_tring, _i_nt, _l_ong, or _d_ouble)

- Single space

- Field name

- Colon and space

- Field value

- Newline (\n)

*field*

*...*

*field*

**%%**

- File or
- stream

## Sample event records

- t DATE: 2004-04-15T21:36:01.422069
- s LVL: Debug
- s HOST: 131.243.2.143
- s TGT: dbg
- s EVNT: loop.start
- s FILE: ../../samples/sample_logger.c
- i LINE: 41
- s FN: main
- i start: 0
- i stop: 10
- %%

- t DATE: 2004-04-15T21:36:01.425059
- s LVL: Info
- s HOST: 131.243.2.143
- s TGT: node
- s EVNT: cpu.utilization.user
- i VAL: 50
- s UNITS: %
- %%
- t DATE: 2004-04-15T21:36:01.425059
- s LVL: Info
- s SRC: 123.45.67.8
- s DST: 87.65.43.21
- s TGT: path
- s EVNT: delay.roundtrip
- d VAL: 93.4
- %%

## NL-Lite C API Prologue

- There are actually multiple C APIs
- Format an “object” into a string
- Format a stream of “objects”
- Format an event belonging to a “module” to a stream using write() with variable arguments
- Format a module’s event at a given logging level for a given target type

## NL-Lite C API

- Create logger:
- NL_logger_module()
- Write messages:
-  NL_write() & friends
-  Cleanup:
- NL_logger_del()

- Note: The entire C ApI is now in one .c and one .h file
- Less than 3,000 lines of code

## NL_logger_module overview

- Creates a new (global) module
- Each time you write an event, you write it as part of a logger “module”
- An event’s module determines its..
- Log destination (URL)
- Logging level
- Target type

## NL_logger_module syntax

**NL_result_t NL_logger_module(**
**	****const char *  module,**
**	****const char *  url,**
**	****NL_level_t    level,**
**	****NL_tgttype_t  ttype,**
**	****... );**

** **

- Target types app, debug, and node can pass the host IP address, or “” for automatic lookup (by reverse DNS of hostname)

## NL_write

**NL write(level, module, event, fmt, args...);**

- Log level:
**NL_LVL_{NOLOG, FATAL, ERROR, WARN, INFO, DEBUG, DEBUG1, DEBUG2, DEBUG3}**
- Syntactic sugar for writing these log levels directly:
**NL_fatal(module, etc.);**

- Module name

- Event name

**“****name=<type-code>”** pairs, separated by spaces

- Values for each pair in *fmt*

## Example

- #**include “nl_log.h”**
**main() { **
**  ****int i;**
**	****NL_logger_module(“foo",NULL, NL_LVL_DEBUG, ****	****	****NL_TYPE_DBG, "");**
**  ****for ( i=0; i < 10; i++ ) {**
**	****	****NL_debug1(“foo","doit.start","index=i", i);**
**	****	****/* doit(); */**
**	****	****NL_debug1(“foo","doit.end","index=i", i);**
**	****}**
**}**

## Dynamic Log Levels

- Changing logging on the fly is cool
- NL-Lite has a really simple way of doing this:
**NLCONFIGLEVEL=/tmp/level myprog &**
**echo 3 > /tmp/level**
- You can even do per-module levels:
**NLCONFIGLEVEL=/tmp/level myprog &**
**echo “5 foo\n3” > /tmp/level**

## Python API

- Two separate APIs
- Homegrown
- Compatible with Python *logging* API
- Both can work with/without the C library
- Right now, the homegrown API is much faster than the Python logging one
- I’m working on this!
- Can’t say for sure the Python API is “done”

## Python logging API

**log = ****nllogging****.Logger('default')**
**handler = logging.StreamHandler(strm=sys.stderr)**
**handler.setFormatter(****nllogging****.Formatter())**
**log.addHandler(handler)**
**log.setLevel(logging.DEBUG)**
**log.info("loop.start")**
**for i in range(0,10):**
**    ****log.log(DEBUG,"loop.iter", ("index",i) )**
**log.info("loop.end")**

- Instead of a string, can also add any number of (“name”,value) pairs

- Standard Python
- Logging call

## Java API

- Still in the early stages. Just formats a string:
**NL nl = new NL();**
**Formatter fmt = ****nl****.getFormatter();**
**for ( int i=0; i < 10; i++ ) {**
**	****log.info(nl.setEvent(“foo").add("index",i).toString());**
**  ****log.info(“Non-netlogger message\n");**
**}**

## Compatibility & Tools

- There is a (Python) netlogd that can read in the “nllite” format and spit out the “ULM” (ASCII) one
- Also a fast, C program to convert “ULM” logs to the “nllite”, or vice-versa
- Some new gnuplot-producing scripts

## DSD Projects?

- C is the most mature API right now
- NLV  done!
- Akenti?
- Python is “close” but still needs some requirements feedback and the $6M-man treatment
- {py,swig}Globus?
- Jabber code
- And of course there’s Java, which basically I need to finish..
- Karlo no longer believes I’ll *ever* finish it

## URL

- http://www-didc.lbl.gov/nllite/