From bresnaha at mcs.anl.gov  Mon Jul  3 08:44:33 2006
From: bresnaha at mcs.anl.gov (John Bresnahan)
Date: Mon Jul  3 08:44:11 2006
Subject: [SC06xmon] internalized summarizer API
In-Reply-To: <4499DA4D.10604@lbl.gov>
References: <4499DA4D.10604@lbl.gov>
Message-ID: <44A93B61.1030500@mcs.anl.gov>

i am trying to get to this today.  sorry about the delay i have been 
switched onto an emergency deadline project and getting up to speed on 
it has taken all of my time.

a file seems to be missing from the distro:

| configure: error: cannot find sources (src/nl_log.h) in . or ..



Dan Gunter wrote:
> Hi,
> 
> This is mostly for John Bresnahan, but I figured it would be good to let 
> everyone know the progress. I finally got through adding code to 
> NetLogger so the "summarization" tool is integrated with the NetLogger C 
> API. Code examples can be found in c/src/perf/ioperf.c and its config 
> files perf-config.{server,client}, and  also c/src/perf/eventrate.c . It 
> just occurred to me that I may not have put the right automake magic to 
> get the configuration files into the distro., so I'm attaching them to 
> this email. Oh, joy, now you can all read the XML fun-ness.
> 
> Preliminary performance results, after some optimization, are good. 
> Summarizing events is about 50% faster than writing them to /dev/null, 
> and a lot more informative :-) But I am eager to integrate this code 
> with the XIO code to see if the performance results pan out in practice. 
> Based on what I've done so far, I am hopeful that NetLogger activity 
> overlaps enough with I/O that the only perturbation we see is from the 
> occasional disk I/O of writing out the summary records (which we may be 
> able to minimize by using a separate disk for logs).
> 
> Like before, the basic idea is that a configuration file (in XML -- "we 
> drank the kool-aid" as Martin would say) sets up everything about the 
> summarization, and you just need to point at that configuration file 
> when you start up NetLogger. The XIO instrumentation needs to be 
> rewritten to use the higher-level C API, but this should be a relief, 
> since it's pretty close to printf().
> 
> Clearly, we still need to push the summary records through Martin's 
> CDF-o-matic. But that part of the problem hasn't changed (for better or 
> worse), unless Martin knows something he's not telling us.
> 
> -Dan
> 
> I placed a tarball of NetLogger that contains the new code at:
> 
> http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz
> 
> This is "beta" software, so please expect crashes and the like the first 
> few times.
> Also, thanks to XML configuration, you need to have libxml2 on your 
> systems.
> 
> 
> ------------------------------------------------------------------------
> 
> <?xml version='1.0'?>
> <config xmlns='http://dsd.lbl.gov/NetLogger/ns/summ'>
>   <event-name-keys>EVNT</event-name-keys>
>   <function id='f1' type='tsumm' url='/tmp/ioperf-client-summ.log'>"
>     <param name='interval' >100000</param>
>     <param name='event' >csumm</param>
>     <param name='value' >value</param>
>   </function>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>sock</id-keys>
>     <event>read.start</event>
>     <event>read.end</event>
>   </event-group>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>sock</id-keys>
>     <event>write.start</event>
>     <event>write.end</event>
>   </event-group>
> </config>
> 
> 
> ------------------------------------------------------------------------
> 
> <?xml version='1.0'?>
> <config xmlns='http://dsd.lbl.gov/NetLogger/ns/summ'>
>   <event-name-keys>EVNT</event-name-keys>
>   <function id='f1' type='tsumm' url='/tmp/ioperf-summ-server.log'>"
>     <param name='interval' >100000</param>
>     <param name='event' >iosumm</param>
>     <param name='value' >nbytes</param>
>   </function>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>fd</id-keys>
>     <event>read.start</event>
>     <event>read.end</event>
>   </event-group>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>fd</id-keys>
>     <event>write.start</event>
>     <event>write.end</event>
>   </event-group>
> </config>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> SC06xmon mailing list
> SC06xmon@george.lbl.gov
> http://dsd.lbl.gov/mailman/listinfo/sc06xmon

From bresnaha at mcs.anl.gov  Mon Jul  3 08:44:33 2006
From: bresnaha at mcs.anl.gov (John Bresnahan)
Date: Mon, 03 Jul 2006 10:44:33 -0500
Subject: [SC06xmon] internalized summarizer API
In-Reply-To: <4499DA4D.10604@lbl.gov>
References: <4499DA4D.10604@lbl.gov>
Message-ID: <44A93B61.1030500@mcs.anl.gov>

i am trying to get to this today.  sorry about the delay i have been 
switched onto an emergency deadline project and getting up to speed on 
it has taken all of my time.

a file seems to be missing from the distro:

| configure: error: cannot find sources (src/nl_log.h) in . or ..



Dan Gunter wrote:
> Hi,
> 
> This is mostly for John Bresnahan, but I figured it would be good to let 
> everyone know the progress. I finally got through adding code to 
> NetLogger so the "summarization" tool is integrated with the NetLogger C 
> API. Code examples can be found in c/src/perf/ioperf.c and its config 
> files perf-config.{server,client}, and  also c/src/perf/eventrate.c . It 
> just occurred to me that I may not have put the right automake magic to 
> get the configuration files into the distro., so I'm attaching them to 
> this email. Oh, joy, now you can all read the XML fun-ness.
> 
> Preliminary performance results, after some optimization, are good. 
> Summarizing events is about 50% faster than writing them to /dev/null, 
> and a lot more informative :-) But I am eager to integrate this code 
> with the XIO code to see if the performance results pan out in practice. 
> Based on what I've done so far, I am hopeful that NetLogger activity 
> overlaps enough with I/O that the only perturbation we see is from the 
> occasional disk I/O of writing out the summary records (which we may be 
> able to minimize by using a separate disk for logs).
> 
> Like before, the basic idea is that a configuration file (in XML -- "we 
> drank the kool-aid" as Martin would say) sets up everything about the 
> summarization, and you just need to point at that configuration file 
> when you start up NetLogger. The XIO instrumentation needs to be 
> rewritten to use the higher-level C API, but this should be a relief, 
> since it's pretty close to printf().
> 
> Clearly, we still need to push the summary records through Martin's 
> CDF-o-matic. But that part of the problem hasn't changed (for better or 
> worse), unless Martin knows something he's not telling us.
> 
> -Dan
> 
> I placed a tarball of NetLogger that contains the new code at:
> 
> http://dsd.lbl.gov/NetLogger/download/dev/NetLogger-3.4.0.tar.gz
> 
> This is "beta" software, so please expect crashes and the like the first 
> few times.
> Also, thanks to XML configuration, you need to have libxml2 on your 
> systems.
> 
> 
> ------------------------------------------------------------------------
> 
> <?xml version='1.0'?>
> <config xmlns='http://dsd.lbl.gov/NetLogger/ns/summ'>
>   <event-name-keys>EVNT</event-name-keys>
>   <function id='f1' type='tsumm' url='/tmp/ioperf-client-summ.log'>"
>     <param name='interval' >100000</param>
>     <param name='event' >csumm</param>
>     <param name='value' >value</param>
>   </function>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>sock</id-keys>
>     <event>read.start</event>
>     <event>read.end</event>
>   </event-group>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>sock</id-keys>
>     <event>write.start</event>
>     <event>write.end</event>
>   </event-group>
> </config>
> 
> 
> ------------------------------------------------------------------------
> 
> <?xml version='1.0'?>
> <config xmlns='http://dsd.lbl.gov/NetLogger/ns/summ'>
>   <event-name-keys>EVNT</event-name-keys>
>   <function id='f1' type='tsumm' url='/tmp/ioperf-summ-server.log'>"
>     <param name='interval' >100000</param>
>     <param name='event' >iosumm</param>
>     <param name='value' >nbytes</param>
>   </function>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>fd</id-keys>
>     <event>read.start</event>
>     <event>read.end</event>
>   </event-group>
>   <event-group>
>     <consume/>
>     <function id='f1'/>
>     <id-keys>fd</id-keys>
>     <event>write.start</event>
>     <event>write.end</event>
>   </event-group>
> </config>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> SC06xmon mailing list
> SC06xmon at george.lbl.gov
> http://dsd.lbl.gov/mailman/listinfo/sc06xmon