Package: collective.stats
IntroductionThis isEnfold Systems’low level ZODB stats tool[1].InstallationInside PloneTo install in Plone, addcollective.statsto yourplone.recipe.zope2instancesection’s eggs parameter e.g.:[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    …
    collective.statsPlone 3.3 usage requires appending [oldzope] tocollective.stats:[instance]
recipe = plone.recipe.zope2instance
eggs =
    Plone
    …
    collective.stats [oldzope]Plone 3.3 on Windows requires a Python 2.4 compilation of thepsutilpackage which can be found athttp://dist.enfoldsystems.com/simpleRun buildout and run Plone in the foreground, and you will see output like this:2011-09-22 22:25:30 INFO Zope Ready to handle requests
2011-09-22 22:25:50 INFO collective.stats | 0.0021 0.0014 0.0018 0.0004 0.0000 0000 0000 0000 | GET:/favicon.ico | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 116708 - 116744
2011-09-22 22:25:55 INFO collective.stats | 0.1783 0.0021 0.1779 0.0004 0.0000 0000 0000 0000 | GET:/manage_main | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 116756 - 116948
2011-09-22 22:25:55 INFO collective.stats | 0.0020 0.0013 0.0017 0.0004 0.0000 0000 0000 0000 | GET:/misc_/OFSP/dtmlmethod.gif | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 117076 - 117076
2011-09-22 22:25:55 INFO collective.stats | 0.0018 0.0012 0.0016 0.0004 0.0000 0000 0000 0000 | GET:/misc_/TemporaryFolder/tempfolder.gif | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 117076 - 117076
2011-09-22 22:25:55 INFO collective.stats | 0.0021 0.0013 0.0018 0.0004 0.0000 0000 0000 0000 | GET:/misc_/SiteAccess/VirtualHostMonster.gif | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 117076 - 117076
2011-09-22 22:25:58 INFO collective.stats | 0.3960 0.0015 0.3957 0.0004 0.0000 0000 0000 0000 | GET:/@@plone-addsite | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 117076 - 126352
2011-09-22 22:25:58 INFO collective.stats | 0.0020 0.0014 0.0017 0.0004 0.0000 0000 0000 0000 | GET:/++resource++plone-admin-ui.css | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 126352 - 126356
2011-09-22 22:25:58 INFO collective.stats | 0.0014 0.0009 0.0012 0.0004 0.0000 0000 0000 0000 | GET:/++resource++plone-logo.png | t: 0.0000, t_c: 0.0000, t_nc: 0.0000 | RSS: 126356 - 126360If you want to use this product in a production environment, and you don’t want your logs to fill up, you can remove logging by providing an environment variableCOLLECTIVE_STATS_DISABLE_LOG. The only value that will causecollective.statsnot to log, is the number 1:environment-vars =
    COLLECTIVE_STATS_DISABLE_LOG 1You still get theX-Statsresponse header.Outside PloneTo use it outside of Plone, after configuringcollective.statsto be used inside Plone (as described above) configure azc.recipe.eggsection in your buildout like so:[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
scripts = collective-statsRun buildout, and this will create a script calledcollective-statsyou can use to parse Plone logs and produce a.csvfile:$ bin/collective-stats var/log/instance.log
…
1.3170 0.0196 1.3139 0.0004 0.0000 0000 0000 0003 | GET:/Plone
0.0283 0.0274 0.0278 0.0004 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/member.css
0.0153 0.0147 0.0150 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/forms.css
0.0176 0.0167 0.0171 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/ploneKss.css
0.0704 0.0694 0.0699 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/++resource++plone.app.discussion.stylesheets/discussion.css
0.0096 0.0090 0.0093 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/print.css
0.0067 0.0061 0.0064 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/mobile.css
0.0920 0.0709 0.0915 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/++resource++tinymce.stylesheets/tinymce.css
0.0319 0.0313 0.0316 0.0003 0.0034 0004 0000 0000 | GET:/Plone/portal_css/Sunburst Theme/ploneCustom.css
0.0078 0.0069 0.0075 0.0003 0.0000 0000 0000 0000 | GET:/Plone/portal_kss/Sunburst Theme/plone.kssCheck the current directory and you should see astats.csvfile:$ ls -1
…
lib/
parts/
setup.py
src/
stats.csv
var/Take a peek at the top of that file to see the column headers:$ head stats.csv
url,time,t traverse,t commit,t transchain,setstate,total,total cached,modified,rss before,rss after
GET:/favicon.ico,0.0021,0.0014,0.0018,0.0000,0000,0000,0000,116708,116744
GET:/manage_main,0.1783,0.0021,0.1779,0.0000,0000,0000,0000,116756,116948
GET:/manage_page_style.css,0.0110,0.0014,0.0108,0.0000,0000,0000,0000,116948,117012
GET:/p_/ltab,0.0023,0.0008,0.0021,0.0000,0000,0000,0000,117020,117032
GET:/p_/sp,0.0100,0.0054,0.0099,0.0039,0018,0004,0000,116992,117032
GET:/p_/rtab,0.0020,0.0012,0.0018,0.0000,0000,0000,0000,117032,117032
GET:/misc_/OFSP/Folder_icon.gif,0.0018,0.0012,0.0016,0.0000,0000,0000,0000,117032,117032
GET:/p_/ControlPanel_icon,0.0017,0.0009,0.0015,0.0000,0000,0000,0000,117036,117036
GET:/misc_/OFSP/UserFolder_icon.gif,0.0017,0.0012,0.0015,0.0000,0000,0000,0000,117036,117036DocumentationColumn header details:HeaderDetailtimeTotal time inside publishert traverseThis is time when zope publisher gets publishable objectt commitTime on transaction.commit()t transchainTime in plone.transformchain.applyTransformsetstateTotal time inside Connection.setstatetotalTotal zodb object loadstotal cachedTotal loads from cachemodifiedTotal modified objectsrss beforeRAM usage before requestrss afterRAM usage after requestAn ExampleIf you enable collective.stats to emit stats in response headers you will see a response line such as:X-Stats:4.5556 0.0232 1.2539 0.6334 9266 1244 0000Deciphering:4.555 - (time) is total time in Zope Publisher

0.023 - (t traverse) is after traverse time (callable object inside Publisher)
        time from BEGINNING of request to after TRAVERSE time.

1.253 - (t commit) is before commit() (we have a complete RESPONSE object)
        time from BEGINGING of request to before COMMIT

0.6334 - (setstate) total time in __setstate__ (time of ZODB spent unghostifying # of LOAD objects)

9266 - (total) total number of LOADS

1244 - (total cached) total number of HOT LOADS (cache hits in ZODB)

0000 - (modified) total number of MODIFIED objects.Summaryt_time - t_commit = total time to commit() (time executed in IPubBeforeCommit)
If you are using plone.app.caching or plone.app.theming both of which use
commit events; so depending how collective.stats gets registered - its possible
those are not being captured.In this example 3.3 seconds is “lost” in commit.  In this particular case
it was due to unoptimized plone.app.theming / diazo rules file.Enjoy![1]collective.statshas been donated to the Plone collective by Enfold Systems under a BSD-like license (ZPL 2.1).Changelog1.1.0 (2022-08-25)Add support for Python 3
[mamico]1.0.2 (2015-12-01)Provide a way to still have the functionality, without logging
[frapell]1.0.1 (2015-06-23)FixesAttributeErrorwiththread._local, adapt to newpsutilAPI.
Ref:http://stackoverflow.com/questions/20640679[thet]1.0.0 (2012-03-29)Add stats info for plone.transformchain applyTransform
[fafhrd91]Add stats info to failure requests.
[fafhrd91]0.9.1 (2011-12-01)Add output header detail to README
[aclark]Fix typo
[jean]0.9.0 (2011-09-22)Fix docs
[aclark]0.8 (2011-09-22)Fix docs
[aclark]0.7 (2011-09-22)Initial import from enfold.stats 0.7 tarball
[aclark]