Package: toolk0s
The Story of Toolbox====================Toolbox is fundamentally a document-oriented approach to resourceindexing.  A "tool" consists three mandatory string fields -- name,description, and URL -- that are generic to the large class of problemsof web resources, as well as classifiers, such as author, usage, type,etc. A tool may have an arbitrary number of classifier fields asneeded.  Each classifier consists of a set of values with which a toolis tagged. This gives toolbox the flexibility to fit a large number ofdata models, such as PYPI, DOAP, and others.Running Toolbox---------------You can download and run the toolbox software yourself:http://github.com/k0s/toolboxTo serve in baseline mode, install the software and run::paster serve paste.iniThis will serve the handlers and static content using the paste(http://pythonpaste.org) webserver using ``README.txt`` as the``/about`` page and serving the data in ``sample``.The dispatcher (``toolbox.dispatcher:Dispatcher``) is the central (WSGI)webapp that designates per-request to a number of handlers (from``handlers.py``).  The dispatcher has a few options:* about: path to a restructured text file to serve at ``/about``* model_type: name of the backend to use (memory_cache, file_cache, or couch)* template_dir: extra directory to look for templatesThese may be configured in the ``paste.ini`` file in the``[app:toolbox]`` section by prepending with the namespace``toolbox.``. It is advisable that you copy the example ``paste.ini``file for your own usage needs.  Additional ``toolbox.``-namespacedarguments will be passed to the model.  For instance, to specify thedirectory for the ``file_cache`` model, the provided ``paste.ini`` uses``toolbox.directory = %(here)s/sample``.Architecture------------Toolbox uses a fairly simple architecture with a single abstract datamodel allowing an arbitrary number of implementations to be constructed::Interfaces            Implementations+----+              +-+-----+|HTTP|              | |files|+----+---\  +-----+ | +-----+|-|model|-+-+-----++------+-/  +-----+ | |couch||script|            | +-----++------+            +-+------+| |memory|| +------++-+---+|...|+---+Toolbox was originally intended to use a directory of files, one per project,as the backend. These were originally intended to be HTML files as theabove model may be clearly mapped as HTML::<div class="project"><h1><a href="{{url}}">{{name}}</a></h1><p class="description">{{description}}</p>{{for field in fields}}<ul class="{{field}}">{{for value in values[field]}}<li>{{value}}</li>{{endfor}}{{endfor}}</div>This microformat approach allows not only easy editing of the HTMLdocuments, but the documents may be indepently served and displayedwithout the toolbox server-side.The HTML microformat was never implemented (though, since the modelbackend is pluggable, it easily could be). Instead, the originalimplementation used JSON blobs stored in one file per tool. Thisapproach loses the displayable aspect, though since JSON is a definedformat with several good tools for exploring and manipulating the dataperhaps this disavantage is offset.A couch backend was also written.+------------+-----------+------------+|Displayable?|File-based?|Concurrency?|+-----+------------+-----------+------------+|HTML |Yes         |Yes        |No          |+-----+------------+-----------+------------+|JSON |Not really  |Yes        |No          |+-----+------------+-----------+------------+|Couch|No          |No         |Yes?        |+-----+------------+-----------+------------+The concurrency issue with file-based documennt backends may beovercome by using locked files.  Ideally, this is accomplished at thefilesystem level.  If your filesystem does not promote thisfunctionality, it may be introduced programmatically.  A rough cartoonof a good implementation is as follows:1. A worker thread is spawned to write the data asynchronously. Thedata is sent to the worker thread.2. The worker checks for the presence of a lockfile (herein furtherdetailed). If the lockfile exists and is owned by an active process,the worker waits until said process is done with it. (For a morerobust implementation, the worker sends a request to write the file tosome controller.)3. The worker owns a lockfile based on its PID in some directoryparallel to the directory root under consideration (for example,``/tmp/toolbox/lock/${PID}-${filename}.lck``).4. The worker writes to the file.5. The worker removes the lockThe toolbox web service uses a dispatcher->handler framework.  Thehandlers are loosely pluggable (they are assigned in the dispatcher),but could (and probably should) be made completely pluggable.  Thatsaid, the toolbox web system features an integration of templates,static resources (javascript, css, images), and handlers, so truepluggability is further away than just supporting pluggable handlersin the dispatcher.Deployment, however, may be tailored as desired.  Any of the giventemplates may be overridden via passing a ``template_dir`` parameterwith a path to a directory that have templates of the appropriatenames as found in toolbox's ``templates`` directory.Likewise, the static files (css, js, etc.) are served using ``paste``'s``StaticURLParser`` out of toolbox's ``static`` directory. (Seetoolbox's ``factory.py``.) Notably this is *not* done using the WSGIapp itself.  Doing it with middleware allows the deployment to becustomizable by writing your own factory.  For example, instead ofusing the ``paste`` webserver and the included ``paste.ini``, youcould use nginx or apache and ``mod_wsgi`` with a factory fileinvoking ``Dispatcher`` with the desired arguments and serving thestatic files with an arbitrary static file server.It is common sense, if rarely followed, that deployment should besimple.  If you want to get toolbox running on your desktop and/or fortesting, you should be able to do this easily (see the ``INSTALL.sh``for a simple installation using ``bash``; you'll probably want toperform these steps by hand for any sort of real-world deployment).If you want a highly customized deployment, then this will requiremore expertise and manual setup.The template data and the JSON are closely tied together.  This has thedistinct advantage of avoiding data translation steps and avoidingcode duplication.Toolbox uses several light-footprint libraries:* webob for Request/Response handling: http://pythonpaste.org/webob/* tempita for (HTML) templates: http://pythonpaste.org/tempita/* whoosh for search.  This pure-python implementation of full-textsearch is relatively fast (for python) and should scale decently tothe target scale of toolbox (1000s or 10000s of tools). While not asfast as lucene, whoosh is easy to deploy and has a good API andpreserves toolbox as a deployable software product versus aninstance that requires the expert configuration, maintainence, andtuning of several disparate software products that is bothnon-automatable (cannot be installed with a script) andtime-consuming. http://packages.python.org/Whoosh/* jQuery: jQuery is the best JavaScript library and everyoneshould use it. http://jquery.com/* jeditable for AJAXy editing: http://www.appelsiini.net/projects/jeditable* jquery-token for autocomplete: http://loopj.com/jquery-tokeninput/* less for dynamic stylesheets: http://lesscss.org/User Interaction----------------A user will typically interact with Toolbox through the AJAX webinterface.  The server side returns relatively simple (HTML) markup,but structured in such a way that JavaScript may be utilized topromote rich interaction.  The simple HTML + complex JS manifestsseveral things:1. The document is a document. The tools HTML presented to the user (withthe current objectionable exception of the per-project Delete button)is a document form of the data. It can be clearly and easilytranslated to data (for e.g. import/export) or simply marked up using(e.g.) JS to add functionality. By keeping concerns seperate(presentation layer vs. interaction layer) a self-evident clarity ismaintained.2. Computation is shifted client-side. Often, an otherwise lightweightwebapp loses considerable performance rendering complex templates. Bykeeping the templates light-weight and doing control presentation andhandling in JS, high performance is preserved.What Toolbox Doesn't Do-----------------------* versioning: toolbox exposes editing towards a canonical document.It doesn't do versioning.  A model instance may do whateverversioning it desires, and since the models are pluggable, it wouldbe relatively painless to subclass e.g. the file-based model andhave a post-save hook which does an e.g. ``hg commit``. Customizedtemplates could be used to display this information.* authentication: the information presented by toolbox is freelyreadable and editable. This is by intention, as by going to a "wiki"model and presenting a easy to use, context-switching-free interfacecuration is encouraged (ignoring the possibly imaginary problem ofwiki-spam). Access-level auth could be implemented using WSGImiddleware (e.g. repoze.who or bitsyauth) or through a front end"webserver" integration layer such as Apache or nginx. Finer grainedcontrol of the presentation layer could be realized by using customtemplates.What Toolbox Would Like To Do-----------------------------Ultimately, toolbox should be as federated as possible.  The basicarchitecture of toolbox as a web service + supporting scripts makesthis feasible and more self-contained than most proposed federatedservices.  The basic federated model has proved, in practice,difficult to achieve through purely the (HTTP) client-server model, aswithout complete federation and adherence to protocol offline cronjobs should be utilized to pull external data sources. If a webserviceonly desires to talk to others of its own type and are willing to keepa queue of requests for when hosts are offline, entire HTTP federationmay be implemented with only a configuration-specified discoveryservice to find the nodes.Evolution---------Often, a piece software is presented as a state out of context (thatis minus the evolution which led it to be and led it to look furtherout towards beyond the horizon).  While this is an interesting specialeffect for an art project, software being communication thisis only conducive to software in the darkest of black-box approaches."Beers are like web frameworks: if they're not micro, you don't knowwhat you're talking about." - hipsterhackerFor sites that fit the architecture of a given framework, it may beadvisable to make use of them.  However, for most webapp/webservicecategories which have a finite scope and definitive intent, it isoften easier, more maintainable, and more legible to build a completeHTTP->WSGI->app architecture than to try to hammer a framework intofitting your problem or redefining the problem to fit the framework.This approach was used for toolbox.The GenshiView template, http://k0s.org/hg/GenshiView, was invoked togenerate a basic dispatcher->handler system.  The cruft was removed,leaving only the basic structure and the TempitaHandler since tempitais lightweight and it was envisioned that filesystem tempita templates(MakeItSo!) would be used elsewhere in the project.  The basichandlers (projects views, field-sorted view, new, etc.) were writtenand soon a usable interface was constructed.A ``sample`` directory was created to hold the JSON blobs. Becausethis was done early on, two goals were achieved:1. the software could be dogfooded immediately using actual applicabledata. This helped expose a number of issues concerning the data formatright away.2. There was a place to put tools before the project reached adeployable state (previously, a few had lived in a static state usinga rough sketch of the HTML microformat discussed above onk0s.org). Since the main point of toolbox is to record Mozilla tools,the wealth of references mentioned in passing could be put somewhere,instead of passed by and forgotten.  One wishes that they do not missthe train while purchasing a ticket.The original intent, when the file-based JSON blob approach was to bethe deployed backend, was to have two repositories: one for the codeand one for the JSON blobs.  When this approach was scrapped, thefile-based JSON blobs were relegated to the ``sample`` directory, withthe intent to be to import them into e.g. a couch database on actualdeployment (using an import script). The samples could then be usedfor testing.The model has a single "setter" function, ``def update``, used forboth creating and updating projects.  Due to this and due to the factthe model was ABC/pluggable from the beginning, a converter ``export``function could be trivially written at the ABC-level::def export(self, other):"""export the current model to another model instance"""for project in self.get():other.update(project)This with an accompanying CLI utility was used to migrate from JSONblob files in the ``sample`` directory to the couch instance.  Thisparticular methodology as applied to an unexpected problem (theunanticipated switch from JSON blobs to couch) is a good example ofthe power of using a problem to drive the software forward (in thiscase, creation of a universal export function and associated commandline utility). The alternative, a one-off manual data migration, wouldhave been just as time consuming, would not be repeatable, would nothave extended toolbox, and may have (like many one-offs do) infectedthe code base with associated semi-permanant vestiges.  In general,problems should be used to drive innovation.  This can only be done ifthe software is kept in a reasonably good state.  Otherwiseconsiderable (though probably worthwhile) refactoring should be doneprior to feature extension which will become cost-prohibitive intime-critical situations where a one-off is (more) likely to be employed.Use Cases---------The target use-case is software tools for Mozilla, or, more generally,a software index.  For this case, the default fields uses are given inthe paste.ini file: usage, author, type, language. More fields may beadded to the running instance in the future.However, the classifier classification can be used for a wide varietyof web-locatable resources.  A few examples:* songs: artist, album, genre, instruments* de.li.cio.us: type, media, author, siteResources---------* http://readthedocs.org/