Package: zope.kgs
===============================Zope 3 Controlled Package Index===============================This package has been developed to support the maintenance of a stable set ofZope project distributions. It manages the controlled packages configurationfile and supports the generation of buildout configuration files that can beused by developers.Another use of this package is to use it for testing new distributions againstthe index. Here is the workflow for testing a new package against stable set:1. Install the correct version of this package.(a) Download the version of this package that manages the stable set thatyou are interested in. Currently only the trunk exists, which managesthe Zope 3.4 release::$ svn co svn://svn.zope.org/repos/main/zope.release/trunk zope3.4$ cd zope3.4(b) Bootstrap the checkout::$ python ./bootstrap.py(c) Run buildout to create the scripts::$ ./bin/buildout(d) Run the ``buildout.cfg`` generation script to build a configurationfile that can be used for testing:$ ./bin/generate-buildout2. From the generated configuration file, you can now build a testingenvironment.(a) Enter the test directory and create a buildout:$ cd test$ python ../bootstrap.py$ ./bin/buildout(b) Run all the tests to verify that all tests are initially passing:$ ./bin/test -vpc13. Modify the ``buildout.cfg`` to look for your the new distribution to betested:(a) Comment out the "index" option. This needs to be done, so that the newpackage is going to be picked up.(b) Change the version number of the package of interest in the "versions"section.Alternative:(a) Check out the new distribution from SVN.(b) Add a "develop path/to/my/package" line in the "buildout" section of``buildout.cfg``.4. Run the tests, making sure that they all pass.5. Modify ``controlled-packages.cfg`` by adding the new version of the packageto the package's version list.6. Generate all files again and upload them:$ cd ..$ ./bin/generate-buildout$ ./bin/generate-versions$ ./bin/uploadOnce the files are uploaded, a crontab-job, running every minute, willdetect the changes in ``controlled-pages.cfg`` and will generate the newcontrolled package pages.Note: I think the process is still a tiny bit too long. I probably write ascript that makes testing a new version of a package easier, but let's seewhether this process is workable first.===============Known Good Sets===============This package provides a set of scripts and tools to manage Good-Known-Sets, orshort KGSs. A KGS is a set of package distributions that are known to workwell together. You can verify this, for example, by running all the tests ofall the packages at once.Let me show you how a typical controlled packages configuration file lookslike:>>> import tempfile>>> cfgFile = tempfile.mktemp('-cp.cfg')>>> open(cfgFile, 'w').write('''\... [DEFAULT]... tested = true...... [KGS]... name = zope-dev... version = 1.2.0... date = 2009-01-01... changelog = CHANGES.txt... announcement = ANNOUNCEMENT.txt... files =...     zope-dev-1.2.0.tgz...     zope-dev-1.2.0.zip...     zope-dev-1.2.0.exe...... [packageA]... versions = 1.0.0...            1.0.1...... [packageB]... versions = 1.2.3... test-extras = test...... [packageC]... # Do not test this package.... tested = false... versions = 4.3.1... ''')As you can see, this file uses an INI-style format. The "DEFAULT" section isspecial, as it will insert the specified options into all other sections asdefault. The "KGS" section specifies some global information about the KGS,such as the name of the KGS. Since this section references several externalfiles, we should quickly create those.>>> import os>>> dir = os.path.dirname(cfgFile)>>> open(os.path.join(dir, 'CHANGES.txt'), 'w').write('''\... =======... Changes... =======...... packageA... ========...... Version 1.0.0... -------------...... * Initial Release... ''')>>> open(os.path.join(dir, 'ANNOUNCEMENT.txt'), 'w').write('''\... =======================... zope-dev 1.2.0 Released... =======================...... The announcement text!... ''')>>> open(os.path.join(dir, 'zope-dev-1.2.0.tgz'), 'w').write('tgz')>>> open(os.path.join(dir, 'zope-dev-1.2.0.exe'), 'w').write('exe')All other sections refer to package names. Currently each package sectionsupports two options. The "versions" option lists all versions that are knownto work in the KGS. Those versions should *always* only be bug fixes to thefirst listed version. The second option, "tested", specifies whether thepackage should be part of the KGS test suite. By default, we want all packagesto be tested, but some packages require very specific test setups that cannotbe easily reproduced _[1], so we turn off those tests.You can also stack controlled package configurations on top of eachother. Base configurations can be specified using the `extends` option:>>> import tempfile>>> cfgFile2 = tempfile.mktemp('-cp.cfg')>>> open(cfgFile2, 'w').write('''\... [DEFAULT]... tested = true...... [KGS]... name = grok-dev... version = 0.1.0... extends = %s...... [packageA]... versions = 1.0.2...... [packageD]... versions = 2.2.3...            2.2.4... ''' %cfgFile)As you can see, you can completely override another package's versionspecification as well.Generating the configuration file and managing it is actually the hardpart. Let's now see what we can do with it... [1]: This is usually due to bugs in setuptools or buildout, such as PYCfiles not containing the correct reference to their PY file.Generate Versions-----------------One of the easiest scripts, is the version generation. This script willgenerate a "versions" section that is compatible with buildout.>>> versionsFile = tempfile.mktemp('-versions.cfg')>>> from zope.kgs import version>>> version.main((cfgFile, versionsFile))>>> print open(versionsFile, 'r').read()[versions]packageA = 1.0.1packageB = 1.2.3packageC = 4.3.1Let's now ensure that the versions also work for the extended configuration:>>> versionsFile2 = tempfile.mktemp('-versions.cfg')>>> version.main((cfgFile2, versionsFile2))>>> print open(versionsFile2, 'r').read()[versions]packageA = 1.0.2packageB = 1.2.3packageC = 4.3.1packageD = 2.2.4Generate Buildout-----------------In order to be able to test the KGS, you can also generate a full buildoutfile that will create and install a testrunner over all packages for you:>>> buildoutFile = tempfile.mktemp('-buildout.cfg')>>> from zope.kgs import buildout>>> buildout.main((cfgFile, buildoutFile))>>> print open(buildoutFile, 'r').read()[buildout]parts = testversions = versions<BLANKLINE>[test]recipe = zc.recipe.testrunnereggs = packageApackageB [test]<BLANKLINE>[versions]packageA = 1.0.1packageB = 1.2.3packageC = 4.3.1<BLANKLINE>Let's make sure that the buildout generation also honors the extensions:>>> buildoutFile2 = tempfile.mktemp('-buildout.cfg')>>> buildout.main((cfgFile2, buildoutFile2))>>> print open(buildoutFile2, 'r').read()[buildout]parts = testversions = versions<BLANKLINE>[test]recipe = zc.recipe.testrunnereggs = packageApackageB [test]packageD<BLANKLINE>[versions]packageA = 1.0.2packageB = 1.2.3packageC = 4.3.1packageD = 2.2.4<BLANKLINE>Flat Links Pages----------------We can also create a flat links page that can be used in the`dependency_links` argument in your `setup.py` file. Since this moduleaccesses the original PyPI to ask for the download locations and filenames, wehave to create a controlled packages configuration file that contains realpackages with real version numbers:>>> cfgFileReal = tempfile.mktemp('-cp.cfg')>>> open(cfgFileReal, 'w').write('''\... [DEFAULT]... tested = true...... [KGS]... name = zope-dev... version = 3.4.0b2...... [PIL]... versions = 1.1.6...... [zope.component]... versions = 3.4.0...... [zope.interface]... versions = 3.4.0...            3.4.1...... [z3c.formdemo]... versions = 1.1.0... ''')Let's now create the links page:>>> linksFile = tempfile.mktemp('-links.html')>>> from zope.kgs import link>>> link.main((cfgFileReal, linksFile))>>> print open(linksFile, 'r').read()<html><head><title>Links for the "zope-dev" KGS (version 3.4.0b2)</title></head><body><h1>Links for the "zope-dev" KGS (version 3.4.0b2)</h1><a href="http://pypi.python.org/packages/2.4/z/z3c.formdemo/z3c.formdemo-1.1.0-py2.4.egg#md5=9d605bd559ea33ac57ce11f5c80fa3d3">z3c.formdemo-1.1.0-py2.4.egg</a><br/><a href="http://pypi.python.org/packages/source/z/z3c.formdemo/z3c.formdemo-1.1.0.tar.gz#md5=f224a49cea737112284f74b859e3eed0">z3c.formdemo-1.1.0.tar.gz</a><br/><a href="http://pypi.python.org/packages/2.4/z/zope.component/zope.component-3.4.0-py2.4.egg#md5=c0763e94912e4a8ac1e321a068c916ba">zope.component-3.4.0-py2.4.egg</a><br/><a href="http://pypi.python.org/packages/source/z/zope.component/zope.component-3.4.0.tar.gz#md5=94afb57dfe605d7235ff562d1eaa3bed">zope.component-3.4.0.tar.gz</a><br/><a href="http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.4.0.tar.gz#md5=0be9fd80b7bb6bee520e56eba7d29c90">zope.interface-3.4.0.tar.gz</a><br/><a href="http://pypi.python.org/packages/2.4/z/zope.interface/zope.interface-3.4.0-py2.4-win32.egg#md5=3fa5e992271375eac597622d8e2fd5ec">zope.interface-3.4.0-py2.4-win32.egg</a><br/><a href="http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.4.1.tar.gz#md5=b085f4a774adab688e037ad32fbbf08e">zope.interface-3.4.1.tar.gz</a><br/></body></html>PPIX Support------------You can also use the KGS to limit the available packages in a package indexgenerated ``zc.mirrorcheeseshopslashsimple``. This script also uses PyPI tolook up distribution file, so wave to use the real configuration file again.Let's create the pages:>>> indexDir = tempfile.mkdtemp('-ppix')>>> from zope.kgs import ppix>>> ppix.main((cfgFileReal, indexDir))The index contains one directory per package. So let's have a look:>>> import os>>> sorted(os.listdir(indexDir))['PIL', 'z3c.formdemo', 'zope.component', 'zope.interface']Each directory contains a single "index.html" file with the download links:>>> pkgDir = os.path.join(indexDir, 'zope.component')>>> sorted(os.listdir(pkgDir))['index.html']>>> pkgIndex = os.path.join(pkgDir, 'index.html')>>> print open(pkgIndex, 'r').read()<html><head><title>Links for "zope.component"</title></head><body><h1>Links for "zope.component"</h1><a href="http://pypi.python.org/packages/2.4/z/zope.component/zope.component-3.4.0-py2.4.egg#md5=c0763e94912e4a8ac1e321a068c916ba">zope.component-3.4.0-py2.4.egg</a><br/><a href="http://pypi.python.org/packages/source/z/zope.component/zope.component-3.4.0.tar.gz#md5=94afb57dfe605d7235ff562d1eaa3bed">zope.component-3.4.0.tar.gz</a><br/></body></html>PIL is an interesting case, because it does not upload its distribution filesyet, at least not for version 1.1.6:>>> pkgIndex = os.path.join(indexDir, 'PIL', 'index.html')>>> print open(pkgIndex, 'r').read()<html><head><title>Links for PIL</title></head><body><h1>Links for PIL</h1><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5 home_page</a><br/><a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5 download_url</a><br/><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a2 home_page</a><br/><a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a2 download_url</a><br/><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a1 home_page</a><br/><a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a1 download_url</a><br/><a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.4 home_page</a><br/><a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.3 home_page</a><br/><a href='http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz' rel="download">1.1.3 download_url</a><br/><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.6 home_page</a><br/><a href='http://effbot.org/downloads/#Imaging' rel="download">1.1.6 download_url</a><br/></body></html>Optionally, you can also specify the `-i` option to generate an overview:>>> ppix.main(('-i', cfgFileReal, indexDir))>>> sorted(os.listdir(indexDir))['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']Let's now look at the file:>>> indexPage = os.path.join(indexDir, 'index.html')>>> print open(indexPage, 'r').read()<html><head><title>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</title></head><body><h1>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</h1><a href="PIL">PIL</a><br/><a href="z3c.formdemo">z3c.formdemo</a><br/><a href="zope.component">zope.component</a><br/><a href="zope.interface">zope.interface</a><br/></body></html>Allowing exisitng package pages to be overwritten and making the main indexpage an optional feature makes it possible to use this script for two usecases: (1) Merge the constraints into a PPIX index created by``zc.mirrorcheeseshopslashsimple``, and (2) create a standalone index whichonly provides the packages of the KGS.Getting the Latest Versions---------------------------When updating the KGS, it is often useful to know for which packages have newreleases.>>> from zope.kgs import latest>>> latest.main((cfgFileReal,))z3c.formdemo: 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.3.0b1, 1.4.0, ...However, it is often desired only to show new minor versions; in this case, wecan pass an option to exclude all versions that have a different majorversion:>>> latest.main(('-m', cfgFileReal))z3c.formdemo: 1.1.1, 1.1.2Sometimes you're only interested in changes that apply to a single package,and you won't want to wait for the script to query all of the others>>> latest.main(('-m', cfgFileReal, 'zope.app.server'))>>> latest.main(('-m', cfgFileReal, 'z3c.formdemo'))z3c.formdemo: 1.1.1, 1.1.2Extracting Change Information-----------------------------When releasing a version of the KGS, it is desirable to produce a list ofchanges since the last release. Changes are commonly compared to an olderversion.>>> cfgFileRealOrig = tempfile.mktemp('-cp.cfg')>>> open(cfgFileRealOrig, 'w').write('''\... [DEFAULT]... tested = true...... [KGS]... name = zope-dev... version = 3.4.0b1...... [PIL]... versions = 1.1.6...... [zope.component]... versions = 3.4.0...... [zope.interface]... versions = 3.4.0... ''')Let's now produce the changes:>>> from zope.kgs import change>>> change.main((cfgFileReal, cfgFileRealOrig))Processing ('PIL', '1.1.6')Processing ('z3c.formdemo', '1.1.0')Processing ('zope.component', '3.4.0')Processing ('zope.interface', '3.4.1')===PIL===<BLANKLINE>No changes or information not found.<BLANKLINE>============z3c.formdemo============<BLANKLINE>1.1.0 (unknown)---------------<BLANKLINE>- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used withnon-object data. Specificically, this small application demonstrates using aGadfly database using pure SQL calls without any ORM.<BLANKLINE>- Feature: New "Address Book" demo that demonstrates more complex use cases,such as subforms, composite widgets, and mappings/lists<BLANKLINE><BLANKLINE>==============zope.component==============<BLANKLINE>3.4.0 (2007-09-29)------------------<BLANKLINE>No further changes since 3.4.0a1.<BLANKLINE><BLANKLINE>==============zope.interface==============<BLANKLINE>3.4.1 (unknown)---------------<BLANKLINE>Fixed a setup bug that prevented installation from source on systemswithout setuptools.<BLANKLINE>3.4.0 (unknown)---------------<BLANKLINE>Final release for 3.4.0.<BLANKLINE><BLANKLINE>You can also create the changes without an original file, in which case onlythe versions listed in the current KGS are considered.>>> change.main((cfgFileReal,))Processing ('PIL', '1.1.6')Processing ('z3c.formdemo', '1.1.0')Processing ('zope.component', '3.4.0')Processing ('zope.interface', '3.4.1')===PIL===<BLANKLINE>No changes or information not found.<BLANKLINE>============z3c.formdemo============<BLANKLINE>1.1.0 (unknown)---------------<BLANKLINE>- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used withnon-object data. Specificically, this small application demonstrates using aGadfly database using pure SQL calls without any ORM.<BLANKLINE>- Feature: New "Address Book" demo that demonstrates more complex use cases,such as subforms, composite widgets, and mappings/lists<BLANKLINE><BLANKLINE>==============zope.component==============<BLANKLINE>3.4.0 (2007-09-29)------------------<BLANKLINE>No further changes since 3.4.0a1.<BLANKLINE><BLANKLINE>==============zope.interface==============<BLANKLINE>3.4.1 (unknown)---------------<BLANKLINE>Fixed a setup bug that prevented installation from source on systemswithout setuptools.<BLANKLINE>3.4.0 (unknown)---------------<BLANKLINE>Final release for 3.4.0.<BLANKLINE><BLANKLINE>The Site Generator------------------The easiest way to publish the KGS is via a directory published by a Webserver. Whenever a new `controlled-packages.cfg` file is uploaded, a script isrun that generates all the files. I usually set up a crontab job to dothis. The site generator script acts upon a directory, in which it assumes a`controlled-packages.cfg` file was placed:>>> siteDir = tempfile.mkdtemp()>>> cfgFileSite = os.path.join(siteDir, 'controlled-packages.cfg')>>> import shutil>>> shutil.copy(cfgFileReal, cfgFileSite)>>> from zope.kgs import site>>> site.main(['-s', siteDir])Let's have a look at the generated files:>>> from pprint import pprint>>> pprint(sorted(os.listdir(siteDir)))['3.4.0b2', 'index.html', 'intro.html', 'resources']>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2')))['ANNOUNCEMENT.html', 'CHANGES.html','buildout.cfg', 'controlled-packages.cfg', 'index', 'index.html','links.html', 'minimal', 'versions.cfg']>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2', 'minimal')))['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']If you try to generate the site again without adding the controlled packagesconfig file to the site directory again, it will simply return:>>> site.main(['-s', siteDir])Basic Parser API----------------The ``kgs.py`` module provides a simple class that parses the KGSconfiguration file and provides all data in an object-oriented manner.>>> from zope.kgs import kgsThe class is simply instnatiated using the path to the config file:>>> myKGS = kgs.KGS(cfgFile)>>> myKGS<KGS 'zope-dev'>The name, version and date of the KGS is available via:>>> myKGS.name'zope-dev'>>> myKGS.version'1.2.0'>>> myKGS.datedatetime.date(2009, 1, 1)When the changelog and/or announcement files are available, the KGS referencesthe absolute path:>>> myKGS.changelog'.../CHANGES.txt'>>> myKGS.announcement'.../ANNOUNCEMENT.txt'The same is true for other release-related files:>>> myKGS.files('.../zope-dev-1.2.0.tgz','.../zope-dev-1.2.0.exe')The packages are available under `packages`:>>> myKGS.packages[<Package 'packageA'>, <Package 'packageB'>, <Package 'packageC'>]Each package is also an object:>>> pkgA = myKGS.packages[0]>>> pkgA<Package 'packageA'>>>> pkgA.name'packageA'>>> pkgA.versions['1.0.0', '1.0.1']>>> pkgA.testedTrueAs we have seen in the scripts above, the KGS class also supports the`entends` option. Thus, let's load the KGS for the config file 2:>>> myKGS2 = kgs.KGS(cfgFile2)>>> myKGS2<KGS 'grok-dev'>>>> myKGS2.name'grok-dev'>>> myKGS2.packages[<Package 'packageA'>,<Package 'packageB'>,<Package 'packageC'>,<Package 'packageD'>]=======CHANGES=======1.2.0 (2009-07-24)------------------- Add ability to specify the extra-includes for running tests.1.1.0 (2009-02-01)------------------- Added '--no-links', '--no-index', and '--no-minimal-index' options to thesite generation sctipt to make it faster.- Generating list of latest versions does not fail anymore, if downloading andparsing of the index page fails.- Update links to PyPI web sites.1.0.1 (2009-01-29)------------------- Fix documentation in all scripts, fixing missing imports and incorrectwording.- The package should depend on `python-dateutl` and not `datetutil`, since thelatter is not available in PyPI anymore.1.0.0 (2009-01-29)------------------- Initial version as ``zope.kgs``.* A script that extracts the relevant part of the changelog of each packagein the KGS.* A script that lists all versions of a package released after the latestversion listed in the KGS.* A script that manages the generation of the entire KGS site.+ Generates generic and version-specific pages.+ Page generation is template-based for easy customization.* Generate `links.html` file which lists all controlled packages files.* Features copied from ``zope.release``:+ Parser for KGS configuration files.+ Generate `versions.cfg` and `buildout.cfg` script.* Features copied from ``zc.mirrorcheeseshopslashsimple``:+ Generate new index pages for the controlled packages.