Package: collective.recipe.filestorage
Detailed DocumentationIntroductionThis recipe adds additional filestorage and zodb_db stanzas to the zope.conf
and zeo.conf files generated by the plone.recipe.zope2instance and
plone.recipe.[zope2]zeoserver recipes.  It also creates the directories in which
the extra filestorage data files will be created.  This makes it easy to add
additional filestorages and mountpoints to a Zope 2 instance.Supported optionspartsA list of filestorage sub-parts to be generated, one per line.  (This is
different from buildout parts.)zeoThe name of a plone.recipe.zope2zeoserver or plone.recipe.zeoserver part to
which we want to add the extra filestorage.  Defaults to the first such part
in the buildout, if any.zopesA list of names of plone.recipe.zope2instance parts to which we want to add
the extra filestorage.  Defaults to all plone.recipe.zope2instance parts
connected to the associated zeoserver part, if any, or all
plone.recipe.zope2instance parts, if no ZEO is found.The following options affect the generated zope.conf and zeo.conf.  Each may be
specified for all filestorage subparts in the collective.recipe.filestorage
buildout part, or for one particular filestorage subpart by placing the option
in a new buildout part called ‘filestorage_subpart’, where subpart is the name
of the subpart listed in the ‘parts’ option of this recipe.  The name of the
subpart may be interpolated by using ‘%(fs_part_name)s’ in the option.locationThe location of the Data.fs file, relative to the buildout root directory.
Defaults to var/filestorage/%(fs_part_name)s/Data.fszodb-nameThe name of the ZODB.  Defaults to ‘%(fs_part_name)s’.zodb-cache-sizeSet the ZODB cache size, i.e. the number of objects which the ZODB cache
will try to hold.  Inherits from the associated zope part.  Defaults to
5000.zodb-mountpointSet the path to the mountpoint.  Defaults to ‘/%(fs_part_name)s’.zodb-container-classSet the class of the object being mounted.  Defaults to not being set.zeo-addressSet the port of the associated ZEO server.  Inherits from the associated
Zope and ZEO parts.  Defaults to 8100.zeo-client-cache-sizeSet the size of the ZEO client cache. Inherits from the associated Zope
part.  Defaults to ‘30MB’.zeo-storageSet the id of the ZEO storage. Defaults to ‘%(fs_part_name)s’.zeo-client-nameSet the name of the ZEO client. Defaults to ‘%(fs_part_name)s_zeostorage’.zeo-client-clientSet the persistent cache name that is used to construct the cache
filenames. Persistent cache files are disabled by default.blob-storageSet the directory to be used to store blobs for a standalone Zope instance
or a ZEO server. Optional. Required if you’re going to store blobs, though.
Recommended value: var/blobstorage-%(fs_part_name)szeo-blob-storageSet the directory to be used to store blobs for a ZEO client. Defaults to
using the same value asblob-storage.zeo-shared-blob-dirBoolean that should be ‘on’ if the blob dir is being shared by the ZEO
server and client. Defaults to ‘on’.Example usageLet’s create and run a minimal buildout that adds an extra filestorage:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')Our zope.conf should get the extra filestorage stanza automatically injected into it:>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs>
    cache-size 5000
    <filestorage >
      path .../var/filestorage/my-fs/my-fs.fs
    </filestorage>
    mount-point /my-fs
</zodb_db>
<BLANKLINE>The recipe will also create a directory for the new filestorage:>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
TrueLet’s make sure that the conf files will be regenerated whenever we make a change to a filestorage part,
even if the direct configuration for the zope/zeo parts hasn’t changed:>>> open('buildout.cfg', 'a').write("    my-fs-2\n")
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs-2' in open('parts/instance/etc/zope.conf').read()
TrueLet’s make sure that the filestorage directory is not clobbered even if the filestorage part is removed
from the buildout:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
TrueWe can override the defaults for a number of settings:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... blob-storage = var/blobstorage-%(fs_part_name)s
... zodb-name = %(fs_part_name)s_db
... zodb-cache-size = 1000
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... zodb-container-class = Products.ATContentTypes.content.folder.ATFolder
... parts =
...     my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs_db>
    cache-size 1000
    <blobstorage >
      blob-dir .../var/blobstorage-my-fs
      <filestorage >
        path .../var/filestorage/my-fs/Data.fs
      </filestorage>
    </blobstorage>
    mount-point /my-fs_mountpoint
    container-class Products.ATContentTypes.content.folder.ATFolder
</zodb_db>
<BLANKLINE>A setting can also be modified just for one particular filestorage, by creating a new part with
thefilestorage_prefix, like so:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
...
... [filestorage_my-fs]
... zodb-cache-size = 1000
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs>
    cache-size 1000
    <filestorage >
      path .../var/filestorage/my-fs/my-fs.fs
    </filestorage>
    mount-point /my-fs
</zodb_db>
<BLANKLINE>By default, the recipe adds the extra filestorages to each plone.recipe.zope2instance part in the buildout,
but you can tell it to only add it to certain parts:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance1
...     instance2
...
... [instance1]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [instance2]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = instance1
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/instance1/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/instance2/etc/zope.conf').read()
FalseExample Usage with ZEOHere is a minimal buildout including a ZEO server and two ZODB clients:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...     secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')This should result in the appropriate additions to zeo.conf and both zope.conf’s:>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
    <filestorage my-fs>
      path /sample-buildout/var/filestorage/my-fs/my-fs.fs
    </filestorage>
<BLANKLINE>

>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs>
 cache-size 5000
 <zeoclient>
   server 8100
   storage my-fs
   name my-fs_zeostorage
   var /sample-buildout/parts/primary/var
   cache-size 30MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs
</zodb_db>
<BLANKLINE>

>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs>
 cache-size 5000
 <zeoclient>
   server 8100
   storage my-fs
   name my-fs_zeostorage
   var /sample-buildout/parts/secondary/var
   cache-size 30MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs
</zodb_db>
<BLANKLINE>As above, we can override a number of the default parameters:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...     secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... blob-storage = var/blobstorage-%(fs_part_name)s
... zodb-cache-size = 1000
... zodb-name = %(fs_part_name)s_db
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... zeo-address = 8101
... zeo-client-cache-size = 50MB
... zeo-storage = %(fs_part_name)s_storage
... zeo-client-name = %(fs_part_name)s_zeostorage_name
... parts =
...     my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
    <blobstorage my-fs_storage>
      blob-dir /sample-buildout/var/blobstorage-my-fs
      <filestorage my-fs_storage>
        path /sample-buildout/var/filestorage/my-fs/Data.fs
      </filestorage>
    </blobstorage>
<BLANKLINE>
>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs_db>
 cache-size 1000
 <zeoclient>
   blob-dir /sample-buildout/var/blobstorage-my-fs
   shared-blob-dir on
   server 8101
   storage my-fs_storage
   name my-fs_zeostorage_name
   var /sample-buildout/parts/primary/var
   cache-size 50MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>
>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs_db>
 cache-size 1000
 <zeoclient>
   blob-dir /sample-buildout/var/blobstorage-my-fs
   shared-blob-dir on
   server 8101
   storage my-fs_storage
   name my-fs_zeostorage_name
   var /sample-buildout/parts/secondary/var
   cache-size 50MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>By default, the recipe adds the extra filestorages to the first
plone.recipe.zope2zeoserver part in the buildout, and will throw an error if
there is more than one part using this recipe.  However, you can override this
behavior by specifying a particular ZEO part.  In this case, the filestorages
will only be added to the Zopes using that ZEO, by default:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver1
...     zeoserver2
...     primary
...     secondary
...     other-zope
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8100
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8101
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [other-zope]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8100
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = zeoserver2
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/zeoserver2/etc/zeo.conf').read()
True
>>> 'my-fs' in open('parts/zeoserver1/etc/zeo.conf').read()
False
>>> 'my-fs' in open('parts/primary/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/other-zope/etc/zope.conf').read()
FalseError conditionsImportant note: You must place all parts using the
collective.recipe.filestorage recipe before the part for the instances and
zeoservers that you are adding the filestorage to.  Otherwise you’ll get an
error:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     instance
...     filestorage
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part must be listed before the following parts in ${buildout:parts}: instance
<BLANKLINE>Buildouts with multiple zeoserver parts will result in an
error if the desired ZEO to associate with is not explicitly specified:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver1
...     zeoserver2
...     primary
...     secondary
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part found multiple zeoserver parts; please specify which one to use with the "zeo" option.Specifying a nonexistent zeo should result in an error:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = foobar
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part specifies nonexistant zeo part "foobar".So should specifying a nonexistent zope part:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = foobar
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part expected but failed to find the following parts in ${buildout:parts}: foobarIf the Zope/ZEO parts are being automatically identified, let’s make sure
that we don’t accidentally “wake up” parts that would not otherwise be
included in the buildout:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
...
... [foobar]
... recipe = plone.recipe.distros
... urls =
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'foobar' in os.listdir(os.path.join(sample_buildout, 'parts'))
FalseMake sure that instance parts are found correctly in buildouts using ‘extends’
and the += or -= options:>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     extendstest
... ''' % globals())
>>> write('prod.cfg',
... '''
... [buildout]
... extends = buildout.cfg
... parts +=
...     foobar
...
... [foobar]
... recipe = plone.recipe.distros
... urls =
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q -c prod.cfg')
>>> 'extendstest' in open(os.path.join(instance, 'etc', 'zope.conf')).read()
TrueRunning the testsThe subversion checkout of collective.recipe.filestorage includes a buildout
which installs a script for running the tests.Just run:python2.4 bootstrap.py
bin/buildout
bin/testKnown issue: The tests run buildout in a separate process, so it’s currently
impossible to put a pdb breakpoint in the recipe and debug during the test.
If you need to do this, set up another buildout which uses collective.recipe.filestorage
as a development egg.Reporting bugs or asking questionsThere is a shared bugtracker and help desk on Launchpad:https://bugs.launchpad.net/collective.buildout/Change history0.6 (2010-08-13)Support use of the recipe when the ZEO server and client are in different
buildouts.
[ramon]0.5 (2010-02-03)Support plone.recipe.zeoserver in addition to plone.recipe.zope2zeoserver.
[davisagli]0.4 (2009-12-30)Add support for creating blob storages associated with the created
filestorage parts.
[davisagli]Fixed message indicating which parts the filestorage part must be installed
before.  This fixeshttps://bugs.launchpad.net/bugs/433877(Thanks, Jean
Jordaan)
[davisagli]0.3 (2009-03-19)Avoid causing the installation of all parts, even those not listed in the
[buildout] parts variable.  Thanks to Dylan Jay for the diagnosis and fix.
[davisagli]Fixed restructured text of long description (README.txt).
[maurits]0.2 (2008-04-30)Added zodb_container_class option.
[davisagli]0.1 (2008-04-22)Happy Earth Day!
[davisagli]Created recipe with ZopeSkel.  Initial implementation.
[davisagli]ContributorsDavid Glick [davisagli], AuthorThanks to:Dylan JayRamon Navarro Bosch