Package: iw.fss
ByIngeniweb.Table of contents1About FileSystemStorage2Copyright and license3Requirements4Upgrading4.1From FSS 2.5.x and older4.2From iw.fss 2.7.15Migrating between strategies6Installation6.1With buildout6.2From SVN reposo6.3Configuring your Plone sites7Storage strategies7.1Strategies overview7.2Flat storage strategy7.3Directory storage strategy7.4Site storage strategy 17.5Site storage strategy 28FSS main configuration9Configuration panel9.1FSS and RDF9.2FSS maintenance10Zope backups11Caveats and pittfalls12Developer’s corner12.1Using FSS in your content types12.2Monkey patching a third party content type12.2.1The Python way12.2.2The ZCML way12.3Storing Plone standard content types with FSS12.4Customize RDF info set12.4.1RDF script interface12.4.2Simple example12.4.3More complex example12.4.4Recommanded practices on custom RDF13Testing14Other documentation15Downloads16Subversion repository17Support and feedback18Credits1About FileSystemStorageFileSystemStorage (FSS) is an Archetypes storage for storing fields
raw values on the file system. This storage is used to avoid
unnecessary growth of the ZODB’s FileStorage (Data.fs) when using a
lot of large files.Please note that FSS is a Plone component forcontent types
developers. Do not expect anything more in your Plone site if you
don’t use Plone components that may use FSS, such asAttachmentField.2Copyright and licenseCopyright (c) 2005 - 2008IngeniwebSASThis software is subject to the provisions of the GNU General Public License,
Version 2.0 (GPL).  A copy of the GPL should accompany this distribution.
THIS SOFTWARE IS PROVIDED “AS IS” AND ANY AND ALL EXPRESS OR IMPLIED
WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
FOR A PARTICULAR PURPOSESee theLICENSEfile that comes with this product.3RequirementsPlone 3.0+ bundle4Upgrading4.1From FSS 2.5.x and olderThe way we configure the storage and backup paths as well as the
storage strategiesHAVE CHANGED IN DEPTH. There’s no automatic
migration from this version. Please create a configuration file that
clones your old configlet preferences as indicated inFSS main
configuration.4.2From iw.fss 2.7.1From this version and later version, iw.fss registers upgrade steps in
GenericSetup. When upgrading to a newer version of iw.fss, just have a
look at the “Upgrade” tab of the portal_setup tool in your Plone site.Warning, due to the removal of theportal_fsstool upgrading from
2.7.1 does not retain the RDF options (activated and specific
script). If you used RDF with 2.7.1, you need to redo the same RDF
setting in the FileSystemStorage config panel.5Migrating between strategiesUse thebin/strategymigrator.pyshell utility that ships with FSS. Get
more info on this utility using the–helpoption.Note that only flat -> directory and directory -> flat are available
today. Sponsorship is welcome - as indicated inSupport and
feedback- if you need other strategy migrations.6Installation6.1With buildoutThis example speaks of itself:[buildout]
...
eggs =
  ...
  iw.fss
  ...

[instance]
...
zcml =
  ...
  iw.fss
  iw.fss-meta
  ...You may also use theiw.recipe.fssrecipe egg to configure fully an
instance.6.2From SVN reposoInflate iw.fss package into your zope instance lib/python directory.
I’m sure it’s already done.Add$INSTANCE_HOME/etc/package-includes/iw.fss-meta.zcmlfile with
this line:<include package="iw.fss" file="meta.zcml"/>Add$INSTANCE_HOME/etc/package-includes/iw.fss-configure.zcmlfile
with this line:<include package="iw.fss" />6.3Configuring your Plone sitesRead carefullyStorage strategiesandFSS main
configurationbelow and configure your instance inplone-filesystemstorage.confaccordingly. You may use theiw.recipe.fssrecipe in yourbuildout.cfgto do this.Start your Zope instanceIn ZMI, go into your plone site and install the product using
portal_quickinstallerOptionally, tweak some - self explained - preferences in the
FileSystemStorage configuration panel of your Plone site.7Storage strategiesA storage strategy defines how field values will be stored in your
files system.7.1Strategies overviewFSS comes with 4 storage strategies. Each strategy requires two directories:The storage directory stores the values of relevant fields in files
according to the selected strategy.The backup directory that keeps the values of the fields of deleted
contents for “Undo” purpose.Only one storage strategy can be selected on a given Plone site,
but different Plone site in the same Zope instance may have distinct
storage strategies.You need the best performances: choose theFlat storage strategyorDirectory storage strategy.You need a storage that looks like your Plone site (in example to
publish the storage directory in a read only samba or NFS share):
choose theSite storage strategy 1or theSite storage strategy 2.If you care about CMFEdition support (means that one of the content
types using FSS is versioned with CMFEditions), you shouldnotuse
theSite storage strategy 1or theSite storage strategy 2.SeeFSS main configurationto configure the best suited strategy on
your Zope instance and Plone sites.7.2Flat storage strategyAll field values are stored in a flat structure. This strategy is the
default one.Filename of these stored values:<uid of content>_<field name>Filename of rdf file:<uid of content>_<field name>.rdfFilename of backup values:<uid of content>_<field name>.bakRdf files are not backed up. They are automatically generated.Example of storage:fssDirectory
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak7.3Directory storage strategyAll field values are stored in a directory structure. Sub directories
are defined on two level. First level of directory uses the 2
characters of content uid.  Second level of directory uses the 4
characters of content uid. Backup files are stored in a flat
structure.Filename of these stored values:<uid of content>_<field name>Filename of rdf file:<uid of content>_<field name>.rdfFilename of backup values:<uid of content>_<field name>.bakRDF files are not backed up. They are automatically generatedExample of storage:fssDirectory
|
|- f42
   |
   |- f42ad
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bakDepending in one hand on your file system performances facing a huge
amount of files in the same directory, and in the other hand on the
number of contents relying on FSS, you might choose this strategy or
theFlat storage strategy.7.4Site storage strategy 1All field values are stored in a directory structure mirroring
structure of PloneSite.  Backup files are stored in a flat structure.Filename of these stored values: Filename of field value or field
name if not definedFilename of rdf file:<field name>.rdfFilename of backup values:<uid of content>_<field name>.bakRdf files are not backed up. They are automatically generatedExample of storage:fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- image
         |  |
         |  |- moutain.jpg
         |  |
         |  |- image.rdf
         |
         |- file
            |
            |- diary.odt
            |
            |- file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak7.5Site storage strategy 2All field values are stored in a directory structure mirroring
structure of PloneSite. Backup files are stored in a flat structure.Filename of these stored values: Filename of field value or field
name if not definedFilename of rdf file:<field filename>.rdfFilename of backup values:<uid of content>_<field name>.bakRdf files are not backed up. They are automatically generatedExample of storage:fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- fss.cfg
         |
         |- moutain.jpg
         |
         |- mountain.jpg.rdf
         |
         |- diary.odt
         |
         |- diary.odt.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak8FSS main configurationFSS is mainly configured the ZConfig way. At startup, the
configuration file will be searched (in that order) in:$INSTANCE_HOME/etc/plone-filesystemstorage.conf/path/to/iw/fss/etc/plone-filesystemstorage.conf/path/to/iw/fss/etc/plone-filesystemstorage.conf.inA sample working configuration in provided in this last file. It
assumes you have $INSTANCE_HOME/var/fss_storage and
$INSTANCE_HOME/var/fss_backup directories, both being read/write
enabled to the user that runs the Zope process, unless Zope will raise
an error at startup.All configuration doc you need is inplone-filesystemstorage.conf.incomments.Inconsistent configuration features raise explicit error messages at
Zope startup.Note that we didn’t include this inzope.confin order to keep Zope
2.7 and Zope 2.8 compatibility.9Configuration panel9.1FSS and RDFAs mentioned above inStorage strategies, FSS can optionally store
RDF files in the back-end storage. These RDF files conform theDCMES-XML standardon XML expression of DublinCore elements.If you select theFlat storage strategy, the RDF files may be used
to build a files tree as close as possible of the Plone tree structure
from the storage back-end, with thebuild_fs_tree.pyutility:$ cd /path/to/iw/fss/bin
$ python build_fs_tree.py --help9.2FSS maintenanceThe “Maintenance” tab of FSS configuration panel shows some statistics
about the amount of files managed by FSS.In addition 2 additional buttons are provided:Update FSS files: Cleans up the back-end storage
directories. Files not referenced from a content object are removed.Update RDF files: Builds all RDF files, in the event you chose
to select the generetion of RDF files after having inserted relevant
contents.10Zope backupsOf course, you need to synch your Data.fs backups with the storage
paths backups, unless you may restore corrupted/incomplete sites.11Caveats and pittfallsNote that whatever’s the selected strategies,DO NEVER CHANGE ANY
FILE IN THE STORAGE PATHS UNLESS YOU KNOW WHAT YOU DO, otherwise
you’ll loose your contents, your job, your money, your friends, your
wife/husband and children.DO NEVER CHANGE THE STRATEGY FOR A PLONE SITE ONCE THIS SITE HAS
FIELDS STORED IN ITS FSS STORAGE PATHunless you ran the appropriate
strategy migrator - seeMigrating between strategiesotherwise
you’re loose your contents, your job, …Thought it is possible to share the same storage path and backup path
within various Plone sites, since you don’t mind of your data in
development sites, you should really configure FSS such each Plone
site has its own private storage path and backup path. In cas you
ignore this warning,DO NEVER CLICK ANY BUTTON IN THE MAINTENANCE
TAB OF THE FSS CONFIGURATION PANELotherwise you’ll loose your
contents, your job, …Zexp exportsdon’t embed the associated FSS storage. So don’t move
your Plone site within Zope instances using zexp exports unless you
move the storage and backup directories along with the zexp file.After changing your configuration (seeFSS main configuration),
always restart your Zope instance in foreground since configuration
errors are not reported when Zope is started as daemon or Windows
service.12Developer’s corner12.1Using FSS in your content typesWe assume that creating AT based content types is a familiar activity.When creating the schema of a content type, you usually don’t mention
the storage of your fields since AT provides a default ZODB
storage. You can choose another storage, either another one that ships
in the Archetypes bundle, or a third party storage like FSS.Here is a small example that doesn’t need much comments:# Usual Zope/CMF/Plone/Archetypes imports
...
from iw.fss.FileSystemStorage import FileSystemStorage
...
my_schema = Schema((
    FileField('file',
              ...
              storage=FileSystemStorage(),
              widget=FileWidget(...)
              ),
    ...
    ))
 ...You may have a look atexamples/FSSItem.pyto see a demo content
type using FSS. If you want to play with this content type:$ $INSTANCE_HOME/bin/zopectl stop
$ export FSS_INSTALL_EXAMPLE_TYPES=1
$ $INSTANCE_HOME/bin/zopectl start12.2Monkey patching a third party content type12.2.1The Python wayThis works with any Zope version listed in theRequirements. If you
run a Zope 2.9 or later instance, you should preferThe ZCML way.This simple example shows how to plug the ATFile standard type from
any custom product. Let’s say we’re in the__init__.pyof your
custom product:...
from iw.fss import zcml as fss_utils
from Products.ATContentTypes import atct
...
fss_utils.patchATType(atct.ATFile, ['file'])
...12.2.2The ZCML wayThis works only with Zope 2.9 and up, and is the recommanded way to
wire your AT based content types with FSS since this way hides the API
details that may be deprecated in the future.You just need to add the “fss” namespace support and some directives
in your “configure.zcml” like this to have the same effect as in the
example ofThe Python wayabove:<configure
  ...
  xmlns:zcml="http://namespaces.zope.org/zcml"
  xmlns:fss="http://namespaces.ingeniweb.com/filesystemstorage"
  ...>
  ...
  <fss:typeWithFSS
    zcml:condition="installed iw.fss"
    class="Products.ATContentTypes.atct.ATFile"
    fields="file" />
  ...
</configure>The (required) attributes of this directive are:class: the dotted name of the AT based content type class.fields: one or more (space separated) field names to wire with FSS.12.3Storing Plone standard content types with FSSInclude this ZCML element either in your instancepackage-includesdirectory or in any personalconfigure.zcmlin your instance
packages:<include package="iw.fss" file="atct.zcml" />This ZCML file wires FSS with the relevant content types that ship with Plone bundle:FileImageNews Item12.4Customize RDF info setYou can add data to the RDF info set via a FSS dedicated - and
optional - hook.Open the FSS configuration panel and give the name of your personal
RDF data script, let’s call itfss_custom_rdffor this example. This
script will add or change the RDF default data set.12.4.1RDF script interfaceThe script may be in any Plone layer likecustomor any other in the
skins path.Such script is expected to have 4 parameters:name: the name of the fieldinstance: the content object being processedproperties: FSS field properties mapping, depending on the type of
field associated to FSS storage.default_rdf: a mapping structure of the RDF data, including the
namespaces, and the elements.Such script is expected to return an updateddefault_rdfmapping
structure.Please read the source of classesFSSInfo,FSSFileInfoandFSSImageInfofrom moduleFileSystemStorage.py,RDFWriterfrom
modulerdf.pyfor detailed information on formats of expected data.12.4.2Simple exampleWe need to add the date of RDF file generation in an element like
this:<fss:rdf_date>2007-05-13 13:21:00</fss:rdf_date>Here’s the script:## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
"""We add the date of RDF file generation"""

rdf_args = default_rdf.copy()

# Add the date of RDF generation
new_props = []
new_props.extend(rdf_args['field_props'])
new_props.append({'id': 'fss:rdf_date', 'value': DateTime().ISO()})
rdf_args['field_props'] = new_props

return rdf_args12.4.3More complex exampleIn this more complex (and stupid - don’t do this ;) example, we remove
the elements fromfssnamespace and simulate the standard behaviour
for other elements (dcandrdfelements).Here’s the code:## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
from Products.CMFCore.utils import getToolByName

rdf_args = {}

# Set RDF charset
ptool = getToolByName(instance, 'portal_properties')
rdf_args['charset'] = ptool.site_properties.default_charset

# Set RDF namespaces
rdf_args['namespaces'] = (
    {'id': 'xmlns:rdf', 'value': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'},
    {'id': 'xmlns:dc', 'value': 'http://purl.org/dc/elements/1.1/'},
    {'id': 'xmlns:fss', 'value': 'http://namespace.ingeniweb.com/fss'},)

# Set field url
utool = getToolByName(instance, 'portal_url')
portal_path = utool.getPortalObject().getPhysicalPath()
portal_path_len = len(portal_path)
rel_path = '/'.join(instance.getPhysicalPath()[portal_path_len:])
fss_path = '%s/%s' % (rel_path, name)
rdf_args['field_url'] = fss_path

# Set RDF properties
props = (
   {'id': 'dc:title', 'value': instance.title_or_id()},
   {'id': 'dc:description', 'value': instance.Description()},
   {'id': 'dc:language', 'value': instance.Language()},
   {'id': 'dc:creator', 'value': instance.Creator()},
   {'id': 'dc:date', 'value': instance.modified()},
   {'id': 'dc:format', 'value': properties.get('mimetype', 'text/plain')},
   )
rdf_args['field_props'] = props

# Returns the RDF structure
return rdf_args12.4.4Recommanded practices on custom RDFDo not change the existing elements underfssnamespace unless thebin/build_fs_tree.pyutility won’t work. In other terms, do not use
the script of theMore complex example.Adding or changing elements to thedcandrdfnamespaces may screw up
external utilities based on canonicalDCMES-XML standard.Consider adding your own namespace for your custom extra elements.13TestingPlease read./tests/README.txt.14Other documentationSee./doc/*and./locales/README.txt15DownloadsYou may find newer stable versions of FSS and pointers to related
informations (tracker, doc, …) fromhttp://pypi.python.org/pypi/iw.fss16Subversion repositoryStay in tune with the freshest (maybe unstable) versions or participate to
the FileSystemStorage evolutions:https://svn.plone.org/svn/collective/iw.fss17Support and feedbackPlease read all the documentation that comes with this product before
asking for support, unless you might get a RTFM reply ;)Localisation issues - other than french - should be reported to the
relevant translators (seeCreditsbelow).Report bugs using the tracker (theTrackerlink fromhttp://plone.org/iw.fss). Please provide in your
bug report:Your configuration (Operating system+Zope+Plone+Products/versions).The storage strategy in use.The full traceback if available.One or more scenario that triggers the bug.Note that we do not support bug reports on Subversion trunk or
branches checkouts.Mail to Ingeniweb supportin English or
French to ask for specific support.Donations are welcome for new features requests18CreditsMain developer:Cyrille LebeaupinZConfig and ZCML support:Gilles LenfantHTTP “Range” support:Youenn BoussardParis 2008 sprint additional contributors:
-Mehdi Benammar-Romain GriffithsSeelocales/README.txtabout localization credits.