Package: parse2plone
ContentsParse2PloneIntroductionWarningGetting startedJustificationInstallationExecutionExampleCustomizationTroubleshootingContactCreditsHistoryParse2PloneEasily import static websites into PloneIntroductionParse2Ploneis a “one off” HTML parser (in the form of a Buildout recipe that
creates a script for you) to easily get content from static HTML websites
(on the file system) into Plone.It is designed to satisfy only the most trivial of use cases (e.g. a simple static
site in /var/www/html); for more serious deployments, you might enjoyfunnelweb.WarningThis is aBuildoutrecipe for use withPlone; by itself it does nothing. If you
don’t know what Plone is, please see:http://plone.org. If you don’t know
what Buildout is, please see:http://www.buildout.org/.Getting startedBecause it drives the author nuts whenever he has to dig for a recipe’s options,
here are this recipe’s options:[import]
recipe = parse2plone

# core features
path = /Plone
illegal_chars = _ .
html_extensions = html
image_extensions = gif jpg jpeg png
file_extensions = mp3
target_tags = a div h1 h2 p

# additional bells & whistles
force = false
publish = false
collapse = false
replacetypes =
rename =
match =
paths =NoteThe parameters listed above are configured with their default values. Edit these
values if you would like to change the default behavior; they are (mostly)
self-explanatory. Now you can just cut and paste to get started or keep reading if
you would like to know more.JustificationWhy did you createParse2Plonewhencollective.transmogrifieralready
exists?Here are some reasons:BecauseParse2Ploneis aimed at lowering the bar for folks who don’t already
know (or want to know) what a “transmogrifier blueprint” is but are able to update
theirbuildout.cfgfile; runBuildout; then run a single command; all
without having to think too much.collective.transmogrify provides a framework for creating reusable pipes
(whose definitions are called blueprints).Parse2Ploneprovides
a single, non-reusable script.Transmogrifier and friends appear to be “developer’s tools”, while the author wantsParse2Ploneto be an “end user’s tool”.If you are a developer looking to create repeatable migrations, you probably want to be
usingcollective.transmogrifier. If you are an end user that just wants to see your
static website in Plone, then you might want to giveParse2Plonea try.There is also this user/contributor comment, which captures the author’s sentiment:Parse2Plone's release was very timely as I need either this or something very
similar - and while I've no doubt I could make transmogrify do the job, it's a
lot of work for a one-shot loading of legacy pages.

                                                -Derek Broughton, Pointer
                                                 Stop Consulting, Inc.That’s great, but why didn’t you try to improvecollective.transmogrifierand make it more user friendly?Here are some reasons:The author is a minimalist. One of the design aims ofParse2Plonewas to
have as few dependencies as possible. There are two “big” dependencies,zc.buildoutandlxml. Using Buildout was a design compromise, and
usinglxmlis pretty much a must if you want to “parse”. Oh, andBeautifulSoupwas thrown in after the author read this:http://codespeak.net/lxml/elementsoup.htmlparse2plonein addition to its primary role as a content importer, is
intended to serve as an educational tool; both for the author and consumer.
Python coding best practices, and demonstrating how to script tasks in Plone
with “bin/instance run” are the aim. Forking and pull requests are encouraged.The author had an itch to scratch; it will be nice for him to be able to say
“just go write a script” and then point to an example.All of that said, the author understands that reusability and conservation of
developer resources are important goals, especially for the Plone project.
But for better or worse, these were not the goals ofParse2Plone. However,
you can be sure that the lessons learned while developingParse2Plonewill
be applied outside of it, particularly with regard to the conservation of developer
resources within the Plone project.InstallationYou can installParse2Ploneby editing yourbuildout.cfgfile like
so. First add animportsection:[import]
recipe = parse2ploneThen add theimportsection to the list of parts:[buildout]
...
parts =
    ...
    importNow runbin/buildoutas usual.NoteThe section nameimportis arbitrary, you can call it whatever you
want. Just keep in mind that the section name corresponds directly to the
script name. In other words, whatever you name the section - that’s what
the script will be called.ExecutionNow you can runParse2Plonelike this:$ bin/plone run bin/import /path/to/filesNoteIn the example above and examples below,bin/plonerefers to aZope 2
instancescript created byplone.recipe.zope2instance.Yourbin/plonescript may be calledbin/instanceorbin/client, etc. instead.ExampleIf you have a site in /var/www/html that contains the following:/var/www/html/index.html
/var/www/html/about/index.htmlYou should run:$ bin/plone run bin/import /var/www/htmlAnd the following will be created:http://localhost:8080/Plone/index.htmlhttp://localhost:8080/Plone/about/index.htmlCustomizationModifying the default behavior ofparse2ploneis easy; just use the command
line options or add parameters to yourbuildout.cfgfile. Both approaches
allow customization of the exact same set of options, but the command line
arguments will trump any settings found in yourbuildout.cfgfile.Buildout optionsYou can configure the following parameters in yourbuildout.cfgfile in
theparse2plonerecipe section.OptionsParameterDefaultvalueDescriptionpath/PloneSpecify an alternate location in the
database for the import to occur.illegal_chars_ .Specify illegal characters.parse2plonewill ignore files that
contain these characters.html_extensionshtmlSpecify HTML file extensions.parse2plonewill import HTML files
with these extensionsimage_extensionspng, gif,
jpg, jpeg,Specify image file extensions.parse2plonewill import image files
with these extensions.file_extensionsmp3Specify image file extensions.parse2plonewill import files with
with these extensions.target_tagsa h1 h2 pSpecify target tags.parse2plonewill parse the contents of HTML tags
listed. If any tag is provided as an
XPath expression (any expression
begining with /) the matching elements
will first be extracted from the root
document.  Selections for the contents
of other tags will then be performed
only on the document subset.
If only XPath expressions are given,
then the entire subtree of the matched
elements are returned (including HTML)forcefalseForce create folders that do not exist.
For example, if you do
–path=/Plone/foo and foo does not
exist, you will get an error message.
Use –force to tellparse2ploneto
create it.publishfalsePublish newly created content.collapsefalse“collapse” content. (see
collapse_parts() in parse2plone.py)renameRename content. (see rename_parts()
in parse2plone.pyreplacetypesUse custom types. (see replace_types())matchMatch files. (see match_files())pathsSpecify a series of locations on the
filesystem, with corresponding
locations in the database for imports,
with syntax:
–paths=import_dirs:object_paths
(–path will be ignored)ExampleInstead of accepting the defaultparse2plonebehaviour, in yourbuildout.cfgfile you may specify the following:[import]
recipe = parse2plone
path = /Plone/foo
html_extensions = htm
image_extensions = png
target_tags = pThis will configureparse2ploneto (only) import contentfrom:Images ending in.pngHTML files ending in.htmText withinptagsto:A folder named/Plone/foo.Command line optionsThe followingparse2plonecommand line options are supported.Options'--path','-p'You can specify an alternate import path (‘/Plone’ by default)
with--pathor-p:$ bin/plone run bin/import /path/to/files --path=/Plone/foo'--html-extensions'You can specify HTML file extensions with the--html-extensionsoption:$ bin/plone run bin/import /path/to/files --html-extensions=htm'--image-extensions'You can specify image file extensions with the--image-extensionsoption:$ bin/plone run bin/import /path/to/files --image-extensions=png'--file-extensions'You can specify generic file extensions with the--file-extensionsoption:$ bin/plone run bin/import /path/to/files --file-extensions=pdf'--target-tags'You can specify the target tags to parse with the--target-tagsoption:$ bin/plone run bin/import /path/to/files --target-tags=p'--force'Force create folders that do not exist.'--publish'Publish newly created content.'--collapse'“collapse” content (see collapse_parts()).'--rename'Rename content (see rename_files()).'--replacetypes'Customize types (see replace_types() in parse2plone.py).'--match'Match files (see match_files()).'--paths'You can specify a series of import paths and corresponding object paths:$ bin/plone run bin/import --paths=sample:Plone/sample,sample2:Plone/sample2'--help'And lastly, you can always askparse2ploneto tell you about its available options with
the--helpor-hoption:$ bin/plone run bin/import -hExampleInstead of accepting the defaultparse2plonebehaviour, on the command line you
may specify the following:$ bin/plone run bin/import /path/to/files -p /Plone/foo --html-extensions=html \
    --image-extensions=png --target-tags=pThis will configureparse2ploneto (only) import contentfrom:Images ending in.pngHTML files ending in.htmText withinptagsto:A Plone site folder named/Plone/foo.TroubleshootingHere are some trouble-shooting comments/tips.Compiling lxmlParse2Plonerequireslxmlwhich in turn requireslibxml2andlibxslt. If you do not havelxmlinstalled “globally” (i.e. in your
system Python’s site-packages directory) then Buildout will try to install it
for you. At this pointlxmlwill look for the libxml2/libxslt2 development
libraries to build against, and if you don’t have them installed on your system
alreadyyour mileage may vary(i.e. Buildout will fail).Database accessBefore runningparse2plone, you must either stop your Plone site or
use ZEO. Otherwiseparse2plonewill not be able to access the
database.ContactQuestions, comments, or concerns? Please e-mail:aclark@aclark.net.CreditsDevelopment sponsored by Radio Free AsiaHistory0.9.9 (11/16/2010)Added a large number of tests; performed associated refactoring; 50% test coverage0.9.8 (11/12/2010)Add “paths” feature to allow multi-import dirs (on the
file system), and corresponding object paths (in Plone)
to be specified.0.9.7 (11/08/2010)Fix import errorAdd file handler to logger; saves output to a file called “parse2plone.log”0.9.6 (11/08/2010)Fixes to “match” featureCombine all modules into oneRemove a stray pdb (!)Add tests (we’re at 20% test coverage people!)Update docs0.9.5 (11/08/2010)Add match featureAdd more project justifications to the docs0.9.4 (11/06/2010)Removebin/importscript whenever recipe is uninstalled [aclark4life]Add support for XPath syntax in target_tags [derek]Add “typeswap” feature [aclark4life]Update docs [aclark4life]0.9.3 (11/04/2010)Add Plone 2.5 compatBug fixesBetter handling of file system path; better base dir calculation0.9.2 (11/03/2010)More doc fixes0.9.1 (11/03/2010)Doc fixes0.9.0 (11/03/2010)Fix regressions introduced (or unresolved as of) 0.8.2. Thanks Derek
Broughton for the bug report(s)Many fixes to convert_parameter_values() method which converts
recipe parameters to arguments passed to main()Fix “slugify” feature0.8.2 (11/02/2010)Add rename featureFix regressions introduced in 0.8.10.8.1 (10/29/2010)Refactor options/parameters functionality to universally support _SETTINGS dictAdd “slugify” featureDoc fixesAdd support to optionally publish content after creationAdd support for generic file import0.8 (10/27/2010)Support the importing of content to folders within the Plone site object0.7 (10/25/2010)Documentation fixes0.6 (10/25/2010)Support customization via recipe parameters and command line arguments0.5 (10/22/2010)Revert ‘Add Plone to install_requires’0.4 (10/22/2010)Add ‘Plone’ to install_requires0.3 (10/22/2010)Another setuptools fix0.2 (10/22/2010)Setuptools fix0.1 (10/21/2010)Initial release