Added a bunch of skip_suffixes and one skip_string.
.
Don't import sitemaps; fixed indentation on several lines.
Removed testNeedsCrawledSitemap now that we want to use the sitemap.
Factored out a log_in method from the make_http_request method.  Also removed all the passing of full_url that was happening.
Added an extract_sort_criterion function to isolate the name of the sort criterion.  I set this up as a top-level function rather than an object method.  I also converted strip_plone_suffix to be such a function.
Added queing of any folder's default page. Also added a few more image filenames to ignore.
Added proper handling of collections. Not all criteria types handled yet, but searching by type and relative location is working.
Re-worked the gathering of criteria for collections so as to flag any instances of criteria the script doesn't know how to handle.
Added a couple more files to exclude.
Added capture of creation data criteria for collections.
Added ImportLink class and added unquoting of URL strings for traversal during object creation.
Added handling of Link objects.
This commit got too big.  Main items: handling of more than one type criterion in RemoteObject (not in ImportObject yet), treating http and https as the same for caching and crawling purposes, smarter handling of the case when calling /Type returns an entire web page.
Added retrieval of creator, creation date, and modification date. Setting modification date in new site isn't working, though.
Now sets creator correctly.
Added handling of Large Folders coming out of the Plone 3 site.
Proper setting of modification date (using trick to keep Plone from resetting mod date when reindexing).
Added file created by MacOS when mounting code folder.