Link checker: DB layer and similar changes for URL Verification

Schema and IDL changes for URL Verification functionality
Start building URL-Validation related funcitons
Add Rose::URI to the CPAN_MODULES list
Add ON INSERT trigger to parse the URL as it is added

Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: middle layer work, actual parallelized machinery to check links

Added is_event check to AppUtils
URLVerify.pm getting started
URLVerify.pm; firing batch url verify calls
URLVerify.pm; docs / cleanup
Constraints need unique names
Liberalize res_code constraint
URLVerify.pm; resume options; docs
URLVerify.pm; initial testing tweaks
Move Fieldmapper API call to Application.pm
Move export of opensrf.open-ils.system.fieldmapper API call from
    Fieldmapper.pm into Application.pm with the rest of the shared API
    calls.  This allows us to remove the OpenSRF::Application base from
    Fieldmapper, which was causing some method name collisions.  For
    example, if a Fieldmapper object had a field called "session", which,
    incidentally, is also a method of OpenSRF::Application, the version
    from OpenSRF::Application would get called instead of the Fieldmapper
    version, since Fieldmapper methods are defined during AUTOLOAD.
    Hilarity was guaranteed to ensue.
URLVerify.pm; more testing tweaks
URLVerify.pm; redirects / error handling
url_verify perm/org setting seed data
url_verify perm/org setting seed data
url_verify perm/org setting seed data
url_verify seed data (trigger)
url verify seed data / null constraint repairs
URLVerify.pm; settings, tmp caching, cleanup / misc
url verify seed data repairs
url_verify seed data / sql manifest
url_verify schema repair / res-code constraint
Do the same thing I did to fix constrain names in upgr scripts to baseline
URLVerify.pm; move to lwp to support ftp and simplify
URLVerify.pm; apply timeout to lwp useragent; comments
URLVerify.pm; avoid re-processing same url within the same attempt
URLVerify.pm; avoid re-processing same url within the same attempt (thinko)
URLVerify.pm; avoid re-processing same url repairs; honor delay=0
URLVerify.pm; docs; url shuffling
URLVerify.pm; docs; url shuffling
URLVerify.pm; docs; url domain looping; cleanup
URLVerify.pm; docs; tested redirect max/loops and repairs

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: DB layer fixes

Function for ingesting URLs from container item + session
New container type for url verification sessions

Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: user interface and supporting fixes (part 1)

Add open-ils.url_verify service to example OpenSRF configs
ML methods to create sessions and do the searching/bucketing
    We can't use PCRUD to create url_verify.session objects because a) you
    couldn't trust the creator field if we allowed that, and b) the
    container foreign key has a not-null constraint, so you have to create
    that first, and you can't do that with PCRUD.
    I've removed the C, U and D perms for PCRUD for url_verify.session, but
    I left the R in case we wind up using that.
Beginnings for the big session kick-off UI.  Not yet functional.
Get all search results, not just first 10
Check for session ownership and for previous searchitude
Deal with moved publish_fieldmapper() method
    This is a companion commit to
    fac45ab9b1cb8924 / Move Fieldmapper API call to Application.pm
    Without it, Flattener and Action/Trigger stop working with errors like
    this:
    [Mon Aug 20 13:50:18 2012] [error] [client XXX.XXX.XXX.XXX] Exception:
    OpenSRF::EX::ERROR 2012-08-20T13:50:18 main -e:0 System ERROR:
    Exception: OpenSRF::DomainObject::oilsMethodException
    2012-08-20T13:50:18 OpenSRF::AppRequest
    /usr/local/share/perl/5.10.1/OpenSRF/AppSession.pm:1064 <500>   *** Call
    to [open-ils.fielder.flattened_search.execute.atomic] failed for session
    [1345485018.767884163.96534353976], thread trace [1]:\nNo field by the
    name publish_fieldmapper in Fieldmapper! at
    /usr/local/share/perl/5.10.1/OpenILS/Utils/Fieldmapper.pm line
    270.\n\n\n\n, referer:
    http://XXXXXXX/eg/conify/global/actor/search_filter_group
Use a perm that actually exists
More UI work. Saved search selector & search scope OU selector & cosmetics
Fix subtle Perl issue
    Not a syntax error that the compiler will catch, but see
    "perldoc -f do" which will lead you do "perldoc perlsyn"
Buckets and their items aren't designed to be PCRUD accessible,
    so we need a handy view to link URL Verify Sessions to the bib
    contained.  We can leverage this in flattener queries.
Pretty much finished session create UI but for cloning
Permisison fixing
whitespace
Fix previously nonfunctional stored procedure url_verify.extract_urls(INT,INT)
Call URL extraction phase from UI
Fix xpath generation to match what works
Various fixes, largely UI
Refactor create_session as dojo module.
Fix IDL permissions that require jumps
Essentials for URL selecting interface
Verification sorta works
A note about open-ils.url_verify.verify_url for future reference

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: URLVerify.pm; response throttling repairs

Signed-off-by: Bill Erickson <berick@esilibrary.com>
Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: user interface and supporting fixes (part 2)

Started verification review UI, also SCHEMA CHANGES
    It just doesn't work for me to not have url_verify.url directly related
    to url_verify.session.  When dealing with the "root" URL in a redirect
    chain, you can get the related session through url_selector, but not when
    you have any later URL in the chain.  The only way for IDL perms to work
    would be to have a link to a view using a CTE to find the "root" URL.
    That's too complex, so instead of that I've just added a session fkey on
    url_verify.url.
Corrections to the preceding commit
Vertical scrolling UI glitches fixed
Fix broken display of verification attempt in progress
Implement the "process immediately" switch, hitherto unhooked up
Verify-all now means all-matching-my-search-terms, not necessarily all-in-uvs
let's do filter sets a little more generalized-like
Permission fixing
Filter set loading works.
Filter loading: gracefully skip unknown fields, remove inital empty row
Saving filter sets
Fix filter dialog for pkey fields, scrolliness issue, saved filters issue
Pretty start page for staff client menu to land on
Staff client menu entry
User settings for saved columns
Session cloning, working and rather tested
show name of session on url select page ...
    ... and link back to that on review attempt page
IN / NOT IN for filter somewhat working, but doesn't save/load yet
Saving/loading filter rows for IN, NOT IN operators
Printing

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: technical overview (documentation)

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: Allow configurable User Agent string

Credit to Bill Erickson for noticing that tests were resulting in an
inordinate number of 403 Forbidden responses, which turned out to be
due to discrimination by sites against a libwww/* user agent string.

We now use "Evergreen <version> Link Checker" by default, and it's
configurable in opensrf.xml (grep for user_agent).

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: Links to MARC Editor in verification review interface

Now you can click on the number in the Record ID column on any row in
the verification review interface to launch the MARC Editor for that
record.

This is only supports the most basic of possible mediation workflows to
follow a link checker run, but more could be done with future
development.

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: make bib ID visible by default in attempt review view

Since a piece of workflow (opening the bib in the MARC editor) hangs
off of it, may as well not hide it.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: Some UI tweaks suggested by George Duimovich

  - Make the "Filter" link above FlattenerGrids a button and not a link.

  - Instead of IDs as links in some grid columns, have the ID show up in
    plain text and have links with a more descriptive name sit next to the
    ID.

  - Correct the settings for saving grid columns on the Select URLs and
    Review Attempt interfaces.

  - Tiny i18n fixes (page titles)

  - Fix lack of horizontal scrollbar on Select URLs interface, and also
    fix the way that if you clicked on said scrollbar in a case where
    your grid was taller than your browser window, the page would
    automatically scroll up to focus on your grid header row, and you
    couldn't actually manipulate the horizontal scrollbar.  We sadly
    pay for our horiz scrollbar with a doubled vertical scrollbar, but
    possibly someone can figure the Right way to fix such layout
    problems, which actually occur widely in similar interfaces in
    Evergreen.

  - Add buttons to download CSV on Select URLs and Review Attempt
    interfaces.

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Link checker: URL extraction bugfix and usability improvements

The database procedure for URL extraction did not correctly handle
URLs in repeating subfields in the same tag.  Now fixed.

You now get a hint if you try to create a session with a name you have
used before.

Some UI elements on the create session interface have been reordered to
be more logical.

Signed-off-by: Lebbeous Fogle-Weekley <lebbeous@esilibrary.com>
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Stamping upgrades for Link Checker

Signed-off-by: Mike Rylander <mrylander@gmail.com>