Refactor FsAdaptor.getDocumentContent: extracting methods

In order to support the changes for the following bugs:

b/6513820: Enumerate share points or Namespaces given a hostname
b/12897821: Enumerate Namespace target links

the retriever will be enhanced to support returning ACLs and
content given a docid that could be just a hostname, a sharepoint,
a DFS Namespace, a DFS link, a directory, or a regular file.
In all cases other than regular file, the content returned will
be generated HTML consisting of links to enumerated content,
in much the same way that directories are handled now.  For
shares, DFS Namespaces, and DFS links, ACL extraction and ACL
inheritance must be handled differently than with ordinary
files and directories.

In anticipation of these changes, this refactoring extracts
chunks of FsAdaptor.getDocumentContent() into separate methods.
The code for adding the filesystem ACLs, file and directory
content to the response were pulled out.

Future refactorings will target share ACLs and delay resolving
DFS links.

Code Review: https://codereview.appspot.com/179920043/
Refactor FsAdaptor share ACL handling to support multiple shares

In order to support the changes for the following bugs:

b/6513820: Enumerate share points or Namespaces given a hostname
b/12897821: Enumerate Namespace target links

Supplying only hostname or DFS Namespace as a startpoint
could eventually resolve to multiple shares being traversed.
In anticipation of these changes, this refactoring alters the
FsAdaptor code that reads and pushes DFS share ACLs and share ACLs
to not assume the rootPath is the share.

Code Review: http://codereview.appspot.com/180940043
Refactor FsAdaptor share ACL handling to feed share ACLs at crawl time

In order to support the changes for the following bugs:

b/6513820: Enumerate share points or Namespaces given a hostname
b/12897821: Enumerate Namespace target links

Supplying only hostname or DFS Namespace as a startpoint
could eventually resolve to multiple shares being traversed.
In anticipation of these changes, this refactoring alters the
FsAdaptor code to feed the share ACLs at crawl time, rather
than in the lister.

It should be mentioned that this reverses a previous change
that moved feeding the share ACLs from the crawler to the
lister.  See https://codereview.appspot.com/14294044
Even though I performed that modification, the motivation
for doing so eludes me. IIRC, there were occasional
issues where the share ACL would not get fed, leaving a
broken inheritance chain for all documents.

Code Review: http://codereview.appspot.com/176110043
Refactor FsAdaptor startPoint validation via method extraction.

In order to support the changes for the following bugs:

b/6513820: Enumerate share points or Namespaces given a hostname
b/12897821: Enumerate Namespace target links

Supplying only hostname or DFS Namespace as a startpoint
could eventually resolve to multiple shares being traversed.
In anticipation of these changes, this refactoring alters the
FsAdaptor.init code, extracting the startPoint validation into
a separate method.  This change also coalesces pulling the
configuration options out of the config.

Code Review: http://codereview.appspot.com/178320043
Refactor FsAdaptor filesystem monitors to monitor multiple startpoints.

In order to support the changes for the following bugs:

b/6513820: Enumerate share points or Namespaces given a hostname
b/12897821: Enumerate Namespace target links

Supplying only hostname or DFS Namespace as a startpoint
could eventually resolve to multiple shares being traversed.
In anticipation of these changes, this refactoring alters the
WindowsFileDelegate to allow multiple file system monitors.
To simplify this change, this refactoring also removes the
unused stopMonitorPath method from the FileDelegate interface.
Fix b/12897821: Enumerate Namespace target links.

This change allows the administrator to specify a DFS Namespace
(aka DFS Root), as the startpoint for the filesystem adaptor.
If this is done, the adaptor will enumerate the DFS links in
that namespace at crawl time, and supply them as links in a
generated HTML document (similar to how directories were already
handled).

The DFS links are resolved when the GSA crawls them, the share ACLs
for the link are returned in the response, and a new filesystem
monitor is started for the target filesystem (if one has not
already running).

The two biggest structual changes relate to the multiple targets
being traversed:

1) There can now be multiple filesystem monitors running.

2) The names of the Share ACL and DFS Share ACL have changed.
   In the past, they were naked fragment names.  Now they are
   fragments on the filesystem root or DFS link DocIds.
   This change is incompatible with existing indexed content,
   and some or all documents may require a recrawl.

Code Review: http://codereview.appspot.com/184950043
Remove PollingIncrementalLister implementation from FsAdaptor

Now that FsAdaptor.getModifiedDocIds() does nothing, drop the
PollingIncrementalLister implementation, and remove the
CONFIG_MAX_INCREMENTAL_LATENCY configuration parameter.

Code Review: http://codereview.appspot.com/184290043
Allow DFS Links to be placed deeply under DFS Namespace Root

While reading the Microsoft DFS tutorial, I found that DFS links
may be placed in non-targetted folders under the DFS namespace.
For instance: \\server\namespace\folder1\folder2\link.

The code assumed that the links were directly under the namespace,
and this corrects that poor assumption.  However this complicated
WindowsFileDelegate.isDfsLink().  That method is called for every
crawled document where we would be performing a network DFS getInfo
call for every file.

A test of calling isDfsLink() against 250,000 pathnames via a DFS
server linked to a local network file system took 515 millisecs.
So I don't believe calling isDfsLink() for every crawled file will
significantly impact performance.

Code Review: http://codereview.appspot.com/184360043
Better File System Adaptor and ACL documentation.

Code Review: http://codereview.appspot.com/183360043
Rename DfsRoot to DfsNamespace to avoid confusion.

Even though MSDN developer documentation refers to a DFS Namespace as a
DFS Root, we have multiple uses of the work "root" in the adaptor.  So
to avoid some confusion, we have decided to unambiguously refer to a
DFS Namespace as DfsNamespace, rather than DfsRoot.

Code Review: http://codereview.appspot.com/187280043
Fix b/16619976 Part Deux: Adaptor Feeds improper DFS share ACLs.

From the Properties panel when setting permissions on a DFS link:

“Specify how DFS Namespaces controls which users can see this folder.
 By default, permissions are inherited from local file system of the
 namespace server, granting all domain users the ability to view all
 folders in the namespace.”

This change replaces the call to NetDfsGetSecurity() with a call
to GetFileSecurity() on the namespace when there is no explicit
ACL on the DFS link.

Code Review:  http://codereview.appspot.com/191130043