system-test: build authorized-keys file

The admin interface supports authentication via SSH pub/priv keys,
which are often stored in well-known locations.

This patch takes the user's public keys, if found, and uses them to
build an authorized-keys file for the admin interface.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5754/
Acked-by: Gerd Behrmann
fix servlet classpath clash

The servlet API jar files are signed.  Recent versions of java (or
possibly bug http://bugs.sun.com/view_bug.do?bug_id=6805618 ) means
that, on some platforms, all the classes in a package must be signed
by the same certificate.  A failure results in an error message and
dCache failing to startup.

Currently, dCache includes a Jetty-supplied 'javax.servlet' jar file
(v3.0), and a 'javax.servlet'-supplied 'servlet-api' jar file (v2.5).
These are signed by different certificates and, since the v3.0 file
contains additional classes, if v2.5 precedes v3.0 in the classpath
then the classloader will find classes in the same package with
different signatures.

The previous version of Jetty depended on servlet-api jar file and,
while this too had two jar files (a Jetty-supplied and a
'javax.servlet'-supplied) since the two jar files have the same set of
classes, the classloader will take javax.servlet.* classes from one
jar file, independent of classpath order.

This patch fixes this problem by excluding the references to
servlet-api dependency in axis and axisg jar files.

Target: master
Requires-nodes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5757
Acked-by: Gerd Behrmann
Add DB driver to the list of classes to load

The chimera-cli command was overhauled to make it significantly faster.
One technique used was to limit the classes in the classpath to only
those needed.

Unfortunately, this work excluded the database-driver jar. The current
chimera-cli works for system-test, as the plugin mechanism ensures the
database-driver jar is included; however, the PostgreSQL driver is not
included, resulting in chimera-cli no longer working for machines using
PostgreSQL.  The error message is a stack-trace that starts:

[root@bes-srm dcache]# chimera-cli ls /
Exception in thread "main" java.lang.ClassNotFoundException:
org/postgresql/Driver

[reported in user-forum]

Target: master
Request: 2.6
Requires-notes: yes
Requires-book: no
Patch: http://rb.dcache.org/r/5755/
Acked-by: Gerd Behrmann
info: fix concurrent modification

Gerd reported seeing this:

06 jun 2013 21:44:33 (info) [] Uncaught exception in thread Thread-79
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.remove(HashMap.java:912) ~[na:1.7.0_11]
        at org.dcache.services.info.gathers.MessageHandlerChain.flushOldMetadata(MessageHandlerChain.java:230) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.MessageHandlerChain.getMetricTTL(MessageHandlerChain.java:173) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.MessageHandlerChain.getMetricTTL(MessageHandlerChain.java:37) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.CellMessageHandlerSkel.answerArrived(CellMessageHandlerSkel.java:150) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:971) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:929) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_11]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]

The problem is triggered by having multiple messenger threads and the
class not being thread-safe.

This patch updates the class to make it thread-safe

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5756/
Acked-by: Gerd Behrmann
ssh2: Honor admin.colors.enable

Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by:
Target: trunk
Require-book: no
Require-notes: no
Remove line that sets colors from environment param and not from within dCache
Change parameter to no type definition and switch line to above cell related properties
ssh2: Honor admin.colors.enable

Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
Merge color fix branch with current master because not all changes got committed with the last push.

Sorry.

ssh2: Honor admin.colors.enable
Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
loginbroker: Comply with new configuration policy

Drops the service srm-loginbroker. All doors now register with
a single login broker called LoginBroker and SRM and all
monitoring services query this one LoginBroker to locate doors.
SRM is updated to exclude srm as a transfer protocol, thus
making it safe to register all doors in the same login broker.

The dcache.service.loginbroker property is updated to take
a comma separated list of loginbrokers to register with or
to query. httpd now uses this property to configure which
loginbrokers to query.

New properties

loginbroker.cell.name
srm.protocols.disallowed.get
srm.protocols.disallowed.put
srm.protocols.preferred

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5742/
cleaner: Clean up cleaner.properties

Besides cleaning up the comments, this patch turns the previously
obsoleted property cleaner.hsm into a deprecated property.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5744/
spacemanager: Allow previously obsoleted values for deprecated srmSpaceManagerEnabled

This patch adds legacy values for srmSpaceManaerEnabled. Thus one can
once again use true, on, yes, enabled, false, off, no, disabled, but
only when using the legacy property.

Also fixed a bug in srm.batch introduced when previously deprecating
srmSpaceManagerEnabled.

Require-book: no
Require-notes: yes
Target: trunk
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5748/
replica: Restored legacy values of deprecated replicaManager property

When renaming the configuration properties of replica manager, the
replicaManager was marked deprecated and the allowed values changed
to true and false rather than yes and no. This patch restored the
legacy values such that yes and no can be used. The new
dcache.enable.replica property still requires the values true and
false.

The default value of the property has been changed from yes/true to
no/false, as this better reflect most dCache instances.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5749/
gplazma2-htpasswd: Add htpasswd support

This plugin adds support for the htpasswd file - a file format
used by many web servers.

The plugin only supports MD5 hashes. SHA is not supported
at the moment.

The htpasswd can be managed with the htpasswd utility available on
most OSes.

Target: trunk
Require-book: yes
Require-notes: yes
gplazma2-ndgf: Dynamic Principal Mapping plugin

This is an experimental gPlazma 2 mapping and session plugin. It was
created with focus on what we would need from a general authorization
system at NDGF.

Notice that I deliberately ignored whatever strange mappings our
existing system supports. It may very well be possible to combine old
and new, however for this excercise I forgot about old and created what
I think should be how we do it in the future.

The plugin maps each principal to either:

- a user record, or
- a group record,
- nothing

A user record got a UID. A group record got a GID. These provide the UID
and GIDs to include in the LoginReply. Each user record and group record
can have any number of attributes, and these are included as session
attributes in the LoginReply. Each user record and each group record
contains an authorization flag with four possible values: BANNED,
READ-ONLY, ALLOWED, UNKNOWN. The authorization decision for the login
operation depends on these flags.

The whole mapping process doesn't know about specific Principal types or
LoginAtribute types. Instead everything is driven by descriptions of
possible classes in the database. The idea is that this plugin can work
with new principals introduced by other plugins. Thus other plugins are
not limited to the small number of principals shipped with core dCache.

The database essentially maintains 5 types of objects (I use Alias as a
class name to describe a Principal that maps to a user or group; that's
primarily to avoid a naming conflict in the class name space).

- AliasClass describes a Princial class, e.g. GlobusPrincipal

- Alias describes a specific principal, e.g. my DN

    Alias got two subclasses: UserAlias and GroupAlias, which are mapped
    to a user record and a group record respectively

- User describes a user record. It is associated with the
  UserAliases that map to this user.

- Group describes a group record. It is associated with the GroupAliases
  that map to this group.

- AttributeClass describes a LoginAttribute class (e.g. root directory).

- Attribute describes a specific LoginAttribute instance (e.g. my root
  directory being /pnfs/ndgf.org/data/behrmann).

Notice that users and groups can have multiple principals, however each
principal only means one thing. It was a deliberate choice NOT to
introduce a separate account concept. The intention is that a user has
exactly one uid and a uid describes exactly one user. Of course a user
may have different credentials, e.g. multiple certificates, or multiple
user name password pairs. The database allows these to be mapped to the
same user record, however this has to be an explicit action by the
admin.

The AliasClass contains various pieces of information about how to treat
Principals of the given type. The most important is whether it describes
a user or a group and whether to auto create a corresponding user or
group. The intention is that one would typically authorize a complete VO
or subgroup of a VO (e.g. through an FQAN). The group gives rise to a
GID. However we don't need an apriori list of all users in the VO. User
records get created on the fly and a new uid is assigned.

Notice that a new user or group created this way doesn't authorize the
user or group: The default is UNKNOWN. The typical case is however that
the user got an FQAN that maps to an existing group which IS authorized.
This way we authorize the user to login and still give her a unique uid.

AttributeClass similary contains information about how an attribute is
used. In particular it tells us if a LoginReply may contain multiple
instances of the same LoginAttribute class. It also tells us whether it
only applies to the primary groups. A typical usecase is that one would
define RootDirectory as an AttributeClass, bind it to a group record
that is authorized and then flag it to apply to primary groups. Thus
when a user logs in, she gets authorized by being a member of this group
(per having an FQAN that maps to that group). She will get the root
directory associated with that group. If a user is a member of two VOs,
then this scheme will ensure that the root directory depends on which
group is the primary group (in case, say, atlas and cms got different
root directories).

Determining the primary group is non-trivial, as there are many ways to
influence the process:

- If an explicit gid was specified during login (DCAP allows that), then
  that selects the primary group. The user however already has to be a
  member of that group!

- If an explicit login name was specified (this would be the role
  argument in DCAP, or the login name in gsiftp or kerberos ftp), then
  it is considered a group name and is used to select the primary group.
  The user however already has to be a member of that group!

- If a principal is flagged as a primary group, then the group that it maps
  to is the primary group. If there is more than one, a ranking value
  associated with AliasClass is used to pick one.

- Otherwise we pick the highest rank principal (non primary) that maps
  to a group.

- Otherwise we consult the user record. Each user must have a group that
  will be used as a fallback.

The grid-mapfile mapping is special. Since each user maps to one and
only one user record, there isn't much sense in considering the
grid-mapfile as a map to user names: The DN is usually mapped to a user
record directly anyway. In practice what we use the grid-mapfile for is
a fallback for VOMS when the client didn't provide a VOMS proxy
certificate. We generate the grid-mapfile by exporting group information
from the VOMS server. That means the grid-mapfile actually maps users to
groups, not users. That is also how the plugin uses the grid-mapfile. It
takes the DN of the user, maps it to a name which is considered as a
GroupNamePrincipal and which is then pushed through regular user
mapping.

There is one detail in the above mapping. The mapping happens even when
the login didn't specify a DN: A login with a user name + password is
first resolved to a user record. Then we determine other aliases for the
user and if one is a DN we push it through grid-mapfile. The use case is
that it is the *user* that is a member of the VO, not the DN. We don't
want to have to duplicate the group membership in the database when we
already have it described in grid-mapfile. This also means we cannot use
the existing gridmap file plugin to perform this mapping.

The final LoginReply (or rather the Subject) will contain the UID and
GIDs as principals. The other principals may be included, depending on
the database configuration. Specifically all aliases of the user and the
groups may be included (actually the Principal corresponding to the
alias). The idea is that after login it shouldn't matter whether the
user logged in one way or the other. E.g. if stage authorization was
configured using DN, then the user with that DN should be allowed to
stage no matter whether she authorized with X509, Kerberos or a user
name. Whether the principal is included depends on the embodied field of
the AliasClass. If set to true, then the principal is included in the
subject.

This process also applies to principals that didn't map to a user or
group alias. E.g. the Origin principal class also gets an entry in the
database (as an AliasClass) and if it is marked as embodied, then it is
included in the LoginReply.

Thus the admin is in absolute control of what kind of principals to pass
on to the rest of dCache. If we know that we only ever use UID and GID,
then we can limit the system to this. If we still use DN and FQAN and
user name to authorize in various components, then we can include that
information too.

Authorizations success and failures are logged. There is a special log
target for this information and nothing else should be logged to this
log target. The log format and destination is configurable through
logback. Our operators didn't want this information to rely on a
database, however one can use a database appender if one likes to have
it in a database. This should eventually be moved to gPlazma so it works
for all setups.

Forward and reverse mappings are not quite implemented yet.

I updated the WebDAV door to use the reverse mapping to show the owner
in directory listings. This is just for proof of concept: We need to
pipeline and cache the reverseMap calls and we need to work on the
presentation (probably only show common name and have the complete DN as
a mouse over).

I use JDO and DataNucleus for database access. I use LiquiBase for
schema change management. Database access is abstracted through two DAO
interfaces, with one class implementing both.

Besides the plugin there is an administrator service that implements a
dCache admin shell CLI for manipulating the database contents (I of
course expect that there will be support in webadmin for these
operations too). The DataNucleus second level cache is disabled since we
otherwise get inconsistencies between multiple plugin instances and the
admin component. I use Spring for transaction demarcation.

The patch is not yet complete, as there are things I want to add before
we consider to deploy it at NDGF:

- exceptions need to be refined
- More javadoc must be written
- some unit tests are needed
- a bit of pollish here and there
- the ORM needs to have another look; maybe my choice of primary keys
  isn't the best

Let me reiterate that this is an experimental component. It will pretty
much do what we need at NDGF, however I have already thought of tons of
ways it could be generalized. My primary interest at the moment is
however to get input from the rest of you.

I believe the plugin can be very useful in any context. I envision a
setup in which various gPlazma modules would do initial mappings like
extracting DN, FQAN, call out to external identity services, etc. but
where the result would not be a final UID and GID; rather the result
would be new principals, that would then be mapped by the above
component.

This is how I would integrate the federated identity mapping and group
system that is currently being developed in SweGrid (the COIP idea). The
user identity (X.509, username + password, ssh, etc) would be mapped to
globally unique user identifiers and group identifiers (strings not
numbers). The plugin would represent these as new principals, and then
the database driven mapping module would map them to internal UIDs and
GIDs.

There is one final comment about how I handle groups. I intentionally
did not add the ability to map principals to mulitple groups. The reason
is that I believe group membership management should be dealt with in
another plugin (like a callout to the COIP system). More importantly,
any system that would allow me to map principals to multiple groups in
the database resulted in issues with reverse mappings. In the end it
made everything much simpler to just have it map to one group.

Ah, and another final comment :-) How FQANs are handled is not hardcoded
in the system. The database can be setup to treat them in various ways.
However I imagine a system in which FQANs auto generates new groups and
a GID is assigned to each of them (remember that auto created users and
groups are not authorized; that's still an explicit action by the
admin). Thus a user that got 15 FQANs will end up having 15 GIDs (and
probably a few more). Thus at the file system level we can very
specifically control access to particular FQANs (as each corresponds to
one specific GID). This will also make it much easier to expose our ACLs
through the ACL support in SRM: We can easily map GIDs back to FQANs.