# Secure, Extensible, Token Authentication for Department of Energy High Performance Computing

## Abstract

There has been a high level of interest in the deployment of One Time
Passwords (OTP) at Department of Energy (DOE) Office of Science Labs.
The aggressive deployment of OTP systems can create a usability issue
for users of computing resources, both in terms of managing the physical
tokens and with respect to the impact of OTP on workflows and
applications. Recent collaborative work between NERSC, ESNet, Argonne
National Laboratory (ANL) and Oak Ridge National Laboratory (ORNL) has
resulted in a functioning prototype of a DOE wide RADIUS Authentication
Fabric that addresses the issue of minimizing the proliferation of OTP
tokens across the DOE user community.

In this paper, we review the work on the RADIUS Fabric, and propose that
further work begin on a Secure Extensible Token Authentication (SETA)
system, based on Kerberos that integrates directly with OTP/RADIUS based
systems. We propose that the project be based on standard, reliable open
source platforms such as Kerberos and FreeRADIUS, and that the source
code be made available to all DOE sites so that it can be customized to
support the individual OTP systems and security policies in place at
each site. This will create a community across DOE that collectively
possesses the skills to modify and extend SETA to support future
security initiatives. With this in place, we believe that as the Federal
Personal Identity Verification (PIV) project begins to roll out across
DOE, it will be possible to modify the services to cleanly support PIV,
cutting down on deployment costs as well as minimizing incompatibilities
due to vendor specific implementations.

## Background and Summary of Work

Readers of this document are probably aware of the expensive and
widespread security compromises in early 2004, as well as the
identification of One Time Password (OTP) technology as a potential
solution. The plans for widespread adoption of OTP across DOE
laboratories motivated the creation of an inter-laboratory collaboration
to examine the possibility of sharing OTP token deployment in order to
minimize the total cost to DOE, as well as avoiding token deployments
that would prove onerous to our user community. This group eventually
consisted of representatives from ESNet, NERSC, ANL and ORNL and was
originally known as the NOPS Group.

The NOPS group has demonstrated that a framework for sharing OTP tokens
across DOE labs is technically feasible as an initial step in addressing
the burden of widespread OTP deployment. The overall architecture can be
summarized as follows:

-   Use RADIUS as a standard protocol for communicating OTP requests
    between DOE sites.

-   OTP systems are mandated to support RADIUS authentication. All of
    the major OTP vendors support RADIUS, and the NOPS group has tested
    OTP authentication over RADIUS on the following OTP systems:

    -   SecureID by RSA Technologies

    -   CRYPTOCard from CRYPTOCard Corporation

    -   SafeWord from Secure Computing

-   Translations between a local site's namespace and a shared, fully
    qualified username/domainname namespace can be handled with fairly
    straightforward modules on FreeRADIUS servers.

-   Within each site, Radius proxying is used to forward authentication
    requests to the appropriate destination site.

    -   Individual clients only need to forward to a sitewide hub that
        is aware of the routing details for each domain, as well as
        which users are local. This hub then routes requests as
        appropriate and forwards the responses.

    -   Radius "routing" information is centrally managed, in a way very
        similar to network routing. The hub is essentially a RADIUS
        router.

-   When a user with a token from a remote site attempts to log into the
    local site, they are able to use their remote token to authenticate
    to systems at the local site

    -   The local machine routes the request to the site RADIUS router,
        where it is rewritten (if necessary) and then forwarded to the
        appropriate OTP server (see Figures 1 and 2).

Each of the sites mentioned above has built a RADIUS infrastructure and
has tested it locally for shell and Web authentication, as well as made
it available to remote sites for Web authentication. NERSC volunteered a
system for shell logins (via ssh) and has successfully demonstrated Unix
logins with One Time Password tokens from remote sites.

![](media/image1.jpeg){width="5.9944444444444445in"
height="4.631944444444445in"}

Figure 1: Sample OTP Deployment (NERSC RADIUS testbed)

The diagram above shows our deployment within NERSC. The important
architectural details are:

1.  **radical.nersc.gov** -- this machine is the "router" that all
    RADIUS requests must pass through. It has information about the
    local and remote OTP servers. In our testing, we configured the
    system to forward all remote domains to ESNet for final routing. In
    the current configuration, it is an InfoBlox RADIUS appliance.

2.  **otp1.nersc.gov** and **otp2.nersc.gov** -- these machines are OTP
    servers with software from RSA SecureID, CryptoCard and Secure
    Computing's SafeWord. We have tested the RADIUS configuration with
    all 3 systems and found them to be compatible with the overall
    architecture

3.  **gridlock.nersc.gov** -- this is a system running the FreeRADIUS
    package. It has been configured to execute a module that maps from
    an unqualified local username (such as "sychan") into a fully
    qualified username (such as sychan123@nersc.gov). It then forwards
    the modified RADIUS request to radical.nersc.gov for final routing.

4.  **Cluster** -- at NERSC, we tested OTP with sample nodes from 2
    production clusters, PDSF and Newton. Both systems were able to use
    RADIUS easily using only modifications to the PAM configuration and
    setting up the necessary RADIUS configuration files.

ESNet managed the transport of RADIUS requests between sites. ESNet has
written an excellent paper on the RADIUS Authentication Fabric that
proposes a permanent infrastructure for routing authentication requests
between sites. Here is a diagram that shows the relationships between
different sites and ESNet.

![](media/image2.jpeg){width="5.989583333333333in"
height="4.623611111111111in"}

Figure 2: Diagram of Wide Area RADIUS infrastructure (RADIUS
Authentication Fabric)

In developing and using this testbed, we believe that we have examined
and found solutions for all the basic technical issues of routing RADIUS
requests between sites, and being able to share tokens among sites that
choose to participate. There are still policy, funding and procedural
issues that need to be addressed, but there are few (if any) technical
hurdles to be dealt with at the level of sharing OTP tokens.

In looking at the deployment of OTP across NERSC, and its effects on our
users and their work, further usability issues have become clear. The
usability issue goes beyond minimizing the number of tokens users must
carry. It includes the following issues:

-   Not requiring users to use the token for each and every access to a
    system.

    -   This can be problematic with OTP systems that have a 1 minute
        limit per login

-   Support for batch job submissions.

-   Support for distributed computing applications.

    -   Grid computing is becoming increasingly important, with
        scientists working in High Energy Physics aggressively deploying
        Grid technology

-   Support for bulk file copies and other data management tasks (often
    related to the batch job and distributed computing issues listed
    above)

The approach that many sites use to deal with these is to use a gateway
based model where a hard perimeter requires OTP technology to pass
through; once through this perimeter the systems have no additional OTP
requirements. This approach has the benefit of being relatively simple
and inexpensive to implement, however it is an eggshell or Maginot Line
type defense -- it bets everything on a thin perimeter, and has little
protection against password hijacking once this perimeter has been
breached. It lacks the "defense in depth" that is recommended for robust
security.

The other drawback to the gateway approach is that it only supports
batch jobs, distributed computing and bulk file copies within the
perimeter of the gateway: if usage or applications cross this perimeter,
OTP intrudes itself into the process, requiring manual entry of One Time
Passwords. This is the Kiss of Death for Grid computing and similar
collaborative projects within DOE.

Sites that have deployed OTP systems have successfully integrated
Kerberos with OTP. This addresses every one of the issues listed above,
and also provides deeper defense because once a Kerberos token is
acquired, passwords are no longer used and are no longer vulnerable to
theft. Kerberos is also a mature, well understood technology that is
supported on virtually all DOE computing platforms, as well as Windows.
It has been vetted by the computer security community and is generally
respected.

> Existing Kerberos/OTP implementations, however, have limitations:

-   There is no allowance for integration with RADIUS

    -   Current Kerberos/OTP authentication solutions require that
        either the Kerberos Key Distribution Center(KDC) knows the OTP
        in advance or, the ticket be sent back to the client encrypted
        with the users long lived Kerberos key(requiring the user to
        provide both their OTP, and their Kerberos password.)

    -   Prior knowledge of the OTP by the KDC is not possible if the OTP
        is to be passed on to another server via the RADIUS fabric for
        validation)

    -   If the user\'s long term key is used to encrypt the ticket, it
        becomes possible to capture Kerberos tickets via network
        sniffing of Kerberos authentication exchanges once the Kerberos
        password has been captured using tty sniffing techniques like
        those we have already seen in use by hackers.

-   Kerberos tokens themselves are vulnerable to theft.

    -   Current Unix Kerberos implementations store user credentials in
        the local file system. This allows any process running as a
        normal user to access all credentials acquired in other sessions
        by that user, and additionally allows accounts with
        administrative privileges to access all currently stored
        credentials for any user.

    -   In order to mitigate the danger of credential theft it is
        desirable to isolate credentials to a session (a process and
        those of its descendants which have not requested a new
        session.)

    -   Secure session based credential caches exist for Linux, but need
        to be more aggressively tested and deployed

    -   Kerberos credential caches based on generic kernel credential
        caches need to be implemented

    -   Some form of more secure credential cache needs to be developed
        for non-Linux platforms and made available to DOE sites

    -   Kerberos implementations need to be configured and extended to
        support real time credential cancellation (similar to the
        service that OCSP provides for certificates)

Radius integration, and secure credential caches for Linux are
relatively straightforward integration problems, and do not require
extensive research and development: the code base is open sourced, and
many of the problems have been solved either in piecemeal form or
solutions can be adapted from other, existing packages.

Further investigation will be required to determine which Unix platforms
provide the necessary kernel interfaces to develop a secure session
based credential cache, and what would be necessary to develop such a
thing for those platforms that don't.

Kerberos currently does not provide for the revocation of tickets. In
order to support this sort of functionality a new type of Kerberos
transaction would need to be added to the protocol in order to determine
if the user key from which a ticket was derived is still valid.
Additional information would also need to be stored in the ticket
(possibly in the authorization-data field of the ticket) to relate the
ticket back to the key upon which it is based.

## Proposed Work

Our proposal is that DOE laboratories agree to support a cross site
RADIUS based OTP standard. Participating labs would agree to require
RADIUS support in their OTP systems, and allow remote sites to
authenticate against their OTP services when appropriate. It is
understood that further negotiations will be necessary to establish the
policies around federation, standards of security and related issues; we
propose that ESNet be chartered with this task, due to their experience
with creating federations for the DOE Grids project. ESNet has proposed
a Radius Authentication Fabric that describes the proposed architecture,
as well as other topics related to it. The ESNet proposal grew directly
from the work described in this paper, and we endorse the Radius
Authentication Fabric as an excellent step in creating a DOE wide OTP
authentication strategy.

To address the shortcomings in a system that is dependent solely on OTP
tokens, as well as planning for the future F.I.V. deployment, we further
propose that work begin on the following:

1.  A mechanism for authentication and acquisition of a Kerberos ticket
    granting ticket (TGT) via the use of a user's OTP.

    a.  This could be implemented as an extension of the standard
        Kerberos authentication exchange, taking advantage of the padata
        field of the authentication request to transmit the OTP, and
        negotiate the key to be used in encrypting the authentication
        reply.

    b.  Alternatively this could be implemented as part of the RADIUS
        exchange using a custom FreeRadius (<http://www.freeradius.org/>
        )module to acquire the TGT, and a custom Pluggable
        Authentication Module (PAM) based on the open source PAM radius
        and Open1x XSupplicant (<http://www.open1x.org/> )clients which
        would place the tgt in the session's credential cache.

2.  A FreeRadius (<http://www.freeradius.org/> ) module that makes use
    of the impending Open Sourcing of the CryptoCard OTP server to
    create an integrated OTP solution that is free of the licensing cost
    and restrictions of commercial products. This can result in
    significant long term savings for DOE.

3.  A Pluggable Authentication Module (PAM) based on the open source PAM
    radius and Open1x XSupplicant (http://www.open1x.org/ )clients.

    a.  This will serve as the client portion of the RADIUS/Kerberos
        service mentioned above in 1b.

    b.  Typical RADIUS clients have encryption protocols that are
        adequate, but the rigid key management strategies have
        difficulties when scaling. The Xsupplicant code supports PEAP
        and EAP-TTLS, two RADIUS authentication protocols that support
        more flexible (and secure) encryption -- we propose modifying
        Xsupplicant to interface into a standard PAM installation.

4.  Promoting the use and development of secure session-private
    credential caches across DOE.

    a.  Credential theft is a fact of life -- passwords are being
        routinely stolen automatically. How long until ssh keys,
        Kerberos credentials and grid certificates are harvested
        automatically?

    b.  Kerberos credentials have sound cryptography in the design;
        however they are vulnerable when stored locally on a computer
        system. There has already been instances of Kerberos tokens
        being stolen and abused. SSH keys are vulnerable to the same
        problems.

    c.  The Linux community has an in-kernel keyring management patch (
        http://lwn.net/Articles/97061/ The latest patch can be found at
        http://people.redhat.com/\~dhowells/keys/)

        i.  This facility and others like it should be developed into
            more mature tools and promoted across DOE labs.

5.  Examine methods of real-time Kerberos credential revocation

    a.  Despite efforts to make sure that credentials are stored
        securely, they will inevitable be stolen. How can we be
        proactive about damage control?

The result of this work would be an open sourced, secure OTP
infrastructure that is maintained by and for the DOE Science Community.
By developing the code within DOE, and open sourcing it, we create a
community that has the programming and operational experience to extend
it to support future systems such as F.I.V. or to deal with new security
threats as they materialize.

## Detailed Discussion of Proposed Solution

1.  Extensions to existing Kerberos implementations supporting
    authentication with OTPs over a RADIUS fabric could be accomplished
    as follows.

    a.  When issuing a KRB_AS_REQ message the program requesting
        authentication (pam module, kinit, etc.) would include the OTP
        encrypted with a secret key shared by both the host from which
        the request is issued, and the KDC as part of the padata field.

    b.  The KDC can then use the OTP to authenticate the user over the
        RADIUS fabric, and if the OTP authentication succeeds the reply
        can be encrypted using the shared secret which was used to
        encrypt the OTP in the request.

    c.  This requires that all programs used to request a TGT(pam,
        kinit, etc.) have access to the key used to generate the
        authentication request(likely by making them setuid to a user
        able to read a keytab.)

2.  As an alternative to section 1 above, create a RADIUS server that
    provides integrated OTP authentication and Kerberos credential
    acquisition.

    a.  The RADUIS server could acquire the ticket on the users' behalf
        after verifying the OTP via the RADIUS fabric, and return it as
        part of the RADIUS response.

    b.  This would require that the RADIUS conversation be protected by
        strong encryption such as that provided by PEAP or EAP-TTLS.

    c.  The RADIUS server would also need access to the users Kerberos
        key. Since the RADIUS server is already trusted to establish the
        authenticity of a user, it shouldn't be an additional security
        risk to store the users Kerberos keys there.

3.  CryptoCard is a One Time Password solution provider that has been
    integrated with Kerberos at several sites such as FermiLab and the
    Naval Research Lab (NRL). These sites use the CryptoCard tokens, but
    do not have to purchase the server or per seat licenses (resulting
    immediate and long term savings). In addition, CryptoCard has
    informed NERSC and LBL that they intend to open source their server
    toolkit to enable customers to build custom OTP servers. We propose
    that the following work be done:

    a.  Using the code already developed and shared by FermiLab, NRL and
        CryptoCard, develop a standalone program that can authenticate
        users of the CryptoCard OTP tokens.

        i.  Other OTP tokens will be supported if implementation details
            are made available by the vendors

        ii. The code that FermiLab and NRL have developed already
            performs the necessary authentication function -- but it
            needs to be modified to be a standalone program.

        iii. CryptoCard offers an SDK, including a library that performs
             the necessary authentication functions.

    b.  Integrate this service with FreeRadius via the published module
        interface, creating an open source and freely available OTP
        service that runs under FreeRADIUS.

        i.  This service would make it far less expensive for sites to
            deploy OTP solutions, lowering costs across DOE overall and
            helping to promote more deployments of OTP solutions

    c.  This OTP authentication module would be used for the integrated
        OTP/Kerberos service described above.

        i.  FreeRadius modules can return additional, virtually
            arbitrary attributes to a client.

        ii. Using the techniques described in the Kerberos section, a
            Kerberos ticket can be acquired and returned as an attribute
            to the client after successful authentication.

        iii. The client will then be responsible for decrypting the
             Kerberos ticket and inserting it into the credential cache

4.  The Pluggable Authentication Module standard is widely available and
    many authentication methods are supported. The current RADIUS
    clients available under PAM do not support the PEAP or EAP-TTLS
    protocols. These protocols provide secure, encrypted communications
    between machines where users are attempting to login, and the back
    end servers that perform the actual authentication.

    a.  The code base for using PEAP and EAP-TTLS for RADIUS already
        exists in the Open1x project ( <http://www.open1x.org/> )

    b.  Using the available source code from the existing PAM Radius
        client libraries as a wrapper, the Open1x code can be made into
        a PAM module. The initial versions of this client will likely be
        Linux.

    c.  This will allow PEAP and/or EAP-TTLS to be used with Unix
        clients, vastly simplifying the problem of RADIUS key
        management, as well increasing the cryptographic security of the
        RADIUS traffic

    d.  After extracting the Kerberos ticket from the RADIUS
        transaction, the credential will then be stored in the user's
        credential cache using standard Kerberos techniques.

5.  The availability of secure session-private credential caches can be
    encouraged in the following ways.

    a.  Support the effort to get keyring support adopted into the
        mainline Linux kernel source tree, as well as encouraging
        similar efforts for other Unix operating systems.

    b.  Investigate whether the necessary features to implement similar
        functionality exist in other Unix-like operating systems.

    c.  Develop, or support existing efforts to develop Kerberos
        credential caches which make use of these advanced os security
        features.

    d.  Adopt the use of session-private credential caches where they
        are available in preference to the current file based cache
        implementations.

## Conclusion

The recent rise in credential theft has spurred an interest in the use
of OTPs. We have proposed a method to reduce the burden on DOE users by
developing a mechanism of using RADIUS to allow for cross site
authentication, thus reducing the number of tokens required by each user
and reducing the amount of replication at each site. The NOPS group has
shown that this approach is technically viable.

The development of a DOE Science wide authentication fabric will
increase the overall security of all sites. Many sites are currently
pursuing similar, yet different routes toward OTP. This mish mash of
deployments has the potential of resulting in some OTPs that are either
deployed improperly or provide a false sense of security with a hardened
external shell, but soft inside. As has been shown with recent security
incidents, incidents that occur at one site can adversely affect other
sites. A more unified approach to OTP can potentially reduce the overall
impacts of credential thefts and security incidents.

## Appendix 1: Sample configuration for FreeRADIUS username rewriting

The following is an excerpt from an email sent to the otp-eng mailing
list describing how to configure FreeRADIUS to perform rewriting of
local username into a fully qualified name of the form username@domain
which can be used to route RADIUS authentication requests. The prototype
is written in Perl, but can be easily rewritten in 'C' for speed and
security concerns.

>     In addition, I found a hint on the FreeRADIUS mailing list about
> how\
> to get a more scalable username mapping working. I\'ve tested locally
> and\
> it seems to work.\
> \
>     I\'ve created a local alias for Tom Barron of \"tbarron\" that
> maps to\
> \"<tpb@ornl.gov>\" and one for Suzanne of \"willoughby\" that maps to\
> \"<sdq@ornl.gov>\". Please give them a try, as well as your actual,
> fully\
> qualified usernames.\
> \
>     The basic approach is to setup a lookup table that translates from
> a\
> local username to the \"actual\" username (which may be local, or
> remote).\
> Early on in the processing of the radius request, you take the
> username\
> and perform the lookup, then replace the original username with the\
> actual username - then allow the radius server to do the normal\
> processing (in our case, normal processing means proxying to the\
> appropriate server).\
> \
>     I used FreeRadius\'s rlm_perl and rlm_attr_rewrite modules to do
> the\
> translation. You need to use 2 different modules because:\
>      - the rlm_perl module can do a table lookup for translation\
> purposes, but is incapable of rewriting the original User-Name value
> in\
> the request\
>     - the rlm_rewrite_attr module can rewrite the original User-Name,\
> but is incapable of doing table lookups\
> \
>     I took the sample example.pl script that comes with the rlm_perl\
> module and put this function into it:\
> \
> sub authorize\
> {\
>         \# Perform a lookup of the given User-Name and see if we have
> a\
>         \# new username to map it to. If so, set a check item for it\
>         \# so that a subsequent rewrite_attr function can replace the\
>         \# username with this new value. If we don\'t have a match\
>         \# then set the check item to the given User-Name\
>         \#\
> \
>         \# lookup hash\
>         my %nameLookup = ( sychan =\> <'nobody@nersc.gov'>,\
>                         tbarron =\> <'tpb@ornl.gov'>,\
>                         willoughby =\> <'sdq@ornl.gov'> );\
> \
>         for (keys %RAD_REQUEST) {\
>                 \# This is for test only\
>                 &radiusd::radlog(0, \"rlm_perl:: \$\_ =
> \$RAD_REQUEST{\$\_} \");\
>         }\
>         if (\$nameLookup{\$RAD_REQUEST{\'User-Name\'}}) {\
>             \$RAD_CHECK{\'User-Name\'} =\
> \$nameLookup{\$RAD_REQUEST{\'User-Name\'}};\
>         } else {\
>             \$RAD_CHECK{\'User-Name\'} =
> \$RAD_REQUEST{\'User-Name\'};\
>         }\
>         #\...\
>         \#\
>         &radiusd::radlog(0, \"rlm_perl::Set new User-Name attribute
> to\
> \$RAD_CHECK{\'User-Name\'} \");\
> \
>         return RLM_MODULE_OK;\
> }\
> \
>     You then configure the FreeRadius server with the appropriate\
> configuration directives to execute the perl script, then run a
> rewrite\
> action on the subsequent packet:\
> \
> from radiusd.conf:\
> \[skip to appropriate section for rewrite definition\]\
>        #  rewrite arbitrary packets.  Useful in accounting and\
> authorization.\
>         \#\
>         \## This module is highly experimental at the moment.  Please
> give\
>         \## feedback to the mailing list.\
>         \#\
>         #  The module can also use the Rewrite-Rule attribute. If it\
>         #  is set and matches the name of the module instance, then\
>         #  that module instance will be the only one which runs.\
>         \#\
>         #  Also if new_attribute is set to yes then a new attribute\
>         #  will be created containing the value replacewith and it\
>         #  will be added to searchin (packet, reply or config).\
>         \# searchfor,ignore_case and max_matches will be ignored in
> that\
> case.\
> \
>         \#\
>         attr_rewrite newname {\
>                 attribute = User-Name\
>                 \# may be \"packet\", \"reply\", or \"config\"\
>                 searchin = packet\
>                 searchfor =\~ \".\*\"\
>                 replacewith = \"%{check:User-Name}\"\
>                 ignore_case = no\
>                 new_attribute = no\
>                 max_matches = 1\
>                 \## If set to yes then the replace string will be\
> appended to the original string\
>                 append = no\
> \
>         }\
> \
> \[skip to modules definition section\]\
>         \# Lets try the rlm_perl module - sychan\
>         perl {\
>                 module = \${confdir}/example.pl\
>                 func_authorization = \"authorize\"\
>         }\
> \
> \[skip to authorize definition section\]\
> authorize {\
>         \#\
>         #  The preprocess module takes care of sanitizing some
> bizarre\
>         #  attributes in the request, and turning them into
> attributes\
>         #  which are more standard.\
>         \#\
>         #  It takes care of processing the \'raddb/hints\' and the\
>         #  \'raddb/huntgroups\' files.\
>         \#\
>         #  It also adds a Client-IP-Address attribute to the request.\
>         preprocess\
> \
> \
>         \# lets try the perl module\
>         perl\
> \
>         \# lets try rewriting\
>         newname\
> \
>     \[rest of authorize section ignored\]\
> \
> \
>     After setting this up, the name translation seems to work.\
>     Note that in the example.pl script, I am using a hardcoded hash -
> it\
> would be a trivial change to make it a tied hash to a Berkeley DB
> file,\
> allowing you to get a much more scalable and manageable way to do the\
> translations.\
> \
>     Since this is just a prototype, hacking it up in perl is fine. I\
> suspect that you could probably actually run this in production if
> you\
> wanted to: the rlm_perl and rlm_attr_rewrite modules are labelled as\
> \"experimental\" but it seems that many sites are using them in\
> production. The perl module is an embedded interpreter, and I suspect\
> that it is doing bytecode caching, so the overhead is probably very\
> small. My experience is that the embedded perl interpreters are\
> generally very quick, and they run pretty cleanly unless you start to\
> depend too much on the perl garbage collector. With complex data\
> structures, the perl reference counting garbage collector used to
> miss\
> things. Maybe things have changed since then. In any case, I believe\
> that if you avoid creating lots of stuff in perl\'s heap, you should
> be\
> safe from GC issues.