# The integration of BBFTP with the SAM system

## Status Report

Gabriele Garzoglio, Feb 08, 2001

### The goal

Goal of the project was configuring and testing a SAM station to use
BBFTP[^1] instead of rcp as file transfer method.

### What is available

The sam station droide is now configured to use bbftp.

I have set up a cvs repository called bbftp on the CD/ODS cvs area. It
includes the original bbftp v1_9_4, the modified code to change the
authentication procedure, the bbrcp wrapping script and some
documentation. Note that the original bbftp release comes with the
libraries libcrypto.a and libz.a for several platforms: these libraries
gets corrupted checking them in and out of the cvs repository for
reasons that I didn't have time to investigate. If problems occur at
building time, the original libraries can be found in the tar ball that
bbftp distributes at http://ccweb.in2p3.fr/bbftp.

### What was done

A new sam station, named droide, was installed and tested on
droide.fnal.gov. This station runs master, fss and stager servers.
During testing a few inconsistencies in the documentation and running
problems were found and are reported later in this document.

BBFTP v1_9_4 was used for this test. The release comes with 2 source
packages: the bbftp client and the daemon. The packages built easily on
Linux (droide) and IRIX (d0test) (see INSTALL note in the package).

BBFTP was tested running the daemon in background as the sam user
looking at port 4021 (the bbftp default port). Note that this port can
be easily changed by the use of the flag "-w port_num" for both client
and server or by hard coding the port number in the bbftp sources. The
bbftp client accepts a list of commands in input from a file only (no
user interface is implemented); the commands are very similar to the
standard ftp and are well documented in the man pages that come with the
package. I couldn't find the man page of the daemon, instead.

The bbftp client can run several parallel tcp streams, performing
compression on the fly. By some tests that Michael Diesburg has done, it
looks like 7 streams are a good choice for an optimized transfer rate;
on the fly compression was also used during the sam tests.

In order to simplify the inclusion of bbftp within sam, I have written a
wrapping script that mimics the syntax of rcp. The script writes a
temporary file in /tmp that bbftp uses as list of commands. Only the
user sam is allowed to use the script. I have performed many tests on
this script especially to check the consistency of the exit status (see
file TESTS in the cvs repository); for example, if bbftp can connect to
the server but tries to transfer a non existing file, it exits 0 (no
error); the wrapping script checks if conditions like these occurs and
exits 1 (error). This preserves the sam error handling mechanisms.

This script was renamed rcp and put in the directory
\$SAM_STATION_DIR/bin/ together with bbftp (the client): for the station
master the path to the rcp wrapper comes before the unix rcp; this way
sam uses bbftp transparently.

### Details on the retrieve test

I've created a project named ggtest to retrieve a file cached on the
d0test sam station, and the relative python script to be used with sam
run project gg_project.py.

I've run the bbftp daemon in background (bbftpd --b) as user sam on
d0test.

From droide:

> \> sam run project gg_project.py

The file was delivered using bbftp in a transparent way. This is a line
from the log file \~sam/station\_\_droide\_\_dev\_\_droide/trace :

02/07/01 16:37:31 droide.SM.SAMINFO 22249:

INFO FileDelivered Station=droide
FileName=sim.qcd_02_05.2_1000evts.355-0953-c5.02

EncpOutput=Using bbftp v.1.9.4 to transfer file.

Starting under pid 22891 on Wed Feb 07 16:37:01

\>\> USERNAME sam PASS

\<\< bbftpd version 1.9.4 : OK

\>\> COMMAND : get
/sam/cache1/sam/do_not_erase/sim.qcd_02_05.2_1000evts.355-0953-c5.02
/sam/cache1//boo/sim.qcd_02_05.2_1000evts.355-0953-c5.02

\<\< OK 283008482 bytes got in 28.8 secs (9.61e+03 Kbytes/sec or 75.1
Mbits/s)

Storing was not tried for problems in writing directly to the cache of
d0test.

### Open issues: the authentication method

The authentication between the bbftp client and the server is still an
open issue. The bbftp daemon supports several authentication methods and
protocols (shadow password, security password, AFS, PAM). The daemon
looks through the local or yp passwords to authenticate the client. This
method can be broken where the Kerberos authentication system is
strictly enforced. Finding an alternative way of authenticating the
client and the server was not the focus of this test. A final decision
on how to cope with this issue from a general point of view has not been
taken so far.

The test with sam was performed hard coding the sam password in the
client executable. The password was masked character by character to
avoid easy ways of finding it out (like using the "strings" command).
The user sam was then given a local passwd on d0test.

### Problems encountered during the droide sam station testing.

1.  The product sam_user v2_2_6 does not have the files globals.py and
    globals.pyc in \$SAM_USER_DIR/ v2_2_6/src/python which are needed by
    run_project.py in the same directory. The script run_project.py is
    invoked when issuing a\
    \>sam run progect file.py.

2.  The example python script to store motecarlo files, which comes with
    the product sam_user v2_2_8 (and v2_2_4b), does not work. The error
    reported is\
    Code: method unimplemented (Category Operating System)

3.  When storing, the auto-destination does not work. I was using sam
    v2_1_2 --q dev on droide; station, fss and stager v2_2_6.

4.  Using the web interface to the sam database of data file
    (development), I have retrieved the file names for the file location
    d0test.fnal.gov:/sam/cache1/sam/do_not_erase; clicking on these file
    names I've got information for files with a similar but different
    name; these latest files are not in the database.

### Acknowledgements

This project was a joint effort between Igor Terekhov and myself: he
promptly helped me whenever I needed a jump-start in the sam land.

Luciano Piccoli helped a lot for the set up of the droide linux box and
the sam station.

Michael Diesburg contributed to the setup of bbftp with his previous
tests, and Maarten Litmaath with very useful example of code related to
the authentication issue.

[^1]: **References**

    ^?^ http://ccweb.in2p3.fr/bbftp