From: smf%isis@ajax.rsre.mod.uk (Steve Folkes)
Newsgroups: comp.sys.next.advocacy
Subject: TDF/ANDF (long) (Was: NeXT's binaries need a diet, or the solution to NeXT's multiprocessor problem.)
Date: 20 Nov 92 09:56:12 GMT
Organization: DRA Malvern, Worcs., UK

Hi,

There have been a number of postings recently about fat binaries and
Douglas Floyd's NeXTCode idea. I think that TDF/ANDF is relevant to this
discussion, so I thought I would try to provide some information about it
especially when compared to fat binaries. I work on the team developing TDF
at DRA, so I am probably slightly biased. What follows is my own opinion,
not that of DRA.

[For those who haven't heard of ANDF (Architecture Neutral Distribution
Format) or TDF (Ten15 Distribution Format) yet, here is a very brief
description. ANDF is the Open Software Foundation's name for a software
distribution format that allows applications to be represented in a manner
that is independent of any particular hardware architecture. This means
that software distributed in this format can be installed on any machine
that provides an ANDF installer and implements the APIs that the software
uses. TDF is such a format, developed at the UK's Defence Research Agency.
TDF was selected by OSF as the base technology for ANDF. USL have also
signed a letter of intent to help commercialise the TDF technology.]

I can see two problems that fat binaries are trying to solve (perhaps there
are others that I am unaware of). Firstly they are a means of distributing
an application so that it can be executed on more than one architecture.
Secondly they are a means of allowing applications to reside on a file
server with clients of different architectures, without needing several
copies of the entire application.

I think that fat binaries are probably a reasonable solution to these
problems, but I am not convinced that they are a good solution. I will deal
with the second issue first. I think that the problem with fat binaries as
a solution to the second problem is that the solution is limited only to
binaries. There are however other things that you may wish to share in the
same way (e.g. libraries, initialisation files etc). Fat binaries do not
solve this problem.

Another possible solution (which lends itself to sharing any type of file)
is something like the HP-UX CDF (context dependent file) mechanism. A CDF
is a special type of directory that behaves like a file. Each process has a
context which is a list of strings. These strings represent information,
such as the name of the machine the process is running on, the type of
executables it is capable of executing, and a default value. It would be
possible to add other information like operating system version as well.
When a process tries to open or execute a CDF, the system looks inside the
directory for a file whose name matches one of the strings in the process'
context (they are searched in a specified order, so that a process may
indicate it prefers one binary format to another - e.g. prefer 68040 to
68030, but can execute both). That file is then opened or executed instead.
This is only a brief description - I am no expert on HP-UX.

I think CDFs are a better solution to this problem, because they are more
general. In an application, you could just make the binary component a CDF,
with an entry for each processor type. The only advantage I can think that
fat binaries might have is a small amount of space saving (e.g. sharing
TIFFs or similar portable data, but this could always be stored in a
separate file).

On the first issue, I think fat binaries are no worse a solution than
anything else that is currently in use. I think however that it lacks a lot
of the advantages of something like TDF. Below is a list of advantages I
believe TDF has over fat binaries for distributing software.

For any given application there only needs to be one piece of TDF. This TDF
can be installed and executed on any machine that has a TDF installer and
provides an implementation of any APIs that the application needs.

The single piece of TDF means that the size of distributable won't increase
as support for new architectures is added (actually, it isn't necessary to
add support for new architectures - it's there automatically).

With TDF, users can buy an application for their current machine (e.g. a
68040 NeXT) and still use it when they upgrade to a more powerful machine
with a different processor (e.g. the rumoured NRW). It is not necessary to
purchase a new copy of the software to run on the new machine (at least
there is no technical reason, although there may be licence related reasons
why this is not the case). With fat binaries, this is only true if the copy
you bought came with the correct components for your new machine.

From a developer's point of view, if a new machine comes out with the same
operating system/APIs as existing machines then all of their TDF
applications for the existing machines will just work on the new machine.
With fat binaries, it would be necessary to recompile the application for
the new machine (which requires getting a new machine, or a cross compiler
for the new machine), produce a new distributable with support for the new
architecture, and re-distribute the new version. This seems like a lot of
extra work, for no real gain.

I suspect that TDF applications are more likely to survive operating system
upgrades without breaking. When writing an application with TDF, the APIs
the application uses are specified in a portable manner (and only the
public parts of the API are specified). This means that the TDF that is
distributed should be portable to anything that implements the API. The TDF
contains no part of the API implementation itself.

When using normal C, macros are used fairly frequently. This means that
whilst it may be possible to replace most of the API implementation by
changing a shared library, certain parts of the API implementation are
compiled into the application itself. This can cause applications to break
when the new operating system changes the implementation of part of the API
that the macro accessed. As well as the API implementation changing, the
system ABI may change (although this is probably less likely). Again, a TDF
application will just need re-installing, whereas a binary won't work after
the ABI change.

Probably the biggest disadvantage of TDF for the user is the fact that
installation takes longer than installing a binary (as you need to do a
partial compilation). The time is typically 50-75% of the compilation time.
However as you would normally only need to do this once per architecture it
is probably not too great a price to pay.

Various statements have been made about how performance would be affected
by compiling through an ANDF. The latest set of facts and figures (for SPEC
1.2B) indicate that TDF is no more than 2% slower than the native compiler
on all platforms that support it (on systems where gcc 1.37.1 produced
faster code than the native compiler, we used gcc for the comparison
instead). Actually, on all but one platform the binaries compiled through
TDF are faster than those compiled with the native (or gcc) compiler.
Running other applications has shown that there is very little in it either
way (i.e. you don't lose efficiency by compiling through TDF).

For those who are interested, TDF can currently be installed on the
following platforms:

	SCO UNIX	80x86
	SVR4.2		80x86
	HP-UX 7.05	680x0
	Ultrix 4.2A	MIPS
	SUNOS 4.1.1	SPARC
	AIX		RS/6000

There used to be support for Ultrix 4.2 on VAX, but this is currently not
being updated. There is also an installer for a transputer box, which is
being updated to use the current TDF specification. A port to MSDOS has
been made, although it only supports the ANSI C API at present. In order to
get TDF on a NeXT, you would need the following:

	Language -> TDF producers

		There is already an ANSI C->TDF producer. Producers to turn
		other languages into TDF are being written (as part of the
		ESPRIT project in Europe, producers are being implemented
		for FORTRAN, Ada, C++ and a LISP dialect; there are
		probably others). I don't know of an Objective C to TDF
		producer, but it should be possible to write one.

	TDF -> NeXT installers

		There are already TDF->680x0 and TDF->80x86 installers in
		existence. The 680x0 installer is for HP-UX, but the author
		is porting it to my NeXT in his spare time. I believe the
		installer itself is okay, and the only problem is that
		NeXT's header files contain some gcc specific code (so they
		can't be used to compile programs through TDF). This could
		be fixed by re-writing the header files, or by getting the
		C->TDF producer to understand the gcc extensions (I would
		like to see the latter, but this may not be done).

	API specifications and implementations

		It would be necessary to convert the NeXTSTEP API into the
		special header files that the producer uses to specify the
		API without knowing its implementation.

Hopefully this will give people some idea of what TDF is and what it can
do. It is only a very brief description of the system, and has only
described how I think TDF compares to fat binaries as a distribution
mechanism.

Despite its name, there is a lot more to TDF than just being a distribution
format. TDF is a porting technology. TDF allows a much greater degree of
separation between modules during a compilation than in a conventional
system (it is possible to delay the binding of expressions, types,
representations etc until install time). This is particularly useful for
separating portable code from non-portable code.

It should be obvious from the above that TDF relies heavily on APIs in
order to allow portable programs to be written. TDF does not allow people
to magically write programs that will run on every machine ever built, it
just provides mechanisms to make it easier for programmers to write
portable code. APIs are one of the most important mechanisms for writing
portable code. APIs are more than just C library interfaces such as ANSI C,
POSIX, XPG3, or the NeXT appkit. APIs include things like assumptions about
the size of an integer or the floating point format. TDF will let you write
code that assumes the target machine has 32 bit integers (although you will
probably need to tell the producer this to stop it complaining about the
code being non-portable - by default it will complain about all non-
portable code that it can detect) but the TDF produced will then only
install on machines that implement 32 bit integers.

The producer cannot detect every use of non-portable code, but a lot of
work is being done on trying to make it easier to specify what assumptions
are being made, and to detect violations of these assumptions.

The TDF specification and encoding are now in the public domain, as are a
set of routines for encoding and decoding TDF. For more information on TDF,
people should contact Dr Nic Peeling, email:

		peeling%hermes.mod.uk@relay.mod.uk

	I hope some people find this of interest,

			- Steve.
--
===============================================================================
| Steve Folkes.
| Internet: smf%hermes.mod.uk@relay.mod.uk
| JANET:    smf@uk.mod.hermes
===============================================================================
Newsgroups: comp.sys.next.advocacy