From: dgh@validgh.com (David G. Hough on validgh)
Newsgroups: comp.benchmarks,comp.sys.sun.misc,comp.lang.c,comp.lang.fortran
Subject: Comparing Compilers on SunOS 4.1.2
Date: 2 Oct 92 18:14:12 GMT
Followup-To: poster
Organization: validgh, PO Box 20370, San Jose, CA 95160

tbl | troff -ms source for the following report is available from
dgh@validgh.com

Summary

     Many SPARCstation customers wonder if they can get better C and Fortran
compilation speed or execution speed or correctness or robustness by obtaining
compilers from other sources than SunPro, but they often lack the resources to
thoroughly study the matter.   In many cases, however, the answer appears to
be "no".  I compared some C and Fortran compilers, among those available in
March 1992, on a SPARCstation-2 running SunOS 4.1.2.   Some results:

Robustness

*    GCC 2.1 and F2C are robust "free" alternatives to SunPro SC1.0 compilers.

*    Lucid-C 2.1 and KAP are much less robust than the SC1.0 compilers they
     intend to replace or enhance, often generating incorrectly optimized
     code.

*    SunOS 4.1.2 bundled /bin/cc is not as robust at -O4 as SC1.0 cc; many
     Fortran programs translated with F2C require compilation at -O2.

Execution-time performance
     I found no better alternative to SC 1.0 compilers that provides con-
     sistently superior execution-time performance, especially on floating-
     point programs, except SC2.0.1 compilers.

Compile-time performance
     I found no better alternative to SC1.0 cc for compile and execution time
     performance with -g, the normal debugging mode for C programs.

     "SC1.0" compilers are SunPro's current release; "SC2.0.1" compilers are
the next release under development.

Definition of Execution-Time Performance

     The performance comparison between two compilers is based on relative
user+sys performance of programs that ran correctly and for at least ten
seconds with both compilers.  On a particular test program, 100% is defined to
be the best correct performance obtained so far on a SPARCstation-2 with any
compiler, any options, and any operating system.

     Thus a rating of 50% corresponds to 2X slower than the best.  All runs
were ranked in order, and first and third quartile percentages recorded.  In
the following tables, for instance, "47-70" means that one fourth of the tests
achieved less than 47% of best known correct performance, one fourth achieved
more than 70% of best known correct performance, and half achieved between 47%
and 70%.

     Only execution times of ten seconds or more, from realistic applications,
were tabulated in the execution time comparisons.  Data was collected in
multi-user mode, and so may vary somewhat from run to run; hence differences
of a few percentage points aren't significant.

     ~SPECint92 and ~SPECfp92 are also included.   The usefulness of these
single measures is reduced by lack of a corresponding measure of dispersion.
The "~" serves as a reminder that these numbers are based on user+sys time
under the SunPro test harness rather than real time under official SPEC
Makefiles, and so vary from official SPEC results.

     The SunPro test harness is a collection of Makefiles and sh and awk
scripts, combined with modified versions of test programs from a variety of
sources, that is used to test Unix system-level correctness and measure Unix
system-level performance of CPU-bound programs. As such it tests hardware,
operating system, compilers, and libraries.  For this report, in order to com-
pare compilers and their libraries, the hardware and operating system were
kept constant: a SPARCstation-2 with 32 MB RAM running SunOS 4.1.2.

Definition of Compile-Time Performance

     Like execution times, compilation times are evaluated as a percentage of
the best user+sys compile time ever recorded on a SPARCstation-2, from any
compiler, with any options.  Compile times are given for all programs in a
particular language, whether realistic applications or synthesized test pro-
grams.  Because compilation involves substantial disk and net I/O, and sub-
stantial overhead for the SunPro test harness shell scripts and makefiles,
times are more variable and less reproducible than execution times, which are
mostly CPU-bound.

     There is no one-to-one correspondence between compilation times and exe-
cution times.   One executable such as SPICE 2G6 or SPICE 3E2 may be run with
several different input decks to generate different execution times.   And
whereas only realistic applications are used to compare execution times, a
number of synthetic test programs that don't yield useful execution time com-
parisons are included in the compilation time comparison.

Compiler Versions and Options Tested

     I tested the following compiler versions available in March 1992:

*    Lucid C 2.1

*    GCC 2.1 with some patches supplied by Cygnus

*    /bin/cc bundled with SunOS 4.1.2

*    SunPro "SC1.0" compilers C 1.1 and Fortran 1.4

*    Kuck and Associates' KAP released for SunPro Fortran 1.4

     I also tested "sc2" compilers, developmental versions of C 2.0.1 and For-
tran 2.0.1; final released versions may yield slightly different results.

     More current releases, GCC 2.2.2 and Lucid-C 2.2, may yield better
results on SPARC under SunOS 4.1.2.

group       Compiler       Options

gcc.g       gcc-2.1        -g
gcc.g       f2c            -R -r -P | gcc

lucid.g     lcc-2.1        -g
lucid.g     f2c            -R -r -P | lcc

sc1.g       SC1.0 cc       -g
sc1.g       SC1.0 f77      -g

gcc.max     gcc-2.1        -O2 -funroll-loops -static
gcc.max     f2c            -R -r -P | gcc

lucid.max   lcc-2.1        -O5 -Ou -Xa -Bstatic
lucid.max   f2c            -R -r -P | lcc

bincc.max   /bin/cc        -O4 -dalign -fsingle /usr/lib/libm.il -Qoption as -Ff0 -Bstatic
bincc.max   f2c            -R -r | cc without libm.il

sc1.max     SC1.0 cc       -O4 -cg89 -dalign -libmil -Bstatic -fsingle
sc1.max     SC1.0 f77      -O4 -cg89 -dalign -libmil -Bstatic

sc2.max     sc2 acc        -xO4 -dalign -libmil -Bstatic -Xa
sc2.max     sc2 f77        -O4 -dalign -libmil -Bstatic

sc1.f2c     SC1.0 cc       -O4 -cg89 -dalign -libmil -Bstatic -fsingle
sc1.f2c     f2c            -R -r | cc without libm.il

sc1.kap     SC1.0 f77      -O4 -cg89 -dalign -libmil -Bstatic
sc1.kap     KAP            -inlr -inll=4 -ag=a | f77 <max opts>

sc1.O       SC1.0 cc       -O
sc1.O       SC1.0 f77      -O

sc1.fast    SC1.0 cc       -fast
sc1.fast    SC1.0 f77      -fast

For compiling F2C-translated code, the -P option to F2C and the "ANSI" mode of
the gcc or lucid compiler were used, unless the Fortran source contained con-
flicting array type usages; then -P was not used and the "K&R" mode of the C
compiler was used (GCC -traditional or Lucid -Xk).  To conserve storage on
systems that lack virtual memory, many large Fortran applications use the same
common storage areas to hold different types of data.

The following tables use these abbreviations:

       Abbreviations for classes of test programs

C      C programs
F      Fortran programs
int    dominated by integer arithmetic
dp     dominated by double-precision floating-point arithmetic
sp     dominated by single-precision floating-point arithmetic
apps   realistic applications

The 15 C integer applications, 24 C double-precision floating-point applica-
tions, 18 Fortran single-precision floating-point applications, and 44 Fortran
double-precision floating-point applications, are listed at the end of this
report.

Alternatives to C 1.1

     For these tests, F2C was used to translate Fortran programs into C.  Thus
only the C compiler varied.

C Compiler   22 C programs   15 C int apps   24 C dp apps   ~SPECint92
                   compile         execute        execute

gcc.g                63-82           22-55          35-40            7.4
lucid.g              60-88           30-47          47-54            8.5
sc1.g                76-99           49-65          49-64           12.1

bincc.max            29-55           72-82          72-85           16.9
lucid.max            30-61           58-93          52-75           17.3
gcc.max              39-70           72-91          69-80           17.9
sc1.max              23-40           79-91          91-99           18.7
sc2.max              18-42           87-97          92-97           19.6

There are 23 sc2.max C compilations, rather than 22, because one program  uses
the long double type, which is not supported by the other compilers.

       C Compiler with F2C   73 F programs   44 F dp apps   ~SPECfp92
                                   compile        execute

       gcc.g                         38-51          10-23           3.4
       lucid.g                       37-52          23-50           7.4

       gcc.max                       24-36          29-65          11.8
       lucid.max                     22-35          56-74          14.2
       bincc.max                     18-29          54-74          15.3
       sc1.f2c                       13-21          65-83          16.8

"Compile time" includes both F2C and C compiler time.

     Comments: GCC 2.1 is competent in most respects - robust and well  optim-
ized  for  integer  programs,  but  incompletely  optimized for SPARC floating
point.  The free combination of GCC 2.1 and F2C with "max opts"  yielded  less
than  29%  of  best  known  correct  performance for one quarter of 44 Fortran
floating-point programs, and less than 65% for three quarters.

     Lucid C 2.1 does well on stones, C linpack, and SPEC int benchmarks,  but
appears not to have been tested much on realistic floating-point applications.
See the robustness notes below.

     The /bin/cc bundled with SunOS 4.1.2 is not as advanced an  optimizer  as
the unbundled products.

Alternatives to Fortran 1.4

     Customers dissatisfied with Fortran 1.4  price  might  use  F2C,  a  free
Fortran-to-C  converter  from  netlib. Customers dissatisfied with Fortran 1.4
performance might buy the KAP preprocessor in addition.   What do they gain?

     F2C is a very robust translator.  I found only one library bug,  which  I
fixed myself.  Like GCC, it requires some configuring before it can be used.

     KAP sometimes requires extensive  experimentation  for  each  program  to
determine  KAP  options that will improve performance without creating invalid
results.  The following KAP results mostly use the three KAP options  required
for SPEC89fp.

    Compiler     73 F programs   18 F sp apps   44 F dp apps   ~SPECfp92
                       compile        execute        execute

    bincc.max            18-29          48-72          54-74          15.3
    sc1.f2c              13-21          58-79          65-83          16.8
    sc1.max              19-27          82-96          86-94          20.7
    sc1.kap            -                83-99          87-97          21.4
    sc2.max              16-22          79-91          88-98          22.3

There are 80 sc1.max and sc2.max 80 Fortran compilations, rather than 73,  be-
cause  7  programs  use the real*16 data type, which is translated by F2C into
long double, which is not supported by SC1.0 cc.

     Three quarters of Fortran double precision programs, compiled  with  F2C,
obtained less than 83% of the available performance.  Yet with SC1.0 f77, they
obtained at least 86% of the available performance.

     A small additional performance gain is sometimes available  on  realistic
applications by using KAP.  Of course, on some programs like matrix300, if you
can determine the right options, KAP can provide an enormous improvement.  KAP
was  originally  developed for vector and parallel processors operating on ap-
plications large enough to justify considerable investment in performance tun-
ing.    On  typical scalar RISC workstations most of the potential KAP benefit
lies in rearranging memory accesses to better exploit the data cache, so  pro-
grams that don't access data in a predictable pattern show no benefit.

     Because KAP preprocessing occurred on a  remote  SPARCstation-1  machine,
comparable compile times aren't available.

SC1.0 compiler optimization effects

     SunPro's compiler technology  was  evaluated  by  comparing  default  -g,
-fast, -O, and "max" compilation options.

    Options    22 C programs   15 C int apps   24 C dp apps   ~SPECint92
                     compile         execute        execute

    sc1.g              76-99           49-65          49-64           12.1

    sc1.O              39-56           70-81          69-82           15.9
    sc1.fast           39-55           66-83          86-97           16.3
    sc1.max            23-40           79-91          91-99           18.7
    sc2.max            18-42           87-97          92-97           19.6


     Options    80 F programs   18 F sp apps   44 F dp apps   ~SPECfp92
                      compile        execute        execute

     sc1.g              86-99          28-59          28-46           8.2

     sc1.O              19-30          69-86          64-81          15.7
     sc1.fast           25-36          77-94          74-87        -
     sc1.max            19-27          82-96          86-94          20.7
     sc2.max            16-22          79-91          88-98          22.3

No ~SPECfp92 is reported for -fast since the -fnonstd  part  of  -fast  causes
early  termination on some tests when certain floating-point exceptions arise.
Similarly only 43 Fortran double-precision applications are counted for -fast.

     Thus on three quarters of these tests, SC1.0 compilers got at  least  86%
of  the  best  known  correct  performance  for  44  Fortran  double-precision
floating-point applications, and 79% for 15 C integer applications.   Compared
to  SC1.0 results, C integer and Fortran double-precision applications execute
measurably faster with "sc2" compilers.

Robustness Issues

     Compiler and run-time failures are listed below.  It is in the nature  of
realistic  applications such as those reported here that, like all other large
programs, they contain errors of various kinds, that may or may not affect the
validity  of the results.  These may be undetected until a new compiler or new
optimization is tested.  When I isolate such errors  in  the  applications,  I
correct them in the SunPro test harness and report them to the developers.

     Customers differ as to the value of compilers that are sensitive to  such
bugs.   Some  are  grateful to discover the problems, even if the debugging is
painful; others can't afford that and greatly value  compilers  that  can  run
their existing programs unchanged.

     The definition of "correct results" for a program that runs to completion
with  zero  exit status is usually the unoptimized -g results if they are any-
thing like the results obtained with other compilers.  Optimized results  from
the  same compiler are deemed incorrect if they differ from unoptimized, other
than in minor floating-point differences.

lucid.max: F2C and Lucid C 2.1 -O5 -Ou

     Correct execution was determined by compiling with Lucid C -g,  which  is
supposed  to  produce  the same result as -O-.  There were substantial differ-
ences in output, on many programs, between -g and -O5 -Ou.  Optimization  lev-
els  were lowered from -O5 -Ou until compilation completed and correct results
were obtained.  Although many of the problematic programs  were  F2C  transla-
tions  from  Fortran,  most  of the large C floating-point applications in the
SunPro test harness had to be recompiled at -O- or -O0.

         executables   compiled with       examples

             327       -O5 -Ou (maximum)
               6       -O4                 052.alvinn (C)
               6       -O3                 048.ora
               1       -O2                 mm1sim (C)
               2       -O1                 perfect.arc2d
              10       -O0 (default)       015.doduc rayshade406 (C)
               5       -O-                 spice3e2 (C) imsl c (C)
                                                            -

Reduced optimizations were required for 6 out of 20 SPEC92 programs, and for 4
out of 12 PERFECT Club programs.

     Lucid C 2.1 compiled codes got less than 95% of best  performance  in  24
out  of  24  C floating-point programs, 12 out of 15 C integer programs, and 6
out of 8 SPEC C programs.

sc1.kap: Fortran 1.4 plus KAP with -inlr -inll=4 -ag=a


executables   KAP options with F1.4   examples and comments

    324       -inlr -inll=4 -ag=a     required for max SPEC89fp
      8       -inlr -inll=4           (perfect.bdna perfect.dyfesm) problem with test programs
      9       <none - KAP default>    (perfect.mg3d) problem with KAP
      5       O=1 R=1 S=1             (mcnp) problem with KAP
      1       SO=0                    (perfect.trfd) problem with KAP

8 test programs can't be compiled with -ag=a, because they don't  declare  and
use common blocks consistently. This is a problem with the test programs rath-
er than with KAP, but such inconsistent usages are common in Fortran.

     9 test programs that work correctly with default KAP options,  fail  with
the  KAP options required for maximum SPEC performance, timing out during com-
pilation or execution, failing with nonzero exit status during compilation  or
execution, or producing incorrect results.

     6 test programs wouldn't compile or execute correctly even with  KAP  de-
fault options and required lower-than-default optimization settings.

     KAP+F1.4 got less than 95% of best performance on 10 out  of  18  single-
precision  programs,  and 26 out of 44 double precision, 4 out of 12 SPEC For-
tran programs, and 10 out of 11 PERFECT Club programs.

gcc.max: F2C and GCC -O2 -funroll-loops

     After translation via F2C, perfect.adm could not be compiled  by  gcc  at
-O2 on the system available.  It was compiled unoptimized instead.

bincc.max: F2C and /bin/cc -O4  -dalign -fsingle /usr/lib/libm.il -Qoption  as
-Ff0 -Bstatic

     65 FCVS  tests,  6  other  Fortran  programs  including  perfect.adm  and
perfect.flo52,  and  085.gcc  could  not be compiled at -O3 or -O4 by /bin/cc.
They were compiled at -O2 instead.

sc1.fast: C1.1 and F1.4 compiling -fast

     FCVS test #405 was not compiled correctly with -fast.   It  was  compiled
unoptimized instead.
-- 

David Hough
dgh@validgh.com
Newsgroups: comp.benchmarks