**[FIM HOW TO Manual]{.underline}**

2 Jan 2008 version

**[Outline]{.underline}**

1.  **Software -- source, running**

2.  **Dynamics Options**

3.  **Physics Options**

```{=html}
<!-- -->
```
1.  **[Software]{.underline}**

**[Structure and Operation (on wjet) of FIM source and FIM
run]{.underline}**

> Jacques Middlecoff \-- 13 Nov 2007

**[FIMrun/ FIMsrc/ README]{.underline}**

The FIMsrc directory contains everything necessary to build FIM and the
FIMrun directory contains everything necessary to run FIM. The FIMsrc
and FIMrun directories are together here for convenience, but on wjet
typically they are separated with FIMsrc in the home directory and
FIMrun on the /tg filesystem.

a.  **[FIMsrc directory]{.underline}** contains:

bin/ cntl/ fim/ lib/ Makefile post/ prep/

-   FIMsrc/bin will contain the executables.

-   FIMsrc/cntl contains the control subroutines used by the
    pre-processing, FIM, and the post processing.

-   FIMsrc/fim contains the FIM code in two subdirectories,

    -   FIMsrc/fim/column and

    -   FIMsrc/fim/horizontal.

-   FIMsrc/lib will contain the libraries.

-   FIMsrc/post contains the source for post processing and

-   FIMsrc/prep contains the source for pre-processing.

**FIM Directory Tree available at**

<http://www-ad.fsl.noaa.gov/ac/source_browsers/fimbrowser/>

(account -- visitor, PW -- noaaaccess1)

a.  **[Compiling and linking the model]{.underline}**

cd to FIMsrc and enter (for instance):

*make G=8 K=50 P=240*

for

-   grid level 8 \-- 655,352 points

```{=html}
<!-- -->
```
-   50 vertical levels

-   240 processors.

which will build executables for

-   prep (ss2icos8_50 and ssfc2icos8_50),

-   fim (fim8_50_240), and

-   post

NOTE: run_time checks for FIM

-   The new FIM system requires only 3 keystrokes in the main Makefile
    (uncommenting 3 diagnostic FFLAGS) to enable run-time checks for
    prep, fim, and post. When the new system is archived (FIM 3.0), a
    requirement for changing the code will be to make a test run with
    run-time checks enabled.

The executables are put in the FIMsrc/bin. FIM is a *static memory*
model so you must build an executable for the desired number of
processors. The parallel part of FIM is in FIMsrc/fim/horizontal. In
order to avoid a partial recompile with a different number of
processors, and facilitate saving compilations for different processor
counts and levels, the makefile creates a subdirectory called
FIMsrc/fim/horizontal_8_50_240 (for this case) and links to
FIMsrc/fim/horizontal and does the build in horizontal_8_50_240. A
\"make clean\" removes these subdirectories and the contents of
FIMsrc/lib. A \"make cleanall\" also removes the contents of FIMsrc/bin.
FIM is parallelized using the [Scalable Modeling System
(SMS)](http://www-ad.fsl.noaa.gov/ac/sms.html)
(http://www-ad.fsl.noaa.gov/ac/sms.html). The path to SMS is set in
FIMscr/horizontal/Makefile.sms.

a.  **[Creating initial conditions for model]{.underline}**

> **TO BE ADDED in near future - Jacques**

b.  **[Running the model]{.underline}**

The FIMrun directory contains the files

FIMnamelist qsubfim qsubTemplate

To run the model, cd to FIMrun and use the qsub script qsubfim. The

qsubfim script calls the qsubTemplate script which sets the three

environment variables:

> setenv DATADIR /tg2/projects/fim/fimdata
>
> setenv SRCDIR /whome/jacquesm/FIM/FIMsrc
>
> setenv PREPDIR /whome/jacquesm/FIM/FIMrun/fim5_50_16_241300/prep
>
> which set the path to fimdata, FIMsrc, and prep respectively.

-   DATADIR contains the input data needed by prep

-   SRCDIR/bin contains the executables.

-   Setting PREPDIR allows the use of an existing prep directory and
    thus saves recalculating the pre-processing info.

-   The qsubTemplate script also sets the account number which is
    currently set to \"fim\". To run the case above, enter

*qsubfim 8 50 240*

The qsubTemplate script creates a directory of the form

FIMrun/fim8_50_240_133867 where 133867 is the job ID. Under this

directory qsubTemplate creates the subdirectories

data/ fim/ post/ prep/

for running the pre-processing (prep), FIM (fim) and the post processing
(post). All three of these operations read FIMrun/FIMnamelist and write
data to FIMrun/data.

1\. qsubTemplate copies all the input needed for prep to FIMrun/prep and
then cd\'s to prep and runs ss2icos_5_50 and ssfc2icos_5_50 located in
FIMsrc/bin. All the output of prep is written to FIMrun/data. Standard
out goes to a file called stdout in FIMrun/fim8_50_240_133867/prep.

2\. qsubTemplate then cd\'s to FIMrun/fim and runs FIM (fim8_50_240)
also

located in FIMsrc/bin. All the input for FIM is read from FIMrun/data.
For each 3D output variable, FIM produces a separate output file in
FIMrun/data. All the 2D variables are written to one file. These output
files should be bit-wise exact for different numbers of processors on
the same machine with the same compile options. Standard out goes to a
file called *stdout* in FIMrun/fim8_50_240_133867/fim. At each output,
FIM writes out a line labeled MAXMIN to *stdout* from output.F90 with
the following print statement:

> print\"(\'MAXMIN\',i10,1p6e10.3)\",its,maxqv3d,aveqv3d,minqv3d,maxdp3d,
> avedp3d,mindp3d

These four digit numbers should all be non-negative and for different
machines and different compile options, the averages should match to
three digits and the MINs and MAXs should match to two digits.

The total time for a run is given by \"Elapsed Time\" in the last line
of the file *stdout*:

> SMS:: Program complete, exiting: 2007:07:17::02:14:49 Elapsed Time =
> 6301 sec.

a.  **[Output files from FIM model]{.underline}**

**[TO BE FILLED IN]{.underline}**

Include format, units, variables

b.  **[Generation of the icosahedral grid]{.underline}**

Generation of the icosahedral grid involves dividing the globe into 10
rhombi. The points in each rhombus are currently generated in prep using
a Hilbert curve which means that the rhombuses are best divided by
powers of two. On 10\*4\*\*N processors (N=0,1,2,\...), the halos are
nearly optimum, optimum being 4\*sqrt(TotalPoints/(10\*4\*\*N)). The
halos are reasonable small on N\*40 processors where N has factors about
the same size. The sweet spot for proposed applications is about 2500
points per processor and, for level 8, 240 processors gives 2731 points
per processor.

c.  **[Cache blocking]{.underline}**

To be filled in by Jacques in the future.

d.  **[Version control]{.underline}**

-   FIM releases using Subversion

    -   Latest revision: svn checkout svn+ssh://g0511/repos/fim/

    -   FIM version 2.0 (Revision 6): svn checkout
        svn+ssh://g0511/repos/fim/ -r6

    -   FIM version 1.1 (Revision 4): svn checkout
        svn+ssh://g0511/repos/fim/ -r4

(Taken from <http://fim.noaa.gov/internal/fim-version.html> )

2.  **[Dynamics options]{.underline}**

**More information at <http://fim.noaa.gov/fimdocu_rb.pdf>**

a.  **[Horizontal grid]{.underline}**

Future entry on use of unstructured grid, indexing, indirect addressing

b.  **[Vertical grid]{.underline}**

Use of hybgen.F90 options

-   Sigma

    1.  cp hybgen_sig.F90 over hybgen.F90 and recompile

-   Hybrid isentropic-sigma

    1.  cp hybgen_theta.F90 over hybgen.F90

        a.  Use PCM or PPM options

    2.  Dpsig options

Different options for sigma layer thickness currently available. To be
specified in ./grid directory.

dpsig.txt

> dpsig.txt-40mb
>
> dpsig.txt-15mb
>
> dpsig.txt-7.5mb
>
> dpsig.txt-20mb
>
> dpsig.txt_RB

Different options for reference potential temperature values currently
available. To be specified in ./grid directory.

theta_coor.txt

> theta_coor.txt_RB
>
> theta_coor.txt_RUC

3.  **[Physics options]{.underline}**

**More information at <http://fim.noaa.gov/fimdocu_rb.pdf>**