TSUBAME4.0 User's Guide

TSUBAME Computing Services, Global Scientific Information and Computing Center
2024-10-10

TSUBAME4.0 User's Guide

Table of contents

1. Introduction to TSUBAME4.0

41.1. System architecture4

1.2. Compute node configiration

5

1.3. Software configuration

6

1.4. Storage configuration

6

2. Getting Started

7

2.1. Get an account7

2.2. Login7

2.3. Password Management

8

2.4. Changing default login shell

9

2.5. How to check TSUBAME points

9

3. Storage system

10

3.1. Home directory10

3.2. Group disk

10

3.3. CIFS access from inside campus

11

3.4. Local scratch area

12

4. Software Environment

134.1. Switch User Environment

13

4.2. Intel Compiler

14

4.3. NVIDIA HPC SDK

16

4.4. AOCC

17

4.5. GPU Environment

174.6. Use containers

18

5. Job Scheduler

20

5.1. Job Scheduling System Configuration

20

5.2 Normal queue

21

5.3. Interactive job queue

30

5.4. Storage use on Compute Nodes

31

6. Commercial Applications

32

6.1. ANSYS

33

6.2. ANSYS/Fluent

34

6.3. ANSYS/LS-DYNA

36

6.4. ABAQUS

37

6.5. ABAQUS CAE

38

6.6. Gaussian38- 2/80 -2024-10-10

TSUBAME4.0 User's Guide

6.7. GaussView

39

6.8. AMBER

41

6.9. Materials Studio43

6.10. Discovery Studio45

6.11. Mathematica

46

6.12. COMSOL

48

6.13. Schrodinger

49

6.14. MATLAB

50

6.15. VASP

51

6.16. Linaro forge(ex:Arm Forge)52

7. Freeware

55

7.1. Quantum chemistry/MD related software57

7.2. CFD related software60

7.3. Numerical calculation library for GPU

60

7.4. Machine learning and big data analysis related software

61

7.5. Visualization related software

64

7.6. Other freeware

68

Appendix. Comparison of old and new TSUBAME

75

Appendix.1. Architecture75

Appendix.2. Compute nodes

75

Appendix.3. Software

76

Appendix.3.1 System software

76

Appendix.3.2 Commercial application

77

Appendix.3.3 Freeware

78

Appendx.4. Storage

79

Revision History80- 3/80 -2024-10-10

TSUBAME4.0 User's Guide

1. Introduction to TSUBAME4.0
1.1. System architectureThis system is a shared computer that can be used from various research and development departments at Tokyo Institute of Technology.
The system's theoretical peak double precision performance is 66.8PFLOPS, theoretical peak half precision performance is 952PFLOPS, total main
memory is 180TiB, total SSD capacity is 327TB and total HDD capacity is 44.2PB.
Each compute node and storage system are connected to the high-speed network by InfiniBand and are connected to the internet at a speed of
100Gbps via SINET6 directry from Suzukakedai campus, Tokyo Institute of Technology.
The system architecture of TSUBAME4.0 is shown below.

- 4/80 -2024-10-10

TSUBAME4.0 User's Guide

1.2. Compute node configiration
The computing node of this system a large scale cluster system consisting of HPE Cray XD665 240 nodes.
One compute node is equipped with two AMD EPYC 9654 (96 core, 2.4GHz), and the total number of cores is 46,080.
the main memory capacity is 768GiB per compute node, total memory capacity is 180TiB.
Each compute node has 4 port of InfiniBand NDR200 interface and consistutes a fat tree topology by InfiniBand switch.

- 5/80 -2024-10-10

TSUBAME4.0 User's Guide

The spacifications of each TSUBAME4.0 node are as follows:
Components

Model/SpecficationCPU

AMD EPYC 9654 2.4GHz x 2 Socket

Core/Thread

96 core / 192 thread x 2 Socket

Memory768GiB (DDR5-4800)

GPU

NVIDIA H100 SXM5 94GB HBM2e x 4

SSD

1.92TB NVMe U.2 SSD

Interconnect

InfiniBand NDR200 200Gbps x 4

1.3. Software configuration
The operating system (OS) of this system has the following environment.
• RedHat Enterprise Linux Server 9.3
The OS configuration is dynamically changed according to the service execution configuration.
For application software that is available in this system, please refer to Commercial applications and Freeware.

1.4. Storage configuration
This system has high speed / large capacity storage for storing various simulation results.
On the compute node, the Lustre file system is used as the high-speed storage area, the large-scale storage area, Home directory and common
appplicaion deployments.
In addition, 1.92TB NVMe SSD is installed as a local scratch area in each compute node.
The file systems available in this system are listed below:
Usage

Mount point

Capacity

Filesystem

High-speed storage area

/gs/fs

372TB

LustreHome directory/home

44.2PB

LustreEach node 1.92TB

xfs

(SSD)
Large-scale storage area

/gs/bsCommon application deployment

/apps

(HDD)
Local scrach area (SSD)/localFor more details about local scrach area in each resorce type, please refer to Available resource type.

- 6/80 -2024-10-10

TSUBAME4.0 User's Guide

2. Getting Started
2.1. Get an accountIn order to use this system, it is necessary to register a user account.
Please refer Getting Accounts Page for details, as the procedure depends on your affiliation and program to apply.

2.2. LoginYou need to upload the SSH public key to access the login node.
Please refer to TSUBAME portal User's Guide "SSH public key registration" for the operation of public key registration.
When you finish registration of the SSH public key, you can login to the login node. You will be redirected to one of the login nodes automatically by
DNS round-robin.

Warning
As login nodes are shared with many users, please do not execute heavy operations there.

The use case is shown below.

Connect to the login node with SSH. And you can transfer files using SFTP.
login.t4.gsic.titech.ac.jp

To connect to a specific login node, log in with the following hostname (FQDN).
login1.t4.gsic.titech.ac.jp
login2.t4.gsic.titech.ac.jp

The following example shows a connection method with X transfer option enabled from Linux / Mac / Windows (Cygwin).
example) UserName gsic_user , and Private key: ~/.ssh/t4-key$ ssh gsic_user@login.t4.gsic.titech.ac.jp -i ~/.ssh/t4-key -YC- 7/80 -2024-10-10

TSUBAME4.0 User's GuideInfo
If you have the key pair in the standard path and with the standard file name, the -i option is not required.

In the first connection, the following message may be sent depending on the client's setting. In that case, enter yes .
(fingerprint type depens on your ssh version)
The authenticity of host 'login.t4.gsic.titech.ac.jp (131.112.133.51)' can't be established.
ECDSA key fingerprint is SHA256:CyUFJvJ9ExAE4QEM3BTcKHZ2g7Mw57NnCzbvqIM0ewo.
ECDSA key fingerprint is MD5:d0:99:50:c7:ae:d4:32:62:a6:b8:5e:c0:bc:11:37:96.
ED25519 key fingerprint is SHA256:l++Qi9lvnHIRQbQ+638MgWsZdSyZvdAWicwDsYq5KBI.
Are you sure you want to continue connecting (yes/no)?

2.2.1. Restrictions for heavy work in login nodes
As login nodes (login, login0, login1) are shared with many users at the same time, please do not execute programs which dominate CPU time. For
parallel or long-time computation, please use compute nodes using qsub and qrsh commands, Followings are examples of judgment criteria. Whenthe system administrator noticed your program is preventing others, it will be terminated even if it is permitted or not prohibited here,
Permitted operations• File transfer, compression, decompression (e.g., scp, sftp, rsync, tar)
• Program compilation (If you consume lots of resources by parallel compilation etc., use compute nodes)
Prohibited operations• Calculation using commercial applications, freeware, or programs made by yourself• Execution of programs that exceeds 10 minutes (except for file transfer)• Execution of parallel programs (including python and MPI)
• Execution of programs that consumes lots of memory• Execution of lots of processes simultaneously (e.g., parallel compilation)• Execution of server programs or auto-restart programs (e.g., VSCode Server, Jupyter Notebook)• Other operations which use lots of CPU resources

Info
Jupyter Lab and code-server(VS Code clone) can be invoked in compute nodes via Open OnDemand.

The login nodes have 4 GB memory limit per process. The system administrator will terminate programs with excessive loads without prior notice. If
you want to execute such operations, or you feel login nodes are too heavy, please use compute nodes as interactive jobs via job scheduler.

2.3. Password Management
The user account of this system is managed by the LDAP, and authentication in the system is done by SSH key authentication.
For this reason, you do not need a password to use the compute nodes, but you will need a password to access the Windows/Mac terminal in theuniversity, storage system.
If you need to change the password, please change from the TSUBAME4.0 portal. The rules of available passwords are described on the
TSUBAME4.0 portal password setting page.

- 8/80 -2024-10-10

TSUBAME4.0 User's Guide

2.4. Changing default login shell
At the time of user registration, the login shell of each user account is /bin/bash. If you want to change the default login shell, please use the chshcommand. The available login shells are bash, tcsh and zsh.
You can check the available login shell with the chsh command without arguments.
$ chshUsage: chsh shell(/bin/bash /usr/bin/zsh /usr/bin/tcsh).

Info
It may take one hour to complete the change at most.

The following is an example of changing the default login shell to tcsh.
$ chsh /usr/bin/tcsh
Please input Web Portal Password(not SSH Passphrase)
Enter LDAP Password: xxxxxx<-- Enter your password.
Changing shell succeded!!

2.5. How to check TSUBAME points
You can check TSUBAME points with t4-user-info group point command as below.
$ t4-user-info group point -g TESTGROUP
gid
group_namedepositbalance-------------------------------------------------------------xxxx TESTGROUP
10
5000

In this case, current deposit point is 10, and remaining TSUBAME point of TESTGROUP is 5000.

- 9/80 -2024-10-10

TSUBAME4.0 User's Guide

3. Storage system
In this system, you can use the Lustre file system for the high-speed and the large-scale storage area as well as home directory. And local scratch
area on each compute node is also available.

3.1. Home directory
HOME directory is 25GiB availalbe per user.
You can check usage status with t4-user-info disk home command.
$ t4-user-info disk home
uid nameb_size(GB) b_quota(GB)i_filesi_quota--------------------------------------------------------------2011 TESTUSER
7
251014462000000

Of the 25 GB quota limit, 7 GB is used, regarding the inode limit, we can check the situation that we are using approximately 100,000 out of the 2
million quota limit.
Please note that new writing cannot be performed if the quota limit is exceeded. If you remove some files and free space is available again, you cancreate new files.
Even if you delete files, sometimes the quota limit status has remains. In this case, it takes a day at most until refreshing the status to recover.

3.2. Group disk
Group disk is used to share files among specific group members. The high-speed or the large-scale storage area are available depend on yourpurpose.
To purchase group disk, please refer TSUBAME Portal User''s Guide.
The characteristics of each storage area are as follows.
Storage area

PATH

Group disk capacity

Available sizes

settingHigh-speed storage area

Number of files that can besaved(inode)/gs/fs100GB unit

MAX 3TB

2,000,000 / 1TB/gs/bs1TB unit

MAX 100TB

2,000,000 / 1TB(SSD)
Large-scale (Big) storage
area (HDD)

3.2.1. High-speed storage area
High-speed storage area is mounted at /gs/fs.
The high-speed storage area consists of the Lustre file system using SSD and you can be used by purchasing it as a group disk.
Compared to the Large-scale (Big) storage area, the total disk space is smaller, but file access is faster, so it is recommended for “execution
environments where file access speed is important” and “frequently accessed data files”.

3.2.2. Large-scale (Big) storage area
Large-scale (Big) storage area is mounted at /gs/bs.
The large-scale (big) storage area consists of the Lustre file system using HDD and you can be used by purchasing it as a group disk.
Although file access is slower than in high-speed storage areas, the available storage size is large and the cost is low, so it is recommended for “datafor which file access speed is not required” and “storage of files such as backup data.

- 10/80 -2024-10-10

TSUBAME4.0 User's Guide

3.2.3. Usage status of group disk
You can check usage status of specific group disk with t4-user-info disk group command.
$ t4-user-info disk group -g TESTGROUP/gs/bs/gs/fsgid group_namesize(TB) quota(TB)
file(M) quota(M) size(TB) quota(TB)
file(M) quota(M)---------------------------------------------------------------------------------------------------------------xxxx TESTGROUP
59.78
100
7.50
200
0.00
0
0.00
0

In this case, TESTGROUP purchased /gs/bs and used 60TB out of 100TB quota limit, its inode used 7.5M out ob 200M.

3.2.4 Confirmation of group disk usage
Group disks are purchased on a monthly basis.
If the amount of group disk purchased in the previous month is different from the current month (decrease), the actual usage may be greater than theamount of group disk purchased, depending on the usage situation.
Please use one of the following procedures to check whether the amount of group disk usage exceeds the purchased amount.
[How to check]
1. when checking on TSUBAME4.0 : Usage status of group disk
2. When checking on TSUBAME Portal : Confirmation of group disk usage

Info
Only group administrators and authorized group members can check on TSUBAME portal.

3.2.4.1 What to do when the group disk usage exceeds the reserved size
If the amount of group disk usage exceeds the purchased amount, the following two levels of restrictions will apply
1. prohibition of writing files that exceed the purchased amount (at the beginning of the month)
2. prohibition of access to anything on the group disk (specific timing)
In the case of condition 1, please delete files to keep the purchased amount within the limit, or change the purchased size of the group disk byreferring to Increase or decrease group disk capacity to exceed the current usage amount.
In case of condition 2., file deletion is not possible. Please change the purchased size of the group disk with reference to Increase or decrease groupdisk capacity to exceed the current usage.

3.3. CIFS access from inside campus
In TSUBAME4.0, you can access group disk from the Windows / Mac terminal in the university using the CIFS protocol.
\\gshs.t4.gsic.titech.ac.jp

You need TSUBAME4.0 account. The password is the one you set in the portal.
Note that "@TSUBAME" is needed subsequent your account when you use Windows.

User(TSUBAME4.0_ACCOUNT)@TSUBAME

Password(TSUBAME4.0_PASSWORD)/gs/bs, /gs/fs are displayed as t4_bs and t4_fs respectively. Access to your group disk beneath them.

- 11/80 -2024-10-10

TSUBAME4.0 User's Guide

Info
Home directory is not accessible with CIFS.

3.4. Local scratch area
Local scratch area is located in SSD on each compute node and available for workspace or temporarily area to read/write data rapidly.
By using the local scratch area, the fastest file access can be achieved without crossing over nodes.
Note, however, that the file transfer process from the group disk or home directory to the local scratch area must be performed at the beginning of thejob, which may result in slower execution time if the number of files to be transferred is large or the access frequency is low.
Files in the local scratch area are deleted at the end of the job, so users should explicitly save necessary files in the Home directory or group disk.
The capacity of local scratch area in each resource type is described here.

- 12/80 -2024-10-10

TSUBAME4.0 User's Guide

4. Software Environment
4.1. Switch User Environment
In this system, you can switch the compiler and application use environment by using the Environment Modules system.
For Environment Modules system, please refer to the man command or Environment Modules.

4.1.1. List available modules
You can check available modules with “module avail" or “module ava".
$ module avail

For each version available, please refer to following pages:
System software
Supported applictions

4.1.2. Module information
If you want to know information about a module, module watis MODULE command.
$ module whatis intel/2024.0.2-------------------------------- /apps/t4/rhel9/modules/modulefiles/compiler -------------------------------intel/2024.0.2: Intel oneAPI compiler 2024.0 and MKL

4.1.3. Load modules
Modules are loaded with module load MODULE command.
$ module load intel/2024.0.2

You need to use same modules in a job script as complied/build.

4.1.4. List loaded modules
module list displays currently loaded modules.
$ module list
Currently Loaded Modulefiles:
1) intel/2024.0.2
2) cuda/12.3.2

4.1.5. Unload modules
If you want to unload modules in use, type module unload MODULE .
$ module list
Currently Loaded Modulefiles:
1) intel/2024.0.2
2) cuda/12.3.2$ module unload cuda
$ module list
Currently Loaded Modulefiles:
1) intel/2024.0.2

4.1.6. Refresh module environment
module purge can refresh module environment. All modules currently loaded will be unloaded.
$ module list
Currently Loaded Modulefiles:- 13/80 -2024-10-10

TSUBAME4.0 User's Guide1) intel/2024.0.2
2) cuda/12.3.2$ module purge$ module list
No Modulefiles Currently Loaded.

4.2. Intel Compiler
In this system, you can use Intel compiler (oneAPI), AMD compiler (AOCC), NVIDIA compiler (NVIDIA HPC SDK) and GNU compiler.
Intel compiler's commands are listed below:
CommandLanguage

SyntaxFormer command

ifx

Fortran 77/90/95$ ifx [OPTION] source_file

ifort

icx

C

$ icx [OPTION] source_file

icc

icpx

C++

$ icpx [OPTION] source_file

icpc

When you use Intel compiler, load intel using module command. --help option shows various compiler options.

Info
icc and icpc commands are no loger supported in Intel oneAPI 2024. You should use icx and icpx instead. ifort is also suspected to decommit innear future, we recommend you to use ifx instead.

Info
Since Intel oneAPI 2024, the default standard for C++ has changed from C++14 to C++17. Therefore, syntax errors may occur. For more information, see
here.

- 14/80 -2024-10-10

TSUBAME4.0 User's Guide

4.2.1. Compiler options
The compiler options are listed below.
OptionDescription-O

Same as O2-O0

Disables all optimizations. Using for debugging,etc.

-O1

Affects code size and locality. Disables specific optimizations.

-O2

Default optimizations. Same as -O.
Enables optimizations for speed, including global code scheduling, software pipelining, predication.

-O3

Aggressive optimizations for maximum speed (, but does not guarantee higher performance).
Optimization including data prefetching, scalar replacement, loop transformations.

-xCORE-AVX512

Generates Intel Advanced Vector Extensions 512 (Intel AVX512), Intel AVX2, AVX, SSE4.2, SSE4.1, SSSE3, SSE3, SSE2, SSE
instructions for Intel processors. Optimizes for Intel processors with Intel AVX512 instruction set support.

-xCORE-AVX2

Generates Intel Advanced Vector Extensions 2 (Intel AVX2), Intel AVX, SSE4.2, SSE4.1, SSSE3, SSE3, SSE2, and SSE
instructions for Intel processors. Optimized for Intel processors supporting the Intel AVX2 instruction set.

-xSSE4.2

Generates Intel SSE4 high-efficiency and high-speed string processing instructions, Intel SSE4 vectorized compiler andmedia accelerator instructions, and Intel SSSE3, SSE3, SSE2, and SSE instructions for Intel processors. Optimizes for Intelprocessors supporting the Intel SSE4.2 instruction set.

-xSSSE3

Generates Intel SSSE3, SSE3, SSE2, and SSE instructions for Intel processors, optimized for Intel processors supporting
the Intel SSSE3 instruction set.

-qopt-report=nGenerates an optimization report. By default, the report is output to a file with an .optrpt extension, where n is the level ofdetail from 0 (no report) to 5 (most detailed). The default is 2.

-fp-model

Controls the semantics of floating-point operations. Disables optimizations that affect the precision of floating-point data

preciseand rounds intermediate results to the precision defined by the source.

-gThe -g option instructs the compiler to generate symbolic debugging information in the object file that increases the sizeof the object file.

-traceback

This option instructs the compiler to generate supplemental information in the object file so that when a fatal error occurs
at run-time, the source file traceback information can be displayed.
When a fatal error occurs, correlation information for the source file, routine name, and line number is displayed along withthe hex address of the call stack (program counter trace).
The map file and the hex address of the stack displayed when an error occurs can be used to determine the cause of theerror.
This option increases the size of the executable program.

4.2.2. Recommended optimization options
Recommended compile optimization options are listed below. The AMD EPYC 9654 in this system supports the Intel AVX512 instruction set, so the xCORE-AVX512 option can be specified. When -xCORE-AVX512 is specified, the compiler will analyze the source code and generate the optimal
AVX512, AVX2, AVX, and SSE instructions. The recommended optimization option is an aggressive and safe option. Optimization may change theorder of calculations and may introduce errors in the results.

- 15/80 -2024-10-10

TSUBAME4.0 User's Guide

Info
AVX512, adopted in Intel Xeon (Skylake and later), is now supported by AMD in the 4th generation EPYC of the Zen4 architecture, which is included in thissystem.

OptionDescription-O3

O2 optimization to enable more powerful loop transformations such as fusion, unroll and jam blocks, and IF statement
folding.

-xCORE-AVX512

Generates Intel Advanced Vector Extensions 512 (Intel AVX512), Intel AVX2, AVX, SSE4.2, SSE4.1, SSSE3, SSE3, SSE2, SSE
instructions for Intel processors. Optimizes for Intel processors with Intel AVX512 instruction set support.

If the above options worsen the performance of your program, reduce the level of optimization to -O2 or change the vectorization options. Also try thefloating point option if the results do not match.

4.2.3. Intel 64 architecture memory model
Create an executable binary using one of the following memory models
Memory model

Description

small ( -mcmodel=small )Code and data are limited to the first 2GB of address space so that all access to code and data is instruction pointer(IP) relative addressing.
If -mcmodel option is not specified, this is the default.

medium ( -Code is limited to the first 2GB of address space, but data is not. Code can be accessed via IP relative addressing,mcmodel=medium )but data must be accessed via absolute addressing.

large ( -mcmodel=large )Neither code nor data is restricted. Both code and data access uses absolute addressing.

IP relative addressing requires only 32 bits, while absolute addressing requires 64 bits. This affects code size and performance. (IP relative
addressing is slightly faster to access.)
When the total amount of common blocks, global data, and static data in a program exceeds 2 GB, the following error message is output at link time<some lib.a library>(some .o): In Function <function>:: relocation truncated to fit: R_X86_64_PC32 <some symbol>…………………
: relocation truncated to fit: R_X86_64_PC32 <some symbol>

In this case, compile/link with -mcmodel=medium and -shared-intel .
If you specify the medium or large memory model, you must also specify the -shared-intel compiler option to ensure that the appropriate dynamic
version of Intel's runtime libraries is used.

4.3. NVIDIA HPC SDK
Each command of NVIDIA HPC SDK (formerly PGI compiler) is as follows.
CommandLanguage

SyntaxFormer command

nvfortran

Fortran 77/90/95/2003/2008/2018$ nvfortran [OPTION] source_file

pgfortrannvc

C$ nvcc [OPTION] source_file

pgcc

nvc++

C++

$ nvc++ [OPTION] source_filepgc++

For more details about each command, see $ man nvcc .

- 16/80 -2024-10-10

TSUBAME4.0 User's Guide

To use it, load nvhpc with the module command. If you specify the --help option, a list of compiler options will be displayed.

4.4. AOCC
Each command of AMD Optimizing C/C++ and Fortran Compilers (AOCC) is as follows.
CommandLanguage

Syntaxflang (clang)

Fortran 95/2003/2008$ flang [OPTION] source_file

clang

C

$ clang [OPTION] source_fileclang-cpp (clang)C++$ clang-cpp [OPTION] source_file

To use it, load aocc with the module command. If you specify the --help option, a list of compiler options will be displayed.

4.5. GPU Environment
This system provides you GPU (NVIDIA H100 SXM5) environment in conjection with CPU.

4.5.1. Execution and debug for interactive job
The login nodes (login, login1, login2) do not have GPUs and can only compile and link. Also, execution of high-load programs on login nodes islimited.
You can run GPU codes with interactive and debug on compute nodes by batch system. Please refer Interactive job for more details.

4.5.2. Supported applications for GPU
GPU supported applications are listed below (as of 2024.4.1).
• ABAQUS 2024 --- Refer to ABAQUS Usage guide.
• ANSYS 2024R1 --- Refer to ANSYS Usage guide.
• AMBER 22up05 --- Refer to AMBER Usage guide.
• Mathematica 14 --- Refer to Mathematica Usage guide.
• MATLAB 2024 --- Refer to MATLAB Usage guide.
• Linaro forge(ex:Arm Forge) --- Refer to [並列プログラミング] in Lectures• NVIDIA HPC SDK --- Refer to NVIDIA HPC Usage guide.
For other applications, we will provide it sequentially.

4.5.3. MPI Environment with CUDAIMPI Environment with CUDA is available.
OpenMPI + gcc environment# load OpenMPI, GCC$ module load openmpi/5.0.2-gcc
Loading openmpi/5.0.2-gcc
Loading requirement: cuda/12.3.2

Info
The required related modules are loaded automatically.

- 17/80 -2024-10-10

TSUBAME4.0 User's Guide

OpenMPI + NVIDIA HPC SDK environment# load OpenMPI、NVIDIA HCP SDK$ module load openmpi/5.0.2-nvhpc
Loading openmpi/5.0.2-nvhpc

OpenMPI + Intel environment# load OpenMPI, Intel$ module load openmpi/5.0.2-intel
Loading openmpi/5.0.2-intel
Loading requirement: intel/2024.0.2 cuda/12.3.2

Info
The required related modules are loaded automatically.

4.6. Use containers
In TSUBAME4.0, Apptainer (Singularity) is available as a container environment for HPC.
An example of how to use Apptainer is shown below.

4.6.1. Build image
An example of the Apptainer image creation process is shown below. This example uses the latest Ubuntu Docker image.
[Options]• -nv : Use GPU• -B : Mounting a file system• -s : Build images in a sandbox format$ mkdir $HOME/apptainer$ cd $HOME/apptainer$ apptainer build -s ubuntu/ docker://ubuntu:latest
INFO:
Starting build...
Getting image source signatures
Copying blob 49b384cc7b4a done
Copying config bf3dc08bfe done
Writing manifest to image destination
Storing signatures
2024/05/29 13:07:49 info unpack layer: sha256:49b384cc7b4aa0dfd16ff7817ad0ea04f1d0a8072e62114efcd99119f8ceb9ed
2024/05/29 13:07:50 warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2024/05/29 13:07:50 warn xattr{$HOME/apptainer/build-temp-2088960457/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings
will be suppressed
INFO:
Creating sandbox directory...
INFO:
Build complete: ubuntu/$ cd ubuntu/$ mkdir gs apps
$ cd ../# Create directories to mount /gs and /apps

4.6.2. Shell start
The following is an example of how to start a shell with Apptainer. Here, the image created by Build image is used.
[Options]• -nv : Use GPU• -B : Mounting a file system• -f (--fakeroot) : Exercise root privileges on containers• -w (--writable) : Allows writing into the container- 18/80 -2024-10-10

TSUBAME4.0 User's Guide

Info
The -w (--writable) option must be specified.
If not specified, we have confirmed that the shell will not start properly.

$ cd $HOME/apptainer$ apptainer shell -B /gs -B /apps -B /home --nv -f -w ubuntu/
INFO:
User not listed in /etc/subuid, trying root-mapped namespace
INFO:
Using fakeroot command combined with root-mapped namespace
WARNING: nv files may not be bound with --writable
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount /bin/nvidia-smi [files]: /usr/bin/nvidia-smi doesn't exist in container
WARNING: Skipping mount /bin/nvidia-debugdump [files]: /usr/bin/nvidia-debugdump doesn't exist in container
WARNING: Skipping mount /bin/nvidia-persistenced [files]: /usr/bin/nvidia-persistenced doesn't exist in container
WARNING: Skipping mount /bin/nvidia-cuda-mps-control [files]: /usr/bin/nvidia-cuda-mps-control doesn't exist in container
WARNING: Skipping mount /bin/nvidia-cuda-mps-server [files]: /usr/bin/nvidia-cuda-mps-server doesn't exist in container
Apptainer> id
uid=0(root) gid=0(root) groups=0(root)

4.7.2.1. precautions when using fakeroot
When using the fakeroot function (--fakeroot) in Apptainer, the libc version must match between the host and container because Apptainer uses thehost's fakeroot as a bindmap.
If the host's libc library is newer than the corresponding library in the container, the fakeroot command may output an error that the GLIBC version is
missing. ( Reference:Fakeroot feature )Example :/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.33' not found (required by /.singularity.d/libs/faked)/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.34' not found (required by /.singularity.d/libs/faked)
fakeroot: error while starting the `faked' daemon.
/.singularity.d/libs/fakeroot: 1: kill: Usage: kill [-s sigspec | -signum |-sigspec] [pid | job]... orkill -l [exitstatus]

If the above error occurs, recreate the container so that the libc library versions of the host and container match.

- 19/80 -2024-10-10

TSUBAME4.0 User's Guide

5. Job Scheduler
The system's job scheduling uses Altair Grid Engine, which efficiently schedules single and parallel jobs according to priority and required resources.

5.1. Job Scheduling System Configuration
TSUBAME4.0 provides the following queue types/job types for different purposes.
Queue types

Job types

command

Resource types

Normal queue

Batch jobqsub

Selectable

Interactive job

qrsh

Selectable

Interactive job

iqrsh

Fixed

Interactive job queue[Queue types]• Normal queue
System resources are allocated and occupied for use in units of logically divided resource types.
This is usually used. Node reservation is also available.
• Interactive job queue
An interactive queue is an environment where the same resource is shared among users, making it less likely to fail to secure a node even whencongested, and providing a quick start to visualization and interactive programs.
The programs with intermittent processor usage, such as debuggers, visualizers, Jupyter Lab, are expected and do not use this service forprograms that dominate processors continuously.
Please be sure to read List of various limits before use.

Info
Interactive job queue is available free of charge only for intramural users (tgz-edu) and access card users.
See Interactive job queue for usage.

[Job types]• Batch job
Create and submit a job script. See Batch job for details.
• Interactive job
Execute a program or shell script interactively. Usage differs depending on the queue type.
When using "Normal queue", please refer to Interactive job.
When using "Interactive job queue", please refer to Interactive job queue.
[Resource types]• Use Normal queue
Please refer to Resource types for available resource types.
• Use Interactive job queue
Assigned resources number of physical CPU cores 24 cores, 96GB memory, 1MIG, but up to 12 people share the same resources.

5.1.1. Resource types
This system uses resource types in which compute nodes are logically divided to reserve system resources.

- 20/80 -2024-10-10

TSUBAME4.0 User's Guide

When submitting a job, specify how many resource types will be used (e.g., -l node_f=2 ). The available resource types are listed below.
Resource typePhysical CPU cores

Memory (GB)GPUs

Local scratch area (GB)node_f

192

768

4

1920

node_h

96

384

2

960

node_q

48

192

1

480

node_o

24

96

1/2

240

gpu_1

8

96

1

240

gpu_h

4

48

1/2

120

cpu_160

160

368

0

96

cpu_80

80

184

0

48

cpu_40

40

92

0

24

cpu_16

16

36.8

0

9.6

cpu_8

8

18.4

0

4.8

cpu_4

4

9.2

0

2.4

• "Physical CPU cores", "Memory (GB), "GPUs" are the available resources per resource type.
• Same resource type can be used with [Resource type]=[Num]. Different resource types combinations are not available.
• Maximum run time is 24 hours.
• TSUBAME4 has various limits such as "the number of jobs that can be executed at the same time" and "the total number of slots that can be
executed". (slots=Physical cores in each resoruce type x nodes number (same as slots of qstat))Current limits are shown below:
https://www.t4.gsic.titech.ac.jp/en/resource-limit
Note that it may change depends on usage any time.

5.2 Normal queue
5.2.1 Batch job
To run a job on this system, log in to the login node and execute qsub command.

5.2.1.1 Job submission flow
To submit a job, create and submit a job script. The submission command is qsub .
• Create a job script• Submit a job using qsub
• Status check using qstat• Cancel a job using qdel
• Check job result
The qsub command confirms billing information (TSUBAME points) and accepts jobs.

- 21/80 -2024-10-10

TSUBAME4.0 User's Guide

5.2.1.2. Creating job script
Here is a job script format:#!/bin/sh#$ -cwd#$ -l [Resource type] =[Number]#$ -l h_rt=[Maximum run time]#$ -p [Priority][Load relevant modules requried for the job]
[Your program]

Warningshebang( #!/bin/sh line) must be located at the first of the job script.

• [Load relevant modules requried for the job] Load required related modules that you need. For example, loading Intel compiler is as below:
module load intel• [Your program]
Execute your program Running a.out binary is like below:./a.out

Resource type can be set at command line, or in comment block ( #$ )at the top of job script file.
Make sure to set resource type and maximun run time, which are requirement to submit a job.

- 22/80 -2024-10-10

TSUBAME4.0 User's Guide

The main options for the qsub command are listed below.
OptionDescription-l [Resource type Name]

Specify the resource type.

=Number-l h_rt=[Maximum run

Specify the maximum run time (hours, minutes and seconds).

time] (Required)[[HH:]MM:]SS, HH:MM:S, MM:SS or SS-N

name of the job (Script file name if not specified))-oname of the standard output file-ename of the standard error output filename of the standard error output file-j yIntegrate standard error output into a standard output file.

-mWill send email when job ends or aborts. The conditions for the -m argument include:
a: mail is sent when the job is aborted.
b: mail is sent when the job begins.
e: mail is sent when the job ends.
It is also possible to combine like abe.
When a large number of jobs with mail option are submitted, a large amount of mail is also sent, heavy load isapplied to the mail server, and it may be detected as an attack and the mail from Tokyo Tech may be blocked. If you
need to execute such jobs, please remove the mail option or review the script so that it can be executed with one job.

-p (Premium Options)

Specify the job execution priority. If -4 or -3 is specified, a charge factor higher than -5 is applied. The setting values-5, -4, -3 correspond to the priority 0, 1, 2 of the charging rule.
-5: Standard execution priority. (Default)-4: The execution priority is higher than -5 and lower than -3.
-3 :Highest execution priority.
Note that all priority number to specify is negative value. Do not forget preceding minus sign.

-t

Submits a Array Job
specified with start-end[:step]-hold_jid

Defines the job dependency list of the submitted job.
The job is executed after the specified dependent job is finished.

-arSpecify the reserved AR ID when using the reserved node.

Note that -V option for passing environment variables in the job submission environment is not available in this system.

5.2.1.3. Job script examples
5.2.1.3.1. SINGLE JOB/GPU JOB

The following is an example of a job script created when executing a single job (job not parallelized) or GPU job.
For GPU job, please replace -l cpu_4=1 with -l gpu_1=1 and load required modules such as CUDA environment.
#!/bin/sh# run in current working directory#$ -cwd#$ -l cpu_4=1# maximum run time#$ -l h_rt=1:00:00#$ -N serial# load cuda module
module load cuda
# load Intel Compiler
module load intel./a.out- 23/80 -2024-10-10

TSUBAME4.0 User's Guide

5.2.1.3.2. SMP JOB

An example of a job script created when executing an SMP parallel job is shown below. Hyper-threading is enabled for compute nodes. Please
explicitly specify the number of threads to use.
#!/bin/sh#$ -cwd# node_f 1 node#$ -l node_f=1#$ -l h_rt=1:00:00#$ -N openmp
module load cuda
module load intel# 192 therads per node
export OMP_NUM_THREADS=192./a.out

5.2.1.3.3. MPI JOB

An example of a job script created when executing an MPI parallel job is shown below. Please specify an MPI environment according to the MPI
library used by you for MPI jobs as follows. For OpenMPI, to pass library environment variables to all nodes, you need to use -x LD_LIBRARY_PATH.
Intel MPI#!/bin/sh#$ -cwd# node_f x 4#$ -l node_f=4#$ -l h_rt=1:00:00#$ -N flatmpi
module load cuda
module load intel# Intel MPI
module load intel-mpi# 8 process/node, all MPI process is 32
mpiexec.hydra -ppn 8 -n 32 ./a.out

OpenMPI#!/bin/sh#$ -cwd# node_f x 4#$ -l node_f=4#$ -l h_rt=1:00:00#$ -N flatmpi# Load OpenMPI: Intel compiler, CUDA are loaded automaticallymodule load openmpi/5.0.2-intel# 8 process/node, all MPI process is 32
mpirun -npernode 8 -n 32 -x LD_LIBRARY_PATH ./a.out

5.2.1.3.4. HYBRID PARALLEL (HYBRID, MPI+OPENMP)

An example of a job script created when executing a process/thread parallel (hybrid) job is shown below. Please specify an MPI environment
according to the MPI library used by you for MPI jobs as follows. For OpenMPI, to pass library environment variables to all nodes, you need to use =-x
LD_LIBRARY_PATH`.
Intel MPI#!/bin/sh#$ -cwd# node_f x 4#$ -l node_f=4#$ -l h_rt=1:00:00#$ -N hybrid
module load cuda
module load intel
module load intel-mpi# 192 threads per node
export OMP_NUM_THREADS=192# 1 MPI process per node, all MPI process is 4
mpiexec.hydra -ppn 1 -n 4 ./a.out

OpenMPI#!/bin/sh#$ -cwd- 24/80 -2024-10-10

TSUBAME4.0 User's Guide# node_f x 4#$ -l node_f=4#$ -l h_rt=1:00:00#$ -N hybrid# Open MPI: Intel compiler, CUDA are loaded automaticallymodule load openmpi/5.0.2-intel# 192 threads per node
export OMP_NUM_THREADS=192# 1 MPI process per node, all MPI process is 4
mpirun -npernode 1 -n 4 -x LD_LIBRARY_PATH ./a.out

5.2.1.4. Job submission
Job is queued and executed by specifying the job submission script in the qsub command. You can submit a job using qsub as follows.
$ qsub -g [TSUBAME group] SCRIPTFILE

OptionDescription-gSpecify the TSUBAME group name.
Please add as qsub command option, not in script.

-q prior

Subscription job.
Wait one hour at most until execution.

5.2.1.5. Job status
The qstat is a job status display command.
$ qstat [option]

The options of qstat is listed below.
OptionDescripion-r

Displays job resource infromation-j [job-ID]

Displays additional information about the job

Here is an example of qstat:$ qstatjob-IDprior nameuser
statesubmit/start at queuejclass slotsja-task-ID----------------------------------------------------------------------------------- 25/80 -2024-10-10

TSUBAME4.0 User's Guide

307 0.55500 sample.sh testuser r 02/12/2023 17:48:10(snip)

all.q@r8n1A.default32

Item

Description

Job-ID

Job-ID number

prior

Priority of job

name

Name of the job

user

ID or user who submitted job

stateState of the jobr runnig
qw waiting in the queue
h on holdd deletingt a transition like during job-starts suspended
S suspended by the queue
T has reached the limit of the tailE errorRq Rrescheduled and waiting for run
Rr rescheduled and running

submit/start at

Submit or start time and date of the job

queue

queue namejclassjob class nameslots

The number of slot the job is taking(slot=physical cores in each resource type x node number)ja-task-ID

Array job task-id

5.2.1.6. Deleting job
To delete your job, use the qdel command.
$ qdel [Job-ID]

An example of deleting job is shown as below:$ qstatjob-IDprior nameuser
statesubmit/start at queuejclass slotsja-task-ID---------------------------------------------------------------------------------307 0.55500 sample.sh testuser r 02/12/2023 17:48:10 all.q@r8n1A.default32$ qdel 307
testuser has registered the job 307 for deletion$ qstatjob-IDprior nameuser
statesubmit/start at queuejclass slotsja-task-ID----------------------------------------------------------------------------------

5.2.1.7. Job result
The standard output of AGE jobs is stored in the file "script file name.o job ID" in the directory where the job was executed. Also, the standard error
output is "script file name.eJob ID".

5.2.1.8. Array job
An array job is as a function to parameterize and execute the operation contained in the job script repeatedly.

- 26/80 -2024-10-10

TSUBAME4.0 User's Guide

Each job executed in an array job is called a task and is managed by a task ID. A job ID that does not specify a task ID has the entire task ID as its
range.

Info
Each task in an array job is scheduled as a separate job, so the schedule wait time is proportional to the number of tasks.
If each task is short or the number of tasks is large, it is strongly recommended to reduce the number of tasks by grouping several tasks together.
Example: 10000 tasks are combined into 100 tasks that each process 100 tasks.

The task number can be specified as an option of the qsub command or defined in a job script. Submission options are specified as -t(start
number)-(end number):(step size) . If the step size is 1, it can be omitted. An example is shown below.
# describe below in a job script#$ -t 2-10:2

The above example (2-10:2) specifies a start number of 2, an end number of 10, a step size of 2 (one skipped index), and consists of five tasks withtask numbers 2, 4, 6, 8, and 10.
The task number for each task is set in an environment variable named $SGE_TASK_ID, so this environment variable can be used in the job script toenable parameter study. The result file will be output with the job name followed by the task ID.
Also, if you want to remove a specific task ID before/while it is running, use qdel's -t option as follows.
$ qdel[Job-ID] -t [Task-ID]

5.2.2. Interactive job
The system's job scheduler has the ability to run programs and shell scripts interactively. To run an interactive job, use the qrsh command and specify
the resource type and elapsed time with -l. After submitting a job with qrsh, a command prompt will be returned when the job is dispatched. The
following shows how to use an interactive job.
$ qrsh -g [TSUBAME group] -l [Resource type]=[number] -l h_rt=[maximum run time]
Directory: /home/N/username(job start time)username@rXnY:~> [commands to run]username@rXnY:~> exit

If the -g option group is not specified, up to two resource types, up to 10 minutes of elapsed time, and a "trial run" with a priority of -5 will be
performed.
Example specifying resource type node_f, 1 node, elapsed time 10 minutes$ qrsh -g [TSUBAME group] -l node_f=1 -l h_rt=0:10:00
Directory: /home/N/username(job start time)username@rXnY:~> [commands to run]username@rXnY:~> exit

Enter exit can stop the interactive job.

5.2.2.1. X forwarding using interactive nodes
To perform X forwarding directly from a node connected by qrsh, please follow the procedure below.
1. Enable X forwarding and connect to login node with ssh.
2. Execute qrsh command with X11 forwarding like the following example.
Example)The example shows a 2-hour job with resource type cpu_4 and 1 node.
The assigned node is dispatched by the schedular from the free nodes at the time the command is executed, so you cannot explicitly specify a node.

- 27/80 -2024-10-10

TSUBAME4.0 User's Guide# Execution of qrsh command$ qrsh -g [TSUBAME group] -l cpu_4=1 -l h_rt=2:00:00
username@rXnY:~> module load [application modules to load]username@rXnY:~> [command to run X11 application]
username@rXnY:~> exit

Info
For X forwarding using interactive nodes, Open OnDemand is also available.

5.4.2.2 Connecting to the network applications
If you need to operate an application using a Web browser or other means, you can use SSH port forwarding to access the application from a Web
browser at local.
(1) Get hostname of interactive node connected by qrsh
$ qrsh -g tga-hpe_group00 -l cpu_4=1 -l h_rt 0:10:00$ hostname
r1n1$ [execute the program that requires Web browser]

After starting an interactive job with qrsh, get the hostname of the machine. In the above example, r1n1 is the hostname. There is nothing to do
anymore in the console, but keep it until the work by the application is finished.
(2) Connect with SSH port forwarding enabled from the console from which you are connecting to ssh. (not on the login node or interactive job)ssh -i /path/to/key -l username -L 8888::<> login.t4.gsic.titech.ac.jp

The network port of the application to be connected is different for each application. For details, please refer to the documentation of each applicationor the application's startup message.

Tips
SSH port forwarding settings depend on the SSH console (or terminal software) used to SSH into TSUBAME4. For details, please check the manual ofeach SSH console or refer to the FAQ.

(3) Connect to the application with a web browser.
Launch a web browser (Microsoft Edge, Firefox, Safari, etc.) on the console at hand and go to http://localhost:8888/.

5.2.3. Trial run
InfoThis feature is available only for TSUBAME account holders. It is designed mainly for Tokyo Tech users who can sign up by themselves.

TSUBAME provides the "trial run" feature, in which users can execute jobs without consuming points, for those who are anxious whether TSUBAME
applies to their research or not. To use this feature, submit jobs without specifying a group via -g option. In this case, the job is limited to 2 nodes, 10
minutes of running time, and priority -5 (worst).

Warning
The trial run feature is only for testing whether your program works or not. Do not use it for actual execution for your research and measurement.
It does not mean that you can execute jobs freely without charge if the job size meets the limitation written in above.

- 28/80 -2024-10-10

TSUBAME4.0 User's Guide

The trial run function is provided that users who are not sure whether they can use TSUBAME for their own research can check the operation before
purchasing points. Please do not use the trial run to perform calculations that deviate significantly from these purposes or that directly lead toresearch results.
Please consider using interactive queue if you wish to perform small calculations for educational purposes in class.
For trial runs, the following restrictions apply to the amount of resources

Maximum number of the resource type specified

2

Maximum run time

10 minutes

number of concurrent runs

1

Resource typeno limitation

For Trial run, it is necessary to run a job without specifying a TSUBAME group. Note that the points are consumed when you submit a job with the
TSUBAME group.

5.2.4. Subscription job
Submittin a job for compute node subscription requires -q prior option. Other options are same as the other jobs.
$ qsub -q prior -g [TSUBAME group] SCRIPTFILE

For more details about compute node subscription, check here.

Warning
Even if a job for the subscription group, note that if -q prior is not specified, the job will be processed as a pay-as-you-go job.

5.2.5. Reserving compute nodes
It is possible to execute jobs exceeding 24 hours and/or 72 nodes by reserving computation nodes.
Steps for reservation is as follows.
• Make a reservation from TSUBAME portal• Check reservation status, cancel a reservation from TSUBAME portal• Submit a job using qsub for reserved node
• Cancel a job using qdel
• Check job result• Check the reservation status and AR ID from the command line
Please refer to TSUBAME Portal User's Guide "Reserving compute nodes" on reservation from the portal, confirmation of reservation status andcancellation of the reservation.
When the reservation time is reached, the job can be executed in the reservation group account. An example of job submission specifying an AR ID,
which is a reservation ID, is shown below.
• Submitting a job to a reserved node with qsub
$ qsub -g [TSUBAME group] -ar [AR ID] SCRIPTFILE• Submitting an interactive job to a reserved node with qrsh
$ qrsh -g [TSUBAME group] -l [Resource type]=[number] -l h_rt=[maximum run time] -ar [AR ID]- 29/80 -2024-10-10

TSUBAME4.0 User's Guide

The resource types available for reserved execution are node_f, node_h, node_q, and node_o. Other resource types are not available for reservation.
The qstat command is used to check the status of a job after it has been submitted, and the qdel command is used to delete a job.
Also, the format of the script is the same as that of the normal execution.
Use t4-user-info compute ar to check the reservation status and AR ID from the command line.
xxxxx@login1:~> t4-user-info compute arar_id
uid user_namegid group_namestatestart_date
end_date
time_hour node_countpoint return_point------------------------------------------------------------------------------------------------------------------------------------------------------1320 2005 A2901247
2015 tga-red000r2023-01-29 12:00:00 2023-01-29 13:00:00
1
1
18000
0
1321 2005 A2901247
2015 tga-red000r2023-01-29 13:00:00 2023-01-29 14:00:00
1
1
18000
0
1322 2005 A2901247
2015 tga-red000w
2023-01-29 14:00:00 2023-02-02 14:00:00
96
11728000
1728000
1323 2005 A2901247
2015 tga-red000r2023-01-29 14:00:00 2023-02-02 14:00:00
96
11728000
1728000
1324 2005 A2901247
2015 tga-red000r2023-01-29 15:00:00 2023-01-29 16:00:00
1
1730600001341 2005 A2901247
2015 tga-red000w
2023-02-25 12:00:00 2023-02-25 13:00:00
1
181620001620003112 2004 A2901239
2349 tgz-trainingr2023-04-24 12:00:00 2023-04-24 18:00:00
6
20
54000003113 2004 A2901239
2349 tgz-trainingr2023-04-25 12:00:00 2023-04-25 18:00:00
6
20
54000003116 2005 A2901247
2015 tga-red000r2023-04-18 17:00:00 2023-04-25 16:00:00
167
1
3006000
03122 2005 A2901247
2014 tga-blue000r2023-04-25 08:00:00 2023-05-02 08:00:00
168
51512000003123 2005 A2901247
2014 tga-blue000r2023-05-02 08:00:00 2023-05-09 08:00:00
168
53780000
03301 2005 A2901247
2015 tga-red000r2023-08-30 14:00:00 2023-08-31 18:00:00
28
150400003302 2005 A2901247
2009 tga-green000r2023-08-30 14:00:00 2023-08-31 18:00:00
28
150400003304 2005 A2901247
2014 tga-blue000r2023-09-03 10:00:00 2023-09-04 10:00:00
24
143200003470 2005 A2901247
2014 tga-blue000w
2023-11-11 22:00:00 2023-11-11 23:00:00
1
1
4500
4500
4148 2004 A2901239
2007 tga-hpe_group00w
2024-04-12 17:00:00 2024-04-12 18:00:00
1
1
4500
4500
4149 2005 A2901247
2015 tga-red000w
2024-04-12 17:00:00 2024-04-13 17:00:00
24
11080001080004150 2004 A2901239
2007 tga-hpe_group00w
2024-04-12 17:00:00 2024-04-12 18:00:00
1
1
4500
4500
------------------------------------------------------------------------------------------------------------------------------------------------------total :818
9728507500
3739500

Use t4-user-info compute ars to check the availability of appointments for the current month from the command line.

5.2.6. SSH login to compute nodes
Nodes that have been submitted jobs with resource type node_f can be logged in directly via ssh. The secured node can be checked by the following
procedure.
$ qstat -j 1463==============================================================
job_number:1463(snip)
exec_host_list1:
r8n3:28, r8n4:28<-- assigned nodes: r8n3, r8n4
(snip)

Info
When you ssh into a compute node, the GID of the sshed process is set to tsubame-users(2000), so you cannot see your job process running immediately
after ssh, except for a trial run, and you cannot attach to it with gdb. To make it visible, execute the following with the name of the group that executed thejob after ssh.
newgrp <groupname>orsg <groupname>5.3. Interactive job queue
An interactive job queue is an environment where the same resource is shared among users, making it less likely to fail to secure a node even whencongested, and providing a quick start to visualization and interactive programs.
Submitting jobs to the interactive queue is as follows- 30/80 -2024-10-10

TSUBAME4.0 User's Guide

Info
Interactive job queue is available free of charge only for intramural users (tgz-edu) and access card users.
To use interactive job queue free of charge, please submit a job without the -g [TSUBAME group] option.
Please note that you will be charged for the target group if the -g option is specified.

iqrsh -g [TSUBAME group] -l h_rt=<maximum run time>

Note that CPU/GPU over-commitment is allowed in the interactive job queue.
Limit for nteractive job queue is descrived here.

5.4. Storage use on Compute Nodes
5.4.1. Local scratch area
SSD on each compute node can be used as a local scratch area.
The local scratch area is set to /local/${JOB_ID} (or /local/${JOB_ID}${SGE_TASK_ID} for array jobs). It can be referenced by specifying the pathof the work area in the job script.
Since the local scratch area is a separate area for each compute node and is not shared, input and output files from within the job script must be
staged on the local host.
The following example copies the input data set from the home directory to the local scratch area and returns the results to the home directory when
there is only one compute node used. (Multiple nodes are not supported.)#!/bin/sh# copy input data to the scratch
cp -rp $HOME/datasets /local/${JOB_ID}# run your program that uses input/output data./a.out /local/${JOB_ID}/datasets /local/${JOB_ID}/results# copy the result back to your home directory
cp -rp /local/${JOB_ID}/results $HOME/results

Tips/local/${JOB_ID} will be deleted when the job completes.

- 31/80 -2024-10-10

TSUBAME4.0 User's Guide

6. Commercial Applications
Under the license agreement, users who can use commercial applications is limited.
Users other than "1. Student/Staff ID" who belong to Tokyo Tech can only use the following commercial applications.
• Gaussian/Gauss View
• AMBER (Only users affiliated with academic institutions)• Intel oneAPI Compiler• NVIDIA HPC SDK Compiler• Linaro forge(ex:Arm Forge)

Info
Each application fee is required for the use of some commercial applications. For more details, please refer to Fare Overview Commercial Applications(Partially charged in TSUBAME4.0).

Commercial applications are listed below:
Software name

Description

ANSYS

Analysis Software

ANSYS/Fluent

Analysis Software

ANSYS/LS-DYNA

Analysis Software

ABAQUS

Analysis Software

ABACUS CAE

Analysis Software

Gaussian

Quantum chemistry calculation program

GaussView

Quantum chemistry calculation program Pre-Post tool

AMBER

Molecular dynamics calculation program

Materials Studio

Chemical Simulation Software

Discovery Studio

Chemical Simulation Software

Mathematica

Mathematical Processing Software

COMSOL

Analysis Software

Schrodinger

Chemical Simulation Software

MATLAB

Numerical calculation software

VASP

Quantum-mechanical molecular dynamics program

Linaro forge(ex:Arm Forge)

DebuggerIntel oneAPI Compiler

Compiler, Development tool

NVIDIA HPC SDK Compiler

Compiler, Development tool

InfoFor "module" command, please refer to Switch User Environment.

- 32/80 -2024-10-10

TSUBAME4.0 User's Guide

6.1. ANSYS
The procedure for using GUI is shown below.
$ module load ansys$ launcher

The procedure for using CLI is shown below.
$ module load ansys$ mapdl

Following command is also available instead of mapdl.
（As of ANSYS24.1. It depends on the version.）$ ansys241

Enter exit , then the program ends.
Specify input file, you can run it interactively.
Example-1$ mapdl [options] < inputfile > outputfile
Example-2$ mapdl [options] -i inputfile -o outputfile

To use the batch queue system, create a shell script and run it in the CLI as follows$ qsub sample.sh

Script example：MPI parallel processing#!/bin/bash#$ -cwd#$ -V#$ -l node_f=2#$ -l h_rt=0:10:0
module load ansys
mapdl -b -dis -np 56 < inputfile > outputfile- 33/80 -2024-10-10

TSUBAME4.0 User's Guide

Script example：With GPU#!/bin/bash#$ -cwd#$ -V#$ -l node_f=1#$ -l h_rt=0:10:0. /etc/profile.d/modules.shmodule load ansys
mapdl -b -dis -np 28 -acc nvidia -na 4 < inputfile > outputfile

The license usage of ANSYS can be checked as below:$ lmutil lmstat -S ansyslmd -c *****@kvm5.ini.t4.gsic.titech.ac.jp:*****@kvm6.ini.t4.gsic.titech.ac.jp:*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/ansys_inc/t4-license
The port number is changed in mid April every year.

6.2. ANSYS/Fluent
ANSYS/Fluent is a thermal fluid analysis application. The usage procedure is shown below.
GUI startup procedure is shown below.
$ module load ansys$ fluent- 34/80 -2024-10-10

TSUBAME4.0 User's Guide

CLI startup procedure is shown below.
$ module load ansys$ fluent -g

Entering exit ends the program.
To run interactively using the journal file, execute the command as follows- 35/80 -2024-10-10

TSUBAME4.0 User's Guide

journal file is fluentbench.jou, 3D$fluent 3d -g -i fluentbench.jou

To use the batch queue system, create a shell script and run it in the CLI as follows$ qsub sample.sh

Script example：MPI Parallel processing (node_f)#!/bin/bash#$ -cwd#$ -V#$ -l node_f=2#$ -l h_rt=0:10:0

module load ansys
JOURNAL=journalfile
OUTPUT=outputfile
VERSION=3d
fluent -mpi=intel -g ${VERSION} -cnf=${PE_HOSTFILE} -i ${JOURNAL} > ${OUTPUT} 2>&1

Script example：MPI Parallel processing (node_h)#!/bin/bash#$ -cwd#$ -V#$ -l node_h=1#$ -l h_rt=0:10:0

module load ansys
JOURNAL=journalfile
OUTPUT=outputfile
VERSION=3d
fluent -ncheck -mpi=intel -g ${VERSION} -cnf=${PE_HOSTFILE} -i ${JOURNAL} > ${OUTPUT} 2>&1

Since settings across resources are not possible when using anything other than node_f , please use #$ -l {resource name}=1 (for example, #$ l node_h=1 for node_h ) and include the -ncheck option in the command.

The license usage of ANSYS/Fluent can be checked as below:$ lmutil lmstat -S ansyslmd -c *****@kvm5.ini.t4.gsic.titech.ac.jp:*****@kvm6.ini.t4.gsic.titech.ac.jp:*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/ansys_inc/t4-license
The port number is changed in mid April every year.

6.3. ANSYS/LS-DYNA
6.3.1. ANSYS/LS-DYNA overview
LS-DYNA is a program that analyzes the large deformation behavior of structures with an explicit time history, and is a highly reliable program with aworld-class installation record in these fields, demonstrating its power in crash/impact analysis, drop analysis, plastic forming analysis, andpenetration/crack/fracture analysis.

6.3.2. Launching ANSYS/LS-DYNA
ANSYS/LS-DYNA is used in batch jobs. An example of a batch script is shown below.
Please read and execute the script according to the version you wish to use.

- 36/80 -2024-10-10

TSUBAME4.0 User's Guide

Script example：MPP single-precision version#!/bin/bash#$ -cwd#$ -V#$ -l node_h=1#$ -l h_rt=5:00:0

module load ansys intel-mpi
export dynadir=/apps/t4/rhel9/isv/ansys_inc/v241/ansys/bin/linx64
export exe=$dynadir/lsdyna_sp_mpp.eexport dbo=$dynadir/lsl2a_sp.e
export NCPUS=4
export INPUT=$base_dir/sample/airbag_deploy.k
mpiexec -np ${NCPUS} ${exe} i=${INPUT}${dbo} binout*

Script example ：MPP double-precision edition#!/bin/bash#$ -cwd#$ -V#$ -l node_h=1#$ -l h_rt=5:00:0
module load ansys intel-mpi
export dynadir=/apps/t4/rhel9/isv/ansys_inc/v241/ansys/bin/linx64
export exe=$dynadir/lsdyna_dp_mpp.eexport dbo=$dynadir/lsl2a_dp.e
export NCPUS=4
export INPUT=$base_dir/sample/airbag_deploy.k
mpiexec -np ${NCPUS} ${exe} i=${INPUT}${dbo} binout*

Scripts should be modified to suit the user's environment. In the example script above, the input file is specified as INPUT=inputfile in the shell script.

6.4. ABAQUS
The procedure for interactive use is shown below.
$ module load abaqus$ abaqus job=inputfile [options]

To use the batch queue system, create a shell script and run it in the CLI as follows$ qsub sample.sh

Scrip example：MPI Parallel processing#!/bin/bash#$ -cwd#$ -V#$ -l node_o=1#$ -l h_rt=0:10:0

module load abaqus# ABAQUS settings.
INPUT=s2a
ABAQUS_VER=2024
ABAQUS_CMD=abq${ABAQUS_VER}
SCRATCH=${TMPDIR}
NCPUS=4${ABAQUS_CMD} interactive \job=${INPUT} \cpus=${NCPUS} \scratch=${SCRATCH} \mp_mode=mpi > ${INPUT}.`date '+%Y%m%d%H%M%S'`log 2>&1

- 37/80 -2024-10-10

TSUBAME4.0 User's Guide

6.5. ABAQUS CAE
The procedure for using ABAQUS CAE is shown below.
$ module load abaqus$ abaqus cae

Click File > Exit on the menu bar to exit.

6.6. Gaussian
The interactive usage procedure is shown below.
$ module load gaussian/revision$ g16 inputfile

Specify the revision you are using for revision; for Gaussian16 Rev.C02, it is as follows$ module load gaussian/16C2_cpu$ g16 inputfile

To use the batch queue system, create a shell script and run it in the CLI as follows$ qsub sample.sh

Script example：Intra-node parallel processing
Sample scripts for calculating structural optimization and vibrational analysis (IR+Raman intensity) of Glycine.
The calculation can be performed by placing the following files, glycine.sh and glycine.gjf, in the same directory and executing the following
command. After calculation, glycine.log and glycine.chk are generated.

- 38/80 -2024-10-10

TSUBAME4.0 User's Guide

The results of the analysis are explained in GaussView.
$ qsub glycine.shglycine.sh#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -V

module load gaussian
g16 glycine.gjf

glycine.gjf
chk=glycine.chk
cpu=0-191<--Not necessary if the module that automatically sets the environment variables GAUSS_CDEF/GAUSS_GDEF is loaded.
mem=700GB
P opt=(calcfc,tight,rfo) freq=(raman)
glycine Test Job
2
N
H
C
C
H
H
O
O
H

0
0
0
0
0
0
0
0
0

-2.15739574-1.15783574-2.84434974-1.83982674-3.46918274-3.46918274-0.63259574-2.22368674-2.68286796-1.69517043-1.72483643-0.41935843
0.72406557-0.34255543-0.34255543
0.49377357
1.89158057-2.54598119-0.01896033 H-0.01896033 H-0.01896033 H-0.01896033 H-0.90878333 H
0.87086267 H-0.01896033 H-0.01896033 H-0.01896033 H

1 2 1.0 3 1.0 9 1.0
2
3 4 1.0 5 1.0 6 1.0
4 7 1.5 8 1.5
5
6
7
8
9

6.7. GaussView
GaussView is an application that visualizes Gaussian results.
The procedure for using GaussView is shown below.
$ module load gaussian gaussview$ gview.exe- 39/80 -2024-10-10

TSUBAME4.0 User's Guide

Click File > Exit from the menu bar to exit.
Example: glycine.log
The following explanation uses as an example the analysis of a result file obtained by executing the script shown as a sample in the Gaussiansection.
$ module load gaussian gaussview$ gview.exe glycine.log

You can check the analysis results from Result.
You can see a summary of the calculation in Result>Summary, the charge information in Result>ChageDistribution..., and the results of the vibration
analysis in Vibration....
In the sample, vibration analysis is performed, so you can see the vibration from StartAnimation in the Vibration dialog.

- 40/80 -2024-10-10

TSUBAME4.0 User's Guide

6.8. AMBER
AMBER was originally developed for molecular dynamics calculations of proteins and nucleic acids, but recently parameters for sugars have also
been developed, making it an increasingly useful tool for chemical and biological research. If you wish to use AMBER in your own research, you
should carefully study the manual and related papers to understand the limitations of the models and theories employed by AMBER, as well as thescope of application. Currently, AMBER does not allow unlimited copying of its source code, but it is possible to use it internally at Tokyo Tech, so you
can incorporate further developed methods based on it.
Please read the following as appropriate for the version you wish to use.
The following is the procedure for interactive sequential processing.
$ module load amber$ sander [-O|A] -i mdin -o mdout -p prmtop -c inpcrd -r restrt

The following is the procedure for using parallel processing (sander.MPI) in an interactive manner.
$ module load amber$ mpirun -np -[parallel number] -x LD_LIBRARY_PATH sander.MPI [-O|A] -i mdin -o mdout -p prmtop -c inpcrd -r restrt

The following is the procedure for using GPU sequential processing (pmemd.cuda) in interactive mode.
$ module load amber$ pmemd.cuda [-O] -i mdin -o mdout -p prmtop -c inpcrd -r restrt

The following is the procedure for using GPU parallel processing (pmemd.cuda.MPI) in interactive.
$ module load amber$ mpirun -np -[parallel number] -x LD_LIBRARY_PATH pmemd.cuda.MPI [-O] -i mdin -o mdout -p prmtop -c inpcrd -r restrt- 41/80 -2024-10-10

TSUBAME4.0 User's Guide

The following is the procedure for using the system in the case of a batch queue system.
$ qsub parallel.shScript example：CPU Parallel processing#!/bin/bash#$ -cwd#$ -l node_f=2#$ -l h_rt=0:10:00#$ -V
export NSLOTS=56
in=./mdin
out=./mdout_para
inpcrd=./inpcrd
top=./top
cat <<eof > $in
Relaxtion of trip cage using&cntrlimin=1,maxcyc=5000,irest=0, ntx=1,
nstlim=10, dt=0.001,
ntc=1, ntf=1, ioutfm=1
ntt=9, tautp=0.5,
tempi=298.0, temp0=298.0,
ntpr=1, ntwx=20,
ntb=0, igb=8,
nkija=3, gamma_ln=0.01,
cut=999.0,rgbmax=999.0,
idistr=0/
eof

module load amber
mpirun -np $NSLOTS -x LD_LIBRARY_PATH \sander.MPI -O -i $in -c $inpcrd -p $top -o $out < /dev/null/bin/rm -f $in restrt

Script example：GPU Parallel processing#!/bin/bash#$ -cwd#$ -l node_f=2#$ -l h_rt=0:10:0#$ -V
export NSLOTS=56
in=./mdin
out=./mdout
inpcrd=./inpcrd
top=./top
cat <<eof > $in
FIX (active) full dynamics ( constraint dynamics: constant volume)&cntrlntx = 7,
irest = 1,
ntpr = 100,
ntwx = 0,
ntwr = 0,
ntf = 2,
ntc = 2,
tol = 0.000001,
cut = 8.0,
nstlim = 500, dt = 0.00150,
nscm = 250,
ntt = 0,lastist = 4000000,
lastrst = 6000000,/
eof

module load amber
mpirun -np $NSLOTS -x LD_LIBRARY_PATH \pmemd.cuda.MPI -O -i $in -c $inpcrd -p $top -o $out < /dev/null/bin/rm -f $in restrt- 42/80 -2024-10-10

TSUBAME4.0 User's Guide

6.9. Materials Studio
6.9.1. License connection setting
BIOVIA > Licensing > License Administrator x.x.xx from the Windows [Start menu] with system administrator privileges.

Click [Connections] -[Set] , and open "Set License Server" dialog.

Select Redundant Server and type each host name and a port number.

Host namekvm5.ini.t4.gsic.titech.ac.jp

Host namekvm6.ini.t4.gsic.titech.ac.jp

Host nameldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is seen after you purchase the application with Application distribution for Labs. They are described in the install manual.
The port number is changed in mid April every year.

- 43/80 -2024-10-10

TSUBAME4.0 User's Guide

If the Server Status displays "Connected", the setting is complete.
In order to use Materials Studio, a connection must be established to the license server on two or more hosts.

6.9.2. License Usage Status
6.9.2.1. How to check on Windows
BIOVIA > Licensing > License Administrator x.x.xx > Utilities (FLEXlm LMTOOLs) from the Windows [Start menu] .
Open [Service/License File] tab and slect [Configulation using License File] .
Make sure that MSI_LICENSE_FILE is displayed.

Open [Server Status] tab, click [Perform Status Enqurity] and you can see usage status of the license.
If you want to display only specific licenses, enter the license name that you want to display in [Individual Feature] and execute [Perform Status
Enqurity].

6.9.2.2. How to check on login nodes
oExecute the following command to view the usage status.
$ lmutil lmstat -S msi -c *****@kvm5.ini.t4.gsic.titech.ac.jp,*****@kvm6.ini.t4.gsic.titech.ac.jp,*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/materialsstudio/t4-license
The port number is changed in mid April every year.

6.9.3. Launching Materials Studio
On Windows that Materials Studio installed, go to Start menu and click BIOVIA > Materials Studio 2024 to launch it.

- 44/80 -2024-10-10

TSUBAME4.0 User's Guide

6.10. Discovery Studio6.10.1. How to set up a license connection
From the Start menu, run BIOVIA > Licensing > License Administrator X.X.X as administrator.

Open [Connections] and click [Set] to open the Set License Server dialog.

Check Redundant servers, enter the host name and port number as shown below, and click OK.

Host namekvm5.ini.t4.gsic.titech.ac.jp

Host namekvm6.ini.t4.gsic.titech.ac.jp

Host nameldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is seen after you purchase the application with Application distribution for Labs. They are described in the install manual.
The port number is changed in mid April every year.

- 45/80 -2024-10-10

TSUBAME4.0 User's Guide

If the Server Status shows "Connected", the setup is complete.
In order to use Discovery Studio, a connection must be established to at least two license server hosts.

6.10.2. How to check license usage
6.10.2.1. How to check on Windows
Run BIOVIA > Licensing > License Administrator X.X.X > Utilities (FLEXlm LMTOOLs) from Windows start menu.
Open [Service/License File] tab, select [Configulation using License File].
Make sure MSI_LICENSE_FILE is displayed.

Open [Server Status] tab, click [Perform Status Enqurity] then the staus of license usage is shown.
If you want to check only specific license, input the license name into [Individual Feature], then run [Perform Status Enqurity].

6.10.3. Launching Discovery Studio
On Windows that Discovery Studio is installed, click BIOVIA > Discovery Studio 2024 64-bit Client from Start menu.

6.11. Mathematica
Info
Starting with Mathematica version 14.1, the application startup command names have changed.
Please refer to the instructions according to the version you are using.
To check the version that is enabled by default and the list of available versions, execute the following command.
$ module avail mathematica
The underlined versions in the output version list are enabled by default.

When using Mathematica 14.0
The following is the procedure for starting with the CLI.

- 46/80 -2024-10-10

TSUBAME4.0 User's Guide$ module load mathematica/14.0$ mathMathematica 14.0.0 Kernel for Linux x86 (64-bit)
Copyright 1988-2023 Wolfram Research, Inc.
In[1]:=

Entering Quit ends the program.
You can use it with GUI as follows.
$ module load mathematica/14.0$ Mathematica

When using Mathematica 14.1 or later
The following is the procedure for starting with the CLI.
$ module load mathematica/14.1$ mathWolfram 14.1.0 Kernel for Linux x86 (64-bit)
Copyright 1988-2024 Wolfram Research, Inc.
In[1]:=

Entering Quit ends the program.
You can use it with GUI as follows.
$ module load Mathematica/14.1$ WolframNB- 47/80 -2024-10-10

TSUBAME4.0 User's Guide

6.12. COMSOLCOMSOLの利用手順を以下に示します。
You can launch COMSOL as follows.
$ module load comsol$ comsol- 48/80 -2024-10-10

TSUBAME4.0 User's Guide

Click File > Exit from menu bar, then the program ends.
You can check the status of COMSOL licenses as follows.
$ lmutil lmstat -S LMCOMSOL -c *****@kvm5.ini.t4.gsic.titech.ac.jp:*****@kvm6.ini.t4.gsic.titech.ac.jp:*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/comsol/t4-license
The port number is changed in mid April every year.

6.13. Schrodinger
The procedure for using Schrodinger is shown below.
Ligprep with CLI：
$ module load schrodinger
SMILES format input file, MAE format output file$ ligprep -ismiinputfile -omaeoutputfile

If you want to use it with GUI, launch Maestro.
$ module load schrodinger$ maestro- 49/80 -2024-10-10

TSUBAME4.0 User's Guide

Click File > Exit on menu bar will ends the program.
You can check license usage of Schrodinger as follows.
$ lmutil lmstat -S SCHROD -c *****@kvm5.ini.t4.gsic.titech.ac.jp:*****@kvm6.ini.t4.gsic.titech.ac.jp:*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/schrodinger/t4-license
The port number is changed in mid April every year.

6.14. MATLAB
MATLAB is an application that enables numerical calculations such as matrix calculations and data visualization.
Examples of how to use MATLAB are shown below.
$ module load matlab$ matlab- 50/80 -2024-10-10

TSUBAME4.0 User's Guide

CLI$ module load matlab$ matlab -nodisplay

Enter exit when you stop using the application.
You can check MATLAB license usage status as follows.
$ lmutil lmstat -S MLM -c *****@kvm5.ini.t4.gsic.titech.ac.jp:****@kvm6.ini.t4.gsic.titech.ac.jp:*****@ldap2.ini.t4.gsic.titech.ac.jp

Info
The port number for the license is visible after you purchase the application with Application activation (TSUBAME4).
/apps/t4/rhel9/isv/matlab/t4-license
The port number is changed in mid April every year.

6.15. VASPInfoWhen using VASP, users must have their own license.

- 51/80 -2024-10-10

TSUBAME4.0 User's Guide

Info
For VASP inquiries, the response by TSUBAME4.0 contacts is limited due to licensing terms.
Please refer to TSUBAME4.0 FAQ or refer to and post on VASP Forum.

There are two ways to use VASP with TSUBAME 4.0.
• Use pre-compiled binaries
We've acquired VASP licenses for supercomputers and provide compiled binaries to VASP license holders.
If you have a valid VASP6 license and want to use compiled binaries on TSUBAME4.0, please register through this application form.
After the procedure is completed, set permissions to access binaries on TSUBAME. (It may take some time as confirmation with the license issuer
is required.)
VASP compiled binary usage is shown below.

$ module load VASP
$ vasp_std (or vasp_ncl etc...)• Build the source code yourself
If you build the source code yourself, no application is required. Please prepare the source code yourself.
Also, you do not need to specify the module command. Specify environment variables according to your own environment.
As reference information, the procedure for building VASP6.4.2 on TSUBAME4.0 is available.
• I would like to know the procedure to build a VASP with TSUBAME4.0.

6.16. Linaro forge(ex:Arm Forge)
Run Linaro forge as follows.
$ module load forge$ forge- 52/80 -2024-10-10

TSUBAME4.0 User's Guide

Select File > Exit will end the program.

- 53/80 -2024-10-10

TSUBAME4.0 User's Guide- 54/80 -2024-10-10

TSUBAME4.0 User's Guide

7. Freeware
Freesoftware available in this system is listed below.

- 55/80 -2024-10-10

TSUBAME4.0 User's Guide

SoftwareDescription

GAMESS

Solver Simulator

TinkerSolver Simulator

GROMACS

Solver Simulator

LAMMPS

Solver Simulator

NAMMD

Solver Simulator

QUANTUM ESPRESSO

Solver Simulator

CP2K

Solver Simulator

OpenFOAM

Solver Simulator, Visualization

CUDA

GPU library

CuDNNGPU library

NCCL

GPU libraryTensorFlow

DeepLearning framework

DeePMD-kit

MD DeepLearning framework

PyTorchMachine larning

R

Interpriter (Rmpi,rpud)

HadoopDistributed Data Processing Tools

POV-Ray

Visualization

ParaView

Visualization

VisIt

Visualization

vmd

Visualization

VESTA

Visualization

turbovnc

Remote GUI(X11)

VirtualGL

Remote GUI

Open OnDemand

Web poral site for HPC

gnuplot

Data grapher/visualization

GIMP

Image Viewing and Editing

Tgif

Image Viewing and Editing

ImageMagick

Image Viewing and Editing

TeX Live

TeX distribution

OpenJDK

Development tool

pythonDevelopment toolruby

Development toolperl

Development tool

PHP

Development toolgolangDevelopment tool

- 56/80 -2024-10-10

TSUBAME4.0 User's Guide

SoftwareDescription

Emacs

Editorvim

EditorPETSc

Linear system solvers, libraries

FFTW

Fast Fourier Transform Library

Apptainer

Container platform

Spack

Software package management

miniconda

Software package management

PyPl

Software package management

Rbenv

Software package management

Alphafold

Bio

tmux

Terminal mutiplexer

NetCDF

Multidimensional data format

HDF5

Hierarchical data format

ffmpegVideo and audio processing

InfoFor "module" command, please refer to Switch User Environment.

7.1. Quantum chemistry/MD related software
7.1.1. GAMESSGAMESS is an open source ab initio molecular quantum chemistry calculation application.
An example of how GAMESS can be used with the batch queue system is shown below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N gamessmodule load gamess$GAMESS_DIR/rungms exam01 00 2 2 1

A detailed description is provided below.
http://www.msg.ameslab.gov/gamess/index.html

7.1.2. Tinker
Tinker is a modeling software for molecular dynamics with special features for biopolymers.
An example of how Tinker can be used with the batch queue system is shown below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N tinker- 57/80 -2024-10-10

TSUBAME4.0 User's Guide

module load tinker
cp -rp $TINKER_DIR/example $TMPDIR
cd $TMPDIR/example
dynamic waterbox.xyz -k waterbox.key 100 1 1 2 300
cp -rp $TMPDIR/example $HOME

A detailed description is provided below.
https://dasher.wustl.edu/tinker/

7.1.3. GROMACSGROMACS is an engine for molecular dynamics simulation and energy minimization.
An example of how to use GROMACS with the batch queue system is shown below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N gromacs

module load gromacs
cp -rp $GROMACS_DIR/examples/water_GMX50_bare.tar.gz $TMPDIR
cd $TMPDIR
tar xf water_GMX50_bare.tar.gz
cd water-cut1.0_GMX50_bare/3072
gmx_mpi grompp -f pme.mdp
OMP_NUM_THREADS=2 mpiexec -np 4 gmx_mpi mdrun
cp -rp $TMPDIR/water-cut1.0_GMX50_bare $HOME

A detailed description is provided below.
http://www.gromacs.org/

Info
GROMACS 2023 and later support the CUDA Graphs feature.
This feature may lead to improved execution performance when using multiple GPUs, but it is your own responsibility to determine whether or not it will
speed up your case, as described below.
"Note that this remains an experimental feature which has had limited testing, so care should be taken to ensure the results are as expected."A detailed description is provided below.
https://developer.nvidia.com/blog/a-guide-to-cuda-graphs-in-gromacs-2023/
Example ( TSUBAME4.0 node_f=1 ) :mpiexec -x OMP_NUM_THREADS=2 -x GMX_ENABLE_DIRECT_GPU_COMM=1 -np 4 gmx_mpi mdrun -nb gpu -bonded gpu -pme gpu -update gpu -npme 1

7.1.4. LAMMPSLAMMPS is a classical molecular dynamics code that models populations of liquid, solid, and gaseous particles.
An example of how LAMMPS can be used with a batch queue system is described below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N lammps

module load lammps
cp -rp $LAMMPS_DIR/examples/VISCOSITY $TMPDIR
cd $TMPDIR/VISCOSITY
mpirun -x PATH -x LD_LIBRARY_PATH -np 4 lmp -sf gpu -in in.gk.2d
cp -rp $TMPDIR/VISCOSITY $HOME

A detailed description is provided below.
http://lammps.sandia.gov/- 58/80 -2024-10-10

TSUBAME4.0 User's Guide

7.1.5. NAMD
NAMD is an object-oriented parallel molecular dynamics code designed for high-performance simulations of large biomolecular systems.
An example of how NAMD can be used with a batch queue system is described below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N namd

module load namd
cp -rp $NAMD_DIR/examples/stmv.tar.gz $TMPDIR
cd $TMPDIR
tar xf stmv.tar.gz
cd stmv
namd3 +idlepoll +p4 +devices 0,1,2,3 stmv.namd
cp -rp $TMPDIR/stmv $HOME

A detailed description is provided below.
https://www.ks.uiuc.edu/Research/namd/3.0/ug/

7.1.6. CP2K
CP2K is a quantum chemistry and solid state physics software package that can run atomic simulations of solid, liquid, molecular, periodic, material,
crystalline, and biological systems.
An example of how CP2K can be used with a batch queue system is described below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N cp2k

module load cp2k
cp -rp $CP2K_DIR/benchmarks/QS $TMPDIR
cd $TMPDIR/QS
export OMP_NUM_THREADS=1
mpirun -x PATH -x LD_LIBRARY_PATH -np 4 cp2k.psmp -i H2O-32.inp -o H2O-32.out
cp -rp $TMPDIR/QS $HOME

A detailed description is provided below.
https://www.cp2k.org/

7.1.7. QUANTUM ESPRESSOQUANTUM ESPRESSO is a suite for first-principles electronic structure calculations and materials modeling.
An example of how to use QUANTUM ESPRESSO with the batch queue system is described below.
#!/bin/sh#$ -cwd#$ -l h_rt=00:10:00#$ -l node_f=1#$ -N q-e

module purge
module load quantumespresso
cp -p $QUANTUMESPRESSO_DIR/test-suite/pw_scf/scf.in .
cp -p $QUANTUMESPRESSO_DIR/example/Si.pz-vbc.UPF .
mpirun -x ESPRESSO_PSEUDO=$PWD -x PATH -x LD_LIBRARY_PATH -np 4 pw.x < scf.in

A detailed description is provided below.
https://www.quantum-espresso.org/- 59/80 -2024-10-10

TSUBAME4.0 User's Guide

7.2. CFD related software
7.2.1. OpenFOAM
OpenFOAM is an open source fluid/continuum simulation code. There are two versions installed: the Foudation version (openfoam) and the ESI
version (openfoam-esi).
An example of how to use OpenFOAM with the batch queue system is described below.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N openformmodule load openfoam
mkdir -p $TMPDIR/$FOAM_RUN
cd $TMPDIR/$FOAM_RUN
cp -rp $FOAM_TUTORIALS .
cd tutorials/legacy/incompressible/icoFoam/cavity/cavity
blockMesh
icoFoamparaFoamIf you are using the ESI version of OpenFOAM, replace module load above with module openfoam-esi .
A detailed explanation is given below.
https://openfoam.org/resources/
http://www.openfoam.com/documentation/

7.3. Numerical calculation library for GPU
7.3.1. cuBLAS
cuBLAS is a BLAS (Basic Linear Algebra Subprograms) library that runs on GPUs.
How to use$ module load cuda
$ nvcc -gencode arch=compute_90,code=sm_90 -o sample sample.cu -lcublas

When calling cuBLAS in a normal C program, it must be specified with -I, -L, or -l at compile time.
$ module load cuda$ gcc -o blas blas.c -I${CUDA_HOME}/include -L${CUDA_HOME}/lib64 -lcublas

7.3.2. cuSPARSE
cuSPARSE is a library for sparse matrix calculations on NVIDIA GPUs.
How to use$ module load cuda
$ nvcc -gencode arch=compute_90,code=sm_90 sample.cu -lcusparse -o sample

When calling cuSPARSE in a normal C program, it must be specified with -I, -L, or -l at compile time.
$ module load cuda
$ g++ sample.c -lcusparse_static -I${CUDA_HOME}/include -L${CUDA_HOME}/lib64-lculibos -lcudart_static -lpthread -ldl -o sample

7.3.3. cuFFT
cuFFT is a library for parallel FFT (Fast Fourier Transform) on NVIDIA GPUs.

- 60/80 -2024-10-10

TSUBAME4.0 User's Guide

How to use$ module load cuda
$ nvcc -gencode arch=compute_90,code=sm_90 -o sample sample.cu -lcufft

If cuFFT is called during a normal C program, it must be specified with -I, -L, or -l at compile time.
$ module load cuda$ gcc -o blas blas.c -I${CUDA_HOME}/include -L${CUDA_HOME}/lib64 -lcufft

7.4. Machine learning and big data analysis related software
7.4.1. CuDNN
CuDNN is a library for GPU-based Deep Neural Networks.
The usage of CuDNN is described below.
$ module load cuda cudnn

7.4.2. NCCL
NCCL is a collective communication library for multiple GPUs.
An example of how to use NCCL is shown below.
$ module load cuda nccl

7.4.3. PyTorch
PyTorch is an open source machine learning library available for machine learning in Python.
The installation procedure for PyTorch is described below.
$ python3 -m pip install --user torch

Info
PyTorch is installed in the user environment.

7.4.4. TensorFlow
TensorFlow is an open source library for machine learning and AI using data flow graphs.
Instructions for installing TensorFlow are provided below.
$ python3 -m pip install --user tensorflow

Info
TensorFlow is installed in the user environment.

A detailed description is provided below.
https://www.tensorflow.org/- 61/80 -2024-10-10

TSUBAME4.0 User's GuideInfo
To run TensorFlow on a GPU, the versions of python, cudnn, and cuda must match the table linked below.
https://www.tensorflow.org/install/source#gpu
Example)TensorFlow 2.17.0
Pytnon 3.9～3.12
CuDNN 8.9
CUDA 12.3
Please note that TSUBAME4.0 does not introduce CuDNN 8.9, which supports CUDA12.
In addition, as a general rule, we do not introduce versions older than the currently introduced version.
If you want to run TensorFlow 2.17.0 on a GPU, please install CuDNN 8.9 on your own environment using pip or other software.
(We have confirmed that neither without CuDNN nor with CuDNN 9.0.0 will work on GPUs.)
In addition, depending on the construction environment and combination, there may be cases where installation cannot be performed normally even if theconditions are met.
Our verified procedure is described below for reference information.
If you have trouble installing other versions, procedures, or environments, you will need to check the release notes of each application yourself to resolve
the problem.
[Reference]
Here is a log of the process of installing TensorFlow 2.17.0 and CuDNN 8.9 with CUDA12 support in a python virtual environment and confirming the useof GPUs.
shell-session# Check python version$ python -V
Python 3.9.18# Load cuda12.3.2$ module load cuda/12.3.2# Create python virtual environment$ python -m venv venv
# Call python virtual environment$ source venv/bin/activate# Upgrade pip$ pip install --upgrade pip# Install cudnn8.9 for cuda12 and tenserflow2.17.0$ pip install nvidia-cudnn-cu12==8.9.7.29 tensorflow==2.17.0# Check that the GPU is visible$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"<< Omit logs >>[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]7.4.5. DeePMD-kit
DeePMD-kit is a machine learning framework for MD. An example job script for DeePMD-kit is shown below.

7.4.5.1. 1 DeePMD-kit + LAMMPS
7.4.5.1.1. DEEPMD-KIT LAMMPS 1 NODE

An example job script for DeePMD-kit + LAMMPS (1 node, 4 GPUs) is shown below.
#!/bin/sh#$ -l h_rt=6:00:00#$ -l node_f=1#$ -cwd

module purge
module load deepmd-kit lammps
module li 2>&1
# enable DeePMD-kit for lammps/2aug2023_u3
export LAMMPS_PLUGIN_PATH=$DEEPMD_KIT_DIR/lib/deepmd_lmp- 62/80 -2024-10-10

TSUBAME4.0 User's Guide# https://tutorials.deepmodeling.com/en/latest/Tutorials/DeePMD-kit/learnDoc/Handson-Tutorial%28v2.0.3%29.html
wget https://dp-public.oss-cn-beijing.aliyuncs.com/community/CH4.tar
tar xf CH4.tar
cd CH4/00.data
python3 <<EOF
import dpdata
import numpy as np
data = dpdata.LabeledSystem('OUTCAR', fmt = 'vasp/outcar')print('# the data contains %d frames' % len(data))# random choose 40 index for validation_data
index_validation = np.random.choice(200,size=40,replace=False)# other indexes are training_data
index_training = list(set(range(200))-set(index_validation))
data_training = data.sub_system(index_training)
data_validation = data.sub_system(index_validation)# all training data put into directory:"training_data"
data_training.to_deepmd_npy('training_data')# all validation data put into directory:"validation_data"
data_validation.to_deepmd_npy('validation_data')
print('# the training data contains %d frames' % len(data_training))
print('# the validation data contains %d frames' % len(data_validation))
EOF

cd ../01.train
dp train input.json
dp freeze -o graph.pb
dp compress -i graph.pb -o graph-compress.pb
dp test -m graph-compress.pb -s ../00.data/validation_data -n 40 -d results
cd ../02.lmp
ln -s ../01.train/graph-compress.pb
mpirun -x PATH -x LD_LIBRARY_PATH -np 4 lmp -sf gpu -in in.lammps

7.4.5.1.2. DEEPMD-KIT LAMMPS 2 NODESAn example job script for DeePMD-kit + LAMMPS (2 nodes, 8 GPUs) is shown below.
#!/bin/sh#$ -l h_rt=12:00:00#$ -l node_f=2#$ -cwd

module purge
module load deepmd-kit lammps
module li 2>&1
# enable DeePMD-kit
export LAMMPS_PLUGIN_PATH=$DEEPMD_KIT_DIR/lib/deepmd_lmp# https://tutorials.deepmodeling.com/en/latest/Tutorials/DeePMD-kit/learnDoc/Handson-Tutorial%28v2.0.3%29.html
wget https://dp-public.oss-cn-beijing.aliyuncs.com/community/CH4.tar
tar xf CH4.tar
cd CH4/00.data
python3 <<EOF
import dpdata
import numpy as np
data = dpdata.LabeledSystem('OUTCAR', fmt = 'vasp/outcar')print('# the data contains %d frames' % len(data))# random choose 40 index for validation_data
index_validation = np.random.choice(200,size=40,replace=False)# other indexes are training_data
index_training = list(set(range(200))-set(index_validation))
data_training = data.sub_system(index_training)
data_validation = data.sub_system(index_validation)# all training data put into directory:"training_data"
data_training.to_deepmd_npy('training_data')# all validation data put into directory:"validation_data"
data_validation.to_deepmd_npy('validation_data')
print('# the training data contains %d frames' % len(data_training))
print('# the validation data contains %d frames' % len(data_validation))
EOF
cd ../01.train
mpirun -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x NCCL_BUFFSIZE=1048576 -npernode 4 -np 8 dp train input.json
dp freeze -o graph.pb
dp compress -i graph.pb -o graph-compress.pb
dp test -m graph-compress.pb -s ../00.data/validation_data -n 40 -d results
cd ../02.lmp
ln -s ../01.train/graph-compress.pb
mpirun -x PATH -x LD_LIBRARY_PATH -npernode 4 -np 8 lmp -sf gpu -in in.lammps

A detailed description is provided below. https://docs.deepmodeling.com/projects/deepmd/en/master/index.html- 63/80 -2024-10-10

TSUBAME4.0 User's Guide

7.4.6. R
R is an interpreted programming language for data analysis and graphics.
Rmpi is installed for parallel processing and rpud for GPU.
Examples of how to use R are described below.
$ module load R$ mpirun -np 2 Rscript test.R

7.4.7. Apache Hadoop
The Apache Hadoop software library is a framework for distributed processing of large data sets using a simple programming model.
Examples of how to use Apache Hadoop are described below.
$ module load hadoop$ mkdir input$ cp -p $HADOOP_HOME/etc/hadoop/*.xml input$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output 'dfs[a-z.]+'$ cat output/part-r-00000
1
dfsadmin

The following is the procedure for using the system in the case of a batch queue system.
#!/bin/bash#$ -cwd#$ -l node_f=1#$ -l h_rt=0:10:0#$ -N hadoop

module load hadoop
cd $TMPDIR
mkdir input
cp -p $HADOOP_HOME/etc/hadoop/*.xml input
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output 'dfs[a-z.]+'cp -rp output $HOME

7.5. Visualization related software
7.5.1. POV-Ray
POV-Ray is a free ray-tracing software.
Examples of how POV-Ray can be used are described below.
$ module load pov-ray$ povray -benchmark

A detailed description is provided below.
http://www.povray.org/

7.5.2. ParaView
ParaView is an open source, multi-platform data analysis and visualization application.
Examples of how ParaView can be used are described below.
$ module load paraview$ paraview- 64/80 -2024-10-10

TSUBAME4.0 User's Guide

7.5.2.1. For visualization with multiple GPUs
You can use paraview/5.12.0 , paraview/5.12.0-egl , to visualize with multiple GPUs on multiple nodes. Note that paraview/5.12.0-egl does notinclude the paraview command.
Here is an example of using 8 GPUs with node_f=2 .
• wrap.sh#!/bin/sh
num_gpus_per_node=4
mod=$((OMPI_COMM_WORLD_RANK%num_gpus_per_node))
if [ $mod -eq 0 ];thenexport VTK_DEFAULT_EGL_DEVICE_INDEX=0
elif [ $mod -eq 1 ];thenexport VTK_DEFAULT_EGL_DEVICE_INDEX=1
elif [ $mod -eq 2 ];thenexport VTK_DEFAULT_EGL_DEVICE_INDEX=2
elif [ $mod -eq 3 ];thenexport VTK_DEFAULT_EGL_DEVICE_INDEX=3fi$*• job.sh#!/bin/sh#$ -cwd#$ -V#$ -l h_rt=8:0:0#$ -l node_f=2module purge
module load paraview
mpirun -x PATH -x LD_LIBRARY_PATH -npernode 4 -np 8 ./wrap.sh pvserver

Don't forget to give wrap.sh execute permission. ( chmod 755 wrap.sh )With the above job script
qsub -g <グループ名> job.shand submit the job.
Confirm that the job is flowing with qstat .
yyyyyyyy@login1:~> qstat
job-ID
prior
name
user
state submit/start atqueue
jclassslots ja-task-ID-----------------------------------------------------------------------------------------------------------------------------------------------xxxxxxx 0.55354 job.shyyyyyyyyr05/31/2024 09:24:19 all.q@rXnY
56

Ssh to the node where the job is flowing via X forwarding and start paraview.
yyyyyyyy@login1:~> ssh -CY rXnY
yyyyyyyy@rXnY:~> module load paraview
paraview

turbovnc can also be used.
After starting, click "File"->"Connect" and click "Add Server".
Enter "Name" appropriately (in this case, "test") and click "Configure".

- 65/80 -2024-10-10

TSUBAME4.0 User's Guide

Then click "Connect".
Once connected, test(cs://localhost:11111) will appear in the "Pipeline Browser" field.

- 66/80 -2024-10-10

TSUBAME4.0 User's Guide

The sample data of paraview can be downloaded from here.

- 67/80 -2024-10-10

TSUBAME4.0 User's Guide

A detailed description is provided below.
https://www.paraview.org/

7.5.3. VisIt
VisIt is an open source visualization application.
Examples of how VisIt can be used are described below.
$ module visit$ visitA detailed description is provided below.
https://wci.llnl.gov/simulation/computer-codes/visit/

7.6. Other freeware
7.6.1. turbovnc
turbovnc is open source VNC software. An example of how to use turbovnc is shown below.
Please allocate a computation node with qrsh and execute on the computation node.
• Get compute nodes$ qrsh -g <TSUBAME group> -l <Resource type>=<number> -l h_rt=<Maximum run time>• Execute the following on the secured compute node and start vncserver$ module load turbovnc$ vncserver -xstartup xfce.sh
You will require a password to access your desktops.
Password: # <-- Enter a password
Verify:Would you like to enter a view-only password (y/n)? n- 68/80 -2024-10-10

TSUBAME4.0 User's Guide

Desktop 'TurboVNC: rXnY:1 ()' started on display rXnY:1 # <-- Make sure to memolize VNC display number:1
Creating default startup script /home/n/xxxx/.vnc/xstartup.turbovnc
Starting applications specified in /home/n/xxxx/.vnc/xstartup.turbovnc
Log file is /home/n/xxxx/.vnc/rXiYnZ:1.log

If you want to increase the screen size, specify the size as vncserver -geometry <WIDTH>x<HEIGHT> .
• Then download the installer for your own PC from https://sourceforge.net/projects/turbovnc/files/, Install turbovnc viewer• From the terminal software connected to the compute node, configure SSH port forwarding to forward local port 5901 to port 5901 on thecompute node (if the display number is rXiYnZ:n, the port number for port forwarding is set to 5900+n).
• Start turbovnc viewer from your own PC, connect to localhost:5901 and enter the password you set.

Tips
The VNC display number counts up each time vncserver is started; check the port number in the SSH port forwarding configuration each time.

7.6.1.1. How to use VNC client from MobaXterm
MobaXterm has a built-in VNC client, so you can use VNC connections without installing a VNC client.
• After securing the node with qrsh, select "Sessions"->"New session"->"VNC" from MobaXterm.

• Then, in "Basic Vnc settings", enter the hostname of the secured compute node in "Remote hostname or IP address" and 5900+n in "Port", click"Connect through SSH gateway( jump host)" in "Network settings", enter login.t4.gsic.titech.ac.jp in "Gateway SSH server", and click "Connect
through SSH server" in "Network settings". Enter login.t4.gsic.titech.ac.jp in "Gateway SSH server", leave "Port" as 22, enter your TSUBAME loginname in "User", check "Use private key" and enter your Check "Use private key" and enter your private key.

- 69/80 -2024-10-10

TSUBAME4.0 User's Guide

Click OK to start the VNC client.

- 70/80 -2024-10-10

TSUBAME4.0 User's Guide

7.6.1.2. turbovnc + VirtualGL
If you are using a resource type (node_f, node_h, node_q, gpu_1) that allocates one or more GPUs when using turbovnc, you can use VirtualGL tovisualize using GPUs. As an example, the use of VirtualGL in the case of gpu_1 is shown below.
$ qrsh ... -l gpu_1=1$ module load turbovnc$ vncserver -xstartup xfce.sh• Connect with a VNC client and perform the following$ vglrun -d /dev/dri/card<N> <OpenGL application><N> is a number from 1 to 4, corresponding to GPU 0 to 3. The correspondence between the GPU number and card<N> is as follows

Device nameGPU number/dev/dri/card1

GPU1(64:00)/dev/dri/card2

GPU0(04:00)/dev/dri/card3

GPU3(E4:00)/dev/dri/card4

GPU2(84:00)

7.6.2. gnuplot
gnuplot is a command line interactive graph drawing program.
In addition to the standard features, it is built to support X11, latex, PDFlib-lite and Qt4.
Examples of how to use gnuplot are given below.
$ gnuplot

7.6.3. Tgif
tgif is an open source drawing tool.
The following is a description of how to use tgif.
$ module load tgif
$ tgifIf you found an error saying Cannot open the Default(Msg) Font '-*-courier-medium-r-normal-*-14-*-*-*-*-*-iso8859-1' , add the following
line to ~/.Xdefaults
Tgif.DefFixedWidthFont:
Tgif.DefFixedWidthRulerFont:
Tgif.MenuFont:
Tgif.BoldMsgFont:
Tgif.MsgFont:-*-fixed-medium-r-semicondensed--13-*-*-*-*-*-*-*-*-fixed-medium-r-semicondensed--13-*-*-*-*-*-*-*-*-fixed-medium-r-semicondensed--13-*-*-*-*-*-*-*-*-fixed-medium-r-semicondensed--13-*-*-*-*-*-*-*-*-fixed-medium-r-semicondensed--13-*-*-*-*-*-*-*

7.6.4. GIMP
GIMP is an open source image manipulation program.
Examples of how to use GIMP are described below.
$ gimp7.6.5. ImageMagick
ImageMagick is an image processing tool.

- 71/80 -2024-10-10

TSUBAME4.0 User's Guide

In addition to the standard features, it is built to support X11, HDRI, libwmf, and jpeg.
An example of how to use ImageMagick is shown below.
$ module load imagemagick$ convert -size 48x1024 -colorspace RGB 'gradient:#000000-#ffffff' -rotate 90 -gamma 0.5 -gamma 2.0 result.jpg

7.6.6. Tex Live
TeX Live is an integrated TeX package.
An example of how to use TeX Live is shown below.
$ lualatex test.tex

Info
A PDF file will be created.

7.6.7. Java SDK
The following versions of OpenJDK are installed as Java SDK.
- openjdk version "1.8.0_402"- openjdk version "11.0.22" (Default)- openjdk version "21.0.2"

The module command allows you to switch between versions.
The method of switching is as follows
module unload openjdk
module load openjdk/<version>

OpenJDK version

version

1.8.0_402

1.8.0

11.0.2211.0.2221.0.221.0.2The current version can be checked by following the steps below.
java -version

An example of how to use the Java SDK is shown below.
$ javac Test.java$ java Test

7.6.8. PETSc
PETSc is an open source parallel numerical library. It can perform linear equation solving, etc.
Two versions, one for real numbers and the other for complex numbers, are installed.
Examples of how to use PETSc are described below.
$ module load petsc/3.20.4-realor← real number usage- 72/80 -2024-10-10

TSUBAME4.0 User's Guide$ module load petsc/3.20.4-complex$ mpiifort test.F -lpetsc← for complex numbers

7.6.9. FFTW
FFTW is an open source library for Fast Fourier Transform.
Since the FFTW 2x and 3x series are incompatible, two versions, Series 2 and Series 3, are installed.
Examples of how to use FFTW are described below.
$ module load fftw/3.3.10-intel intel-mpi/2021.11
or$ module load fftw/3.3.10-gcc← Open MPI$ gfortran test.f90 -lfftw3← Intel MPI

7.6.10. Apptainer (Singularity)
Apptainer (Singularity) is a Linux container for HPC.
For more information, see Use containers.

7.6.11. Alphafold
Alphafold is a protein structure prediction program using machine learning. An example of using Alphafold is shown below.
• Initialization (login node or compute node)
module purge
module load alphafold
cp -pr $ALPHAFOLD_DIR .
cd alphafold
git pull # Update latest version• At runtime (example job script for alphafold/2.3.2)#!/bin/sh#$ -l h_rt=24:00:00#$ -l node_f=1#$ -cwd

module purge
module load alphafold
module li
cd alphafold./run_alphafold.sh -a 0,1,2,3 -d $ALPHAFOLD_DATA_DIR -o dummy_test/ -m model_1 -f ./example/query.fasta -t 2020-05-14

Due to the large size of the database files, please avoid individual downloads whenever possible.
For a detailed description of Alphafold, please see below. https://github.com/deepmind/alphafold

7.6.12. miniconda
miniconda is a python virtual environment creation software. An example of using miniconda is shown below.
module load miniconda
eval "$(/apps/t4/rhel9/free/miniconda/24.1.2/bin/conda shell.bash hook)"
conda create -n test
conda acivate test

For a detailed description of miniconda, please see below. https://docs.anaconda.com/free/miniconda/index.html

7.6.13. spack
spack is a package manager for HPC. An example of using spack is shown below.

- 73/80 -2024-10-10

TSUBAME4.0 User's Guide

module load spack
spack install tree

For more information on spack, please see below. https://spack.io/- 74/80 -2024-10-10

TSUBAME4.0 User's Guide

Appendix. Comparison of old and new TSUBAMEAppendix.1. Architecture
TSUBAME3.0

TSUBAME4.0

Theoretical peak double precision performance

12.5PFLOPS

66.8PFLOPS

Theoretical peak half precision performance

47.2PFLOPS

952PFLOPS

Total main memory

135TiB

180TiB

Total HDD capacity

15.9PB

44.2PB

Total SSD capacity

327TB

Compute nodes

540

240

Total cores

15,12046,080Total GPUs

2,160

960

Interconnect

Intel Omni-Path HFI 100Gbps

InfiniBand NDR200 200Gbps

Interenet

SINET5 100Gbps

SINET6 100Gbps

Appendix.2. Compute nodes
TSUBAME3.0

TSUBAME4.0

Compute node HPE SGI ICE-XA 540 nodes

Compute node HPE Cray XD665 240 nodes

CPUIntel Xeon E5-2680 v4 2.4GHz x 2 Socket

AMD EPYC 9654 2.4GHz x 2 Socket

Cores/Threads

14cores / 28threads x 2CPU

96cores / 192threads x 2CPU

Memory256GiB

768GiB (DDR5-4800)

GPU

NVIDIA TESLA P100 for NVlink-Optimized Servers x 4

NVIDIA H100 SXM5 94GB HBM2e x 4

SSD

2TB

1.92TB NVMe U.2 SSD

Interconnect

Intel Omni-Path HFI 100Gbps x 4

InfiniBand NDR200 200Gbps x 4

Computing Unit
Components (per node)- 75/80 -2024-10-10

TSUBAME4.0 User's Guide

Appendix.3. Software
Appendix.3.1 System software
TSUBAME3.0 (as of 2023.4.6)

TSUBAME4.0

OS

SUSE Linux Enterprise Server 12 SP5

RedHat Enterprise Linux Server 9.3

Job scheduler

Univa Grid Engine 8.6.11

Altair Grid Engine 2023.1.1

Compilers

GCC 4.8.5, 12.2.0

GCC 11.4.1

Intel Compiler 23.0.0

Intel oneAPI compiler 2024.0 and MKL

NVIDIA HPC SDK 22.2

NVIDIA HPC SDK 24.1
AOCC 4.1.0

MPIIntel MPI 21.8.0

Intel MPI 2021.11

SGI MPT 2.16
OpenMPI 3.1.4

OpeMPI 5.0.2

CUDA library

11.0.3 (12.1.0)

12.3.2CUDA driver450.172.01

545.23.08

OmniPath driver (OPA)

10.10.3.1.1

InfiniBand driver (OFED)

23.10-1.1.9- 76/80 -2024-10-10

TSUBAME4.0 User's Guide

Appendix.3.2 Commercial application
SoftwareDescription

TSUBAME3.0

TSUBAME4.0

ANSYS

Analysis Software○○

ABAQUS

Analysis Software○○

ABACUS CAE

Analysis Software○○

MSC NastranAnalysis Software○

MSC Patran

Analysis Software○

Gaussian

Quantum chemistry calculation program○○

GaussView

Quantum chemistry calculation program Pre-Post tool

○

○

AMBER

Molecular dynamics calculation program○○

Materials Studio

Chemical Simulation Software○○

Discovery Studio

Chemical Simulation Software○○

Mathematica

Mathematical Processing Software

○

○

Maple

Mathematical Processing Software○

AVS/Express

Visualization○

AVS/Express PCE

Visualization○

LS-DYNA

Analysis Software○○ Included in ANSYSCOMSOL

Analysis Software○○

Schrodinger

Chemical Simulation Software○○

MATLAB

Numerical calculation software

○

○

VASP

Quantum molecular dynamics calculation program○○

Linaro forge(ex:Arm Forge)

Debugger○○Intel Compiler

Compiler○○ oneAPIPGI Compiler

Compiler○○ NVIDIA HPC SDK- 77/80 -2024-10-10

TSUBAME4.0 User's Guide

Appendix.3.3 Freeware
SoftwareDescription

TSUBAME3.0

TSUBAME4.0

GAMESS

Solver Simulator○○

TinkerSolver Simulator○○

GROMACS

Solver Simulator○○

LAMMPS

Solver Simulator

○

○

NAMMD

Solver Simulator○○

QUANTUM ESPRESSO

Solver Simulator

○

○

CP2K

Solver Simulator○○

OpenFOAM

Solver Simulator, Visualization

○

○

CuDNN

GPU library

○

○

NCCL

GPU library

○

○

Caffe

DeepLearning framework○

Chainer

DeepLearning framework○

TensorFlow

DeepLearning framework○○

DeePMD-kit

MD DeepLearning framework

○

○

R

Interpriter (Rmpi,rpud)

○

○

clang

Compiler○○ AOCC

Apache Hadoop

Distributed Data Processing Tools○○

POV-Ray

Visualization○○

ParaView

Visualization

○

○

VisIt

Visualization○○

turbovnc

Remote GUI(X11)○○

gnuplot

Data grapher/visualization

○

○

Tgif

Image Viewing and Editing

○

○

GIMP

Image Viewing and Editing○○

ImageMagick

Image Viewing and Editing○○

TeX Live

TeX distribution○○

Java SDK

Development tool

○

○

PETSc

Linear system solvers, libraries

○

○

FFTW

Fast Fourier Transform Library

○

○

DMTCP

Checkpoint Restart○○Singularity

Linux container for HPC○○ Apptainer

Open OnDemand

Web portal site for HPC○- 78/80 -2024-10-10

TSUBAME4.0 User's Guide

Appendx.4. Storage
TSUBAME3.0

StorageMount point

Capacity

Filesystem

Home directory

/home

40TB

GPFS+cNFS

Shared application deployment

/apps

(HDD)

TSUBAME4.0

Hight-speed storage area 1 (HDD)/gs/hs0

4.8PB

LustreHight-speed storage area 2 (HDD)/gs/hs1

4.8PB

LustreHight-speed storage area 3 (HDD)/gs/hs2

4.8PB

LustreLocal scratch area (SSD)/scr

1.9TB/node

xfs

Shared scratch area

/beeond

Depends on compute node

BeeGFS

StorageMount point

Capacity

Filesystem

High-speed storage area

/gs/fs

372TB

LustreHome directory/home

44.2PB

Lustre

1.92TB/node

xfs

(SSD)
Large-scale (Big) storage area

/gs/bsShared application deployment

/apps

(HDD)
Local scratch area (SSD)/local- 79/80 -2024-10-10

TSUBAME4.0 User's Guide

Revision History
Date

Change2024/04/21

First edition- 80/80 -2024-10-10