TSUBAME4.0 FAQ(English)

TSUBAME Computing Services, Global Scientific Information and Computing Center
2024-09-30

TSUBAME4.0 FAQ (English)

Table of contents

Table of contents

6

General6

Accounts

7

TSUBAME point and TSUBAME portal

7

Job Execution (Scheduler)

7

Applications

8

Migration from TSUBAME3

8

Tips

8

General9What is TSUBAME4.0?

9

For what purpose can TSUBAME be used?

9

Conditions for using TSUBAME4.0

9

How to get started with TSUBAME4.0

9

How to write acknowledgments in a paper using TSUBAME?

9

Differences between login node and compute node

9

How many the number of files is acceptable per one directory?

10

I'm a beginner, I don't know what to do.

10

About file transfer

11

I want to copy a large amount of data from/to TSUBAME

12

How to synchronize data between TSUBAME and PC

13

How to terminate the programs executed accidentally

13

To allow other members to read and write on a group disk

13

The basic configuration of Module file14

CPU / GPU allocation at resource designation in Altair Grid Engine(AGE)14

How to create an SSH key pair in Linux/Mac/Windows(Cygwin/OpenSSH)

14

How to create an SSH key pair using PuTTY or MobaXterm

15

About common errors in Linux16

How to solve "Disk quota exceeded" error

18

Session suddenly disconnected while working on TSUBAME4.0

18

Available SSH client on Windows

18

The combination of compiler and mpi module

19

When trying to save a file with Emacs during an interactive job, the screen froze

20

Cannot login to TSUBAME4.0 (ssh, Permission denied (publickey,hostbased) etc.)

20

Can not establish CIFS connection to the group disk, Unable to open TSUBAME group disk on Windows

22

FAQ about group disk22

- 2/59 -2024-09-30

TSUBAME4.0 FAQ (English)

About the IP address of the gateway server for compute nodes(connection to license servers outside TSUBAME, etc.)

23

An error such as "fork: Resource temporarily unavailable" is displayed on the login node.

23

Can Docker be used with TSUBAME4.0?

23

Can Singularity be used with TSUBAME4.0?

23

Can container be used with TSUBAME4.0?

23

Can I use Jupyter Lab with Tsubame 4.0?

23

The group disk suddenly became unusable.

23

I would like to know how to utilize the GPU in TSUBAME4.0.

24

I want to use a debugger/profiler that supports multi-threading/multi-processing

24

OpenOnDemand or Jupyter fails to start

24

GLIBC not found error when using Apptainer

27

I want to use VS Code.

27

Accounts

28

Can not find user even though searched when adding a user to a group28

Account Procedures for Re-Issuance of Tokyo Tech IC Card after Expiration Date

28

TSUBAME point and TSUBAME portal

29

TSUBAME portal is not displayed, or does not work properly

29

The screen does not display properly in the process of joining the group

29

Time lag between synchronizing with the login node of the information updated by the portal system

29

Supplement for TSUBAME point

29

The status of "Payment code status" on TSUBAME portal remains "予算責任者の追認待ち利用停止"

30

How to check TSUBAME points, group disk usage, home directory usage

30

When are TSUBAME points consumed and returned?(liquidation of temporary points).

31

The link to TSUBAME Portal in the auto-mail does not work properly.

31

My application for a payment code was rejected, but why?

32

Approved payment code cannot be added to group

33

Job Execution (Scheduler)

34

I get an error when submitting a job, but I do not know which option is bad

34

The job status is "Eqw" and it is not executed.

34

How to terminate the job submitted to batch job scheduler

36

I'd like to check the congestion status of compute node37

How to use scrath area

37

SSH login to compute nodes

38

Submission of dependent job

38

Want to execute multiple calculations at once in a batch job

38

Calculation starts on the login node before executing the qsub command38

The error when executing the qrsh command

39

Check the detail of an error message printed the log file39

- 3/59 -2024-09-30

TSUBAME4.0 FAQ (English)

How to transfer X with qrsh

40

"Warning: Permanently added ECDSA host key for IP address 'XXX.XXX.XXX.XXX' to the list of known hosts." in the error log

42

Errors and remedies of qsub command execution

42

About specification of batch job scheduler

42

About troubleshooting at reservation execution

43

About the difference between h_rt and actual execution time

44

A "command not found" error occured in qrsh/job script44

run some programs on different CPUs/GPUs in a job script

45

Applications

46

Commercial applications (Mathematica, Gaussian, etc.) do not work. Command not found.

46

I purchased a commercial application, but it is not available.

46

A commercial application I purchased suddenly becomes unusable.

46

How to use Python packages on PyPI (e.g. Theano)46

X application (GUI) doesn't work

47

I would like to use an application not provided by TSUBAME 4.0

48

Installing applications not provided by TSUBAME 4.0

48

There is a problem with the operation of the distribution application

49

Pre/post processing of commercial applications

49

The range of support by T4 Helpdesk about the program error such as segmentation fault

50

Error handling for each commercial application

51

How to install numpy, mpi4py, chainer, tensorflow, cupy etc. using python/3.9.18

51

License restriction on commercial application usage51

I want to install and use my library in R

51

An error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called

52

Port forward configuration for each terminal software

52

file output stops by doing mpirun ... >& log.txt & with intel MPI

54

I want to link Intel oneAPI Math Kernel Library, what are the link options?

55

How to use VNC from MobaXterm

55

An error such that "X fatal error. ***ABAQUS/ABQcaeG rank 0 terminated by signal 6 " occurs at modeling

55

icc, icpc commands cannot be used with Intel OneAPI

55

C++17 related errors occur when using Intel oneAPI.

55

I'm having trouble building and running VASP.

56

I would like to know the procedure to build a VASP with TSUBAME4.0.

56"UCX ERROR failed to insert region" error occurs when running VASP

56

Unable to module load python

56

When running OpenMPI/Intel MPI, an hcoll-related error or segmentation fault occurs.

57

Binaries compiled with the -lblas option in Intel Compiler terminate abnormally at runtime

57

GPU not used when running TensorFlow

57- 4/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Migration from TSUBAME3

58

Comparison of old and new TSUBAME

58

Tips

59

TSUBAME4.0 some workarounds of known issues

59- 5/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Table of contents
Warning
Most items are copies of TSUBAME3 FAQs, and some descriptions are outdated.

If necessary, please use the search box in the top right.

General• What is TSUBAME4.0?
• For what purpose can TSUBAME be used?
• Conditions for using TSUBAME4.0• How to get started with TSUBAME4.0• How to write acknowledgments in a paper using TSUBAME?
• Differences between login node and compute node
• How many the number of files is acceptable per one directory?
• I'm a beginner, I don't know what to do.
• About file transfer• I want to copy a large amount of data from/to TSUBAME• How to synchronize data between TSUBAME and PC• How to terminate the programs executed accidentally• To allow other members to read and write on a group disk
• The basic configuration of Module file• CPU / GPU allocation at resource designation in Altair Grid Engine(AGE)• How to create an SSH key pair in Linux/Mac/Windows(Cygwin/OpenSSH)• How to create an SSH key pair using PuTTY or MobaXterm• About common errors in Linux• How to solve "Disk quota exceeded" error• Session suddenly disconnected while working on TSUBAME4.0• Available SSH client on Windows• The combination of compiler and mpi module• When trying to save a file with Emacs during an interactive job, the screen froze• Cannot login to TSUBAME4.0 (ssh, Permission denied (publickey,hostbased) etc.)• Can not establish CIFS connection to the group disk, Unable to open TSUBAME group disk on Windows• FAQ about group disk
• About the IP address of the gateway server for compute nodes(connection to license servers outside TSUBAME, etc.)• An error such as "fork: Resource temporarily unavailable" is displayed on the login node.
• Can Docker be used with TSUBAME4.0?
• Can Singularity be used with TSUBAME4.0?
• Can container be used with TSUBAME4.0?
• Can I use Jupyter Lab with Tsubame 4.0?

- 6/59 -2024-09-30

TSUBAME4.0 FAQ (English)• The group disk suddenly became unusable.
• I would like to know how to utilize the GPU in TSUBAME4.0.
• I want to use a debugger/profiler that supports multi-threading/multi-processing• OpenOnDemand or Jupyter fails to start• GLIBC not found error when using Apptainer• I want to use VS Code.

Accounts• Can not find user even though searched when adding a user to a group• Account Procedures for Re-Issuance of Tokyo Tech IC Card after Expiration Date

TSUBAME point and TSUBAME portal• TSUBAME portal is not displayed, or does not work properly• The screen does not display properly in the process of joining the group• Time lag between synchronizing with the login node of the information updated by the portal system• Supplement for TSUBAME point• The status of "Payment code status" on TSUBAME portal remains "予算責任者の追認待ち利用停止"• How to check TSUBAME points, group disk usage, home directory usage• When are TSUBAME points consumed and returned?(liquidation of temporary points).
• The link to TSUBAME Portal in the auto-mail does not work properly.
• My application for a payment code was rejected, but why?
• Approved payment code cannot be added to group

Job Execution (Scheduler)• I get an error when submitting a job, but I do not know which option is bad• The job status is "Eqw" and it is not executed.
• How to terminate the job submitted to batch job scheduler• I'd like to check the congestion status of compute node
• How to use scrath area
• SSH login to compute nodes• Submission of dependent job• Want to execute multiple calculations at once in a batch job• Calculation starts on the login node before executing the qsub command• The error when executing the qrsh command• Check the detail of an error message printed the log file
• How to transfer X with qrsh
• "Warning: Permanently added ECDSA host key for IP address 'XXX.XXX.XXX.XXX' to the list of known hosts." in the error log• Errors and remedies of qsub command execution• About specification of batch job scheduler• About troubleshooting at reservation execution• About the difference between h_rt and actual execution time- 7/59 -2024-09-30

TSUBAME4.0 FAQ (English)• A "command not found" error occured in qrsh/job script• run some programs on different CPUs/GPUs in a job script

Applications• Commercial applications (Mathematica, Gaussian, etc.) do not work. Command not found.
• I purchased a commercial application, but it is not available.
• A commercial application I purchased suddenly becomes unusable.
• How to use Python packages on PyPI (e.g. Theano)• X application (GUI) doesn't work
• I would like to use an application not provided by TSUBAME 4.0• Installing applications not provided by TSUBAME 4.0• There is a problem with the operation of the distribution application• Pre/post processing of commercial applications• The range of support by T4 Helpdesk about the program error such as segmentation fault• Error handling for each commercial application• How to install numpy, mpi4py, chainer, tensorflow, cupy etc. using python/3.9.18• License restriction on commercial application usage• I want to install and use my library in R
• An error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called• Port forward configuration for each terminal software• file output stops by doing mpirun ... >& log.txt & with intel MPI• I want to link Intel oneAPI Math Kernel Library, what are the link options?
• How to use VNC from MobaXterm• An error such that "X fatal error. ***ABAQUS/ABQcaeG rank 0 terminated by signal 6 " occurs at modeling• icc, icpc commands cannot be used with Intel OneAPI• C++17 related errors occur when using Intel oneAPI.
• I'm having trouble building and running VASP.
• I would like to know the procedure to build a VASP with TSUBAME4.0.
• "UCX ERROR failed to insert region" error occurs when running VASP
• Unable to module load python• When running OpenMPI/Intel MPI, an hcoll-related error or segmentation fault occurs.
• Binaries compiled with the -lblas option in Intel Compiler terminate abnormally at runtime• GPU not used when running TensorFlow

Migration from TSUBAME3
Tips• TSUBAME4.0 some workarounds of known issues- 8/59 -2024-09-30

TSUBAME4.0 FAQ (English)

GeneralWhat is TSUBAME4.0?
TSUBAME 4.0 is a supercomputer operated and managed by Global Scientific Information and Computing Center (GSIC) of the Tokyo Institute ofTechnology. TSUBAME 4.0 has a theoretical calculation performance of 952 PFlops (half precision) and is expected to be the largest supercomputerin Japan handling a wide range of workloads including big data and AI in addition to conventional High Performance Computing.

For what purpose can TSUBAME be used?
Use of TSUBAME is limited to education, research, clerical work and social contribution purpose only. It can not be used for applications that directly
lead to private financial interests. For example, mining virtual currency using block chain technology.

Conditions for using TSUBAME4.0
An account is required to use TSUBAME4.0.
Account types vary depending on affiliation and system. See How to apply for a TSUBAME account.

How to get started with TSUBAME4.0
This part shows the flow until setup an environment for running program.
There are 6 steps necessary to use TSUBAME4.
When step 1 and 2 are done, login is enabled. To submit jobs, you need to complete additional steps 3 - 5. In addition to your home directory of 25GiB,
if you need additional volumes, do step 6.
1. Getting an account
2. SSH key pair generation and the public key registration
3. Creation of a group (group administrator only)
4. Addition of users to the group(group administrator and its members)
5. Tsubame point purchase(group administrator only)
6. Setup of group disk(group administrator only)

How to write acknowledgments in a paper using TSUBAME?
Please refer to the following page for an example of how to write an acknowledgement. Please note that this is just an example, and you may adjust
the description to match the description of other supercomputers or research funds.
Please mention TSUBAME usage in acknowledgement of publications
In addition, please submit reports on your use of TSUBAME, such as bibliographic information, through TSUBAME Portal to help us understand how
TSUBAME is being used. Please refer to the following User's Guide for how to submit usage reports.
TSUBAME portal User's Guide Management of TSUBAME usage report

Differences between login node and compute node
The difference between login node and compute node is as follows.

- 9/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Hardware
Login nodeCompute node

# of nodes

2

240

CPU

AMD EPYC 7443 24-Core/2.85GHz x 2

AMD EPYC 9654 96-Core/2.4GHz x 2

Memory256GiB

768GiBGPU

None

NVIDIA H100 SXM5 × 4

Local storage

None

1.92TBThe login nodes are shared servers and are not assumed to be used for calculation purpose. Please avoid high-load processing such as program
execution at the login nodes, execute it on compute nodes though job scheduler.
Please refer to TSUBAME4.0 User's Guide for details.

How many the number of files is acceptable per one directory?
As the number of files per directory increases, the processing time for metadata operations (file creation, deletion, and opening) on the files under the
directory increases, or the file system may generate errors, making it impossible to create files.
Even when using a group disk, it is recommended to arrange files in a hierarchical manner so that there are no more than 100,000 files per directory.
Example:• NG: 000000.dat ～ 999999.dat• If a million files are placed flat in one directory, the load during file access will increase, causing performance degradation and failure.
• OK: 000/000000.dat ～ 000/000999.dat, 001/001000.dat ～ 001/001999.dat, …
• The hierarchical arrangement minimizes the cost of file system operations by limiting the number of files per directory to about 1000.

I'm a beginner, I don't know what to do.
The content depends on what you are a beginner for.
Beginners of UNIX/LINUXUpon using TSUBAME 4.0, users are required to master UNIX/Linux proficiently levels. Handbooks are made on this assumption.
If you do not understand the content of the handbooks, please read the UNIX/Limux beginner's book at the library, and understand how to use UNIX
shells and commands.
There are various publications on the operation of the "terminal emulator" software. Please check this according to your application software too.
First, please understand the operation of UNIX/Linux, then read our guidebooks, then check the section 3.
Beginners of supercomputer
If you have used UNIX/Linux but never used a job scheduler, please read "Job Scheduler" of TSUBAME 4.0 User's Guide.
In addition, TSUBAME 4.0 regularly conducts workshops, check the page of Lectures.
Please also refer to "Introduction to TSUBAME (Linux basics)" and "TSUBAME4.0利用者ガイダンス" posted on Lectures.
Beginners of TSUBAME 4.0
If you have been using TSUBAME 3, please refer to "Migration from TSUBAME 3".
Please also refer to "TSUBAME4.0利用者ガイダンス" posted on Lectures.
Beginners of compile
If you have used UNIX/Linux but never used a commandline compile, please read Intel compiler or NVIDIA HPC SDK or AOCC.
In addition, if you use intel compiler, refer to "並列プログラミング" in Lectures.

- 10/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Beginners of parallel programming(OpenMP,MPI)
Please refer to "並列プログラミング" posted on Lectures.
Beginners of GPU programming
TSUBAME 4.0 regularly conducts workshops, check the page of Lectures.
Please also refer to "GPUプログラミング" and "GPUプログラミング・ハンズオン" posted on Lectures.
Beginners of ISV application software
Please check the application software guide for each. In addition, TSUBAME 3.0 regularly conducts workshops, check the page of Lectures.

About file transfer
File transfer by rsync, scp, and sftp is available on TSUBAME4.0. As well as login, you need to access with your SSH private key which is a pair of yourSSH public key registered in TSUBAME3 portal. Also, please check the settings of the application you are using carefully, as some applications may
time out.
• To install a file transfer application• If you are using Linux/Mac/Cygwin (Windows) (rsync, scp, sftp commands)• To use CIFS accessTo install a file transfer application
If you are using MobaXterm or RLogin, it is easier to use the built-in file transfer function of these software.
If you are using other software such as PuTTY for connection, you need to install a file transfer application such as FileZilla or WinSCP that supports
sftp and rsync protocols. In this case, as well as login, you need to access using SSH private key which is a pair of SSH public key registered in
TSUBAME4.0 portal. For Filezilla and WinSCP, you can use the .ppk format key files that you usually use for PuTTY. For details on how to use eachsoftware, please refer to the manual of each software.
If the option feature "OpenSSH Client" in Windows 10/11 is enabled, you can use scp and sftp command from command prompt or powershell.

If you are using Linux/Mac/Cygwin (Windows) (rsync, scp, sftp commands)
In these environments, rsync, scp, and sftp commands are available. Describes three ways each, rsync, scp, sftp.
rsync:To transfer from the local to the remote host, execute the following command. If you set the standard path/file name as the key pair location, the -i
option is not required.
$ rsync -av --progress -e "ssh -i <Private_Key_File> -l Login_Name" <Local_Directory> <Remote_Host:Remote_Directory>

is the transfer source, is the transfer destination. For example, the command when the user with login name "GSCIUSER00" copies the current directory
to /gs/bs/GSIC of TSUBAME 4.0 using ~/.ssh/ecdsa of private key is as follows.
$ rsync -av --progress -e "ssh -i ~/.ssh/ecdsa -l GSICUSER00" ./ login.t4.gsic.titech.ac.jp:/gs/bs/GSIC

For details such as how to specify the transfer source and tranfer destination, please execute the following command and confirm it.
$ man rsync

scp:
To transfer from local to remote host, execute the following command. If you set the standard path/file name as the key pair location, the -i option is
not required.

- 11/59 -2024-09-30

TSUBAME4.0 FAQ (English)$ scp -i <Private_Key_File> <Login_Name*@*Remote_Host>:<Remote_Directory> <Local_directory>

Please enter the suitable phrase for your situation in the < >.For example, the command when the user with login name "GSCIUSER00" copies thecurrent directory to /gs/bs/GSIC of TSUBAME 4.0 using ~/.ssh/ecdsa of private key is as follows.
$ scp -i ~/.ssh/ecdsa GSICUSER00@login.t4.gsic.titech.ac.jp:/gs/bs/GSIC .

For details such as how to specify the transfer source and tranfer destination, please execute the following command and confirm it.
$ man scp

sftp:
To transfer interactively, execute the following command.
If you set the standard path/file name as the key pair location, the -i option is not required.
$ sftp -i <Private_Key_File> <Login_Name>@<Remote_Host>

For example, the command when the user with login name "GSCIUSER00" copies the current directory to /gs/bs/GSIC of TSUBAME 4.0 using ~/.ssh/
ecdsa of private key is as follows.
$ sftp -i ~/.ssh/ecdsa GSICUSER00@login.t4.gsic.titech.ac.jp

For details such as how to specify the transfer source and tranfer destination, please execute the following command and confirm it.
$ man sftp

To use CIFS accessIn addition, only on-campus terminals can be accessed via CIFS.
The CIFS address is \gshs.t4.gsic.titech.ac.jp. Please note that even in the campus network, CIFS may be blocked by routers on the way, in whichcase it cannot be used.
For storages provided on TSUBAME3, refer to "Storage service (CIFS)" in TSUBAME4.0 User's Guide.

I want to copy a large amount of data from/to TSUBAME
Please consider the following topics to improve the performance of data transfer between TSUBAME and external computers.

Pack the files to appropriate size
Large amounts of small files reduce transfer speed. Pack such files using the tar command to archives of 1GB size each.

Change transfer protocols
If you do not get enough speed with scp / sftp, consider using rsync or CIFS(Tokyo Tech users only) protocols.
For more details on the CIFS connection, please refer "Storage Service (CIFS)" section of the TSUBAME4.0 users guide.

Remove the bottleneck on the network route• If you have old LAN cables (CAT-3 or CAT-5 (not CAT-5e)), switching hubs, or routers whose link speed is lower than 1000 Mbps, replace them withnewer ones.
• When using a router (WiFi router, NAT router, broadband router, etc.), connect your computer to the external network (in Tokyo Tech, IP address
starting with 131.112 or 172.16-31) directly.
For details of the network at Tokyo Tech, please contact the network administrator of the laboratory. If you are not sure, please contact the branchmanager for each building or organization.

- 12/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Tokyo Tech Users Only) Use the iMac terminal of Education Computer Systems
If it is difficult to change the network configuration, you can bring your HDD to the GSIC and connect it to the iMac terminal of the Education
Computer Systems in the exercise room to transfer the data. Please check the opening hours.
Terminal room location and Opening Hours (in Japanese)
Hardware (in Japanese)

How to synchronize data between TSUBAME and PC
The advantage of the rsync command is that it transfers only the difference. If the transfer is interrupted for any reason, you can start again, or if you
run it again after a certain period of time, you can transfer only those files that have changed their content. Data deleted from the source can also bedeleted at the destination for complete synchronization.
An example command is shown below. It's a good idea to check the log or run it multiple times, in case the command fails along the way.
Synchronize TSUBAME with the data of the terminal on your local PC.
rsync -auv (source directory) (your login name)@login.t4.gsic.titech.ac.jp:(full path of the destination directory)

Synchronize TSUBAME data to the terminal on your local PC.
rsync -auv (your login name)@login.t4.gsic.titech.ac.jp:(full path of the source directory) (destination directory)

How to terminate the programs executed accidentally
Terminate the program according to the following procedure when you accidentally executed a program on the login node where the program
execution is prohibited.
See "How to terminate the job submitted to the batch job scheduler" for the deletion of the jobs submitted to the batch job scheduler.

To allow other members to read and write on a group disk
Warning
This article is about [group disks (/gs/bs,/bs/fs)]((../../handbook.ja/storage/#group), do not run the following sample in your home directory.

Users are not allowed to change the owner of their files. Therefore, please change the group permissions so that it can be read and written. The pointis,• Change permissions for all files and directories below the directory, not just the top-level directory.
• Add read (R) as well as write (W) permissions to the file. If you don't have a write (w), you can't erase it later.
• The directory should contain not only read (r) but also write (w) and execute (x). You can't access the directory without the execution (x).
Some example commands are shown below. Depending on the original permissions of the file, some errors may occur, in which case, try re-runningthe command until the output no longer changes.
Find your own directories under /gs/bs/tgX-XXXXXX/ and make them readable and writable by group members.
find /gs/bs/tgX-XXXXXX/ -type d -user $USER ! -perm -2770 -print0 | xargs -r0 chmod -v ug+rwx,g+s

Find your own files under /gs/bs/tgX-XXXXXX/ and make them readable and writable by group members.
find /gs/bs/tgX-XXXXXX/ -type f -user $USER ! -perm -660 -print0 | xargs -r0 chmod -v ug+rw- 13/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Find your own files under /gs/bs/tgX-XXXXXX/ and match the ownership group to the TSUBAME group.
find /gs/bs/tgX-XXXXXX/ -user $USER ! -group (TSUBAME group name) -print0 | xargs -r0 chgrp -v (TTSUBAME group name)

The basic configuration of Module file
The basic configuration of Module file is described below.
• Module file listed as [Application name]/[Version].
• If you do not specify a version, it loads the preset default version
If multiple versions exist, the default version is loaded.
$ module avail cuda--------------------cuda/12.0.0 cuda/12.1.0 cuda/12.3.2$ module list
Currently Loaded Modulefiles:
1) cuda/12.3.2• For applications with dependencies such as MPI, they can be used by loading them in advance.
$ module load openmpi/5.0.2-intel
Loading openmpi/5.0.2-intel
Loading requirement: intel/2024.0.2 cuda/12.3.2

CPU / GPU allocation at resource designation in Altair Grid Engine(AGE)
AGE assigns virtual CPUID / GPUID according to the specified number of resources except node_f.
• In case of CPU
As an example of cpu_8 of a resource type that reserves only one CPU and resource type cpu_4 that reserves 4 CPUs, When s_core=7 is specified,seven nodes are allocated and 1 core of each node is allocated. When q_core=7 is specified, seven nodes are allocated and 4 cores of each node areallocated.
• In case of GPU
In the case of resource type gpu_1 that reserves only one GPU, When s_gpu=4 is specified, 4 nodes are reserved and the GPU of each node is virtuallyassigned as GPU 0. Just because you secured 4, it does not mean GPU 0, 1, 2, 3.
In node_h which is a resource type that reserves 2 GPUs, 2 GPUs are allocated within the node, in this case GPU 0 and 1 are allocated.

How to create an SSH key pair in Linux/Mac/Windows(Cygwin/OpenSSH)
Warning
If your SSH private key is leaked, your account will be misused by the third-party. Please secure your private key with setting the passphrase.

Info
TSUBAME4.0 login node cannot be connected with RSA key (SHA-1). ecdsa key or ed25519 key is recommended.

SSH key pair creation method in Linux / Mac / Windows (Cygwin or OpenSSH) is as follows.
Please check man ssh-keygen command for key type difference.
There are correspondence / unsupported types depending on the version of openssh.

- 14/59 -2024-09-30

TSUBAME4.0 FAQ (English)

ecdsa key type:$ ssh-keygen -t ecdsa

ed25519 key type:$ ssh-keygen -t ed25519

When you execute one of the above commands, you will be asked for the save location as follows.
If there is special circumstance to avoid, such as the same filename is already used for other purpose, just press Enter key to use the default value.
(If you are already using SSH key pair for other sites, you can reuse the same file for TSUBAME)
Generating public/private <keytype> key pair.
Enter file in which to save the key <$HOME/.ssh/id_keytype:> (No need to type filename)[Enter]

Then you will be prompted for a passphrase, so enter it.
Enter passphrase (empty for no passphrase): (Set passphrase; What you type will not appear in screen) [Enter]

Re-enter your passphrase for confirmation.
Enter same passphrase again: (Enter the same passphrase again for confirmation; What you type will not appear in screen) [Enter]

A key pair is created and saved to two files. The upper line shows the location of private key, and the lower line shows that of public key. Register the
public key via TSUBAME portal.
our identification has been saved in *$HOME/.ssh/id_keytype*
Your public key has been saved in *$HOME/.ssh/id_keytype* .pub.
The key fingerprint is:
SHA256: *random number:username@hostname*
The key's randomart image is:(鍵Some text specific to the generated key pair will be shown)

Check the file with the following command.
$ ls ~/.ssh/ -l
drwx------ 2 *user* *group* 512 Oct 6 10:50 .
drwx------ 31 *user* *group* 4096 Oct 6 10:41 ..
-rw------- 1 *user* *group* 411 Oct 6 10:50 *private_key*-rw-r--r-- 1 *user* *group*
97 Oct 6 10:50 *public_key*

If the permissions are incorrect, correct them with the following command.
$ chmod 700 ~/.ssh$ chmod 600 ~/.ssh/*private_key*

How to create an SSH key pair using PuTTY or MobaXterm
Warning
If your SSH private key is leaked, your account will be misused by the third-party. Please secure your private key with setting the passphrase.

Info
TSUBAME4.0 login node cannot be connected with RSA key (SHA-1). ecdsa key or ed25519 key is recommended.

This article describes how to create SSH keypair to be used TSUBAME3 using PuTTYgen, which will be installed with PuTTY. MobaKeyGen from
MobaXterm has the same functionarity and UI.
You will get a dialog similar to this by executing PuTTYgen:- 15/59 -2024-09-30

TSUBAME4.0 FAQ (English)

4. Copy from here

2. Generate Key-pair

3. Save Private Key
1. Choose Key Type1. Select the key type in "Type of key to generate".
ECDSA or EdDSA format is recommended. For other formats, you may not be able to connect to TSUBAME4.0.
2. Press "Generate" to create SSH key-pair
You can adjust key-pair configurations using "Parameters" box, but you don't have to do so in most cases.
3. Press "Save private key" to save a private key file of generated key-pair for future login.
After registering public key to TSUBAME4.0, anyone who can read this private-key file can log in to TSUBAME4.0 with your account.
Please keep this file safe, DO NOT carry with USB stick, or send via e-mail etc.
You can force to enter passphrase to use this key file, by inputting them in "Key passphrase" and "Confirm passphrase" boxes before saving.
4. Copy all texts in "Public key for pasting..." box, paste it into "Enter SSH public key code" in "Register SSH public key" menu in TSUBAME portal, and thensubmit this.
When you want to login into TSUBAME, open the file saved in step 3. with Pageant (.ppk files are associated with Pageant if you install PuTTY using
installer with default options) in advance, or specify the file in "Private key file for authentication" box under "Connection"-"SSH"-"Auth" menu in PuTTY
connection setting dialog.

Warning
Never choose Parameters' SSH-1 (RSA).

Please refer to the manual for information on how to use PuTTY.

About common errors in Linux
Here we have a FAQ on Linux common errors.
For details on how to use the described command, please check with the man command etc.

- 16/59 -2024-09-30

TSUBAME4.0 FAQ (English)

No such file or directory
There is no required file or directory.
It occurs when specifying a nonexistent file, directory name, etc., typing, or incorrect path specification.
Also, depending on the application, it may occur when the line feed code is CR + LF on windows.
Measures
Please review the file and directory name carefully. Also, please check FAQ "The job status is "Eqw" and it is not executed." about the newline
character.
There are related errors as follows.
error while loading shared libraries: ****.so: cannot open shared object file:

No such file or directoryThis is an error that occurs when there is no library required by the program or can not be read.
Measures
Please check with ldd command. There is a way to set the environment variable LD_LIBRARY_PATH, explicitly specify the library at compile time, andso on.

command not foundThe command you entered does not exist.
Depending on the type of program you wish to run, perform the following checks.
• Commercial Application• You may not have purchased the application. Please refer to the FAQ below.
• Commercial applications (Mathematica, Gaussian, etc.) do not work. Command not found.
• I purchased a commercial application, but it is not available.
• You may not have executed the module command. Please load the necessary module.
• Freeware• You may not have executed the module command. Please load the necessary modules.
• Program you've installed yourself• The environment variable PATH may not be set correctly.
Make sure you use the "echo $PATH" command to set the path to the directory that contains the command you need, and if not, set it.
Example of adding the "hoge" directory directly under the home directory ($HOME) to the existing environment variable "PATH"$ export PATH=~/hoge:$PATH

Permission denied
You are not authorized to perform the operation you attempted to perform.
Linux and user and group permissions are set on a file / directory basis.
Check the authority of the target file or directory you want to read or write or execute with the following command.
(When checking the hoge file for an example)$ ls -l hoge

Measures If you are trying to create files in / etc, / lib etc which are system directories etc, please make it in the user directory.
If it occurs in a user directory such as a group disk, check the authority and please do.

Disk quota exceeded
Please check FAQ How to solve "Disk quota exceeded" error"".

- 17/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Out Of Memory
This error occurs when memory runs out.
Measures
Change the resource type to one with more memory capacity.
Divide the memory usage per node with mpi etc.
Related FAQ "Check the detail of an error message printed the log file"

Related URL• How to solve "Disk quota exceeded" error• The error when executing the qrsh command• Check the detail of an error message printed the log file• "Warning: Permanently added ECDSA host key for IP address 'XXX.XXX.XXX.XXX' to the list of known hosts." in the error log• Errors and remedies of qsub command execution• The range of support by T4 Helpdesk about the program error such as segmentation fault• Error handling for each ISV application

How to solve "Disk quota exceeded" error
This message indicates there is no space left in ether a home directory or a group disk.
When you face it, you should delete unused files or purchase an additional group disk to keep enough free disk space.
Please note that temporary files are generated at the home directory in some cases, and an application sometimes needs over 25 GB of a disk spacefor creating temporary files. (25 GB is the capacity of a home directory)
To avoid running out of disk space, we recommend not to use home directory but to use local scratch area or shared scratch area for temporary filelocation.
** Related FAQ ** How to check TSUBAME points, group disk usage, home directory usage

Session suddenly disconnected while working on TSUBAME4.0
For TSUBAME4.0, session timeout is set for security measures.
Sessions that do not have any input at a certain time are disconnected.
Even if the GUI application is started up and operated, if there is no input to the terminal, it will be disconnected.
If you want to avoid this, please set keep alive on the terminal side.
Please check the user guide of the terminal for keep alive setting.

Available SSH client on Windows
The following SSH clients on Windows are available to connect TSUBAME.

OpenSSH client (Windows 10/11 functionality)
OpenSSH client can be installed via [Apps]-[Manage optional features] section in Settings app.
ssh, ssh-keygen, etc commnads(same as linux) are available after the installation.

PuTTY
Official Site

- 18/59 -2024-09-30

TSUBAME4.0 FAQ (English)

PuTTY is a free SSH Client software. Please refer to this article to generate the SSH key.

MobaXterm
Official Site
This software includes an SSH client and an X11 server.
Most X11 applications on TSUBAME seem to work fine.
Please refer to this article to generate the SSH key.

Window Subsystem for Linux (WSL)
Linux environments can be constructed on Windows by downloading Linux distribution(such as Ubuntu, OpenSUSE) from Windows 10/11 store.
ssh, ssh-keygen commands are available from that.

CygwinOfficial Site
Cygwin provides a pseudo-Linux environment on Windows.
ssh, ssh-keygen commands are available from that. We strongly recommend using other software.

The combination of compiler and mpi module
GNU, Intel oneAPI, NVDIA HPC SDK and OpenMPI can be used in combination.
gcc is provided by the OS. Please check the available version with the following command.
$ gcc --version$ module avail

1. Intel OpenMPI$ module load openmpi/5.0.2-intel
Loading openmpi/5.0.2-intel
Loading requirement: intel/2024.0.2 cuda/12.3.2$ mpicc -v
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /apps/t4/rhel9/isv/intel/compiler/2024.0/bin/compiler
Configuration file: /apps/t4/rhel9/isv/intel/compiler/2024.0/bin/compiler/../icx.cfg
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/11
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/11
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /apps/t4/rhel9/cuda/12.3.2, version

2. GNU OpenMPI$ module load openmpi/5.0.2-gcc
Loading openmpi/5.0.2-gcc
Loading requirement: cuda/12.3.2$ mpicc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-host-pie --enable-host-bind-now --enable-languages=c,c++,f
ortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/
bugzilla --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit -disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-plugin --enable-initfini-array --without-isl --enable-multilib --with-linker-hash-style=gnu --enable-offload-ta
rgets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_6
4=x86-64-v2 --with-arch_32=x86-64 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serial
ization=1
Thread model: posix- 19/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Supported LTO compression algorithms: zlib zstd
gcc version 11.4.1 20230605 (Red Hat 11.4.1-2) (GCC)

3. NVIDIA HPC SDK OpenMPI$ module load openmpi/5.0.2-nvhpc
Loading openmpi/5.0.2-nvhpc
Loading requirement: nvhpc/24.1$ mpicc -v
Export NVCOMPILER=/apps/t4/rhel9/isv/nvidia/hpc_sdk/Linux_x86_64/24.1
Export PGI=/apps/t4/rhel9/isv/nvidia/hpc_sdk
nvc-Warning-No files to process

When trying to save a file with Emacs during an interactive job, the screen froze
It is caused by the flow control by specific input characters, enabled on the default terminal setting.
Flow control is a function to temporarily hold a data transfer in order to prevent overflow of the receiving side, for example, when the transmission
speed exceeds the reception packet speed in data transfer. In general, Ctrl+S is used for pending transfers, and Ctrl+Q is used for restart as control
characters.
When editing with Emacs interactively and overwrite saving, you have to enter Ctrl+S, but it is also a flow control character, so the packet will not betransferred and it will be as if it were frozen. To fix, please enter Ctrl+Q.
To disable flow control, you need to execute the following command before running interactive job.
$ stty -ixon

If you want to always disable flow control, add the above command to .bashrc in your home directory.

Cannot login to TSUBAME4.0 (ssh, Permission denied (publickey,hostbased) etc.)
Please refer the following checklist before contacting us for help.
1. Is your account name correct?
Please confirm you are using TSUBAME4.0 account.
• TSUBAME4.0 account is different from TSUBAME3 account. We are receiving an increasing number of inquiries using TSUBAME3 accounts.
• Your TSUBAME4.0 account is different from your student number or faculty and staff number.
• Do you have a TSUBAME4.0 account? For information on obtaining an account, please refer to How to obtain an account.
2. Did you registered a public key in correct format?
Please confirm that you have registered a public key in OpenSSH format to TSUBAME portal.
You can not login to TSUBAME3 if you registered a public key in PuTTY format.
Please refer to the following links for how to create a key pair.
• How to create an SSH key pair in Linux/Mac/Windows(Cygwin/OpenSSH)• How to create an SSH key pair using PuTTY or MobaXterm
Please refer to the following for how to register the public key.
Registering SSH public key

Info
TSUBAME4.0 login node cannot be connected with RSA key (SHA-1). ecdsa key or ed25519 key is recommended.

3. Is the command you entered correctly? ( for Linux / Mac / Windows(Cygwin) )- 20/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Please make sure your login name and path to your private key file (*) are specified in the command line correctly.
$ ssh <TSUBAME4 account>@login.t4.gsic.titech.ac.jp -i <private key>

Example) When your login name is gsic_user and the location of a secret key is ~/.ssh/t4-key,$ ssh gsic_user@login.t4.gsic.titech.ac.jp -i ~/.ssh/t4-key

Tips
If your private key file is stored in one of the following locations under your home directory (the case that you have not modified the location from thedefault value), you can omit "-i " part.
.ssh/id_rsa, .ssh/id_dsa, .ssh/id_ecdsa, .ssh/id_ed25519

Info
TSUBAME4.0 login node cannot be connected with RSA key (SHA-1). ecdsa key or ed25519 key is recommended.

Please refer to the following man command for options of ssh command.
$ man ssh

4. Does the symptom reproduce in another terminal environment?
There are various types of terminal software for Windows.Please check whether it reproduces even with another terminal software.
If not reproduce, it may be a software-specific problem.In that case we can not respond to your inquiries, so please understand.
5. Do the symptoms reproduce with different network?
There may be a problem with the network from which you are accessing the site and you may not be able to connect.
If there are multiple access routes, change the access source and see if the problem reproduces itself.
• On campus/off campus• Lab/other labs
• Home/public WiFi
If the connection status changes when the network is changed, there may be a problem with the router, firewall, or other settings. Please check thesettings with your administrator.
Also, in the case of Windows, security software may be blocking communication. Please temporarily turn off your security software and see if you
can connect. Please refer to the manual of your security software for the workaround.
If you do not solve any of the above solutions, please contact us with the following information.
• Operation system (Windows10, Debian12, macOS 14.4.1 and so on)• Terminal software and the version (Cygwin, PuTTY, Rlogin and so on)
For details on how to check the version, please refer to the terminal software manual.
For Linux / Mac OS, please send SSH version. You can check with the following command.
$ ssh -V

• The operation you tried. If you get an error, please send the details.
For Linux / Mac OS, please send the output of ssh command with -v option (debug mode) including the command line itself.
Example) When an account name is gsic_user and the location of a secret key is ~/.ssh/t4-key,$ ssh gsic_user@login.t4.gsic.titech.ac.jp -i ~/.ssh/t4-key -v- 21/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Can not establish CIFS connection to the group disk, Unable to open TSUBAME group disk on
Windows
Info
Access to the TSUBAME group disk using CIFS is only available within the university. It cannot be accessed from off-campus.

Please check network communication to the CIFS server.
Please check the PING using a DOS prompt etc.
C:\> ping gshs.t4.gsic.titech.ac.jp

To access the group disk from Windows, you need to set a TSUBAME password. Please configure from the TSUBAME portal. For the setting method,
please see here. And, if the message "Password is incorrect" or "Password has expired" is displayed, please reset your TSUBAME password.

FAQ about group disk
About group disk
The Group disk is the high-speed storage area (SSD) and large-capacity storage area (HDD) described in the "Usage Guide". This is a shared storage
that allows each group to use the capacity set on the TSUBAME portal.
Usage period: In one-month increments until the end of the fiscal year (end of March) including the month of purchase.
Point and inodes per purchase unit
Type

Purchase unit

Point

inode

large-capacity storage area (HDD)

1TB

0.5

2,000,000

high-speed storage area (SSD)

100GB

0.2

200,000

How to set: you can set from the TSUBAME Portal Reference:TSUBAME Portal User's Guide "10. Management of Group Disk"

What is the group disk grace period ?
Group disks are reset once at the end of the fiscal year, and all group disks are in a grace state that can only be read/deleted. This period is called thegrace period, and usually it will be maintained around the middle of April.
Reference:TSUBAME Portal User's Guide
If the data of the previous year remains and you purchase it after the grace period, it becomes as follows.
For example, if you purchased 50TB in the previous year and you used a capacity of 45TB.
1) When 45TB is deleted during the grace period and the used capacity is 0.
You can purchase from 1TB which is the minimum capacity.
2) When 25TB is deleted during the grace period and the used capacity is 20TB.
Available from over 20TB.
3) If the used capacity is not deleted during the grace period (used capacity is 45TB)
Available from over 45TB.
If you do not need the previous year's data, please delete it during the grace period.

- 22/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Related FAQ• Checking the usage of group disks with command• Can not establish CIFS connection to the group disk
• "Disk quota exceeded" error is output

About the IP address of the gateway server for compute nodes(connection to license servers
outside TSUBAME, etc.)
The IP address range of the compute node gateway server is as follows.
131.112.133.241, 131.112.133.242

When computing on TSUBAME by using a campus or university license server, please set so that communication within the above range is permitted.
Please keep in mind that the above address may be changed without notice from the circumstances of operation.
If your software requires communication with a license server outside of TSUBAME (e.g., in a laboratory), please confirm that you can communicate
with the license server from a network outside of TSUBAME and outside of the license server before contacting us with the following information.
• Global IP address of the license server• Port number of the license server (or all ports if there are more than one)
• IP address of the host where the communication test was performed

An error such as "fork: Resource temporarily unavailable" is displayed on the login node.
The login node has a limitation of 50 processes per user. Therefore, if you create a process that exceeds the limit, you will get an error like this. For
more information, please refer to Please refrain from occupying the CPU in the login nodes..

Can Docker be used with TSUBAME4.0?
Can Singularity be used with TSUBAME4.0?
Can container be used with TSUBAME4.0?
Docker cannot be used with TSUBAME4.0. Apptainer(Singularity) is available.
For more information, see Use containers.

Can I use Jupyter Lab with Tsubame 4.0?
Jupyter Lab is available in Tsubame4.0. See Open OnDemand User's Guide.

The group disk suddenly became unusable.
Since a group disk is allocated every month, the amount of the group disk used may exceed the allocated size when the month is crossed.
If this situation continues, all access to the target group disk will be prohibited at a specific time.
- If you wish to check the usage status of the group disk, please refer to Confirmation of group disk usage.
- For information on what to do when the amount of group disk usage is exceeded, please refer to What to do when the group disk usage exceeds thereserved size.
Please also refer to FAQ about group disk.

- 23/59 -2024-09-30

TSUBAME4.0 FAQ (English)

I would like to know how to utilize the GPU in TSUBAME4.0.
The following documents, which are available on the Lectures page, may be helpful.
- TSUBAME4のGPUを最大限活用する方法

I want to use a debugger/profiler that supports multi-threading/multi-processing
Linaro Forge(ex:Arm forge) is available.
Please refer to "Parallel Programming" on the Lectures page for information on how to use it.

OpenOnDemand or Jupyter fails to start
If you experience any of the following problems when using OpenOnDemand (TSUBAME Desktop) or Jupyter, your configuration file may becorrupted.
- Application does not start - Application terminates abnormally after startup
Delete the following directories
Application

Directory path

OpenOnDemand(TSUBAME Desktop)~/ondemand

Jupyter~/.jupyter

Info
These directories are automatically created at the time of use. Deletion of these directories usually has no effect.
If you edit files in these directories and experience problems, your inquiry is not covered.

- 24/59 -2024-09-30

TSUBAME4.0 FAQ (English)

The procedure for deleting a directory on OpenOnDemand is as follows- 25/59 -2024-09-30

TSUBAME4.0 FAQ (English)

1. Log in to OpenOnDemand. For instructions, see Login to Open OnDemand for instructions.
2. Click Files - Home Directory.

3. Check the Show Dotfiles checkbox.

4. Select the directory to be deleted.
4-1. For Jupyter Click ⋮ next to the .jupyter directory and click Delete. (Directories beginning with . )4-2. OpenOnDemand(TSUBAME Desktop)の場合- 26/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Click ⋮ next to the ondemand directory and click Delete.

5. A confirmation dialog box will appear. Confirm that it is the directory to be deleted and click OK.

GLIBC not found error when using Apptainer
When using the fake root function (--fakeroot) in Apptainer, the libc version must be matched between the host and the container.
If they do not match, you may get the following error.
/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.33' not found (required by /.singularity.d/libs/faked)/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.34' not found (required by /.singularity.d/libs/faked)
fakeroot: error while starting the `faked' daemon.
/.singularity.d/libs/fakeroot: 1: kill: Usage: kill [-s sigspec | -signum |-sigspec] [pid | job]... orkill -l [exitstatus]

For details, see Precautions when using fakeroot .

I want to use VS Code.
Access to login nodes using VS Code is prohibited because it places a heavy load on the login nodes.
Please refrain from occupying the CPU in the login nodes.
In TSUBAME4.0, it is possible to connect to compute nodes via Open OnDemand using code server.
Please consider using code server. Please refer to Open OnDemand User's Guide for details on how to use it.

- 27/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Accounts
Can not find user even though searched when adding a user to a group
If the target user can not be searched on the user addition screen of the portal, there are the following reasons.

TSUBAME account application has not been completed
The TSUBAME account is not automatically created as you enroll, and each user needs to apply for an account via the Tokyo Tech Portal. Until theaccount application is completed, it will not appear in the account search by the group administrator.
In addition, if the applicant is an owner of an access card, the account application from the TSUBAME portal will only be valid if it has the same status
as the Tokyo Tech staff/student.
Also, because it is not automatically approved, it is necessary to send a document certifying your identity. So please complete the procedure withreference to the explanation of that page.
For details of account application, please refer to "Getting accounts".

Attribute of user and attribute of TSUBAME group do not match
When creating a TSUBAME group, you should have set up what kind of users can belong as group divisions. You can not add users who are out ofthat condition. For example, you can not add external users to "Only teachers and students at Tokyo Institute of Technology Group". In such a case, amessage like "This user is not eligible to participate" is displayed. Because group classification can not be changed afterwards,so please create agroup of appropriate attribute that allows you to subscribe to that user.
In addition, if a new entitlement to an existing account occurs, such as when a user using HPCI newly started a joint research with Tokyo Institute ofTechnology with competitive funding, by appending an existing account name when applying for the new qualified account, you will be able to belong
to that group.

Account Procedures for Re-Issuance of Tokyo Tech IC Card after Expiration Date
If your Tokyo Tech IC card has expired and your card has been reissued, you can activate your account by logging in to Tokyo Tech Portal, and thenlogging in to the TSUBAME Portal page from “General Information Services” - “TSUBAME4 Portal”. General Information Services” - ‘TSUBAME4 Portal’and log in to the TSUBAME Portal page to activate your account.
After your account is activated, you can login to TSUBAME Portal Page directly.

- 28/59 -2024-09-30

TSUBAME4.0 FAQ (English)

TSUBAME point and TSUBAME portal
TSUBAME portal is not displayed, or does not work properly• If JavaScript is disabled, please enable it and try again.
• If it does not work with your browser, please use another browser such as Edge, Chrome, Firefox, Safari.

The screen does not display properly in the process of joining the group
If you can not join the group just by displaying the login screen, when clicking on the URL on the mail titled "TSUBAME 4.0 TSBAME group user
invitation", please check below.
• Login and keeping the logged-in browser open, click the mail link again.
• Depending on the mailer, the end character "=" may not be included in the link. In that case, copy paste it to the browser including "=".
• The deadline of the invitation URL is one week. If it expires, try to join the group again.

Time lag between synchronizing with the login node of the information updated by the portal system
It may take up to 5 minutes for the information to be reflected on the login node after adding users, groups from the portal system. If information is
not updated from the login node, please wait for about 5 minutes before operation.

Supplement for TSUBAME point
TSUBAME point is consumed when you submit a job or purchase a group disc.
1. How to confirm points
You can check the following information at TSUBAME4.0 Portal and command line. In the case of the portal, you can check it below.
• Number of points consumed for each job submitted to the batch job scheduler• Points consumed in purchasing group disc
For the command line please check the following
FAQ "How to check TSUBAME points, group disk usage, home directory usage"
2. Consumption of points
Points will be consumed based on Article 13 here.
The points consumed depend on the user and usage.
Please confirm here for details.
3. Purchase points
Points can be purchased at the TSUBAME 4.0 portal.
For details, please check TSUBAME Portal User's Guide.
4. Point expiration date
Points purchased within the fiscal year will expire at the end of that fiscal year. For details, please refer to the following (in Japanese).
• Points available for purchase and validity period• "Treatment concerning billing. 6. points are valid only for purchased fiscal year"- 29/59 -2024-09-30

TSUBAME4.0 FAQ (English)

The status of "Payment code status" on TSUBAME portal remains "予算責任者の追認待ち利用停止""予算責任者の追認待ち" is a state waiting for approval from the budget manager of applied payment code.
Please ask the budget manager to approve in TSUBAME portal.

How to check TSUBAME points, group disk usage, home directory usage
TSUBAME points, group disk usage status, home directory usage status can now be checked with commands. The command "t4-user-info" can beused only with the login node(login1,login2) . It can not be executed on compute nodes.
Each command option is required for confirmation.
GSICUSER@login1:~> t4-user-info
usage: t3-user-info [command] [sub command] [option][command] [sub_command] [option]
group
point
: Output the points of the Tsubame Group.
: Without option : show all belonging groups.
[-g] <group name> : extract the specified group.
disk
group
: Output the purchase amount and use amount of Tsubame Group disk.
: Without option : show all belonging groups.
[-g] <group name> : extract the specified group.
home: Output the use limit and used of the home disk.

Command examples for checking group points.
In the following example, it is assumed that "GSICUSER" user participating in the group "GSIC_GROUP", "GSIC" executes the command.
Please use "your user name" and "participating group name" when actually executing the command.
1. When checking the situation of all participating groups
You can check the situation where TSUBAME points of "GSIC_GROUP" and "GSIC" of participating group are 17218631 and 95680000 , and deposit is0, 124000 spectively.
GSICUSER@login1:~> t3-user-info group point
gidgroup_namedepositbalance-------------------------------------------------------------0007
GSIC_GROUP
017218631
0451
GSIC
124000 995680000

2. When checking the status of the designated group
You can check the situation where the TSUBAME point of the specified GSIC_GROUP group is 17218631 and deposit is 124000.
GSICUSER@login1:~> t4-user-info group point -g GSIC_GROUP
gid
group_namedepositbalance-------------------------------------------------------------2007
tga-hpe_group00
12400017218631

To check the usage status of the group disk
In the specified GSIC_GROUP group, only /gs/bs is purchased, about 60 TB of the 100 TB quota limit is used,
Regarding the inode limit, we can check the situation of using 7.5 million out of the 200 million quota limit.
GSICUSER@login1:~> t4-user-info disk group -g GSIC_GROUP/gs/fs/gs/bsgid group_namesize(TB) quota(TB)
file(M) quota(M) size(TB) quota(TB)
file(M) quota(M)--------------------------------------------------------------------------------------------------------------0007 GSIC_GROUP
0.00
0
0.00
0
59.78
100
7.50
200

When checking the use status of the home directory
Of the 25 GB quota limit, 7 GB is used,
Regarding the inode limit, we can check the situation that we are using approximately 100,000 out of the 2 million quota limit.

- 30/59 -2024-09-30

TSUBAME4.0 FAQ (English)

GSICUSER@login1:~> t4-user-info disk home
uid nameb_size(GB) b_quota(GB)i_filesi_quota--------------------------------------------------------------0177 GSICUSER
7251014462000000

When are TSUBAME points consumed and returned?(liquidation of temporary points).
TSUBAME4.0 collects TSUBAME points as "temporary points" that are expected to be needed when submitting a job, and settles the actual
consumption points after the job is finished. The timing of TSUBAME point consumption and return is as follows.
However, delays may occur under certain load conditions, such as when a large number of jobs are deleted at once or when a failure occurs.
The timing of TSUBAME point consumption and return is as follows.
1. When a job is submitted or qrsh command is executed
The maximum TSUBAME points that a job can consume are collected as temporary points.
2. When a job is terminated
The timing of settling provisional points is different in the following three ways
a. When a job finishes as usual, or when a job is canceled by the qdel command after it has started running
Recalculate the consumption points based on the time actually used by the job, and return the difference immediately.
b. When a job is canceled by the qdel command before execution starts, or when the qrsh command fails to start
The temporary points will be automatically returned.
c. When a job is deleted by the scheduler or the system administrator due to a system error
When a job is deleted by the scheduler or the system administrator due to a system error, it is basically settled based on the time when the job wasdeleted as in a. If the cancellation is clearly due to system reasons, we will compensate you for the wasted TSUBAME points if you contact us.
If you have any questions about TSUBAME points, please contact us using the inquiry form with the following information.
• The TSUBAME4.0 group that submitted the job• User who submitted the job• Job ID
Reference : The error when executing the qrsh command

The link to TSUBAME Portal in the auto-mail does not work properly.
Some links automatically generated by the TSUBAME Portal, such as links to approval pages of TSUBAME group invitations, may not function
properly depending on your browser or mailer environment. Specifically,• The login page of the TSUBAME Portal is displayed instead of being redirected to the correct page.
• The message like "The referenced URL has expired" is displayed.
If you see this kind of symptom, try copying and pasting the address sent by e-mail to the address bar of the window in which you are logged in fromthe Tokyo Tech Portal to TSUBAME Portal, and then press the Enter key.
If you still think the system is not working properly, please contact us using the contact form with the following information.
• Your TSUBAME account login name
• The date and time of the email sent to you• The approximate date and time of the first click on the link
• Other related TSUBAME Group information, etc.

- 31/59 -2024-09-30

TSUBAME4.0 FAQ (English)

My application for a payment code was rejected, but why?
When you apply for a payment code on TSUBAME Portal, you will receive a notice of rejection from the TSUBAME Portal if your application isincomplete.
If your application is incomplete, you will receive a rejection notice from TSUBAME Portal. Please check the reason for the rejection in the "Commentsfrom the system administrator" section of the notice, and make corrections if necessary.
The following is a list of typical reasons for rejection and what you need to check when you resubmit.
1. The existence of the corresponding budget could not be confirmed (該当する予算の存在を確認できませんでした)
The above message is sent when the person in charge has checked the existence of the budget code and budget name pair entered in the payment
code application on the financial accounting system, but could not find the corresponding budget, including the possibility of a typo.
If the budget is for external funds or Grant-in-Aid for Scientific Research, please make sure that the budget code for the current fiscal year has alreadybeen created in the Request for Supplies system.
Also, please make sure that the budget code, budget name, budget department, budget category, and person in charge of the budget are enteredcorrectly, because if they are not, we will not be able to search them.
In particular, the budget code (32 digits) and budget name have a large number of digits in the financial accounting system from FY2020, and some
applications may be missing some of them when transcribing, so we strongly recommend that you follow the procedure below to obtain them.
• Log in to the New Goods Request System• Select Budget Management• Select Check Budget Execution Status• Select Create CSV• Post the "Budget Code" (32 digits) and "Budget Name" in the output CSV file.
2. I was able to confirm that I had the budget code, but XXX was different. (予算コードがあることまでは確認できましたが、XXXが異なっていました)The above message is sent when a budget that seems to be applicable is found based on the information entered in the payment code application,
but some items do not match the registered contents on the financial accounting system and are incorrect beyond the scope that can be corrected at
the discretion of the person in charge as a typo.
Please refer to the previous section, check the registration details displayed in the goods billing system, and then reapply with the correct details.
Please refer to the previous section to check the registration details displayed in the requisition system, and reapply with the correct details. Please
note that if the XXX part is the name of the person responsible for the budget, please refer to the next section.
3. I was able to confirm that there was a budget code, but the name of the person responsible for the budget was different. (予算コードがあることまでは確認できましたが、予算責任者氏名が異なっていました)
The above message will be sent when the budget manager's account information on the payment code application does not match the budget
manager's information on the financial accounting system.
TSUBAME requires budget managers, including those who do not directly use TSUBAME's computing resources, to obtain a TSUBAME account, agree
to the Terms of Service, and fill out their account information in order to ensure that we can communicate with them about their billing needs.
In addition, if the user of the payment code (the main administrator of the TSUBAME group) is different from the budget manager, the budget
manager is required to approve the application for the payment code on the TSUBAME portal in order to confirm that the use of the budget is allowedin advance.
4. Rejected because the expense category is not "Other" (費目がその他ではないため、却下します) (Grant-in-Aid for Scientific Research)
You will receive the above message when you specify a budget code for an expense category other than "Other" of the Grant-in-Aid for Scientific
Research (goods, travel, and personnel expenses) in the payment code application.
There are four types of budget codes for Grants-in-Aid for Scientific Research, and TSUBAME's computer usage fees are classified as "Other"
expenses. (This corresponds to "Facility and equipment usage fees within the research institution" in the table of expense categories common to all
government ministries and agencies.)
For this reason, applications for payment codes for expenses other than "Other" will be rejected as an incorrect expense category.

- 32/59 -2024-09-30

TSUBAME4.0 FAQ (English)

5. If you are not a faculty or staff member, you must be responsible for your own budget. (教職員以外の方は、自身が予算責任者となっている申請に限らせていただいております)
The above message is sent to non-faculty members when they apply for a payment code for a budget for which they are not the budget manager.
Non-faculty members are not allowed to be the payer of TSUBAME for budgets other than those for which they are the budget manager (e.g., Grant-inAid for Scientific Research by JSPS Postdoctoral Fellows).
If you need to use such a budget, please ask an appropriate faculty or staff member to be the payer and reapply for the payment code.
6. The period in which new claims can be generated has already passed. (既に新規請求事項発生可能期間を過ぎています)The above message will be sent when you cannot purchase new TSUBAME points for your requested budget.
In TSUBAME, purchase operations on the TSUBAME portal are grouped together on a monthly basis, and the budget is transferred to the next monthor later.
We have established a period of availability for payment codes so that this transfer process can be done within the budget's available period (within
the fiscal year and research period), but we will not accept applications for budgets that have already passed this period and cannot be used topurchase points.
Please note that during the period from January to March, only corporate operating expenses and scholarship donations can be registered and used
for payment codes, but the amount used from January to March will be charged to the same budget for the following year.
7. The system administrator has already approved the application for the same fiscal year and the same budget code. (既に同じ利用年度かつ同じ
予算コードの申請をシステム管理者が承認済みです)
You will receive the above message when a payment code has already been approved for the same budget.
Please use the approved payment code.

Approved payment code cannot be added to group
The only payment codes that can be set for a group are those applied for by the group manager.
Payment codes applied by anyone other than the Group manager cannot be used.
For detail, see Management of payment codes and Set payment code to group.

- 33/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Job Execution (Scheduler)
I get an error when submitting a job, but I do not know which option is bad
The correspondence varies depending on the error.

qsub: Unknown option
The "qsub: Unknown option" error also occurs when there is an error in the line description starting with "#$" in the job script, besides the option of theqsub command. A common mistake is putting a space before and after the character "=". Please try deleting the space around "=".

Job is rejected, h_rt can not be longer than 10 mins with this group
If you do not specify the TSUBAME group with the -g option or newgrp, it is considered "Trial run".
"Trial run" has limitation only within 10 minutes, this error occurs when the specified h_rt option specifies more than 10 minutes.
In case of "Trial run" please modify the h_rt option to 0:10:0.
If you want to execute it other than "Trial execution", specify the TSUBAME group with the -g option or newgrp.

Info
In this case, please confirm that you are participating in the appropriate TSUBAME group and that the TSUBAME group has points. For the TSUBAME
group please check TSUBAME portal usage guide

Unable to run job: Job is rejected. h_rt must be specified.
It can not be executed because there is no description of h_rt option. Please set time and execute.

Unable to run job: the job duration is longer than duration of the advance resavation id AR-ID.
This error occurred because you specified a time longer than the reserved time.
Please refer the reservation of FAQ below.
Related FAQ About specification of batch job scheduler

error: commlib error: can't set CA chain
This error occurs when the home directory of TSUBAME3.0 is migrated (copied) to TSUBAME4.0 as it is.The error occurs because the certificate
remains in the ~/.sge directory used by the TSUBAME3.0 scheduler and conflicts with the certificate used by the TSUBAME4.0 scheduler.
Since ~/.sge directory is not used by TSUBAME4.0, please delete it as follows.
$ cd $HOME$ mv .sge .sge.back

If you remove ~/.sge and a job cannot be submitted even though the .sge directory does not exist, [Contact us]((https://www.t4.gsic.titech.ac.jp/en/
contact-t4).

The job status is "Eqw" and it is not executed.
There is a possibility of a system failure, but it may be due to a job script mistake.
Please confirm with the following command.

- 34/59 -2024-09-30

TSUBAME4.0 FAQ (English)$ qstat -j <job ID> | grep error

Please confirm the following points. After checking, delete jobs with "Eqw" status with qdel command.
Example)When there is a problem with file permission.
error reason

1:

time of occurrence [5226:17074]: error: can't open stdout output file "<File of the cause>": Permission denied

When there is no line feed code problem, directory does not exist, or job script is invalid.
error reason
1:
time of occurrence [5378:990988]: execvp(/var/spool/age/<hostname>/job_scripts/<jobID>, "/var/spool/age/<hostname>/job_scripts/<jobID>") failed: No such file or directory

1. The line feed code of the job script is not in UNIX format(LF).
If the line feed code is set to CR + LF on windows, it will also occur, so please confirm with the actual script together.
You can confirm with the file command.
$ file <script file name>• Output in case of CR + LF
<Script file name>: ASCII text, with CRLF line terminators• Output in case of LF<Script file name>: ASCII text

You can also confirm cat command.
$ cat -e <script file name>• Output in case of CR + LF
The end of line is displayed as ^M$#!/bin/bash^M$#$ -cwd^M$#$ -l node_f=1^M$#$ -l h_rt=0:10:00^M$
module load intel^M$• Output in case of LF
The end of line is displayed as $#!/bin/bash$#$ -cwd$#$ -l node_f=1$#$ -l h_rt=0:10:00$
module load intel$

The measures for line feed code are as follows• Do not edit scripts on Windows• When editing a script on Windows, make sure to check the line feed code by using an editor corresponding to the line feed code.
• Correct the line feed code to LF with nkf command.
In case of other than LF, execute below command.
$ nkf -Lu file1.sh > file2.sh- 35/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Warning
file1.sh is an original file (before conversion) and file2.sh is a converted file, respectively. Both file names must be different. If their names are identical, it
will be corrupted.

2. There is no such directory to be executed
Occurs when the execution directory described in the job script does not exist.
Please confirm with the following command.
$ qstat -j<Job ID> | grep ^error

error reason

1:

09/13/2024 12:00:00 [2222:19999]: error: can't chdir to /gs/bs/test-g/user00/no-dir: No such file or directory

3. The job script is described in the background job (with "&")
It will not be executed if it is entered and submitted as a background job (with "&") as shown below.
Example）#!/bash/sh#$ -cwd#$ -l node_f=1#$ -l h_rt=1:00:00#$ -N test
module load intel./a.out &4. When there is no file permission
Please set permissions appropriately.
Example) Grant read and execute permission to myself$ chmod u+rx <script_file>

5. Disk Quota
Please check the group disk quota.
It is about 2 million inodes per 1 TB.
Please refer to FAQ below.
Checking the usage of group disks with command

How to terminate the job submitted to batch job scheduler
See "How to terminate the programs executed accidentally" for the deletion of the processes running on the login nodes.
1. When the job-ID is known
Terminate the job with qdel command as follows.
Example : If job-ID is 10056$ qdel 10056

2. When the job-ID is unknown
Confirm the job-ID with qstat command, then incompleted jobs of the user are displayed.
Example: When GSIC user confirms the incompleted jobs, displayed as follows.
$ qstatjob-ID prior name user state submit/start atqueue jclass slots ja-task-ID------------------------------------------------------------------------------------------- 36/59 -2024-09-30

TSUBAME4.0 FAQ (English)

10053 0.555 ts1
10054 0.555 ts2
10055 0.555 ts310056 0.555 eq1

GSIC
GSIC
GSIC
GSIC

r
qw
hqw
Eqw

08/28/2024 22:53:44 all.q
08/28/2024 22:53:44 all.q
08/28/2024 22:53:45 all.q
08/28/2024 22:58:42 all.q

28
112
56
7

Tips
Delete jobs with Eqw by yourself. See here for the cause of it.
Refer to here if you want to change the status of a job to hqw.

State

Explanationr

Runningqw

Waiting in orderhqw

Waiting for other jobs to finish because of the dependency

Eqw

Error for some reasont

In transition from qw to r

I'd like to check the congestion status of compute node
Please check a stacked line chart in Job monitoring page. To check whether there are free compute nodes, see the green area on the chart.
Color

Status■Idle Nodes■

Running Nodes■

Reserved Waiting Nodes■

Reserved Running Nodes

How to use scrath area
TSUBAME 4.0 provides the following scratch area. For details, please refer to Storage use on Compute Nodes in the TSUBAME4.0 User's Guide
1. Local scratch area
The environment variable \$TMPDIR that is allocated only on compute node is the local scratch area.
\$TMPDIR is usually a unique directory for each job under /local.
For more information, please see here.

Info
You can not write directly under /local.

2. /tmp directory
There is a 2GB capacity limit for the /tmp directory.
There is a concern that problems such as hanging up of the execution program may occur when creating large scratch files.
Please consider using the scratch directories in 1.

- 37/59 -2024-09-30

TSUBAME4.0 FAQ (English)

SSH login to compute nodes
For SSH login to compute nodes only node_f is possible.
Please use node_f when executing applications that use SSH when doing MPI communication.
For details, please check the section SSH login of "TSUBAME 4.0 User's Gude".

Submission of dependent job
If you want to execute batch job A-2, as soon as the batch job named A-1 finishes, please use the -hold_jid option to submit the job as shown below.
$ qsub -N A-1 MM.sh$ qsub -N A-2 -hold_jid A-1 MD.sh

If you issue the qstat command afer submission, the status will be "hqw".

Want to execute multiple calculations at once in a batch job
If you want to multiple calculations in one job by executing batch, for example executing the four commands exec1, exec2, exec3, exec4 at onece,
write the batch script as follows.
#!/bin/sh#$ -cwd#$ -l node_f=1#$ -l h_rt=1:00:00
module load cuda
module load intel
exec1 &
exec2 &
exec3 &
exec4 &
wait

The above is only an example.
If you want to execute programs located in different directories at onece, you need to write the executable file from the path.
For example, if you want to directly execute a.out in folder1 of the home directory, you specify as below.
~/folder1/a.out &If you need to the directory of the executable file and execute it.
cd ~/folder1./a.out &Or,
cd ~/folder1 ; ./a.out &Warning
If the last line of the script file ends with "&", the job wil not run.
Do not forget to write the last wait command of the script.

Calculation starts on the login node before executing the qsub command
Typing the commands as described in the manual, the calculation starts on the login node before executing the qsub command.

- 38/59 -2024-09-30

TSUBAME4.0 FAQ (English)

GSICUSER@login1:~> #!/bin/bash
GSICUSER@login1:~> #$ -cwd
GSICUSER@login1:~> #$ -l node_f=2
GSICUSER@login1:~> #$ -l h_rt=0:30:0
GSICUSER@login1:~> module load matlab
GSICUSER@login1:~> matlab -nodisplay -r AlignMultipleSequencesExample

This is because you are directly executing commands on the shell that you need to write in the batch script.
Instead of executing them directly, create a batch script file and specify it with the qsub command.

If you are not familiar with terms such as batch script and shell, please see "1.Beginners of UNIX/LINUX" at "I'm a beginner, I don't know what to do."The error when executing the qrsh command
Explain the error when running qrsh.
1.Your "qrsh" request could not be scheduled, try again later.
The error above indicates that there are no available vacant resource for interactive job.
Please retry it after the resource become available.
See "I'd like to check the congestion status of compute node" for the status of the use of compute node.
2.Job is rejected. You do NOT have enough point to finish this job
This error indicates that there is no TSUBAME points required for assuring the node.
Please check the point balance.
Reference: FAQ "How long will it take for TSUBAME points to be returned?"
3.Unable to run job: unable to send message to qmaster using port 6444 on host "jobconX": got send error. Exiting.
This error occurs when the AGE server side is under heavy load.
Please wait a while and try again.

Check the detail of an error message printed the log file
The following message may be printed to the log file in some case.
/var/spool/age/hostname/job_scripts/JOB-ID: line XX: Process-ID Killed

Program_Name

In this case, type the qacct command to check the job in detail.
$ qacct -j JOB-ID

The following is an output example of the qacct command. (Excerpt)
Please check the details using the man command.

1. Example when the memory resource is exceeded$ qacct -j 4500000
qname
all.q
hostname
r1n1
group
GSIC
owner
GSICUSER00
project
NONE
department
defaultdepartment
jobname
SAMPLE.shjobnumber
4500000
taskidundefinedaccount
0 0 1 0 0 0 3600 0 0 0 0 0 0priority0
cwd/path-to-current
submit_host login1 or login2
submit_cmd
qsub -A GSICGROUP SAMPLE.shqsub_time%M/%D/%Y %H:%M:%S.%3N- 39/59 -2024-09-30

TSUBAME4.0 FAQ (English)

start_time
end_time
granted_pe
slots
faileddeleted_by
exit_statusmaxvmemmaxrssmaxpssarid
jc_name%M/%D/%Y %H:%M:%S.%3N%M/%D/%Y %H:%M:%S.%3N
node_o70
NONE
137
120.000G
0.000
0.000
undefined
NONE

you need pay atttention to exit_status, accont and maxvmem in the example.
exit_status provides the cause of the error by exit code. The exit_status 137 indicates 128 + 9, but since the status occurs in various problem, you
may not determine.
Then check the granted_pe and maxvmem.
The maximum memory usage, respectively.
It is estimated that 120 GB of memory was about to be actually used although up to 96 GB is available in node_o according to the User's Guide.
In TSUBAME, the job is killed automatically if the job used more memory size than assigned.
2. Example when the reserved time is exceeded$ qacct -j 50000000
qname
all.q
hostname
r1n1
group
GSIC
owner
GSICUSER00
project
NONE
department
defaultdepartment
jobname
SAMPLE.shjobnumber
50000000
taskidundefinedaccount
0 0 1 0 0 0 600 0 0 0 0 0 0priority0
cwd/path-to-current
submit_host login0 or login1
submit_cmd
qsub -A GSICGROUP SAMPLE.sh
granted_pe
node_fslots
7
failed0deleted_by
NONE
exit_status 137
wallclock614.711maxvmem
12.000G
maxrss0.000
maxpss0.000
arid
undefined
jc_nameNONE

you need pay atttention to exit_status, wallclock in the example.
exit_status provides the cause of the error by exit code. The exit_status 137 indicates 128 + 9, but since the status occurs in various problem, you
may not determine.
So I will focus on account, wallclock.
The seventh digit of the account space break indicates the time (sec) for securing resources.
In this example it is 600 seconds.

Wallclock shows the elapsed time, which is 614 seconds in this example.
Since the calculation did not end within the resource securing time, it can be inferred that the job was forcibly terminated.

How to transfer X with qrsh
This FAQ will explain how to transfer X with qrsh.
In this method, you can use GUI applications on other than node_f.
Please follow the procedure below.

- 40/59 -2024-09-30

TSUBAME4.0 FAQ (English)(Preliminary Work)
Enable X forwarding and ssh to the login node.
Reference: FAQ "X application (GUI) doesn't work" section 1 and 2.
1. After logging in to the login node, execute the following command.
In the example below, GSICUSER will use node_o from login1 for one hour.
GSICUSER@login1:~> qrsh -g GSICGROUP -l node_o=1,h_rt=1:00:00

2.Run the X application you want to use.
The following is an example with imagemagick.
GSICUSER@r1n3:~> module load imagemagick
GSICUSER@r1n3:~> display

Warning
Depending on the GUI application, there are applications that can not be activated or calculated by limiting memory or SSH.

- 41/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Info• For memory, please use the appropriate resource type.
• ANSYS Fluent can not be launched due to SSH restriction. To avoid this, use the -ncheck option (not supported by manufacturer).
• Schrodinger can be launched but can not compute by SSH restriction. You can use on f_node only.
• For OpenGL applications, export __GLX_VENDOR_LIBRARY_NAME=mesa is required"Warning: Permanently added ECDSA host key for IP address 'XXX.XXX.XXX.XXX' to the list of
known hosts." in the error log
It is the message that sytstem added the certificate of IP address XXX.XXX.XXX.XXX in the known_host file(SSH server certificate list) , when there isa node connected for the first time or when the certificate of the host which had previously connected has been changed. This is a normal operation,and it does not affect the calculation result and can be ignored.

Errors and remedies of qsub command execution
This section explains the error messages that occurs after executing qsub command and its remdy.
Unable to run job: Job is rejected because too few parameters are specified.
A required parameter is not specified. You need to specify resource type and number of resources, and execution time.
qsub: Unknown option
There is an roor in qsub option specification. Please refer to this.
Unable to run job: Job is rejected. core must be between 1 and 2.
3 or more resources per job can not be used for trial execution. Specify 1 or 2 as the number of resources.
Unable to run job: Job is rejected, h_rt can not be longer than 10 mins with this group.
For trial execution, you can not submit jobs whose execution time exceed 10 minutes. Please refer to this.
Unable to run job: Job is rejected. You do Not enough point to finish this job.
Points to secure the specified resources and time are insufficient.
Please check the point status from the TSUBAME portal page.
Unable to run job: failed receiving gdi request response for mid=1 (got syncron message receive timeout error).
orUnable to run job: got no response from JSV script"/apps/t4/rhel9/age/customize/jsv.pl".
Communication with the job scheduler will time out and the above error message may be displayed if the management node becomes in a state ofhigh load due to a large amount of job input in a short time. The state of high load is temporary. Please try again after waiting a while.

About specification of batch job scheduler
TSUBAME 4.0 uses the batch job scheduler

Resource TypeSee here for more information on resource types.

Job submission method
See here for more information on job submission method.
Also, please check the related FAQs below. * How to use scrath area * Submission of dependent job * How to transfer X with qrsh

- 42/59 -2024-09-30

TSUBAME4.0 FAQ (English)

About job limit
Please check "Various limit value list" about the current limit. If the submitted job exceeds the per-user limit, it will be kept in wait state "qw" eventhough there are enough idle nodes in TSUBAME4.0. Once the other jobs terminate and the job fits in the per-user limit, it becomes running state "r", ifthere's enough idel nodes.

About reservation
Reservation can be set in units of one hour and the node can be used 5 minutes before the reservation end time.
When submitting a job, it needs to be executed with the following command. AR ID can be confirmed on the portal.
$ qsub -g [TSUBAME Group] –ar [AR ID] <YOURSCRIPTNAME>

Since it is used up to 5 minutes before the reservation end time, you need to devise the -l option of the job script.
Example) Resource specification when reservation period is 2 days#$ -l h_rt=47:55:00"Reservation" does not apply to the above "Job limits", and has the "Reservation" restriction. Please check "Various limit value list" about the current
limit.

Coping with error
Please check the related FAQ below for coping with error.
• I get an error when submitting a job, but I do not know which option is bad• The job status is "Eqw" and it is not executed.
• The error when executing the qrsh command• Check the detail of an error message printed the log file

About troubleshooting at reservation execution
We summarize the troubleshooting when jobs can not be submitted during reservation execution.
The following command is an example where the GSIC group executes the AR number 20190108 which is used on 2days.
1.Forgot to add ARID
If the "ar" option is not attached, the job will be executed as a normal job.
(points will be consumed as usual).
Example of NG
When the following command is executed, it is executed as a normal job.
$ qsub -g GSIC hoge.sh

OK example
Be sure to use the -ar option when making reservation execution.
$ qsub -g GSIC -ar 20190108 hoge.sh

2.h_rt longer than reserved time
If the h_rt option time specification is longer than the reserved time, the job will not flow.
Also, because it is a specification that will be used 5 minutes before the reservation end time, please shorten the specified time by 5 minutes from thereservation time.
Example of NG
It is not executed because reservation time is full.

- 43/59 -2024-09-30

TSUBAME4.0 FAQ (English)$ grep h_rt hoge.sh#$ -l h_rt=48:00:00$ qsub -g GSIC -ar 20190108 hoge.sh

OK example (end time is -5 minutes)$ grep h_rt hoge.sh#$ -l h_rt=47:55:00$ qsub -g GSIC -ar 20190108 hoge.sh

When executing after the reservation start time, such as when the program terminates abnormally or when a job can not be submitted before thereservation start time, it is necessary to consider elapsed time.
For example, if you submit a job after 2 hours from the reservation start time, it will be the following script. (When one minute of internal processing
time from qsub command execution to allocation of compute nodes)$ grep h_rt hoge.sh#$ -l h_rt=45:54:00$ qsub -g GSIC -ar 20190108 hoge.sh

Related URL* TSUBAME4.0 User’s Guide "Reserve compute nodes" * TSUBAME portal User's Guide "Reserving compute nodes" * About specification of batch job
scheduler

About the difference between h_rt and actual execution time
The time specified by h_rt also includes the time for preparation processing to execute the job submitted by the user. Therefore, the time specified byh_rt does not become actual job execution time.
The points to be consumed are calculated based on the job execution time excluding the preparation processing time. And the preparation process
time is not constant because it varies depending on the status of the node where the job is to be executed.

A "command not found" error occured in qrsh/job script
Info
In TSUBAME 3.0, the following description was required in batch scripts, but is no longer necessary in TSUBAME 4.0.
. /etc/profile.d/modules.sh

If "command not found" error occurred when executing a command which is installed by external installer such as pip, please try the following on thelogin node:$ type <command><command> is hasehd (/path/to/<command>)

Then you can confirm the path, and add the following to the job script:
export PATH=$PATH:/path/toHere, /path/to is the directory where the command is located.

Related URLs
About common errors in Linux- 44/59 -2024-09-30

TSUBAME4.0 FAQ (English)

run some programs on different CPUs/GPUs in a job script
It is possible to run some programs on different CPUs/GPUs as follows.
In this example, a.out uses CPU0-6+GPU0, b.out uses CPU7-13+GPU1, c.out uses CPU14-20+GPU2, d.out uses CPU21-27+GPU3.
#!/bin/sh#$ -cwd#$ -V#$ -l node_f=1#$ -l h_rt=00:30:00
a[0]=./a.out
a[1]=./b.out
a[2]=./c.out
a[3]=./d.out
for i in $(seq 0 3)
doexport CUDA_VISIBLE_DEVICES=$i
numactl -C $((i*7))-$((i*7+6)) ${a[$i]} &
done
wait

- 45/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Applications
Commercial applications (Mathematica, Gaussian, etc.) do not work. Command not found.
TSUBAME4.0 requires a fee to use some commercial applications.
Detail : Fare overview
Initially, the target commercial application is not available, so it appears that the application is not installed.
$ module load gaussian$ g16 inputfile-bash: g16: command not foundTo use the target commercial application, go to the TSUBAME Portal and perform Application Activation.
After application activation, run it again.
$ module load gaussian$ g16 inputfile

Please note that app purchases must be made on an account-by-account basis.
Please also check I purchased a commercial application, but it is not available. .

When using Mathematica 14.1 or later
Mathematica has changed the names of GUI executable commands since version 14.1.
Version
14.0
14.1 and later

Execution command namemathematica
WolframNB

See Mathematica for details.

I purchased a commercial application, but it is not available.
Commercial applications must be purchased by account, not by group.
For example, if a group admin makes a purchase, only the group admin's account can use it.
Please complete the purchase process separately for all accounts you wish to use.

A commercial application I purchased suddenly becomes unusable.
Purchased commercial applications are available from the time of purchase until the end of the specified month.
The period of purchase may have exceeded. See Application Activation, and check the status.

How to use Python packages on PyPI (e.g. Theano)
You can install modules into your home directory (Example: Theano case)$ pip install --user theano- 46/59 -2024-09-30

TSUBAME4.0 FAQ (English)

X application (GUI) doesn't work
In this page, X applicatoin indicates the application that is installed in TSUBAME4.0 and can work on the X environment, that is GUI application.
Please check the troubleshooting below.

X server application is installed and active on the client PC
Windows
There are a lot of X server applications for Windows.
Please confirm that one of them is installed and on active.
Mac
Please confirm XQuartz is installed and configured.
https://support.apple.com/en-us/100724
Linux
Please confirm both of the X11 server application and its libraries are installed.

The X transfer option in the terminal is enabled.
A Terminal on Windows (Except for Cygwin)
The setting method differs depending on your terminal and X server application.
Please check the manual of each application.
Linux/Mac/Windows(Cygwin) Please confirm the ssh command contains the option -y and -c (these are the options for X transfer)$ ssh <account_name>@login.t4.gsic.titech.ac.jp -i <key> -YC

Example: in case of gsic_user as account_name and ~/.ssh/t4-key as key, then$ ssh gsic_user@login.t4.gsic.titech.ac.jp -i ~/.ssh/t4-key -YC

Please refer to the output of the following command for ssh option.
$ man ssh

Error reproduces in another terminal/Xserver
There are various free terminal softwares/X server applications for Windows.
Please check the same error occurs another terminal/X server.
It may be due to compatibility between terminal and X server.
It may be compatible with commercial application.
If it does not reproduce in other applications, there is a possibility that it is an application specific problem.
In that case we can not respond even if you contact us, please understand.
In addition, depending on the X application, command options may be required.
Please check the manual of X application you want to use.
Some GL applications that do not work with normal X forwarding/VNC connection may work with VirtualGL, so please give it a try if needed.
For the detail of VirtualGL, please refer to User's Guide.

Operation check
If it is in an interactive node, the standard terminal emulator of X Window System is started with the following command. Please confirm whether tostart.
$ xterm

If xterm works but the X application you want to use does not work, please try "3. Error reproduces in another terminal/Xserver"- 47/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Example of failure
xterm: Xt error: Can't open display:xterm: DISPLAY is not set

Please check 1 and 2 if the error occurs.

Application use
Do not execute programs that occupy the CPU at login nodes. Please use compute nodes for Full-scale use including visualization.
Please refer to the FAQ below for information on using the GUI application at the compute node. Reference: FAQ "How to transfer X with qrsh"
When using node_f, X transfer can be performed with the ssh -Y command.
Please inform us of the following when you inquire• Operating System you use (Example: Windows10,Debian12,macOS 14.4.1)• Terminal environment that the error occurs (Cygwin, PuTTY/VcXsrv, Rlogin/Xming)• Version
For Windows, the both versions of the terminal and X server application.
see the manuals for applications for checking versions.
Please inform the version of SSH in case of using Linux/Mac with the command below.
$ ssh -V

• Please send us the contents you tried so far, or if you get an error, please describe the error.

I would like to use an application not provided by TSUBAME 4.0
Installing applications not provided by TSUBAME 4.0
Please check if it applies to the following items.
If applicable, you can install it freely at your own risk.
Please check the installation manual and the license agreement of the application.
• Works with OS installed in TSUBAME4.0(Red Hat Enterprise Linux 9.3). Software requiring Windows or Mac OS won't work.
• Not requiring administrator privilege (root) to install it.
• Possible to install it to your own home directory or group disk. (It is not allowed to install it to any specified nodes' local disk.)• With a valid license.
• Not requiring the change to the settings for the kernel, libraries or the system itself.
• If only under these conditions, you can install it and use it on your own responsibility.
• No need for GSIC support.
0.Notes
As described above, GSIC will not help anything about the applications brought by users, as we do not know anything about it.
In case of problems, users themselves must distinguish whether it comes from the application itself or the general issue of TSUBAME, and ask
application vendors for application-specific problems.
The versions of libraries and drivers may be changed at the time of the regular maintenance of TSUBAME etc. In that case, you might need toreconfigure the application you had used. Please be aware of the risk of losing compatibility in the future.
1.Installation directory
You can install in the following two places.
Please choose suited according to your operation.
If you need to share within the TSUBAME group such as members of the laboratory, please use the high speed storage area.

- 48/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Even if you change the permissions by chmod or some commands in the home directory, you can not share that.

• Home directory• Group disk
Reference: TSUBAME 4.0 User's Guide "Strage system"
2.Installation method
Please install application according to the manual or README or community forum of the application to be installed.
Depending on the application, it is necessary to compile the library or module or something from the source file by yourself.
Below are some notes and a typical installation.
• Application management software such as zypper can not be used, you have to compile from the source file basically.
• If CUDA is used, it must be compiled on the compute node. (since the login node does not have a GPU).
• The TSUBAME4.0 OS is not a Debian distribution, so the apt command is not available.
• dnf commands cannot be used because they require root privileges. If you want to use them, please use Use Containers.
Example 1) executing configure script, generating makefile, then make, make test and make install:$ ./configure --prefix=$HOME/install$ make && make test
$ make install

Example 2) creating a directory for buid, cmake and make install:$ mkdir build && cd build$ cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/install$ make install

Example 3) example 3)installin with install script:$ ./install.sh

There is a problem with the operation of the distribution application
Since most of the troubles arising with regard to the distribution application are caused by the environment, we do not support them individually.
Please solve yourself as you accepted at the time of application.
Even if you contact us, we can not respond.

Pre/post processing of commercial applications
When using the commercial application on TSUBAME4.0, there are the following two cases.
Perform all processing of pre / solver / post in TSUBAME4.0 Perform pre / post processing on client and perform solver processing with
TSUBAME4.0
1.When performing processing of pre / solver / post in TSUBAME4.0
In TSUBAME 4.0, basically all the functions of pre / solver / post are introduced, and in case of execution at interactive node, it is possible to perform
all processing of "pre, solver, post".
How to run in interactive jobs and how to use each process depends on commercial application. Please check the manual and user's guide of eachapplication.
2.When performing pre / post processing on client and perform solver processing with TSUBAME4.0
Operation on TSUBAME may be unstable due to compatibility of X server. This problem can be avoided by performing pre-post processing on theclient, so we distribute software.
Software is provided for improved convenience. Please note that distribution may be canceled depending on the situation.

- 49/59 -2024-09-30

TSUBAME4.0 FAQ (English)

The following procedure is necessary to perform pre / post processing on client and perform solver processing with TSUBAME4.0
Step 1: Apply for software usage and obtain it
Step 2: Install the software on the client
Step 3: Perform pre processing with software on the client
Step 4: Transfer the data created in Step 3 to TSUBAMEStep 5: Create a batch script for submitting the job scheduler
Step 6: Execute the qsub command in TSUBAME and execute the batch script created in Step 5
Step 7: Transfer the result data of Step 6 to the client
Step 8: Perform post processing with software installed on the client

The range of support by T4 Helpdesk about the program error such as segmentation fault
GeneralPlease check the following related FAQ first
About common errors in Linux"Disk quta exceeded" error is output
Error handling for each commercial application

1.For commercial applications
Supported without ABAQUS/ABAQUS CAE. Please inform the following information through inquiry.
• Application nameEg)Abaqus/Explicit• Error messageEg)buffer overflow detected• JOB_IDEg)181938• Host name where the error occurredEg)r6n5• The situation in detailEg)The error occured when logged into r6n5 interactively with qrsh and executed the following command. Details are as follows:$ module load abaqus intel$ abq2017 interactive job=TEST input=Job1 cpus=6 scratch=$TMPDIR mp_mode=mpi#Error#
Run package*** buffer overflow detected ***:
/pathto/package terminated======= Backtrace: =========/lib64/libc.so.6(+0x721af)[0x2aaab0c001af]...
(The rest is ommited)

ABAQUS is an academic license, so there is no technical support.
It is necessary to register on the SIMULIA documentation site and resolve it yourself.
For information on the documentation site, please contact us from "Contact Us".
2.For the application compiled yourself
Not supported. Please resolve it yourself. See "I would like to use an application not provided by TSUBAME 4.0".
Error information is output when compiling with the traceback option.

- 50/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Error handling for each commercial application
GeneralThe following error occurs immediately after the program runs:
unable to connect to forwarded X server: Network error: Connection refused
Error: Can't open display: localhost:13.0.
Application name: Xt error: Can't open display:
Application name: DISPLAY is not set

The X server configration may be wrong. See the section 1 and 2 of the FAQ "X application doesn't work."
The GUI program suddenly terminates
Please check the keep alive setting in the terminal you use. See the FAQ "Session suddenly disconnected while working on TSUBAME4.0."
A job abruptly aborted
Although various reasons can be considered, please check the gollowing. * Check of batch error file (usually script_name.e.$JOBID ) * Check
program-specific log file * Check the free space of the directoryreference: FAQ
About common errors in Linux"Disk quta exceeded" error is output
The range of support by T4 Helpdesk about the program error such as segmentation fault

How to install numpy, mpi4py, chainer, tensorflow, cupy etc. using python/3.9.18
If you want to install numpy, mpi4py, chainer etc. using python/3.9.18, do as follows.
$ module load intel cuda openmpi$ python3 -m pip install --user python_modules

If you want to specify the version, do:$ python3 -m pip install --user python_modules==version

When installing a module that uses GPU like cupy etc., please keep computing nodes with qrsh command. before you install it.

License restriction on commercial application usage
Please refer to "License restriction on commercial application usage'

I want to install and use my library in R
In TSUBAME4.0, R-4.3.2 is available.
In addition to the basic package, the libraries available as default are as follows.
Rmpi, rpud

Please use library() command to check other available libraries.

If you wish to use a library other than the above, you will need your own installation operation.
Since the installation directory of R is impossible due to the permission relationship, you can install / manage your own library after specifying thelibrary path. The procedure is as follows.
Assuming that the library path is $HOME/Rlib, the library name is testlib, and the testlib.tar.gz is the source package, and operate as follows.
Load modules:$ module load cuda openmpi R- 51/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Create library installation directory (if nothing):$ mkdir ~/Rlib

Downlod package:$ wget https://cran.r-project.org/src/contrib/testlib.tar.gz

Install library:$ R CMD INSTALL -l $HOME/Rlib testlib.tar.gz

Your own installation library settings:$ export R_LIBS_USER=$HOME/Rlib

Use your library:$ R> library(testlib)

An error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called
Sometimes like the following error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called.
[r5n2:26205] [[60041,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_dyn.c at line 87

If you faced the error, please try ether the one of the following:
1. mpirun -np python3 -m mpi4py.futures ./test.py
1. use mpi4py with intel MPI

Port forward configuration for each terminal software
How to configure port forwarding for each terminal software as follows.
Please try the followings with allocating a compute node by qrsh/qsub.
As an example, suppose a compute node r7n7 is allocated, and connect local PC port 5901 to r7n7 port 5901.

1. MobaXterm
Tunneling -> New SSH Tunnel -> My computer with MobaXterm, input 5901 into "Forwarded port", in SSH server, input login.t4.gsic.titech.ac.jp into
"SSH server", input username into "defaultuser", input 22 into "SSH port", in Remote server, input r7n7 into "Remote server", input 5901 into "Remoteport" and save、choose key icon under Settings tab, and start the configured tunnel- 52/59 -2024-09-30

TSUBAME4.0 FAQ (English)

2. OpenSSH/WSL$

ssh -L 5901:r7n7:5901 -i <private key> -f -N <uesrname>@login.t4.gsic.titech.ac.jp

3. PuTTYPuTTY Configuration -> Connection -> SSH -> Tunnels, input 5901 into "Source Port", r7n7:5901 into "Destination" and click "Add" and Open

- 53/59 -2024-09-30

TSUBAME4.0 FAQ (English)

4. teraterm
Setup->SSH forwarding->Add->input 5901 into "Forward local port", input r7n7 into "to remote machine", and input 5901 into "port" then click "OK"file output stops by doing mpirun ... >& log.txt & with intel MPI
With intel MPI, output might stop by doing background execution like the following.

- 54/59 -2024-09-30

TSUBAME4.0 FAQ (English)

mpirun ... ./a.out >& log.txt &In this case, it can be avoided by the following:
mpirun ... ./a.out **< /dev/null >**& log.txt &I want to link Intel oneAPI Math Kernel Library, what are the link options?
If you want to link Intel oneAPI Math Kernel Library、please fill the appropriate contents into Intel oneAPI Math Kernel Library Link Line Advisor, andget the link opthion from "Use this link line”.

How to use VNC from MobaXterm
If you have a GUI application on TSUBAME and X forwarding fails to draw or the performance is insufficient, TurboVNC may improve the situation.
Since MobaXterm has a built-in VNC client function, it is relatively easy to use.
Please refer to User's Guide. for how to start a VNC server on a compute node and how to connect it from MobaXterm.

An error such that "X fatal error. ***ABAQUS/ABQcaeG rank 0 terminated by signal 6 " occurs at
modeling
An error such that "X fatal error. ***ABAQUS/ABQcaeG rank 0 terminated by signal 6 " occurs at modeling.
The error seems to occur when ABAQUS CAE is started, and modeling is performed in X transfer with MobaXterm, etc.
You can avoid this error by using VNC+VirtualGL, so please use it.
Please refer here for more information on how to use VNC from MobaXterm.
Please refer here for more information on how to use VNC via noVNC.
Please refer here for more information on how to use VirtualGL from VNC.

icc, icpc commands cannot be used with Intel OneAPIStarting with Intel oneAPI 2024, the icc and icpc commands are no longer available.
Please use the icx and icpx commands.
We also recommend using the ifx command instead of the ifort command.
https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-fortran-compiler-release-notes.html

C++17 related errors occur when using Intel oneAPI.
Since Intel oneAPI 2024, the default standard for C++ has changed from C++14 to C++17.
As a result, the following error may occur• error: ISO C++17 does not allow 'register' storage class specifier [-Wregister]• error: ISO C++17 does not allow dynamic exception specifications [-Wdynamic-exception-spec]
For more information, see here.
If this error occurs, do one of the following• Modify the source code to conform to the C++17 standard.
• Specify the option -std=c++14 during translation to compile with the C++14 standard.

- 55/59 -2024-09-30

TSUBAME4.0 FAQ (English)

I'm having trouble building and running VASP.
Even if you inquire about VASP, we may not be able to respond due to licensing restrictions.
VASP Forum has been established as official user support.
Please also use the VASP Forum.
And, we have also posted some FAQs for your reference.
I would like to know the procedure to build a VASP with TSUBAME4.0.
"UCX ERROR failed to insert region" error occurs when running VASP
C++17 related errors occur when using Intel oneAPI.

I would like to know the procedure to build a VASP with TSUBAME4.0.
This is the procedure when VASP6.4.2 was built on TSUBAME4.0.
1. module load
$ module load nvhpc openmpi intel

2. Copy arch/makefile.include.nvhpc_ompi_mkl_omp_acc file and modify the following$ cp arch/makefile.include.nvhpc_ompi_mkl_omp_acc makefile.include$ vi makefile.include$ diff arch/makefile.include.nvhpc_ompi_mkl_omp_acc makefile.include
20,21c20,21< FC= mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp< FCL= mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp -c++libs--> FC= mpif90 -acc -gpu=cc60,cc70,cc80,cc90,cuda12.3 -mp> FCL= mpif90 -acc -gpu=cc60,cc70,cc80,cc90,cuda12.3 -mp -c++libs
81c81< MKLROOT?= /path/to/your/mkl/installation--> MKLROOT?= /apps/t4/rhel9/isv/intel/mkl/2024.0/
87,88c87,88< SCALAPACK_ROOT ?= /path/to/your/scalapack/installation< LLIBS_MKL= -L$(SCALAPACK_ROOT)/lib -lscalapack -Mmkl--> #SCALAPACK_ROOT ?= /path/to/your/scalapack/installation> #LLIBS_MKL= -L$(SCALAPACK_ROOT)/lib -lscalapack -Mmkl

3. make$ make DEPS=1 -j12

2>&1 |tee make.log

Disable HCOLL when executing.
$ export OMPI_MCA_coll=^hcoll$ make test

"UCX ERROR failed to insert region" error occurs when running VASP
Add runtime option -mca coll_hcoll_enable 0.
Example：mpirun -mca coll_hcoll_enable 0 vasp_std >vasp.log

Unable to module load python
Python 3 is available by default in TSUBAME4.0.
No module load required.

- 56/59 -2024-09-30

TSUBAME4.0 FAQ (English)

When running OpenMPI/Intel MPI, an hcoll-related error or segmentation fault occurs.
This may be improved by specifying the following environment variables.
OpenMPI：export OMPI_MCA_coll=^hcoll
Intel MPI：export I_MPI_COLL_EXTERNAL=0

Binaries compiled with the -lblas option in Intel Compiler terminate abnormally at runtime
Intel Compiler provides a numerical calculation library called Intel MKL.
When using BLAS with Intel Compiler, please change the option to link the Intel MKL library instead of -lblas. For details, please refer to How to linkIntel MKL.
Note that a wide variety of numerical calculation libraries exist, including dedicated libraries such as Intel MKL.Some of them have dependencies orconflicts among libraries.
If an error occurs in a numerical calculation library, please consider checking for conflicts or using another library.

GPU not used when running TensorFlow
When running TensorFlow on a GPU, there are requirements for the combination of TensorFlow, python, cudnn, and cuda.
Please refer to Tensorflow for details.

- 57/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Migration from TSUBAME3
For information on migrating from TSUBAME3, please see Migration from TSUBAME3.0 to TSUBAME4.0.

Comparison of old and new TSUBAME
For a comparison of TSUBAME3 and TSUBAME4.0, please see Comparison of old and new TSUBAME.

- 58/59 -2024-09-30

TSUBAME4.0 FAQ (English)

Tips
TSUBAME4.0 some workarounds of known issues- 59/59 -2024-09-30