175th Parallel Programming Workshop with Trial AccountSupercomputing for Beginners
22nd Apr, 2022
This is for participants who use
OpenSSH or Cygwin on Windows, Mac OS, or Linux.

20th Apr. 2022 v1.3

2022/4/22

Supercomputing for Beginners

Agenda13:00 – 14:00
Introduction, How to use Zoom for this tutorial
What are supercomputers ? (lecture)
Login tutorial (lecture + exercise)
14:15 – 15:45
Write, compile, and run programs (lecture + exercise)Compile and run parallel programs (lecture + exercise)
16:00 – 17:00
How to run machine learning on supercomputers(lecture + exercise)
For advanced usage （lecture）

2

2022/4/22

Supercomputing for Beginners

3

Important notices
n Your account will be valid for one month (from today).
l Comply with “Terms of use” (https://www.cc.u-tokyo.ac.jp/en/guide/application/rules.php)üll

You may use supercomptuers only for contributing to academic research, education, and society.

Your account and your files on the supercomputer will be deleted afterwards.
The account will be disabled if you do not keep attending to this lecture till its end.

n For participants from industries: please finish one of the series of theseworkshops@ITC, if you want to apply to “trial usage”.
n If you have any questions, do not ask ITC official e-mail. Please write insteadto the lecturer in charge:
shiba [at] cc.u-tokyo.ac.jp

2022/4/22

Supercomputing for Beginners

Supercomputer workshop (tutorials) at ITC, U-Tokyo
Roadmap

Introductory

Intermediate/Advanced

ScienceModeling

Algorithm

Software

OpenACC+MPI
Multi-GPU

Using Libraries
OpenFOAM
Advanced

Parallel FEM

Hardware

OFP Practice
MPI Advanced

Hybrid programming

Hyperworks
Parallel FEM
BasicsOpenFOAM
Introduction

Streamlining Scientific
Computation
Introduction

Learning steps
Relevance

OpenMP

Introduction to
GPU programming

MPI Introduction
Supercomputing
for Beginners

Start

25

2022/4/22

Supercomputing for Beginners

How to use supercomputer
login node = entrance
where you can run
commands[tut138@obcx02~]$ ls -l
drwxr-x--- 2 shiba group 10 1
Apr 13:00 test.out[tut138@obcx02~]$ ./test.out
Hello world[tut138@obcx02~]$ qsub a.sh

2. Jobsubmission

1. LoginYour terminal

SSH

You need to prepare
your SSH key toestablish connection

3. Program execution

Login nodes

Compute nodes= “main body”

26

2022/4/22

Supercomputing for Beginners

Secure Shell protocoln Shell = command base software,connecting the user and the OS
n SSH = encrypted connection thatenables remote connection

encryption

OBCX

Via the SSH connection, you cann Copy files
n Forward the GUIn Tunnelingn Mount to your directory

Shell command example after SSH login

272022/4/22

Supercomputing for Beginners

SSH key authentication
enables secure connection to supercomputers
Use “SSH key pairs” instead of (plain text) passwords.
Initial settings (for the first login)n Generate key pairs
n Register the public keys on the login node
①request login③reply encrypted message④decrypt thesecret key⑤send the hashed value

OBCX②encryption using
the public key fromthe client⑥confirm⑦allow login

Begin the session28

2022/4/22

Supercomputing for Beginners

Cautions on SSH key authentication
Your secret key should be kept strictly confidential
ü to prevent illegal login of other people.
l Do not copy your secret key to any other places= The key pairs should not be recycled (on other machines)
n Set PASSPHRASE when generating your SSH key!
n The passphrase of your key should be different from otherpasswords, including those of OBCX user portal and user
password on OBCX.

29

2022/4/22

Supercomputing for Beginners

SSH key authentication
Generate your key pair on your local computer (1/3)
Open Cygwin or Terminal and begin the following operation$ ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): Return
Enter passphrase (empty for no passphrase): Set your passphrase
ReturnEnter same passphrase again: The same passphrase
ReturnYour identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:vt88O+PTcscHkOyabvxGjeRsMWLAWds+ENsDcReNwKo tut138@ITCUT-VAIO
The key's randomart image is:+---[RSA 2048]----+|. o=oo.o+ |How to generate key|+ O... .||.+o+.
|• ssh-keygen -t rsa <Return>
|
+oB.
|
|
So *o*|• <Return>
|
.E
B.o |
|
.. = . o |• Passphrase as you like<Return>|.=oB o +||.+o+*O ..|• The same passphrase<Return>+----[SHA256]-----+

30

2022/4/22

Supercomputing for Beginners

31

SSH key authentication
Confirm that there are both secret and publickeys on your local computer（2/3）
If you do not understand the meaning, type letters and confirm a similar output$ cd .ssh$ ls

id_rsaid_rsa.pub⇒⇒

Private Key
Public Key$ cat id_rsa.pub

ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQDa6InmOYYaCrWjQDukjiNEfdW8veUwJyZtEI3oDu0A28eey6p0wbtI7JB
09xnI17O7HG4yYvOM81+/nlAHy5tAfJly0dsPzjTgdTBLdgi3cSf5pWEY6U96yaEr0Ei8Wge1HkXrhcwUjGDVTz
vT0Refe6zLdRziL/KNmmesSQfR5lsZ/ihsjMgFxGaKsHHq/IErCtHIIIf9V/Ds2yj6vkAaWH6asBn+ZsRiRFvwH
PhkYAnp/j3LY6b8Qfqg0p4WZRenh/HgySWTYIGi8x67VzMaUlm9qlK0QFMCaK2rivX1fmbwyWJ/vrWDqiek6YXo
xLDu+GPeQ4CPvxJcZnqF9gf3 tut138@ITCUT-VAIO

2022/4/22

Supercomputing for Beginners

32

SSH key authentication
Copy public key （3/3）
Cut and paste your “id_rsa.pub” file
$ cd .ssh$ ls

id_rsaid_rsa.pub

Procedure• cat id_rsa.pub <Return>• Drag andDrop from “ssh-rsa”⇒秘密鍵（PrivateKey） to the filnal⇒公開鍵（Public Key）in this case)part(“tut138@ITCUT-VAIO”$ cat id_rsa.pub

ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQDa6InmOYYaCrWjQDukjiNEfdW8veUwJyZtEI3oDu0A28eey6p0wbtI7JB
09xnI17O7HG4yYvOM81+/nlAHy5tAfJly0dsPzjTgdTBLdgi3cSf5pWEY6U96yaEr0Ei8Wge1HkXrhcwUjGDVTz
vT0Refe6zLdRziL/KNmmesSQfR5lsZ/ihsjMgFxGaKsHHq/IErCtHIIIf9V/Ds2yj6vkAaWH6asBn+ZsRiRFvwH
PhkYAnp/j3LY6b8Qfqg0p4WZRenh/HgySWTYIGi8x67VzMaUlm9qlK0QFMCaK2rivX1fmbwyWJ/vrWDqiek6YXo
xLDu+GPeQ4CPvxJcZnqF9gf3 tut138@ITCUT-VAIO

2022/4/22

Supercomputing for Beginners

SSH key authentication
id_rsa• Private Key

ー keep it on your PC• Keep it confidential, do not move it from where it is generated, do notsend it to others

id_rsa.pub• Public Key ー

put on supercomputer• You may copy and send it to others by e-mail.
• If you want to log into a supercomputer from multiple local

computers, then generate a pair of public and private keys oneach local computer. You can register multiple public keys.

332022/4/22

Supercomputing for Beginners(1) Login to OBCX user portal
https://obcx-www.cc.u-tokyo.ac.jp/cgi-bin/hpcportal.en/index.cgi

USER ID(tUVXYZ）

Initial password
sent from ITC, UT

34

2022/4/22

35

Supercomputing for Beginners(2) Change your portal password

Initial password sent from
ITC, University of Tokyo
New password (twice)2022/4/22

Supercomputing for Beginners(3) public key registration（id_rsa.pub）

1. Choose “SSH Public key”
2. Paste the public key（ id_rsa.pub )3. Click “register”
Ask the staffs if you have troubles

362022/4/22

Supercomputing for Beginners

More than one public keys can be registered on OBCX

37

2022/4/22

Supercomputing for Beginners

Get Manual PDF files on OBCX portal （1/2）

382022/4/22

Supercomputing for Beginners

Get Manual PDF files on OBCX portal （2/2）

392022/4/22

40

Supercomputing for Beginners

Login to OBCX supercomputer$ ssh tUVXYZ@obcx.cc.u-tokyo.ac.jp
Enter passphrase for key '/home/tut138/.ssh/id_rsa:

Your Passphrase

1. ssh tUVXYZ@obcx.cc.u-tokyo.ac.jp <Return>2. Passphrase <Return>

Return2022/4/22

Supercomputing for Beginners

Login to OBCXLast login: Sun Apr 12 15:05:47 2020 from obcx01.cc.u-tokyo.ac.jp----------------------------------------------------------------------------Oakbridge-CX Information
Date: Apr. 03, 2020---------------------------------------------------------------------------Welcome to Oakbridge-CX system* Operation Schedule
04/24(Fri) 09:00 – 04/24(Fri) 20:00
04/24(Fri) 20:00 -You will findmaintenanceschedule (+etc)
System Maintenance
Normal Operation If you successfully
logged in

For more information about this service, see
https://www.cc.u-tokyo.ac.jp/supercomputer/schedule.php* How to use
Users Guide can be found at the User Portal (https://obcx-www.cc.utokyo.ac.jp/).
If you have any questions, please refer to the following URL and contact us:
https://www.cc.u-tokyo.ac.jp/supports/contact/* Updated OBCX Users Guide
10/01(Tue): v1.0
Set your email address on the User Portal [https://obcx-www.cc.u-tokyo.ac.jp][tUVXYZ@obcx01 ~]$41

2022/4/22

Supercomputing for Beginners

After login
You will see the linux shell[tUVXYZ@obcx01 ~]$ pwd→ type commands

Return/home/tUVXYZ

pwd ー shows the present directory442022/4/22

Supercomputing for Beginners

Break

45

2022/4/22

Supercomputing for Beginners

How to use supercomputer
login node = entrance
where you can run
commands[tut138@obcx02~]$ ls -l
drwxr-x--- 2 shiba group 10 1
Apr 13:00 test.out[tut138@obcx02~]$ ./test.out
Hello world[tut138@obcx02~]$ qsub a.sh

2. Jobsubmission

1. LoginYour terminal

SSH

You need to prepare
your SSH key toestablish connection

3. Program execution

Login nodes

Compute nodes= “main body”

46

2022/4/22

Supercomputing for Beginners

How to use supercomputers ?
SSH login

Putty、Cygwin、SSH、key authentication

Write codes

How to use editors (emacs, vi, nano), Linux basics

CompileEnvironment module, compiler options

Job submission

Writing job scripts, how to manage jobs

Confirm results

47

2022/4/22

51

Supercomputing for Beginners

Linux Directories on OBCX Linux/home/home/tUVXYZ

You are here
just after login/home/kUVXYZ/home/mUVXYZ/
Root
directory/work/work/gt00/work/gt00/tUVXYZ/work/po****/work/jh******

LUSTRE file system
/usr
/etc

n Fast write/read, large capacityn All the jobs are run from herel You cannot run jobs from /homen Separated from /home , so that you can login evenwhen /work is in trouble

2022/4/22

Supercomputing for Beginners

After you logged in…
$ pwd

Return/home/tUVXYZ$ cd /work/gt00/tUVXYZ$ pwd Return/work/gt00/tUVXYZ
$ cd
$ pwd

ReturnReturn/home/t00XYZ

Return

1. You are at “/home/tUVXYZ”
2. Move to working directory:“ /work/gt00/tUVXYZ”
3. You can get back by “cd”

55

2022/4/22

Supercomputing for Beginners

Exercise: computing Fibonacci sequence
defined as a recurrence relation

F0 = 1F1 = 1
Fn = Fn 1 + Fn 2
<latexit sha1_base64="aGpfKio9rBd4elCeerhjKtm4B6M=">AAADTHichVJLa9RQGD1Ntdbx0aluBDfB6ZSKtNzMxiIIVaG47MNpC80wJOntGJrcxCQzWsP8Af+AC1cKRcSf4caN7lT6E8RlC4IP8OROxEcZvSHJ+c73ne+e+3DjwE8zIfZHjNFjx8dOjJ+snDp95uxEdfLcWhp1E082vSiIkg3XSWXgK9nM/CyQG3EindAN5Lq7c6vIr/dkkvqRupPtxrIVOh3lb/uek5FqV29Mm4ttYV43LdO2K0Vg/R4oBovtXM1affPKADX69r2us2XOKLsjzcbldrUm5oQe5lFglaCGcixF1S+wsYUIHroIIaGQEQdwkPLZhAWBmFwLObmEyNd5iT4q1HZZJVnhkN3ht8Nos2QV46JnqtUeZwn4JlSaqIt34oU4EK/FS/FRfBvaK9c9Ci+7/LsDrYzbE48urH7+ryrkP8PdX6p/es6wjXnt1af3WDPFKryBvvfw8cHqtZV6Pi2eiU/0/1Tsi1dcgeodenvLcuUJu9eHOnLpZvie5ZzrAeeP9AmkZaVi5r7euVCvRfGscvIueswUvooTammu2J2IPXLsaRSw9uf6TUyhxsopKtiZ18T6+1IcBWuNOYt4uVFbuFlemHFcxCXMsNNVLOA2ltCki+d4g/f4YLw1Do2vxvdBqTFSas7jjzE69gNCK76f</latexit>
sha1_base64="j41D7YQITxu9gm7MzpYA7IjDnag=">AAAC+3ichVI9T9tQFD0YaCHQJmxILBEhqBO66QJiqtSFEQgBJBoh2zyohb9kO2mpxR/IysDA1EoZEEu3irlL/wADP6HqCFIlxMDxixGiKPRZ9rs+9557z33vWqHrxInI5YAxODT84uXIaGFsvPDqdbE0vh4HrchWDTtwg2jTMmPlOr5qJE7iqs0wUqZnuWrD2n+f+TfaKoqdwF9LDkLV9Mw939l1bDMhtLxdqsic6FV+atRyo4J8BaUbfMAOAthowYOCj4S2CxMxny3UIAiJNZESi2g52q9wiAK5LUYpRphE9/nd499Wjvr8z3LGmm2ziss3IrOMqlzIqVzJLzmT33LbN1eqc2RaDrhbPa4Kt4udyfrf/7I87gk+PrCe1ZxgFwtaq0PtoUayLuwev/3l+Kq+uFpNZ+Wb/KH+r3IpP9mB3762uytq9YTZq30VWVTT/8xS1vrM+oG+gTiP9On5pE/O0734vKuUuIU2PZmu7IaaGstOJ2COFF1tuYy977+MGVQYOUMGM3NKav/OxFNj/e1cjfaKYARTmMYbJpjHOyxhGQ0W30EHR0bH+G786E2TMZCP1QQeLeP8DrWFqbU=</latexit>
sha1_base64="/Be7ntYvb07kiOZmbXjDM/1v+Og=">AAADQXichVJLa9RQGD2NrzpWO7oS3ASnUypiuZlNiyAoQnHZh9MWmiEk6e00NLmJSWa0hvkD/gEXrhSKiD/DjRvdqfQniMsWBB/gyZ2IjzJ6Q5Lzne873+Pe6yVhkOVC7I8Zx46fOHlq/HTtzMTZc5P18xOrWdxLfdn24zBO1z03k2GgZDsP8lCuJ6l0Iy+Ua97O7dK/1pdpFsTqbr6byE7kdlWwFfhuTsqp35o2Fxxh3jAt07ZrpWH9bigaC06hrlkD8+oQtQb2vZ67ac4ouyvN1hWn3hCzQi/zKLAq0EC1FuP6F9jYRAwfPUSQUMiJQ7jI+GzAgkBCroOCXEoUaL/EADVqe4ySjHDJ7vDbpbVRsYp2mTPTap9VQr4plSaa4p14IQ7Ea/FSfBTfRuYqdI6yl13+vaFWJs7ko4srn/+rivjPsf1L9c+ec2xhXvcasPdEM+UU/lDff/j4YOX6crOYFs/EJ/b/VOyLV5xA9Q/9vSW5/ITZmyM78tjN6D0rWOsB68f6BLIqUtFzX+9cpGdRPKuCvIc+PWVf5Ql1NFfuTswcBfY0Chn7c34TU2gwcooKZuY1sf6+FEfBamvWIl4SGMclXMYME8zhJu5gEW0Wf443eI8Pxlvj0Pg6vFDGWHWzLuCPZXz/AQk4veg=</latexit>
sha1_base64="GfKpEPNEnlEk/e0vHUBFZo7JDCk=">AAADTHichVJLa9RQFD5Ntdbx0WndCG4uTqdUpOVkNpaCUBWKyz6cttAMIUlvx9DkJiaZ0RrmD/gHXLhSKCL+DDdudKfSnyBdtlDwAX65E/FRRm9I8p3vnO/c7z7cOPDTjHl/yBg+dXrkzOjZyrnzFy6OVccn1tKok3iy6UVBlGy4TioDX8lm5meB3IgT6YRuINfdnTtFfr0rk9SP1L1sN5at0Gkrf9v3nAyUXb01JRZtFjeFKSyrUgTm74FCsGjnasbsiet91OhZDzrOlphWVluKxjW7WuNZ1kOcBGYJalSOpaj6hSzaoog86lBIkhRlwAE5lOLZJJOYYnAtysElQL7OS+pRBdoOqiQqHLA7+LYRbZasQlz0TLXawywB3gRKQXX+wK/4kN/ya/7M3wb2ynWPwssu/m5fK2N77Mnl1eP/qkL8M7r/S/VPzxlt05z26sN7rJliFV5f33389HB1fqWeT/ELPoD/57zPb7AC1T3y9pblyjN0rw905MLN4D3LMdcjzB/pE0jLSoXMQ71zoV6Lwlnl4F3qIlP4Kk6opblidyL0yGlPowC1P9cvaJJqqJyEAp1xTcy/L8VJsNaYNYGXubZwu7wwo3SFrtI0Ot2gBbpLS9SEi5f0jj7SJ+O9cWR8Nb73S42hUnOJ/hjDIz8AQYu+nQ==</latexit>
sha1_base64="8EKbeTwsttEMzvq93/DNDJ5xU4g=">AAADTHichVJLa9RQGD1Ntdbx0VE3gpvgdEpFLDeDoAhCVSgu+5q20AxDkt6OoclNTDKjNcwf8A+4cKVQRPwZbtzoTqU/QbpsQfABntyJ+CijNyQ53/m+891zH24c+GkmxO6IMXrk6Nix8eOVEydPnZ6onjm7kkbdxJNNLwqiZM11Uhn4SjYzPwvkWpxIJ3QDuepu3Snyqz2ZpH6klrPtWLZCp6P8Td9zMlLt6q0pc64tzJumZdp2pQis3wPFYK6dqytW37w8QI2+fb/rbJjTyu5Is3GpXa2JGaGHeRhYJaihHPNR9QtsbCCChy5CSChkxAEcpHzWYUEgJtdCTi4h8nVeoo8KtV1WSVY4ZLf47TBaL1nFuOiZarXHWQK+CZUm6uK9eCn2xRvxSnwS34b2ynWPwss2/+5AK+P2xOPzS5//qwr5z3Dvl+qfnjNs4rr26tN7rJliFd5A33v0ZH/pxmI9nxLPxR79PxO74jVXoHoH3s6CXHzK7vWhjly6Gb5nOed6yPkjfQJpWamYeaB3LtRrUTyrnLyLHjOFr+KEWpordidijxw7GgWs/bl+E5OosXKSCnbmNbH+vhSHwUpjxiJeuFqbvV1emHFcwEVMs9M1zOIu5tGkixd4iw/4aLwzDoyvxvdBqTFSas7hjzE69gNCy76h</latexit>(n2)

1, 1, 2, 3, 5, 8, 13, 21 ….
Iterative computation is a starting
point for using supercomputers
From Wikipedia
https://ja.wikipedia.org/wiki/フィボナッチ数56

2022/4/22

Supercomputing for Beginners

57

Fibonacci sequence, up to 92nd
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
10946
17711
28657
46368

75025
121393
1964183178115142298320401346269217830935245785702887922746514930352
24157817
39088169
63245986
102334155
165580141
267914296
433494437
701408733
1134903170
1836311903
2971215073
4807526976

7778742049
12586269025
20365011074
32951280099
53316291173
86267571272
139583862445
225851433717
365435296162
591286729879
956722026041
1548008755920
2504730781961
4052739537881
6557470319842
10610209857723
17167680177565
27777890035288
44945570212853
72723460248141
117669030460994
190392490709135
308061521170129
498454011879264

806515533049393
1304969544928657
2111485077978050
3416454622906707
5527939700884757
8944394323791464
14472334024676221
23416728348467685
37889062373143906
61305790721611591
99194853094755497
160500643816367088
259695496911122585
420196140727489673
679891637638612258
1100087778366101931
1779979416004714189
2880067194370816120
4660046610375530309
7540113804746346429

Task:
Compute this sequence, and find
“747031” in the result

2022/4/22

58

Supercomputing for Beginners

Compose a program using text editors
First, make a directory where you work on.
Return[tUVXYZ@obcx01 ~]$ cd[tUVXYZ@obcx01 ~]$ mkdir fibonacci Return[tUVXYZ@obcx01 ~]$ cd fibonacci Return

Please use text editors such as Emacs, vim, and nano.
例1： Compose a Fortran code using Emacs[tUVXYZ@obcx01 fibonacci]$ emacs fibonacci.f90

Return例2： Compose a Python script using vim[tUVXYZ@obcx01 fibonacci]$ vim fibonacci.py

Return例3： Compose a C code using nano[tUVXYZ@obcx01 fibonacci]$ nano fibonacci.c

Return2022/4/22

62

Supercomputing for Beginners

Exercise: computing Fibonacci sequence
C
Edit the code#include <stdio.h>
int main(void) {int i;
long a, b, tmp;
a=1;
b=1;
printf("%ld\n", a);for (i=2; i<=92; i++) {tmp = b;b = a + b;
a = tmp;
printf("%ld\n", a);}}[tUVXYZ@obcx01 fibonacci]$ emacs fibonacci.c

Return2022/4/22

63

Supercomputing for Beginners

Exercise: computing Fibonacci sequence
FortranEdit a code[tUVXYZ@obcx01 fibonacci]$ emacs fibonacci.f90

program main
implicit none
integer(kind=4) i
integer(kind=8) a,b,tmp
a=1
b=1
do i=2, 92
tmp = bb= a + b
a = tmp
print *, aend doend program main

Return2022/4/22

64

Supercomputing for Beginners

Exercise: computing Fibonacci sequence
Python
Edit a code[tUVXYZ@obcx01 fibonacci]$ emacs fibonacci.py

a, b = 1, 1
print(a)for i in range(1,92):a, b = b, a+b
print(a)In python…
ー Data type is auto detected (e.g. 64 bit integrer)
ー Multiple variables can be updated at once.
ー Compilation not necessary.

Return2022/4/22

Supercomputing for Beginners

How to use supercomputers ?
SSH login

Putty、Cygwin、SSH、key authentication

Write codes

How to use editors (emacs, vi, nano), Linux basics

CompileEnvironment module, compiler options

Job submission

Writing job scripts, how to manage jobs

Confirm results652022/4/22

Supercomputing for Beginners

Environment modules (1)We can choose compilers (etc)
gnu compiler, intel compiler, python interpreters, …

We can choose libraries (for accelerating computations)
Fast Fourier Transform, linear algebra/simultaneous linear equations, and etc…

Show modules which is in use
Return[tUVXYZ@obcx01 ~]$ module list
Currently Loaded Modulefiles:
1) impi/2019.9.304
2) intel/2020.4.304

Intel MPI libraryIntel compiler

Clear up current settings (we want to switch to others)[tUVXYZ@obcx01 ~]$ module purge[tUVXYZ@obcx01 ~]$ module list
No Modulefiles Currently Loaded.

ReturnReturn66

2022/4/22

Supercomputing for Beginners

Displaying available environment modules[tUVXYZ@obcx01 ~]$ module avail------ /home/opt/local/modulefiles/L/mpi/intel/2019.5.281/impi/2019.5.281 -----alps/2.3.0(default)
phdf5/1.10.5(default)
feram/0.26.04(default)
pnetcdf/1.11.2(default)
frontflow_blue/8.1(default)
ppohBEM/0.5.0(default)
frontistr/4.5(default)
ppohDEM_util/1.0.0(default)
modylas/1.0.4(default)
ppohFDM/0.3.1(default)
mpi-fftw/3.3.8(default)
ppohFEM/1.0.1(default)
netcdf-fortran-parallel/4.4.5(default) ppohFVM/0.3.0(default)
netcdf-parallel/4.7.0(default)
pt-scotch/6.0.6(default)
openmx/3.8(default)
revocap_coupler/2.1(default)parmetis/4.0.3(default)
superlu_dist/6.1.1(default)
petsc/3.11.2(default)
xtapp/rc-150401(default)
phase/2019.01(default)----------- /home/opt/local/modulefiles/L/compiler/intel/2019.5.281 -----------R/3.6.0(default)
metis/5.1.0(default)akaikkr/cpa2002v010(default)
mt-metis/0.6.0(default)
bioconductor/3.10(default)
netcdf/4.7.0(default)
blast/2.9.0(default)
netcdf-cxx/4.3.0(default)
bwa/0.7.17(default)
netcdf-fortran/4.4.5(default)
fftw/3.3.8(default)
paraview/5.6.1(default)
gsl/2.5(default)
povray/3.7.0.8(default)
hdf5/1.10.5(default)
ppohAT/1.0.0(default)
hdf5/1.8.21
revocap_refiner/1.1.04(default)
impi/2019.5.281(default)
samtools/1.9(default)
intelpython/2.7
scotch/6.0.7(default)
intelpython/3.6(default)
superlu/5.2.1(default)
mesa/19.0.6(default)
superlu_mt/3.1(default)
metis/4.0.3
xabclib/1.03(default)---------------------- /home/opt/local/modulefiles/L/core ---------------------acusolve/2019.1.0(default)
intel/2018.3.222
advisor/2019.3.0.591490
intel/2019.3.199
advisor/2019.4.0.597843
intel/2019.4.243
advisor/2019.5.0.602216(default)
intel/2019.5.281(default)
anaconda/2-2019.03
intel/2020.1.217
anaconda/3-2019.03(default)
itac/2019.4.036
bioperl/1.007002(default)
itac/2019.5.041(default)
bioruby/1.5.2(default)
julia/1.4.0(default)
cmake/3.0.2
llvm/7.1.0(default)
cmake/3.14.5(default)
massivethreads/0.97

Return67

2022/4/22

Supercomputing for Beginners

68

To see at once complex module dependencies on OBCX[tUVXYZ@obcx01 ~]$ show_module

ReturnApplicationName
ModuleName
Node
BaseCompiler/MPI-------------------------------------------------------------------------------------------------------------ALPS
alps/2.3.0
login,computeintel/2020.4.304/impi/2019.9.304
Acusolve
acusolve/2019.1.0
login,compute
Advisoradvisor/2019.4.0.597843
login,compute
Advisoradvisor/2019.5.0.602216
login,compute
Advisoradvisor/2020.3.0.607294
login,compute
Advisoradvisor/2019.3.0.591490
login,compute
AkaiKKRakaikkr/cpa2002v010
login,computeintel/2020.4.304
Anaconda
anaconda/2-2019.03
login,compute
Anaconda
anaconda/3-2019.03
login,compute
Arm DDT
ddt/19.1
computeArm DDT
ddt/20.2.1
computeArm DDT
ddt/20.0.2
computeBLAST
blast/2.11.0
login,computeintel/2020.4.304
BWA
bwa/0.7.17
login,computeintel/2020.4.304
BioPerl
bioperl/1.007002
login,compute
BioRubybioruby/1.5.2
login,compute
Bioconductor
bioconductor/3.10
login,computeintel/2020.4.304
CMake
cmake/3.0.2
login,compute
CMake
cmake/3.14.5
login,computeCP2K
cp2k/v8.1
login,computeintel/2020.4.304/impi/2019.9.304
Devtoolset
devtoolset/7
login,compute
FFTW
fftw/3.3.8
login,computeintel/2020.4.304
FFTW
mpi-fftw/3.3.8
login,computeintel/2020.4.304/impi/2019.9.304
FeRAM
feram/0.26.04
login,computeintel/2020.4.304/impi/2019.9.304
GATK
gatk/4.1.2.0
login,computeGCC
gcc/4.8.5
login,compute
GCC
gcc/7.5.0
login,compute
GNU Octave
octave/6.1.0
login,compute
GNU Scientific Library
gsl/2.6
login,computeintel/2020.4.304
GROMACS
gromacs/2020.5
login,computeintel/2020.4.304/impi/2019.9.304
Gogo/1.12.6
login,compute
HDF5
hdf5/1.12.0
login,computeintel/2020.4.304
HDF5
hdf5/1.8.22
login,computeintel/2020.4.304
HDF5
phdf5/1.12.0
login,computeintel/2020.4.304/impi/2019.9.304
HDF5
phdf5/1.8.22
login,computeintel/2020.4.304/impi/2019.9.304
HyperWorks
hyperworks/2019.1.0
login,compute
IASP91
iasp91/default
computeInspector
inspector/2019.5.0.602103
login,compute
Inspector
inspector/2019.3.0.591484
login,compute-

BaseCompiler/MPI needs
to be loaded before
loading each module2022/4/22

Supercomputing for Beginners

Compilation & setting up your job
By environment modules$ module [option] args
optionavail

show available environment modules

list

show loaded environment modules

load

load specified environment module(s)unloadUnload specified environment module(s)switchload and unload

purge

unload all the current modules

If you use Python3:$ module load python/3.7.3

692022/4/22

Supercomputing for Beginners

Environment modules (2)Load a previous version of Intel compiler[tUVXYZ@obcx01 ~]$ module load intel/2020.1.217[tUVXYZ@obcx04 ~]$ module list Return
Currently Loaded Modulefiles:
1) impi/2019.7.217
2) intel/2020.1.217

ReturnAn appropriate Intel MPI library is also loaded

Load newer Python3 (system default is Python 2.7.5)[tUVXYZ@obcx01 ~]$ module load python/3.7.3
Return[tUVXYZ@obcx01 ~]$ module list
Currently Loaded Modulefiles:
1) impi/2019.7.217
2) intel/2020.1.217

Return3) python/3.7.3

702022/4/22

Supercomputing for Beginners

How to use supercomputer
login node = entrance
where you can run
commands[tut138@obcx02~]$ ls -l
drwxr-x--- 2 shiba group 10 1
Apr 13:00 test.out[tut138@obcx02~]$ ./test.out
Hello world[tut138@obcx02~]$ qsub a.sh

2. Jobsubmission

1. LoginYour terminal

SSH

You need to prepare
your SSH key toestablish connection

3. Program execution

Login nodes

Compute nodes= “main body”

712022/4/22

Supercomputing for Beginners

Job script = work instructions for supercomputers
Write a script to let your program run on compute nodes.
It is forbidden to run your program (binary) on the login node after compilation.
ー login nodes are shared with other users, so should not be under load
C, Fortran (fibonacci.sh)#!/bin/bash#PJM -L rscgrp=tutorial#PJM -L node=1#PJM -L elapse=0:01:00#PJM -g gt00
#PJM -N fibonacci#PJM -o stdout.txt#PJM -jPython (fibonacci.sh)［resource group］［number of nodes］[ maximum time］［group name］［job name］［stdout file]［merge error into stdout]

module purge
module load intel/2020.1.217./fibonacci.out

module load python/3.7.3
python ./fibonacci.py

722022/4/22

73

Supercomputing for Beginners

We have already arranged your job scripts
Copy a job script we have prepared in advance.
Return[tUVXYZ@obcx01 ~]$ cd[tUVXYZ@obcx01 ~]$ cd fibonacci

ReturnC, Fortran[tUVXYZ@obcx01 fibonacci]$ cp /work/gt00/share/z30122/fibo_c/fibonacci.sh .

PythonReturn[tUVXYZ@obcx01 fibonacci]$ cp /work/gt00/share/z30122/fibo_python/fibonacci.sh .
Return2022/4/22

74

Supercomputing for Beginners

In case you could not edit the program in time
You may copy the one we have prepared in advance.
C[tUVXYZ@obcx01 fibonacci]$ cp /work/gt00/share/z30122/fibo_c/fibonacci.c .

FortranReturn[tUVXYZ@obcx01 fibonacci]$ cp /work/gt00/share/z30122/fibo_fortran/fibonacci.f90 .

PythonReturn[tUVXYZ@obcx01 fibonacci]$ cp /work/gt00/share/z30122/fibo_python/fibonacci.py .
Return2022/4/22

75

Supercomputing for Beginners

Compile the program
If you use C or Fortran→ Compile the code to generate an executable (binary) file.
C[tUVXYZ@obcx01 fibonacci]$ icc fibonacci.c -o fibonacci.out
programReturnexecutable

Fortran[tUVXYZ@obcx01 fibonacci]$ ifort fibonacci.f90 -o fibonacci.out
Return2022/4/22

Supercomputing for Beginners

CompilersIntel compiler / GNU compilers are available
Source code typeIntel
Serial codes
GNU
Intel
MPI parallel codes
GNU

lang

commandC

icc

Fortran

ifort

C

gcc

FortrangfortranC

mpiiccFortranmpiifortC

mpicc

Fortranmpif77 or mpif90

Ex 1) Compiling a C code by Intel compiler$ icc [option] “source.c” –o “exe.out”
Ex 2) Compiling a Fortran90/95 code by GNU compiler$ gfortran [option] -free-line-length-none “source.f90” –o “exe.out”-free-line-length-none : specifying Fortran free form

76

2022/4/22

Supercomputing for Beginners

Compiler option examples
Languageparallelization C/Fortran

C/Fortran

Intel option

GNU option-qopenmp-fopenmp-g

Embed debugging information
into the executables-Wall

Display all warnings (potential
bugs)

debugging

Fortranoptimization

Enable OpenMP-traceback-fbacktrace

Trace where an error happensin running the executable-checkbounds-fbounds-check

Detect whether a variable iswithin bounds-O0、-O1、-O2、-O3

Optimization (larger values
are more aggressive)-xHostCPU-architechture-specificoptimization

C/Fortran-march=native77

2022/4/22

78

Supercomputing for Beginners

Running your jobs on OBCXYou cannot run your code from /home directory on OBCX
→ Copy your program into your /work directory
Return[tUVXYZ@obcx01 ~]$ cd
Return[tUVXYZ@obcx01 ~]$ ls
fibonacci[tUVXYZ@obcx01 ~]$ cp –r fibonacci[tUVXYZ@obcx01 ~]$ cd/work/gt00/tUVXYZ//work/gt00/tUVXYZ/fibonacci

ReturnReturnSubmit the job script, then your program will be run on the compute nodes.
[tUVXYZ@obcx01 fibonacci]$ pjsub fibonacci.sh

Return2022/4/22

Supercomputing for Beginners

Check you job status & display the results
Confirm your jobs that are now running.
（Perhaps nothing will be displayed if they terminate at once).
[tUVXYZ@obcx01 fibonacci]$ pjstat

Display the results.
[tUVXYZ@obcx01 fibonacci]$ more result_fibo.txt［

Returnwill let you go downwards ]Search 747031 in the results.
[tUVXYZ@obcx01 fibonacci]$ grep 747031 result_fibo.txt

792022/4/22

Supercomputing for Beginners

How to manage your job – scheduler commands
commandrole

how to use

pjsub

submit a job$pjsub “script.sh”

pjdel

delete a job$pjdel “job ID”

pjstatquaery job status$pjstatCaution) these commands are specific to Fujitsu Supercomputers

pjstat optionsn -H: confirm jobs that have finishedn --rsc -b : query job “congestion” status in each resource groupn --rsc -x : query the number of nodes (resources) a user can request in a jobn --nodeuse： Check the current resource usage over the whole OBCX system802022/4/22

Supercomputing for Beginners

On your trial account for this workshop
You can use this account for one month.
“lecture” resource group is available after the workshop.
“tutorial” is available only today (13-17 pm).
Job script
fibonacci.sh#!/bin/bash#PJM -L rscgrp=lecture#PJM -L node=1#PJM -L elapse=0:01:00#PJM -g gt00
#PJM -N fibonacci#PJM -o stdout.txt#PJM -jmodule purge
module load intel/2020.1.217./fibonacci.out［resource group]812022/4/22

Supercomputing for Beginners

Exercise: parallel computation of πI=<latexit sha1_base64="YIu/MegrnHwiO6kVQlrJjYvJwyI=">AAADJXichVJNaxRBEH2Z+BHjRzbmIgRkcBMRhKUmBAyCEOLF3PK1SSC7WWZme5MmMz3DTO9m47A3T/4BD54UgqhH/QdevOSmQk6exWMEPXiwpndQNCT2MN3Vr+pVv6puLw5kqokOB6zBM2fPnR+6MHzx0uUrI6XRq6tp1E58UfWjIErWPTcVgVSiqqUOxHqcCDf0ArHm7dzP/WsdkaQyUit6Lxb10N1SsiV9VzPUKF2ft+/ZNal0gzYdu9ZKXD+b7mXO7e7mVK/ZbZTKVCEz7OOGUxhlFGMhKv1ADU1E8NFGCAEFzXYAFyl/G3BAiBmrI2MsYUsav0APw8xtc5TgCJfRHZ63eLdRoIr3ec7UsH0+JeA/YaaNSfpIL+mI3tNr+kI/T8yVmRy5lj1evT5XxI2Rx9eWv/+XFfKqsf2HdapmjRZmjFbJ2mOD5FX4fX7n4ZOj5btLk9lNek5fWf8zOqR3XIHqfPP3F8XS01P0eKzl5I5lfFKXT49M/9MiUrFn1/QtNJUovqkM+5yrw55cVX4/dYPlvYk4R9+fd7r5u3obEyhz5AQzevxEnH8fxHFjdariUMVZnC7PzhWPZQjjuIFbnOcOZvEAC6iyhkd4hTd4a72wDqwP1qd+qDVQcMbw17A+/wKS17fL</latexit>

Z 1
0

=⇡

4
dx
2
1+x

4
3

quadrature by parts↓
sub-billion meshes

2
1
0
0

0.2

0.4

0.6

0.8

1

x

We will see parallelization speedup.
We prepared 3 sample programs: C, Fortran, Python
Exercise: measure execution time for various degree of parallelism(understanding of MPI is not our focus now, just run ! )82

2022/4/22

Supercomputing for Beginners

C source codes
pi.c [serial]

pi_mpi.c [parallel]#include <stdlib.h>#include <stdio.h>#include <stdlib.h>#include <stdio.h>#include <mpi.h>

int main(void) {int i;
int ndiv = 10000000;

double width = 1.0 / (double)ndiv;
printf("width = %.15f\n", width);
double sum = 0.0;
double x = 0.0;

for (i=0; i<ndiv; i++) {x = (i+0.5)*width;
sum += width * 4.0 / (1.0 + x*x);}

for (i = myrank*ndiv_local;i < (myrank+1)*ndiv_local; i++) {x = (i + 0.5) * width;
sum = sum + width * 4.0 / (1.0 + x*x);}

int main(int argc, char* argv[]) {int ndiv = 5600000000;
int ierr, myrank, nprocs;
int ndiv_local, i;
double x, width, sum, total_sum;
double t1, t2;

MPI_Reduce(&sum, &total_sum, 1, MPI_DOUBLE,
MPI_SUM, 0, MPI_COMM_WORLD);
t2 = MPI_Wtime();width = 1.0 / (double)ndiv;
if (myrank == 0) {printf("PI(MPI) = %.18f\n", total_sum);printf("Number of cores utilized = %d\n", nprocs);
printf("Execution time = %.8f (sec.)\n", t2 - t1);}

ierr = MPI_Init(&argc, &argv);
ierr = MPI_Comm_rank(MPI_COMM_WORLD,&myrank);ierr = MPI_Comm_size(MPI_COMM_WORLD,&nprocs);

ierr = MPI_Finalize();t1 = MPI_Wtime();printf("PI = %.15f\n", sum);
return 0;}

return 0;
sum = 0.0;
ndiv_local = ndiv / nprocs;}

compile[tUVXYZ@obcx01 calc_pi_mpi]$ mpiicc pi_mpi.c –o pi_mpi.out

862022/4/22

Supercomputing for Beginners

Fortran source codes
pi.f90 [serial]

pi_mpi.f90 [parallel]

program main

program main

implicit none

Use mpi
implicit none

integer i, ndiv
real(kind=8) unit, width, sum, x“_8” indicates 8-byte integer (larger # of digits)
do i=myrank*ndiv_local+1, (myrank+1)*ndiv_local
x = ( dble(i-1) + 0.5)*width
sum = sum + width * 4.0 / (1.0 + x*x)
end docall MPI_REDUCE(sum, total_sum, 1, MPI_REAL8,MPI_SUM, 0, MPI_COMM_WORLD, ierr)ndiv = 560000000
width = 1.0 / dble(ndiv)

integer ndiv, ierr, myrank, nprocs
integer ndiv_local, i;real(kind=8) unit, width, sum, x, total_sum
real(kind=8) t1, t2

print *, "width"
print '(F18.14)', widthndiv = 5600000000_8
width = 1.0 / dble(ndiv)

sum = 0.0d0
x = 0.0d0

call MPI_INIT(ierr)call MPI_COMM_RANK(MPI_COMM_WORLD,
myrank, ierr)call MPI_COMM_SIZE(MPI_COMM_WORLD,
nprocs, ierr)

if (myrank .eq. 0) thenprint "('PI(MPI) = ', F18.16)", total_sum
print "('Number of cores utilized = ', i0)", nprocs
print "('Execution time = ', F12.8)", t2 - t1
endif

do i = 1, ndiv
x = ( dble(i-1) + 0.5) * width
sum += width * 4.0 / (1.0 + x*x)
end doprint *, "sum"print '(F18.14)', sum

t2 = MPI_Wtime()call MPI_FINALIZE(ierr)end program main

sum = 0.0d0
ndiv_local = ndiv / nprocs
t1 = MPI_Wtime()

end program main

Compile[tUVXYZ@obcx01 calc_pi_fortran]$ mpiifort pi_mpi.f90 –o pi_mpi.out

872022/4/22

Supercomputing for Beginners

Python source codespi.py [serial]

pi_mpi.py [parallel]

program main
implicit none

from mpi4py import MPI
import numpy

integer i, ndiv
real(kind=8) unit, width, sum, xcomm = MPI.COMM_WORLD
myrank = comm.Get_rank();
nprocs = comm.Get_size();ndiv = 560000000
width = 1.0 / dble(ndiv)
print *, "width"
print '(F18.14)', width
sum = 0.0
x = 0.0
do i = 1, ndiv
x = ( dble(i-1) + 0.5) * width
sum = sum + width * 4.0 / (1.0 + x*x)
end doprint *, "sum"print '(F18.14)', sum

ndiv = 560000000
width = 1.0/ndiv
t1 = MPI.Wtime()
sum = numpy.zeros(1)total_sum = numpy.zeros(1)ndiv_local = ndiv // nprocs
for i in range (myrank*ndiv_local, (myrank+1)*ndiv_local):x = width * (i+0.5)
sum[0] = sum[0] + width * 4.0 / (1.0 + x*x)comm.Reduce([sum, MPI.DOUBLE], total_sum, op=MPI.SUM, root=0)
t2 = MPI.Wtime()

end program main
if comm.rank == 0:
print("PI(MPI) = ", total_sum[0])print("Number of cores utilized = ", nprocs)
print("Exectution time = ", t2 - t1, " (sec.)"882022/4/22

Supercomputing for Beginners

Copy the codes into your own “/work” directory[tUVXYZ@obcx01 ~]$ cd /work/gt00/tUVXYZ[tUVXYZ@obcx01 tUVXYZ]$ mkdir calc_pi_mpi[tUVXYZ@obcx01 tUVXYZ]$ cd calc_pi_mpi[tUVXYZ@obcx01 calc_pi_mpi]$ pwd/work/gt00/tUVXYZ/calc_pi_mpi

Fortran$ cp /work/gt00/share/z30122/pi_fortran_mpi/* .

Casterisk = wild card (everything)$ cp /work/gt00/share/z30122/pi_c_mpi/* .

Python$ cp /work/gt00/share/z30122/pi_python_mpi/* .

89

2022/4/22

Supercomputing for Beginners

Compile parallel programs
For C & Fortran, we need to compile the source code.
C[tUVXYZ@obcx01 calc_pi_mpi]$ mpiicc pi_mpi.c –o pi_mpi.out

Fortran[tUVXYZ@obcx01 calc_pi_mpi]$ mpiifort pi_mpi.f90 –o pi_mpi.out

902022/4/22

Supercomputing for Beginners

For Python users
You need to set up in advance[tUVXYZ@obcx01 ~]$ cd /work/gt00/tUVXYZ/calc_pi_mpi[tUVXYZ@obcx01 calc_pi_mpi]$ emacs setenv.sh(or vim setenv.sh)
Change the following:

export PYTHONUSERBASE=/work/gt00/tUVXYZ/.local⇩
your own user ID
Run setup.sh(install numpy and mpi4py)[tUVXYZ@obcx01 ~]$ ./setup.sh

91

2022/4/22

Supercomputing for Beginners

Submit parallel jobs
We prepared job scripts with various degrees of parallelism,
to see performance increase.
[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n1c0004.sh- 1 node, 4 cores[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n1c0028.sh- 1 node, 28 cores[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n1c0056.sh- 1 node, 56 cores[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n2c0112.sh- 2 nodes, 112 cores[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n4c0224.sh- 4 nodes, 224 cores[tUVXYZ@obcx01 calc_pi_mpi]$ pjsub run_n8c0448.sh- 8 nodes, 448 cores922022/4/22

Supercomputing for Beginners

Confirm the job status
Check your jobs and wait until all your jobs finish.
[tUVXYZ@obcx01 calc_pi_mpi]$ pjstat

You will get the results after the jobs finish.
[tUVXYZ@obcx01 calc_pi_mpi]$ ls
pi_mpi.c
pi_mpi.out
result_n1c0004.txt
result_n1c0028.txt
result_n1c0056.txt

result_n2c0112.txt
result_n4c0224.txt
result_n4c0448.txt
run_n1c0004.sh
run_n1c0028.sh

run_n1c0056.sh
run_n2c0112.sh
run_n4c0224.sh
run_n8c0448.sh“result_n*c****.txt” are output files wherein the results are stored.

932022/4/22

Supercomputing for Beginners

Confirm the results and compare execution times
You will find elapse times of your jobs in the output files.
Let’s see the beginning part of each output file.
[tUVXYZ@obcx01 calc_pi_mpi]$ head result*.txt

We find that the execution time is shorter with smaller number of nodes.
==> result_n1c0004.txt <==PI(MPI) = 3.141592653589913464
Number of cores utilized = 4
Execution time = **.******** (sec.)==> result_n2c0112.txt <==PI(MPI) = 3.141592653589794892
Number of cores utilized = 112
Execution time = **.******** (sec.)==> result_n1c0028.txt <==PI(MPI) = 3.141592653589770912
Number of cores utilized = 28
Execution time = **.******** (sec.)==> result_n4c0224.txt <==PI(MPI) = 3.141592653589791340
Number of cores utilized = 224
Execution time = **.******** (sec.)==> result_n1c0056.txt <==PI(MPI) = 3.141592653589800221
Number of cores utilized = 56
Execution time = **.******** (sec.)==> result_n8c0448.txt <==PI(MPI) = 3.141592653589797557
Number of cores utilized = 448
Execution time = **.******** (sec.)

94

2022/4/22

Supercomputing for Beginners

97

If you want to “log in” to the computation node…

Interactive job
We can run a job on the compute nodes as if we were
working in the bash shell on them.
[tUVXYZ@obcx04 tUVXYZ]$ pjsub --interact -g gt00 -L rscgrp=interactive,node=1[INFO] PJM 0000 pjsub Job 517079 submitted.
[INFO] PJM 0081 .connected.
[INFO] PJM 0082 pjsub Interactive job 517079 started.
[tUVXYZ@cx0065 tUVXYZ]$

You are “virtually” logging into a compute node (cx0065) via the login node (node04).

Compute nodes are not shared with other users but are for your exclusive use.
You may run any jobs you like in the interactive job.

2022/4/22

Supercomputing for Beginners：実習編

98

Finally: transfer your simulation data to your laptop
Log out from supercomputer[tUVXYZ@obcx01 ****]$ exit
Mac-mini:~ shiba$

Transfer a file using “sftp”
Mac-mini:~ shiba$ sftp tUVXYZ@obcx.cc.u-tokyo.ac.jp
sftp > cd /work/gt00/tUVXYZ/fibo
sftp > get fibonacci.txt
Fetching /work/00/gt00/tUVXYZ/fibonacci.txt fibonacci.txt/work/00/gt00/tUVXYZ/fibonacci.txt
100% 171
11.9KB/s
sftp > exit
Mac-mini:~ shiba$ ls
fibonacci.txt

00:00