4.
Revision 1.2

Copyright© 2011, XLsoft K.K. All rights reserved.

1

icc

®C

icpc

®Cifort® Fortran

TA/TC® Trace Analyzer/ Collector

VTune® VTune Amplifier XE<Install dir>
: <TA/TC Install dir> = /opt/intel/itac/8.0.3.007/

mpiiccmpicc

mpiicpc

mpic++, mpicxx® C++

mpiifort

mpif77,mpif90® Fortran

mpirunmpiexec.hydra®C® MPI(MPDHydra)

Copyright© 2011, XLsoft K.K. All rights reserved.

2® VTune Amplifier XE•Hotspot
Concurrency
Lock & Wait
Hardware Event
Visual Studio* 2010
® VTune Amplifier XE

Copyright© 2011, XLsoft K.K. All rights reserved.

3® VTune Amplifier XECPU••• GUI•

MPICopyright© 2011, XLsoft K.K. All rights reserved.

4• GUI<Analysis Type>hotspots
concurrency
locksandwaits:

core2-general-exploration
nehalem-general-exploration
snb-general-exploration
GUICopyright© 2011, XLsoft K.K. All rights reserved.

5® VTune Amplifier XE※MPI®

–x86_64VTune Amplifier XEsource <VTune Install dir>/amplxe-vars.sh
2011source /opt/intel/vtune_amplifier_xe/amplxe-vars.shVTune–()amplxe-cl -collect <Analysis Type> -- <program>–mpiexec -n 1024 amplxe-cl -collect <Analysis Type> -r <name> -- <program>※ <name>※ <Analysis Type>

Copyright© 2011, XLsoft K.K. All rights reserved.

6® VTune Amplifier XE•® VTune Amplifier XE
Performance Monitoring Unit (PMU)
PMU<Analysis Type> =lightweight-hotspots, or snb-general-exploration

• PMU
•

1

1
® VTune Amplifier XE

PMU
11
mpiexec -host host001 -n 1 amplxe-cl -collect <Analysis Type> -r <name>-- <program>: -host host001 -n 15 <program>: -host host 002 -n 1
amplxe-cl -collect <Analysis Type> -r <name> -- <program>: -hosthost002 -n 15 …

Copyright© 2011, XLsoft K.K. All rights reserved.

7

Concurrency(Concurrency)(1.42

Copyright© 2011, XLsoft K.K. All rights reserved.

)

0.92

8

฀

฀ ฀-Copyright© 2011, XLsoft K.K. All rights reserved.

9

Copyright© 2011, XLsoft K.K. All rights reserved.

10snb_general-exploration(Sandy Bridge

Copyright© 2011, XLsoft K.K. All rights reserved.

)

11Performance Monitoring Unit (PMU)

PMUPerformance•
Monitoring Unit (PMU)• VTunePMU

Copyright© 2011, XLsoft K.K. All rights reserved.

12•Core 2

CPU_CLK_UNHALTED.CORE

MEM_LOAD_RETIRED.L2_LINE_MISS

Core i7 2010

CPU_CLK_UNHALTED. THREAD_P

MEM_LOAD_RETIRED. LLC_MISS

Core i7 2011

CPU_CLK_UNHALTED. THREAD_P

MEM_LOAD_UOPS_MISC_RETIRED. LLC_MISS_PS

Copyright© 2011, XLsoft K.K. All rights reserved.

13

MEM_LOAD_UOPS_RETIRED.L1_HIT_PS
MEM_LOAD_UOPS_RETIRED.L2_HIT_PS
MEM_LOAD_UOPS_RETIRED.L3_HIT_PS
MEM_LOAD_UOPS_LLC_HIT_RETIRED.XSNP_HIT_PS
MEM_LOAD_UOPS_LLC_HIT_RETIRED.XSNP_HITM_PS
MEM_LOAD_UOPS_MISC_RETIRED.LLC_MISS_PS
MEM_LOAD_UOPS_RETIRED.HIT_LFB_PS<Analysis Type>

Copyright© 2011, XLsoft K.K. All rights reserved.

14® Trace Analyzer / Collector® Trace Analyzer/Collector•– MPI–hotspot•–– MPI–––– MPI

Copyright© 2011, XLsoft K.K. All rights reserved.

15® Trace Analyzer/Collector– x86_64®

source <composer Install dir>/bin/compilervars.sh intel64
source <icc Install dir>/bin/iccvars.sh intel64
source <ifort Install dir>/bin/ifortvars.sh intel64®

– x86_64

MPIsource <Intel MPI Install dir>/bin64/mpivars.sh®

–Trace Analyzer/Collector

source <TA/TC Install dir>/bin/itacvars.sh
Info.

composerC

icc

ifort.sh.csh

Copyright© 2011, XLsoft K.K. All rights reserved.

16® Trace Analyzer/ Collector

1•>mpiicc test.c –o test.exe -trace>mpiifort test.f –o test.exe -trace•(stf)>mpirun -n 8 ./test.exe>mpiexec –n 8 ./test.exe•Trace Analyzer> <TA/TC Install dir>/bin/traceanalyzer text.exe.stf
GUIstf> <TA/TC Install dir>/bin/traceanalyzer( /opt/intel/itac/8.0.3.007/bin/traceanalyzer )Copyright© 2011, XLsoft K.K. All rights reserved.

17® Trace Analyzer/ Collector

• stf
•

2

: [File]-[Open]stf: [Charts]-[Event Timeline]••

Function ProrileTips(()::i): or

Function Profile

Copyright© 2011, XLsoft K.K. All rights reserved.

18® Trace Analyzer® Trace Collector® Tracer Collector– MPI– API® Tracer Analyzer––Copyright© 2011, XLsoft K.K. All rights reserved.

® Trace Collector
/MPI

19

® Trace Collector (TC)

TC/TC

TC

Copyright© 2011, XLsoft K.K. All rights reserved.

20® Trace Collector

1•––Copyright© 2011, XLsoft K.K. All rights reserved.

21® Trace Collector2•>mpiicc sample.c -L${VT_LIB_DIR} -lVT ${VT_ADD_LIBS}>mpiifort sample.f -L${VT_LIB_DIR} -lVT ${VT_ADD_LIBS}>mpiicc sample.c -L${VT_LIB_DIR} -lVT ${VT_ADD_LIBS}
A

A-L${VT_LIB_DIR}

TC

B-lVT

MPI

C${VT_ADD_LIBS}

B

CVTnull VTfsTCVTmc

VTcs※–trace = –L${VT_LIB_DIR} –lVT ${VT_ADD_LIBS}> mpiicc sample.c –trace

Copyright© 2011, XLsoft K.K. All rights reserved.

22® Trace Collector

3• TC>mpiicc sample.c -L${VT_LIB_DIR} -lVT ${VT_ADD_LIBS}-lVT-lVTnulllibVTnull.so-lVT

libVT.so-lVTfs

libVTfs.so-lVTmc

libVTmc.so-lVTcs

libVTcs.so

Fale-safe(Copyright© 2011, XLsoft K.K. All rights reserved.

)23® Trace Collector1) –g4

VT_PCTRACE
export (setenv)>mpiicc –g sample.c –o sample.exe -trace>mpirun -genv VT_PCTRACE on -n 8 ./sample.exe

2) TA
3)

sample.stf[Charts]-[Event Timeline][Details on Function]4) [Show Source]
5)
VT_PCTRACE(PC)

Copyright© 2011, XLsoft K.K. All rights reserved.

24

: MPI

Copyright© 2011, XLsoft K.K. All rights reserved.

25

: MPI1• Message Checking
MPI>mpiicc -g sample.c -L${VT_LIB_DIR} -lVTmc ${VT_ADD_LIBS}-lVTmcMessage Checking>mpirun -genv VT_CHECK_TRACING on -genv VT_DEADLOCK_TIMEOUT 20s -genv
VT_DEADLOCK_WARNING 25s -n 8 ./sample.exe
VT_CHECK_TRACING on

Message Checking

VT_DEADLOCK_TIMEOUT 20s(20)VT_DEADLOCK_WARNING 25s(25Copyright© 2011, XLsoft K.K. All rights reserved.

)

26

: MPI2

http://www.shodor.org/refdesk/Resources/Tutorials/BasicMPI/deadlock.c libVTmc.so()>mpiicc -g deadlock.c -o deadlock.exe -L${VT_LIB_DIR} -lVTmc ${VT_ADD_LIBS}>mpirun -genv VT_CHECK_TRACING on -genv VT_DEADLOCK_TIMEOUT 20s -genv
VT_DEADLOCK_WARNING 25s -n 2 ./deadlock.exe 0 80000[0][0] ERROR:ERROR: nono progress
progress observed
observed inin any
any processprocess forfor over
over 0:20
0:20 minutes,minutes, aborting
aborting application
application[0][0] WARNING:WARNING: starting
starting emergency
emergency trace
trace file
file writing
writing[0][0] ERROR:ERROR: GLOBAL:DEADLOCK:HARD:GLOBAL:DEADLOCK:HARD: fatal
fatal error
error
[0]
[0] ERROR:ERROR: Application
Application aborted
aborted because
because nono progress
progress waswas observed
observed forfor over
over 0:20
0:20 minutes,minutes,[0][0] ERROR:ERROR: check
check forfor real
real deadlock
deadlock (cycle(cycle ofof processes
processes waiting
waiting forfor data)data) oror
[0][0] ERROR:ERROR: potential
potential deadlock
deadlock (processes(processes sending
sending data
data toto each
each other
other andand getting
getting blocked
blocked[0][0] ERROR:ERROR: because
because thethe MPIMPI might
might wait
wait forfor thethe corresponding
corresponding receive).
receive).
[0][0] ERROR:ERROR: [0][0] nono progress
progress observed
observed forfor over
over 0:20
0:20 minutes,
minutes, processprocess is
is currently
currently inin MPIMPI call:call:[0]
MPI_Recv(*buf=0x7fff1cdb2474,[0] ERROR:ERROR:
MPI_Recv(*buf=0x7fff1cdb2474, count=800000,count=800000, datatype=MPI_INT,
datatype=MPI_INT, source=1,source=1,
tag=999,
tag=999, comm=MPI_COMM_WORLD,
comm=MPI_COMM_WORLD, *status=0x7fff1d0bf874)*status=0x7fff1d0bf874)[0]main[0] ERROR:ERROR:
main (/tmp/inteltest/focus/deadlock.c:49)(/tmp/inteltest/focus/deadlock.c:49)[0](/lib64/libc-2.12.so)[0] ERROR:ERROR:(/lib64/libc-2.12.so)[0](/tmp/inteltest/focus/odeadlock.exe)[0] ERROR:ERROR:(/tmp/inteltest/focus/odeadlock.exe)[0][0] ERROR:ERROR: [1][1] nono progress
progress observed
observed forfor overover 0:20
0:20 minutes,
minutes, processprocess is
is currently
currently inin MPIMPI call:call:[0]
MPI_Recv(*buf=0x7fffa4bd4ce4,[0] ERROR:ERROR:
MPI_Recv(*buf=0x7fffa4bd4ce4, count=800000,count=800000, datatype=MPI_INT,
datatype=MPI_INT, source=0,source=0,tag=999,
tag=999, comm=MPI_COMM_WORLD,
comm=MPI_COMM_WORLD, *status=0x7fffa4ee20e4)*status=0x7fffa4ee20e4)[0]main[0] ERROR:ERROR:
main (/tmp/inteltest/focus/deadlock.c:49)(/tmp/inteltest/focus/deadlock.c:49)[0](/lib64/libc-2.12.so)[0] ERROR:ERROR:(/lib64/libc-2.12.so)[0](/tmp/inteltest/focus/odeadlock.exe)[0] ERROR:ERROR:(/tmp/inteltest/focus/odeadlock.exe)[0][0] INFO:INFO: Writing
Writing tracefile
tracefile odeadlock.exe.stf
odeadlock.exe.stf inin /tmp/inteltest/focus/stf/tmp/inteltest/focus/stf[0][0] INFO:INFO: GLOBAL:DEADLOCK:HARD:GLOBAL:DEADLOCK:HARD: found
found 1
1 timetime (1(1 error
error ++0
0 warnings),warnings), 00 reports
reports were
were suppressed
suppressed[0][0] INFO:INFO: FoundFound 1
1 problem
problem (1(1 error
error ++00 warnings),warnings), 00 reports
reports werewere suppressed.
suppressed.
APPLICATIONAPPLICATION TERMINATEDTERMINATED WITH
WITH THETHE EXIT
EXIT STRING:STRING: Hangup
Hangup (signal(signal 1)
1)

Copyright© 2011, XLsoft K.K. All rights reserved.

27

: MPI

3TA[Details on Function][Charts]-[Event Timeline][Details on Function][Issue]Copyright© 2011, XLsoft K.K. All rights reserved.

28

: itcpin

1•––

Copyright© 2011, XLsoft K.K. All rights reserved.

29

: itcpin
itcpin2TC•mpirun <mpi options> itcpin [<TC options>] -- <app command line>※ "--"TC•>mpiicc sample.c -o sample.exe>mpirun -n 10 itcpin -–run -- ./sample.exe>mpiexec -genv VT_CHECK_TRACING on -genv VT_DEADLOCK_TIMEOUT 20s -genv
VT_DEADLOCK_WARNING 25s –n 8 itcpin --insert libVTmc.so -–run -./sample.exe

Copyright© 2011, XLsoft K.K. All rights reserved.

30

: itcpin3• TC
mpirun <mpi options> itcpin [<TC options>] -- <app command line>

TC--runitcpin--insert

TClibVT.so: --insert libVTfs.solibVTnull.so libVTmc.so libVTcs.soTC® Trace Collector<TA/TC Install dir>/doc/ITC_Reference_Guide.pdf[3.4 Tracing of Binaries and Binary Instrumentation]

Copyright© 2011, XLsoft K.K. All rights reserved.

31

1
•
–MPI

–

Copyright© 2011, XLsoft K.K. All rights reserved.

32® Trace Analyzer•

Intel® Trace Analyzer Reference Guide(ITA_Reference_Guide.pdf) [1.7 For the Impatient]•<TA/TC Install dir>/examples/traces/poisson_sendrecv.single.stf: MPI_sendrecv
poisson_icomm.single.stf:•® Trace Analyzer

Copyright© 2011, XLsoft K.K. All rights reserved.

33® Trace Analyzer•[Charts]-[Event Timeline]–() MPI ()

• MPI
•

•
– Event Timeline
Group MPI][MPI Ungroup
MPI

Copyright© 2011, XLsoft K.K. All rights reserved.

34

Advanced Aggregation:•® Trace Analyzer 8.0.3
Aggregation–– 1000

1
Advanced768

1
Copyright© 2011, XLsoft K.K. All rights reserved.

35

Advanced Aggregation:

2•[File]-[Edit Configuration]

Copyright© 2011, XLsoft K.K. All rights reserved.

36

Event TimelineQualitative Timeline

Flat Profile

Message Profile

MPI Error Details

Filtering

Copyright© 2011, XLsoft K.K. All rights reserved.

37

Copyright© 2011, XLsoft K.K. All rights reserved.

38