

&)+-/'  !ö "$÷£2Q°·põ ä¦õ É¶õõ ĔĔúċČĊĒ ŏ ĔĢĦĞĝõ ĔęġĤĜĝġúċĝěĜõ ČġĥęġĤĝĥĪõ ĊĢėĦĤęĘõ ĒğĥģĖĤĢĦġĘ Ŏ¨ ´ ŉıśĬį   ¤ ?
 Â : ŮƄƉƎƑŮƊƐŲžŸťŦŢŎ° · ç « İƀƏůŦŪŸŉç « ĽŝŃô T a   b Ø ï   ŮƄƉƎƑŮƊƐŲžŸťŦŢŉıŜĭļŎƀƏůŦŪŸŉŏĬƋŤžŭŤŧƐŰ& ä ŉŎŮƄƉƎƑŮƊƐŲžŸ
ťŦŢş

ō; ĺç « ŴƉƑźƐūşË ĲĬô l Æ Ò º  ş¥ ĲŃN Ð  Ó ¿ ŌŮƄƉƎƑŮƊƐş

S ¢ ľĬłŎ} 0 l ş² ŀş­  Ŏ ŇōÄ ĲňĲŜĭõ

õ  " ō%  ŃÇ © ōV ŀŜ ª ŊľňĬ " ş' śç ĸ' ë ŀŜz  Ĭæ ş ņŃ¹ +  ª ŊĲņŃÇ ©  L Ŏâ  ōųƆƑůş ĴŜ ª ķË ŞŝňĲŜĭÜ _ Ĭ ŒŎųƆƑůŎY ŌĲ ª ŊľňĬì  Ø ï  ōřśÇ © Ŏŕş  ŀŜ ª  ķç « ĽŝŃĭļŎq  ŏĬô T a   bØ ï   Ŋ> ŐŝňĵśĬ È ¬ ōŏ ċČĊĒöċĝěĜõ ČġĥęġĤĝĥĪõ ĊĢėĦĤęĘõ ĒğĥģĖĤĢĦġĘ÷ ª Ŋľň¯ ŚŝňĲŜĭċČĊĒ  ª ŏĬ L â ĶŚØ ï  ş " ŎÇ © â ōì  ĽŁĬłŎ  ŉ« ¤ ŀŜ ŧŻƍũōřśÇ © ş  ĽŁŜŗŎŉıŜĭċČĊĒ  ª Ŏ½ a Ř0 ¡ ş ĻŜ ĽŚōŏ ċČĊĒ  ª
DEVELOPMENT OF HIFU SIMULATOR FOR LARGE-SCALE COMPUTING A Ŏç « ōŏĬØ ï  ķ " şŋŎřĳō u ŀŜĶş½ T ō¯Ŝj Î ķıśĬy ľĲŮƄƉƎƑ

63ŮƊƐŲžŸťŦŢşç « ľňĸŃĭ H < ŉŏłŎl Æ ōŇĲňH < ŀŜĭõ (.%,*õ ĔĔúċČĊĒ ŉŏØ ï  ķ " ş u ŀŜà µ ş# ¢ ľĬĽŚōì  Ø ï  ōřśc ĸ× ļĽŝŜa & ] Ŏ| è K 2 şŮƄƉƎƑŮƊƐŀŜĭ ō ª â  ŉŎ¢ Ô ş| è « [ ¬ ōŮƄƉƎƑŮƊƐ
ŀŜķĬØ ï  ŏ " ş u ŀŜŃŖĬ ª â  = Ù ŉ× ļņňĲŜ¢ Ô ŗ9 śr ĳj Î ķıŜĭĴŐĬñ â şV Ô ŊľŃŮƄƉƎƑŮƊƐŎI : Ĭñ Ê ó ōřś8 W ľŃØ ï  ķ  ­ ¬ ŊľŃì   L Ŏp ō ¬ Ōì   ş ŜI : ķıśĬ­ ¬ Ç ©  L Ŏ ^ ŌÀ Ã ōt  ş7 œŀķıśĬļŝŚş

 ŀŜōŏÇ © â Ü  ŎZ p ¬ ð F ńĺŉŌĹĬ` ¼ B Ōð F şÒ º V Ô Ŋ

Figure 20. Voxel model of human skull and the array transducer bounded by the numerical domain. The colored contour of the cross section represents the distribution of thevolume fraction of bone. The numerical domain is 120×160×120 mm and is resolved by
ŀŜj Î
ķıŜĭõ
DEVELOPMENT
OF HIFU SIMULATOR FOR LARGE-SCALE COMPUTING
65
600×800×600 grid points with grid spacing of 0.2 mm.

õõFigure 21. Focusing ultrasound propagation through the human skull using an array transducer without
control.
Distribution offrom
the instantaneous
Figure 23. Schematic diagram of the data flows to the HIFU simulator: volumefocus
data
constructed
CT pressure and maximum pressure is visualized with theskull bone: (a) Instantaneous pressure; and (b) Maximum pressure ( pmax = 2.50 MPa).

C ƔĬċČĊĒ ŮƄƉƎƑŮƊƐ k C õdata of a living human body and the volume data of the signed distance function (SDF) representing anõobject defined by CAD are employed for an HIFU simulation.
Copyright ! 2010 John Wiley & Sons, Ltd.

Int. J. Numer. Meth. Fluids 2011; 65:43–66
DOI: 10.1002/fldõ m Å  ĮŎ n şŮƄƉƎƑŮƊƐV Ô ŊŀŜŃŖĬÒ º V Ô ŎƇŷƍd  ŘÇ ©  Ä Řd  Ŋfocal point
placementat the target even when the ultrasoundpropagatedthrough
an inhomogeneous ŉŏĬÿĈćđ ŉ9ĲņŃşŋŎřĳōŮƄƉƎƑŮƊƐō9śÚ ŠŉËĹĶķ@ò ŉıŜĭĔĔúċČĊĒ
medium.
As a realistic application, transcranial HIFU therapy for a brain tumor was modeled well by the
HIFU simulator, where inhomogeneity caused not only displacement of the focal point but also
diffusion of the focusing ultrasound. Even in such cases, phase control based on pre-computation
was highly effective for solving both problems.

When considering pressure waves with viscous attenuation, the momentum equation is
described as
"
#
$%
!
1
!u i
!p
!m
#m ekk $ij +2%m eij − ekk $ij ,
(2)
+
=−
3
!t
!xi !x j

f ĽŝŃ 

3 ª §  şƂŪűƍ¸ è ōã Ä ľŃƂŪűƍ  Ƈŷƍş ¥ ŀŜŉļŎ@ ò ş

where eij = 1/2(!u i /!x j +!u j /!xi ) is the deformation tensor, and #m and %m are the bulk and shearÑ  ľňĲŜĭÒ
ŉthe¥ ŀŜƂŪűƍŏ
ýĠĠisŎh¿sinceŌshearş¸ èōD viscosity,
respectively.ºHere,constitutive equation
for 
fluid
used,
modulusof ō ¥ ŀŜj Î ķı
tissueexceptºforÐbone
is negligibly
than bulk modulus. Althoughbone should be treated asĲĬ0 ¡ řĹÁ  ķ
śĬłŎ
 ŏî
^ ōNsmallerĸĹŌŜĭłŎŃŖĬċĎć
; ĺōŴƉƑźƐūşË
elastic body, bone is modeled as a multicomponent mixture to take into account not shear elasticity
f ŚŝŜ
ōŀŜj
Î ķıŜĭõ
butheterogeneities
of bones
following Aubry et al. [7], in fact CT images can give the internal
structure of bones. They show that the simulated wave fronts passing through such modeled bonesõ Ø ï  «  Ì Ä ŊľňŢƎŤE ŎŸƋƐŰŷƉƑŭƑş ¥ ľňĵśĬ Ň ŇŎŸƋƐŰŷƉƑŭ
of a skull are very close to the corresponding experimental wave fronts propagation through theskull.
The®constitutive
equationfor fluid
can be acceptablefor
discussing
focusing
ultrasoundƑŎşá ' ō*g ŀŜōřśĬØï  ŎŎ Ä theŘNĸĽş*g ŉĸŜÌ Ä şV Ô Ŋľň
through bone. Modeling of bone as anisotropic elastic material will be studied in future to achieve
ĲŜĭõ
more
accurate HIFU simulation.
The
to material compressibility
is higher
thandueŀŜG
to the convectiontermsõ  nonlinear
ŏÍ weffectŎodue& ĶŚŌśĬM
O Ö  Ŏï
 Ŏ
uthat
ōV
± b ş¸è ġ  Ĭ| è  ½ aof continuity and momentum equation depending on the amplitude and nonlinearity coefficient.
Ŏ} ê
\&
ōřņňí
v 2intoľĬĊĈđĈ
nonlinearō Ŀň|«[
şÒº ŀŜĭ
( 2 ŏÒ º ð F şż
Tait’sequation
is employed
to takeaccount theeffectèdue
to theequationof stateof the medium,ƑŹ4  ŉð F & - ľĬżƑŹ" ŏƍƑƀ&- ŀŜŽŤſƌŵŹ ( şs ¥ ľňĲŜĭłŎ ( 2 ō#$
p + A1k 1/&k
ŏ ĐĎċĉďĉ ş ¥ ľňĵśĬð
ŶƑſƍ o ŌŋŎç!k =F
!k0& - Ŏã ( ƆƇƌ» £ Ĭ! % . * g ĬÝ (3)p0 + A1k«   ş0 ¡ 2 ľňĲŜĭƀƏūƋƅ  ŏ ĊĢģĥĖġąü ŉ o ĽŝňĵśĬĐĎċĉďĉ Ŏ* g ōé ŞŜ

2 /& − p and & = (B/A) +1. Here, B/A is the nonlinearity coefficientwhere A1k = !k0 csk0
0
k
k
k
pŏ
ćøøş
¥ ľňĲŜĭƁŵŸŰƃŵŸâ
& ŏtheĊĢģĖĥģĖġąü
ŉŬƑŷţƐūĽŝňĲŜĭõ
[13,
14].
The density
and sonic speed of medium from
Tait’s equation
are used to obtain thedensity and sonic speed of mixture.

õ G ± b ŊľňĬÖ å  P b Ŋß 1 å  P b ş¥ ĲŜĭƒ  Õ x 6  Ɠõ G ± b ş¸ è Ɩ ½ a

2.2.
matchedboundaries
 iPerfectly\ & ŉí
v 2layers
ľĬ|forènonreflectingz;ŏ ½ a Ŏ i \ & ş¥ ĲŜĭõA
Öperfectlyå  Pmatchedb Ɨõ layer (PML) [15] is employed to eliminate the reflection of ultrasound at theboundary of a numerical domain. In the region of PML, Equation (1) is split for each component
with a damping term by decomposing the pressure as p = px + p y + pz :1 ⎛ ∂ px
⎞ ∂u$
+ σ x px ⎟ + # = 0,
2 ⎜
!u
⎝ ∂t
⎠1 ∂x ! px
ρ m csm+'x px +
= 0,
2
!
c
!t
!x
m⎞ sm∂v
1 ⎛ ∂ py$
+ σ y py ⎟1+ # !=p0,
2 ⎜⎝ ∂t⎠ ∂y y +' y p y + !v = 0,
ρ m csm(4)2!m csm!t!y
1 ⎛ ∂ pz∂w⎞$
+ σ z pz ⎟1+ # !=p0z
2 ⎜
!w
⎝
⎠
ρ
c
∂t
∂z
p
+'
= 0.
m sm
z z +
õ
2
!m csm
!t õ
!z

ßthe
1 sameå way,Pb
IntheƗõdamping term is introduced to the momentum equation without decompositionof the velocity according to PML [15],
#
$
"
#
$%
!p
!
1
!u i
!m
#m ekk $ij +2%m eij − ekk $ij ,(5)++'i u i = −3!t!xi !x jõõInt. J. Numer. Meth. Fluids 2011; 65:43–66
DOI: 10.1002/fld

Copyright ! 2010 John Wiley & Sons, Ltd.

  õ l Æ  R ōŏĬ£ 2 Q ° · p Ŏ ďČćć ĵřő Ċēý ş¥ ĲŃĭõ  õĮŎƀƏūƋƅŎS Ë | è ŏ þüüûýüþă ³ ŉıśĬļŝŎĲĹŇĶŎŴƉƑźƐūş{ ŀýĂăûĄĂÿý ³ Û ô Þ 2 ŀŜōřśĬōo / ľŃĭļŝŏĬƀƏūƋƅŎŴƉƑźƐūōřśĬŨƈŵŮƉƄŰŎ Ř º å Ŏ,  şË ĲĬłŎÁ  Ĭ º | è ō5 ŖŜ$ £ | è Ŏ- : ķM ĲĬėĖğĕĥùõ ėĖğĕħùõ ėĖğĕĨùõ ėĖğĕĦ Ŏô Þ 2 şS ¢ ľŃĭÓ ¿ ŏ  Õ x  đĒčĉýā Ŋ đĒčĉĂ ş6  Ŏĭõ õ Ċēý ŉ) ¥ ŉĸŜU J Ý ƀƏžšŤƋş ķÏ Ú ŔŝŜĽŝŜķ& ĶśĬ

ķ& ĶņŃĭõō ĐĎĆďćĂĀīĚĩ ŉŎl Æ ş ľŃĭ¾ ƕ Ŏl Æ ;ŊľňĬ 1 X w   º e Ņķ,  ĽŝĬl Æ ķ;  ŀŜķ~ e

ZZ-HIFU評価報告
野田

茂穂、沖田

浩平、青山

幸也

11/10/2011 @ SS研マルチコアクラスタ性能WG

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

アプリケーション紹介

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

number, we can assume adiabatic change of the medium and neglect the advection term during
ultrasound propagation. The mass conservation equation for the mixture then becomes
1 ! p !u j
= 0,
+
2 !t
!mDEVELOPMENT
csm!x
OF jHIFU SIMULATOR FOR LARGE-SCALE COMPUTING(1)65

高密度焦点式超音波法ソルバ
sound in mixture, csm , arewhere the mixture density, !m , and the adiabatic speed of !
! defined in
DEVELOPMENT OF HIFU SIMULATOR FOR LARGE-SCALE COMPUTING

63

2
2 =
terms of the volume fraction of the medium, f k , as !m = k f k !k and 1/!m csm
k f k /!k csk
under the assumption of a constant volume fraction. The sound source, f p A cos("t), is added to• 超音波を集束させ，焦点で加熱によって腫瘍を焼灼すthe right-handof UltrasoundEquation(1) to represent the ultrasound generated by the transducers.
る治療法（HighIntensityside
Focused: HIFU）- 超音波が体内を伝播する過程や，温度分布および治When considering pressure waves with viscous attenuation, the momentum equation is
療部位の時間発展をシミュレーション
describedas- 医療画像から作成されたボクセル人体モデルを利用
"
#
$%
!
1
!u i
!p
- 治療機器による焦点制御が困難な体深部の腫瘍治療!m#m ekk $ij +2%m eij − ekk $ij ,(2)+=−におけるシミュレーションによる焦点制御支援
3!t!xi !x j- 各個人の体型に対応したシミュレーションによる術

Figure 20. Voxel model of human skull and the array transducer bounded by the numerical domain. The colored contour of the cross section represents the distribution of thevolume fraction of bone. The numerical domain is 120×160×120 mm and is resolved by
600×800×600 grid points with grid spacing of 0.2 mm.

前の治療計画の検討
whereeij = 1/2(!u i /!x j +!u j /!xi ) is the deformation tensor, and #m and %m are the bulk and shear
viscosity,
respectively. Here, the constitutive equation for fluid is used, since
shear modulus of
63
る安全性の検討
tissueexceptforboneisnegligibly
smallerthanbulk
modulus.
Althoughbone
shouldbe treated as• 多成分からなる媒質中の音波の伝播に対する基礎式を
elastic body, bone is modeledFigure
as a23.
multicomponent
mixtureinto
account
notelasticity
Schematic diagram of thedata flowsto
to takethe HIFU
simulator:
volumedatashearconstructedfrom CT
空間n次，時間2次精度の有限差分法によって離散化
of a living human body and the volume data of the signed distance function (SDF) representing an
し，FDTD法に準じて時間発展を行う．領域分割とルー
but heterogeneities of bones data
following
Aubry
et
al.
[7],
in
fact
CT
imagescangivetheinternal
object defined by CAD are employed for an HIFU simulation.
プ分割によるハイブリッド並列化によって大規模並列
structure of bones. They show that the simulated wave fronts passing through such modeled bones
計算が可能となっている．
focalcorresponding
point placement at theexperimental
target even when theultrasoundpropagatedthrough an throughinhomogeneousof a skull are very close to the
wave
frontspropagationthe• Fortran90, C++medium.
skull.
The
constitutive equation
for
fluid
can
be
acceptablefor
discussing
thefocusing
ultrasound• MPI,
OpenMP,SPHEREAs a realistic application, transcranial HIFU therapy for a brain tumor was modeled well by thethrough bone. Modeling of boneanisotropic
elastic material
willstudied ofin thefutureto achieve
HIFU assimulator,
where inhomogeneity
caused not
onlybedisplacement
focal point
but also
diffusion of the focusing ultrasound. Even in such cases, phase control based on pre-computation
more accurate HIFU simulation.
was highly effective for solving both problems.
The nonlinear effect due to material compressibility is higher than that due to the convection terms
All Rights Reserved,
Copyright© ISLiM, RIKEN 2011~ coefficient.
of continuity and momentum equation depending on theamplitude
andnonlinearity
ACKNOWLEDGEMENTS
Figure 20. Voxel model of human skull and the array transducer bounded by the numerTait’s equation is employed to
take
into
nonlineareffectdue
domain.
The accountcolored contour ofthe
the crossrepresents theof the to the equation of statePart ofical
this
study
by thesection‘Researchanddistribution
Developmentof the Next-Generation Integrated
volumefractionofwasbone. supported
The numerical domain
is 120×160×120mm
andis resolved by
600×800×600
grid points
spacing ofof
0.2Education,mm.
Living Matter Simulation’projectof with
the grid
Ministry
Culture, Sports, Science and Technology.
of the medium,The calculations were performed using the RIKEN Integrated Cluster of Clusters (RICC).
#$
p + A1k 1/&k
REFERENCES!k = !k0
(3)
p0 +
A1k
1. Duck FA, Baker AC, Starritt
HC. Ultrasoundin Medicine. Medical Science Series. Institute of Physics Publishing:- 治療機器の設計および認可に向けた臨床試験におけ

DEVELOPMENT OF HIFU SIMULATOR FOR LARGE-SCALE COMPUTING

Figure 21. Focusing ultrasound propagation through the human skull using an array transducer without
focus control. Distribution of the instantaneous pressure and maximum pressure is visualized with theskull bone: (a) Instantaneous pressure; and (b) Maximum pressure ( pmax = 2.50 MPa).
Copyright ! 2010 John Wiley & Sons, Ltd.

Int. J. Numer. Meth. Fluids 2011; 65:43–66
DOI: 10.1002/fld

Bristol and Philadelphia, 1998.

2 /& − p 2. and
Kennedy JE,=
ter(B/A)
Haar GR, Cranston
D. High intensity focused ultrasound: surgery of the future? The British
where A1k = !k0 csk0
0 Journal &ofk Radiology
k +1. Here, B/A is the nonlinearity coefficientk2003; 76:590–599.
3. Umemura
Kawabata
K, Sasaki fromK. In vivothe
acceleration
tissue areheatingusedby microbubble
agent. IEEE
[13, 14]. The density and sonic
speedS, of
mediumTait’sof ultrasonicequationto obtain
theTransactions on Ultrasonics, Ferroelectrics, and Frequency Control 2005; 52:1690–1698.
基礎式と計算手法
density and sonic speed of mixture.
4. Coussios CC, Roy RA. Applications of acoustic and cavitation to noninvasive therapy and drug delivery. Annual

Review of Fluid Mechanics 2008; 40:395–420.
5. Fink M. Time reversal of ultrasonic fields. Part I: Basic principles. IEEE Transactions on Ultrasonics,
Ferroelectrics, and Frequency Control 1992; 39(5):555–566.
• 式１、式２をFDTDにて計算
6. Fink M, Montaldo G, Tanter M. Time-reversal acoustics in biomedical engineering. Annual Review of Biomedical
Engineering 2003; 5:456–497.
• 時間積分は2次精度の陽解法ですが、拡散項がある場合は1次精度の陽解法
7. Aubry J-F, Tanter M, Pernot M, Tomas J-L, Fink M. Experimental demonstration of noninvasive transskull
adaptive focusing based on prior computed tomography scans. Journal of the Acoustical Society of America• 空間は６次精度中心差分
1993; 113(1):84–93.
8. Sun J, Hynynen K. Focusing of therapeutic ultrasoundskull: a numerical study. Journal of thex throughy a humanz
Acoustical Society of America 1998; 104(3):1705–1715.
式１：質量保存式（PML:Perfectly
matchedlayer）
9. Curra
FP, Mourad
PD, Khokhlova VA, Cleveland RO, Crum LA. Numerical simulations of heating patterns
and tissue temperaturex response due to high-intensity focused ultrasound. IEEE Transactions on Ultrasonics,
Figure21.
Focusing
ultrasoundpropagation Control
thehuman47(4):1077–1089.
skull using an array transducer without
Ferroelectrics, and Frequency
2000;
x throughx pressure
focus control.
of the instantaneousand maximum pressure is visualized with the
2 Distributionx (a)10. HariharanP,
Myers
MR, Banerjee
RK.
HIFU procedures
moderate= 2.50MPa). intensities-effect of large blood vessels.
skull
bone:
Instantaneouspressure;
and(b)Maximumpressure ( pmaxat
m smin Medicinex andx Biology 2007; 52:3493–3513.
2
Physics
Copyright
!
2010
John
Wiley
&
Sons,
Ltd.
Int.
J.
Numer.
Meth.
Fluids2011; 65:43–66
m sm

2.2. Perfectly matched layers for nonreflecting boundaries

A perfectly matched layer (PML) [15] is employed to eliminate the reflection of ultrasound at theboundary of a numerical domain. In the region of PML, Equation (1) is split for each component
with a damping term by decomposing the pressure as p = p + p + p :
#
$
1
!p
!u
1! ⎛c ∂ p !t +'⎞ p∂u + !x = 0,
+σ p ⎟ +
= 0,
⎜
⎠ ∂x$
ρ c ⎝ ∂t#! py
1!v
Copyright ! 2010 John Wiley
Int. J. Numer. Meth. Fluids 2011; 65:43–66
(4)
+
= 0,
y Ltd.
⎞ &y pSons,
1! ⎛c2∂ py !t +'∂v
DOI: 10.1002/fld
!y
+
σ
p
+
=
0,
m
sm
2 ⎜
⎝ ∂t# y y ⎟⎠ ∂y$ρ m csm
! pz
!w
1
+'
= 0.
1 ⎛ 2∂ pz
z p∂w
z +
⎞
!2m ⎜csm +!t
σ z pz ⎟ +
= 0!z
⎠ ∂z
ρ m csm ⎝ ∂t
In the same way, the damping term is introduced to the momentum equation without decompositionof the velocity according to PML [15],式２：運動量保存式
#
$
"
#
$%
!p
!
1
!u i
!m
#m ekk $ij +2%m eij − ekk $ij ,(5)++'i u i = −3!t!xi !x j
DOI: 10.1002/fld

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

Copyright ! 2010 John Wiley & Sons, Ltd.

Int. J. Numer. Meth. Fluids 2011; 65:43–66
DOI: 10.1002/fld

性能測定

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

プロファイル取得方法• コンパイルFXFFLAGS= -Cfpp -Kfast,OMP -Am -X9 -Ktl_trt
fx : $(FILE)$(FC) $(FXFFLAGS) $(NSTEPMAX) $(DSIZE) $(SPLIT) $(CALOPT) $(FILE)• 計測実行方法- 通常
fpcoll -Ihwm -Hevent=Statistics ./a.out- PA tool# performance measurement Vol1
fpcoll -tcsv -o output_prof_1.csv -i3600000 Ulocal_event="cycle_counts,op_stv_wait,op_stv_wait_nc_pend,op_stv_wait_sxmiss_ex,branch_instructions,op_stv_wait_ex,op_stv_wait_sxmiss,instruction_counts",section=range ./a.out# performance measurement Vol2
fpcoll -tcsv -o output_prof_2.csv -i3600000 Ulocal_event="eu_comp_wait,cse_window_empty,load_store_instructions,branch_comp_wait,fl_comp_wait,cse_window_e
mpty_sp_full,trap_DMMU_miss,cycle_counts",section=range ./a.out# performance measurement Vol3
fpcoll -tcsv -o output_prof_3.csv -i3600000 -Ulocal_event="cycle_counts,inh_cmit_gpr_2write,
0endop,floating_instructions,impdep2_instructions,3endop,2endop,1endop",section=range ./a.out# performance measurement Vol4
fpcoll -tcsv -o output_prof_4.csv -i3600000 Ulocal_event="cycle_counts,prefetch_instructions,sx_miss_count_pf,sx_miss_count_dm,op_r_iu_req_mi_go,instruction_flo
w_counts,dvp_count_pf,dvp_count_dm",section=range ./a.out

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

BeforeAfter--------------------------------------------------------------------------------------------Fujitsu Performance Profiler Version 3.0
Measured time
: Thu Sep 15 11:58:44 2011
CPU frequency: Process0 2520 (MHz)Type of program: Thread (OpenMP)
Average at sampling interval : 10.0 (ms)Measured range: All ranges--------------------------------------------------------------------------------------------_____________________________________________________________________________________________
Time statistics--------------------------------------------------------------------------------------------Fujitsu Performance Profiler Version 3.0
Measured time
: Thu Sep 15 12:01:26 2011
CPU frequency: Process0 2520 (MHz)Type of program: Thread (OpenMP)
Average at sampling interval : 10.0 (ms)Measured range: All ranges--------------------------------------------------------------------------------------------_____________________________________________________________________________________________
Time statistics

Elapsed(s)User(s)System(s) Barrier(s)-----------------------------------------------------------200.1027
676.9999
0.01200.0100 Application-----------------------------------------------------------200.1027
676.9999
0.01200.0100 Process0

Elapsed(s)User(s)System(s) Barrier(s)-----------------------------------------------------------167.8631
551.9431
0.01110.0000 Application-----------------------------------------------------------167.8631
551.9432
0.01110.0000 Process0

Elapsed(s)User(s)System(s) Barrier(s)-----------------------------------------------------------200.1027
676.9999
0.01200.0100 Process0-----------------------------------------------------------200.1027
171.5852
0.00210.0000 Thread0
199.9842
168.8950
0.00310.0000 Thread1
199.9827
168.7949
0.00320.0100 Thread3
199.9841
167.7248
0.00360.0000 Thread2_____________________________________________________________________________________________
Procedures profile

Elapsed(s)User(s)System(s) Barrier(s)-----------------------------------------------------------167.8631
551.9432
0.01110.0000 Process0-----------------------------------------------------------167.8631
139.6659
0.00200.0000 Thread0167.6828
138.1258
0.00300.0000 Thread2
167.6810
137.1857
0.00330.0000 Thread3
167.6829
136.9657
0.00280.0000 Thread1_____________________________________________________________________________________________
Procedures profile*************************************************************************************
Application - procedures**************************************************************************************************************************************************************************
Application - procedures*************************************************************************************

Cost%
Barrier% StartEnd---------------------------------------------------------------------67694 100.0000
1 0.0015--- Application---------------------------------------------------------------------12509 18.4787
0 0.0000 1377 1849 hifu3d.hifu3d_cal_viscous_
8501 12.5580
0 0.0000 2920 3020 hifu3d.hifu3d_cal_t_
7108 10.5002
0 0.0000 2057 2157 hifu3d.hifu3d_cal_v_
6933 10.2417
0 0.0000 2263 2363 hifu3d.hifu3d_cal_w_
6674 9.8591
0 0.0000 1158 1265 hifu3d.hifu3d_update_mixture_properties_
4915 7.2606
0 0.0000 1856 1956 hifu3d.hifu3d_cal_u_4709 6.9563
0 0.0000 2766 2913 hifu3d.hifu3d_cal_grad_t_3624 5.3535
0 0.0000 2476 2547 hifu3d.hifu3d_cal_p_
3466 5.1201
0 0.0000--- jwe_gpwd
2838 4.1924
0 0.0000 3158 3244 hifu3d.hifu3d_cal_grad_phi_

Cost%
Barrier% StartEnd---------------------------------------------------------------------55192 100.0000
0 0.0000--- Application---------------------------------------------------------------------12233 22.1644
0 0.0000 1407 1927 hifu3d.hifu3d_cal_viscous_
6776 12.2771
0 0.0000 1183 1295 hifu3d.hifu3d_update_mixture_properties_
4661 8.4451
0 0.0000 2866 3013 hifu3d.hifu3d_cal_grad_t_3994 7.2366
0 0.0000 2356 2463 hifu3d.hifu3d_cal_w_
3879 7.0282
0 0.0000 1934 2041 hifu3d.hifu3d_cal_u_3844 6.9648
0 0.0000 2143 2250 hifu3d.hifu3d_cal_v_
3638 6.5915
0 0.0000 2576 2647 hifu3d.hifu3d_cal_p_3604 6.5299
0 0.0000 3020 3166 hifu3d.hifu3d_cal_t_3428 6.2110
0 0.0000--- jwe_gpwd
2851 5.1656
0 0.0000 3304 3390 hifu3d.hifu3d_cal_grad_phi_

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

Before

After

13000

9750

6500

3250

0
l_v
ca

s

ou

isc

l_t

ca

_v

ca

l_w

ca

ert

rop

xtu

mi

_
ate

p
re_

ies al_u
c

_t
rad

l_g

ca

l_p

ca

jwe

hi
_p
rad

d
pw

_g

g
al_

c

d
up

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

チューニング内容• ループ分割によるキャッシュミスの低減：
cal_v, cal_w• 演算量の削減：cal_u,cal_v,cal_w

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

チューニング効果• オリジナル版とチューニング版を8スレッド並列で実- 従ってチューニング版を8スレッドで実行したの行したところ、チューニング版があまり速くないと同様の速度- 並列に実行して効果があるチューニングとほとん- また、チューニング版はループを４分割し、作業

ど変らないか、却って遅くなるチューニングがあ配列を導入しており、これがオリジナル版と比較る。

してオーバーヘッドになる場合は、並列に実行す
ると却って遅くなる• オリジナル版とチューニング版で、速度が変らない
か却って遅くなる理由は、• 並列性能（８スレッド）- オリジナル版では、ループ内で配列を多く使用し- オリジナル：5.3倍ているのでキャッシュミスが発生- チューニング：4.9倍- チューニング版では、ループを4つに分割し、1つのループで使用する配列の数を減らしてキャッシ
ュミスを減少- 一方、オリジナル版を8スレッドで並列に実行すると、キャッシュの大きさが8倍になるため、(本プログラムの場合)、何も修正しなくてもキャッシュミスが減少

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~ 
#9D@A9D@A  2KLB7G@:H58=< 2KLBEFI58=< 2!"9D@A13
133 2  2'(58=< 2.+*.+*!&!!&!%&!

%&!

$&!

$&!

#&!

#&!

!&!

!&!

&!

&!

%& %& $& $& #& #& !& !& & 
#9D@AJ;<>04
/()9D@A9D@A

 
2 2
<A5
KLB7G@:H58=<2
2
KLBEFI58=< 2J;<>04!"9D@A13
C6@?
133 22KLB7G@:H58=<
 22'(58=<KLBEFI58=<2
2

 J;<>04
/()9D@A 2
<A5 2

J;<>04C6@? 2 KLB7G@:H58=< 2 KLBEFI58=< 2&)-,*+..)-,*+..)-,*+..)-,*+..

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

1*,:(,-,*,:(,,
AWTJLXX#,

A6qnAWTJLXX#,AWTJLXX#,# 

#DPRL- #DPRL. #DPRL/ #DPRL0

>-vl

>-vl
&PSXYWZJYPTS'>-vl
&QK+XY'

>.vl

>.vl
&PSXYWZJYPTS'>.vl
&QK+XY'-*53:(,.

1*25:(,5-*/..%

0*--1%-*.0:(,5,*.43%,*450%-*53:(,.

-*53:(,.

-*53:(,.

lrda   =+@dgml      
a
a
a
a
a
a

>-$)>.$"
wx|l}
l}upr
upr&<7+X'&<7+X'-*41:(,,-*5/:(,,?;>@AC?=AC

D>7vl

D>7vl&PSXYWZJYPTS'

D>7vl&QK+XY'

4*/.:(,.

.*-5:(,/.*-.:(,/,*,,,%,*,,,%<WHUO#&XLJ'
"h{t   
atepo
Z@Aivpraa AWTJLXX#,-*./:),-

"h{t
 #&%'AWTJLXX#,

1*35:(,,/*33:(,,   
a
a

,*,2%

lrda

,*,,:(,,-*/,:(,-

0 ivpr- ivpr asfypjzdgmla
-*5-%
,*,,%
swx|dgmla.*5/%.*3,:(,-.*05:(,,

3*-/:(,- 

1*4-:),-

0*54:(,,

   =+@dgml      
a
a
a
a
a
a

2*13%

.+/ ivpr&`b'`bba a=+@dgmla-/*3,%-*.2%/*15:(,,

tepo
Z@Aivpra ~kln_!c
Z@Aivpralrda
,*.5%.*1.%/2*-,%

~kln  `bb
- ivpr _!c ~kln_
a

!c

-*0,:(,,-*05:(,-

3*55:(,,,*,,:(,,

~kln  `bb
- ivpr _!c ~kln_a~kln_!c!ctepoasfypjzdgmlaswx|dgmla
,*3-%
3*10%
0*,1%




-*4.%

.+/ ivpr&`b 0 ivpr
'-*1,:(,-.*11:(,-.+/ ivpr&`b 0 ivpr
'

,*,,%

3*15%

-.*5/%

-*53:(,.

-,,*,,%

"&XLJ'
A6#96D6#&-.'AWTJLXX#,

A6#96D6#&/0'.*1:(,.

J^JQLGJTZSYX TUGXY[G\HPY
0*52:(---*-2:(--J^JQLGJTZSYXPSOGJRPYGNUWG.\WPYLTUGXY[G\HPYGS TUGXY[G\HPYGX IWHSJOGPSXYWZ TUGXY[G\HPYGL TUGXY[G\HPYGX PSXYWZJYPTSGJJXLG\PSKT\G QTHKGXYTWLGPS IWHSJOGJTRUJXLG\PSKT\G YWHUG9??EGLZGJTRUG\HPY
MQGJTRUG\HPY
J^JQLGJTZSYXJGULSK]RPXXGL]
JYPTSX
]
]RPXX
TZSYX
LRUY^
XYWZJYPTSX
G\HPY
LRUY^GXUGMZQQ
RPXX.*,:(,. .*,-:(-,
0*53:(--AWTJLXX#,,*,,:(,,,LSKTU/*,5:(,4-*.0:(-,

MQTHYPSNGPSXYW PRUKLU.GPSXYZJYPTSXWZJYPTSX/*/4:(--

5*-.:(-,/*2.:(-,/*/,:(-,-*05:(-,

0*/,:(--/LSKTU.LSKTU-LSKTU.*,-:(-,/*35:(-,/*31:(-,-*42:(--J^JQLGJTZSYX.*.,:(-,-*/4:(---*02:(,5-*4,:(--

5*1-:(,5.*-.:(,/

0*51:(--UWLMLYJOGPSXY X]GRPXXGJTZS X]GRPXXGJTZS TUGWGPZGWLVGR PSXYWZJYPTSGMQK[UGJTZSYGKK[UGJTZSYGUM
WZJYPTSX
YGUM
YGKR
PGNT
T\GJTZSYX
R

0*52:(--

0*-.:(-,-*..:(,5-*0-:(,3

1*25:(,50*/5:(--

5*3-:(,45*03:(,2CA6B820F===M] <WHUO#&XLJ'
"h{t   
aalrda

AWTJLXX#,# -*1:(,. -*01:(,,/*,3:),.

5*0/:),-

"h{t
 #&%'   
aa

AWTJLXX#,# 

,*,2%

.*22%

lrda

   =+@dgml      
a
a
a
a
a
a

,*,,:(,,

0*5,:(,,-*,.:(,--*13:(,,

0*05:(,,/*22:),-

   =+@dgml      
a
a
a
a
a
a

-*3/%

,*,,%

4*55%

-4*31%

.*43%

4*./%

,*23%

tepo
Z@Aivpra/*-/:),-.*.2:(,,

tepo
Z@Aivpra,*13%

~kln  `bb
- ivpr _!c ~kln_
a

!c

4*41:),-

1*1,:(,,.*45:(,,,*,,:(,,

~kln  `bb
- ivpr _!c ~kln_
a

!c

0*-1%

-*2.%

-,*,5%

1*/-%.+/ ivpr&`b 0 ivpr'

2*/5:(,,-*./:(,-.+/ ivpr&`b 0 ivpr
'

,*,,%

--*3-%..*15%



1*01:(,-

-,,*,,%-*,:(,.

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~

1*,:(,-,*,:(,,
AWTJLXX#,

A6qnAWTJLXX#,AWTJLXX#,##DPRL- #DPRL. #DPRL/ #DPRL0

>-vl

>-vl
&PSXYWZJYPTS'>-vl
&QK+XY'

>.vl

>.vl
&PSXYWZJYPTS'>.vl
&QK+XY'-*21:(,.

1*,.:(,5-*/45%

0*4,-%-*-0:(,5,*/-2%-*,5-%-*21:(,.

-*21:(,.

-*21:(,.

lrda   =+@dgml      
a
a
a
a
a
a



>-$)>.$"
wx|l}
l}upr
upr&<7+X'&<7+X'-*51:(,,?;>@AC?=AC

D>7vl

D>7vl&PSXYWZJYPTS'

D>7vl&QK+XY'.*-/:(,,

5*//:(,.

.*.,:(,/.*/2:(,/,*,,,%,*,,,%`bb
a

~kln  
ivpr _!c ~kln_!c.+/ ivpr&`b 0
'

ivpr



-*,4:(,-.*/3:(,--*21:(,.

.+/ ivpr&`b 0
'

ivpr



-0*/5%-,,*,,%.+/ ivpr&`b 0
'

ivpr



0*02:(,,-*-.:(,-

0*//:(,-.+/ ivpr&`b 0
'

ivpr



-,*.5%.1*5,%-,,*,,%<WHUO#&XLJ'
"h{tAWTJLXX#,

5*40:),.

"h{t
 #&%'AWTJLXX#,AWTJLXX#,

A6#96D6#&/0'
AWTJLXX#,CA6B820F===M]
"h{t



"h{t
 #&%'AWTJLXX#,#

/*./:(,,/*3-:(,,   
a
a

lrda

,*,2%

A6#96D6#&-.'AWTJLXX#,#   
a
a



-*52%

J^JQLGJTZSYX TUGXY[G\HPY.*.1%,*,,:(,,-*.-:(,-.*.1:(,--*24:(,,

2*/1:(,-

1*2-:),-

   =+@dgml      
a
a
a
a
a
a

,*,,%

3*/2%

-/*22%-*,.%/4*1/%

,*/0%

tepo
Z@Aivpra

0*40:(,,-*2/:(,,

tepo
Z@Aivpr
a

.*50%

,*55%

3*24:),-

`bb
a

,*03%

5*31:(,,

1*5/:(,,,*,,:(,,

~kln  
ivpr _!c ~kln_

!c

1*5.%

/*2,%

,*,,%

2*1/%

TUGXY[G\HPYGS TUGXY[G\HPYGX IWHSJOGPSXYWZ TUGXY[G\HPYGL TUGXY[G\HPYGX PSXYWZJYPTSGJJXLG\PSKT\G QTHKGXYTWLGPS IWHSJOGJTRUJXLG\PSKT\G YWHUG9??EGLZGJTRUG\HPY
MQGJTRUG\HPY
J^JQLGJTZSYXJGULSK]RPXXGL]
JYPTSX
]
]RPXX
TZSYX
LRUY^
XYWZJYPTSX
G\HPY
LRUY^GXUGMZQQ
RPXX

0*-1:(--

5*13:(-,,*,,:(,,J^JQLGJTZSYXPSOGJRPYGNUWG.\WPYL

0*-1:(---*05:(-,.*45:(--   
aalrda,LSKTU.*04:(,4-*--:(-,

MQTHYPSNGPSXYW PRUKLU.GPSXYZJYPTSXWZJYPTSX
4*05:(-,/*00:(-,/*,4:(-,

4*0,:(,5/*2.:(--/LSKTU.LSKTU-LSKTU-*0/:(-,.*33:(-,.*02:(-,-*20:(--J^JQLGJTZSYX
0*-2:(--.*-2:(-,-*,1:(---*0-:(,5-*2,:(--

5*/0:(,5.*/2:(,/

0*-2:(--UWLMLYJOGPSXY X]GRPXXGJTZS X]GRPXXGJTZS TUGWGPZGWLVGR PSXYWZJYPTSGMQK[UGJTZSYGKK[UGJTZSYGUM
WZJYPTSX
YGUM
YGKR
PGNT
T\GJTZSYX
R/*5,:(-,-*-/:(,54*42:(,21*,.:(,5/*22:(--

4*53:(,43*//:(,2<WHUO#&XLJ'.*02:),.

4*,5:),-

5*.3:),-   
a
a

lrda

,*,2%

-*43%

.*-0%

   =+@dgml      
a
a
a
a
a
a

,*,,:(,,

0*14:(,,

4*1-:(,,-*,2:(,,

0*,,:(,,/*1/:),-

   =+@dgml      
a
a
a
a
a
a

,*,,%

-,*14%-5*2/%

.*00%

5*./%

,*4.%

tepo
Z@Aivpra/*,1:),--*,/:(,,

tepo
Z@Aivpr
a

,*3,%

.*/3%

`bb
a

0*40:),-

`bb
a

-*-.%

~kln  
ivpr _!c ~kln_!c

/*03:(,,.*--:(,,,*,,:(,,

~kln  
ivpr _!c ~kln_

!c

4*,-%

0*42%

,*,,%

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~ #9D@A9D@A  2KLB7G@:H58=< 2KLBEFI58=< 2!"9D@A13
133 2  2'(58=< 2 J;<>04
/()9D@A 2
<A5 2

J;<>04C6@? 2 KLB7G@:H58=< 2 KLBEFI58=< 2.+*!&!%&!

$&!

#&!

!&!

&!

%& $& #& !& &)-,*+..)-,*+..

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~ 
#9D@A9D@A  2KLB7G@:H58=< 2KLBEFI58=< 2!"9D@A13
133 2  2'(58=< 2 J;<>04
/()9D@A 2
<A5 2

J;<>04C6@? 2 KLB7G@:H58=< 2 KLBEFI58=< 2.+*!&!%&!

$&!

#&!

!&!

&!

%& $& #& !& &)-,*+..)-,*+..

All Rights Reserved, Copyright © ISLiM, RIKEN 2011~*************************************************************************************
Application - performance monitors*************************************************************************************

Befor

Time(s) InstructionsMIPS
MFLOPS
Cover(%) Start
End------------------------------------------------------------------------------------------------------Elapsed
200.1027 1.554858e+127770.3003
3038.8860
87.1965--- Application------------------------------------------------------------------------------------------------------Elapsed
33.9091 309480667484
9126.7686
4700.0615
96.2603 1377 1849 hifu3d.hifu3d_cal_viscous_
Elapsed21.9806 185927060668
8458.6934
1721.6554
99.2490 2920 3020 hifu3d.hifu3d_cal_t_
Elapsed18.5975 189785811427 10204.9199
3616.7947
98.4694 2057 2157 hifu3d.hifu3d_cal_v_
Elapsed18.3742 186768878699 10164.7500
3586.7241
98.3019 2263 2363 hifu3d.hifu3d_cal_w_
Elapsed28.4768 96586898288
3391.7786
747.4074
60.7027 1158 1265 hifu3d.hifu3d_update_mixture_properties_
Elapsed12.8303 139624795714 10882.4238
5393.4106
99.1439 1856 1956 hifu3d.hifu3d_cal_u_
Elapsed12.7157 104721324854
8235.5811
3514.5505
97.0267 2766 2913 hifu3d.hifu3d_cal_grad_t_
Elapsed9.3909 92206173601
9818.6475
5363.4482
99.0825 2476 2547 hifu3d.hifu3d_cal_p_
Elapsed16.6106 54691619543
3292.5637
810.1867
55.2351--- jwe_gpwd
Elapsed7.5414 46972190323
6228.5947
3524.1199
97.5748 3158 3244 hifu3d.hifu3d_cal_grad_phi_
Time(s) L2 miss(%) mTLB-is(%) mTLB-op(%)
Cover(%) Start
End------------------------------------------------------------------------------------------------------Elapsed
200.1027
0.30720.00000.000087.1965--- Application------------------------------------------------------------------------------------------------------Elapsed
33.9091
0.40900.00000.000096.2603 1377 1849 hifu3d.hifu3d_cal_viscous_
Elapsed21.9806
0.19600.00000.000099.2490 2920 3020 hifu3d.hifu3d_cal_t_
Elapsed18.59750.18110.00000.000098.4694 2057 2157 hifu3d.hifu3d_cal_v_
Elapsed18.37420.39530.00000.000098.3019 2263 2363 hifu3d.hifu3d_cal_w_
Elapsed28.4768
0.14130.00000.000060.7027 1158 1265 hifu3d.hifu3d_update_mixture_properties_
Elapsed12.83030.24570.00000.000099.1439 1856 1956 hifu3d.hifu3d_cal_u_
Elapsed12.7157
0.43200.00000.000097.0267 2766 2913 hifu3d.hifu3d_cal_grad_t_
Elapsed9.39090.38840.00000.000099.0825 2476 2547 hifu3d.hifu3d_cal_p_
Elapsed16.6106
0.05670.00000.000055.2351--- jwe_gpwd
Elapsed7.5414
0.59140.00000.000097.5748 3158 3244 hifu3d.hifu3d_cal_grad_phi_
Time(s) InstructionsMIPS
MFLOPS
Cover(%) Start
End*************************************************************************************------------------------------------------------------------------------------------------------------Elapsed
167.8631 1.279574e+127622.7266
3379.7988
84.7377--- Application------------------------------------------------------------------------------------------------------Elapsed
32.9582 303491811692
9208.3789
5005.4312
96.4877 1407 1927 hifu3d.hifu3d_cal_viscous_
Elapsed28.9257 98329499601
3399.3813
760.7977
60.1695 1183 1295 hifu3d.hifu3d_update_mixture_properties_
Elapsed12.4409 103532620467
8321.9326
3587.9478
96.9842 2866 3013 hifu3d.hifu3d_cal_grad_t_
Elapsed10.5489 106238752306 10071.0410
4376.0215
98.6138 2356 2463 hifu3d.hifu3d_cal_w_
Elapsed10.0296 104768611662 10445.9932
4825.9990
99.2187 1934 2041 hifu3d.hifu3d_cal_u_
Elapsed10.1546 106700586942 10507.6123
4580.7646
98.0174 2143 2250 hifu3d.hifu3d_cal_v_
Elapsed9.4881 92693027995
9769.4053
5309.7554
98.7432 2576 2647 hifu3d.hifu3d_cal_p_
Elapsed9.5217 114825479098 12059.3682
5313.0464
98.2709 3020 3166 hifu3d.hifu3d_cal_t_
Elapsed16.1134 54403140381
3376.2605
844.6558
55.1470--- jwe_gpwd
Elapsed7.5706 49470042876
6534.5195
3790.9597
97.8672 3304 3390 hifu3d.hifu3d_cal_grad_phi_

AfterTime(s) L2 miss(%) mTLB-is(%) mTLB-op(%)
Cover(%) Start
End------------------------------------------------------------------------------------------------------Elapsed
167.8631
0.34420.00000.000084.7377--- Application------------------------------------------------------------------------------------------------------Elapsed
32.9582
0.41670.00000.000096.4877 1407 1927 hifu3d.hifu3d_cal_viscous_
Elapsed28.9257
0.14200.00000.000060.1695 1183 1295 hifu3d.hifu3d_update_mixture_properties_
Elapsed12.44090.43220.00000.000096.9842 2866 3013 hifu3d.hifu3d_cal_grad_t_
Elapsed10.54890.40720.00000.000098.6138 2356 2463 hifu3d.hifu3d_cal_w_
Elapsed10.02960.28020.00000.000099.2187 1934 2041 hifu3d.hifu3d_cal_u_
Elapsed10.1546
0.26900.00000.000098.0174 2143 2250 hifu3d.hifu3d_cal_v_
Elapsed9.48810.38890.00000.000098.7432 2576 2647 hifu3d.hifu3d_cal_p_
Elapsed9.52170.34610.00000.000098.2709 3020 3166 hifu3d.hifu3d_cal_t_
Elapsed16.1134
0.05490.00000.000055.1470--- jwe_gpwd
All Rights Reserved, Copyright © ISLiM, RIKEN 2011~
Elapsed7.5706
0.59150.00000.000097.8672 3304 3390 hifu3d.hifu3d_cal_grad_phi_