I tried the Sun Studio Performance Analyzer and it was able to profile
to application but not the openMPI source code.  The source code was
empty in Sun Studio Performance Analyzer.  I tried but i was not able to
get the profiling information of openMPI. 

So i installed Vampir trace and i was able to get otf while suggested i
got the profiling information.  But i fail to understand what i should
do now.  1.  Am i suppose to download additional information for trace
information visualization?  Is the additional software this https://www.ssl-id.net/www.vampir.eu/index.html
2.  If not how can i visualize the trace information i got vampir-trace. 

Sorry for bugging everyone so much about this.  But i spend a lot of
time and this alone and i am not getting output. 

It's probably fine to bug people about some of this.  OMPI would
benefit from having a tool ecosystem around it.  There's VampirTrace
and PERUSE instrumentation and stuff, but some more activity/attention
in this area would be better. 

I don't know that VampirTrace will give what you're looking for.  You
seem to want to profile the internals of OMPI.  VT basically just
instruments entry into and exit out of MPI.  In contrast, PERUSE
instruments MPI internals. 

Sun Studio Performance Analyzer should also work.  I know I've used it
to profile both MPI apps and the internals of OMPI. 

One of the problems... I mean, one of the *features* of Sun
Performance Analyzer is that it *HIDES* the internals of the MPI
library.  There is a concept of user and expert models and stuff.  Most
users just want to see their program the way they wrote it (whether for
Java, OpenMP, MPI, etc.).  So, Performance Analyzer hides the "black
box" stuff (internals of Java, OpenMP, MPI, etc.).  But, *you* want
"expert" capabilities.  You want to see what's under the hood.  So,
after you have collected data and have started the Analyzer GUI, choose
"View" -> "Show/Hide Functions..." -> "Show All".  Maybe there
are other things you're encountering, but for me that changes MPI calls
from being black boxes to exposing where OMPI is spending its time:
PML functions, BTL functions, etc. 

To get source code information, you also need to build OMPI with -g
turned on.  That will include debugging information.  With some
compilers, turning -g on might turn off optimizations... I just don't
know.  With Sun Studio compilers, -g will not change your optimizations
-- it will only add debugging/symbolic information, compiler commentary
on optimizations, etc. 

If you want to ask Nik or me other questions, feel free.  I'll send you
Nik's home phone number! : ^)