Timing framework improvement

@nkogteva please, review.
On Dec 5, 2014, at 1:57 AM, Artem Polyakov <notifications@github.com> wrote:

> 	• The idea of OPAL_TIMING_REPORT into a file is to have all timing events from all processes in one place and isolated from other output.

(my bias: I actually don't have any religious opinion that opal_output() *must* be used I just want to make sure you're not re-inventing the wheel when some other part of OMPI infrastructure would work ok... with that in mind, let's look at each of your analysis points...)

Ok.  This means that you send all the data to a single process and have that one process output the file, right?  (i.e., there's no other way to ensure that all data from all processes ends up in a single file without losing any of the data)

In that case, opal_output() should still be fine.

> Since timing framework provides global time synchronisation feature (if run with mpirun_prof) we can later sort events by their timestamps and see in what sequence and with what delays they appear on different nodes/processes.
> 
> 	• If I use opal_output service to output to stdout everything works perfectly and output is merged. Line ordering is not an issue as we can sort them later. The only problem here is to separate timing data from all other output. This can be done using known string prefix that is common for all timing lines.

Sure, but keep in mind that you do not have to opal_output(0, ...).  You can open your own stream (which won't be 0) and then use that.  Hence, it doesn't have to go to stdout.  Or it could go to stdout *and* a file.  Or ...<whatever you want>...

> 	• If I use opal_output service to output into a file the problem with output separation goes away, however:
> a. I can't create a file at arbitrary location. The placement is application-wide and setted at ORTE level at ess initialization with opal_output_set_output_file_info.

Hmm.  This sounds like you're relying on the filesystem to combine the data from all processes into a single file.  I think that this is a losing proposition -- you're going to find cases where multiple processes write to the (network-filesystem hosted) file simultaneously, and therefore the data from some process(es) get lost/overwritten.

I don't honestly remember if opal_output allows you to open a file in arbitrary locations, or whether they're always under the session directory...  But even if they're in the session directory, is that a problem?  (especially if you take the approach that all processes send data to a single process and a single process writes out the data -- then a process-wide filesystem location is irrelevant)

> b. I can't merge the output from different processes toghether at runtime. To acheive the initial goal I'll need a distributed postprocessing that will gather outputs from all processes of the application on all nodes.

Yes, I think you're going to need this -- regardless of whether you use opal_output or not.

> 	• Current implementation that rely on "append" feature and fprintf may not work on all filesystems. If I recall correctly I've seen that some lines was disappearing on NFS. So I need an advice here: what is the easiest way to collect output from all nodes in one file without data loss.
> a. The easiest way that I see is to write to stdout (see p. 2) and grep after by a prefix.
> b. Other option can be to forward all timing info to the mpirun and do the centralized output. The problem is that we work at OPAL layer that has no communication facilities.

Mmm.  Yes.  This is a problem (you're down in OPAL and don't have communication facilities available).

I think there are 3 general options:

1. output to stdout/stderr with a specific prefix that you can easily grep for (i.e., your option a, above).

2. have every process write to a unique file name with high-resolution timestamps.

3. some kind of MPI_Gather-like operation at the end where all procs send their timing data to some master process (e.g., vpid=0 or mpirun or something) and that one process does the output.  But you're down at OPAL, so...  I guess you'd have to have OPAL just generate the data, and then have another agent up in ORTE (or OMPI?) do this synchronization.

Keep in mind that all 3 options have ordering issues -- none of them will *guarantee* to output data from each process in the correct order.

> I'd prefer something that would be close to fopen(fname, "a").

I think only option #2 is close to that.

-- 
Jeff Squyres
jsquyres@cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/