lattice/quda — Issue #46: Allow user to specify logical topology for multi-GPU communications
Author: rbabich
Created: 2012-01-31 03:19:51+00:00
State: open

At present, to properly run an application built with QUDA over QMP, it's necessary to specify "-geom Px Py Pz Pt" on the command-line. This is awkward in cases where the application has built-in logic to determine the best layout and is also incompatible with QDP/C, as summarized by James Osborn:

> One issue with interfacing multi-GPU to QDP at the moment will be that
> QDP isn't currently setting the logical topology. This was changed to
> support multi-lattice in QDP, since one might not want the same node
> mapping on each lattice, and QMP didn't have communicator support. Now
> that QMP does, I could create a new communicator for each lattice and
> set each's topology, but my concern is that MPI communicators could be
> expensive in memory and I don't want to rely on this. I'm planning to
> add some sort of light-weight communicators to QMP to address this.
>
> Another issue is that the QMP topology has it's own fixed mapping of the
> ranks to the logical topology, which may not be optimal. Right now QDP
> is using a different mapping which was a little better in some cases. I
> am also planning to allow the QMP mapping to be more flexible, but
> haven't gotten to this yet.
>
> Anyway, the main point is that it would be nice if QUDA didn't rely on
> the QMP topology, but instead allowed the user to pass in a function (or
> functions) that specified the rank->coords and coords->rank mappings.
> That would allow much greater flexibility for the applications using
> QUDA. Additionally, allowing a QMP communicator to be specified would be
> ever better. You said that some groups may want to port QMP and not use
> communicators, but it should be possible for those ports to still keep
> the same API (with the communicator structure) and just have it always
> be the same one (basically make QMP_comm_split always fail).

At this stage, I'd suggest not going so far as to rely on QMP communicators, which are still an "alpha" feature, but allowing the user to pass in mapping function seems like a nice solution. This would also add much-needed flexibility to the MPI code path, which currently assumes a simple lexicographical ordering when assigning logical grid coordinates to MPI ranks.

To summarize, I propose replacing this declaration:

void initCommsQuda(int argc, char **argv, const int *X, const int nDim);

with:

typedef int (*QudaCommsMap)(const int *x, void *fdata);
void initCommsQuda(const int *X, const int nDim, QudaCommsMap func, void *fdata);

Here `fdata` points to any auxiliary data required by the user-supplied mapping function `func()`. Passing `NULL` for `fdata` is perfectly valid. As an implementation detail, note that since we'll no longer be able to assume the existence of a QMP logical topology, we'll have to eliminate the use of "relative" sends and receives in face_qmp.cpp. This is a minor inconvenience but again quoting James Osborn:

> The relative sends were just a cached version of the calculation of (get my coords) -> (add 1 mod length) -> (get rank). They aren't necessary (and were never used by QDP), since you can just create the neighbor table yourself and use the regular send.

Comments?