Fixes #3577: Move opal_datatype assert fix to v1.7

---svn-pre-commit-ignore-below---

r28398 [[BR]]
As initially reported by Eric Chamberland in
http://www.open-mpi.org/community/lists/users/2013/04/21689.php, the
assert in opal_datatype_is_contiguous_memory_layout() is not always
correct -- he supplied a test case where it was not valid,
essentially:

 1. Call MPI_Type_create_indexed_block(0, ..., &newtype) and commit newtype
 1. Call MPI_Type_create_resized(newtype, 0, nonzero_value, &resized) and commit resized
 1. Call MPI_File_set_view with resized

This will eventually call opal_datatype_is_contiguous_memory_layout(),
and the assert will fail.  After some consultation with George, it was
determined that the assert() is basically good, but it needs to also
check for (count != 0).
Fixes #3576: Cleanup stale filem code
Fixes #3572: Move r28381 to v1.7 branch

---svn-pre-commit-ignore-below---

r28381 [[BR]]
Fix the @#$!@# ompi_info --version option. Via long chat with Jeff, simplify this option a lot by dumping all the silly suboptions it covered. Instead, just provide the very basic "ompi_info -V|--version" like all other tools do

cmr:v1.7.2


r28382 [[BR]]
Crud - missed this file, needs to go with prior commit, will add to cmr
Fixes #3402: CMR: Move r27601 to v1.7 (enforce minimum flex version)

---svn-pre-commit-ignore-below---

r27601 [[BR]]
enforce minimum flex version in autogen.pl
Fixes #3394: Move r27571 to v1.7 branch (cleanup flex parsing properly)

---svn-pre-commit-ignore-below---

r27571 [[BR]]
Remove the finish_parsing function from various .l files. The function is incomplete (doesn't clean up the lex state) and should be replaced by *_yylex_destroy which correctly cleans up the state.

Checked with the flex 2.5.35. Verified with valgrind that this fixes several "still reachable" leaks.

cmr:v1.7


r27577 [[BR]]
instead of relying on yyterminate to clean up the lex context call the destroy functions directly (after closing the file)

r27580 [[BR]]
add prototypes for lex destroy functions

r27647 [[BR]]
Per discussion on the OMPI teleconf this morning:

 * Remove the check for a specific m4 version.  We don't require it.
   Autoconf requires a specific version of m4, but we don't.
 * Reduce flex min version to 2.5.4
 * Update a bunch of language in HACKING, including:
   * General text updates
   * OMPI doesn't require a specific m4; Autoconf does
   * Soften language about flex version
   * Reserve the right to upgrade min flex version to 2.5.35 someday
Revert r24815 - the trunk no longer checks minimum flex version, and the test fails on Mac anyway
Sigh - revert r24816 as well. The flex patch just isn't complete - will request it be combined and tested with CMR #3583
Fixes #3580: Move openib "no BTLs" fix to v1.7

---svn-pre-commit-ignore-below---

r28402 [[BR]]
If there are no BTLs, do ''not'' actually shut down the fd listener,
because a) it may still be needed to shut down the CPCs, and b) it
will be shut down during component_close().
Fixes #3353: Move r27451 to v1.7 branch (remove deprecated stuff and cmd_line processor changes)

RHC: Note that I still had to hand-edit orterun.c to fix one incorrect cmd line declaration

---svn-pre-commit-ignore-below---

r27437 [[BR]]
remove unused, deprecated function opal_cmd_line_make_opt

r27451 [[BR]]
MCA: remove deprecated mca_base_param functions (mca_base_param_register_int, mca_base_param_register_string, mca_base_param_environ_variable). Remove all uses of deprecated functions.
cmr:v1.7
Fixes #3583: CMR: Move more flex fixes to 1.7

---svn-pre-commit-ignore-below---

r27571 [[BR]]
Remove the finish_parsing function from various .l files. The function is incomplete (doesn't clean up the lex state) and should be replaced by *_yylex_destroy which correctly cleans up the state.

Checked with the flex 2.5.35. Verified with valgrind that this fixes several "still reachable" leaks.

cmr:v1.7


r27574 [[BR]]
A prior commit apparently broke the trunk when something was inadvertently left behind - so remove a reference to a no-longer-existing function


r27577 [[BR]]
instead of relying on yyterminate to clean up the lex context call the destroy functions directly (after closing the file)

r27580 [[BR]]
add prototypes for lex destroy functions

r27647 [[BR]]
Per discussion on the OMPI teleconf this morning:

 * Remove the check for a specific m4 version.  We don't require it.
   Autoconf requires a specific version of m4, but we don't.
 * Reduce flex min version to 2.5.4
 * Update a bunch of language in HACKING, including:
   * General text updates
   * OMPI doesn't require a specific m4; Autoconf does
   * Soften language about flex version
   * Reserve the right to upgrade min flex version to 2.5.35 someday

r27657 [[BR]]
Re-add support for old flex (2.5.4a and earlier) while still cleaning up properly in new flex.
Fixes #3584: Move r28430 MRECV STATUS_IGNORE fix to v1.7.2

---svn-pre-commit-ignore-below---

r28430 [[BR]]
Don't assign the status if MPI_STATUS_IGNORE is passed in.  Thanks to
Lisandro Dalcin for finding the issue.
Fixes #3587: CMR: Bring useful opal_list_t macros to 1.7

---svn-pre-commit-ignore-below---

r27985 [[BR]]
add iterator macros for opal_list_t

r28029 [[BR]]
The opal_list_t destructor doesn't release the items on the list prior to destructing or releasing it. Provide two convenience macros for doing so.
Fixes #3595: Move r28455 to v1.7 branch (debugger attach fix)

---svn-pre-commit-ignore-below---

r28455 [[BR]]
Ensure that debugger connect can occur by getting the rml contact info updated before calling init_after_spawn
Fixes #3597: Move hwloc dependency_lib fix to v1.7

---svn-pre-commit-ignore-below---

r28457 [[BR]]
Fix issue raised by Stefan Friedel: remove an extraneous -L that is
added by hwloc's embedding so that it doesn't appear in
libhwloc_embedded.la (and therefore propogate all the way up to
libmpi.la). 

Committed upstream in hwloc SVN r5588.

r28458 [[BR]]
Update the list of post-1.5.2 r numbers from hwloc that we have
committed here.
Fixes #3592: btl_openib: change SRQ defaults

---svn-pre-commit-ignore-below---

r28440 [[BR]]
* btl_openib: changed default SRQ settings:
  * increase number of wqe to minimize number of RNRs
  * it is better to have high watermark and post relatively small number of wqes
  * increased TX queue size

r28441 [[BR]]
* correction of r28440
Fixes #3594: Move r28442 to v1.7 branch (coll sm fixes)

---svn-pre-commit-ignore-below---

r28442 [[BR]]
In coll_sm, we alloc a huge chunk of shared memory, divvy it into lots
of individual regions (each region is a multiple of page size in
length), and each process claims its own regions by binding it to its
local memory.  Each process would end up membining something like 16
individual regions in the overall shmem segment.

There were two errors in this code relating to the memory affinity
pinning.  Some combination of these two errors would lead to kernel
panics (!) on my RHEL 6.2 x86_64 machines when used with mmap'ed
shared memory (not posix or sysv shared memory, curiously enough):

1. The shared memory segment is initially divided into two regions:
control and data.  The control starts at the beginning of the shmem
segment, the data starts after that.  The data portion, unfortunately,
was ''not'' aligned to a page.  So all the multiple-of-page-size
regions that we divvy up were also not alined on page boundaries.  And
therefore all the regions we tried to membind were not on page
boundaries.

The solution was to ensure that the data portion started on a page
boundary.  Then all of the individual regions were on page boundaries,
too.

That being said, in my tests, Linux mbind() fails gracefully when the
address is not on a page boundary.  So I'm not sure how this worked at
all / led to a kernel panic...

2. There was some bad pointer math that resulted in membinding regions
larger than they should have been, resulting in region overlaps.
There were definitely overlaps between regions in the same process;
it's likely that there were overlaps between regions of multiple
processes, too -- I'm not sure (and don't care to figure out :-) ).

The solution was to fix the pointer math so that each region membinds
exactly only itself and no neighboring/overlapping regions.

cmr:v1.7.2:reviewer=samuel
Fixes #3280: Move LAMA to v1.7 branch (rollup)

---svn-pre-commit-ignore-below---

r27203 [[BR]]
Add some help messages to the RAS simulator for PEBKAC issues.

r27204 [[BR]]
Fixes, improvements, and enhancements to the opal_tree class (used by
the LAMA RMAPS component, to be committed shortly).

r27206 [[BR]]
= Overview =

First revision of the Locatation Aware Mapping Algorithm (LAMA) RMAPS
component.  This component is used to effect many different types of
regular of process/processor affinity patterns.  Although quite
flexible in the patterns that it provides, it is ''not'' a
fully-arbitrary, rankfile-like solution for process/processor
affinity.  

Inspiried by !BlueGene-like network specifications, LAMA has a core
algorithm that is quite good at specifying regular patterns in
multiple "dimensions" (where "dimensions" are expressed in terms of
different hardware elements: processor hardware threads, cores,
sockets, ...etc.).  The LAMA core algorithm is described here:

  http://www.open-mpi.org/papers/cluster-2011-lama/

= LAMA Usage Levels =

LAMA allows specifying affinity multiple different ways:

 1. None: Speciying no affinity options to mpirun results in exactly
    the same behavior as today: no affinity is used.
 1. Simple: Using the mpirun options "--bind-to <WIDTH>" and "--map-to
    <LEVEL>" to indicate how "wide" each process should be bound
    (i.e., bind to a processor core, or to a processor socket, etc.)
    and how to lay out the processes (i.e., round robin by cores,
    sockets, etc.).  
 1. Expert: Using four new MCA parameters to effect process mapping
    and binding to processors.  These options are a bit complex, and
    are not for the faint at heart, but offer a high degree of
    (regular pattern) flexibility (each of these are described more
    fully below): 
    * rmaps_lama_map: a sequence of characters describing how to lay
      out processes
    * rmaps_lama_bind: a sequence of characters describing the
      resources to bind to each process
    * rmaps_lama_mppr: a sequence of characters describing the maximum
      number of processes to allow per resource (i.e., a specific
      definition of "oversubscription")
    * rmaps_lama_ordering: once all processes are in place, how to
      order the ranks in MPI_COMM_WORLD

We anticipate that most users will utilize the "None" and "Simple"
levels of affinity, and they continue to work just as they do with the
v1.6 series and SVN trunk.  

The Expert level was designed for two purposes:

 1. To provide a precise definition for the "Simple" level (i.e.,
 every
    --bind-to/--map-by option in the "Simple" level has a
 corresponding
    precise specification in the "Expert" level)
 1. As modern computing platforms become more complex, we simply
    cannot predict what application developers will need in terms of
    processor affinity.  LAMA is an attempt to provide a highly
    flexible mechanism that allows applications to utilize a variety
    of complex, unique affinity patterns beyond the common "bind to
    core" and "bind to socket" patterns.

= LAMA Simple Level =

The "Simple" level is pretty much the same as what Open MPI has
offered for years.  It supports the same --bind-to and --map-by
options that Open MPI has supported for a while, but expands their
scope a bit.

Specifically, the following options are available for both --bind-to
and --map-by:

 * slot
 * hwthread
 * core
 * l1cache
 * l2cache
 * l3cache
 * socket
 * numa
 * board
 * node

= LAMA Expert Level =

The "Expert" level requires some explanation.  I'll repeat my
disclaimer here: the LAMA Expert level is not for the meek.  It is
flexible, but complex.  '''Most users won't need the Expert level.'''

LAMA works in three phases: mapping, binding, and ordering.  Each is
described below.

== Expert: Mapping ==

Processes are paired with sets of resources.  For example, each
process may be paired with a single processor core.  Or each process
may be paired with an entire processor socket.  LAMA performs this
mapping, obeying the Max Processes Per Resource ("MPPR", pronounced
"mipper") limits.  More on MPPR, below.

Mapping can be performed across multiple hardware levels:

 * h: Hardware thread
 * c: Processor core
 * s: Processor socket
 * L1: L1 cache
 * L2: L2 cache
 * L3: L3 cache
 * N: NUMA node
 * b: Processor board
 * n: Server node

If the act of mapping is that of pairing MPI processes to the
resources that have been allocated to a job, one can easily imagine
looping through all the resources and assigning processes to them.

But to effect different process process layout patterns across those
resources, one may want to loop over those resources ''in a different
order.''  That is, if the above-mentioned nine hardware resources
(hardware thread, processor core, etc.) can be thought of as an
nine-dimensional space, you can imagine nine nested loops to traverse
all of them.  And you can imagine that changing the order of nesting
would change the traversal pattern.

LAMA accepts a sequence of tokens representing the above-mentioned
nine hardware resources to specify the order of looping when mapping
resources to processes.

For example, consider a "simple" traversal: csL1L2L3Nbnh.  Reading
that sequence of letters from left-to-right, it specifies mapping by
processor core, processor socket, L1 cache, L2 cache, L3 cache, NUMA
node, processor board, server node, and finally hardware thread.

Wait... what?  That string specifies resources from "smallest" to
"largest" -- with the exception of hardware threads.  Why are they
tacked on to the end?

In short, this string of letters means "map by round robin by core" --
(indeed, it exactly corresponds to the Simple level "--map-by core").
Specifically, LAMA traverses the string from left-to-right and maps
processes to all the resources indicated by that token (e.g., "c" for
processor core).  When there are no more resources indicated by that
token, it goes on to the next token.

Hence, in this case, LAMA will map the first process to the first
core, then it will map the second process to the second core, and so
on.

Once all the cores are exhausted, LAMA effectively ignores all the
other letters until "h" (because all the other resources are made up
of cores; when cores are exhausted, those resources are exhausted,
too).  

If there are still more processes to be mapped, LAMA will then
traverse all the hyperthreads -- meaning that the next process will be
mapped to the second hyperthread on the first core.  And the next
process will be mapped to the second hyperthread on the second core.
And so on.

Keep in mind that the cores involved may span many server nodes; we're
not just talking about the cores (etc.) in a single machine.

As another example, the sequence "sL1L2L3Nbnch" is exactly equivalent
to "--map-by socket" (i.e., LAMA maps the first process to the first
socket, the second process to the second socket, and so on).

The sequence of letter can be combined in many, many different ways to
produce many different regular mapping patterns.

=== Max Processes Per Resource (MPPR) ===

The MPPR is an expression that precisely defines the maximum number of
processes that can be mapped to any single resource.  In effect, it
defines the concept of "oversubscription."  Specifically, traditional
HPC wisdom is that "oversubscription" is when there is more than one
MPI process per processor core.

This conventional defintion is expressed in a MPPR string of "1:c"
(one process per core).  

But what if your MPI processes are multi-threaded, and they need
multiple processes per core?  You'd need a different description of
"oversubscription" in this case.  Perhaps you want to have one MPI
process per socket.  This would be expressed in a MPPR string of
"1:s".

The general form of an individual MPPR specification is an integer
follow by a colon, followed by any of the tokens from mapping can be
used in the MPPR specification.  For example "1:c" is pronounced "one
process per core."

Multiple MPPR specifications can be strung together into a
comma-delimited list, too.  All of these MPPR values and then taken
into account when mapping.  Here's some examples:

 * 1:c -- allow, at most, one process per processor core (i.e., don't
   schedule by hyperthread)
 * 1:s -- allow, at most, one process per processor socket (e.g., 
   that process may be multithreaded, or wants exclusive use of the
   socket's caches)
 * 1:s,2:n -- only allow one process per processor socket, but, at
   most, two processes per server node (e.g., if the two MPI processes
   will consume all the RAM on the server node, even if there are more
   processor cores available)

If mapping all processes to resources would exceed a MPPR limit, this
job is ruled to be oversubscribed.  If --oversubscribe was specified
on the mpirun command line, the job continues.  Otherwise, LAMA will
abort the job.

Additionally, if --oversubscribe is specified, LAMA will endlessly
cycle through the mapping token string untill all processes have been
mapped.

== Expert: Binding == 

Once processes have been paired with resources during the Mapping
stage, they are optionally bound to a (potentially different) set of
resources.  For example, processes may be mapped round robin by
processor socket, but bound to an individual processor core.

To be clear: if binding is not used, then mapping is effectively
reduced to "counting how many processes end up on each server node."
Without binding, there's no enforcement that a process will stay where
LAMA thinks it was placed.  

With binding, however, processes are bound to a set of hardware
threads.  The number of threads to which the process is bound is
sometimes referred to as the "binding width".  For example, if a
process is bound to all the hardware threads in a processor socket,
its "width" is the processor socket.

(note that we specifically do not say that the hardware threads are
sequential, even if they are all within a single resource such as a
processor core or socket.  BIOS ordering of hardware threads can be
wonky; so we only refer to "sets of hardware threads")

Bindings are expressed as an integer and a token from the mapping
string.  For example "1s" means "bind each process to one processor
socket" (there is no ":" in the binding string because the ":" is
pronounced as "per" when reading the MPPR string).

Note that it only makes sense to bind processes to a single resource
specification (unlike the MPPR specification, where multiple limits
can be specified).

== Expert: Ordering ==

Finally, processes are assigned a rank in MPI_COMM_WORLD.  LAMA
currently offers two ordering modes: sequential or natural:

 * Sequential: if you laid out all the hardware resources in a single
   line, and then overlaid all the MPI processes on top of them, they
   are ordered from 0 to (N-1) from left-to-right.
 * Natural: the ordering of ranks follows the mapping ordering.  For
   example, consider a server node with two processor sockets, each
   containing four cores.  The command line "mpirun -np 8 --bind-to
   core --map-by socket --order n a.out" would result in MCW ranks
   that look like this: [0 2 4 6] [1 3 5 7].

= Execution =

At this point, the job is fully mapped, optionally bound, and its
ranks in MPI_COMM_WORLD are ordered.  It now starts its execution.

= Final Notes =

Note that at this point, lama is not the default mapper.  It must be
activiated with "--mca rmaps lama".  We'll continue to do further
testing and comparitive analysis with the current set of ORTE mappers.

Also, note that the LAMA algorithm can handle heterogeneity between
hardware resources (e.g., an MPI job spanning server nodes with
differing numbers of processor sockets).  For lack of a longer
explanation (this commit message already long enough!), LAMA considers
each server node individually during mapping and binding.

See the LAMA paper for more details:
http://www.open-mpi.org/papers/cluster-2011-lama/

r27207 [[BR]]
 * Fix a few warnings in ompi_rb_tree
 * Add the get_key function to the opal_tree test

r27208 [[BR]]
Oops -- we want the priority to be low, not high (for now).  :-)

r27227 [[BR]]
As opal_hwloc_topo_data_t has to create a class instance in orte, its definition has to be exported. Otherwise, there will be unresolved variable error on Windows.
Fixes #3599: Better LAMA help messages

---svn-pre-commit-ignore-below---

r28470 [[BR]]
Proper show_help error messages for LAMA.

r28471 [[BR]]
Remove redundant opal_output() messages from the module; the called
functions will now show_help() their own error messages if something
goes wrong (per r28470).
Fixes #3281: Move verbs-based improvements to v1.7 (rollup)

---svn-pre-commit-ignore-below---

Custom Mercurial branch from the ticket based on the following r numbers:

r27212 [[BR]]
Per some discussions between LANL, Cisco, ORNAL, and Mellanox, move
some new common OpenFabrics functionality to ompi/mca/common/verbs.
Also move everything that was in ompi/mca/common/ofautils under
ompi/mca/common/verbs.  

 * Move ofautils -> verbs
 * Add new functionality in ompi/mca/common/verbs (see doxygen
 * comments in ompi/mca/common/verbs/common_verbs.h for details):
   * ompi_common_verbs_find_ibv_ports()
   * ompi_common_verbs_port_bw()
   * ompi_common_verbs_mtu()
   * '''If you're writing verbs-based code, you should be using this
     common functionality'''
 * Adapt openib BTL to use some trivial common functionality in
   common/verbs
 * Don't use "#ifdef OMPI_HAVE_RDMAOE",use 
   "#if defined(HAVE_IBV_LINK_LAYER_ETHERNET)"
 * Update the following to include/link against common/verbs
   * bcol/iboffload
   * sbgp/ibnet
   * btl/openib

r27229 [[BR]]
Always define OMPI_HAVE_RDMAOE; don't define it conditionally.

r27251 [[BR]]
OMPI_HAVE_IBV_LINK_LAYER does not exist.  Instead, check defined(HAVE_IBV_LINK_LAYER_ETHERNET).

r27332 [[BR]]
More updates to common verbs infrastructure:

 * Moved "check basics" sanity check from openib BTL to common/verbs
   (which also allows us to have openib ''not'' include
   <infiniband/driver.h>, which is a Very Good Thing)
 * Add new ompi_common_verbs_qp_test() function, which tests to see
   whether a device supports RC and/or UD QPs.  The openib BTL now
   uses this function to ensure that the device supports RC QPs.
 * Rename ompi_common_verbs_find_ibv_ports() to be
   ompi_common_verbs_find_ports() -- the "ibv" was redundant.
 * Re-work ompi_common_verbs_find_ports() to use
   ompi_common_verbs_qp_test() instead of testing for RC/UD QPs itself
 * Add bunches of opal_output_verbose() to the find_ports() routine
   (to help diagnosing connectivity problems -- imaging running with
   --mca btl_base_verbose 10; you'll see all the find_ports() test
   results)
 * Make ompi_common_verbs_qp_test() warn if devices/ports are supplied
   in the if_include/if_exclude strings that do not exists (quite
   similar to what the openib BTL does today).
 * Add ompi_common_verbs_mca_register() function, which registers
   common verbs MCA params.  It will also register MCA param synonyms
   for thse MCA params to upper-level components (e.g.,
   btl_<upper-level-component>_<the-mca-param>). 
   * common_verbs_warn_nonexistent_if: warn if
     if_include/if_exclude-specified devices or ports do not exist.  

r27393 [[BR]]
More common/verbs improvements:

 * Add OMPI_COMMON_VERBS_FLAGS_NOT_RC, which looks for a device	that
   does ''not'' support RC
 * Add ompi_common_verbs_find_max_inline(), and	remove that code from
   the openib BTL component

r27556 [[BR]]
Remove redundant common_verbs.h include.
Fixes #3557: Move openib init/finalize fix to v1.7

---svn-pre-commit-ignore-below---

r28317 [[BR]]
Be a little more conservative about initializing devices and modules
(i.e., ensure that more data items get zeroed out/set to NULL) so that
if something goes wrong during initialization, we don't try to clean
up something that isn't there (and segv).

The chance of this happening on the trunk is very low (and will also
be low once the verbs improvements are brought over to v1.7).  But it
can actually happen in the v1.6 branch (e.g., if no CPC is available,
we'll try to get the length of the endpoints list, but the endpoints
list is NULL).  

Hence, even though the real goal is to get this functionality over to
v1.6, I figured I'd commit to the trunk/CMR to v1.7 just to try to
keep commonality in the openib between all three where possible.