fabric: Move endpoint and domain caps into fabric.h

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_EP_MAX enum value

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Add PSM protocol to fi_proto enum

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Convert some named enum to anonymous

Enums that can be extended using vendor specific values
should be anonyous.  They are only used to define the values.

Reduce the size of fields that reference these enums from
uint64 to uint32, with the upper bit used to determine a
provider specific value.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Rename fi_info:type to ep_type

Clarify the type.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Introduce 'mode' bitfield into fi_info

There are some features/flags/capabilities/not-sure-what-to-call-this
that a provider needs to expose to an app in order for the app to
get the best performance out of the provider. BUT, the app usually
will NOT want this "feature", so would never request it. The
domain capability FI_LOCAL_MR is an example. Apps don't want to
deal with memory registration, but may need to in order to get
decent performance or work with a provider at all (such as a verbs
provider) it may need to.

Add a mode field to fi_info.  Mode is defined such that an app
will request what modes of operation that it can support.  The
provider will clear mode bits that are not required, and keep
set only those that are.  An app must check the resulting mode
bits that are set and react accordingly.

If an app does not set a mode bit that is required by a provider,
that provider will return FI_ENODATA to the fi_getinfo request.

Note that FI_WRITE_COHERENT moves from a domain capability to a
mode bit.  This results in the meaning of FI_WRITE_COHERENT inverting.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Define FI_MSG_PREFIX mode

Introduce a new operation mode by which the application
provides the buffer space needed to implement the underlying
protocol.  This is an optimization for the provider, which
can avoid it needing to use separate protocol buffers.

The advantage is that a single SGE will contain both the
protocol headers and the application data.  This reduces
the number of buffers that HW must access in order to transmit
the data and places the received headers and data together.

The disadvantage to an app is that it must allocate the
buffer space when sending or receive messages.  FI_MSG_PREFIX
does not apply to RMA data.  Applications that do not want
to support this mode can disable this optimization, which
will require that the provider allocate its own protocol
headers.

Patch is derived from work by Reese Faucette <rfaucett@cisco.com>

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Merge ep_cap and domain_cap

Replace the separate capability fields with a single field.  This
field applies to general output, which includes the fabric, domain,
endpoint, and transmit/receive contexts.

This simplifies things for the app, which no longer needs to know
if a capability is an endpoint or domain cap.  It also allows a
single capability to apply to multiple objects collectively.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_PASSIVE

FI_PASSIVE implies FI_SOURCE.  Since FI_SOURCE is needed for
unconnected endpoints, replace the use of FI_PASSIVE with
FI_SOURCE.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Move FI_USER_MR_KEY from cap to mode

User selectable MR keys is a desirable feature that
most applications using RMA will want.  Move it from
being a capability to its inverse being a mode --
FI_PROV_MR_KEY.  Apps can indicate that they support
this resriction by setting the mode bit.  Otherwise,
MR keys will be selected by the app, which improves
scalability.

Remove FI_MR_KEY flag from the memory registration calls as
part of this change.  Support for provider versus user requested
keys will now be determined based on the selected mode when the
domain is opened.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Fix typo in fi_rx_context

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Merge pull request #179 from shefty/master

API updates to simplify usage