prov/psm: support fi_cq_write

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: initialize cq wait object

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: define optimized version of tagged send/recv

Remove branches when the default EP op_flags is 0 and when the AV
type is known.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: use free list to optimize CQ event allocation

A set of macros are defined for general free list management:

	PSMX_FREE_LIST_INIT(head,tail,type,count)
	PSMX_FREE_LIST_GET(head,tail,type,item)
	PSMX_FREE_LIST_PUT(head,tail,type,item)
	PSMX_FREE_LIST_FINALIZE(head,tail,type)

The free list can be initialized with a certain size and is
expanded automatically when needed. Items are zero'ed when put
back to the list.

As the first example, CQ event allocation is moved from calling
calloc directly to allocating from the free list. The same usage
pattern can be applied to other resource allocation such as AM
request.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: add event suppression to tagged send/recv optimization

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: add debug output for tagged ops selection

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
Merge pull request #176 from j-xiong/master

prov/psm: cq_write, wait obj, optimized tsend/trecv and cq event allocation
fabric: Move endpoint and domain caps into fabric.h

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_EP_MAX enum value

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Add PSM protocol to fi_proto enum

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Convert some named enum to anonymous

Enums that can be extended using vendor specific values
should be anonyous.  They are only used to define the values.

Reduce the size of fields that reference these enums from
uint64 to uint32, with the upper bit used to determine a
provider specific value.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Rename fi_info:type to ep_type

Clarify the type.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Introduce 'mode' bitfield into fi_info

There are some features/flags/capabilities/not-sure-what-to-call-this
that a provider needs to expose to an app in order for the app to
get the best performance out of the provider. BUT, the app usually
will NOT want this "feature", so would never request it. The
domain capability FI_LOCAL_MR is an example. Apps don't want to
deal with memory registration, but may need to in order to get
decent performance or work with a provider at all (such as a verbs
provider) it may need to.

Add a mode field to fi_info.  Mode is defined such that an app
will request what modes of operation that it can support.  The
provider will clear mode bits that are not required, and keep
set only those that are.  An app must check the resulting mode
bits that are set and react accordingly.

If an app does not set a mode bit that is required by a provider,
that provider will return FI_ENODATA to the fi_getinfo request.

Note that FI_WRITE_COHERENT moves from a domain capability to a
mode bit.  This results in the meaning of FI_WRITE_COHERENT inverting.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Define FI_MSG_PREFIX mode

Introduce a new operation mode by which the application
provides the buffer space needed to implement the underlying
protocol.  This is an optimization for the provider, which
can avoid it needing to use separate protocol buffers.

The advantage is that a single SGE will contain both the
protocol headers and the application data.  This reduces
the number of buffers that HW must access in order to transmit
the data and places the received headers and data together.

The disadvantage to an app is that it must allocate the
buffer space when sending or receive messages.  FI_MSG_PREFIX
does not apply to RMA data.  Applications that do not want
to support this mode can disable this optimization, which
will require that the provider allocate its own protocol
headers.

Patch is derived from work by Reese Faucette <rfaucett@cisco.com>

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Merge ep_cap and domain_cap

Replace the separate capability fields with a single field.  This
field applies to general output, which includes the fabric, domain,
endpoint, and transmit/receive contexts.

This simplifies things for the app, which no longer needs to know
if a capability is an endpoint or domain cap.  It also allows a
single capability to apply to multiple objects collectively.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_PASSIVE

FI_PASSIVE implies FI_SOURCE.  Since FI_SOURCE is needed for
unconnected endpoints, replace the use of FI_PASSIVE with
FI_SOURCE.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Move FI_USER_MR_KEY from cap to mode

User selectable MR keys is a desirable feature that
most applications using RMA will want.  Move it from
being a capability to its inverse being a mode --
FI_PROV_MR_KEY.  Apps can indicate that they support
this resriction by setting the mode bit.  Otherwise,
MR keys will be selected by the app, which improves
scalability.

Remove FI_MR_KEY flag from the memory registration calls as
part of this change.  Support for provider versus user requested
keys will now be determined based on the selected mode when the
domain is opened.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Fix typo in fi_rx_context

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Merge pull request #179 from shefty/master

API updates to simplify usage
Use symbol visibility attribute to define exported symbols.

To be used in conjunction with the linker script.

Should allow for better optimization, see:

http://www.akkadia.org/drepper/dsohowto.pdf

Section 2.1