Clean up dl-open features:

Add --with-dlopen=[yes,no] option to compile without loadable provider support.
Selecting no also disables the check for -ldl.

Rename FI_EXTPATH to FI_EXTDIR for consistency with variable names and configure options.

Remove unused includes in fabric.c

Re-include common.c and enosys.c in dl-loadable providers, these are needed.

Address issues from review:

Remove the ./configure-able extdir: this isn't really needed, if one wants to not
scandir a directory on libfabric load do:

1) build with: --enable-dlopen=no
2) set FI_EXTDIR=/dev/null

Not addressed in this patch:

common sources. This is a seperate issue, in the meantime
re-add these so dl-loadable providers are not broken.

More error-checking in configure script for mistakes (ie dlopen=no and providers
built with =dl). I have more ideas for this and will address in a seperate patch.
Merge pull request #161 from pmmccorm/master

Clean up dl-open features:
Merge pull request #167 from shefty/master

prov/psm: Fix PSM build -- missed EQ rename ro CQ
Merge pull request #168 from shefty/master

prov/verbs: Return -FI_ENOMEM if fi_allocinfo fails
prov/psm: support fi_cq_write

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: initialize cq wait object

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: define optimized version of tagged send/recv

Remove branches when the default EP op_flags is 0 and when the AV
type is known.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: use free list to optimize CQ event allocation

A set of macros are defined for general free list management:

	PSMX_FREE_LIST_INIT(head,tail,type,count)
	PSMX_FREE_LIST_GET(head,tail,type,item)
	PSMX_FREE_LIST_PUT(head,tail,type,item)
	PSMX_FREE_LIST_FINALIZE(head,tail,type)

The free list can be initialized with a certain size and is
expanded automatically when needed. Items are zero'ed when put
back to the list.

As the first example, CQ event allocation is moved from calling
calloc directly to allocating from the free list. The same usage
pattern can be applied to other resource allocation such as AM
request.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: add event suppression to tagged send/recv optimization

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
prov/psm: add debug output for tagged ops selection

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
Merge pull request #176 from j-xiong/master

prov/psm: cq_write, wait obj, optimized tsend/trecv and cq event allocation
fabric: Move endpoint and domain caps into fabric.h

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_EP_MAX enum value

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Add PSM protocol to fi_proto enum

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Convert some named enum to anonymous

Enums that can be extended using vendor specific values
should be anonyous.  They are only used to define the values.

Reduce the size of fields that reference these enums from
uint64 to uint32, with the upper bit used to determine a
provider specific value.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Rename fi_info:type to ep_type

Clarify the type.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Introduce 'mode' bitfield into fi_info

There are some features/flags/capabilities/not-sure-what-to-call-this
that a provider needs to expose to an app in order for the app to
get the best performance out of the provider. BUT, the app usually
will NOT want this "feature", so would never request it. The
domain capability FI_LOCAL_MR is an example. Apps don't want to
deal with memory registration, but may need to in order to get
decent performance or work with a provider at all (such as a verbs
provider) it may need to.

Add a mode field to fi_info.  Mode is defined such that an app
will request what modes of operation that it can support.  The
provider will clear mode bits that are not required, and keep
set only those that are.  An app must check the resulting mode
bits that are set and react accordingly.

If an app does not set a mode bit that is required by a provider,
that provider will return FI_ENODATA to the fi_getinfo request.

Note that FI_WRITE_COHERENT moves from a domain capability to a
mode bit.  This results in the meaning of FI_WRITE_COHERENT inverting.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Define FI_MSG_PREFIX mode

Introduce a new operation mode by which the application
provides the buffer space needed to implement the underlying
protocol.  This is an optimization for the provider, which
can avoid it needing to use separate protocol buffers.

The advantage is that a single SGE will contain both the
protocol headers and the application data.  This reduces
the number of buffers that HW must access in order to transmit
the data and places the received headers and data together.

The disadvantage to an app is that it must allocate the
buffer space when sending or receive messages.  FI_MSG_PREFIX
does not apply to RMA data.  Applications that do not want
to support this mode can disable this optimization, which
will require that the provider allocate its own protocol
headers.

Patch is derived from work by Reese Faucette <rfaucett@cisco.com>

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Merge ep_cap and domain_cap

Replace the separate capability fields with a single field.  This
field applies to general output, which includes the fabric, domain,
endpoint, and transmit/receive contexts.

This simplifies things for the app, which no longer needs to know
if a capability is an endpoint or domain cap.  It also allows a
single capability to apply to multiple objects collectively.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Remove FI_PASSIVE

FI_PASSIVE implies FI_SOURCE.  Since FI_SOURCE is needed for
unconnected endpoints, replace the use of FI_PASSIVE with
FI_SOURCE.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>