prov/verbs: Remove mlx4 direct support

Remove unsupported mlx4 direct support from the verbs configure

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
prov/verbs: Actually enable building the verbs provider

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Add pretty-printer for enums, OR-able int members, and structures
from fi_info. The current list of printable structures is defined
in fabric.h - fi_pp_type.

A call like:

char *buf = fi_tostr(cur, FI_INFO);
printf("%s", buf);

Will produce a string containing (subject to change!):

fi_info:
  type: FI_EP_RDM
  ep_cap: [FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMICS, FI_BUFFERED_RECV, ]
  op_flags: []
  fi_addr_format:       FI_ADDR_PROTO
  connreq:      0
  source addr:  (null)
  dest addr:    (null)
  fi_ep_attr:
    protocol:   FI_PROTO_IB_RC
    max_msg_size:       -1
    inject_size:        64
    total_buffered_recv:        -1
    max_order_raw_size: 0
    max_order_war_size: 0
    max_order_waw_size: 0
    mem_tag_format:     aaaaaaaaaaaaaaaa
    msg_order: [FI_ORDER_SAS, ]
    tx_ctx_cnt: 0
    rx_ctx_cnt: 0
  fi_domain_attr:
    name:       psm
    caps: [FI_WRITE_COHERENT, FI_CONTEXT, ]
    threading:  FI_THREAD_PROGRESS
    control_progress:   FI_PROGRESS_MANUAL
    data_progress:      FI_PROGRESS_MANUAL
    mr_key_size:        0
    eq_data_size:       0
    ep_cnt:     0
    tx_ctx_cnt: 0
    rx_ctx_cnt: 0
    max_ep_tx_ctx:      0
    max_ep_rx_ctx:      0
  fi_fabric_attr:
    name:               psm
    prov_name:  PSM
    prov_version:       9

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Add TX/RX class types

Transmit and receive contexts use the same structure as fid_ep,
but the class types should be different, so that apps and
providers can distinguish between them, separate from directing
them to different entry points.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Move domain capability flag into fi_info

Applications will likely care about requesting specific domain
capabilities, similar to endpoint capabilities.  Rather than
having an app declare a domain attr structure and link to it
from fi_info, move the dom_cap field into fi_info, similar to
the ep_cap field.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
fabric: Define size of tx/rx queues

Define a mechanism that can be used to describe the size of
transmit and receive contexts.  Rather than defining that all
queues contain a fixed number of entries of a fixed size, we
describe a generic set of attributes that allow an app to
determine whether the next operation can be queued with the
tx/rx context.

This supports fixed sized queues, but also allows for queues
that accept a variable number of requests, based on the size
of the request.  There is a restriction that the size of the
underlying queue is a fixed size, in terms of memory.  But
trying to suport a dynamically sizable queue would require
a lot more involvement with the application.

Link from fi_info to the tx/rx attributes, so that the
application can specify the context sizes on endpoint
creation.  This also allows the provider to return the
size information from fi_getinfo.  As part of the change,
move the op_flags from fi_info into the tx/rx attributes,
which will allow the app to specify different flags for
transmit and receive operations.

Update the tostr function accordingly, plus toss in a few
bonus changes to improve code consistency.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Merge pull request #158 from shefty/master

Define TX/RX sizes plus 'to-string' function for printing structures
Merge pull request #156 from j-xiong/master

prov/psm: writedata/writedatato, FI_MR_OFFSET, etc