ssa.c: Add upper limit for number of access workers

The limit could be configurable from compiler command line.
Example:
./configure CFLAGS="-DMAX_ACCESS_POOL_WORKERS_NUM=2"
By default, the number of access workers is unlimited.

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
[ssa.c, acme.c, distrib/ibssa_opts.cfg]: Limit spacing for rejoin timeout

If rejoin timeout reaches a maximum it becomes constant.
Maximum timeout is
     MAX_REJOIN_TIMEOUT_FACTOR * rejoin_timeout.

Constant MAX_REJOIN_TIMEOUT_FACTOR can be changed from
compiler command line.
For example:
     ./configure CFLAGS="-DMAX_REJOIN_TIMEOUT_FACTOR=2"

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa.c: Add small random shift to rejoin timeout

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
Revert "[acm distrib].c: Pass umad send errors to SSA framework"

This reverts commit d1509db5f80d07188ccc0b64ae271f3d97ce17d8.

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
[distrib acm].c: Pass umad send errors to SSA framework

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa.c: Add log message before and after closing rclose thread pool

After commit 54326af6e7d7352953507bfbe22608ed72890376,
g_thread_pool_free could take a long time because it waits
for finish all rsocket closings.

Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa_release_notes.txt: Updated librdmacm information

Sasha found that 1.0.19-1 was missing a critical (for SSA)
rclose patch.

Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa_release_notes.txt: Added SSA framework workaround for librdmacm rclose issue

Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa.c: In ssa_calculate_prdb, return original/current PRDB when ret != 1

Either new PRDB is same as/equal to current PRDB or
invalid PRDB structure is detected during compare.

It is necessary to return original/current PRDB so update
is generated in case of restarts of access and ACM.

If there is some optimization for this, it can occur
in downstream thread (not done currently).

Signed-off-by: Hal Rosenstock <hal@mellanox.com>
ssa.c: Cosmetic change to reconnect log message in ssa_upstream_handler

Signed-off-by: Hal Rosenstock <hal@mellanox.com>
Eliminate race between core initialization and SSA MAD receipt

It is possible for SSA MAD to be received by core prior to
core->list_lock mutex being initialized by core_init_svc (and
this has been occasionally observed). Receipt of SSA MAD
causes core->list_lock mutex to be taken but if it's not already
initialized, the lock never completes.

For ACM connecting to combined core node, the core layer SSA
log (ibssa.log) contains:
Dec 17 22:24:37 482714 [CTRL]: ssa_ctrl_run: port event on fd 11
Dec 17 22:24:37 482763 [CTRL]: ssa_ctrl_port: mlx4_0:1 receiving MAD
Dec 17 22:24:37 482795 [UP mlx4_0:1:1]: core_process_msg: mlx4_0:1:1
Dec 17 22:24:39 482259 [CTRL]: ssa_ctrl_run: port event on fd 11
Dec 17 22:24:39 482289 [CTRL]: ssa_ctrl_port: mlx4_0:1 receiving MAD
Dec 17 22:24:43 482248 [CTRL]: ssa_ctrl_run: port event on fd 11
Dec 17 22:24:43 482282 [CTRL]: ssa_ctrl_port: mlx4_0:1 receiving MAD
Dec 17 22:24:51 482225 [CTRL]: ssa_ctrl_run: port event on fd 11
Dec 17 22:24:51 482262 [CTRL]: ssa_ctrl_port: mlx4_0:1 receiving MAD
Dec 17 22:25:07 482239 [CTRL]: ssa_ctrl_run: port event on fd 11
Dec 17 22:25:07 482273 [CTRL]: ssa_ctrl_port: mlx4_0:1 receiving MAD

In order to fix this, move the application specific initialization
into ssa_start_svc after the ssa_svc structure allocation occurs.
If the application specific initialization does not succeed,
then application specific destruction is needed as well.

So ssa_start_svc now contains 2 additional routines: init_svc
and destroy_svc which are application specific. distrib and acm
have default routines whereas core uses the already implemented
core_init_svc and core_destroy_svc routines.

Signed-off-by: Hal Rosenstock <hal@mellanox.com>
core.c: Add rebalancing mechanism for supporting unordered bringup

The pupose of algorithm for distribution tree rebalancing
is to balance the number of children for distribution
and access layers.

Balanced tree assures that maximum difference in
fanout (number of children) between nodes, that belong
to the same layer, is not more than 1.

The algorithm works as follows:

- Maximum fanout is being calculated per layer

- For each node whose actual fanout is larger
  than the maximum allowed, we take the proper number
  of children, reset their member_record parent_gid
  field and add them to orphan list.

- Orphan list members are being adopted (assigned with
  new parents)

Algorithm purpose is to keep ssa distribution tree
balanced in case of unordered ssa fabric bringup.

IMPORTANT NOTE:
Current rebalancing mechanism is currently only accurate at
initial ssa bring-up phase, because it relies
on children counters that are not currently updated when
some node leaves the distribution tree.

Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com>
core.c: Make core_start_dtree_timer() method more generic

core_start_dtree_timer() method was renamed to core_start_timer()

Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com>
core.c: Add timer for calling core_rebalance_tree()

Currently core_rebalance_tree() is called only once:
CORE_BALANCE_TIMEOUT seconds after the first subnet up
event.

Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com>
core.c: Change children counters update mechanism

Rather than updating ssa members children counters
when path records are received, they are updated in
'core_build_tree()' method and in case of unsuccessful
PR request are decremented.

Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com>
ssa.c: Reset prdb_epoch on new upstream connection

Currently only conn_dataup.epoch is reset on new
connection establishment.

In case of ACM connecting to ACCESS layer that doesn't
have SMDB, no PRDB will be calculated and thus ACM
prdb_epoch buffer won't be updated and will contain
some old value, that might be different from
conn_dataup.epoch.

This scenario will lead to quering non-existing PRDB
from ACCESS layer.

Signed-off-by: Ilya Nelkenbaum <ilyan@mellanox.com>