MB-9534: avoid use of binary:split while cleaning backtraces

We found out that binary:split is using binary:matches under
covers. And that binary:matches has logic to yield control back to
erlang scheduler on heavy enough invokations. For whatever reason they
decided to keep their intermediate state in "digits" of large
bignum. And those bignums get expose in backtraces we capture, causing
massive slowness if bignums are big enough.

We avoid binary:split completely. We use binary:match and incremental
splitting of backtrace in order to avoid this problem. Filtering and
massaging of backtrace is now also incremental for some small wins in
memory consumption (i.e. by avoiding intermediate list).

Change-Id: I25acc43936ca6dec81a7a3ef7685f9979c3efcfd
Reviewed-on: http://review.couchbase.org/31634
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-9834 Log how long certificate and private key generation takes.

Change-Id: Iaa289481651b4bba132ceb6e654c1f36f252a3ea
Reviewed-on: http://review.couchbase.org/31687
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100 Remove unneeded intermediate function.

Change-Id: I49340c1f58454055a8dd414001ff6d48b4fc6281
Reviewed-on: http://review.couchbase.org/31688
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9834 Move certificate generation out of config transaction.

Certificate generation might take seconds. And if config is changed
during this period of time, the transaction will need to restart and
hence regenerate the certificate again.

Change-Id: Iaf3914a716719daad3fecac12e03be09d1367954
Reviewed-on: http://review.couchbase.org/31689
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
use infinite timeout in work_queue:submit_sync_work/2

I.e. default timeout of 5 seconds is just too low.

Change-Id: Iead7d0840715b9845d5dcf66506c52bc698ae65d
Reviewed-on: http://review.couchbase.org/31691
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
hibernate work_queue after every piece of work

I.e. because it's state is very small, there's no need allow it to
accumulate tons of garbage

Change-Id: Ic8f35d3ad315f887ebe8949541d0dde88e938203
Reviewed-on: http://review.couchbase.org/31692
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-9451: serialize diag_per_node invokations

I.e. so that it's not consuming excessive amounts of CPU and/or
memory.

Change-Id: Ied973771c219656ed94a2f827a7277f2f42377b3
Reviewed-on: http://review.couchbase.org/31693
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-9451: deallocate memory sooner while sending diags

We're now doing parts of pretty-printing work in separate short-lived
processes. The idea is that when those processes die, their memory is
released immediately.

Change-Id: I63dd276bae10f26661011dc748a8b04095b887cf
Reviewed-on: http://review.couchbase.org/31694
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-9826: use new connection for remote cluster/bucket check

I.e. so that we can detect certificate mismatch.

Note that it does not prevent us from reusing previously established
connections for data traffic. But that's relatively fine and a
different problem anyways (i.e. solving it requires making sure _all
cluster nodes_ stopped using connections established using old
certificate).

Change-Id: I63ddc669a236be0665811f717718e0b5ecaf797d
Reviewed-on: http://review.couchbase.org/31695
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-9858: don't try to pass ssl connect options for http requests

Change-Id: I322f0c35448e66c8af432ec7a02c146c78bd4f6e
Reviewed-on: http://review.couchbase.org/31731
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9830: formatting for 'Create Cluster Reference'

Change-Id: I8e652b7ab724dbdedcb75e68167c6a4eb504a8b9
Reviewed-on: http://review.couchbase.org/31727
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9815: moved cluster certificate section

Change-Id: I5e150b4747ef77c43c0e3e83860a7fb36c2e85f6
Reviewed-on: http://review.couchbase.org/31728
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9815: removed unnecessary code

Change-Id: I24eab9ef74a0d3071dad799b1e33959ba7c6716e
Reviewed-on: http://review.couchbase.org/31729
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9818: cluster certificate is for enterprise only

Change-Id: Ib008d98f95419f0a9c0f54bc0b6e451ea93acc2f
Reviewed-on: http://review.couchbase.org/31730
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9839 Set up SSL before starting menelaus_web.

Change-Id: Iabed7766e21f6b811ac1d5241f8bbc0d2cfa97c3
Reviewed-on: http://review.couchbase.org/31746
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9858 Fix request_remote_bucket_on_new_conn's return value.

It must be {ok, RemoteBucket} instead of
{ok, {RemoteCluster, RemoteBucket}}.

Change-Id: I28ad238b68172ef336d74678e4a1e86ca7e590fd
Reviewed-on: http://review.couchbase.org/31747
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9862: reuse memcached sockets in find_missing_revs

pooled_memcached_client:execute_with_socket logic was only treating
{ok, _} as successfull return. Causing find_missing_revs which returns
triple to not return sockets to pool.

This is now fixed by treating any return of tuple which starts with ok
as successful response.

Change-Id: If7c1640beee74b4b1d153c3b34a4ce2d72f5bf80
Reviewed-on: http://review.couchbase.org/31750
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
[cbcollecinfo] gather ssl_proxy logs too

Change-Id: I759b2ab5164dcba39f58895af834ef082f1cd2e3
Reviewed-on: http://review.couchbase.org/31751
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
remove accidently committed excessive debug logging

Change-Id: I913246695cf1d114ad56d3b4edc3913753bfe6c4
Reviewed-on: http://review.couchbase.org/31736
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-100 Start ns_ssl_proxy as permanent application.

So that if it terminates for some reason, entire VM is also
terminated.

Change-Id: Ie9c040f83c26e5d07eba53d56fe0fab6cb4892d4
Reviewed-on: http://review.couchbase.org/31752
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>