Repository: FeatureBaseDB/featurebase

Issue #1247: Simplify inter-node communication interfaces
Created: 2018-05-03T23:17:33Z
Description
Pilosa has 3 interfaces for setting up inter-node communication - you'll notice that two are very similar:
// EventReceiver is the interface for the object which will listen for and
// decode broadcast messages before passing them to pilosa to handle. The
// implementation of this could be an http server which listens for messages,
// gets the protobuf payload, and then passes it to
// EventHandler.ReceiveMessage.
type EventReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(EventHandler) error
}

// MemberSet represents an interface for Node membership and inter-node communication.
type MemberSet interface {
	// Open starts any network activity implemented by the MemberSet
	// Node is the local node, used for membership broadcasts.
	Open(n *Node) error
}

// BroadcastReceiver is the interface for the object which will listen for and
// decode broadcast messages before passing them to pilosa to handle. The
// implementation of this could be an http server which listens for messages,
// gets the protobuf payload, and then passes it to
// BroadcastHandler.ReceiveMessage.
type BroadcastReceiver interface {
	// Start starts listening for broadcast messages - it should return
	// immediately, spawning a goroutine if necessary.
	Start(BroadcastHandler) error
}
Pilosa currently defines several interfaces for sending, and receiving inter-node communication. These are:
// BroadcastHandler is the interface for the pilosa object which knows how to
// handle broadcast messages. (Hint: this is implemented by pilosa.Server)
type BroadcastHandler interface {
	ReceiveMessage(pb proto.Message) error
}

// EventHandler is the interface for the pilosa object which knows how to
// handle broadcast messages. (Hint: this is implemented by pilosa.Server)
type EventHandler interface {
	ReceiveEvent(e *NodeEvent) error
}

// StatusHandler specifies the methods which an object must implement to share
// state in the cluster. These are used by the GossipMemberSet to implement the
// LocalState and MergeRemoteState methods of memberlist.Delegate
type StatusHandler interface {
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
	HandleRemoteStatus(proto.Message) error
}

// Gossiper is an interface for sharing messages via gossip.
type Gossiper interface {
	SendAsync(pb proto.Message) error
}

// Broadcaster is an interface for broadcasting messages.
type Broadcaster interface {
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
	SendTo(to *Node, pb proto.Message) error
}

// InternalClient should be implemented by any struct that enables any transport between nodes
type InternalClient interface {
    // Very long, here's a summary
    // Schema stuff get, create, delete, etc.
    // Import, Export, Backup, Restore
    // FragmentBlocks, BlockData, AttrDiff
    // Query, MaxSlice
    SendMessage(ctx context.Context, pb proto.Message)
}
We can roughly categorize these into sending and receiving:
// Sending
StatusHandler
	LocalStatus() (proto.Message, error)
	ClusterStatus() (proto.Message, error)
Gossiper
	SendAsync(pb proto.Message) error
Broadcaster 
	SendSync(pb proto.Message) error
	SendAsync(pb proto.Message) error
	SendTo(to *Node, pb proto.Message) error
InternalClient
  SendMessage(ctx context.Context, pb proto.Message)
  ...

// Receiving
BroadcastHandler
	ReceiveMessage(pb proto.Message) error
EventHandler 
	ReceiveEvent(e *NodeEvent) error
StatusHandler
	HandleRemoteStatus(proto.Message) error
Confusingly, there are both pull and push based interfaces here.
Gossip/memberlist has a facility for sharing state between nodes asynchronously
that it polls on its own time (the LocalStatus() and ClusterStatus() methods).
I'm not convinced that it's a good idea for us to use this - it might be better
to use SendAsync or SendSync for consistency.
Currently, Pilosa has two types of communication:


A handler (now HTTP), which has some endpoints that are purely external, some
which are internal, and some which can be either.


Gossip/memberlist which is used purely for internal communication. Within
this, there are cluster membership events (EventHandler), state sharing
events (StatusHandler), and generic messaging (BroadcastHandler).


It's clear to me that this all can be simplified, but there are a number of
considerations.
At a high level, Pilosa should expose the operations that can be done to it, and
the information that can be gotten from it, this is mostly in API right now, but
not completely. Implementations of external interfaces (HTTP, GRPC, etc.) can
take an API and expose it however they like.
Pilosa should define an interface for talking to other nodes. As part of its
initialization, it needs to be given an implementation of this interface. It is
up to the operator of the cluster to ensure that the sender on each node is
compatible with a receiver on every other node. This interface is mostly
InternalClient, but there is a lot of other messiness.
Success criteria (What criteria will consider this ticket closeable?)
This ticket defines the desired end state at a high level. (one way to send, one
way to receive). Work toward this goal will need to be broken out into subtasks.

Duplicate issue #1317: Broadcast/Messaging simplification and unification
Created: 2018-05-21T20:54:25Z
Description
Pilosa currently employs several, very similar, interfaces to send/receive/handle messages among nodes in the cluster. We should combine these into a single interface which manages inter-node messaging.
Success criteria (What criteria will consider this ticket closeable?)
There is a single, obvious method by which a node sends and receives messages.

Closing comment (2018-06-02T14:30:09Z): closing as a duplicate of #1247