nyse taq: Simplify decoding functions

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Add byte order helper functions

The NASDAQ ITCH 4.1 protocol, for example, needs byte order conversion
helpers.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Use <endian.h> for byte order helpers

Also add missing helper functions.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Byte Vector support introduced

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Default operator should occupy a bit inside the PMAP

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: session configuration structure introduced

This patch introduces per-session configuration
data structure which is to be used to initialize
FAST session properly (see fast_session_new()).
The principles are the same as with FIX protocol.

At the moment the following settings are supported:

* Preamble - some venues put a few bytes in front of
each FAST encoded message. These bytes do not obey to
FAST encoding rules and present auxiliary information
(e.g. MsgSeqNum). At the moment libtrading do not use
such a preamble but can skip a particular number of bytes
before it starts to decode a message.

* Reset - some venues make a decoder to reset dictionary
explicitly (e.g. they incorporate a particular bytes
combination into the stream  - 0xC0 0xF8) but some venues
assume that a decoder should reset dictionary without any
notification (e.g. if unreliable UDP protocol is used to
disseminate the data). In the latter case we raise a flag
within the FAST session data structure to indicate that
reset should be performed without any notification
received.

* Source specific multicast - some venues use source
specific multicast (IP_ADD_SOURCE_MEMBERSHIP). Libtrading
should support it also

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Increase the maximum number of sequence elements supported

It's not unusual to receive an order book updates of the depth 50.
In such a case the Snapshot message would contain of 100 sequence
elements (50 for bids and 50 for ask).

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Delta operator for ascii strings implemented

This implementation is naive and might be optimized later

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: preserve pmap_bit whenever data are not fully received

pmap_bit is a bit checked whenever FAST field is decoded.
If buffer does not contain enough data to decode the whole
field we should preserver the previous pmap_bit value and
wait until necessary bytes will be received.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: use glib hash table implementation to store FAST fields

hsearch_r() API is not available on Darwin (a.k.a. Mac OS X)
so this is causing a build regression. Since glib is supported
on all relevant platforms, let use that library for the moment.

Reported-by: Pekka Enberg <penberg@kernel.org>
Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
Document build prerequisites on Fedora

Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Limit order book data structures modification

There are two general ways the limit order book can
be distributed:
 * Price levels distribution. In this case order book
is distributed explicitly. That is order book's dynamics
represents the successive changes applied to a particular
price level i.e. price level modification, price level
removal, price level creation.
 * Orders' flow distribution. In this case order book
is distributed implicitly. That is one receives the
orders' flow and creates a limit order book by processing
orders received.

Libtrading is to support both ways. However, linear arrays
of bids and asks can create some difficulties in the future.
Hence, the current patch presents a few amendments which are
to be applied: instead of using linear arrays hash tables,
binary trees and lists are used.

Whenever a particular level is sought - hash tables are to
be examined. In order to traverse an order book binary trees
or lists are good candidates.

At the moment glib implementation is used but this option
is adjustable.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Identify order book by symbol

It's possible to identify order book both with number and
with symbol. We should support both variants.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: TradingSessionID support introduced

Security can be traded in different modes (trading sessions).
Hence, we should distinguish between trading sessions when
order book is constructed.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Decimal volume support introduced

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Market data entries may have different names

Market data entries may have different names under the
snapshot/increment messages which depend on the template.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: New options in order books' configuration files

The following options are included:
* Source specific multicast option
* Preamble option
* Implicit reset option

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Reset a session if implicit reset option is set

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST parser: Keep reading raw data until signal is sent

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Apply incremental messages on the fly

We had the following logic implemented whenever
an orderbook is being subscribed (aka late join):
collect incremental messages and wait until snapshot
is arrived. Apply the snapshot and then incremental
messages stored in memory.

Such an implementation has an evident flaw i.e. we
cannot guarantee the number of incremental messages
being stored in memory. It leads to a lot of memory
allocated/reallocated which influence on the library
performance.

This patch allow incremental messages to be applied
without snapshot received. In other words we do not
have to store incremental messages in memory but
rather apply them on the fly.

All we have to know is incremental messages's sequence
number for a particular instrument. Once it is known,
it becomes easy to apply a snapshot message: we should
accept newer changes and discard older ones.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>