itch41: Avoid memcpy() in itch41_decode()

Remove the implicit memcpy() in itch41_decode() and let the caller
decide what to do with the message. This improves ITCH message decoding
performance up to 5%:

Before:

[penberg@golgotha libtrading]$ perf stat ./tools/test-itch41 ../data/nasdaq/S123110-v41.txt.gz
Decoding messages: 100%
 Performance counter stats for './tools/test-itch41 ../data/nasdaq/S123110-v41.txt.gz':

      18211.313992 task-clock                #    0.997 CPUs utilized
             8,450 context-switches          #    0.464 K/sec
                 0 cpu-migrations            #    0.000 K/sec
               572 page-faults               #    0.031 K/sec
    68,847,572,721 cycles                    #    3.780 GHz                     [83.33%]
    23,200,259,317 stalled-cycles-frontend   #   33.70% frontend cycles idle    [83.34%]
    15,248,176,165 stalled-cycles-backend    #   22.15% backend  cycles idle    [66.66%]
    88,276,934,750 instructions              #    1.28  insns per cycle
                                             #    0.26  stalled cycles per insn [83.34%]
    14,898,170,670 branches                  #  818.072 M/sec                   [83.32%]
     1,151,099,496 branch-misses             #    7.73% of all branches         [83.35%]

      18.265550473 seconds time elapsed

After:

[penberg@golgotha libtrading]$ perf stat ./tools/test-itch41 ../data/nasdaq/S123110-v41.txt.gz
Decoding messages: 100%
 Performance counter stats for './tools/test-itch41 ../data/nasdaq/S123110-v41.txt.gz':

      17345.070875 task-clock                #    0.997 CPUs utilized
             8,362 context-switches          #    0.482 K/sec
                 4 cpu-migrations            #    0.000 K/sec
               572 page-faults               #    0.033 K/sec
    65,569,032,443 cycles                    #    3.780 GHz                     [83.32%]
    24,233,187,125 stalled-cycles-frontend   #   36.96% frontend cycles idle    [83.34%]
    15,307,048,233 stalled-cycles-backend    #   23.34% backend  cycles idle    [66.67%]
    84,022,007,842 instructions              #    1.28  insns per cycle
                                             #    0.29  stalled cycles per insn [83.34%]
    13,990,083,749 branches                  #  806.574 M/sec                   [83.33%]
     1,036,831,608 branch-misses             #    7.41% of all branches         [83.34%]

      17.396997744 seconds time elapsed

Signed-off-by: Pekka Enberg <penberg@kernel.org>
tools/test-itch41: Message type stats

Add message type stat printing to test-itc41 program. Example output:

[penberg@golgotha libtrading]$ ./tools/test-itch41 ../data/nasdaq/S123110-v41.txt.gz
 Message type stats for '../data/nasdaq/S123110-v41.txt.gz':

        46,825  Timestamp - Seconds
             6  System Event
         7,776  Stock Directory
         7,779  Stock Trading Action
             0  REG SHO Restriction
       318,355  Market Participant Position
    60,072,466  Add Order
     9,106,358  Add Order - MPID Attribution
     2,814,336  Order Executed
        47,097  Order Executed With Price
       501,104  Order Cancel
    66,998,982  Order Delete
     9,362,883  Order Replace
       271,518  Trade (non-cross)
        15,550  Cross Trade
             4  Broken Trade
     1,119,600  NOII

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Remove unused includes from test-fix-client.c

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Rename ITCH to NASDAQ ITCH

ITCH is a widely used name for market data protocols. Therefore, rename
ITCH to NASDAQ ITCH to avoid ambiguity.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Generic order book support

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Level 2 market data support

This patch implements the core logic of FIX over FAST
market data receiving process. Its main purpose is to
let users follow an arbitrary number of order books
in real time i.e. order books of various instruments
and different depths.

To complete the goal, this patch introduces a few
data structures to be used by an external user.
'struct fast_book' represents a particular order
book which state is managed via FIX over FAST data
flow. 'struct fast_book_set' represents a collection
of order books which are updated through snapshot and
incremental data feeds.

The patch assumes that an order book's levels are updated
with help of two types of FIX messages: Snapshot Full
Refresh and Incremental Refresh. Whenever a user subscribes
to a particular order book we first reconstruct its state
by following snapshot messages. Once we are able to catch
order book's snapshot, we start following incremental
messages and apply them one by one.

It could happen that incremental messages disrupt the
incoming order i.e. incoming gaps are detected. In such
a case we assume that all order books' states are invalid.
We apply recovery procedure for each order book separately.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
FAST: Limit orderbook vizualization

The patch implements a sample application which is
intended to follow and display orderbook dynamics.
The application demonstrates how libtrading can be
used to achieve such a goal.

Here a few notes regarding the application structure:

* template file is an input parameter provided by the
FAST market data disseminator

* configuration file is an input parameter which defines
necessary market data feeds and orderbooks to follow.

The snippet below defines two incremental and one snapshot
UDP multicast data feeds. 'ip' is a multicast group ip address,
'port' port number to be used, 'lip' ip address of local interface
(see IP_ADD_MEMBERSHIP socket option for details)

	<feeds>
		<feed type="increment">
			<ip value="239.1.1.5" />
			<lip value="0.0.0.0" />
			<port value="15005" />
		</feed>
		<feed type="increment">
			<ip value="239.1.2.5" />
			<lip value="0.0.0.0" />
			<port value="16005" />
		</feed>
		<feed type="snapshot">
			<ip value="239.1.1.6" />
			<lip value="0.0.0.0" />
			<port value="15006" />
		</feed>
	</feeds>

The snippet below defines a single order book to follow

	<books>
		<book symbol="RIM3">
			<depth value="5" />
			<tick_exp value="0" />
			<tick_mnt value="10" />
			<id value="45166150" />
		</book>
	</books>

'symbol' - security symbol; 'depth' - order book depth;
'tick_exp', 'tick_mnt' - exponent and mantissa respectively
(this parameters will be obtained via Securities' Definitions
data feeds in the future); 'id' - security id

* To use libtrading functionality one has to follow a few steps:
	- book_set = calloc(1, sizeof(struct fast_book_set));
	book_set is a data structure which contains of all orderbooks
	- fast_books_init(book_set);
	book_set initialization
	- fast_books_subscribe(book_set, book);
	subscribe to the books we are interested in
	- fast_books_update(book_set);
	update books collection periodically

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
Clean up Makefile

Split long PROGRAMS line to multiple lines.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Update .gitignore

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Add install target to Makefile

Add install target to Makefile that installs header files, libraries,
and a shiny new libtrading-config utility to user-defined directory. By
default, everything is installed under $HOME.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Remove die() call from buffer_inflate()

In preparation for moving buffer_inflate() to generic code, remove the
call to die().

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Move buffer_inflate() to generic code

Signed-off-by: Pekka Enberg <penberg@kernel.org>
test-nasdaq-itch41: Display percentage after done

If we are unable to process a file completely, make sure the percentage
is not hidden by stats.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
test-nasdaq-itch41: Die if buffer_inflate() fails

Stop execution if buffer_inflate() fails and print out an error message.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
test-nasdaq-itch41: Prettify program name

Use the short version of program name everywhere.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Fix buffer_inflate() Z_BUF_ERROR handling

The zlib documentation states that:

  [...] Z_BUF_ERROR is not fatal, and inflate() can be called again
  with more input and more output space to continue decompressing.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Use "processing" for progress indicator

Signed-off-by: Pekka Enberg <penberg@kernel.org>
FIX: client should return negative value if unexpected message is received

Each time the client receives a message from the server it
compares the message received against the predefined template.
If those messages differ the client should exit with negative
value returned.

If tosend_elem is not NULL then the client was interrupted
by unexpected message from the server and negative value
must be returned.

Signed-off-by: Marat Stanichenko <mstanichenko@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
ITCH41: Retail Interest Message ("N")

This adds support for Retail Interest Message ("N") messages. The NASDAQ
TotalView-ITCH specification changelog says the message has been added
on January 8, 2013.

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Fix install Makefile target

The install target fails like this:

[penberg@tux libtrading]$ make install
  CC       libtrading-config.o
  LINK     libtrading-config
  INSTALL
removed `/home/penberg/bin/libtrading-config'
`libtrading-config' -> `/home/penberg/bin/libtrading-config'
removed `/home/penberg/lib/libtrading.a'
`libtrading.a' -> `/home/penberg/lib/libtrading.a'
install: invalid option -- 'l'
Try `install --help' for more information.
make: *** [install] Error 1

Fix up TEST_PROGRAM libraries properly and remove "-lz" from LIBS.

Signed-off-by: Pekka Enberg <penberg@kernel.org>