bitcoin/bitcoin — pull request #5084 by chjj (2014-10-13 23:17:05+00:00)
Title: Allow compiling bitcoind as a shared library: libbitcoind.so

We are all aware of alternative implementations of the bitcoin protocol. I've
even helped personally implement [two][bcoin] of [them][bitcore]. There is an
obvious fundamental problem with all alternative implementations that I'm sure
everyone here sees: They're not bug-for-bug compatible with Satoshi's original
implementation.

As we see more and more production platforms ([BitPay][bitpay],
[Coinbase][coinbase], [Circle][circle], [Blockchain][blockchain], etc.) that
_absolutely_ depend on the adamant stability of their platform, with __so__
much at stake, it is imperative that their backend protocol implementation be
100% compatible with the original protocol. That is non-trivial, and maybe even
impossible to do with an alternative implementation.

Platforms and services like this are forced to use bitcoind as a backend on
their servers if they want stability. Using just bitcoind with RPC calls has a
number of limitations and hurdles.

- - -

This pull request alters the autoconf files to allow something like this:

``` bash
$ git clone git://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ ./autogen.sh
$ ./configure --enable-daemonlib
$ make
...
$ ls -la src/libbitcoind.so
```

Any program, or binding for a platform (python, php, node.js, ruby, etc), can
now link to the official bitcoin implementation. Luckily bitcoind decided not
to use `static` functions and put function templates in every header file,
which means linkage to any function in bitcoind should be fairly easily as
every function is exposed to the program which links to it.

## An Example

``` bash
# Write a program which links to libbitcoind
echo '#include "bitcoind.h"' > my_program.cc
echo '#include "..."' >> my_program.cc
echo 'extern bool AppInit(int, char **);' >> my_program.cc
echo 'int main(int argc, char **argv) {' >> my_program.cc
echo ' return AppInit(argc, argv) ? 0 : 1;' >> my_program.cc
echo '}' >> my_program.cc

# Compile our program
gcc -I$HOME/bitcoin/src -I$HOME/bitcoin/src/obj \
-I$HOME/bitcoin/src/config -I$HOME/bitcoin/src/leveldb/include \
-DHAVE_CONFIG_H -g -O2 -fexceptions -frtti -fpermissive \
-o my_program my_program.cc \
-lboost_system -lboost_filesystem -lboost_program_options \
-lboost_thread -lboost_chrono -lssl -lcrypto \
$HOME/bitcoin/src/libbitcoind.so

# Start our process
./my_program -server -daemon

# Bitcoind logs should be spitting out
tail -f ~/.bitcoin/debug.log
```

A contrived example: Obviously this capability would most likely be used to
link to different platforms' modules/bindings with that binding being
dynamically loaded into python's/node.js'/php's/ruby's memory.

### A More Complete Example

At BitPay, I have written a fully fledged node.js binding which links to
`libbitcoind.so` and can do almost anything an alternative bitcoin
implementation can do. It is called [bitcoind.js][bitcoindjs].

Feel free to examine the [documentation][doc] and [source][cc] to see the
actual [benefits][js] something like this would provide.

## What this PR does:

- Moves `main()` out of `bitcoind.cpp` and into `bitcoind_main.cpp`. Moves all
includes and variables out of `bitcoind.cpp` and into `bitcoind.h` so it can
be included by both `bitcoind.cpp` and `bitcoind_main.cpp` if necessary.

- Exposes two or three functions in header files which will be useful for
different platforms linking to libbitcoind. This required creating an
`rpcwallet.h` header file.

- Adds `--enable-daemonlib` argument to `configure.ac` - this enables `-fPIC`
for all object files instead of -`fPIE`. It also sets an AM conditional which
is checked for in the automake file. Also resets `-fvisibility` to its
default.

- When compiling the library, automake makes use of this `AM_CONDITIONAL` such
that targets are overridden by one `all:` target compiling the library, and
bitcoin-qt(+tests) is ignored.

- Now that all object files can be compiled as position independent code, the
custom `Makefile` target I've added (vanilla Make syntax - I'm sure automake
can do something fancier and compile a shared object through the help of
libtool, but I couldn't manage to make it work) can link these object files
together into `libbitcoind.so` with `-shared -fPIC`.

## Issues and potential annoyances:

I figure I'll address these since I'm fairly certain these questions will be
raised.

- If you want to compile bitcoind or bitcoin-qt, the object files compiled for
libbitcoind are essentially useless since they're not `-fPIE`. You have to
clean and recompile for scratch.

- Since bitcoind was never structured like a proper library, it's essentially
one big mess: there is logging and arg parsing all over the place. A coder's
best bet is to write their own stripped down version of `AppInit2()` which
starts up the net.cpp threads and listens for `RequestShutdown`, eventually
joining all threads. Currently, bitcoind.js simply starts up `AppInit2()` on
a separate thread.

- Name Mangling: Node.js module bindings are written in C++. Lua and Ruby
bindings are canonically written in C, which means in order to use
libbitcoind without guessing bizarre function names, functions will have to
be exposed with `extern "C"` on non-OO functions to force the stopping of
mangled names. Of course, the workaround would be for Lua and Ruby users to
write a C++ wapper and then link to them from their C binding.

- Internal API: When I first started writing bitcoind.js, I forked bitcoin
v0.9.0 and turned it into a shared library. Once I rebased it onto the latest
master, a lot of functions had changed and took differen params, a lot of
classes were changed or added. I had to rewrite several functions. Any
program depending on libbitcoind.so will have to deal with this as bitcoind
is developed since it is not developed like a library.

- The Build: I'm sure there's a slicker way to compile this as an `.so`. I'm
really not an Automake guru. I've fooled around a bit with the build
here, but at the end of the day, someone smarter than me could probably do a
better job at writing a `src/Makefile.daemon.include`.

The `noinst_LIBRARIES` libraries and other magical features of automake are
simply ignored when `--enable-daemonlib` is used. The .mm.o target compiles
all the object files necessary to link them up into a shared library with a
vanilla make target.

- Platforms: I'm not terribly familiar with OSX, but right now, I've only
tested this build on Linux (Arch). I've had two friends try to build on OSX
with unfortunate results that were apparently common in trying to compile
bitcoind on OSX. I'm still unsure if this will work with Macintoshes. I have
even less of an idea of how windows works compared to OSX. Maybe it could be
made into a `.dll` somehow.

## The Score

There may be better ways of doing this, but I think if the future of bitcoin
and the bitcoin world is to be a bright one, bitcoind _needs_ to be reshaped
into an optional library in some form. I don't only speak on behalf of my
employer here ([BitPay][bitpay]), even though I'm sure we all agree. This also
happens to be my personal opinion on the matter.

I open this to discussion. I'm willing to do whatever it takes to make Bitcoin
into a library.

[bcoin]: https://github.com/indutny/bcoin
[bitcore]: https://github.com/bitpay/bitcore
[bitcoindjs]: https://github.com/bitpay/bitcoind.js
[bitpay]: https://bitpay.com/
[coinbase]: https://www.coinbase.com/
[circle]: https://www.circle.com/
[blockchain]: https://blockchain.info/
[bindinggyp]: https://github.com/bitpay/bitcoind.js/blob/master/binding.gyp#L31
[doc]: https://github.com/bitpay/bitcoind.js/blob/master/README.md#documentation
[js]: https://github.com/bitpay/bitcoind.js/blob/master/lib/bitcoind.js
[cc]: https://github.com/bitpay/bitcoind.js/blob/master/src/bitcoindjs.cc