varnish/libvmod-curl — pull request #21 by valdemon (2014-10-23 10:36:40+00:00)
Title: The asynchronous, non-blocking cURL calls using multi interface and libuv

Hi,

this commit introduces the asynchronous cURL, as we discussed in #19.
1. All transfer functions use the *multi interface*, so they work asynchronously.
2. The response related functions wait until the response is ready.
3. You can do any number of tasks between the transfer call (like get,post) and a first call to any of the response related functions (like status, header).
4. The transfer function call and corresponding response related function call can be spread through different VCL subroutines
5. There's a *no_wait* option provided (the new *set_no_wait* function in the module interface), for the *fire & forget* mode.
6. All previous VTC tests (unchanged) are passing
7. Additionally 2 tests are provided:
- test09.vtc - proves the asynchronous capability as stated in points 1 to 4
- test10.vtc - checks the 'no_wait' option (point 5)

Some highlights:

- the *cURL multi interface* is used with the asynchronous socket IO option (the *MULTI_SOCKET* option, see: http://curl.haxx.se/libcurl/c/libcurl-multi.html), which allowed to serve efficiently a large numbers of simultaneous cURL calls in a single multi interface thread
- the *libuv* (https://github.com/joyent/libuv) is used for the socket/time event handling. This gives an abstraction for the OS dependent scalable event multiplexing (thus portability).
Requires *libuv-devel* package for buid, and/or *libuv* for runtime, which one can install as following:
```
yum install libuv libuv-devel
```
for CentOS, or
```
sudo apt-get install libuv-dev
```
for Ubuntu
- stress tests were performed on different virtual box'es (CentOS 6.5, CentOS 7 and Ubuntu 13.10), showing no issues with the module.
- 500 requests /s,
- 2 cURL calls per request (one of them in *no_wait* mode)

Known issues:
- timeouts are not served properly by the multi-interface in the cURL version 7.19.7 (current RedHat/Centos 6 package). Don't exactly know in which cURL version this was fixed, but it's not an issue anymore in 7.32.0 (Ubuntu 13.10) or in 7.29.0 (CentOS 7).
One is able to solve it on CentOS 6 by upgrading to higher version of libcurl, eg. to the latest (I can confirm that it works), using eg. the city-fan.org repo:
```
rpm -Uvh http://www.city-fan.org/ftp/contrib/yum-repo/city-fan.org-release-1-12.rhel6.noarch.rpm
yum install libcurl
```

Best regards,

Waldek