Fix race on thread shutdown causing deadlock

This patch and the next are somewhat a revert of 318e9f2, but the previous
fix didn't quite close the race. This only happens when we create threads
for a backstore that turns out to be invalid, which we then tear down.

See https://bugzilla.redhat.com/show_bug.cgi?id=848585 .

This is occurring because there's still a window where a thread misses
seeing info->stop == 1 but is not yet in cond_wait so it misses the
broadcast:

thread_close:              thread_worker_fn:
                           info->stop is seen as 0
info->stop = 1
pthread_cond_broadcast     -- misses broadcast
                           pthread_cond_wait
pthread_join (hangs)

I believe the solution is to go back to using pthread_cancel. We can call
it before pthread_cond_wait is called (or after) and it will do the right
thing: pop out and exit. The only tricky bit is we need to use the
pthread_cleanup_push mechanism to properly release info->pending_lock.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Remove startup_lock

The purpose of the startup lock was to hold threads at the startup_lock
so if a thread creation fails, it's still possible to have them
end themselves by looking at info->stop. Since we are back to using
pthread_cancel instead of info->stop, we don't need this any more.
The only wrinkle is we need to only send pthread_cancels to
successfully-created threads, so now check this in the destroy_threads
reverse-loop.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Replace if()/goto with while()

Is equivalent and nicer-looking.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Remove unneeded pthread_cond_broadcast

Previous patchset neglected to remove a now unneeded call.

Signed-off-by: Andy Grover <agrover@redhat.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
tgt 1.0.47

Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Fix tgtd.spec to build on SLES

The docbook package name is different between RedHat and SLES.

Signed-off-by: Roi Dayan <roid@mellanox.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
added support for bsoflags parameter

This patch will allow one to use bsoflags with tgt-admin.

Signed-off-by: Cédric Milesi <cedric.milesi@hp.com>
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
iscsi: add helper functions for checking iostate of iscsi connections

This patch adds helper functions for checking iostate of iscsi
connections from outside of iscsid.c. These functions will be used in
iscsi_tcp.c in the future commits.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
iscsi: decompose iscsi_[rt]x_handler()

Basic strategy of parallelizing sending/receiving iSCSI PDUs is like
this: offload do_recv(), do_send() and crc checking to worker threads
and do other part in the main event loop thread.

To follow the above strategy, this patch decomposes iscsi_rx_handler()
and iscsi_tx_handler() into some parts. Some of them can be called in
worker threads and others are not. In the future commit,
multi-threaded TCP event handler will be implemented based on
them. This patch re-implements single threaded TCP event handler
(past iscsi_tcp_event_handler(), renamed iscsi_tcp_st_event_handler())
based on these decomposed parts. iscsi_tcp_st_event_handler() is
also required because under some workload the multi-threaded version
degrades performance because of frequent communication between
threads.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
tgtd, iscsi: add a new option "-T" for specifying a number of threads which send/recv iSCSI PDUs

Currently, nr_iothreads, specified with "-t" option, is used as a
number of worker threads per logical unit conventionally in some
backing store drivers.

This patch adds a new option "-T" to tgtd for specified a number of
threads which send/recv iSCSI PDUs. Because using the "-t" option for
this purpose would be introduce inconvenience of performance tuning.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
tgtd: implement a deferred event modification mechanism

In the next commit, worker threads for sending/receiving iSCSI PDUs
will be implemented. When the feature is used, the worker threads read
and write fds created by accept(2). For avoiding conflicts between the
main event loop thread and the worker threads, no event should be
notified to the main event via epoll_wait(2) when the worker threads
are running.

For this purpose, this patch implements a deferred event modification
mechanism. Two new members are added to iscsi_tcp_connection:
used_in_worker_threads and restore_events. The first one is used for
indicating that the connection is used by worker threads. The second
one is used for storing an event mask which should be set to the
fd. When control of the fd is passed back to the main event, the
stored event mask is set via tgt_event_modify().

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
iscsi: offload iSCSI PDU send/recv to worker threads

Current tgtd sends and receives iSCSI PDUs in its main event
loop. This design can cause bottleneck when many iSCSI clients connect
to single tgtd process. For example, we need multiple tgtd processes
for utilizing fast network like 10 GbE because typical single
processor core isn't fast enough for processing a bunch of requests.

This patch lets tgtd offload send/recv iSCSI PDUs and check digests to
worker threads when a parameter of "-T" option is larger than 1. The
offloading is done by a newly added event handler,
iscsi_tcp_mt_event_handler(). When "-T" isn't passed or value 1 is
passed, single threaded version (iscsi_tcp_st_event_handler()) is
used. The single threaded version is provided because in some cases
the multi-threaded version degrades performance (e.g. a number of
initiators are not so large).

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
iscsi: let conn_close() transfer state of a connection

Current tgtd seems to have a possibility that it has a connection
whose state isn't STATE_CLOSE even after the connection is passed to
conn_close(). Such a connection is confusing because
iscsi_scsi_cmd_done() cannot detect the closing of the connection.

This patch lets conn_close() transfer state of a connection to
STATE_CLOSE for avoiding this purpose.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
iscsi: per worker thread work queue for rx/tx PDU

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>