MB-6860 Fix old couchdb view cleanup when there are no ddocs

If there were no design documents, it wouldn't delete existing
useless index files.

Change-Id: I88b1f5950eed8bb0b04287fd14a4ccdf90083e55
Reviewed-on: http://review.couchbase.org/21745
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-6860 Shutdown outdated dev index processes

After development design documents get updated or deleted,
ensure the corresponding view group processes (and files)
are shutdown/closed. This is used to work over an year ago,
and got fixed in Apache CouchDB, but since then Couchbase
uses the old couchdb views in a different way - the design
documents are always located in separate database (master)
from the database where data comes from (vbucket database).
This Couchbase specific way of using couchdb views broke
the shutdown logic of outdated and unnecessary view group
processes and their respecive index files.

This only affected development views that don't use the
full_set=true parameter.

Change-Id: I42d66cd1fdd9798ce43debb7b6d6579cd72ae273
Reviewed-on: http://review.couchbase.org/21746
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-100: Don't throw error on non-existing directory

Don't throw an error if the directory you want to delete
doesn't exist.

Change-Id: Id7e4dbfacc952ac3a2a2069b66c61c29303be4fe
Reviewed-on: http://review.couchbase.org/21711
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-6860 Ensure dev index file deleted after db deletion

Often view group would receive a db process monitor DOWN
message before couch_view received the database deleted
event, in which case the index file would end up not being
deleted. The solution here is to shutdown the view group
only through couch_view when the database is deleted.

Change-Id: Ife6311b3a50d5c9b5576bcb553b207c5b162c3d0
Reviewed-on: http://review.couchbase.org/21828
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-6988 Add more data to output of queries with ?debug=true

Add some more index meta data, from all nodes, to the output
of queries with ?debug=true. Also make the same debug information
available for reduce queries as well, which was not working
after some optimizations done long ago.

Change-Id: I975130d95de196a29528ec061b11e1c89a080520
Reviewed-on: http://review.couchbase.org/21869
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
MB-6969 add snappy-stubs-public.h.in to snappy dist target

Change-Id: I111d01a73aa44df2defb477ee4c0f1a5d1d31888
Reviewed-on: http://review.couchbase.org/21801
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
MB-6969 add lhttpc.hrl to DIST target

Change-Id: I717ef85de9398eeafa31a950ea249efeb46e327f
Reviewed-on: http://review.couchbase.org/21802
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
MB-6969 add missing .erl file to couchdb Makefile.am

also removed couch_file.erl_config_writer.erl from
source_files as it was causing make dist to fail since
this file does not exist anymore

Change-Id: I33ab1fb23cef072ae0dc440c7096a7ee5dae4f4c
Reviewed-on: http://review.couchbase.org/21804
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
MB-6990: Add validation of the spatial function

Design Docs that contained invalid spatial functions weren't
rejected, now they are.

This commit depends on http://review.couchbase.org/21890

Change-Id: If775539e965f1f3f8f1af807559c61e57ca15054
Reviewed-on: http://review.couchbase.org/21891
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-6990 Ignore missing module during make check

During make check, or when running CouchDB standalone for
testing purpose, the spatial modules (GeoCouch) are not in
the Erlang path, whence just ignore the function call in
this case.

Change-Id: Ia02b37eef52960852189c2dc2b689198cecf89dc
Reviewed-on: http://review.couchbase.org/21962
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-6947 Don't log set_view_outdated errors in httpd layer

These are already logged by the index/view merger, with a
log level of 'info' and with more useful details.

Change-Id: Id919d4ffedc60f58bf708f209689d0ab7eb2408a
Reviewed-on: http://review.couchbase.org/21963
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
MB-6995 Don't restart couch_{server,set_view} on dir changes.

If some external (meaning ns_server) process calls anything from
couchdb at the time of restart, it may cause this process to terminate
in the hard way with reached_max_restart_intensity error. To avoid
this we're going to stop couchdb from ns_server in coordination with
all the other ns_server processes.

Change-Id: Ifea0c6e3bbebbdd36efedd4fe10bb2a7ccb27892
Reviewed-on: http://review.couchbase.org/21946
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6995 Stop related apps when couchdb application is stopped.

When ns_server needs to restart couchdb, we want it to restart not
only couch app but also all the related apps.

Change-Id: Ic8e1d316fd1c2e765c7783e5009cbf9448cc9b13
Reviewed-on: http://review.couchbase.org/21947
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7025 Shutdown databases in couch_server:terminate correctly.

Since recently couch_dbs_by_name contains only PID and nothing
more.

It's not clear though if this actually fixes referred bug.

Change-Id: Ifb3a0bd9a9c011c4836e0513a4adbaba94c3c8c1
Reviewed-on: http://review.couchbase.org/22021
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-6107 Do less fsyncs on index compaction

When reaching the retry phase, there's really no need to
do an fsync for each retry iteration. Instead just do a
single one at the end.

Also change the order of some flush calls (without affecting
correctness) just to increase parallelism.

Change-Id: I7021cde3aa54b5c9c7a04b0513301bce6c30ee8d
Reviewed-on: http://review.couchbase.org/21964
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-6107 More efficient index compaction retry phase

Ensure that during the index compaction retry phase, we
don't insert/remove many small batches into the btrees.
Instead do less batch operations but with larger batches,
reducing IO and causing less fragmentation.

Change-Id: Idc9a8059c2e74f4f3cfe19a7be98e3a9d556500d
Reviewed-on: http://review.couchbase.org/21965
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-6107 Make index btree chunk thresholds configurable

Allow via couch_config to change the thresholds for the
btrees created for views. The purpose is just to allow
for evperf team to easily do tests with different values
for these thresholds.

Change-Id: I0e68127159b38081dd64391d4dc2b43004233113
Reviewed-on: http://review.couchbase.org/21968
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-7030 Always send new group snapshot to parent

Instead of sending it with a minimum periodicity of 5 seconds,
every time a batch of changes is applied to the btrees, send the
new view group snapshot to the parent. This prevents discarding
some indexing progress when there's a quick succession of partition
state transitions, as it happens during rebalance.

This restriction was added in the past with the goal of reducing
memory copying (group snapshots were structures with much more
fat).

Change-Id: I4a56b1eaf1019e3203dcaf4e23422145a0b64d21
Reviewed-on: http://review.couchbase.org/22006
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-7030 Use some cheaper BIFs for list and orddicts

lists:member/2 is cheaper than ordsets:is_element/2, and
lists:keyfind/3 is cheaper than orddict:is_key/2 and
orddict:fetch/2.

Change-Id: I657f644d7b675097cec46cfbabb0aba4ff02fcd7
Reviewed-on: http://review.couchbase.org/22007
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-7030 Check mail box for new group snapshot

When stopping the updater, check if there's currently
any new group snapshot sent by him to us. If so, process
it.
Before this change it was ignored, meaning that the next
time the updater was restarted, it would repeat some work,
wasting CPU and IO.

Change-Id: I840bc797567a1d7c81f9970aaf4e1b9d20271b1e
Reviewed-on: http://review.couchbase.org/22008
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>