Bump version to 2.2
Add a Simple Unit testsuite

Summary: We have used to at facebook for code sanity checking
use thrift rate limitting

Summary: add max_conn directive to the scribe main configuration file.
         default 4000.

Reviewed By: agiardullo

Test Plan: 1. set max_conn=5 or any small number in the scribe.conf on any
           port, e.g. 1463
           2. start a scribd instance and watch the err log
           3. keep telnet to scribe port, e.g. 1463
           4. once telnet session exceeds max_conn.  you should see the
           following gets output to the scribe err console:
           "
           [Wed Sep 23 02:43:00 2009] "Throtle max_conn to 2"
           Thrift: Wed Sep 23 02:43:00 2009 libevent 1.4.3-stable method
           epoll
           Thrift: Wed Sep 23 02:43:37 2009 thrift non-blocking server
           overload condition
           "

Revert: OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@14907 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Replace "smc" with "service" in open source facing code

Summary:
SAT

Reviewed By: agiardullo

Test Plan:
checked that code compiles. check smc fetching still works with my test
for the
datacenter filtering.

Revert: OK

DiffCamp Revision: 94534

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@23387 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
deprecate max_queue_length

Summary:
deprecate max_queue_length. If someone sends a message vector with
	large number of messages then first attempt storing it in the primary
	store rather than directly going to the secondary store.
	1

Test Plan:
tested with scribe/examples

DiffCamp Revision: 99859
Reviewed By: groys
CC: agiardullo, groys, scribe-dev@lists
Tasks:
#183054

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@24789 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
add use_tree flag to create date-based directory tree

Summary:(from emaland @ twitter)  adds a config parameter use_tree which
will allow creation of
date based directory tree instead of the current default category based.
This is turned off by default so it is non-effecting unless specified in
config

Task ID: #180169

Blame Rev:

Reviewers:agiardullo,groys

CC:scribe-dev@lists, scribe-ops@lists

Test Plan: compile this and see that it does not effect us

Revert Plan:

Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

DiffCamp Revision: 100349

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@24802 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
max_size=0 in filestores will allow unlimited size

Summary:
from (Eric mayland @ twitter ) same as title

Test Plan:
run testsuite. simple enough for not testing much

DiffCamp Revision: 100368
Reviewed By: agiardullo
Commenters: pkhemani
CC: scribe-ops@lists, agiardullo, pkhemani, groys, scribe-dev@lists
Tasks:
#180169

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@24803 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Revert "add use_tree flag to create date-based directory tree"

Summary:
This reverts commit in the title, which was put in by
mistake. Somebody accept this

Test Plan:
compile run example

DiffCamp Revision: 100636
Reviewed By: pkhemani
CC: agiardullo, pkhemani, scribe-dev@lists
Tasks:
#180169

Revert Plan:
re revert recursion :)

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@24820 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Fix error in counters

Summary:
fix bug that incremented overall counter twice

Test Plan:
none

DiffCamp Revision: 100716
Reviewed By: pkhemani
CC: agiardullo, pkhemani, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@24844 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Add a filecat test

Summary:
A simple test that will send a file to the required scribe
server. This can help verifying that there is no data loss while testing
"a one scribe server sending to another scribe server" scenario. Also
lets get the required path in tests.php. For opensourcing I will add it
to the script to revert those lines

Test Plan:
tested that this works

DiffCamp Revision: 103042
Reviewed By: pkhemani
CC: pkhemani, groys
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@25476 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Ability to double log default and prefix categories

Summary:
Allow default and prefix categories to have multiple stores.

In the process (1) removed temporary data strucutres that were used only
during config time (2) had to fix locking to allow multiple stores to be
created for a category at run time

Test Plan:
test multiple file stores for default
test multiple file stores for test*
test multiple file stores for "default test*"
test multiple file stores for "xyz test*"
test multiple file stores for "test*"

DiffCamp Revision: 103186
Reviewed By: groys
Commenters: arivin, agiardullo
CC: agiardullo, arivin, pkhemani, groys, scribe-dev@lists
Tasks:
#179680

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@25744 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
get rid of lastWriteTime

Summary:
this variable has no consequence in the current code

Test Plan:
none

DiffCamp Revision: 104925
Reviewed By: agiardullo
CC: agiardullo, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@25783 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Make exception handling consistent and catch const&

Summary:
just a small clean up of our exception handling code. We should
always catch by const &

Test Plan:
test suite

DiffCamp Revision: 105479
Reviewed By: pkhemani
CC: agiardullo, pkhemani, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@25919 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
code cleanup - pcategories and pcategory_prefixes are now statically
allocated

Summary:
This task was identified as a followon task to "add ability to
	double log default and prefix categories"

	changes pcategories to categories. pcategory_prefixes to
	category_prefixes

Test Plan:
tested with examples/*. tested with scribe_ctrl reload
	tesuite tested

DiffCamp Revision: 106173
Reviewed By: groys
CC: agiardullo, groys, scribe-dev@lists
Tasks:
#192430

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26015 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Remove locking from getsize

Summary:
Change getSize to not acquire a lock to read the queue size. The lock = true
part was just a leftover from a previous  version of the 600 buckets diff.

Test Plan:
testsuite

DiffCamp Revision: 100884
Reviewed By: jsong
Commenters: agiardullo, pkhemani
CC: agiardullo, jsong, pkhemani, groys, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26065 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Send dummy log message to check for dfqs

Summary:
Currently when a scribe server denies for queue size, it has
already received the whole set of log messages for the call. This is
wasteful of network resources, especially in cases where a bufferstore
has a network store as primary, and the bufferstore gets backed up. The
bufferstore will then send one whole backed up file at a time. With this
change,
1. a bufferstore will try sending a dummy message before doing a
secondary to primary transfer. As an additional improvement to adaptive
backoff behavior, a bufferstore will not change its retry_interval if
the dummy log fails
2. If a scribe server receives a dummy log call, it will do throttle
checks and return OK or TRY_LATER.

This in essence implements a hasCapacity rpc function for the scribe
server without changing its thrift interface

Test Plan:
run testsuite. Check by simulating backoff that adaptive
backoff and non-adaptive backoff behavior still works

DiffCamp Revision: 104810
Reviewed By: pkhemani
Commenters: agiardullo, arivin
CC: agiardullo, arivin, pkhemani, groys, scribe-dev@lists
Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26066 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
categories=default leads to double logging default

Summary:
regression introduced when implementing "enable double log for prefix and
default categories"

Test Plan:
In the last iteration I did not have a test with just categories=default.
Carried that out this time.
	Tested with file store.
	Tested with testsuite

DiffCamp Revision: 107618
Reviewed By: groys
CC: agiardullo, pkhemani, groys, scribe-dev@lists
Tasks:
#179680

Revert Plan:
OK

Blame Revision:
103186

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26259 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
NetWorkStore:: fix connection recycling on error

Summary:
For service based connections entire store is closed and reopened.
        Their is additional connpool logic to replace the old connection
        with the new connection. This should take care of fetching
        the most recent smc updates.

        don't close a connection by resetting it. this will take care of
        ECONNRESET error messages

        don't close non-service based connections when errors occur. this
        should take care of broken-pipe erros between mid-tier and
writers.

Test Plan:
1. service based, conn_pool=yes
	2. service based, conn_pool=no

	For the above 2 cases do the following
		start scribe client in disconnected state
		start the scribe servers (2 of them)
		make sure logs can be sent to one of the scribe servers
		STOP the scribe server that was logging the message
		scribe client picks up the other scribe server (after
		    going through DISCONNECTED state and file write/read which
		    sucks)
		bring down the other scribe server
		scribe client now in disconnected state
		change the SMC scribe_test setting
		start 2 new scribe servers as per the new setting
		scribe client goes from DISCONNECTED to SENDING_BUFFER to
		    STREAMING.
		This time while the smc servers are running change the SMC
		    setting
		send some logs ... the logs still go to the old servers
		bring up the new servers
		stop the old servers
		scribe client becomes DISCONNECTED and then STREAMS to the new
		    servers

	3. ip:port based, conn_pool=yes
	4. ip:port based, conn_pool=no

		STOP and restart the servers. scribe client DISCONNECTS and
		then reconnects

		simulate a dfqs from the server, thrift connection doesn't
		close, scribe client goes to disconnected state, and to
		streaming when dfqs goes away.

	5. all 4 cases above, set small dfqs threshold (1 message) - send
	    logs directly to the scribe servers and to via the scribe clients
	    by running scribe_cat in shell loops.

DiffCamp Revision: 107742
Reviewed By: groys
CC: agiardullo, pkhemani, groys, scribe-dev@lists
Tasks:
#178283
#123207
#188199

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26334 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
stop default stores on reload

Summary:
default stores were not being stopped. This has been an issue for a
	while. And that is why the connection pool was not getting recycled
	on reload.

Test Plan:
Before the change
		verified that default store thread was not being stopped by
		going dropping into the debugger. In the debugger it was very
		easy to cause a crash because of this.

		also verified that this was an issue with scribed that is
		is deployed in production and doesn't have many of the recent
		changes

	After the change
		connection pool is getting cleared. default stores are being
		cleared.

DiffCamp Revision: 108213
Reviewed By: groys
CC: agiardullo, pkhemani, groys, scribe-dev@lists
Tasks:
#194684

Revert Plan:
OK

git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26394 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
Enable scribe conf parameter inheritance, and allow simple buffered bucket store
configuration

Summary:
Allow parameter inheritance in scirbe.  Scribe consists of various store each
of which contain a tree like sub stores.  Currently, each store's configuration
have to be specified in the conf section of that store.  

With this diff, we allow sub stores to get config parameter from its ancestor
stores, if it is not found in its configuration.  This has the benefits of
simplifying configuration.

Parameter inheritance implementation can be grouped into the following change
lists:
1. adding a getStoreConf() which returns a StoreConf object for Store().
2. adding a Store* parent parameter to Store creation such that we can trace
ancestor stores from with any store.  parent store can be obtained through
Store::getParent() call.
3. adding a Store* store to StoreConf
4. change StoreConf::getString implementation such that if a parameter is not
found in the current StoreConf object, it uses StoreConf::store parameter to get
to the Store this configuration is attached to.  And then use the getParent()
call to get its parent store.  And the parent store's StoreConf is retrieved
with getStoreConf() call.  Check with parameter is specified in the parent
store's StoreConf.  If not found, using the same logic to keep searching until
the parent store is NULL.
5. placing a global StoreConf inside g_handler.  If a parameter is not found
within the Store ancestor chain, then search the global StoreConf as the last
resort.
6. "type" parameter is not inheritable.

The second purpose of this diff, actually the main reason, is to use the
parameter inheritance to simply configuration of a large number of buffered
bucket store.  E.g. for a bucket store consists of 1000 buckets, current
configuration runs over 20k+ lines.  Most of those lines are identical.  With
parameter inheritance, we can place most of the configuration parameter in the
parent bucket store configuration instead of placing them under each buffer
store.

As such:
1. we extends <bucket> configuration under bucket store to allow buffer store. 
Majority of buffer store's configuration can be placed under parent bucket
store's configuration section to avoid repetition.  file_path parameter will be
overwritten with bucket id info for each buffer store within a bucket store,
during creation.

With this change, we can reduce the aforementioned 20k+ line configuration to
less than 20 lines.

Test Plan:
tested with testsuite.php.  Also added paramtest to testsuite to test parameter
inheritance and simplified buffer store creation under a bucket store.

DiffCamp Revision: 105894
Reviewed By: agiardullo
Commenters: pkhemani, zshao, groys
CC: agiardullo, zshao, jsong, pkhemani, groys
Revert Plan:
OK



git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@26470 2248de34-8caa-4a3c-bc55-5e52d9d7b73a