fix incorrect recover for hidden debug output of basic auth

* While doing debug log output, basic auth info will be temporarily
  replaced with "<hidden>" for protecting sensitive info.
* The value is not correctly recovered when the base64 param in original
  request is empty in a bad request.

Change-Id: I07a6e1578439d16ffd4e6ee3a9eae73dfdfca0f9
Signed-off-by: Kevin Cai <kevinc@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57155
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
add helper classes for unit testing

* add HijackConfigVar for changing the config var temporarily.
* add TakesAtLeast for asserting an test takes at least certain time span.
* refactor TimeConstraint a little bit to reduce memory usage

Change-Id: Ib6ac41ad855e476cbe1725a040e5a8ccf4752466
Signed-off-by: Kefu Chai <chaik@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57133
Reviewed-by: Kevin Cai <kevinc@mozy.com>
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
Added logging for turning off throttling

refs #102632

Signed-off-by: russs <russs@mozy.com>
Change-Id: I070e0244e4c8bf857c1784d538636fdf5b2ee2c2
Reviewed-on: https://gerrit.dechocorp.com/56757
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kefu Chai <chaik@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
Reviewed-by: Pancho Avila <panchoa@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
Disable 64 to 32 bit warnings on Mac

Disable 20 warnings occuring in the xcode
compilation of mordor and mordorprotobuf target.

Practically all the warnings are coming from boost and
protocol buffer library code in places where long and int are interchanged.

If there is a better way to fix these please, or if they are really valid
then let me know of a better fix.

Example log with the errors:

http://build.dechocorp.com/job/sync-release/dist=seblddev-mac/3557/consoleFull

Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Change-Id: I5b2adfcc22625e44264d0776ed6e2eee3995d1df
Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57503
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kefu Chai <chaik@mozy.com>
compiles with c++11 and boost 1.54, tested using clang 5.0

* as c++11 provides `nullptr`, if the compiler does implement it, boost tends
  not to provide the implicit bool() for smart_ptr, so we need to call `bool()`
  explicitly.

Change-Id: I4653ec1b57ed1ac045a7bcd41af72922949986f0
Signed-off-by: Kefu Chai <chaik@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57539
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
Fix ZeroStream->seek to actually be called

as an inline virtual function, seek was not being called, but
rather the base class Mordor::Stream::seek was being called,
which simply threw a MORDOR_NOTREACHED exception.

We either need to have supportsSeek be false, or put the
implementation in the .cpp.  I chose the second option.

Change-Id: I36219477a7c69979e6653293c49e5873aaa3403a
Signed-Off-by: Jeff Tulley <jefft@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57850
move ZeroStream into Mordor namespace

Change-Id: I673034f45796ea31fa2d517e0f62bada828df7be
Signed-off-by: Kefu Chai <chaik@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/57880
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
Reviewed-by: Eric Wang <leiw@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
add protobuf to vcxproj

For some reason, this wasn't already there.

Signed-off-by: Jeff Tulley <jefft@mozy.com>
Change-Id: I3409c5e37f752d80801a18613613d7b06518e991
Reviewed-on: https://gerrit.dechocorp.com/57946
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Bandwidth throttling works with multiple connections per host

refs #111665

The change in throttle.cpp was changing max to min when we should
"Never sleep for longer than a tenth of a second"

getActiveConnections() returns the number of connections that are
active and have pending requests.

Testing done: tested bandwidth throttling with 1-3 connections per
host. The total bandwidth used was in line with the throttle.


Signed-off-by: russs <russs@mozy.com>
Change-Id: I41531810e84abd898a52e330134e06a07df8c3c0
Reviewed-on: https://gerrit.dechocorp.com/59056
Reviewed-by: Pancho Avila <panchoa@mozy.com>
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
improve performance of ThrottleStream a little bit

* refs #111665
* save a syscall in most cases

Change-Id: Ibb0fc5991e16260b440b9493ad5a0ba9eabc727b
Signed-off-by: Kefu Chai <chaik@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59177
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Russ Simons <russs@mozy.com>
Add comments to IOManager (Windows implementation)

Add some more statements explaining how the
IOManager works, hopefully shedding some light on
the mysterious tickle, the add that is actually a
subtraction and the stopping() check that returns a time!

Change-Id: I17ac1af804b74d5fa297a1221a582cc2916e0e6a
Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59108
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kefu Chai <chaik@mozy.com>
Reviewed-by: Xiangdong Lin <xiangdongl@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
Reviewed-by: Russ Simons <russs@mozy.com>
Add IOCP error threshold.

refs #111932
refs #112528

- Allow a number of IOCP errors to ignore when calling
PostQueuedCompletionStatus in the tickle() method (the default value is 0).

- If we reach the number of specified errors in the specified timeframe we
will stop ignoring the errors.

- The Sync product has experienced some "Insufficient system resources" errors
returned by PostQueuedCompletionStatus, possibly indicating the the IOCP
queue is full. There is a possibility that this is a rare error that we can
recover from by ignoring it and continuing on.

- Be careful when using this method as it is possible for the corresponding
GetQueuedCompletionStatusEx call in idle() to wait infinitely which could cause
deadlock if the PostQueuedCompletionStatus failure is ignored. (Sync makes use
of many Timers which should ensure that GetQueuedCompletionStatusEx will not
ever wait infinitely.)

Change-Id: I586ba09e92a6112645740bf5b2f56e63c2f6c5cb
Signed-off-by: Mat Bess<mathew.bess@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59456
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Async Socket::cancelIo call may cause coredump

* If Socket::cancelIo was registered to IOManager as
  timeout handler and then Socket object destroyed,
  it might cause coredump when Socket::cancelIo being
  called due to timeout, because the use of wild pointer

Change-Id: I6a0a8599aa6a8db457371c5e846204bcf562bb84
Signed-off-by: Bert Yuan <berty@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59481
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-by: Kefu Chai <chaik@mozy.com>
Fix for Windows XP

GetTickCount64 is not available on Windows XP so clients using mordor
would crash on startup.  Replace by TimeManager::now, which returns
microseconds instead of milliseconds.

refs #112976

Change-Id: Ifdbeec4bfd544d645d67875076496b569806aee1
Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59781
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Mat Bess <mathew.bess@mozy.com>
Minor fix for Windows iomanager logging

The logging of successful iomanager calls was highly misleading
because it shows the lasterror.  When the call succeeded the
error has nothing to do with the call being logged, it is just
whatever most recent error happened on the thread.

For example, with this fix the output will look like this:

GetQueuedCompletionStatusEx(000001D8, 997): 1, (1) (0, "The operation completed successfully.")

Instead of:

GetQueuedCompletionStatusEx(0000028C, 32): 1, (1) (258, "The wait operation timed out.")

Tip: the error string is printed via this operator:
std::ostream &operator <<(std::ostream &os, error_t error)

Also add a few more comments to explain the alternative
registerEvent() signature and relationship between registerFile
and registerEvent().

Change-Id: If82f0d65fbf3664042f83c9014ee885cc9fc804b
Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/59827
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Mat Bess <mathew.bess@mozy.com>
add parallelism to parallel_do interface

* provide the ability to control the parallelism of parallel_do just
  like the parallel_foreach

Change-Id: I76799211dfc19bf9be1ab1e9904eb7fd643cc145
Signed-off-by: Kevin Cai <kevinc@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/60013
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Bert Yuan <berty@mozy.com>
Reviewed-by: Bo Shen <bos@mozy.com>
Reviewed-by: Andrew Skowronski <andrews@mozy.com>
refs #111616: update for boost 1.55

* add boost_system to resolve linking issues
* pass args to configure from build.sh

Change-Id: I0855528b9fdccdaa3f661abf67abf034b62f86b2
Signed-off-by: Xiangdong Lin <xiangdongl@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/60235
Reviewed-by: Kefu Chai <chaik@mozy.com>
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
closes #113805: add NDEBUG_PERF to disable perf sensitive assertion

* some perf data from FiberMutex::mutexPerformance
  (testcase running on a 8-core CPU DELL 2950 server)
  +-----------+----------------+-------------+-------+
  | repeatness| no NDEBUG_PERF | NDEBUG_PERF | diff  |
  +-----------+----------------+-------------+-------+
  |  10,000   |    3.24s       |   0.54s     |  600% |
  |  50,000   |  107.69s       |   2.60s     | 4142% |
  | 100,000   | 1075.00s       |   5.01s     |21457% |
  +-----------+----------------+-------------+-------+
* The test data shows the larger the size of waiting list, the worse
  the FibeMutex performance impacted by the ASSERT.

Change-Id: Ied0b4966c1f3fce7bf1ad452201693cea3f0f855
Signed-off-by: Kevin Cai <kevinc@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/60451
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kefu Chai <chaik@mozy.com>
Demonstrate problem with using FiberEvent in multiple thread scenario.

New test that shows the danger of using FiberEvent to try to
coordinate destruction.  The main conclusion is not to use
FiberEvent at all!  But for education purposes these test scenarios
are informative.

Note: earlier patch versions demontrate a temporary sleep
in the mordor code that forces the test to hit the race condition
more easily.  That is removed and the bad code is disabled so that this
can safely run.

refs #114320

Change-Id: Ibd741abef6b29b8d029698ab4ec65d1db88cebae
Signed-off-by: Andrew Skowronski <andrews@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/60812
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>
refs #112434, add toUtf16() on linux

* add unicode.cpp to run_test on Xcode project and Makefile.am
* add test case of `Unicode::toUtf16`
* fix the toUtf16(const char * str) on OSX

Change-Id: Iab05dba6a30d5dbeb477b8ea12aec3d5c76fcf73
Signed-off-by: Kefu Chai <chaik@mozy.com>
Reviewed-on: https://gerrit.dechocorp.com/61526
Reviewed-by: Jenkins <hudsonbuild@mozy.com>
Reviewed-by: Zhaoding Wu <zhaoding.wu@mozy.com>
Reviewed-by: Kevin Cai <kevinc@mozy.com>