Initial import of hack generator

Summary:

Test Plan:

Blame Rev:
Two preprocessing-related fixes

Summary: Facebook: #import and #if inside macro expansion are not standard C things. GCC accepts them but we needn't.

Test Plan: part of the warp testing

Reviewed By: davejwatson@fb.com

FB internal diff: D1227580

@override-unit-failures
Add Krb5Principal::operator== overload, and Krb5Keytab iterator

Summary: The keytab iterator works like the ccache one does.

Test Plan: run regression tests

Reviewed By: andrei.bajenov@fb.com

FB internal diff: D1227645
Check for client in keytab before getting initial credentials

Summary:
If the keytab doesn't contain the client, raise an exception.
This will result in the same external behavior as before, but without
sending an unnecessary request to the KDC.

Test Plan: run regression tests

Reviewed By: andrei.bajenov@fb.com

FB internal diff: D1227650
Remove race condition in thread startup in TEventJobQueue

Summary:

In TEventJobQueue::run(), we were setting ##thread_## to the current
thread, and in TEventJobQueue::join(), we are using ##thread_##.  So
code that looks like this has a terrible race condition:

auto thread = std::shared_ptr<JobThread>(new JobThread(this));
threadFactory->newThread(thread)->start();
thread->join();

I changed it so that inside ##newThread()##, when the factory sets the
Thread of its runnable, the JobThread grabs its copy then.

This is only dangerous in the case that ##start()## fails and throws
an exception, but in that case, TEventJobQueue::shutdown() will crash
anyway because some of the elements of ##threads_## will be null.

If we don't want to intercept Runnable::thread(), then we will have to
change JobThread::run() to notify its parent after it sets thread_,
and have TEventJobQueue::init() wait for all its threads to have fully
started before returning.

Test Plan:

I added a test that failed accessing a null about one out of three
times.  It now never fails.

Reviewed By: davejwatson@fb.com

FB internal diff: D1228774

Blame Rev:
Fixing off-by-one bug in FrozenHashMap

Summary: One of the optimizations I added late actually broke certain lookup cases.

Test Plan: Error checking in benchmarks

Reviewed By: marcelo.juchem@fb.com

FB internal diff: D1225842

Blame Revision: D1163100
repro for union list TCompact deserialization bug

Summary:
Hey thrift team, I surfaced this bug where unions getting deserialized out of a list are set to empty.

This is a repro of the bug. Might not be minimal.

Somehow the deserializer extracts a T_STOP TType when reading the union type field, resulting in __clear being
called and an empty union returned. I believe this only happens on the second element or further.

@tgeorgiou looked at the java deserialization code and noticed the difference of an extra readFieldBegin call.
I added that call in a similar spot (included in this patch). It eats the T_STOP and fixes the problem. Disclaimer: I have no idea
whether this fix is appropriate or if it breaks other cases, but I figured it might be a helpful lead so I included it.

Remove the fix to see the breakage.

Test Plan: fbmake runtests

Reviewed By: davejwatson@fb.com

FB internal diff: D1217806
Py CppServer

Summary: Perf tester for py TCppServer.

Test Plan:
fbconfig -r thrift/perf; fbmake opt
_bin/thrift/perf/py/TestServer.par -s TCppServer
_bin/thrift/perf/cpp/loadgen --weight_noop=1 --num_threads=100 -header

QPS     Latency PCT Latency  Tot. QPS  Tot. Latency Tot. PCT Latency
1700 56550/30225      107949      1700   56550/30225           107949

_bin/thrift/perf/py/TestServer.par -s TNonblockingServer
_bin/thrift/perf/cpp/loadgen --weight_noop=1 --num_threads=100
QPS     Latency PCT Latency  Tot. QPS  Tot. Latency Tot. PCT Latency |
2200  45617/1419       85421      2216   44900/31468            84074

Due to GIL neither scale with --num_workers.  It seems TGEventServer perf tester isn't working at the moment(?), but it used to scale since it used separate processes.

QPS difference between TCppServer and TNonblocking is probably due to framed vs. header transport, since it looks like some of the header parsing is done in Python for TCppServer

Reviewed By: mhorowitz@fb.com

FB internal diff: D1224228
Also set cleitnt imeout for twoway request

Summary: Set for oneway, also call for twoway.  Looks like a refactoring oversight

Test Plan: set breakpoint, verified maybeSetTimeout was called.

Reviewed By: haijunz@fb.com

FB internal diff: D1229547

Blame Revision: D1015779
Fix logging bug

Summary: Looks like a copypasta bug, prints the wrong variable

Test Plan:
From D1224024,
fbconfig -r thrift/perf; fbmake dbg
_bin/thrift/perf/d/NonblockingServer --num_threads=4
prints 4 and not 1

Reviewed By: simpkins@fb.com

FB internal diff: D1224054
Perf tester

Summary: Adds a perf tester for d NonblockingServer

Test Plan:
fbconfig -r thrift/perf; fbmake opt

IO thread mode - pure async
_bin/thrift/perf/d/Nonblocking --num_io_threads=4
_bin/thrift/perf/cpp/loadgen --weight_noop=1 -async -num_threads=100
QPS     Latency PCT Latency  Tot. QPS  Tot. Latency Tot. PCT Latency
135338   3969/2440        8255    134468     3994/2448             8227

tm_async - Async, but ops don't block IO threads.
_bin/thrift/perf/cpp/ThriftServer -num_threads=4
_bin/thrift/perf/cpp/loadgen --weight_noop=1 -async -num_threads=100 -cpp2
QPS     Latency PCT Latency  Tot. QPS  Tot. Latency Tot. PCT Latency
319500   3113/3121        7835    319467     3113/3121             7835 |

Task queue mode
_bin/thrift/perf/d/NonblockingServer -num_threads=4
_bin/thrift/perf/cpp/loadgen --weight_noop=1 -async -num_threads=100
QPS     Latency PCT Latency  Tot. QPS  Tot. Latency Tot. PCT Latency
24249 22410/11865       43571     24253   22410/11865            43571

Conclusions: task queue mode in D Nonblocking server is almost unusable for prod: bad lock contention.  async mode is reasonable, but still the C++ version is almost 2.5x faster (due to IOBuf pipelining probably).  It is definitely worth the time to provide bindings to the C++ server

Reviewed By: alandau@fb.com

FB internal diff: D1224024
Support float type for Haskell in thrift

Summary: Support float type for Haskell in thrift

Test Plan: compile and run test/hs/{Server.Client}.hs

Reviewed By: alandau@fb.com

FB internal diff: D1226720
allow changing name of thrift worker thread

Summary: title

Test Plan: unit tests using this feature pass. Any more testing suggestions are welcome

Reviewed By: dcapel@fb.com

FB internal diff: D1223764
More verbose error messages.

Summary: More verbose error messages.

Test Plan: 1. Compiled, saw the message.

Reviewed By: mhorowitz@fb.com

FB internal diff: D1232847
borrow in protocols

Summary: Perf optimization, same as most other languages.  Saves a copy when the backing transport supports it

Test Plan:
fbconfig -r thrift/perf; fbmake opt
_bin/thrift/perf/d/Nonblocking
_bin/thrift/perf/cpp/loadgen --weight_noop=1  -num_threads=4 -async -async_ops=10

Used perf top, found about 10% perf improvement from saving the array copy after this fix.

Reviewed By: alandau@fb.com

FB internal diff: D1231486
Less aggressive thrift krb5 cc manager error logs

Summary: If an error message is equal to the last logged, it will be ignored.

Test Plan: 1. Compiled and saw.

Reviewed By: andrei.bajenov@fb.com

FB internal diff: D1233196
Give a name to the threadmanager threads if using service framework and no other name is provided

Summary: as title

Test Plan: ran test server and saw the names with top

Reviewed By: andrewcox@fb.com

FB internal diff: D1234916

@override-unit-failures
Kill common/base/Deprecation.h

Summary:
Expanded some old macros.

Facebook: This is part of a larger effort to clean up common/base

Deprecation.h contains some macros that are disabled on old versions of
the compiler. Since our lowest gcc version is 4.7 and clang supports
these options as well, the macros are now bitrot.

This diff:

- removes all #includes of Deprecation.h
- replaces all FBCODE_DEPRECATED macros with __attribute__((deprecated))

Codemods:

codemod --extensions=h,cpp 'FBCODE_DEPRECATED' '__attribute__((deprecated))'

codemod --extensions=h,cpp '(^ *)FBCODE_DISABLE_DEPRECATED_WARNING\(([^()]*)\)' '\1// \2\n\1#pragma GCC diagnostic push\n\1#pragma GCC diagnostic ignored "-Wdeprecated-declarations"'

codemod --extensions=h,cpp '(^ *)FBCODE_RESTORE_DEPRECATED_WARNING\(\)' '\1#pragma GCC diagnostic pop'

Note that the *_46 class of macros is unused in fbcode.

Test Plan: arc diff --preview

Reviewed By: andrewjcg@fb.com

FB internal diff: D1230537
ServiceFramework: don't override thread pool name if user provided one

Summary:

Currently we just set it regardless of whether `ThriftServer::setPoolThreadName` was called or not. Not cool.

@override-unit-failures

Test Plan: Used it on unicorn servers.

Reviewed By: marccelani@fb.com

FB internal diff: D1237775
recursive structs

Summary: cpp2 version of D799611.

Test Plan: Updated test to build cpp2.

Reviewed By: jeffp@fb.com

FB internal diff: D1227064