Fixed stack imbalance and other issues in test.data.table

rbindlist.c had an issue with UNPROTECT. Fixed now and the tests are okay. Hope this fixes it.