hpcc-systems/HPCC-Platform — Issue #1667: Roxie write access to DFS is inconsistent
Author: rengolin
Created: 2012-02-23 15:11:40+00:00
State: open

Span out from #886, when executing the following query with a local dali server, we get dead-locks, since Roxie is using separate packages to cache the same file (one on main thread and another on an asynchronous thread):

rec := record
string line;
end;

raw := dataset([{'foo'},{'bar'},{'baz'}], rec);
output(raw,,'~regress::roxie::fileaccess',CSV,OVERWRITE);

ds := dataset('~regress::roxie::fileaccess', rec, CSV);
output(ds);

First access, object 1, caches

18 CRoxiePackage::lookupFile() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:369 0x00007ffff77b318f
17 CRoxiePackage::lookupFileName() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:529 0x00007ffff77b3f95
16 CRoxieServerDiskReadActivityFactory::CRoxieServerDiskReadActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:20517 0x00007ffff76fce73
15 createRoxieServerDiskReadActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:20575 0x00007ffff769cfe7
14 CQueryFactory::createActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:264 0x00007ffff7793472
13 CQueryFactory::loadNode() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:630 0x00007ffff77957b5
12 CQueryFactory::loadSubgraph() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:646 0x00007ffff77958d1
11 CRoxieServerQueryFactory::loadGraph() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1159 0x00007ffff7798961
10 CQueryFactory::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:816 0x00007ffff7796ba5
9 createServerQueryFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1271 0x00007ffff7791fa6
8 CRoxieServerQuerySetManager::loadQueryFromDll() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:939 0x00007ffff77b5e6a
7 CRoxieQuerySetManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:794 0x00007ffff77b5045
6 CStandaloneQueryPackageManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1291 0x00007ffff77b7a8a
5 CRoxiePackageSetManager::loadStandaloneQuery() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:2335 0x00007ffff77bd8c1
4 CRoxiePackageSetManager::reload() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1408 0x00007ffff77b8785
3 CRoxiePackageSetManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1317 0x00007ffff77b81c1
2 start_query() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdmain.cpp:932 0x00007ffff778fa40
1 main() /home/rengolin/workspace/hpcc/src/testing/ecl/a.out.cpp:258 0x000000000040173b

Second access, object 1, uses cache

19 CRoxiePackage::lookupFile() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:369 0x00007ffff77b318f
18 CRoxiePackage::lookupFileName() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:529 0x00007ffff77b3f95
17 CRoxieDiskBaseActivityFactory::CRoxieDiskBaseActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdactivities.cpp:924 0x00007ffff767cc8b
16 CRoxieCsvReadActivityFactory::CRoxieCsvReadActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdactivities.cpp:1121 0x00007ffff767e237
15 createRoxieCsvReadActivityFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdactivities.cpp:1625 0x00007ffff7672465
14 CSlaveQueryFactory::loadSlaveNode() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1349 0x00007ffff77993e9
13 CSlaveQueryFactory::loadOuterSubgraph() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1452 0x00007ffff7799b0b
12 CSlaveQueryFactory::loadGraph() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1495 0x00007ffff7799d89
11 CQueryFactory::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:816 0x00007ffff7796ba5
10 createSlaveQueryFactory() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdquery.cpp:1519 0x00007ffff77921b1
9 CRoxieSlaveQuerySetManager::loadQueryFromDll() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:963 0x00007ffff77b5f3e
8 CRoxieQuerySetManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:794 0x00007ffff77b5045
7 CRoxieSlaveQuerySetManagerSet::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1011 0x00007ffff77b6410
6 CStandaloneQueryPackageManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1292 0x00007ffff77b7ace
5 CRoxiePackageSetManager::loadStandaloneQuery() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:2335 0x00007ffff77bd8c1
4 CRoxiePackageSetManager::reload() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1408 0x00007ffff77b8785
3 CRoxiePackageSetManager::load() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:1317 0x00007ffff77b81c1
2 start_query() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdmain.cpp:932 0x00007ffff778fa40
1 main() /home/rengolin/workspace/hpcc/src/testing/ecl/a.out.cpp:258 0x000000000040173b

Third access, object 2, separate thread, read-locks the same file twice

13 CRoxiePackage::lookupFile() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:369 0x00007ffff77b318f
12 CRoxiePackage::createFileName() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdstate.cpp:544 0x00007ffff77b40ec
11 CRoxieServerContext::createLFN() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:29892 0x00007ffff772ddc0
10 CRoxieServerDiskWriteActivity::resolve() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:10583 0x00007ffff76d0d78
9 CRoxieServerDiskWriteActivity::start() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:10613 0x00007ffff76d13be
8 CRoxieServerInternalSinkActivity::execute() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:2201 0x00007ffff76af552
7 CActivityGraph::casyncfor::Do() /home/rengolin/workspace/hpcc/src/roxie/ccd/ccdserver.cpp:26822 0x00007ffff771c65c
6 CAsyncFor::cdothread::run() /home/rengolin/workspace/hpcc/src/system/jlib/jthread.cpp:648 0x00007ffff4b9ca6c
5 Thread::begin() /home/rengolin/workspace/hpcc/src/system/jlib/jthread.cpp:247 0x00007ffff4b9b5e0
4 Thread::_threadmain() /home/rengolin/workspace/hpcc/src/system/jlib/jthread.cpp:98 0x00007ffff4b9b0b2
3 start_thread() 0x00007ffff3d28d8c
2 clone() 0x00007ffff697804d
1 0x0000000000000000

Later on, on ```CRoxiePackage::createFileName()```, the ```resolved->remove();``` line will lock, since it tries to acquire a write-lock to delete the file, but the cached version in the main thread's package is read-locked.

There is no correct answer to this, Roxie locks all files for the whole duration of the work unit by design. The question is, if that's the design, why perform deletion/writes in Roxie, especially on other threads that is not main? Shouldn't we give an error sooner?