opened
bblum
username_0
There's still a race that makes the ID wrapper hang waiting for landslide jobs after TIME_UP.
issue
created
bblum
username_0
It's not to do with particular jobs.

All I know is the last line output is ```[MESSAGING]     suspending time``` and it's resistant to kill -9.
comment
created
bblum
username_0
```[JOB 136] State space too big (32 brs elapsed, time rem 205, eta 2316) -- blocking```
is printed
but its status never changes from ```Running...```
comment
created
bblum
username_0
infodump

i was stumped for a long time until i noticed in the contents of the job struct, the state of blocking_cvar is clobbered. turns out it's never cond_init'ed. i'm running some tests with that fixed and hoping this never comes up again.

mortals were not meant to understand the sequence of events in which uninitialized reads in blocking_cvar corrupted the state of its neighbour done_cvar.
```
(gdb) thread 6
[Switching to thread 6 (Thread 0x7f7bb53ee700 (LWP 10521))]
(gdb) bt
#0  0x00007f7bb8fc4f7d in __lll_lock_wait ()
#1  0x00007f7bb8fc74ec in _L_cond_lock_792 ()
#2  0x00007f7bb8fc73c8 in __pthread_mutex_cond_lock ()
#3  0x00007f7bb8fc2795 in pthread_cond_wait@@GLIBC_2.3.2 ()
#4  0x00000000004092e3 in wait_on_job (j=0x7f7b98002110) at job.c:277
#5  0x0000000000417b02 in process_work (j=0x7f7b98002110, was_blocked=false) at work.c:261
#6  0x0000000000417e18 in workqueue_thread (arg=0x6) at work.c:290
#7  0x00007f7bb8fbedf5 in start_thread ()
#8  0x00007f7bb8cec1ad in clone ()

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f7bb33ea700 (LWP 22078))]
(gdb) bt
#0  0x00007f7bb8fc4f7d in __lll_lock_wait ()
#1  0x00007f7bb8fc27e2 in pthread_cond_wait@@GLIBC_2.3.2 ()
#2  0x0000000000408fd9 in job_block (j=0x7f7b98002110) at job.c:251
#3  0x000000000040a719 in handle_estimate (...) at messaging.c:214
#4  0x000000000040ba27 in talk_to_child (...) at messaging.c:362
#5  0x000000000040890c in run_job (arg=0x7f7b98002110) at job.c:198
#6  0x00007f7bb8fbedf5 in start_thread ()
#7  0x00007f7bb8cec1ad in clone ()

(gdb) print *(struct job *)0x7f7b98002110
$1 = {
config = 0x7f7b98002af0,
id = 117,
generation = 3,
should_reproduce = true,
config_file = {fd = 68,  filename = "../pebsim/config-id.landslide.Bk1nx8"},
log_stdout = { fd = 69, filename = "ls-setup.log.gTNRPI"},
log_stderr = {fd = 70,  filename = "ls-output.log.djBt8i"},
stats_lock = {__data = {__lock = 0,
      __nr_readers = 0, __readers_wakeup = 0, __writer_wakeup = 0,
      __nr_readers_queued = 0, __nr_writers_queued = 0, __writer = 0, __shared = 0,
      __pad1 = 0, __pad2 = 0, __flags = 0}, __size = '\000' <repeats 55 times>,
    __align = 0},
elapsed_branches = 32,
estimate_proportion = 0.0146484375,
estimate_elapsed = {secs = 38, mins = 0, hours = 0, days = 0, years = 0, inf = false}
estimate_eta = {secs = 43, mins = 37, hours = 0, days = 0, years = 0, inf = false}
estimate_eta_numeric = 2263454568,
cancelled = false,
complete = false,
timed_out = false,
kill_job = false,
log_filename = "ls-output.log.djBt8i",
trace_filename = 0x0,
status = JOB_BLOCKED,
done_cvar = {__data = {__lock = 0, __futex = 2, __total_seq = 1,
      __wakeup_seq = 1, __woken_seq = 1, __mutex = 0x7f7b980022a0, __nwaiters = 0,
      __broadcast_seq = 1},
    __size = "\000\000\000\000\002\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\240\"\000\230{\177\000\000\000\000\000\000\001\000\000", __align = 8589934592},
blocking_cvar = {__data = {__lock = 2,  __futex = 32635,
      __total_seq = 140168740738608, __wakeup_seq = 140168740739104,
      __woken_seq = 140168740739600, __mutex = 0x7f7b980022a0, __nwaiters = 2483035632,
      __broadcast_seq = 32635},
    __size = "\002\000\000\000{\177\000\000\060\026\000\224{\177\000\000 \030\000\224{\177\000\000\020\032\000\224{\177\000\000\240\"\000\230{\177\000\000\360\035\000\224{\177\000",
    __align = 140166257704962},
lifecycle_lock = {__data = {__lock = 2, __count = 0,
      __owner = 22078, __nusers = 2, __kind = 0, __spins = 0, __list = {__prev = 0x0,
        __next = 0x0}},
    __size = "\002\000\000\000\000\000\000\000>V\000\000\002", '\000' <repeats 26 times>,
    __align = 2}}
```
comment
created
bblum
username_0
Figured out why this broke the way it did.

Workqueue thread (6) has been broadcasted, but is waiting for ```lifecycle_lock``` to be released before it can exit ```cond_wait```.

Job thread (2) is holding ```lifecycle_lock```, getting ready to AU&D on ```blocking_cvar```. But before it can let go of ```lifecycle_lock```, it needs to get the internal ```blocking_cvar``` lock.

Because the lock is uninitialized, it randomly appears to be already locked. No thread holds it so the job thread (2) waits forever. Hence ```lifecycle_lock``` is never dropped and WQ thread 6 is stuck at the very end of ```cond_wait``` forever.
comment
closed
bblum
username_0
issue