Revert "Revert "epoll: fix up lock ordering issues""

This reverts commit 408a982f3f2fbd4a3aaddafb60233909eb6b7b83.

I never see the build issue mentioned by Nadav. I tested this patch with
tomcat + nio connector which uses epoll intensively and I did not see any
issue.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
epoll: Do not remove _file map twice

The problem is that when epoll_wait removes the key from _file map, it
does not remove the key from _epoll map, so when epoll_ctrl(DEL) tries
to remove _file_map again, we hit the assert.

This fixes the following.

After:
    Revert "Revert "epoll: fix up lock ordering issues""

I saw:

   $ sudo scripts/run.py -c1 -nv -m4g  -e "/tests/tst-epoll.so"
   ...

   PASS: epoll_wait
   PASS: write
   PASS: epoll_wait
   PASS: epoll_ctl MOD
   PASS: epoll_wait
   PASS: epoll_ctl ADD
   PASS: errno == EEXIST
   Assertion failed: i != f_epolls->end() (/home/asias/src/cloudius-systems/osv/fs/vfs/kern_descrip.cc: epoll_del: 249)

   [backtrace]
   0x0000000000223648 <__assert_fail+24>
   0x000000000041743a <file::epoll_del(epoll_ptr)+186>
   0x00000000003f7682 <epoll_ctl+786>
   0x0000100000c032db <???+12595931>
   0x000000000040510a <osv::application::run_main(std::string, int, char**)+266>
   0x0000000000405847 <osv::application::run_main()+551>
   0x000000000020bbd1 <osv::application::main()+97>
   0x0000000000406698 <???+4220568>
   0x0000000000433745 <???+4405061>
   0x00000000003e38aa <thread_main_c+26>
   0x0000000000389c75 <???+3710069>

   (gdb) bt
   #0  0x00000000003f8e92 in cli_hlt () at /home/asias/src/cloudius-systems/osv/arch/x64/processor.hh:242
   #1  halt_no_interrupts () at /home/asias/src/cloudius-systems/osv/arch/x64/arch.hh:49
   #2  osv::halt () at /home/asias/src/cloudius-systems/osv/core/power.cc:34
   #3  0x000000000022360f in abort (fmt=fmt@entry=0x603b60 "Assertion failed: %s (%s: %s: %d)\n") at /home/asias/src/cloudius-systems/osv/runtime.cc:150
   #4  0x0000000000223649 in __assert_fail (expr=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>)
       at /home/asias/src/cloudius-systems/osv/runtime.cc:156
   #5  0x000000000041743b in file::epoll_del (this=<optimized out>, ep=...) at /home/asias/src/cloudius-systems/osv/fs/vfs/kern_descrip.cc:249
   #6  0x00000000003f7683 in del (key=..., this=0xffff800001746040) at /home/asias/src/cloudius-systems/osv/core/epoll.cc:96
   #7  epoll_ctl (epfd=<optimized out>, op=<optimized out>, fd=<optimized out>, event=0x2000002f9d50) at /home/asias/src/cloudius-systems/osv/core/epoll.cc:283
   #8  0x0000100000c032dc in test_epolloneshot () at /home/asias/src/cloudius-systems/osv/tests/tst-epoll.cc:78
   #9  main (ac=<optimized out>, av=<optimized out>) at /home/asias/src/cloudius-systems/osv/tests/tst-epoll.cc:231
   #10 0x000000000040510b in osv::application::run_main (this=this@entry=0xffffa00001561f18, path="/tests/tst-epoll.so", argc=argc@entry=1,
       argv=argv@entry=0xffffa00002994760) at /home/asias/src/cloudius-systems/osv/core/app.cc:182
   #11 0x0000000000405848 in osv::application::run_main (this=this@entry=0xffffa00001561f18) at /home/asias/src/cloudius-systems/osv/core/app.cc:197
   #12 0x000000000020bbd2 in osv::application::main (this=0xffffa00001561f18) at /home/asias/src/cloudius-systems/osv/core/app.cc:144
   #13 0x0000000000406699 in operator() (__closure=0x0, app=<optimized out>) at /home/asias/src/cloudius-systems/osv/core/app.cc:104
   #14 osv::application::__lambda3::_FUN (app=<optimized out>) at /home/asias/src/cloudius-systems/osv/core/app.cc:106
   #15 0x0000000000433746 in operator() (__closure=0xffffa00003320600) at /home/asias/src/cloudius-systems/osv/libc/pthread.cc:97
   #16 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::__lambda7>::_M_invoke(const std::_Any_data &) (__functor=...) at /home/asias/src/cloudius-systems/osv/external/x64/gcc.bin/usr/include/c++/4.8.2/functional:2071
   #17 0x00000000003e38ab in main (this=0xffff80000354d050) at /home/asias/src/cloudius-systems/osv/core/sched.cc:940
   #18 sched::thread_main_c (t=0xffff80000354d050) at /home/asias/src/cloudius-systems/osv/arch/x64/arch-switch.hh:137
   #19 0x0000000000389c76 in thread_main () at /home/asias/src/cloudius-systems/osv/arch/x64/entry.S:113

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
epoll: Get rid of registered_epoll

After commit 3082e79f14339be0737fed55f2d3ab3f8c260983 (epoll: a real
epoll), our epoll is not on top of poll any more. last_poll_wake_count
is not used. Remove the leftover.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
poll: Remove poll_wake_count

After commit 3082e79f14339be0737fed55f2d3ab3f8c260983 (epoll: a real
epoll), our epoll is not on top of poll any more. We can remove hacks
for implementing epoll() over poll().

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
epoll: Drop leftover comments

epoll is not implemented on top of poll any more. Remove incorrect
comments.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>