genodelabs/genode — pull request #717 by myrkr (2013-04-24 10:22:29+00:00)
Title: linux: Mark a local fd as closed after mmap

Please check if this alternate approach to the problem with lx_sendmsg to a closed FD looks good to you. I already tested in for quite some time in our software.

The method Platform_env_base::Rm_session_mmap::_map_local() calls
lx_close on the file descriptor it maps. This can be fatal, the
file descriptor is still stored within the dataspace and a call to
Ram_session_component::_revoke_ram_ds() will run lx_close on that
file descriptor, even if the file descriptor is used for some other
purposes now.

In our scenario this caused a lx_reply call to fail with:

lx_sendmsg failed with -9 in lx_reply()

because the file descriptor returned by lx_wait was closed before
Ipc_server::_reply() method was called.

To avoid it, I added a _dataspace_closed() method which will mark
the closed file descriptor as closed. There are two cases, the
dataspace is local or remote.

In the remote case nothing happens (because the lx_close call has
no effect out-side of the local process). This is always the case
for a non-core process, this is why the implementation within
base-linux/src/base/env/platform_env.cc is empty.

In the local case the fd(-1) is performed, this is the value the
Ram_session_component::_revoke_ram_ds() method checks for when
deciding if the file descriptor is to be closed.