Revert "Wrap accesses to the fd_sets in struct fdtable"

This reverts commit b8788928d8ab7ff317979883aac2eca01ff66d62.

Reverting because this commit is not needed for the 3.4 android binder to work.