Revert network/port authz changes

We will be enforcing this with policy.json and nova-side
changes instead.

RM10027 RM9305 RM9376 RM9709