Fix PortNotFound exception during sync_routers

This trace is observed when an L3 agent invokes
sync_routers right about the same time a port
interface is removed from a router.

Related-bug: #1355409

Change-Id: I825b25080cbf054462318fc01248692b9e0e4ecb
Merge "Fix PortNotFound exception during sync_routers"