Remove ghost networking

Deps on #5234