Revert "include X-Forwarded-Port for non proxyprotocol connections"

This reverts commit b8f414490eaa11fe4eeb9ec6378ab7395f72464b.