Typically, you can guess who owns a fence by seeing where the rails are, with the fence typically facing away from their property so that their neighbor gets the 'good' side of the fence. This is the most secure way of facing fencing so there are no rails for anyone to use to climb into your garden.