doh, --relative was the option I needed for rsync
push and pull both working, yay!