Finally it seems to work when I cahnged the IP and removed it form
~/.ssh/known_hosts - seems some way the auth was cached.

Follow up using:
https://github.com/jmdeldin/box/blob/master/ansible/setup.yml
Fix it for real.