Deny passwd access to infra, keys only